Skip to content

GPT-OSS Puzzle 88B Heavy enablement

Publication redaction: Operator-specific GPU UUIDs in linked raw evidence were replaced with stable labels. Hardware class, measurements, and event ordering are unchanged.

Captured: 2026-07-18
Host: Primary Node; NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 96 GB
Checkpoint: nvidia/gpt-oss-puzzle-88B at 9c0e0746a0d2218b28cc7b2cb3ce4e1a2f50fdb2
Engine source: Anvil vLLM commits 3fbe020fb04afb27885196dabd1a72392074d49c and 485463b3498ed3ffcf0c8fcb52c1670a21be5d82
Result: promote Puzzle 88B to the local default Heavy tier; retain official Gemma 4 12B IT QAT W4A16 as the immediate managed rollback

This record is the promotion follow-up to the July 17 local qualification. That run established the model architecture, weighted KV grouping, real checkpoint load, ordinary GPT-OSS control path, deterministic Chat, Responses, prefix-cache boundary crossings, a 130,696-prompt-token near-limit retrieval, five-request capacity, and the full official GPT-OSS GPQA sampler. Its remaining promotion blocker was the prompt-specific record_weather_zip tool-parser crash addressed here.

Why the serving branch is required

The model-support commit adds the Puzzle architecture from the work proposed in upstream vLLM PR #38135. The pinned checkpoint also needs a narrow generation-config compatibility fix. Its generation_config.json identifies tokens 200002 and 199999 as EOS but omits Harmony token 200012, <|call|>. A valid tool call could therefore continue into another assistant turn. The OpenAI tool parser then received token 173781 (assistant) where Harmony requires token 200006 (<|start|>) and returned HTTP 500. Adding 200012 as a request-level stop token made the same tool case pass 10/10, isolating termination rather than model quality as the failure.

The production command now supplies:

--override-generation-config '{"eos_token_id":[200002,199999,200012]}'

The prior fork revision did not apply special-token fields from this full generation-config override. Commit 485463b3 merges the override in ModelConfig.try_get_generation_config() and adds a focused unit test for eos_token_id and stop_token_ids. This behavior is also proposed upstream in vLLM PR #45978; the fork commit is a local backport, not a competing upstream contribution. The broader GPT-OSS generation-config limitation is tracked in vLLM issue #28041.

Default Heavy shape

The heavy managed service and router tier now select gpt-oss-puzzle-88b. The reproducible shape is:

Setting Value
Image anvil-vllm:gpt-oss-puzzle-485463b3498ed3ffcf0c8fcb52c1670a21be5d82
Endpoint http://127.0.0.1:30002/v1
Context 131,072 tokens
Quantization checkpoint-native GPT-OSS MXFP4; Marlin MoE backend
KV cache FP8
Admission 8 sequences; 8,192 batched tokens
Tool path native Harmony template; OpenAI parser; automatic tool choice
Router default reasoning_effort=high
Immediate rollback heavy-gemma4-rollback (gemma4-12b-it-w4a16-ct)

The image and checkpoint revision are both immutable. Gemma 4 keeps its prior 256K recipe and evidence as the immediate rollback; ThinkingCap and GPT-OSS-120B remain second- and third-line rollbacks.

Functional isolation run

Before the expensive full-source image completed, a temporary overlay copied only vllm/config/model.py from commit 485463b3 onto the previously qualified Puzzle image. This isolated the generation-config merge behavior but is not promotion evidence and supports no performance claim. On the RTX PRO 6000 it passed:

  • health and model identity;
  • smoke and structured JSON;
  • a 120,000-token requested needle check (99,100 prompt tokens observed);
  • 20/20 shared-prefix tool calls with reasoning evidence;
  • the original parser regression 10/10 without request-level stop_token_ids;
  • Responses API completion and streaming SSE with [DONE];
  • a complete tool-call, tool-result, final-answer exchange.

The raw preflight is overlay-functional-preflight.json, and the explicitly non-promotable summary is overlay-compatibility-smoke.json.

Exact-image and router qualification

The final source build produced image sha256:470f7b7e39c4363696d5a79fd041d6a45253229a9ba1c055d089ddbdc0ed120c with both build and OCI revision labels set to 485463b3. The managed Heavy container resolved that exact image and reported vLLM 0.23.1rc1.dev1229+g485463b34. Startup resolved GptOssPuzzleForCausalLM, the pinned checkpoint revision, corrected EOS list, Marlin MXFP4, 38.72 GiB FP8 KV cache, and 2,819,653 KV tokens before health returned HTTP 200.

The exact image then passed the same full promotion preflight: coding smoke, structured JSON, 99,100-prompt-token retrieval, 20/20 shared-prefix tools, and required reasoning evidence. The original record_weather_zip regression passed 10/10 with HTTP 200, finish_reason=tool_calls, exact ZIP 98101, and no request-level stop_token_ids. Responses API, streaming SSE with [DONE], and a tool-result continuation also passed. See runtime-identity.json, promotion-functional-preflight.json, tool-call-regression.json, and exact-interface-smoke.json. Artifact hashes are retained in SHA256SUMS.

The router profile and config were written atomically, then the pinned local router image was started without model-service dependency changes. Its first start failed closed because the automatically selected ~/.anvil-serving/.env exists but has no router-token entry, shadowing the existing token in ~/.env. Restarting through Anvil Serving with the explicit existing env file restored auth without exposing or replacing the secret. The tailnet front door then reported Heavy ready and admitting, with expected and observed model both gpt-oss-puzzle-88b. An authenticated routed tool request returned the exact model identity and record_weather_zip(zip="98101"). See router-validation.json.

The checked-in promotion and rollback router profiles were validated against the deployed router image before any state change. The OpenClaw harness sync also passed in dry-run mode with the Heavy-backed intents at a 131,072-token context. No remote OpenClaw configuration was applied as part of this local Heavy change.

Live capacity and repeated quality benchmark

After the exact image and router were resident, the production Heavy endpoint was requalified and measured without restarting either service. The repeated preflight passed coding, JSON, a 99,100-observed-prompt-token retrieval, and 20/20 shared-prefix tool calls. All checks retained non-empty reasoning evidence with reasoning_effort=low.

The bounded 8K-context capacity probes used the repository benchmark harness:

Path Requests Concurrency TTFT p50 / p95 E2E p50 / p95 Aggregate output
Direct Heavy 10/10 1 0.393 / 0.956 s 0.473 / 1.035 s 3.85 tok/s
Direct Heavy 40/40 8 0.766 / 1.075 s 0.906 / 1.148 s 17.85 tok/s
Authenticated router planning 10/10 1 0.484 / 0.718 s 0.484 / 0.718 s 2.00 tok/s

These capacity prompts produced only 20, 86, and 10 output tokens respectively. They establish bounded request completion and latency under the declared load; they are not a controlled long-decode throughput result.

The protocol-v3 quality run used three attempts per deterministic check, 256 visible-answer tokens, 2,048 reasoning-headroom tokens, and verified reasoning_effort=low control. The 32,768- and 128,000-token context targets both passed; the latter consumed 127,916 prompt tokens and had 25.906-second TTFT. Session recall passed 3/3, the tool-call check passed 3/3, and timeout triage passed 3/3. The strict unified-diff check passed 2/3 because one response inserted a space after the diff marker. The harness therefore correctly marked the overall repeated quality gate failed at its 100% threshold.

This run closes the former parser failure: the pre-fix image returned HTTP 500 for the same tool suite 0/3, while the final image completed it 3/3. It does not establish a general quality win over Gemma 4. The live artifacts are live-preflight-20260718.json, live-capacity-c1-8k-r10-20260718.json, live-capacity-c8-8k-r40-20260718.json, live-router-capacity-c1-8k-r10-20260718.json, and live-quality-protocol-v3-r3-20260718.json.

Scope and caveats

  • This change proves local serving compatibility, bounded capacity, and the listed protocol features. It does not claim Puzzle has better quality or throughput than the prior Gemma Heavy; no controlled cross-model A/B was run, and the repeated strict quality suite retained one 2/3 deterministic failure.
  • FP8 KV cache trades some numerical precision for context capacity. All live gates use the same FP8-KV production shape.
  • The 120K gate requests approximately 120,000 characters; the tokenizer reported 99,100 prompt tokens. The served 131,072-token window is an engine identity claim for this rerun. The separate July 17 exact-image qualification retains the 130,696-prompt-token near-limit evidence.
  • The local fork commit is retained only until the equivalent upstream generation-config behavior is available in the selected vLLM base.

Reproduction surface

The complete pinned build, cache, serve, benchmark, router, and rollback procedure is the GPT-OSS Puzzle 88B recipe.

Serve lifecycle changes used Anvil Serving rather than raw Docker mutation:

uv run --no-sync anvil-serving serves up heavy --manifest examples/primary-node/serves.toml --recreate --no-router --confirm
uv run --no-sync anvil-serving eval preflight --tier heavy --manifest examples/primary-node/serves.toml --checks smoke,json,needle,tools --needle-ctx 120000 --tool-batch 20 --thinking-mode default --reasoning-effort low --visible-answer-tokens 512 --reasoning-headroom-tokens 4096 --reasoning-evidence required --confirm
uv run --no-sync anvil-serving eval benchmark capacity --tier heavy --manifest examples/primary-node/serves.toml --requests 40 --concurrency 8 --ctx-tokens 8192 --max-tokens 256 --reasoning-effort low --confirm
uv run --no-sync anvil-serving eval benchmark quality --tier heavy --manifest examples/primary-node/serves.toml --suite chat,context,tool,session,intelligence --context-targets 32768,128000 --eval-repetitions 3 --eval-min-pass-rate 1.0 --visible-answer-tokens 256 --reasoning-headroom-tokens 2048 --reasoning-effort low --confirm
uv run --no-sync anvil-serving harness sync openclaw --config examples/primary-node/anvil-router.live.toml --dry-run

The current and rollback shapes are defined in examples/primary-node/docker-compose.yml, examples/primary-node/serves.toml, and configs/serve-recipes.toml.