Qwen3.8 27B official FP8 1M-context continuation¶

Captured: 2026-08-14
Decision: challenger, no-promotion
Measured hardware: one RTX PRO 6000 Blackwell Max-Q 96 GB card, with the second equal card continuing its independent BF16 serve
Evidence classes: functional, capacity, bounded long-context retrieval

Outcome¶

The official FP8 checkpoint extended well beyond the earlier native-window test on one local SM120 card. A 1,010,000-token configured serve completed a monotonic retrieval ladder through 825,049 actual prompt tokens. The largest case passed 3/3, returned the exact buried code each time, used finish_reason=stop, and averaged 956.739 seconds request-to-completion. The spread was only 0.562 seconds.

That is a stable offline/batch capability, not an interactive default. The largest request took almost 16 minutes for a 14-token answer. No route, client, or promotion changed. After the ladder, the candidate again passed coding, JSON, 128K retrieval, and 20/20 tool calls. The original 262K FP8 lane was then restored through its exact managed recipe and passed the same full gate.

Exact configuration¶

Model: Qwen/Qwen3.8-27B-FP8 revision 017b9c7af6b5689d5dd426a76e0bc077eb5ca20a.
Runtime image: sha256:4a2f33a884222f7049b983263ad9976f89452bb81affecf5b67d89ad35c1bc31, vLLM revision 3a0914114705fa38d4c3171d0746c1a6b6f10209.
Topology: TP=1 on one of two equal PCIe-attached RTX PRO 6000 cards; the BF16 lane stayed on the other card.
Weights/KV: official block-scaled FP8 weights, FP8 KV.
Context: --max-model-len 1010000 plus nested text_config.max_position_embeddings=1010000.
Admission: one sequence, 4,096 batched tokens, GPU utilization 0.92.
Control choices: text-only, chunked prefill, thinking disabled, prefix cache off, and no MTP.
Portable recipes: configs/qwen38-27b-official-fp8-1m-recipe.toml and configs/qwen38-27b-official-fp8-262k-control-recipe.toml.

Startup allocated 56.68 GiB for KV cache and reported 1,845,432 KV tokens, or 1.83 configured full-context requests. This proves the memory shape fits; the campaign still constrained admission to one and did not test concurrent 1M-sized requests.

Retrieval results¶

The harness target is approximate, so the table uses API-reported prompt tokens as the capacity claim. Latency is non-streaming request start through the complete 14-token answer; the derived input rate includes all request and generation overhead and is not a kernel-only prefill rate.

Harness target	Actual prompt tokens	Passes	E2E latency	Derived effective input rate
384,000	316,849	1/1	167.252 s	1,894.4 tok/s
512,000	422,449	1/1	279.367 s	1,512.2 tok/s
768,000	633,649	1/1	585.176 s	1,082.8 tok/s
1,000,000	825,049	3/3	956.528-957.090 s; mean 956.739 s	mean 862.3 tok/s

The falling effective input rate shows a steep upper-window latency cost. The result establishes repeated retrieval and process stability at 825,049 actual tokens; it does not establish a one-million-token API prompt, general long-context reasoning quality, or acceptable interactive latency.

Recipe research update¶

The initial official 27B recipe revision already contained the 1M launch flags. The later 27B update changed the declared vLLM floor and difficulty label, not the context recipe. The newly present official sibling is Qwen3.8-2.4T-A95B, whose multi-node/high-GPU-count Blackwell assumptions make it unsuitable for this two-card 192 GB host.

The continuation stayed on the pinned official Qwen FP8 revision. No third-party NVFP4 checkpoint was downloaded or loaded.

Gates and restoration¶

Before mutation, both existing BF16 and FP8 endpoints passed their functional preflights. The 1M candidate passed the full gate before and after the retrieval ladder. The candidate was then unloaded through Anvil Serving, and the exact 262K FP8 control was restored on the same logical GPU role. Its final preflight passed coding, JSON, 128K retrieval, and 20/20 tools. The untouched BF16 lane passed a final coding/JSON smoke. Shared memory ended available and clean with zero reclaimable files.

Caveats and next experiment¶

vLLM retained the warning that missing FP8 attention q/prob scaling factors default to 1.0. These retrieval passes do not prove broad quality equivalence.
The largest prompt was 825,049 actual tokens, not one million.
No 1M concurrency, MTP, prefix-cache, routed alias, multimedia, durable worker agentic, or SWE-bench run was performed.
A useful next performance experiment is a matched MTP=3 versus no-MTP A/B at a bounded mid-window context, not another 16-minute ceiling repetition.
Promotion remains separately human-gated.