Qwen3.8 27B GGUF 250K qualification on RTX 5090¶
Date: 2026-08-21
Routed-client follow-up: 2026-08-22
Decision: FAST-TIER challenger; no-promotion
Measured hardware: one NVIDIA RTX 5090, 32,607 MiB, Blackwell sm_120
Topology: direct managed candidate serve; RTX PRO 6000 hardware was not involved
The measured host exposed 32 GiB of physical RAM, not the planning assumption of 96 GiB. Docker/WSL had about 16.2 GiB available; late in the MTP run the container used 12.57 GiB of its 15.08 GiB limit. The result therefore does not depend on hidden 96 GiB host memory or model-weight CPU offload.
Outcome¶
Unsloth Qwen3.8 27B Q4_0 with its Q4_0 MTP head is the first measured RTX 5090 recipe in this repository to preserve an 8,192-token output reserve while accepting 253,822 actual prompt tokens. It passed exact long-context retrieval, tools after more than 100K tokens, repeated multimodal tasks, and 101-request agent sessions. MTP raised short decode throughput from 69.1 to 104.1 tok/s and reduced end-to-end latency by 18.7%, at the cost of 20.8% worse short TTFT and 17.0% lower short prefill throughput.
This evidence supports a FAST-TIER routing role, but not promotion. The independent
SWE-bench worker could not be reached through its required controller transport, the
upstream container health probe targets its image-default port rather than the recipe
port, and llama.cpp emitted non-consecutive-position warnings during the endurance run.
The exact pre-test RadixArk NVFP4 128K recipe was restored and passed its complete
preflight. No route or promotion state changed.
Immutable artifacts¶
| Component | Exact identity | Size / digest | License |
|---|---|---|---|
| Official base | Qwen/Qwen3.8-27B@1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 |
BF16 source identity; not loaded in this arm | Apache-2.0 |
| Q4_0 weights | unsloth/Qwen3.8-27B-GGUF@4ca720788d1e01f1bff70c033e0d0028fd02e502, Qwen3.8-27B-Q4_0.gguf |
16,056,478,688 bytes; SHA-256 ede16c7b36e578ca87a8c70e011e4b4633a32c831c0ce76d0f474582384e671d |
Apache-2.0 |
| MTP head | same repository/revision, MTP/mtp-Qwen3.8-27B-Q4_0.gguf |
1,369,590,656 bytes; SHA-256 50d9ce5a6da381bbcfb31061cf73df94a90e6faf8efeddee379a9cb8f1501c6e |
Apache-2.0 |
| Vision projector | same repository/revision, F16 projector | 927,607,488 bytes; SHA-256 cbb841a9ee0636b2ec172f5bb8df2ea8dfeb01e90fe7c6126581d662a0b4e43e |
Apache-2.0 |
| Q6_K control | bartowski/Qwen3.8-27B-GGUF@f0eec4a4bb4975114a030d048952d83c0a53c034, Qwen3.8-27B-Q6_K.gguf |
23,463,130,720 bytes; SHA-256 7d590099e0a0fe7b8df812045faa2ae12bf4dbf3492b8eb7c7ab24c94d36ed |
Apache-2.0 |
| Runtime | ghcr.io/ggml-org/llama.cpp:server-cuda@sha256:cf2e30bc855cf58cdbdc65d05b5b5e02afa95fb788343a5334d704367ac5c9ac |
llama.cpp b10548, commit a298422da78eb75e440a7de0ca408af64d323d93 |
MIT |
The evaluated 4-bit artifact is explicitly GGUF Q4_0; it is not relabeled as
ModelOpt NVFP4 or an undocumented N4_0. The conventional Q6_K control comes from a
different quantization repository, so even a future load would retain a quantizer and
calibration provenance confound.
Served configuration¶
Both loaded arms used 262,144 served tokens, concurrency one, batch 2,048, microbatch 512, full GPU layer offload, Flash Attention, Q4_0 K/V cache, mmap loading, Jinja chat templating, Qwen reasoning disabled, the DeepSeek reasoning wire format, and the F16 vision projector. Sampling was temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5, and repeat penalty 1.0. The speculative arm added draft-MTP, three maximum draft tokens, all draft layers on GPU, and Q4_0 draft K/V.
The embedded GGUF chat template was used through --jinja; no external chat
template override was supplied. Tool calls used llama.cpp's Jinja/OpenAI server
path rather than a separately named tool-call parser. Reasoning was disabled
and --reasoning-format deepseek selected the wire representation if reasoning
content appeared. The container's exact CUDA toolkit build was not separately
captured, so it remains unknown; successful execution with driver 610.88 and
native sm_120 kernels is local compatibility evidence, not a generic minimum
CUDA-version claim.
Startup logs prove the MTP model was loaded. The server's /props response did not
identify the active speculative type, so acceptance counters from runtime metrics and
logs are the authoritative speculative evidence for this run.
Mathematical screen¶
The policy requirement was 250,000 prompt tokens plus 8,192 output tokens, concurrency
one, with 3 GiB operational headroom and 0.75 GiB estimation uncertainty. The usable
VRAM budget was therefore 30,164,385,792 bytes. Let W be resident weights and runtime
state, D the MTP drafter, K(t) the KV allocation at t tokens, A transient
activations, and H the reserved headroom:
V_required(t) = W + D + K(t) + A + H.
Bounded KV-byte uncertainty produced these feasibility intervals:
| Recipe | Estimated maximum tokens | Margin at 258,192 tokens | Decision |
|---|---|---|---|
| Q4_0, no spec | 635,349–755,309 | +6.115 to +7.585 GiB | load and measure |
| Q4_0 + MTP3 | 486,463–611,411 | +4.217 to +5.921 GiB | load and measure |
| Q6_K, no spec | 197,111–303,243 | -0.990 to +0.687 GiB | unresolved; do not promote |
| Q6_K + same MTP | 101,867–199,930 | -2.888 to -0.977 GiB | policy-infeasible; disqualify before download |
The Q6_K+MTP interval fails even at its optimistic bound. Running it could only prove an unsafe allocation, host offload, reduced context, or a violated reserve; none satisfies the requested recipe.
Matched performance¶
| Arm | Startup VRAM | Short TTFT | Short prefill | Short decode | Short E2E | Result |
|---|---|---|---|---|---|---|
| Q4_0 no spec | 22,254 MiB | 0.24 s | 14,881 tok/s | 69.1 tok/s | 0.91 s | control passed |
| Q4_0 + MTP3 | 25,408 MiB; 25,460 MiB late | 0.29 s | 12,352 tok/s | 104.1 tok/s | 0.74 s | preferred interactive arm |
Across 775 requests the cumulative MTP acceptance was 17,606/19,227, or 91.57%. That aggregate is repetition-biased by the context and endurance corpora. Before the long ladder, the mixed short/tool workload accepted 792/1,053 drafts, or 75.2%.
On the first 250K MTP request, runtime logs reported 1,050.16 prompt tok/s over 237.95 seconds, 60.49 decode tok/s, and 65/66 accepted draft tokens. GPU load was about 99%, power about 460 W, and temperature 59–61 °C during long prefill; VRAM allocation remained fixed rather than growing with each request. The runtime did not expose standalone per-request KV bytes, so this finding records the Q4_0 KV format and total allocation without inventing a KV-byte estimate.
llm.secondary routed real-client evaluation¶
A bounded 2026-08-22 acceptance used the existing llm.secondary test alias from
Companion Node through the Primary Node router to the RTX 5090 candidate. No router
assignment or client default was changed. OpenClaw and Hermes each passed an exact
marker turn and a real shell-tool/result-continuation turn. OpenClaw reported the
requested anvil/llm.secondary winner with fallback disabled. Hermes reported the
requested alias under its normalized custom usage identity; its tool turn made two
API calls and retained one terminal call in session evidence.
Two deliberate Hermes negative-control attempts used the wrong provider selector. The visible marker was correct, but usage evidence proved that a configured fallback model produced it. Those attempts are failures. Routed acceptance must validate the client-observed provider and model rather than grade answer text alone.
The short routed and tool-continuation gates pass, but the 250K routed gate remains closed. During the test the router advertised a 131,072-token compatibility profile with stale SGLang/NVFP4 fingerprint fields, while the measured backend was the 262,144-token llama.cpp Q4_0/MTP3 recipe. A truthful 262K router profile and matching client catalog metadata must be deployed before a 250,000-token-minimum routed acceptance can pass. Native video remains unsupported by this recipe and must not be inherited from the compatibility profile.
llm.secondary gate |
Result | Evidence |
|---|---|---|
| OpenClaw exact model identity | pass | requested and winning model were anvil/llm.secondary; fallback was disabled and unused |
| OpenClaw shell tool/result continuation | pass | real shell call completed and the model continued from its result |
| Hermes exact model identity | pass | usage recorded the custom provider and llm.secondary model |
| Hermes shell tool/result continuation | pass | one terminal call completed across two model API turns |
| Fallback rejection control | pass | two wrong-provider attempts returned the right visible marker but were rejected because usage proved fallback |
| Truthful runtime/quant fingerprint | fail | router described SGLang/NVFP4 while the candidate was llama.cpp Q4_0/MTP3 |
| Advertised context at least 250K | fail | router and clients exposed 131,072 rather than the candidate's 262,144 tokens |
| Promotion authorization | not granted | the evaluation made no persistent route or client-default change |
The llm.secondary verdict is therefore short-path qualified, long-context
not qualified, no promotion. It is suitable for continued live evaluation
only while the caller understands that the current routed metadata does not
truthfully expose the measured 250K capability. Updating the fingerprint is a
prerequisite, not an administrative cleanup: without it the harness cannot
distinguish the candidate from the previous SGLang/NVFP4 serve or size client
compaction safely.
Context and correctness gates¶
| Requested prompt bucket | No spec | MTP3 | Mean MTP3 elapsed | Result |
|---|---|---|---|---|
| 32K | 12/12 | 12/12 | 12.47 s | pass |
| 131K | 7/7 | 12/12 | 83.25 s | pass |
| 200K | 7/7 | 7/7 | 164.98 s | pass |
| 250K | 7/7 | 7/7 | 240.72 s | pass |
| 253,952 requested + 8,192 reserve | 1/1 at 253,822 actual | 1/1 at 253,822 actual | 246.72 s | exact cross-link retrieval pass |
The corpus placed exact names, UUID-like values, numbers, IP addresses, code symbols,
and cross-document relationships near 10%, 25%, 50%, 75%, 90%, and 97% of the prompt.
The MTP arm remained functional after the full envelope: JSON, streaming, tool-result
continuation, and tools 20/20 all passed without a restart. A separate long-tool request
reported 110,875 actual prompt tokens and selected get_weather with valid nested JSON.
| Gate | MTP3 result | Caveat |
|---|---|---|
| Functional preflight | pass | coding, JSON, streaming, continuation, Responses, tools 20/20 |
| Agentic scout | 16/18 | both failures were known debug-loop extra-read protocol desynchronization |
| Neutral 101-request endurance | 3/3 | exact final recall; prompt growth 29 to 3,099 tokens |
| Privacy-worded endurance control | 0/3 | model explicitly refused the prompt's request to remember a private marker; prompt-sensitivity control retained |
| Deterministic image corpus | 18/18 | screenshot, OCR, chart, UI controls, spatial count, two-image comparison |
| SWE-bench Verified scout | incomplete | required controller credential was unavailable; no SSH fallback is permitted |
| Native video | unsupported | not claimed by this llama.cpp recipe |
Vision request latency was 0.621 s p50 and 1.902 s p95. A startup warning notes that
Qwen-VL grounding may require --image-min-tokens 1024; the bounded corpus passed
without changing that setting.
Heavy-model context¶
The heavy references were not rerun in this qualification, so these are retained controls, not a matched head-to-head. The existing single-RTX-PRO Qwen3.5 122B NVFP4 record passed tools 20/20 and a 231,426-token prompt at 68.91 s TTFT; its dual-PRO record passed the bounded intelligence and endurance gates. GPT-OSS 120B retains tools 20/20 and strong generation throughput at 131K, but its old deterministic quality record is marked historical-invalid because the reasoning budget was not normalized. These records are insufficient to claim that Q4_0 has no coding-quality loss relative to either heavy model.
Promotion and routing decision¶
- Keep Q4_0 + MTP3 as the RTX 5090
FAST-TIERchallenger for interactive tools, repository inspection, long-context retrieval, and screenshot/OCR work. - Escalate difficult multi-file coding, ambiguous architecture, repeated tool recovery, and high-consequence long-horizon tasks to the qualified heavy tier until a matched SWE and time-to-success campaign is complete.
- Do not promote until the managed health probe reflects the recipe port, the llama.cpp position warnings are explained or eliminated, the isolated SWE worker completes, and a truthful 262K fingerprint/capacity profile passes routed client acceptance at the 250K minimum.
- Do not load conventional Q6_K + the same MTP head at 262K on this 32 GB card. Revisit Q6_K only without MTP, with a smaller context, or after measured KV/residency changes make the entire conservative interval positive.