Skip to content

Gemma 4 vLLM 0.25.1 WSL2 and concurrency-128 retest

Date: 2026-07-16

Host: Primary Node under Docker Desktop/WSL2

Hardware: RTX 5090 32 GB (Fast) and RTX PRO 6000 96 GB (Heavy)

Runtime: vLLM 0.25.1, FlashInfer 0.6.13, FP8 KV cache

Outcome

The earlier single-stream result was accurate but incomplete. Unsloth 12B NVFP4 is slower than official QAT at concurrency 1 on both cards. At concurrency 128, however, it is 35.7% faster than QAT at 1K context and 45.0% faster at 8K context on the PRO 6000. On the RTX 5090, that gain only appears with Model Runner V2: NVFP4 is 35.2% faster than QAT at c128/8K, while the two are effectively tied at c128/1K.

The production engine upgrade is safe and complete:

  • Fast and Heavy now use the same pinned vLLM 0.25.1 image digest.
  • Both set VLLM_WSL2_ENABLE_PIN_MEMORY=1.
  • Both deliberately keep VLLM_USE_V2_MODEL_RUNNER=0. The V2 lab serve reached health, but vLLM warns that V2 does not yet support thinking_token_budget, which is part of the Gemma behavior surface.
  • Fast passed smoke, structured JSON, and 20/20 tool calls after the upgrade. Heavy passed the same gate at the corrected 512-token visible budget; the retained 256-token attempt hit finish_reason=length after producing an otherwise valid but overlong answer.

No NVFP4 checkpoint was promoted. The prior repeated quality failures remain unresolved, and this run measured serving capacity rather than re-ranking answer quality.

Matched 12B QAT versus NVFP4

All rows use 128 requests, concurrency 128, a 128-token generation cap, thinking disabled, FP8 KV, and the same vLLM 0.25.1 container. Throughput is aggregate output tokens/second from the repository's capacity harness.

GPU Runner Quant c128 / 1K c128 / 8K 8K p95 TTFT
RTX 5090 V1 QAT W4A16 1,991 1,135 1.91 s
RTX 5090 V1 Unsloth NVFP4 1,264 1,124 2.19 s
RTX 5090 V2 QAT W4A16 2,074 1,195 1.89 s
RTX 5090 V2 Unsloth NVFP4 1,998 1,616 1.23 s
RTX PRO 6000 V1 QAT W4A16 2,042 1,053 1.58 s
RTX PRO 6000 V1 Unsloth NVFP4 2,770 1,526 1.16 s

At low concurrency the result goes the other way:

GPU Quant c1 / 1K c8 / 1K c1 / 30K
RTX 5090 QAT W4A16, V1 74 611 27
RTX 5090 Unsloth 12B NVFP4, V1 53 462 27
RTX PRO 6000 QAT W4A16, V1 71 578 26
RTX PRO 6000 Unsloth 12B NVFP4, V1 65 516 26

This reconciles the apparent contradiction with Unsloth's published B200, concurrency-128 results: the checkpoint is not a universal single-stream decode win, but it can be a large continuous-batching win on the right runner and hardware.

Larger-model sweep

Model GPU c1 / 1K c8 / 1K c128 / 1K c128 / 8K c1 / 30K
Unsloth 26B-A4B NVFP4 RTX 5090 101 1,006 3,538 1,320 62
Unsloth 26B-A4B NVFP4 RTX PRO 6000 91 853 3,227 1,466 53
Unsloth 31B NVFP4 RTX PRO 6000 36 335 1,720 799 10

The 26B-A4B checkpoint is the throughput leader at short context on both cards. Its 8K Heavy result is slightly below 12B NVFP4, and its earlier repeated quality gate failed, so it remains a lab-only candidate. The 31B model remains materially slower despite its earlier quality pass.

How the speed is achieved

The local evidence points to a stack of effects rather than one hidden optimization:

  1. The Unsloth checkpoint mixes W8A8 FP8 and W4A4 NVFP4 layers, with FP8 KV cache. vLLM selected CutlassFP8ScaledMMLinearKernel for FP8 layers and FlashInferCutlassNvFp4LinearKernel for dense NVFP4 layers.
  2. The 26B-A4B model only activates about 4B parameters per token. vLLM auto-selected the FLASHINFER_CUTLASS NVFP4 MoE backend, which explains why it can outrun the dense 12B and 31B models despite having more total weights.
  3. Continuous batching is decisive. The 12B NVFP4 advantage appears at c128, not c1 or c8.
  4. Model Runner V2 materially improves the 5090 NVFP4 high-concurrency path. The matched QAT V2 control only improved 4-5%, so the larger NVFP4 uplift is not merely a generic runner gain.
  5. FlashInfer B12x did not produce these numbers. Stable vLLM 0.25.1 selected CUTLASS, and the current upstream dense-B12x selector fix was still unmerged during this run.

Unsloth documents the layer-level format and the benchmark hardware/concurrency, but not the exact Gemma calibration or important-layer selection algorithm. That remaining recipe detail cannot be recovered from the published checkpoint alone.

Evidence and caveats

  • Raw capacity artifacts and production preflights are in 2026-07-16-gemma4-vllm0251-wsl2-evidence.
  • Exact external sources, dates, evidence class, and decision impact are in source-registry.json.
  • Capacity prompts share a fixed-context shape and are cache/order sensitive. These numbers are serving-capacity evidence, not an isolated decode-only microbenchmark.
  • The WSL2 feature was proven by healthy V2 startup with VLLM_WSL2_ENABLE_PIN_MEMORY=1; no CPU offload was used, so the benchmark is not distorted by host-memory paging.
  • Quality promotion remains blocked by the previous repeated gates. V2 also lacks the required thinking-budget behavior, independently blocking production promotion.