Skip to content

T013 — Vision model serve and preset (Qwen3-VL-4B-Instruct, first evictable reservation)

Task: gpu-reservations:T013 · Date: 2026-07-13 · Box: primary-node (RTX 5090, WSL2/Docker) Evidence: docs/findings/2026-07-13-t013-vision-evidence/

What shipped

  • vision intent preset (intent.PRESETS) + declared-preset-only vision work class (classify.WORK_CLASSES) — general image understanding (describe/ground a screenshot), distinct from ocr's verbatim document extraction. /v1/models advertises it with no other edit (T004 no-drift design; pinned by tests/router/test_discovery.py).
  • vision serve (examples/primary-node/serves.toml + docker-compose.yml): Qwen/Qwen3-VL-4B-Instruct @ ebb281e… on the pinned vLLM digest, :30008, loopback-only, UUID-pinned to the 5090 — the first ADR-0017 evictable reservation (12288 MiB) on dark-fast, with router_tier = "vision-local" declared for the T005/ADR-0018 drain.
  • vision-local router tier in anvil-router.live.toml/anvil-router.agentic.toml (byte-identical), tool-free, 16 K window, model_identity = true (see finding 2).

Live verification (all through product verbs; evidence files in parentheses)

  1. Honest denial: with the full resident set running (fast+embeddings+reranker+ocr = 26112/27999 MiB), serves up vision is denied by the ledger — over budget by 10401 MiB, nothing started (admission-denied-full-resident-set.txt, ledger-0…). Deliberate: evictable serves run opportunistically; the T015 resident-set decision owns any rebalance.
  2. Bring-up: after serves down ocr reranker embeddings, serves up vision admits (26624 committed / 1375 free — ledger-1…); engine healthy, 8.63 GiB weights, 18,192-token fp8 KV.
  3. Grounded description: a synthetic dashboard screenshot with known ground truth (RTX 5090 / 30057 MiB / HEALTHY dot / "Error 503 … 'planning'" dialog / Retry button / heavy-fast-ocr bar chart) is described correctly both direct (direct-vision-description.txt) and through the routed vision preset on a worktree router built from this branch (routed-vision-description.txt; decision log routed-decision-log.json: work_class=vision, served_tier=vision-local, verifier_passed=true; v1-models-advertises-vision.txt).
  4. Eviction (T005 flow, simulated on-demand acquisition): serves up comfyui-sim --evict (8192 MiB on-demand stand-in; ADR-0017 accounting is declarative, so an alpine container commits the reservation faithfully — eviction-sim-manifest.toml/-compose.yml) chose vision as the victim, quiesced + drained vision-local through the router's ADR-0018 boundary before docker stop, then admitted the sim (eviction-on-demand-acquisition.txt, ledger-2…: vision exited/not committed, comfyui-sim on-demand/running). Routed vision requests exhaust cleanly (HTTP 503) while evicted.
  5. Restore: serves down comfyui-sim + serves up vision re-admits (ledger-3…), guarded router readmit passes health + exact identity (observed_model == qwen3-vl-4b-instruct), and the routed request grounds again (finish stop, grounded True).
  6. Box restored: resident set back to 26112/27999 (free 1887, ledger-4…); live router (:8000) healthy and untouched; voice sidecars untouched (up 10h through the whole task).

Findings (fixed in this change)

  1. vLLM auto memory profiling is unusable on the busy WSL2 card (new CLAUDE.md gotcha #22): cudaMemGetInfo reported 30.2 GiB free while nvidia-smi showed ~12.6; the profiler blamed the reconciliation delta on itself (~13.8 GiB phantom "non-torch forward increase") and computed negative KV at any admissible budget. Fix: --kv-cache-memory-bytes 1342177280 (1.25 GiB fp8 = 18,192 tokens, 1.11x @16 K) bypasses measurement-based sizing. Measured engine footprint: 11,738 MiB device delta — inside the 12288 MiB reservation.
  2. model_identity = true is REQUIRED on a tier backed by an evictable serve: the guarded router readmit refuses (identity_not_configured) without it, so an evicted tier could never re-enter rotation. Verified live, then re-verified end-to-end with the flag.

Follow-ups (operator)

  • Live router promotion: the updated anvil-router.live.toml (vision tier + preset, mapping_version 2026-07-13.primary-node-t013-vision) is NOT yet promoted to the anvil-router-cfg volume, and /v1/models on the deployed image will not advertise vision until the router image is rebuilt from a main containing this change. Promote via the guarded router promote path at the next router release (T011 precedent). Until then the deployed router treats config-declared vision as a custom preset if promoted config-only.
  • T015 resident-set decision: vision (12288 evictable) + full resident set oversubscribes by 10401 MiB; today it runs only when residents are freed. If routine vision traffic is wanted, T015 should weigh which residents yield.