Benchmark methodology and evidence rules¶

The benchmark guide is designed to grow without turning unlike measurements into a misleading leaderboard. This page defines the vocabulary, comparison classes, execution pipeline, and minimum publication record.

Evidence and decision labels¶

Every published run carries one or more evidence labels: external-prior, compatibility-only, functional, capacity, quality, and historical-invalid. The decision is recorded separately as current, rollback, challenger, no-promotion, or rejected.

The measured GPU is the card that executed the workload. protected/co-resident means another card or service remained available during the run; topology-only means the hardware was described but not exercised. Neither is measurement evidence. In particular, the July 28 ASR qualification measured the RTX 5090 while protecting the RTX PRO 6000.

Measurement classes¶

Only rows with the same label, hardware, workload, and material recipe settings should be ranked directly.

Metric dictionary¶

Time to first visible token (TTFT). Time from request start until the first non-empty streamed content delta. For a model without a separate reasoning channel, it includes queueing, prefill, and first-token work. For a reasoning model, hidden reasoning can occur before this point, so TTFT is a user-visible latency measure and must not be treated as a prefill boundary.

Time to first output (TTFO). Time from request start until the first non-empty streamed reasoning or content delta. capacity-v4-reasoning records TTFO separately from TTFT so hidden reasoning is not mislabeled as prefill. For a non-reasoning response, TTFO and TTFT are the same event.

End-to-end latency (E2E). Time from request start until the response finishes. Always publish output-token counts or workload identity beside it.

Standalone prefill throughput. Prompt tokens divided by an isolated prefill interval. Do not derive this from TTFT unless the harness separately records the queueing and first-token components. When that evidence is absent, publish the long-context TTFT as a labeled prefill-latency proxy and leave prefill tok/s blank.

Effective client-observed prefill rate. Prompt tokens divided by the protocol's first-output boundary: TTFT in capacity-v3, TTFO in capacity-v4-reasoning. This includes queueing, scheduling, prompt processing, and first-output work, so it is useful for matched end-to-end A/Bs but is not standalone or kernel-only prefill throughput. Both protocols label the field effective_prefill_tok_s; the protocol identity is therefore required before comparison.

Generation duration. Client-observed E2E minus the protocol's first-output boundary. In capacity-v3 that is TTFT. In capacity-v4-reasoning it is TTFO, so the interval includes reasoning and visible generation. The reasoning-aware protocol also retains visible_generation_ms from first visible content to the end of the response.

Controlled decode rate. For a long-generation workload, approximately:

(output tokens - 1) / generation seconds

The harness should retain its direct timing fields and token counts. Avoid this derived rate for very short outputs, where timer and scheduling overhead dominate. When usage completion tokens include reasoning tokens, the reasoning-aware protocol's rate describes the combined completion stream, not visible-only decode.

TPOT and the legacy mean-ITL key. Generation duration divided by output tokens - 1. tpot_ms, mean_time_per_output_token_ms, and the backward-compatible mean_inter_token_latency_ms field describe the same per-request aggregate proxy. They are not timestamped token-arrival intervals, so their percentiles summarize request-level mean TPOT values, not token-level ITL. Prefer TPOT in new prose and publish token-level ITL only when a native artifact retains token event timestamps and defines transport-chunk handling.

Aggregate output throughput. Total output tokens across all requests divided by wall time. This is useful for batch capacity at a stated concurrency, but it is not interchangeable with per-request controlled decode. The headline is performance-eligible only when every attempted request completes and satisfies its active output/canary contract with exact token usage. Failed responses retain bounded, sanitized timing and response metadata but are excluded from performance distributions; prompts and full response bodies are not retained. An incomplete run keeps its diagnostic samples and failures while aggregate throughput_tok_s is null. The local harness uses usage.completion_tokens when the endpoint returns it. Under capacity measurement protocol capacity-v3, usage is required for exact token throughput, effective prefill, decode, and mean inter-token metrics. When usage is absent, exact token-derived values are null and streamed content chunks are retained only as a diagnostic rate. Each successful request retains its planned context, request index, prompt/output token counts, TTFT, generation duration, E2E, effective prefill, decode rate, and mean inter-token latency. Comparison tooling treats different capacity measurement protocols as a workload mismatch. capacity-v4-reasoning additionally retains TTFO, first-visible TTFT, reasoning-chunk count, and a generation interval beginning at first output. Failed reasoning-only completions remain failures rather than being silently converted into visible latency samples.

Controlled-output validation. --response-words N requests exactly N lowercase code words. The default --controlled-output-policy observe records requested and observed word counts, extra words, capture completeness, and exact adherence without rejecting a non-exact response. Use --controlled-output-policy strict for finalist comparisons; it requires a nonzero target and makes non-exact or capture-truncated responses performance-ineligible. The bounded visible-content capture is 8,192 characters, so completion headroom and capture capacity both constrain a strict target. Historical artifacts without these observations are not retroactively exact.

Request canaries. In unique-cache concurrent cells, each response must begin with its own generated marker, contain no other ANVIL_REQ marker, and fit in the complete validation capture. A late/missing marker, any foreign marker, or truncation is a correctness failure. Historical passed fields that only prove substring presence retain their original meaning and are labeled as legacy validation by derived aggregates.

Independent-replica wall time. New exact aggregates record nanosecond Unix start/finish fields and the same explicit clock_domain_id only when every replica process shares one physical host clock. A matching clock-domain label must never be inferred or reused across hosts. Replicas also carry the same declared SHA-256 configuration fingerprint; matching declarations bind the comparison but are not independent attestation of the launch.

For legacy artifacts with only matching whole-second UTC buckets, the combiner uses each replica's monotonic duration to derive lower and upper union-wall bounds. metrics.throughput_tok_s is the conservative lower bound and throughput_tok_s_upper_bound is published with it. Whole-second evidence, missing fingerprints, or older output validation must remain labeled bounded or legacy-unverified, never exact synchronization.

Context ladders retain the same timing and token fields on every context row, not only in the run-level percentile summary. A ladder may be cold, warmed, or prefix-cached; publish that state and the endpoint-reported cached-token count per row. Do not merge cold and warmed rows into one percentile or describe a client-observed TTFO-derived rate as standalone kernel prefill.

Percentiles. Capacity summaries use the nearest-rank method over successful samples. Sort the values, compute ceil(percentile * sample_count / 100), and select that one-based rank. Failed requests are reported separately rather than inserted as synthetic latency values.

Concurrency. The number of active requests driven by the harness. Report it separately from the engine's admission cap (max_num_seqs). Prefix caching or RadixAttention changes results when prompts share prefixes, so publish cache policy and prompt independence.

Served context. The configured engine limit. Validated context is the largest length that passed a retained correctness check. Advertised context is only a model-card claim. Never substitute one for another.

Reasoning headroom. The allowance added to a separate visible-answer allocation. Protocol v3 sends their sum as the endpoint's single completion cap; it cannot hard-partition hidden and visible channels. A 4,096-token headroom plus the standard 256-token visible allocation therefore sends max_tokens=4352. A wrong visible answer and a truncated/no-visible-answer attempt are different failure modes and must be counted separately.

Stable quality. An item is stable only when every retained repeat is correct. Publish both stable items and total correct attempts, such as 8/10 stable, 23/30 attempts.

Sampled context plan. Capacity sampling uses measured cumulative context buckets and a recorded pseudorandom seed. The full plan is resolved before workers are submitted, so thread scheduling cannot change which context lengths the run attempts. Prefix-cache bursts share a byte-identical bounded prefix and retain a unique request suffix.

Effective context. The context-degradation workflow calibrates prompt size from endpoint-reported token usage, places independently hidden targets at the profile's declared positions, and retains every raw observation. Effective context is the highest attempted bucket that still meets both the profile's pass-rate floor and maximum relative drop from its attempted baseline. It is not the advertised window, the configured engine limit, or an unattempted interpolation. Missing engine prefill/decode telemetry stays unavailable rather than being inferred from wall-clock latency.

Agentic stage score. Deterministic agentic fixtures score protocol, tool arguments, result incorporation, recovery, history, applicable reasoning, and final answer independently. Tool results come from the harness, never from the model under test. Tool scenarios accept a natural-language final answer when it contains every deterministic result marker; they do not require an undisclosed marker-only response. The reasoning stage is applicable only to an explicit reasoning scenario and is never inferred from exact-answer formatting. Ordered planning fixtures accept equivalent numbered or delimited workflows, and required tool arguments are disclosed in the prompt rather than hidden in the grader. Artifacts retain the visible final answer, its SHA-256, per-turn latency and token counts, finish reasons, tool-call counts, and whether a separate reasoning channel was present. A malformed call, reasoning-budget exhaustion, incorrect recovery, final-answer failure, and worker failure are separate outcomes. Long-session fixtures execute every scripted user turn as a real request, retain the model's actual replies in history, and require strictly increasing endpoint-reported prompt-token counts. A single prefilled transcript cannot be reported as a multi-turn endurance pass.

Long-context tool gate. The long-tools preflight check calibrates a real tool request and passes only when endpoint usage reports at least 100,000 actual prompt tokens, the model selects the expected tool, and its arguments are schema-valid. Nominal filler length alone is not capacity evidence.

Official SWE resolution. An agent trajectory and patch are intermediate evidence. SWE resolution is complete only when the pinned official SWE-bench grader emits a report for the exact explicitly selected instance. Agent, dataset, image, prediction, grader, and model reasoning-control identities remain bound in the artifact. See the durable job workflow.

Evaluation pipeline¶

1. Research. Prefer current official sources and recent hardware-matched community data. External evidence selects a candidate; it never supplies a local pass.
2. Pin. Record the model commit, engine version and image digest, parser, quantization, KV format, context limit, speculative decoding, cache policy, and admission settings.
3. Preflight. Run the exact final endpoint through independent correctness, context, and tool gates.
4. Measure. Keep capacity, controlled decode, and quality as separate workloads. Rerun the same suite across candidates before ranking them.
5. Publish. Add a dated narrative under docs/findings/, link raw JSON rather than copying it, update the findings index, and update the maintained guide when a recommendation or comparison changes.
6. Decide. Promotion, residency, and router calibration require an explicit human gate after evidence review.

Replica capacity-scheduler qualification (separately authorized)¶

Synthetic scheduler tests and a direct measurement of individual replicas are not evidence that a routed scheduler works. The following is an inert method for a later, separately authorized real-hardware qualification. It does not authorize a recipe load, router change, benchmark, promotion, or deployment.

1. Prepare and pin. Use the managed recipe load and lifecycle surfaces, not an ad hoc launcher. Before the first request, retain the repository revision, router and topology inputs, recipe and registry digests, image and software identities, declared tier/member ceilings, scheduler strategy, and an admission/transition starting-state snapshot. Record the live endpoint's exact served model identity separately from its declared recipe provenance; neither substitutes for the other. Start with managed status/log evidence and retain managed unload evidence during restoration.
2. Measure matched cells. Run the same pinned workload and engine controls for (a) each direct member baseline, (b) routed round-robin, and (c) routed capacity scheduling. A direct aggregate is a baseline, not routed-scheduler proof. Use strict controlled output, unique-cache request canaries, the same seed/context/completion controls, and at least 100 attempted requests per comparison cell. Retain every request, including failures and excluded samples. Exercise sustained concurrent levels through each declared member ceiling and the aggregate tier ceiling, including at least one level above each; record the exact configured levels instead of assuming a topology in this method.
3. Prove terminal and adverse paths. Retain selection/admission evidence showing the selected member, pre-reservation score, readiness and pressure freshness class, and one backend attempt for every routed request. Include errors, timeout handling, and ordinary and streamed response close evidence so release after stream close is observable. Under separately approved controlled transitions, quiesce/drain/readmit a member and a tier, then record stale and unknown telemetry cases. Those cells must show no replay, retry, implicit fallback, or second-member selection.
4. Restore and decide. Restore the captured admission, member-intent, recipe, router, and topology starting state with the managed router lifecycle and tier transition procedures. Retain post-restoration status/log evidence and the separately authorized real-client gates. Keep raw evidence, failures, configuration/identity, and restoration records in the campaign artifact set and publish only sanitized derived claims. Human review may then decide whether promotion is warranted; a complete qualification artifact never promotes a route by itself.

This method keeps hardware feasibility, synthetic coverage, real qualification, and promotion separate. It makes no claim that a particular scheduler has been measured, is qualified, or is deployed.

Required publication record¶

Every new benchmark finding should preserve:

Use 127.0.0.1 in local endpoint examples. Credentials belong in environment resolution only and must be redacted from artifacts.

Source freshness¶

Record both the date a source was observed and its publication/update date:

A model card is primary evidence for its own format and advertised limits. An engine cookbook is primary evidence for that engine's recipe. Reddit and forum posts are discovery leads unless the publisher owns the post and local testing corroborates it. llmrequirements.com is an advisory fit prior, not hardware-matched local benchmark evidence.

Known invalid comparisons¶

The earlier fixed 256–384-token deterministic planning suite allowed hidden reasoning to consume the entire completion budget. The resulting Qwen3.5-122B 1/5, Nemotron Puzzle 0/5, and GPT-OSS-120B 0/5 scores are retained as historical-invalid evidence, not cross-model quality rankings. Protocol v2 repairs this by separating reasoning headroom from the visible-answer contract and recording truncation explicitly.

Similarly, a conventional short benchmark can report a much lower aggregate tok/s value than a controlled long-generation run without contradiction. The workloads answer different questions.

Adding a result¶

Inventory and compare retained evidence through the supported read-only surface:

anvil-serving eval benchmark evidence list --root docs/findings --model MODEL --format json
anvil-serving eval benchmark evidence show ARTIFACT.json --format json
anvil-serving eval benchmark evidence compare FIRST.json SECOND.json --format json

This summary surface omits prompts, full responses, reasoning text, and arbitrary stored command/method text; bounds recursive scans and artifact sizes; and flags material workload differences before displaying rows. It replaces ad hoc find/rg plus JSON extraction for routine evidence navigation. Comparison is fail-closed: missing controls or provenance make the result non-comparable, and the command exits non-zero unless the operator passes --allow-mismatch for exploratory reporting. Quality-suite identity requires an immutable source SHA-256. Engine, recipe, and method differences are retained as separate implementation-provenance warnings so a same-workload cross-model comparison remains expressible without hiding how it was served. A speculative A/B method hash remains a blocking workload field, and malformed numeric or thinking-control values make the artifact invalid for comparison.

Use the durable CLI surfaces for repeatable operations:

anvil-serving models pull OWNER/REPO --revision COMMIT_SHA --confirm
anvil-serving serves up SERVICE --manifest examples/primary-node/serves.toml --recreate --confirm
anvil-serving eval preflight --base-url http://127.0.0.1:PORT/v1 --model SERVED_NAME --confirm
anvil-serving eval benchmark quality --base-url http://127.0.0.1:PORT/v1 \
  --model SERVED_NAME --suite-file SUITE.json --candidate-id CANDIDATE \
  --config-id CONFIG --output quality.json --confirm

Then:

1. Put raw evidence in a dated docs/findings/YYYY-MM-DD-topic-evidence/ directory and retain immutable source revisions.
2. Add the dated narrative and update docs/findings/README.md.
3. Update the appropriate table in the benchmark guide and the recipe notes in the model guide.
4. Update the chronological archive if the reference deployment, recommendation, or comparison history changed.
5. Run the strict documentation build and link checker, then request an adversarial review before publication.

The benchmark guide should summarize evidence, never silently upgrade its confidence. A one-pass calibration stays one-pass; an operator-observed result without a retained artifact stays incomplete; a failed load remains a failure.