Flare upstream overlap review — 2026-10-23 Pacific¶
Checked 2026-10-24 UTC against NVIDIA/flare-local-inference d5fd9824593c2a70f6bdc001b39f16b4c5b3b0b4. The GitHub inventory contains all 37 PRs and 34 non-PR issues returned by the all-state listing (71 entries, below the 100-entry page limit). Inventory screening is not a claim to have reviewed every unrelated diff. Focused reviews examined cache retention, state restoration, scheduler latency, numerical diagnostics, export integrity, and adjacent frontend changes. Public sources are linked below; raw inventory is in source/github-inventory.json.
Decision: do not transplant the old diagnostic patch wholesale. Several parts already exist upstream. The host row/page allocation remains an independent candidate, but its benefit must be re-established against the new retention policy. No reviewed PR establishes a fix for our protected warm-hit regression or sampled-output inconsistency.
What overlaps and what could help¶
| Upstream work | Current status | Relationship to our work | Decision |
|---|---|---|---|
| #38: prefix-cache retention | Merged, included in selected baseline | Eviction by actual backing pressure, fallback to a cheaper shallower checkpoint, deferred store retries, compacted-row refills, and host-decision metrics overlap old local diagnostics. It still leaves bytes / 4 / row_record in slab allocation. |
Use current implementation and counters. The baseline is now complete; run matched allocation arms before claiming a current benefit. Do not submit duplicate retention/telemetry changes. |
| #16: shifted private draft pages | Open, c9f1c4f9 |
Directly applies to GLM draft-state capture. Our measured per-rank row size matches the old one-page layout. A missing draft-history page could reduce acceptance after restore. | Test existing PR, do not recreate it. It needs a new exported package. Author reports target outputs stay exact; it does not establish the cause of our output mismatch. |
| #35: requested logprob token IDs | Merged, included in selected baseline | Up to 128 prospectively fixed token scores can compare the same IDs in cold/restored paths; stronger than changing top-20 sets. | Reuse after re-export. It is a diagnostic capability, not a numerical correction or full-logit oracle. |
| #43 / issue #42: feedback starvation | Open, 6dd92500 |
Makes feedback/final-prefill work visible while ordinary decode is in flight. Relevant to mixed concurrent traffic. | Separate candidate. Our measured protected windows had zero prefill and host transfer; observed aggregate speculative counters did not change, while per-request speculative work remains unknown. The proposed trigger is not demonstrated there. |
| #69: absorb a short final message | Open, 95938ae1 |
Avoids an extra prefill graph for a short final segment; drops an intermediate checkpoint while preserving prompt-end caching. Author reports improved TTFT on four GB10 Sparks. | Worth a separate cold/chat-prefix test. Cannot explain the old zero-prefill warm-hit measurement, and could change the row-pressure experiment's checkpoint population. |
| #57: request lifecycle metrics | Open, 21a2030b |
Supplies aggregate TTFT/E2E/queue/prefill/decode/ITL and speculative counters we should not recreate. | Use if selected and qualified. It does not link an HTTP request to scheduler slot/generation or time its host-index call; those causal gaps remain. |
| #27: message rendering cost | Closed, superseded by 67ba265e |
Current main uses one-pass annotated rendering instead of repeatedly rendering conversation prefixes. | Do not revive the closed patch. Check retained GLM package-template compatibility before attempting GPU startup. |
| #29 / issue #23: multi-host disk cache | Merged | Shared CLI/Orbit forwarding changed; separate-host caching is disk-only. | Follow current shared serving-options path for any row-share knob. Our same-host TP2 memory pool is a different case. |
| Issue #47: snapshot-slot limit | Open | Related capacity symptom, but GPU snapshot slots on DeepSeek TP4, not the host slab row/page partition. | Cross-reference, not a duplicate resolution or proof of our mechanism. |
| Issue #14, LIL FlashInfer #43 | Open | Demonstrated NaN propagation from unused V-cache tails in paged attention. Dependency patch has a second reported partial Gemma reproduction. | Track, but GLM uses sparse MLA/DSA rather than the demonstrated paged route. No claim this fixes our GLM case; affected prepared packages need re-export. |
| #18: pinned export reads | Open, old implementation superseded | Current native export removed Python _Cuda.download and already uses pinned 8-MiB staging slots. |
Do not cherry-pick the old patch or equate export reads with runtime offload DMA. Current native export still needs its own GPU validation. |
| #64: test compilation | Open, cited omission already fixed | Current prompt.rs contains the missing initializer after e952c8a7. |
Do not cherry-pick. Static source resolution is distinct from running the full test suite. |
PR #8 primarily changes vision-window leasing, includes a terminal-work priority change, and reports its own latency tradeoffs. It is not a suitable bundle for this text-cache comparison. #59, #67, and issues #30/#32/#39 concern tool/parser failures, not the plain-text fixed-work mismatch. #19 corrects JSON-schema prompt freezing in an upstream qualification harness; use that work if adopting that harness, without relaxing prompt equality. Dense-Qwen proposals/issues #20/#44/#45/#46 and DeepSeek draft-head #65/Qwen #66 do not supply a GLM cache fix. Preparation changes #68/#70/#71 are included in the chosen source; they do not affect kernels already embedded in our old package.
What the review rules out¶
The known GLM selection-width export fix 22887c1 and page-table-extent change eb770324 already precede our old 47581cca source. They are not new remedies. The upstream GLM defect note also records a remaining timing-sensitive KDA/NaN failure at TP4/C16. That warrants keeping invalid-distribution failures, but does not establish the same cause at our TP2/C4. Its neighbouring-record answer category is a model miss, not the documented corruption defect.
A byte-exact Snapshot→Restore→Snapshot is insufficient to disprove #16: both snapshots can omit the same required draft page. Conversely, a draft-page omission does not prove target-logit corruption. Keep byte fidelity, complete state capture, numerical equivalence and sampled output repeatability separate.
Our protected bracket still lacks request-correlated attribution. No reviewed change has been measured on our rig as a remedy for its roughly 3-ms increase. Current upstream metrics can narrow a future run, but histograms cannot supply missing per-request timing retroactively.
Completed prerequisites and next experiment order¶
The retained-package compatibility check and current-upstream-plus-compatibility baseline are complete. The exact legacy template is recognized by the existing upgrader; four focused CPU tests and the release build passed. Six signed GPU startups, one smoke and all 391 diagnostic requests passed. The closed report, decision summary and raw closure retain the result and exact restoration. This does not describe unmodified current main: the frontend compatibility patch was present, while cache allocation and package graphs were unchanged.
Three 52-request reverse replays measured median TTFT 449.93/452.90/460.25 ms, each with 10 host fetch completions and 199,464 prefill tokens. The 42 host no_match outcomes per repetition can include GPU hits. All 192 measured one-token output/prompt/body comparisons matched; full speculative work is unknown. No direct current row/page pressure counter or 75% arm was measured. The historical long-output/protected failures remain unresolved.
- Review the separate compatibility PR now. It is a concrete startup compatibility correction, not an allocator or numerical fix.
- For allocation, compare current default 25% against a minimal startup-only 75% option, including the unchanged 25% patched-build control, three matched reverse-replay repetitions and protected small/page-heavy brackets. Preserve the physical budget/minimums and existing shared CLI/Orbit configuration path. The prior 14-incarnation/864-request plan remains conditional and was not executed in this refresh.
- To exercise #16 and #35, export packages with the same checkpoint/settings. Hold the chosen package fixed across allocation arms. Causal attribution to #16 requires matched current-main and current-main+#16 exports or a component oracle; old versus new packages change too much.
- Treat MTP-disabled cold/restored probes separately and preserve the earlier invalid-distribution/rank-divergence failure. A successful later probe cannot erase it.
- Evaluate #43/#69 only with their triggering workload. Reuse #38 counters; do not recreate #57 histograms. Add request-to-slot/generation and host-index spans only if the remaining attribution question needs them.
- Keep exact incumbent restoration and independent correctness gates for every outage. No default change, promotion or upstream submission follows from this diagnostic result.
The earlier CPU inspection used only the 2,645,178-byte primary index under a one-CPU/2-GiB build, with no CUDA/weights/network. It established the unannotated template SHA 0c4099f3382d6c92700dfb99725025360966fd73032f0ecf32377c0d9e6309c5; it did not itself verify signatures. The later six GPU startups separately supplied signed package/weight verification. The clean unmodified image built successfully, but the measured runtime is the compatibility image.
Contribution path¶
Repository AGENTS.md permits a focused PR for a demonstrated defect or measured problem; it does not require an issue first. The design review and truthful validation disclosure are required. The useful contribution is either residual row-capacity evidence and a small allocation control, or validation/feedback on an existing PR—not another implementation of #16, #38 or #57. The revised issue draft now includes the completed baseline. An allocation-code PR still needs a matched current 25%/75% result; the separate compatibility PR is ready for user review.
Detailed reviews: cache overlap, correctness, latency. No upstream message was sent.
Concrete contribution uncovered by the review¶
The exact-byte check found that today's 11,022-byte GLM template, after removing its six literal boundary annotations, equals all 10,950 bytes of the retained template. A full re-export is therefore not the only remedy for this frontend rejection: the existing legacy-template upgrader can recognize GLM using the same shared mechanism already used for Qwen, Step and Kimi.
A small compatibility patch is now prepared on codex/legacy-glm-template-compat-20261023, with meaningful CPU rendering/token/boundary regressions and the existing independent MiniJinja oracle. It does not modify the old signed asset or package graphs. The code review found no material source blocker, conditional on CPU tests and release build. Exact-byte feasibility and PR draft retain the details. All four focused CPU tests and the release build passed. The first build failed because its test command omitted runtime-package; correcting the command to upstream CI's feature set required no source change. Validation retains both attempts. This historical CPU receipt is complemented by the completed GPU baseline; its original CPU-only fields remain unchanged.
A focused fresh GitHub search for GLM/template issues and PRs found only output-parser/Gemma-related results, alongside the full inventory review; none duplicates this legacy GLM startup fix. The older statement that unmodified current main rejects the package still stands. A subsequent baseline using this patch must be labeled current upstream plus frontend compatibility, not unmodified upstream. PR #16/#35 still require re-export independently.
The compatibility patch is committed locally as 0b21d7f4883ac6f2b099df55813e8be06777b9df; its committed code matches the tested build context. It has not been pushed. The managed recipe and final collector passed review and the full bounded baseline. Operational driver, monitoring-contract and response-identity collector failures are retained separately in the report, including their exact restoration. The incumbent is restored, not promoted to the candidate. The final GitHub recheck still finds the same main commit and unchanged 71-item inventory.