GLM-5.3-Flash SGLang SM120 campaign friction log

Failures and fix-forward changes are recorded during the campaign. The first entry is the pre-run source audit, not a local runtime failure.

Time	Category	Command or stage	Observation	Workaround	Durable follow-up
2026-09-02T08:27:06Z	ambiguous-output	source translation	The stable release, quick-start command, launcher defaults, and retained lock metadata do not describe one byte-identical launch contract.	Pin one explicit local command and record every divergence before load.	Preserve the exact image, model, flags, environment, and startup log in the evidence bundle.
2026-09-02T08:29:00Z	manual-workaround	controller-first snapshot	The restricted controller returned HTTP 401 and the required user-local credential was not present. No mutation was attempted through it.	Use the exact worktree-backed local CLI with the resolved operator home; retain controller coverage as missing.	Do not add or copy a credential solely for this campaign.
2026-09-02T08:46:00Z	ambiguous-output	current-source refresh	Upstream main advanced beyond stable v0.1.0 to v0.1.1-rc.14; its digest-pinned registry image exists, while stack.lock.json and the README still label the candidate not built or upstream-qualified.	Pin the registry digest and source commit independently, treat all stable performance numbers as external priors only, and qualify rc.14 locally from zero claims.	Preserve both the tag-to-digest resolution and the source/documentation discrepancy in upstream-source-lock.json.
2026-09-02T09:26:28Z	environment-compatibility	131K no-spec first boot with upstream PCIe IPC allreduce enabled	Both TP ranks reached Torch distributed initialization, then their first CUDA allocation failed with CUDA driver error: unknown error; the container exited before weights loaded. The same pinned image subsequently enumerated and allocated on both GPUs in minimal CUDA probes with and without CUDA_VISIBLE_DEVICES.	Remove only SGLANG_ENABLE_PCIE_IPC_ALLREDUCE and SGLANG_PCIE_IPC_MAX_NUMEL, use a fresh v54 cache volume, and retry the otherwise identical no-spec profile through the managed recipe lifecycle.	Retain the enabled failure as the local WSL2 transport result; credit no performance to the upstream PCIe IPC extension unless a later exact build passes it.
2026-09-02T09:29:00Z	environment-compatibility	131K no-spec retry with upstream PCIe IPC allreduce disabled	The clean-cache managed retry reproduced the same CUDA 999 failure on both ranks before weight loading, proving that the optional SGLang PCIe IPC extension was not the root cause.	Unload through the managed lifecycle and isolate Torch distributed initialization in the same image.	Preserve the negative A/B; do not attribute the failure to the optional extension.
2026-09-02T09:30:31Z	environment-compatibility	exact-image two-rank NCCL collective	The default NCCL 2.30.7 path failed in ncclCuMemMapAndSetAccess with CUDA error 999. Repeating the same probe with only NCCL_CUMEM_ENABLE=0 completed rank-0/rank-1 all-reduce with value 2.0 over SHM/direct.	Add only the proven cuMem disable plus temporary NCCL INFO logging to a fresh v55 managed cache and retry.	Keep the local WSL2 NCCL control explicit in every surviving recipe; do not disable P2P or host cuMem without separate evidence.
2026-09-02T09:32:41Z	environment-compatibility	131K no-spec v55 boot and matched allocator probe	With NCCL cuMem disabled, the managed server still failed at torch.ones. The exact two-rank probe reproduced it only when the upstream PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True was present; expandable_segments:False allocated on both ranks and completed the all-reduce.	Change only the allocator setting to false, retain the proven NCCL control, use a fresh v56 cache, and retry through the managed lifecycle.	Keep the allocator override scoped to this WSL2 image/runtime and requalify if Torch, CUDA, driver, or image changes.
2026-09-02T09:40:24Z	environment-compatibility	131K no-spec v56 decode-graph capture	The managed serve completed distributed initialization, loaded the exact checkpoint, ran FlashInfer autotuning, and began batch-one decode-graph capture. Rank 0 then raised SIGFPE in Torch symmetric-memory rendezvous from SGLang's MultimemAllGatherer; rank 1 logged its own failed multimem fallback.	Add a hash- and anchor-guarded entrypoint patch that sets only the logits gatherer's enabled argument false, preserving its standard NCCL fallback, then retry with a fresh v57 cache.	Keep the source hash and postcondition fail-closed; remove the patch only after an exact upgraded runtime proves symmetric memory on this WSL2 TP pair.
2026-09-02T09:53:18Z	protocol-correctness	first complete no-spec preflight	The managed v57 serve became healthy and passed 100K-target retrieval, tools 20/20, streaming tools, and tool-result continuation. The checkpoint template unconditionally opened <think> and ignored enable_thinking=false, so hidden reasoning plus </think> remained in visible content and failed strict JSON and Responses exactness. The image itself was understood and transcribed correctly; the configured expectations belonged to a different fixture. The 100K nominal long-tool prompt measured only 84,543 tokens.	Derive a hash-guarded runtime-cache chat template that defaults thinking on but honors enable_thinking=false; restart with the same kernel cache. Rerun the correct image expectations and raise the long-tool nominal target to 120K so measured prompt tokens exceed 100K.	Retain the initial failures as harness/template evidence; do not mislabel the image or long-tool threshold misses as model correctness failures.
2026-09-02T10:02:09Z	protocol-correctness	corrected 131K no-spec preflight	The source-hash-gated derived template produced zero reasoning leakage with thinking disabled, and the corrected image plus 101K-measured long-tool checks passed.	Preserve the derived template hash, source hash, and fail-closed anchor in every surviving recipe.	Remove the patch only after the exact checkpoint template natively honors the requested thinking control.
2026-09-02T10:17:32Z	performance	matched 131K adaptive-MTP A/B	Adaptive EAGLE raised median decode 74.10 to 103.52 tok/s at 4K and 73.56 to 95.06 at 120K. Long-context prefill moved 5,926 to 5,759 tok/s, a 2.8% reduction inside the predeclared 3% boundary.	Advance adaptive MTP to context-envelope qualification.	Keep no speculation as the exact reliability/performance control.
2026-09-02T10:49:18Z	capacity-policy	exact upstream 499,712-token/C4 profile	Full functional preflight passed, but idle host-visible reserve was only 49 MiB per card. C4 short reached 147.54 aggregate output tok/s; four nominal-120K requests degraded to 148.86 s median TTFT and 0.61 aggregate output tok/s.	Reduce running requests, recurrent-state slots, graph batch, and then context while holding model/runtime/kernel controls fixed.	Retain the profile as policy-infeasible rather than interpreting a successful health check as qualification.
2026-09-02T11:00:00Z	capacity-policy	499,712-token C1 reduction	C1 raised host-visible reserve to 2,347 MiB per card, still below the 3 GiB requirement.	Continue the monotonic context reduction.	Keep the C1 profile policy-infeasible.
2026-09-02T11:21:43Z	capacity-policy	393,216-token C1 full qualification	Idle reserve initially passed at 3,351 MiB per card and all functional, deep-context, quality, image, and endurance work passed, but retained lazy/media/kernel allocations reduced post-workload reserve to 2,101 MiB.	Reduce the selected context pool to 245,760 and rerun the entire acceptance sequence.	Require post-workload rather than startup-only reserve evidence for recommended profiles.
2026-09-02T11:19:01Z	benchmark-integrity	first 393K coding-agent quality run	Four checks passed; the Windows safety-plan answer reached the 768-token visible cap with finish reason length. The validators did not indicate an incorrect answer.	Classify it as visible-answer budget exhaustion and rerun the unchanged suite at 2,048 visible tokens.	Retain both artifacts; do not score a budget truncation as a model-quality defect.
2026-09-02T11:40:50Z	qualification	245,760-token C1 final profile	All functional, bidirectional thinking-control, capacity, coding, image, and 60-request endurance gates passed. Post-workload reserve was 3,487 MiB on each card, 415 MiB above policy.	Select this profile as the verified local challenger.	Keep promotion separately human-gated; C2+ at long context remains unqualified.
2026-09-02T11:41:00Z	kernel-optimization	final candidate log audit	SM120 sparse-MLA CPB calibration rejected implausible constants and used the C++ heuristic, but the endpoint remained healthy and all end-to-end gates passed.	Retain the fallback as qualified behavior and do not create an unmeasured tune.	Treat an exact default-versus-tuned end-to-end A/B as a future optimization gate.
2026-09-02T11:52:26Z	restoration	managed exclusive-mode restore	The first restore attempt was refused before mutation for an incompatible split restore group. The correct transaction then reached health but router readmission returned 401 because the local process lacked its configured token; rollback preserved and stopped the incumbent.	Load only ANVIL_ROUTER_TOKEN from the user-local credential file into the restore process and repeat the managed transaction with the manifest-declared rollback group.	The retry restored the original retained container ID, image, model, loopback port, exclusive owner, router readiness, and empty shared-memory state.