Skip to content

Replacement evaluation friction log

Status: Evaluation gates are terminal: no challenger qualified. DeepSeek r4 passed protocol and resource gates, then failed the first mandatory Oracle8 round. It was unloaded and the exact retained GLM passed direct, routed, and actual-Pi restoration checks. The final artifact set records independent reviews and the exact retained evidence.

Area Observed friction Disposition and evidence boundary
MiMo r4, Oracle 7 Two planning outputs used a visually similar non-ASCII arrow where the earlier prompt did not require literal ASCII ->. A separate structured-edit output wrapped otherwise-valid JSON in a Markdown fence. The arrow cases are instrument ambiguity, retained without rescoring. The fenced JSON is a genuine format defect. The r4 result remains diagnostic rather than a promotion result; its independent SWE result also fails the campaign gate.
Oracle 8 instrument The updated prompt explicitly requires literal ASCII ->; the scorer is unchanged. This is a prompt clarification for a new oracle revision, not a repair or reinterpretation of Oracle 7 observations. Future native artifacts must identify the oracle revision.
MiMo r5 sampling The r5 recipe separately set server generation defaults to temperature 0.0 and top-p 0.95. Pinned runtime source describes greedy normalization for zero temperature. Keep this configuration change distinct from the Oracle 8 prompt correction. It supports the recorded r5 configuration only and does not revise r4 results.
MiMo r5 SWE The native resolve rate is 1.0 only across the two graded submissions. Three assigned tasks lack submissions; the official grader is incomplete and the required Pylint sentinel failed. Report campaign outcome as 2/5 assigned tasks, not 2/2 graded-only. The denominator boundary rejects ranking or promotion from the native rate.
Mistral r2 The protocol run passed, but its measured memory policy exceeded the campaign ceiling. The r2 configuration is rejected on resource policy, regardless of protocol success.
Mistral r3 Reducing GPU memory utilization to 0.94 created bounded headroom, but the frozen nine-scenario, two-repetition Oracle 8 population had one parallel-tools failure: 17/18 in round one rather than the required 18/18 per round and 54/54 aggregate. The 0.94 change is a headroom remedy, not a reliability remedy. The configuration remains rejected; the retained native 17/18 pass flag must not be presented as campaign acceptance.
DeepSeek build r1 The cached compiler base reported Torch 2.14.0+cu130 where the build gate required Torch 2.13.0+cu130. Stopped before compilation; this is a toolchain-identity failure, not model quality evidence.
DeepSeek build r2 The exact toolchain gate passed, then compilation exceeded the managed 8 GiB memory cgroup. Resource bound raised to 16 GiB with no swap and one build job. This does not establish an SM120 source or kernel incompatibility.
DeepSeek build r3 Native compiles and CPU/import tests completed, but the final inspection helper imported the extension before Torch and failed to resolve a Torch shared library. The r4 build path imports Torch first for that lookup, retaining source pins and numerical assertions. CPU build correction is not GPU correctness evidence.
DeepSeek build r4 The bounded CPU build passed its recorded CPU/import and architecture gates. The separately retained Stage 2 r6 numerical gate passed. The CPU build result alone establishes no model quality.
DeepSeek GPU identity Runtime CUDA UUID text omitted the conventional GPU- prefix even though it represented the requested device. The harness now compares only a full canonical UUID with an optional prefix, preserving mismatch refusal. Later attempts verified the canonical identity before executing numerical groups; that closes the representation guard without claiming the full numerical gate passed.
DeepSeek JIT tmpfs The first numerical group passed; the next JIT-built shared object failed to map from the bounded /tmp tmpfs. Primary Moby source defaults to noexec, and the recorded mount request omitted exec; the old effective mount flags were not retained. The harness requests executable /tmp while retaining the 4 GiB cap, read-only root, nosuid, nodev, network isolation, limits, and cleanup. The r3 and r4 receipts proved effective executable tmpfs and completed the basic Engram shared-object checks. The complete numerical gate remained blocked by later, separate failures.
DeepSeek PID limit The r3 spin job exhausted its fixed 512-task cgroup (pids.current=512, pids.events.max=1) and reached its inner deadline. Frozen source requires two 256-worker tables plus runtime tasks; the captured 473 child threads are a post-exhaustion snapshot, not a baseline or exact peak. The reviewed fixed 1,024-task envelope retains before/after cgroup evidence. r4 completed the spin checks with zero task-limit events. The original wait site remains unlocalized; no partial r3 result transferred to r4.
DeepSeek Triton cache In r4, the next Engram/vLLM group tried to create /root/.triton under the read-only root and raised OSError: [Errno 30]. Seven failure markers were printed before the bounded output cap stopped the command; no completed JUnit count was retained for that group. The minimum correction sets Triton's supported cache directory inside the existing /tmp tmpfs and checks launch readback. Independent review verified the supported setting against retained pinned upstream source; the exact installed Triton revision remains unknown. The fresh r5 run passed the Engram/vLLM group (7/7) and registration (7/7), verifying the override on the pinned image. The final group then failed the separate strict skip gate; the complete numerical gate remained incomplete. This is a harness write-path failure, not demonstrated model or kernel numerical error.
DeepSeek optional benchmark selection r5 completed all six groups. The final pytest invocation reported 136 passed and six skipped, so the zero-skip guard rejected it. Four predeclared optional benchmark functions expanded to six nodes; absolute exclusion prefixes did not match collected relative node IDs. Independent source and environment review identified those six benchmark nodes. A new spec corrects only the four exclusion prefixes to cwd-relative IDs; source assertions, resources, and the zero-skip gate stay unchanged. r5 remains failed. The fresh r6 run selected exactly the prior 142-node set minus those six optional nodes and passed all 136, with zero skips or failures. Independent review accepted all six numerical groups; the result remains single-GPU kernel evidence only. Per-case JUnit identities were not retained in r5.
Engram microtimings Standalone source tests print host-node/spin timing ratios. Their fixture files are in the job's memory-backed temporary filesystem. Retain the raw output for reconstruction; it is not NVMe latency, serving throughput, end-to-end model speed, or comparative capacity evidence.
DeepSeek Stage 3 r7-r12 r7 stopped at host-memory admission; r8-a1 found its required metadata snapshot absent; r8-a2 reached dummy readiness but later proved an incomplete EXL3 residency proxy. Corrected r9 failed CUDA OOM in dense-mrow startup, r10 failed custom-all-reduce IPC initialization after capture progress, and r11 reached health but missed the frozen 4-GiB idle reserve. The narrow remedies were a bounded metadata pull, restoring original EXL3 controls, disabling dense-mrow, disabling custom all-reduce, and preserving the reserve gate. r12 then passed the bounded dummy-path integrity, containment, reserve, health, and one text-probe graph evidence gate. It remains dummy evidence only: no main-pack load, 327K, C4, image8, quality, throughput, or promotion claim transfers.
DeepSeek Stage 4 All six managed main-pack pull groups completed. The corrected r8 loader verified all 178 integrity pairs before engine start and passed the 64K/C1/image0 resource and basic 2/2 correctness gates. This is bounded real-weight evidence only. Full-role context, C4, vision, native quality, and promotion remain separate gates.
Pi selected-route acceptance An interim Pi fixture failed before its prompt because get_available_models emitted one catalog JSONL record larger than the client's 262,144-byte bound. get_state had already identified the selected provider/model and idle state. For the narrow selected-route read-turn acceptance, omit only the full-catalog request and its count; retain selected-state checks before/after, the exact read tool turn, agent_end, and settlement proof. Full-catalog propagation acceptance remains a separate contract and must keep its catalog digest check. This is a framing limit, not a model, provider, or prompt failure.
Router readmit invocation A direct .command attempt supplied only the command stem where the managed readmit surface requires explicit arguments. Treat this as a command-shape error, not an authentication failure. The subsequent managed preview and confirmed readmit receipt supply the operational evidence; no credential conclusion follows from the stem-only attempt.

Retained public native artifacts bind the completed results by basename: mimo-r4-oracle7.native.json, mimo-r5-oracle8-r1.native.json, mimo-r5-swe5-terminal.native.json, mistral-r2-protocol.native.json, mistral-r3-protocol.native.json, and mistral-r3-oracle8-r1.native.json. The final artifact manifest binds their exact sanitized bytes; the provenance ledger separately binds protected originals.

Stage 4 r7 addendum

Area Observed friction Disposition and evidence boundary
DeepSeek Stage 4 r7 metadata integrity The real-text r7 entrypoint stopped before engine initialization because its recipe literal for model.safetensors.index.json transposed the two characters at one-based digest positions 25 and 26 and failed checksum validation. The authoritative artifact index and pinned Hugging Face LFS digest agree on d7368ca2c8ac48554b5a71e7aeca4bf330ebdf51d3929832cbc3c12ea7b50dc0; the five other critical metadata checks passed. This is a recipe-metadata integrity failure, not model, engine, quality, or resource evidence. The authoritative terminal receipt, logs, status, and managed unload receipt are retained as raw/deepseek-stage4-r7-terminal.json, raw/deepseek-stage4-r7-logs-failure.json, raw/deepseek-stage4-r7-status-failure.json, and raw/deepseek-stage4-r7-unload-apply.json. The independently reviewed r8 correction passed all 178 integrity checks, resource/readiness checks, and basic 2/2 correctness before managed unload.
DeepSeek Stage 4 r7 containment observation After the early entrypoint exit, managed status could not obtain cgroup values and reported RAM/swap as unavailable. Its recorded oom_killed field is false. Treat the containment observation as unavailable after early exit. It neither proves OOM nor ineffective configured memory limits.
Model-cache inventory topology A cache-inventory invocation with optional topology selectors was refused because the selected target declared zero model-catalog owners. The documented direct-local named-volume inventory then succeeded. The successful local inventory used the supported direct-local function; it did not bypass topology authority. Retain the refused preview and the local inventory receipt as raw/deepseek-engram-index-pull-preview.json, raw/deepseek-engram-index-pull-preview-topology-friction.md, and raw/deepseek-stage4-r7-main-cache-inventory-local.json.

Evaluation packet preparation

Area Observed friction Disposition and evidence boundary
Full-role recipe preparation The first draft raised batch tokens from 2048 to 8192 without measured need. Rejected before execution; r2 preserves 2048. No outcome transfers from an unexecuted draft.
Workload binding Reviews found stale incumbent bindings, incorrect whole-job contract hashes, incomplete C1 and stability declarations, and protocol sampling labels that did not match the runner. Relative output paths also depended on cwd. Corrected before execution, with rejected versions retained. Fresh identity, absolute execution paths, canonical hashes, and independent review remain mandatory.
Global CLI option audit A leaf-help audit incorrectly classified --json as unsupported. Top-level CLI source proves that the global option is removed before leaf dispatch. The rejected audit is retained; no valid flag was removed.
Paired oracle control Existing GLM 54/54 artifacts use Oracle 7; the prepared Oracle 8 GLM control has not run. Do not reuse Oracle 7 as the paired Oracle 8 result. Run the fresh paired control only if a challenger survives to the comparison gate.
Benchmark source binding The prepared command map selected the operator checkout while job metadata named frozen benchmark revisions. Source inspection shows campaign_source is metadata only and detached workers inherit the submitter working directory and environment. Corrected before requests: each benchmark runs from its exact frozen checkout with the selected interpreter and no PYTHONPATH override. Operator lifecycle commands remain separate.
Retrospective launcher provenance Earlier MiMo, Mistral and GLM Oracle receipts name the intended source revision but do not retain worker CWD, PYTHONPATH, imported-module path or digest. MiMo SWE separately binds the official grader adapter, not the entire agent launcher. Keep observed outcomes and intended contracts, but do not claim uniform proof of the actually imported source. Fresh qualification requests require an execution/import receipt; no historical result was rescored or treated as a matched Oracle8 GLM control.
Full-role r2 active reserve The 2-GiB KV configuration started with 4,615 MiB/card and answered both basic checks correctly, but active sampling fell to 4,087 MiB/card, below 4,096. No OOM or swap occurred. Exact r2 failed resource policy and was unloaded. The 1.5-GiB KV r3 successor was tested separately; no result or capacity outcome transfers.
Full-role r3 active reserve The 1.5-GiB KV configuration passed basic 2/2 and model-aware 5/5 protocol controls, including its 20-request tool batch. Sampling inside the protocol window reached 3,961 MiB/card against the 4,096-MiB requirement; no OOM or swap was observed. Exact r3 was unloaded and rejected on resource policy. R4 changes only CPU weight offload from 4 to 6 GiB per GPU, retaining all role and containment settings. R4 subsequently passed the measured reserve gate; no latency or host-cache improvement was established.
Bound-packet metadata Independent review caught stale embedded container identities, superseded scenario fingerprints, and status text copied from inert preparation. Versioning an executed basic plan also required a historical-path correction. Rejected drafts and the immutable executed plan remain retained. Execution progressed only through separately accepted gates. Later packets keep workload contracts immutable and record progress in external receipts.

Final full-role decision

Area Observed friction Disposition and evidence boundary
DeepSeek r4 Oracle8 Round one passed17/18. In debug-loop repetition0, the model first called run_tests(scope="all"), then corrected to unit, making five calls where the frozen prompt required exactly four ordered calls. Independent review verified a genuine exact-argument/order failure. Preserve the complete native observation; no rescore, retry, or rounds2/3/downstream finalist work. This rejects only the exact configuration.
DeepSeek r4 resource accounting The continuous monitor collected934 samples across idle and active periods. Basic, model-aware, and Oracle8 active windows contain14,54,and100 samples respectively. Whole-monitor counts are not active-workload counts. The exact windows and native controls are retained; resources passed but do not excuse the reliability failure.
Benchmark logs CLI A bounded logs --cursor 0 --limit 100 call with global --json was rejected by the follow-output policy. Omit only global --json for this leaf; bounded logs already return their native JSON object. The rejected read did not affect the running job.
R4 public projection The first sanitized draft omitted the complete failed observation and executed budgets. Rejected before publication. The additive v2 retains all18 observations and controls, removes only private history/identifiers and raw reasoning, and passed independent projection review.
Final incumbent restoration Exact retained GLM passed direct5/routed6 and an actual Pi read turn with122 events, one read call, one agent_end and post-end idle settlement. Restoration preserves the original52GiB/1MiB swap allowance and grandfathered GPU reserve. It does not constitute fresh challenger qualification, full-catalog validation, or provider migration.