Qualification reflection¶
This campaign clarified a discipline that is easy to state and hard to keep: a model result is only about the model after the serving path, request template, admission policy, grading harness, resource envelope, and evidence custody are all known to be doing what the record says. Several early results were useful precisely because they did not meet that standard. They identified a broken launch, a wrong artifact interpretation, an unavailable lane, or an incomplete observer; they did not establish that a quantization or model family was better or worse.
The terminal Ignis fixed-five SWE run reached five attempted cases, two officially graded and resolved cases, and three empty cases. Django resolved, but Pylint was ungraded with an empty patch, so the required sentinel pair was not met. The outcome is incomplete on the fixed denominator and below the 4/5 floor; it is not a promotion result and does not turn empty cases into a score. The previously queued Qwen top-k-20 log R2 was deliberately left unrun, and the optional GLM T1 SWE branch was cancelled. Top-k-20 is therefore a prepared control, not a demonstrated quality or speed improvement.
The scorer is intentionally narrow: five fixed SWE-bench Verified tasks, a pinned agent and grader, independent official grading, complete grading, and a 4/5 floor. That fixed denominator protects the campaign from cap fishing and from swapping in easier cases after an inconvenient result. It is also not a general measure of model quality. The retained baseline scored 2/5. It remains the continuity and contract baseline, not a universal-quality winner. The current restore recommendation is consequently conservative: retain the last-good non-abliterated official-base GLM TR3 EXL3 4-bpw serve; its BF16 teacher is lineage evidence, not the serving precision, with the APC/no-spec, 327K, C4, vision, and structured-output contract. R19 independently accepted the exact restoration identity and direct/routed/client evidence, with the thinking-off client gap preserved. No unmeasured kernel setting, sampler setting, or speedup belongs in that recommendation.
The strongest positive learning was about separating model behavior from runtime behavior. Quality preflight, structured-output probes, tool calls, streaming, and SWE execution all exercise different contracts. A candidate that can answer a short control has not shown it can admit concurrent work; a candidate that loads has not shown it can keep the required context; and a quality score cannot repair a timeout, a malformed parser record, or a grader map. Ignis R4 is a good example. It has six of six serial C1 protocol observations and twelve of twelve strict quality observations, while its broader preflight has six passes from thirteen observations because seven requests received immediate 503 admission refusals. Its JSON_OBJECT boundary returned 400 unsupported_value. Those are useful, bounded runtime facts. They do not establish full-context behavior, C4 behavior, vision or Hindsight behavior, performance, or promotion readiness. The separate terminal SWE record is 2/5 overall: two graded and resolved, three empty. Twenty-four recovered SWE engine-full 503 retry notices are neither quality observations nor evidence explaining the empty patches or 60-step churn. The ignis-health-r4-preflight-native.json, ignis-health-r4-c1-preflight-native.json, ignis-health-r4-json-object-boundary-response.json, and ignis-health-r4-quality-r1-native.json records should be linked together so the partial success is not detached from its contract limits.
MiMo made the same point from the resource side. The first selector chose zero offload; a later selector chose 2 GiB. The later conversion still stopped on policy/trust grounds. That sequence fixes a selector defect, but it does not demonstrate that a different quantization is causally responsible for the observed outcome. Changes to template, engine, quantization, offload, resource reservation, and admission were too entangled to assign bit-width causality. The retained mimo-mopd-nvfp4-tp2-32k-failure-decision.json and mimo-routed-offload-r6-failure-decision.json should continue to say policy failure, not physical non-fit or model-quality failure. Similarly, max cache events are a distinct observation from an OOM or OOM kill; they cannot be silently converted into an OOM classification.
Several durable fixes reduced accidental campaign work without widening the claims. PR 652 made the zero-swap containment path available after a swap-threshold admission error. PR 653 separated a quality timeout from a quality result. PR 654 repaired Xet acquisition. PR 656 added the optional non-root cache mount and its managed candidate path was used; global installed-CLI state remains a separate fact. The schema-aware Tabby operator patch 997 was tested through nine CPU cases plus typed streaming, full-13 protocol evidence, and quality-12 evidence. PR 656 was merged and its reviewed source CLI was actually used for the managed Ignis candidate; the global installed CLI remained unchanged. The schema patch also has bounded native protocol verification. Neither fact implies a production route, promotion, or an optimized model setting. Every final finding should preserve that four-part boundary: code/source, installed artifact, deployed configuration, and live-verified behavior.
The campaign also accumulated errors that should shorten the next one. The initial Ignis attempt lacked a conversion sidecar and later exposed a health-path 404; repaired recipes made R4 readiness testable, but not complete. Observer state, unwrap, and an extra -- launch defect produced ambiguous records. An invalid SWE paired-image map invalidated its binding. The CLI help/parser mismatch made a preview flag look supported when the frozen parser rejected it. The native evidence reader could not consume a retained native schema. Linux host-memory support was WSL-only. A quiesced primary tier unexpectedly readmitted and the actor remains unproven. Each deserves a durable ticket/watch disposition, rather than a retry being recorded as a fix. The closure friction log links the relevant basenames, including ignis-trellis25-tp1-131k-failure-decision.json, ignis-sidecar-r3-health-path-diagnostic.json, deepseek-swe-paired-image-map-erratum.json, and router-quiesce-drift-analysis.md.
Evidence publication became a correctness problem in its own right. R12 and R13 accepted an over-broad public projection that corrupted semantic slash-bearing strings. R16-r2 and R17 repaired the affected staging with stronger structural and semantic checks; later reviews required multiple iterations, including six cache/projection checks, before R18/R19 were accepted. The lesson is not merely to redact more carefully. A projection checker must preserve schema keys, types, list order, valid model and dataset identities, public URLs, image and artifact hashes, and known private aliases at the same time. Numeric-only equality was insufficient. Publication acceptance must remain independent from the producer, and a public-safe bundle is still not source, deployment, or live acceptance.
The source set was productive when used as a prior rather than a benchmark replacement. The campaign compiled 100 public sources, selected for model, runtime, and hardware relevance. Private Discord material remained discovery-only. The next campaign should start with the exact official lineage, artifact revision, serving contract, hardware envelope, and one small fixed task matrix, then eliminate infeasible candidates before loading them. It should run one baseline control, one contract probe, and one independent quality gate before any long SWE job. It should record every failed admission with the owning component's earliest error, then change one dimension at a time. Model parameter changes need evidence; safe operations, cache, and source fixes can be applied when they remove a demonstrated defect, but blind tuning is not learning.
The shorter effective campaign is therefore not a smaller gate. It is an earlier stop rule. Fix the command surface and evidence reader before asking them to certify results. Freeze the baseline and sentinel tasks before candidate work. Keep context, concurrency, modality, structured-output, memory, quality, and grader outcomes in separate cells. Require a terminal denominator before comparing candidates. Start the expensive SWE run only after the candidate has passed the relevant protocol and quality gates under the same retained contract. Preserve the strict no-cap-fishing policy. The terminal Ignis run did not meet those gates, so retain the exact last-good GLM identity with the accepted R19 restoration proof and its explicit thinking-off limitation and report the gap without inventing a runner-up claim.
R19 proves bounded restoration: direct 13/13; routed tools, SSE, JSON_OBJECT, vision/OCR; Pi settled read-tool acceptance; Hermes; and OpenClaw high 9/9. OpenClaw thinking-off failed twice with reasoning visible, while max was rejected locally without a query. Unified evaluation remains non-green where it forces thinking-off. Hindsight recall and retain passed; default cloud providers, routes, and native memories were preserved. The restore command timed out externally at 60 seconds, was not retried, and the exact retained container was subsequently healthy/ready. No new engine, kernel, or sampler optimization was applied. Baseline containment remained 52 GiB RAM plus 1 MiB swap, with 745,472 bytes of swap observed; startup free GPU memory was 2,974 MiB/card and later 1,875/1,897 MiB, below the separate 4 GiB candidate floor. This is restoration acceptance with stated limits, not blanket client acceptance or a new performance claim.
Cleanup added a final lesson about operation contracts. Eleven exact model-cache snapshots and 106 Docker image IDs were removed with managed preview/apply and absence checks. The retained 50 images have container, configuration, or child-image references; active and declared rollback/library model assets were protected. The cleanup summary separates 624,806,192,173 reported logical bytes for Docker caches from 402,592,907,264 native allocated-plan bytes, and does not add them into a filesystem-space claim. A default mutable cache-inspector image was acquired during a read-only probe and then removed; subsequent inspection used an already cached pinned helper. Pin the helper before cleaning its image cache.
The native removal guard also needed a real root-cause correction: a persistent Hugging Face lock path is not proof of an active download. Checking the actual nonblocking POSIX lock and holding its descriptor through deletion allowed safe cleanup while retaining the lock inode that waiting downloaders share. Parsed configuration dependencies, aliases, generic YAML, and read-only host observers needed semantic auditing rather than filename exceptions. Independent source review, fresh guards, and post-inventory checks were required before applying that repair. Preserve this as source/operation evidence; it did not optimize model quality or change the installed router, controller, providers, or model profile.