Campaign failures and fix-forward dispositions¶
Closed means the investigation has a recorded disposition, not that the model passed. Failed attempts remain in native evidence.
| Observation | Owning layer and disposition | Durable evidence / verification | Status |
|---|---|---|---|
| Earlier tool fixtures omitted literal edit and planning requirements. | Harness: oracle 7 discloses exact arguments and planning output. Older ambiguous scores remain historical. | Product PR #567; independent review; 9559 tests passed, 53 skipped; focused 55-test gate. | Closed: source correction merged. |
| Requests smoke exhausted 40 calls without submitting a patch; old native output called it a test failure. | Harness: preserve LimitsExceeded as model budget failure and still invoke the official grader. Do not rewrite historical artifacts. | Product PR #567; native/baseline-swe-native.json; SWE regressions. | Closed: correction merged; model failure retained. |
| Docker may omit a tag from a repository digest. | Identity: permit optional omitted tags only with exact repository and digest; preserve registry ports. | Product PR #567; identity tests and independent review. | Closed: source correction merged. |
| Qwen r1 exceeded 52 GiB host RAM during warm-up. | Recipe: r2 moves PLE embeddings to GPU with VLLM_PLE_CPU_OFFLOAD=0, keeping containment. Initial OOM remains a failed configuration. | native/qwen-r1-kernel-faults.json; configuration/qwen-recipe-r2.toml; r2 preflight and memory status. | Closed: r2 loads; r1 rejected for this envelope. |
| GLM Spark recovered from a small allocator mapping failure during startup. | Runtime watch: thin transient headroom; require repeated cold starts before reconsidering this exact profile. Health does not prove startup reliability. | review/glm-spark-startup-independent-review.json; observed feasibility receipt. | Deferred: quality gates failed; no promotion. |
| GLM Spark counting streams overshot or reached the output cap. | Model/request: temperature 0.6 failed count recovery; both temperatures failed strict JSON. Preserve failures and exclude their throughput. | Both strict-capacity and oracle-7 native artifacts; independent policy review. | Closed: bounded repair failed; not qualified. |
| Qwen reasoning exhausted strict-cell budgets; medium exposed separator/count errors; disabled thinking still miscounted. | Request/runtime: two native repairs failed. Later raw probe confirms two leading newlines for that request only. No supported Rust reasoning selector in this pin; no router trimming. | configuration/qwen-parser-boundary-research.json; medium/disabled plans and native results. | Closed: medium is a coding lead only. |
| GLM Spark submitted only three of five patches; Qwen medium submitted five but failed Pylint. | Model: count all five attempts. GLM hit two 60-call limits. Qwen accepted malformed regular expressions. These are not transport failures. | Five-task native results, official reports and independent SWE reviews. | Closed: bounded quality failures retained. |
| Long guides passed markers but violated length or integration requirements. | Validator scope: chapter markers and syntax are diagnostics, not executed application correctness. Preserve separate 32K truncations and 64K trials. | Native long-output artifacts and independent critiques. | Closed: no broad long-session or coding claim. |
| Offline product tests overlapped part of GLM Spark agent execution. | Measurement: retain coding outcomes but withhold a controlled elapsed-time ranking. | review/glm-spark-swe-scout5-co-residency.json. | Closed: performance limitation recorded. |
| Private evidence umask affected unrelated fixture permissions. | Test setup: retain 0700 evidence directory; use normal 022 permissions inside isolated test environment. No source security workaround. | Final full-test receipt and independent product review. | Closed: required checks passed. |
| Historical bounded logs call reported a follow/JSON conflict. | Command use: current module-bound models recipes logs --since 1m --tail 1 --json succeeds without --follow. Earlier cause unresolved; no reproducible defect claimed. | review/bounded-logs-readback.json. | Closed: supported readback verified; revisit if reproduced. |
| Generic evidence inspection rejects native SWE stage schemas. | Existing surface: use digest-bound eval benchmark swe artifact. Do not convert or discard native schemas. | Managed Qwen medium readback and retained native SWE evidence. | Closed: per-suite reader works; generic discovery outside scope. |
| Incumbent Django official report marks one passing test as failed in addition to a genuine failure. | Upstream grader watch: retain raw output and official JSON at pinned grader revision f7bbbb2c. Do not use this report for per-test pass counts; the genuine admin Unicode failure independently preserves the unresolved task outcome. | native/baseline-swe-reports/ and review/baseline-swe-scout5-independent-review.json. | Closed for campaign scoring; upstream per-test reporting discrepancy retained for future investigation. |
Scoped process lesson¶
The material correction belongs to the product fixture and failure-classification layer, delivered in PR #567. Existing qualification instructions already require explicit contracts, independent review, immutable failures and supported trials. No skill text was changed and no behavioral gain is claimed for new instructions. Revisit if the oracle prompt/scorer contract or SWE termination metadata changes.