Skip to content

Independent coding source review

Date: 2026-10-16 (America/Los_Angeles)

Reviewer: GPT-6 independent quality-critic session. The coding outputs were generated by the evaluated Qwen QUASAR and Swift15 candidates; the fixture/grader implementation was authored separately with GPT-5.6 Terra.

Execution authorization and scope

All 21 visible Python source blocks in each retained matched-32k artifact were read before any offline execution:

  • quasar-text32k-coding-raw.json — SHA-256 dead4f3902454f62fed20bbbcc965e10a883178ccc600e5c28583b7347770c22
  • swift15-text32k-coding-raw.json — SHA-256 e5f0d4104e217d3bdfaf2f9556a4a383788787d4e2236dd0408d3d7c4439b30c

Every reviewed block was a bounded pure-Python function. None imported modules, accessed files or networks, traversed private/dunder objects, invoked arbitrary callables, or contained unbounded work for the frozen finite inputs. Offline grading was therefore authorized with --reviewed --expected-repetitions 3. No model or endpoint was called by the grader.

Both raw artifacts matched frozen coding-fixture/suite.json SHA-256 3f7f061135313c81e9002f9971f682ee80d03ae485f566cfdb2fd17ea9b7c57a. Each contained exactly 21 native attempts with status=passed, finish_reason=stop, failure_class=null, non-empty reasoning_content, and per-attempt budgets of 4096 visible-answer plus 8192 reasoning-headroom tokens.

Frozen list-probe grades

  • QUASAR: 119/123 cases and 20/21 whole solutions. The only failed solution was stable_toposort repetition 2, which raised TypeError when the grader's JSON IPC delivered edge pairs as lists.
  • Swift15: 120/123 cases and 20/21 whole solutions. The only failed solution was decode_uvarint repetition 1 (7/10 cases); it accumulated varint groups in big-endian order and failed three behavioral cases.

There were no native collection failures, source rejections, harness incompatibilities, execution timeouts, sandbox errors, or input-mutation failures in either grade.

Tuple-edge representation sensitivity

The frozen prompt describes edges as “a list of (before, after) string pairs,” while JSON IPC represents pairs as lists. The frozen list-probe files remain unchanged. Separately named post hoc sensitivity files reconstruct only stable_toposort edge pairs as tuples immediately before candidate invocation, while preserving the same five cases, expected outputs, prompts, and retained model responses.

  • QUASAR tuple-edge sensitivity: 123/123 cases and 21/21 whole solutions.
  • Swift15 tuple-edge sensitivity: unchanged at 120/123 cases and 20/21 whole solutions.

The QUASAR list-probe miss is therefore a representation ambiguity/list-robustness result, not an unconditional algorithm failure. Swift15's uvarint failure is representation-independent in these probes.