Repository navigation
No committed parity receipt names an external comparator — the 0.98 threshold's own basis is apr-vs-apr self-consistency, not just the Qwen 3.5 rows #3576
Description
Measured 2026-09-20, answering a [U] raised in review: what do our parity receipts name as their comparator?
$ grep -rl 'comparator_sha' evidence/
evidence/parity/arb-apr-001/lambda-labs/qwen3.5-9b-q4_k_m.json
count: 1
One file. The one written today, an hour ago, after the review asked the question. No other committed parity receipt in this repository names a comparator at all.
The scope is wider than Qwen 3.5
The review suspected the Qwen 3.5 figures were self-consistency. They are — thresholds.yaml says so in its own measured_by:
qwen3.5-0.8b: measured_by: 'apr parity ./Qwen3.5-0.8B-Q4_K_M.gguf --prompt "$PROMPT" --json'
qwen3.5-9b: measured_by: 'apr parity ./Qwen3.5-9B-Q4_K_M.gguf --prompt "$PROMPT" --json'apr parity compares apr's CPU arm against apr's CUDA arm. But so does the default row — the 0.98 threshold that governs every dense model:
known-good
qwen2.5-coder-7b-instruct-q4_k_m@ lambda … apr 0.65.2 cuda … 0.998607 ×5
known-badqwen2.5-coder-1.5b-instruct-q4_k_m@ lambda … 0.950827 ×5
Both populations are apr-vs-apr. So the entire threshold system — the discriminating number as well as the readings judged against it — is self-consistency. Two of our own implementations agreeing. The 0.941807 incident was precisely a CUDA arm disagreeing with a CPU arm; agreement excludes that class and nothing beyond it.
Two different defects, which must not be conflated
| dense models (qwen2.5-coder, …) | Qwen 3.5 | |
|---|---|---|
can llama.cpp @ build_commit load it? |
yes | no — exits 1, unknown model architecture: 'qwen35', 0 hits for qwen35 in the pinned checkout |
| so the missing oracle arm is | a gap we chose — it was available and not used | a constraint — unavailable at the current pin |
The first is fixable today with no ruling. The second needs a decision.
Credit where the file is already honest
thresholds.yaml does not pretend. The one row that would be an oracle comparison — qwen3.5-0.8b@cpu-vs-llama.cpp — carries min_cosine: deliberately absent, measured_by: null, and a basis marked [U] explaining that both sides refuse the model. judge() fails closed on it. The file distinguishes the two comparisons by key; what was missing is that nothing carried that distinction into the receipts, so a reader of a receipt could not tell which kind they held.
What lands regardless of the ruling
comparator.kindbecomes schema-required in the receipt, as an enum —self|llama.cpp|hf-transformers— withcomparator_sharequired for every kind butself.comparator: {kind: self, comparator_sha: null}was the right record; it should be the schema, not a one-off.thresholds.yamlgainscomparator_kindper model row, so the key convention (@cpu-vs-llama.cpp) stops being the only place the distinction lives.- Historic receipts are re-marked, not silently reinterpreted: every existing Qwen 3.5 and dense figure gets
comparator: selfrecorded against it. They remain valid as what they are — CPU/CUDA agreement — and stop reading as oracle parity.
Consequence to state plainly: goal B's positive control as written ("differential corpus vs pinned llama.cpp; a corrupted kernel turns it RED") is unsatisfiable for the blessed model today. That is not a reason to weaken it — it is the thing the ruling has to fix.
Activity
noahgift commentedon Sep 20, 2026
The ruling this needs — and a fourth option the measurement opens
Three options were put forward for Qwen 3.5's missing oracle. The finding that the dense baseline is self-consistency too adds one that none of them cover, and it is the cheapest thing on the list.
llama_pin.tomlto a build supportingqwen35transformersreference logits, evidence path, never the gate paththresholds.yamlRecommendation: (d) immediately, (b) for Qwen 3.5 this week, (a) evaluated at the 0.70 cut.
Reasoning: (d) is free and fixes the threshold, which is the part every other judgement leans on — a discriminating number grounded in two of our own implementations is the weakest link, and it is repairable today with the pin we already have. (b) then gives Qwen 3.5 an oracle without invalidating anything. (a) remains the right end state but its cost is real and it should be paid deliberately at a cut, not absorbed mid-train.
(c) must not ship as if it were parity — if it is chosen, goal B's floor changes wording in the same PR.
Whatever is chosen, the three schema changes in the issue body land regardless:
comparator.kindrequired as an enum,comparator_kindper row inthresholds.yaml, and historic figures re-markedselfrather than reinterpreted.noahgift commentedon Sep 23, 2026
milestone → 0.71.0 per docs/roadmaps/milestone-triage-2026-09-23.md (#4024), operator approval 2026-09-23 (relayed by aprender-cf). Reason: model/backend correctness or the certified matrix: Don't Leave Behind (#3994)
noahgift commentedon Sep 26, 2026
Parked by APR-EPIC-001: over E6 budget (rank 5/5). Reopen = pull. (Swap for a CRUX-PERF child; no claim, no branch, no PR at park time.)
release(0.69): batch — 15 receipted PRs + fleet-pv andon fix, one CI …
PMAT-3685: apr parity REFUSES (exit 12) a model the CUDA capability g…