Evaluator: not_attempted conflates 'never ran' with 'comparator couldn't extract a value', and the deterministic grader overrides evidence-backed verdicts to 0.0

[laoliu5280]

Summary

The verify phase scores a claim not_attempted (value 0.0, counted in the denominator) in two very different situations that it cannot tell apart:

1. The experiment genuinely never ran — no evidence was produced (a fair failure).
2. The experiment ran and produced evidence, but the deterministic grader could not extract a clean number from it — a figure instead of a numeric table, an unparsed paper_range, a value present in prose/JSON in an unexpected shape, etc. This is a tooling/extraction gap, not a reproduction failure, yet it is scored identically to (1).

Worse, _apply_deterministic_grade overrides the LLM comparator's evidence-based verdict with not_attempted whenever it can't produce its own number — discarding a match/partial the comparator already justified. The comparator's reproduction reasoning is then preserved in the rationale, so the file's status and rationale openly contradict each other.

At scale this dominates the Replication Score and makes it a large under-estimate of paper reproducibility.

Where it happens

• src/veritas/core/grading.py returns ("not_attempted", ...) whenever the comparator's extracted value is absent or unparseable:
  • _grade_scalar L138–139 (value_found=False), L152 (replicated or paper value is missing / non-numeric)
  • _grade_scalar_range L169–170, L174 (replicated value or paper_range missing / malformed)
  • _grade_table L214–215 (no replicated table was produced)
• src/veritas/core/runner.py::_apply_deterministic_grade (L1649–1681) overwrites verdict.status with the grader's output for every scalar | scalar_range | table claim (L1677), even when the comparator proposed match/partial. The old verdict survives only as text: rationale = f"[deterministic grade] {why}. Comparator notes: {comparator_rationale}" (L1680).
• src/veritas/core/verify.py then scores not_attempted = 0.0 and keeps it in the denominator (only not_applicable and missing-file claims are excluded), so an extraction failure is charged against the paper exactly like a real no_match.

Evidence (105 ICML 2026 oral runs)

• 1,022 of 1,713 scored claims (59.7%) are not_attempted = 0.0 — they dominate the ~0.20 median score far more than real failures (only 92 claims, 5.3%, are no_match).
• 674 of those 1,022 (66%) were produced by the deterministic grader ([deterministic grade] in the rationale).
• 43 are cases where the LLM comparator explicitly proposed match/partial/no_match and was overridden to not_attempted.

Concrete status vs rationale contradictions:

• characterizing-evaluating-and-optimizing / C10 — status=not_attempted (0.0), rationale: "assets/reasoning.png exists and is structurally identical to Figure 5 … all five rows above 0.5." The figure reproduces the claim; it scored 0 because a table-typed claim's evidence was a PNG.
• controlled-llm-training / C14 — status=not_attempted (0.0), rationale: "re-ran the full benchmark on an A100 … exit_code 0, producing results…" — the run executed and produced data; it tripped paper_range missing / malformed.
• ai-engram / C9 — status=not_attempted (0.0), rationale notes the run "reproduces the main paper conditions (plain a=0.6 → ~0.703, adaptive a=1.0 → ~0.810)" — only the specific ablation table wasn't extracted.

Why it matters

The Replication Score is meant to measure whether the paper reproduces. Right now it also silently absorbs:

• our comparator's inability to read figures / non-standard output formats,
• claims blocked by our infrastructure (OOM, rate-limit, timeout) that never produced evidence, and
• authors shipping no artifacts,

all collapsed into one not_attempted = 0. Downstream consumers (the score, the report, external analysis) cannot separate "the paper failed" from "our tooling/compute failed," so the headline reproducibility number is understated by an unknown, material amount.

Recommended direction: make the evaluator an agentic subagent that grades dynamically

The root problem is architectural: a fixed deterministic matcher cannot cope with the open-ended shapes real replication evidence takes — figures, prose, logs, JSON in arbitrary schemas, values that need light computation or unit/format reconciliation. Every case above is the matcher hitting evidence it can't parse and bailing to not_attempted = 0. Patching the matcher case-by-case will keep losing this game.

Instead, the evaluator should be an LLM subagent that grades each claim dynamically, the way a human referee would. For each claim it gets the claim + its verification instructions, the paper's reported value, and whatever evidence the run produced (numbers, tables, figures, logs, notebooks), and it decides match | partial | no_match | not_attempted with a cited justification — reading a figure when the result is a plot, reconciling formats/units, and doing the small reasoning a rule table can't.

To keep the auditability/independence that motivated the current deterministic split (notes/2026-06-01-verifier-split-design.md):

• keep extraction and grading conceptually separate, but let the grader be an agent, not a rule table;
• require the evaluator to cite the exact evidence and show its comparison (the value it read, the tolerance reasoning) so the verdict stays reproducible and reviewable;
• reserve not_attempted strictly for "no evidence was produced" — never for "evidence existed but I couldn't parse it."

Under an agentic evaluator the specific bugs above dissolve: the figure gets read (characterizing/C10 → partial/match), the executed benchmark gets graded (controlled-llm/C14), and the comparator's evidence-backed judgment is no longer thrown away by a matcher that lacked a number.

Supporting changes that still apply:

1. Distinguish "no evidence produced" from "evidence produced but ungradeable/blocked." A structured not_attempted_reason ∈ {no_evidence, blocked_infra, authors_missing_artifact} so the score can treat them differently.
2. Fix the scoring denominator (verify.py). Exclude our-infra-blocked claims from the denominator (like not_applicable), so the score reflects the paper's reproducibility rather than our compute/extraction limits. Keep genuine no_match and authors-missing-artifact in.
3. Record grading/blocking provenance on every claim (did the experiment run? was evidence produced? what did the evaluator read?) so a run is auditable and re-scoreable.

Impact of fixing

On the 105-run corpus, a large share of the 1,022 zeros are extraction artifacts or our-infra blocks rather than genuine non-reproductions, so the current median (~0.20) materially understates reproducibility. Separating these cleanly is a prerequisite for an honest Replication Score.

---

Filed from an audit of all 105 finished ICML-2026-oral runs; per-claim justifications available on request.