apr code eval: task suite, sandbox, metrics and nightly receipts (APR-CODE-EVAL-001) #5342

Scope: The implementation stages of the apr code evaluation (spec work in #5282): task-suite schema and loader, a hermetic sandbox runner, the verdict and metric engine (pass@k, Wilson CI), the agentic driver, contamination control, and receipts from a nightly CPU/GPU run.

Why it matters: apr code needs a measured pass rate before any claim about coding quality; the spec defines it, these rows build it.

Done when: A nightly run on CPU and GPU writes a receipt with pass@1 and its confidence interval over the dated task suite, run in the sandbox with no network.

Epic: #4000. Folded under triage rule 7b (2026-10-14): each row below was its own issue, closed as not planned and kept here as a sub-ticket. Reopen an issue when its work starts or its PR opens.

Sub-tickets

• apr code eval E2: task-suite schema + loader (repo tasks, function tasks, edit tasks, terminal tasks) #5283 apr code eval E2: task-suite schema + loader (repo tasks, function tasks, edit tasks, terminal tasks) (milestone none); branch la-71/5283-code-eval-build
  • accept: Acceptance: FALSIFY-ACE-TASK-* tests green.
• apr code eval E3: hermetic sandbox runner (no network, pinned digest, limits, fixed seed) #5284 apr code eval E3: hermetic sandbox runner (no network, pinned digest, limits, fixed seed) (milestone none)
  • accept: Acceptance: FALSIFY-ACE-SANDBOX-* tests green, including a planted network call that must fail.
• apr code eval E4: verdict + metric engine (pass@1, unbiased pass@k, Wilson CI, refusals) #5285 apr code eval E4: verdict + metric engine (pass@1, unbiased pass@k, Wilson CI, refusals) (milestone none)
  • accept: Acceptance: FALSIFY-ACE-VERDICT-* and FALSIFY-ACE-METRIC-* tests green.
• apr code eval E5: apr code driver — agentic loop + edit-format adapters #5286 apr code eval E5: apr code driver — agentic loop + edit-format adapters (milestone none)
  • accept: Acceptance: a fixed-seed run of the smoke suite produces a receipt byte-identical across two runs.
• apr code eval E6: contamination control (dated tasks, canary, train-set overlap) #5287 apr code eval E6: contamination control (dated tasks, canary, train-set overlap) (milestone none)
  • accept: Acceptance: the receipt carries the post-cutoff score, and a planted leaked task gets flagged.
• apr code eval E7: receipts + nightly CPU/GPU run via the fast-verb matrix (#5254) #5288 apr code eval E7: receipts + nightly CPU/GPU run via the fast-verb matrix (ci(lab): nightly fast-verbs CRUX smoke + tok/s on the ARM64 CUDA edge runner, ranked gap list #5254) (milestone none)
  • accept: Acceptance: two consecutive nights produce receipts, and the comment ranks tasks by failure.

Merged in (TRIAGE-100, 2026-10-14)

• apr code eval E1: ontology contracts + spec (task, sandbox, verdict, metric) #5282 — apr code eval E1: ontology contracts + spec (task, sandbox, verdict, metric)
  • done when: pv validate is green on all four contracts; a C314 quorum receipt is bound to the PR head.