Repository navigation
apr code eval E4: verdict + metric engine (pass@1, unbiased pass@k, Wilson CI, refusals) #5285
Closed as not planned
Description
Per-task verdict from test results (fail-to-pass all pass AND pass-to-pass all pass). Metrics: pass@1, the unbiased pass@k estimator (Chen et al. 2021), Wilson 95% CI, edit-format conformance rate, cost (tokens, wall). The engine refuses (no number at all) on 0 tasks, 0 executed tests, empty output or a sandbox error counted as a pass.
Acceptance: FALSIFY-ACE-VERDICT-* and FALSIFY-ACE-METRIC-* tests green.
Part of #5281. Spec: docs/specifications/apr-code-eval-v1.md (APR-CODE-EVAL-001).
keep-open: child of #5281
Activity
noahgift commentedon Oct 14, 2026
Follow-ups from the #5290 seat reviews (spec APR-CODE-EVAL-001, head c814c01), to fold into this ticket's build:
edits(out)is computed by parsing, a non-empty output that does not parse lands in EMPTY, so the¬parsesFORMAT_ERROR arm never fires. Define EMPTY apart from the parser (raw output is blank or whitespace only) and add a falsify row: non-empty unparseable output ⇒ FORMAT_ERROR, never EMPTY.Refs #5290
noahgift commentedon Oct 14, 2026
merged into #5314 (master ticket for epic #4000; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5342 (apr code eval: task suite, sandbox, metrics and nightly receipts (APR-CODE-EVAL-001)), split out of #5314 by theme.