Repository navigation
apr code eval E6: contamination control (dated tasks, canary, train-set overlap) #5287
Closed as not planned
Description
LiveCodeBench practice: every task carries a release date, and a run reports its score split by the model's training cutoff. A canary GUID string is embedded in every task. An n-gram overlap check against the training manifest flags tasks whose solution text may have leaked.
Acceptance: the receipt carries the post-cutoff score, and a planted leaked task gets flagged.
Part of #5281. Spec: docs/specifications/apr-code-eval-v1.md (APR-CODE-EVAL-001).
keep-open: child of #5281
Activity
noahgift commentedon Oct 14, 2026
merged into #5314 (master ticket for epic #4000; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5342 (apr code eval: task suite, sandbox, metrics and nightly receipts (APR-CODE-EVAL-001)), split out of #5314 by theme.
feat(code-eval): contamination control, canary and 13-gram overlap (E6)
feat(apr-cli): apr code eval, the receipted eval of apr code (E5, par…