Repository navigation
epic: apr code eval — our own coding-agent benchmark (pv ontology, Rust harness) #5281
Closed as not planned
Description
owner: aprender-flow (set 2026-10-19 by aprender-triage-worker at the cop's word; picks up children only as batch constituents after its current timed task)
Problem
apr code has no evaluation. Ollama has no counterpart to compare against, so there is no external score to beat or match. Without our own eval, any claim that apr code "solves coding tasks" cannot be checked.
Goal
Build our own coding-agent eval, in Rust, designed as a pv ontology (task, sandbox, verdict, metric) and based on established public benchmarks:
| Source practice | What we take from it |
|---|---|
| SWE-bench Verified | real-repo tasks, hidden fail-to-pass and pass-to-pass tests, pass@1 |
| EvalPlus (HumanEval+ / MBPP+) | extended hidden tests, so a weak test suite cannot pass a wrong answer |
| Aider polyglot | edit-format conformance, multiple languages, two attempts with test feedback |
| Terminal-Bench | agentic tasks in a container, judged by checking the final state |
| LiveCodeBench | tasks dated after the model's cutoff, as a contamination control |
Every run is hermetic (no network, pinned image digest, fixed seeds, wall/CPU/memory limits) and writes a receipt (task-set digest, model sha256, apr version + sha, per-task verdicts, metric with a confidence interval).
Rules
- Rust harness only. Python appears only as an external target language inside the sandbox, never as harness code.
- A run with 0 tasks, 0 executed tests or empty model output must never report a pass.
- LAB first. It runs nightly on CPU and GPU through the fast-verb matrix (ci(lab): nightly fast-verbs CRUX smoke + tok/s on the ARM64 CUDA edge runner, ranked gap list #5254) once built. It is not on any PR, merge-queue or release path.
Children
- apr code eval E1: ontology contracts + spec (task, sandbox, verdict, metric) #5282 E1 ontology contracts + spec
- apr code eval E2: task-suite schema + loader (repo tasks, function tasks, edit tasks, terminal tasks) #5283 E2 task-suite schema + loader
- apr code eval E3: hermetic sandbox runner (no network, pinned digest, limits, fixed seed) #5284 E3 hermetic sandbox runner
- apr code eval E4: verdict + metric engine (pass@1, unbiased pass@k, Wilson CI, refusals) #5285 E4 verdict + metric engine
- apr code eval E5: apr code driver — agentic loop + edit-format adapters #5286 E5 apr code driver + edit-format adapters
- apr code eval E6: contamination control (dated tasks, canary, train-set overlap) #5287 E6 contamination control
- apr code eval E7: receipts + nightly CPU/GPU run via the fast-verb matrix (#5254) #5288 E7 receipts + nightly CPU/GPU via ci(lab): nightly fast-verbs CRUX smoke + tok/s on the ARM64 CUDA edge runner, ranked gap list #5254
keep-open: epic
Activity
noahgift commentedon Oct 22, 2026
tracked in #5348