Repository navigation
Evals: cheaper cadence, one case per invocation, free structural checks #13
Description
Learnings from adding three task-executor cases for #2 on 2026-10-13, where each run is about twenty cents and the agent run, not the judge, is the cost.
Cadence in the README. evals/README.md says three runs is the confirmation. Record the cheaper rule: run only the component the change touched, never the suite; iterate at --runs 1; confirm at two runs for a case whose graders are all deterministic on a deterministic prompt, three otherwise.
One case per invocation. claude plugin eval honours only the last --case flag, so EVAL_ARGS="--case a --case b" runs b alone. Say so in the README, or make the Makefile accept a list.
Free checks on case structure. What needs no model goes under checks/: every case carries its component's tag, every grader's frontmatter parses, every scaffold.sh is a regular executable file, no --case typo. Overlaps with #3.
Fewer variants. A twin case that differs only in mode (stops-on-vague-bundle-standalone) doubles a hypothesis's cost for one extra fact. Drop or demote such twins once they have stayed green across a few agent changes.
Activity
sabind commentedon Oct 13, 2026
Evidence from 2026-10-13, so this ticket stands on its own.
Cost per run (list price,
claude plugin evalon the task-executor cases added in #14, agent model opus):Adding the llm grader moved a run by about $0.03. The agent run itself is the cost, so the levers are how many runs and which cases, not how a case is graded.
Only the last
--caseis honoured. This command ran onlystops-on-vague-bundle-standalone:The first case was silently skipped, and the table listed one case. One invocation per case is the workaround until the Makefile or the README covers it.
Cadence agreed with the maintainer: run only the
make eval-<kind>-<name>target for the component the change touched, nevermake eval; iterate with--runs 1; confirm at two runs when every grader of the case is deterministic (regex, tool_used, file_exists) and the prompt leaves the agent one sensible path; confirm at three runs otherwise, including any case with an llm grader. The README currently says three for everything.Free checks: the structural checks bullet overlaps with #3, which already covers eval tag hygiene. Do that part under #3 and leave this ticket the README, the
--casebehaviour and the twin-case guidance.sabind commentedon Oct 13, 2026
Status (2026-10-13): three of the four items ship with the
3-eval-checks-and-cadencebranch, which closes this issue.evals/README.mdnow says to run only the component touched, iterate at--runs 1, and confirm at two runs when every grader of the case isregex,tool_usedorfile_exists, three when any isllmor the target mixes both. The Makefile header and the top-level README agree.--casetakes one glob and the last flag wins, so several named cases need one invocation each. Making the Makefile accept a list was judged more complexity than the cost warrants for now; revisit if evals get expensive.checks/test_eval_cases.py, undermake validate: every case tagged with its component, every grader parsing with a known type and its required fields, every scaffold a regular executable file named by itscase.yaml.stops-on-vague-bundle-standalonestays; no rule about twins goes into the README.sabind commentedon Oct 14, 2026
Closed by PR #22 (merged 2026-10-14). The PR body said "Closes #3, #13 and #4" but GitHub only auto-closes the first reference after a keyword, so closing by hand.