[sabind]

Why

Closes #3, #13 and #4. A case that lost its component tag, a grader whose frontmatter did not parse, or a scaffold that was not a regular executable file was silently never run or never scored, and nothing free caught it. The eval README still said three runs was the confirmation and did not say that --case takes one glob. And the SubagentStart hook had only ever been seen firing inside a sub-agent on macOS, while CI runs on Linux.

What

make validate now fails on any eval case that cannot run or cannot be scored. The eval README, the Makefile header and the top-level README record a cheaper cadence: run only the component touched, iterate at one run, confirm at two for deterministic graders and three when an LLM judge is involved. The hook eval was run once under Linux and recorded on #4.

Changes

• The free check on eval case structure (checks/test_eval_cases.py, .github/workflows/validate.yml): a stdlib unittest module that finds every case, requires its component's tag, requires every grader to parse with a known type and that type's fields, and requires every scaffold to be a regular executable file named by its case.yaml; the CI workflow comment describes the checks instead of listing them. The goal is to look for cases that cannot run or don't validate. A broken case at run time is not good and it can be prevented.
• The cadence and --case rule (evals/README.md, Makefile header, README.md): run only the component touched; iterate at --runs 1; confirm at two when every grader of the case is deterministic, three when any is llm or the target mixes both; --case takes one glob and the last flag wins; the eval README names the check as what keeps cases valid and the top-level README points to it. The goal is to minimize unnecessary invocations. For things that are mostly deterministic, two invocations is enough on the plugin side; three for LLMs on the validate side, since there are two places where non-deterministic behaviour can be introduced, so we have more rigor. The rest is to make sure we keep things working and aligned, which is why the free check test_eval_cases.py is called out.
• The spec (.claude/specs/eval-hygiene/): requirements, design and the ticked task list. We added the requirements that outlined this work; it's important to know how we got here.

Testing

make validate: 15 checks OK, both plugin validations pass. The new check alone: 4 tests in 0.07 s. Broken-input proof, run twice on scratch copies of a case: a missing tag, an unknown grader type, a missing pattern, an empty llm body, a missing exists, a non-executable scaffold and an unwired scaffold.sh each produce one problem line naming the file. All 29 existing cases pass the check unchanged.

Hook eval under Linux, once, in a scratch node:22-bookworm container with Claude Code 2.1.296, recorded with the command on #4:

Case	Score	Pass	Runs	Cost
rules-loaded	1.00	100%	1	$0.04
subagent-rules-loaded	1.00	100%	1	$0.07

Risk and rollout

Limited. The check runs in CI on every PR from now on, so a new case that is untagged, has a malformed grader or an unwired scaffold fails make validate. Nothing else changes behaviour; the behaviour was validated and the goal is to increase testing robustness. Roll back by reverting the commit.

Notes for reviewers

The only code is checks/test_eval_cases.py; the rest is documentation. The fewer-variants item of #13 was dropped by decision and is recorded on the issue.

🤖 Generated with Claude Code