diff --git a/.claude/specs/eval-hygiene/design.md b/.claude/specs/eval-hygiene/design.md new file mode 100644 index 0000000..7439cce --- /dev/null +++ b/.claude/specs/eval-hygiene/design.md @@ -0,0 +1,114 @@ +# Design Document + +## Overview + +One new free check, one README edit, one Makefile comment edit, and one scratch container run that leaves nothing behind. `checks/test_eval_cases.py` walks every `prompt.md` under `evals/`, derives the component from the path, and asserts the tag, the graders and the scaffold wiring, in the style of `checks/test_issue_forms.py` (Requirement 1). `evals/README.md`'s Running and Writing-a-case sections gain the cadence and the `--case` rule, and the Makefile header agrees (Requirement 2). The Linux hook eval runs once from a container built in the scratchpad with the token passed as an environment variable, and its table goes on issue #4 (Requirement 3). The PR closes the three issues (Requirement 4). + +The one fact established by reading rather than from the docs: an `llm` grader carries only `type: llm` in its frontmatter and its judging prompt as the body, so its required field is a non-empty body, not a frontmatter key. + +## Architecture + +```mermaid +flowchart TD + V["make validate"] --> C["make check: unittest discover -s checks"] + C --> T["test_eval_cases.py"] + T --> D["discover evals/**/prompt.md"] + D --> P{"for each case"} + P --> TAG["tag == component from path"] + P --> G["each graders/*.md: fences parse, type known, required fields present"] + P --> S["case.yaml scaffold_script <-> scaffold.sh: both or neither; regular, executable"] + TAG & G & S --> R["one failure per problem, naming the file"] +``` + +Key decision, one test method per concern, not one per case. `unittest` has no parametrisation; the issue-forms check loops inside a method and reports the first failing file with its name. The same shape here: a method for tags, one for graders, one for scaffolds, each looping over all cases and collecting every problem into one assertion message, so a single run lists everything wrong (Requirement 1.3, 1.5, 1.6). Trade-off: one red test per concern rather than per case; the message carries the detail. + +Key decision, a hand-rolled frontmatter reader rather than a YAML dependency. The repository's checks are standard-library only (Requirement 1.8), and grader frontmatter is flat `key: value` lines between `---` fences, with one inline-map value (`target: { source: file, path: ... }`) that the check treats as an opaque string. Rejected alternative, PyYAML: not installed, and the code-changes rule forbids a new dependency without asking. + +Key decision, the known-type table lives in the check. Four types with their required fields (Requirement 1.5, 1.9). A new type fails with a message to add it, so the list is edited on purpose. Rejected alternative, accept any `type`: would let a typo (`regx`) pass and the grader silently never score. + +Key decision, the Linux run is scratch, not shipped. The author chose a one-off over a committed script; the comment on #4 holds the exact command so it can be repeated by hand (Requirement 3.3, 3.4). + +## Components and Interfaces + +| Component | Responsibility | Requirements | +|-----------|----------------|--------------| +| `checks/test_eval_cases.py`, `cases()` | Yield `(path, component)` for every `evals/**/prompt.md`; component from the path segments; fail if none | 1.2, 1.3 | +| `checks/test_eval_cases.py`, `frontmatter(path)` | Return the `key: value` pairs between the first two `---` lines, and the body after; raise a clear error when the fences are missing | 1.5 | +| `checks/test_eval_cases.py`, `KNOWN_TYPES` | `{"regex": ("pattern",), "tool_used": ("tool",), "file_exists": ("path", "exists"), "llm": ()}`; `llm` additionally needs a non-empty body | 1.5, 1.9 | +| `test_every_case_is_tagged_with_its_component` | Loop; collect `(case, missing tag)`; one assertion | 1.3 | +| `test_every_case_has_valid_graders` | Loop; at least one grader; each parses, type known, fields present, llm body non-empty; collect problems | 1.4, 1.5, 1.9 | +| `test_scaffolds_are_wired_and_executable` | `case.yaml` present: the named script exists, `is_file()` and not `is_symlink()`, owner-executable; `scaffold.sh` present without `case.yaml`: problem | 1.6, 1.7 | +| `evals/README.md`, Running | Cadence paragraph rewritten: component only, iterate at 1, confirm at 2 or 3; `--case` one glob, last wins | 2.1, 2.2, 2.3 | +| `evals/README.md`, Writing a case | The check named as what keeps tags, graders and scaffolds valid | 2.5 | +| `Makefile` header comment | Cadence and `--case` lines agree with the README | 2.4 | +| Scratch container (scratchpad only) | `node:22-bookworm`-family tagged image, `npm install -g @anthropic-ai/claude-code`, checkout copied in, `make eval-hook` with the token from `-e` | 3.1, 3.2, 3.4, 3.5 | +| Issue #4 comment | Table, image tag, Claude Code version, command | 3.3 | +| Issue #3 and #13 comments, PR description | Size-check half shipped in #21; fewer-variants item dropped by decision; PR closes #3, #13, #4 | 4.1, 4.2 | + +## Data Models + +Case, as the check sees it: + +| Field | Source | Constraint | +|-------|--------|------------| +| `path` | directory of `prompt.md` | under `evals/` | +| `component` | path: `evals////` gives `name`; `evals/hook//` gives `hook` | must appear in `tags` | +| `tags` | `prompt.md` frontmatter, `[a, b]` list syntax | contains `component` | +| `graders` | files `graders/*.md` | at least one | +| `scaffold` | `case.yaml` `context.scaffold_script`, read by a line match, no YAML parser | exists, regular, owner-executable; and `scaffold.sh` implies `case.yaml` | + +Grader frontmatter: + +| Field | Required for | Note | +|-------|--------------|------| +| `type` | all | one of `KNOWN_TYPES` | +| `pattern` | `regex` | string, quoted or not | +| `tool` | `tool_used` | string | +| `path`, `exists` | `file_exists` | string, boolean | +| body | `llm` | non-empty after the closing fence | +| `flags`, `match`, `target`, `min`, `max`, `arm`, `weight` | optional | not validated | + +Cadence rule text, for the README's Running section and the Makefile header: run only the `make` target of the component touched, the whole suite only for a change to `hooks/` or the Makefile; iterate at `--runs 1`; confirm at two runs when every grader is `regex`, `tool_used` or `file_exists`, at three when any grader is `llm`; `--case` takes one glob and the last `--case` flag wins, so several named cases need one invocation each. + + +## Correctness Properties + +### Property 1: Every case is reachable by its target + +*For any* case under `evals/`, the `make eval-*` target enumerated from its parent directory selects it, because its tags contain the component the path names. + +**Validates: Requirements 1.2, 1.3** + +### Property 2: Every grader can score + +*For any* grader file under any case, its frontmatter parses, its `type` is one the harness supports, and the fields that type needs are present, so the harness never throws on it. + +**Validates: Requirements 1.4, 1.5, 1.9** + +### Property 3: Scaffolds run when named and are named when present + +*For any* case, `case.yaml` names a scaffold if and only if a scaffold file exists, and the named file is a regular, owner-executable file. + +**Validates: Requirements 1.6, 1.7** + +## Error Handling + +| Condition | Category | Behaviour | Requirements | +|-----------|----------|-----------|--------------| +| No `prompt.md` under `evals/` | EMPTY | Fail: "no eval cases found" | 1.2 | +| Tag missing | UNTAGGED | Fail, listing `case: expected tag ` for each | 1.3 | +| Case has no graders | UNGRADED | Fail, listing the case | 1.4 | +| Grader fences missing or a line is not `key: value` | MALFORMED | Fail, listing `file: ` | 1.5 | +| Grader `type` unknown | UNKNOWN_TYPE | Fail, listing the file and "add it to KNOWN_TYPES with its required fields" | 1.9 | +| Required field missing, or `llm` body empty | INCOMPLETE | Fail, listing the file and the field | 1.5 | +| `case.yaml` names a missing, symlinked or non-executable script | BAD_SCAFFOLD | Fail, listing the case and which condition | 1.6 | +| `scaffold.sh` without `case.yaml` | UNWIRED | Fail, listing the case | 1.7 | +| Docker daemon down or token unset (Linux run) | STOP | Say so and stop; no run elsewhere | 3.5 | + +## Testing Strategy + +- The check is itself the test for Properties 1 to 3 against the live suite. Its own correctness is proved by an ephemeral check: copy one case into a scratch directory under the scratchpad, break it four ways in turn (drop the tag, misspell a grader type, remove `pattern`, `chmod -x` the scaffold), point the check at that root via its `ROOT` constant or an environment variable, and confirm each produces the matching failure message; then delete the scratch copy. +- Run time: the check file alone, `python3 -B -m unittest discover -s checks -p test_eval_cases.py`, under five seconds (Requirement 1.8). +- Existing suite: `make validate` green with all 29 cases passing the new check unchanged, or the offending case fixed and named in the PR (Requirement 4.3). +- README and Makefile: read back against the requirement text; `make help` unchanged. +- The Linux run proves nothing about this branch's code; it is the evidence #4 asks for, recorded there with the image tag, the Claude Code version and the command. diff --git a/.claude/specs/eval-hygiene/requirements.md b/.claude/specs/eval-hygiene/requirements.md new file mode 100644 index 0000000..f5abdd9 --- /dev/null +++ b/.claude/specs/eval-hygiene/requirements.md @@ -0,0 +1,73 @@ +# Requirements Document + +## Introduction + +The handbook's evals are real model calls, run by hand per component, and three things make them costlier or less trustworthy than they need to be. A case that loses its component tag, a grader whose frontmatter does not parse, or a scaffold that is not a regular executable file is silently never run or never scored, and nothing free catches it. The eval README still says three runs is the confirmation and does not say that `--case` takes one glob, so contributors over-run and mis-select. And the SubagentStart hook has only ever been seen firing inside a sub-agent on macOS, while CI runs on Linux. This task adds free structural checks under `checks/` that `make validate` runs, records the cheaper cadence and the case-selection rule in the README, and runs the hook eval once under Linux in a scratch container, recording the result on the issue. Issues #3, #13 and #4. + +## Glossary + +- **Case**: A directory under `evals/` holding a `prompt.md` with frontmatter, a `graders/` directory, and optionally `case.yaml` and `scaffold.sh`. +- **Component**: A skill, agent, rule or hook of the plugin, named by the directory under `skills/`, `agents/` or `rules/`, or `hook`. +- **Tag**: An entry in a Case's `tags` frontmatter list; `make eval-*` selects Cases with `--tag `. +- **Grader**: A file under a Case's `graders/` directory whose frontmatter holds a `type` and that type's fields. +- **Known_Type**: One of the grader types the harness supports and the repository uses: `regex`, `tool_used`, `file_exists`, `llm`. +- **Required_Fields**: The fields a Grader of a Known_Type must carry: `pattern` for `regex`; `tool` for `tool_used`; `path` and `exists` for `file_exists`; `prompt` or the field the harness documents for `llm`. +- **Scaffold**: A Case's `scaffold.sh`, named by `case.yaml`'s `context.scaffold_script`, which builds the Case's workspace before the run. +- **Free_Check**: A standard-library `unittest` file under `checks/`, discovered by `make check` and run by `make validate` and CI, making no model call. +- **Cadence**: The number of runs per Case a contributor uses: one while iterating, and a confirmation count before trusting a change. +- **Deterministic_Grader**: A Grader of type `regex`, `tool_used` or `file_exists`, whose verdict depends only on the transcript and the workspace. +- **Hook_Eval**: The two Cases tagged `hook`, run by `make eval-hook`, which show the handbook rules loaded into a session and into a sub-agent. +- **Linux_Run**: A one-off run of the Hook_Eval inside a scratch Linux container on the contributor's machine, with the plugin loaded from a copy of the checkout. +- **Token**: The credential the container needs to call the model: an `ANTHROPIC_API_KEY` or a `CLAUDE_CODE_OAUTH_TOKEN` from `claude setup-token`, passed as an environment variable and never written to disk in the repository. + +## Requirements + +### Requirement 1: Free checks on case structure + +**User Story:** As a contributor, I want a case that cannot run or cannot be scored to fail `make validate`, so that a silent gap in the suite is caught before merge rather than discovered when a change goes untested. + +#### Acceptance Criteria + +1. THE plugin SHALL ship a Free_Check `checks/test_eval_cases.py` in the style of `checks/test_issue_forms.py`. +2. THE Free_Check SHALL discover every Case under `evals/` by its `prompt.md`, and SHALL fail if there are none. +3. THE Free_Check SHALL assert that every Case's `tags` list contains the Component name derived from the Case's path (`evals/skills//`, `evals/agents//`, `evals/rules//` give ``; `evals/hook/` gives `hook`), naming the Case and the missing Tag on failure. +4. THE Free_Check SHALL assert that every Case has at least one Grader. +5. THE Free_Check SHALL assert that every Grader's frontmatter parses as `key: value` lines between `---` fences, that its `type` is a Known_Type, and that its Required_Fields are present, naming the Grader file and the problem on failure. +6. WHEN a Case has a `case.yaml` naming a `scaffold_script`, THE Free_Check SHALL assert that the named file exists beside it, is a regular file (not a symlink), and is executable by its owner. +7. THE Free_Check SHALL assert that a Case with a `scaffold.sh` also has a `case.yaml` naming it, so a scaffold is never silently skipped. +8. THE Free_Check SHALL use only the standard library, and SHALL run in under five seconds on the current suite. +9. IF a Grader uses a `type` the repository has not used before, THEN THE Free_Check SHALL fail with a message saying to add the type and its Required_Fields to the check, so the list stays deliberate. + +### Requirement 2: The README records the cheaper cadence + +**User Story:** As a contributor, I want the eval README to tell me how many runs to make and how to select cases, so that I spend the least on my own seat for a trustworthy answer. + +#### Acceptance Criteria + +1. THE eval README SHALL say to run only the `make` target of the Component the change touched, never the whole suite, unless the change is to the hooks or the Makefile. +2. THE eval README SHALL say to iterate at `--runs 1` and to confirm at two runs when every Grader of the Case is a Deterministic_Grader, and at three runs when any Grader is an `llm` judge. +3. THE eval README SHALL say that `--case` takes one glob and that the last `--case` flag wins, so several named Cases need one invocation each. +4. THE Makefile's header comment SHALL agree with the README on Cadence and on `--case`. +5. THE eval README SHALL name the Free_Check of Requirement 1 as what keeps Tags, Graders and Scaffolds valid, in the "Writing a case" section, without listing its individual assertions. + +### Requirement 3: The hook eval is seen under Linux once + +**User Story:** As a maintainer, I want to see the SubagentStart hook fire inside a sub-agent on Linux once, so that the one behaviour CI cannot exercise is confirmed on the platform CI uses, without adding anything to the repository for it. + +#### Acceptance Criteria + +1. THE Linux_Run SHALL be performed once in this task from a scratch Docker container on the contributor's Mac: a tagged Node LTS image with Claude Code installed, the checkout copied in, and `make eval-hook` run with the plugin loaded from that copy. +2. THE container SHALL receive the Token only as an environment variable, and the Token SHALL appear in no file, no shell history the task writes, and no issue comment. +3. THE Linux_Run's result table, the image tag, the Claude Code version and the exact command used SHALL be recorded as a comment on issue #4, so the run can be repeated by hand. +4. THE task SHALL add no file to the repository for the Linux_Run; the scratch container, its image build context and any copied checkout SHALL be removed when the run is done. +5. IF Docker is not running or the Token is unset, THEN THE task SHALL stop and say so rather than run the eval elsewhere. + +### Requirement 4: Issue housekeeping + +**User Story:** As a maintainer, I want the three issues to reflect what shipped where, so that the board is honest. + +#### Acceptance Criteria + +1. THE pull request for this task SHALL close #3, #13 and #4. +2. A comment on #3 SHALL note that its rules-size half shipped in #21 as `checks/test_rules_size.py`, and a comment on #13 SHALL note that its fewer-variants item was dropped by decision, before this task's PR closes them. +3. THE `make validate` target SHALL pass, and the existing Cases SHALL all pass the new Free_Check without modification; where one does not, the Case is fixed in this task and the fix named in the PR. diff --git a/.claude/specs/eval-hygiene/tasks.md b/.claude/specs/eval-hygiene/tasks.md new file mode 100644 index 0000000..ec9f61e --- /dev/null +++ b/.claude/specs/eval-hygiene/tasks.md @@ -0,0 +1,63 @@ +# Implementation Plan + +## Overview + +The check comes first, since the README names it. Then the documentation edits in one wave, since they touch different files. The Linux run and the issue comments come last, since the run proves nothing about this branch and the comments describe the finished state. The handbook's test framework is the stdlib `unittest` files under `checks/`, so the check is both the deliverable and its own proof against the live suite; its correctness on broken input is proved by an ephemeral check. Every command runs in the worktree at `/Users/dansabin/autohdr/developer-handbook-evals`. + +## Tasks + +- [x] 1. The check + - [x] 1.1 Write `checks/test_eval_cases.py` + - Module docstring in the style of `checks/test_issue_forms.py`: one sentence saying what the check keeps true. `ROOT` from the file's location. `KNOWN_TYPES` as the design's table, with a comment-free structure: a dict from type to its required frontmatter fields, and `llm` handled by a non-empty body. `cases()` yields `(case_dir, component)` for every `evals/**/prompt.md`, component from the path (`evals////` gives `name`; `evals/hook//` gives `hook`). `frontmatter(path)` returns the `key: value` pairs between the first two `---` lines and the body after, raising `ValueError` with the file name when the fences are missing or a line is not `key: value`; an inline-map value such as `target: { ... }` is kept as its raw string. Three test methods: `test_every_case_is_tagged_with_its_component`, `test_every_case_has_valid_graders`, `test_scaffolds_are_wired_and_executable`, each looping over all cases, collecting every problem as `path: reason`, and failing once with the list joined by newlines; a fourth, `test_there_are_cases`, fails if none are found. The scaffold test reads `scaffold_script:` from `case.yaml` by a line match, checks the named file exists, `is_file()`, not `is_symlink()`, and `os.access(path, os.X_OK)`, and separately that a `scaffold.sh` present without `case.yaml` is a problem. Run `make check` and `time make check` + - **Property 1: Every case is reachable by its target** + - **Property 2: Every grader can score** + - **Property 3: Scaffolds run when named and are named when present** + - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9_ + - [x] 1.2 Prove the check on broken input + - Ephemeral: copy `evals/agents/task-executor/standalone-mode/` into a scratch tree under the scratchpad shaped `evals/agents/task-executor/standalone-mode/`, make the check accept an alternative root (an `EVAL_CHECK_ROOT` environment variable read at import, defaulting to `ROOT`, is the smallest way; keep it only if it stays under three lines, otherwise monkeypatch `ROOT` from the scratch script), then break the copy four ways in turn and assert the matching message: remove `task-executor` from `tags`; change a grader's `type` to `regx`; remove `pattern` from a regex grader; `chmod -x scaffold.sh`. Delete the scratch tree. Record the four messages in the Summary + - _Requirements: 1.3, 1.5, 1.6, 1.9_ + +- [x] 2. Documentation and the twin + - [x] 2.1 Rewrite the cadence paragraph in `evals/README.md`, Running + - Replace the paragraph beginning "A single run is noisy" with the design's cadence rule text: run only the `make` target of the component touched, the whole suite only for a change to `hooks/` or the Makefile; iterate at `--runs 1`; confirm at two runs when every grader is `regex`, `tool_used` or `file_exists`, at three when any grader is `llm`; `--case` takes one glob and the last `--case` flag wins, so several named cases need one invocation each. Keep the example commands block; drop the `--case` example's claim if it implies more than one. Read back + - _Requirements: 2.1, 2.2, 2.3_ + - [x] 2.2 Name the check in `evals/README.md`, Writing a case + - One or two sentences at the end of the section: `checks/test_eval_cases.py`, run by `make validate`, keeps tags, graders and scaffolds valid, without listing its assertions. Read back + - _Requirements: 2.5_ + - [x] 2.3 Make the Makefile header agree + - In the Makefile's header comment, the lines about defaults and runs say the same cadence as the README in one or two lines; nothing else in the Makefile changes. `make help` unchanged + - _Requirements: 2.4_ + +- [x] 3. The Linux run and the issues + - [x] 3.1 Run the hook eval under Linux once, done by the author with Claude driving + - In the scratchpad, write a Dockerfile from a tagged Node LTS image (`node:22-bookworm` or the current LTS tag) that installs `@anthropic-ai/claude-code` globally and `make`; copy the worktree's checkout into the build context without `.git`, `evals/results` and `.claude/settings.json`; run the container with `-e ANTHROPIC_API_KEY` or `-e CLAUDE_CODE_OAUTH_TOKEN` from the author's shell, never written to a file, executing `make eval-hook EVAL_ARGS="--runs 1"` in the copied checkout; capture the result table, `claude --version` and the image tag. Then remove the image, the container and the scratch build context. Stop and say so if Docker is not running or neither token is set + - _Requirements: 3.1, 3.2, 3.4, 3.5_ + - [x] 3.2 Record the run on issue #4 + - A `gh issue comment 4` with the table, the image tag, the Claude Code version and the exact `docker build` and `docker run` commands with the token shown as a placeholder, so the run can be repeated by hand. No token in the comment + - _Requirements: 3.3_ + - [x] 3.3 Note on issues #3 and #13 what shipped and what was dropped + - A `gh issue comment 3` saying the rules-size half shipped in #21 as `checks/test_rules_size.py` and the tag half ships with this branch's PR; a `gh issue comment 13` saying the fewer-variants item was dropped by decision on 2026-10-13 and the rest ships with this branch's PR + - _Requirements: 4.2_ + - [x] 3.4 Validate the finished branch + - `make validate` green; `make help` lists the same targets as before; `git status` shows only the intended files. Record the result for the PR's Testing section + - _Requirements: 4.3_ + +## Notes + +- Every command runs in the worktree at `/Users/dansabin/autohdr/developer-handbook-evals`, never in the main checkout, which another session is using. +- The check follows `checks/test_issue_forms.py` for docstring style, root discovery and failure messages; standard library only. +- Model calls happen only in task 3.1, once, at one run per case; nothing else in this plan costs a seat. +- The PR closes #3, #13 and #4 (Requirement 4.1), through the `pr` skill. + +## Task Dependency Graph + +```json +{ + "waves": [ + { "id": 0, "tasks": ["1.1"] }, + { "id": 1, "tasks": ["1.2", "2.1", "2.2", "2.3"] }, + { "id": 2, "tasks": ["3.1", "3.3"] }, + { "id": 3, "tasks": ["3.2", "3.4"] } + ] +} +``` diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index 7e15173..246835d 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -1,9 +1,9 @@ name: Validate plugin # Runs the same check developers run locally before opening a PR: make validate, -# which runs the Python checks under checks/ (the hook encoder round-trip and the -# agent frontmatter and handbook-reference checks) plus claude plugin validate --strict, -# so a stray field, a missing version or a mis-escaped rule cannot land. +# which runs the unittest files under checks/, each one concern, plus +# claude plugin validate --strict, so a stray field, a missing version or a +# mis-escaped rule cannot land. on: pull_request: push: diff --git a/Makefile b/Makefile index 8e755ed..735c1a4 100644 --- a/Makefile +++ b/Makefile @@ -13,6 +13,9 @@ # in CI. Defaults: one arm (plugin on), the case's own runs count, list-price # ceiling $(EVAL_MAX_COST_USD). Add flags with EVAL_ARGS, for example # make eval-skill-pr EVAL_ARGS="--runs 2 --ablation with-without" +# Iterate at --runs 1; confirm at 2 when every grader of the case is +# deterministic, 3 when any is llm or the target mixes both. --case takes one +# glob and the last --case wins, so run one invocation per named case. # The eval-skill-*, eval-rule-* and eval-agent-* targets are enumerated from # the evals/ directories and select their cases with --tag, so every case # carries its component's name as a tag and a new case needs no Makefile diff --git a/README.md b/README.md index db56d97..350fb39 100644 --- a/README.md +++ b/README.md @@ -192,8 +192,9 @@ the four that grant Bash put `bin` first on the PATH for the wrappers the scaffolds write. `eval` carries the union so the whole suite runs. Each invocation runs under a list-price ceiling, `EVAL_MAX_COST_USD`, default $2. -Each case runs three times by default; iterate with `EVAL_ARGS="--runs 1"`, and -run the table you paste for a rule or skill-description change with +Each case runs three times by default; iterate with `EVAL_ARGS="--runs 1"`, +confirm as `evals/README.md` says (2 runs for deterministic graders, 3 when any +is llm), and run the table you paste for a rule or skill-description change with `EVAL_ARGS="--ablation with-without"` so the Δ column shows what the plugin added. Results land in `evals/results//` (gitignored). diff --git a/checks/test_eval_cases.py b/checks/test_eval_cases.py new file mode 100644 index 0000000..700c25b --- /dev/null +++ b/checks/test_eval_cases.py @@ -0,0 +1,194 @@ +"""Every eval case is reachable by its make target, has graders the harness can score, and runs the scaffold it carries. An llm grader needs no field: its prompt is the body after the closing fence.""" + +import os +import re +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +EVALS_DIR = ROOT / "evals" + +KNOWN_TYPES = { + "regex": ["pattern"], + "tool_used": ["tool"], + "file_exists": ["path", "exists"], + "llm": [], +} + +FENCE = "---" +SCAFFOLD_SCRIPT = re.compile(r"^\s*scaffold_script:\s*(?P[^\s#]+)\s*(?:#.*)?$", re.MULTILINE) +TAGS = re.compile(r"^tags:(?P.*)$") + + +def cases(): + """Yield ``(case_dir, component)`` for every ``prompt.md`` under ``evals/``. + + ``evals////`` names component ``name``; ``evals/hook//`` names ``hook``. + A ``prompt.md`` under neither names component ``None``. + """ + for prompt in sorted(EVALS_DIR.glob("**/prompt.md")): + case_dir = prompt.parent + parts = case_dir.relative_to(EVALS_DIR).parts + if parts[:1] == ("hook",): + yield case_dir, "hook" + else: + yield case_dir, parts[1] if len(parts) >= 2 else None + + +def fenced(path): + """Return ``(lines, body)``: the lines between the first two ``---`` lines of ``path`` and the text after them. + + Raises ``ValueError`` naming the file when either fence is missing. + """ + lines = path.read_text(encoding="utf-8").splitlines() + if not lines or lines[0].strip() != FENCE: + raise ValueError("{}: no opening --- fence".format(path.name)) + try: + end = next(i for i, line in enumerate(lines[1:], start=1) if line.strip() == FENCE) + except StopIteration: + raise ValueError("{}: no closing --- fence".format(path.name)) from None + return lines[1:end], "\n".join(lines[end + 1 :]) + + +def frontmatter(path): + """Return ``(fields, body)`` from the ``key: value`` lines between the first two ``---`` lines of ``path``. + + Values are kept as raw strings, an inline map such as ``target: { ... }`` included. + Raises ``ValueError`` naming the file when the fences are missing or a line is not ``key: value``. + """ + lines, body = fenced(path) + fields = {} + for line in lines: + if not line.strip(): + continue + key, sep, value = line.partition(":") + if not sep or not key.strip() or " " in key.strip(): + raise ValueError("{}: not a key: value line: {!r}".format(path.name, line)) + fields[key.strip()] = value.strip() + return fields, body + + +def unquote(value): + """Return ``value`` stripped, without one matching pair of surrounding single or double quotes.""" + value = value.strip() + if len(value) >= 2 and value[0] == value[-1] and value[0] in "'\"": + return value[1:-1] + return value + + +def tags(value): + """Return the entries of a ``[a, b]`` frontmatter list.""" + return [unquote(tag) for tag in value.strip().strip("[]").split(",") if tag.strip()] + + +def prompt_tags(path): + """Return the ``tags`` in the frontmatter of ``path``, an inline ``[a, b]`` or one ``- a`` per line. + + Returns an empty list when there is no ``tags:`` line. Raises ``ValueError`` when a fence is missing. + """ + lines, _ = fenced(path) + for i, line in enumerate(lines): + match = TAGS.match(line) + if match is None: + continue + if match.group("value").strip(): + return tags(match.group("value")) + items = [] + for item in lines[i + 1 :]: + if not item.strip().startswith("-"): + break + items.append(unquote(item.strip()[1:])) + return items + return [] + + +def relative(path): + return path.relative_to(ROOT) + + +class EvalCasesTest(unittest.TestCase): + """The eval cases under evals/ are tagged, gradable and scaffolded as the harness expects.""" + + def assertNoProblems(self, problems): + if problems: + self.fail("\n".join(problems)) + + def test_there_are_cases(self): + self.assertTrue(list(cases()), "no eval cases found") + + def test_every_case_is_tagged_with_its_component(self): + problems = [] + for case_dir, component in cases(): + if component is None: + problems.append( + "{}: not under evals/// or evals/hook/".format(relative(case_dir / "prompt.md")) + ) + continue + try: + found = prompt_tags(case_dir / "prompt.md") + except ValueError as error: + problems.append("{}/{}".format(relative(case_dir), error)) + continue + if component not in found: + problems.append("{}: expected tag {}".format(relative(case_dir), component)) + self.assertNoProblems(problems) + + def test_every_case_has_valid_graders(self): + problems = [] + for case_dir, _ in cases(): + graders_dir = case_dir / "graders" + graders = sorted(p for p in graders_dir.glob("*.md") if p.is_file()) if graders_dir.is_dir() else [] + if not graders: + problems.append("{}: no graders".format(relative(case_dir))) + for grader in graders: + try: + fields, body = frontmatter(grader) + except ValueError as error: + problems.append("{}/{}".format(relative(grader.parent), error)) + continue + kind = unquote(fields["type"]) if "type" in fields else None + if kind is None: + problems.append("{}: no type".format(relative(grader))) + elif kind not in KNOWN_TYPES: + problems.append( + "{}: unknown type {}; add it to KNOWN_TYPES with its required fields".format( + relative(grader), kind + ) + ) + else: + for field in KNOWN_TYPES[kind]: + if not fields.get(field): + problems.append("{}: {} grader has no {}".format(relative(grader), kind, field)) + if kind == "llm" and not body.strip(): + problems.append("{}: llm grader has an empty body".format(relative(grader))) + self.assertNoProblems(problems) + + def test_scaffolds_are_wired_and_executable(self): + problems = [] + for case_dir, _ in cases(): + case_yaml = case_dir / "case.yaml" + named = None + if case_yaml.is_file(): + match = SCAFFOLD_SCRIPT.search(case_yaml.read_text(encoding="utf-8")) + if match is not None: + named = unquote(match.group("name")) + if named is not None: + script = case_dir / named + if not (script.exists() or script.is_symlink()): + problems.append("{}: scaffold {} does not exist".format(relative(case_dir), named)) + elif script.is_symlink(): + problems.append("{}: scaffold {} is a symlink".format(relative(case_dir), named)) + elif not script.is_file(): + problems.append("{}: scaffold {} is not a regular file".format(relative(case_dir), named)) + elif not os.access(script, os.X_OK): + problems.append("{}: scaffold {} is not executable".format(relative(case_dir), named)) + scaffold = case_dir / "scaffold.sh" + if (scaffold.exists() or scaffold.is_symlink()) and ( + named is None or (case_dir / named).resolve() != scaffold.resolve() + ): + problems.append("{}: scaffold.sh is not named by case.yaml".format(relative(case_dir))) + self.assertNoProblems(problems) + + +if __name__ == "__main__": + unittest.main() diff --git a/evals/README.md b/evals/README.md index 7632d90..15a28fa 100644 --- a/evals/README.md +++ b/evals/README.md @@ -37,9 +37,17 @@ make eval-rule-code-changes EVAL_ARGS="--case smallest-change" # one case make eval EVAL_MAX_COST_USD=5 ``` -A single run is noisy, so a plain `make eval-…` runs each case three times, the -documented confirmation, and the table shows the mean. Iterate with -`EVAL_ARGS="--runs 1"`, then confirm at three before trusting a change. +Run only the target of the component your change touched, never the whole +suite, unless the change is to `hooks/` or the Makefile. A single run is noisy, +so iterate with `EVAL_ARGS="--runs 1"`, then confirm before trusting a change: +pass `--runs 2` when every grader of the case is `regex`, `tool_used` or +`file_exists`, and `--runs 3` when any grader is `llm`. A target whose cases +mix the two confirms at 3, or run it at 2 and then each `llm` case at 3 with +one `--case` each. Without `--runs`, a plain `make eval-…` still runs each case +three times and the table shows the mean. + +`--case` takes one glob, and when you pass it more than once the last flag +wins, so run several named cases with one invocation each. Every target runs one arm, the plugin loaded, which answers "does the skill work". For a change to a rule or to a skill's `description`, the question is @@ -173,4 +181,7 @@ so it is scored the same with and without the plugin. A case that needs flags gets one line in the Makefile; a new skill or rule folder gets its `make` target with no Makefile change. +`checks/test_eval_cases.py`, run by `make validate`, keeps every case's tag, +graders and scaffold valid. + Reference: https://code.claude.com/docs/en/plugin-evals