registry.yaml records eval_cases and eval_executed per policy. For at least four policies
those numbers disagree with the suite they describe:
| policy |
cases in evals/suite.yaml |
eval_cases in registry |
protect-agent-config |
24 |
8 |
protect-commit-privacy |
20 |
7 |
scan-secrets |
20 |
11 |
block-wildcard-agent-permissions |
11 |
7 |
Every one of those suites has an execute: block on every case, so these are executable cases
that the registry is undercounting — not aspirational ones.
Why this matters more than a wrong number. This repository's entire proposition is that its
labels can be trusted: 39 policies, 17 enforced, 22 advisory, each claim backed by something. The
README publishes those counts and tools/check_readme.py verifies them. But eval_cases and
eval_executed are published in the same file and tools/check_registry.py does not look at
them at all — grep it, there are zero references. So a number that speaks to how well-tested a
guard is can drift indefinitely without anything noticing. That is exactly the class of unchecked
claim the catalog exists to argue against.
What to do:
- Work out what
eval_executed is actually supposed to mean. It is 0 for several policies
whose evals demonstrably do run (chock check --only evals reports hundreds of passes), so
either the field means something narrower than "cases that ran" or it is simply never updated.
Whichever it is, the answer belongs in a comment beside the field.
- Extend
tools/check_registry.py to verify both counts against the suites, the way
check_readme.py already verifies the README's counts against the registry.
- Correct the current values so the new check passes.
Do not fix this by deleting the fields. A count that is hard to keep honest is still worth
keeping honest — and if the conclusion really is that eval_executed cannot be defined
meaningfully, say that explicitly in the PR and remove it deliberately rather than quietly.
Sized to an afternoon. Files: registry.yaml, tools/check_registry.py, and the evals/suite.yaml
under each base/<policy>/.
Filed with the wrong numbers initially and corrected in place — the original text repeated
the registry's own stale counts instead of checking them against the suites, which is the very
mistake this issue is about.
registry.yamlrecordseval_casesandeval_executedper policy. For at least four policiesthose numbers disagree with the suite they describe:
evals/suite.yamleval_casesin registryprotect-agent-configprotect-commit-privacyscan-secretsblock-wildcard-agent-permissionsEvery one of those suites has an
execute:block on every case, so these are executable casesthat the registry is undercounting — not aspirational ones.
Why this matters more than a wrong number. This repository's entire proposition is that its
labels can be trusted: 39 policies, 17 enforced, 22 advisory, each claim backed by something. The
README publishes those counts and
tools/check_readme.pyverifies them. Buteval_casesandeval_executedare published in the same file andtools/check_registry.pydoes not look atthem at all — grep it, there are zero references. So a number that speaks to how well-tested a
guard is can drift indefinitely without anything noticing. That is exactly the class of unchecked
claim the catalog exists to argue against.
What to do:
eval_executedis actually supposed to mean. It is0for several policieswhose evals demonstrably do run (
chock check --only evalsreports hundreds of passes), soeither the field means something narrower than "cases that ran" or it is simply never updated.
Whichever it is, the answer belongs in a comment beside the field.
tools/check_registry.pyto verify both counts against the suites, the waycheck_readme.pyalready verifies the README's counts against the registry.Do not fix this by deleting the fields. A count that is hard to keep honest is still worth
keeping honest — and if the conclusion really is that
eval_executedcannot be definedmeaningfully, say that explicitly in the PR and remove it deliberately rather than quietly.
Sized to an afternoon. Files:
registry.yaml,tools/check_registry.py, and theevals/suite.yamlunder each
base/<policy>/.Filed with the wrong numbers initially and corrected in place — the original text repeated
the registry's own stale counts instead of checking them against the suites, which is the very
mistake this issue is about.