Skip to content

registry.yaml's eval counts are stale, and no checker verifies them #67

Description

@jothimani-rajendran

registry.yaml records eval_cases and eval_executed per policy. For at least four policies
those numbers disagree with the suite they describe:

policy cases in evals/suite.yaml eval_cases in registry
protect-agent-config 24 8
protect-commit-privacy 20 7
scan-secrets 20 11
block-wildcard-agent-permissions 11 7

Every one of those suites has an execute: block on every case, so these are executable cases
that the registry is undercounting — not aspirational ones.

Why this matters more than a wrong number. This repository's entire proposition is that its
labels can be trusted: 39 policies, 17 enforced, 22 advisory, each claim backed by something. The
README publishes those counts and tools/check_readme.py verifies them. But eval_cases and
eval_executed are published in the same file and tools/check_registry.py does not look at
them at all
— grep it, there are zero references. So a number that speaks to how well-tested a
guard is can drift indefinitely without anything noticing. That is exactly the class of unchecked
claim the catalog exists to argue against.

What to do:

  1. Work out what eval_executed is actually supposed to mean. It is 0 for several policies
    whose evals demonstrably do run (chock check --only evals reports hundreds of passes), so
    either the field means something narrower than "cases that ran" or it is simply never updated.
    Whichever it is, the answer belongs in a comment beside the field.
  2. Extend tools/check_registry.py to verify both counts against the suites, the way
    check_readme.py already verifies the README's counts against the registry.
  3. Correct the current values so the new check passes.

Do not fix this by deleting the fields. A count that is hard to keep honest is still worth
keeping honest — and if the conclusion really is that eval_executed cannot be defined
meaningfully, say that explicitly in the PR and remove it deliberately rather than quietly.

Sized to an afternoon. Files: registry.yaml, tools/check_registry.py, and the evals/suite.yaml
under each base/<policy>/.

Filed with the wrong numbers initially and corrected in place — the original text repeated
the registry's own stale counts instead of checking them against the suites, which is the very
mistake this issue is about.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions