Make assay's own guards provably able to fail

[hseshadr]

The claim this PR touches

Every property assay states out loud — the ranking metrics being trec_eval's arithmetic, the refusals refusing, the envelope pinning its signer, the ledger's three integrity checks, and the numeric literals the README promises — is now guarded by a test that has been watched go red for the right reason, by a harness anyone can re-run.

Why now. assay's ranking-face red runs existed only as prose in commit 1c382ce. No mutation harness was committed anywhere in the repo, so no reader could re-run them. A guard nobody can watch fail is not evidence — and that is especially sharp here, since assay is the repo that supplies measurement tooling to the rest of the portfolio.

One command

uv run poe mutants

Evidence: 18/18 guards fired

mutation	claim broken	before	after	verdict
ranking-order-reaches-trec-eval	the ranked ORDER is what trec_eval scores, not merely the retrieved set	0	1	RED
ranking-k-reaches-trec-eval	the cut-off k is handed to the engine, not ignored	0	1	RED
ranking-recall-is-not-precision	recall@k divides by the relevant set, never by k	0	1	RED
ranking-graded-gains-reach-trec-eval	graded relevance is graded, not collapsed to binary	0	1	RED
ranking-average-precision-is-trec-eval-map	average_precision is trec_eval's AP	0	1	RED
ranking-f1-is-the-harmonic-mean	F1@k is harmonic, not arithmetic	0	1	RED
ranking-refuses-an-empty-relevant-set	refused, not scored 0.0	0	1	RED
ranking-refuses-a-fractional-gain	refused, never silently rounded	0	1	RED
ranking-refuses-a-duplicate-document	refused, not de-duplicated	0	1	RED
envelope-recomputes-the-payload-hash	the hash is re-derived, not believed	0	1	RED
envelope-pins-the-signer	an unpinned key is rejected before any maths	0	1	RED
ledger-chains-each-entry-to-the-one-before	the chain is walked link by link	0	1	RED
ledger-requires-the-pinned-head	a truncated ledger is refused against the pin	0	1	RED
ledger-verifies-every-entry-signature	a re-hashed forgery cannot launder itself	0	1	RED
documented-sample-floor-is-30	the README's floor of 30 samples ships	0	1	RED
documented-confidence-level-is-95pc	the README's "95% interval" is 95%	0	1	RED
documented-ranking-cutoff-is-10	the documented first-page depth ships	0	1	RED
documented-golden-vector-count-is-12	9 canonical + 3 receipt vectors still ship	0	1	RED

18/18 guards fired when their claim was broken.
whole suite after restore: exit 0 (green)

The verdict is the exit code, never grepped output

0 all passed, 1 a test failed, 4 usage error, 5 nothing collected. Only 1 counts as a guard firing; every other code gets its own name and fails the run. A harness elsewhere in this portfolio passed --reporter=basic to a Vitest with no such reporter, every run crashed, the grep read the crash as "no failures", and all 12 mutations reported green.

Each mutation is also read back off disk before its verdict is trusted, and the anchor must occur exactly once or the harness hard-stops — a mutation that silently did not apply can never be scored as a guard that held.

The harness was itself watched failing, three ways

negative control	result	harness exit
anchor no longer in the source	MutationNotAppliedError: anchor occurs 0x, expected exactly 1	1
guard pointed at a test that cannot notice	SURVIVED — the guard is blind to this break	1
guard node id that does not exist	BASELINE NOT GREEN (exit 4) — the guard was not passing to start	1

A real finding: the ledger chain walk had no isolating test

Deleting the chain-link check left the whole suite green. The existing splice test survived it because a spliced entry also changes the entry count, so the pinned-head check caught it first — the chain walk was defended only in depth.

test_should_reject_a_ledger_whose_chain_link_was_rewritten isolates it: rewrite an interior entry's prev_hash and nothing else. The last entry is untouched so the pinned head still matches on count and hash, and prev_hash sits outside the signed receipt so every signature still verifies. Only the chain walk can object.

A second finding: two vector-count assertions measured shape

test_vectors.py asserted >= 8 canonicalization vectors and >= 1 receipts while the README promises 9, 3 and 12. Three vectors could go missing with the suite green. tests/test_documented_constants.py now pins the literals. No existing assertion was weakened or deleted — the loose bounds stay as the shape check they are.

Also in here

• .github/dependabot.yml (github-actions, weekly). Verified: assay had no Dependabot config, which is exactly why it was the only repo in the portfolio still on astral-sh/setup-uv v8.3.2 while everything else moved to v9.0.0. A SHA pin does not move on its own.
• Both pins bumped to v9.0.0 (c771a70e6277c0a99b617c7a806ffedaca235ff9, verified against the upstream tag), with the exact version in the comment — never a floating # v9.
• Vulnerability alerts confirmed on: GET /repos/hseshadr/assay/vulnerability-alerts → 204; automated-security-fixes → {"enabled":true,"paused":false}.
• A mutation-gate CI job, so the evidence regenerates on every PR instead of being captured once. Deliberately not a step inside gate: gate asks whether the tests pass, this asks whether they can fail.
• The gate now covers scripts/ (ruff, ruff-format, mypy --strict, xenon A). The harness that proves the guards can fail is not exempt from the gate that guards them.
• README Status corrected — it claimed 220 tests and avow 0.2.0 as published; it is 258 and 0.3.0.

Gates

gate	exit
uv run poe gate	0 — 187 tests, 100% statement and branch coverage (803 stmts, 68 branches, none missed)
pnpm -r --include-workspace-root gate (Node 24)	0 — 40 @edgeproc/avow + 31 @edgeproc/receipt-ui, 100% coverage both
uv run poe mutants	0 — 18/18

Nothing merged, tagged or published.

🤖 Generated with Claude Code

https://claude.ai/code/session_015o7tjWLFZvzRv4KyNfDukx

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.