Skip to content

ci: run the parity repository-acceptance suite on the unfiltered leg - #2582

Merged
ryanbr merged 1 commit into
mainfrom
fix/2567-acceptance-runs-unfiltered
Sep 29, 2026
Merged

ryanbr merged 1 commit into
mainfrom
fix/2567-acceptance-runs-unfiltered

Conversation

@ryanbr

@ryanbr ryanbr commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Tools/tests/test_parity_governance_acceptance.py does not test the scanner. It asserts that the checked-in authority and baseline still reproduce from the current product source, so a product change is what invalidates it.

It ran only in parity-governance.yml, whose filter names Tools/parity_*. So a change could move the declaration and call-site graph the ledger measures without the gate that measures it ever running.

This is what happened today

#2557 took the only production call to primarySessionRestingHR on both platforms, turning it test-only. The two resulting findings went unrecorded. It had seventeen green checks and none of them was this one.

main then carried a red ratchet until #2581, and the failure surfaced on #2577, an outside contributor's PR that had caused none of it. parity-governance.yml's own header notes that this shape had already happened four times, three of them landing on an outside contributor.

The header also names the cause exactly:

One of these tests is not about the scanner at all: the repository acceptance asserts that the checked-in authority still reproduces from the CURRENT product source. Product changes are what invalidates it, and product changes are exactly what both filters exclude.

It chose a daily schedule as the mitigation, which bounds exposure to about a day. Today's exposure was about an hour and still reached a contributor.

Why on the unfiltered leg and not behind a product-path filter

Because the guard test in this very file already gives the reason, about a different suite:

Naming the product paths instead would only move the trap, since the list is whatever the assertions happen to read today.

#1691 is the precedent: a Tools/** filter silenced a suite that reads TodayScreen.kt and AppModel.swift, and main carried it red through nine merges. The remedy this repository settled on is no filter, which is why the core leg has none. This suite reads product source for a living, so it belongs there.

Its own step and its own floor, so the core floor of 114 stays independent. That is the reasoning already written beside that floor.

Cost, measured rather than estimated

About 134 s for 49 tests on the CI runner, against about 6 s for the core step beside it. Confirmed from this PR's own run, which is the first time the step executed: it collected 49 tests, matching its floor.

Two corrections worth recording, because both were mine and both were caught by measuring rather than by thinking harder.

I first assumed this suite cost 4 to 6 minutes and designed a separate parallel workflow around that. The 4 to 6 minutes is parity_ratchet.py, which checks out the base tree and derives twice; this suite derives once. That removed the need for the more complicated design.

Then I wrote 51 s into the workflow comment, which is what it takes on my workstation. On the runner it is about 134 s. In a CI file "on this runner" reads as the CI runner, so that number was misleading where it sat. It now says both, and says not to size it from a laptop.

The practical effect: on any product PR this runs in parallel with the Swift and Android legs, which are minutes long, so the wall-clock cost is roughly nothing. A docs-only PR pays the full 134 s.

The scanner's own multi-minute unit tests stay path-filtered in parity-governance.yml, where the "repeated self-testing of the scanner" argument in its header genuinely applies.

The guard

test_core_tools_filter_covers_every_governance_tool_path now asserts the unfiltered leg actually invokes the module.

The failure mode is not someone arguing against this change. It is someone dropping the step, or hanging a filter off it, in six months. That is precisely what #1691 was.

  • Mutation-verified: pointing the step at a different module makes the guard fail.
  • The assertion reads the dotted module path, which occurs only in the invocation and not in the prose above it, so a comment cannot satisfy it. I checked, because a guard satisfiable by its own documentation is not a guard.

Verification

tests.test_parity_governance_acceptance and tests.test_parity_disposition_kinds: 51 tests, pass. The workflow parses.

Refs #2567, #2557, #1691

`Tools/tests/test_parity_governance_acceptance.py` does not test the scanner. It
asserts that the CHECKED-IN authority and baseline still reproduce from the
CURRENT product source, so a product change is what invalidates it. It ran only
in parity-governance.yml, whose filter names `Tools/parity_*`.

So a change could move the declaration and call-site graph the ledger measures
without the gate that measures it ever running.

That is not hypothetical. Earlier today #2557 took the only production call to
`primarySessionRestingHR` on both platforms, turning it test-only, and the two
resulting findings went unrecorded. Seventeen green checks, none of them this
one. `main` carried a red ratchet until #2581, and it surfaced on #2577, an
outside contributor's PR that had caused none of it. parity-governance.yml's own
header notes that shape had already happened four times, three of them on an
outside contributor.

Here, and not behind a filter naming product paths, for the reason the guard test
below already gives about a different suite: the list would be whatever the
assertions happen to read today, so a filter only moves the trap. #1691 is the
precedent, where a `Tools/**` filter silenced a suite that reads
`TodayScreen.kt` and `AppModel.swift`, and main carried it red through nine
merges.

Its own step and its own floor, so the core floor of 114 stays independent, which
is the reasoning already written beside that floor.

Cost, measured on this runner rather than estimated: 51 s for 49 tests, against
roughly 25 s for the rest of the leg. I first assumed this suite cost 4 to 6
minutes and designed a separate parallel workflow around that; the 4 to 6 minutes
is `parity_ratchet.py`, which checks out the base tree and derives twice. This
suite derives once.

The guard is extended to assert the unfiltered leg actually invokes the module.
The failure mode is not someone arguing against this, it is someone dropping the
step or hanging a filter off it later, which is exactly what #1691 was.
Mutation-verified: pointing the step at a different module makes the guard fail.
The assertion reads the dotted module path, which appears only in the invocation
and not in the prose above it, so a comment cannot satisfy it.

Verified: `tests.test_parity_governance_acceptance` plus
`tests.test_parity_disposition_kinds`, 51 tests, pass. The workflow parses.

Refs #2567, #2557, #1691
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant