From e2d12bdd1203666f133be3f766d908950e7f4ee0 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 11:54:35 +0200 Subject: [PATCH 1/4] ci(diet): coverage on tags only, perf-gate table path-scoped, model suites nightly, mutants only on .rs, gate's skipped rule tightened (#3676) Refs #3676 #3668 #3658 Operator 2026-09-21: aprender releases as fast as possible, intel idle; "YES, coverage on tags release only." All five workflow items approved by Noah directly (CLAUDE.md check-in for .github/workflows edits). 1. COVERAGE -- ci.yml `skip_coverage: true` on the sovereign-ci call. That job gated nothing (no coverage_min) and measured nothing (`--lib` on the facade root: 0 tests, per the existing NOTE), at 237 s per run (main 35563537942). coverage-nightly.yml now also triggers on `push: tags: ['v*']` (COV_FLOOR via `make coverage`), concurrency per ref so a tag run and the nightly do not cancel each other. Every consumer enumerated first (the rmedia trap aprender-04 relayed): the pre-publish dogfood's `make coverage-check` (produces its own llvm-cov), the queue steward (reacts only to `failure`), two tests reading an embedded string / the Makefile, a legacy prepare-release.sh (local llvm-cov). None reads CI's number. The one output lost: that job's advisory Codecov upload. NEW scripts/check_coverage_has_producers.sh asserts the chain: R1 `make -n coverage-check` reaches llvm-cov; R2 dogfood runs it (non-comment); R3 coverage-nightly's `on:` block has schedule AND tags [v*] (comments and other top-level keys don't count); R4 numeric COV_FLOOR. 10 rows, each link broken -> RED; missing file -> ENV 2. NOT done here, stated: making T-4 WAIT on the tag's coverage run is an autopilot change, deliberately not made on the 0.69 cut day. 2. BOOK -- no change. book.yml and book-contracts.yml are already path-filtered; book-contracts watches crates/aprender-core/src/** because chapter examples run aprender-core code, which check_workflow_path_filters.sh RULE 2 requires ("a gate that runs code must watch the code it runs"). Narrowing it would break that rule. 3a. PERF GATE TABLE -- the explicit guard-tree step (405 s on main 35563537942) is replaced by NEW scripts/check_perf_gate_selftest_scoped.sh, dispatched by guard_tree.sh --no-cargo: it runs `perf_gate.sh --selftest` only when the tree diff vs origin/main touches its DERIVED scope -- every $ROOT/ path perf_gate.sh reads, followed transitively through the scripts it calls (8 paths today) -- and otherwise prints a SUMMARY skip line. Unknown comparand or a failed diff -> RUN, never skip. The table runs whole nightly (guards-nightly.yml + manifest, 405 s). 11-row case table; mutants RED: scope not transitive, renames hiding the old side, unknown -> skip, prefix-string matching. Registered in check_no_timing_in_required.sh's META_GUARDS (reads no clock). 3b. MODEL SUITES -- the PR step keeps ONLY falsification_spec_v10_tests: it holds the SATD ratchet measured against origin/main, the one property here a diff can change (and vacuous on the nightly, where HEAD == origin/main). The stress and gpu_state suites judge the whole tree; guards-nightly.yml already runs all three. check_model_tests_wired: all 3 still wired. 4. MUTANTS -- the diff step emits has_rs (`git diff --no-renames --name-only`); image pull, `cargo install cargo-mutants`, the run and the upload are gated on it. A diff with no .rs passes quickly instead of installing the tool. 5. GATE -- the mutants rule, between GATE-MUTANTS-RULE markers: on a pull_request mutants must SUCCEED (a no-.rs diff succeeds); `skipped` passes only on other events, where the job's own `if:` skips it. The old rule ("success/skipped both pass") passed a PR whose mutants never ran. NEW scripts/check_ci_gate_mutants_rule.sh EXTRACTS that block from ci.yml and executes it for 10 (event, result) pairs; its --self-test plants the old rule and it goes RED (3 rows); no block -> ENV 2. Gates: guard_tree --no-cargo: all PASS except check_fleet_pv_shapes_gate.sh, the known lambda-box fleet-pv condition (identical on main; fixed in #3669); guard_tree_test 23/0; guards-are-wired, guards-nightly-manifest (8 entries), no-timing-in-required, model-tests-wired PASS; bashrs 0 errors on the new files, bashrs-gate PASS; actionlint: the same 7 shellcheck infos as main, nothing new; roadmap aggregate idempotent; fmt, deny, contracts 1666. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 69 ++++++++-- .github/workflows/coverage-nightly.yml | 10 +- .github/workflows/guards-nightly.yml | 2 + docs/roadmaps/entries/PMAT-3676.yaml | 17 +++ docs/roadmaps/roadmap.yaml | 17 +++ scripts/check_ci_gate_mutants_rule.sh | 78 +++++++++++ scripts/check_coverage_has_producers.sh | 105 +++++++++++++++ scripts/check_no_timing_in_required.sh | 7 + scripts/check_perf_gate_selftest_scoped.sh | 147 +++++++++++++++++++++ scripts/guards_nightly_manifest.txt | 5 +- 10 files changed, 444 insertions(+), 13 deletions(-) create mode 100644 docs/roadmaps/entries/PMAT-3676.yaml create mode 100644 scripts/check_ci_gate_mutants_rule.sh create mode 100644 scripts/check_coverage_has_producers.sh create mode 100644 scripts/check_perf_gate_selftest_scoped.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 92fa61de8b..500fcbaf92 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -112,6 +112,17 @@ jobs: # MECHANISM is live fleet-wide via sovereign-ci (#37); a single-crate repo is # the natural first coverage pilot. aprender's blocking-quality pilot is the # diff-scoped mutation gate below. + # + # #3676 (operator 2026-09-21: "YES, coverage on tags release only"): so this + # job gated nothing AND measured nothing (0 tests on the facade root), at the + # cost of an instrumented rebuild on every PR, merge group and push (237 s + # on main run 35563537942). Skipped here. Every consumer of aprender's + # coverage number has its own producer, asserted by + # scripts/check_coverage_has_producers.sh: the pre-publish dogfood runs + # `make coverage-check` (-> `make coverage`: workspace llvm-cov, COV_FLOOR), + # and coverage-nightly.yml runs nightly AND on every `v*` tag. The one + # output lost is this job's advisory Codecov upload. + skip_coverage: true secrets: inherit # APR-MONO: Workspace-wide test (all 75 crates) @@ -1151,8 +1162,14 @@ jobs: # unwiring the §4.9.1 signature arm drops it to 23 passed, 6 broken), so a # gate that silently stops discriminating now fails here rather than # certifying a release later. - - name: Perf gate can still discriminate (APR-PERF-GATE-001, case table) - run: bash scripts/perf_gate.sh --selftest + # + # #3676: PATH-SCOPED now. The table's verdict changes only when perf_gate.sh + # or a file it reads changes, and it cost 405 s on every PR, merge group and + # push (main run 35563537942). scripts/check_perf_gate_selftest_scoped.sh -- + # dispatched by guard_tree.sh --no-cargo above, no step here -- runs it when + # the diff vs origin/main touches its DERIVED scope (every $ROOT/ path it + # reads, followed through the scripts it calls) and says so on a SUMMARY + # line when it does not; guards-nightly.yml runs the table whole. # PERF-007, the HOST half of §4.9.1. `perf_gate.sh --selftest` already # runs this table, but check_guards_are_wired.sh cannot see one level of # indirection and its remedy for an unwired guard is to wire it, not to @@ -2157,9 +2174,13 @@ jobs: # 3. the suites, judged against the measured ceiling DR_ENV=(-e "SATD_BASELINE=$n") dr "$GITHUB_WORKSPACE" cargo test -p aprender-core --features model-tests \ - --test falsification_spec_v10_tests \ - --test falsification_stress_tests \ - --test falsification_gpu_state_tests + --test falsification_spec_v10_tests + # #3676: the PR path keeps ONLY falsification_spec_v10_tests -- it holds the + # SATD ratchet (F-CHECKLIST-005) measured against origin/main above, the one + # property here a diff can change, and the reason this step was KEPT at the + # 67-E3 merge (on the nightly HEAD == origin/main, so there the ratchet is + # vacuous). falsification_stress_tests and falsification_gpu_state_tests + # judge the whole tree, not the diff; guards-nightly.yml runs all three. # 4. the satd class of the ratchet-semantics case table (rebuilds nothing) DR_ENV=() dr "$GITHUB_WORKSPACE" bash scripts/tests/ratchet_semantics_test.sh --class satd @@ -3073,11 +3094,22 @@ jobs: # main-branch pushes are not blocked by a job that intentionally # did not run. Only an explicit `failure` blocks. MUT="${{ needs.mutants.result }}" - if [ "$MUT" = "failure" ]; then - echo "mutants (diff-scoped mutation) failed: $MUT" - exit 1 - fi - echo "mutants result: $MUT (success/skipped both pass)" + EVT="${{ github.event_name }}" + # --- GATE-MUTANTS-RULE-BEGIN (#3676; scripts/check_ci_gate_mutants_rule.sh runs this block) --- + # `skipped` is a pass ONLY when the skip is the job's own filter: + # mutants declares `if: github.event_name == 'pull_request'`, so on every + # other event it is skipped by design. On a pull_request it must have RUN + # and succeeded -- a diff with no .rs still succeeds (its steps skip, the + # job passes), so a pull_request `skipped` or `cancelled` means the job + # never ran, and the old rule ("success/skipped both pass") passed that. + case "$EVT:$MUT" in + pull_request:success) echo "mutants result: success on a pull_request" ;; + pull_request:*) echo "mutants on a pull_request must succeed; got '$MUT' -- a skip here is not the job's filter"; exit 1 ;; + *:skipped) echo "mutants result: skipped on $EVT (its filter: pull_request only)" ;; + *:success) echo "mutants result: success on $EVT" ;; + *) echo "mutants (diff-scoped mutation) on $EVT: $MUT"; exit 1 ;; + esac + # --- GATE-MUTANTS-RULE-END --- # Receipt presence is NOT read here any more (PP-066 C0-5, PRQ-013, # #2982). This job runs under the PR head's definition of ci.yml, so a # receipt clause here was a check the PR under review could edit. The @@ -3137,6 +3169,7 @@ jobs: # Need history + base branch to compute the PR diff for --in-diff. fetch-depth: 0 - name: Compute PR diff for mutation scoping + id: prdiff # cargo-mutants --in-diff takes a unified diff and mutates ONLY the # lines it adds/changes. We diff the PR head against the merge-base with # the target branch so the scope is exactly "what this PR introduces". @@ -3159,7 +3192,19 @@ jobs: if [ ! -s pr.diff ]; then echo "Empty diff — no code to mutate" fi + # #3676: cargo-mutants mutates Rust. A diff with no .rs path has nothing to + # mutate, but used to pull the image and `cargo install cargo-mutants` + # anyway. --no-renames: a .rs file renamed away still counts. + git diff --no-renames --name-only "$MERGE_BASE"...HEAD > pr.files + if grep -qE '\.rs$' pr.files; then + echo "has_rs=true" >> "$GITHUB_OUTPUT" + echo "Rust paths in the diff: $(grep -cE '\.rs$' pr.files)" + else + echo "has_rs=false" >> "$GITHUB_OUTPUT" + echo "No .rs path in the diff ($(grep -c . pr.files || true) path(s)) -- nothing to mutate; the mutation steps skip and the job passes" + fi - name: Pull sovereign-ci image (with retry + local-cache fallback) + if: steps.prdiff.outputs.has_rs == 'true' # Same two-layer resilience as workspace-test — see that job for full context. run: | if docker image inspect "$IMAGE" > /dev/null 2>&1; then @@ -3182,6 +3227,7 @@ jobs: delay=$((delay + 6)) done - name: Install cargo-mutants + if: steps.prdiff.outputs.has_rs == 'true' run: | docker run --rm \ -e CI -e GITHUB_ACTIONS -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_RUN_ID -e GITHUB_EVENT_NAME -e GITHUB_WORKFLOW \ @@ -3190,6 +3236,7 @@ jobs: "$IMAGE" \ cargo install cargo-mutants --locked - name: Run diff-scoped mutation testing (BLOCKING) + if: steps.prdiff.outputs.has_rs == 'true' # No continue-on-error: a missed mutant on the PR diff fails the job, # which fails `gate`, which blocks merge. --in-diff pr.diff restricts # mutation to PR-touched lines. Empty diff → 0 mutants → clean pass. @@ -3290,7 +3337,7 @@ jobs: exit 0 ' - name: Upload mutation results - if: always() + if: always() && steps.prdiff.outputs.has_rs == 'true' uses: actions/upload-artifact@v7 with: name: mutation-results diff --git a/.github/workflows/coverage-nightly.yml b/.github/workflows/coverage-nightly.yml index 19e49ea0e9..52ff9ea760 100644 --- a/.github/workflows/coverage-nightly.yml +++ b/.github/workflows/coverage-nightly.yml @@ -33,13 +33,21 @@ on: # `gh workflow run` (#3292); until then, re-measure before trusting. - cron: '0 22 * * *' # 22:00 UTC -> lands ~03:00 UTC workflow_dispatch: {} + # #3676 (operator 2026-09-21: "YES, coverage on tags release only"): coverage + # left the PR / merge-group / push path, so the RELEASE TAG runs it here, with + # the same COV_FLOOR `make coverage` enforces nightly. A floor breach reds the + # tag's run; scripts/check_coverage_has_producers.sh keeps this trigger present. + push: + tags: ['v*'] permissions: contents: read issues: write # open/update the single coverage tracking issue concurrency: - group: coverage-nightly + # Per ref (#3676): a tag's run and the nightly on main must not cancel each + # other. A second nightly on the same ref still supersedes the first. + group: coverage-nightly-${{ github.ref }} cancel-in-progress: true jobs: diff --git a/.github/workflows/guards-nightly.yml b/.github/workflows/guards-nightly.yml index deef9af9d2..9ac9ed527b 100644 --- a/.github/workflows/guards-nightly.yml +++ b/.github/workflows/guards-nightly.yml @@ -170,6 +170,8 @@ jobs: run: bash scripts/check_hermetic_stdin_tests.sh - name: Publish safety (binary/oversize files in published packages) run: bash scripts/check_publish_safety.sh + - name: Perf gate can still discriminate (APR-PERF-GATE-001, case table) + run: bash scripts/perf_gate.sh --selftest - name: Book rust examples compile run: bash scripts/check_book_examples_compile.sh - name: aprender-profile lib tests (was gated by nothing) diff --git a/docs/roadmaps/entries/PMAT-3676.yaml b/docs/roadmaps/entries/PMAT-3676.yaml new file mode 100644 index 0000000000..eab79f976b --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3676.yaml @@ -0,0 +1,17 @@ +- id: PMAT-3676 + github_issue: 3676 + item_type: task + title: 'CI diet: aprender itself burns 99 intel runner-hours/day — coverage on tag/nightly only, path-filter book jobs, perf/model suites nightly, mutants only on .rs diffs' + status: planned + priority: critical + assigned_to: null + created: 2026-09-21T09:50:38Z + updated: 2026-09-21T09:50:38Z + spec: null + acceptance_criteria: [] + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'ACCEPTANCE (quoted from issue #3676 "Proposal" + "done_when"; operator 08:38Z: "YES, coverage on tags release only"; Noah approved all five workflow items directly to aprender-b3): 1. Coverage (ci / coverage) on the release tag and the nightly only, never on pull_request or merge_group. COV_FLOOR enforcement moves to the tag gate (T-2) and the nightly, where a regression still blocks the release. 2. Book Chapter Examples and mdBook: path-filtered. They run when book/** or the crates a chapter imports change, and nightly always. 3. The perf gate case table and the model-tests falsification suites move to nightly. The PR path keeps a smoke row. 4. mutants runs only when the diff has .rs. 5. Every moved job still runs on the tag. check_workflow_path_filters.sh and the gate aggregator are updated so a skipped job is a pass only when the skip reason is the documented filter. done_when: intel runner-minutes per 24 h drop by >= 40% at equal PR throughput (measured after merge); the tag gate still runs coverage with COV_FLOOR, and a planted coverage drop fails the tag (control); a book-only PR runs the book jobs, a crate-only PR not imported by a chapter does not (both rows); gate accepts skipped only for jobs whose filter says skip, any other skipped state fails (mutation). COP ADDENDUM (aprender-04, relaying rmedia-88): before removing coverage from ci.yml, enumerate every consumer of the coverage number and prove each still has a producer, with a guard asserting the chain.' diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index 84e48c530f..8549de6139 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -19007,3 +19007,20 @@ roadmap: labels: - kind:code notes: 'ACCEPTANCE (hand-entered from issue #3629 done_when; minted on the PR branch so the AD-04 quorum can run -- pmat work status refuses a missing item). Done when: scripts/fleet_utilization.sh --selftest has no pipe into grep -q; a mutation that restores the pipe AND forces EPIPE (out padded to >64 KiB so the write cannot complete before grep exits) turns the self-test RED deterministically — the positive control for the race, in the PR body the 55-site count is in the PR body from the command that produced it, and a guard refuses a new site (check_guards_are_wired.sh names it; ratchet baseline committed) main''s guard-tree green on the merge commit without rerun ' +- id: PMAT-3676 + github_issue: 3676 + item_type: task + title: 'CI diet: aprender itself burns 99 intel runner-hours/day — coverage on tag/nightly only, path-filter book jobs, perf/model suites nightly, mutants only on .rs diffs' + status: planned + priority: critical + assigned_to: null + created: 2026-09-21T09:50:38Z + updated: 2026-09-21T09:50:38Z + spec: null + acceptance_criteria: [] + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'ACCEPTANCE (quoted from issue #3676 "Proposal" + "done_when"; operator 08:38Z: "YES, coverage on tags release only"; Noah approved all five workflow items directly to aprender-b3): 1. Coverage (ci / coverage) on the release tag and the nightly only, never on pull_request or merge_group. COV_FLOOR enforcement moves to the tag gate (T-2) and the nightly, where a regression still blocks the release. 2. Book Chapter Examples and mdBook: path-filtered. They run when book/** or the crates a chapter imports change, and nightly always. 3. The perf gate case table and the model-tests falsification suites move to nightly. The PR path keeps a smoke row. 4. mutants runs only when the diff has .rs. 5. Every moved job still runs on the tag. check_workflow_path_filters.sh and the gate aggregator are updated so a skipped job is a pass only when the skip reason is the documented filter. done_when: intel runner-minutes per 24 h drop by >= 40% at equal PR throughput (measured after merge); the tag gate still runs coverage with COV_FLOOR, and a planted coverage drop fails the tag (control); a book-only PR runs the book jobs, a crate-only PR not imported by a chapter does not (both rows); gate accepts skipped only for jobs whose filter says skip, any other skipped state fails (mutation). COP ADDENDUM (aprender-04, relaying rmedia-88): before removing coverage from ci.yml, enumerate every consumer of the coverage number and prove each still has a producer, with a guard asserting the chain.' diff --git a/scripts/check_ci_gate_mutants_rule.sh b/scripts/check_ci_gate_mutants_rule.sh new file mode 100644 index 0000000000..e4b0a06a71 --- /dev/null +++ b/scripts/check_ci_gate_mutants_rule.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash +# check_ci_gate_mutants_rule.sh -- the `gate` job accepts a SKIPPED mutants job only +# when the skip is that job's own filter (#3676). +# +# mutants runs only on pull_request (`if: github.event_name == 'pull_request'`), +# so on merge_group, push and workflow_dispatch it is skipped by design. On a +# pull_request it must RUN: a diff with no .rs still succeeds, because its +# mutation steps skip and the job passes. The old rule read "success/skipped both +# pass", which also passed a pull_request whose mutants job never ran at all. +# +# This guard does not re-implement the rule. It EXTRACTS the block between the +# GATE-MUTANTS-RULE-BEGIN/END markers from .github/workflows/ci.yml and executes +# it for every (event, result) pair, so it judges the code the gate runs. A +# missing or empty block is ENV rc=2, never a pass. +# +# check_ci_gate_mutants_rule.sh the case table over ci.yml's block +# check_ci_gate_mutants_rule.sh --self-test the old rule, planted: it must go RED +set -uo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" || exit 2 +WF="${GATE_RULE_WORKFLOW:-$ROOT/.github/workflows/ci.yml}" + +# extract -> the rule block, dedented; empty when the markers are absent +extract() { + awk '/GATE-MUTANTS-RULE-BEGIN/{f=1; next} /GATE-MUTANTS-RULE-END/{f=0} f' "$1" | sed -E 's/^ {10}//' +} + +# verdict -> exit status of the block with EVT/MUT set +verdict() { + EVT=$2 MUT=$3 bash -c "$1" >/dev/null 2>&1 +} + +table() { # table -> 0 iff every row holds + local blk bad=0 n=0 evt mut want got + blk=$(extract "$1") + [ -n "$blk" ] || { printf 'ENV no GATE-MUTANTS-RULE block in %s -- cannot judge, not a pass\n' "$1" >&2; return 2; } + while read -r evt mut want; do + [ -n "$evt" ] || continue + n=$((n + 1)); got=0; verdict "$blk" "$evt" "$mut" || got=1 + if [ "$got" = "$want" ]; then printf 'ok row %-2s %-14s %-10s -> %s\n' "$n" "$evt" "$mut" "$([ "$want" = 0 ] && echo pass || echo FAIL)" + else printf 'FAIL row %-2s %-14s %-10s wanted %s, got %s\n' "$n" "$evt" "$mut" "$want" "$got" >&2; bad=1; fi + done <<'ROWS' +pull_request success 0 +pull_request skipped 1 +pull_request cancelled 1 +pull_request failure 1 +merge_group skipped 0 +push skipped 0 +workflow_dispatch skipped 0 +merge_group success 0 +merge_group failure 1 +merge_group cancelled 1 +ROWS + return "$bad" +} + +case "${1:-}" in -h|--help) sed -n '2,18p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; esac +[ -f "$WF" ] || { printf 'ENV %s is missing\n' "$WF" >&2; exit 2; } + +if [ "${1:-}" = "--self-test" ]; then + echo "=== gate mutants rule: the old rule must turn the table RED ===" + d=$(mktemp -d "${TMPDIR:-/tmp}/gate-rule.XXXXXX") || exit 2 + rmtree() { case "${1:-}" in ''|/) return 0 ;; *) [ -d "$1" ] && rm -rf -- "$1" ;; esac; return 0; } + trap 'rmtree "${d:-}"' EXIT + printf ' # --- GATE-MUTANTS-RULE-BEGIN ---\n if [ "$MUT" = "failure" ]; then exit 1; fi\n # --- GATE-MUTANTS-RULE-END ---\n' > "$d/old.yml" + printf 'jobs: {}\n' > "$d/none.yml" + bad=0 + if table "$d/old.yml" > "$d/out" 2>&1; then printf 'FAIL the OLD rule ("success/skipped both pass") passed the table\n'; bad=1 + else printf 'ok the OLD rule is RED: %s\n' "$(grep -c '^FAIL' "$d/out") row(s), e.g. $(grep -m1 '^FAIL' "$d/out" | cut -c7-70)"; fi + rc=0; table "$d/none.yml" > /dev/null 2>&1 || rc=$? + [ "$rc" = 2 ] && printf 'ok a workflow with no rule block is ENV rc=2, never a pass\n' || { printf 'FAIL no rule block gave rc=%s\n' "$rc"; bad=1; } + [ "$bad" = 0 ] && { echo "SELF-TEST PASSED"; exit 0; } + echo "SELF-TEST FAILED" >&2; exit 1 +fi + +echo "=== gate accepts a skipped mutants job only when the skip is its filter (check_ci_gate_mutants_rule.sh) ===" +table "$WF"; rc=$? +[ "$rc" = 0 ] && echo PASS || echo "FAIL (rc=$rc)" >&2 +exit "$rc" diff --git a/scripts/check_coverage_has_producers.sh b/scripts/check_coverage_has_producers.sh new file mode 100644 index 0000000000..22d911eb69 --- /dev/null +++ b/scripts/check_coverage_has_producers.sh @@ -0,0 +1,105 @@ +#!/usr/bin/env bash +# check_coverage_has_producers.sh -- every consumer of aprender's coverage number +# still has a producer (#3676). +# +# Operator 2026-09-21: "YES, coverage on tags release only." `ci / coverage` left +# the PR / merge-group / push path (ci.yml `skip_coverage: true`; it had gated +# nothing and measured 0 tests on the facade root). A deletion like that can +# starve a gate that READ the number from CI -- rmedia's release gate did +# exactly that ("could not look"). So the chain is asserted, not remembered: +# +# R1 `make -n coverage-check` reaches llvm-cov -- the PRODUCER the +# pre-publish dogfood uses measures coverage itself, reads nothing from CI. +# R2 scripts/dogfood.sh still runs `make ... coverage-check` -- the release +# CONSUMER exists (without it the chain would be vacuously intact). +# R3 .github/workflows/coverage-nightly.yml triggers on `schedule:` AND on a +# `push:` of `v*` tags -- the tag gets its own COV_FLOOR run. +# R4 the Makefile defines a numeric COV_FLOOR -- the floor `make coverage` enforces. +# Any link missing is FAIL; a file that cannot be read is ENV rc=2. +# +# check_coverage_has_producers.sh judge the tree +# check_coverage_has_producers.sh --self-test each rule against fixtures that break it +# +# Test seams: COVCHAIN_ROOT (the tree judged). +set -uo pipefail +ROOT="${COVCHAIN_ROOT:-$(cd "$(dirname "$0")/.." && pwd)}" + +# on_block -> the lines of its top-level `on:` block, comments stripped +on_block() { + awk '/^on:/{f=1; next} f && /^[^[:space:]#]/{exit} f' "$1" | sed 's/#.*$//' +} + +judge() { # judge -> 0 all links hold, 1 a link broke, 2 ENV + local r=$1 bad=0 dry blk + local mk="$r/Makefile" df="$r/scripts/dogfood.sh" wf="$r/.github/workflows/coverage-nightly.yml" + for f in "$mk" "$df" "$wf"; do [ -r "$f" ] || { printf 'ENV %s is not readable -- cannot judge, not a pass\n' "$f"; return 2; }; done + + dry=$(make -n -C "$r" coverage-check 2>/dev/null) || dry="" + if [[ $dry == *"llvm-cov"* ]]; then printf 'ok R1 make -n coverage-check reaches llvm-cov (the release producer measures, it reads nothing from CI)\n' + else printf 'FAIL R1 make -n coverage-check does not reach llvm-cov -- the dogfood coverage gate would measure nothing\n'; bad=1; fi + + if grep -qE '^[[:space:]]*gate[[:space:]]+coverage[[:space:]]+make[[:space:]].*coverage-check' "$df"; then + printf 'ok R2 scripts/dogfood.sh runs make ... coverage-check (the release consumer exists)\n' + else printf 'FAIL R2 scripts/dogfood.sh no longer runs coverage-check -- nothing checks coverage at the release\n'; bad=1; fi + + blk=$(on_block "$wf") + if grep -qE '^[[:space:]]+schedule:' <<<"$blk" && grep -qE "^[[:space:]]+tags:[[:space:]]*\[[^]]*['\"]?v\*['\"]?" <<<"$blk"; then + printf 'ok R3 coverage-nightly.yml triggers on schedule AND on v* tags\n' + else printf 'FAIL R3 coverage-nightly.yml must trigger on schedule AND on push tags [v*] (on: block, comments ignored)\n'; bad=1; fi + + if grep -qE '^COV_FLOOR[[:space:]]*:?=[[:space:]]*[0-9]+' "$mk"; then + printf 'ok R4 the Makefile defines a numeric COV_FLOOR\n' + else printf 'FAIL R4 the Makefile has no numeric COV_FLOOR -- make coverage enforces nothing\n'; bad=1; fi + return "$bad" +} + +case "${1:-}" in -h|--help) sed -n '2,24p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; esac + +if [ "${1:-}" = "--self-test" ]; then + echo "=== coverage producer chain: each rule must turn RED when its link breaks ===" + d=$(mktemp -d "${TMPDIR:-/tmp}/covchain.XXXXXX") || exit 2 + rmtree() { case "${1:-}" in ''|/) return 0 ;; *) [ -d "$1" ] && rm -rf -- "$1" ;; esac; return 0; } + trap 'rmtree "${d:-}"' EXIT + bad=0; n=0 + fixture() { # fixture -> a tree where every link holds + mkdir -p "$1/scripts" "$1/.github/workflows" + printf 'COV_FLOOR := 88\nLLVMCOV := llvm-cov\ncoverage:\n\t$(LLVMCOV) report --fail-under-lines $(COV_FLOOR)\ncoverage-check: coverage\n' > "$1/Makefile" + printf '#!/usr/bin/env bash\n gate coverage make -C "$d" coverage-check\n' > "$1/scripts/dogfood.sh" + printf "on:\n schedule:\n - cron: '0 22 * * *'\n push:\n tags: ['v*']\njobs: {}\n" > "$1/.github/workflows/coverage-nightly.yml" + } + # one mutation per row, as a function (no eval: the mutation is code, not a string) + m_none() { :; } + m_r1() { sed -i 's/\$(LLVMCOV) report.*/echo nothing/' Makefile; } + m_r2_gone() { printf '#!/usr/bin/env bash\n' > scripts/dogfood.sh; } + m_r2_comment() { printf '#!/usr/bin/env bash\n# gate coverage make -C x coverage-check\n' > scripts/dogfood.sh; } + m_r3_gone() { sed -i '/push:/d; /tags:/d' .github/workflows/coverage-nightly.yml; } + m_r3_comment() { sed -i "s/^ tags: \['v\*'\]/ # tags: ['v*']/" .github/workflows/coverage-nightly.yml; } + m_r3_nosched() { sed -i '/schedule:/d; /cron:/d' .github/workflows/coverage-nightly.yml; } + m_r3_otherkey() { printf "on:\n schedule:\n - cron: '0 22 * * *'\nenv:\n tags: ['v*']\njobs: {}\n" > .github/workflows/coverage-nightly.yml; } + m_r4() { sed -i '/^COV_FLOOR/d' Makefile; } + m_missing() { rm -f .github/workflows/coverage-nightly.yml; } + row() { # row WANT-RC LABEL MUTATION-FUNCTION + local want=$1 label=$2 rc=0; n=$((n + 1)); rmtree "$d/t"; fixture "$d/t" + ( cd "$d/t" && "$3" ) || { printf 'FAIL row %s fixture mutation failed: %s\n' "$n" "$label"; bad=1; return; } + judge "$d/t" > "$d/out" 2>&1 || rc=$? + if [ "$rc" = "$want" ]; then printf 'ok row %-2s rc=%s %s\n' "$n" "$rc" "$label" + else printf 'FAIL row %-2s rc=%s (wanted %s) %s\n' "$n" "$rc" "$want" "$label"; sed 's/^/ /' "$d/out"; bad=1; fi + } + row 0 "every link holds -> PASS" m_none + row 1 "R1: coverage no longer runs llvm-cov -> RED" m_r1 + row 1 "R2: dogfood stops calling coverage-check -> RED" m_r2_gone + row 1 "R2: the call only in a COMMENT -> RED" m_r2_comment + row 1 "R3: the tag trigger removed -> RED" m_r3_gone + row 1 "R3: the tag trigger only in a COMMENT -> RED" m_r3_comment + row 1 "R3: schedule removed (tags only) -> RED" m_r3_nosched + row 1 "R3: tags: under a DIFFERENT top-level key, not on: -> RED" m_r3_otherkey + row 1 "R4: COV_FLOOR removed -> RED" m_r4 + row 2 "a missing file is ENV rc=2, never a pass" m_missing + [ "$bad" = 0 ] && { printf 'SELF-TEST PASSED: %s rows\n' "$n"; exit 0; } + printf 'SELF-TEST FAILED\n' >&2; exit 1 +fi + +echo "=== every consumer of aprender's coverage number has a producer (check_coverage_has_producers.sh) ===" +judge "$ROOT"; rc=$? +[ "$rc" = 0 ] && echo PASS || echo "FAIL (rc=$rc)" >&2 +exit "$rc" diff --git a/scripts/check_no_timing_in_required.sh b/scripts/check_no_timing_in_required.sh index bfd0ee39cd..55454bcd20 100755 --- a/scripts/check_no_timing_in_required.sh +++ b/scripts/check_no_timing_in_required.sh @@ -182,9 +182,16 @@ check_no_fabricated_baselines.sh check_no_timing_in_required.sh check_perf_claims_cite_receipts.sh check_perf_concurrency_groups.sh +check_perf_gate_selftest_scoped.sh check_perf_matrix_schema.sh check_perf_receipt_fields_have_producers.sh " +# check_perf_gate_selftest_scoped.sh (#3676) matched on `perf`. It reads no clock: +# it takes a git tree diff against origin/main, decides whether perf_gate.sh's +# DERIVED input set was touched, and if so runs `perf_gate.sh --selftest` -- the +# fixture-driven case table that was ALREADY an explicit required step in +# guard-tree before #3676 moved it behind this scope. A statement about which +# files changed, not about duration. # check_perf_claims_cite_receipts.sh (PERF-010) matches `check_*perf*.sh` and # turned this guard RED the moment the file appeared — verified before wiring # anything, not after: diff --git a/scripts/check_perf_gate_selftest_scoped.sh b/scripts/check_perf_gate_selftest_scoped.sh new file mode 100644 index 0000000000..6482eb99b8 --- /dev/null +++ b/scripts/check_perf_gate_selftest_scoped.sh @@ -0,0 +1,147 @@ +#!/usr/bin/env bash +# check_perf_gate_selftest_scoped.sh -- run perf_gate.sh's case table on a pull +# request ONLY when the change touches something that table reads (#3676). +# +# THE COST. `perf_gate.sh --selftest` is APR-PERF-GATE-001's 29-case table. It +# was an explicit guard-tree step on every PR, merge group and push: 405 s on +# main run 35563537942. Its verdict can only change when perf_gate.sh or a file +# it reads changes; on every other PR it re-proves the same fact about the same +# bytes. Operator 2026-09-21: aprender releases as fast as possible, intel idle. +# +# THE SCOPE IS DERIVED, NOT LISTED. The files the table reads are the +# `$ROOT/` references in perf_gate.sh, plus perf_gate.sh, followed through +# every shell script it references (transitively). A new reference widens the +# scope with no edit here, so the scope cannot silently go +# stale -- the failure a hand-kept path filter has (check_workflow_path_filters.sh +# RULE 2: a gate that runs code must watch the code it runs). +# +# THE DECISION, fail-closed toward RUNNING: +# the diff vs origin/main (a tree diff, --no-renames: a rename lists both sides) +# touches a scoped path (the path itself, or anything under a scoped directory) +# -> run the table; its verdict is this guard's verdict +# touches none of them -> skip, and SAY so on a SUMMARY line guard_tree.sh +# surfaces (#3651): which diff, how many scoped paths, where it runs whole +# origin/main cannot be read, or the diff fails -> RUN (never skip on unknown) +# The table still runs whole every night (.github/workflows/guards-nightly.yml). +# +# check_perf_gate_selftest_scoped.sh decide, and run it if in scope +# check_perf_gate_selftest_scoped.sh --self-test the decision's case table +# +# Test seams: PERF_GATE_SUBJECT (the script whose refs define the scope and which +# is run), PERF_GATE_BASE (the comparand ref, default origin/main). +set -uo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" || exit 2 +SUBJECT="${PERF_GATE_SUBJECT:-$ROOT/scripts/perf_gate.sh}" +BASE="${PERF_GATE_BASE:-origin/main}" + +# scope_of -> the repo-relative paths it reads, one per line (itself first). +# TRANSITIVE over shell scripts: a scoped .sh file's own $ROOT/ refs are in scope too +# (perf_gate.sh -> perf_receipt_sign.sh -> whatever that reads). The python libs it +# reads import only the standard library (measured), so .py files end the walk. +scope_of() { + local s=$1 rel queue next f + rel="${s#"$ROOT/"}" + printf '%s\n' "$rel" + local seen=" $rel " + queue="$s" + while [ -n "$queue" ]; do + next="" + for f in $queue; do + [ -f "$f" ] || continue + while IFS= read -r rel; do + [ -n "$rel" ] || continue + case "$seen" in *" $rel "*) continue ;; esac + seen="$seen$rel " + printf '%s\n' "$rel" + case "$rel" in *.sh) next="$next $ROOT/$rel" ;; esac + done < <(grep -oE '\$ROOT/[A-Za-z0-9_./-]+' "$f" | sed 's|^\$ROOT/||' | sort -u) + done + queue="$next" + done +} + +# in_scope -> 0 iff path is a scoped path or under a scoped directory +in_scope() { + local p=$1 s + while IFS= read -r s; do + [ -n "$s" ] || continue + case "$p" in "$s"|"$s"/*) return 0 ;; esac + done < "$2" + return 1 +} + +rmtree() { case "${1:-}" in ''|/) return 0 ;; *) [ -d "$1" ] && rm -rf -- "$1" ;; esac; return 0; } + +# decide -> prints RUN or SKIP ; never fails toward SKIP +decide() { + local t scope touched p n_scope n_touched + t=$(mktemp -d "${TMPDIR:-/tmp}/perf-gate-scope.XXXXXX") || { printf 'RUN cannot make a temp dir\n'; return 0; } + scope_of "$SUBJECT" > "$t/scope" + n_scope=$(grep -c . "$t/scope") + if ! git -C "$ROOT" rev-parse -q --verify "$BASE^{tree}" >/dev/null 2>&1; then + printf 'RUN the comparand %s cannot be read -- never skip on unknown\n' "$BASE" + rmtree "$t"; return 0 + fi + if ! git -C "$ROOT" diff --no-renames --name-only "$BASE" HEAD > "$t/touched" 2>/dev/null; then + printf 'RUN git diff %s HEAD failed -- never skip on unknown\n' "$BASE" + rmtree "$t"; return 0 + fi + n_touched=$(grep -c . "$t/touched") + while IFS= read -r p; do + [ -n "$p" ] || continue + if in_scope "$p" "$t/scope"; then + printf 'RUN the diff vs %s touches %s, which the table reads\n' "$BASE" "$p" + rmtree "$t"; return 0 + fi + done < "$t/touched" + printf 'SKIP the diff vs %s touches %s path(s), none of the %s the table reads; guards-nightly.yml runs it whole\n' "$BASE" "$n_touched" "$n_scope" + rmtree "$t"; return 0 +} + +case "${1:-}" in -h|--help) sed -n '2,32p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; esac + +if [ "${1:-}" = "--self-test" ]; then + echo "=== perf_gate --selftest scoping: case table ===" + d=$(mktemp -d "${TMPDIR:-/tmp}/perf-gate-scope-st.XXXXXX") || exit 2 + trap 'rmtree "${d:-}"' EXIT + bad=0; n=0 + row() { n=$((n + 1)); if [ "$2" = 0 ]; then printf 'ok row %-2s %s\n' "$n" "$1"; else printf 'FAIL row %-2s %s -- got: %s\n' "$n" "$1" "$OUT" >&2; bad=1; fi; } + r="$d/repo"; mkdir -p "$r/scripts/lib" "$r/tests/fixtures/perf-gate" "$r/src" + git init -q -b main "$r" + printf '#!/usr/bin/env bash\nROOT=x\npython3 "$ROOT/scripts/lib/sig.py"\ncat "$ROOT/tests/fixtures/perf-gate"\nbash "$ROOT/scripts/helper.sh"\necho selftest-ran\n' > "$r/scripts/perf_gate.sh" + printf '#!/usr/bin/env bash\ncat "$ROOT/scripts/deep.txt"\n' > "$r/scripts/helper.sh" + printf 'x\n' > "$r/scripts/lib/sig.py"; printf 'x\n' > "$r/tests/fixtures/perf-gate/a.json"; printf 'x\n' > "$r/src/lib.rs"; mkdir -p "$r/tests/fixtures/perf-gate-old"; printf 'x\n' > "$r/tests/fixtures/perf-gate-old/x.json"; printf 'x\n' > "$r/scripts/deep.txt" + ( cd "$r" && export GIT_AUTHOR_NAME=t GIT_AUTHOR_EMAIL=t@t GIT_COMMITTER_NAME=t GIT_COMMITTER_EMAIL=t@t GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=core.hooksPath GIT_CONFIG_VALUE_0=/dev/null + git add -A && git commit -q -m base && git branch base ) || exit 2 + edit() { # edit ... -> a fresh commit on main touching exactly those paths, vs branch base + ( cd "$r" && git checkout -q -B main base && for f in "$@"; do printf 'changed\n' >> "$f"; done \ + && GIT_AUTHOR_NAME=t GIT_AUTHOR_EMAIL=t@t GIT_COMMITTER_NAME=t GIT_COMMITTER_EMAIL=t@t git -c core.hooksPath=/dev/null commit -q -am e ) || exit 2 + } + judge() { OUT=$(cd "$r" && ROOT="$r" PERF_GATE_SUBJECT="$r/scripts/perf_gate.sh" PERF_GATE_BASE="${1:-base}" bash -c "$(declare -f scope_of in_scope rmtree decide); SUBJECT=\$PERF_GATE_SUBJECT; BASE=\$PERF_GATE_BASE; decide"); } + edit src/lib.rs; judge; [[ $OUT == SKIP* ]]; row "a diff touching none of the table's inputs -> SKIP, and says where it runs whole" $? + [[ $OUT == *"none of the 5 the table reads"* ]]; row " ...the scope is DERIVED from the \$ROOT/ refs: 5 paths (itself, a lib, a fixture dir, a helper script and what IT reads)" $? + edit scripts/deep.txt; judge; [[ $OUT == RUN* ]]; row "a file read only by a HELPER script the table calls -> RUN (the scope is transitive over .sh)" $? + edit scripts/perf_gate.sh; judge; [[ $OUT == "RUN "*"scripts/perf_gate.sh"* ]]; row "a diff touching perf_gate.sh itself -> RUN" $? + edit scripts/lib/sig.py; judge; [[ $OUT == RUN* ]]; row "a diff touching a file the table reads (\$ROOT/scripts/lib/sig.py) -> RUN" $? + edit tests/fixtures/perf-gate/a.json; judge; [[ $OUT == RUN* ]]; row "a diff under a scoped DIRECTORY (tests/fixtures/perf-gate/) -> RUN" $? + edit tests/fixtures/perf-gate-old/x.json; judge; [[ $OUT == SKIP* ]]; row "a sibling dir whose name only STARTS like a scoped dir (perf-gate-old/) is not in scope" $? + edit src/lib.rs; judge no-such-ref; [[ $OUT == RUN* ]]; row "the comparand cannot be read -> RUN, never skip on unknown" $? + ( cd "$r" && git checkout -q -B main base && git mv src/lib.rs tests/fixtures/perf-gate/moved.json && GIT_AUTHOR_NAME=t GIT_AUTHOR_EMAIL=t@t GIT_COMMITTER_NAME=t GIT_COMMITTER_EMAIL=t@t git -c core.hooksPath=/dev/null commit -q -m mv ) || exit 2 + judge; [[ $OUT == RUN* ]]; row "a rename INTO a scoped directory -> RUN" $? + ( cd "$r" && git checkout -q -B main base && git mv scripts/lib/sig.py src/sig.py && GIT_AUTHOR_NAME=t GIT_AUTHOR_EMAIL=t@t GIT_COMMITTER_NAME=t GIT_COMMITTER_EMAIL=t@t git -c core.hooksPath=/dev/null commit -q -m mv2 ) || exit 2 + judge; [[ $OUT == RUN* ]]; row "a rename OUT of scope still lists the removed side -> RUN (--no-renames)" $? + # the real subject's derived scope must include the dirs and files it is known to read + OUT=$(scope_of "$ROOT/scripts/perf_gate.sh" | tr '\n' ' ') + [[ $OUT == *"scripts/perf_gate.sh"* && $OUT == *"scripts/lib/receipt_sig.py"* && $OUT == *"tests/fixtures/perf-gate"* ]] + row "the REAL perf_gate.sh's derived scope names itself, receipt_sig.py and its fixture dir" $? + [ "$bad" = 0 ] && { printf 'SELF-TEST PASSED: %s rows\n' "$n"; exit 0; } + printf 'SELF-TEST FAILED\n' >&2; exit 1 +fi + +echo "=== perf_gate.sh --selftest, path-scoped (check_perf_gate_selftest_scoped.sh) ===" +[ -f "$SUBJECT" ] || { printf 'ENV %s is missing -- cannot judge, not a pass\n' "$SUBJECT" >&2; exit 2; } +verdict=$(decide) +case "$verdict" in + SKIP*) printf 'SUMMARY perf_gate --selftest skipped: %s\n' "${verdict#SKIP }"; echo PASS; exit 0 ;; + *) printf '%s\n' "$verdict"; bash "$SUBJECT" --selftest; exit $? ;; +esac diff --git a/scripts/guards_nightly_manifest.txt b/scripts/guards_nightly_manifest.txt index 0abb91a4f4..8eaddc126e 100644 --- a/scripts/guards_nightly_manifest.txt +++ b/scripts/guards_nightly_manifest.txt @@ -21,13 +21,16 @@ # (`scripts/dogfood.sh --phase pre-publish`), instead of being blocked at the # PR. That is decision D-1 of the 06x release schedule. # -# FORMAT: ``. The step +# FORMAT: ``. (The perf-gate +# line, added by #3676, is seconds on main run 35563537942: it stays at PR time +# PATH-SCOPED via scripts/check_perf_gate_selftest_scoped.sh and runs whole here.) The step # name is the join key: it must appear VERBATIM as a `- name:` in # .github/workflows/guards-nightly.yml (that workflow's first step asserts it, # and fails closed if a line here names a step nothing runs), and it is also # what keeps scripts/check_guards_are_wired.sh finding these guards wired -- # that guard greps every file under .github/workflows/, not ci.yml alone. 586 Test-tier decision case table (BSE-17): quick/full/reuse, drift is ENV +405 Perf gate can still discriminate (APR-PERF-GATE-001, case table) 256 model-tests falsification suites (was gated by nothing, aprender#2522) -- SATD ceiling measured on the comparand (BSE-03) 206 Book rust examples compile 200 aprender-profile lib tests (was gated by nothing) From f7b25a6c6e64432e2ea62d039eee455cbb1bec6e Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 12:05:05 +0200 Subject: [PATCH 2/4] =?UTF-8?q?ci(diet):=20quorum=20round=201=20findings?= =?UTF-8?q?=20=E2=80=94=20book=20nightly=20+=20path-filter=20RULE=203,=20p?= =?UTF-8?q?erf-gate=20smoke=20row,=20gate-rule=20self-test=20independent?= =?UTF-8?q?=20of=20ci.yml=20(#3676)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Refs #3676 Round 1 (e2d12bdd1) was FAIL/FAIL/PASS. The two FAIL lanes held the diff to the ticket's text and were right on three counts: 1. Item 2 says the book jobs are path-filtered "and nightly always". They were filtered, but NO path-filtered workflow had a schedule. book.yml, book-contracts.yml and install-script.yml now carry a nightly cron (22:30 / 22:45 / 23:00 UTC, ~5h early for the #3292 dispatch delay; book.yml's pages deploy stays push-only by its own `if:`). 2. Item 5 says check_workflow_path_filters.sh is updated. It gains RULE 3: a path-FILTERED workflow must also trigger on `schedule:` with a cron -- the filter is a claim that nothing outside it can break the gate, and the nightly is what re-checks it (book.yml sat dark three months, per the guard's own header). scripts/lib/workflow_path_filters.py emits a SCHEDULE row. Self-test 4 -> 6: no schedule -> RED; `schedule: []` -> RED. On the tree before the crons: 3 workflows RED; after: PASS. 3. Item 3 says "The PR path keeps a smoke row" for the perf table. On a skip, check_perf_gate_selftest_scoped.sh now requires `perf_gate.sh --list-selftests` to exit 0 and enumerate >= 1 case (109 today, ~60 ms), and says so on the SUMMARY line. Three end-to-end rows (skip + smoke; in scope -> full table; a table that cannot enumerate -> RED). 14 rows; five mutants RED (not transitive, renames, unknown -> skip, prefix match, smoke removed). Lane 1's other finding also held: check_ci_gate_mutants_rule.sh checked for ci.yml before --self-test, so its self-test could not run without the workflow. The check now sits on the bare-run path only. Gates: guard_tree --no-cargo all PASS except check_fleet_pv_shapes_gate.sh (the lambda-box fleet-pv condition, fixed in #3669); guard_tree_test 23/0; bashrs 0 errors on every changed script, bashrs-gate PASS; actionlint: no finding beyond main's shellcheck infos. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/book-contracts.yml | 5 +++ .github/workflows/book.yml | 5 +++ .github/workflows/install-script.yml | 5 +++ scripts/check_ci_gate_mutants_rule.sh | 3 +- scripts/check_perf_gate_selftest_scoped.sh | 40 +++++++++++++++---- scripts/check_workflow_path_filters.sh | 46 +++++++++++++++++++--- scripts/lib/workflow_path_filters.py | 4 ++ 7 files changed, 94 insertions(+), 14 deletions(-) diff --git a/.github/workflows/book-contracts.yml b/.github/workflows/book-contracts.yml index 5a52062a7e..e1ca6e9c65 100644 --- a/.github/workflows/book-contracts.yml +++ b/.github/workflows/book-contracts.yml @@ -33,6 +33,11 @@ on: - "crates/aprender-core/src/**" - "scripts/book-gate.sh" - ".github/workflows/book-contracts.yml" + # #3676 rule 3 (scripts/check_workflow_path_filters.sh): a path filter is a + # claim that nothing outside these paths can break this gate; the nightly run + # re-checks the claim every day. Set ~5h early for the dispatch delay (#3292). + schedule: + - cron: '45 22 * * *' permissions: contents: read diff --git a/.github/workflows/book.yml b/.github/workflows/book.yml index 9f17e1c7e5..ba40bbd70b 100644 --- a/.github/workflows/book.yml +++ b/.github/workflows/book.yml @@ -49,6 +49,11 @@ on: - "crates/aprender-core/src/lib.rs" - "crates/apr-cli/src/extended_commands.rs" - "crates/apr-cli/src/tool_commands.rs" + # #3676 rule 3 (scripts/check_workflow_path_filters.sh): a path filter is a + # claim that nothing outside these paths can break this gate; the nightly run + # re-checks the claim every day. Set ~5h early for the dispatch delay (#3292). + schedule: + - cron: '30 22 * * *' permissions: contents: read diff --git a/.github/workflows/install-script.yml b/.github/workflows/install-script.yml index 933806814a..df4b38dae6 100644 --- a/.github/workflows/install-script.yml +++ b/.github/workflows/install-script.yml @@ -31,6 +31,11 @@ on: - ".github/workflows/install-script.yml" - ".github/workflows/binary-release.yml" - ".github/workflows/nightly.yml" + # #3676 rule 3 (scripts/check_workflow_path_filters.sh): a path filter is a + # claim that nothing outside these paths can break this gate; the nightly run + # re-checks the claim every day. Set ~5h early for the dispatch delay (#3292). + schedule: + - cron: '0 23 * * *' permissions: contents: read diff --git a/scripts/check_ci_gate_mutants_rule.sh b/scripts/check_ci_gate_mutants_rule.sh index e4b0a06a71..544a64b362 100644 --- a/scripts/check_ci_gate_mutants_rule.sh +++ b/scripts/check_ci_gate_mutants_rule.sh @@ -54,7 +54,6 @@ ROWS } case "${1:-}" in -h|--help) sed -n '2,18p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; esac -[ -f "$WF" ] || { printf 'ENV %s is missing\n' "$WF" >&2; exit 2; } if [ "${1:-}" = "--self-test" ]; then echo "=== gate mutants rule: the old rule must turn the table RED ===" @@ -73,6 +72,8 @@ if [ "${1:-}" = "--self-test" ]; then fi echo "=== gate accepts a skipped mutants job only when the skip is its filter (check_ci_gate_mutants_rule.sh) ===" +# the workflow is needed only here: --self-test builds its own fixtures (quorum lane 1 on #3688) +[ -f "$WF" ] || { printf 'ENV %s is missing\n' "$WF" >&2; exit 2; } table "$WF"; rc=$? [ "$rc" = 0 ] && echo PASS || echo "FAIL (rc=$rc)" >&2 exit "$rc" diff --git a/scripts/check_perf_gate_selftest_scoped.sh b/scripts/check_perf_gate_selftest_scoped.sh index 6482eb99b8..d4b79b3185 100644 --- a/scripts/check_perf_gate_selftest_scoped.sh +++ b/scripts/check_perf_gate_selftest_scoped.sh @@ -27,10 +27,16 @@ # check_perf_gate_selftest_scoped.sh decide, and run it if in scope # check_perf_gate_selftest_scoped.sh --self-test the decision's case table # -# Test seams: PERF_GATE_SUBJECT (the script whose refs define the scope and which -# is run), PERF_GATE_BASE (the comparand ref, default origin/main). +# THE SMOKE ROW (the ticket's words: "The PR path keeps a smoke row"): on a skip, +# `perf_gate.sh --list-selftests` must still exit 0 and enumerate at least one +# case (109 today, ~60 ms). It proves the table still parses and registers on +# this runner; the full table runs when its inputs change, and nightly. +# +# Test seams: PERF_GATE_ROOT (the tree), PERF_GATE_SUBJECT (the script whose refs +# define the scope and which is run), PERF_GATE_BASE (the comparand, default origin/main). set -uo pipefail -ROOT="$(cd "$(dirname "$0")/.." && pwd)" || exit 2 +SELF="$(cd "$(dirname "$0")" && pwd)/$(basename "$0")" || exit 2 +ROOT="${PERF_GATE_ROOT:-$(cd "$(dirname "$0")/.." && pwd)}" || exit 2 SUBJECT="${PERF_GATE_SUBJECT:-$ROOT/scripts/perf_gate.sh}" BASE="${PERF_GATE_BASE:-origin/main}" @@ -108,13 +114,13 @@ if [ "${1:-}" = "--self-test" ]; then row() { n=$((n + 1)); if [ "$2" = 0 ]; then printf 'ok row %-2s %s\n' "$n" "$1"; else printf 'FAIL row %-2s %s -- got: %s\n' "$n" "$1" "$OUT" >&2; bad=1; fi; } r="$d/repo"; mkdir -p "$r/scripts/lib" "$r/tests/fixtures/perf-gate" "$r/src" git init -q -b main "$r" - printf '#!/usr/bin/env bash\nROOT=x\npython3 "$ROOT/scripts/lib/sig.py"\ncat "$ROOT/tests/fixtures/perf-gate"\nbash "$ROOT/scripts/helper.sh"\necho selftest-ran\n' > "$r/scripts/perf_gate.sh" + printf '#!/usr/bin/env bash\nROOT=x\n: python3 "$ROOT/scripts/lib/sig.py"\n: cat "$ROOT/tests/fixtures/perf-gate"\n: bash "$ROOT/scripts/helper.sh"\ncase "$1" in --list-selftests) printf "case_a\\ncase_b\\n" ;; --selftest) echo selftest-ran ;; esac\n' > "$r/scripts/perf_gate.sh" printf '#!/usr/bin/env bash\ncat "$ROOT/scripts/deep.txt"\n' > "$r/scripts/helper.sh" printf 'x\n' > "$r/scripts/lib/sig.py"; printf 'x\n' > "$r/tests/fixtures/perf-gate/a.json"; printf 'x\n' > "$r/src/lib.rs"; mkdir -p "$r/tests/fixtures/perf-gate-old"; printf 'x\n' > "$r/tests/fixtures/perf-gate-old/x.json"; printf 'x\n' > "$r/scripts/deep.txt" ( cd "$r" && export GIT_AUTHOR_NAME=t GIT_AUTHOR_EMAIL=t@t GIT_COMMITTER_NAME=t GIT_COMMITTER_EMAIL=t@t GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=core.hooksPath GIT_CONFIG_VALUE_0=/dev/null git add -A && git commit -q -m base && git branch base ) || exit 2 edit() { # edit ... -> a fresh commit on main touching exactly those paths, vs branch base - ( cd "$r" && git checkout -q -B main base && for f in "$@"; do printf 'changed\n' >> "$f"; done \ + ( cd "$r" && git checkout -q -B main base && for f in "$@"; do printf '# changed\n' >> "$f"; done \ && GIT_AUTHOR_NAME=t GIT_AUTHOR_EMAIL=t@t GIT_COMMITTER_NAME=t GIT_COMMITTER_EMAIL=t@t git -c core.hooksPath=/dev/null commit -q -am e ) || exit 2 } judge() { OUT=$(cd "$r" && ROOT="$r" PERF_GATE_SUBJECT="$r/scripts/perf_gate.sh" PERF_GATE_BASE="${1:-base}" bash -c "$(declare -f scope_of in_scope rmtree decide); SUBJECT=\$PERF_GATE_SUBJECT; BASE=\$PERF_GATE_BASE; decide"); } @@ -130,6 +136,18 @@ if [ "${1:-}" = "--self-test" ]; then judge; [[ $OUT == RUN* ]]; row "a rename INTO a scoped directory -> RUN" $? ( cd "$r" && git checkout -q -B main base && git mv scripts/lib/sig.py src/sig.py && GIT_AUTHOR_NAME=t GIT_AUTHOR_EMAIL=t@t GIT_COMMITTER_NAME=t GIT_COMMITTER_EMAIL=t@t git -c core.hooksPath=/dev/null commit -q -m mv2 ) || exit 2 judge; [[ $OUT == RUN* ]]; row "a rename OUT of scope still lists the removed side -> RUN (--no-renames)" $? + # END TO END: the whole guard, not just decide() -- the smoke row on a skip, the table on a RUN + whole() { OUT=$(cd "$r" && PERF_GATE_ROOT="$r" PERF_GATE_SUBJECT="$r/scripts/perf_gate.sh" PERF_GATE_BASE=base bash "$SELF" 2>&1); RC=$?; } + edit src/lib.rs; whole + [ "$RC" = 0 ] && [[ $OUT == *"SUMMARY perf_gate --selftest skipped:"*"smoke: 2 case(s) enumerated"* ]]; row "end to end, out of scope: rc 0 and a SUMMARY line with the SMOKE row (2 cases enumerated)" $? + edit scripts/perf_gate.sh; whole + [ "$RC" = 0 ] && [[ $OUT == *"selftest-ran"* ]]; row "end to end, in scope: the full table runs" $? + ( cd "$r" && export GIT_AUTHOR_NAME=t GIT_AUTHOR_EMAIL=t@t GIT_COMMITTER_NAME=t GIT_COMMITTER_EMAIL=t@t \ + && git checkout -q -B brk base && printf '#!/usr/bin/env bash\nexit 3\n' > scripts/perf_gate.sh \ + && git -c core.hooksPath=/dev/null commit -q -am brk1 && printf 'y\n' >> src/lib.rs \ + && git -c core.hooksPath=/dev/null commit -q -am brk2 ) || exit 2 # HEAD~1 already broken; HEAD touches only src/ + OUT=$(cd "$r" && PERF_GATE_ROOT="$r" PERF_GATE_SUBJECT="$r/scripts/perf_gate.sh" PERF_GATE_BASE=HEAD~1 bash "$SELF" 2>&1); RC=$? + [ "$RC" = 1 ] && [[ $OUT == *"FAIL smoke"* ]]; row "end to end, out of scope but the table cannot even enumerate -> the smoke row is RED" $? # the real subject's derived scope must include the dirs and files it is known to read OUT=$(scope_of "$ROOT/scripts/perf_gate.sh" | tr '\n' ' ') [[ $OUT == *"scripts/perf_gate.sh"* && $OUT == *"scripts/lib/receipt_sig.py"* && $OUT == *"tests/fixtures/perf-gate"* ]] @@ -142,6 +160,14 @@ echo "=== perf_gate.sh --selftest, path-scoped (check_perf_gate_selftest_scoped. [ -f "$SUBJECT" ] || { printf 'ENV %s is missing -- cannot judge, not a pass\n' "$SUBJECT" >&2; exit 2; } verdict=$(decide) case "$verdict" in - SKIP*) printf 'SUMMARY perf_gate --selftest skipped: %s\n' "${verdict#SKIP }"; echo PASS; exit 0 ;; - *) printf '%s\n' "$verdict"; bash "$SUBJECT" --selftest; exit $? ;; + SKIP*) + cases=$(bash "$SUBJECT" --list-selftests 2>/dev/null); lrc=$? + n_cases=$(grep -c . <<<"$cases") + if [ "$lrc" -ne 0 ] || [ "$n_cases" -lt 1 ]; then + printf 'FAIL smoke: %s --list-selftests exited %s with %s case(s) -- the table does not even enumerate\n' "${SUBJECT#"$ROOT/"}" "$lrc" "$n_cases" >&2 + exit 1 + fi + printf 'SUMMARY perf_gate --selftest skipped: %s; smoke: %s case(s) enumerated\n' "${verdict#SKIP }" "$n_cases" + echo PASS; exit 0 ;; + *) printf '%s\n' "$verdict"; bash "$SUBJECT" --selftest; exit $? ;; esac diff --git a/scripts/check_workflow_path_filters.sh b/scripts/check_workflow_path_filters.sh index a7abc07c24..7b47f2fd6a 100644 --- a/scripts/check_workflow_path_filters.sh +++ b/scripts/check_workflow_path_filters.sh @@ -69,9 +69,24 @@ check_workflow() { rm -f /tmp/wfpf_err.$$ [ -z "$out" ] && return 0 - local push pr + local push pr sched push="$(printf '%s\n' "$out" | awk -F'\t' '$1=="PUSH"{print $2}' | sort -u)" pr="$(printf '%s\n' "$out" | awk -F'\t' '$1=="PR"{print $2}' | sort -u)" + sched="$(printf '%s\n' "$out" | awk -F'\t' '$1=="SCHEDULE"{print $2}')" + + # Rule 3 (#3676): a path-FILTERED workflow must also run on `schedule:`. The + # filter is a claim that nothing outside its paths can break the gate; the + # nightly run is what turns a wrong claim into a red within a day instead of + # the three months book.yml sat dark. Applies when either event is filtered. + local filtered=0 + if [ -n "$push" ] && [ "$push" != '' ]; then filtered=1; fi + if [ -n "$pr" ] && [ "$pr" != '' ]; then filtered=1; fi + if [ "$filtered" = 1 ] && [ "$sched" != yes ]; then + printf '\nFAIL %s: path-filtered, but no `schedule:` trigger.\n' "$name" + printf ' A change outside the filter that breaks this gate is never seen;\n' + printf ' add a nightly cron so the claim is re-checked every day.\n' + return 1 + fi # An unfiltered event cannot go dark; only compare when BOTH are filtered. case "$push" in *''*) return 0 ;; esac @@ -129,13 +144,32 @@ on: pull_request: paths: ["book/**"] YML - # 2 - symmetric. MUST pass. + # 2 - symmetric, with a nightly. MUST pass. cat > "$TD/wf2.yml" <<'YML' on: push: paths: ["book/**"] pull_request: paths: ["book/**"] + schedule: + - cron: '30 22 * * *' +YML + # 5 - symmetric but NO schedule (rule 3, #3676). MUST fail. + cat > "$TD/wf5.yml" <<'YML' +on: + push: + paths: ["book/**"] + pull_request: + paths: ["book/**"] +YML + # 6 - a schedule key with no cron entry is not a nightly (rule 3). MUST fail. + cat > "$TD/wf6.yml" <<'YML' +on: + push: + paths: ["book/**"] + pull_request: + paths: ["book/**"] + schedule: [] YML # 3 - no path filter at all: cannot go dark. MUST pass. cat > "$TD/wf3.yml" <<'YML' @@ -155,7 +189,7 @@ on: YML fails=0 - for c in 1 4; do + for c in 1 4 5 6; do if check_workflow "$TD/wf${c}.yml" >/dev/null 2>&1; then printf 'FAIL row %s NOT flagged - the guard is blind to a real defect shape\n' "$c" fails=$((fails + 1)) @@ -173,9 +207,9 @@ YML done if [ "$fails" -ne 0 ]; then - printf '\nSELF-TEST FAILED (%s/4 wrong)\n' "$fails"; exit 1 + printf '\nSELF-TEST FAILED (%s/6 wrong)\n' "$fails"; exit 1 fi - printf '\nSELF-TEST PASSED (4/4)\n' + printf '\nSELF-TEST PASSED (6/6)\n' exit 0 fi @@ -212,5 +246,5 @@ if [ "$violations" -ne 0 ]; then exit 1 fi -printf 'PASS: every path-filtered workflow gates PRs and main identically.\n' +printf 'PASS: every path-filtered workflow gates PRs and main identically, and runs nightly.\n' exit 0 diff --git a/scripts/lib/workflow_path_filters.py b/scripts/lib/workflow_path_filters.py index 40f561bb1b..88cb240933 100644 --- a/scripts/lib/workflow_path_filters.py +++ b/scripts/lib/workflow_path_filters.py @@ -20,6 +20,10 @@ on = d.get(True) if d.get(True) is not None else d.get('on') if not isinstance(on, dict): sys.exit(0) +# RULE 3 (#3676): a path filter is a claim; the nightly run is its backstop. +sched = on.get('schedule') +if isinstance(sched, list) and any(isinstance(e, dict) and e.get('cron') for e in sched): + print("SCHEDULE\tyes") for ev, tag in (('push', 'PUSH'), ('pull_request', 'PR')): spec = on.get(ev) if not isinstance(spec, dict): From 5d7dbda0affae6fdbab76b8909bb861ecd72305a Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 17:31:42 +0200 Subject: [PATCH 3/4] wip(PMAT-3676): coverage_on: tag + v* tag trigger + producers R3/R5 + closes-issue --self-test (provisional pin 5997b41) --- .github/workflows/ci.yml | 36 +++++---- .github/workflows/coverage-nightly.yml | 9 +-- scripts/check_coverage_has_producers.sh | 102 ++++++++++++++++-------- 3 files changed, 93 insertions(+), 54 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 500fcbaf92..336274b959 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -13,6 +13,10 @@ name: CI on: push: branches: [main, master] + # #3676: sovereign-ci's `coverage_on: tag` (below) runs coverage only on a push of a + # v* tag, and its gate REQUIRES it to succeed there. A push filtered to branches never + # fires for a tag, so this is the reusable's second required edit. + tags: ['v*'] pull_request: branches: [main, master] # Merge queue: GitHub dispatches a `merge_group` event for each batch it forms. @@ -63,7 +67,7 @@ jobs: # 23916c00d adds an OPTIONAL `runs_on` input whose default is the label set # this repo already gets, and 70e51ec04 swaps the digest on every job. No # input this repo passes changes meaning. - uses: paiml/.github/.github/workflows/sovereign-ci.yml@70e51ec045d95983e9a3d02342d13283bf5f9dc1 + uses: paiml/.github/.github/workflows/sovereign-ci.yml@5997b411a371d0902f2ade4cba31e04ac6f87580 with: repo: ${{ github.event.repository.name }} # Phase 3 pilot (heavy workload) — build-performance.md §7 Phase 3. @@ -116,13 +120,16 @@ jobs: # #3676 (operator 2026-09-21: "YES, coverage on tags release only"): so this # job gated nothing AND measured nothing (0 tests on the facade root), at the # cost of an instrumented rebuild on every PR, merge group and push (237 s - # on main run 35563537942). Skipped here. Every consumer of aprender's - # coverage number has its own producer, asserted by - # scripts/check_coverage_has_producers.sh: the pre-publish dogfood runs - # `make coverage-check` (-> `make coverage`: workspace llvm-cov, COV_FLOOR), - # and coverage-nightly.yml runs nightly AND on every `v*` tag. The one - # output lost is this job's advisory Codecov upload. - skip_coverage: true + # on main run 35563537942). `coverage_on: tag` runs it only on a push of a v* + # tag (the trigger above) and on workflow_dispatch; everywhere else the job is + # skipped and sovereign-ci's gate prints "coverage: NOT MEASURED", never a + # green. On a v* tag the gate REQUIRES it. The first spelling, + # `skip_coverage: true`, was read by that gate as a mandatory job that did not + # succeed, so `ci / gate` went red on every PR (#3707). Every consumer of the + # number keeps a producer, asserted by scripts/check_coverage_has_producers.sh + # (R1-R5): the pre-publish dogfood's `make coverage-check`, coverage-nightly's + # schedule, and this job on each release tag. + coverage_on: tag secrets: inherit # APR-MONO: Workspace-wide test (all 75 crates) @@ -1361,12 +1368,12 @@ jobs: - name: "Receipt-wiring guard: every rule it states, mutated (15/15 kill)" run: bash scripts/mutate_pr_review_wiring_guard.sh # APR-RELEASE-001 §6 R-2, WIRED (#3200 shipped the guard; this is its - # caller). check_pr_closes_issue.sh is a check_*.sh, so guard_tree.sh - # auto-discovers it and runs it BARE -- and bare runs only its own - # self-test, "self-test OK: 9 case(s)", exit 0. Nine green rows about a - # regex, on every PR, judging NO PR body. The 0.67.0 reconcile found 48 - # merged PRs since the previous tag of which only 9 closed anything; that - # is what this guard exists to stop, and until now it had never read one. + # caller). This step is its ONLY runner: guard_tree.sh skips it as + # wired-with-args (it needs a body), so its case table ran in NO job until + # this step called `--self-test` beside `--body` (found by aprender-f0, + # 2026-09-21; the comment here used to claim guard_tree ran it bare). The + # 0.67.0 reconcile found 48 merged PRs since the previous tag of which only + # 9 closed anything; that is what this guard exists to stop. # # THE BODY ARRIVES THROUGH `env:`, NEVER INTERPOLATED INTO THE SCRIPT. # `${{ }}` is substituted textually BEFORE bash sees the line, so a body @@ -1390,6 +1397,7 @@ jobs: PR_BODY: ${{ github.event.pull_request.body }} run: | set -euo pipefail + bash scripts/check_pr_closes_issue.sh --self-test printf '%s' "$PR_BODY" > "$RUNNER_TEMP/pr-body.txt" bash scripts/check_pr_closes_issue.sh --body "$RUNNER_TEMP/pr-body.txt" # APR-RELEASE-001 §11.1, WIRED. The section shipped as prose in #3268 and diff --git a/.github/workflows/coverage-nightly.yml b/.github/workflows/coverage-nightly.yml index 52ff9ea760..c1047285f0 100644 --- a/.github/workflows/coverage-nightly.yml +++ b/.github/workflows/coverage-nightly.yml @@ -33,12 +33,9 @@ on: # `gh workflow run` (#3292); until then, re-measure before trusting. - cron: '0 22 * * *' # 22:00 UTC -> lands ~03:00 UTC workflow_dispatch: {} - # #3676 (operator 2026-09-21: "YES, coverage on tags release only"): coverage - # left the PR / merge-group / push path, so the RELEASE TAG runs it here, with - # the same COV_FLOOR `make coverage` enforces nightly. A floor breach reds the - # tag's run; scripts/check_coverage_has_producers.sh keeps this trigger present. - push: - tags: ['v*'] + # #3676: no v* tag trigger here. The release tag's coverage is owned by ci.yml + # (`coverage_on: tag`, which sovereign-ci's gate requires on a tag push), so a tag + # is measured once. scripts/check_coverage_has_producers.sh R3 keeps it that way. permissions: contents: read diff --git a/scripts/check_coverage_has_producers.sh b/scripts/check_coverage_has_producers.sh index 22d911eb69..97758facb3 100644 --- a/scripts/check_coverage_has_producers.sh +++ b/scripts/check_coverage_has_producers.sh @@ -3,18 +3,24 @@ # still has a producer (#3676). # # Operator 2026-09-21: "YES, coverage on tags release only." `ci / coverage` left -# the PR / merge-group / push path (ci.yml `skip_coverage: true`; it had gated -# nothing and measured 0 tests on the facade root). A deletion like that can -# starve a gate that READ the number from CI -- rmedia's release gate did -# exactly that ("could not look"). So the chain is asserted, not remembered: +# the PR / merge-group / push path: it had gated nothing and measured 0 tests on +# the facade root. A deletion like that can starve a gate that READ the number +# from CI -- rmedia's release gate did exactly that ("could not look"). So the +# chain is asserted, not remembered: # # R1 `make -n coverage-check` reaches llvm-cov -- the PRODUCER the # pre-publish dogfood uses measures coverage itself, reads nothing from CI. # R2 scripts/dogfood.sh still runs `make ... coverage-check` -- the release # CONSUMER exists (without it the chain would be vacuously intact). -# R3 .github/workflows/coverage-nightly.yml triggers on `schedule:` AND on a -# `push:` of `v*` tags -- the tag gets its own COV_FLOOR run. +# R3 .github/workflows/coverage-nightly.yml triggers on `schedule:` and NOT on +# `v*` tags -- the nightly trend stays, and a tag is measured ONCE, by ci.yml. # R4 the Makefile defines a numeric COV_FLOOR -- the floor `make coverage` enforces. +# R5 ci.yml hands sovereign-ci `coverage_on: tag` (no `skip_coverage: true`) +# AND its own `on:` has `push: tags: ['v*']` -- the reusable's two required +# edits. sovereign-ci's gate then prints "coverage: NOT MEASURED" on a pull +# request and REQUIRES coverage to succeed on a v* tag. `skip_coverage: true` +# was the first spelling (#3688): that gate counts a skipped coverage job as +# a mandatory failure, so it turned `ci / gate` red on every PR. # Any link missing is FAIL; a file that cannot be read is ENV rc=2. # # check_coverage_has_producers.sh judge the tree @@ -28,11 +34,14 @@ ROOT="${COVCHAIN_ROOT:-$(cd "$(dirname "$0")/.." && pwd)}" on_block() { awk '/^on:/{f=1; next} f && /^[^[:space:]#]/{exit} f' "$1" | sed 's/#.*$//' } +# live -> the file with comments stripped (a setting in a comment is not a setting) +live() { sed 's/#.*$//' "$1"; } +V_TAGS_RE="^[[:space:]]+tags:[[:space:]]*\[[^]]*['\"]?v\*['\"]?" judge() { # judge -> 0 all links hold, 1 a link broke, 2 ENV - local r=$1 bad=0 dry blk - local mk="$r/Makefile" df="$r/scripts/dogfood.sh" wf="$r/.github/workflows/coverage-nightly.yml" - for f in "$mk" "$df" "$wf"; do [ -r "$f" ] || { printf 'ENV %s is not readable -- cannot judge, not a pass\n' "$f"; return 2; }; done + local r=$1 bad=0 dry blk ci + local mk="$r/Makefile" df="$r/scripts/dogfood.sh" wf="$r/.github/workflows/coverage-nightly.yml" cy="$r/.github/workflows/ci.yml" + for f in "$mk" "$df" "$wf" "$cy"; do [ -r "$f" ] || { printf 'ENV %s is not readable -- cannot judge, not a pass\n' "$f"; return 2; }; done dry=$(make -n -C "$r" coverage-check 2>/dev/null) || dry="" if [[ $dry == *"llvm-cov"* ]]; then printf 'ok R1 make -n coverage-check reaches llvm-cov (the release producer measures, it reads nothing from CI)\n' @@ -43,17 +52,29 @@ judge() { # judge -> 0 all links hold, 1 a link broke, 2 ENV else printf 'FAIL R2 scripts/dogfood.sh no longer runs coverage-check -- nothing checks coverage at the release\n'; bad=1; fi blk=$(on_block "$wf") - if grep -qE '^[[:space:]]+schedule:' <<<"$blk" && grep -qE "^[[:space:]]+tags:[[:space:]]*\[[^]]*['\"]?v\*['\"]?" <<<"$blk"; then - printf 'ok R3 coverage-nightly.yml triggers on schedule AND on v* tags\n' - else printf 'FAIL R3 coverage-nightly.yml must trigger on schedule AND on push tags [v*] (on: block, comments ignored)\n'; bad=1; fi + if ! grep -qE '^[[:space:]]+schedule:' <<<"$blk"; then + printf 'FAIL R3 coverage-nightly.yml must trigger on schedule (on: block, comments ignored)\n'; bad=1 + elif grep -qE "$V_TAGS_RE" <<<"$blk"; then + printf 'FAIL R3 coverage-nightly.yml also triggers on v* tags -- ci.yml owns tag coverage (R5); a tag would be measured twice\n'; bad=1 + else printf 'ok R3 coverage-nightly.yml triggers on schedule and not on v* tags\n'; fi if grep -qE '^COV_FLOOR[[:space:]]*:?=[[:space:]]*[0-9]+' "$mk"; then printf 'ok R4 the Makefile defines a numeric COV_FLOOR\n' else printf 'FAIL R4 the Makefile has no numeric COV_FLOOR -- make coverage enforces nothing\n'; bad=1; fi + + ci=$(live "$cy") + if grep -qE '^[[:space:]]+skip_coverage:[[:space:]]*true' <<<"$ci"; then + printf 'FAIL R5 ci.yml sets skip_coverage: true -- sovereign-ci'"'"'s gate reads that skip as a mandatory failure (ci / gate red on every PR); use coverage_on: tag\n'; bad=1 + elif ! grep -qE "^[[:space:]]+coverage_on:[[:space:]]*['\"]?tag['\"]?[[:space:]]*$" <<<"$ci"; then + printf 'FAIL R5 ci.yml does not hand sovereign-ci coverage_on: tag (on a live line)\n'; bad=1 + elif ! grep -qE "$V_TAGS_RE" <<<"$(on_block "$cy")"; then + printf "FAIL R5 ci.yml's on: block has no push tags [v*] -- coverage_on: tag alone never fires on a tag (the reusable's second required edit)\n"; bad=1 + else printf 'ok R5 ci.yml: coverage_on: tag, and on: push tags [v*] (coverage is owed on every release tag)\n'; fi return "$bad" } -case "${1:-}" in -h|--help) sed -n '2,24p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; esac +# The header is every leading comment line, found rather than counted. +case "${1:-}" in -h|--help) awk 'NR == 1 { next } !/^#/ { exit } { sub(/^# ?/, ""); print }' "$0"; exit 0 ;; esac if [ "${1:-}" = "--self-test" ]; then echo "=== coverage producer chain: each rule must turn RED when its link breaks ===" @@ -65,19 +86,26 @@ if [ "${1:-}" = "--self-test" ]; then mkdir -p "$1/scripts" "$1/.github/workflows" printf 'COV_FLOOR := 88\nLLVMCOV := llvm-cov\ncoverage:\n\t$(LLVMCOV) report --fail-under-lines $(COV_FLOOR)\ncoverage-check: coverage\n' > "$1/Makefile" printf '#!/usr/bin/env bash\n gate coverage make -C "$d" coverage-check\n' > "$1/scripts/dogfood.sh" - printf "on:\n schedule:\n - cron: '0 22 * * *'\n push:\n tags: ['v*']\njobs: {}\n" > "$1/.github/workflows/coverage-nightly.yml" + printf "on:\n schedule:\n - cron: '0 22 * * *'\n workflow_dispatch: {}\njobs: {}\n" > "$1/.github/workflows/coverage-nightly.yml" + printf "on:\n push:\n branches: [main, master]\n tags: ['v*']\n pull_request:\n branches: [main, master]\njobs:\n ci:\n uses: paiml/.github/.github/workflows/sovereign-ci.yml@x\n with:\n coverage_on: tag\n" > "$1/.github/workflows/ci.yml" } # one mutation per row, as a function (no eval: the mutation is code, not a string) - m_none() { :; } - m_r1() { sed -i 's/\$(LLVMCOV) report.*/echo nothing/' Makefile; } - m_r2_gone() { printf '#!/usr/bin/env bash\n' > scripts/dogfood.sh; } - m_r2_comment() { printf '#!/usr/bin/env bash\n# gate coverage make -C x coverage-check\n' > scripts/dogfood.sh; } - m_r3_gone() { sed -i '/push:/d; /tags:/d' .github/workflows/coverage-nightly.yml; } - m_r3_comment() { sed -i "s/^ tags: \['v\*'\]/ # tags: ['v*']/" .github/workflows/coverage-nightly.yml; } - m_r3_nosched() { sed -i '/schedule:/d; /cron:/d' .github/workflows/coverage-nightly.yml; } - m_r3_otherkey() { printf "on:\n schedule:\n - cron: '0 22 * * *'\nenv:\n tags: ['v*']\njobs: {}\n" > .github/workflows/coverage-nightly.yml; } - m_r4() { sed -i '/^COV_FLOOR/d' Makefile; } - m_missing() { rm -f .github/workflows/coverage-nightly.yml; } + m_none() { :; } + m_r1() { sed -i 's/\$(LLVMCOV) report.*/echo nothing/' Makefile; } + m_r2_gone() { printf '#!/usr/bin/env bash\n' > scripts/dogfood.sh; } + m_r2_comment() { printf '#!/usr/bin/env bash\n# gate coverage make -C x coverage-check\n' > scripts/dogfood.sh; } + m_r3_nosched() { sed -i '/schedule:/d; /cron:/d' .github/workflows/coverage-nightly.yml; } + m_r3_tagsback() { printf "on:\n schedule:\n - cron: '0 22 * * *'\n push:\n tags: ['v*']\njobs: {}\n" > .github/workflows/coverage-nightly.yml; } + m_r4() { sed -i '/^COV_FLOOR/d' Makefile; } + m_r5_skip() { sed -i 's/coverage_on: tag/skip_coverage: true/' .github/workflows/ci.yml; } + m_r5_both() { printf ' skip_coverage: true\n' >> .github/workflows/ci.yml; } + m_r5_nocov() { sed -i '/coverage_on:/d' .github/workflows/ci.yml; } + m_r5_comment() { sed -i 's/ coverage_on: tag/ # coverage_on: tag/' .github/workflows/ci.yml; } + m_r5_always() { sed -i 's/coverage_on: tag/coverage_on: always/' .github/workflows/ci.yml; } + m_r5_notag() { sed -i "/ tags: \['v\*'\]/d" .github/workflows/ci.yml; } + m_r5_tagcomment() { sed -i "s/^ tags: \['v\*'\]/ # tags: ['v*']/" .github/workflows/ci.yml; } + m_missing() { rm -f .github/workflows/coverage-nightly.yml; } + m_missing_ci() { rm -f .github/workflows/ci.yml; } row() { # row WANT-RC LABEL MUTATION-FUNCTION local want=$1 label=$2 rc=0; n=$((n + 1)); rmtree "$d/t"; fixture "$d/t" ( cd "$d/t" && "$3" ) || { printf 'FAIL row %s fixture mutation failed: %s\n' "$n" "$label"; bad=1; return; } @@ -85,16 +113,22 @@ if [ "${1:-}" = "--self-test" ]; then if [ "$rc" = "$want" ]; then printf 'ok row %-2s rc=%s %s\n' "$n" "$rc" "$label" else printf 'FAIL row %-2s rc=%s (wanted %s) %s\n' "$n" "$rc" "$want" "$label"; sed 's/^/ /' "$d/out"; bad=1; fi } - row 0 "every link holds -> PASS" m_none - row 1 "R1: coverage no longer runs llvm-cov -> RED" m_r1 - row 1 "R2: dogfood stops calling coverage-check -> RED" m_r2_gone - row 1 "R2: the call only in a COMMENT -> RED" m_r2_comment - row 1 "R3: the tag trigger removed -> RED" m_r3_gone - row 1 "R3: the tag trigger only in a COMMENT -> RED" m_r3_comment - row 1 "R3: schedule removed (tags only) -> RED" m_r3_nosched - row 1 "R3: tags: under a DIFFERENT top-level key, not on: -> RED" m_r3_otherkey - row 1 "R4: COV_FLOOR removed -> RED" m_r4 - row 2 "a missing file is ENV rc=2, never a pass" m_missing + row 0 "every link holds -> PASS" m_none + row 1 "R1: coverage no longer runs llvm-cov -> RED" m_r1 + row 1 "R2: dogfood stops calling coverage-check -> RED" m_r2_gone + row 1 "R2: the call only in a COMMENT -> RED" m_r2_comment + row 1 "R3: nightly schedule removed -> RED" m_r3_nosched + row 1 "R3: the nightly's v* tag trigger back (a tag measured twice) -> RED" m_r3_tagsback + row 1 "R4: COV_FLOOR removed -> RED" m_r4 + row 1 "R5: skip_coverage: true instead of coverage_on (the #3688 red) -> RED" m_r5_skip + row 1 "R5: skip_coverage: true alongside coverage_on -> RED" m_r5_both + row 1 "R5: coverage_on removed -> RED" m_r5_nocov + row 1 "R5: coverage_on only in a COMMENT -> RED" m_r5_comment + row 1 "R5: coverage_on: always (coverage back on every PR) -> RED" m_r5_always + row 1 "R5: ci.yml's v* tag trigger removed -> RED" m_r5_notag + row 1 "R5: ci.yml's v* tag trigger only in a COMMENT -> RED" m_r5_tagcomment + row 2 "coverage-nightly.yml missing is ENV rc=2, never a pass" m_missing + row 2 "ci.yml missing is ENV rc=2, never a pass" m_missing_ci [ "$bad" = 0 ] && { printf 'SELF-TEST PASSED: %s rows\n' "$n"; exit 0; } printf 'SELF-TEST FAILED\n' >&2; exit 1 fi From a738b926c889c40c27c8e85b81c7b3b0e257bcfe Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 17:37:22 +0200 Subject: [PATCH 4/4] =?UTF-8?q?fix(ci):=20coverage=20on=20tags=20only,=20d?= =?UTF-8?q?one=20so=20sovereign-ci's=20gate=20reads=20it=20=E2=80=94=20`co?= =?UTF-8?q?verage=5Fon:=20tag`=20on=20the=20merged=20#77=20pin,=20not=20`s?= =?UTF-8?q?kip=5Fcoverage:=20true`=20(PMAT-3676)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `skip_coverage: true` on sovereign-ci@70e51ec made `ci / gate` RED on every PR: that reusable's gate counts a skipped coverage job as a mandatory failure ("coverage did not succeed (result: skipped)"). #3688's own CI (run 35585906113) and batch-1 #3707 both hit it, and #3707 left this row out. - sovereign-ci is pinned to paiml/.github f713290c86fcd70d6f26a0faab14e67e6713586f: #74's `coverage_on` input, plus #77, which keeps #73's roadmap-fragment-parity NOT-ACTIVE while roadmap.yaml is tracked at the base (aprender still commits it; #3709 migrates it). - `coverage_on: tag` replaces `skip_coverage: true`, and ci.yml gets `push: tags: ['v*']`, the reusable's second required edit. - coverage-nightly's v* tag trigger is removed, so ci.yml measures each tag once. - check_coverage_has_producers.sh R3/R5 assert all of this. Self-test 16/16, including "skip_coverage: true instead of coverage_on -> RED". On f7b25a6c6 it reports R3+R5 RED; after the change it PASSES. - ci.yml now runs `check_pr_closes_issue.sh --self-test` beside `--body` (guard_tree skips it as wired-with-args, so its case table ran in no job; found by aprender-f0), and the comment claiming otherwise is corrected. Local: guard_tree --no-cargo 82/0 (after merging main 225b2a9ab). The proof before re-folding into batch-2 is this PR's own CI: green, with the gate logging `coverage: NOT MEASURED` and `roadmap-fragment-parity: NOT-ACTIVE`. Refs #3676 #3707 #3709 paiml/.github#77 Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 2 +- docs/roadmaps/entries/PMAT-3676.yaml | 2 +- docs/roadmaps/roadmap.yaml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7e176066cf..6f5bfc10f7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -67,7 +67,7 @@ jobs: # 23916c00d adds an OPTIONAL `runs_on` input whose default is the label set # this repo already gets, and 70e51ec04 swaps the digest on every job. No # input this repo passes changes meaning. - uses: paiml/.github/.github/workflows/sovereign-ci.yml@5997b411a371d0902f2ade4cba31e04ac6f87580 + uses: paiml/.github/.github/workflows/sovereign-ci.yml@f713290c86fcd70d6f26a0faab14e67e6713586f with: repo: ${{ github.event.repository.name }} # Phase 3 pilot (heavy workload) — build-performance.md §7 Phase 3. diff --git a/docs/roadmaps/entries/PMAT-3676.yaml b/docs/roadmaps/entries/PMAT-3676.yaml index eab79f976b..46e139e00f 100644 --- a/docs/roadmaps/entries/PMAT-3676.yaml +++ b/docs/roadmaps/entries/PMAT-3676.yaml @@ -14,4 +14,4 @@ estimated_effort: null labels: - kind:code - notes: 'ACCEPTANCE (quoted from issue #3676 "Proposal" + "done_when"; operator 08:38Z: "YES, coverage on tags release only"; Noah approved all five workflow items directly to aprender-b3): 1. Coverage (ci / coverage) on the release tag and the nightly only, never on pull_request or merge_group. COV_FLOOR enforcement moves to the tag gate (T-2) and the nightly, where a regression still blocks the release. 2. Book Chapter Examples and mdBook: path-filtered. They run when book/** or the crates a chapter imports change, and nightly always. 3. The perf gate case table and the model-tests falsification suites move to nightly. The PR path keeps a smoke row. 4. mutants runs only when the diff has .rs. 5. Every moved job still runs on the tag. check_workflow_path_filters.sh and the gate aggregator are updated so a skipped job is a pass only when the skip reason is the documented filter. done_when: intel runner-minutes per 24 h drop by >= 40% at equal PR throughput (measured after merge); the tag gate still runs coverage with COV_FLOOR, and a planted coverage drop fails the tag (control); a book-only PR runs the book jobs, a crate-only PR not imported by a chapter does not (both rows); gate accepts skipped only for jobs whose filter says skip, any other skipped state fails (mutation). COP ADDENDUM (aprender-04, relaying rmedia-88): before removing coverage from ci.yml, enumerate every consumer of the coverage number and prove each still has a producer, with a guard asserting the chain.' + notes: 'ACCEPTANCE (quoted from issue #3676 "Proposal" + "done_when"; operator 08:38Z: "YES, coverage on tags release only"; Noah approved all five workflow items directly to aprender-b3): 1. Coverage (ci / coverage) on the release tag and the nightly only, never on pull_request or merge_group. COV_FLOOR enforcement moves to the tag gate (T-2) and the nightly, where a regression still blocks the release. 2. Book Chapter Examples and mdBook: path-filtered. They run when book/** or the crates a chapter imports change, and nightly always. 3. The perf gate case table and the model-tests falsification suites move to nightly. The PR path keeps a smoke row. 4. mutants runs only when the diff has .rs. 5. Every moved job still runs on the tag. check_workflow_path_filters.sh and the gate aggregator are updated so a skipped job is a pass only when the skip reason is the documented filter. done_when: intel runner-minutes per 24 h drop by >= 40% at equal PR throughput (measured after merge); the tag gate still runs coverage with COV_FLOOR, and a planted coverage drop fails the tag (control); a book-only PR runs the book jobs, a crate-only PR not imported by a chapter does not (both rows); gate accepts skipped only for jobs whose filter says skip, any other skipped state fails (mutation). COP ADDENDUM (aprender-04, relaying rmedia-88): before removing coverage from ci.yml, enumerate every consumer of the coverage number and prove each still has a producer, with a guard asserting the chain. (A), THE ci / gate FIX (cop ruling 2026-09-21 ~15:00Z; batch-1 #3707 left this row out): the first spelling, `skip_coverage: true` on sovereign-ci@70e51ec, made `ci / gate` RED on every PR: that reusable''s gate counts a skipped coverage job as a mandatory failure ("coverage did not succeed (result: skipped)"). This PR''s own CI was red on it (run 35585906113, cancelled unread) and so was batch-1. The fix: pin sovereign-ci to paiml/.github f713290c86fcd70d6f26a0faab14e67e6713586f (#74 `coverage_on` + #77), pass `coverage_on: tag`, add `push: tags: [v*]` to ci.yml (the reusable''s second required edit), and remove coverage-nightly''s v* tag trigger so each tag is measured once, by ci.yml. The same pin also brings #73''s roadmap-fragment-parity, which would have refused every aprender PR (aprender still commits roadmap.yaml). #77 keys it on the BASE tree: NOT-ACTIVE while roadmap.yaml is tracked, and it arms by itself when #3709 migrates. scripts/check_coverage_has_producers.sh now asserts R3 (the nightly runs on schedule, NOT on v* tags) and R5 (ci.yml has coverage_on: tag, no live skip_coverage: true, and on: push tags [v*]), with 16/16 self-test rows including "skip_coverage: true instead of coverage_on -> RED". On f7b25a6c6 the guard reports R3 and R5 RED; after the change it PASSES. Also in this change (cop, found by aprender-f0): ci.yml runs `check_pr_closes_issue.sh --self-test` beside `--body`, because guard_tree skips it as wired-with-args and its case table ran in no job, and the ci.yml comment that claimed otherwise is corrected. PROOF REQUIRED before re-folding: this PR''s own CI green, with the gate logging `coverage: NOT MEASURED` and `roadmap-fragment-parity: NOT-ACTIVE`.' diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index 03104398c4..172b8678f6 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -19864,7 +19864,7 @@ roadmap: estimated_effort: null labels: - kind:code - notes: 'ACCEPTANCE (quoted from issue #3676 "Proposal" + "done_when"; operator 08:38Z: "YES, coverage on tags release only"; Noah approved all five workflow items directly to aprender-b3): 1. Coverage (ci / coverage) on the release tag and the nightly only, never on pull_request or merge_group. COV_FLOOR enforcement moves to the tag gate (T-2) and the nightly, where a regression still blocks the release. 2. Book Chapter Examples and mdBook: path-filtered. They run when book/** or the crates a chapter imports change, and nightly always. 3. The perf gate case table and the model-tests falsification suites move to nightly. The PR path keeps a smoke row. 4. mutants runs only when the diff has .rs. 5. Every moved job still runs on the tag. check_workflow_path_filters.sh and the gate aggregator are updated so a skipped job is a pass only when the skip reason is the documented filter. done_when: intel runner-minutes per 24 h drop by >= 40% at equal PR throughput (measured after merge); the tag gate still runs coverage with COV_FLOOR, and a planted coverage drop fails the tag (control); a book-only PR runs the book jobs, a crate-only PR not imported by a chapter does not (both rows); gate accepts skipped only for jobs whose filter says skip, any other skipped state fails (mutation). COP ADDENDUM (aprender-04, relaying rmedia-88): before removing coverage from ci.yml, enumerate every consumer of the coverage number and prove each still has a producer, with a guard asserting the chain.' + notes: 'ACCEPTANCE (quoted from issue #3676 "Proposal" + "done_when"; operator 08:38Z: "YES, coverage on tags release only"; Noah approved all five workflow items directly to aprender-b3): 1. Coverage (ci / coverage) on the release tag and the nightly only, never on pull_request or merge_group. COV_FLOOR enforcement moves to the tag gate (T-2) and the nightly, where a regression still blocks the release. 2. Book Chapter Examples and mdBook: path-filtered. They run when book/** or the crates a chapter imports change, and nightly always. 3. The perf gate case table and the model-tests falsification suites move to nightly. The PR path keeps a smoke row. 4. mutants runs only when the diff has .rs. 5. Every moved job still runs on the tag. check_workflow_path_filters.sh and the gate aggregator are updated so a skipped job is a pass only when the skip reason is the documented filter. done_when: intel runner-minutes per 24 h drop by >= 40% at equal PR throughput (measured after merge); the tag gate still runs coverage with COV_FLOOR, and a planted coverage drop fails the tag (control); a book-only PR runs the book jobs, a crate-only PR not imported by a chapter does not (both rows); gate accepts skipped only for jobs whose filter says skip, any other skipped state fails (mutation). COP ADDENDUM (aprender-04, relaying rmedia-88): before removing coverage from ci.yml, enumerate every consumer of the coverage number and prove each still has a producer, with a guard asserting the chain. (A), THE ci / gate FIX (cop ruling 2026-09-21 ~15:00Z; batch-1 #3707 left this row out): the first spelling, `skip_coverage: true` on sovereign-ci@70e51ec, made `ci / gate` RED on every PR: that reusable''s gate counts a skipped coverage job as a mandatory failure ("coverage did not succeed (result: skipped)"). This PR''s own CI was red on it (run 35585906113, cancelled unread) and so was batch-1. The fix: pin sovereign-ci to paiml/.github f713290c86fcd70d6f26a0faab14e67e6713586f (#74 `coverage_on` + #77), pass `coverage_on: tag`, add `push: tags: [v*]` to ci.yml (the reusable''s second required edit), and remove coverage-nightly''s v* tag trigger so each tag is measured once, by ci.yml. The same pin also brings #73''s roadmap-fragment-parity, which would have refused every aprender PR (aprender still commits roadmap.yaml). #77 keys it on the BASE tree: NOT-ACTIVE while roadmap.yaml is tracked, and it arms by itself when #3709 migrates. scripts/check_coverage_has_producers.sh now asserts R3 (the nightly runs on schedule, NOT on v* tags) and R5 (ci.yml has coverage_on: tag, no live skip_coverage: true, and on: push tags [v*]), with 16/16 self-test rows including "skip_coverage: true instead of coverage_on -> RED". On f7b25a6c6 the guard reports R3 and R5 RED; after the change it PASSES. Also in this change (cop, found by aprender-f0): ci.yml runs `check_pr_closes_issue.sh --self-test` beside `--body`, because guard_tree skips it as wired-with-args and its case table ran in no job, and the ci.yml comment that claimed otherwise is corrected. PROOF REQUIRED before re-folding: this PR''s own CI green, with the gate logging `coverage: NOT MEASURED` and `roadmap-fragment-parity: NOT-ACTIVE`.' - id: PMAT-3681 github_issue: 3681 item_type: task