From 85cf9a3ecab7d44b2d485dc8c6a452a933e1f449 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 08:06:06 +0200 Subject: [PATCH 01/28] =?UTF-8?q?fix(guard):=20check=5Fpr=5Freview=5Fcount?= =?UTF-8?q?s.sh=20was=20RED=20on=20main=20and=20ran=20nowhere=20=E2=80=94?= =?UTF-8?q?=20a=20comment=20classified=20it=20out=20of=20guard-tree=20(PMA?= =?UTF-8?q?T-3646)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PRQ-013 (#2985) deleted ci.yml's receipt job and, with it, the step names that stated six of the counts this guard checks. The guard went RED ("6 row(s) disagree with the tree") and nobody saw it, because a comment at :196 naming the build tool matched guard_tree.sh's CARGO_RE. That dropped the guard from guard-tree's --no-cargo run, and guard-cargo never names it. - The site table drops the five ci.yml rows whose sentences left with the job (the roadmap records them as a STATED LOSS). ci.yml's "43-row" goes from 2 sites to 1. The bats_tests derivation is removed: no file states that count any more, and a derivation with no site checks nothing. check_pr_review_wiring.sh's "43-row" joins the table as a new site, with a self-test row (stale-wiring-rows). - The :196 comment no longer names the tool, so `guard_tree.sh --dry-run --no-cargo` prints `run: scripts/check_pr_review_counts.sh`. Restoring the old wording moves it back to --cargo-only (mutation-verified). - In --list mode, mutate-guard.sh and mutate_quorum_arm.sh now require only awk. They demanded bats/jq/minisign/check-jsonschema before printing a catalogue that uses none of them, so the counts derivation exited ENV=2 on a runner without them. Reproduced under a PATH without those four: rc=2 before, rc=0 after. guard-tree runs on the clean-room hosts, including the gx10/yoga docker runners, so wiring the guard without this would have swapped a dark guard for a red job. The real sweeps still refuse to run without the tools (rc=1). Verified: check rc=0 (22 rows); --self-test 16/16; guard_tree_test.sh 23/23; check_pr_review_wiring.sh PASS, self-test 11/11; check_guards_are_wired.sh PASS, self-test 7/7. Item 2 of #3646 needs no code. check_pr_review_receipt.sh is invoked with a receipt by check_pr_review_arm4.sh (A3 and A4), which pr-review-quorum.yml runs. The ledger line #3647 adds for this guard is deleted here once #3647 lands (the ledger is shrink-only). Refs #3646 Co-Authored-By: Claude Opus 5 (1M context) --- docs/roadmaps/entries/PMAT-3646.yaml | 23 +++++++++++ docs/roadmaps/roadmap.yaml | 23 +++++++++++ scripts/check_pr_review_counts.sh | 57 +++++++++++++++------------- scripts/mutate-guard.sh | 7 +++- scripts/mutate_quorum_arm.sh | 5 ++- 5 files changed, 86 insertions(+), 29 deletions(-) create mode 100644 docs/roadmaps/entries/PMAT-3646.yaml diff --git a/docs/roadmaps/entries/PMAT-3646.yaml b/docs/roadmaps/entries/PMAT-3646.yaml new file mode 100644 index 0000000000..b978739a78 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3646.yaml @@ -0,0 +1,23 @@ +- id: PMAT-3646 + github_issue: 3646 + item_type: task + title: check_pr_review_counts.sh is RED on main (6 rows disagree) and check_pr_review_receipt.sh has no caller — both dark since PRQ-013 moved the receipt job, classified out of --no-cargo by a comment + status: planned + priority: high + assigned_to: null + created: 2026-09-21T06:00:51Z + updated: 2026-09-21T06:00:51Z + spec: null + acceptance_criteria: + - 'Issue #3646 "What lands" item 1, verbatim: "Fix the 6 disagreeing rows in PR-REVIEW-SKILL-002''s files (or the derivation), then reword the one comment so the guard leaves the cargo population and `guard-tree` runs it; delete its ledger line in the same commit (shrink-only)."' + - 'Issue #3646 "What lands" item 2, verbatim: "Decide `check_pr_review_receipt.sh`''s caller: either `pr-review-quorum.yml` invokes it on the PR''s receipt (a workflow edit — operator''s call), or it is folded into `check_pr_review_arm4.sh`, which is already wired with args. Delete its ledger line with whichever lands."' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'Handed over by aprender-b3 (2026-09-21); claimed on the issue. Measured on origin/main @ 237fbc32f per the issue: `bash scripts/check_pr_review_counts.sh` rc=1 "--- 6 row(s) disagree with the tree ---"; `bash scripts/check_pr_review_receipt.sh` rc=1 "A run over zero receipts is not a pass (S6.3: a missing receipt is RED)." The ledger line for each guard is added by #3647 (scripts/unwired_guards_baseline.txt, shrink-only); it is deleted in the commit that wires that guard, after #3647 merges. Item 1 lands first; item 2 lands here only if it folds into check_pr_review_arm4.sh with no workflow edit, otherwise the ticket stays open for the operator''s call and the PR says Refs, not Closes. + + What the item-1 PR does. (a) The 6 RED rows were ci.yml sites whose sentences PRQ-013 deleted along with the receipt job (roadmap: "STATED LOSS"), so the rows go. The bats_tests derivation goes too, because no file states that count any more; ci.yml''s fixture_rows site goes from 2 to 1; and scripts/check_pr_review_wiring.sh''s "43-row" joins the table with a self-test row of its own. (b) The :196 comment is reworded so CARGO_RE stops matching: `guard_tree.sh --dry-run --no-cargo` prints `run: scripts/check_pr_review_counts.sh`, and restoring the old wording moves it back to --cargo-only (mutation-verified). (c) The --list mode of mutate-guard.sh and mutate_quorum_arm.sh requires only awk. Before this it demanded bats/jq/minisign/check-jsonschema, so wiring the counts guard into guard-tree, which runs on clean-room HOSTS including the gx10/yoga docker runners, would have turned a dark guard into an ENV-red job (reproduced: rc=2 under a PATH without those four). Sweeps still refuse without them (rc=1). + + Item 2, measured: check_pr_review_receipt.sh HAS a caller with a receipt. check_pr_review_arm4.sh runs `bash "$guard" "$scratch"` (A3) and `bash "$guard" "$best_dir"` (A4) with guard defaulting to it, and pr-review-quorum.yml:141 runs arm4. So there is nothing to fold. What remains is whether guard_tree.sh and the unwired ledger should treat a guard reachable only through another wired guard as wired. That is recorded on the issue and not changed here.' diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index 618b9a93ca..68d106c69b 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -18990,3 +18990,26 @@ roadmap: labels: - kind:code notes: 'ACCEPTANCE (hand-entered from issue #3627 done_when; minted on the PR branch so the AD-04 quorum can run -- pmat work status refuses a missing item). Done when: contracts/rmsnorm-kernel-v1.yaml Scale invariance obligation carries either the ε>0 formal or the α²·mean(x²) ≫ ε precondition; pv validate passes ~~regenerated with pv probar --binding~~ **WITHDRAWN 21:40Z** — b3 measured that pv probar --binding emits 7 #[ignore = "no binding available"] stubs for this contract; the tracked file''s 5 real tests were produced by the pre-monorepo external generator (20dda945b, 2026-02-18) and the header is false (class row: #3630). Instead: the tracked test is edited **directly**, the commit says so, and the file''s header is replaced by a true one (hand-maintained, contract-bound) the minimal input above ([0.0, -0.15964308], α=0.012254778) is a committed regression case that PASSES under the new assertion (i.e. the fixed test is exercised on the exact input that killed #3621), and a mutation that restores the ε=0 assertion turns it RED cargo' +- id: PMAT-3646 + github_issue: 3646 + item_type: task + title: check_pr_review_counts.sh is RED on main (6 rows disagree) and check_pr_review_receipt.sh has no caller — both dark since PRQ-013 moved the receipt job, classified out of --no-cargo by a comment + status: planned + priority: high + assigned_to: null + created: 2026-09-21T06:00:51Z + updated: 2026-09-21T06:00:51Z + spec: null + acceptance_criteria: + - 'Issue #3646 "What lands" item 1, verbatim: "Fix the 6 disagreeing rows in PR-REVIEW-SKILL-002''s files (or the derivation), then reword the one comment so the guard leaves the cargo population and `guard-tree` runs it; delete its ledger line in the same commit (shrink-only)."' + - 'Issue #3646 "What lands" item 2, verbatim: "Decide `check_pr_review_receipt.sh`''s caller: either `pr-review-quorum.yml` invokes it on the PR''s receipt (a workflow edit — operator''s call), or it is folded into `check_pr_review_arm4.sh`, which is already wired with args. Delete its ledger line with whichever lands."' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'Handed over by aprender-b3 (2026-09-21); claimed on the issue. Measured on origin/main @ 237fbc32f per the issue: `bash scripts/check_pr_review_counts.sh` rc=1 "--- 6 row(s) disagree with the tree ---"; `bash scripts/check_pr_review_receipt.sh` rc=1 "A run over zero receipts is not a pass (S6.3: a missing receipt is RED)." The ledger line for each guard is added by #3647 (scripts/unwired_guards_baseline.txt, shrink-only); it is deleted in the commit that wires that guard, after #3647 merges. Item 1 lands first; item 2 lands here only if it folds into check_pr_review_arm4.sh with no workflow edit, otherwise the ticket stays open for the operator''s call and the PR says Refs, not Closes. + + What the item-1 PR does. (a) The 6 RED rows were ci.yml sites whose sentences PRQ-013 deleted along with the receipt job (roadmap: "STATED LOSS"), so the rows go. The bats_tests derivation goes too, because no file states that count any more; ci.yml''s fixture_rows site goes from 2 to 1; and scripts/check_pr_review_wiring.sh''s "43-row" joins the table with a self-test row of its own. (b) The :196 comment is reworded so CARGO_RE stops matching: `guard_tree.sh --dry-run --no-cargo` prints `run: scripts/check_pr_review_counts.sh`, and restoring the old wording moves it back to --cargo-only (mutation-verified). (c) The --list mode of mutate-guard.sh and mutate_quorum_arm.sh requires only awk. Before this it demanded bats/jq/minisign/check-jsonschema, so wiring the counts guard into guard-tree, which runs on clean-room HOSTS including the gx10/yoga docker runners, would have turned a dark guard into an ENV-red job (reproduced: rc=2 under a PATH without those four). Sweeps still refuse without them (rc=1). + + Item 2, measured: check_pr_review_receipt.sh HAS a caller with a receipt. check_pr_review_arm4.sh runs `bash "$guard" "$scratch"` (A3) and `bash "$guard" "$best_dir"` (A4) with guard defaulting to it, and pr-review-quorum.yml:141 runs arm4. So there is nothing to fold. What remains is whether guard_tree.sh and the unwired ledger should treat a guard reachable only through another wired guard as wired. That is recorded on the issue and not changed here.' diff --git a/scripts/check_pr_review_counts.sh b/scripts/check_pr_review_counts.sh index d2ba36f814..9955aa41ca 100755 --- a/scripts/check_pr_review_counts.sh +++ b/scripts/check_pr_review_counts.sh @@ -101,15 +101,6 @@ derive_fixture_rows() { printf '%s\n' "$n" } -# bats tests: `@test` at column 0 in the file the CI step actually runs. -derive_bats_tests() { - local root=$1 n - [ -f "$root/tests/pr-review.bats" ] || return 1 - n=$(grep -c '^@test ' "$root/tests/pr-review.bats") - [ "$n" -gt 0 ] || return 1 - printf '%s\n' "$n" -} - # S13 (PRREV-015) adds a SECOND table, a SECOND bats file and a SECOND mutation set. # They are derived here rather than left uncounted, because the universe this guard # iterates is what decides whether it can fail at all: a table it has never heard of is @@ -193,9 +184,11 @@ derive_publish_rows() { # Falsification tests: the contract's own list, counted from its `- id:` labels rather # than from the prose that states the total. It went stale unseen - `pass_criteria` said # 11 over 15 entries at PRREV-013 and over 17 at PRREV-015 - and the only thing that -# noticed was `cargo test -p aprender-contracts --test validate_contracts` printing +# noticed was the aprender-contracts `validate_contracts` integration test printing # `pr-review-skill-v2: pass_criteria says 11 tests, actual 17` into a corpus-wide failure -# nobody reads per-contract. Counting labels and not lines, the same reason S7's rows +# nobody reads per-contract. (Name that test runner's binary in prose here and +# guard_tree.sh's CARGO_RE classifies this whole guard as build-tool-using, which +# drops it from guard-tree's --no-cargo run. That is how it went dark: #3646.) Counting labels and not lines, the same reason S7's rows # carry ids B1..B6. derive_falsification_tests() { local root=$1 n @@ -218,32 +211,43 @@ derive_falsification_tests() { # --------------------------------------------------------------------------- # The site table. id | file | occurrences | template (@N@ -> derived value) +# +# SIX ci.yml ROWS LEFT WITH THE JOB THAT STATED THEM (#3646). PRQ-013 (#2982, #2985) +# deleted ci.yml's receipt job, and with it the step names that carried "233/233", +# "165 tests", "134-mutant" x2, "83 rows" and "Arm 4 case table: 10 rows", plus one +# of the two "43-row" sites. That is the "sentence was deleted" case, and here it was +# deliberate: the roadmap records the sweeps and the fixture table as a STATED LOSS. +# So the rows go, not the sentences. Nobody noticed for weeks because a comment +# naming the build tool had classified this guard out of guard-tree's --no-cargo +# population, so it went RED on main and ran nowhere. The ci.yml rows that remain are +# for step names that still exist. +# +# The bats_tests count went with them. No file states the size of tests/pr-review.bats +# any more, and a derivation with no site checks nothing, so it was removed rather +# than left to read as coverage. If a file states that count again, derive it again +# and add the row. # --------------------------------------------------------------------------- SITES=' mutants|.claude/skills/pr-review/SKILL.md|2|@N@/@N@ mutants|contracts/binding.yaml|1|@N@/@N@ -mutants|.github/workflows/ci.yml|1|@N@/@N@ mutants|docs/specifications/PR-REVIEW-SKILL-002-v2.md|1|@N@/@N@ -fixture_rows|.github/workflows/ci.yml|2|@N@-row +fixture_rows|.github/workflows/ci.yml|1|@N@-row +fixture_rows|scripts/check_pr_review_wiring.sh|1|@N@-row fixture_rows|tests/pr-review.bats|1|@N@ row fixture_rows|tests/pr-review.bats|1|-eq @N@ ] -bats_tests|.github/workflows/ci.yml|2|@N@ tests quorum_mutants|docs/specifications/PR-REVIEW-SKILL-002-v2.md|3|@N@/@N@ quorum_mutants|.claude/skills/pr-review/SKILL.md|1|@N@/@N@ quorum_mutants|contracts/pr-review-skill-v2.yaml|1|@N@/@N@ quorum_mutants|.claude/skills/pr-review/SKILL.md|1|@N@-mutant -quorum_mutants|.github/workflows/ci.yml|2|@N@-mutant quorum_mutants|docs/specifications/PR-REVIEW-SKILL-002-v2.md|2|@N@ mutants quorum_bats_tests|docs/specifications/PR-REVIEW-SKILL-002-v2.md|1|@N@ rows quorum_bats_tests|docs/specifications/PR-REVIEW-SKILL-002-v2.md|1|@N@-row quorum_bats_tests|.claude/skills/pr-review/SKILL.md|2|@N@ rows -quorum_bats_tests|.github/workflows/ci.yml|1|@N@ rows quorum_rows|tests/pr-review-quorum.bats|1|-eq @N@ ] quorum_rows|tests/pr-review-quorum.bats|1|expected @N@ q-* falsification_tests|contracts/pr-review-skill-v2.yaml|1|All @N@ falsification tests shadow_rows|.github/workflows/ci.yml|1|case table: @N@ rows publish_rows|.github/workflows/ci.yml|1|Publisher case table: @N@ rows -arm4_rows|.github/workflows/ci.yml|1|Arm 4 case table: @N@ rows arm4_rows|.github/workflows/pr-review-quorum.yml|1|Arm 4 case table: @N@ rows signer_rows|.github/workflows/ci.yml|1|Signer case table: @N@ rows ' @@ -259,8 +263,6 @@ check() { derived[mutants]=$v v=$(derive_fixture_rows "$root") || die_env "cannot derive the fixture-row count" derived[fixture_rows]=$v - v=$(derive_bats_tests "$root") || die_env "cannot derive the bats test count" - derived[bats_tests]=$v v=$(derive_quorum_mutants "$root") || die_env "cannot derive the S13 mutation-set size (mutate_quorum_arm.sh --list)" derived[quorum_mutants]=$v v=$(derive_quorum_rows "$root") || die_env "cannot derive the S13 q-* fixture-row count" @@ -280,7 +282,7 @@ check() { [ -n "$quiet" ] || { echo "=== the counts these files state as measured, against the tree ($PROG) ===" - echo "derived: mutants=${derived[mutants]} fixture_rows=${derived[fixture_rows]} bats_tests=${derived[bats_tests]}" + echo "derived: mutants=${derived[mutants]} fixture_rows=${derived[fixture_rows]}" echo " quorum_mutants=${derived[quorum_mutants]} quorum_rows=${derived[quorum_rows]} quorum_bats_tests=${derived[quorum_bats_tests]}" echo " falsification_tests=${derived[falsification_tests]} shadow_rows=${derived[shadow_rows]} publish_rows=${derived[publish_rows]} arm4_rows=${derived[arm4_rows]} signer_rows=${derived[signer_rows]}" } @@ -381,7 +383,8 @@ self_test() { scripts/pr_review_quorum_arm.sh \ scripts/pr_review_shadow_record.sh \ scripts/pr_review_shadow_publish.sh \ - scripts/check_pr_review_arm4.sh scripts/pr_review_sign_receipt.sh ) \ + scripts/check_pr_review_arm4.sh scripts/pr_review_sign_receipt.sh \ + scripts/check_pr_review_wiring.sh ) \ | ( cd "$base" && tar -xf - ) || die_env "could not stage a copy of the tree" local nrows=0 @@ -428,8 +431,9 @@ self_test() { sed -i "s#$n/$n#$((n - 1))/$((n - 1))#" "$1/.claude/skills/pr-review/SKILL.md"; } stale_rows() { local n; n=$(derive_fixture_rows "$1") sed -i "s#$n-row#$((n - 1))-row#" "$1/.github/workflows/ci.yml"; } - stale_tests() { local n; n=$(derive_bats_tests "$1") - sed -i "s#$n tests#$((n - 1)) tests#" "$1/.github/workflows/ci.yml"; } + # The site #3646 added: check_pr_review_wiring.sh states the fixture-row count too. + stale_wrows() { local n; n=$(derive_fixture_rows "$1") + sed -i "s#$n-row#$((n - 1))-row#" "$1/scripts/check_pr_review_wiring.sh"; } delete_claim() { local n; n=$(derive_mutants "$1") sed -i "0,\#$n/$n#{\#$n/$n#d}" "$1/contracts/binding.yaml"; } stale_ftests() { local n; n=$(derive_falsification_tests "$1") @@ -440,7 +444,6 @@ self_test() { # let exactly this edit through green. grow_ftests() { sed -i 's#^- id: F-PRREV-002$#- id: F-INVENTED-999\n rule: a falsification test the contract does not count\n\n&#' \ "$1/contracts/pr-review-skill-v2.yaml"; } - grow_bats() { printf '\n@test "a new test the docs do not count" {\n true\n}\n' >> "$1/tests/pr-review.bats"; } grow_rows() { mkdir -p "$1/tests/fixtures/pr-review/row-27-invented"; } grow_mutants() { sed -i 's#^ \[ -n "$head" \] || reject B1 "predicate.head_sha is absent" || return 1#&\n [ -n "$head" ] || reject B1 "an invented rule nothing documents" || return 1#' \ "$1/scripts/check_pr_review_receipt.sh"; } @@ -470,9 +473,10 @@ self_test() { row baseline 0 "the tree as committed" noop row stale-mutation-score 1 "the mutation score written back by one" stale_mutants row stale-fixture-rows 1 "the fixture-row count written back by one" stale_rows - row stale-bats-count 1 "the bats test count written back by one" stale_tests + row stale-wiring-rows 1 "the wiring guard's fixture-row count written back by one" stale_wrows + # stale-bats-count and tree-grew-a-test left with the bats_tests derivation (#3646): + # no file states that count, so neither edit has a claim to falsify. row claim-deleted 1 "the sentence stating the count is deleted" delete_claim - row tree-grew-a-test 1 "a bats test is added and no file says so" grow_bats row tree-grew-a-fixture-row 1 "a row-* fixture is added and no file says so" grow_rows row tree-grew-a-mutant 1 "a reject site is added and no file says so" grow_mutants row stale-quorum-mutation 1 "the S13 mutation score written back by 18" stale_qmutants @@ -505,7 +509,6 @@ case "${1:-}" in --show) printf 'mutants %s\n' "$(derive_mutants "$REPO_ROOT")" printf 'fixture_rows %s\n' "$(derive_fixture_rows "$REPO_ROOT")" - printf 'bats_tests %s\n' "$(derive_bats_tests "$REPO_ROOT")" printf 'quorum_mutants %s\n' "$(derive_quorum_mutants "$REPO_ROOT")" printf 'quorum_rows %s\n' "$(derive_quorum_rows "$REPO_ROOT")" printf 'quorum_bats_tests %s\n' "$(derive_quorum_bats_tests "$REPO_ROOT")" diff --git a/scripts/mutate-guard.sh b/scripts/mutate-guard.sh index 450fc4fced..ad466564b9 100755 --- a/scripts/mutate-guard.sh +++ b/scripts/mutate-guard.sh @@ -104,7 +104,12 @@ while [ "$#" -gt 0 ]; do esac done -for t in bats jq awk minisign check-jsonschema git; do +# --list prints the catalogue and runs no mutant, so it needs only what catalogue() +# reads with. Demanding the sweep's tools there made every count derived from the +# catalogue (check_pr_review_counts.sh) an ENV failure on a runner that cannot run the +# sweep, and guard-tree runs on clean-room hosts, not in a sweep image (#3646). +if [ "$LIST_ONLY" = 1 ]; then need='awk'; else need='bats jq awk minisign check-jsonschema git'; fi +for t in $need; do command -v "$t" >/dev/null 2>&1 || { echo "$PROG: FAIL - $t is not on PATH. A mutation score that could not be measured" >&2 echo " must not be reported as one (S8: guard_mutation_score is a one, not a floor)." >&2 diff --git a/scripts/mutate_quorum_arm.sh b/scripts/mutate_quorum_arm.sh index 995ca8b6da..0139c4de2d 100755 --- a/scripts/mutate_quorum_arm.sh +++ b/scripts/mutate_quorum_arm.sh @@ -72,7 +72,10 @@ while [ "$#" -gt 0 ]; do esac done -for t in bats jq awk minisign check-jsonschema git; do +# --list runs no mutant; it needs only what catalogue() reads with (see +# mutate-guard.sh, #3646). +if [ "$LIST_ONLY" = 1 ]; then need='awk'; else need='bats jq awk minisign check-jsonschema git'; fi +for t in $need; do command -v "$t" >/dev/null 2>&1 || { echo "$PROG: FAIL - $t is not on PATH. A mutation score that could not be measured" >&2 echo " must not be reported as one (S13.10: this number is a one, not a floor)." >&2 From e3ceb27f2b844fa6f5e9204dc9074ae7c7f4815a Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 09:55:50 +0200 Subject: [PATCH 02/28] PMAT-3076: F16/BF16 CPU matvec reads the row with AVX2+F16C+FMA instead of a fn pointer per weight float16_matmul called the decode through a `fn(u16) -> f32` pointer for every element: an indirect call plus a bounds check per weight, which nothing could inline or vectorise. On a Threadripper 7960X, Qwen2.5-0.5B-Instruct-f16 decoded at ~10 tok/s; BRICK-PROFILE put ~99% of the time in the F16 matvecs, and the 272 MB LM head read at ~27 GB/s. Each row is now one float16_row_dot over its bytes. x86_64 CPUs with avx2+fma (+f16c for F16) take a fused kernel, runtime-detected: vcvtph2ps, or u16->u32 <<16 for BF16, into four FMA accumulators. Everything else takes a chunked decode-then-dot the compiler vectorises. The loop structure is unchanged, and a row that runs past the end of the buffer is dotted over the whole elements that exist, as the per-element bounds check did. Tests: parity against an f64 reference at 16 lengths that reach every loop and tail (both kinds, and the portable path on its own); NaN, inf and subnormals at every lane position; and float16_matmul against a transcription of the pre-fix loop, including a truncated buffer with an odd trailing byte. Mutation-checked: skipping the tail, BF16 <<15, F16 routed to the BF16 kernel, dropping the portable remainder, and removing the end clamp each turn tests RED. Co-Authored-By: Claude Opus 5 (1M context) --- .../src/gguf/inference/float16_dot.rs | 176 +++++++++++++++++ .../src/gguf/inference/float16_dot_tests.rs | 177 ++++++++++++++++++ .../src/gguf/inference/matmul_fused.rs | 46 +++-- .../aprender-serve/src/gguf/inference/mod.rs | 2 + docs/roadmaps/entries/PMAT-3076.yaml | 17 ++ docs/roadmaps/roadmap.yaml | 17 ++ 6 files changed, 417 insertions(+), 18 deletions(-) create mode 100644 crates/aprender-serve/src/gguf/inference/float16_dot.rs create mode 100644 crates/aprender-serve/src/gguf/inference/float16_dot_tests.rs create mode 100644 docs/roadmaps/entries/PMAT-3076.yaml diff --git a/crates/aprender-serve/src/gguf/inference/float16_dot.rs b/crates/aprender-serve/src/gguf/inference/float16_dot.rs new file mode 100644 index 0000000000..c9022160cd --- /dev/null +++ b/crates/aprender-serve/src/gguf/inference/float16_dot.rs @@ -0,0 +1,176 @@ +// #3076: F16 / BF16 weight-row · f32 activation dot products for `float16_matmul`. +// +// The row loop used to decode each weight through a `fn(u16) -> f32` pointer: one indirect +// call per element, which nothing could inline or vectorise. Measured on a Threadripper 7960X +// (Zen 4, 24 cores), Qwen2.5-0.5B-Instruct-f16 decoded at ~10 tok/s, and the 272 MB LM head +// read at ~27 GB/s. These kernels read the row's bytes directly: AVX2 + F16C + FMA on x86_64 +// when the CPU has them (runtime-detected), and a chunked decode-then-dot everywhere else. + +/// The 2-byte float format a weight row is stored in. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) enum Float16Kind { + /// IEEE 754 binary16 (GGUF type 1). + F16, + /// bfloat16, the top half of an f32 (GGUF type 30). + Bf16, +} + +/// Dot product of a row of little-endian 2-byte floats with `x`. +/// +/// Covers `min(row.len() / 2, x.len())` elements: a trailing odd byte is not an element, which +/// is exactly what the per-element `offset + 1 < data.len()` check it replaces admitted. +pub(super) fn float16_row_dot(kind: Float16Kind, row: &[u8], x: &[f32]) -> f32 { + let n = (row.len() / 2).min(x.len()); + let (row, x) = (&row[..n * 2], &x[..n]); + + #[cfg(target_arch = "x86_64")] + { + if is_x86_feature_detected!("avx2") && is_x86_feature_detected!("fma") { + match kind { + Float16Kind::F16 if is_x86_feature_detected!("f16c") => { + // SAFETY: avx2, fma and f16c verified at runtime; `row.len() == 2 * x.len()`. + return unsafe { f16_row_dot_avx2(row, x) }; + }, + Float16Kind::Bf16 => { + // SAFETY: avx2 and fma verified at runtime; `row.len() == 2 * x.len()`. + return unsafe { bf16_row_dot_avx2(row, x) }; + }, + Float16Kind::F16 => {}, + } + } + } + float16_row_dot_portable(kind, row, x) +} + +/// A bfloat16 is the top 16 bits of an f32. +#[inline] +fn bf16_bits_to_f32(bits: u16) -> f32 { + f32::from_bits(u32::from(bits) << 16) +} + +/// Decode one element (`bits` is the little-endian u16 of the weight). +#[inline] +fn decode_float16(kind: Float16Kind, bits: u16) -> f32 { + match kind { + Float16Kind::F16 => crate::quantize::f16_to_f32_lut(bits), + Float16Kind::Bf16 => bf16_bits_to_f32(bits), + } +} + +/// The non-x86 path, and the x86 path on a CPU without the features above: decode a chunk into +/// a stack buffer, then accumulate it in eight independent lanes the compiler can vectorise. +pub(super) fn float16_row_dot_portable(kind: Float16Kind, row: &[u8], x: &[f32]) -> f32 { + const CHUNK: usize = 64; + let mut buf = [0.0f32; CHUNK]; + let mut lanes = [0.0f32; 8]; + let mut tail = 0.0f32; + for (bytes, xs) in row.chunks(CHUNK * 2).zip(x.chunks(CHUNK)) { + let m = xs.len(); + for (w, b) in buf[..m].iter_mut().zip(bytes.chunks_exact(2)) { + *w = decode_float16(kind, u16::from_le_bytes([b[0], b[1]])); + } + let (ws, xs8) = (buf[..m].chunks_exact(8), xs.chunks_exact(8)); + let (w_rem, x_rem) = (ws.remainder(), xs8.remainder()); + for (w8, x8) in ws.zip(xs8) { + for l in 0..8 { + lanes[l] += w8[l] * x8[l]; + } + } + tail += w_rem.iter().zip(x_rem).map(|(w, x)| w * x).sum::(); + } + lanes.iter().sum::() + tail +} + +/// The shared AVX2 loop: four accumulators of fused multiply-adds over 32 elements, then +/// 8 at a time, then a scalar tail. A macro rather than a generic function, so that each +/// expansion sits inside a kernel whose target features include its `$load8`, and the 8-lane +/// load inlines. A closure would not inherit those features, and would cost a call per 8 +/// weights. +#[cfg(target_arch = "x86_64")] +macro_rules! avx2_float16_row_dot { + ($row:ident, $x:ident, $load8:ident, $decode1:expr) => {{ + use std::arch::x86_64::{ + _mm256_add_ps, _mm256_castps256_ps128, _mm256_extractf128_ps, _mm256_fmadd_ps, + _mm256_loadu_ps, _mm256_setzero_ps, _mm_add_ps, _mm_cvtss_f32, _mm_hadd_ps, + }; + let n = $x.len(); + let (rp, xp) = ($row.as_ptr(), $x.as_ptr()); + let (mut a0, mut a1, mut a2, mut a3) = + (_mm256_setzero_ps(), _mm256_setzero_ps(), _mm256_setzero_ps(), _mm256_setzero_ps()); + let mut i = 0; + // `i + 32 <= n` (then `i + 8 <= n`) bounds every 16-byte `row` load at byte `2 * i` + // and every 8-float `x` load at `i`, given `row.len() == 2 * x.len()`. + while i + 32 <= n { + a0 = _mm256_fmadd_ps($load8(rp.add(2 * i)), _mm256_loadu_ps(xp.add(i)), a0); + a1 = _mm256_fmadd_ps($load8(rp.add(2 * i + 16)), _mm256_loadu_ps(xp.add(i + 8)), a1); + a2 = _mm256_fmadd_ps($load8(rp.add(2 * i + 32)), _mm256_loadu_ps(xp.add(i + 16)), a2); + a3 = _mm256_fmadd_ps($load8(rp.add(2 * i + 48)), _mm256_loadu_ps(xp.add(i + 24)), a3); + i += 32; + } + while i + 8 <= n { + a0 = _mm256_fmadd_ps($load8(rp.add(2 * i)), _mm256_loadu_ps(xp.add(i)), a0); + i += 8; + } + let acc = _mm256_add_ps(_mm256_add_ps(a0, a1), _mm256_add_ps(a2, a3)); + let s = _mm_add_ps(_mm256_castps256_ps128(acc), _mm256_extractf128_ps::<1>(acc)); + let s = _mm_hadd_ps(s, s); + let mut sum = _mm_cvtss_f32(_mm_hadd_ps(s, s)); + for j in i..n { + sum += $decode1(u16::from_le_bytes([$row[2 * j], $row[2 * j + 1]])) * $x[j]; + } + sum + }}; +} + +/// 8 F16 weights at `p` as f32 lanes (`vcvtph2ps`). +/// +/// # Safety +/// The CPU must support avx2 and f16c, and `p..p + 16` must be readable. +#[cfg(target_arch = "x86_64")] +#[target_feature(enable = "avx2,f16c")] +#[inline] +unsafe fn load8_f16(p: *const u8) -> std::arch::x86_64::__m256 { + use std::arch::x86_64::{__m128i, _mm256_cvtph_ps, _mm_loadu_si128}; + // SAFETY: the caller guarantees 16 readable bytes at `p` and the target features. + unsafe { _mm256_cvtph_ps(_mm_loadu_si128(p.cast::<__m128i>())) } +} + +/// 8 BF16 weights at `p` as f32 lanes: widen u16 to u32, shift into the top half. +/// +/// # Safety +/// The CPU must support avx2, and `p..p + 16` must be readable. +#[cfg(target_arch = "x86_64")] +#[target_feature(enable = "avx2")] +#[inline] +unsafe fn load8_bf16(p: *const u8) -> std::arch::x86_64::__m256 { + use std::arch::x86_64::{ + __m128i, _mm256_castsi256_ps, _mm256_cvtepu16_epi32, _mm256_slli_epi32, _mm_loadu_si128, + }; + // SAFETY: the caller guarantees 16 readable bytes at `p` and the target feature. + unsafe { + let wide = _mm256_cvtepu16_epi32(_mm_loadu_si128(p.cast::<__m128i>())); + _mm256_castsi256_ps(_mm256_slli_epi32::<16>(wide)) + } +} + +/// F16 row dot on AVX2 + F16C + FMA. +/// +/// # Safety +/// The CPU must support avx2, fma and f16c, and `row.len()` must be `2 * x.len()`. +#[cfg(target_arch = "x86_64")] +#[target_feature(enable = "avx2,fma,f16c")] +unsafe fn f16_row_dot_avx2(row: &[u8], x: &[f32]) -> f32 { + // SAFETY: loads are bounded as the macro documents; features are this fn's contract. + unsafe { avx2_float16_row_dot!(row, x, load8_f16, crate::quantize::f16_to_f32_lut) } +} + +/// BF16 row dot on AVX2 + FMA. +/// +/// # Safety +/// The CPU must support avx2 and fma, and `row.len()` must be `2 * x.len()`. +#[cfg(target_arch = "x86_64")] +#[target_feature(enable = "avx2,fma")] +unsafe fn bf16_row_dot_avx2(row: &[u8], x: &[f32]) -> f32 { + // SAFETY: loads are bounded as the macro documents; features are this fn's contract. + unsafe { avx2_float16_row_dot!(row, x, load8_bf16, bf16_bits_to_f32) } +} diff --git a/crates/aprender-serve/src/gguf/inference/float16_dot_tests.rs b/crates/aprender-serve/src/gguf/inference/float16_dot_tests.rs new file mode 100644 index 0000000000..924a87ae56 --- /dev/null +++ b/crates/aprender-serve/src/gguf/inference/float16_dot_tests.rs @@ -0,0 +1,177 @@ +//! #3076: the F16/BF16 row dot that replaced `float16_matmul`'s per-element `fn`-pointer loop. +//! +//! Every length below is checked against an f64 reference, which reaches every branch of the +//! kernels: the 32-wide unrolled loop, the 8-wide loop, the scalar tail, and the portable +//! path's 64-element chunks and remainders. `float16_matmul` itself is checked against a +//! transcription of the loop it replaced, including a truncated weight buffer. + +use super::matmul::{float16_matmul, float16_row_dot, float16_row_dot_portable, Float16Kind}; + +/// Deterministic values in roughly [-2, 2]: a 64-bit LCG, so a failure reproduces exactly. +fn values(n: usize, seed: u64) -> Vec { + let mut s = seed.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1); + (0..n) + .map(|_| { + s = s + .wrapping_mul(6_364_136_223_846_793_005) + .wrapping_add(1_442_695_040_888_963_407); + ((s >> 40) as f32 / (1u64 << 24) as f32) * 4.0 - 2.0 + }) + .collect() +} + +fn encode(kind: Float16Kind, w: &[f32]) -> Vec { + w.iter() + .flat_map(|&v| match kind { + Float16Kind::F16 => half::f16::from_f32(v).to_bits().to_le_bytes(), + Float16Kind::Bf16 => half::bf16::from_f32(v).to_bits().to_le_bytes(), + }) + .collect() +} + +fn decode(kind: Float16Kind, b: [u8; 2]) -> f32 { + let bits = u16::from_le_bytes(b); + match kind { + Float16Kind::F16 => half::f16::from_bits(bits).to_f32(), + Float16Kind::Bf16 => half::bf16::from_bits(bits).to_f32(), + } +} + +/// (exact dot in f64 of the decoded weights, sum of |w * x| for the error bound). +fn reference(kind: Float16Kind, row: &[u8], x: &[f32]) -> (f64, f64) { + row.chunks_exact(2) + .zip(x) + .fold((0.0, 0.0), |(dot, mag), (b, &xv)| { + let p = f64::from(decode(kind, [b[0], b[1]])) * f64::from(xv); + (dot + p, mag + p.abs()) + }) +} + +const LENGTHS: [usize; 16] = [ + 0, 1, 7, 8, 9, 31, 32, 33, 63, 64, 65, 100, 129, 896, 4864, 4867, +]; + +fn assert_matches_reference(kind: Float16Kind, dot: fn(Float16Kind, &[u8], &[f32]) -> f32) { + for (k, &n) in LENGTHS.iter().enumerate() { + let row = encode(kind, &values(n, 2 * k as u64 + 1)); + let x = values(n, 2 * k as u64 + 2); + let (exact, mag) = reference(kind, &row, &x); + let got = f64::from(dot(kind, &row, &x)); + // f32 accumulation over n terms: error well inside 1e-5 of the magnitude sum here. + assert!( + (got - exact).abs() <= 1e-5 * mag + 1e-6, + "{kind:?} n={n}: got {got}, exact {exact}, |w*x| sum {mag}" + ); + } +} + +#[test] +fn test_3076_f16_row_dot_matches_the_f64_reference_at_every_length() { + assert_matches_reference(Float16Kind::F16, float16_row_dot); +} + +#[test] +fn test_3076_bf16_row_dot_matches_the_f64_reference_at_every_length() { + assert_matches_reference(Float16Kind::Bf16, float16_row_dot); +} + +/// The fallback must be right on its own: on an AVX2 host `float16_row_dot` never reaches it. +#[test] +fn test_3076_portable_path_matches_the_f64_reference_at_every_length() { + assert_matches_reference(Float16Kind::F16, float16_row_dot_portable); + assert_matches_reference(Float16Kind::Bf16, float16_row_dot_portable); +} + +/// Non-finite and subnormal weights reach the sum the way the scalar loop let them. +#[test] +fn test_3076_specials_propagate_through_every_lane_position() { + for kind in [Float16Kind::F16, Float16Kind::Bf16] { + let (nan, inf) = match kind { + Float16Kind::F16 => (0x7E00u16, 0x7C00u16), + Float16Kind::Bf16 => (0x7FC0u16, 0x7F80u16), + }; + // Position 3 sits in the 32-wide loop, 37 in the 8-wide loop, 42 in the tail. + for pos in [3usize, 37, 42] { + let mut row = encode(kind, &vec![0.5; 43]); + row[2 * pos..2 * pos + 2].copy_from_slice(&nan.to_le_bytes()); + assert!( + float16_row_dot(kind, &row, &vec![1.0; 43]).is_nan(), + "{kind:?} NaN at {pos}" + ); + row[2 * pos..2 * pos + 2].copy_from_slice(&inf.to_le_bytes()); + let got = float16_row_dot(kind, &row, &vec![1.0; 43]); + assert!( + got.is_infinite() && got > 0.0, + "{kind:?} +inf at {pos}: {got}" + ); + } + } + // The smallest F16 subnormal is 2^-24, not zero: 40 of them times 2^20 is 40 / 16 = 2.5. + let row: Vec = std::iter::repeat_n(0x0001u16.to_le_bytes(), 40) + .flatten() + .collect(); + let got = float16_row_dot(Float16Kind::F16, &row, &vec![(1u32 << 20) as f32; 40]); + assert!((got - 2.5).abs() < 1e-6, "F16 subnormals: {got}"); +} + +/// The element count is `min(row.len() / 2, x.len())`: an odd trailing byte is not an element. +#[test] +fn test_3076_element_count_is_whole_elements_of_the_shorter_operand() { + let kind = Float16Kind::F16; + let row = encode(kind, &[1.0; 20]); + let mut odd = row.clone(); + odd.push(0x3C); // half an element: must be ignored, not read as a weight + assert_eq!(float16_row_dot(kind, &odd, &[2.0; 20]), 40.0); + assert_eq!(float16_row_dot(kind, &row, &[2.0; 9]), 18.0); + assert_eq!(float16_row_dot(kind, &row[..10], &[2.0; 20]), 10.0); +} + +/// `float16_matmul` against the loop it replaced, transcribed: one element at a time, an +/// element counted only when both of its bytes are inside `data`. +fn pre_3076_float16_matmul( + kind: Float16Kind, + input: &[f32], + data: &[u8], + in_dim: usize, + out_dim: usize, + seq_len: usize, +) -> Vec { + let mut out = Vec::with_capacity(seq_len * out_dim); + for s in 0..seq_len { + let x = &input[s * in_dim..(s + 1) * in_dim]; + for row in 0..out_dim { + let mut sum = 0.0f64; + for (col, &xv) in x.iter().enumerate() { + let offset = row * in_dim * 2 + col * 2; + if offset + 1 < data.len() { + sum += + f64::from(decode(kind, [data[offset], data[offset + 1]])) * f64::from(xv); + } + } + out.push(sum); + } + } + out +} + +#[test] +fn test_3076_float16_matmul_matches_the_pre_3076_loop_including_a_truncated_buffer() { + let (in_dim, out_dim, seq_len) = (37, 5, 3); + let input = values(in_dim * seq_len, 7); + for kind in [Float16Kind::F16, Float16Kind::Bf16] { + let full = encode(kind, &values(in_dim * out_dim, 8)); + // Full buffer, then one cut mid-row that also leaves an odd trailing byte. + for data in [&full[..], &full[..full.len() - 2 * in_dim - 21]] { + let got = float16_matmul(&input, data, in_dim, out_dim, seq_len, kind); + let want = pre_3076_float16_matmul(kind, &input, data, in_dim, out_dim, seq_len); + assert_eq!(got.len(), want.len()); + for (i, (&g, &w)) in got.iter().zip(&want).enumerate() { + assert!( + (f64::from(g) - w).abs() <= 1e-5 * w.abs().max(1.0), + "{kind:?} len={} out[{i}]: got {g}, want {w}", + data.len() + ); + } + } + } +} diff --git a/crates/aprender-serve/src/gguf/inference/matmul_fused.rs b/crates/aprender-serve/src/gguf/inference/matmul_fused.rs index c810b80719..056d623107 100644 --- a/crates/aprender-serve/src/gguf/inference/matmul_fused.rs +++ b/crates/aprender-serve/src/gguf/inference/matmul_fused.rs @@ -1,15 +1,22 @@ +include!("float16_dot.rs"); + /// CPU matmul for 2-byte-per-element float formats (BF16, F16) /// Shared by BF16 and F16 paths — same structure, different decode. -fn float16_matmul( +/// +/// #3076: each row is one `float16_row_dot` over the row's bytes. The per-element loop it +/// replaces called the decode through a `fn` pointer. A row that runs past the end of `data` +/// is dotted over the whole elements that exist, as that loop's per-element bounds check did. +pub(super) fn float16_matmul( input: &[f32], data: &[u8], in_dim: usize, out_dim: usize, seq_len: usize, - decode: fn(u16) -> f32, + kind: Float16Kind, ) -> Vec { use rayon::prelude::*; + let row_bytes = in_dim * 2; let mut all_output = Vec::with_capacity(seq_len * out_dim); for s in 0..seq_len { let x = &input[s * in_dim..(s + 1) * in_dim]; @@ -17,16 +24,9 @@ fn float16_matmul( let row_output: Vec = (0..out_dim) .into_par_iter() .map(|row| { - let row_byte_start = row * in_dim * 2; - let mut sum = 0.0f32; - for col in 0..in_dim { - let offset = row_byte_start + col * 2; - if offset + 1 < data.len() { - let bits = u16::from_le_bytes([data[offset], data[offset + 1]]); - sum += decode(bits) * x[col]; - } - } - sum + let start = (row * row_bytes).min(data.len()); + let end = (start + row_bytes).min(data.len()); + float16_row_dot(kind, &data[start..end], x) }) .collect(); @@ -169,12 +169,22 @@ impl OwnedQuantizedModel { let data = &weight.data; match weight.qtype { GGUF_TYPE_F32 => Ok(self.fused_matmul_f32(input, data, in_dim, out_dim, seq_len)), - GGUF_TYPE_BF16 => Ok(float16_matmul(input, data, in_dim, out_dim, seq_len, |b| { - f32::from_bits((b as u32) << 16) - })), - GGUF_TYPE_F16 => Ok(float16_matmul(input, data, in_dim, out_dim, seq_len, |b| { - half::f16::from_bits(b).to_f32() - })), + GGUF_TYPE_BF16 => Ok(float16_matmul( + input, + data, + in_dim, + out_dim, + seq_len, + Float16Kind::Bf16, + )), + GGUF_TYPE_F16 => Ok(float16_matmul( + input, + data, + in_dim, + out_dim, + seq_len, + Float16Kind::F16, + )), GGUF_TYPE_Q4_0 | GGUF_TYPE_Q8_0 => { self.fused_matmul_q4_q8(input, weight, in_dim, out_dim, seq_len) }, diff --git a/crates/aprender-serve/src/gguf/inference/mod.rs b/crates/aprender-serve/src/gguf/inference/mod.rs index e9541c9bf6..a57f376a60 100644 --- a/crates/aprender-serve/src/gguf/inference/mod.rs +++ b/crates/aprender-serve/src/gguf/inference/mod.rs @@ -23,6 +23,8 @@ mod attention_flash_tests; #[cfg(test)] mod attention_gqa_tests; #[cfg(test)] +mod float16_dot_tests; +#[cfg(test)] mod generate_quantized_topk_tests; #[cfg(test)] mod generate_quantized_topp_tests; diff --git a/docs/roadmaps/entries/PMAT-3076.yaml b/docs/roadmaps/entries/PMAT-3076.yaml new file mode 100644 index 0000000000..96ef530a8f --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3076.yaml @@ -0,0 +1,17 @@ +- id: PMAT-3076 + github_issue: 3076 + item_type: task + title: 'F16/BF16 GGUF CPU matmul has no SIMD path — unquantized models appear to hang' + status: planned + priority: high + assigned_to: null + created: 2026-09-21T07:52:58Z + updated: 2026-09-21T07:52:58Z + spec: null + acceptance_criteria: [] + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'ACCEPTANCE (hand-entered; #3076 has no done_when section, so this quotes its last plan comment verbatim, minted on the PR branch so the AD-04 quorum can run). "(1) the GGUF CPU path gets a SIMD f16/bf16 matvec (dequant-to-f32 tiles through the existing fused dispatcher) with a greedy-parity gate against llama.cpp on an unquantized ≤1B GGUF, so an F16/BF16 model decodes at a measured tok/s instead of appearing to hang; (2) until that lands the loader prints a one-line notice that the F16/BF16 path is scalar and slow rather than silently crawling." Since (1) lands here, (2) is moot. DEVIATION from the issue body: it suggests an integration test "with a wall-clock assertion"; the project bans wall-clock assertions in required checks, so the regression guard is behavioural (kernel parity against an f64 reference at lengths hitting every loop and tail, and float16_matmul against a transcription of the pre-fix loop including a truncated buffer), and tok/s is measured and reported rather than asserted. Not in scope: prefill running one position at a time at decode rate (#2787 / #2801).' diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index 84e48c530f..c08aa41d3b 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -17150,6 +17150,23 @@ roadmap: - orch:fable - ci notes: null +- id: PMAT-3076 + github_issue: 3076 + item_type: task + title: 'F16/BF16 GGUF CPU matmul has no SIMD path — unquantized models appear to hang' + status: planned + priority: high + assigned_to: null + created: 2026-09-21T07:52:58Z + updated: 2026-09-21T07:52:58Z + spec: null + acceptance_criteria: [] + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'ACCEPTANCE (hand-entered; #3076 has no done_when section, so this quotes its last plan comment verbatim, minted on the PR branch so the AD-04 quorum can run). "(1) the GGUF CPU path gets a SIMD f16/bf16 matvec (dequant-to-f32 tiles through the existing fused dispatcher) with a greedy-parity gate against llama.cpp on an unquantized ≤1B GGUF, so an F16/BF16 model decodes at a measured tok/s instead of appearing to hang; (2) until that lands the loader prints a one-line notice that the F16/BF16 path is scalar and slow rather than silently crawling." Since (1) lands here, (2) is moot. DEVIATION from the issue body: it suggests an integration test "with a wall-clock assertion"; the project bans wall-clock assertions in required checks, so the regression guard is behavioural (kernel parity against an f64 reference at lengths hitting every loop and tail, and float16_matmul against a transcription of the pre-fix loop including a truncated buffer), and tok/s is measured and reported rather than asserted. Not in scope: prefill running one position at a time at decode rate (#2787 / #2801).' - id: PMAT-3118 github_issue: 3118 item_type: task From eebd642c85a3cbe974d8177a58c72b6c0d0e1801 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 10:26:01 +0200 Subject: [PATCH 03/28] =?UTF-8?q?feat(release):=20batching=20is=20the=20de?= =?UTF-8?q?fault=20merge=20path=20=E2=80=94=20scripts/batch=5Ffold.sh=20+?= =?UTF-8?q?=20APR-RELEASE-001=20=C2=A713=20(#3673)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Refs #3673 #3669 #3650 paiml-implement#317 Operator 2026-09-21: "we are arbitrarily going slow because each PR adds time, but in most cases 80% of PR can be batched." Code handed off by the cop (aprender-04) as an uncommitted draft; rewritten here. scripts/batch_fold.sh -- fold receipted PR branches into release/-batch, --no-ff, in order. It resolves ONLY the generated set: roadmap.yaml, census.json, contracts.nt, shapes.ttl (pv extract writes it), and the README CONTRACT_COUNT blocks. Every other conflict aborts that branch: SKIP + paths. - README is merged three-way with the counts normalised out (git merge-file), so a prose conflict is a SKIP. The draft took README wholesale with --theirs, which silently dropped the batch side's prose. - A CLEAN fold that touches the generated set marks it stale: two PRs that each add a contract merge textually and the census is then off. Stale without --regen ends `REGEN REQUIRED`, exit 3, never silent. - --regen regenerates once with the tree-built pv (pv_bin.sh), asserts the three fixed points (make roadmap-aggregate-check, pv extract --check, readme_sync --check), refuses writes outside the set, commits. - refuses a dirty batch worktree; exit 0/1/2/3 documented. scripts/check_batch_fold.sh -- the case table, 21 rows over throwaway repos with stub pv/make/readme_sync, plus --self-test with 4 mutants, each RED: README wholly generated (the draft's rule), every conflict taken, a clean fold never stale, a failed fixed point ignored. It is a check_*.sh on purpose: the draft's own `--self-test)` case made check_guards_are_wired.sh RED ("NEW: batch_fold.sh", measured), and guard_tree.sh --no-cargo dispatches this file with no workflow edit. The draft, judged by this table: 13 of 21 rows RED (3 of them only on output wording). APR-RELEASE-001: - §13 (new): roles (author stops at the receipt; never arms, never merges main to stay current), assembly, the generated set, left-out rules, what needs a new quorum, push/disarm, body/arm/close, a red batch, what it costs (a SQUASH queue lands a batch as ONE commit: bisect on main stops at the batch, so the batch branch is kept until the next tag), falsifiers FX-B1..3. - hard rule 9 AMENDED. It read "No hand-squash of batches ... each PR lands as its own commit ... never hand-bisected", which the new default contradicts. The old text is kept verbatim as superseded, with the reason: the queue it described (group size 8) was not the queue we had (SQUASH, maximumEntriesToMerge=1, measured on #3658). - §8 stop "a hand-squashed batch" -> "a batch assembled by hand instead of scripts/batch_fold.sh (§13)". - revision note 2026-09-21. Gates: check_batch_fold 21/21 + self-test 4/4; bashrs 0 errors on both files; bashrs-gate PASS; guards-are-wired PASS; guard_tree dispatches it; guard_tree_test 23/0; pipe-into-grep-q at ceiling; hardcoded-paths +0; apr pinned; no-timing-in-required; check_ci_unwedge and check_ont_ratchet (read the spec) green; roadmap aggregate + sorted/unique/additive; fmt, deny, aprender-contracts --lib 1666. Co-Authored-By: Claude Opus 5 (1M context) --- docs/roadmaps/entries/PMAT-3673.yaml | 17 ++ docs/roadmaps/roadmap.yaml | 17 ++ .../APR-RELEASE-001-train-and-build-kaizen.md | 121 +++++++++++- scripts/batch_fold.sh | 183 ++++++++++++++++++ scripts/check_batch_fold.sh | 169 ++++++++++++++++ 5 files changed, 503 insertions(+), 4 deletions(-) create mode 100644 docs/roadmaps/entries/PMAT-3673.yaml create mode 100755 scripts/batch_fold.sh create mode 100644 scripts/check_batch_fold.sh diff --git a/docs/roadmaps/entries/PMAT-3673.yaml b/docs/roadmaps/entries/PMAT-3673.yaml new file mode 100644 index 0000000000..4e0e102b9e --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3673.yaml @@ -0,0 +1,17 @@ +- id: PMAT-3673 + github_issue: 3673 + item_type: task + title: 'Batching is the default merge path: scripts/batch_fold.sh + APR-RELEASE-001 §13 (operator 2026-09-21)' + status: planned + priority: critical + assigned_to: null + created: 2026-09-21T08:23:21Z + updated: 2026-09-21T08:23:21Z + spec: docs/specifications/APR-RELEASE-001-train-and-build-kaizen.md + acceptance_criteria: [] + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'ACCEPTANCE (quoted from issue #3673 "What lands"): 1. scripts/batch_fold.sh — fold receipted PR branches into one release/-batch branch, git merge --no-ff in the given order: a conflict ONLY in the generated set {docs/roadmaps/roadmap.yaml, contracts/census.json, contracts/contracts.nt, contracts/shapes.ttl, the CONTRACT_COUNT blocks of README.md} is taken and marked for regeneration; a conflict ANYWHERE else — including README prose outside a count block — aborts that branch''s merge and reports SKIP with the paths, never resolved by picking a side; --regen regenerates the generated set ONCE with the tree-built pv (scripts/pv_bin.sh), asserts the fixed points (aggregate --check, pv extract --check, readme_sync.sh --check), and commits the regeneration; exit 0 all folded; 1 any SKIP; 2 usage/git/regeneration failure; a folded-but-unregenerated generated set is never silent; --self-test case table, wired into CI. 2. APR-RELEASE-001 §13 — the batching procedure as the default path (cop prompt §7): assembly, generated-set regeneration, left-out rules, what needs a new quorum (only content new to the batch), constituent disarm/close, a red batch, and the review-lane VOID rule while paiml-implement#317 is open. AUTHOR NOTES (not criteria): the case table lives in scripts/check_batch_fold.sh (guard-tree runs it) because a --self-test case inside batch_fold.sh made check_guards_are_wired.sh RED; §13 required amending hard rule 9 ("No hand-squash of batches") and the matching §8 stop, which the new default contradicts.' diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index 84e48c530f..174da9efbf 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -19007,3 +19007,20 @@ roadmap: labels: - kind:code notes: 'ACCEPTANCE (hand-entered from issue #3629 done_when; minted on the PR branch so the AD-04 quorum can run -- pmat work status refuses a missing item). Done when: scripts/fleet_utilization.sh --selftest has no pipe into grep -q; a mutation that restores the pipe AND forces EPIPE (out padded to >64 KiB so the write cannot complete before grep exits) turns the self-test RED deterministically — the positive control for the race, in the PR body the 55-site count is in the PR body from the command that produced it, and a guard refuses a new site (check_guards_are_wired.sh names it; ratchet baseline committed) main''s guard-tree green on the merge commit without rerun ' +- id: PMAT-3673 + github_issue: 3673 + item_type: task + title: 'Batching is the default merge path: scripts/batch_fold.sh + APR-RELEASE-001 §13 (operator 2026-09-21)' + status: planned + priority: critical + assigned_to: null + created: 2026-09-21T08:23:21Z + updated: 2026-09-21T08:23:21Z + spec: docs/specifications/APR-RELEASE-001-train-and-build-kaizen.md + acceptance_criteria: [] + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'ACCEPTANCE (quoted from issue #3673 "What lands"): 1. scripts/batch_fold.sh — fold receipted PR branches into one release/-batch branch, git merge --no-ff in the given order: a conflict ONLY in the generated set {docs/roadmaps/roadmap.yaml, contracts/census.json, contracts/contracts.nt, contracts/shapes.ttl, the CONTRACT_COUNT blocks of README.md} is taken and marked for regeneration; a conflict ANYWHERE else — including README prose outside a count block — aborts that branch''s merge and reports SKIP with the paths, never resolved by picking a side; --regen regenerates the generated set ONCE with the tree-built pv (scripts/pv_bin.sh), asserts the fixed points (aggregate --check, pv extract --check, readme_sync.sh --check), and commits the regeneration; exit 0 all folded; 1 any SKIP; 2 usage/git/regeneration failure; a folded-but-unregenerated generated set is never silent; --self-test case table, wired into CI. 2. APR-RELEASE-001 §13 — the batching procedure as the default path (cop prompt §7): assembly, generated-set regeneration, left-out rules, what needs a new quorum (only content new to the batch), constituent disarm/close, a red batch, and the review-lane VOID rule while paiml-implement#317 is open. AUTHOR NOTES (not criteria): the case table lives in scripts/check_batch_fold.sh (guard-tree runs it) because a --self-test case inside batch_fold.sh made check_guards_are_wired.sh RED; §13 required amending hard rule 9 ("No hand-squash of batches") and the matching §8 stop, which the new default contradicts.' diff --git a/docs/specifications/APR-RELEASE-001-train-and-build-kaizen.md b/docs/specifications/APR-RELEASE-001-train-and-build-kaizen.md index b96f30d350..814b114df8 100644 --- a/docs/specifications/APR-RELEASE-001-train-and-build-kaizen.md +++ b/docs/specifications/APR-RELEASE-001-train-and-build-kaizen.md @@ -13,6 +13,10 @@ tag inheritance from the release commit, B2 split by capability (cpu shards + gp derived from `cargo metadata` (TIERS deleted), unattended cascade under standing authorization with receipts, v0.68.0 = GitHub-only / 0.68.1 = crates.io, P0·Fan-out as the 0.69 train's first PR. Earlier revisions kept where still true. *(Merged 2026-09-18 onto the `main` lineage — #3268 §4.1–4.3, §5.1, §6.1–6.4, §10–§12 and #3455 `check_milestone_cut.sh` — which the 09-15/16/17 working copy had been cut before; where the two disagreed the later operator ruling wins and the older text is kept as history.)* +**Revision 2026-09-21** — §13: integration batches are the DEFAULT merge path (operator: "we are +arbitrarily going slow because each PR adds time, but in most cases 80% of PR can be batched"), +assembled by `scripts/batch_fold.sh`. Hard rule 9 and the §8 "hand-squashed batch" stop are amended +to match (§13.9 says what that costs). **Revision 2026-09-16** — adds §1.6 critical path for 0.68, milestone hygiene (§6), the CI-wedge mechanism and its two sanctioned rules (P0·Unwedge), clean-room-on-tag (T-3), reaper budget as a manifest fact, and the capacity-source rule (no runners API, ever). @@ -178,9 +182,15 @@ that decides a gate. 2026-09-13 stands: "T4 is never mine … releases are automated by release train … for ALL releases".)* 8. `pmat work add` from the driver session only. Stop the line on RED — no reruns-as-passes, no `--skip`, no waivers. -9. **No hand-squash of batches.** Batch verification is the merge queue's job (merge_group, - group size 8); each PR lands as its own commit. A red batch fans out one job per PR, never - hand-bisected. +9. **A batch is an integration PR assembled by `scripts/batch_fold.sh`, never by hand (§13, + operator 2026-09-21).** Receipted PRs fold into `release/-batch`; only the generated set is + ever resolved by the tool, every other conflict leaves that PR out, and a red batch is traced to + its constituent with `git log --merges` on the batch branch (§13.7). + *(Superseded 2026-09-21. Until then this rule read: "No hand-squash of batches. Batch + verification is the merge queue's job (merge_group, group size 8); each PR lands as its own + commit. A red batch fans out one job per PR, never hand-bisected." The queue it described was + not the queue we had: mergeMethod SQUASH, maximumEntriesToMerge=1, measured on #3658 — so every + PR paid its own CI run, queue slot and roadmap re-merge.)* 10. **WIP is enforced at the producer.** paiml-implement's pre-push hook refuses to mint a PR when open aprender PRs ≥ 13 (basis: measured drain 8.8/day × 1.5 day, 2026-09-15; ratchet after 20 records) and emits an issue instead. Sweeps emit issues, not PRs. @@ -654,7 +664,7 @@ next: train eligible at - Any queue entry ejected twice on the same cause → stop, five-whys, no third re-queue. - Any measurement whose tree line shows `behind≠0`, or taken outside a per-session worktree → discard and re-measure from `origin/main`. -- Any step would need an invented threshold, a hand-squashed batch, a hand-cancelled run with started +- Any step would need an invented threshold, a batch assembled by hand instead of `scripts/batch_fold.sh` (§13), a hand-cancelled run with started jobs, a label edit to dodge a runner, an env bypass of a prejob gate, `--allow-dirty`, or a host CONFIG change made over SSH instead of through forjar (§3.5 — SSH for measurement and for unclogging is expected) → stop. - The upstream ontology spec, at the version this spec cites, is not reachable at a committed sha @@ -987,3 +997,106 @@ them are report lines. | FX-3 | §12.1.3 | every counter on a §7 line has a named ratchet target and moves one way | move one backward by hand → RED | | FX-4 | §12.2.C | `beats:` says `measured-on published` only when the dogfood ran on a `cargo install aprender` binary (G13: no embedded SHA, `DOGFOOD_ALLOW_UNPINNED=1`) | report `published` from a dev build → RED | | FX-5 | §12.2.C | a beat's status in `docs/BEATS.md` matches its contract | change the status without the contract → `readme_contract` RED | + +## §13 Integration batches — the default merge path (operator 2026-09-21) + +> "we are arbitrarily going slow because each PR adds time, but in most cases 80% of PR can be batched." + +The per-PR queue is the slow path. Every PR paid one CI run (~32 min median wall, ~68 runner-min), +one queue slot (~26 min per merge, measured), and, while `roadmap.yaml` is committed (#3650), about +one re-merge of the generated files per landing. The default is now ONE integration batch PR that +carries every receipted PR that is ready. First run: #3669, 18 PRs plus an andon fix. + +### §13.1 Who does what + +- **An author** stops at the receipt: branch, roadmap fragment, AD-04 quorum, + `docs/audits/quorum-.json` committed on the branch, lanes path to the batch driver. **Never + arms, and never merges main just to stay current.** +- **The batch driver** (the traffic cop) folds, regenerates, pushes and arms the batch. +- **A folded PR** is never pushed to again. Fixes go to the batch branch, or to the author's + branch followed by a re-fold. + +### §13.2 Assembly + +``` +git worktree add -b release/-batch origin/main +cd && bash scripts/batch_fold.sh --regen ... # priority order, cut-blockers first +``` + +Run it under **bash**. zsh does not word-split `$var`, which breaks a hand-rolled loop over branch +names. `batch_fold.sh` merges each branch `--no-ff`. It resolves ONLY the **generated set**: +`docs/roadmaps/roadmap.yaml`, `contracts/census.json`, `contracts/contracts.nt`, +`contracts/shapes.ttl`, and the `CONTRACT_COUNT` blocks of `README.md`. README is merged three-way +with the counts normalised out, so a README conflict outside a count block is a real conflict. Every +other conflict aborts that branch's merge (`SKIP : conflict in `). A clean fold that +touches the generated set still marks it stale, because two PRs that each add a contract merge +textually and the census is then wrong. + +`--regen` regenerates the set ONCE with the pv built from the batch tree (`scripts/pv_bin.sh`): +roadmap aggregate, `pv census`, `pv extract`, `readme_sync.sh --write`. It then asserts the fixed +points (`make roadmap-aggregate-check`, `pv extract --check`, `readme_sync.sh --check`) and commits +the regeneration. Exit codes: + +| Exit | Meaning | +|---|---| +| 0 | all folded, set consistent | +| 1 | a SKIP | +| 2 | a usage, git or regeneration failure, including a failed fixed point | +| 3 | folded, but the generated set is stale (`REGEN REQUIRED`) | + +### §13.3 Left out, and said so + +A PR is left out of the batch for exactly two reasons, and the batch body names which: + +- a real conflict (`SKIP`, e.g. #3606 against #3634); +- a tree-wide gate it fails on its own (e.g. `pv census` refused #3642's YAML). + +Nothing else keeps a receipted PR out. + +### §13.4 What needs a new quorum + +Each constituent's receipt stands, because its diff is unchanged inside the batch. Only content +NEW to the batch needs a quorum: fixes made in the batch branch, run with `--base `. The +regeneration needs none; its fixed points are asserted by `--regen`. While paiml-implement#317 is +open, a review lane whose agy conversation store (`~/.gemini/antigravity-cli/conversations/.db`) +never references the judged worktree is **VOID**. Stale shared scratch clones have produced +`measured` PASSes from another branch with rewritten baselines (#3664). A NON-author's measured +verification in the judged tree, posted on the PR with commands and outputs, fills the seat. + +### §13.5 Push, then disarm + +1. Push the batch first so CI starts. +2. Run `cargo check --workspace --all-targets` locally in parallel. +3. Comment "folded into #N" on each constituent and disarm it. +4. Cancel in-flight PR CI outside the batch; it goes stale when the batch moves main. +5. Dequeue anything the batch supersedes. + +### §13.6 The batch body, arming, closing + +The body has a table of constituents: PR, receipt path, head sha. It carries every constituent's +closing keywords, plus a `keep-open:` line naming the constituent PR numbers, or +`check_pr_closes_issue.sh` fails. Arm the batch. After it merges, close each constituent as "landed +in #N". + +### §13.7 A red batch + +1. Run `git log --merges` on the batch branch. +2. Find the constituent whose merge introduced the red, and reproduce it on that merge. +3. Either fix it in the batch (new content, so §13.4 applies) or drop it: rebuild the batch without + it, then `--regen`. +4. Push once. + +### §13.8 What this costs, stated rather than found + +The queue is SQUASH, so a batch lands on main as ONE commit. `git bisect` on main therefore stops at +the batch, not the constituent. The constituent history survives on the batch branch (the `--no-ff` +merges) and in the body's head-sha table, so the batch branch is kept until the train's next tag. +Deleting it early turns "which PR broke main" into a rebuild. + +### §13.9 Falsifiers + +| # | Assertion | Mutation | +|---|---|---| +| FX-B1 | `batch_fold.sh` resolves only the generated set: a code conflict and a README prose conflict are SKIP, a count-block conflict folds (`check_batch_fold.sh`, run by guard-tree on every PR) | treat README as wholly generated, or take every conflict → `check_batch_fold.sh --self-test` RED | +| FX-B2 | a stale generated set is never silent: a clean fold touching the census exits 3 | never mark a clean fold stale → RED | +| FX-B3 | a failed fixed point stops the batch (exit 2, the check named) | ignore `pv extract --check` → RED | diff --git a/scripts/batch_fold.sh b/scripts/batch_fold.sh new file mode 100755 index 0000000000..c3b9994b97 --- /dev/null +++ b/scripts/batch_fold.sh @@ -0,0 +1,183 @@ +#!/usr/bin/env bash +# batch_fold.sh -- fold receipted PR branches into ONE integration batch branch +# (APR-RELEASE-001 §13; operator 2026-09-21: "we are arbitrarily going slow +# because each PR adds time, but in most cases 80% of PR can be batched"). +# +# Run INSIDE the batch worktree (release/-batch, cut from origin/main). +# Each branch is merged --no-ff, in the order given (cut-blockers first). +# +# THE GENERATED SET is the only thing this script ever resolves by itself: +# docs/roadmaps/roadmap.yaml aggregate of docs/roadmaps/entries/ +# contracts/census.json pv census +# contracts/contracts.nt pv extract +# contracts/shapes.ttl pv extract (every `shape:` block) +# README.md CONTRACT_COUNT scripts/readme_sync.sh -- the COUNT BLOCKS only +# A conflict in those is taken and marked for regeneration. README.md is merged +# three-way with the counts normalised out, so a README conflict OUTSIDE a count +# block is a real conflict, not a generated one. Any real conflict aborts that +# branch's merge and reports SKIP with the paths -- never resolved by picking a +# side. The branch is left out of the batch and named. +# +# A merge that folds CLEANLY can still leave the generated set wrong: two PRs +# that each add a contract merge textually and the census count is then off by +# one. So every fold that touches a generated file marks the set STALE, and a +# stale set is never silent: --regen regenerates it ONCE, with the pv built from +# THIS tree (scripts/pv_bin.sh), asserts the three fixed points, and commits the +# regeneration; without --regen the run ends `REGEN REQUIRED` and exits 3. +# +# bash scripts/batch_fold.sh [--regen] BRANCH... +# bash scripts/batch_fold.sh --regen regenerate only +# +# Output, one line per branch: +# folded generated=[] merged; were taken for regeneration +# SKIP : conflict in merge aborted, branch left out +# ERROR : git could not merge it at all +# Exit: 0 every branch folded and the generated set is consistent; 1 any SKIP; +# 2 a usage, git or regeneration failure (including a failed fixed-point +# check); 3 folded, but the generated set is stale (run --regen). +# The case table is scripts/check_batch_fold.sh (guard-tree runs it). +# +# Test seam: BATCH_FOLD_PV= skips building pv (scripts/pv_bin.sh) and uses it. +set -uo pipefail + +GENERATED_WHOLE="docs/roadmaps/roadmap.yaml contracts/census.json contracts/contracts.nt contracts/shapes.ttl" +COUNT_RE='()[0-9]+()' + +die() { printf 'ERROR %s\n' "$*" >&2; exit 2; } +rmtree() { case "${1:-}" in ''|/) return 0 ;; *) [ -d "$1" ] && rm -rf -- "$1" ;; esac; return 0; } + +is_generated_whole() { # PATH -> 0 iff the whole file is derived + local g + for g in $GENERATED_WHOLE; do [ "$1" = "$g" ] && return 0; done + return 1 +} + +touches_generated() { # REV_A REV_B -> prints the generated paths that differ + local p + while IFS= read -r p; do + [ -n "$p" ] || continue + if is_generated_whole "$p" || [ "$p" = README.md ]; then printf '%s\n' "$p"; fi + done < <(git diff --no-renames --name-only "$1" "$2") +} + +# readme_counts_only -> 0 iff README.md's conflict is ONLY in CONTRACT_COUNT blocks. +# On 0 the file is written merged (counts left for readme_sync.sh) and staged. +readme_counts_only() { + local t rc=0 + t=$(mktemp -d "${TMPDIR:-/tmp}/batch-fold-readme.XXXXXX") || return 1 + if ! { git show :1:README.md > "$t/base" && git show :2:README.md > "$t/ours" && git show :3:README.md > "$t/theirs"; } 2>/dev/null; then + rmtree "$t"; return 1 # an add/add or delete conflict is not a count conflict + fi + local n + n=$(grep -m1 -oE "$COUNT_RE" "$t/ours" | sed -E 's/[^0-9]//g') + sed -E -i "s/$COUNT_RE/\\1N\\2/g" "$t/base" "$t/ours" "$t/theirs" + git merge-file -p "$t/ours" "$t/base" "$t/theirs" > "$t/merged" 2>/dev/null || rc=$? + if [ "$rc" = 0 ]; then + # put OUR count back so the file stays well-formed; --regen rewrites every block + sed -E -i "s/()N()/\\1${n:-0}\\2/g" "$t/merged" + cp -- "$t/merged" README.md && git add -- README.md || rc=1 + fi + rmtree "$t" + return "$rc" +} + +fold_one() { # BRANCH -> one verdict line; 0 folded, 1 skipped, 2 git failure + local br=$1 f real="" gen="" conf before + before=$(git rev-parse HEAD) || return 2 + if git merge --no-ff --no-edit "$br" >/dev/null 2>&1; then + gen=$(touches_generated "$before" HEAD | tr '\n' ' ') + [ -z "$gen" ] || STALE=1 + printf 'folded %s generated=[%s]\n' "$br" "${gen% }"; return 0 + fi + if ! git rev-parse -q --verify MERGE_HEAD >/dev/null 2>&1; then + printf 'ERROR %s: git merge failed without a conflict state\n' "$br"; return 2 + fi + conf=$(git diff --name-only --diff-filter=U) + while IFS= read -r f; do + [ -n "$f" ] || continue + if is_generated_whole "$f"; then + gen="$gen $f" + elif [ "$f" = README.md ] && readme_counts_only; then + gen="$gen README.md" + else + real="$real $f" + fi + done <<< "$conf" + if [ -n "$real" ]; then + git merge --abort >/dev/null 2>&1 + printf 'SKIP %s: conflict in%s\n' "$br" "$real"; return 1 + fi + for f in $gen; do + [ "$f" = README.md ] && continue # already merged and staged above + { git checkout --theirs -- "$f" && git add -- "$f"; } >/dev/null 2>&1 \ + || { git merge --abort >/dev/null 2>&1; printf 'ERROR %s: cannot take %s\n' "$br" "$f"; return 2; } + done + git -c commit.gpgsign=false commit -q --no-edit >/dev/null 2>&1 \ + || { git merge --abort >/dev/null 2>&1; printf 'ERROR %s: commit failed\n' "$br"; return 2; } + STALE=1 + printf 'folded %s generated=[%s]\n' "$br" "${gen# }"; return 0 +} + +regen() { # regenerate the generated set once, assert the fixed points, commit it + local PV extra + if [ -n "${BATCH_FOLD_PV:-}" ]; then + PV=$BATCH_FOLD_PV + else + # shellcheck source=scripts/pv_bin.sh + . scripts/pv_bin.sh || die "regen: scripts/pv_bin.sh could not build/resolve pv from this tree" + fi + make roadmap-aggregate >/dev/null 2>&1 || die "regen: make roadmap-aggregate failed" + "$PV" census contracts --format json > contracts/census.json 2>/dev/null || die "regen: pv census failed" + "$PV" extract contracts >/dev/null 2>&1 || die "regen: pv extract failed" + bash scripts/readme_sync.sh --write >/dev/null 2>&1 || die "regen: readme_sync.sh --write failed" + # the fixed points: each regenerated artifact must equal a second regeneration + make roadmap-aggregate-check >/dev/null 2>&1 || die "regen: fixed point FAILED -- make roadmap-aggregate-check" + "$PV" extract contracts --check >/dev/null 2>&1 || die "regen: fixed point FAILED -- pv extract contracts --check" + bash scripts/readme_sync.sh --check >/dev/null 2>&1 || die "regen: fixed point FAILED -- readme_sync.sh --check" + # only the generated set may have moved; anything else is a tool writing where it should not + extra=$(git status --porcelain --untracked-files=no | awk '{print $2}' \ + | grep -vxE 'docs/roadmaps/roadmap\.yaml|contracts/census\.json|contracts/contracts\.nt|contracts/shapes\.ttl|README\.md' || true) + [ -z "$extra" ] || die "regen: wrote outside the generated set: $(printf '%s ' $extra)" + if [ -n "$(git status --porcelain --untracked-files=no)" ]; then + git add -- docs/roadmaps/roadmap.yaml contracts/census.json contracts/contracts.nt contracts/shapes.ttl README.md 2>/dev/null + git -c commit.gpgsign=false commit -q -m "batch: regenerate the generated set once (batch_fold.sh --regen) + +roadmap aggregate, pv census, pv extract (contracts.nt + shapes.ttl) and the +README CONTRACT_COUNT blocks, regenerated with the pv built from this tree; +fixed points asserted: make roadmap-aggregate-check, pv extract --check, +readme_sync.sh --check." >/dev/null 2>&1 || die "regen: commit failed" + printf 'regenerated the generated set and committed it (fixed points asserted)\n' + else + printf 'regenerated the generated set: already consistent (fixed points asserted)\n' + fi +} + +case "${1:-}" in -h|--help) sed -n '2,42p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; esac + +REGEN=0; branches=() +while [ $# -gt 0 ]; do + case "$1" in + --regen) REGEN=1; shift ;; + -*) printf 'batch_fold.sh: unknown flag %s\n' "$1" >&2; exit 2 ;; + *) branches+=("$1"); shift ;; + esac +done +[ "${#branches[@]}" -gt 0 ] || [ "$REGEN" = 1 ] \ + || { printf 'usage: bash scripts/batch_fold.sh [--regen] BRANCH... | --regen\n' >&2; exit 2; } +git rev-parse --is-inside-work-tree >/dev/null 2>&1 || die "not inside a git work tree" +[ -z "$(git status --porcelain --untracked-files=no)" ] \ + || die "the batch worktree has uncommitted changes; fold into a clean tree only" + +STALE=0; status=0 +for br in "${branches[@]}"; do + fold_one "$br"; rc=$? + [ "$rc" = 2 ] && exit 2 + [ "$rc" = 1 ] && status=1 +done +if [ "$REGEN" = 1 ]; then + regen +elif [ "$STALE" = 1 ]; then + printf 'REGEN REQUIRED: a folded branch touched the generated set; run: bash scripts/batch_fold.sh --regen\n' + [ "$status" = 0 ] && status=3 +fi +exit "$status" diff --git a/scripts/check_batch_fold.sh b/scripts/check_batch_fold.sh new file mode 100644 index 0000000000..37d980b225 --- /dev/null +++ b/scripts/check_batch_fold.sh @@ -0,0 +1,169 @@ +#!/usr/bin/env bash +# check_batch_fold.sh -- the case table for scripts/batch_fold.sh, the integration-batch +# folder (#3673, APR-RELEASE-001 §13). Batching is the DEFAULT merge path now (operator +# 2026-09-21), so the folder decides what lands with a batch: it may resolve ONLY the +# generated set, and must leave out -- never silently resolve -- every real conflict. +# +# The table runs the real batch_fold.sh against throwaway git repos. Regeneration uses +# stub tools (a stub pv via BATCH_FOLD_PV, a stub Makefile and readme_sync.sh in the +# fixture), so no build and no network are needed. +# +# WHY THIS FILE AND NOT `batch_fold.sh --self-test`: a script that carries its own +# --self-test case claims to be a guard, and check_guards_are_wired.sh requires a +# workflow to run it -- measured: the handed-off draft did exactly that and turned +# the meta-guard RED ("NEW: batch_fold.sh"). A check_*.sh is dispatched by +# guard_tree.sh --no-cargo, so the table runs on every PR with no workflow edit. +# +# check_batch_fold.sh the case table +# check_batch_fold.sh --self-test the mutants: four wrong folders, each must turn a row RED +set -uo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" || exit 2 +SUBJECT="${BATCH_FOLD_SUBJECT:-$ROOT/scripts/batch_fold.sh}" + +rmtree() { case "${1:-}" in ''|/) return 0 ;; *) [ -d "$1" ] && rm -rf -- "$1" ;; esac; return 0; } + +# mkfixture DIR -- a repo with a generated set, README count blocks, code, and branches: +# plain (no generated path) gen1/gen2 (roadmap-only conflict) code1/code2 (real conflict) +# rc1/rc2 (README count-only conflict) rp1/rp2 (README prose conflict) +# cen (clean census edit) plus stub tools for --regen +mkfixture() { + local d=$1 + git init -q -b main "$d" || return 2 + ( + cd "$d" || exit 2 + export GIT_AUTHOR_NAME=t GIT_AUTHOR_EMAIL=t@t GIT_COMMITTER_NAME=t GIT_COMMITTER_EMAIL=t@t + export GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=core.hooksPath GIT_CONFIG_VALUE_0=/dev/null + mkdir -p docs/roadmaps contracts src scripts + printf 'a\n' > docs/roadmaps/roadmap.yaml + printf '{"n_files": 5}\n' > contracts/census.json + printf 't0\n' > contracts/contracts.nt + printf 's0\n' > contracts/shapes.ttl + printf 'x\n' > src/lib.rs + printf '# T\nprose line one\ncount: 5\nprose line two\n' > README.md + printf 'roadmap-aggregate:\n\t@printf %s > docs/roadmaps/roadmap.yaml\nroadmap-aggregate-check:\n\t@test "$$(cat docs/roadmaps/roadmap.yaml)" = aggregated\n' "'aggregated\\n'" > Makefile + cat > scripts/readme_sync.sh <<'SH' +#!/usr/bin/env bash +case "$1" in + --write) sed -E -i 's/()[0-9]+()/\17\2/g' README.md ;; + --check) grep -q 'CONTRACT_COUNT_START -->7510511[0-9]+7