Repository navigation
PMAT-3587: fail-fast discards verdicts — the default flips, and skipped becomes a number - #3616
Merged
Merged
Conversation
RED FIXTURE FIRST. Only the case table moves; scripts/ci_run_explicit_test_commands.sh
is untouched, so these rows fail against the runner as it stands today.
`bash scripts/check_explicit_test_commands.sh --self-test` on THIS commit: rc=1,
6 failing of 72 rows (66 ok -- the fixture discriminates, it does not fail everything):
FAIL row 30 a failing command no longer stops the run (rc 1 = ran, measured failure)
FAIL row 32 ...and the command AFTER it ran too -- no fail-fast
FAIL row 33 the receipt reports ran/skipped/total as NUMBERS, never an absence
FAIL row 34 ...a command's own rc is NOT the runner's rc (7 must not leak out)
FAIL row 35 --fail-fast stops early and is UNKNOWN (rc 2), not Fail
FAIL row 37 all ran, none failed -> rc 0 and skipped=0
This is the ticket's own falsifier: plant a failing command early in the list and
require the EXACT count that still ran and the EXACT count skipped. A receipt that
reports only the failure fails the control.
The exit vocabulary these rows assert -- the runner's rc is its OWN verdict and never
a passthrough of a command's rc, because propagating one would let a test exiting 2
mean the same thing as "commands were not run":
0 every command ran, none failed
1 every command ran, F >= 1 FAILED (a measured failure)
2 UNMEASURED commands exist (Unknown, doctrine 4)
Row 34 exists specifically to pin that: the fixture's failing command exits 7, and
the runner must NOT exit 7.
Two defects in my own fixture, found by running it rather than reading it, both worth
recording because each produced a wrong count:
- a malformed `notfact ... ; test $? -eq 7` aborted the self-test after row 33 under
`set -e`, hiding rows 34-37. A fail-fast abort concealing later rows, inside the
harness written to fix exactly that -- it reported 3 failures when there were 6.
- the all-pass fixture used `true` for both fragments; the runner correctly REFUSED
a duplicate command (rc 2). The runner was right and the fixture was wrong.
Refs #3587, #3581, #3586, #3561, #3080
Pmat-Ticket: PMAT-3587
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…er, not an absence Same fixture, unchanged. Only scripts/ci_run_explicit_test_commands.sh moves. `--self-test` now rc=0, 72/72 rows. THE DEFAULT FLIPS, rather than a flag being added for the workflow to pass. The operator's framing is "fail-fast is a verdict-discarding DEFAULT", and a flag would leave the caller choosing, per invocation, whether to discard verdicts -- which is the defect restated, not removed. #3581 is what it cost: one test failing on a missing interpreter produced "fail-fast: the 15 command(s) after it did not run", and one shard hid 8,576 tests. Fifteen unknowns became an absence, and an absence is indistinguishable from "passed" to every consumer downstream. default every command runs; a red run still measures everything else --fail-fast opt-in, restores the old stop-at-first behaviour A RESULT line is printed on BOTH paths, always: RESULT ran=N skipped=S total=T failed=F RESULT first-failure [i/T] exit R: <cmd> skipped is a NUMBER. That is the whole point: an absence reads as "nothing to report"; a 2 reads as two commands nobody measured. EXIT CODE is the runner's OWN vocabulary and never a passthrough of a command's rc. Propagating one would let a test exiting 2 mean the same thing as "commands were not run" -- two meanings, one spelling, the defect this repo keeps finding: 0 every command ran, none failed 1 every command ran, F >= 1 FAILED -- a measured failure 2 UNMEASURED commands exist -- Unknown, doctrine 4; also every refusal Measured end to end on a 4-command fixture whose 2nd exits 7: default ran=4 skipped=0 total=4 failed=1 rc=1 (NOT 7 -- row 34 pins this) --fail-fast ran=2 skipped=2 total=4 failed=1 rc=2 UNKNOWN Callers checked on origin/main before flipping, not locally: the only invocation is ci.yml:634, a plain `run:` step whose rc is the step's rc, so 1 and 2 both fail the step as before and nothing branches on the value. ci.yml is NOT touched by this PR. Arg parsing takes --shard and --fail-fast in either order; both orders measured. Gates, fresh worktree off origin/main 00e5d2d: check_explicit_test_commands.sh and --self-test rc=0 (72 rows) check_hardcoded_paths.sh --full-if-capable rc=0 check_bashrs_gate.sh · check_apr_bin_pinned.sh rc=0 check_no_pipe_into_grep_q.sh rc=0 check_shell_lint_ratchet.sh rc=0 check_guards_are_wired.sh rc=0 check_sourced_libs_option_neutral.sh rc=0 Refs #3587, #3581, #3586, #3561, #3080 Pmat-Ticket: PMAT-3587 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
This was referenced Sep 20, 2026
PMAT-3587 (nextest half): [profile.ci] fail-fast = false — 3 of 6 verdicts discarded, measured
#3626
Merged
Merged
guyernest
pushed a commit
to guyernest/aprender
that referenced
this pull request
Sep 29, 2026
…dicts discarded, measured (paiml#3626) * PMAT-3587 nextest half, step 1 (RED): the guard that refuses a fail-fast [profile.ci] RED FIRST. Only the guard lands; .config/nextest.toml is untouched, so on this commit: $ bash scripts/check_nextest_ci_profile_no_fail_fast.sh FAIL .config/nextest.toml: [profile.ci].fail-fast = True -- a failing test cancels every test still queued and their verdicts are discarded rc=1 $ bash scripts/check_nextest_ci_profile_no_fail_fast.sh --self-test ok rows 1-9 (every polarity: false / true / absent / wrong profile / no profile / false-in-a-comment / the string "false" / unparseable / missing file) FAIL row 10 the real .config/nextest.toml is RED rc=1 Row 10 is deliberate: the case table judges the real config too, so a red tree cannot hide behind green fixtures. paiml#3616 landed --no-fail-fast as the default for the explicit-command runner. The workspace-test shards run under nextest's OWN profile, where fail-fast = true still stands, and that is the half the operator's "on the shard" most plausibly meant. The behaviour, measured on a 6-test probe whose 2nd test fails (~60 ms), the rest 400-550 ms, -j 2 -- not read from docs: fail-fast = true Summary 3/6 tests run: 2 passed, 1 failed Cancelling due to test failure: 1 test still running fail-fast = false Summary 6 tests run: 5 passed, 1 failed (still rc 100) (key absent) Summary 3/6 tests run -- nextest's DEFAULT is fail-fast ON So an absent key is RED, not neutral, and the guard says so. The tree's own number for this class is the prior instance, not tonight's: "nextest fail-fast hid every other dark failure: 7 rounds for 4 tests." Tonight's 11,757 on paiml#3621 were NOT this -- that log reads `Cancelling due to signal`, nextest's wording for receiving SIGTERM itself, which fail-fast does not govern; the probe above produced fail-fast's actual wording, `Cancelling due to test failure`, and the two are distinct. Those 11,757 are a receipt row (a signal-killed shard must read as Unknown), filed separately. The guard reads the key STRUCTURALLY (tomllib, [profile.ci], the bool) and runs no build tool, so guard_tree.sh --no-cargo picks it up with two rows and no ci.yml step is needed. Its first draft's header comment named the build tool with a space after it and matched CARGO_RE on its own -- a substring test satisfied by the documentation of the thing it looks for, in a guard written the day that class was named. Refs paiml#3587, paiml#3616, paiml#3621, paiml#3607 Pmat-Ticket: PMAT-3587 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * PMAT-3587 nextest half, step 2 (GREEN): [profile.ci] fail-fast = false One line in .config/nextest.toml, plus the comment that says why. Same guard as the RED commit, unchanged: now `run` PASS and `--self-test` 10/10. Every workspace-test shard runs `cargo nextest run --profile ci`, so this is the setting they inherit. A red shard now still measures every test it was going to measure, and the Summary line counts them -- which is exactly when the data is most wanted. The failure is still a failure (rc 100); it is just no longer the only thing the shard knows. nextest re-reads the config cleanly (`cargo nextest show-config version` rc 0). Not touched: ci.yml:497's comment "[profile.ci]: retries=0, fail-fast" -- a workflow file, and already stale on retries (paiml#2616). It now drifts on both keys; noted in the PR body rather than edited here. Refs paiml#3587, paiml#3616 Pmat-Ticket: PMAT-3587 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * PMAT-3587: register the ticket (fragment + aggregate) so the AD-04 quorum can run docs/roadmaps/entries/PMAT-3587.yaml hand-entered from issue paiml#3587, roadmap.yaml regenerated by make roadmap-aggregate. Four roadmap guards rc=0. Pmat-Ticket: PMAT-3587 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * PMAT-3587: AD-04 quorum receipt for paiml#3626 — 3/3 PASS, three independent lanes docs/audits/quorum-PMAT-3587.json, executor agy, width 3, judged at head 0981864, author model claude-opus-5, lane models gemini-3.1-pro-high / gemini-3.7-flash-high / gemini-3.6-flash-high (the trio ruled for everyone tonight after 3.8-flash's silences, paiml-implement#311). Three distinct conversation ids, verdicts via structured_output, zero sibling-lane or $WORK references. Dissent: none. First attempt. Pmat-Ticket: PMAT-3587 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(guard): the nextest fail-fast guard read a missing tomllib as "the config is RED" -- ENV is never 1 Measured on intel-clean-room-6 (paiml#3626, guard-tree job 106169486109, head 2be40b4): Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'tomllib' FAIL (rc=1) and the self-test on the same runner: rows 1, 8, 10 FAIL, rows 2-7 "ok" -- because those rows WANTED rc=1 and a dead interpreter delivers rc=1 too. tomllib is python 3.11+; the runner's python3 is older; lambda's is 3.13, which is why every local run was green. The `import tomllib` sat at module level, outside the try, so the import error was never classified: an environment death was reported as a code defect, on a guard that exists to stop verdicts being discarded. Reproduced locally against the old guard with /usr/bin/python3.10 -S (no tomllib, no tomli): identical traceback, `FAIL (rc=1)`. What changes (all in scripts/check_nextest_ci_profile_no_fail_fast.sh): - Readers, in order: tomllib, tomli, then a purpose-built reader for the ONE question this guard asks (under [profile.ci], what is fail-fast?). The reader refuses -- ENV rc=2, never a guess -- anything outside its scope: multi-line strings, values continuing past the line, dotted keys, top-level keys, duplicate tables, [[profile.ci]] as an array of tables, a fail-fast value it cannot type. The real .config/nextest.toml (inline tables, [[...overrides]], sub-tables) judges GREEN through it: `NEXTEST_GUARD_PYTHON=python3.10 NEXTEST_GUARD_FORCE_FALLBACK=1` -> `reader=purpose-built reader`, rc=0. - The judge SAYS its verdict (a `VERDICT ok|FAIL|ENV` line). The shell accepts rc 0/1 only with that line; rc=1 without it -- a traceback before the first statement, a SyntaxError on an old python, no python3 at all (rc=127) -- is ENV rc=2 naming the interpreter and $RUNNER_NAME. A BaseException catch-all inside the judge classifies the rest as ENV too. - Self-test: 10 -> 30 rows. The case table runs twice, once per reader (NEXTEST_GUARD_FORCE_FALLBACK=1 sets sys.modules['tomllib'] = sys.modules['tomli'] = None so the imports fail for real, not a flag that skips around them); three refusal rows for the purpose-built reader; two death rows (/bin/false as the interpreter, a missing interpreter) that want ENV 2; two new table rows (false under [[profile.ci.overrides]] only -> RED; the real file's shape -> PASS). Passes under python 3.13.1 and 3.10.12 on this box. Mutants, each turning the self-test RED (baseline PASSED 30 rows): M1 rc=1 accepted with no VERDICT line row 28 rc=1 (wanted 2) M2 purpose-built reader removed 10 rows M3 multi-line string opener skipped, not refused 1 row (fixture: `note = """title"`, which a quote-count check passes) M4 [[profile.ci]] read as the profile 1 row M5 the string "false" typed as the bool 1 row M6 judge always returns 0 21 rows M8 continuation lines not refused 1 row M7 (library reader: a list profile.ci not refused explicitly) SURVIVES and is equivalent by construction: the catch-all classifies the AttributeError as the same ENV 2. Kept for the message it gives. Gates on the tree: bashrs-gate PASS (354 files), pipe-into-grep-q 75/75, hardcoded paths delta +0, apr pinned OK; the guard's --help still says self-test, so guard_tree runs both rows; no `cargo ` substring, so it stays inside the --no-cargo run. AD-04: docs/audits/quorum-PMAT-3587.json was judged at head 0981864 with diff_sha256 428d6936d2f8478c7de2da67b93566df08a18f2f43eb78ba6314027eff16a772. This commit MOVES diff_sha256 (the guard is inside the judged diff); the receipt is left as judged, per the cop's ruling that a fix on the same intent stands on it and says so here. New value: 41cdea8a2e887122bfa3e3d2ec8d37166ae1b52a88532d3f21f980e0e24e71c9. Refs paiml#3587 paiml#3626 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * PMAT-3587: AD-04 quorum receipt round 2 for paiml#3626 at 3df1c64 — 3/3 PASS Round 1 (0981864, diff_sha256 428d6936…) attested a diff that the tomllib fix (0464c2d) replaced; the cop disarmed on that ground. Round 2 judges the merge head 3df1c64 against origin/main (diff_sha256 e54d3397…): lanes gemini-3.1-pro-high / gemini-3.1-pro-low / gemini-3.6-flash-high, PASS/PASS/PASS. Independence: lane 1 ran `--self-test` itself, lane 3 ran the guard against its own fixture under NEXTEST_CONF_OVERRIDE, lane 2 read only; no lane transcript reads a sibling lane file (lane-3's one match is the receipt named as a diff item, not a verdict). Refs paiml#3587 paiml#3626 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fail-fast in the shard command runner converts unknowns into an absence, and an absence is indistinguishable from "passed" to every consumer downstream.
Two commits, so the RED is in the history and a reviewer can run it:
What it cost
Measured on #3581: one test failing on a missing interpreter (
git rev-parse --show-toplevel, #3586) producedFifteen commands never ran and nothing downstream said so. The shard reported one failure; the truth was one failure and fifteen unmeasured. One earlier shard hid 8,576 tests. Among the fifteen was the PR's own new fragment — the change most in need of proof was the least exercised, and the receipt could not tell anyone.
The same defect, in the harness written to fix it
Worth more than the 8,576, because it is the defect happening to someone who already knew about it.
While writing the RED fixture I wrote a malformed row —
notfact … ; test $? -eq 7. Underset -ethat aborted the self-test after row 33, hiding rows 34-37, and it reported 3 failures where there were 6.A fail-fast abort concealing later rows, inside the harness built to stop fail-fast aborts concealing later rows. Nothing flagged it. I caught it only because the count looked too small for the number of behaviours I had asserted — which is the only way anyone ever catches this, and it is precisely the check a consumer of a shard receipt cannot make today.
A second one, kept for the same reason: my all-pass fixture used
truefor both fragments and the runner correctly refused a duplicate command (rc 2). My first reading was "the runner is broken". Both are recorded in the RED commit rather than quietly fixed — the wrong count is the lesson.What lands
The default flips. Not a flag for the workflow to pass: the operator's framing is "fail-fast is a verdict-discarding default", and a flag would leave the caller choosing, per invocation, whether to discard verdicts — the defect restated, not removed.
--fail-fastA
RESULTline on both paths, always:skippedis a number. That is the whole point — an absence reads as "nothing to report"; a2reads as two commands nobody measured.The exit code is the runner's own vocabulary and never a passthrough of a command's rc. Propagating one would let a test exiting 2 mean the same thing as "commands were not run" — two meanings, one spelling:
F ≥ 1failed — a measured failureThe individual rc lives in
RESULT first-failure, where it is information. Row 34 pins this: the fixture's failing command exits 7, and the runner must not exit 7.Measured end to end
A 4-command fixture whose 2nd exits 7:
ran=4 skipped=0 total=4 failed=1--fail-fastran=2 skipped=2 total=4 failed=1No workflow edit, and that is a measurement not a hope
Callers checked on
origin/main, not on a local checkout: the only invocation isci.yml:634, a plainrun:step whose rc is the step's rc, so 1 and 2 both fail it exactly as before and nothing branches on the value..github/workflows/is untouched by this PR.--shardand--fail-fastparse in either order; both orders measured.Gates
Fresh worktree off
origin/main(00e5d2dde):check_explicit_test_commands.shand--self-testcheck_hardcoded_paths.sh --full-if-capablecheck_bashrs_gate.sh·check_apr_bin_pinned.shcheck_no_pipe_into_grep_q.sh·check_shell_lint_ratchet.shcheck_guards_are_wired.sh·check_sourced_libs_option_neutral.shkeep-open: #3581 and #3586 are their own rows; #3561 is the same class in the T-0 preflight and is #3614; #3080 is the parent epic. Refs #3080, #3561, #3581, #3586.
ont-delta: none — a runner default and its case table, with no ontology type, shape or known-red list touched.
🤖 Generated with Claude Code