Skip to content

PMAT-3587: fail-fast discards verdicts — the default flips, and skipped becomes a number - #3616

Merged
noahgift merged 2 commits into
mainfrom
PMAT-3587-no-fail-fast
Sep 20, 2026
Merged

noahgift merged 2 commits into
mainfrom
PMAT-3587-no-fail-fast

Conversation

@noahgift

Copy link
Copy Markdown
Member

Fail-fast in the shard command runner converts unknowns into an absence, and an absence is indistinguishable from "passed" to every consumer downstream.

Two commits, so the RED is in the history and a reviewer can run it:

git checkout <RED>  && bash scripts/check_explicit_test_commands.sh --self-test   # rc=1, 6 failing of 72
git checkout <HEAD> && bash scripts/check_explicit_test_commands.sh --self-test   # rc=0, 72/72

What it cost

Measured on #3581: one test failing on a missing interpreter (git rev-parse --show-toplevel, #3586) produced

FAIL  [1/16] exit 101: cargo test -p aprender-core --test monorepo_invariants
      fail-fast: the 15 command(s) after it did not run

Fifteen commands never ran and nothing downstream said so. The shard reported one failure; the truth was one failure and fifteen unmeasured. One earlier shard hid 8,576 tests. Among the fifteen was the PR's own new fragment — the change most in need of proof was the least exercised, and the receipt could not tell anyone.

The same defect, in the harness written to fix it

Worth more than the 8,576, because it is the defect happening to someone who already knew about it.

While writing the RED fixture I wrote a malformed row — notfact … ; test $? -eq 7. Under set -e that aborted the self-test after row 33, hiding rows 34-37, and it reported 3 failures where there were 6.

A fail-fast abort concealing later rows, inside the harness built to stop fail-fast aborts concealing later rows. Nothing flagged it. I caught it only because the count looked too small for the number of behaviours I had asserted — which is the only way anyone ever catches this, and it is precisely the check a consumer of a shard receipt cannot make today.

A second one, kept for the same reason: my all-pass fixture used true for both fragments and the runner correctly refused a duplicate command (rc 2). My first reading was "the runner is broken". Both are recorded in the RED commit rather than quietly fixed — the wrong count is the lesson.

What lands

The default flips. Not a flag for the workflow to pass: the operator's framing is "fail-fast is a verdict-discarding default", and a flag would leave the caller choosing, per invocation, whether to discard verdicts — the defect restated, not removed.

default every command runs; a red run still measures everything else it was going to measure
--fail-fast opt-in, restores the old stop-at-first behaviour

A RESULT line on both paths, always:

RESULT ran=N skipped=S total=T failed=F
RESULT first-failure [i/T] exit R: <cmd>

skipped is a number. That is the whole point — an absence reads as "nothing to report"; a 2 reads as two commands nobody measured.

The exit code is the runner's own vocabulary and never a passthrough of a command's rc. Propagating one would let a test exiting 2 mean the same thing as "commands were not run" — two meanings, one spelling:

rc meaning
0 every command ran, none failed
1 every command ran, F ≥ 1 failed — a measured failure
2 unmeasured commands exist — Unknown, doctrine 4; also every refusal

The individual rc lives in RESULT first-failure, where it is information. Row 34 pins this: the fixture's failing command exits 7, and the runner must not exit 7.

Measured end to end

A 4-command fixture whose 2nd exits 7:

RESULT rc
default ran=4 skipped=0 total=4 failed=1 1 — not 7
--fail-fast ran=2 skipped=2 total=4 failed=1 2 — UNKNOWN

No workflow edit, and that is a measurement not a hope

Callers checked on origin/main, not on a local checkout: the only invocation is ci.yml:634, a plain run: step whose rc is the step's rc, so 1 and 2 both fail it exactly as before and nothing branches on the value. .github/workflows/ is untouched by this PR. --shard and --fail-fast parse in either order; both orders measured.

Gates

Fresh worktree off origin/main (00e5d2dde):

gate rc
check_explicit_test_commands.sh and --self-test 0 (72 rows)
check_hardcoded_paths.sh --full-if-capable 0
check_bashrs_gate.sh · check_apr_bin_pinned.sh 0
check_no_pipe_into_grep_q.sh · check_shell_lint_ratchet.sh 0
check_guards_are_wired.sh · check_sourced_libs_option_neutral.sh 0

keep-open: #3581 and #3586 are their own rows; #3561 is the same class in the T-0 preflight and is #3614; #3080 is the parent epic. Refs #3080, #3561, #3581, #3586.
ont-delta: none — a runner default and its case table, with no ontology type, shape or known-red list touched.

🤖 Generated with Claude Code

noahgift and others added 2 commits September 20, 2026 18:54
RED FIXTURE FIRST. Only the case table moves; scripts/ci_run_explicit_test_commands.sh
is untouched, so these rows fail against the runner as it stands today.

`bash scripts/check_explicit_test_commands.sh --self-test` on THIS commit: rc=1,
6 failing of 72 rows (66 ok -- the fixture discriminates, it does not fail everything):

  FAIL row 30  a failing command no longer stops the run (rc 1 = ran, measured failure)
  FAIL row 32    ...and the command AFTER it ran too -- no fail-fast
  FAIL row 33  the receipt reports ran/skipped/total as NUMBERS, never an absence
  FAIL row 34    ...a command's own rc is NOT the runner's rc (7 must not leak out)
  FAIL row 35  --fail-fast stops early and is UNKNOWN (rc 2), not Fail
  FAIL row 37  all ran, none failed -> rc 0 and skipped=0

This is the ticket's own falsifier: plant a failing command early in the list and
require the EXACT count that still ran and the EXACT count skipped. A receipt that
reports only the failure fails the control.

The exit vocabulary these rows assert -- the runner's rc is its OWN verdict and never
a passthrough of a command's rc, because propagating one would let a test exiting 2
mean the same thing as "commands were not run":

  0  every command ran, none failed
  1  every command ran, F >= 1 FAILED     (a measured failure)
  2  UNMEASURED commands exist            (Unknown, doctrine 4)

Row 34 exists specifically to pin that: the fixture's failing command exits 7, and
the runner must NOT exit 7.

Two defects in my own fixture, found by running it rather than reading it, both worth
recording because each produced a wrong count:
  - a malformed `notfact ... ; test $? -eq 7` aborted the self-test after row 33 under
    `set -e`, hiding rows 34-37. A fail-fast abort concealing later rows, inside the
    harness written to fix exactly that -- it reported 3 failures when there were 6.
  - the all-pass fixture used `true` for both fragments; the runner correctly REFUSED
    a duplicate command (rc 2). The runner was right and the fixture was wrong.

Refs #3587, #3581, #3586, #3561, #3080

Pmat-Ticket: PMAT-3587
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…er, not an absence

Same fixture, unchanged. Only scripts/ci_run_explicit_test_commands.sh moves.
`--self-test` now rc=0, 72/72 rows.

THE DEFAULT FLIPS, rather than a flag being added for the workflow to pass. The
operator's framing is "fail-fast is a verdict-discarding DEFAULT", and a flag would
leave the caller choosing, per invocation, whether to discard verdicts -- which is
the defect restated, not removed. #3581 is what it cost: one test failing on a
missing interpreter produced "fail-fast: the 15 command(s) after it did not run",
and one shard hid 8,576 tests. Fifteen unknowns became an absence, and an absence
is indistinguishable from "passed" to every consumer downstream.

  default      every command runs; a red run still measures everything else
  --fail-fast  opt-in, restores the old stop-at-first behaviour

A RESULT line is printed on BOTH paths, always:

  RESULT ran=N skipped=S total=T failed=F
  RESULT first-failure [i/T] exit R: <cmd>

skipped is a NUMBER. That is the whole point: an absence reads as "nothing to
report"; a 2 reads as two commands nobody measured.

EXIT CODE is the runner's OWN vocabulary and never a passthrough of a command's rc.
Propagating one would let a test exiting 2 mean the same thing as "commands were not
run" -- two meanings, one spelling, the defect this repo keeps finding:

  0  every command ran, none failed
  1  every command ran, F >= 1 FAILED     -- a measured failure
  2  UNMEASURED commands exist            -- Unknown, doctrine 4; also every refusal

Measured end to end on a 4-command fixture whose 2nd exits 7:
  default      ran=4 skipped=0 total=4 failed=1   rc=1   (NOT 7 -- row 34 pins this)
  --fail-fast  ran=2 skipped=2 total=4 failed=1   rc=2   UNKNOWN

Callers checked on origin/main before flipping, not locally: the only invocation is
ci.yml:634, a plain `run:` step whose rc is the step's rc, so 1 and 2 both fail the
step as before and nothing branches on the value. ci.yml is NOT touched by this PR.
Arg parsing takes --shard and --fail-fast in either order; both orders measured.

Gates, fresh worktree off origin/main 00e5d2d:
  check_explicit_test_commands.sh and --self-test   rc=0  (72 rows)
  check_hardcoded_paths.sh --full-if-capable        rc=0
  check_bashrs_gate.sh · check_apr_bin_pinned.sh    rc=0
  check_no_pipe_into_grep_q.sh                      rc=0
  check_shell_lint_ratchet.sh                       rc=0
  check_guards_are_wired.sh                         rc=0
  check_sourced_libs_option_neutral.sh              rc=0

Refs #3587, #3581, #3586, #3561, #3080

Pmat-Ticket: PMAT-3587
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=3616 head=1cd876418d65380a06ae253ee53fc3ebc4b421c1 verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

@noahgift
noahgift added this pull request to the merge queue Sep 20, 2026
Merged via the queue into main with commit 11a1038 Sep 20, 2026
26 of 27 checks passed
@noahgift
noahgift deleted the PMAT-3587-no-fail-fast branch September 20, 2026 20:52
guyernest pushed a commit to guyernest/aprender that referenced this pull request Sep 29, 2026
…dicts discarded, measured (paiml#3626)

* PMAT-3587 nextest half, step 1 (RED): the guard that refuses a fail-fast [profile.ci]

RED FIRST. Only the guard lands; .config/nextest.toml is untouched, so on this commit:

  $ bash scripts/check_nextest_ci_profile_no_fail_fast.sh
  FAIL  .config/nextest.toml: [profile.ci].fail-fast = True -- a failing test cancels
        every test still queued and their verdicts are discarded                   rc=1
  $ bash scripts/check_nextest_ci_profile_no_fail_fast.sh --self-test
  ok    rows 1-9 (every polarity: false / true / absent / wrong profile / no profile /
        false-in-a-comment / the string "false" / unparseable / missing file)
  FAIL  row 10  the real .config/nextest.toml is RED                                rc=1

Row 10 is deliberate: the case table judges the real config too, so a red tree cannot
hide behind green fixtures.

paiml#3616 landed --no-fail-fast as the default for the explicit-command runner. The
workspace-test shards run under nextest's OWN profile, where fail-fast = true still
stands, and that is the half the operator's "on the shard" most plausibly meant.

The behaviour, measured on a 6-test probe whose 2nd test fails (~60 ms), the rest
400-550 ms, -j 2 -- not read from docs:

  fail-fast = true    Summary  3/6 tests run: 2 passed, 1 failed
                      Cancelling due to test failure: 1 test still running
  fail-fast = false   Summary  6 tests run: 5 passed, 1 failed          (still rc 100)
  (key absent)        Summary  3/6 tests run  -- nextest's DEFAULT is fail-fast ON

So an absent key is RED, not neutral, and the guard says so. The tree's own number for
this class is the prior instance, not tonight's: "nextest fail-fast hid every other
dark failure: 7 rounds for 4 tests." Tonight's 11,757 on paiml#3621 were NOT this -- that
log reads `Cancelling due to signal`, nextest's wording for receiving SIGTERM itself,
which fail-fast does not govern; the probe above produced fail-fast's actual wording,
`Cancelling due to test failure`, and the two are distinct. Those 11,757 are a receipt
row (a signal-killed shard must read as Unknown), filed separately.

The guard reads the key STRUCTURALLY (tomllib, [profile.ci], the bool) and runs no
build tool, so guard_tree.sh --no-cargo picks it up with two rows and no ci.yml step
is needed. Its first draft's header comment named the build tool with a space after
it and matched CARGO_RE on its own -- a substring test satisfied by the documentation
of the thing it looks for, in a guard written the day that class was named.

Refs paiml#3587, paiml#3616, paiml#3621, paiml#3607

Pmat-Ticket: PMAT-3587
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* PMAT-3587 nextest half, step 2 (GREEN): [profile.ci] fail-fast = false

One line in .config/nextest.toml, plus the comment that says why. Same guard as the
RED commit, unchanged: now `run` PASS and `--self-test` 10/10.

Every workspace-test shard runs `cargo nextest run --profile ci`, so this is the
setting they inherit. A red shard now still measures every test it was going to
measure, and the Summary line counts them -- which is exactly when the data is most
wanted. The failure is still a failure (rc 100); it is just no longer the only thing
the shard knows.

nextest re-reads the config cleanly (`cargo nextest show-config version` rc 0).

Not touched: ci.yml:497's comment "[profile.ci]: retries=0, fail-fast" -- a workflow
file, and already stale on retries (paiml#2616). It now drifts on both keys; noted in the
PR body rather than edited here.

Refs paiml#3587, paiml#3616

Pmat-Ticket: PMAT-3587
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* PMAT-3587: register the ticket (fragment + aggregate) so the AD-04 quorum can run

docs/roadmaps/entries/PMAT-3587.yaml hand-entered from issue paiml#3587, roadmap.yaml
regenerated by make roadmap-aggregate. Four roadmap guards rc=0.

Pmat-Ticket: PMAT-3587
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* PMAT-3587: AD-04 quorum receipt for paiml#3626 — 3/3 PASS, three independent lanes

docs/audits/quorum-PMAT-3587.json, executor agy, width 3, judged at head 0981864,
author model claude-opus-5, lane models gemini-3.1-pro-high / gemini-3.7-flash-high /
gemini-3.6-flash-high (the trio ruled for everyone tonight after 3.8-flash's silences,
paiml-implement#311). Three distinct conversation ids, verdicts via structured_output,
zero sibling-lane or $WORK references. Dissent: none. First attempt.

Pmat-Ticket: PMAT-3587
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(guard): the nextest fail-fast guard read a missing tomllib as "the config is RED" -- ENV is never 1

Measured on intel-clean-room-6 (paiml#3626, guard-tree job 106169486109, head 2be40b4):

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ModuleNotFoundError: No module named 'tomllib'
    FAIL (rc=1)

and the self-test on the same runner: rows 1, 8, 10 FAIL, rows 2-7 "ok" -- because
those rows WANTED rc=1 and a dead interpreter delivers rc=1 too. tomllib is python
3.11+; the runner's python3 is older; lambda's is 3.13, which is why every local run
was green. The `import tomllib` sat at module level, outside the try, so the import
error was never classified: an environment death was reported as a code defect, on a
guard that exists to stop verdicts being discarded.

Reproduced locally against the old guard with /usr/bin/python3.10 -S (no tomllib,
no tomli): identical traceback, `FAIL (rc=1)`.

What changes (all in scripts/check_nextest_ci_profile_no_fail_fast.sh):

- Readers, in order: tomllib, tomli, then a purpose-built reader for the ONE
  question this guard asks (under [profile.ci], what is fail-fast?). The reader
  refuses -- ENV rc=2, never a guess -- anything outside its scope: multi-line
  strings, values continuing past the line, dotted keys, top-level keys, duplicate
  tables, [[profile.ci]] as an array of tables, a fail-fast value it cannot type.
  The real .config/nextest.toml (inline tables, [[...overrides]], sub-tables)
  judges GREEN through it: `NEXTEST_GUARD_PYTHON=python3.10
  NEXTEST_GUARD_FORCE_FALLBACK=1` -> `reader=purpose-built reader`, rc=0.
- The judge SAYS its verdict (a `VERDICT ok|FAIL|ENV` line). The shell accepts
  rc 0/1 only with that line; rc=1 without it -- a traceback before the first
  statement, a SyntaxError on an old python, no python3 at all (rc=127) -- is ENV
  rc=2 naming the interpreter and $RUNNER_NAME. A BaseException catch-all inside
  the judge classifies the rest as ENV too.
- Self-test: 10 -> 30 rows. The case table runs twice, once per reader
  (NEXTEST_GUARD_FORCE_FALLBACK=1 sets sys.modules['tomllib'] = sys.modules['tomli']
  = None so the imports fail for real, not a flag that skips around them); three
  refusal rows for the purpose-built reader; two death rows (/bin/false as the
  interpreter, a missing interpreter) that want ENV 2; two new table rows
  (false under [[profile.ci.overrides]] only -> RED; the real file's shape ->
  PASS). Passes under python 3.13.1 and 3.10.12 on this box.

Mutants, each turning the self-test RED (baseline PASSED 30 rows):
  M1 rc=1 accepted with no VERDICT line            row 28 rc=1 (wanted 2)
  M2 purpose-built reader removed                  10 rows
  M3 multi-line string opener skipped, not refused  1 row  (fixture: `note = """title"`,
                                                     which a quote-count check passes)
  M4 [[profile.ci]] read as the profile             1 row
  M5 the string "false" typed as the bool           1 row
  M6 judge always returns 0                        21 rows
  M8 continuation lines not refused                 1 row
  M7 (library reader: a list profile.ci not refused explicitly) SURVIVES and is
     equivalent by construction: the catch-all classifies the AttributeError as the
     same ENV 2. Kept for the message it gives.

Gates on the tree: bashrs-gate PASS (354 files), pipe-into-grep-q 75/75, hardcoded
paths delta +0, apr pinned OK; the guard's --help still says self-test, so guard_tree
runs both rows; no `cargo ` substring, so it stays inside the --no-cargo run.

AD-04: docs/audits/quorum-PMAT-3587.json was judged at head 0981864 with
diff_sha256 428d6936d2f8478c7de2da67b93566df08a18f2f43eb78ba6314027eff16a772. This
commit MOVES diff_sha256 (the guard is inside the judged diff); the receipt is left
as judged, per the cop's ruling that a fix on the same intent stands on it and says
so here. New value: 41cdea8a2e887122bfa3e3d2ec8d37166ae1b52a88532d3f21f980e0e24e71c9.

Refs paiml#3587 paiml#3626

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* PMAT-3587: AD-04 quorum receipt round 2 for paiml#3626 at 3df1c64 — 3/3 PASS

Round 1 (0981864, diff_sha256 428d6936…) attested a diff that the tomllib fix
(0464c2d) replaced; the cop disarmed on that ground. Round 2 judges the merge
head 3df1c64 against origin/main (diff_sha256 e54d3397…): lanes
gemini-3.1-pro-high / gemini-3.1-pro-low / gemini-3.6-flash-high, PASS/PASS/PASS.
Independence: lane 1 ran `--self-test` itself, lane 3 ran the guard against its
own fixture under NEXTEST_CONF_OVERRIDE, lane 2 read only; no lane transcript
reads a sibling lane file (lane-3's one match is the receipt named as a diff
item, not a verdict).

Refs paiml#3587 paiml#3626

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant