Repository navigation
nextest fail-fast discards verdicts: one failure hid 15 unmeasured commands and the shard read as '1 failure', not '1 failure + 15 Unknown' #3587
Description
Operator ruling, 2026-09-20: fail-fast is a verdict-discarding default.
What happened
workspace-test-shard (3/3) on #3581 failed one test — falsify_ggml_003_extractor_case_table_passes, from the unguarded git rev-parse --show-toplevel (#3586). Then:
FAIL [1/16] exit 101: cargo test -p aprender-core --test monorepo_invariants
fail-fast: the 15 command(s) after it did not run
Fifteen commands never ran, and nothing downstream said so. The shard reports one failure. The truth is one failure and fifteen unmeasured. Among the fifteen was the PR's own new 350-aprender-quant-ggml-traits-fixture.cmd fragment — so the change most in need of proof on that head was the change least exercised, and the receipt could not tell anyone.
Why it is a verdict defect, not a speed setting
Fail-fast saves runner time by converting fourteen unknowns into an absence. An absence reads as "nothing to report", which is indistinguishable from "passed" to every consumer downstream — the same shape as a gate that returns GO because its producer emitted no rows (#3561), and as a status reader taking the oldest line of an append-only file. Skipped is Unknown. It is not green.
What lands
--no-fail-faston the shard. The cost is runner minutes on an already-red shard; the benefit is that a red run still measures everything else it was going to measure — which is exactly when you most want the data.- The reducer reports
ran / skipped / totalin the shard receipt. The skipped count is a number, never an absence. - A receipt with
skipped > 0isUnknown, notFailand notPass— it names what was not measured, per doctrine 4.
Falsifier: plant a failing test early in a shard's command list and require the receipt to report the exact count of commands that still ran, and the exact count skipped. A receipt that reports only the failure fails the control.
Note for whoever takes it
This interacts with a standing lesson — nextest fail-fast hid every other dark failure: 7 rounds for 4 tests. That was the same mechanism costing round-trips; this is the same mechanism costing coverage nobody knew was missing. One fix addresses both.
Activity
PMAT-3430: the case table must run where the code runs — awk, not pyt…
noahgift commentedon Sep 20, 2026
Four instances measured on 2026-09-20, all on #3581 or #3420. The fourth is the number that decides this.
workspace-test-shard (3/3), job 106075655182falsify_ggml_003(git refused the bind-mounted tree)fail-fast: the 15 command(s) after it did not runworkspace-test-shard (1/3), job 106078925945the_tracked_repo_graph_is_fresh(stalecontracts.nt)FAIL [13/17] … the 4 command(s) after it did not runworkspace-test-shard (3/3), job 106087864812falsify_ggml_003(python3: command not found)FAIL [1/16] … the 15 command(s) after it did not runworkspace-test-shard (1/3), job 105966962481qdc_dispatch_sites_match_oracle8576/21113 tests were not run due to test failureInstances 1–3 are commands; instance 4 is 8,576 individual tests, 41% of that shard, discarded because one test in it failed. nextest says so itself: "run with
--no-fail-fastto run all tests, or run with--max-fail".Why the count matters more than the principle. "A fail-fast shard might be hiding something" is a shrug. "One shard hid 8,576 tests on a PR in the 0.69 must-carry set, and nobody can say whether any of them were red" is a measurement, and it is unfalsifiable after the fact — the run is gone and the only honest verdict for those 8,576 is
Unknown, neverpass.The compounding cost, measured on #3581 today. Adding one contract file staled three tracked derivatives —
census.json,contracts.nt, and the READMECONTRACT_COUNTblock. Each was invisible until the one before it was fixed, because the first failure ended the run. Three CI rounds, on a pool at zero idle, to learn three facts that a single--no-fail-fastrun would have reported at once. Fail-fast does not only discard verdicts; it serializes discovery at one fact per round.So the
ran / skipped / totalfield this issue asks for is not bookkeeping — on instance 4 it is the difference between reading "1 failure" and reading "1 failure + 8,576 Unknown", and those are different states of the world.Refs #3581, #3420, #3430
PMAT-3430: receipt §6a — what adding ONE contract file actually cost
PMAT-3587: register the ticket (fragment + aggregate) so the AD-04 qu…
fix(guard): the nextest fail-fast guard read a missing tomllib as "th…
PMAT-3587: AD-04 quorum receipt round 2 for #3626 at 3df1c64 — 3/3 PASS
noahgift commentedon Sep 25, 2026
Parked by APR-EPIC-001: over S1 budget (rank 33/163). Reopen = pull. Applied by aprender-d1: claude+agy both said close, apr shadow row recorded (handoff/close-quorum-2026-09-25.json).
PMAT-3430 M1: one ggml tensor-type table, extracted from upstream rat…
PMAT-3587: fail-fast discards verdicts — the default flips, and skipp…
PMAT-3587 (nextest half): [profile.ci] fail-fast = false — 3 of 6 ver…