From 99d2d7a93ff68dc6b343b2e86f0e698bad198b70 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Sun, 20 Sep 2026 23:17:25 +0200 Subject: [PATCH 1/6] PMAT-3587 nextest half, step 1 (RED): the guard that refuses a fail-fast [profile.ci] RED FIRST. Only the guard lands; .config/nextest.toml is untouched, so on this commit: $ bash scripts/check_nextest_ci_profile_no_fail_fast.sh FAIL .config/nextest.toml: [profile.ci].fail-fast = True -- a failing test cancels every test still queued and their verdicts are discarded rc=1 $ bash scripts/check_nextest_ci_profile_no_fail_fast.sh --self-test ok rows 1-9 (every polarity: false / true / absent / wrong profile / no profile / false-in-a-comment / the string "false" / unparseable / missing file) FAIL row 10 the real .config/nextest.toml is RED rc=1 Row 10 is deliberate: the case table judges the real config too, so a red tree cannot hide behind green fixtures. #3616 landed --no-fail-fast as the default for the explicit-command runner. The workspace-test shards run under nextest's OWN profile, where fail-fast = true still stands, and that is the half the operator's "on the shard" most plausibly meant. The behaviour, measured on a 6-test probe whose 2nd test fails (~60 ms), the rest 400-550 ms, -j 2 -- not read from docs: fail-fast = true Summary 3/6 tests run: 2 passed, 1 failed Cancelling due to test failure: 1 test still running fail-fast = false Summary 6 tests run: 5 passed, 1 failed (still rc 100) (key absent) Summary 3/6 tests run -- nextest's DEFAULT is fail-fast ON So an absent key is RED, not neutral, and the guard says so. The tree's own number for this class is the prior instance, not tonight's: "nextest fail-fast hid every other dark failure: 7 rounds for 4 tests." Tonight's 11,757 on #3621 were NOT this -- that log reads `Cancelling due to signal`, nextest's wording for receiving SIGTERM itself, which fail-fast does not govern; the probe above produced fail-fast's actual wording, `Cancelling due to test failure`, and the two are distinct. Those 11,757 are a receipt row (a signal-killed shard must read as Unknown), filed separately. The guard reads the key STRUCTURALLY (tomllib, [profile.ci], the bool) and runs no build tool, so guard_tree.sh --no-cargo picks it up with two rows and no ci.yml step is needed. Its first draft's header comment named the build tool with a space after it and matched CARGO_RE on its own -- a substring test satisfied by the documentation of the thing it looks for, in a guard written the day that class was named. Refs #3587, #3616, #3621, #3607 Pmat-Ticket: PMAT-3587 Co-Authored-By: Claude Opus 5 (1M context) --- .../check_nextest_ci_profile_no_fail_fast.sh | 94 +++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100755 scripts/check_nextest_ci_profile_no_fail_fast.sh diff --git a/scripts/check_nextest_ci_profile_no_fail_fast.sh b/scripts/check_nextest_ci_profile_no_fail_fast.sh new file mode 100755 index 0000000000..a975c5b390 --- /dev/null +++ b/scripts/check_nextest_ci_profile_no_fail_fast.sh @@ -0,0 +1,94 @@ +#!/usr/bin/env bash +# check_nextest_ci_profile_no_fail_fast.sh -- nextest's [profile.ci] must declare +# fail-fast = false, explicitly (PMAT-3587, the nextest half). +# +# THE DEFECT. fail-fast is a verdict-discarding default: on the first failing test +# nextest cancels every test still queued, and the shard reports the one failure with +# no count of what it did not measure. Measured on a 6-test probe whose 2nd test fails: +# fail-fast = true Summary 3/6 tests run: 2 passed, 1 failed +# Cancelling due to test failure: 1 test still running +# fail-fast = false Summary 6 tests run: 5 passed, 1 failed +# (key absent) Summary 3/6 tests run -- nextest's DEFAULT is fail-fast ON +# The tree paid for this once already: "nextest fail-fast hid every other dark failure: +# 7 rounds for 4 tests." A red run that still measures everything else it was going +# to measure is exactly the run whose data is most wanted. +# +# WHY THIS GUARD READS A TOML KEY AND DOES NOT RUN NEXTEST. The property IS the config +# value, read structurally (tomllib), under the profile CI actually uses -- not a +# substring anywhere in the file. A guard that invoked the build tool would match +# guard_tree.sh's CARGO_RE, be excluded from its --no-cargo run, and need its own +# ci.yml step; the behavioural proof above was run once and recorded in the PR that +# landed this. (This comment once said the tool's name with a space after it, and +# THAT was enough to match CARGO_RE: a substring test satisfied by the documentation +# of the thing it looks for, inside a guard written the same day that was named.) +# `--self-test` covers every polarity: false, true, absent, wrong profile, no profile, +# unparseable, missing file -- and only the first is green. +# +# check_nextest_ci_profile_no_fail_fast.sh judge .config/nextest.toml +# check_nextest_ci_profile_no_fail_fast.sh --self-test the case table (fixtures, no cargo) +set -uo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" || exit 2 +CONF="${NEXTEST_CONF_OVERRIDE:-$ROOT/.config/nextest.toml}" + +# judge -> 0 when [profile.ci].fail-fast is literally false; 1 otherwise; 2 ENV. +judge() { + local f=$1 + [ -r "$f" ] || { printf 'ENV %s: not readable -- cannot judge, not a pass\n' "$f" >&2; return 2; } + python3 - "$f" <<'PY' +import sys, tomllib +p = sys.argv[1] +try: + with open(p, "rb") as fh: + d = tomllib.load(fh) +except Exception as e: # unparseable config is ENV, never a pass + print(f"ENV {p}: does not parse as TOML ({e}) -- cannot judge, not a pass", file=sys.stderr); sys.exit(2) +prof = d.get("profile", {}).get("ci") +if prof is None: + print(f"FAIL {p}: no [profile.ci] -- CI runs --profile ci, so the run would take nextest's default, which is fail-fast ON", file=sys.stderr); sys.exit(1) +v = prof.get("fail-fast") +if v is None: + print(f"FAIL {p}: [profile.ci] does not set fail-fast -- nextest's default is ON (measured: 3/6 tests run on a 6-test probe)", file=sys.stderr); sys.exit(1) +if v is not False: + print(f"FAIL {p}: [profile.ci].fail-fast = {v!r} -- a failing test cancels every test still queued and their verdicts are discarded", file=sys.stderr); sys.exit(1) +print(f"ok {p}: [profile.ci].fail-fast = false -- a red run still measures everything else") +PY +} + +case "${1:-}" in -h|--help) sed -n '2,24p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; esac + +if [ "${1:-}" = "--self-test" ]; then + echo "=== nextest [profile.ci] fail-fast guard: case table ===" + d=$(mktemp -d) || exit 2 + rmtree() { case "${1:-}" in ''|/) return 0 ;; *) [ -d "$1" ] && rm -rf -- "$1" ;; esac; return 0; } + trap 'rmtree "${d:-}"' EXIT + bad=0; n=0 + row() { # row WANT_RC LABEL TOML-BODY + local want=$1 label=$2 body=$3 rc=0; n=$((n + 1)) + printf '%s' "$body" > "$d/c.toml" + judge "$d/c.toml" > /dev/null 2>&1 || rc=$? + if [ "$rc" = "$want" ]; then printf 'ok row %-2s rc=%s %s\n' "$n" "$rc" "$label" + else printf 'FAIL row %-2s rc=%s (wanted %s) %s\n' "$n" "$rc" "$want" "$label" >&2; bad=1; fi + } + row 0 "fail-fast = false under [profile.ci] -> PASS" $'[profile.ci]\nretries = 2\nfail-fast = false\n' + row 1 "fail-fast = true under [profile.ci] -> RED" $'[profile.ci]\nretries = 2\nfail-fast = true\n' + row 1 "key ABSENT under [profile.ci] -> RED (nextest default is ON)" $'[profile.ci]\nretries = 2\n' + row 1 "false under a DIFFERENT profile only -> RED" $'[profile.default]\nfail-fast = false\n[profile.ci]\nretries = 2\n' + row 1 "no [profile.ci] at all -> RED" $'[profile.default]\nfail-fast = false\n' + row 1 "false in a COMMENT, true in the key -> RED (not a substring test)" $'[profile.ci]\n# fail-fast = false\nfail-fast = true\n' + row 1 "the string \"false\" (a string, not a bool) -> RED" $'[profile.ci]\nfail-fast = "false"\n' + row 2 "unparseable TOML -> ENV rc=2, never a pass" $'[profile.ci\nfail-fast = false\n' + n=$((n + 1)); rc=0; judge "$d/absent.toml" > /dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] && printf 'ok row %-2s rc=2 missing file -> ENV rc=2, never a pass\n' "$n" \ + || { printf 'FAIL row %-2s rc=%s (wanted 2) missing file -> ENV\n' "$n" "$rc" >&2; bad=1; } + # and the real config, so a red tree cannot hide behind green fixtures + n=$((n + 1)); rc=0; judge "$CONF" > /dev/null 2>&1 || rc=$? + [ "$rc" -eq 0 ] && printf 'ok row %-2s rc=0 the real %s is GREEN\n' "$n" "${CONF#"$ROOT/"}" \ + || { printf 'FAIL row %-2s rc=%s the real %s is RED\n' "$n" "$rc" "${CONF#"$ROOT/"}" >&2; bad=1; } + [ "$bad" -eq 0 ] && { printf 'SELF-TEST PASSED: %s rows\n' "$n"; exit 0; } + printf 'SELF-TEST FAILED\n' >&2; exit 1 +fi + +echo "=== nextest [profile.ci] must not discard verdicts on the first failure (check_nextest_ci_profile_no_fail_fast.sh) ===" +judge "$CONF"; rc=$? +[ "$rc" -eq 0 ] && echo "PASS" || echo "FAIL (rc=$rc)" >&2 +exit "$rc" From 14915e051c2635f340053c7d911ec8b88c2b682b Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Sun, 20 Sep 2026 23:18:26 +0200 Subject: [PATCH 2/6] PMAT-3587 nextest half, step 2 (GREEN): [profile.ci] fail-fast = false One line in .config/nextest.toml, plus the comment that says why. Same guard as the RED commit, unchanged: now `run` PASS and `--self-test` 10/10. Every workspace-test shard runs `cargo nextest run --profile ci`, so this is the setting they inherit. A red shard now still measures every test it was going to measure, and the Summary line counts them -- which is exactly when the data is most wanted. The failure is still a failure (rc 100); it is just no longer the only thing the shard knows. nextest re-reads the config cleanly (`cargo nextest show-config version` rc 0). Not touched: ci.yml:497's comment "[profile.ci]: retries=0, fail-fast" -- a workflow file, and already stale on retries (#2616). It now drifts on both keys; noted in the PR body rather than edited here. Refs #3587, #3616 Pmat-Ticket: PMAT-3587 Co-Authored-By: Claude Opus 5 (1M context) --- .config/nextest.toml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.config/nextest.toml b/.config/nextest.toml index b0c188a42a..ba48acda18 100644 --- a/.config/nextest.toml +++ b/.config/nextest.toml @@ -16,7 +16,16 @@ # now fails on any key nextest ignores. [profile.ci] retries = 2 -fail-fast = true +# false, not true (PMAT-3587). fail-fast is a verdict-discarding default: on the first +# failing test nextest cancels every test still queued and reports the one failure with +# no count of what it did not measure. Measured on a 6-test probe whose 2nd test fails: +# true -> "3/6 tests run ... Cancelling due to test failure"; false -> "6 tests run: +# 5 passed, 1 failed" (still rc 100 -- a failure is still a failure, it is just no +# longer the only thing measured). An ABSENT key is the same as true: nextest's default +# is ON, so scripts/check_nextest_ci_profile_no_fail_fast.sh requires this line to be +# present and literally false. The tree paid for the old value once: "nextest fail-fast +# hid every other dark failure: 7 rounds for 4 tests." +fail-fast = false # Measured on an idle 48-core box, 80806 tests, 353s wall: slowest single test # is 202.7s (aprender-orchestrate bug_hunter::test_bh_mod_001_hunt_all_modes), # only 2 tests exceed 120s and NONE exceeds 300s. terminate-after = 20 periods From 0981864480f7087175b062bd19b272b0de6ca08a Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 01:32:43 +0200 Subject: [PATCH 3/6] PMAT-3587: register the ticket (fragment + aggregate) so the AD-04 quorum can run docs/roadmaps/entries/PMAT-3587.yaml hand-entered from issue #3587, roadmap.yaml regenerated by make roadmap-aggregate. Four roadmap guards rc=0. Pmat-Ticket: PMAT-3587 Co-Authored-By: Claude Opus 5 (1M context) --- docs/roadmaps/entries/PMAT-3587.yaml | 17 +++++++++++++++++ docs/roadmaps/roadmap.yaml | 17 +++++++++++++++++ 2 files changed, 34 insertions(+) create mode 100644 docs/roadmaps/entries/PMAT-3587.yaml diff --git a/docs/roadmaps/entries/PMAT-3587.yaml b/docs/roadmaps/entries/PMAT-3587.yaml new file mode 100644 index 0000000000..5d77bde6a0 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3587.yaml @@ -0,0 +1,17 @@ +- id: PMAT-3587 + github_issue: 3587 + item_type: task + title: 'nextest fail-fast discards verdicts: one failure hid 15 unmeasured commands and the shard read as ''1 failure'', not ''1 failure + 15 Unknown''' + status: planned + priority: critical + assigned_to: null + created: 2026-09-20T12:37:48Z + updated: 2026-09-20T12:37:48Z + spec: null + acceptance_criteria: [] + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'ACCEPTANCE (hand-entered from issue #3587 "What lands"; minted on the PR branch so the AD-04 quorum can run -- pmat work status refuses a missing item). Two halves: the explicit-command runner (#3616, merged) and the nextest [profile.ci] key (#3626). Lands: 1. **--no-fail-fast on the shard.** The cost is runner minutes on an already-red shard; the benefit is that a red run still measures everything else it was going to measure — which is exactly when you most want the data. 2. **The reducer reports ran / skipped / total** in the shard receipt. The skipped count is a **number**, never an absence. 3. **A receipt with skipped > 0 is Unknown, not Fail and not Pass** — it names what was not measured, per doctrine 4. ' diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index 3a90bd6297..d7016b8d80 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -18848,3 +18848,20 @@ roadmap: labels: - kind:code notes: 'ACCEPTANCE (hand-entered; pmat work add derived neither spec: nor acceptance_criteria: — defect paiml-mcp-agent-toolkit#1414). Spec: docs/specifications/ruling-receipts-under-contract.md (operator ruling 2026-09-20). Done when: contracts/parity-receipt-v1.yaml with its shape on origin/main; extract:parity-receipt implemented; pv lint --gate shapes --path evidence/ in the required check; PR body shows 7 older receipts RED before back-fill and 0 after, plant violation = 1, mutation RED, pc_shape fired; check_parity_receipt.sh folded into the shape or its remainder listed under not_expressible; ONT-4c3 bound with the parity receipt as focus node; verdict rendered on a fleet host once the pv pin lands, else checkout-only: true. Sigma parent bound: json. Back-fill denominator: 7. STOP: subset-insufficient; shared-file-touched without the guard label; any threshold typed into the shape instead of resolved from thresholds.yaml.' +- id: PMAT-3587 + github_issue: 3587 + item_type: task + title: 'nextest fail-fast discards verdicts: one failure hid 15 unmeasured commands and the shard read as ''1 failure'', not ''1 failure + 15 Unknown''' + status: planned + priority: critical + assigned_to: null + created: 2026-09-20T12:37:48Z + updated: 2026-09-20T12:37:48Z + spec: null + acceptance_criteria: [] + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'ACCEPTANCE (hand-entered from issue #3587 "What lands"; minted on the PR branch so the AD-04 quorum can run -- pmat work status refuses a missing item). Two halves: the explicit-command runner (#3616, merged) and the nextest [profile.ci] key (#3626). Lands: 1. **--no-fail-fast on the shard.** The cost is runner minutes on an already-red shard; the benefit is that a red run still measures everything else it was going to measure — which is exactly when you most want the data. 2. **The reducer reports ran / skipped / total** in the shard receipt. The skipped count is a **number**, never an absence. 3. **A receipt with skipped > 0 is Unknown, not Fail and not Pass** — it names what was not measured, per doctrine 4. ' From 2be40b44413e2a59208fbe6f72785e84858db7e4 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 01:35:28 +0200 Subject: [PATCH 4/6] =?UTF-8?q?PMAT-3587:=20AD-04=20quorum=20receipt=20for?= =?UTF-8?q?=20#3626=20=E2=80=94=203/3=20PASS,=20three=20independent=20lane?= =?UTF-8?q?s?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/audits/quorum-PMAT-3587.json, executor agy, width 3, judged at head 098186448, author model claude-opus-5, lane models gemini-3.1-pro-high / gemini-3.7-flash-high / gemini-3.6-flash-high (the trio ruled for everyone tonight after 3.8-flash's silences, paiml-implement#311). Three distinct conversation ids, verdicts via structured_output, zero sibling-lane or $WORK references. Dissent: none. First attempt. Pmat-Ticket: PMAT-3587 Co-Authored-By: Claude Opus 5 (1M context) --- docs/audits/quorum-PMAT-3587.json | 163 ++++++++++++++++++++++++++++++ 1 file changed, 163 insertions(+) create mode 100644 docs/audits/quorum-PMAT-3587.json diff --git a/docs/audits/quorum-PMAT-3587.json b/docs/audits/quorum-PMAT-3587.json new file mode 100644 index 0000000000..3d7266d6b5 --- /dev/null +++ b/docs/audits/quorum-PMAT-3587.json @@ -0,0 +1,163 @@ +{ + "ticket": "PMAT-3587", + "base": "origin/main", + "base_resolved": "origin/main", + "base_note": "no origin/origin/main exists; judged against the local ref", + "head": "0981864480f7087175b062bd19b272b0de6ca08a", + "diff_sha256": "428d6936d2f8478c7de2da67b93566df08a18f2f43eb78ba6314027eff16a772", + "width": 3, + "executor": "agy", + "prompt_mode": "inline", + "prompt_bytes": 12302, + "author": { + "model": "claude-opus-5", + "family": "claude", + "source": "flag" + }, + "agreed": true, + "lanes": [ + { + "lane": 1, + "status": "SUCCESS", + "verdict": "PASS", + "summary": "The diff implements the 'nextest [profile.ci] key (#3626)' half of PMAT-3587 exactly as requested. It correctly sets `fail-fast = false` under `[profile.ci]` in `.config/nextest.toml` and adds a robust guard script (`scripts/check_nextest_ci_profile_no_fail_fast.sh`) to prevent regression. The guard script incorporates a rigorous `--self-test` mechanism testing every polarity and correctly exits with code 2 for environment failures (missing file or unparseable TOML), enforcing anti-vacuity. Using `tomllib` in Python is a sound technical choice to avoid error-prone regex/string searching on the TOML format. The script passes `bash -n` and `shellcheck -S error`. There are no weakened gates, missing roadmap updates, or contrary testing assertions.", + "findings": [], + "raw_bytes": 3657, + "err_bytes": 0, + "envelope_status": "SUCCESS", + "verdict_source": "structured_output", + "grounding_check": "parity", + "model": "gemini-3.1-pro-high", + "model_measured": "gemini-3.1-pro-high", + "model_source": "measured", + "family": "gemini" + }, + { + "lane": 2, + "status": "SUCCESS", + "verdict": "PASS", + "summary": "As Lane 2 reviewer, I examined the diff against ticket PMAT-3587 and attempted to refute the change. The diff implements exactly what the ticket specifies without weakening gates or introducing extraneous modifications:\n1. `.config/nextest.toml`: Sets `fail-fast = false` under `[profile.ci]` so nextest does not cancel remaining tests when a failure occurs.\n2. `scripts/check_nextest_ci_profile_no_fail_fast.sh`: Adds an anti-vacuous guard using Python's `tomllib` to enforce `fail-fast = false` under `[profile.ci]`, passing a 10-row self-test table covering all polarities and failure modes.\n3. `docs/roadmaps/entries/PMAT-3587.yaml` and `docs/roadmaps/roadmap.yaml`: Adds ticket PMAT-3587 roadmap tracking.\n\nNo gates are weakened, no tests refute the ticket, and the implementation adheres to requirements. Verdict is PASS.", + "findings": [ + { + "claim": "The guard script scripts/check_nextest_ci_profile_no_fail_fast.sh structurally checks .config/nextest.toml using tomllib to ensure [profile.ci].fail-fast = false, and passes its complete 10-row self-test matrix covering true, false, absent keys, profile scoping, comments, non-boolean values, unparseable TOML, and missing files.", + "command": "scripts/check_nextest_ci_profile_no_fail_fast.sh --self-test", + "file": "scripts/check_nextest_ci_profile_no_fail_fast.sh", + "grounding": "measured", + "line": 57, + "output": "=== nextest [profile.ci] fail-fast guard: case table ===\nok row 1 rc=0 fail-fast = false under [profile.ci] -> PASS\nok row 2 rc=1 fail-fast = true under [profile.ci] -> RED\nok row 3 rc=1 key ABSENT under [profile.ci] -> RED (nextest default is ON)\nok row 4 rc=1 false under a DIFFERENT profile only -> RED\nok row 5 rc=1 no [profile.ci] at all -> RED\nok row 6 rc=1 false in a COMMENT, true in the key -> RED (not a substring test)\nok row 7 rc=1 the string \"false\" (a string, not a bool) -> RED\nok row 8 rc=2 unparseable TOML -> ENV rc=2, never a pass\nok row 9 rc=2 missing file -> ENV rc=2, never a pass\nok row 10 rc=0 the real .config/nextest.toml is GREEN\nSELF-TEST PASSED: 10 rows" + }, + { + "claim": ".config/nextest.toml explicitly configures fail-fast = false under [profile.ci], satisfying the nextest profile configuration requirement of ticket PMAT-3587.", + "file": ".config/nextest.toml", + "grounding": "cited", + "line": 28 + }, + { + "claim": "docs/roadmaps/entries/PMAT-3587.yaml and docs/roadmaps/roadmap.yaml record ticket PMAT-3587 with status planned and its acceptance criteria notes.", + "file": "docs/roadmaps/entries/PMAT-3587.yaml", + "grounding": "cited", + "line": 1 + } + ], + "raw_bytes": 7505, + "err_bytes": 0, + "envelope_status": "SUCCESS", + "verdict_source": "structured_output", + "grounding_check": "parity", + "model": "gemini-3.7-flash-high", + "model_measured": "gemini-3.7-flash-high", + "model_source": "measured", + "family": "gemini" + }, + { + "lane": 3, + "status": "SUCCESS", + "verdict": "PASS", + "summary": "As Lane 3 reviewer, I judged the diff against PMAT-3587 and attempted to refute it. All changes adhere strictly to the ticket requirements:\n1. `.config/nextest.toml` sets `fail-fast = false` under `[profile.ci]`, ensuring nextest does not cancel queued tests or discard verdicts on failure.\n2. `scripts/check_nextest_ci_profile_no_fail_fast.sh` introduces a guard script using `tomllib` with a 10-row self-test verifying edge cases (bool false vs string \"false\", comment tricks, profile scoping, missing keys, parse errors, missing files).\n3. `docs/roadmaps/entries/PMAT-3587.yaml` and `docs/roadmaps/roadmap.yaml` register ticket PMAT-3587.\n\nNo gate was weakened, no assertions violate the ticket, and no out-of-scope modifications were introduced. Verdict is PASS.", + "findings": [ + { + "claim": "The guard script scripts/check_nextest_ci_profile_no_fail_fast.sh correctly implements structural TOML validation and passes its 10-row self-test suite covering true, false, absent, string, comment, missing file, and invalid TOML cases.", + "command": "/tmp/test_pmat_3587/check_nextest_ci_profile_no_fail_fast.sh --self-test", + "file": "scripts/check_nextest_ci_profile_no_fail_fast.sh", + "grounding": "measured", + "line": 57, + "output": "=== nextest [profile.ci] fail-fast guard: case table ===\nok row 1 rc=0 fail-fast = false under [profile.ci] -> PASS\nok row 2 rc=1 fail-fast = true under [profile.ci] -> RED\nok row 3 rc=1 key ABSENT under [profile.ci] -> RED (nextest default is ON)\nok row 4 rc=1 false under a DIFFERENT profile only -> RED\nok row 5 rc=1 no [profile.ci] at all -> RED\nok row 6 rc=1 false in a COMMENT, true in the key -> RED (not a substring test)\nok row 7 rc=1 the string \"false\" (a string, not a bool) -> RED\nok row 8 rc=2 unparseable TOML -> ENV rc=2, never a pass\nok row 9 rc=2 missing file -> ENV rc=2, never a pass\nok row 10 rc=0 the real .config/nextest.toml is GREEN\nSELF-TEST PASSED: 10 rows" + }, + { + "claim": ".config/nextest.toml explicitly sets fail-fast = false under [profile.ci] as required by PMAT-3587.", + "file": ".config/nextest.toml", + "grounding": "cited", + "line": 28 + }, + { + "claim": "docs/roadmaps/entries/PMAT-3587.yaml and docs/roadmaps/roadmap.yaml properly add the tracking entry for ticket PMAT-3587 with acceptance notes.", + "file": "docs/roadmaps/entries/PMAT-3587.yaml", + "grounding": "cited", + "line": 1 + } + ], + "raw_bytes": 7100, + "err_bytes": 0, + "envelope_status": "SUCCESS", + "verdict_source": "structured_output", + "grounding_check": "parity", + "model": "gemini-3.6-flash-high", + "model_measured": "gemini-3.6-flash-high", + "model_source": "measured", + "family": "gemini" + } + ], + "dissent": [], + "dedup": [ + { + "file": ".config/nextest.toml", + "line": 28, + "lanes_agreeing": [ + 2, + 3 + ], + "claims": [ + ".config/nextest.toml explicitly configures fail-fast = false under [profile.ci], satisfying the nextest profile configuration requirement of ticket PMAT-3587.", + ".config/nextest.toml explicitly sets fail-fast = false under [profile.ci] as required by PMAT-3587." + ] + }, + { + "file": "docs/roadmaps/entries/PMAT-3587.yaml", + "line": 1, + "lanes_agreeing": [ + 2, + 3 + ], + "claims": [ + "docs/roadmaps/entries/PMAT-3587.yaml and docs/roadmaps/roadmap.yaml properly add the tracking entry for ticket PMAT-3587 with acceptance notes.", + "docs/roadmaps/entries/PMAT-3587.yaml and docs/roadmaps/roadmap.yaml record ticket PMAT-3587 with status planned and its acceptance criteria notes." + ] + }, + { + "file": "scripts/check_nextest_ci_profile_no_fail_fast.sh", + "line": 57, + "lanes_agreeing": [ + 2, + 3 + ], + "claims": [ + "The guard script scripts/check_nextest_ci_profile_no_fail_fast.sh correctly implements structural TOML validation and passes its 10-row self-test suite covering true, false, absent, string, comment, missing file, and invalid TOML cases.", + "The guard script scripts/check_nextest_ci_profile_no_fail_fast.sh structurally checks .config/nextest.toml using tomllib to ensure [profile.ci].fail-fast = false, and passes its complete 10-row self-test matrix covering true, false, absent keys, profile scoping, comments, non-boolean values, unparseable TOML, and missing files." + ] + } + ], + "uncovered": [], + "coverage_source": "lanes", + "partial": false, + "partial_reasons": [], + "auto_merge": { + "checked": true, + "was_armed": false, + "disarmed": false, + "note": "auto-merge not armed" + }, + "lint": { + "ok": true, + "output": "receipt complete: kind=artifact lanes=3 author=claude-opus-5/claude" + } +} From 0464c2da234a146cdf0f7ec7f936f762441094fd Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 03:16:30 +0200 Subject: [PATCH 5/6] fix(guard): the nextest fail-fast guard read a missing tomllib as "the config is RED" -- ENV is never 1 Measured on intel-clean-room-6 (#3626, guard-tree job 106169486109, head 2be40b444): Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'tomllib' FAIL (rc=1) and the self-test on the same runner: rows 1, 8, 10 FAIL, rows 2-7 "ok" -- because those rows WANTED rc=1 and a dead interpreter delivers rc=1 too. tomllib is python 3.11+; the runner's python3 is older; lambda's is 3.13, which is why every local run was green. The `import tomllib` sat at module level, outside the try, so the import error was never classified: an environment death was reported as a code defect, on a guard that exists to stop verdicts being discarded. Reproduced locally against the old guard with /usr/bin/python3.10 -S (no tomllib, no tomli): identical traceback, `FAIL (rc=1)`. What changes (all in scripts/check_nextest_ci_profile_no_fail_fast.sh): - Readers, in order: tomllib, tomli, then a purpose-built reader for the ONE question this guard asks (under [profile.ci], what is fail-fast?). The reader refuses -- ENV rc=2, never a guess -- anything outside its scope: multi-line strings, values continuing past the line, dotted keys, top-level keys, duplicate tables, [[profile.ci]] as an array of tables, a fail-fast value it cannot type. The real .config/nextest.toml (inline tables, [[...overrides]], sub-tables) judges GREEN through it: `NEXTEST_GUARD_PYTHON=python3.10 NEXTEST_GUARD_FORCE_FALLBACK=1` -> `reader=purpose-built reader`, rc=0. - The judge SAYS its verdict (a `VERDICT ok|FAIL|ENV` line). The shell accepts rc 0/1 only with that line; rc=1 without it -- a traceback before the first statement, a SyntaxError on an old python, no python3 at all (rc=127) -- is ENV rc=2 naming the interpreter and $RUNNER_NAME. A BaseException catch-all inside the judge classifies the rest as ENV too. - Self-test: 10 -> 30 rows. The case table runs twice, once per reader (NEXTEST_GUARD_FORCE_FALLBACK=1 sets sys.modules['tomllib'] = sys.modules['tomli'] = None so the imports fail for real, not a flag that skips around them); three refusal rows for the purpose-built reader; two death rows (/bin/false as the interpreter, a missing interpreter) that want ENV 2; two new table rows (false under [[profile.ci.overrides]] only -> RED; the real file's shape -> PASS). Passes under python 3.13.1 and 3.10.12 on this box. Mutants, each turning the self-test RED (baseline PASSED 30 rows): M1 rc=1 accepted with no VERDICT line row 28 rc=1 (wanted 2) M2 purpose-built reader removed 10 rows M3 multi-line string opener skipped, not refused 1 row (fixture: `note = """title"`, which a quote-count check passes) M4 [[profile.ci]] read as the profile 1 row M5 the string "false" typed as the bool 1 row M6 judge always returns 0 21 rows M8 continuation lines not refused 1 row M7 (library reader: a list profile.ci not refused explicitly) SURVIVES and is equivalent by construction: the catch-all classifies the AttributeError as the same ENV 2. Kept for the message it gives. Gates on the tree: bashrs-gate PASS (354 files), pipe-into-grep-q 75/75, hardcoded paths delta +0, apr pinned OK; the guard's --help still says self-test, so guard_tree runs both rows; no `cargo ` substring, so it stays inside the --no-cargo run. AD-04: docs/audits/quorum-PMAT-3587.json was judged at head 0981864 with diff_sha256 428d6936d2f8478c7de2da67b93566df08a18f2f43eb78ba6314027eff16a772. This commit MOVES diff_sha256 (the guard is inside the judged diff); the receipt is left as judged, per the cop's ruling that a fix on the same intent stands on it and says so here. New value: 41cdea8a2e887122bfa3e3d2ec8d37166ae1b52a88532d3f21f980e0e24e71c9. Refs #3587 #3626 Co-Authored-By: Claude Opus 5 (1M context) --- .../check_nextest_ci_profile_no_fail_fast.sh | 232 ++++++++++++++---- 1 file changed, 191 insertions(+), 41 deletions(-) diff --git a/scripts/check_nextest_ci_profile_no_fail_fast.sh b/scripts/check_nextest_ci_profile_no_fail_fast.sh index a975c5b390..406b360b2c 100755 --- a/scripts/check_nextest_ci_profile_no_fail_fast.sh +++ b/scripts/check_nextest_ci_profile_no_fail_fast.sh @@ -14,47 +14,171 @@ # to measure is exactly the run whose data is most wanted. # # WHY THIS GUARD READS A TOML KEY AND DOES NOT RUN NEXTEST. The property IS the config -# value, read structurally (tomllib), under the profile CI actually uses -- not a -# substring anywhere in the file. A guard that invoked the build tool would match -# guard_tree.sh's CARGO_RE, be excluded from its --no-cargo run, and need its own -# ci.yml step; the behavioural proof above was run once and recorded in the PR that -# landed this. (This comment once said the tool's name with a space after it, and -# THAT was enough to match CARGO_RE: a substring test satisfied by the documentation -# of the thing it looks for, inside a guard written the same day that was named.) -# `--self-test` covers every polarity: false, true, absent, wrong profile, no profile, -# unparseable, missing file -- and only the first is green. +# value, read structurally, under the profile CI actually uses -- not a substring +# anywhere in the file. A guard that invoked the build tool would match guard_tree.sh's +# CARGO_RE, be excluded from its --no-cargo run, and need its own ci.yml step; the +# behavioural proof above was run once and recorded in the PR that landed this. +# +# ENV IS NEVER 1. The first cut imported tomllib (python 3.11+) at module level. On +# intel-clean-room-6 (#3626, job 106169486109) python3 is older, the import traceback +# exited 1, and the guard reported "the real .config/nextest.toml is RED" -- an +# environment death read as a code defect; rows 2-7 of the self-test even "passed", +# because they wanted rc=1 anyway. Now: tomllib, else tomli, else a purpose-built reader +# for the one key this guard judges; the shell side accepts a verdict only when the +# judge SAID one (a VERDICT line), and anything else -- a traceback, a dead +# interpreter -- is ENV rc=2 naming the runner. --self-test runs the whole case table +# twice, once per reader, plus the death rows. # # check_nextest_ci_profile_no_fail_fast.sh judge .config/nextest.toml # check_nextest_ci_profile_no_fail_fast.sh --self-test the case table (fixtures, no cargo) set -uo pipefail ROOT="$(cd "$(dirname "$0")/.." && pwd)" || exit 2 CONF="${NEXTEST_CONF_OVERRIDE:-$ROOT/.config/nextest.toml}" +# test seams (read per call, so a self-test row can set them): NEXTEST_GUARD_PYTHON, the +# interpreter (a dead one reproduces the intel shape); NEXTEST_GUARD_FORCE_FALLBACK=1, a +# real import failure of both TOML libraries (reproduces an old python on a new one). # judge -> 0 when [profile.ci].fail-fast is literally false; 1 otherwise; 2 ENV. judge() { - local f=$1 + local f=$1 out rc=0 PYTHON="${NEXTEST_GUARD_PYTHON:-python3}" [ -r "$f" ] || { printf 'ENV %s: not readable -- cannot judge, not a pass\n' "$f" >&2; return 2; } - python3 - "$f" <<'PY' -import sys, tomllib + out=$(NEXTEST_GUARD_RUNNER="${RUNNER_NAME:-unknown}" "$PYTHON" - "$f" 2>&1 <<'PY' +import os, re, sys p = sys.argv[1] +runner = os.environ.get("NEXTEST_GUARD_RUNNER", "unknown") +if os.environ.get("NEXTEST_GUARD_FORCE_FALLBACK") == "1": + # make `import tomllib` / `import tomli` raise for real, so the fallback is the + # path an old interpreter takes and not a flag that skips around it + sys.modules["tomllib"] = None + sys.modules["tomli"] = None + +def verdict(code, tag, msg): + print("VERDICT %s %s: %s" % (tag, p, msg)) + sys.exit(code) + +def load_with_lib(path): + """(profile.ci table or None, reader name) via tomllib/tomli; None,None when neither imports.""" + for name in ("tomllib", "tomli"): + try: + mod = __import__(name) + except ImportError: + continue + with open(path, "rb") as fh: + d = mod.load(fh) + ci = d.get("profile", {}).get("ci") + if ci is not None and not isinstance(ci, dict): + raise ValueError("profile.ci is a %s, not a table" % type(ci).__name__) + return ci, name + return None, None + +_QUOTED = re.compile(r'"(?:[^"\\]|\\.)*"|\'[^\']*\'') +_HEADER = re.compile(r'^\[(\[)?\s*([A-Za-z0-9_.\-"\' ]+?)\s*\](\])?$') +_KEY = re.compile(r'^([A-Za-z0-9_\-]+|"[^"]+"|\'[^\']+\')\s*=\s*(.*)$') + +def load_minimal(path): + """Purpose-built reader for ONE question: under the table [profile.ci], what is the + value of fail-fast? Table headers set the current table; every other line must be a + single-line `key = value` that is balanced and opens no multi-line construct -- such + lines are only SKIPPED unless they are the judged key. Anything it does not + understand raises, and the caller reports ENV: it never guesses. Returns + (None, ...) when no [profile.ci] table exists, else the dict of judged keys.""" + tables = {} + cur = None + with open(path, "r", encoding="utf-8") as fh: + for n, raw in enumerate(fh, 1): + line = raw.strip() + if not line or line.startswith("#"): + continue + m = _HEADER.match(line) + if m: + aot = bool(m.group(1)) or bool(m.group(3)) + if bool(m.group(1)) != bool(m.group(3)): + raise ValueError("line %d: unbalanced table header" % n) + name = m.group(2).replace('"', "").replace("'", "") + if aot and name == "profile.ci": + raise ValueError("line %d: [[profile.ci]] is an array of tables, not the profile" % n) + cur = name + ("[]" if aot else "") + if not aot and cur in tables: + raise ValueError("line %d: duplicate table [%s]" % (n, name)) + tables.setdefault(cur, {}) + continue + m = _KEY.match(line) + if not m: + raise ValueError("line %d: not a table header or a `key = value` line" % n) + key, val = m.group(1).strip('"\''), m.group(2) + if "." in key: + raise ValueError("line %d: dotted key %r is outside this reader's scope" % (n, key)) + if val.startswith('"""') or val.startswith("'''"): + raise ValueError("line %d: multi-line string is outside this reader's scope" % n) + bare = _QUOTED.sub("", val) + if bare.count("[") != bare.count("]") or bare.count("{") != bare.count("}"): + raise ValueError("line %d: value continues past the line -- outside this reader's scope" % n) + if bare.count('"') % 2 or bare.count("'") % 2: + raise ValueError("line %d: unterminated string" % n) + if cur is None: + raise ValueError("line %d: top-level key %r is outside this reader's scope" % (n, key)) + if cur != "profile.ci": + continue + v = bare.split("#", 1)[0].strip() if not val.lstrip().startswith(('"', "'")) else val.strip() + if v == "true": + v = True + elif v == "false": + v = False + elif re.match(r'^-?\d+$', v): + v = int(v) + elif len(v) >= 2 and v[0] == v[-1] and v[0] in "\"'": + v = v[1:-1] + elif key == "fail-fast": + raise ValueError("line %d: fail-fast = %r is not a value this reader can type" % (n, v)) + if key in tables[cur]: + raise ValueError("line %d: duplicate key %r" % (n, key)) + tables[cur][key] = v + return tables.get("profile.ci") + try: - with open(p, "rb") as fh: - d = tomllib.load(fh) -except Exception as e: # unparseable config is ENV, never a pass - print(f"ENV {p}: does not parse as TOML ({e}) -- cannot judge, not a pass", file=sys.stderr); sys.exit(2) -prof = d.get("profile", {}).get("ci") -if prof is None: - print(f"FAIL {p}: no [profile.ci] -- CI runs --profile ci, so the run would take nextest's default, which is fail-fast ON", file=sys.stderr); sys.exit(1) -v = prof.get("fail-fast") -if v is None: - print(f"FAIL {p}: [profile.ci] does not set fail-fast -- nextest's default is ON (measured: 3/6 tests run on a 6-test probe)", file=sys.stderr); sys.exit(1) -if v is not False: - print(f"FAIL {p}: [profile.ci].fail-fast = {v!r} -- a failing test cancels every test still queued and their verdicts are discarded", file=sys.stderr); sys.exit(1) -print(f"ok {p}: [profile.ci].fail-fast = false -- a red run still measures everything else") + try: + prof, how = load_with_lib(p) + if how is None: + prof, how = load_minimal(p), "purpose-built reader (no tomllib/tomli on this interpreter)" + except Exception as e: + verdict(2, "ENV", "cannot be parsed on runner=%s (%s: %s) -- cannot judge, not a pass and not a fail" + % (runner, type(e).__name__, e)) + if prof is None: + verdict(1, "FAIL", "no [profile.ci] -- CI runs --profile ci, so the run would take nextest's default, " + "which is fail-fast ON (reader=%s)" % how) + v = prof.get("fail-fast") + if v is None: + verdict(1, "FAIL", "[profile.ci] does not set fail-fast -- nextest's default is ON (measured: 3/6 tests " + "run on a 6-test probe) (reader=%s)" % how) + if v is not False: + verdict(1, "FAIL", "[profile.ci].fail-fast = %r -- a failing test cancels every test still queued and " + "their verdicts are discarded (reader=%s)" % (v, how)) + verdict(0, "ok", "[profile.ci].fail-fast = false -- a red run still measures everything else " + "(reader=%s, runner=%s)" % (how, runner)) +except SystemExit: + raise +except BaseException as e: # the judge itself died: say so, as ENV + verdict(2, "ENV", "judge crashed on runner=%s (%s: %s) -- cannot judge, not a pass and not a fail" + % (runner, type(e).__name__, e)) PY + ) || rc=$? + # A verdict is something the judge SAID. rc=1 with no VERDICT line is the interpreter + # dying (a traceback before the first statement, a SyntaxError on an old python, no + # python3 at all) -- the exact shape that read as "RED" on intel-clean-room-6. + case "$out" in + *"VERDICT ok "*|*"VERDICT FAIL "*|*"VERDICT ENV "*) ;; + *) printf 'ENV %s: the judge did not judge (interpreter=%s rc=%s runner=%s) -- not a pass and not a fail\n%s\n' \ + "$f" "$PYTHON" "$rc" "${RUNNER_NAME:-unknown}" "$out" >&2; return 2 ;; + esac + case "$rc" in + 0) printf '%s\n' "${out#VERDICT }" ;; + 1|2) printf '%s\n' "${out#VERDICT }" >&2 ;; + *) printf 'ENV %s: judge rc=%s is not a verdict (runner=%s)\n%s\n' "$f" "$rc" "${RUNNER_NAME:-unknown}" "$out" >&2; return 2 ;; + esac + return "$rc" } -case "${1:-}" in -h|--help) sed -n '2,24p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; esac +case "${1:-}" in -h|--help) sed -n '2,33p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; esac if [ "${1:-}" = "--self-test" ]; then echo "=== nextest [profile.ci] fail-fast guard: case table ===" @@ -62,28 +186,54 @@ if [ "${1:-}" = "--self-test" ]; then rmtree() { case "${1:-}" in ''|/) return 0 ;; *) [ -d "$1" ] && rm -rf -- "$1" ;; esac; return 0; } trap 'rmtree "${d:-}"' EXIT bad=0; n=0 - row() { # row WANT_RC LABEL TOML-BODY + row() { # row WANT_RC LABEL TOML-BODY (reader named by $READER, set by table()) local want=$1 label=$2 body=$3 rc=0; n=$((n + 1)) printf '%s' "$body" > "$d/c.toml" judge "$d/c.toml" > /dev/null 2>&1 || rc=$? - if [ "$rc" = "$want" ]; then printf 'ok row %-2s rc=%s %s\n' "$n" "$rc" "$label" - else printf 'FAIL row %-2s rc=%s (wanted %s) %s\n' "$n" "$rc" "$want" "$label" >&2; bad=1; fi + if [ "$rc" = "$want" ]; then printf 'ok row %-2s rc=%s [%s] %s\n' "$n" "$rc" "$READER" "$label" + else printf 'FAIL row %-2s rc=%s (wanted %s) [%s] %s\n' "$n" "$rc" "$want" "$READER" "$label" >&2; bad=1; fi + } + table() { # the same table under BOTH readers; an old-python runner must judge the same property + row 0 "fail-fast = false under [profile.ci] -> PASS" $'[profile.ci]\nretries = 2\nfail-fast = false\n' + row 1 "fail-fast = true under [profile.ci] -> RED" $'[profile.ci]\nretries = 2\nfail-fast = true\n' + row 1 "key ABSENT under [profile.ci] -> RED (nextest default is ON)" $'[profile.ci]\nretries = 2\n' + row 1 "false under a DIFFERENT profile only -> RED" $'[profile.default]\nfail-fast = false\n[profile.ci]\nretries = 2\n' + row 1 "no [profile.ci] at all -> RED" $'[profile.default]\nfail-fast = false\n' + row 1 "false in a COMMENT, true in the key -> RED (not a substring test)" $'[profile.ci]\n# fail-fast = false\nfail-fast = true\n' + row 1 "the string \"false\" (a string, not a bool) -> RED" $'[profile.ci]\nfail-fast = "false"\n' + row 1 "false under [[profile.ci.overrides]] only -> RED (an override entry is not the profile)" \ + $'[profile.ci]\nretries = 2\n[[profile.ci.overrides]]\nfilter = "test(/x/)"\nfail-fast = false\n' + row 0 "false after an inline table and before a sub-table -> PASS (the real file's shape)" \ + $'[profile.ci]\nretries = 2\nfail-fast = false\nslow-timeout = { period = "60s", terminate-after = 20 }\nstatus-level = "slow"\n[profile.ci.junit]\npath = "junit.xml"\n' + row 2 "[[profile.ci]] (an array of tables, not the profile) -> ENV rc=2, never a pass" $'[[profile.ci]]\nfail-fast = false\n' + row 2 "unparseable TOML -> ENV rc=2, never a pass" $'[profile.ci\nfail-fast = false\n' + # and the real config, so a red tree cannot hide behind green fixtures + n=$((n + 1)); rc=0; judge "$CONF" > /dev/null 2>&1 || rc=$? + [ "$rc" -eq 0 ] && printf 'ok row %-2s rc=0 [%s] the real %s is GREEN\n' "$n" "$READER" "${CONF#"$ROOT/"}" \ + || { printf 'FAIL row %-2s rc=%s [%s] the real %s is RED\n' "$n" "$rc" "$READER" "${CONF#"$ROOT/"}" >&2; bad=1; } } - row 0 "fail-fast = false under [profile.ci] -> PASS" $'[profile.ci]\nretries = 2\nfail-fast = false\n' - row 1 "fail-fast = true under [profile.ci] -> RED" $'[profile.ci]\nretries = 2\nfail-fast = true\n' - row 1 "key ABSENT under [profile.ci] -> RED (nextest default is ON)" $'[profile.ci]\nretries = 2\n' - row 1 "false under a DIFFERENT profile only -> RED" $'[profile.default]\nfail-fast = false\n[profile.ci]\nretries = 2\n' - row 1 "no [profile.ci] at all -> RED" $'[profile.default]\nfail-fast = false\n' - row 1 "false in a COMMENT, true in the key -> RED (not a substring test)" $'[profile.ci]\n# fail-fast = false\nfail-fast = true\n' - row 1 "the string \"false\" (a string, not a bool) -> RED" $'[profile.ci]\nfail-fast = "false"\n' - row 2 "unparseable TOML -> ENV rc=2, never a pass" $'[profile.ci\nfail-fast = false\n' + READER=library; table + READER=fallback; NEXTEST_GUARD_FORCE_FALLBACK=1 table + # what the purpose-built reader must REFUSE rather than guess (ENV, never a pass) + READER=fallback + NEXTEST_GUARD_FORCE_FALLBACK=1 row 2 "a multi-line string hiding a [profile.ci] header -> ENV, never a guess (a line reader that skipped it would say PASS)" \ + $'[profile.default]\nnote = """title"\n[profile.ci]\nfail-fast = false\nx = """"\n' + NEXTEST_GUARD_FORCE_FALLBACK=1 row 2 "a value continuing past its line -> ENV, never a guess" \ + $'[profile.ci]\nfail-fast = false\nslow-timeout = { period = "60s",\n terminate-after = 20 }\n' + NEXTEST_GUARD_FORCE_FALLBACK=1 row 2 "fail-fast = fals (not a value it can type) -> ENV, never a guess" \ + $'[profile.ci]\nfail-fast = fals\n' + # the death rows: the shape that read as RED on intel-clean-room-6 + n=$((n + 1)); rc=0; printf '[profile.ci]\nfail-fast = false\n' > "$d/c.toml" + NEXTEST_GUARD_PYTHON=/bin/false judge "$d/c.toml" > /dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] && printf 'ok row %-2s rc=2 interpreter exits 1 with no verdict -> ENV rc=2, never 1\n' "$n" \ + || { printf 'FAIL row %-2s rc=%s (wanted 2) interpreter exits 1 with no verdict must be ENV, not RED\n' "$n" "$rc" >&2; bad=1; } + n=$((n + 1)); rc=0 + NEXTEST_GUARD_PYTHON="$d/no-such-interpreter" judge "$d/c.toml" > /dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] && printf 'ok row %-2s rc=2 no interpreter at all (rc=127) -> ENV rc=2\n' "$n" \ + || { printf 'FAIL row %-2s rc=%s (wanted 2) no interpreter must be ENV\n' "$n" "$rc" >&2; bad=1; } n=$((n + 1)); rc=0; judge "$d/absent.toml" > /dev/null 2>&1 || rc=$? [ "$rc" -eq 2 ] && printf 'ok row %-2s rc=2 missing file -> ENV rc=2, never a pass\n' "$n" \ || { printf 'FAIL row %-2s rc=%s (wanted 2) missing file -> ENV\n' "$n" "$rc" >&2; bad=1; } - # and the real config, so a red tree cannot hide behind green fixtures - n=$((n + 1)); rc=0; judge "$CONF" > /dev/null 2>&1 || rc=$? - [ "$rc" -eq 0 ] && printf 'ok row %-2s rc=0 the real %s is GREEN\n' "$n" "${CONF#"$ROOT/"}" \ - || { printf 'FAIL row %-2s rc=%s the real %s is RED\n' "$n" "$rc" "${CONF#"$ROOT/"}" >&2; bad=1; } [ "$bad" -eq 0 ] && { printf 'SELF-TEST PASSED: %s rows\n' "$n"; exit 0; } printf 'SELF-TEST FAILED\n' >&2; exit 1 fi From ef8064cde316c9286bc7602aded3cd2e7bee0fbb Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 03:29:47 +0200 Subject: [PATCH 6/6] =?UTF-8?q?PMAT-3587:=20AD-04=20quorum=20receipt=20rou?= =?UTF-8?q?nd=202=20for=20#3626=20at=203df1c64aa=20=E2=80=94=203/3=20PASS?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 1 (0981864, diff_sha256 428d6936…) attested a diff that the tomllib fix (0464c2da2) replaced; the cop disarmed on that ground. Round 2 judges the merge head 3df1c64aa against origin/main (diff_sha256 e54d3397…): lanes gemini-3.1-pro-high / gemini-3.1-pro-low / gemini-3.6-flash-high, PASS/PASS/PASS. Independence: lane 1 ran `--self-test` itself, lane 3 ran the guard against its own fixture under NEXTEST_CONF_OVERRIDE, lane 2 read only; no lane transcript reads a sibling lane file (lane-3's one match is the receipt named as a diff item, not a verdict). Refs #3587 #3626 Co-Authored-By: Claude Opus 5 (1M context) --- docs/audits/quorum-PMAT-3587.json | 119 +++++++++++++++++++++--------- 1 file changed, 84 insertions(+), 35 deletions(-) diff --git a/docs/audits/quorum-PMAT-3587.json b/docs/audits/quorum-PMAT-3587.json index 3d7266d6b5..6e4adb26e8 100644 --- a/docs/audits/quorum-PMAT-3587.json +++ b/docs/audits/quorum-PMAT-3587.json @@ -3,12 +3,12 @@ "base": "origin/main", "base_resolved": "origin/main", "base_note": "no origin/origin/main exists; judged against the local ref", - "head": "0981864480f7087175b062bd19b272b0de6ca08a", - "diff_sha256": "428d6936d2f8478c7de2da67b93566df08a18f2f43eb78ba6314027eff16a772", + "head": "3df1c64aac8c2efb04e3fba6c36667707d13960e", + "diff_sha256": "e54d3397340ac956739ce78c9cef906d56644eb982ff52db619882149eae3942", "width": 3, "executor": "agy", "prompt_mode": "inline", - "prompt_bytes": 12302, + "prompt_bytes": 31749, "author": { "model": "claude-opus-5", "family": "claude", @@ -20,9 +20,30 @@ "lane": 1, "status": "SUCCESS", "verdict": "PASS", - "summary": "The diff implements the 'nextest [profile.ci] key (#3626)' half of PMAT-3587 exactly as requested. It correctly sets `fail-fast = false` under `[profile.ci]` in `.config/nextest.toml` and adds a robust guard script (`scripts/check_nextest_ci_profile_no_fail_fast.sh`) to prevent regression. The guard script incorporates a rigorous `--self-test` mechanism testing every polarity and correctly exits with code 2 for environment failures (missing file or unparseable TOML), enforcing anti-vacuity. Using `tomllib` in Python is a sound technical choice to avoid error-prone regex/string searching on the TOML format. The script passes `bash -n` and `shellcheck -S error`. There are no weakened gates, missing roadmap updates, or contrary testing assertions.", - "findings": [], - "raw_bytes": 3657, + "summary": "The diff accurately implements the PMAT-3587 ticket requirements by setting `fail-fast = false` under `[profile.ci]` in `.config/nextest.toml`, effectively resolving the issue of nextest prematurely discarding test verdicts upon first failure. This prevents false metrics (like '1 failure + 15 Unknown') when executing CI tests. Additionally, the introduced script `scripts/check_nextest_ci_profile_no_fail_fast.sh` correctly establishes a robust mechanism to guarantee the property structurally, bypassing brittle substring checks. The script's 30-case `--self-test` suite has been executed and evaluated; it covers syntax variants and gracefully exits correctly when checking all edge cases. The required tracking items (docs/roadmaps/entries/PMAT-3587.yaml and docs/roadmaps/roadmap.yaml) are present. There are no violations, weakened gates, or instances of non-compliance with the ticket parameters. The outcome is PASS.", + "findings": [ + { + "claim": "The diff correctly updates .config/nextest.toml to explicitly set fail-fast = false under [profile.ci], which meets the PMAT-3587 requirement to prevent discarding of verdicts after a failure.", + "file": ".config/nextest.toml", + "grounding": "cited", + "line": 28 + }, + { + "claim": "The guard script scripts/check_nextest_ci_profile_no_fail_fast.sh implements anti-vacuous structural TOML validation that correctly asserts fail-fast = false is set under [profile.ci] and passes its comprehensive 30-row self-test table covering true/false/absent cases and malformed inputs.", + "command": "./scripts/check_nextest_ci_profile_no_fail_fast.sh --self-test", + "file": "scripts/check_nextest_ci_profile_no_fail_fast.sh", + "grounding": "measured", + "line": 204, + "output": "=== nextest [profile.ci] fail-fast guard: case table ===\\nok row 1 rc=0 [library] fail-fast = false under [profile.ci] -> PASS\\nok row 2 rc=1 [library] fail-fast = true under [profile.ci] -> RED\\nok row 3 rc=1 [library] key ABSENT under [profile.ci] -> RED (nextest default is ON)\\nok row 4 rc=1 [library] false under a DIFFERENT profile only -> RED\\nok row 5 rc=1 [library] no [profile.ci] at all -> RED\\nok row 6 rc=1 [library] false in a COMMENT, true in the key -> RED (not a substring test)\\nok row 7 rc=1 [library] the string \"false\" (a string, not a bool) -> RED\\nok row 8 rc=1 [library] false under [[profile.ci.overrides]] only -> RED (an override entry is not the profile)\\nok row 9 rc=0 [library] false after an inline table and before a sub-table -> PASS (the real file's shape)\\nok row 10 rc=2 [library] [[profile.ci]] (an array of tables, not the profile) -> ENV rc=2, never a pass\\nok row 11 rc=2 [library] unparseable TOML -> ENV rc=2, never a pass\\nok row 12 rc=0 [library] the real .config/nextest.toml is GREEN\\nok row 13 rc=0 [fallback] fail-fast = false under [profile.ci] -> PASS\\nok row 14 rc=1 [fallback] fail-fast = true under [profile.ci] -> RED\\nok row 15 rc=1 [fallback] key ABSENT under [profile.ci] -> RED (nextest default is ON)\\nok row 16 rc=1 [fallback] false under a DIFFERENT profile only -> RED\\nok row 17 rc=1 [fallback] no [profile.ci] at all -> RED\\nok row 18 rc=1 [fallback] false in a COMMENT, true in the key -> RED (not a substring test)\\nok row 19 rc=1 [fallback] the string \"false\" (a string, not a bool) -> RED\\nok row 20 rc=1 [fallback] false under [[profile.ci.overrides]] only -> RED (an override entry is not the profile)\\nok row 21 rc=0 [fallback] false after an inline table and before a sub-table -> PASS (the real file's shape)\\nok row 22 rc=2 [fallback] [[profile.ci]] (an array of tables, not the profile) -> ENV rc=2, never a pass\\nok row 23 rc=2 [fallback] unparseable TOML -> ENV rc=2, never a pass\\nok row 24 rc=0 [fallback] the real .config/nextest.toml is GREEN\\nok row 25 rc=2 [fallback] a multi-line string hiding a [profile.ci] header -> ENV, never a guess (a line reader that skipped it would say PASS)\\nok row 26 rc=2 [fallback] a value continuing past its line -> ENV, never a guess\\nok row 27 rc=2 [fallback] fail-fast = fals (not a value it can type) -> ENV, never a guess\\nok row 28 rc=2 interpreter exits 1 with no verdict -> ENV rc=2, never 1\\nok row 29 rc=2 no interpreter at all (rc=127) -> ENV rc=2\\nok row 30 rc=2 missing file -> ENV rc=2, never a pass\\nSELF-TEST PASSED: 30 rows" + }, + { + "claim": "docs/roadmaps/entries/PMAT-3587.yaml appropriately tracks the PMAT-3587 ticket and establishes its roadmap entry with correctly specified acceptance criteria for preventing verdict discards.", + "file": "docs/roadmaps/entries/PMAT-3587.yaml", + "grounding": "cited", + "line": 1 + } + ], + "raw_bytes": 11924, "err_bytes": 0, "envelope_status": "SUCCESS", "verdict_source": "structured_output", @@ -36,36 +57,34 @@ "lane": 2, "status": "SUCCESS", "verdict": "PASS", - "summary": "As Lane 2 reviewer, I examined the diff against ticket PMAT-3587 and attempted to refute the change. The diff implements exactly what the ticket specifies without weakening gates or introducing extraneous modifications:\n1. `.config/nextest.toml`: Sets `fail-fast = false` under `[profile.ci]` so nextest does not cancel remaining tests when a failure occurs.\n2. `scripts/check_nextest_ci_profile_no_fail_fast.sh`: Adds an anti-vacuous guard using Python's `tomllib` to enforce `fail-fast = false` under `[profile.ci]`, passing a 10-row self-test table covering all polarities and failure modes.\n3. `docs/roadmaps/entries/PMAT-3587.yaml` and `docs/roadmaps/roadmap.yaml`: Adds ticket PMAT-3587 roadmap tracking.\n\nNo gates are weakened, no tests refute the ticket, and the implementation adheres to requirements. Verdict is PASS.", + "summary": "The diff implements the required changes for PMAT-3587 by configuring `fail-fast = false` in `.config/nextest.toml` under `[profile.ci]`. It also includes a robust structural validation script with a `--self-test` to prevent regressions, satisfying the anti-vacuity and self-testing requirements. No gates are weakened and the changes directly fulfill the stated acceptance criteria.", "findings": [ { - "claim": "The guard script scripts/check_nextest_ci_profile_no_fail_fast.sh structurally checks .config/nextest.toml using tomllib to ensure [profile.ci].fail-fast = false, and passes its complete 10-row self-test matrix covering true, false, absent keys, profile scoping, comments, non-boolean values, unparseable TOML, and missing files.", - "command": "scripts/check_nextest_ci_profile_no_fail_fast.sh --self-test", - "file": "scripts/check_nextest_ci_profile_no_fail_fast.sh", - "grounding": "measured", - "line": 57, - "output": "=== nextest [profile.ci] fail-fast guard: case table ===\nok row 1 rc=0 fail-fast = false under [profile.ci] -> PASS\nok row 2 rc=1 fail-fast = true under [profile.ci] -> RED\nok row 3 rc=1 key ABSENT under [profile.ci] -> RED (nextest default is ON)\nok row 4 rc=1 false under a DIFFERENT profile only -> RED\nok row 5 rc=1 no [profile.ci] at all -> RED\nok row 6 rc=1 false in a COMMENT, true in the key -> RED (not a substring test)\nok row 7 rc=1 the string \"false\" (a string, not a bool) -> RED\nok row 8 rc=2 unparseable TOML -> ENV rc=2, never a pass\nok row 9 rc=2 missing file -> ENV rc=2, never a pass\nok row 10 rc=0 the real .config/nextest.toml is GREEN\nSELF-TEST PASSED: 10 rows" + "claim": "The diff sets fail-fast to false under [profile.ci] to ensure nextest does not cancel remaining tests on failure.", + "file": ".config/nextest.toml", + "grounding": "cited", + "line": 27 }, { - "claim": ".config/nextest.toml explicitly configures fail-fast = false under [profile.ci], satisfying the nextest profile configuration requirement of ticket PMAT-3587.", - "file": ".config/nextest.toml", + "claim": "A guard script is added to structurally enforce that [profile.ci].fail-fast is false, with a comprehensive --self-test.", + "file": "scripts/check_nextest_ci_profile_no_fail_fast.sh", "grounding": "cited", - "line": 28 + "line": 2 }, { - "claim": "docs/roadmaps/entries/PMAT-3587.yaml and docs/roadmaps/roadmap.yaml record ticket PMAT-3587 with status planned and its acceptance criteria notes.", + "claim": "The ticket is properly added to the roadmap to track the completion of the nextest [profile.ci] requirement.", "file": "docs/roadmaps/entries/PMAT-3587.yaml", "grounding": "cited", "line": 1 } ], - "raw_bytes": 7505, + "raw_bytes": 4155, "err_bytes": 0, "envelope_status": "SUCCESS", "verdict_source": "structured_output", "grounding_check": "parity", - "model": "gemini-3.7-flash-high", - "model_measured": "gemini-3.7-flash-high", + "model": "gemini-3.1-pro-low", + "model_measured": "gemini-3.1-pro-low", "model_source": "measured", "family": "gemini" }, @@ -73,30 +92,30 @@ "lane": 3, "status": "SUCCESS", "verdict": "PASS", - "summary": "As Lane 3 reviewer, I judged the diff against PMAT-3587 and attempted to refute it. All changes adhere strictly to the ticket requirements:\n1. `.config/nextest.toml` sets `fail-fast = false` under `[profile.ci]`, ensuring nextest does not cancel queued tests or discard verdicts on failure.\n2. `scripts/check_nextest_ci_profile_no_fail_fast.sh` introduces a guard script using `tomllib` with a 10-row self-test verifying edge cases (bool false vs string \"false\", comment tricks, profile scoping, missing keys, parse errors, missing files).\n3. `docs/roadmaps/entries/PMAT-3587.yaml` and `docs/roadmaps/roadmap.yaml` register ticket PMAT-3587.\n\nNo gate was weakened, no assertions violate the ticket, and no out-of-scope modifications were introduced. Verdict is PASS.", + "summary": "As Lane 3 reviewer, I evaluated the diff against ticket PMAT-3587 and attempted to refute it.\n\nReview Analysis:\n1. `.config/nextest.toml`: Updated to set `fail-fast = false` under `[profile.ci]`, ensuring nextest does not cancel remaining queued tests on failure or discard verdicts.\n2. `scripts/check_nextest_ci_profile_no_fail_fast.sh`: Added an anti-vacuous guard script using Python (`tomllib`/`tomli` or fallback reader) to enforce `fail-fast = false` under `[profile.ci]`. Measured and verified that `--self-test` passes all 30 test cases covering boolean types, string values, comment tricks, section scoping, unparseable TOML, missing files, and interpreter failure handling (rc=2 ENV exit).\n3. Roadmap tracking (`docs/roadmaps/entries/PMAT-3587.yaml` and `docs/roadmaps/roadmap.yaml`): Added PMAT-3587 tracking metadata and acceptance criteria.\n4. Quorum audit record (`docs/audits/quorum-PMAT-3587.json`): Added audit quorum records.\n\nNo gates were weakened, no assertions violate the ticket requirements, and no unauthorized modifications were introduced. Verdict is PASS.", "findings": [ { - "claim": "The guard script scripts/check_nextest_ci_profile_no_fail_fast.sh correctly implements structural TOML validation and passes its 10-row self-test suite covering true, false, absent, string, comment, missing file, and invalid TOML cases.", - "command": "/tmp/test_pmat_3587/check_nextest_ci_profile_no_fail_fast.sh --self-test", + "claim": "The guard script scripts/check_nextest_ci_profile_no_fail_fast.sh implements structural TOML validation ensuring [profile.ci].fail-fast = false and passes its 30-row self-test matrix covering all polarities, missing keys, comment tricks, string types, invalid TOML, missing files, and interpreter execution failure modes.", + "command": "NEXTEST_CONF_OVERRIDE=/tmp/test_pmat_3587/.config/nextest.toml /tmp/test_pmat_3587/check_nextest_ci_profile_no_fail_fast.sh --self-test", "file": "scripts/check_nextest_ci_profile_no_fail_fast.sh", "grounding": "measured", "line": 57, - "output": "=== nextest [profile.ci] fail-fast guard: case table ===\nok row 1 rc=0 fail-fast = false under [profile.ci] -> PASS\nok row 2 rc=1 fail-fast = true under [profile.ci] -> RED\nok row 3 rc=1 key ABSENT under [profile.ci] -> RED (nextest default is ON)\nok row 4 rc=1 false under a DIFFERENT profile only -> RED\nok row 5 rc=1 no [profile.ci] at all -> RED\nok row 6 rc=1 false in a COMMENT, true in the key -> RED (not a substring test)\nok row 7 rc=1 the string \"false\" (a string, not a bool) -> RED\nok row 8 rc=2 unparseable TOML -> ENV rc=2, never a pass\nok row 9 rc=2 missing file -> ENV rc=2, never a pass\nok row 10 rc=0 the real .config/nextest.toml is GREEN\nSELF-TEST PASSED: 10 rows" + "output": "=== nextest [profile.ci] fail-fast guard: case table ===\nok row 1 rc=0 [library] fail-fast = false under [profile.ci] -> PASS\nok row 2 rc=1 [library] fail-fast = true under [profile.ci] -> RED\nok row 3 rc=1 [library] key ABSENT under [profile.ci] -> RED (nextest default is ON)\n...\nSELF-TEST PASSED: 30 rows" }, { - "claim": ".config/nextest.toml explicitly sets fail-fast = false under [profile.ci] as required by PMAT-3587.", + "claim": ".config/nextest.toml explicitly sets fail-fast = false under [profile.ci], satisfying the nextest profile configuration requirement of ticket PMAT-3587 so test executions are not cancelled early on failure.", "file": ".config/nextest.toml", "grounding": "cited", "line": 28 }, { - "claim": "docs/roadmaps/entries/PMAT-3587.yaml and docs/roadmaps/roadmap.yaml properly add the tracking entry for ticket PMAT-3587 with acceptance notes.", + "claim": "docs/roadmaps/entries/PMAT-3587.yaml and docs/roadmaps/roadmap.yaml properly add the tracking entry and acceptance notes for ticket PMAT-3587.", "file": "docs/roadmaps/entries/PMAT-3587.yaml", "grounding": "cited", "line": 1 } ], - "raw_bytes": 7100, + "raw_bytes": 7282, "err_bytes": 0, "envelope_status": "SUCCESS", "verdict_source": "structured_output", @@ -109,40 +128,70 @@ ], "dissent": [], "dedup": [ + { + "file": ".config/nextest.toml", + "line": 27, + "lanes_agreeing": [ + 2 + ], + "claims": [ + "The diff sets fail-fast to false under [profile.ci] to ensure nextest does not cancel remaining tests on failure." + ] + }, { "file": ".config/nextest.toml", "line": 28, "lanes_agreeing": [ - 2, + 1, 3 ], "claims": [ - ".config/nextest.toml explicitly configures fail-fast = false under [profile.ci], satisfying the nextest profile configuration requirement of ticket PMAT-3587.", - ".config/nextest.toml explicitly sets fail-fast = false under [profile.ci] as required by PMAT-3587." + ".config/nextest.toml explicitly sets fail-fast = false under [profile.ci], satisfying the nextest profile configuration requirement of ticket PMAT-3587 so test executions are not cancelled early on failure.", + "The diff correctly updates .config/nextest.toml to explicitly set fail-fast = false under [profile.ci], which meets the PMAT-3587 requirement to prevent discarding of verdicts after a failure." ] }, { "file": "docs/roadmaps/entries/PMAT-3587.yaml", "line": 1, "lanes_agreeing": [ + 1, 2, 3 ], "claims": [ - "docs/roadmaps/entries/PMAT-3587.yaml and docs/roadmaps/roadmap.yaml properly add the tracking entry for ticket PMAT-3587 with acceptance notes.", - "docs/roadmaps/entries/PMAT-3587.yaml and docs/roadmaps/roadmap.yaml record ticket PMAT-3587 with status planned and its acceptance criteria notes." + "The ticket is properly added to the roadmap to track the completion of the nextest [profile.ci] requirement.", + "docs/roadmaps/entries/PMAT-3587.yaml and docs/roadmaps/roadmap.yaml properly add the tracking entry and acceptance notes for ticket PMAT-3587.", + "docs/roadmaps/entries/PMAT-3587.yaml appropriately tracks the PMAT-3587 ticket and establishes its roadmap entry with correctly specified acceptance criteria for preventing verdict discards." + ] + }, + { + "file": "scripts/check_nextest_ci_profile_no_fail_fast.sh", + "line": 2, + "lanes_agreeing": [ + 2 + ], + "claims": [ + "A guard script is added to structurally enforce that [profile.ci].fail-fast is false, with a comprehensive --self-test." ] }, { "file": "scripts/check_nextest_ci_profile_no_fail_fast.sh", "line": 57, "lanes_agreeing": [ - 2, 3 ], "claims": [ - "The guard script scripts/check_nextest_ci_profile_no_fail_fast.sh correctly implements structural TOML validation and passes its 10-row self-test suite covering true, false, absent, string, comment, missing file, and invalid TOML cases.", - "The guard script scripts/check_nextest_ci_profile_no_fail_fast.sh structurally checks .config/nextest.toml using tomllib to ensure [profile.ci].fail-fast = false, and passes its complete 10-row self-test matrix covering true, false, absent keys, profile scoping, comments, non-boolean values, unparseable TOML, and missing files." + "The guard script scripts/check_nextest_ci_profile_no_fail_fast.sh implements structural TOML validation ensuring [profile.ci].fail-fast = false and passes its 30-row self-test matrix covering all polarities, missing keys, comment tricks, string types, invalid TOML, missing files, and interpreter execution failure modes." + ] + }, + { + "file": "scripts/check_nextest_ci_profile_no_fail_fast.sh", + "line": 204, + "lanes_agreeing": [ + 1 + ], + "claims": [ + "The guard script scripts/check_nextest_ci_profile_no_fail_fast.sh implements anti-vacuous structural TOML validation that correctly asserts fail-fast = false is set under [profile.ci] and passes its comprehensive 30-row self-test table covering true/false/absent cases and malformed inputs." ] } ],