PMAT-3587 (nextest half): [profile.ci] fail-fast = false — 3 of 6 verdicts discarded, measured

.config/nextest.toml

@@ -16,7 +16,16 @@
 # now fails on any key nextest ignores.
 [profile.ci]
 retries = 2
-fail-fast = true
+# false, not true (PMAT-3587). fail-fast is a verdict-discarding default: on the first
+# failing test nextest cancels every test still queued and reports the one failure with
+# no count of what it did not measure. Measured on a 6-test probe whose 2nd test fails:
+# true -> "3/6 tests run ... Cancelling due to test failure"; false -> "6 tests run:
+# 5 passed, 1 failed" (still rc 100 -- a failure is still a failure, it is just no
+# longer the only thing measured). An ABSENT key is the same as true: nextest's default
+# is ON, so scripts/check_nextest_ci_profile_no_fail_fast.sh requires this line to be
+# present and literally false. The tree paid for the old value once: "nextest fail-fast
+# hid every other dark failure: 7 rounds for 4 tests."
+fail-fast = false
 # Measured on an idle 48-core box, 80806 tests, 353s wall: slowest single test
 # is 202.7s (aprender-orchestrate bug_hunter::test_bh_mod_001_hunt_all_modes),
 # only 2 tests exceed 120s and NONE exceeds 300s. terminate-after = 20 periods

docs/audits/quorum-PMAT-3587.json

@@ -0,0 +1,212 @@
+{
+  "ticket": "PMAT-3587",
+  "base": "origin/main",
+  "base_resolved": "origin/main",
+  "base_note": "no origin/origin/main exists; judged against the local ref",
+  "head": "3df1c64aac8c2efb04e3fba6c36667707d13960e",
+  "diff_sha256": "e54d3397340ac956739ce78c9cef906d56644eb982ff52db619882149eae3942",
+  "width": 3,
+  "executor": "agy",
+  "prompt_mode": "inline",
+  "prompt_bytes": 31749,
+  "author": {
+    "model": "claude-opus-5",
+    "family": "claude",
+    "source": "flag"
+  },
+  "agreed": true,
+  "lanes": [
+    {
+      "lane": 1,
+      "status": "SUCCESS",
+      "verdict": "PASS",
+      "summary": "The diff accurately implements the PMAT-3587 ticket requirements by setting `fail-fast = false` under `[profile.ci]` in `.config/nextest.toml`, effectively resolving the issue of nextest prematurely discarding test verdicts upon first failure. This prevents false metrics (like '1 failure + 15 Unknown') when executing CI tests. Additionally, the introduced script `scripts/check_nextest_ci_profile_no_fail_fast.sh` correctly establishes a robust mechanism to guarantee the property structurally, bypassing brittle substring checks. The script's 30-case `--self-test` suite has been executed and evaluated; it covers syntax variants and gracefully exits correctly when checking all edge cases. The required tracking items (docs/roadmaps/entries/PMAT-3587.yaml and docs/roadmaps/roadmap.yaml) are present. There are no violations, weakened gates, or instances of non-compliance with the ticket parameters. The outcome is PASS.",
+      "findings": [
+        {
+          "claim": "The diff correctly updates .config/nextest.toml to explicitly set fail-fast = false under [profile.ci], which meets the PMAT-3587 requirement to prevent discarding of verdicts after a failure.",
+          "file": ".config/nextest.toml",
+          "grounding": "cited",
+          "line": 28
+        },
+        {
+          "claim": "The guard script scripts/check_nextest_ci_profile_no_fail_fast.sh implements anti-vacuous structural TOML validation that correctly asserts fail-fast = false is set under [profile.ci] and passes its comprehensive 30-row self-test table covering true/false/absent cases and malformed inputs.",
+          "command": "./scripts/check_nextest_ci_profile_no_fail_fast.sh --self-test",
+          "file": "scripts/check_nextest_ci_profile_no_fail_fast.sh",
+          "grounding": "measured",
+          "line": 204,
+          "output": "=== nextest [profile.ci] fail-fast guard: case table ===\\nok    row 1  rc=0  [library] fail-fast = false under [profile.ci] -> PASS\\nok    row 2  rc=1  [library] fail-fast = true under [profile.ci] -> RED\\nok    row 3  rc=1  [library] key ABSENT under [profile.ci] -> RED (nextest default is ON)\\nok    row 4  rc=1  [library] false under a DIFFERENT profile only -> RED\\nok    row 5  rc=1  [library] no [profile.ci] at all -> RED\\nok    row 6  rc=1  [library] false in a COMMENT, true in the key -> RED (not a substring test)\\nok    row 7  rc=1  [library] the string \"false\" (a string, not a bool) -> RED\\nok    row 8  rc=1  [library] false under [[profile.ci.overrides]] only -> RED (an override entry is not the profile)\\nok    row 9  rc=0  [library] false after an inline table and before a sub-table -> PASS (the real file's shape)\\nok    row 10 rc=2  [library] [[profile.ci]] (an array of tables, not the profile) -> ENV rc=2, never a pass\\nok    row 11 rc=2  [library] unparseable TOML -> ENV rc=2, never a pass\\nok    row 12 rc=0  [library] the real .config/nextest.toml is GREEN\\nok    row 13 rc=0  [fallback] fail-fast = false under [profile.ci] -> PASS\\nok    row 14 rc=1  [fallback] fail-fast = true under [profile.ci] -> RED\\nok    row 15 rc=1  [fallback] key ABSENT under [profile.ci] -> RED (nextest default is ON)\\nok    row 16 rc=1  [fallback] false under a DIFFERENT profile only -> RED\\nok    row 17 rc=1  [fallback] no [profile.ci] at all -> RED\\nok    row 18 rc=1  [fallback] false in a COMMENT, true in the key -> RED (not a substring test)\\nok    row 19 rc=1  [fallback] the string \"false\" (a string, not a bool) -> RED\\nok    row 20 rc=1  [fallback] false under [[profile.ci.overrides]] only -> RED (an override entry is not the profile)\\nok    row 21 rc=0  [fallback] false after an inline table and before a sub-table -> PASS (the real file's shape)\\nok    row 22 rc=2  [fallback] [[profile.ci]] (an array of tables, not the profile) -> ENV rc=2, never a pass\\nok    row 23 rc=2  [fallback] unparseable TOML -> ENV rc=2, never a pass\\nok    row 24 rc=0  [fallback] the real .config/nextest.toml is GREEN\\nok    row 25 rc=2  [fallback] a multi-line string hiding a [profile.ci] header -> ENV, never a guess (a line reader that skipped it would say PASS)\\nok    row 26 rc=2  [fallback] a value continuing past its line -> ENV, never a guess\\nok    row 27 rc=2  [fallback] fail-fast = fals (not a value it can type) -> ENV, never a guess\\nok    row 28 rc=2  interpreter exits 1 with no verdict -> ENV rc=2, never 1\\nok    row 29 rc=2  no interpreter at all (rc=127) -> ENV rc=2\\nok    row 30 rc=2  missing file -> ENV rc=2, never a pass\\nSELF-TEST PASSED: 30 rows"
+        },
+        {
+          "claim": "docs/roadmaps/entries/PMAT-3587.yaml appropriately tracks the PMAT-3587 ticket and establishes its roadmap entry with correctly specified acceptance criteria for preventing verdict discards.",
+          "file": "docs/roadmaps/entries/PMAT-3587.yaml",
+          "grounding": "cited",
+          "line": 1
+        }
+      ],
+      "raw_bytes": 11924,
+      "err_bytes": 0,
+      "envelope_status": "SUCCESS",
+      "verdict_source": "structured_output",
+      "grounding_check": "parity",
+      "model": "gemini-3.1-pro-high",
+      "model_measured": "gemini-3.1-pro-high",
+      "model_source": "measured",
+      "family": "gemini"
+    },
+    {
+      "lane": 2,
+      "status": "SUCCESS",
+      "verdict": "PASS",
+      "summary": "The diff implements the required changes for PMAT-3587 by configuring `fail-fast = false` in `.config/nextest.toml` under `[profile.ci]`. It also includes a robust structural validation script with a `--self-test` to prevent regressions, satisfying the anti-vacuity and self-testing requirements. No gates are weakened and the changes directly fulfill the stated acceptance criteria.",
+      "findings": [
+        {
+          "claim": "The diff sets fail-fast to false under [profile.ci] to ensure nextest does not cancel remaining tests on failure.",
+          "file": ".config/nextest.toml",
+          "grounding": "cited",
+          "line": 27
+        },
+        {
+          "claim": "A guard script is added to structurally enforce that [profile.ci].fail-fast is false, with a comprehensive --self-test.",
+          "file": "scripts/check_nextest_ci_profile_no_fail_fast.sh",
+          "grounding": "cited",
+          "line": 2
+        },
+        {
+          "claim": "The ticket is properly added to the roadmap to track the completion of the nextest [profile.ci] requirement.",
+          "file": "docs/roadmaps/entries/PMAT-3587.yaml",
+          "grounding": "cited",
+          "line": 1
+        }
+      ],
+      "raw_bytes": 4155,
+      "err_bytes": 0,
+      "envelope_status": "SUCCESS",
+      "verdict_source": "structured_output",
+      "grounding_check": "parity",
+      "model": "gemini-3.1-pro-low",
+      "model_measured": "gemini-3.1-pro-low",
+      "model_source": "measured",
+      "family": "gemini"
+    },
+    {
+      "lane": 3,
+      "status": "SUCCESS",
+      "verdict": "PASS",
+      "summary": "As Lane 3 reviewer, I evaluated the diff against ticket PMAT-3587 and attempted to refute it.\n\nReview Analysis:\n1. `.config/nextest.toml`: Updated to set `fail-fast = false` under `[profile.ci]`, ensuring nextest does not cancel remaining queued tests on failure or discard verdicts.\n2. `scripts/check_nextest_ci_profile_no_fail_fast.sh`: Added an anti-vacuous guard script using Python (`tomllib`/`tomli` or fallback reader) to enforce `fail-fast = false` under `[profile.ci]`. Measured and verified that `--self-test` passes all 30 test cases covering boolean types, string values, comment tricks, section scoping, unparseable TOML, missing files, and interpreter failure handling (rc=2 ENV exit).\n3. Roadmap tracking (`docs/roadmaps/entries/PMAT-3587.yaml` and `docs/roadmaps/roadmap.yaml`): Added PMAT-3587 tracking metadata and acceptance criteria.\n4. Quorum audit record (`docs/audits/quorum-PMAT-3587.json`): Added audit quorum records.\n\nNo gates were weakened, no assertions violate the ticket requirements, and no unauthorized modifications were introduced. Verdict is PASS.",
+      "findings": [
+        {
+          "claim": "The guard script scripts/check_nextest_ci_profile_no_fail_fast.sh implements structural TOML validation ensuring [profile.ci].fail-fast = false and passes its 30-row self-test matrix covering all polarities, missing keys, comment tricks, string types, invalid TOML, missing files, and interpreter execution failure modes.",
+          "command": "NEXTEST_CONF_OVERRIDE=/tmp/test_pmat_3587/.config/nextest.toml /tmp/test_pmat_3587/check_nextest_ci_profile_no_fail_fast.sh --self-test",
+          "file": "scripts/check_nextest_ci_profile_no_fail_fast.sh",
+          "grounding": "measured",
+          "line": 57,
+          "output": "=== nextest [profile.ci] fail-fast guard: case table ===\nok    row 1  rc=0  [library] fail-fast = false under [profile.ci] -> PASS\nok    row 2  rc=1  [library] fail-fast = true under [profile.ci] -> RED\nok    row 3  rc=1  [library] key ABSENT under [profile.ci] -> RED (nextest default is ON)\n...\nSELF-TEST PASSED: 30 rows"
+        },
+        {
+          "claim": ".config/nextest.toml explicitly sets fail-fast = false under [profile.ci], satisfying the nextest profile configuration requirement of ticket PMAT-3587 so test executions are not cancelled early on failure.",
+          "file": ".config/nextest.toml",
+          "grounding": "cited",
+          "line": 28
+        },
+        {
+          "claim": "docs/roadmaps/entries/PMAT-3587.yaml and docs/roadmaps/roadmap.yaml properly add the tracking entry and acceptance notes for ticket PMAT-3587.",
+          "file": "docs/roadmaps/entries/PMAT-3587.yaml",
+          "grounding": "cited",
+          "line": 1
+        }
+      ],
+      "raw_bytes": 7282,
+      "err_bytes": 0,
+      "envelope_status": "SUCCESS",
+      "verdict_source": "structured_output",
+      "grounding_check": "parity",
+      "model": "gemini-3.6-flash-high",
+      "model_measured": "gemini-3.6-flash-high",
+      "model_source": "measured",
+      "family": "gemini"
+    }
+  ],
+  "dissent": [],
+  "dedup": [
+    {
+      "file": ".config/nextest.toml",
+      "line": 27,
+      "lanes_agreeing": [
+        2
+      ],
+      "claims": [
+        "The diff sets fail-fast to false under [profile.ci] to ensure nextest does not cancel remaining tests on failure."
+      ]
+    },
+    {
+      "file": ".config/nextest.toml",
+      "line": 28,
+      "lanes_agreeing": [
+        1,
+        3
+      ],
+      "claims": [
+        ".config/nextest.toml explicitly sets fail-fast = false under [profile.ci], satisfying the nextest profile configuration requirement of ticket PMAT-3587 so test executions are not cancelled early on failure.",
+        "The diff correctly updates .config/nextest.toml to explicitly set fail-fast = false under [profile.ci], which meets the PMAT-3587 requirement to prevent discarding of verdicts after a failure."
+      ]
+    },
+    {
+      "file": "docs/roadmaps/entries/PMAT-3587.yaml",
+      "line": 1,
+      "lanes_agreeing": [
+        1,
+        2,
+        3
+      ],
+      "claims": [
+        "The ticket is properly added to the roadmap to track the completion of the nextest [profile.ci] requirement.",
+        "docs/roadmaps/entries/PMAT-3587.yaml and docs/roadmaps/roadmap.yaml properly add the tracking entry and acceptance notes for ticket PMAT-3587.",
+        "docs/roadmaps/entries/PMAT-3587.yaml appropriately tracks the PMAT-3587 ticket and establishes its roadmap entry with correctly specified acceptance criteria for preventing verdict discards."
+      ]
+    },
+    {
+      "file": "scripts/check_nextest_ci_profile_no_fail_fast.sh",
+      "line": 2,
+      "lanes_agreeing": [
+        2
+      ],
+      "claims": [
+        "A guard script is added to structurally enforce that [profile.ci].fail-fast is false, with a comprehensive --self-test."
+      ]
+    },
+    {
+      "file": "scripts/check_nextest_ci_profile_no_fail_fast.sh",
+      "line": 57,
+      "lanes_agreeing": [
+        3
+      ],
+      "claims": [
+        "The guard script scripts/check_nextest_ci_profile_no_fail_fast.sh implements structural TOML validation ensuring [profile.ci].fail-fast = false and passes its 30-row self-test matrix covering all polarities, missing keys, comment tricks, string types, invalid TOML, missing files, and interpreter execution failure modes."
+      ]
+    },
+    {
+      "file": "scripts/check_nextest_ci_profile_no_fail_fast.sh",
+      "line": 204,
+      "lanes_agreeing": [
+        1
+      ],
+      "claims": [
+        "The guard script scripts/check_nextest_ci_profile_no_fail_fast.sh implements anti-vacuous structural TOML validation that correctly asserts fail-fast = false is set under [profile.ci] and passes its comprehensive 30-row self-test table covering true/false/absent cases and malformed inputs."
+      ]
+    }
+  ],
+  "uncovered": [],
+  "coverage_source": "lanes",
+  "partial": false,
+  "partial_reasons": [],
+  "auto_merge": {
+    "checked": true,
+    "was_armed": false,
+    "disarmed": false,
+    "note": "auto-merge not armed"
+  },
+  "lint": {
+    "ok": true,
+    "output": "receipt complete: kind=artifact lanes=3 author=claude-opus-5/claude"
+  }
+}

docs/roadmaps/entries/PMAT-3587.yaml

@@ -0,0 +1,17 @@
+- id: PMAT-3587
+  github_issue: 3587
+  item_type: task
+  title: 'nextest fail-fast discards verdicts: one failure hid 15 unmeasured commands and the shard read as ''1 failure'', not ''1 failure + 15 Unknown'''
+  status: planned
+  priority: critical
+  assigned_to: null
+  created: 2026-09-20T12:37:48Z
+  updated: 2026-09-20T12:37:48Z
+  spec: null
+  acceptance_criteria: []
+  phases: []
+  subtasks: []
+  estimated_effort: null
+  labels:
+  - kind:code
+  notes: 'ACCEPTANCE (hand-entered from issue #3587 "What lands"; minted on the PR branch so the AD-04 quorum can run -- pmat work status refuses a missing item). Two halves: the explicit-command runner (#3616, merged) and the nextest [profile.ci] key (#3626). Lands: 1. **--no-fail-fast on the shard.** The cost is runner minutes on an already-red shard; the benefit is that a red run still measures everything else it was going to measure — which is exactly when you most want the data. 2. **The reducer reports ran / skipped / total** in the shard receipt. The skipped count is a **number**, never an absence. 3. **A receipt with skipped > 0 is Unknown, not Fail and not Pass** — it names what was not measured, per doctrine 4. '

docs/roadmaps/roadmap.yaml

@@ -18869,6 +18869,23 @@ roadmap:
   labels:
   - kind:code
   notes: 'ACCEPTANCE (hand-entered; pmat work add derived neither spec: nor acceptance_criteria: — defect paiml-mcp-agent-toolkit#1414). Spec: docs/specifications/ruling-receipts-under-contract.md (operator ruling 2026-09-20). Done when: contracts/parity-receipt-v1.yaml with its shape on origin/main; extract:parity-receipt implemented; pv lint --gate shapes --path evidence/ in the required check; PR body shows 7 older receipts RED before back-fill and 0 after, plant violation = 1, mutation RED, pc_shape fired; check_parity_receipt.sh folded into the shape or its remainder listed under not_expressible; ONT-4c3 bound with the parity receipt as focus node; verdict rendered on a fleet host once the pv pin lands, else checkout-only: true. Sigma parent bound: json. Back-fill denominator: 7. STOP: subset-insufficient; shared-file-touched without the guard label; any threshold typed into the shape instead of resolved from thresholds.yaml.'
+- id: PMAT-3587
+  github_issue: 3587
+  item_type: task
+  title: 'nextest fail-fast discards verdicts: one failure hid 15 unmeasured commands and the shard read as ''1 failure'', not ''1 failure + 15 Unknown'''
+  status: planned
+  priority: critical
+  assigned_to: null
+  created: 2026-09-20T12:37:48Z
+  updated: 2026-09-20T12:37:48Z
+  spec: null
+  acceptance_criteria: []
+  phases: []
+  subtasks: []
+  estimated_effort: null
+  labels:
+  - kind:code
+  notes: 'ACCEPTANCE (hand-entered from issue #3587 "What lands"; minted on the PR branch so the AD-04 quorum can run -- pmat work status refuses a missing item). Two halves: the explicit-command runner (#3616, merged) and the nextest [profile.ci] key (#3626). Lands: 1. **--no-fail-fast on the shard.** The cost is runner minutes on an already-red shard; the benefit is that a red run still measures everything else it was going to measure — which is exactly when you most want the data. 2. **The reducer reports ran / skipped / total** in the shard receipt. The skipped count is a **number**, never an absence. 3. **A receipt with skipped > 0 is Unknown, not Fail and not Pass** — it names what was not measured, per doctrine 4. '
 - id: PMAT-3627
   github_issue: 3627
   item_type: task