From dd165ad800e805c79b34e324bd1bb186ef6c4041 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 14 Sep 2026 09:21:46 +0200 Subject: [PATCH 1/4] =?UTF-8?q?docs(spec):=20APR-DARK-TARGETS-001=20?= =?UTF-8?q?=E2=80=94=20what=20the=20unrun=20test=20surface=20costs,=20hold?= =?UTF-8?q?s,=20and=20when=20it=20should=20run?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 654 test binaries have never been LINKED in CI, let alone run. Measured twice on 2026-09-14, one hour apart: merge-queue full tier (#2838) 82,217 tests 73 binaries 37 min intel-clean-room-14 whole --lib --tests 98,621 tests 727 binaries 19 min lambda-vector (48c) A binary that never links cannot fail, flake, or appear in a report. That is why this was invisible rather than noisy, and it is the sentence the rest follows from. `make coverage` is `--lib` too; nightly.yml runs no tests at all. WHAT IT HELD: 37 failures, 5 binaries, and every one a GLOBAL-INVARIANT defect -- a golden oracle, a path to a sibling crate's assets, a contract copy that must equal another file. Not one is a compute regression. That is structural, not luck: BSE-17 selects by CHANGED CRATE and none of these lives in a diff. The quick tier is NOT the gap and this spec does not touch it -- a PR touching crate X does build and run X's integration targets. The argument is golden_v2, not the count. A golden exists ONLY to catch silent format drift and it missed a real format change for two months. #2254 happened to be deliberate and correct; had it been wrong, nothing here would have said so. A dark golden is worse than no golden, because its presence reads as coverage. DECISION: nightly primary; pre-release (T-1) as a HARD gate, since "green on the cut" over 73 of 727 binaries is a weaker claim than it reads; NOT the merge queue; plus one addition to "nightly and pre-release only" -- PRs that touch the SELECTOR (ci_test_tier.sh, gate_touched_crates.sh, tree_reader_tests.txt, ci.yml's --test chain), because that is exactly when the dark set moves. It starts ratcheted or it is born red at 37 and becomes `present` within a week. But Phase B is reachable: with #3248/#3249/#3238 landed the count is ONE, so the ratchet is scaffolding with a named exit condition, not a resting place. I WAS WRONG ABOUT THE COST AND THE MEASUREMENT CAUGHT ME. The first draft asserted the --tests delta was ">> 20 min" of linking, on the strength of a progress check reading a stale PID. Measured, same box, cold target dir: cargo build --workspace --lib 76 s 715 crates + --tests incrementally 145 s 228 crates -> 808 binaries, 39 GB Two and a half minutes, not twenty. The cost is the RUN, not the link. That makes the merge-queue question closer than the first draft implied -- ~+14 min (+35%) on a 37-min critical path by throughput-calibrated extrapolation -- so the verdict is unchanged but the REASON is now a number, and §3 names the condition under which it flips. The number was already being measured when the claim was written; the claim should have waited. §7.1 records that. Ships with the failing set as measured, so the first ratchet baseline is not re-derived from memory. Refs #3239 Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) --- ...ARK-TARGETS-001-failing-set-2026-09-14.txt | 37 +++ ...R-DARK-TARGETS-001-full-surface-cadence.md | 236 ++++++++++++++++++ 2 files changed, 273 insertions(+) create mode 100644 docs/specifications/APR-DARK-TARGETS-001-failing-set-2026-09-14.txt create mode 100644 docs/specifications/APR-DARK-TARGETS-001-full-surface-cadence.md diff --git a/docs/specifications/APR-DARK-TARGETS-001-failing-set-2026-09-14.txt b/docs/specifications/APR-DARK-TARGETS-001-failing-set-2026-09-14.txt new file mode 100644 index 0000000000..11ac3b0239 --- /dev/null +++ b/docs/specifications/APR-DARK-TARGETS-001-failing-set-2026-09-14.txt @@ -0,0 +1,37 @@ +aprender-cgp::falsify falsify_cgp_061_doctor_speed_real +aprender-mcp::falsify_mcp_008 contract_copy_matches_workspace_root +aprender-present-yaml::examples test_ald_001_data_card_basic +aprender-present-yaml::examples test_ald_002_data_table_virtualized +aprender-present-yaml::examples test_ald_003_data_distribution_chart +aprender-present-yaml::examples test_ald_004_data_scatter_plot +aprender-present-yaml::examples test_ald_006_data_time_series +aprender-present-yaml::examples test_ald_008_data_class_balance +aprender-present-yaml::examples test_all_ald_examples_exist +aprender-present-yaml::examples test_all_apr_examples_exist +aprender-present-yaml::examples test_all_chart_examples_exist +aprender-present-yaml::examples test_all_dashboard_examples_exist +aprender-present-yaml::examples test_all_edge_case_examples_exist +aprender-present-yaml::examples test_apr_001_model_card_basic +aprender-present-yaml::examples test_apr_002_model_comparison +aprender-present-yaml::examples test_apr_003_model_metrics_chart +aprender-present-yaml::examples test_apr_005_model_inference_demo +aprender-present-yaml::examples test_apr_007_model_gradient_flow +aprender-present-yaml::examples test_apr_010_model_export_preview +aprender-present-yaml::examples test_cht_001_line_chart_basic +aprender-present-yaml::examples test_cht_002_bar_chart_grouped +aprender-present-yaml::examples test_cht_003_pie_chart_basic +aprender-present-yaml::examples test_dsh_001_training_dashboard +aprender-present-yaml::examples test_dsh_002_dataset_explorer +aprender-present-yaml::examples test_dsh_003_model_comparison_dashboard +aprender-present-yaml::examples test_dsh_005_experiment_tracker +aprender-present-yaml::examples test_dsh_008_confusion_matrix +aprender-present-yaml::examples test_edg_001_empty_dataset +aprender-present-yaml::examples test_edg_002_large_dataset +aprender-present-yaml::prs_examples individual_examples::test_data_explorer_has_dataset +aprender-present-yaml::prs_examples individual_examples::test_image_classifier_has_bindings +aprender-present-yaml::prs_examples individual_examples::test_minimal_example +aprender-present-yaml::prs_examples individual_examples::test_parameter_tuner_has_sliders +aprender-present-yaml::prs_examples individual_examples::test_sentiment_demo_has_resources +aprender-present-yaml::prs_examples qa_checklist::test_all_examples_pass_qa_checklist +aprender-present-yaml::prs_examples test_all_prs_examples_valid +apr-format::golden_fixtures golden_v2_f32_writer_is_byte_identical diff --git a/docs/specifications/APR-DARK-TARGETS-001-full-surface-cadence.md b/docs/specifications/APR-DARK-TARGETS-001-full-surface-cadence.md new file mode 100644 index 0000000000..2410e592a3 --- /dev/null +++ b/docs/specifications/APR-DARK-TARGETS-001-full-surface-cadence.md @@ -0,0 +1,236 @@ +# APR-DARK-TARGETS-001 — the full test surface: what it costs, what it catches, when it runs + +**Status:** proposed · **Opened:** 2026-09-14 · **Refs:** #3239, #3242, #3248, #3249, #3245 +**Supersedes nothing.** BSE-17 (the quick tier) is correct and this specification does not change it. + +--- + +## 1. The measurement + +Two runs, one hour apart, on 2026-09-14. + +| | tests | binaries | run time | where | +|---|---|---:|---:|---| +| merge-queue full tier (#2838, run `34809681888`) | 82,217 | **73** | 2,226 s (37 min) | `intel-clean-room-14` | +| whole `--workspace --lib --tests` | **98,621** | **727** | 1,141 s (19 min) | lambda-vector, 48 cores | + +```bash +cargo nextest run --workspace --lib --tests --no-fail-fast \ + --exclude aprender-gpu --exclude aprender-cuda-edge --exclude aprender-compute +``` + +**The gap is not "20% more tests" — it is 10× the binaries.** 654 test binaries have never been +*linked* in CI, let alone run. A binary that never links cannot fail, cannot flake, and cannot +appear in a report. That is why this was invisible rather than noisy, and it is the single most +important sentence in this document. + +Nothing else covers them either. `make coverage` — the only other whole-workspace run — is also +`--lib` (`Makefile`: `cargo llvm-cov test --workspace --exclude aprender-gpu --lib`), and +`nightly.yml` is a binary-release workflow that runs no tests at all. + +### 1.1 Cost, isolated — and the link cost is NOT the problem + +Same box, same cache state, cold target dir (`scratchpad/testdelta.sh`): + +| phase | wall | crates | +|---|---:|---:| +| `cargo build --workspace --lib` (what CI already pays) | **76 s** | 715 | +| `+ --tests` incrementally (the delta) | **145 s** | 228 | +| total | **221 s** | 808 binaries, 39 GB | + +**This falsifies the obvious guess, including the one this document made in its first draft.** +650 extra test binaries were assumed to be link-bound and expensive. They are not: two and a half +minutes. The cost of the full surface is almost entirely the *run*, and the run is embarrassingly +parallel. + +### 1.2 What that means on a clean-room runner + +The two boxes are not comparable directly — lambda-vector is 48 cores, and an +`intel-clean-room-*` runner is one of sixteen sharing a 32-core box. Calibrate on measured +throughput instead: + +| | tests/s | +|---|---:| +| lambda-vector | 86.4 | +| intel-clean-room-14 | 36.9 | +| ratio | **2.34×** | + +Extrapolated (and it is an extrapolation, §7.1): the full surface on a clean-room runner is +**≈ 45 min run + ≈ 6 min extra build ≈ 51 min**, against the **37 min** the queue's full tier +costs today. **+14 min, ~+35%.** + +--- + +## 2. What the dark surface actually held + +37 failures, 5 binaries: + +| binary | n | defect | +|---|---:|---| +| `aprender-present-yaml::examples` | 27 | `../../examples/…` resolves to the repo root; the assets are in the **sibling** crate. Dark since APR-MONO | +| `aprender-present-yaml::prs_examples` | 7 | anchored on `CARGO_MANIFEST_DIR`, then walked to the same wrong root | +| `apr-format::golden_fixtures` | 1 | the golden predates #2254's `skip_serializing_if`; 1092 vs 516 bytes | +| `aprender-mcp::falsify_mcp_008` | 1 | the in-crate contract copy ≠ the workspace-root copy | +| `aprender-cgp::falsify` `doctor_speed_real` | 1 | asserts a runner capability — see §5 | + +**Not one is a compute or correctness regression.** Every one is a *global invariant*: a golden +oracle, a path to another crate's assets, a file that must equal another file. + +**That is structural, not luck.** BSE-17 selects by **changed crate**. None of these lives in +anybody's diff, so no selector can reach them. The corollary matters as much as the finding: + +> A PR touching crate X **does** build and run X's integration targets. Change-proximate +> coverage exists and is proportional to the change. The hole is exactly the class that no diff +> can point at. + +### 2.1 The argument is `golden_v2`, not the count + +A golden test exists *only* to catch silent format drift. This one missed a real format change +for two months. #2254 happened to be deliberate and correct; had it been wrong, nothing in this +repository would have said so. + +The risk is not "tests are red". It is **"the alarm was disconnected, and we found out by +accident."** A dark golden is strictly worse than no golden, because its presence is read as +coverage. + +--- + +## 3. Decision: where it runs + +| surface | verdict | why | +|---|---|---| +| **Nightly** | **yes — primary** | ~21 min end-to-end on a 48-core box; no PR or merge latency. | +| **Pre-release (T-1)** | **yes — hard gate** | APR-RELEASE-001 T-1 asserts "green on the cut". With 654 binaries never linked, "green" is a materially weaker claim than it reads. The release is precisely where the whole surface is owed. | +| **Merge queue (full tier)** | **no — but it is close** | ≈ +14 min (+35%) on a 37-min critical path, ×3 parallel groups, against defects that move on a scale of *months*. Queue latency is a measured throughput problem here: main went 8.5 h with no merge on 2026-09-13, and ~2 h on 2026-09-14. 24-hour detection latency is the right trade for this defect class. **Revisit if** queue depth is routinely below `max_entries_to_build` — then the 14 min is spare capacity and the answer flips. | +| **Every PR** | **no** | This is what BSE-17 exists to avoid, and §2 shows the quick tier is not the gap. | +| **PRs touching the SELECTOR** | **yes** | `scripts/ci_test_tier.sh`, `scripts/gate_touched_crates.sh`, `scripts/tree_reader_tests.txt`, and ci.yml's explicit `--test` chain. A change to the selector is exactly when the dark set moves. A `paths:` trigger; fires a few times a month. | + +"Nightly and pre-release **only**" is nearly right, and the selector-touch trigger is the one +addition worth making: it is the cheapest possible guard against this specification silently +ceasing to be true. + +--- + +## 4. It must not become `present` + +A job that is red for months is a job everyone learns to ignore +(`feedback_present_check_is_a_review_backlog_not_a_broken_gate`). At 37 failures it would be +born red. + +**Therefore: start ratcheted, finish hard-failing.** + +1. **Phase A — shrink-only ratchet.** Baseline = the measured failing set, keyed by test id. + A *new* failing id fails the job; a removed one is re-recorded with `--update`. Same shape as + `unwired_guards_baseline.txt` and `pipe_grep_q_baseline.txt`. +2. **Phase B — hard fail.** When the baseline reaches 0, delete it and the `--update` path. + +**Phase B is reachable now, which is why the ratchet is scaffolding rather than a resting place.** +With #3248, #3249 and #3238 landed the count is **1**: `aprender-cgp::falsify +falsify_cgp_061_doctor_speed_real`, another runner-capability assertion (§5). The exit condition +is that one target, not an open-ended backlog. + +--- + +## 5. Precondition: capability gating must be fixed first + +Three targets failed in the *quick* tier on 2026-09-14 for a reason that will also make the +nightly permanently red if it is not fixed first: **a test that needs a box capability was gated +by a cargo feature a sibling crate can enable.** + +`--tests` builds dev-dependencies, so `aprender-orchestrate`'s +`jugar-probar = { …, features = ["browser"] }` armed `aprender-test-lib`'s browser falsifiers on +a browser-less clean-room runner. Feature unification is a property of the **build graph**; +"default features" is a property of a **package**. + +Fixed by giving each a name nothing else asks for — `browser-falsify`, `python-oracle` — with **no +skip introduced**: where the capability feature is on, the test still fails hard without the +capability. + +`scripts/check_feature_gated_test_leak.sh` (#3246) measures the remaining surface: **33 edges over +22 (crate, feature) gates**, 46 of the gated targets being `cuda`. It is shrink-only for the same +reason as §4 — most are probably harmless, and failing on all 33 would assert what the tree has +not earned. + +`aprender-cgp::falsify doctor_speed_real` is the last known instance and the Phase-B blocker. + +--- + +## 6. Shape of the job + +```yaml +name: Full test surface +on: + schedule: [{ cron: '0 4 * * *' }] # after the nightly image rebuild + workflow_dispatch: + pull_request: + paths: # §3, the selector-touch trigger + - 'scripts/ci_test_tier.sh' + - 'scripts/gate_touched_crates.sh' + - 'scripts/tree_reader_tests.txt' + - '.github/workflows/ci.yml' +``` + +Non-negotiables, each earned by a defect this repository has already paid for: + +- **`--no-fail-fast`.** Plain `cargo test` stops at the first failing binary; seven rounds were + spent on four tests before that was understood (`feedback_nextest_fail_fast_hides_dark_failures`). +- **Classify env-death.** Route the cargo invocation through `scripts/cargo_step.sh` (#3241) so a + full disk or an un-spawnable `rustc` reads as `ENV`, not as 650 new defects. Both happened on + 2026-09-14. +- **Vacuity floor.** Fewer than ~600 binaries built ⟹ the *selection* is broken, not the tree. + A sweep that silently runs nothing is the exact failure mode this job exists to end. +- **Not a required check.** It is long and it is not on the merge path. + +--- + +## 7. Open, with owners + +### 7.1 The clean-room figure in §1.2 is an EXTRAPOLATION, not a measurement +It scales one box's throughput by a ratio taken from a single pair of runs. The nightly's +`timeout-minutes` must come from real runs of this job (BSE-05: `T = max(15, ceil(1.5·p99), +p99+20)` over its own history), never from §1.2. Until three runs exist, set it generously and +say so in a comment. + +The first draft of this document asserted the `--tests` delta was "≫ 20 min" on the strength of a +progress check that was reading a stale PID. It is 145 s. The number was already being measured +when the claim was written; the claim should have waited. Recorded here because the same +temptation will recur every time a build "looks" expensive. + +### 7.2 The 329 skipped tests +`nextest` reported 329 skipped in the full sweep vs 131 in the queue's subset. The delta is +unexamined; some are `#[ignore]`, which this repository bans — e.g. +`test_timeout_handling_example` carries `#[ignore] // Timeout handling currently hangs - needs +executor fix`. Each needs a ticket or a removal. + +### 7.3 GPU crates stay excluded +`aprender-gpu`, `aprender-cuda-edge` and `aprender-compute` are excluded here exactly as the full +tier excludes them, because feature unification pulls the CUDA driver in under `--workspace`. +Their targets are owed a separate CUDA-runner sweep; `cuda-nightly.yml` is the natural home. + +### 7.4 `make coverage` is `--lib` too +The coverage floor (88%) is therefore measured over the same partial surface. Whether the floor +should move to the full surface is a separate decision with its own baseline, and it should not +be bundled with this one. + +--- + +## 8. Falsifiers + +Claims here are checkable, and each should be re-run rather than re-read: + +| claim | how to falsify | +|---|---| +| 654 binaries never link in CI | `Starting N tests across M binaries` in any merge-queue `workspace-test` log vs the sweep's | +| the quick tier covers change-proximate targets | touch one file in crate X; the tier prints `crates=… X …` and builds X's `--tests` | +| the defects are global-invariant only | read the five: none is reachable from a diff to the crate that owns it | +| Phase B is reachable | after #3248/#3249/#3238, re-run the sweep; expect exactly one failing id | +| the `--tests` delta is cheap | `scratchpad/testdelta.sh`: 76 s for `--lib`, +145 s for `--tests`, 808 binaries | +| the merge-queue cost is ≈ +35% | re-derive §1.2's ratio from any two same-day runs and redo the arithmetic | + +--- + +## 9. What this does not propose + +- Changing BSE-17 or the quick tier. §2 is the argument that it is not the gap. +- Adding anything to the merge queue's critical path. +- A permanent ratchet. §4 gives the exit condition and names the one target that blocks it. From 1ed5cbefe3d7fccc5bfcf570f41a4af4fbd1c807 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 14 Sep 2026 09:22:51 +0200 Subject: [PATCH 2/4] =?UTF-8?q?docs(spec):=20the=20present-yaml=20fix=20is?= =?UTF-8?q?=20#3250=20=E2=80=94=20this=20document=20took=20#3249?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Written before the PR was opened, so it cited the number this document then got. The three PRs that take the failing set from 37 to 1 are #3248 (golden_v2), #3250 (present-yaml paths) and #3238 (mcp contract copy). Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) --- .../APR-DARK-TARGETS-001-full-surface-cadence.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/specifications/APR-DARK-TARGETS-001-full-surface-cadence.md b/docs/specifications/APR-DARK-TARGETS-001-full-surface-cadence.md index 2410e592a3..cfec276eab 100644 --- a/docs/specifications/APR-DARK-TARGETS-001-full-surface-cadence.md +++ b/docs/specifications/APR-DARK-TARGETS-001-full-surface-cadence.md @@ -1,6 +1,6 @@ # APR-DARK-TARGETS-001 — the full test surface: what it costs, what it catches, when it runs -**Status:** proposed · **Opened:** 2026-09-14 · **Refs:** #3239, #3242, #3248, #3249, #3245 +**Status:** proposed · **Opened:** 2026-09-14 · **Refs:** #3239, #3242, #3248, #3250, #3245 **Supersedes nothing.** BSE-17 (the quick tier) is correct and this specification does not change it. --- @@ -125,7 +125,7 @@ born red. 2. **Phase B — hard fail.** When the baseline reaches 0, delete it and the `--update` path. **Phase B is reachable now, which is why the ratchet is scaffolding rather than a resting place.** -With #3248, #3249 and #3238 landed the count is **1**: `aprender-cgp::falsify +With #3248, #3250 and #3238 landed the count is **1**: `aprender-cgp::falsify falsify_cgp_061_doctor_speed_real`, another runner-capability assertion (§5). The exit condition is that one target, not an open-ended backlog. From 179bddae2c4cbb45c0d5fb69874140ed457a88e8 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 14 Sep 2026 09:23:06 +0200 Subject: [PATCH 3/4] docs(spec): the last stale #3249 in the falsifier table Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) --- .../specifications/APR-DARK-TARGETS-001-full-surface-cadence.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/specifications/APR-DARK-TARGETS-001-full-surface-cadence.md b/docs/specifications/APR-DARK-TARGETS-001-full-surface-cadence.md index cfec276eab..a6ad297856 100644 --- a/docs/specifications/APR-DARK-TARGETS-001-full-surface-cadence.md +++ b/docs/specifications/APR-DARK-TARGETS-001-full-surface-cadence.md @@ -223,7 +223,7 @@ Claims here are checkable, and each should be re-run rather than re-read: | 654 binaries never link in CI | `Starting N tests across M binaries` in any merge-queue `workspace-test` log vs the sweep's | | the quick tier covers change-proximate targets | touch one file in crate X; the tier prints `crates=… X …` and builds X's `--tests` | | the defects are global-invariant only | read the five: none is reachable from a diff to the crate that owns it | -| Phase B is reachable | after #3248/#3249/#3238, re-run the sweep; expect exactly one failing id | +| Phase B is reachable | after #3248/#3250/#3238, re-run the sweep; expect exactly one failing id | | the `--tests` delta is cheap | `scratchpad/testdelta.sh`: 76 s for `--lib`, +145 s for `--tests`, 808 binaries | | the merge-queue cost is ≈ +35% | re-derive §1.2's ratio from any two same-day runs and redo the arithmetic | From 0c6839ca784eaddd291b4cd19640104a20d77c2d Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 14 Sep 2026 10:14:24 +0200 Subject: [PATCH 4/4] docs(spec): a 4-lane quorum returned do-not-implement-as-written, and it was right on three counts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All four agy lanes (3 plan + 1 teamwork) refused the draft. Three findings stand and the spec is revised rather than defended. 1. THE RATCHET HAD NO WRITER. The draft proposed shrink-only, tightening to hard-fail. Three lanes refused it independently and the defect is mechanical, not stylistic: `--update` writes a file and needs a commit, and a scheduled workflow has no PR and no author. Every other ratchet here (unwired_guards, pipe_grep_q, shell_lint) is updated by a human inside a PR BECAUSE it runs in one. A nightly's baseline would only ever be edited by someone who already noticed the job was red -- the exact population the ratchet exists to stop relying on. So: land #3250/#3248/#3238, gate doctor_speed_real, ship hard-failing on day one. The exit condition was already ONE test when the draft was written; there was never an interval in which a ratchet carried information. Fallback if the gating stalls: ONE named exclusion with its issue number, auditable in a line. 2. THE 39 GB TARGET DIR WAS UNBUDGETED. Unanimous, and the draft did not mention it once -- on the same day I spent an hour taking gx10 from 0 bytes free after it dequeued a PR from merge-queue position 1 (#3247). New §4bis makes it a constraint: not on gx10 (916 GB, the box that hit the wall), a per-run target dir reclaimed in-job rather than left to a reaper that already cannot reclaim under keep=3, and a preflight free-space assertion that fails ENV: instead of letting cargo die half-way and read as 650 new defects. 3. THE REVISIT CONDITION WAS WRONG. I keyed it on queue depth vs max_entries_to_build; lane 3 pointed out that is a CAP, not a measure of slack -- a queue can sit at depth 3 with idle capacity. Keyed on arrival rate vs drain rate instead. Also sharpened, from a measurement all lanes converged on: T-1 does not "add the full surface", it must BUILD `ci / deep`. That job has 0 hits in every workflow including the SHA-pinned upstream sovereign-ci.yml, and its other named parts run nowhere either -- `cargo test --doc` 0, `cargo run --example` 0, feature matrix 0, `--no-default-features` 1 hit that is a Windows BUILD flag. And the fast-pool pin leaves this spec entirely (§8bis). All four lanes rejected it with a computable break-even rather than a caveat: `runs-on` is a hard label filter with no fallback, so a pin is non-work-conserving and is a NET LOSS whenever fast-pool wait exceeds the ~18-22 min runtime delta -- which is the condition that holds exactly when the latency mattered. The asymmetry is real (#3251); the actuator is wrong. Lane 4 was briefed as /teamwork but agy 1.2.2 ran ZERO invoke_subagent calls and created no teamwork_projects dir (PMAT-126 reproduced). It is one model answering once and is not counted as a team review anywhere in this document. Refs #3239, #3251 Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) --- ...R-DARK-TARGETS-001-full-surface-cadence.md | 90 +++++++++++++++---- 1 file changed, 73 insertions(+), 17 deletions(-) diff --git a/docs/specifications/APR-DARK-TARGETS-001-full-surface-cadence.md b/docs/specifications/APR-DARK-TARGETS-001-full-surface-cadence.md index a6ad297856..5051be7dcd 100644 --- a/docs/specifications/APR-DARK-TARGETS-001-full-surface-cadence.md +++ b/docs/specifications/APR-DARK-TARGETS-001-full-surface-cadence.md @@ -100,8 +100,8 @@ coverage. | surface | verdict | why | |---|---|---| | **Nightly** | **yes — primary** | ~21 min end-to-end on a 48-core box; no PR or merge latency. | -| **Pre-release (T-1)** | **yes — hard gate** | APR-RELEASE-001 T-1 asserts "green on the cut". With 654 binaries never linked, "green" is a materially weaker claim than it reads. The release is precisely where the whole surface is owed. | -| **Merge queue (full tier)** | **no — but it is close** | ≈ +14 min (+35%) on a 37-min critical path, ×3 parallel groups, against defects that move on a scale of *months*. Queue latency is a measured throughput problem here: main went 8.5 h with no merge on 2026-09-13, and ~2 h on 2026-09-14. 24-hour detection latency is the right trade for this defect class. **Revisit if** queue depth is routinely below `max_entries_to_build` — then the 14 min is spare capacity and the answer flips. | +| **Pre-release (T-1)** | **yes — but the gate must first EXIST** | APR-RELEASE-001 T-1 asserts "green on the cut" via `ci / deep`. Measured: **`ci / deep` has 0 hits in every workflow, including the SHA-pinned upstream `sovereign-ci.yml`.** So do not "add the full surface to T-1" — **build `ci / deep`**, emitting that exact check name, triggered on the cut, and make the full surface one of its parts. Its other named parts run nowhere either: `cargo test --doc` 0 hits, `cargo run --example` 0 hits, feature matrix 0 hits, `--no-default-features` 1 hit and it is a Windows *build* flag in `nightly.yml`. | +| **Merge queue (full tier)** | **no — but it is close** | ≈ +14 min (+35%) on a 37-min critical path, ×3 parallel groups, against defects that move on a scale of *months*. Queue latency is a measured throughput problem here: main went 8.5 h with no merge on 2026-09-13, and ~2 h on 2026-09-14. 24-hour detection latency is the right trade for this defect class. **Revisit if** the ARRIVAL RATE falls below what the queue drains — `max_entries_to_build` is a cap, not a measure of slack, and a queue can sit at depth 3 while idle capacity exists. Key it on arrivals/hour vs merges/hour. | | **Every PR** | **no** | This is what BSE-17 exists to avoid, and §2 shows the quick tier is not the gap. | | **PRs touching the SELECTOR** | **yes** | `scripts/ci_test_tier.sh`, `scripts/gate_touched_crates.sh`, `scripts/tree_reader_tests.txt`, and ci.yml's explicit `--test` chain. A change to the selector is exactly when the dark set moves. A `paths:` trigger; fires a few times a month. | @@ -111,26 +111,60 @@ ceasing to be true. --- -## 4. It must not become `present` +## 4. NO ratchet. Land the fixes, then hard-fail from day one. -A job that is red for months is a job everyone learns to ignore -(`feedback_present_check_is_a_review_backlog_not_a_broken_gate`). At 37 failures it would be -born red. +*(Revised 2026-09-14 after a 4-lane agy quorum returned `do-not-implement-as-written` on all four +lanes. The draft proposed a shrink-only ratchet; three lanes independently refused it, and they +were right.)* -**Therefore: start ratcheted, finish hard-failing.** +The draft's reasoning was: a job born red at 37 failures becomes `present` — ignored within a week +(`feedback_present_check_is_a_review_backlog_not_a_broken_gate`) — so start ratcheted and tighten. -1. **Phase A — shrink-only ratchet.** Baseline = the measured failing set, keyed by test id. - A *new* failing id fails the job; a removed one is re-recorded with `--update`. Same shape as - `unwired_guards_baseline.txt` and `pipe_grep_q_baseline.txt`. -2. **Phase B — hard fail.** When the baseline reaches 0, delete it and the `--update` path. +**The defect in that plan is mechanical, not stylistic: a scheduled workflow cannot update its own +baseline.** `--update` writes a file and needs a commit. Every other ratchet in this repository +(`unwired_guards_baseline.txt`, `pipe_grep_q_baseline.txt`, `shell_lint_baseline.txt`) is updated +by a human inside a PR, because it runs *in* a PR. A nightly cron has no PR and no author. Its +baseline would therefore only ever be edited by someone who noticed the job was red — which is +precisely the population the ratchet exists to stop relying on. -**Phase B is reachable now, which is why the ratchet is scaffolding rather than a resting place.** -With #3248, #3250 and #3238 landed the count is **1**: `aprender-cgp::falsify -falsify_cgp_061_doctor_speed_real`, another runner-capability assertion (§5). The exit condition -is that one target, not an open-ended backlog. +So a ratchet here is not scaffolding. It is a backlog with no writer. + +**The order instead:** + +1. **Land the three fixes.** #3250 (34 rows), #3248 (1), #3238 (1). Each carries its own fix and, + where applicable, its own CI wiring — the wiring never lands ahead of the fix. +2. **Gate `aprender-cgp::falsify falsify_cgp_061_doctor_speed_real`** behind a capability feature, + per §5. It is the last of the 37 and it is the same class as the other three fixed today. +3. **Ship the job hard-failing.** No baseline file, no `--update` path, nothing to maintain. + +The exit condition was already one test when the draft was written. There is no interval during +which a ratchet would have carried real information. + +**If step 2 stalls**, the job ships with that ONE test excluded by name in the workflow, with the +issue number in the comment — a single named exclusion a reader can audit in one line, not a +37-row file nobody reads. That is the fallback, and it is bounded. --- +## 4bis. Budget the disk before scheduling anything + +*(Added 2026-09-14. All four quorum lanes raised this and the draft did not mention it once.)* + +The full-surface build produces a **39 GB target directory** (measured: 808 binaries, +`scratchpad/testdelta.sh`). That is not a footnote here — it is the same resource that took gx10 +to **0 bytes free** on 2026-09-14 and dequeued a PR from merge-queue position 1 (#3247). + +Non-negotiable for this job: + +- **Not on gx10.** It is the smallest of the three Linux hosts (916 GB vs intel 3.6 TB, yoga + 935 GB) and the one that hit the wall. `runs-on` must exclude it or pin away from it. +- **A per-run target dir that is reclaimed in the same job**, not left for the reaper. The reaper's + critical mode already cannot reclaim under `keep=3 per repo` (#3247), so a nightly that leaks + 39 GB compounds a known-unfixed defect. +- **A preflight free-space assertion** that fails the job with `ENV:` rather than letting cargo die + half-way — the failure mode measured twice today is a step reporting a bare exit 101 that reads + as 650 new defects. + ## 5. Precondition: capability gating must be fixed first Three targets failed in the *quick* tier on 2026-09-14 for a reason that will also make the @@ -196,7 +230,7 @@ progress check that was reading a stale PID. It is 145 s. The number was already when the claim was written; the claim should have waited. Recorded here because the same temptation will recur every time a build "looks" expensive. -### 7.2 The 329 skipped tests +### 7.2 The 329 skipped tests — and they must be resolved BEFORE the job hard-fails `nextest` reported 329 skipped in the full sweep vs 131 in the queue's subset. The delta is unexamined; some are `#[ignore]`, which this repository bans — e.g. `test_timeout_handling_example` carries `#[ignore] // Timeout handling currently hangs - needs @@ -229,8 +263,30 @@ Claims here are checkable, and each should be re-run rather than re-read: --- +## 8bis. The fast-pool pin is NOT part of this spec + +A separate proposal — pin the `merge_group` `workspace-test` to the fast gx10/yoga pool, because +14 of 21 clean-room runners are 2.5× slower and two thirds of merges draw one — was reviewed +alongside this document. **All four quorum lanes rejected it as written**, and the objection is +sharper than "risky": + +GitHub `runs-on` is a **hard label filter with no fallback**. A pin is therefore +**non-work-conserving**: pinned to a busy fast pool, the job waits while intel sits idle. That +converts a scheduling preference into head-of-line blocking, and it has a computable break-even — + +> **net loss whenever fast-pool queue wait exceeds the runtime delta (~18–22 min).** + +Which is exactly the condition that holds when the queue is deep, i.e. when the latency mattered. + +It belongs in an infra scheduling specification with that break-even as its gate, not here. The +asymmetry it is reacting to is real and measured (#3251); the actuator is wrong. + +--- + ## 9. What this does not propose - Changing BSE-17 or the quick tier. §2 is the argument that it is not the gap. - Adding anything to the merge queue's critical path. -- A permanent ratchet. §4 gives the exit condition and names the one target that blocks it. +- **A ratchet of any kind.** §4 — a scheduled job cannot commit its own baseline update, so a + ratchet here is a backlog with no writer. Land the fixes first and ship hard-failing. +- Pinning anything to a runner pool. §8bis.