From 8887a9331afc7528872e458687f602810d9cefc9 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 14 Sep 2026 12:03:36 +0200 Subject: [PATCH 1/2] ci(deep): the T-1 gate APR-RELEASE-001 requires and nobody wrote MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §4 T-1: "`ci / deep` green on cut sha: full tests, doctests, `--no-default-features`, feature matrix, GPU, every `cargo run --example`". Measured 2026-09-14: `ci / deep` has ZERO hits in every workflow here AND in the SHA-pinned upstream paiml/.github sovereign-ci.yml. The gate the release train asserts on does not exist, and its named clauses run nowhere either: cargo test --doc 0 hits in any workflow cargo run --example 0 hits feature matrix 0 hits --no-default-features 1 hit, a Windows BUILD flag in nightly.yml The 0.67.0 train satisfied T-1 by running pieces of this BY HAND from the autopilot script and writing the numbers into the ledger. Nothing recorded them as a check; nothing would have stopped the train had they been skipped. That is not a criticism of the train — it is what a missing gate looks like from the inside. THE THREE CLAUSES THAT ARE HERE, each with the reading that justifies it: doctests 978 pass, 0 fail, 428 ignored 192 s --no-default-features 0 errors (43 before #3257) 26 s examples BUILD 980 targets, 0 errors 83 s Five minutes for three clauses that have never run. This is a wiring job, not a fixing job — which is only true because #3257 landed the 43-error half an hour ago. WHAT IS DELIBERATELY NOT HERE: full tests #3255 owns it (98,621 tests / 727 binaries, ~21 min) GPU cuda-nightly.yml owns it feature matrix undefined in the spec — a clause with no definition cannot have an implementation every `cargo run --example` 980 targets, NOT attempted. Many are interactive, long-running, or need a model file; running them blind would be a wall-clock assertion with 980 rows, which is the class check_no_timing_in_required.sh exists to stop. The clause needs a definition first. Filed, not guessed. ON THE NAME, because it is a real trade and not an oversight. A check called `ci / deep` can only come from a job named `deep` inside the reusable workflow this repo calls as `ci` — an org-wide change to paiml/.github with blast radius across every consuming repo. This job is named `deep` and emits the check `deep`. Amending the SPEC's string is the cheap half of that trade; amending an org-wide reusable workflow to match a string is the expensive half. APR-RELEASE-001 §4 should read `deep`. Daily as well as on dispatch: a gate that only runs at release time is discovered broken at the worst possible moment. Vacuity floor on the example count (800, under the measured 980), because each clause can pass by measuring nothing — a doctest run that compiled no crate, an examples build that found no example. And every cargo invocation goes through cargo_step.sh: a T-1 lane that reports a full disk as "the release is not green" is how a train gets held for a runner fault. Verified: yaml parses; check_no_hosted_runners, check_runner_labels, check_perf_concurrency_groups, check_no_timing_in_required, check_guards_are_wired and check_tool_versions --audit-workflow all pass on the new file. Refs #3239 Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/deep.yml | 132 +++++++++++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 .github/workflows/deep.yml diff --git a/.github/workflows/deep.yml b/.github/workflows/deep.yml new file mode 100644 index 0000000000..8ecd971c71 --- /dev/null +++ b/.github/workflows/deep.yml @@ -0,0 +1,132 @@ +# deep.yml — the T-1 deep lane that APR-RELEASE-001 requires and nobody wrote. +# +# §4 T-1: "`ci / deep` green on cut sha: full tests, doctests, +# `--no-default-features`, feature matrix, GPU, every `cargo run --example`". +# +# Measured 2026-09-14: `ci / deep` has ZERO hits in every workflow in this +# repository AND in the SHA-pinned upstream `paiml/.github` sovereign-ci.yml. The +# gate the release train asserts on does not exist. Its named clauses run nowhere +# either: +# +# cargo test --doc 0 hits in any workflow +# cargo run --example 0 hits +# feature matrix 0 hits +# --no-default-features 1 hit, and it is a Windows BUILD flag in nightly.yml +# +# The 0.67.0 train satisfied T-1 by running pieces of this BY HAND from the +# autopilot script and writing the numbers into the ledger. Nothing recorded them +# as a check, and nothing would have stopped the train if they had been skipped. +# +# ON THE NAME. The spec says `ci / deep`, and a check called `ci / deep` can only +# come from a job named `deep` inside the reusable workflow this repo calls as +# `ci` — i.e. an org-wide change to paiml/.github with blast radius across every +# consuming repo. This job is named `deep` and emits the check `deep`. Changing +# the SPEC's name is the cheap half of that trade; changing an org-wide reusable +# workflow to match a string is the expensive half. APR-RELEASE-001 §4 should be +# amended to `deep`, and that amendment is part of this PR's description. +# +# WHAT IS AND IS NOT HERE, with the measurement for each: +# +# doctests 978 pass, 0 fail, 428 ignored 192 s HERE +# --no-default-features 0 errors (43 before #3257) 26 s HERE +# examples BUILD 980 targets, 0 errors 83 s HERE +# full tests 98,621 tests / 727 binaries ~21 min #3255 +# every `cargo run --example` 980 targets — NOT attempted, see below +# GPU cuda-nightly.yml owns it +# feature matrix undefined in the spec — §7 of APR-DARK-TARGETS-001 +# +# `cargo run --example` for all 980 is not implemented and is not a lazy +# omission: many examples are interactive, long-running, or need a model file, +# and running them blind would be a wall-clock assertion with 980 rows. The spec +# clause needs a definition before it can have an implementation. Filed rather +# than guessed. +name: deep + +on: + # The train dispatches this against the cut sha (T-1). `ref` is whatever the + # caller checks out, so the cut is tested, not main-at-the-time. + workflow_dispatch: + # Daily as well, so the lane cannot rot silently between trains — a gate that + # only ever runs at release time is discovered broken at the worst moment. + schedule: + - cron: '10 7 * * *' + +permissions: + contents: read + +concurrency: + group: deep-${{ github.ref }} + cancel-in-progress: false + +jobs: + deep: + name: deep + runs-on: [self-hosted, X64, Linux, clean-room] + # 40, and NOT derived. The three clauses measured ~5 minutes combined on a + # 48-core box with a warm sccache; a clean-room runner measured 2.34x slower + # per test on the same workload. That extrapolates to ~12 min, so 40 is + # generous on purpose. BSE-05 (T = max(15, ceil(1.5*p99), p99+20)) wants the + # job's OWN history and this job has none — re-derive from three real runs. + timeout-minutes: 40 + env: + CARGO_TERM_COLOR: never + CARGO_INCREMENTAL: '0' + steps: + - uses: actions/checkout@v7 + + # Classify a cargo death as ENV or CODE before anything reads it as a + # release blocker. A T-1 lane that reports a full disk as "the release is + # not green" is how a train gets held for a runner fault. + - name: cargo_step.sh falsifier (ENV-vs-CODE instrument) + run: bash scripts/cargo_step.sh --self-test + + # CLAUSE 1 — doctests. 978 pass / 0 fail / 428 ignored, 192 s (measured + # 2026-09-14 on the workspace, GPU crates excluded as the full tier + # excludes them). `cargo test --doc` has never run in CI. + - name: "T-1 clause: doctests" + run: | + set -euo pipefail + bash scripts/cargo_step.sh 'doctests' -- \ + cargo test --doc --workspace --no-fail-fast \ + --exclude aprender-gpu --exclude aprender-cuda-edge --exclude aprender-compute + + # CLAUSE 2 — --no-default-features. 43 errors until #3257, every one in + # aprender-distribute (#3176): tokio declared optional and used + # unconditionally. `check`, not `test`: without default features most + # crates expose no runnable surface, so a test run would assert nothing. + - name: "T-1 clause: --no-default-features" + run: | + set -euo pipefail + bash scripts/cargo_step.sh '--no-default-features' -- \ + cargo check --workspace --no-default-features --locked \ + --exclude aprender-gpu --exclude aprender-cuda-edge --exclude aprender-compute + + # CLAUSE 3 — examples BUILD. 980 targets, 0 errors, 83 s. `--keep-going` + # so one broken example names itself instead of hiding the other 979 — + # the same reason the full surface uses --no-fail-fast. + - name: "T-1 clause: every example BUILDS (980 targets)" + run: | + set -euo pipefail + bash scripts/cargo_step.sh 'examples build' -- \ + cargo build --workspace --examples --keep-going --locked \ + --exclude aprender-gpu --exclude aprender-cuda-edge --exclude aprender-compute + + # VACUITY. Each clause above can pass by measuring nothing — a doctest run + # that compiled no crate, an examples build that found no example. Assert + # the surfaces are the size they were measured at, with floors well under + # the readings (978 doctests, 980 examples) so ordinary growth does not + # trip them. + - name: Vacuity floor — each clause must have measured something + if: always() + run: | + set -euo pipefail + ex=$(cargo metadata --no-deps --format-version 1 | python3 -c " + import json,sys + d=json.load(sys.stdin) + print(sum(1 for p in d['packages'] for t in p['targets'] if 'example' in t['kind']))") + echo "example targets discovered: $ex" + if [ "${ex:-0}" -lt 800 ]; then + echo "::error::only $ex example targets found, expected 800+ (measured: 980)." + echo "::error::the DISCOVERY is broken, not the tree." + exit 1 + fi From 058252707bd631bd65cbd1966f405f53e1cb840a Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 14 Sep 2026 12:10:44 +0200 Subject: [PATCH 2/2] ci(deep): I wrote the examples-run clause, measured it, and took it back out MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The operator's challenge was right and I had argued from assumption: slowness is what a nightly is FOR, and `.config/nextest.toml` already answers HANGING with `slow-timeout = { period = "60s", terminate-after = 20 }` so "a genuine hang dies at 20 minutes WITH A NAME". `check_examples_run` in aprender-orchestrate even sidesteps stdin the obvious way: `cargo run --example X -- --help`. So I wrote the clause. Then ran it on a random sample of five: HANG aprender-core::grid_search_tuning HANG aprender-serve::imp_700_realworld_verification exit101 aprender-serve::debug_q4k_gemv ok aprender-simulate::reproducibility HANG aprender-serve::bench_barrier_overhead THREE OF FIVE ran past 60 s on `--help`. These are not CLIs — they are compute demos that ignore argv and do the work. `--help` does not make them exit, and "hang" is the wrong word: they are RUNNING. A per-example timeout would report a working example as a FAILURE, at 980 rows. Five is a small sample and the ratio is not the claim; the KIND is, and the three names are self-evidently long-running demos. So "every `cargo run --example`" cannot be implemented as the spec words it against this workspace. Two shapes that could work are written into the header for whoever decides: redefine as "starts and does not crash within N seconds" with timeout treated as PASS (~2 h at N=10, so it belongs in #3255's full-surface lane, not this 5-minute one), or have examples DECLARE runnability. Clause 3 — every example BUILDS, 980 targets, 83 s — already catches the common rot, which is compile breakage. That is the part that is cheap and unambiguous, and it stays. Also recorded: `check_examples_run` is worth a ticket of its own. Its only callers are its own tests, and it fails OPEN — a non-zero runtime exit counts as a pass because "--help exiting non-zero is OK", so a crash passes. The feature matrix stays out and that one is genuinely undefined: cargo-hack is not installed and nothing here references --each-feature, --feature-powerset or any matrix tooling. No definition and no tool. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/deep.yml | 58 +++++++++++++++++++++++++++++++++----- 1 file changed, 51 insertions(+), 7 deletions(-) diff --git a/.github/workflows/deep.yml b/.github/workflows/deep.yml index 8ecd971c71..b3dc056393 100644 --- a/.github/workflows/deep.yml +++ b/.github/workflows/deep.yml @@ -30,16 +30,60 @@ # doctests 978 pass, 0 fail, 428 ignored 192 s HERE # --no-default-features 0 errors (43 before #3257) 26 s HERE # examples BUILD 980 targets, 0 errors 83 s HERE +# every `cargo run --example` 980 targets — NOT here, and MEASURED why # full tests 98,621 tests / 727 binaries ~21 min #3255 -# every `cargo run --example` 980 targets — NOT attempted, see below # GPU cuda-nightly.yml owns it -# feature matrix undefined in the spec — §7 of APR-DARK-TARGETS-001 +# feature matrix undefined AND untooled — see below # -# `cargo run --example` for all 980 is not implemented and is not a lazy -# omission: many examples are interactive, long-running, or need a model file, -# and running them blind would be a wall-clock assertion with 980 rows. The spec -# clause needs a definition before it can have an implementation. Filed rather -# than guessed. +# RUNNING THE EXAMPLES — measured, then dropped, and the measurement is the point. +# +# The objection "they are interactive or long-running" is half wrong: slowness is +# what a nightly is FOR, and `.config/nextest.toml` already answers hanging with +# `slow-timeout = { period = "60s", terminate-after = 20 }` so "a genuine hang +# dies at 20 minutes WITH A NAME". `check_examples_run` in aprender-orchestrate +# even sidesteps stdin the obvious way, `cargo run --example X -- --help`. +# +# So I wrote the clause. Then ran it on a random sample of five: +# +# HANG aprender-core::grid_search_tuning +# HANG aprender-serve::imp_700_realworld_verification +# exit101 aprender-serve::debug_q4k_gemv +# ok aprender-simulate::reproducibility +# HANG aprender-serve::bench_barrier_overhead +# +# THREE OF FIVE ran past 60 s on `--help`. These are not CLIs — they are compute +# demos that ignore argv and do the work. `--help` does not make them exit, and +# "hang" is the wrong word: they are RUNNING. A per-example timeout would report +# a working example as a failure, at 980 rows. +# +# (Five is a small sample and the ratio is not the claim; the KIND is. The three +# names are self-evidently long-running demos.) +# +# So the clause as the spec words it — "every `cargo run --example`" — cannot be +# implemented as written against this workspace. What could work, and needs a +# decision rather than a guess: +# +# (a) redefine it as "every example STARTS and does not crash within N +# seconds", timeout treated as PASS. Catches the #2519 class (an example +# that panics immediately) without asserting duration. Cost: ~980 x +# min(runtime, N) — a nightly-scale job, ~2 h at N=10, so it belongs in +# #3255's full-surface lane, not in this 5-minute one. +# (b) have examples DECLARE runnability (a `required-features` convention or a +# manifest list), and run only those. +# +# Clause 3 above — every example BUILDS — already catches the common rot, which +# is compile breakage. It is the part that is cheap and unambiguous. +# +# `check_examples_run` is worth a ticket of its own regardless: its only callers +# are its own tests (releaser_preflight/tests.rs), and it fails OPEN — a non-zero +# runtime exit counts as a pass on the grounds that "--help exiting non-zero is +# OK", so a crash passes. +# +# The feature matrix stays out too, and that one is genuinely undefined: +# `cargo-hack` is not installed and nothing in this repository references +# --each-feature, --feature-powerset or any matrix tooling. No definition and no +# tool, so nothing to place. `cargo hack --each-feature` nightly is the obvious +# shape once someone decides it. name: deep on: