Release process for 0.70+: CRUX-smoke gate on the release binary; full ladder + full CRUX move to nightly; #4033 levers wired into the workflow

Part of #4033.

Operator words (2026-09-23):

"lets update our process for .70 and one of your workers on that epic ensure new optimization is part of workflow and we change nightly for the long releaes"
"anything huge, must be nightly only"

0.69.1 shipped under an OPERATOR EMERGENCY SCOPE: CRUX smoke only. This ticket makes that shape the normal process, with no override needed.

Deliverables:

Release gate (normal, not emergency). On the RELEASE BINARY on every GPU host, run CRUX smoke: the certified models × the modes the certification admits, plus the positive control. Add a delta ladder: only the cells whose inference path the diff touches (certification: carry a cell's receipt forward when the diff since its apr_sha doesn't touch that cell's inference path #4037). Also a 1-model per-host smoke plus the MCP surface digest.
Nightly (the long run).
- Full model ladder (all cells, including 27B) plus full-depth CRUX on main, on lambda and gx10, off-hours, under gpu-q (nightly full ladder + CRUX on main; the release sweep runs only the delta since the last green nightly #4040).
- A release is admitted only if the last green nightly on its merge-base is at most 24 h old; the delta is re-measured.
- A red nightly files an issue automatically.
- Coverage moves to nightly as well (Coverage owes 0.69.1: 87.82% vs floor 88, and COVERAGE_EXCLUDE_REGEX has described a different tree since April #3839, fix(coverage): port #3839 to main — exact skips + --ignore-run-fail so coverage-nightly produces a number again #4023).
Every Epic: faster release certification — ladder + CRUX (operator: "do all", 2026-09-23) #4033 lever wired into the workflow as it lands. Each must have a must-RED proving it can't turn a real RED green:
- ladder: run a cell's CPU and CUDA lanes concurrently (the CPU lane doesn't hold the GPU lock) #4034 CPU and CUDA lanes run concurrently;
- ladder/CRUX: load each model once per cell — one persistent apr serve, every gate as a request #4035 load the model once per cell;
- CRUX: reference cache keyed by model sha + prompt ids + oracle version; only the apr legs run per host #4036 CRUX reference cache;
- certification: carry a cell's receipt forward when the diff since its apr_sha doesn't touch that cell's inference path #4037 carry-forward;
- ladder: CPU==GPU think-block parity on a fixed prefix (e.g. 256 tokens); full 2048-token budget on GPU only #4038 prefix parity;
- sweep: order cells so CRUX-certified models run first (CRUX can start early), then largest first to cut the tail #4039 cell ordering;
- nightly full ladder + CRUX on main; the release sweep runs only the delta since the last green nightly #4040 nightly.
Update the documents that describe the release:
- APR-RELEASE-001 (the release spec);
- the final_sweep runbook;
- check_model_ladder.sh (the scope becomes a normal mode);
- the dogfood/release skill;
- CLAUDE.md's release section.
  Retire phase 2 as a release step.
Measure the target: release-night wall time from cut to crates.io, with 0.69.1 as the baseline (freeze 17:00 → publish ~21:00 CEST). Record it per release.

Prerequisite: #4051. Receipts must stamp per-gate/engine t_start/t_end/lock_wait before any #4033 lever can be judged against the baseline.

#4051 (CRUX half): every CRUX engine call is stamped, and an unstampe…

Gate item G-ONT: ONT-001 complete is a MUST-RED for 0.70.0. This is on the operator's instruction, relayed by the cop, verbatim: "the ontology spec MUST be finished in .7". Owner of the gate: aprender-6c [8b6b78]. Rows: infra-83 owns the infra rows; the aprender-side rows ship in 0.70.0, starting with #4047 (the capability-cells extractor), then #3972, #3847 and the rest of the #3269 set.

The rule. 0.70.0 is NO-GO unless every ONT row in infra/docs/specifications/paiml-ontology.md §5, on both the infra and aprender sides, is closed with its done_when measured:

scripts/ont/precondition-lint.sh docs/specifications/paiml-ontology.md --ledger <the ONT ledger> reports unbound=0 violations=0. Every row is bound in the ledger to a merged sha.
Every row's probe scripts/ont/done_when/ONT-<n>.sh exits 0, run against the release commit (aprender side) and infra main at the cut (infra side). A probe that DECLINEs counts as not done.
The infra spec is read at a pinned sha, recorded in the verdict. An untracked or unpinned spec is itself RED (EPIC E0 0.70: Ontology (ONT-001) #3269's v4.3 lesson).

Measured now (infra main @ 29a84779, precondition-lint with no ledger): rows=27, 27 done_when probes. It also reports 3 UNACCOUNTED paths (ONT-4c1, ONT-4b2, ONT-4c3) and one BLOCKED-EXTERNAL: ONT-4c4 waits on aprender#3522 O-1. The bound count needs the ledger, and infra-83 will post it as the baseline. The gate runs the probes; it never quotes a count.

Falsifiers (built with the gate, in the #4045 release-gate PR):

one fixture row unbound → NO-GO naming it;
one bound row whose done_when exits 1 → NO-GO naming it;
a declined probe → NO-GO, never skipped;
a spec read without a pin → NO-GO;
0 rows parsed → decline (the vacuity form).
Mutants: drop each clause; the named row must go RED.

Blocker to flag now: ONT-4c4 depends on aprender#3522 row O-1 (a real #[kernel] in the workspace). Unless O-1 lands in 0.70.0, G-ONT cannot go green. That is a scope decision for the cop and operator, not something this gate may relax.

ONT-001 ledger-bound baseline for G-ONT (infra-83, measured 2026-09-23 ~20:05 CEST with scripts/ont/precondition-lint.sh over the spec and docs/audits/ONT-001/ledger.jsonl at each commit, extracted with git archive):

infra commit	rows	bound	unbound	probe_paths	declared	blocked
`29a84779` (v4.11 merge)	27	13	14	23	8	ONT-4c4 BLOCKED-EXTERNAL on #3522 O-1
`1c8e0202` (main now)	27	13	14	23	8	same
`PMAT-921-v412` (#950, armed)	28	13	15	26	11	same; adds ONT-4c5 (this repo's #4047)

Correction to the request's premise: there are no UNACCOUNTED rows at 29a84779 or on main. The "3 UNACCOUNTED (ONT-4c1/4b2/4c3)" shape is exactly what this lint prints when it runs without the ledger: every row then reads as unbound, and bound rows that declare nothing (4c1, 4b2, 4c3) report their tracked paths as UNACCOUNTED. If G-ONT reproduces that, check that its --ledger path resolves.

Bound (13): ONT-0 1 2a 2b 4 4b 4b2 4c1 4c3 6 6b D P. Unbound on v4.12 (15): ONT-E (infra), ONT-11 (pmat), and 13 aprender rows: 4c4 4c5 4c 4d 2c 3a 3b 5 4e 7 8 9 10.
Per-row done_when rc was not run here: the probes need live aprender and pmat worktrees plus pv. The selector's discover run is the measurement for that.

M8: SHIFT-LEFT, for 0.70 on. Operator, verbatim, relayed by the cop: "all of these trivial issues should be caught hours early and are "fake" gates update .70 to fix and move this release to crates.io". Owner: aprender-6c [8b6b78].

Rule. Every gate that can block a publish runs continuously on the release branch from the freeze onward, on every push to it and at least hourly, against the candidate sha. The branch is already merged with main, so merge-back gates are measured there too. No publish-blocking gate may be first evaluated at the publish step. The measure: 0 gates first-seen-red at publish, counted per release in the ledger (M7).

Classification. Each gate is one of two kinds:

REAL gates protect users. A red one stops the train (andon).
BOOKKEEPING gates protect the repo's own accounting. They auto-fix, or report and ticket, but never block the publish. They still block merges to main in normal CI, as today.

gate (0.69.1's late surprise)	class	from 0.70
complexity ratchet vs main (10 fns)	bookkeeping	reported on the branch from freeze; a ticket per function; never blocks publish
census / README counts (1831 vs 1833)	bookkeeping	auto-regenerated and committed on the release branch
bashrs findings (8)	bookkeeping (release scripts, not shipped code)	from freeze; justify or fix in-branch; never blocks publish
pmat-comply CB-200	bookkeeping	report + ticket
claim literals	bookkeeping	regenerated from the measured source
ladder / CRUX scope	real	M4: `--scope release`, smoke on the release binary + the admitted nightly (#4040), run from freeze
publish preflight R4 (the squash)	real: what is tagged is what was judged	from freeze on the candidate; any rule whose input is not known before the tag is named, and checked at the tag only
clean-room build of the candidate sha	real: reproducible build	dispatched at freeze and on every candidate change, not at publish
dogfood declared gates	real	from freeze on the candidate binary
publish preflight R1–R3, R5–R7	classified rule by rule in the M8 PR, from its rule table	—

Mechanism (proposal): one release-candidate-watch driver on a self-hosted lane (never hosted runners).

It runs the whole publish-blocking set on the branch head and writes ONE table per run to the release issue: gate, class, verdict, first-red-at.
Bookkeeping gates run their auto-fix and commit it to the branch. Real gates andon.
If this needs a .github/workflows change, I check in with the cop first (standing rule). Otherwise it's a forjar timer like nightly full ladder + CRUX on main; the release sweep runs only the delta since the last green nightly #4040's.

Falsifiers:

a planted bookkeeping red must NOT block the dry-run publish;
a planted real red MUST;
a gate added to the publish path without a class is refused (the list is derived from the publish step's own gate invocations, never hand-kept);
a gate first-seen-red at publish is counted and reported, never hidden.

M8 (CI hygiene): #4102 — a single #4046 shard left 64G of per-run target on gx10 (infra-d3's measurement). An always() cleanup step, then a size lever chosen by measured kind.

#4045 M4: end-to-end rows for the normal release gate -- a RED smoke …

#4045 M6: the 0.70+ release process in the documents that describe it

#4045 M8: shift-left -- every publish-blocking gate, classified real|…

G-ONT gate built: scripts/check_ont_complete.sh (feat/4045-release-gate-normal). Self-test has 9 rows and 8 mutants, each killed by its named row.

Live baseline (aprender-6c [8b6b78]), with infra main @ b8ee0419e PINNED and the real ONT ledger: RED.

rows 28, bound 13, UNBOUND 15, violations 0.
16 done_when probes are unmet against the release tree: ONT-1, 2c, 3a, 3b, 4c, 4c4, 4c5, 4d, 4e, 5, 7, 8, 9, 10, 11, E.
BLOCKED-EXTERNAL: ONT-4c4 waits on aprender#3522 O-1 (in 0.70.0 as a hard dependency, per the cop).

0.70.0 cannot pass G-ONT until all 28 rows are bound and every probe exits 0. The gate runs the probes; it never quotes a count.

#4045 G-ONT: check_ont_complete.sh -- ONT-001 finished on both sides …

#4045 M8: the publish RE-READS the candidate watch verdict (cop rulin…

#4045: guards run correctly bare under guard_tree (CI)

#4045 M8: the gate-class guard is cargo-free, so it runs on every PR …

merge origin/chore/0.69.1-merge-back into #4045 -- the base's cut -> …

merge #4117's squash (#4187) into #4045 over the merge-back merge

fix(#4045): the nightly window is the contract's max_age_h -- it was …

audit(#4045): degraded same-family quorum receipt -- 3/3 PASS on 0e1fd86

roadmap(#4045): entry for the batch/0.70.0 port (M7 + G-ONT; the rest…

fix(#4045): quorum findings -- a crashed lint is not a pass; the unca…

audit(#4045): degraded same-family quorum receipt -- 3/3 PASS on c8c6f7e

fold: #4045 M7 release wall-time + G-ONT check_ont_complete (feat/404…

Parked by APR-EPIC-001: over E1 budget (rank 44/93). Reopen = pull. Applied by aprender-d1: claude+agy both said close, apr shadow row recorded (handoff/close-quorum-2026-09-25.json).

fix(roadmap): B1's three roadmap picks spliced their entries into car…

roadmap: GH-4915, the standing CRUX-smoke release policy

Navigation Menu

Repository navigation

Release process for 0.70+: CRUX-smoke gate on the release binary; full ladder + full CRUX move to nightly; #4033 levers wired into the workflow #4045

4 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Release process for 0.70+: CRUX-smoke gate on the release binary; full ladder + full CRUX move to nightly; #4033 levers wired into the workflow #4045

Description

Activity

noahgift commented on Sep 23, 2026

noahgift commented on Sep 23, 2026

noahgift commented on Sep 23, 2026

noahgift commented on Sep 23, 2026

noahgift commented on Sep 23, 2026

noahgift commented on Sep 23, 2026

4 remaining items

noahgift commented on Sep 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions