Skip to content

feat(release): rc tagger gates on fleet cells before tagging + #4314 salvage (Refs #4327) - #4463

Closed
noahgift wants to merge 26 commits into
mainfrom
feat/4327-rc-tagger
Closed

noahgift wants to merge 26 commits into
mainfrom
feat/4327-rc-tagger

Conversation

@noahgift

Copy link
Copy Markdown
Member

Refs #4327

One PR for the rc tagger plus the #4314 salvage. #4314 was closed and its salvage commits are carried here (cop ruling 2026-09-25).

What

Quorum receipts

Verification

bash scripts/release/rc_tag_main.sh --self-test passes. It covers:

  • the decision table, including the ?unread row;
  • fake-API end-to-end runs: green, RED cell, existing tag, 422, and an unreadable tag list;
  • a live run with each override set → rc 2;
  • the gate-bypass mutant, which turns it RED.

bash scripts/release/fleet_cells_gate.sh --self-test also passes.

🤖 Generated with Claude Code

noahgift and others added 25 commits September 25, 2026 21:13
…mit (#4275)

The smoke tested `case "$v" in *"${TAG#v}"*)`, a substring match. It refused
every rc (an rc asset reports 0.69.3, which does not contain 0.69.3-rc.1:
Cargo stays X.Y.Z, cop ruling 2026-09-24) and it accepted 0.69.30 for v0.69.3.

scripts/release/asset_version_check.sh TAG COMMIT "VERSION_LINE" compares the
version whole, with the tag's -rc.N stripped, and requires the asset's sha to
be a prefix of the tag's commit. That is what binds an rc asset to ITS commit.
Case table: 14 rows (the issue's 4 plus 0.69.30, no-git, short/empty sha, a
-rc Cargo version, a non-full commit, a pv line). Six mutants, all killed.

The sha had to be made real first. The published v0.69.3-rc.1 cpu asset
reports `apr 0.69.3 (v0.69.3+no-git)`: the bullseye container sees no usable
git, so no release binary has ever carried its commit. Both apr build lanes
now pass APR_GIT_SHA_OVERRIDE=$(git rev-parse --short=9 HEAD) (build.rs's
first source; it is rerun-if-env-changed, so the cached target rebuilds).

Closes #4275

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 8d2cbe2)
…rebuild (#4286)

scripts/release/promote_rc.sh vX.Y.Z-rc.N [--dry-run] | --self-test
- refuses unless the rc is a prerelease, the final does not exist (nor a draft),
  and check_release_assets.sh says the rc's asset set is complete
- downloads every rc tarball, verifies it against its .sha256, renames it to the
  final name (the -rc.N label must occur exactly once), writes a same-hash .sha256,
  creates a DRAFT final at the rc's commit, uploads, reads every asset back and
  compares hashes, and only then publishes it
- live --dry-run on v0.69.3-rc.1: 8 tarballs staged, cmp byte-equal, final
  .sha256 hash == rc hash

Findings fixed on the way:
- a promoted tarball keeps its rc inner directory (apr-v0.69.3-rc.1-.../apr).
  install.sh, the binary-release smoke and autopilot.sh assumed ${asset}/apr, so
  a byte-identical promotion broke installs (hermetic fake-curl test: RED
  "Binary 'apr' not found in archive" before, installs apr 0.69.3 after; an
  archive with two apr-* dirs is refused). All three now take the single apr-* dir.
- publishing the promoted final fires `release: published`, whose uploads would
  clobber the rc bytes with a rebuild. binary-release.yml gains an `assets` gate
  job: builds run only when check_release_assets.sh says assets are missing; an
  ENV error fails the gate, so the builds skip (fail closed). Smokes run on
  !cancelled() && verify success, so they still check promoted assets.

Refs #4286 — "released apr --version names the rc" needs #4110 provenance;
identical bytes cannot carry a new label, so the release body records the rc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 804e81d)
…rtial (#4286)

The assets gate skips every build when the release already holds all its
assets, so needs.build.result is skipped and the summary printed
'partial (status=skipped)' on every promoted release. Say it was not rebuilt.
Quorum finding (sonnet lane, PR #4314).

Refs #4286

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 0b3463d)
…+no-git (#4110)

The 0.69.1 crates.io install AND the release binaries both printed
`apr 0.69.1 (v0.69.1+no-git)` (0.68.1 did too). There were two causes.

(a) crates.io: the tarball carries `.cargo_vcs_info.json` (git.sha1 =
d8a6df5…), which `cargo package` writes, but build.rs never read it. It
is now rung 2, AHEAD of `git rev-parse`, because an unpacked crate can sit
inside an unrelated repository whose HEAD is not ours. The value is short
9-hex, with -dirty when cargo recorded a dirty tree; anything that is not
40 hex is ignored.

(b) release binaries: binary-release.yml builds as root in a container
over the runner-owned checkout, and git refuses it as "dubious ownership".
Reproduced in rust:1.93.0-bookworm: main's build script prints
v0.69.1+no-git and this one prints the HEAD. The git rung now retries with
ONLY the enclosing checkout marked safe.directory, via a private
GIT_CONFIG_GLOBAL under OUT_DIR. `-c safe.directory` is ignored by git
before 2.39 (measured on 2.34.1). This needs no workflow edit.

Must-RED: crates/apr-cli/src/build_rs_vcs_sha_tests.rs compiles the
SHIPPED build.rs with rustc and runs it as cargo does. Rows: packaged
(byte-exact 0.69.1 vcs file), foreign-repo, dirty, malformed, no-vcs,
override, dev-checkout, and foreign-owner (via git's
GIT_TEST_ASSUME_DIFFERENT_OWNER, with a precondition that plain git really
refuses). Three plants must each go RED: vcs rung removed, vcs rung behind
git, retry removed. The tests run in workspace-test (apr-cli --lib).
Contract apr-version-traceability-v1 goes to 1.2.0 with FALSIFY-VERSION-005.

Also: Cargo.lock chacha20 0.10.0 (yanked) -> 0.10.2. It is lock-only; no
default-graph package builds it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f1a97f7)
…ted exactly (#4110, #4256, #4290)

The v0.69.3-rc.2 asset prints `apr 0.69.3 (v0.69.3+no-git)`: no rc, no commit.
Operator 2026-09-24: "we need actual version numbers", "version number needs
release canidate info in it".

- scripts/release/stamp_rc_version.sh: binary-release.yml stamps the checked-out
  rc tag tree to X.Y.Z-rc.N (workspace version, every path pin incl. partial
  "0.69" ones, workspace Cargo.lock entries) before `cargo build --locked`, so
  CARGO_PKG_VERSION itself carries the rc. Final tags are a no-op; a tag whose
  X.Y.Z is not the workspace version is refused. Self-test: cargo metadata
  --locked is the oracle, 4 mutants (lock / path-first pins / version-first
  pins / partial pins dropped) each make cargo refuse. Real tree: the
  v0.69.3-rc.2 archive stamps and resolves --locked, 79/79 members at rc.2.
- asset_version_check.sh: the printed version must equal the tag EXACTLY,
  -rc.N included; bare 0.69.3 on an rc tag is refused. Against the real rc.2
  asset: "bad asset reports 0.69.3; tag v0.69.3-rc.2 wants 0.69.3-rc.2".
- #4290: the F2 receipt key is the build, not the version: version plus the
  running executable's sha256 (catches rc.N, dev and dirty rebuilds); an
  unreadable executable gets a key no receipt can hold.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 2b3cb55)
…t while a fleet cell is RED (#4327, #4328)

Operator, 2026-09-24: "THE INSTANT we do a release candidate it needs to be on all
hardware either before or same time. change process".

C2 (#4327):
- rc_cut.sh creates the rc as a DRAFT prerelease. Drafts are invisible to install.sh
  and to the fleet poller.
- binary-release.yml reads the release by listing it, because releases/tags/ never
  returns a draft.
- binary-release.yml builds a fifth apr asset, aarch64-apple-darwin-cpu, on the mini
  runner. It is stamped and verified like the others, so mini can install an rc at
  all. check_release_assets now expects 18 assets.
- rc_fleet_stage.sh (new) stages the rc on every host: lambda, gx10, yoga, intel and
  mini (jetson is retired). For each host it copies the assets and installs them,
  with `fleet-bins install` or an atomic built-in installer. It then verifies what
  PATH serves with asset_version_check (apr X.Y.Z-rc.N (<sha>) of the tag commit)
  and checks pv's version.
- It publishes only when every host passes. An unreachable host holds the rc unless
  a dated waiver covers it. It prints a per-host receipt.
- Self-test:
  - a verdict table;
  - two verdict mutants;
  - a fake three-host fleet in which one lying installer holds the publish;
  - a mutant with the verify deleted, which publishes past that host.

C4 (#4328): fleet_cells_gate.sh (new) refuses the cut while any fleet cell is RED
without a dated, recorded waiver (scripts/release/fleet-waivers.tsv).
- It also refuses missing or stale cells.
- rc_cut.sh runs it before writing the tag.
- The cells come from infra-64's C3 monitor, at fleet/cells.tsv on the fleet-state
  branch.
- Self-test: a 14-row table, 2 mutants, and a wiring row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 90b070f)
…ame (#4327)

releases/tags/ never returns a draft. A promote run before rc_fleet_stage.sh
published the rc therefore reported "cannot read release" as an ENV error, when
the real state was that the rc was not on the fleet yet. It now finds the rc by
tag_name and refuses a draft with the staging command to run. Found by the
sonnet review lane on 90b070f.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit ca7332f)
…e over the ceiling of 74 (#4327 line, surfaced on #4314)

grep -qx now reads a process substitution. Self-test 12/12; the planted
darwin-name mutant turns row 12 RED (11/12). check_no_pipe_into_grep_q 74/74.

Pmat-Ticket: PMAT-4275

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 15b03ae)
…n_pinned read it as a bare apr call (guard-cargo red on #4314)

The mutant deletes the asset_version_check.sh call, so the label now names
that. check_apr_bin_pinned OK (357 files); rc_fleet_stage --self-test 14 ok, rc 0.

Pmat-Ticket: PMAT-4275

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 7d06cad)
…ed — cuda-unit did not compile

OwnedQuantizedLayer gained post_attn_norm_weight/post_ffw_norm_weight and
GGUFConfig gained query_pre_attn_scalar; this cuda-only test target was never
updated, so cuda-unit failed E0063 x10. Same file is red on origin/main.

Pmat-Ticket: PMAT-4275

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit dffcfef)
…te goes RED without them (#4353)

Operator 2026-09-25: "ensure these gaps never occur again as mac is critical
and also wgpu enabled binary for intel".

- build-pv-darwin: native pv build on the mini runner, pv-<tag>-aarch64-apple-darwin.
- build-apr-cpu gains a flavour axis; the x86_64 wgpu row builds with --features wgpu
  (apr-<tag>-x86_64-unknown-linux-gnu-wgpu). Flavour is proven by BEHAVIOUR: a cpu
  build refuses `apr run --gpu` (exit 9), a GPU build does not (exit 3 on the missing
  model). A byte grep for wgpu cannot tell them apart (125-127 hits in every build).
  Both polarities are asserted, so each lane is the other's positive control.
- check_release_assets.sh expects 22 assets (6 apr + 6 sha + 10 pv); new must-fail
  rows for darwin pv and the wgpu apr. There is no waiver path.
- Fix: `printf | grep -qxF` under pipefail read a PRESENT asset as MISSING when
  grep -q closed the pipe early (SIGPIPE 141) — 5/30 self-test runs red at 22 lines.
- tests/check_release_assets_test.sh: its hand-written complete set lacked the
  darwin apr (#4327), so row 3 was red on the base; now carries all 22 + mutants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 9c35d85)
…ck_runner_labels.sh requires (#4353)

[self-hosted, macOS, ARM64, mini] has none of apple-silicon|m4, so guard-tree
failed on build-apr-darwin and build-pv-darwin. Same selector as ci.yml mac-check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit fbca016)
…stops fetching a check it never runs (#4353 quorum nits)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit adb895c)
…the superseded rc_cut.sh (#4314 salvage)

rc_cut.sh was dropped with #4314 (superseded by the 16:58 rc-on-main ruling).
The row stays RED until the rc tagger calls this gate: an unwired gate blocks
nothing. RC_TAGGER points it at the tagger once one exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mments stop citing rc_cut.sh (#4314 salvage, quorum r1)

Gemini lane r1 (conv 2da40e08): with rc_cut.sh gone nothing creates the draft,
so the listing lookup raised StopIteration. main's releases/tags/ lookup 404s
the same way on a bare tag, so this is not a regression; the rc tagger owns
draft creation now. All 7 lookups exit naming the tag and that contract.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#4314 salvage, quorum r2)

Haiku lane r2: the two download-side asset lookups kept a bare next(). A miss
already failed the step; now it names the missing asset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ound 3, not degraded

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g hardcoded in prose (#4314 salvage, quorum r4)

agy lane r4 (conv 8ba6854b): smoke-cpu fetched only -cpu, so #4353's wgpu
asset never got #4275's tag/commit binding. The smoke matrix now carries a
flavour and a clean-room wgpu row. The '16'/'18' asset counts in comments
(22 now) are replaced by a pointer to check_release_assets.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s fail by name (#4314 salvage, quorum r5)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r drafts, then dispatches (#4314 salvage, quorum r6)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6f83d92

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ate before it writes the tag (Refs #4327)

Under the 2026-09-25 16:58 ruling an rc is a tag on queue-green main, and rc_cut.sh
was dropped. That left fleet_cells_gate.sh with no tagger to call it, so its WIRING
row was RED. rc_tag_main.sh runs its steps in this order:
- judge: the sha is on main, `ci / gate` succeeded, Cargo reads X.Y.Z, the tag is new
- fleet gate
- git/refs tag
- DRAFT prerelease release
- dispatch binary-release.yml

The WIRING row now defaults to rc_tag_main.sh.

Self-test:
- a 13-row decision table
- a fake-API end-to-end (write order, RED cell, existing tag, 422)
- a gate-bypass mutant that tags past RED, proving the gate is load-bearing
- a large Cargo.toml fixture, which kills the pipefail/SIGPIPE version-read mutant

A live dry-run on main shows the real cells refuse a cut today because the RED
cells have no waivers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t overrides refused on live runs (Refs #4327)

From the agy quorum lane on 32d991b. That lane was voided by exit 3 (other sessions' ref churn), but two of its findings were real:
- A failed matching-refs read set D_TAG_AT='' and was judged as "tag is new". It now reads '?unread' and is refused.
- RC_TAG_HERE/CELLS/NOW were honoured on live runs, so a green cells file could get a real tag past the fleet gate. A live run now exits 2 when any of them is set.

Both new self-test rows are mutant-checked: reverting either fix turns the self-test FAIL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… ruling 2026-09-25; Refs #4327)

The waivers cover forjar#639, bashrs#393, pzsh#34, copia#73/74, pmat#1428 and apex#153/154, plus lambda llama.cpp until infra#1155 deploys. All expire 2026-09-27.
There is NO waiver for apr/pv: pv no-sha is #4219, which must go green.
Against the 20:32Z cells, the gate now refuses on only pv x4 and lambda-labs/rmedia.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (cop ruling 2026-09-25; Refs #4327)

- The pv waiver expires 2026-09-26 and must be deleted at the 0.70.0 fleet install.
- llama.cpp now carries the reason from infra-64.
- The gate now passes against the 20:32Z cells.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=4463 head=2a68a75af12f202a310f5dc152e6233807a949b3 verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

…resh session finishes a killed cut (#4327)

lib_cut_receipt.sh: one ledger row `intent` before each write the world sees (tag, draft,
dispatch, each host's deploy, publish) and `done` after. Ledger outside any worktree.

- rc_tag_main.sh: a tag at THIS sha that the ledger says this cut wrote is `resume` (never
  re-tagged, fleet not re-judged); draft/dispatch skipped when done, and after a kill asked
  of GitHub (release list / workflow_dispatch runs since the intent row). A tag the ledger
  does not know still refuses.
- rc_fleet_stage.sh: a host whose deploy is done is re-verified only; an asset already on a
  host at its sha256 (asset + .sha256) is not copied again; a publish that landed is recorded,
  never re-edited (notes not appended twice).
- rc_cut_resumable.sh: tag -> bounded asset wait -> stage --publish, one command.

Falsifier (rc_cut_resumable.sh --self-test): kill -9 after the tag, before the first deploy;
a fresh session finishes with 1 tag, 1 draft, 1 dispatch, 0 duplicate copies, 1 publish,
3/3 hosts on the rc sha. Same for kills after:tag/draft/install:<host>/publish. Mutants:
per-session ledger strands the cut (rc 1, never re-tags); resume check removed strands it;
sha256 skip removed copies twice. RC_CUT_FAULT is refused on every live run.

Self-tests: rc_tag_main PASS, rc_fleet_stage PASS (17 ok), rc_cut_resumable PASS (9 ok, ~55 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@noahgift

Copy link
Copy Markdown
Member Author

Superseded by #4429 (cop ruling D, 2026-09-27): car/0.70.0 carries newer versions of 22 of the 26 files this PR touches, and #4110's build.rs commit is superseded there by aprender-build-sha (#4219). The unique payload (rc_tag_main.sh, rc_cut_resumable.sh, lib_cut_receipt.sh) is salvaged on e6/4463-salvage @0eb2eeabc9, off car@40d5d0a485, plus a fixture fix (the self-test was reading the live fleet-waivers.tsv). Branch feat/4327-rc-tagger is kept.

@noahgift noahgift closed this Sep 27, 2026
@noahgift
noahgift deleted the feat/4327-rc-tagger branch October 9, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant