Repository navigation
release(rc): an RC is not shipped until every fleet host runs it (install before or with publish) #4327
Closed as not planned
Description
OPERATOR, verbatim (2026-09-24): "THE INSTANT we do a release candidate it needs to be on all hardware either before or same time. change process"
Part of EPIC #4288. Today v0.69.3-rc.2 shipped while lambda still runs rc.1, plus a ~/.cargo/bin/apr "+no-git" shadow. The fleet install runs on an hourly timer, separate from the cut.
New process
- Stage before publish. The rc-cut (release(rc): cut rc on green, promote rc bytes to final, publish gate at the rc — batch #4285 #4275 #4286 #4287 #4314 /
PMAT-4285-rc-cut) builds the assets, then callsfleet-install <tag> <asset-dir>on every fleet host: lambda, gx10 (aarch64), yoga, intel, mini, and jetson when reachable. The install is atomic, verifies sha256 and quarantines shadows. - Verify. On each host, the
apr --versionthat PATH resolves must equal<tag> (<sha>)(apr --version from crates.io install prints +no-git: build.rs ignores .cargo_vcs_info.json #4110/release assets print (vX+no-git): apr_bin.sh STRICT refuses every published apr binary #4256).pvis checked the same way. - Publish only on N/N.
gh release createruns only after every reachable host passes. An unreachable host is recorded as a named RED row, never skipped silently. - Backstop. The fleet-bins poller checks releases every ≤2 min and installs any newer rc or release it finds, for manually cut releases.
- The "SHIPPED" receipt must include a per-host table: host, arch, tag, sha, and path resolved.
Acceptance
- A mutation proves it: a cut where one host fails verify does NOT publish.
- The next RC (rc.3 or 0.69.4-rc.1) ships with 6/6, or with the unreachable hosts named RED.
Activity
noahgift commentedon Sep 24, 2026
Measured 17:58Z (infra-64, evidence/fleet-versions.md): mini (Darwin arm64) cannot take rc.2 because the release ships no aarch64-apple-darwin asset. "All hardware" means the rc-cut must build darwin-arm64 too; otherwise mini is RED on every RC. intel still runs pre-rc.2 apr and pv 0.69.1, because fleet-bins is not deployed there yet (infra-64 is on it). jetson: no route to host.
noahgift commentedon Sep 24, 2026
Host list update. OPERATOR, verbatim, 2026-09-24: "jetson is not online anymore update records". jetson is RETIRED and dropped from the fleet-install host set. Active hosts: lambda, gx10, yoga, intel, mini.
feat(release): an rc is a draft until every fleet host runs it; no cu…
fix(promote): read the rc by listing and refuse a still-draft rc by n…
merge PMAT-4285-rc-cut into batch/0.70.0: version identity (#4110/#4256/
noahgift commentedon Sep 24, 2026
infra side landed on
paiml/infrabranchfleetbins/measure(to be folded into infra#1052):66e41e84,32d0b796.Entrypoint
machines/fleet-hosts/fleet-bins/fleet-install.sh <tag> <asset-dir> [<commit>]. It runs on lambda-labs, the host with the fleet ssh mesh.For each active host (lambda-labs gx10 yoga intel mini), it:
uname -sm. No answer is a REDunreachablerow, never a skip.*-<triple>.tar.gz+.sha256. None for the arch is a REDno-asset-for-<triple>row. This is mini today, until anaarch64-apple-darwinasset exists.fleet-bins installthere. That step verifies the sha256, swaps the binary in atomically, refuses a downgrade, and checks that theapron PATH is the same bytes reporting the tag and commit.Exit 0 only when every host answered, every install exited 0 and every row is
ok. Anything else is nonzero, so the rc-cut must not publish.Receipt: stdout is a TSV table (
host arch bin tag sha path verdict). SetFLEET_INSTALL_RECEIPT=<file>to also write it as a markdown table withhosts ok N/M.rc-cut wiring (aprender side, needs quorum): after assets are built and before
gh release create/cargo publish:It is an ordering change only; the build itself does not change.
Poller (backstop for a host that missed the push):
fleet-bins-poll.timerfires 2 min after the previous poll finishes (OnUnitActiveSec=2min). It runsgit ls-remoteper catalogue row and runs the resolver only when a tag moved. It is live on lambda-labs, gx10, yoga and intel.Tests:
make -C machines/fleet-hosts test-fleet-binsgives fleet-install 12 rows with 10 mutants killed, and fleet-bins 25 mutants killed.Known RED today: mini (no darwin asset). Also, the aprender nightly is 35h stale, so the hourly run exits 1 by design.
noahgift commentedon Sep 24, 2026
Live run against the published v0.69.3-rc.2, all 5 active hosts: 0/5, exit 1. Receipt: paiml/infra
docs/audits/fleet-bins/*-fleet-install-v0.69.3-rc.2.mdon branchfleetbins/measure(a7cec34d).apr --version/pv --versionname no commit, because it was built without git metadata. aprender#4256 now blocks every rc from passing this gate, as intended. Fix the build and the same run turns green.aarch64-apple-darwinasset exists.-gnu-cpu/-gnu-cuda, so the exact-triple match copied pv alone, and the host skipped apr and exited 0. Now every OS+arch variant ships, and a repo's bin missing from the cut is RED (fd54de65, with 2 new mutants).fold(#4292): darwin release lane on the mini-builder runner — check_r…
fix(guard): check_release_assets row 12 piped into grep -q — 75th sit…
fix(release): rc_tag_main fails closed on an unreadable tag list; tes…
chore(release): 48h fleet-cell waivers for non-aprender binaries (cop…
chore(release): waive rmedia#746 and pv#4219 for the v0.70.0 cut only…
noahgift commentedon Sep 25, 2026
Parked by APR-EPIC-001: over E1 budget (rank 50/93). Reopen = pull. Applied by aprender-d1: claude+agy both said close, apr shadow row recorded (handoff/close-quorum-2026-09-25.json).
feat(release): aarch64-apple-darwin apr CPU asset, required by the as…
feat(release): darwin lane runs on the self-hosted mini-builder, not …
fix(release): #4292 darwin lane passes check_runner_labels + grep-q r…
feat(rc): RESUMABLE-CUT — every cut step receipts BEFORE it acts; a f…