feat(cli): CTC-4646 — a self-hosted runner moves to the approved release on its own, between phases

[ryanrozich]

A self-hosted runner moves to the release Catalyst approved, on its own, between phases. It never pulls a tag, and it goes back to the previous images if the new supervisor does not come up.

Depends on coalesce-labs/catalyst-cloud#12379 (CTC-5926), which serves GET /api/v1/hosts/desired-release. Until that is deployed, the read answers 404 and every run keeps the current images.

What a run does

1. Reads the approved release from inside a one-off supervisor container (compose run --rm --no-deps), the same way setup reads enrollment, so the host secret never leaves Docker.
2. Moves only when every check passes:
   • three digests from the three Catalyst repositories, release id matching them, generation newer than the last applied;
   • no phase container for this host;
   • all three images pulled natively by digest;
   • on a Mac with private thoughts storage, re-attested on the new supervisor with the same authority.
3. Rewrites the three image lines in .env (previous file kept as .env.release-previous), runs docker compose up, and waits up to 5 minutes for host_ready from the new supervisor.
4. Not ready: restores the previous .env and images and exits 1. Every other answer keeps the current images. release-state.json keeps the applied generation and the last outcome.

catalyst runner auto-update on|off|status schedules it every 15 minutes: a LaunchAgent on macOS, a systemd user timer on Linux (it says when lingering is off). Setup turns it on after the runner starts, and never fails because of it.

Review

One local Codex review on d8a6ba8 found five issues, all fixed in this head:

• P1: a .env left by an interrupted run was trusted as applied. The run now reconciles against the running containers.
• P1: one idle snapshot did not hold the boundary. The run now stops the supervisor, looks again, and backs off if a phase started.
• P1: Mac custody was not restored on failure. Every failure path now attests it back.
• P2: a watchdog-only change waited for a host_ready that never came.
• P2: the lock was reclaimed by age. It is now reclaimed only when the owning process is dead.

Narrow fix-check in sbx:

• typecheck clean;
• 319 tests passed;
• four mutants caught: no re-check after stop, trusting .env, no custody restore, and a lock that ignores its owner.

No re-review.

Proof

sbx and mini, at 1519f38: typecheck clean, 319 tests passed, build and prose check clean. Mutants caught: no re-check after stop, trusting .env, no custody restore, a lock that ignores its owner, no image cleanup.

Physical, 2026-10-17, test workspace. catalyst-cloud#12379 (CTC-5926) merged and deployed, and its publisher approved generation 1 hr-eb9a0e83273ec210-9a96426c93e32173-63a9b20c3be3dc3e. Then:

Machine	Schedule	Window	Before	After
mini-2 (Mac, Docker Desktop)	launchd	13:30:00–13:31:15, unattended	stock CLI 0.16.15 pins (runner 9db2bf2c, supervisor 3331ebc4, watchdog a5999dbf)	runner eb9a0e83, supervisor 9a96426c, watchdog 63a9b20c
amd64 VM (Ubuntu)	systemd user timer	13:33:48–13:35:50, unattended	the #12288 test supervisor and stock runner	the same three digests

• On both, the new containers had restarts=0, and the logs show supervisor_readopted and host_ready. release-state.json records generation 1 and the previous digests.
• On mini-2, Darwin thoughts custody re-attested on the new supervisor.
• The next --check on both says not_newer.
• A phase on the new images: ADV-17 was granted on mini-2 4 s after it reached Todo. Its intake, plan, implement and validate phases ran on runner eb9a0e83 (research ran on another host), and the ticket reached PR: docs(sample-service): [details removed] — add README Updates note for self-hosted runners catalyst-e2e/sample-service#15.

Found and fixed during the physical run (both in this head):

• launchd StartInterval and RunAtLoad sat as "pended nondemand spawn" for 5 hours on a busy Mac. Calendar minutes 0/15/30/45 fire on time, so the Mac schedule now uses them.
• The ~9 GB runner pull left both machines under the 25 GiB free-disk floor (mini-2 21.5, amd64 17.0), so they advertised 0 slots. The CLI now removes the replaced images after a verified update. Run by hand with the same commands, that removal restored 29.6 and 26 GiB.

Docs: catalyst-cloud#12427 (draft) updates the self-hosting page and lands after the CLI release that carries this.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LGe7UXEGNZW1GenNuUZRDU

[ryanrozich]

Review custody for CTC-4646: this PR has already received its ONE local Codex full review. It found three P1 and two P2 issues. The source owner fixed all five before the current amended head 1519f38aca48170151b4372ff232dc872e7e7c9f:

• Reconcile against running containers instead of trusting .env.
• Stop the supervisor and recheck before switching releases; back off if a phase has started.
• Re-attest Mac custody on every failure path.
• Do not wait for host_ready after a watchdog-only change.
• Reclaim the update lock only after its owner process is dead.

The narrow post-review fix-check recorded by selfhost-lead passed TypeScript, 319 tests, and four targeted mutants. Later image cleanup and calendar-schedule changes received narrow mutant checks, not a second full review. Physical self-update proof on mini-2 (launchd) and amd64 (systemd user timer) completed without a restart, with host-ready readoption; ADV-17 then ran phases on the approved runner generation. Current-head Node 22/24/26, Bun runtime, evals and CodeQL checks are green. The source owner retains the original reviewer session metadata and was asked to add it to the handoff; no second Codex review should be requested for this PR.

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 no unresolved review threads before merge

• #review-threads-unresolved=0

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-10-17 15:31 UTC · Rule: default · triggered by rule enqueue any green, thread-clean, non-held PR
• ✅ Checks passed · on draft merge queue: checking #386 on main (f41b6d6) #391
• ✅ Merged — 2026-10-17 15:38 UTC · at bcb93911e82a1e1c7f535a48a2619135f4b7fd66 · squash

This pull request spent 6 minutes 44 seconds in the queue, including 5 minutes 51 seconds running CI.

Required conditions to merge

• #review-threads-unresolved=0
  • feat(cli): CTC-4646 — a self-hosted runner moves to the approved release on its own, between phases #386
• base=main
• check-success=check
• label!=hold
  • feat(cli): CTC-4646 — a self-hosted runner moves to the approved release on its own, between phases #386
• all of [🛡 Merge Protections rule no unresolved review threads before merge]:
  • #review-threads-unresolved=0
    • feat(cli): CTC-4646 — a self-hosted runner moves to the approved release on its own, between phases #386