release: test a candidate rev on one box, and back it out on failure or at boot

[lionello]

Baseline: 4ee19aa, read on a live native box (agent@20-237-183-155.sslip.io, running 770ef4e).

What prompted this

Somebody asked for an open PR to be applied to one box so it could be tried before merge. There is no supported way to do that, and no way to undo it if it had gone badly. The pieces are mostly built - what is missing is reachability and a commit window.

What already works

• Targeting an arbitrary rev. agentbox update --rev REF takes a tag, branch or commit, fetches it by name, and still goes through the fast-forward guard (source-tree.sh). A PR branch built on current master is a descendant of the running rev, so the guard passes it unchanged.
• Rollback during an update. rollback_to() walks the profile back a generation at a time; post_switch() wraps the whole apply in a handler that rolls back, restores the agent pin, re-applies with the old code and restarts units; recover_handover() catches the case where the new release cannot run its own phase two. This part is good and nothing below changes it.

The three gaps

1. No trigger can name a rev. The rendered agent-box-update.service (bin/agentbox#L2244) has a fixed ExecStart=agentbox update --config <path>, and UPDATE_TRIGGER is the single string the sudo allowlist grants. So every agent-reachable update goes to the tracked branch's HEAD and nowhere else. There is a backend-parity hole underneath it: NixOS has selfUpdate.branch, the native Spec schema (bin/agentbox#L673) has repo and rev and no branch, and the ExecStart carries neither.

2. Rollback stops the moment the update succeeds. Every recovery path lives inside the update transaction. Once post_switch returns 0 the release is permanent: there is no agentbox rollback verb, no unit, no grant. And the success criterion is weak - restart_units() restarts daemons and never asks whether they came back, and the code says so outright: "apply is not a health check with a timeout" (bin/agentbox#L4334). A release that applies cleanly and then wedges caddy or the settings socket is recorded as a success, and the box stays on it.

3. A reboot confirms a bad release rather than reverting it. /nix/var/nix/profiles/agent-box is a plain symlink (observed: agent-box-3-link, generation 3 of 3). Nothing at boot re-evaluates it, and there is no agent-box unit that could. Today the reboot that an operator would reach for as a last resort is the thing that makes a bad release permanent.

Proposed shape

Backing out is a profile generation, not a git revert. Worth stating because it inverts the instinct: generation N-1 is the previous release, already realized in the store, already a GC root. The operation is rollback_to() + agentbox apply + restart_units(), all of which exist, and agent-box-source reset REV already drags the tree back to match. It also sidesteps a trap - returning from a candidate to master is a downgrade in git terms and the fast-forward guard would demand --force, whereas a profile rollback never asks the guard at all.

Four pieces, smallest first. Each is useful alone.

1. agentbox rollback - expose post_switch's recovery branch as a verb, plus agent-box-rollback.service and a sudo grant. On its own this closes gap 2, which is the one with no workaround today.
2. Candidate targeting - add branch to the native Spec for parity with the module, and a templated agent-box-update-candidate@.service whose instance name is the escaped ref, reached through an explicit sudoAllowlist entry rather than something web.enable implies. One box gets pointed at a PR branch; its ordinary trigger then tests it.
3. A commit window - a candidate update writes a marker ({from_generation, target_rev, deadline}). agentbox update --confirm, a settings-page button, or a passing health probe clears it; a timer rolls back if the deadline passes unconfirmed.
4. Boot check - a unit ordered early at boot that reads that marker and rolls back when it is unconfirmed. "Reverts automatically on reboot" falls out of this, and it is the only piece that covers a release which wedges the box badly enough that nobody can reach the button.

The health probe should be verify_units over RESTART_PATTERNS plus a loopback GET of the box's own front door, so that "post_switch returned 0" becomes "the box answers".

Alternatives considered

• Revert the source tree instead of the profile. Rejected: since Agent-box should ship with its own fork #242 the tree is a record of what the profile names, not the source of truth, and a git revert leaves the built closure untouched. It is also strictly slower, because the old closure is already in the store.
• --force back to the previous rev. Works, but spends the one documented downgrade escape hatch on the routine case, and rebuilds what a profile rollback already has.
• A cooldown before boxes adopt a release (raised and declined in Self-update service: verify releases before applying (supply-chain hardening) #46). Orthogonal, and it delays emergency fixes equally; a per-box commit window does not.

Scope

This is the box-side half of #632, whose acceptance ends "Demonstrate rollback" - that issue is scoped to CI and publication, and this is what a box needs in order to participate in it. Compatible with #46's signing story (a candidate is still ancestry-checked, and a signature requirement would apply to it unchanged). Relates to #638.

[defangdevs]

One thing worth surfacing before piece 2 gets built: it revives the exact shape #558 was closed for.

#558 (agent-box-candidate) pointed root's update trigger at an arbitrary branch through sudo, with careful validation (branch-only, ancestor-only, no traversal). It was closed anyway, on this ruling:

Lio, on the local dev-and-revert question: "anything allowed by the sudo exception should come from upstream main only"

The closing comment's reasoning was that narrowing which non-main code root builds doesn't remove the fact that it's non-main code, which the rule forbids outright. It also named a three-item replacement list: an explicit agentbox rollback verb, a deadman timer, and a GC-pin on the known-good generation — plus the phase-two rollback hole, which #586 has since closed.

Pieces 1, 3 and 4 here are exactly that list (rollback verb, commit window ≈ deadman timer, boot check ≈ the "revert on boot" case the timer owner-decision was waiting on). They stay inside the main-only rule: every profile generation they operate over was still built from a reviewed main rev, and #558's closing comment specifically called out that property as what made reverts unambiguous.

Piece 2 (candidate targeting: branch on the native Spec, agent-box-update-candidate@.service, a sudo grant keyed on an arbitrary ref) is the part that doesn't fit inside that rule — it's root building non-main code again, just narrower and per-box instead of fleet-wide. That may be exactly the reconsideration you intend here (the per-box, generation-based framing does answer the "revert target is a half-tested branch state" objection from the closing comment), but the issue body doesn't mention #558 or the rule it's built around, so flagging it in case it's an oversight rather than a deliberate reopening of that question.

Given that, I'll pick up piece 1 (agentbox rollback) first — it's independent, closes the sharpest gap (recovery stops the moment post_switch returns 0 today), and doesn't touch the piece-2 question either way.

[lionello]

Piece 1 is up as #677 (agentbox rollback + its unit + the sudo grant).

Two things it turned up that the other pieces should carry:

The previous generation is not the previous release. cmd_update installs by removing the runtime element and adding it back, and nix profile remove makes a generation of its own - so every successful update leaves an empty generation directly behind the release it installed. Measured on the box this was written from: gen 3 = running release, gen 2 = "elements": [], gen 1 = the install. Anything in pieces 3 and 4 that records a "generation to return to" has to record the one that holds a release, not N-1.

The rollback target has a 7-day shelf life. agent-box-nix-gc.timer runs nix-collect-garbage --delete-older-than 7d, which deletes old generations and their closures. #677 refuses cleanly when the target is gone, but that is a refusal, not a fix: the commit window in piece 3 should hold its own GC root on the generation it promises to return to, or the promise expires on a timer nobody is watching.

[defangdevs]

Standing down on piece 1 - I had an independent agentbox rollback in progress in my own worktree (never pushed) when #677 landed, and #677 is the better version: it caught a bug mine had not (a plain current - 1 lands on the empty generation nix profile remove leaves behind, not the previous release - holds_a_release()/previous_release() in #677 walk past that). No point carrying two implementations forward, so I've discarded mine.