[ajfriesen]

Closes #20. Implements ADR 0009, now accepted.

Summary

An update that cannot come back up undoes itself. Three attempts, then the previous generation — and the refused release is written down so it is not offered again.

Boot counting and the gate that decides what a good boot is were per-board, and two of three boards simply never got a copy. Both now live in modules/core/ and reach every board through core/default.nix, which is what stops the next board being added without them.

 modules/
 ├── core/
+│   ├── good-boot.nix          # bootCounting (3 tries) + the gate, every board
 │   └── default.nix            # imports it, so a new board cannot miss it
 └── hardware/
-    └── generic-x86-disk.nix   # had the gate; rpi4/rpi5 had nothing
+    └── generic-x86-disk.nix   # loader config only
 daemon/
+├── rollback.go                # was this boot a rollback, and from which release
 └── update.go                  # skips refused releases until a newer one ships

The gate no longer requires greetd. A board whose GPU the kernel cannot drive has no kiosk session, so the old condition marked every boot bad and would have rolled the device back repeatedly over a fault no older generation fixes. What is left is the daemon answering /healthz — the device is reachable, so recovery is possible, and nothing more.

How a rollback is noticed

The bootloader decides this two stages before userland and leaves no file and no unit behind, so it is derived from three symlinks the system already maintains:

profile == current == booted     an ordinary boot
profile == current != booted     an update switched, has not rebooted yet
profile != current == booted     the loader abandoned the profile's generation

Reading current-system as well as booted-system is what separates the last line from the second: immediately after an update switches, the profile and the booted system also disagree, and reading only those two would report every pending update as a rollback.

The rejected release's version comes out of that generation's own closure rather than its NixOS label — a label is tags and nixpkgs version joined by hyphens, so dashboard-assistant-1.6.0-rc1-26.11… cannot be split back apart.

The refused versions are persisted in stateDir, which the symlinks cannot carry. Deriving them from the profile alone was wrong for the second rollback in a row:

roll back from 1.6.0  ->  profile names 1.6.0, skip it
install 1.7.0         ->  profile names 1.7.0, 1.6.0 forgotten
roll back from 1.7.0  ->  1.6.0 offered again, fails again, alternates forever

A release leaves the record when the device is running it — the way back for one refused by mistake, since the version picker still offers a rejected release.

Reported to Home Assistant

Three fields on the state snapshot, omitted entirely on a device that never rolled back:

 "update": {
   "installed_version": "1.4.0",
   "latest_version": "1.4.0",
+  "rolled_back": true,
+  "rejected_version": "1.5.0",
+  "rejected_generation": 2
 }

Reported separately from last_error because nothing failed to install: the switch was clean and was reported as done, and the generation only gave up hours later on the next boot. The integration ships from its own repo and does not surface these yet — needs a follow-up issue there.

Evidence

The question that kept this switched off: boot counting needs systemd-boot to set LoaderBootCountPath, a volatile EFI variable, at runtime — from a loader bootctl had to install with --no-variables at the removable fallback path. Unrelated flags in principle, untested upstream (nixpkgs' own bootCounting test runs with canTouchEfiVariables = true).

x86, in a VM — nix build .#checks.x86_64-linux.good-boot, five boots, 116 s:

Before:  no such check
After:   LoaderBootCountPath present
         ha-boot-health[525]: daemon healthy; blessing this boot
         systemd-bless-boot[739]: Marked boot as 'good'. (Boot attempt counter is at 1.)
         gate fails x3 -> entry +0-3 marked bad -> boots the previous generation
         profile != current == booted   (the state daemon/rollback.go reads)

Raspberry Pi 5, on the bench — just hw-test:

Before:  no such assertions
After:   test_the_loader_counted_and_blessed_this_generation PASSED
         test_the_good_boot_gate_never_failed               PASSED

The image's ESP ships nixos-1302d171…+3.conf and the booted board has no counter on any entry. Only systemd-bless-boot removes one, and it runs only when the loader set that variable — so boot counting works through U-Boot's UEFI, --no-variables and all.

The daemon, go test -race ./...:

TestRollbackNamesTheRejectedRelease              names 1.5.0 from the profile's generation
TestNoRollbackOnTheLookAlikeStates               4 states that must not read as rollbacks
TestRejectedReleaseIsNotOffered                  Latest stays at installed, picker marks it
TestANewerReleaseClearsTheSkip                   1.6.0 offered, 1.5.0 stays behind it
TestRejectionsOutliveTheNextUpdate               the two-bad-releases loop, closed
TestInstallingARejectedReleaseClearsIt           the way back
TestSnapshotReportsARollback                     the wire names the integration reads

Neither half can pass vacuously: the new rollback-is-armed flake check asserts the image writes a counter on all three boards, and the bench asserts one was removed. Verified red both ways — enable = false and tries = 5 each fail, naming every board.

All 7 flake checks, 177 bench unit tests, gofmt/go vet/nixfmt/ruff green.

Merge Danger

Door: two-way.

Revert is a revert. Nothing migrates, nothing is destroyed, and the one piece of persisted state (stateDir/rejected-releases) is a plain file that a reverted build simply stops reading. A device already on an older release keeps booting it.

Blast Radius: every board's boot path.

This arms a mechanism that can choose a different generation to boot, on every board, which is the sharpest thing in the repo to get wrong. Worth knowing before merging:

• The expensive direction of wrong is a gate that is too strict — it would spend a healthy release's three attempts and roll back a working update. Hence /healthz only: not greetd, not Home Assistant reachability, not the screen.
• A freshly flashed device has nothing to fall back to. If it never reaches a good boot it keeps trying that one, and the fix is to reflash. Automatic rollback protects updates, not flashing.
• A switch that dies inside switch-to-configuration leaves the same three symlinks as a rollback, because nixos-rebuild sets the profile before it activates. Both mean "told to run a release, not running it", so both get the same treatment; what would tell them apart is the spent counter on the ESP, which is umask=0077 root-only. last_error distinguishes them in practice. Documented in rollback.go.
• good-boot is the slowest CI job — a real systemd-boot booted five times, the only check needing nested reboots.

Three failures in the bench run are not from this branch and are left alone: test_journal_archive_survives_a_reboot (flash-wear.nix, reproducible), the test_journey.py errors (device_ip is session-scoped and first evaluated seconds after that reboot, so it sees end0 DOWN), and test_nix_builds_outside_the_tmpfs (30 s console timeout on nix config show). On an uncounted boot — which the post-reboot boot is — nothing here executes: boot-complete.target has no [Install] and is pulled in only by systemd-bless-boot's generator, which needs the counter blessing just removed.

🤖 Generated with Claude Code