[ajfriesen]

Summary

cache-rpi.yml existed for a good reason: there is no local Pi builder, and
compiling the linux-rpi kernel under binfmt emulation on an x86 box takes
hours, so the aarch64 closure was built on GitHub's native arm64 runner and
pushed to Attic. Nothing about that reasoning is Pi-specific — it is only where
the pain was felt first.

-.github/workflows/
-├── cache-rpi.yml        # rpi4, rpi5: closure (native arm64) + image + R2
-└── release-image.yml    # x86_64: image + R2, no closure, no cache
+.github/workflows/
+├── boards.yml           # the entry point: a matrix over the boards, calling ↓
+└── board.yml            # on: workflow_call — one board, end to end
+    ├── closure   a native runner for this board ──▶ attic push
+    └── image     always x86, needs: closure     ──▶ R2
+
+    rpi4   ubuntu-24.04-arm ─┐
+    rpi5   ubuntu-24.04-arm  ├─ three independent calls, three independent gates
+    x86_64 ubuntu-24.04     ─┘

Why x86_64 needs its closure pushed, and the reason is narrower than cache
coverage. cache.nixos.org does serve the pinned nixos-unstable revision
every board builds from (ADR 0008),
so most of an x86 closure substitutes from upstream. What never substitutes is
a system closure: the daemon, sendspin-go, vboard, the kiosk
configuration and the toplevel itself are this repository's own output, and no
public cache has ever held them. So until now an x86 candidate could not be
fetched by a device at all — an OTA update compiled on the device, and the
release gate's update phase could not be gated on that board. The Pi's
kernel is a difference of degree, not of kind.

Nothing is cross-compiled, and nothing compiles under emulation. Each board
sets its own nixpkgs.hostPlatform, so nixpkgs evaluates a native build —
and an emulated-native aarch64 build yields byte-identical store paths to a
truly native one, which is exactly what lets the closure the arm64 runner just
pushed substitute into the image job and onto a device. pkgsCross would
produce different paths, every one of those lookups would miss, and the
native-runner job would buy nothing. The only emulated work left is
nixos-install's chroot running switch-to-configuration boot — a few
activation scripts. (A comment inherited from cache-rpi.yml called that
"cross-build"; corrected here.)

The two workflows had also drifted, each holding half of what both should have
had:

	was	now
closure pushed, cache warmed on a push to main	Pis only	every board
baked version asserted against the tag	x86_64 only	every board
R2 publish + provenance metadata	two copies, already diverged on what they recorded	one

Board-specific assertions stay board-specific. The U-Boot UEFI loader and the
Pi boot chain on the ESP are Pi-only — an x86 board has no U-Boot in its chain,
and an assertion passing vacuously there would be worse than none. The
seed-file drop zone staged through systemd-boot.extraFiles is shared, so it is
now asserted for every board.

Evidence

Before: two workflows, two shapes, and
labgrid/test_update.py reporting the x86_64 closure as a gap by name.

After, each thing that could have broken a board, checked rather than
assumed:

The assertion promoted from Pi-only to shared actually holds on all three
boards — evaluated before generalising it:

$ for attr in …x86-disk …rpi4 …rpi5; do nix eval --json "$attr.config.boot.loader.systemd-boot.extraFiles"; done
x86-disk → ['README.txt', 'dashboard-assistant.yaml.example']
rpi4     → ['README.txt', 'dashboard-assistant.yaml.example']
rpi5     → ['README.txt', 'dashboard-assistant.yaml.example']

Every image attribute the new matrix builds resolves:

$ nix eval --raw ".#packages.x86_64-linux.{disk-image,rpi4-image,rpi5-image}.drvPath"
disk-image   /nix/store/2ipr792kv113mngf7p125wqx6mgi4zyf--disko-images.drv
rpi4-image   /nix/store/5lvxsqgj9sqyk8b5mxsgsx4lpzk7p6nw--disko-images.drv
rpi5-image   /nix/store/vqwjz699xd47rk2f1v8rnlqsfr8142qn--disko-images.drv

The x86 image now compresses with --long=27 like the Pi images rather than
plain -19. A window log of 27 is exactly zstd's default decoder limit, so the
bare zstd -dc that docs/flash/flash.md tells users to run still works —
compressed and decompressed a 200 MB file to confirm rather than trusting it:

$ zstd -T0 -19 --long=27 big.bin -o big.zst && zstd -dc big.zst >/dev/null
exit=0        # no --long needed on the way out, no warning

Lint and tests:

$ actionlint .github/workflows/*.yml
(clean — all six)

$ pytest labgrid/test_candidate.py test_capability.py test_device.py \
         test_preflight.py test_bench_ha.py test_kiosk.py -q
158 passed

The per-board gate has a guard, and it was written red against the shape it
replaces before being made green (see Review follow-up below):

$ pytest labgrid/test_candidate.py -k waits -q     # two matrixed jobs restored
AssertionError: boards.yml: image needs closure,
                which is matrixed over ['rpi4', 'rpi5', 'x86_64']

labgrid/candidate.py loses its per-board cache-source map and the "and
nothing pushes the x86_64 closure" branch of its failure message, since every
board now has the same answer. Its new guard is that the workflow it names
actually exists on disk — which is what a rename like this one should trip over
next time:

+def test_the_workflow_named_in_the_failure_exists():
+    """The message sends somebody to a path. A rename that left this behind
+    would send them to a file that is not there — which is exactly what
+    happened to `cache-rpi.yml`."""

Not reproducible in review, and not by a dispatch on this branch either:
GitHub only offers workflow_dispatch for a workflow already on the default
branch, so boards.yml cannot be dispatched until this merges. See below.

Merge Danger

Door: two-way to revert, but the release path is live in between.

Reverting restores both files unchanged. What cannot be undone by a revert is a
release cut while this is broken — but the blast radius of that is now one
board, not the fleet: each board's image gates on its own closure alone, so a
Pi failing no longer takes the x86_64 release artifact down with it, which is
the one way the old split was better and is no longer.

Recommended before the next tag: merge, then run boards.yml once via
workflow_dispatch on main — it is not dispatchable before then. That
exercises every path except the tag-only version guard: all three closures
pushed, all three images built and published to dev/<board>/, where the
lifecycle policy sweeps them after 7 days.

Blast Radius: the release flow, and the cache.

• Run names change (rpi4 / closure -> cache, x86_64 / image -> R2 instead
  of build/image). main is not a protected branch, so no required status
  checks need updating.
• The first run pushes an x86_64 closure to Attic for the first time — new
  objects in the cache, nothing displaced.
• A push to main now builds three closures instead of two, so it costs one
  more runner. The image job is still tag- and dispatch-only.
• ATTIC_TOKEN is now needed by a job that runs on an x86 runner too. Same
  secret, same scope; it just has one more consumer.
• Every reference to the old names is updated (flake.nix, justfile,
  modules/core/binary-cache.nix, ops/README.md, release-please.yml,
  labgrid/README.md). No other workflow builds these R2 paths: a candidate's
  notes carry no download links at all and promotion writes them by hand today
  (ADR 0011).
  The only place hard-coding the object names is docs/flash/flash.md, the
  filename table users read while flashing, and those names are unchanged.

Review follow-up (dff3895)

A two-axis review of the first commit found one thing it got wrong and one
thing it claimed wrongly. Both are fixed on the branch.

needs: closure coupled the boards. needs takes a whole job and never
one leg of its matrix, and fail-fast: false does not change that — so the two
matrixed jobs gated every board's image on every board's closure. A flaky
arm64 runner and a v* tag published nothing, x86_64 included, however green
its own closure was. That was a regression this PR introduced, not a wart it
inherited.

Neither obvious fix was available. Merging closure and image into one job per
board is impossible: the Pi closure needs a native arm64 runner while every
image needs x86, because disko boots a QEMU VM and the arm64 runners have no
/dev/kvm. Running the image job past a failed closure with !cancelled() is
unsafe: the tag version guard lives in the closure job, so an image leg that
ran on would publish a wrongly-named object.

So the per-board pair moved to board.yml behind on: workflow_call, and
boards.yml matrixes over the boards and calls it once each — GitHub's own
answer to per-leg needs. Each call is its own pair of jobs, so needs means
that board alone, and x86_64 no longer waits on rpi4's closure either.
board.yml carries no push/workflow_dispatch trigger, so boards.yml is
still the one entry point.

Three things the split forced, none of which change behaviour: workflow-level
env does not cross into a called workflow, so board.yml declares
ATTIC_ENDPOINT/ATTIC_CACHE/R2_ENDPOINT itself (concurrency stays on the
caller and still governs the whole run); secrets: inherit carries
ATTIC_TOKEN and the two R2 keys; and github.ref_type/github.event_name in
a called workflow are the caller's triggering event, so the tag version guard
and the image job's if: are untouched.

The header claimed release-please.yml "builds the same R2 paths". It
builds none. Corrected in the workflow header and in Blast Radius above.
labgrid/README.md also pointed at "boards.yml's closure job", which now
lives in board.yml.

Closes #55

🤖 Generated with Claude Code