feat(ci): ship host-derived buildkit defaults into runners

.github/workflows/build-push-image.yml

@@ -0,0 +1,97 @@
+# Reusable build-and-push for repositories whose Actions jobs run on Watchtower-managed
+# runners (docs/ci-runners/design.md §"Container image builds").
+#
+# Why this exists: it encodes the host knowledge a workflow would otherwise have to carry
+# itself. It deliberately does NOT set up a buildx `docker-container` builder — the build
+# runs on the daemon's default builder (`docker` driver), which uses the host's storage
+# driver (real copy-on-write even where BuildKit's OCI worker would fall back to the
+# copy-less `native` snapshotter, e.g. Synology DSM), keeps its build cache on the host
+# between runs, and inherits the daemon's insecure-registries configuration. The trade-offs
+# are encoded too: `provenance: false` because the docker driver cannot produce
+# attestations, and no registry cache import/export because the daemon's cache persists.
+#
+# Requirements (all provisioned by Watchtower when the repo's CI is enabled and a sync
+# registry is selected): a runner with the Docker socket allowed, the `REGISTRY` Actions
+# variable, and the `REGISTRY_USERNAME`/`REGISTRY_PASSWORD` secrets.
+#
+# Usage in a consuming repo:
+#
+#   jobs:
+#     image:
+#       uses: swimmesberger/Watchtower/.github/workflows/build-push-image.yml@main
+#       with:
+#         image: muxbox
+#       secrets: inherit
+
+name: Build & push image
+
+on:
+  workflow_call:
+    inputs:
+      image:
+        description: Image name without the registry prefix (e.g. `muxbox`).
+        required: true
+        type: string
+      context:
+        description: Build context path.
+        type: string
+        default: .
+      file:
+        description: Dockerfile path; defaults to `<context>/Dockerfile`.
+        type: string
+        default: ""
+      build-args:
+        description: Newline-separated `NAME=value` build arguments.
+        type: string
+        default: ""
+      push:
+        description: Push the image after building.
+        type: boolean
+        default: true
+      tags:
+        description: >-
+          docker/metadata-action tag rules. The default tags `latest` on the default
+          branch, every commit by short SHA, and git tags by name.
+        type: string
+        default: |
+          type=raw,value=latest,enable={{is_default_branch}}
+          type=sha
+          type=ref,event=tag
+    outputs:
+      image:
+        description: First fully-qualified image reference that was built.
+        value: ${{ jobs.build.outputs.image }}
+
+jobs:
+  build:
+    runs-on: [self-hosted, watchtower]
+    outputs:
+      image: ${{ fromJSON(steps.meta.outputs.json).tags[0] }}
+    steps:
+      - uses: actions/checkout@v7
+
+      # `vars.REGISTRY` and the two secrets are the values Watchtower syncs into this
+      # repository's Actions config; `secrets: inherit` in the caller makes them visible here.
+      - uses: docker/login-action@v4
+        with:
+          registry: ${{ vars.REGISTRY }}
+          username: ${{ secrets.REGISTRY_USERNAME }}
+          password: ${{ secrets.REGISTRY_PASSWORD }}
+
+      - id: meta
+        uses: docker/metadata-action@v6
+        with:
+          images: ${{ vars.REGISTRY }}/${{ inputs.image }}
+          tags: ${{ inputs.tags }}
+
+      # No setup-buildx-action on purpose — see the header. The default builder is the
+      # daemon itself, so the layer cache lives on the host and survives the ephemeral runner.
+      - uses: docker/build-push-action@v7
+        with:
+          context: ${{ inputs.context }}
+          file: ${{ inputs.file }}
+          build-args: ${{ inputs.build-args }}
+          push: ${{ inputs.push }}
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          provenance: false

docs/ci-runners/design.md

@@ -171,6 +171,97 @@ the existing Volumes module; GC/pruning is future work. Pre-warming the toolcach
 detected toolchain profile is described in
 [Stack-linked CI](#stack-linked-ci-toolchain-detection--cache-pre-warming).
 
+### Container image builds (BuildKit defaults; issue #65)
+
+A job that builds an image with `docker/setup-buildx-action` gets the `docker-container`
+driver: a BuildKit daemon in its own container, which reads **none** of the host daemon's
+configuration. Two host facts then leak into every consuming repo's workflow YAML unless
+Watchtower delivers them itself:
+
+- **Plain-HTTP registries.** The daemon's `insecure-registries` setting doesn't reach the
+  out-of-daemon BuildKit, so pushing to a local registry needs a `buildkitd-config-inline`
+  stanza in every workflow.
+- **The snapshotter.** BuildKit's OCI worker probes `auto` → overlayfs → fuse-overlayfs →
+  `native`. On kernels without overlayfs (Synology DSM's 4.4) it lands on `native`, which
+  has no copy-on-write: every layer materialisation is a full recursive copy of the
+  accumulated image tree, and builds run ~10× slower with nothing in the job log saying
+  why (the tell is `org.mobyproject.buildkit.worker.snapshotter: native` in the
+  `Set up Docker Buildx` output; the symptom is cache-*hit* steps spending minutes in
+  `extracting`).
+
+**Mechanism — a default buildkitd config shipped into every runner.** buildx reads
+`$BUILDX_CONFIG/buildkitd.default.toml` whenever the workflow passes no config of its own,
+so the orchestrator generates one per reconcile pass (`CiBuildkitConfig`):
+
+- `[registry."…"] http/insecure = true` stanzas for exactly the registries the host
+  daemon itself treats as insecure (`GET /info` → `RegistryConfig.IndexConfigs`) — the
+  daemon is the authority on which registries this box reaches without TLS. This deletes
+  `buildkitd-config-inline` from consuming workflows.
+- `[worker.oci] snapshotter = …`, **auto-detected from the host by default**. The key
+  fact (confirmed by the DSM probe below): BuildKit's own `auto` chain is
+  overlayfs-or-`native` — it never tries fuse-overlayfs on its own — so a kernel without
+  overlayfs silently gets `native` even where FUSE is fully available. Watchtower
+  therefore emits `fuse-overlayfs` exactly when the kernel lacks overlayfs (per
+  `/proc/filesystems`, kernel-global even from a container; an overlay-family daemon
+  storage driver also counts as proof of overlayfs) but has FUSE, and stays silent
+  everywhere else — where overlayfs exists BuildKit picks it unaided and it beats
+  fuse-overlayfs, and where neither exists `native` is all there is. The instance-wide
+  `Ci:BuildkitSnapshotter` option overrides: `auto` (the default), `none` (emit nothing,
+  leave BuildKit's probe alone), or an explicit snapshotter name. Instance-wide because
+  which snapshotter works is a property of the host kernel, not of any repo; the resolved
+  choice is logged once on change.
+
+Delivery is a third per-repo volume, `watchtower-ci-buildx-{repo}`, mounted at
+`/home/runner/_buildx` and exported as `BUILDX_CONFIG` (runner env is inherited by job
+steps). A volume rather than a file bind because of the standing trap: dockerd creates
+missing bind parents as root, and a mount under `~/.docker` would leave that directory
+root-owned and break the next `docker login` in a job. The existing volume-init container
+writes the file (content passed as env, written with `printf '%s'`) and chowns all three
+volume roots; it re-runs whenever the generated content changes — the last-written stamp
+lives on the repo's in-memory status — so a registry added at runtime reaches jobs within
+one pass. A workflow that passes its own `buildkitd-config(-inline)` still wins outright;
+the default only fills the unconfigured case.
+
+**The fast path on hosts without a working OCI snapshotter.** Where neither overlayfs nor
+fuse-overlayfs can work, the `docker-container` driver is simply the wrong tool: the
+daemon's own builder (`driver: docker`, i.e. *no* setup-buildx step) uses the host's
+storage driver — real CoW even on btrfs — keeps its build cache on the host between runs,
+and inherits `insecure-registries` natively. That choice is encoded once in the reusable
+workflow [`build-push-image.yml`](../../.github/workflows/build-push-image.yml)
+(`uses: swimmesberger/Watchtower/.github/workflows/build-push-image.yml@main`), together
+with its consequences: `provenance: false` (the docker driver can't do attestations) and
+no registry cache import/export (the daemon's cache persists anyway). Consuming repos
+carry one `uses:` line, `secrets: inherit`, and no host knowledge — the
+`REGISTRY`/`REGISTRY_USERNAME`/`REGISTRY_PASSWORD` values it reads are the ones the
+registry sync (Secrets §1) already pushes.
+
+**Operational notes.**
+
+- With the docker driver, the daemon's build cache is no longer discarded with the builder
+  container. `docker builder prune` is the relief valve when it grows; wiring it into
+  Watchtower's maintenance/pruning story is future work.
+- Runners on small hosts will not reach GitHub-hosted speeds even once the snapshotter is
+  right — `exporting layers` is mostly gzip on however few cores the box has. That is not
+  a bug to go hunting for.
+- Verifying fuse-overlayfs viability on a host by hand (what the auto-detection decides
+  from, plus the end-to-end check the detection cannot do):
+
+  ```bash
+  grep -E 'overlay|fuse' /proc/filesystems
+  ls -l /dev/fuse
+  docker run --rm --privileged moby/buildkit:latest --oci-worker-snapshotter=fuse-overlayfs --debug 2>&1 | head -20
+  ```
+
+  Run on the DSM NAS 2026-08-28: the kernel has FUSE (`nodev fuse`), `/dev/fuse` exists,
+  and buildkitd starts cleanly with a registered fuse-overlayfs worker
+  (`worker.snapshotter: fuse-overlayfs`) — confirming that BuildKit's `native` fallback
+  there is purely its probe never trying fuse-overlayfs, which is exactly the gap the
+  auto-detection fills; the buildx builder container is privileged, so `/dev/fuse`
+  reaches it. fuse-overlayfs is slower than kernel overlayfs but does metadata copy-up
+  instead of `native`'s full-tree copies. (Startup proves the snapshotter initialises;
+  the first real build is the end-to-end confirmation — `Ci:BuildkitSnapshotter=none`
+  is the escape hatch if a host's FUSE turns out broken in practice.)
+
 ### RPC surface
 
 | Method | Notes |

docs/decisions/0028-ci-buildkit-defaults.md

@@ -0,0 +1,93 @@
+# ADR-0028: CI runners carry the host's BuildKit knowledge — a generated default buildkitd config, and a reusable docker-driver workflow
+
+- Status: Accepted (implemented)
+- Date: 2026-08-28
+- Related: [docs/ci-runners/design.md](../ci-runners/design.md) §"Container image builds",
+  [issue #65](https://github.com/swimmesberger/Watchtower/issues/65) (the evidence and the
+  analysis this decision rests on).
+
+## Context
+
+A repository on a Watchtower-provisioned runner that builds an image the documented way —
+`docker/setup-buildx-action`, i.e. the `docker-container` driver — gets a BuildKit daemon
+in its own container, which reads none of the host daemon's configuration. On the NAS that
+produced issue #65 this had two consequences:
+
+1. **Silent ~10× slower builds.** BuildKit's OCI worker resolves its snapshotter with
+   `auto`: overlayfs, then fuse-overlayfs, then `native`. DSM's 4.4 kernel fails both
+   probes, and `native` has no copy-on-write — every layer materialisation is a full
+   recursive copy of the accumulated stage. A fully-cached MuxBox publish job spent 13m20s
+   with nothing to upload; 26 layer extractions totalled 798s, a 93-byte layer costing a
+   minute on top of the .NET SDK. Nothing in the job log names the cause. The host offers
+   no fix: kernel, storage driver (btrfs — a containerd snapshotter, not an OCI-worker
+   one) and engine version are all pinned by DSM.
+2. **Per-repo host knowledge.** The plain-HTTP registry needs a `buildkitd-config-inline`
+   stanza in every consuming workflow, because the daemon's `insecure-registries` setting
+   does not reach an out-of-daemon BuildKit.
+
+Both violate what the CI-runners design promised: zero-ceremony enablement, workflow YAML
+that stays fully standard, and — per the standing rule that configuration lives in
+Watchtower and is delivered at the point of use — no host facts hand-copied into repos.
+
+## Decision
+
+**1. Watchtower generates a default buildkitd config and ships it into every runner.**
+buildx reads `$BUILDX_CONFIG/buildkitd.default.toml` whenever the workflow passes no
+config of its own, so every `docker/setup-buildx-action` inherits it with no workflow
+change (a workflow's own `buildkitd-config(-inline)` still wins outright). The file
+(`CiBuildkitConfig`) carries:
+
+- `[registry."…"] http = true / insecure = true` for exactly the registries the host
+  daemon treats as insecure, read from `GET /info` — the daemon is the authority on which
+  registries this box reaches without TLS, so this needs no new Watchtower state and
+  tracks the host automatically.
+- `[worker.oci] snapshotter = …`, **auto-detected by default**. Running the probe on the
+  NAS showed BuildKit's `auto` chain is overlayfs-or-`native` — it never tries
+  fuse-overlayfs on its own, even where FUSE is fully available and an explicitly
+  configured fuse-overlayfs worker starts cleanly. Watchtower closes exactly that gap:
+  it emits `fuse-overlayfs` when the kernel lacks overlayfs but has FUSE (read from
+  `/proc/filesystems`, which is kernel-global even from a container; an overlay-family
+  daemon storage driver also proves overlayfs), and emits nothing everywhere else —
+  where overlayfs exists BuildKit picks it unaided and it beats fuse-overlayfs, and
+  where neither exists `native` is all there is. The new instance-wide
+  `Ci:BuildkitSnapshotter` option overrides: `auto` (default), `none` (never emit — the
+  escape hatch if a host's FUSE is broken in practice), or an explicit name.
+  Instance-wide because a working snapshotter is a property of the host kernel, not of
+  any repo.
+
+Delivery is a third per-repo volume (`watchtower-ci-buildx-{repo}`) mounted at
+`/home/runner/_buildx` and exported as `BUILDX_CONFIG`. A volume, not a file bind, because
+dockerd creates missing bind parents root-owned — a mount under `~/.docker` would break
+the next `docker login` in a job (the `_work` trap again). The existing volume-init
+container writes the file and chowns the volume roots, re-running whenever the generated
+content changes, so a registry added at runtime reaches jobs within one reconcile pass.
+The runner spec-hash material gained a version prefix so pre-existing idle runners are
+recycled once and pick the mount up.
+
+**2. The fast path on hosts without a working OCI snapshotter is the docker driver, and
+it is encoded once, in a reusable workflow.** Where no OCI snapshotter works, the config
+file cannot fix the `docker-container` driver, and the right answer is not to use it: the
+daemon's own builder uses the host storage driver (real CoW even on btrfs), keeps its
+cache on the host between ephemeral runners, and inherits `insecure-registries` natively.
+[`build-push-image.yml`](../../.github/workflows/build-push-image.yml) encodes that
+driver choice with its consequences (`provenance: false`; no registry cache
+import/export), reading the `REGISTRY` variable and credentials Watchtower already syncs.
+Consuming repos carry one `uses:` line and `secrets: inherit`.
+
+## Consequences
+
+- MuxBox-class workflows drop `buildkitd-config-inline`, and either switch to the
+  reusable workflow or simply drop `setup-buildx-action`; the ~10× extraction penalty
+  disappears with the `native` snapshotter.
+- The open question from issue #65 — whether fuse-overlayfs can be made to work on DSM —
+  was answered the same day by running the probe on the NAS: buildkitd starts cleanly
+  with a registered fuse-overlayfs worker, so the `native` fallback there is BuildKit's
+  probe never trying fuse-overlayfs at all. That finding is why the snapshotter default
+  became auto-detection rather than a knob the operator must know to set: the DSM host
+  is fixed with **no configuration at all**, and no healthy host changes behaviour
+  (details in the design doc).
+- With the docker driver, build cache accumulates in the daemon instead of dying with the
+  builder container. `docker builder prune` is the manual relief valve; wiring it into
+  Watchtower's maintenance story is recorded as future work in the design doc.
+- Runners on small hosts still will not reach GitHub-hosted speeds (`exporting layers` is
+  gzip-bound); the design doc says so, so nobody hunts for a second bug.

docs/decisions/README.md

@@ -45,3 +45,5 @@ live here.
 - [ADR-0024: PostgreSQL is the only database, and the proxy/auth plane keeps its state in it](0024-postgresql-only-and-state-in-the-database.md)
 - [ADR-0025: Stacks can be stopped and started as a whole, and the stop is a persisted desired state](0025-stack-desired-state.md)
 - [ADR-0026: A Product is the deployable unit; stacks reference it, and releases pin its images](0026-products-are-the-deployable-unit.md)
+- [ADR-0027: Watchtower backs itself up, and a bundle restores it somewhere else](0027-full-instance-backup-and-restore.md)
+- [ADR-0028: CI runners carry the host's BuildKit knowledge — a generated default buildkitd config, and a reusable docker-driver workflow](0028-ci-buildkit-defaults.md)