Skip to content

feat(ci): ship host-derived buildkit defaults into runners - #66

Merged
swimmesberger merged 1 commit into
mainfrom
wt/watchtower-issue-65-edb079
Aug 28, 2026
Merged

feat(ci): ship host-derived buildkit defaults into runners#66
swimmesberger merged 1 commit into
mainfrom
wt/watchtower-issue-65-edb079

Conversation

@swimmesberger

Copy link
Copy Markdown
Owner

Closes #65.

What changed

BuildKit's docker-container driver (the documented setup-buildx-action default) reads none of the host daemon's configuration, and its snapshotter auto-probe is overlayfs-or-native — it never tries fuse-overlayfs on its own. On DSM's 4.4 kernel that means every build silently gets the copy-less native snapshotter (~10× slower; a fully-cached MuxBox publish spent 13m20s with nothing to upload), and the plain-HTTP registry needs a hand-written buildkitd-config-inline stanza in every consuming workflow. Both are host facts that belong in Watchtower, not in repos.

Proposal A — a generated default buildkitd config in every runner

  • CiBuildkitConfig generates buildkitd.default.toml each reconcile pass:
    • [registry."…"] http/insecure = true for exactly the registries the daemon itself marks insecure (GET /infoRegistryConfig.IndexConfigs) — no new Watchtower state, tracks the host automatically, deletes buildkitd-config-inline from consuming workflows.
    • [worker.oci] snapshotter, auto-detected by default: fuse-overlayfs exactly where the kernel lacks overlayfs but has FUSE (read from /proc/filesystems, kernel-global even from a container; an overlay-family storage driver also counts as proof), nothing anywhere else. The new instance-wide Ci:BuildkitSnapshotter option overrides: auto | none | explicit name.
  • Delivered via a third per-repo volume mounted at /home/runner/_buildx, exported as BUILDX_CONFIG, so every setup-buildx-action inherits it with zero workflow changes (a workflow's own buildkitd-config(-inline) still wins). A volume rather than a file bind on purpose — dockerd creates bind parents root-owned (the _work trap; a mount under ~/.docker would break the next docker login). The existing volume-init container writes the file (content via env, printf '%s') and re-runs on content change, converging every pass so runtime registry changes reach repos whose slots are all busy.
  • Runner spec-hash material gained a version prefix so pre-existing idle runners are recycled once and pick up the new mount/env.

Proposal B — the fast path as one uses: line

.github/workflows/build-push-image.yml (reusable) encodes the docker-driver build for hosts where the container driver is the wrong tool: no buildx builder → the daemon's own builder (real CoW via the host storage driver, cache persists across ephemeral runners, insecure-registries inherited natively), provenance: false, reading the REGISTRY variable/secrets Watchtower already syncs. Consumers: uses: swimmesberger/Watchtower/.github/workflows/build-push-image.yml@main + secrets: inherit.

Docs

New "Container image builds" section in docs/ci-runners/design.md and ADR-0028, including the probe procedure and its NAS result, and the operational notes from the issue (docker builder prune as future maintenance work; exporting layers is gzip-bound — not a second bug). Also indexed the missing ADR-0027 line in the decisions README.

The open question — answered

The issue asked whether any OCI snapshotter can work on DSM. The probe was run on the NAS (2026-08-28): FUSE present, /dev/fuse present, and moby/buildkit --oci-worker-snapshotter=fuse-overlayfs starts cleanly with a registered fuse-overlayfs worker. So the native fallback is purely BuildKit's probe never attempting fuse-overlayfs — exactly the gap the auto-detection fills. After merge + deploy, the NAS needs no configuration at all; Ci:BuildkitSnapshotter=none is the escape hatch if a host's FUSE misbehaves in a real build.

Reviewer notes

  • Startup of the probed buildkitd proves the snapshotter initialises; the first real MuxBox build on a runner is the end-to-end confirmation for fuse-overlayfs on DSM.
  • GET /info is called once per reconcile pass and only when at least one CI repo exists.
  • Invalid Ci:BuildkitSnapshotter values warn once per distinct value and fall back to detection; the resolved choice is logged once on change.
  • Tests: CiBuildkitConfigTests (TOML shape, host-detection matrix incl. a DSM-verbatim /proc/filesystems fixture and the fuseblkfuse trap) and extended CiRunnerContainerSpecTests (buildx mount/env, the ~/.docker trap guard, volume-init writes the config without splicing it into the script). Full suites: 1877 + 339 passing.

BuildKit's docker-container driver reads none of the host daemon's
configuration, and its snapshotter auto-probe is overlayfs-or-native —
it never tries fuse-overlayfs. On hosts whose kernel lacks overlayfs
(Synology DSM) builds therefore silently get the copy-less native
snapshotter (~10x slower), and plain-HTTP registries need a hand-written
buildkitd-config-inline stanza in every consuming workflow.

- Generate a default buildkitd.default.toml per reconcile pass
  (CiBuildkitConfig): [registry] http/insecure stanzas for exactly the
  registries the daemon itself treats as insecure (GET /info), plus a
  [worker.oci] snapshotter that is auto-detected from the host by
  default — fuse-overlayfs exactly where the kernel lacks overlayfs but
  has FUSE (probed on the NAS: an explicit fuse-overlayfs worker starts
  cleanly there), nothing anywhere else. The instance-wide
  Ci:BuildkitSnapshotter option overrides (auto | none | a name).
- Deliver it via a third per-repo volume mounted at /home/runner/_buildx
  and exported as BUILDX_CONFIG, so every docker/setup-buildx-action
  inherits it with no workflow change; a volume rather than a file bind
  because dockerd creates bind parents root-owned (the _work trap). The
  volume-init container writes the file and re-runs on content change,
  converging every pass so runtime registry changes reach busy repos.
- Bump the runner spec-hash material so pre-existing idle runners are
  recycled once and pick up the new mount and env.
- Add a reusable build-push-image.yml workflow encoding the docker
  driver fast path (real CoW via the host storage driver, persistent
  daemon cache, provenance off), reading the REGISTRY variable/secrets
  Watchtower already syncs.
- Document the mechanism, the DSM probe result, and the operational
  notes (builder prune, gzip-bound export) in the CI runners design doc
  and ADR-0028; index the missing ADR-0027 line.

Closes #65
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI runners: BuildKit falls back to the native snapshotter on DSM, making container builds ~10x slower

1 participant