CI runners: BuildKit falls back to the `native` snapshotter on DSM, making container builds ~10x slower

[swimmesberger]

Summary

Any repo on a Watchtower-provisioned runner that builds an image with
docker/setup-buildx-action (the docker-container driver, i.e. the documented default)
silently gets BuildKit's native snapshotter. native has no copy-on-write: every layer
materialisation is a full recursive copy of the accumulated image tree. Builds run roughly
an order of magnitude slower than they should, and nothing in the job log says why.

Today the only way out is per-repo knowledge in ci.yml about how this specific host's
Docker is put together. That knowledge belongs in Watchtower, not in every workflow.

Evidence

From MuxBox's publish job (run 33150768868, Build & push image = 15m43s):

#23 extracting sha256:671743…    58.7s     ← a 93-byte layer
#23 extracting sha256:b362e6…    58.4s     ← a 476-byte layer
#23 [api 7/11] RUN dotnet restore  490.5s  ← a CACHE HIT; 471s of it is extraction
#24 [web 4/7] RUN npm ci           216.9s  ← also a cache hit
#26 74.39 truehdd.tar.gz: OK                ← curl+tar of a 5MB binary starts 74s in
#30 96.44 > muxbox-web@1.0.0 prebuild       ← npm run build starts 96s in (12s of real work)
#40 exporting layers               123.4s
#40 pushing layers                  10.4s  ← the actual upload

Extraction cost is proportional to the cumulative stage size, not the layer size — which
is why a 93-byte layer costs a minute on top of the .NET SDK. 26 layer extractions totalled
798.6s.

The previous run (33149601285)
is the cleaner datapoint: every step a cache hit, pushing layers 0.3s (nothing new to
upload), and the step still took 13m20s. That rules out the network and the registry —
both of which are on this same box anyway.

Confirmation in the Set up Docker Buildx output:

org.mobyproject.buildkit.worker.snapshotter:  native

Root cause

docker info on the host:

 Operating System: Synology NAS
 Kernel Version:   4.4.302+
 Server Version:   24.0.2
 Storage Driver:   btrfs
 Docker Root Dir:  /volume3/@docker
 CPUs: 4      Total Memory: 9.567GiB

BuildKit's OCI worker picks its snapshotter with auto: try overlayfs, then fuse-overlayfs,
then fall back to native
(snapshotterFactory).
On DSM's 4.4 kernel both probes fail, so it lands on native.

Note the accepted values are only native, overlayfs, fuse-overlayfs, stargz, auto —
btrfs is not one of them. btrfs is a containerd snapshotter, exposed by BuildKit's
containerd worker but not by the OCI worker that the docker-container driver runs. So
"just point it at btrfs" is not available.

Why this can't be fixed on the host

Every lever is pinned by DSM: the kernel (no overlayfs, and we're not rebuilding a Synology
kernel), the storage driver (btrfs, chosen by the package because overlay2 isn't
available), and the engine version (24.0.2). There is no host-side change that makes auto
resolve to something better.

What a repo has to do today

MuxBox's ci.yml has to carry two host-specific facts: that the plain-HTTP registry needs a
buildkitd-config-inline stanza, and (once fixed) that the docker-container driver must be
avoided in favour of driver: docker, whose builder runs inside the daemon and therefore uses
the btrfs graphdriver — real CoW, and a cache that persists on the host between runs.

That's exactly the "special knowledge crafting the ci.yaml" the CI-runners design set out to
avoid ("Zero-ceremony enablement", "Workflow YAML stays fully standard").

Proposal A — ship a default buildkitd config into runners

buildx reads a default buildkitd config when the workflow doesn't pass one of its own, in
this order
(docs):

$BUILDX_CONFIG/buildkitd.default.toml
$DOCKER_CONFIG/buildx/buildkitd.default.toml
~/.docker/buildx/buildkitd.default.toml

If CiRunnerOrchestrator.BuildRunnerContainerBody mounts such a file, every repo's
docker/setup-buildx-action inherits it with no workflow change:

[worker.oci]
  snapshotter = "<whatever actually works here>"

# Watchtower already knows the registry list (RegistryAuthBuilder.ListResolvedRegistriesAsync
# / CiRepo.SyncRegistryUrl), so it can emit this instead of each repo hand-writing it.
[registry."simondatastore:50000"]
  http = true

The registry half of this is worth doing regardless — it deletes buildkitd-config-inline
from every consuming workflow. It also pairs naturally with mechanism 2 in the design doc
("Registry auth without any secret"), which would mount a pre-authenticated DOCKER_CONFIG
anyway; the buildx config can live in that same directory.

Gotcha: dockerd creates missing bind-mount parents as root, so a naive bind at
/home/runner/.docker/buildx/buildkitd.default.toml leaves /home/runner/.docker root-owned
and will break a later docker login in a job — the same failure mode already documented for
_work and for fresh named volumes. It needs the existing ci-volume-init chown one-shot, or
a DOCKER_CONFIG pointed at a properly-chowned volume.

Open question — is there a working snapshotter at all?

Proposal A only helps if some OCI-worker snapshotter works here. auto already rejected
both candidates, but it's worth knowing whether that's "the kernel has neither" or something
fixable (e.g. fuse-overlayfs failing only because /dev/fuse isn't exposed to the builder
container). To run on the NAS:

grep -E 'overlay|fuse' /proc/filesystems
ls -l /dev/fuse
docker run --rm --privileged moby/buildkit:latest --debug 2>&1 | head -40
docker run --rm --privileged moby/buildkit:latest --oci-worker-snapshotter=fuse-overlayfs 2>&1 | head -20

If neither overlayfs nor fuse-overlayfs can be made to work, Proposal A can still carry the
registry config but cannot fix the snapshotter, and we fall through to B.

Proposal B — make the fast path the documented one

If the docker-container driver is simply not viable on this host, then the guidance for
every repo is "don't use it": either drop docker/setup-buildx-action entirely (the default
builder is the docker driver) or pass driver: docker. Rather than restating that in
each repo, Watchtower could publish a reusable workflow —
swimmesberger/Watchtower/.github/workflows/build-push-image.yml@main — that encodes the
driver choice, provenance: false (the docker driver can't do attestations), and the fact
that registry cache import/export is unnecessary when the daemon's cache persists. Repos then
carry one uses: line and no host knowledge.

Worth surfacing the resulting cache growth on /volume3/@docker too, since the daemon's
build cache is no longer thrown away with the builder container — docker builder prune
belongs somewhere in Watchtower's maintenance story.

Also worth documenting either way

Even after this is fixed, #40 exporting layers was 123s and a good chunk of that is gzip on
4 cores. Runners on this box will not reach GitHub-hosted speeds, and the docs should say so
so the next person doesn't go looking for another bug.

[swimmesberger]

Post-merge correction to how this was fixed. The issue's probe-chain description was right and #66's premise was wrong: auto does call fuseoverlayfs.Supported() before falling back to native (main_oci_worker.go), and Supported() is a real functional check (check.go: LookPath + an actual read-only multi-lowerdir test mount). An explicitly named snapshotter skips that check entirely — which is why the forced buildkitd "started cleanly": it registers the worker without ever mounting anything.

Running the real check on the NAS found the actual root cause: fuse-overlayfs: not found — the standard moby/buildkit image doesn't ship the binary at all (only the -rootless variant does). So auto can never pick fuse-overlayfs under the stock docker-container builder on any host, and forcing it would have failed the first build at the first layer mount. The open question is thereby answered: not viable in the stock setup; a fix would need a custom builder image via driver-opts: image=… plus a capable kernel — per-repo host knowledge again.

#69 reverts the snapshotter auto-detection from #66 to an explicit, verified-only knob; the docker-driver reusable workflow remains the fast path on this host, as Proposal B anticipated.