Skip to content

fix(ci): give docker-socket runners the host's docker group - #62

Merged
swimmesberger merged 1 commit into
mainfrom
wt/watchtower-ci-stack-failure-9716d9
Aug 27, 2026
Merged

fix(ci): give docker-socket runners the host's docker group#62
swimmesberger merged 1 commit into
mainfrom
wt/watchtower-ci-stack-failure-9716d9

Conversation

@swimmesberger

Copy link
Copy Markdown
Owner

A CI job on a stack-linked runner failed at docker/setup-buildx-action with
dial unix /var/run/docker.sock: connect: no such file or directory
(MuxBox run).
Tracing that path turned up two gaps in the runner orchestrator — the second
would have bitten even after the operator enabled the socket.

1. The socket mount alone doesn't grant access

AllowDockerSocket bind-mounted /var/run/docker.sock and nothing else. The
socket belongs to the host's docker group, whose numeric id the operator
grants Watchtower itself (group_add: ["999"] in
deploy/docker/docker-compose.yml), while ghcr.io/actions/actions-runner
gives its non-root runner user a docker group with a hardcoded id of 123.
So the socket was present but unusable — the next failure would have been
permission denied while trying to connect to the Docker daemon socket.

Runner containers now carry Watchtower's own supplementary group ids as
HostConfig.GroupAdd, which is what SelfUpdateService already does for its
socket-mounted coordinator container. The shared /proc/self/status reader
moves out of that service into HostSupplementaryGroups.

2. Settings changes never reached a running runner

Idle runners are long-lived — they sit long-polling GitHub until a job
arrives — and ReconcileRepoAsync only ever topped slots up. Ticking "allow
docker socket" therefore left the already-running runner in place with its old
spec, and the very next build still failed on the socket just granted.

Runner containers now carry watchtower.ci.spec-hash — a hash of the settings
baked in at spawn time (image, docker socket, extra labels). On a mismatch the
reconcile loop retires the runner. Deregistering at GitHub first doubles as the
idleness check: GitHub refuses to delete a runner that is executing a job, so a
busy runner is kept and replaced after it exits on its own. The first refusal
ends the pass, so a repo whose runners are all busy costs one API call per
reconcile interval rather than one per runner, while idle runners are still all
replaced within a single pass.

Notes for the reviewer

  • TryDeleteRegistrationAsync now returns whether GitHub confirmed the runner
    is gone; the two pre-existing call sites ignore it, unchanged in behaviour.
  • The running.Count >= desired early return moved below the recycle step —
    otherwise a full complement of stale runners would never be reconsidered. The
    backoff early return stays above it, so a repo in spawn backoff makes no
    extra API calls.
  • Fix 1 assumes the documented deployment (non-root USER watchtower plus
    group_add with the host docker GID) — the same assumption self-update
    already relies on.
  • SelfUpdateGroupParsingTests is renamed to HostSupplementaryGroupsTests
    along with the code it covers.

Verification

Release build clean, 0 warnings (TreatWarningsAsErrors on). The unit tests
touched and added pass. The rest of Watchtower.Application.Tests could not run
locally — every failure is DockerUnavailableException from Testcontainers
because Docker is not running on this machine; CI is the judge there.

A runner with `AllowDockerSocket` got the socket bind-mounted and nothing
else. The socket belongs to the host's docker group, whose numeric id the
operator grants Watchtower itself (`group_add` in compose), while the
actions-runner image gives its non-root `runner` user a `docker` group with
a hardcoded id of 123 — so the socket was present but unusable. Runners now
carry Watchtower's own supplementary group ids, the same mechanism the
self-update coordinator already uses for its socket mount; the shared
`/proc/self/status` reader moves to `HostSupplementaryGroups`.

Second half of the same failure: idle runners are long-lived (they sit
long-polling GitHub until a job arrives), and reconcile only ever topped
slots up — so ticking "allow docker socket" left the already-running runner
in place and the very next build still failed on the socket just granted.
Runner containers now carry a hash of the settings they were spawned with;
a mismatch retires them. Deregistering at GitHub first doubles as the
idleness check (GitHub refuses to delete a runner that is executing a job),
and the first refusal ends the pass so a busy repo costs one API call per
interval rather than one per runner.
@swimmesberger
swimmesberger merged commit 8c8a3c8 into main Aug 27, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant