fix(ci): give docker-socket runners the host's docker group - #62
Merged
Conversation
A runner with `AllowDockerSocket` got the socket bind-mounted and nothing else. The socket belongs to the host's docker group, whose numeric id the operator grants Watchtower itself (`group_add` in compose), while the actions-runner image gives its non-root `runner` user a `docker` group with a hardcoded id of 123 — so the socket was present but unusable. Runners now carry Watchtower's own supplementary group ids, the same mechanism the self-update coordinator already uses for its socket mount; the shared `/proc/self/status` reader moves to `HostSupplementaryGroups`. Second half of the same failure: idle runners are long-lived (they sit long-polling GitHub until a job arrives), and reconcile only ever topped slots up — so ticking "allow docker socket" left the already-running runner in place and the very next build still failed on the socket just granted. Runner containers now carry a hash of the settings they were spawned with; a mismatch retires them. Deregistering at GitHub first doubles as the idleness check (GitHub refuses to delete a runner that is executing a job), and the first refusal ends the pass so a busy repo costs one API call per interval rather than one per runner.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A CI job on a stack-linked runner failed at
docker/setup-buildx-actionwithdial unix /var/run/docker.sock: connect: no such file or directory(MuxBox run).
Tracing that path turned up two gaps in the runner orchestrator — the second
would have bitten even after the operator enabled the socket.
1. The socket mount alone doesn't grant access
AllowDockerSocketbind-mounted/var/run/docker.sockand nothing else. Thesocket belongs to the host's
dockergroup, whose numeric id the operatorgrants Watchtower itself (
group_add: ["999"]indeploy/docker/docker-compose.yml), whileghcr.io/actions/actions-runnergives its non-root
runneruser adockergroup with a hardcoded id of 123.So the socket was present but unusable — the next failure would have been
permission denied while trying to connect to the Docker daemon socket.Runner containers now carry Watchtower's own supplementary group ids as
HostConfig.GroupAdd, which is whatSelfUpdateServicealready does for itssocket-mounted coordinator container. The shared
/proc/self/statusreadermoves out of that service into
HostSupplementaryGroups.2. Settings changes never reached a running runner
Idle runners are long-lived — they sit long-polling GitHub until a job
arrives — and
ReconcileRepoAsynconly ever topped slots up. Ticking "allowdocker socket" therefore left the already-running runner in place with its old
spec, and the very next build still failed on the socket just granted.
Runner containers now carry
watchtower.ci.spec-hash— a hash of the settingsbaked in at spawn time (image, docker socket, extra labels). On a mismatch the
reconcile loop retires the runner. Deregistering at GitHub first doubles as the
idleness check: GitHub refuses to delete a runner that is executing a job, so a
busy runner is kept and replaced after it exits on its own. The first refusal
ends the pass, so a repo whose runners are all busy costs one API call per
reconcile interval rather than one per runner, while idle runners are still all
replaced within a single pass.
Notes for the reviewer
TryDeleteRegistrationAsyncnow returns whether GitHub confirmed the runneris gone; the two pre-existing call sites ignore it, unchanged in behaviour.
running.Count >= desiredearly return moved below the recycle step —otherwise a full complement of stale runners would never be reconsidered. The
backoff early return stays above it, so a repo in spawn backoff makes no
extra API calls.
USER watchtowerplusgroup_addwith the host docker GID) — the same assumption self-updatealready relies on.
SelfUpdateGroupParsingTestsis renamed toHostSupplementaryGroupsTestsalong with the code it covers.
Verification
Release build clean, 0 warnings (
TreatWarningsAsErrorson). The unit teststouched and added pass. The rest of
Watchtower.Application.Testscould not runlocally — every failure is
DockerUnavailableExceptionfrom Testcontainersbecause Docker is not running on this machine; CI is the judge there.