Skip to content

feat(ci): add operator-requested recycle of runner containers - #68

Merged
swimmesberger merged 1 commit into
mainfrom
wt/action-runner-container-lifecycle-81174f
Aug 28, 2026
Merged

feat(ci): add operator-requested recycle of runner containers#68
swimmesberger merged 1 commit into
mainfrom
wt/action-runner-container-lifecycle-81174f

Conversation

@swimmesberger

Copy link
Copy Markdown
Owner

What

Adds manual lifecycle control over the CI runner containers shown on the product CI tab: recycle (force-recreate) one runner or the whole pool.

  • ci.recycleRunner — recycles one runner container (identified by its 12-char short id; shorter prefixes are rejected so an ambiguous prefix can't silently resolve to a different runner).
  • ci.recycleRunners — recycles the repo's whole pool.
  • CiRunnerOrchestrator.RecycleRunnersAsync — the shared implementation: deregister at GitHub → remove the container → wake the reconcile loop, which respawns a fresh runner under the current settings. The manual counterpart to the automatic spec-hash recycle, without the hash gate.
  • UI — recycle button per runner row (desktop table + mobile card) and a "Recycle all" action on the card, with toasts and query invalidation following the stacks ContainerCard pattern.

Why

Runner containers were fully owned by the reconcile loop; the only manual levers were the enabled toggle and the runner count. There was no way to force-recreate a runner (fresh container, re-pulled image, post-incident cleanup) short of toggling the whole repo off and on.

Busy runners / force

As in the automatic stale recycle, GitHub deregistration doubles as the idleness check: GitHub refuses to delete a runner mid-job, so a busy runner is kept and reported as such. Passing force removes the container anyway — failing the job it was executing — and the UI escalates to that only through an explicit "Kill and recycle" confirm dialog (per-runner and pool-wide).

Plain stop/start/restart is deliberately not offered: a stopped JIT runner is a dead registration the loop would replace within seconds, so recycle is the only per-runner operation that means anything; "temporarily stop" remains the enabled toggle.

Notes for review

  • Container ids resolve only against containers carrying the repo's own watchtower.managed=ci-runner + watchtower.ci.repo-id labels — unrelated containers cannot be targeted through this path.
  • Recycles are audited (ci / runner.recycle), matching ci.updateRepo.
  • GitHubApiClient.TryDeleteRunnerAsync becomes virtual for the test double, matching the other stubbed methods.
  • rpc-schema.json re-exported (152 methods) and the generated web client regenerated.
  • A container that cannot be confirmed idle (missing runner-id label, no credential) is treated as busy, same as the loop does; force is the escape hatch.

Testing

  • 9 new tests in CiRunnerRecycleTests.cs against the AuthTestHost harness with a fake Docker daemon that applies label filters (the cross-repo isolation is part of what's under test): idle/busy/forced recycle, unknown container → NotFound, foreign-repo container unreachable, short-id rejection, pool mix, unknown repo.
  • All 150 CI-related backend tests pass; web app typechecks.
  • Not smoke-tested against a live CI repo — worth one click on a real runner row before merging.

Runner containers were fully owned by the reconcile loop: the only manual
levers were the enabled toggle and the runner count, and the CI tab's
runner table was read-only. Operators had no way to force-recreate a
runner (e.g. to pick up a new image tag or just get a fresh container)
short of toggling the whole repo off and on.

Add ci.recycleRunner (one container, 12-char short id) and
ci.recycleRunners (the whole pool), backed by
CiRunnerOrchestrator.RecycleRunnersAsync: deregister the runner at
GitHub, remove the container, and wake the loop so a replacement spawns
under the current settings. As in the automatic stale recycle,
deregistration doubles as the idleness check - a runner GitHub refuses
to release is executing a job and is kept (reported busy) unless force
is set, which removes it anyway and fails that job. Container ids
resolve only against the repo's own labelled containers, so unrelated
containers cannot be targeted through this path. Recycles are audited.

The CI tab's runner table gets a recycle button per row plus a
"Recycle all" card action; a busy answer escalates through a confirm
dialog to the force variant. Plain stop/start/restart is deliberately
not offered - a stopped JIT runner is a dead registration the loop
would replace within seconds, so recycle is the only per-runner
operation that means anything.

GitHubApiClient.TryDeleteRunnerAsync becomes virtual for the test
double, matching the other stubbed methods.
@swimmesberger
swimmesberger merged commit 5735abb into main Aug 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant