feat(ci): add operator-requested recycle of runner containers - #68
Merged
Merged
Conversation
Runner containers were fully owned by the reconcile loop: the only manual levers were the enabled toggle and the runner count, and the CI tab's runner table was read-only. Operators had no way to force-recreate a runner (e.g. to pick up a new image tag or just get a fresh container) short of toggling the whole repo off and on. Add ci.recycleRunner (one container, 12-char short id) and ci.recycleRunners (the whole pool), backed by CiRunnerOrchestrator.RecycleRunnersAsync: deregister the runner at GitHub, remove the container, and wake the loop so a replacement spawns under the current settings. As in the automatic stale recycle, deregistration doubles as the idleness check - a runner GitHub refuses to release is executing a job and is kept (reported busy) unless force is set, which removes it anyway and fails that job. Container ids resolve only against the repo's own labelled containers, so unrelated containers cannot be targeted through this path. Recycles are audited. The CI tab's runner table gets a recycle button per row plus a "Recycle all" card action; a busy answer escalates through a confirm dialog to the force variant. Plain stop/start/restart is deliberately not offered - a stopped JIT runner is a dead registration the loop would replace within seconds, so recycle is the only per-runner operation that means anything. GitHubApiClient.TryDeleteRunnerAsync becomes virtual for the test double, matching the other stubbed methods.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds manual lifecycle control over the CI runner containers shown on the product CI tab: recycle (force-recreate) one runner or the whole pool.
ci.recycleRunner— recycles one runner container (identified by its 12-char short id; shorter prefixes are rejected so an ambiguous prefix can't silently resolve to a different runner).ci.recycleRunners— recycles the repo's whole pool.CiRunnerOrchestrator.RecycleRunnersAsync— the shared implementation: deregister at GitHub → remove the container → wake the reconcile loop, which respawns a fresh runner under the current settings. The manual counterpart to the automatic spec-hash recycle, without the hash gate.ContainerCardpattern.Why
Runner containers were fully owned by the reconcile loop; the only manual levers were the enabled toggle and the runner count. There was no way to force-recreate a runner (fresh container, re-pulled image, post-incident cleanup) short of toggling the whole repo off and on.
Busy runners /
forceAs in the automatic stale recycle, GitHub deregistration doubles as the idleness check: GitHub refuses to delete a runner mid-job, so a busy runner is kept and reported as such. Passing
forceremoves the container anyway — failing the job it was executing — and the UI escalates to that only through an explicit "Kill and recycle" confirm dialog (per-runner and pool-wide).Plain stop/start/restart is deliberately not offered: a stopped JIT runner is a dead registration the loop would replace within seconds, so recycle is the only per-runner operation that means anything; "temporarily stop" remains the enabled toggle.
Notes for review
watchtower.managed=ci-runner+watchtower.ci.repo-idlabels — unrelated containers cannot be targeted through this path.ci/runner.recycle), matchingci.updateRepo.GitHubApiClient.TryDeleteRunnerAsyncbecomesvirtualfor the test double, matching the other stubbed methods.rpc-schema.jsonre-exported (152 methods) and the generated web client regenerated.forceis the escape hatch.Testing
CiRunnerRecycleTests.csagainst the AuthTestHost harness with a fake Docker daemon that applies label filters (the cross-repo isolation is part of what's under test): idle/busy/forced recycle, unknown container → NotFound, foreign-repo container unreachable, short-id rejection, pool mix, unknown repo.