Let heartbeats re-register testrunners the server has forgotten - #272
Merged
Merged
Conversation
The index page showed "Waiting for test runners to come online" while the API kept accepting and running jobs. The online state comes from an in-memory registry fed only by a runner's start broadcast, while API queue routing lives in separate add-only maps. The registry prune only needs the local clock, but heartbeat delivery needs Redis — so a Redis outage longer than the stale window got a perfectly healthy runner pruned. Once pruned, its heartbeats were deliberately ignored and nothing ever asked it to announce itself again, so the state stuck until a process restart. Heartbeats now carry the runner's serverConfig so the server can re-register an unknown-but-alive runner on the next beat, making the registry self-healing within one heartbeat interval. Old runners without the payload keep the previous behavior. Heartbeat jobs are also cleaned up on completion now — they fire every 30s and were accumulating in Redis forever. Co-authored-by: Claude Fable 5 noreply@anthropic.com
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The index page showed "Waiting for test runners to come online" while the API kept accepting and running jobs. The online state comes from an in-memory registry fed only by a runner's start broadcast, while API queue routing lives in separate add-only maps. The registry prune only needs the local clock, but heartbeat delivery needs Redis — so a Redis outage longer than the stale window got a perfectly healthy runner pruned. Once pruned, its heartbeats were deliberately ignored and nothing ever asked it to announce itself again, so the state stuck until a process restart.
Heartbeats now carry the runner's serverConfig so the server can re-register an unknown-but-alive runner on the next beat, making the registry self-healing within one heartbeat interval. Old runners without the payload keep the previous behavior. Heartbeat jobs are also cleaned up on completion now — they fire every 30s and were accumulating in Redis forever.
Co-authored-by: Claude Fable 5 noreply@anthropic.com