Repository navigation
fix: a shutdown between cache sync and handler registration is not a failure - #102
Merged
Merged
Conversation
…failure An informer refuses new handlers once it has stopped, and the informers stop when the run context is canceled — so a SIGTERM arriving between the sync wait and AddEventHandler made both watchers return an error, taking the controller down with exit 1 on an ordinary stop. It is also what made TestCollectsContainersAndImages flake: that test cancels as soon as it has seen its pods, which is exactly this window. The sync path already drew the distinction; the registration path now does too. The controller's periodic flush pass joins the WaitGroup for the same class of bug one level up: the shutdown pass writes the same payload keys, the spool has no lock, and two writers of one key share a temp file name. The set of writers becomes one list instead of a call sequence per pass — as two hand-written sequences they had already drifted, with the metadata flush in one and not the other. go test -race ./... green, including -count=20 of the previously flaky test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011YRKuL97KRndaopfpixrhp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two bugs of one class, both on the shutdown path.
The watchers. An informer refuses new handlers once it has stopped, and the
informers stop when the run context is canceled — so a SIGTERM arriving between
the cache-sync wait and
AddEventHandlermadePodWatcher.RunandNodeWatcher.Runreturn an error, which takes the controller down with exit 1on an ordinary stop. It is also what made
TestCollectsContainersAndImagesflake in CI: that test cancels as soon as it has seen its pods, which is exactly
this window. The sync path already returned nil for a canceled context; the
registration path now does the same, while still surrendering the watchdog's
verdict when that is what canceled the run.
The flush passes. The periodic pass now joins the
WaitGroup. Without it,cancellation does not wait for the goroutine's current iteration, so it could
still be flushing when the shutdown pass ran — and both write the same payload
keys through a
Spoolthat has no lock and derives one temp file name per key.The comment claiming the shutdown pass was "the only writer" was untrue.
The set of writers is now one list rather than a call sequence per pass. As two
hand-written sequences they had already drifted:
flushMetadatawas in theperiodic pass and absent from the shutdown one — correctly, since it re-derives
a snapshot from watcher indexes that have stopped by then, but nothing said so.
That is now a field with its reason.
Evidence
The two new watcher tests fail before the fix with the flake's own message:
No ADR: no decision changes. No
docs/security.mdchange: nothing collected,stored or transmitted is different.
e2e has not been run — it needs Docker, which is yours to start.
🤖 Generated with Claude Code
https://claude.ai/code/session_011YRKuL97KRndaopfpixrhp