Skip to content

fix: a shutdown between cache sync and handler registration is not a failure - #102

Merged
kfeofantov merged 1 commit into
mainfrom
fix/shutdown-is-not-a-failure
Sep 3, 2026
Merged

kfeofantov merged 1 commit into
mainfrom
fix/shutdown-is-not-a-failure

Conversation

@kfeofantov

Copy link
Copy Markdown
Contributor

Two bugs of one class, both on the shutdown path.

The watchers. An informer refuses new handlers once it has stopped, and the
informers stop when the run context is canceled — so a SIGTERM arriving between
the cache-sync wait and AddEventHandler made PodWatcher.Run and
NodeWatcher.Run return an error, which takes the controller down with exit 1
on an ordinary stop. It is also what made TestCollectsContainersAndImages
flake in CI: that test cancels as soon as it has seen its pods, which is exactly
this window. The sync path already returned nil for a canceled context; the
registration path now does the same, while still surrendering the watchdog's
verdict when that is what canceled the run.

The flush passes. The periodic pass now joins the WaitGroup. Without it,
cancellation does not wait for the goroutine's current iteration, so it could
still be flushing when the shutdown pass ran — and both write the same payload
keys through a Spool that has no lock and derives one temp file name per key.
The comment claiming the shutdown pass was "the only writer" was untrue.

The set of writers is now one list rather than a call sequence per pass. As two
hand-written sequences they had already drifted: flushMetadata was in the
periodic pass and absent from the shutdown one — correctly, since it re-derives
a snapshot from watcher indexes that have stopped by then, but nothing said so.
That is now a field with its reason.

Evidence

The two new watcher tests fail before the fix with the flake's own message:

Run = register pod handler: handler {...} was not added to shared informer
because it has stopped already, want nil

$ go test -race ./...            # green
$ go test -race -count=20 -run TestCollectsContainersAndImages ./internal/collector/
ok  github.com/RebuildStackCo/runtime-agent/internal/collector  6.638s
$ golangci-lint run ./...        # 0 issues
$ make vulncheck chart-lint      # clean

No ADR: no decision changes. No docs/security.md change: nothing collected,
stored or transmitted is different.

e2e has not been run — it needs Docker, which is yours to start.

🤖 Generated with Claude Code

https://claude.ai/code/session_011YRKuL97KRndaopfpixrhp

…failure

An informer refuses new handlers once it has stopped, and the informers stop
when the run context is canceled — so a SIGTERM arriving between the sync wait
and AddEventHandler made both watchers return an error, taking the controller
down with exit 1 on an ordinary stop. It is also what made
TestCollectsContainersAndImages flake: that test cancels as soon as it has seen
its pods, which is exactly this window. The sync path already drew the
distinction; the registration path now does too.

The controller's periodic flush pass joins the WaitGroup for the same class of
bug one level up: the shutdown pass writes the same payload keys, the spool has
no lock, and two writers of one key share a temp file name. The set of writers
becomes one list instead of a call sequence per pass — as two hand-written
sequences they had already drifted, with the metadata flush in one and not the
other.

go test -race ./... green, including -count=20 of the previously flaky test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011YRKuL97KRndaopfpixrhp
@kfeofantov
kfeofantov merged commit e3d21f2 into main Sep 3, 2026
3 checks passed
@kfeofantov
kfeofantov deleted the fix/shutdown-is-not-a-failure branch September 3, 2026 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant