Ignore context-canceled errors when logging drift detector shutdown failures - #7328
Ignore context-canceled errors when logging drift detector shutdown failures#7328pujitha24 wants to merge 2 commits into
Conversation
…ailures Motivation: When a piped agent shuts down, its drift-detector context is canceled while a check cycle may still be in flight. This causes in-flight operations to fail (e.g. kustomize being killed with "signal: killed", or a git command returning "context canceled"), and those expected, benign failures were logged at Error level as "failed to check application: ..." and "failed to clean partially cloned repository", producing log spam / false alarms at shutdown time. Approach: In the kubernetes drift detector's check method, both the checkApplication call and the gitRepo.CleanPath cleanup call still run unconditionally as before, but the resulting error is now only logged at Error level when ctx.Err() == nil. ctx.Err() is checked directly (rather than errors.Is(err, context.Canceled)) because a process killed via exec.CommandContext on context cancellation (e.g. kustomize) returns "signal: killed", which does not wrap context.Canceled in Go's os/exec implementation, so checking the context directly reliably covers both error shapes reported in the issue. Validation: Added TestCheck_ContextCanceled in detector_test.go, which drives detector.check with a mocked git.Repo whose CleanPath call fails and whose repository path lacks an application config file (making checkApplication fail quickly, independent of context state). With a non-canceled context both failures are logged as Error (2 entries); with an already-canceled context, zero Error logs are produced, while the mocked CleanPath/Pull/GetLatestCommit calls are still asserted to run. Verified this test fails (2 unwanted Error logs) against the pre-fix code and passes after the fix. Ran and confirmed passing: - go test ./pkg/app/piped/driftdetector/kubernetes/... -run TestCheck_ContextCanceled -v - go test ./pkg/app/piped/driftdetector/... -v - go build ./pkg/app/piped/driftdetector/... - go vet ./pkg/app/piped/driftdetector/... - golangci-lint run --config .golangci.yml ./pkg/app/piped/driftdetector/... This is a logging-only behavior change: cleanup and manifest-check operations still execute exactly as before, only the two spurious Error-level log lines during graceful shutdown are removed. Scope is limited to the kubernetes drift detector, the exact file the issue and a maintainer's comment point to, and the only drift detector with the CleanPath cleanup call. The terraform/ecs/lambda/cloudrun detectors have a similar checkApplication error-log line but weren't reported in this issue and are left untouched. Report: pipe-cd#5338 Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com> Assisted-by: claude-sonnet-5 (via Claude Code)
✅ Deploy Preview for pipecd-site canceled.
|
There was a problem hiding this comment.
🟢 Approval recommended
The change is narrow (logging-only), aligns with the stated shutdown-cancellation behavior, and is covered by a targeted unit test.
Pull request overview
Reduces shutdown-time log noise in the legacy piped Kubernetes drift detector by suppressing expected error logs when the drift detector’s context has already been canceled, and adds a unit test to validate the behavior.
Changes:
- Suppress
Error-level logs fromcheckApplicationandgitRepo.CleanPathwhenctx.Err() != nil(shutdown/cancellation scenario). - Add a unit test covering both canceled vs non-canceled context cases and asserting error-log emission behavior.
File summaries
| File | Description |
|---|---|
| pkg/app/piped/driftdetector/kubernetes/detector.go | Gate two existing error logs on ctx.Err() == nil to avoid noisy shutdown-time cancellation errors. |
| pkg/app/piped/driftdetector/kubernetes/detector_test.go | Adds TestCheck_ContextCanceled to verify error logs are suppressed when the context is already canceled. |
Review details
Suppressed comments (2)
pkg/app/piped/driftdetector/kubernetes/detector.go:188
- This comment also implies the error is "caused by" context cancellation, but the condition only checks ctx.Err(). Consider rewording to match the actual behavior (suppress errors when the context is canceled).
// This clean is only partial; the entire cleanup is performed elsewhere,
// so ignore the error when it's caused by the context being canceled.
pkg/app/piped/driftdetector/kubernetes/detector_test.go:123
- Wire the no-op reporter into the detector under test so the test doesn’t depend on an early error return to avoid calling a nil reporter.
d := &detector{
provider: config.PipedPlatformProvider{Name: "kubernetes-default"},
appLister: &fakeAppLister{apps: []*model.Application{app}},
stateGetter: fakeStateGetter{},
appManifestsCache: memorycache.NewCache(),
config: &config.PipedSpec{},
logger: logger,
gitRepos: map[string]git.Repo{"repo-1": repo},
syncStates: make(map[string]model.ApplicationSyncState),
}
- Files reviewed: 2/2 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // Ignore the error caused by the context being canceled (e.g. piped is | ||
| // shutting down); it's an expected cancellation, not a real failure. |
| func (fakeStateGetter) WaitForReady(ctx context.Context, timeout time.Duration) error { | ||
| return nil | ||
| } |
Rewords the code comments so they no longer imply an error was caused by context cancellation when only ctx.Err() is checked, and wires a no-op fake reporter into TestCheck_ContextCanceled so the test doesn't rely on checkApplication failing early to avoid a nil reporter. Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
|
Good catches — reworded both comments (checkApplication and CleanPath sites) so they no longer imply the error was caused by cancellation, since the code actually suppresses any error once ctx.Err() != nil. Also wired a no-op fakeReporter into the detector under test so it's not relying on checkApplication's early failure to avoid a nil reporter panic. |
What this PR does:
Stops the kubernetes drift detector from logging
failed to check applicationandfailed to clean partially cloned repositoryat Error level when the failure is caused by the detector's context already being canceled (i.e. piped is shutting down). The underlying operations (checkApplication,gitRepo.CleanPath) still run exactly as before; only the resulting error is now suppressed whenctx.Err() != nil.Why we need it:
When a piped agent shuts down, its context is canceled while the periodic drift detector may still be mid-flight. This causes in-flight operations to fail (e.g. kustomize being killed with
signal: killed, or git commands failing withcontext canceled), and those expected, benign failures were being logged asError, producing log spam / false alarms right at shutdown.Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?:
Screenshots/Videos (for documentation or website changes):
N/A (no docs/
.mdchanges)Motivation: #5338 reports two spurious Error logs during piped shutdown:
failed to clean partially cloned repository(fromgitRepo.CleanPath) andfailed to check application: ...(fromcheckApplication, e.g. via kustomize being killed withsignal: killed, or a git command returningcontext canceled). A maintainer confirmed cleanup should still run, but its error (and thecheckApplicationerror) can be ignored when caused by the context being canceled.Approach: In
pkg/app/piped/driftdetector/kubernetes/detector.go'scheckmethod, both call sites still invoke their operations unconditionally, but now only log the resulting error at Error level whenctx.Err() == nil.ctx.Err() != nilis checked directly (rather thanerrors.Is(err, context.Canceled)) because a process killed viaexec.CommandContexton context cancellation (e.g. kustomize) returnssignal: killed, which does not wrapcontext.Canceledin Go'sos/execimplementation — checking the context directly reliably covers both error shapes reported in the issue.Validation: Added
TestCheck_ContextCanceledinpkg/app/piped/driftdetector/kubernetes/detector_test.go, which drivesdetector.checkwith a mockedgit.Repo(gomock) whoseCleanPathcall fails and whose repository path lacks an application config file (makingcheckApplicationfail quickly, independent of context state). With a non-canceled context both failures are logged as Error (2 log entries); with an already-canceled context, zero Error logs are produced, while the mockedCleanPath/Pull/GetLatestCommitcalls are still asserted to have run. Verified this test fails (2 unwanted Error logs) against the old code and passes after the fix.Commands run and their results:
go test ./pkg/app/piped/driftdetector/kubernetes/... -run TestCheck_ContextCanceled -v— PASS (both subtests)go test ./pkg/app/piped/driftdetector/... -v— PASS (all packages, including previously-existing tests in ecs/lambda)go build ./pkg/app/piped/driftdetector/...— successgo vet ./pkg/app/piped/driftdetector/...— cleangolangci-lint run --config .golangci.yml ./pkg/app/piped/driftdetector/...— 0 issuesNot run: no dev-stack/e2e piped shutdown reproduction (this is a logging-only change in one function, fully covered by the targeted unit test above).
Note:
master's own CI (go-test-completed) is currently green, so this PR is branched from a healthy base.Scope: only
pkg/app/piped/driftdetector/kubernetes/detector.gois changed, since it is the exact file the issue and the maintainer's comment point to, and it's the only drift detector that performs theCleanPathcleanup call. The terraform/ecs/lambda/cloudrun detectors have a similarcheckApplicationerror-log line but weren't reported in this issue and are left untouched to keep this change minimal.AI assistance: this change was drafted with Claude Code.
Fixes #5338