Skip to content

fix: a pod that opts out while running is counted (ADR 0054) - #122

Merged
kfeofantov merged 1 commit into
mainfrom
fix/mid-flight-opt-out-counters
Sep 16, 2026
Merged

kfeofantov merged 1 commit into
mainfrom
fix/mid-flight-opt-out-counters

Conversation

@kfeofantov

Copy link
Copy Markdown
Contributor

The pod informer's AddFunc counted admission decisions and its UpdateFunc did not. A customer who annotated a running pod removed it from collection without moving any counter: filter.excluded_pod_annotation stayed 0 while the record left the go_inventory payload.

Found while diagnosing #121, in the live cluster — the coverage line read excluded_pod_annotation: 0 with the pod excluded and its record already evicted.

This was already decided

ADR 0054 names the exact scenario and chooses against the behaviour that shipped:

The backend sees every collected workload. If one disappears from the collected set at the moment excluded_workload_annotation moves from 0 to 1, which workload was hidden follows by subtraction. […] The alternative is silence, under which the backend concludes the workload was deleted and tells the customer something false.

The counter not moving is the silence that ADR rejected. No new decision here, so no new ADR — this restores documented behaviour.

Why not simply count on update

A pod emits status updates every few seconds and each carries the same answer; counting state would climb forever. countTransition counts a decision that differs from the one recorded in the index — presence there is the previous answer, and the entry carries whether profiling was admitted with it. A change counts once, a repetition counts nothing.

The same rule covers both controls and both directions:

Transition Counted
collected → excluded the matching excluded_* counter
excluded → collected pods_observed
profiled → refused, still collected excluded_profiling_annotation
same answer as last time nothing

Contract change

The filter counters now count decisions rather than pods, so a pod can contribute to pods_observed and to an exclusion counter over its lifetime. Adding them no longer recovers a pod count. docs/backend-requirements.md says so normatively.

Verification

Four of the five new unit tests fail against the prior pods.go:

--- FAIL: TestOptingOutARunningPodIsCounted
--- FAIL: TestRepeatedUpdatesAfterTheOptOutCountOnce
--- FAIL: TestRemovingTheAnnotationCountsThePodObservedAgain
--- FAIL: TestRefusingTheProfilerOnARunningPodIsCounted

The fifth, TestAPodExcludedOnArrivalIsNotCountedAgain, passes both ways by design — it guards the double counting this must not introduce.

In a real cluster, with a new e2e assertion that the coverage explains the departure:

opted runtime-agent-inv-e2e-47343/goworkload-... out at 2026-09-16T10:26:18Z
coverage explains the departure: excluded_pod_annotation = 1
--- PASS: TestGoInventoryEndToEnd (428.25s)
go test ./...                    pass
golangci-lint (pinned v2.11.4)   0 issues
git archive HEAD -> clean tree   builds and tests green

🤖 Generated with Claude Code

https://claude.ai/code/session_01KeZECJXTDHL6Nts95eKMJA

The pod informer's AddFunc counted admission decisions and its UpdateFunc did
not, so a customer who annotated a running pod removed it from collection
without moving any counter. filter.excluded_pod_annotation stayed 0 while the
record left the go_inventory payload.

ADR 0054 already decided this should not happen. It accepts that a workload
leaving the collected set is visible to the backend by subtraction, and rests
on the count explaining why — "the alternative is silence, under which the
backend concludes the workload was deleted and tells the customer something
false". That is what shipped. No new decision here, so no new ADR.

Counting on every update would have been worse: a pod emits status updates
every few seconds and each carries the same answer. countTransition counts a
decision that differs from the one recorded in the index, so a change counts
once and a repetition counts nothing. The same rule covers the narrower
control: a workload that refuses the profiler mid-flight now moves
excluded_profiling_annotation, and removing either annotation counts the pod
observed again.

This makes the filter counters count decisions rather than pods, so adding
pods_observed to the exclusion counters no longer recovers a pod count.
backend-requirements.md says so.

    four of the five new unit tests fail against the prior pods.go; the fifth
    guards against double counting and passes both ways
    make inventory-e2e — PASS (428s), coverage reporting
      excluded_pod_annotation = 1 where it read 0 before

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KeZECJXTDHL6Nts95eKMJA
@kfeofantov
kfeofantov merged commit 28d52d3 into main Sep 16, 2026
4 checks passed
@kfeofantov
kfeofantov deleted the fix/mid-flight-opt-out-counters branch September 16, 2026 10:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant