fix: both halves of the counter reading come from one observation (ADR 0043)

[kfeofantov]

restart_counters reports two numbers a consumer is explicitly told to subtract: the kubelet's counter, and what it already stood at when this agent process first saw the container. They came from different places.

restarts was read from the informer's shared store through the pod lister. restarts_before_observation came from the restartCounts map the agent maintains in its own event handler. client-go updates the shared store before it dispatches to that handler, so for the span of one dispatch the store holds a counter the baseline has not seen.

Usually invisible — the counter only goes up, and the reading is stale in a way nobody could detect. It becomes visible when the counter goes down, which is a StatefulSet replacing a pod under the same name: the store already says 2 while the baseline still says 40.

The payload then asserts that forty restarts predate an observation of a counter standing at two. docs/backend-requirements.md tells a backend to subtract those:

40 restarts predate observation of a counter at 2  →  2 - 40 = -38

This had been surfacing as a test that failed about one run in three under package load — the shape a race takes when nobody has looked at it yet.

The fix

Both values now come from one read of the baseline: restarts is base.last — the counter as the agent last observed it — and restarts_before_observation is base.atFirstSight from the same struct, under the same lock.

That makes the ordering structural rather than probable. A rebaseline sets last and atFirstSight together; an advance only ever raises last. So restarts >= restarts_before_observation holds by construction, whatever the informer is doing at the time.

The lag introduced is one informer dispatch against a payload that flushes once a minute — and it makes this field consistent with everything else in the payload, all of which is already a statement about what this agent process observed. A counter read from somewhere the agent had not yet processed was the odd one out.

The test is the interleaving, not a timing hope

The regression test builds the state directly — baseline at 40, store at 2, handler not yet run — by injecting a stub lister, because that is exactly what the informer produces for a moment on every counter change. Waiting for it to happen by chance is what let this sit unnoticed.

Checked in both directions. With the lister's value restored:

--- FAIL: TestTheReadingNeverClaimsMoreHistoryThanItCounts
    reading claims 40 restarts predate observation of a counter at 2;
    ADR 0034 tells a consumer to subtract these and they would get -38

Evidence

• Five consecutive go test -race -count=1 ./internal/collector runs green, where the previously flaky test failed roughly one in three.
• make restarts-e2e PASS 148.6s, both suites, against a real cluster — including a real day-one reading: Restarts:3 RestartsBeforeObservation:2, difference positive.
• go test -race ./..., make lint (0 issues), make chart-lint, go vet -tags e2e ./... clean.

Docs

• docs/adr/0043-… amends 0034 §7, which describes the reading as assembled from the live pod index at an instant. The rest of the record still is — pod creation, container start, last termination. The counter is not, and the ADR says so rather than hiding the lag.
• docs/backend-requirements.md now states the invariant a backend can rely on: restarts - restarts_before_observation is never negative. It also says it was not always so, so anyone carrying a defensive clamp knows why.
• docs/security.md's payload table follows.

Named, not solved

last_termination still comes from the pod, so it can describe a death the counter has not yet counted. That direction is benign — it under-reports rather than inventing history — and self-corrects on the next flush. Recorded in the ADR so it is a known shape rather than a surprise.

🤖 Generated with Claude Code