feat: observation completeness on measured payloads (ADR 0013)

[kfeofantov]

What and why

A zero was ambiguous. ThrottledPeriods, TotalPeriods and both PSI stall counters were plain counters with a zero default, so a container that was never throttled and a container whose node's /metrics/cadvisor scrape failed for the whole window produced byte-identical records. The cluster-wide signal set existed, but only in a log line, and could not resolve the question for any single record — a scrape fails per node and per path, not per cluster.

A partial window looked like a quiet one, too: Samples counted rate observations but nothing said how much of the window they spanned. A container that started fifty minutes into an hour and one whose node was unreachable for fifty minutes were indistinguishable.

Only the agent can know any of this. From outside the cluster a failed scrape is unrecoverable after the fact, so it belongs in the payload rather than in a log the backend never sees.

What ships

Per record

• cpu.throttling_samples, cpu.psi_samples, memory.psi_samples — how many observations carried each signal. A zero total with a zero sample count now reads as never observed; with a non-zero count it reads as observed and quiet.
• cpu.covered_nanoseconds — the summed duration of the intervals attributed to the record, split across windows exactly as the counters are. Against window_seconds it says how much of the window was seen.

All four add under merge; the generated merge-property tests (commutativity, associativity, partitioned-equals-whole) cover them by construction and pass unchanged.

Per payload

• An observation block: poll cadence, cumulative kubelet requests attempted and failed, and the exposed signal set. Both kubelet paths count as separate requests because they fail independently — which one failed is resolved per record by the sample counts, not here.
• Counters are cumulative since agent start, like every other counter this agent reports; no window-scoped bookkeeping was invented for them.

Provenance completes. usage_snapshot / usage_window → measured, oom_kill → journal, go_inventory → structural. The gap ADR 0012 recorded is closed in one golden migration rather than field by field later.

Bug fixed alongside — it is the same question

Throttling kept its counter baselines under the pod name the cAdvisor exposition labels by, while CPU and memory kept theirs under the pod UID. A name is not an identity: a StatefulSet recreates a pod under the same name with a new UID, and the replacement's counters were then compared against a dead container's baseline — dropped as a phantom counter reset, or, when the new counters happened to exceed the old, stretched into one continuous run. LookupPodByName now returns the UID and all three kubelet signals key on it.

TestThrottlingDoesNotCrossPodRecreation is the guard: with name keying it reports 400/1000 (the replacement's own periods silently lost), with UID keying 403/1010 (each container attributed from its own start).

Golden bytes changed

Four goldens: both usage payloads (source, observation, four new record fields), oom-kill and go-inventory (source). This is a deliberate protocol change, taken all at once. The regenerated usage-snapshot.golden.json is itself the demonstration: index shows throttled_periods: 0 with throttling_samples: 0, web shows throttled_periods: 40 with throttling_samples: 1.

ADR 0013 — written from the implementation

Records where completeness lives (per-record for per-record questions, per-payload for cluster-wide ones), why the counters are cumulative, and the identity fix. Also records what was deliberately not done: no "expected sample count" — it would require the agent to model when a container should have existed, which is a judgment, and covered_nanoseconds carries the same information without it; and no reason string for non-coverage — the agent reports counters, the backend renders the cause.

Invariant claims

• ADR 0001 (one-way): no backend input anywhere in this change.
• ADR 0003 (loss-harmless): no new persistent state; all counters are in-memory and rebaseline after a restart.
• ADR 0004 (no money): counts and nanoseconds only.
• ADR 0006 (merge property): new fields are additive; merge tests unchanged and green.
• Invariant 6: the observation block is cluster-wide totals — no pod, node, or workload is named in it.
• Sink invariant: four goldens regenerated deliberately, none by accident.
• No RBAC change.

Docs

• docs/security.md — new subsection on what the agent reports about its own collection, and why (honesty about gaps), stating that nothing in it identifies anything in the cluster.
• docs/backend-requirements.md — the observation contract, an explicit MUST that a zero total with a zero sample count is not zero, and the signal set moved from self-info to the payload it qualifies.

Tests

• internal/rollup — coverage splits with the interval across a window boundary; a partially observed window covers less than its length; observed-and-quiet versus never-observed produce different records with identical totals.
• internal/collector — throttling does not cross pod recreation (the regression guard); throttling sample count separates quiet from unobserved; observation counts every kubelet request and its failures across three failure combinations (driven through a real REST client against an httptest kubelet); the signal set reports a cluster without PSI as such.
• internal/sink — goldens.

make test (go test -race ./...) and make lint green.

🤖 Generated with Claude Code