feat: container restart journal (ADR 0020)

[kfeofantov]

Slice A of the object journal: non-OOM terminations, as a windowed aggregate. ADR 0020 in this PR.

Also carries a one-line fix for something #45 missed — see the first commit.

What was wrong

The journal had exactly one event type in it. internal/collector/oom.go discarded everything else in one line:

if term == nil || term.Reason != oomKilledReason { continue }

A container killed by the kernel was reported; a container that exited 1 forty times in an hour was invisible. That matters for numbers the agent already sends: a container that keeps dying uses almost no CPU and memory, so its usage rollup looks like a comfortably over-provisioned workload. The restart count is what separates "does not need those resources" from "never lives long enough to use them".

Windowed, not per-event

One payload file per hour-aligned window holding many records — the usage-snapshot shape — keyed by (namespace, pod, container).

Per-event files would have handed the spool's file count to a crash loop. The spool is bounded by age alone (DefaultMaxAge = 24h, no count or byte cap), and a container in CrashLoopBackOff restarts ~12×/hour: a hundred of them write ~1200 files/hour, ~28000 before the first sweep. The latent problem was always there — removing the OOM-only filter is what would have made it live.

A per-pod cap was considered and rejected: it keeps the per-event shape by silently dropping data, and a payload that looks complete while truncated is the failure mode ADR 0018 exists to end.

Exact counts, sampled reasons

The two fields come from data of different quality, so they are not merged:

• restarts — exact, from the restart counter's advance. Counts restarts the agent never saw individually.
• reasons — a sample. A container status carries only its most recent termination.
• reasons_unobserved — the difference. sum(reasons) + reasons_unobserved == restarts holds for every record, and the e2e asserts it.

Folding them would mean either under-reporting restarts or inventing reasons. This is ADR 0013's principle applied to a journal payload.

Reasons outside the known set are counted under other rather than passed through: the set belongs to the CRI, and a map whose keys a container runtime picks is a payload whose shape it picks. Counted, never dropped.

Pod names

The record names the pod, which no other payload does — workload_metadata explicitly counts replicas instead of listing them. Which replica is looping cannot be recovered from a workload total, and it is the first thing anyone asks.

oom_kill has named pods since it shipped and security.md never said so. This makes it a documented decision: new sensitivity-table row, new §8 subsection, and a pointer to the controls that already work (namespace filter, label selector, opt-out annotation — the last of which removes already-collected data since ADR 0018).

Reasons yes, messages no

lastState.terminated.reason comes from the kubelet's own vocabulary. The adjacent message is free text that routinely carries node names, taints and other workloads' details. Same distinction, same answer, as the build-settings allow-list in ADR 0019.

First observation is baselined, not reported

A counter already standing at 40 describes restarts at moments Kubernetes does not record. Reporting them would place a history in whichever window happens to be open at agent start. Knowingly incomplete, and bounded: once per pod.

Verification

• make test (incl. go vet -tags e2e ./...) and make lint green. 7 unit tests on the collector side, 9 on the accumulator, 3 on the payload plus a golden.
• New make restarts-e2e: deploys the controller next to a container that exits 1, asserts the payload in the spool. PASS, 63.30s:

  restart record: {Namespace:runtime-agent-restart-e2e-52598 Pod:crashloop-7c457dbfb5-dhskj
    Container:crashloop Workload:{Kind:Deployment Name:crashloop}
    Restarts:3 Reasons:map[Error:3] ReasonsUnobserved:0 LastExitCode:1}
  

  It reuses the busybox spool-reader image, so no new image is built. The controller is deployed before the crash loop on purpose — a pod already looping at agent start would legitimately report nothing until its next restart.

One thing worth knowing about the tests

The new collector tests deadlocked at first for a reason the existing OOM tests only get away with by luck: WaitForCacheSync proves the initial List landed, but an update arrives over the reflector's watch, which is established afterwards. Every update raced it and the fake clientset silently dropped them (it has no replay). The helper now blocks until a pod created after the watcher started has been delivered, which is proof the watch is live. Not a sleep.

Docs

security.md: RBAC row, sensitivity table, field minimization (restart counters and terminated reason/exit code, and that messages are not read), new §8 subsection on the journal and pod names. backend-requirements.md: the arithmetic rule, the fixed reason set with other, supersede-by-window (deliveries are cumulative within a window, not increments), and that oom_kill overlaps this by design.

🤖 Generated with Claude Code