fix: the Go inventory forgets departed and opted-out workloads (ADR 0018)

[kfeofantov]

inventory.Store was the one place in the agent whose keyspace only ever grew. Ingest had a single operation on it — assign — and no code path anywhere removed a record. A record entered when a node reported a binary that joined, and left only when the controller restarted.

Not an oversight so much as a missing half. Every other payload is derived from state the controller can re-read: workload_metadata is rebuilt from the live informer index on every flush, so it drops what the cluster dropped without anything having to remember that it should. The inventory cannot be rebuilt that way — it joins facts the nodes push with state the controller holds, and reports arrive once per scan interval. Accumulating was the right answer. Eviction was the part that was never written.

Why this is not just stale data

A deleted workload's record kept shipping, and workload_metadata had already forgotten it, so the two payloads disagreed about which workloads exist.

The serious one: an opt-out that did not apply to what had already been collected. Annotating a pod with rebuildstack.co/collect: "false" drops it from PodWatcher's index immediately and stops it being named in the scan scope handed to nodes (ADR 0015). Both halves worked. What followed is that no new fact ever arrived about that pod — and a store that only accumulates has no way to act on the absence of facts. The record already held kept leaving the cluster, with its namespace, workload name, container name and main module path, in every snapshot, for as long as the controller ran.

docs/security.md §11 said "the exclusion applies at the collection stage". For this payload it applied only to facts not yet collected.

What changes

Every flush now drops records whose (namespace, kind, name, container) is absent from the controller's own filtered pod index, and with them the dependency set of any build no surviving record references.

The live index is PodWatcher.Pods() — the same source workload_metadata is derived from. That is the substance, not an implementation detail: the two payloads now agree about which workloads exist by construction, because they read one index. The mechanism was already in place with one consumer too few — indexPod's own comment already promised the index "drops a pod the moment" it is excluded.

Three details that were each a decision, not a default:

• No empty-input guard. Retain with an empty live set empties the inventory. A guard would have been the intuitive safety measure and would have been wrong: a cluster where nothing passes the filters legitimately has an empty inventory, and a guard would pin the last non-empty snapshot forever — the exact failure this fixes. It is safe without one because the index is only ever mutated by informer events and never wiped (handlers register after cache sync; a relist replays as adds/updates), so an empty index means an empty cluster, not an unsynced one. There is a test for this case specifically.
• deps and written are dropped together. Dropping a dependency set while leaving its digest marked as written would mean a returning image is re-ingested and never sent — silently, with nothing failing. Same failure mode the two-step pending/mark split of ADR 0017 exists to make visible.
• nodes_reported stops being cumulative. ADR 0017 (mine, last PR) described it as a running total. Wrong in the same way: its purpose is comparison against the node count in node_metadata, and a count that outlives its nodes exceeds the fleet — so a scaled-down cluster with a broken DaemonSet would have looked complete, which is exactly what the block was added to expose.

Verification

• make test (incl. go vet -tags e2e ./...) and make lint green.
• Seven new unit tests: eviction of a departed workload, survival of a live one with no fresh fact, the empty-cluster case, the written-mark pairing on return, a dependency set shared by a surviving workload, node pruning, and LiveKeys covering init containers (a native sidecar is declared as one and would otherwise flap).
• make inventory-e2e passes in kind (127s): the test finds the record, annotates the running pod out, and asserts the record leaves the payload. Only docs were edited after the image was built, so this run matches the committed Go code exactly.

Docs

ADR 0018 amends ADR 0017 (the nodes_reported lifetime, and the record-lifetime question 0017 recorded as "not addressed here"). docs/security.md §11 now states that the exclusion applies to what was already collected and why the inventory needs telling to forget rather than simply stopping learning; §8 states a record exists only while its workload does. docs/backend-requirements.md gains the absence semantics: a record missing from a snapshot means "no longer collected", the backend must not carry forward records absent from a superseding snapshot, and a CronJob may legitimately flicker.

🤖 Generated with Claude Code