fix: a cache that stopped being fed is noticed (ADR 0035)

[kfeofantov]

Closes the limitation ADR 0033 §5 recorded as known and unfixed.

The hole

HasSynced is a one-way latch. It turns true when the first LIST succeeds and never turns back, so a cache whose watch is refused afterwards keeps reporting availability while its store answers from what it last held. The reflector retries forever in the background.

ADR 0033 §5 judged the damage to be staleness. For a superseding snapshot that is the wrong word: workload_policy and cluster_policy state what is true at captured_at, and the omission of unavailable_sources is itself the claim that nothing was missed. A budget deleted after the refusal keeps shipping; one created after it never ships. The report is not behind — it is wrong, identically at every capture, until someone restarts the agent. And §5's mitigation ("any restart reports correctly") rests on an event this project deliberately made rare: a single-replica Deployment with no volume runs for weeks.

The latch is also not a property of the seven policy caches. It belongs to every informer — and a refusal on pods or the owner chain is not a degraded payload but a blind agent producing payloads of entirely ordinary shape.

There was a third case, worse than the one this started from. WaitForCacheSync has no timeout, so an agent installed without the pods grant never synced, never errored and never exited: a Running pod, a healthy Deployment, no data, and nothing anywhere saying why.

What changed

Watch failures are recorded per cache through SetWatchErrorHandler — the only signal client-go offers, since there is no callback for success.

• The cause is not examined. ADR 0033 §2 already decided the agent does not classify why a source is unreadable; a revoked grant, a deleted ServiceAccount, an API version dropped by an upgrade and an authorization webhook that changed its mind are one state. Persistence does the work classification would: a transient error happens once, a refusal repeats.
• Recovery is inferred from silence, with the reflector's own number. A run of failures ends after two minutes without one — client-go's defaultBackoffReset, for its own reason: while failing it retries at most every [30,60) seconds. A streak is measured between its first and last failure, never up to now, so one dropped connection is never an outage.
• A gating cache failing for five continuous minutes stops the agent. Not the first error: a restart costs the in-memory windows, the restart baselines and the spool (ADR 0026), so dying on a five-second hiccup would cost more than the blindness it prevents. This is what the comment above PodWatcher.Run always claimed happened and the agent could not do.
• The watchdog runs from before the first sync, so the never-synced case and the went-dark case end the same way: an exit naming the resource, and CrashLoopBackOff where a customer can see it.
• A policy cache that failed within the last three minutes declares itself unavailable — through the field that already exists, unchanged.

A real bug found on the way

The watchdog reached its verdict on time and Run returned 30 seconds later. factory.Shutdown() waits for every informer it started, and they were started on the caller's context — so they would stop only when the process was signalled. The informers now run on the watchdog's context. In a live cluster that is the difference between exiting in six minutes and hanging until SIGTERM.

Verified on kind, both directions

	result
Denied the pods grant	agent exits naming the resource, pod restarts — PASS 332s
Revoked a policy grant from a running agent	payload declares the source, metadata keeps collecting, controller never restarts — PASS 366s

With the watchdog and the health check disabled — which is the previous behaviour — the first fails after 11 minutes still running (waiting on a sync that will never come) and the second times out printing the old lie verbatim: last declared sources = [], thirteen minutes after the grant was taken away.

Both e2e tests are slow by nature, and the reason is the mechanism: an established watch is not re-authorized, so a revocation lands only when client-go re-opens the connection, every 5–10 minutes. ADR 0035 §7 records that lag as a decision, along with why polling permissions via SelfSubjectAccessReview was rejected — a write verb in the audit log of a product whose promise is that it only reads.

No payload shape, no new kind, no new RBAC.

🤖 Generated with Claude Code