fix: a cache that stopped being fed is noticed (ADR 0035)

Makefile

@@ -13,7 +13,7 @@ SAMPLE_IMAGE ?= rebuildstack-e2e-goworkload:latest
 # read the controller's spool (the agent image is distroless — no shell).
 SPOOL_READER_IMAGE ?= busybox:1.37
 
-.PHONY: build test lint tidy clean cluster-up cluster-down e2e smoke image sample-image kind-load node-e2e inventory-e2e restarts-e2e lifecycle-e2e policy-e2e profile-gate-e2e profile-capture-e2e
+.PHONY: build test lint tidy clean cluster-up cluster-down e2e smoke image sample-image kind-load node-e2e inventory-e2e restarts-e2e lifecycle-e2e policy-e2e watch-e2e profile-gate-e2e profile-capture-e2e
 
 build: ## Build the agent binary into bin/
 	go build -ldflags '$(LDFLAGS)' -o bin/agent ./cmd/agent
@@ -125,6 +125,18 @@ restarts-e2e: kind-load ## Deploy the controller in kind next to a crash-looping
 		-run 'TestContainerRestartsEndToEnd|TestRestartCountersCarryTheHistoryTheAgentDidNotWatch' 2>&1 \
 		| tee test/e2e/logs/restarts-e2e-$$(date +%Y%m%d-%H%M%S).log
 
+watch-e2e: kind-load ## Deny the pods grant, then revoke a policy grant from a running agent, and assert the first stops the agent while the second only degrades its payload (ADR 0035); slow by nature — an established watch is re-authorized only when client-go re-establishes it, every 5–10 minutes; log goes to test/e2e/logs/
+	@mkdir -p test/e2e/logs
+	docker pull $(SPOOL_READER_IMAGE)
+	go tool kind load docker-image $(SPOOL_READER_IMAGE) --name $(E2E_CLUSTER)
+	set -o pipefail; \
+	E2E_KUBE_CONTEXT=kind-$(E2E_CLUSTER) \
+	E2E_AGENT_IMAGE=$(IMAGE):$(IMAGE_TAG) \
+	E2E_SPOOL_READER_IMAGE=$(SPOOL_READER_IMAGE) \
+	go test -tags e2e -count=1 -timeout 45m -v ./test/e2e/ \
+		-run 'TestAnAgentDeniedAGatingPermissionStopsInsteadOfWaitingForever|TestAPermissionRevokedFromTheRunningAgentReachesThePayload' 2>&1 \
+		| tee test/e2e/logs/watch-e2e-$$(date +%Y%m%d-%H%M%S).log
+
 lifecycle-e2e: kind-load ## Deploy the controller in kind, stage an unschedulable pod and a scheduler preemption, and assert both reach the payloads (ADR 0021); log goes to test/e2e/logs/
 	@mkdir -p test/e2e/logs
 	docker pull $(SPOOL_READER_IMAGE)

docs/adr/0033-policy-sources-degrade.md

@@ -4,6 +4,7 @@ Date: 2026-08-24
 
 Status: Accepted
 Amends: 0032 §1
+Amended by: 0035
 
 Changes how the agent behaves when it cannot read a policy source, and adds an
 `unavailable_sources` line to `workload_policy` and `cluster_policy`. No new

docs/adr/0035-watch-failures-are-noticed.md

@@ -0,0 +1,151 @@
+# 0035. A cache that stopped being fed is noticed: gating caches stop the agent, the rest degrade their payload
+
+Date: 2026-08-25
+
+Status: Accepted
+Amends: 0033 §5
+
+Closes the limitation ADR 0033 recorded as known and unfixed. No new kind, no new
+field, no new RBAC, no change to any payload's shape: `unavailable_sources`
+already exists and now answers a second question with the same word.
+
+## Context
+
+ADR 0033 chose `HasSynced` as the availability signal and got the start of an
+agent's life right: an empty list and a refused read became different states,
+and a customer who declines one of the seven policy grants keeps the usage
+collection they installed the agent for.
+
+`HasSynced` is a one-way latch. It turns true when the first LIST succeeds and
+never turns back. A cache whose watch is refused afterwards therefore keeps
+reporting availability while the store answers every query from what it last
+held, and the reflector retries in the background forever. ADR 0033 §5 recorded
+this, judged the damage to be staleness, and declined to build the detector.
+
+Two things about that judgement were wrong.
+
+**Stale is not the right word for a superseding snapshot.** `workload_policy` and
+`cluster_policy` replace their predecessor and state what is true at
+`captured_at`. A budget deleted after the refusal keeps shipping; one created
+after it never ships; and the payload declares every source read, because the
+omission of `unavailable_sources` is itself the claim that nothing was missed.
+The report is not behind — it is wrong, and it is wrong identically at every
+capture until someone restarts the agent. §5's own mitigation, "any restart
+reports correctly", rests on an event this project deliberately made rare: the
+controller is a single-replica Deployment with no volume and can run for weeks.
+
+**The latch is not a property of the seven policy caches.** It is a property of
+every informer. The pod index, the owner chain and the namespace cache are what
+every other payload is assembled from, and a refusal there is not a degraded
+payload but a blind agent producing payloads of entirely ordinary shape. The
+comment above `PodWatcher.Run` had already stated the intended behaviour —
+*a failing watcher must bring the agent down rather than leave it half-blind* —
+and the agent could not honour it, because nothing told it the watch had died.
+
+There is a third case the same blindness produces, and it is the worst of them.
+`WaitForCacheSync` has no timeout. An agent installed without the `pods` grant
+never syncs, never errors and never exits: a Running pod, a healthy Deployment,
+no data, and nothing anywhere that says why.
+
+## Decision
+
+**1. Watch failures are recorded, and the recording is per cache.**
+`SetWatchErrorHandler` fires on every failed `ListAndWatch`, including a failed
+LIST, and is the only signal client-go offers — there is no matching callback for
+success. Each informer gets a health record holding when its current run of
+failures began and when it was last seen failing.
+
+**2. The cause is not examined.** ADR 0033 §2 decided that the agent does not
+classify why a source is unreadable, and the same holds here. A revoked grant, a
+deleted ServiceAccount whose token stops being accepted, an API version dropped
+by a cluster upgrade and an authorization webhook that changed its mind are one
+state — the cache is not being fed — and the payload has no field in which the
+difference could be expressed.
+
+What classification would have bought, persistence buys instead: a transient
+error happens once, a refusal repeats until someone fixes it.
+
+**3. Recovery is inferred from silence, with the reflector's own number.** A run
+of failures ends after two minutes without one. That is not a guess: while a
+watch is failing the reflector retries at most every [30,60) seconds, and
+client-go states the identical judgement in `defaultBackoffReset` — "if we don't
+backoff for 2min, assume API server is healthy". Adopting its constant keeps the
+two from disagreeing.
+
+A streak's length is measured between its first and last failure, never up to
+now. One failure spans nothing and is therefore never an outage, which is the
+intended reading of a single dropped connection.
+
+**4. A gating cache that fails for five minutes stops the agent.** Gating is the
+split ADR 0033 §1 already drew — a cache that gates a signal against one that
+adds one — applied to the running agent rather than only to its start. Pods,
+replicasets, jobs, deployments, statefulsets, daemonsets, cronjobs, namespaces
+and nodes gate; the seven policy caches do not.
+
+Not the first error, because a restart is not free: the in-memory windows, the
+restart baselines and the spool are lost with the process (ADR 0026), so dying
+on a five-second API server hiccup would cost more than the blindness it
+prevents. Five minutes is several retries, and it bounds how long the agent can
+be blind before it says so.
+
+The watchdog runs from before the first cache sync, so the never-synced case and
+the went-dark case end the same way: an exit whose message names the resource,
+and a pod in CrashLoopBackOff where a customer can see it. The informers run on
+the watchdog's context rather than the caller's, or the factory's shutdown would
+wait for watches that stop only when the process is signalled.
+
+**5. A policy cache that failed within the last three minutes declares itself
+unavailable.** The existing field, unchanged, now covers both halves of one
+statement: a source is unavailable if it never filled *or* if it is failing now.
+The two are not distinguished, for the same reason the causes are not — a
+consumer does the same thing with either.
+
+Three minutes is longer than the [30,60)-second retry interval, so a live
+refusal is declared at every capture rather than at some of them. A permission
+restored is therefore declared unavailable for a capture or two more, and a
+transient error can cost one capture's completeness claim. Both are the safe
+direction: the field overstating what was missed is a smaller error than the
+payload overstating what was read.
+
+**6. What the payload carries is the source name.** The error text is logged and
+goes no further. A refusal names the agent's own ServiceAccount and a resource
+class rather than any customer object, but the rule that identities never leave
+the cluster (CLAUDE.md invariant 6) is not one to defend case by case.
+
+**7. Known limitation: revocation is noticed when the watch is next
+re-established.** An established watch is not re-authorized by the API server, so
+a grant removed from a running agent has no effect until client-go re-opens the
+connection — which it does on its own timeout, between five and ten minutes.
+Detection therefore lags a revocation by up to ten minutes, and the gating exit
+by up to fifteen.
+
+This is not fixable from inside the agent without polling permissions, which
+would mean creating SelfSubjectAccessReviews: a write verb in the audit log of a
+product whose promise is that it only reads, answering a narrower question than
+the handler already answers. The lag is recorded here so it stays a decision.
+
+## Consequences
+
+**Easier.** The three ways an agent could go quietly blind — never synced,
+stopped being fed, and the wait with no timeout — now end in either a named exit
+or a payload that says what it could not see. A customer who narrows the
+ClusterRole of a running agent gets the same honest answer they already got when
+narrowing it before install.
+
+**Harder / given up.** The agent can now stop for a reason that is not its own
+fault, and a sustained API server outage will restart it after five minutes,
+losing an interval of accumulated windows. That is the trade ADR 0026 already
+implies: state that cannot survive a restart is state whose loss is preferable to
+its silent corruption.
+
+`unavailable_sources` also became slightly less precise: it can name a source
+that is readable again but failed within the window. The alternative — a field
+that is precise and occasionally false — is worse for a consumer that must decide
+what a report rests on.
+
+**Not changed.** No payload shape, no RBAC, no kind, no collection behaviour when
+the read succeeds. The seven policy caches still degrade rather than stop, which
+is ADR 0033's decision and the bug it fixed.
+
+This ADR records decisions implemented in the same pull request, per the process
+in [`README.md`](README.md).

docs/adr/README.md

@@ -92,3 +92,4 @@ line to paste.
 - [0032. What bounds a workload from outside its own spec, and the first widening of read access](0032-cluster-policy.md)
 - [0033. A permission the agent was not given degrades one payload, and says so](0033-policy-sources-degrade.md)
 - [0034. The restart counter is shipped as a reading, so the day-one history is not lost](0034-restart-counter-readings.md)
+- [0035. A cache that stopped being fed is noticed: gating caches stop the agent, the rest degrade their payload](0035-watch-failures-are-noticed.md)

docs/backend-requirements.md

@@ -222,6 +222,32 @@ A container whose counter is zero has no record: the payload is a snapshot, so
 absence means zero rather than unknown. An empty record list means no collected
 container has ever restarted, and MUST NOT be read as a failed collection.
 
+**`unavailable_sources` disqualifies the payload's absences, and says nothing
+about why** ([ADR 0033](adr/0033-policy-sources-degrade.md),
+[ADR 0035](adr/0035-watch-failures-are-noticed.md)). `workload_policy` and
+`cluster_policy` each declare the sources they could not read at that capture.
+The field names resource classes, is omitted when every source was read, and
+each payload declares only its own sources.
+
+- The rule it exists for: in these payloads an absent record is a positive
+  statement — "nothing constrains this workload". When a source is listed, that
+  statement is void for anything that source would have supplied, including
+  workloads that vanish from the payload entirely rather than appearing with a
+  field missing. The backend MUST NOT treat the payload as a complete picture of
+  what it declares unreadable.
+- One list means one thing: not available at this capture. It does **not** say
+  whether the permission was never granted, was taken away from the running
+  agent, or was lost to a webhook or an unreachable API server, and the backend
+  MUST NOT infer a cause from it.
+- It is about that capture and no other. A source listed once and not again was
+  readable again; a source that is readable again may still be listed for a
+  capture or two, because the agent errs toward declaring a gap. Neither is a
+  claim that the source is permanently gone, and the backend MUST NOT latch it.
+- A missing grant the agent cannot work without produces no payload at all — the
+  agent exits (ADR 0035). Absence of deliveries and a declared source are
+  therefore different signals, and the backend SHOULD alert on the first through
+  whatever it uses for a silent cluster.
+
 **`pod_disruptions` is what the cluster took away, not what failed**
 ([ADR 0021](adr/0021-pod-lifecycle-journal.md)). One record per pod per
 hour-aligned window: preempted, evicted under node pressure, drained by a taint,

docs/security.md

@@ -136,6 +136,23 @@ indistinguishable from "your cluster has none of those"
 ([ADR 0033](adr/0033-policy-sources-degrade.md)). Granting a permission later
 takes effect at the next flush, without restarting the agent.
 
+**This holds for a permission you narrow later, not only for one you never
+granted.** A grant removed from a running agent is noticed and declared the same
+way, which it was not before
+([ADR 0035](adr/0035-watch-failures-are-noticed.md)); detection lags the change
+by up to ten minutes, because an established watch is not re-authorized by the
+API server until the client re-opens it. What the payload names is the resource
+class, never an object.
+
+**Withdrawing a grant the agent cannot work without stops the agent, visibly.**
+`pods`, `namespaces`, `nodes` and the owner-chain resources are what every
+payload is assembled from, so there is no single payload left to degrade. If one
+of those becomes unreadable — before the first sync or five minutes into a
+refusal that started later — the agent exits with a message naming the resource
+and its pod enters `CrashLoopBackOff`. It does not keep answering from a frozen
+cache, and it does not sit silently waiting, which is what it used to do when a
+grant was missing at install (ADR 0035).
+
 **Authenticating node reports adds no RBAC.** When the node DaemonSet is
 installed (ADR 0010), the controller authenticates each node's projected
 ServiceAccount token by verifying its signature against the cluster's published
@@ -403,8 +420,8 @@ appearing here (ADR 0022).
 | `job_runs` | Per hour: each Job that finished — when it started and finished, whether it succeeded, the failure reason, its pod success/failure counts, and its declared `parallelism`/`completions`/`backoffLimit` | **yes** |
 | `deployment_revisions` | Per flush: each ReplicaSet of a collected Deployment — its revision number, when it was created, its desired/current/ready replica counts, and each container's image reference | **yes** |
 | `workload_metadata` | Declared shape per (namespace, workload, container, image digest): image, requests, limits, ports, QoS, replica counts by phase, by node, and by unscheduled reason, and the pod's reduced placement constraints (ADR 0031) | no |
-| `workload_policy` | Per collected workload, what bounds it from outside its own spec: disruption budgets covering it, autoscalers driving it, and the volume claims its pods mount (ADR 0032); plus `unavailable_sources`, naming any of those the agent could not read (ADR 0033) | no |
-| `cluster_policy` | Per collected namespace, its LimitRanges and ResourceQuotas; plus the cluster's PriorityClasses and StorageClasses, which workloads reference by name (ADR 0032), and `unavailable_sources` for any of those the agent could not read (ADR 0033). StorageClass `parameters` are never read | no |
+| `workload_policy` | Per collected workload, what bounds it from outside its own spec: disruption budgets covering it, autoscalers driving it, and the volume claims its pods mount (ADR 0032); plus `unavailable_sources`, naming any of those the agent could not read — never granted, or no longer being fed (ADR 0033, ADR 0035) | no |
+| `cluster_policy` | Per collected namespace, its LimitRanges and ResourceQuotas; plus the cluster's PriorityClasses and StorageClasses, which workloads reference by name (ADR 0032), and `unavailable_sources` for any of those the agent could not read — never granted, or no longer being fed (ADR 0033, ADR 0035). StorageClass `parameters` are never read | no |
 | `node_metadata` | Per node: name, size, instance and capacity type, zone, region, kernel version, CPU architecture | no |
 | `go_inventory` | Per (namespace, workload, container): Go version, main module, image digest, PGO flag, plus a fleet-coverage block | no |
 | `go_build` | Per image digest, written once: Go version, main module, dependency module **paths**, allow-listed build settings | no |
@@ -585,6 +602,14 @@ records, and the distinction is unrecoverable once the moment has passed
 ([ADR 0013](adr/0013-observation-completeness.md)). No pod, node, or workload is
 named in this data; the failure counts are cluster-wide totals.
 
+The same honesty applies to the API server rather than the kubelet. A cache the
+agent cannot read is declared in `unavailable_sources` on the payload that reads
+it — whether the permission was never granted or was taken away from the running
+agent — and a cache the agent cannot work without stops the agent instead
+([ADR 0035](adr/0035-watch-failures-are-noticed.md)). Both name a resource class
+and nothing else; the underlying error, which names the agent's own
+ServiceAccount, stays in the agent's log and is never part of a payload.
+
 ### Profile filtering (applies on the node, before egress — ADR 0011)
 
 - Stack frames are filtered by Go module path on the node that captured them.