Skip to content

fix: a permission the agent was not given degrades one payload, not the agent (ADR 0033) - #62

Merged
kfeofantov merged 1 commit into
mainfrom
fix/policy-degrades-not-dies
Aug 24, 2026
Merged

kfeofantov merged 1 commit into
mainfrom
fix/policy-degrades-not-dies

Conversation

@kfeofantov

Copy link
Copy Markdown
Contributor

Fixes finding 1 from the review, plus the [planned] marker from finding 2. No new RBAC, no new kind, no change to what is read when the read succeeds.

The failure mode

ADR 0032 put the seven policy caches in the list the agent waits on before starting. A customer who removed one line from the ClusterRole — storageclasses, say, because they did not see why a cost agent needs it — got an agent that failed to start and collected nothing at all, usage included.

Why silence was not the fix either

ADR 0032 §2 decided that a workload nothing constrains produces no record, so in a superseding snapshot that absence is a positive statement. With budgets unreadable it becomes false twice over: the field is missing, and a workload whose only constraint was a budget vanishes from the payload entirely rather than appearing incomplete.

So the absence had to be qualified in the payload, not just survived.

The rule

An informer that gates a signal blocks; an informer that adds one does not. The owner-chain and namespace caches stay blocking — a pod admitted before its controller is cached would be admitted without its workload-level opt-out being checked. The policy caches gate nothing.

Why HasSynced and not "the list came back empty"

This is the whole mechanism, and it answers the question you asked directly:

  • A cluster with no PodDisruptionBudgets lists zero objects and syncs normally → source available, nothing declared.
  • A cluster that denied the permission never syncs → source named in unavailable_sources.

Empty and refused are different states rather than the same silence. It also needs no timeout and no error classification — the cause may be RBAC, a webhook or an unreachable API server, and "not available at this capture" is true in all of them. And it heals: the reflector retries forever, so a permission granted later appears at the next flush without restarting the agent.

Per payload, not pooled

{ "kind": "workload_policy", "unavailable_sources": ["pod_disruption_budgets"], "records": [...] }

At payload level, because it is a property of the capture, not of a workload. Declared separately by each kind because the two have distinct natural keys and are upserted independently — one can arrive without the other, so each must be readable alone.

The storage-class catalog belongs only to cluster_policy: a claim's storage_class is a name read from the PVC, and the catalog only says what that name means.

Known limitation, recorded as a decision

A permission revoked while the agent runs is not detected — HasSynced is a one-way latch, so a cache that filled once and lost its watch keeps reporting availability while serving its last contents. Data stale rather than false, correct again after any restart. Detecting it needs a watch-error handler that can tell Forbidden from an expired resource version; ADR 0033 §5 says why that is not built, so it stays a decision rather than becoming an oversight.

Verification

Negative check: putting the budget cache back in the blocking list fails the new test with no pods collected: the denied source stopped unrelated collection.

The registry's kind-has-one-shape rule refused a second golden for the degraded case, correctly. Both goldens stay healthy so every rich shape is pinned, and the declared line is covered by decoding the written payload — it is a string list, and its serialization is not the subtle part.

make test (16 packages, race detector) and make lint (0 issues) green.

Also in this PR

security.md marks the namespace-scoped installation mode [planned]. It was written in the present tense with no implementation behind it — the informers are cluster-wide — which is the exact defect ADR 0024 exists to prevent, and #61 made it worse by describing how that mode would treat the new cluster-scoped resources.

🤖 Generated with Claude Code

…he agent (ADR 0033)

ADR 0032 put the seven policy caches in the list the agent waits on before
starting. That was a reflex, not a decision, and the failure mode was out of
all proportion to its cause: a customer who removed one line from the
ClusterRole — storageclasses, say, because they did not see why a cost agent
needs it — got an agent that failed to start and collected nothing at all,
usage included.

Quietly skipping unreadable sources would be worse than it looks. ADR 0032
§2 decided that a workload nothing constrains produces no record, so in a
superseding snapshot that absence is a positive statement. With budgets
unreadable the statement becomes false twice over: the field is missing, and
a workload whose only constraint was a budget vanishes from the payload
entirely rather than appearing incomplete.

So the rule is that an informer which gates a signal blocks and one which
adds a signal does not, and the qualification travels in the payload.

The signal is HasSynced rather than "the list came back empty", and that
choice is the mechanism. A cluster with no PodDisruptionBudgets lists zero
objects and syncs normally; a cluster that denied the permission never syncs
at all. Empty and refused become different states instead of the same
silence — which is what a consumer needs and cannot reconstruct afterwards.
It also needs no timeout and no error classification, and it heals: the
reflector retries forever, so a permission granted later shows up at the
next flush without a restart.

Each payload declares its own sources, at payload level rather than inside a
record, because it is a property of the capture. They are not pooled: the
two kinds have distinct natural keys and are upserted independently, so each
must be readable alone. The storage-class catalog belongs only to the
cluster payload — a claim's storage_class is a name read from the PVC, and
the catalog only says what that name means.

Verified negatively: putting the budget cache back in the blocking list
fails the new test with "no pods collected: the denied source stopped
unrelated collection".

The kind-has-one-shape rule in the registry refused a second golden for the
degraded case, correctly. Both goldens stay healthy so every rich shape is
pinned, and the declared line is covered by decoding the written payload —
it is a string list, and its serialization is not the subtle part.

Also marks the namespace-scoped installation mode [planned] in
security.md. It was written in the present tense while no implementation
exists — the informers are cluster-wide — which is the exact defect ADR 0024
was written to prevent, and I made it worse in #61 by describing how that
mode would treat the new cluster-scoped resources.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kfeofantov
kfeofantov merged commit b661ada into main Aug 24, 2026
3 checks passed
@kfeofantov
kfeofantov deleted the fix/policy-degrades-not-dies branch August 24, 2026 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant