Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cmd/agent/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -363,7 +363,7 @@ func run(ctx context.Context, logger *slog.Logger, clientset kubernetes.Interfac
// One capture instant for both payloads of a flush: they describe the
// same cluster state and are joined against each other downstream.
capturedAt := time.Now()
records := metadata.Aggregate(podWatcher.Pods(), podWatcher.Rollouts())
records := metadata.Aggregate(podWatcher.Pods(), podWatcher.UpdateStrategies())
if err := spool.WriteWorkloadMetadata(capturedAt, records); err != nil {
logger.Error("spooling workload metadata", "error", err)
}
Expand Down
51 changes: 34 additions & 17 deletions docs/adr/0048-findings-name-the-fields-they-need.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ Services to `workload_policy`, and a version to each module in `go_build`. Adds
one resource to the ClusterRole (`services`). Extends the informer transform of
ADR 0046 to the probe handlers, which were being cached whole.

*Corrected 2026-08-28, the day it was accepted and before any release: §2 first
named the field `rollout` and gave unread kinds an `unread` marker. The first
run against a real cluster refuted both — see §2.*

## Context

Every decision here so far has answered the same question one fact at a time:
Expand Down Expand Up @@ -62,22 +66,35 @@ The kind is kept and the target is not, because the finding asks whether two
probes are the same check, not what either calls. The numbers are the API
server's defaulted ones, which are the numbers the kubelet will use.

**2. The workload's update strategy ships in a `workload` scope block.** Type,
**2. The workload's update strategy ships as `workload.update_strategy`.** Type,
`maxUnavailable`, `maxSurge`, a StatefulSet's `partition`, and
`minReadySeconds`, read from the Deployment, StatefulSet or DaemonSet the pods
belong to.

Absence has to answer two different questions, and one flag separates them. A
Job, a CronJob, a bare ReplicaSet and an owner-less pod replace no replicas, so
they carry no rollout and that absence is the fact. A pod managed by a custom
resource — an Argo Rollout, a Knative Revision, an in-house operator — is the
opposite case: it almost certainly *does* roll, and the agent holds no RBAC to
read how (`docs/security.md` §11). Reporting nothing there would state "this
does not roll", which is worse than silence. Those records carry
`rollout: {unread: true}` and nothing else — ADR 0013's rule that the agent
reports what it observed rather than what it concluded, applied to a structural
payload. This is the case a first-hour report must not get wrong quietly: Argo
Rollouts is common, and the finding is about exactly the workloads it manages.
belong to. Those three kinds and no others.

**The field is not called `rollout`, and that is a decision rather than a
preference.** `Rollout` is the kind name of the Argo custom resource, which this
agent already reports in `workload_kind`. One record carrying
`workload_kind: Rollout` beside a field called `rollout` puts two meanings of
one word in one object, and the generic English noun is the one that loses.
`updateStrategy` is Kubernetes' own field name on StatefulSet and DaemonSet, so
it is borrowed rather than invented.

**Absence means the agent did not read a strategy, and claims nothing beyond
that.** The first version of this decision tried to say more: kinds known to
replace no replicas carried nothing, and everything else carried
`rollout: {unread: true}` so that a Job could be told from an Argo Rollout. That
was wrong twice over, and one run against a real cluster showed both. It needs a
complete list of owner kinds that do not update themselves, which nobody has —
the list omitted `Node`, so every static control-plane pod on a self-managed
cluster was reported as having an update strategy the agent had failed to read.
And the claim itself is a judgement: whether an unknown custom resource updates
its replicas is not something the agent observes, and this agent reduces rather
than concludes (ADR 0004).

What distinguishes the cases is already in the record. `workload_kind` ships in
the key, the kinds this agent reads are three and are named in
`docs/backend-requirements.md`, and what any other kind implies is a rendering
question for the side that renders.

It is a fact of the workload, and the record's key is narrower than that, so it
is nested and not flattened (ADR 0014) — a `workload` object beside the existing
Expand Down Expand Up @@ -161,9 +178,9 @@ No CRD is read, and the closed ClusterRole that §9 of `docs/security.md` leans
on gains no vendor resource. Reading Argo Rollouts properly is a larger decision
than a field — a first custom-resource grant, an informer for a kind that may not
exist in the cluster, and a strategy whose canary and blue-green shapes are not
this struct — so it is left to its own ADR. What this one owes is that the gap is
legible from the payload rather than inferred from a list of kinds the backend
would have to keep in step with the agent.
this struct — so it is left to its own ADR. Until then the rollout dimension of a
first-hour report is blank for an Argo-managed workload, and blank is what the
payload says.

**Not changed.** No new payload kind, no registry row, no change to any natural
key, no verb that writes, no new privilege on the node, and nothing new is read
Expand Down
10 changes: 5 additions & 5 deletions docs/backend-requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,11 +279,11 @@ was taken away, which is the normal state.
**`workload_metadata.workload` and `.pod` are two different scopes, and neither
is per record** (ADR 0014). A pod's containers each produce a record repeating
both blocks, so **the backend MUST NOT sum either across one workload's
records**. `rollout` is absent for a kind that replaces no replicas — a Job, a
CronJob, a bare ReplicaSet, an owner-less pod — and MUST NOT be rendered as one
that replaces everything at once. A `rollout` carrying only `unread: true` is a
workload managed by a custom resource the agent holds no RBAC for, such as an
Argo Rollout: **unobserved, not absent**, and MUST NOT be read as no strategy.
records**. `workload.update_strategy` is present only for `workload_kind` of
`Deployment`, `StatefulSet` or `DaemonSet` — the kinds the agent reads. For every
other kind it is absent, and **the backend MUST NOT read that absence as "this
workload has no update strategy"**: an Argo Rollout has one and the agent holds
no RBAC for it. Absence says only that the agent did not read one.

**`workload_metadata.pod.unscheduled` explains the replica shortfall, it does
not restate it.** The gap between `replicas` and the sum of `nodes` has always
Expand Down
2 changes: 1 addition & 1 deletion docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -490,7 +490,7 @@ test fails if a kind ships without appearing here (ADR 0022).
| `pod_disruptions` | Per hour: the pods the *cluster* removed — preempted, evicted under node pressure, drained, or removed via the eviction API — with the node and the instant | **yes** |
| `job_runs` | Per hour: each Job that finished — when it started and finished, whether it succeeded, the failure reason, its pod success/failure counts, and its declared `parallelism`/`completions`/`backoffLimit` | **yes** |
| `deployment_revisions` | Per flush: each ReplicaSet of a collected Deployment — its revision number, when it was created, its desired/current/ready replica counts, and each container's image reference | **yes** |
| `workload_metadata` | Declared shape per (namespace, workload, container, image digest): image, requests, limits, ports, probe schedules without what any probe checks, the five runtime knobs of the field-minimization list below, QoS, replica counts by phase, by node, and by unscheduled reason, the pod's reduced placement constraints (ADR 0031), and how the workload replaces its own replicas — or, for a controller the agent does not read, that it did not read it ([ADR 0048](adr/0048-findings-name-the-fields-they-need.md)) | no |
| `workload_metadata` | Declared shape per (namespace, workload, container, image digest): image, requests, limits, ports, probe schedules without what any probe checks, the five runtime knobs of the field-minimization list below, QoS, replica counts by phase, by node, and by unscheduled reason, the pod's reduced placement constraints (ADR 0031), and — for a Deployment, StatefulSet or DaemonSet — the update strategy by which the workload replaces its own replicas ([ADR 0048](adr/0048-findings-name-the-fields-they-need.md)). A workload of any other kind carries no strategy and nothing standing in for one | no |
| `workload_policy` | Per collected workload, what bounds it from outside its own spec: disruption budgets covering it, autoscalers driving it, the volume claims its pods mount (ADR 0032), and the Services routing to it (ADR 0048); plus `unavailable_sources`, naming any of those the agent could not read — never granted, or no longer being fed (ADR 0033, ADR 0035) | not directly — but a claim is named, and a StatefulSet's claim is `<template>-<set>-<ordinal>`, so `data-db-0` identifies pod `db-0` (ADR 0039) |
| `cluster_policy` | Per collected namespace, its LimitRanges and ResourceQuotas; plus the cluster's PriorityClasses and StorageClasses, which workloads reference by name (ADR 0032), and `unavailable_sources` for any of those the agent could not read — never granted, or no longer being fed (ADR 0033, ADR 0035). StorageClass `parameters` are never read | no |
| `node_metadata` | Per node: name, size, instance and capacity type, zone, region, kernel version, CPU architecture. The **name** is your cluster's, and on EKS and on GKE's legacy naming it encodes the node's private address — `ip-10-42-13-201.eu-west-1.compute.internal`. The agent reads no address field; the name it does read may be one (ADR 0039) | no |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,15 @@ type WorkloadKey struct {
Name string
}

// Rollout is how a workload replaces its own replicas: the one thing that
// UpdateStrategy is how a workload replaces its own replicas: the one thing that
// decides whether a routine deploy is also an outage. A budget says what an
// eviction may take away; this says what the workload takes away from itself,
// and no budget is consulted on that path (ADR 0048 §2).
//
// It is a fact of the workload, not of a pod or a build, so it is nested rather
// than flattened onto records keyed more narrowly (ADR 0014).
type Rollout struct {
// It is named for Kubernetes' own field rather than for the process, because
// `Rollout` is the kind name of a custom resource this agent reports in
// `workload_kind` — one word for both would collide in a single record.
type UpdateStrategy struct {
// Type is the strategy in Kubernetes' own vocabulary: RollingUpdate or
// Recreate for a Deployment, RollingUpdate or OnDelete for a StatefulSet or
// DaemonSet.
Expand All @@ -37,35 +38,24 @@ type Rollout struct {
// looks like a stalled rollout from every other angle.
Partition *int32 `json:"partition,omitempty"`
// MinReadySeconds is how long a new replica must stay ready before the
// rollout treats it as available. Zero is the default and is what makes a
// update treats it as available. Zero is the default and is what makes a
// crash-on-first-request roll through every replica.
MinReadySeconds int32 `json:"min_ready_seconds,omitempty"`
// Unread marks a workload whose controller is a kind the agent does not
// read — an Argo Rollout, a Knative Revision, an in-house operator. It is
// the only field set when it is true: the workload may well roll, and this
// says the agent did not look, which is not the same fact as a kind that
// does not roll at all (ADR 0048 §2, ADR 0013).
Unread bool `json:"unread,omitempty"`
}

// rolloutlessKinds are the kinds known to replace no replicas: nothing about
// them is unobserved, so their absence from Rollouts is the fact itself.
var rolloutlessKinds = map[string]bool{
"Job": true, "CronJob": true, "ReplicaSet": true, "none": true, "": true,
}

// Rollouts returns the update strategy of every workload with admitted pods.
// Scope is inherited from the admitted pod index rather than decided again, as
// with revisions (ADR 0030) and policy (ADR 0032). A kind with no strategy — a
// bare Job, a CronJob, an owner-less pod — is absent: a shape, not a gap.
// UpdateStrategies returns the declared strategy of every workload with admitted
// pods whose kind the agent reads: Deployment, StatefulSet and DaemonSet. Any
// other kind is absent, and absence claims nothing about whether that workload
// updates itself — only that this agent did not read how (ADR 0048 §2).
//
// It is read at flush time and not when a pod was described, because a strategy
// can be edited with no pod event following it.
func (w *PodWatcher) Rollouts() map[WorkloadKey]Rollout {
out := make(map[WorkloadKey]Rollout)
// Scope is inherited from the admitted pod index (ADR 0030, ADR 0032). It is
// read at flush time and not when a pod was described, because a strategy can
// be edited with no pod event following it.
func (w *PodWatcher) UpdateStrategies() map[WorkloadKey]UpdateStrategy {
out := make(map[WorkloadKey]UpdateStrategy)
for key := range w.collectedWorkloadKeys() {
if rollout, ok := w.rolloutOf(key); ok {
out[key] = rollout
if strategy, ok := w.strategyOf(key); ok {
out[key] = strategy
}
}
return out
Expand All @@ -86,61 +76,55 @@ func (w *PodWatcher) collectedWorkloadKeys() map[WorkloadKey]struct{} {
return out
}

// rolloutOf reads one workload's strategy from the cache its kind lives in. The
// strategyOf reads one workload's strategy from the cache its kind lives in. The
// three listers are gating caches (ADR 0035), so a miss here is informer lag or
// a workload that has just been deleted, never a permission the agent lacks.
func (w *PodWatcher) rolloutOf(key WorkloadKey) (Rollout, bool) {
if rolloutlessKinds[key.Kind] {
return Rollout{}, false
}
func (w *PodWatcher) strategyOf(key WorkloadKey) (UpdateStrategy, bool) {
switch key.Kind {
case "Deployment":
if w.deployLister == nil {
return Rollout{}, false
return UpdateStrategy{}, false
}
d, err := w.deployLister.Deployments(key.Namespace).Get(key.Name)
if err != nil {
return Rollout{}, false
return UpdateStrategy{}, false
}
r := Rollout{Type: string(d.Spec.Strategy.Type), MinReadySeconds: d.Spec.MinReadySeconds}
s := UpdateStrategy{Type: string(d.Spec.Strategy.Type), MinReadySeconds: d.Spec.MinReadySeconds}
if u := d.Spec.Strategy.RollingUpdate; u != nil {
r.MaxUnavailable = intOrString(u.MaxUnavailable)
r.MaxSurge = intOrString(u.MaxSurge)
s.MaxUnavailable = intOrString(u.MaxUnavailable)
s.MaxSurge = intOrString(u.MaxSurge)
}
return r, true
return s, true
case "StatefulSet":
if w.stsLister == nil {
return Rollout{}, false
return UpdateStrategy{}, false
}
s, err := w.stsLister.StatefulSets(key.Namespace).Get(key.Name)
sts, err := w.stsLister.StatefulSets(key.Namespace).Get(key.Name)
if err != nil {
return Rollout{}, false
return UpdateStrategy{}, false
}
r := Rollout{Type: string(s.Spec.UpdateStrategy.Type), MinReadySeconds: s.Spec.MinReadySeconds}
if u := s.Spec.UpdateStrategy.RollingUpdate; u != nil {
r.Partition = u.Partition
r.MaxUnavailable = intOrString(u.MaxUnavailable)
s := UpdateStrategy{Type: string(sts.Spec.UpdateStrategy.Type), MinReadySeconds: sts.Spec.MinReadySeconds}
if u := sts.Spec.UpdateStrategy.RollingUpdate; u != nil {
s.Partition = u.Partition
s.MaxUnavailable = intOrString(u.MaxUnavailable)
}
return r, true
return s, true
case "DaemonSet":
if w.dsLister == nil {
return Rollout{}, false
return UpdateStrategy{}, false
}
d, err := w.dsLister.DaemonSets(key.Namespace).Get(key.Name)
if err != nil {
return Rollout{}, false
return UpdateStrategy{}, false
}
r := Rollout{Type: string(d.Spec.UpdateStrategy.Type), MinReadySeconds: d.Spec.MinReadySeconds}
s := UpdateStrategy{Type: string(d.Spec.UpdateStrategy.Type), MinReadySeconds: d.Spec.MinReadySeconds}
if u := d.Spec.UpdateStrategy.RollingUpdate; u != nil {
r.MaxUnavailable = intOrString(u.MaxUnavailable)
r.MaxSurge = intOrString(u.MaxSurge)
s.MaxUnavailable = intOrString(u.MaxUnavailable)
s.MaxSurge = intOrString(u.MaxSurge)
}
return r, true
return s, true
}
// A custom resource. Reading it would need RBAC on arbitrary CRDs, which
// this product does not ask for (docs/security.md §11), so what the agent
// can honestly say is that it did not look.
return Rollout{Unread: true}, true
return UpdateStrategy{}, false
}

// intOrString renders a declaration that may be a count or a percentage as it
Expand Down
Loading