diff --git a/docs/adr/0016-no-prometheus-as-a-source.md b/docs/adr/0016-no-prometheus-as-a-source.md new file mode 100644 index 0000000..01c069d --- /dev/null +++ b/docs/adr/0016-no-prometheus-as-a-source.md @@ -0,0 +1,94 @@ +# 0016. Prometheus is not a data source — not as an option, not for backfill + +Date: 2026-08-20 +Status: Accepted + +Supersedes the framing of Prometheus in [ADR 0006](0006-usage-collection-kubelet-rollups.md) +§Context, which rejected it for the usage path while leaving it standing as "a +side channel present in most clusters". ADR 0006 is otherwise unchanged: its +kubelet sources, merge property, and window contract stand. + +## Context + +The security document advertised a Prometheus integration the agent has never +had: a row in the network-access table (`query` / `query_range`, "historical +metrics over the retention window; enables a report on installation day"), a +mention in the `metrics-only` profile, and §10.1 "Prometheus is a side channel", +which disclosed its tenancy weakness as an accepted compromise. + +Nothing in the code reads Prometheus. So the document promised a capability that +did not exist — the mirror image of the §10.2 defect closed in ADR 0015, where +the code fell short of the promise. Both are the same failure: a customer-facing +claim not pinned to a decision. + +The pull to build it is real, and it always arrives phrased as temporary — a +day-one report needs history, the agent has none until it has run, and the +customer's Prometheus already holds months of it. + +**The reason to refuse is not that Prometheus data is low quality. It is that +its quality is unmeasurable.** A `query_range` response is a series of numbers +with no provenance attached. Recording rules, downsampling, and +`metric_relabel_configs` all shape it, and none of them leaves a trace in the +response. The agent cannot tell whether a series is raw, five-minute-averaged, or +dropped and re-derived; it cannot tell whether a workload is missing because it +did not exist or because a relabel rule discarded it. Every safeguard this agent +has built exists to prevent exactly that: ADR 0012 makes each payload declare its +provenance, ADR 0013 makes a measured payload declare how much of the window it +observed and whether each signal was seen at all. A Prometheus series can satisfy +none of those, so it would have to enter as a class whose completeness is +permanently unknown — and be joined, under the same natural keys, with data whose +completeness is exact. + +There is also a tenancy problem, which the old §10.1 already admitted: a +Prometheus endpoint typically serves all namespaces regardless of the caller's +Kubernetes permissions, so the customer's namespace filters would become a soft, +agent-side boundary on that path — while every other path in this agent enforces +them at collection (ADR 0015). + +## Decision + +**Prometheus is not a source of agent data. Not as a default, not as an opt-in, +not as a one-time backfill.** No configuration key, no Helm value, no profile +enables it. The agent's measurements come from the kubelet counters of ADR 0006 +and nothing else. + +The prohibition is on Prometheus as a *source of data about the customer's +workloads*. It does not touch: + +- **the `github.com/prometheus/*` Go modules in `go.mod`**, which parse the + kubelet's own `/metrics/cadvisor` text exposition. That is the kubelet's + format, not a Prometheus server, and the counters are ours to read directly. +- **the customer running Prometheus.** They usually do; the agent simply does not + read it. Its own resource use may of course be scraped by it. +- **exposing agent metrics** in the exposition format, if that is ever wanted. + That is data flowing outward, and it carries no claim about a workload. + +**What we give up, stated plainly: the agent knows nothing about the time before +it was installed.** A first-day report is therefore built from what is true at +install time and needs no history — declared requests and limits, QoS, +topology and placement, probe configuration, the object journal (conditions, +restart counts, ReplicaSet revisions, Job timings) — not from backfilled +measurements. Everything that genuinely requires observation says how long it has +been observing (`covered_nanoseconds`, ADR 0013) and is worth exactly that much. + +## Consequences + +**Easier.** Every measured number in the protocol has a known provenance and a +known completeness, with no exceptions to carve out downstream. Namespace filters +stay enforced at collection on every path. One fewer external dependency, one +fewer endpoint in the network table, one fewer credential. + +**Harder / given up.** No historical backfill, ever. A cluster installed today +has no usage rollups for yesterday, and a report that depends on a long window +must wait for that window. This is the cost, it is accepted deliberately, and +"just this once, to seed the account" is the exact request this ADR exists to +refuse — recording it now, while nothing is built, is the whole point. + +**Cost of the decision itself: zero code.** No Prometheus client was ever +written. This ADR removes three claims from `docs/security.md` and adds a +sentence a future reader can point at. It is being recorded at the cheapest +moment it will ever be available. + +This ADR records a decision implemented in the same pull request — here, as +documentation and a prohibition rather than code — per the process in +[`README.md`](README.md). diff --git a/docs/adr/README.md b/docs/adr/README.md index e3d927f..4a2b5c6 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -55,3 +55,4 @@ stays in place. - [0013. Observation completeness: measured payloads carry what the agent saw](0013-observation-completeness.md) - [0014. Facts of a different scope than the record's key are nested, not flattened](0014-scoped-facts-are-nested.md) - [0015. The node asks the controller which pods it may scan, and fails closed](0015-node-scan-scope.md) +- [0016. Prometheus is not a data source — not as an option, not for backfill](0016-no-prometheus-as-a-source.md) diff --git a/docs/security.md b/docs/security.md index 3eb78ef..a7d8117 100644 --- a/docs/security.md +++ b/docs/security.md @@ -35,7 +35,7 @@ marked **TBD** are being verified and will be resolved before GA. One binary, two roles: ``` -runtime-agent controller # StatefulSet, 1 replica — talks to k8s API, Prometheus, +runtime-agent controller # StatefulSet, 1 replica — talks to k8s API and # pod pprof endpoints; sole egress point runtime-agent node # DaemonSet (optional) — eBPF profiling and Go binary # detection on the node; talks only to the controller @@ -62,7 +62,7 @@ your goals; profiles can be upgraded later. | Profile | Components | Node privileges | What you get | |---|---|---|---| -| `metrics-only` | controller | none | Cost and efficiency findings from usage metrics (kubelet counters via the API server, ADR 0006; Prometheus as a side-channel for history) | +| `metrics-only` | controller | none | Cost and efficiency findings from usage metrics (kubelet counters via the API server, ADR 0006) and workload metadata | | `pprof` | controller | none | Above + CPU/heap profiles pulled from services that already expose `/debug/pprof` | | `ebpf` | controller + node DaemonSet | see [§7](#7-node-privileges) | Above + CPU profiles for services without pprof endpoints | @@ -125,7 +125,6 @@ read, never a write. | Access | Direction | Why | Notes | |---|---|---|---| -| Prometheus HTTP API (`query`, `query_range`) | controller → your Prometheus | Historical metrics over the retention window; enables a report on installation day | Read-only query API. Endpoint set in Helm values. See [§10.1](#101-prometheus-is-a-side-channel) | | Pod pprof endpoints | controller → pods | Pull `/debug/pprof/profile` and `/debug/pprof/heap` from Go services that already expose them | Only pods that pass the workload filters are probed; ports taken from `containerPorts`, never blind scans. `pprof`/`ebpf` profiles only | | kubelet stats, proxied | controller → API server | Poll `/stats/summary` and `/metrics/cadvisor` on every kubelet for usage counters (ADR 0006) | Goes through the API server proxy (`nodes/proxy`, §4) — the agent opens **no direct connection to kubelets**. A direct-kubelet transport for very large clusters would be a documented change here, not a silent one | | Backend egress | controller → one fixed domain | Ship aggregated rollups and filtered profiles | mTLS, pinned domain. The only cross-boundary connection in the system. A NetworkPolicy restricting controller egress to this domain plus in-cluster targets is shipped with the chart | @@ -511,6 +510,10 @@ Stated explicitly so it does not have to be asked: - No cloud provider credentials, IAM roles, or billing API access. Node pricing uses a static price table plus your stated discount. - No external egress from nodes — only the controller crosses the boundary. +- No access to your monitoring stack. The agent does not query Prometheus, + Thanos, Mimir, or any other metrics store — not as an option and not to + backfill history at install time (ADR 0016, [§10.1](#101-the-agent-knows-nothing-about-the-time-before-it-was-installed)). + It measures what it measures itself. - No API access from the node role at all — its ServiceAccount holds zero RBAC, and the only token it mounts is audience-bound to the controller, which the API server rejects ([§7](#7-node-privileges), ADR 0009/0010). The default API @@ -522,15 +525,32 @@ Stated explicitly so it does not have to be asked: ## 10. Known limitations and honest disclosures -### 10.1 Prometheus is a side channel - -Kubernetes RBAC does not govern what the agent can read from your Prometheus: -a Prometheus endpoint typically serves metrics for all namespaces regardless -of the caller's Kubernetes permissions. If you install namespace-scoped, be -aware that the metrics path is only constrained by the agent's own namespace -filters (soft boundary, auditable in the ConfigMap) unless your Prometheus -deployment enforces tenancy itself (e.g. a tenant-scoped query frontend). If -you have a scoped query endpoint, point the agent at it. +### 10.1 The agent knows nothing about the time before it was installed + +Every measurement the agent reports it made itself, from the kubelet counters +of ADR 0006. It does **not** read your Prometheus — not as an option, not for a +one-time backfill (ADR 0016) — so a cluster installed today has no usage data +for yesterday, and a finding that needs a long observation window has to wait +for that window. + +This is a deliberate refusal, not an unbuilt feature. A Prometheus response +carries no record of what shaped it: recording rules, downsampling and +`metric_relabel_configs` all alter a series and leave no trace in the answer, so +the agent could not tell a raw series from a five-minute average, nor a workload +that did not exist from one a relabel rule discarded. Everything else in this +agent's protocol declares its provenance and how completely it was observed +([§8](#8-data-collected-and-data-leaving-the-cluster)); imported series could +declare neither, and would end up joined with exact data under the same keys. +There is also a tenancy asymmetry: a Prometheus endpoint typically serves all +namespaces regardless of the caller's Kubernetes permissions, so your namespace +filters would be enforced only inside the agent on that path, while every other +path enforces them at collection. + +What does not need history still works on day one: declared requests and limits, +QoS, topology and placement, probe configuration, and the object journal +(conditions, restart counts, ReplicaSet revisions, Job timings) are all true at +install time. What genuinely needs observation reports how long it has been +observing, so its weight is visible rather than assumed. ### 10.2 Node-level visibility cannot be namespace-scoped