From 2b23f9110b2af4d81e767c66c4cc7da3bcbf7ca7 Mon Sep 17 00:00:00 2001 From: Konstantins Feofantovs Date: Thu, 20 Aug 2026 23:33:20 +0300 Subject: [PATCH] docs: Prometheus is not a data source (ADR 0016) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit security.md advertised a Prometheus integration the agent has never had: a row in the network-access table (query/query_range, "enables a report on installation day"), a mention in the metrics-only profile, and §10.1 disclosing its tenancy weakness as an accepted compromise. Nothing in the code reads Prometheus. The document promised a capability that did not exist — the mirror image of the §10.2 defect closed in ADR 0015, where the code fell short of the promise. The refusal is not about Prometheus data being low quality. It is that its quality is unmeasurable: recording rules, downsampling and metric_relabel_configs all shape a series and leave no trace in the response, so the agent cannot tell a raw series from a five-minute average, nor a workload that never existed from one a relabel rule discarded. Everything this agent has built lately exists to prevent exactly that — ADR 0012 makes each payload declare its provenance, ADR 0013 makes a measured payload declare how much of the window it observed. An imported series can declare neither, and would be joined with exact data under the same natural keys. There is also a tenancy asymmetry the old §10.1 already admitted: namespace filters would be enforced only inside the agent on that path, while every other path enforces them at collection. What we give up is stated plainly rather than hidden: the agent knows nothing about the time before it was installed, and no backfill will ever change that. §10.1 now says so under its own honest heading, along with what still works on day one — declared resources, QoS, topology, probes, and the object journal, none of which need history. The prohibition covers Prometheus as a source of data about customer workloads. It does not touch the github.com/prometheus/* modules in go.mod, which parse the kubelet's own /metrics/cadvisor exposition, nor the customer running Prometheus, nor exposing agent metrics in that format later. Zero code: no Prometheus client was ever written, which is why this is the cheapest moment this decision will ever be available. ADR 0016 supersedes only the framing of Prometheus in ADR 0006's context; 0006's kubelet sources, merge property and window contract stand unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- docs/adr/0016-no-prometheus-as-a-source.md | 94 ++++++++++++++++++++++ docs/adr/README.md | 1 + docs/security.md | 44 +++++++--- 3 files changed, 127 insertions(+), 12 deletions(-) create mode 100644 docs/adr/0016-no-prometheus-as-a-source.md diff --git a/docs/adr/0016-no-prometheus-as-a-source.md b/docs/adr/0016-no-prometheus-as-a-source.md new file mode 100644 index 0000000..01c069d --- /dev/null +++ b/docs/adr/0016-no-prometheus-as-a-source.md @@ -0,0 +1,94 @@ +# 0016. Prometheus is not a data source — not as an option, not for backfill + +Date: 2026-08-20 +Status: Accepted + +Supersedes the framing of Prometheus in [ADR 0006](0006-usage-collection-kubelet-rollups.md) +§Context, which rejected it for the usage path while leaving it standing as "a +side channel present in most clusters". ADR 0006 is otherwise unchanged: its +kubelet sources, merge property, and window contract stand. + +## Context + +The security document advertised a Prometheus integration the agent has never +had: a row in the network-access table (`query` / `query_range`, "historical +metrics over the retention window; enables a report on installation day"), a +mention in the `metrics-only` profile, and §10.1 "Prometheus is a side channel", +which disclosed its tenancy weakness as an accepted compromise. + +Nothing in the code reads Prometheus. So the document promised a capability that +did not exist — the mirror image of the §10.2 defect closed in ADR 0015, where +the code fell short of the promise. Both are the same failure: a customer-facing +claim not pinned to a decision. + +The pull to build it is real, and it always arrives phrased as temporary — a +day-one report needs history, the agent has none until it has run, and the +customer's Prometheus already holds months of it. + +**The reason to refuse is not that Prometheus data is low quality. It is that +its quality is unmeasurable.** A `query_range` response is a series of numbers +with no provenance attached. Recording rules, downsampling, and +`metric_relabel_configs` all shape it, and none of them leaves a trace in the +response. The agent cannot tell whether a series is raw, five-minute-averaged, or +dropped and re-derived; it cannot tell whether a workload is missing because it +did not exist or because a relabel rule discarded it. Every safeguard this agent +has built exists to prevent exactly that: ADR 0012 makes each payload declare its +provenance, ADR 0013 makes a measured payload declare how much of the window it +observed and whether each signal was seen at all. A Prometheus series can satisfy +none of those, so it would have to enter as a class whose completeness is +permanently unknown — and be joined, under the same natural keys, with data whose +completeness is exact. + +There is also a tenancy problem, which the old §10.1 already admitted: a +Prometheus endpoint typically serves all namespaces regardless of the caller's +Kubernetes permissions, so the customer's namespace filters would become a soft, +agent-side boundary on that path — while every other path in this agent enforces +them at collection (ADR 0015). + +## Decision + +**Prometheus is not a source of agent data. Not as a default, not as an opt-in, +not as a one-time backfill.** No configuration key, no Helm value, no profile +enables it. The agent's measurements come from the kubelet counters of ADR 0006 +and nothing else. + +The prohibition is on Prometheus as a *source of data about the customer's +workloads*. It does not touch: + +- **the `github.com/prometheus/*` Go modules in `go.mod`**, which parse the + kubelet's own `/metrics/cadvisor` text exposition. That is the kubelet's + format, not a Prometheus server, and the counters are ours to read directly. +- **the customer running Prometheus.** They usually do; the agent simply does not + read it. Its own resource use may of course be scraped by it. +- **exposing agent metrics** in the exposition format, if that is ever wanted. + That is data flowing outward, and it carries no claim about a workload. + +**What we give up, stated plainly: the agent knows nothing about the time before +it was installed.** A first-day report is therefore built from what is true at +install time and needs no history — declared requests and limits, QoS, +topology and placement, probe configuration, the object journal (conditions, +restart counts, ReplicaSet revisions, Job timings) — not from backfilled +measurements. Everything that genuinely requires observation says how long it has +been observing (`covered_nanoseconds`, ADR 0013) and is worth exactly that much. + +## Consequences + +**Easier.** Every measured number in the protocol has a known provenance and a +known completeness, with no exceptions to carve out downstream. Namespace filters +stay enforced at collection on every path. One fewer external dependency, one +fewer endpoint in the network table, one fewer credential. + +**Harder / given up.** No historical backfill, ever. A cluster installed today +has no usage rollups for yesterday, and a report that depends on a long window +must wait for that window. This is the cost, it is accepted deliberately, and +"just this once, to seed the account" is the exact request this ADR exists to +refuse — recording it now, while nothing is built, is the whole point. + +**Cost of the decision itself: zero code.** No Prometheus client was ever +written. This ADR removes three claims from `docs/security.md` and adds a +sentence a future reader can point at. It is being recorded at the cheapest +moment it will ever be available. + +This ADR records a decision implemented in the same pull request — here, as +documentation and a prohibition rather than code — per the process in +[`README.md`](README.md). diff --git a/docs/adr/README.md b/docs/adr/README.md index e3d927f..4a2b5c6 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -55,3 +55,4 @@ stays in place. - [0013. Observation completeness: measured payloads carry what the agent saw](0013-observation-completeness.md) - [0014. Facts of a different scope than the record's key are nested, not flattened](0014-scoped-facts-are-nested.md) - [0015. The node asks the controller which pods it may scan, and fails closed](0015-node-scan-scope.md) +- [0016. Prometheus is not a data source — not as an option, not for backfill](0016-no-prometheus-as-a-source.md) diff --git a/docs/security.md b/docs/security.md index 3eb78ef..a7d8117 100644 --- a/docs/security.md +++ b/docs/security.md @@ -35,7 +35,7 @@ marked **TBD** are being verified and will be resolved before GA. One binary, two roles: ``` -runtime-agent controller # StatefulSet, 1 replica — talks to k8s API, Prometheus, +runtime-agent controller # StatefulSet, 1 replica — talks to k8s API and # pod pprof endpoints; sole egress point runtime-agent node # DaemonSet (optional) — eBPF profiling and Go binary # detection on the node; talks only to the controller @@ -62,7 +62,7 @@ your goals; profiles can be upgraded later. | Profile | Components | Node privileges | What you get | |---|---|---|---| -| `metrics-only` | controller | none | Cost and efficiency findings from usage metrics (kubelet counters via the API server, ADR 0006; Prometheus as a side-channel for history) | +| `metrics-only` | controller | none | Cost and efficiency findings from usage metrics (kubelet counters via the API server, ADR 0006) and workload metadata | | `pprof` | controller | none | Above + CPU/heap profiles pulled from services that already expose `/debug/pprof` | | `ebpf` | controller + node DaemonSet | see [§7](#7-node-privileges) | Above + CPU profiles for services without pprof endpoints | @@ -125,7 +125,6 @@ read, never a write. | Access | Direction | Why | Notes | |---|---|---|---| -| Prometheus HTTP API (`query`, `query_range`) | controller → your Prometheus | Historical metrics over the retention window; enables a report on installation day | Read-only query API. Endpoint set in Helm values. See [§10.1](#101-prometheus-is-a-side-channel) | | Pod pprof endpoints | controller → pods | Pull `/debug/pprof/profile` and `/debug/pprof/heap` from Go services that already expose them | Only pods that pass the workload filters are probed; ports taken from `containerPorts`, never blind scans. `pprof`/`ebpf` profiles only | | kubelet stats, proxied | controller → API server | Poll `/stats/summary` and `/metrics/cadvisor` on every kubelet for usage counters (ADR 0006) | Goes through the API server proxy (`nodes/proxy`, §4) — the agent opens **no direct connection to kubelets**. A direct-kubelet transport for very large clusters would be a documented change here, not a silent one | | Backend egress | controller → one fixed domain | Ship aggregated rollups and filtered profiles | mTLS, pinned domain. The only cross-boundary connection in the system. A NetworkPolicy restricting controller egress to this domain plus in-cluster targets is shipped with the chart | @@ -511,6 +510,10 @@ Stated explicitly so it does not have to be asked: - No cloud provider credentials, IAM roles, or billing API access. Node pricing uses a static price table plus your stated discount. - No external egress from nodes — only the controller crosses the boundary. +- No access to your monitoring stack. The agent does not query Prometheus, + Thanos, Mimir, or any other metrics store — not as an option and not to + backfill history at install time (ADR 0016, [§10.1](#101-the-agent-knows-nothing-about-the-time-before-it-was-installed)). + It measures what it measures itself. - No API access from the node role at all — its ServiceAccount holds zero RBAC, and the only token it mounts is audience-bound to the controller, which the API server rejects ([§7](#7-node-privileges), ADR 0009/0010). The default API @@ -522,15 +525,32 @@ Stated explicitly so it does not have to be asked: ## 10. Known limitations and honest disclosures -### 10.1 Prometheus is a side channel - -Kubernetes RBAC does not govern what the agent can read from your Prometheus: -a Prometheus endpoint typically serves metrics for all namespaces regardless -of the caller's Kubernetes permissions. If you install namespace-scoped, be -aware that the metrics path is only constrained by the agent's own namespace -filters (soft boundary, auditable in the ConfigMap) unless your Prometheus -deployment enforces tenancy itself (e.g. a tenant-scoped query frontend). If -you have a scoped query endpoint, point the agent at it. +### 10.1 The agent knows nothing about the time before it was installed + +Every measurement the agent reports it made itself, from the kubelet counters +of ADR 0006. It does **not** read your Prometheus — not as an option, not for a +one-time backfill (ADR 0016) — so a cluster installed today has no usage data +for yesterday, and a finding that needs a long observation window has to wait +for that window. + +This is a deliberate refusal, not an unbuilt feature. A Prometheus response +carries no record of what shaped it: recording rules, downsampling and +`metric_relabel_configs` all alter a series and leave no trace in the answer, so +the agent could not tell a raw series from a five-minute average, nor a workload +that did not exist from one a relabel rule discarded. Everything else in this +agent's protocol declares its provenance and how completely it was observed +([§8](#8-data-collected-and-data-leaving-the-cluster)); imported series could +declare neither, and would end up joined with exact data under the same keys. +There is also a tenancy asymmetry: a Prometheus endpoint typically serves all +namespaces regardless of the caller's Kubernetes permissions, so your namespace +filters would be enforced only inside the agent on that path, while every other +path enforces them at collection. + +What does not need history still works on day one: declared requests and limits, +QoS, topology and placement, probe configuration, and the object journal +(conditions, restart counts, ReplicaSet revisions, Job timings) are all true at +install time. What genuinely needs observation reports how long it has been +observing, so its weight is visible rather than assumed. ### 10.2 Node-level visibility cannot be namespace-scoped