Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions docs/adr/0016-no-prometheus-as-a-source.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# 0016. Prometheus is not a data source — not as an option, not for backfill

Date: 2026-08-20
Status: Accepted

Supersedes the framing of Prometheus in [ADR 0006](0006-usage-collection-kubelet-rollups.md)
§Context, which rejected it for the usage path while leaving it standing as "a
side channel present in most clusters". ADR 0006 is otherwise unchanged: its
kubelet sources, merge property, and window contract stand.

## Context

The security document advertised a Prometheus integration the agent has never
had: a row in the network-access table (`query` / `query_range`, "historical
metrics over the retention window; enables a report on installation day"), a
mention in the `metrics-only` profile, and §10.1 "Prometheus is a side channel",
which disclosed its tenancy weakness as an accepted compromise.

Nothing in the code reads Prometheus. So the document promised a capability that
did not exist — the mirror image of the §10.2 defect closed in ADR 0015, where
the code fell short of the promise. Both are the same failure: a customer-facing
claim not pinned to a decision.

The pull to build it is real, and it always arrives phrased as temporary — a
day-one report needs history, the agent has none until it has run, and the
customer's Prometheus already holds months of it.

**The reason to refuse is not that Prometheus data is low quality. It is that
its quality is unmeasurable.** A `query_range` response is a series of numbers
with no provenance attached. Recording rules, downsampling, and
`metric_relabel_configs` all shape it, and none of them leaves a trace in the
response. The agent cannot tell whether a series is raw, five-minute-averaged, or
dropped and re-derived; it cannot tell whether a workload is missing because it
did not exist or because a relabel rule discarded it. Every safeguard this agent
has built exists to prevent exactly that: ADR 0012 makes each payload declare its
provenance, ADR 0013 makes a measured payload declare how much of the window it
observed and whether each signal was seen at all. A Prometheus series can satisfy
none of those, so it would have to enter as a class whose completeness is
permanently unknown — and be joined, under the same natural keys, with data whose
completeness is exact.

There is also a tenancy problem, which the old §10.1 already admitted: a
Prometheus endpoint typically serves all namespaces regardless of the caller's
Kubernetes permissions, so the customer's namespace filters would become a soft,
agent-side boundary on that path — while every other path in this agent enforces
them at collection (ADR 0015).

## Decision

**Prometheus is not a source of agent data. Not as a default, not as an opt-in,
not as a one-time backfill.** No configuration key, no Helm value, no profile
enables it. The agent's measurements come from the kubelet counters of ADR 0006
and nothing else.

The prohibition is on Prometheus as a *source of data about the customer's
workloads*. It does not touch:

- **the `github.com/prometheus/*` Go modules in `go.mod`**, which parse the
kubelet's own `/metrics/cadvisor` text exposition. That is the kubelet's
format, not a Prometheus server, and the counters are ours to read directly.
- **the customer running Prometheus.** They usually do; the agent simply does not
read it. Its own resource use may of course be scraped by it.
- **exposing agent metrics** in the exposition format, if that is ever wanted.
That is data flowing outward, and it carries no claim about a workload.

**What we give up, stated plainly: the agent knows nothing about the time before
it was installed.** A first-day report is therefore built from what is true at
install time and needs no history — declared requests and limits, QoS,
topology and placement, probe configuration, the object journal (conditions,
restart counts, ReplicaSet revisions, Job timings) — not from backfilled
measurements. Everything that genuinely requires observation says how long it has
been observing (`covered_nanoseconds`, ADR 0013) and is worth exactly that much.

## Consequences

**Easier.** Every measured number in the protocol has a known provenance and a
known completeness, with no exceptions to carve out downstream. Namespace filters
stay enforced at collection on every path. One fewer external dependency, one
fewer endpoint in the network table, one fewer credential.

**Harder / given up.** No historical backfill, ever. A cluster installed today
has no usage rollups for yesterday, and a report that depends on a long window
must wait for that window. This is the cost, it is accepted deliberately, and
"just this once, to seed the account" is the exact request this ADR exists to
refuse — recording it now, while nothing is built, is the whole point.

**Cost of the decision itself: zero code.** No Prometheus client was ever
written. This ADR removes three claims from `docs/security.md` and adds a
sentence a future reader can point at. It is being recorded at the cheapest
moment it will ever be available.

This ADR records a decision implemented in the same pull request — here, as
documentation and a prohibition rather than code — per the process in
[`README.md`](README.md).
1 change: 1 addition & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,4 @@ stays in place.
- [0013. Observation completeness: measured payloads carry what the agent saw](0013-observation-completeness.md)
- [0014. Facts of a different scope than the record's key are nested, not flattened](0014-scoped-facts-are-nested.md)
- [0015. The node asks the controller which pods it may scan, and fails closed](0015-node-scan-scope.md)
- [0016. Prometheus is not a data source — not as an option, not for backfill](0016-no-prometheus-as-a-source.md)
44 changes: 32 additions & 12 deletions docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ marked **TBD** are being verified and will be resolved before GA.
One binary, two roles:

```
runtime-agent controller # StatefulSet, 1 replica — talks to k8s API, Prometheus,
runtime-agent controller # StatefulSet, 1 replica — talks to k8s API and
# pod pprof endpoints; sole egress point
runtime-agent node # DaemonSet (optional) — eBPF profiling and Go binary
# detection on the node; talks only to the controller
Expand All @@ -62,7 +62,7 @@ your goals; profiles can be upgraded later.

| Profile | Components | Node privileges | What you get |
|---|---|---|---|
| `metrics-only` | controller | none | Cost and efficiency findings from usage metrics (kubelet counters via the API server, ADR 0006; Prometheus as a side-channel for history) |
| `metrics-only` | controller | none | Cost and efficiency findings from usage metrics (kubelet counters via the API server, ADR 0006) and workload metadata |
| `pprof` | controller | none | Above + CPU/heap profiles pulled from services that already expose `/debug/pprof` |
| `ebpf` | controller + node DaemonSet | see [§7](#7-node-privileges) | Above + CPU profiles for services without pprof endpoints |

Expand Down Expand Up @@ -125,7 +125,6 @@ read, never a write.

| Access | Direction | Why | Notes |
|---|---|---|---|
| Prometheus HTTP API (`query`, `query_range`) | controller → your Prometheus | Historical metrics over the retention window; enables a report on installation day | Read-only query API. Endpoint set in Helm values. See [§10.1](#101-prometheus-is-a-side-channel) |
| Pod pprof endpoints | controller → pods | Pull `/debug/pprof/profile` and `/debug/pprof/heap` from Go services that already expose them | Only pods that pass the workload filters are probed; ports taken from `containerPorts`, never blind scans. `pprof`/`ebpf` profiles only |
| kubelet stats, proxied | controller → API server | Poll `/stats/summary` and `/metrics/cadvisor` on every kubelet for usage counters (ADR 0006) | Goes through the API server proxy (`nodes/proxy`, §4) — the agent opens **no direct connection to kubelets**. A direct-kubelet transport for very large clusters would be a documented change here, not a silent one |
| Backend egress | controller → one fixed domain | Ship aggregated rollups and filtered profiles | mTLS, pinned domain. The only cross-boundary connection in the system. A NetworkPolicy restricting controller egress to this domain plus in-cluster targets is shipped with the chart |
Expand Down Expand Up @@ -511,6 +510,10 @@ Stated explicitly so it does not have to be asked:
- No cloud provider credentials, IAM roles, or billing API access. Node
pricing uses a static price table plus your stated discount.
- No external egress from nodes — only the controller crosses the boundary.
- No access to your monitoring stack. The agent does not query Prometheus,
Thanos, Mimir, or any other metrics store — not as an option and not to
backfill history at install time (ADR 0016, [§10.1](#101-the-agent-knows-nothing-about-the-time-before-it-was-installed)).
It measures what it measures itself.
- No API access from the node role at all — its ServiceAccount holds zero RBAC,
and the only token it mounts is audience-bound to the controller, which the
API server rejects ([§7](#7-node-privileges), ADR 0009/0010). The default API
Expand All @@ -522,15 +525,32 @@ Stated explicitly so it does not have to be asked:

## 10. Known limitations and honest disclosures

### 10.1 Prometheus is a side channel

Kubernetes RBAC does not govern what the agent can read from your Prometheus:
a Prometheus endpoint typically serves metrics for all namespaces regardless
of the caller's Kubernetes permissions. If you install namespace-scoped, be
aware that the metrics path is only constrained by the agent's own namespace
filters (soft boundary, auditable in the ConfigMap) unless your Prometheus
deployment enforces tenancy itself (e.g. a tenant-scoped query frontend). If
you have a scoped query endpoint, point the agent at it.
### 10.1 The agent knows nothing about the time before it was installed

Every measurement the agent reports it made itself, from the kubelet counters
of ADR 0006. It does **not** read your Prometheus — not as an option, not for a
one-time backfill (ADR 0016) — so a cluster installed today has no usage data
for yesterday, and a finding that needs a long observation window has to wait
for that window.

This is a deliberate refusal, not an unbuilt feature. A Prometheus response
carries no record of what shaped it: recording rules, downsampling and
`metric_relabel_configs` all alter a series and leave no trace in the answer, so
the agent could not tell a raw series from a five-minute average, nor a workload
that did not exist from one a relabel rule discarded. Everything else in this
agent's protocol declares its provenance and how completely it was observed
([§8](#8-data-collected-and-data-leaving-the-cluster)); imported series could
declare neither, and would end up joined with exact data under the same keys.
There is also a tenancy asymmetry: a Prometheus endpoint typically serves all
namespaces regardless of the caller's Kubernetes permissions, so your namespace
filters would be enforced only inside the agent on that path, while every other
path enforces them at collection.

What does not need history still works on day one: declared requests and limits,
QoS, topology and placement, probe configuration, and the object journal
(conditions, restart counts, ReplicaSet revisions, Job timings) are all true at
install time. What genuinely needs observation reports how long it has been
observing, so its weight is visible rather than assumed.

### 10.2 Node-level visibility cannot be namespace-scoped

Expand Down