Skip to content

docs: Prometheus is not a data source (ADR 0016) - #42

Merged
kfeofantov merged 1 commit into
mainfrom
docs/no-prometheus-source
Aug 20, 2026
Merged

kfeofantov merged 1 commit into
mainfrom
docs/no-prometheus-source

Conversation

@kfeofantov

Copy link
Copy Markdown
Contributor

The discrepancy

docs/security.md advertised a Prometheus integration the agent has never had:

  • a row in the §5 network-access table — query / query_range, "historical metrics over the retention window; enables a report on installation day", "endpoint set in Helm values";
  • the metrics-only profile listing "Prometheus as a side-channel for history";
  • §10.1 "Prometheus is a side channel", disclosing its tenancy weakness as an accepted compromise.

Nothing in the code reads Prometheus. There is no client, no config key, no Helm value. The document promised a capability that did not exist — the mirror image of the §10.2 defect closed in ADR 0015, where the code fell short of the promise. Both are the same failure: a customer-facing claim not pinned to a decision.

Why refuse rather than build

Not because Prometheus data is low quality — because its quality is unmeasurable. A query_range response is numbers with no provenance attached. Recording rules, downsampling and metric_relabel_configs all shape a series and none of them leaves a trace in the answer. The agent could not tell a raw series from a five-minute average, nor a workload that never existed from one a relabel rule discarded.

That is precisely what the last four ADRs exist to prevent. ADR 0012 makes every payload declare its provenance; ADR 0013 makes a measured payload declare how much of the window it observed and whether each signal was seen at all. An imported series can declare neither — it would enter as a class whose completeness is permanently unknown, and be joined under the same natural keys with data whose completeness is exact.

There is also a tenancy asymmetry the old §10.1 already admitted: a Prometheus endpoint typically serves all namespaces regardless of the caller's Kubernetes permissions, so namespace filters would be enforced only inside the agent on that path, while every other path enforces them at collection (ADR 0015).

And the pull to build it always arrives phrased as temporary — just to seed the account. Recording the refusal now, while nothing is built, is the point.

What we give up, said out loud

The agent knows nothing about the time before it was installed. §10.1 now carries that under its own honest heading instead of a Prometheus caveat, together with what still works on day one and needs no history: declared requests and limits, QoS, topology and placement, probe configuration, and the object journal (conditions, restart counts, ReplicaSet revisions, Job timings). What genuinely needs observation already reports how long it has been observing (covered_nanoseconds), so its weight is visible rather than assumed.

Scope of the prohibition

It covers Prometheus as a source of data about customer workloads. Explicitly out of scope, stated in the ADR so a future reader does not misread go.mod:

  • the github.com/prometheus/* modules, which parse the kubelet's own /metrics/cadvisor text exposition — that is the kubelet's format, not a Prometheus server;
  • the customer running Prometheus (they usually do; the agent simply does not read it);
  • exposing agent metrics in the exposition format later — data flowing outward, carrying no claim about a workload.

Changes

  • ADR 0016 — the decision, written from the state of the code (which is: none). It supersedes only the framing of Prometheus in ADR 0006's context, which called it "a side channel present in most clusters"; 0006's kubelet sources, merge property and window contract stand unchanged. Note that ADR 0006's cross-reference to §10.1 now points at a section saying the opposite — ADRs are immutable, so 0016 names that explicitly rather than editing history.
  • §2 architecture diagram — the controller no longer "talks to Prometheus".
  • §3 metrics-only profile — Prometheus clause replaced with workload metadata.
  • §5 network table — row removed. One fewer external endpoint and credential.
  • §9 "What the agent does NOT request" — an explicit line: no access to your monitoring stack, Prometheus/Thanos/Mimir or otherwise, not even to backfill at install time.
  • §10.1 — rewritten as "The agent knows nothing about the time before it was installed".

Zero code changes; make test and make lint green.

🤖 Generated with Claude Code

security.md advertised a Prometheus integration the agent has never had: a
row in the network-access table (query/query_range, "enables a report on
installation day"), a mention in the metrics-only profile, and §10.1
disclosing its tenancy weakness as an accepted compromise. Nothing in the
code reads Prometheus. The document promised a capability that did not
exist — the mirror image of the §10.2 defect closed in ADR 0015, where the
code fell short of the promise.

The refusal is not about Prometheus data being low quality. It is that its
quality is unmeasurable: recording rules, downsampling and
metric_relabel_configs all shape a series and leave no trace in the
response, so the agent cannot tell a raw series from a five-minute average,
nor a workload that never existed from one a relabel rule discarded.
Everything this agent has built lately exists to prevent exactly that —
ADR 0012 makes each payload declare its provenance, ADR 0013 makes a
measured payload declare how much of the window it observed. An imported
series can declare neither, and would be joined with exact data under the
same natural keys. There is also a tenancy asymmetry the old §10.1 already
admitted: namespace filters would be enforced only inside the agent on that
path, while every other path enforces them at collection.

What we give up is stated plainly rather than hidden: the agent knows
nothing about the time before it was installed, and no backfill will ever
change that. §10.1 now says so under its own honest heading, along with what
still works on day one — declared resources, QoS, topology, probes, and the
object journal, none of which need history.

The prohibition covers Prometheus as a source of data about customer
workloads. It does not touch the github.com/prometheus/* modules in go.mod,
which parse the kubelet's own /metrics/cadvisor exposition, nor the customer
running Prometheus, nor exposing agent metrics in that format later.

Zero code: no Prometheus client was ever written, which is why this is the
cheapest moment this decision will ever be available. ADR 0016 supersedes
only the framing of Prometheus in ADR 0006's context; 0006's kubelet
sources, merge property and window contract stand unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kfeofantov
kfeofantov merged commit 5454b75 into main Aug 20, 2026
3 checks passed
@kfeofantov
kfeofantov deleted the docs/no-prometheus-source branch August 20, 2026 20:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant