Repository navigation
docs: Prometheus is not a data source (ADR 0016) - #42
Merged
Merged
Conversation
security.md advertised a Prometheus integration the agent has never had: a row in the network-access table (query/query_range, "enables a report on installation day"), a mention in the metrics-only profile, and §10.1 disclosing its tenancy weakness as an accepted compromise. Nothing in the code reads Prometheus. The document promised a capability that did not exist — the mirror image of the §10.2 defect closed in ADR 0015, where the code fell short of the promise. The refusal is not about Prometheus data being low quality. It is that its quality is unmeasurable: recording rules, downsampling and metric_relabel_configs all shape a series and leave no trace in the response, so the agent cannot tell a raw series from a five-minute average, nor a workload that never existed from one a relabel rule discarded. Everything this agent has built lately exists to prevent exactly that — ADR 0012 makes each payload declare its provenance, ADR 0013 makes a measured payload declare how much of the window it observed. An imported series can declare neither, and would be joined with exact data under the same natural keys. There is also a tenancy asymmetry the old §10.1 already admitted: namespace filters would be enforced only inside the agent on that path, while every other path enforces them at collection. What we give up is stated plainly rather than hidden: the agent knows nothing about the time before it was installed, and no backfill will ever change that. §10.1 now says so under its own honest heading, along with what still works on day one — declared resources, QoS, topology, probes, and the object journal, none of which need history. The prohibition covers Prometheus as a source of data about customer workloads. It does not touch the github.com/prometheus/* modules in go.mod, which parse the kubelet's own /metrics/cadvisor exposition, nor the customer running Prometheus, nor exposing agent metrics in that format later. Zero code: no Prometheus client was ever written, which is why this is the cheapest moment this decision will ever be available. ADR 0016 supersedes only the framing of Prometheus in ADR 0006's context; 0006's kubelet sources, merge property and window contract stand unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The discrepancy
docs/security.mdadvertised a Prometheus integration the agent has never had:query/query_range, "historical metrics over the retention window; enables a report on installation day", "endpoint set in Helm values";metrics-onlyprofile listing "Prometheus as a side-channel for history";Nothing in the code reads Prometheus. There is no client, no config key, no Helm value. The document promised a capability that did not exist — the mirror image of the §10.2 defect closed in ADR 0015, where the code fell short of the promise. Both are the same failure: a customer-facing claim not pinned to a decision.
Why refuse rather than build
Not because Prometheus data is low quality — because its quality is unmeasurable. A
query_rangeresponse is numbers with no provenance attached. Recording rules, downsampling andmetric_relabel_configsall shape a series and none of them leaves a trace in the answer. The agent could not tell a raw series from a five-minute average, nor a workload that never existed from one a relabel rule discarded.That is precisely what the last four ADRs exist to prevent. ADR 0012 makes every payload declare its provenance; ADR 0013 makes a measured payload declare how much of the window it observed and whether each signal was seen at all. An imported series can declare neither — it would enter as a class whose completeness is permanently unknown, and be joined under the same natural keys with data whose completeness is exact.
There is also a tenancy asymmetry the old §10.1 already admitted: a Prometheus endpoint typically serves all namespaces regardless of the caller's Kubernetes permissions, so namespace filters would be enforced only inside the agent on that path, while every other path enforces them at collection (ADR 0015).
And the pull to build it always arrives phrased as temporary — just to seed the account. Recording the refusal now, while nothing is built, is the point.
What we give up, said out loud
The agent knows nothing about the time before it was installed. §10.1 now carries that under its own honest heading instead of a Prometheus caveat, together with what still works on day one and needs no history: declared requests and limits, QoS, topology and placement, probe configuration, and the object journal (conditions, restart counts, ReplicaSet revisions, Job timings). What genuinely needs observation already reports how long it has been observing (
covered_nanoseconds), so its weight is visible rather than assumed.Scope of the prohibition
It covers Prometheus as a source of data about customer workloads. Explicitly out of scope, stated in the ADR so a future reader does not misread
go.mod:github.com/prometheus/*modules, which parse the kubelet's own/metrics/cadvisortext exposition — that is the kubelet's format, not a Prometheus server;Changes
metrics-onlyprofile — Prometheus clause replaced with workload metadata.Zero code changes;
make testandmake lintgreen.🤖 Generated with Claude Code