Skip to content

feat: the agent says what it did, on the port it already opens (ADR 0070) - #108

Merged
kfeofantov merged 1 commit into
mainfrom
feat/self-metrics
Sep 6, 2026
Merged

kfeofantov merged 1 commit into
mainfrom
feat/self-metrics

Conversation

@kfeofantov

Copy link
Copy Markdown
Contributor

Adds /metrics to the health listener of both roles (ADR 0070). ADR 0069 §1
picked port 9090 for exactly this — "the metrics endpoint of a later slice joins
this listener as a path" — so no chart template changes.

The two decisions that shaped it

Exposition without a new dependency. prometheus/common/expfmt and
client_model are already direct dependencies: internal/collector/cadvisor.go
decodes this format to read the kubelet. Encoding with the same package adds
no module to the build, so the reachability gate (ADR 0038) gets nothing new,
and the format is still the reference implementation's rather than hand-rolled.

client_golang was rejected for that, and for a second reason that mattered
more: its shape is a registry of counter objects that call sites Inc(), which
invites a second counter beside the one collection_coverage reads. expfmt
has no registry — a family can only be built from a snapshot at scrape time, so
"one set of counters, two audiences" is what the API permits rather than what a
reviewer has to remember.

Labels from a closed set of nine names. role, version, source, path, reason, outcome, kind, state, subject. Pinned by a test, and by a second test
that walks the actual exposition of both roles. No value is a namespace, pod,
node, workload or image name — a cardinality budget and invariant 6 at once,
which matters because ADR 0069 §5's policy rule admits this port from any
source. A node's profiling state crosses the channel from the DaemonSet, so it
is mapped through ADR 0060 §2's enumeration with the rest bucketed to other.

kind is not enumerated here at all: Spool.write now takes the kind, refuses
one with no registry row, and per-kind counters are seeded from sink.Registry()
so every registered kind has a zero series. "No usage_window written in an
hour" is the alert, and an absent series cannot express it.

Source changes (no payload bytes, no goldens)

  • Kubelet poll counters split per path — ADR 0013 §3 always said the two were
    separate requests, then counted them together. Observation() returns the
    sums, so collection_coverage is byte-identical.
  • The spool's own numbers are new: written per kind, write failures, evictions
    by which bound caused them, last sweep's size. ADR 0042 gave the spool hard
    ceilings and no way to see them act. Exposed here; the payload block is a
    protocol change and waits for the transport that gives it a reader.
  • The node counts its report deliveries — a failure only ever reached its own
    log, in the customer's cluster.

Both roles

ADR 0054 §4 makes coverage aggregate-never-per-node; ADR 0067 says a node went
quiet without saying which. A per-node scrape answers the rest, and the agent
contributes no label for it — the node identity comes from the customer's own
service discovery.

Evidence

make test, make lint, make vulncheck, make chart-lint — green, and
re-run against a clean git archive HEAD extraction rather than the working
tree.

docs/security.md gains one §5 row (ceiling 1044 → 1045, argued in ADR 0070 §5:
a scrapeable endpoint is an obligation the document did not have) and §11 now
names both places the same counts appear.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KeZECJXTDHL6Nts95eKMJA

…070)

Adds `/metrics` to the health listener of both roles, exposing the counters the
agent already keeps. The endpoint reads the same accessors `collection_coverage`
is built from — no counter is added beside an existing one — so the SRE's alert
and the backend's report cannot give different answers.

- `internal/metrics`: a per-scrape builder over `prometheus/common/expfmt` and
  `client_model`, both already direct dependencies (the cAdvisor reader decodes
  this format). No new module, so nothing new for the reachability gate; and no
  registry of counter objects to increment, which is the shape a second counter
  appears in.
- Label names are a closed set of nine, pinned by a test that walks the real
  exposition of both roles. No value names a namespace, pod, node, workload or
  image. A node's profiling state is mapped through ADR 0060 §2, so a newer node
  cannot open a series in the controller's exposition.
- `kind` comes from `sink.Registry()`: `Spool.write` takes the kind, refuses one
  with no registry row, and seeds a zero counter per registered kind — "no
  usage_window in an hour" needs the series to exist.
- The kubelet poll counters split per path (ADR 0013 §3 always said they were
  separate requests); `Observation()` returns the sums, so payload bytes are
  unchanged.
- New: the spool's own numbers (written per kind, write failures, evictions by
  which bound, last sweep's size) and the node's report deliveries. ADR 0042 gave
  the spool ceilings and no way to see them act.

No chart change: same port 9090, same NetworkPolicy rule, no ServiceMonitor.
No payload byte, golden file, RBAC rule, capability or mount changed.

make test, make lint, make vulncheck, make chart-lint green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KeZECJXTDHL6Nts95eKMJA
@kfeofantov
kfeofantov merged commit 9c98b38 into main Sep 6, 2026
3 checks passed
@kfeofantov
kfeofantov deleted the feat/self-metrics branch September 6, 2026 10:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant