Skip to content

feat: kubelet usage poller with counter deltas and window snapshots - #14

Merged
kfeofantov merged 1 commit into
mainfrom
feat/kubelet-usage-poller
Aug 9, 2026
Merged

kfeofantov merged 1 commit into
mainfrom
feat/kubelet-usage-poller

Conversation

@kfeofantov

Copy link
Copy Markdown
Contributor

Second of three ADR 0006 slices: actual collection of CPU and memory usage, end-to-end from kubelet to logged rollup snapshots. Builds on the rollup core from #13; the cAdvisor throttling counters and PSI probing come in slice 3.

What's here

  • UsagePoller (internal/collector/usage.go): polls every node's kubelet through the API server proxy (get nodes/proxy) every 30 s — exactly one path in this slice, /stats/summary. CPU comes from the cumulative usageCoreNanoSeconds counter (exact regardless of cadence), memory from the working-set gauge.
  • Delta rules exactly as frozen in ADR 0006: deltas computed from timestamps inside the kubelet response, never scrape time; re-served snapshots (unchanged timestamp) discarded; counter running backwards = container restart → rebaseline, no sample; first observation counts from container start, so short-lived Jobs that survive one poll are fully attributed.
  • Attribution + filter-early: PodWatcher gains a UID → (namespace, workload) index containing only admitted pods. Excluded pods' samples are dropped at the source; unknown pods (informer lag) defer losslessly — the baseline isn't advanced, so the next attributed observation covers the full interval.
  • Delivery shape: once a minute the poller emits deep-copy snapshots of open windows with a monotonic sequence number (the backend's supersede ordering key) and final records for closed windows. Both are logged as one JSON record per line — the spool/sink slice (ADR 0003) will pick these up unchanged.
  • Runtime signal probing: the coverage line now reports usage_signals — which kubelet signals this cluster actually exposes (cpu, memory now; throttling, psi in slice 3).
  • security.md §3/§5 brought in line with implementation: usage collection goes controller → API server only; the agent opens no direct connection to kubelets.

Tests

  • Unit tests for every delta rule, lossless deferral, tracker sweep, flush sequencing, and signal reporting — driven through ingest() with hand-built summaries.
  • e2e (test/e2e/usage_test.go): full pipeline against kind with a CPU-burning deployment under a 300m limit; asserts accumulated core-nanoseconds, sane sampled rates, hour-aligned windows. Makefile e2e timeout bumped 10m → 15m.

New dependency: k8s.io/kubelet (stats API types only).

🤖 Generated with Claude Code

Implements the collection layer of ADR 0006 for CPU and memory:

- UsagePoller polls every node's kubelet through the API server proxy
  (get nodes/proxy, /stats/summary only in this slice) every 30 s and
  reduces samples into the rollup accumulator
- delta rules per ADR 0006: deltas from kubelet response timestamps,
  re-served snapshots discarded, counter resets rebaseline without a
  sample, first observation counts from container start
- PodWatcher gains a UID index for sample attribution; excluded pods are
  dropped before accumulation, unknown pods defer losslessly
- NodeWatcher tracks the live node name set as the polling target list
- open-window snapshots ship once a minute with a monotonic sequence
  number; closed windows emit final records; coverage now reports the
  active usage signal set
- security.md §3/§5: usage collection goes through the API server proxy —
  the agent opens no direct connection to kubelets

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@kfeofantov
kfeofantov merged commit 3c4db5a into main Aug 9, 2026
3 checks passed
@kfeofantov
kfeofantov deleted the feat/kubelet-usage-poller branch August 10, 2026 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant