Skip to content

feat: liveness is the process, readiness is the caches (ADR 0069) - #107

Merged
kfeofantov merged 2 commits into
mainfrom
feat/health-probes
Sep 5, 2026
Merged

kfeofantov merged 2 commits into
mainfrom
feat/health-probes

Conversation

@kfeofantov

Copy link
Copy Markdown
Contributor

Neither role had a probe of any kind — no liveness, no readiness, no startup, on neither workload. A wedged controller was never restarted, and kubectl rollout status returned success while the 18 informers were still listing.

What ships

  • One always-on HTTP listener per role, port 9090, /livez and /readyz. Read-only: GET and HEAD only, no body read, no query parsed, and no reply names an object in the cluster. Separate from 8080, which exists only where a DaemonSet does and carries the node reports. On no Service.
  • Liveness is each loop's own heartbeat — the controller's periodic flush, the node's scan pass — stamped at the top of a pass so a wedged pass goes stale. It touches no cache and opens no socket: an API server outage must not restart the agent on every node at once, and a controller outage must not restart the DaemonSet.
  • Readiness is the caches that gate collection plus an open spool (controller), and the first completed scan pass (node). The nine policy caches are deliberately excluded — ADR 0033 lets a missing permission degrade one payload, and gating on them would turn that into a pod that is never Ready.
  • A startup probe covers the initial sync, with a budget above the five minutes a gating cache may fail before the agent stops itself (ADR 0035), so a refused LIST produces the agent's own diagnostic exit rather than an anonymous probe restart.
  • The NetworkPolicy gains a second ingress rule on the health port with no source: the caller is the kubelet, arriving from the node's address, which no selector in a template can name.
  • Address wiring: health.listenAddress in the controller's config file, -health-address on the node — its file is rendered only by the ebpf profile, and probes are needed in inventory too (ADR 0025).

Why liveness ignores the apiserver, why readiness is not all 18 caches, and why the node's readiness is not the scope query: ADR 0069.

Evidence

make test && make lint && make chart-lint && make vulncheck
→ all pass; 0 lint issues; 3 charts linted, 0 failed; 0 vulnerabilities

Mutation-checked — each of these fails a named test: deleting a probe from either workload, deleting the heartbeat stamp from either loop, deleting the node's readiness latch, deleting the readiness cache gate, removing the policy rule's port restriction, omitting the listen address from either role.

docs/security.md gains one §5 row and its ceiling moves by exactly that row: a port the agent opens is an obligation the document did not have before (ADR 0044).

Not run here

e2e (kind) has not been run for this branch — it needs make cluster-up e2e cluster-down locally, and this PR touches cluster-facing templates.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KeZECJXTDHL6Nts95eKMJA

kfeofantov and others added 2 commits September 4, 2026 16:57
Neither role had a probe of any kind. A wedged controller was never
restarted, and `kubectl rollout status` returned success while the 18
informers were still listing.

Both roles now serve an always-on HTTP listener on port 9090 answering
/livez and /readyz — read-only, GET and HEAD only, and no reply names an
object in the cluster. Liveness is each loop's own heartbeat and depends
on nothing outside the process: an API server outage must not restart the
DaemonSet on every node at once. Readiness is the caches that gate
collection plus an open spool for the controller, and the first completed
scan pass for the node. A startup probe covers the initial sync, with a
budget above the five minutes a gating cache may fail before the agent
stops itself (ADR 0035).

The NetworkPolicy gains a second ingress rule for the health port with no
source: the caller is the kubelet, arriving from the node's address.

The security.md ceiling moves by the one row a newly opened port earns.

  make test && make lint && make chart-lint && make vulncheck
  → all pass; 0 lint issues; 3 charts linted, 0 failed; 0 vulnerabilities

Mutation-checked: deleting a probe, the beat, the readiness latch, the
cache gate, or the policy's port restriction each fails a named test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KeZECJXTDHL6Nts95eKMJA
The compressed flush interval was 20ms, so the liveness deadline under
test was 60ms — shorter than one flush pass takes under -race on CI. The
agent reported itself stale correctly; the test's ratio was wrong. Both
compressed periods now keep the ratio the real ones have.

  go test -race -count 5 -run StaysAlivePastItsDeadline ./cmd/agent → ok
  deleting either beat still fails its test

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KeZECJXTDHL6Nts95eKMJA
@kfeofantov
kfeofantov merged commit d506f6f into main Sep 5, 2026
3 checks passed
@kfeofantov
kfeofantov deleted the feat/health-probes branch September 5, 2026 08:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant