Skip to content

feat: the agent yields first, and two objects it does not ship (ADR 0068) - #106

Merged
kfeofantov merged 1 commit into
mainfrom
feat/the-agent-yields-first
Sep 4, 2026
Merged

kfeofantov merged 1 commit into
mainfrom
feat/the-agent-yields-first

Conversation

@kfeofantov

Copy link
Copy Markdown
Contributor

Four questions a platform team asks at review were answered by omission:
priorityClassName, PodDisruptionBudget, topologySpreadConstraints and
terminationGracePeriodSeconds appeared nowhere in the chart. Two stay absent
with the reason recorded; two become decisions. ADR 0068 carries the rationale.

The two refusals

  • No PodDisruptionBudget. On one replica minAvailable: 1 forbids the only
    eviction there is, so the agent becomes the reason a customer cannot drain
    their own node; maxUnavailable: 1 forbids nothing.
  • No topologySpreadConstraints. One replica has nothing to spread and a
    DaemonSet is spread by construction.

internal/chartrender/guardrail_test.go fails if either reappears, so reversing
one means arguing with a test rather than adding a file.

The two additions

  • priorityClassName on both roles, empty by default. The chart creates no
    PriorityClass — a priority value only means something against the classes a
    cluster already has — and values.yaml carries the manifest of a
    negative-valued class, which is what actually gets the agent underneath
    workloads that sit at zero. A system- name outside kube-system is refused
    at render time: admission rejects such pods rather than scheduling them lower,
    so the failure would be a DaemonSet creating no pods and not saying why.
  • terminationGracePeriodSeconds: 30 on the controller, 10 on the node.

The controller's number comes from the shutdown path. SIGTERM cancels the root
context, wg.Wait() waits for the watchers and the periodic flush goroutine,
then the shutdown pass writes the journals. The one hard bound in it is the
receiver's own 10s Shutdown budget (internal/nodeintake/server.go:76);
watchers stop with their context, kubelet reads are cancelled with it and
bounded at 10s regardless (ADR 0045), and the pass is local writes with no
fsync. Ceiling: about ten seconds.

Below 15 the chart refuses, because what a shorter value costs is invisible:
SIGKILL lands before the pass and up to one coverage interval — 60s
(cmd/agent/main.go:148) — of inventory, container_restarts,
pod_disruptions, job_runs, node_lifecycle and collection_coverage is
gone with no log line, counter or spool entry to notice it by. Usage rollups
survive (cumulative counters) and workload_metadata is deliberately not on
that pass (ADR 0027).

The node gets 10 because it runs no pass and holds nothing that survives a
restart; its exit is bounded by one uninterruptible scan pass, so the number
bounds only how long this agent can hold up somebody else's node drain.

GOMEMLIMIT

80% of the container limit, passed in by the downward API and applied by the
agent. The Go runtime reads the cgroup CPU limit and sets GOMAXPROCS; there is
no memory equivalent, so an over-budget controller is OOM-killed — SIGKILL, and
the 60s above with it.

The downward API cannot express a fraction: resourceFieldRef divides, rounds
up and emits a bare integer, so divisor: 1Mi on a 256Mi limit yields 256,
which GOMEMLIMIT reads as 256 bytes. Helm cannot compute it either — no
quantity parser, and the input is whatever the operator wrote. So the chart
renders MEMORY_LIMIT_BYTES and cmd/agent/memlimit.go calls
runtime/debug.SetMemoryLimit.

The fraction is the decision, and the reserve has to hold what GOMEMLIMIT cannot
see. Measured on the linux/amd64 build of this binary:

.text        27.4 MiB
.rodata       9.9 MiB
.gopclntab   24.1 MiB   = 61.5 MiB of demand-paged, file-backed mapping

At the chart's 256Mi default, 20% is 51 MiB; 90% would leave 26 MiB, thinner
than this binary's resident text is likely to be. The variable is rendered only
where a memory limit is — without one the downward API substitutes the node's
allocatable memory, and the agent would size itself to the node.

GOMAXPROCS is deliberately not set: go 1.26 already reads the CPU limit.

Tests

The chart tests named closed lists for ClusterRole resources, nonResourceURLs,
capabilities and workload kinds — but not for container environment or pod-spec
fields, so an env var could be added with the suite green. This PR adds that
list (TestTheContainerEnvironmentNamesNothingBeyondThisList, plus a bar on
envFrom) alongside the thing it guards.

Five mutations checked to fail: a rendered PodDisruptionBudget, a spread
constraint, a stray env var, a removed grace period, a resourceFieldRef naming
the wrong container.

docs/security.md

Not touched, and the argument rather than a guess: priorityClassName and
terminationGracePeriodSeconds are already-collected fields of the placement
block (ADR 0031, §8), so setting them on the agent's own pods adds no field and
no source. No grant changes — priorityclasses and poddisruptionbudgets are
read already — and no PriorityClass object is created for the agent to read. No
payload byte moves; internal/sink/registry.go is untouched. The env var is
named MEMORY_LIMIT_BYTES, not GOMEMLIMIT, so it is not among the five names
ADR 0047 keeps.

Evidence

make test        ok, all packages
make lint        0 issues.
make chart-lint  1 chart(s) linted, 0 chart(s) failed  (x3 profiles)

Also built and tested from git archive HEAD into a clean tree, so this is what
was committed rather than what is in a working directory.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PuDDpw5fVV4UKgpzdQoqbb

…068)

Four questions a platform team asks at review were answered by omission.
Two stay absent with a reason recorded, two become decisions:

- no PodDisruptionBudget: on one replica minAvailable:1 blocks node drain
  and maxUnavailable:1 forbids nothing;
- no topologySpreadConstraints: nothing to spread across one replica or a
  DaemonSet;
- priorityClassName on both roles, empty by default, no PriorityClass
  created — a value only means something against a cluster's own classes.
  A system- name outside kube-system is refused at render time, since
  admission would reject the pods rather than lower their priority;
- terminationGracePeriodSeconds 30 on the controller, 10 on the node. The
  controller's shutdown path has a ~10s ceiling (the receiver's own
  Shutdown budget); below 15 the chart refuses, because a SIGKILL before
  the flush pass drops up to one coverage interval of the journals with
  nothing to notice it by. The node runs no pass and has no loss, so its
  number bounds only how long the agent delays a drain.

GOMEMLIMIT is set to 80% of the container limit, passed in by the downward
API because the Go runtime reads the CPU limit and not the memory one. The
reserve holds what GOMEMLIMIT cannot see — measured on this binary: .text
27.4 MiB + .rodata 9.9 MiB + .gopclntab 24.1 MiB of demand-paged mapping.
The variable is rendered only where a memory limit is; without one the
downward API substitutes the node's allocatable memory.

Render tests pin all of it, including a closed list of permitted container
environment variables — that list did not exist, so an env var could be
added with the suite green. Five mutations checked to fail: a rendered PDB,
a spread constraint, a stray env var, a removed grace period, a
resourceFieldRef naming the wrong container.

docs/security.md unchanged: priorityClassName and
terminationGracePeriodSeconds are already-collected placement fields
(ADR 0031), no payload, grant or registry row moves.

make test, make lint, make chart-lint green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PuDDpw5fVV4UKgpzdQoqbb
@kfeofantov
kfeofantov merged commit d51023c into main Sep 4, 2026
3 checks passed
@kfeofantov
kfeofantov deleted the feat/the-agent-yields-first branch September 4, 2026 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant