Skip to content

feat: a kind declares how often it is sent (ADR 0073) - #111

Merged
kfeofantov merged 1 commit into
mainfrom
feat/per-kind-send-cadence
Sep 7, 2026
Merged

kfeofantov merged 1 commit into
mainfrom
feat/per-kind-send-cadence

Conversation

@kfeofantov

Copy link
Copy Markdown
Contributor

Every superseding payload rode one one-minute ticker. node_metadata is ~1.5 KB
per node — ~433 MB a day on a 200-node cluster — and workload_metadata ~1.3 KB
per container record, ~3.7 GB a day at two thousand of them. Both are keyed one
row per cluster and superseded in place, so the entire cost is network and write
amplification.

What changes

Cadence becomes a registry column beside the natural key, the delivery
discipline and the provenance: a floor, a ceiling, and a trigger.

Cadence Kinds
every minute, unconditional collection_coverage, usage_snapshot, the four journals, process_counters
changed; floor 1 min, ceiling 15 min node_metadata
changed; floor 5 min, ceiling 15 min workload_metadata, workload_revisions, workload_policy, cluster_policy, restart_counters, go_inventory, process_peaks, listening_ports
event closed windows, oom_kill, go_build, both profile kinds

The ceiling is the load-bearing half. A superseding kind carries no ordering
field (ADR 0027), so a change the predicate missed would otherwise stand as the
answer forever with nothing able to notice.

Four rows were decided against the obvious value:

  • process_counters is never held back. Alone among the superseding
    snapshots its records are differences, not readings (ADR 0062): a skipped
    reading is still true, a skipped difference is an interval nothing else
    carries.
  • collection_coverage is never held back. Its freshness is the liveness
    signal (ADR 0054 §5).
  • node_metadata keeps the one-minute floor, because it earns it — a node's
    conditions carry their last transition, not the kubelet's heartbeat
    (internal/model/node.go), so the payload genuinely holds still between real
    changes. Confirmed rather than assumed.
  • workload_metadata takes the long floor rather than a projection. See
    below.

The comparison is the payload's own bytes with captured_at elided, and
nothing else.
A projection past the noisy fields is a set of fields whose
changes go unseen for a ceiling, and one that has to be maintained by hand every
time a payload grows — a new field added and forgotten in it is invisible to
every test. Where a payload is too noisy for a one-minute predicate the answer is
a longer floor, which cannot be wrong that way.

Decided before the spool write, not before transmission. The spool is the
queue: its filenames are the natural keys and deletion is acknowledgment
(ADR 0003), so a predicate on the far side would leave the transmit path either
deleting files it never sent or ageing them against the spool's bounds. For a
one-per-cluster key the spool holds one file replaced in place, so nothing local
is coarsened except that file's staleness.

Kept honest in both directions. The gate reads the registry row, so there is
no second copy to drift. internal/sink/cadence_test.go exercises floor, ceiling
and predicate per kind over the writers that ship, and fails if a row it does not
cover appears; cmd/agent/cadence_test.go binds the fixed cadences to the tickers
the sink never sees. Suppression is counted per kind and exposed at /metrics.

Contract and docs

docs/backend-requirements.md §4 gains the table and three obligations: a
superseding payload is current until the next delivery of its kind (not only at
its captured_at), silence past a ceiling is a fact about the agent, and a
fleet's effective cadence is its oldest installed agent. The "declared limits"
bullet is corrected — its stated purpose, "so the agent can chunk
deterministically"
, does not describe this agent: a superseding payload is
indivisible.

On folding in size and rate limits: one ADR covers the class — all three are
constants the agent must know without being told — but only cadence gets numbers.
ADR 0039 forbids stating an unmeasured limit and there is no transmit path to
measure.

Amends ADR 0012, and the shared capture instant of ADR 0030 §7 and ADR 0034 §7:
what makes that join sound is no longer the shared number but the reason the
older payload was not rewritten.

Verification

make test   # ok — race, plus `go vet -tags e2e ./...`
make lint   # 0 issues
git archive HEAD | tar -x -C … && go build ./... && go test ./...   # ok

e2e was not run (no cluster here). Eight polls across five files and five
suite timeouts were raised by 5–6 minutes for the new floor — reasoned, not
measured. Shortening the floor for a test installation was the obvious escape
and is not available: a cadence an installation can set is a cadence the backend
cannot rely on.

Left for later, deliberately

workload_metadata still sends every five minutes on a cluster in continuous
rollout. The proper fix is moving pod.phases, pod.nodes and the replica
counts into a kind of their own — a payload change, and its own decision. Same
for splitting go_inventory's coverage block, whose per-node heartbeat is why
its predicate almost never fires.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KeZECJXTDHL6Nts95eKMJA

Every superseding payload rode one one-minute ticker. node_metadata is
~433 MB a day on a 200-node cluster and workload_metadata ~3.7 GB, for
state that changes rarely; both supersede in place, so the whole cost is
network and write amplification.

Cadence becomes a registry column beside the natural key and the delivery
discipline: a floor, a ceiling, and a trigger. The ceiling is what makes
change-triggered sending safe — a superseding kind has no ordering field
(ADR 0027), so a missed change would otherwise stand forever.

The comparison is the payload's own bytes with captured_at elided and
nothing else: a projection is a set of fields whose changes go unseen for
a ceiling and must be maintained by hand. Noisy kinds take a longer floor
instead. process_counters and collection_coverage are never held back.

Decided before the spool write, not before transmission: the spool is the
queue, its filenames are the natural keys and deletion is acknowledgment
(ADR 0003).

Amends ADR 0012, and the shared capture instant of ADR 0030 §7 and 0034 §7.
docs/backend-requirements.md §4 gains the table and the obligations that
come with it, and its "declared limits" bullet is corrected: the agent
never splits a superseding payload, so the limit is not for chunking.

make test: ok (race, plus go vet -tags e2e)
make lint: 0 issues
e2e not run: eight polls and five suite timeouts were raised by 5-6 min
for the new floor, reasoned rather than measured.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KeZECJXTDHL6Nts95eKMJA
@kfeofantov
kfeofantov merged commit 9062ebb into main Sep 7, 2026
3 checks passed
@kfeofantov
kfeofantov deleted the feat/per-kind-send-cadence branch September 7, 2026 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant