Repository navigation
feat: a kind declares how often it is sent (ADR 0073) - #111
Merged
Merged
Conversation
Every superseding payload rode one one-minute ticker. node_metadata is ~433 MB a day on a 200-node cluster and workload_metadata ~3.7 GB, for state that changes rarely; both supersede in place, so the whole cost is network and write amplification. Cadence becomes a registry column beside the natural key and the delivery discipline: a floor, a ceiling, and a trigger. The ceiling is what makes change-triggered sending safe — a superseding kind has no ordering field (ADR 0027), so a missed change would otherwise stand forever. The comparison is the payload's own bytes with captured_at elided and nothing else: a projection is a set of fields whose changes go unseen for a ceiling and must be maintained by hand. Noisy kinds take a longer floor instead. process_counters and collection_coverage are never held back. Decided before the spool write, not before transmission: the spool is the queue, its filenames are the natural keys and deletion is acknowledgment (ADR 0003). Amends ADR 0012, and the shared capture instant of ADR 0030 §7 and 0034 §7. docs/backend-requirements.md §4 gains the table and the obligations that come with it, and its "declared limits" bullet is corrected: the agent never splits a superseding payload, so the limit is not for chunking. make test: ok (race, plus go vet -tags e2e) make lint: 0 issues e2e not run: eight polls and five suite timeouts were raised by 5-6 min for the new floor, reasoned rather than measured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KeZECJXTDHL6Nts95eKMJA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every superseding payload rode one one-minute ticker.
node_metadatais ~1.5 KBper node — ~433 MB a day on a 200-node cluster — and
workload_metadata~1.3 KBper container record, ~3.7 GB a day at two thousand of them. Both are keyed one
row per cluster and superseded in place, so the entire cost is network and write
amplification.
What changes
Cadence becomes a registry column beside the natural key, the delivery
discipline and the provenance: a floor, a ceiling, and a trigger.
collection_coverage,usage_snapshot, the four journals,process_countersnode_metadataworkload_metadata,workload_revisions,workload_policy,cluster_policy,restart_counters,go_inventory,process_peaks,listening_portsoom_kill,go_build, both profile kindsThe ceiling is the load-bearing half. A superseding kind carries no ordering
field (ADR 0027), so a change the predicate missed would otherwise stand as the
answer forever with nothing able to notice.
Four rows were decided against the obvious value:
process_countersis never held back. Alone among the supersedingsnapshots its records are differences, not readings (ADR 0062): a skipped
reading is still true, a skipped difference is an interval nothing else
carries.
collection_coverageis never held back. Its freshness is the livenesssignal (ADR 0054 §5).
node_metadatakeeps the one-minute floor, because it earns it — a node'sconditions carry their last transition, not the kubelet's heartbeat
(
internal/model/node.go), so the payload genuinely holds still between realchanges. Confirmed rather than assumed.
workload_metadatatakes the long floor rather than a projection. Seebelow.
The comparison is the payload's own bytes with
captured_atelided, andnothing else. A projection past the noisy fields is a set of fields whose
changes go unseen for a ceiling, and one that has to be maintained by hand every
time a payload grows — a new field added and forgotten in it is invisible to
every test. Where a payload is too noisy for a one-minute predicate the answer is
a longer floor, which cannot be wrong that way.
Decided before the spool write, not before transmission. The spool is the
queue: its filenames are the natural keys and deletion is acknowledgment
(ADR 0003), so a predicate on the far side would leave the transmit path either
deleting files it never sent or ageing them against the spool's bounds. For a
one-per-cluster key the spool holds one file replaced in place, so nothing local
is coarsened except that file's staleness.
Kept honest in both directions. The gate reads the registry row, so there is
no second copy to drift.
internal/sink/cadence_test.goexercises floor, ceilingand predicate per kind over the writers that ship, and fails if a row it does not
cover appears;
cmd/agent/cadence_test.gobinds the fixed cadences to the tickersthe sink never sees. Suppression is counted per kind and exposed at
/metrics.Contract and docs
docs/backend-requirements.md§4 gains the table and three obligations: asuperseding payload is current until the next delivery of its kind (not only at
its
captured_at), silence past a ceiling is a fact about the agent, and afleet's effective cadence is its oldest installed agent. The "declared limits"
bullet is corrected — its stated purpose, "so the agent can chunk
deterministically", does not describe this agent: a superseding payload is
indivisible.
On folding in size and rate limits: one ADR covers the class — all three are
constants the agent must know without being told — but only cadence gets numbers.
ADR 0039 forbids stating an unmeasured limit and there is no transmit path to
measure.
Amends ADR 0012, and the shared capture instant of ADR 0030 §7 and ADR 0034 §7:
what makes that join sound is no longer the shared number but the reason the
older payload was not rewritten.
Verification
e2e was not run (no cluster here). Eight polls across five files and five
suite timeouts were raised by 5–6 minutes for the new floor — reasoned, not
measured. Shortening the floor for a test installation was the obvious escape
and is not available: a cadence an installation can set is a cadence the backend
cannot rely on.
Left for later, deliberately
workload_metadatastill sends every five minutes on a cluster in continuousrollout. The proper fix is moving
pod.phases,pod.nodesand the replicacounts into a kind of their own — a payload change, and its own decision. Same
for splitting
go_inventory's coverage block, whose per-node heartbeat is whyits predicate almost never fires.
🤖 Generated with Claude Code
https://claude.ai/code/session_01KeZECJXTDHL6Nts95eKMJA