backend-requirements.md

Backend Requirements — the Agent's Perspective

What runtime-agent needs from the backend in order to operate. This is a contract document, not a design: it states behavior and guarantees, not endpoints, schemas, or implementation. Anything the backend does beyond this contract (analysis, cost model, findings, reports, read API, MCP) is a product concern and out of scope here — with one exception: nothing the backend does may violate the principles in §1.

Requirement levels follow RFC 2119 (MUST / SHOULD / MUST NOT).

---

1. Principles the backend MUST preserve

1. One-way flow. The agent initiates every connection. The backend MUST NOT send commands, tasks, or configuration to the agent. Responses to agent requests carry only protocol outcomes (acknowledgments, errors, issued certificates). If a response contains anything else, the agent ignores it by design.
2. Agent autonomy. The agent must remain fully functional for collection with the backend unreachable. Nothing in the contract may require a round-trip to the backend for the agent to decide what to collect.
3. The agent is replaceable, the data is not. The backend is the long-term source of truth for everything the agent ships; the agent keeps only a working window. Loss of an agent's local state MUST be recoverable through this contract alone (re-enroll + continued ingest), with no data history lost on the backend side.

---

2. Identity and enrollment

The full lifecycle is described in security.md §6; the backend obligations:

• Cluster records exist before agents. The backend MUST allow creating a cluster record (name, labels, org) ahead of any agent contact, and MUST treat an enrollment against a non-existent record as an error — never auto-create clusters from enrollment.
• Enrollment. The backend MUST accept a certificate signing request authenticated by a per-cluster enrollment token and return a client certificate carrying the cluster identity. Tokens MUST be short-lived and MUST remain reusable within their TTL (agents restart during rollout).
• Fingerprint check. Enrollment includes a cluster fingerprint (UID of kube-system). The backend MUST reject an enrollment whose fingerprint contradicts the one already bound to the cluster record, and MUST surface the conflict to the operator rather than silently mixing data from two clusters.
• Renewal. The backend MUST issue a renewed certificate over the existing mTLS session before expiry, without any operator involvement.
• Re-enroll. For an existing cluster record, the operator MUST be able to issue a fresh enrollment token; a successful re-enroll MUST revoke the previous certificate and MUST keep the cluster's data history continuous (same cluster identity).
• Revocation. The backend MUST be able to revoke any cluster certificate at any time, effective on the next connection attempt.

3. Transport

• Single fixed domain, mTLS, certificates issued by the backend's private CA. The agent pins that CA; the backend MUST NOT rely on public PKI for its server identity and MUST keep the CA stable across infrastructure changes (rotations follow a published overlap window so pinned agents never break).
• The domain and its addressing MUST be stable enough for enterprise egress allowlists. Changing the domain is a breaking change to every installed agent and requires a migration plan.
• The current stage is plain HTTP with no client identity and none of the above is built: the agent posts to one address its operator configured, and the backend attributes everything that arrives to that one configured source. A backend MUST NOT read a sender identity out of a payload — there is none to read, and adding one would have to be removed again when mTLS makes identity a property of the connection (ADR 0075).
• Wire format: protobuf payloads over plain HTTPS POST (Connect-style RPC is acceptable; bare gRPC is not part of the contract). The data plane MUST work over HTTP/1.1 through enterprise proxies — no dependency on HTTP/2-only features such as gRPC trailers. Rationale: the traffic is batch upload, not streaming, and corporate egress infrastructure is the environment this system lives in.
• Enrollment endpoints MUST be reachable over plain HTTPS (server-auth TLS only) — by definition they are used before a client certificate exists.
• Every request body is gzip-encoded and says so. The agent sends Content-Type: application/json with Content-Encoding: gzip, on every payload of every kind, and the backend MUST decode it. This is not negotiated and there is no uncompressed mode: the agent cannot ask what the backend accepts, because nothing the backend sends may change what the agent does (ADR 0001), so the encoding is a property of the protocol version in §6 exactly as the cadences in §4 are (ADR 0076).
• Every request names the agent that made it. The agent sends User-Agent: runtime-agent/<version>, the same value on every request of a process (ADR 0083). The version is at most 64 characters of A-Za-z0-9.-+_. A build whose version falls outside that sends runtime-agent alone rather than a malformed one, so a backend MUST NOT require a version: an agent built before this header existed sends none, and an intermediary may strip one. The same string travels as collection_coverage.agent.version, on a different schedule.
• The payload is what is encoded, not what is sent. Everything else in this document — sizes, natural keys, supersession, the size ceiling in §5 — describes the decoded bytes. A backend MUST apply its own size limits after decoding, and MUST NOT infer a payload's size from the request's.

4. Ingest

What arrives

The agent ships, per cluster:

• mergeable usage rollups per workload and container over wall-clock-aligned windows (initially hourly; the window length is part of the record key): fixed-boundary histograms for CPU and memory working set, plus exact window totals — CPU core-seconds, CFS throttling periods, and PSI stall time where the cluster exposes it (ADR 0006)
• workload and node metadata: names, namespaces, owner chains, image digests, Go version, module path, declared requests/limits, declared container ports, and — in a per-record pod block — QoS class and replica counts per pod phase and per node; plus instance types, capacity type, allocatable, and node zone/region. These arrive as superseding snapshots of current state, not as a change log (ADR 0012)
• OOM and restart events, plus the restart counter as it stands, which carries the restarts that predate the agent and belong to no window (ADR 0034)
• symbolized, allow-list-filtered profiles, keyed to image digest
• a coverage report: aggregate counts per filter type (discovered / collected / excluded by namespace filter / by annotation / …). Identities of excluded objects — names, namespaces, deny-listed paths — are never transmitted; the backend MUST NOT expect or request them
• a fingerprint (hash + last-change timestamp) of the effective filter configuration, carrying no configuration content. Every upload is attributable to the configuration that produced it, so "the data changed" can be told from "the filters changed"
• agent self-info: version, install profile, and effective RBAC summary, so every report can state what its findings do and do not rest on. The active usage signal set (which kubelet signals — e.g. PSI — this cluster exposes) travels in each measured payload's observation block rather than here, so it is attached to the data it qualifies (ADR 0013)

All payloads are denominated in resource units (core-hours, GiB-hours, counts, ratios). The protocol has no monetary fields: the agent knows nothing about prices or invoices, and pricing is applied backend-side at render time from a versioned pricing snapshot.

Every payload declares a kind, and carries its provenance in a source field: structural (read from a spec or status), measured (polled from an instrument), journal (derived from object history), or sampled (statistical). The backend MUST NOT merge payloads of different provenance under one natural key: a declared value and a measured one are different kinds of claim, and collapsing them yields a confident wrong answer with no failing test to show for it. The registry of kinds, their natural keys, and their delivery discipline is ADR 0012.

Records are container-scoped; a differently scoped fact is nested (ADR 0014). Usage rollups and workload metadata are keyed per container, because requests, limits and CFS quota are declared and enforced per container. A pod with an injected sidecar therefore appears as several records sharing a workload and differing in container. Summing their totals gives the pod or workload total — that is the intended use.

pod.phases is placement and lifecycle, never health. The counts are Kubernetes pod phases verbatim, and the backend MUST NOT read Running as "healthy": a pod in CrashLoopBackOff is Running, and so is one whose readiness probe has never passed. Readiness and restart counts are not in this payload. Succeeded and Failed pods are reported too — a CronJob's workload legitimately shows dozens consuming nothing — so pod.replicas should be read with its breakdown, not alone; which of them count is not the agent's call.

Measured payloads state what was observed, and the backend MUST use it (ADR 0013). Each usage payload carries an observation block — poll cadence, cumulative counts of kubelet requests attempted and failed, and the signal set this cluster exposes — and each record carries its own completeness:

• cpu.covered_nanoseconds — how much of the window this container was actually observed for. Against window_seconds it bounds what the record can support.
• cpu.throttling_samples, cpu.psi_samples, memory.psi_samples — how many observations carried each signal. A zero total with a zero sample count means "never observed", not "zero". A backend that reads the first as the second will report a throttling-free workload on the strength of a failed scrape.

The counters are cumulative since agent start; the backend takes differences between deliveries, exactly as it does for every other counter here.

container_restarts counts exactly and explains partially (ADR 0020). Each record covers one (namespace, pod, container) within one hour-aligned window, and its three restart fields are not interchangeable:

• restarts is exact. It comes from the restart counter's advance, so it counts restarts the agent never saw individually.
• reasons is a sample of that total. A container status carries only its most recent termination, so restarts that happened between two status updates are counted without a reason.
• reasons_unobserved is the difference. sum(reasons) + reasons_unobserved == restarts holds for every record, and a backend that renders the breakdown as if it were the whole must not: a container restarting faster than the informer delivers updates would appear to restart less than it does.

The reason keys are a fixed set (OOMKilled, Error, Completed, ContainerCannotRun, ContainerStatusUnknown, DeadlineExceeded, StartError, Evicted) plus other for anything the container runtime reports outside it. The backend MUST tolerate other and MUST NOT assume the set never grows.

The payload supersedes by (window start, window length): each flush rewrites the open window's payload with the window's running totals, and the last write before the window ends is final. The backend MUST upsert by that key and MUST NOT add successive deliveries together — they are cumulative within the window, not increments. Windows before the agent started watching a container are absent by design: a restart Kubernetes does not timestamp cannot be placed in one.

oom_kill and container_restarts describe the same events from two angles and do overlap: an OOM-killed container appears as an oom_kill event and within the window's reasons["OOMKilled"]. This is not double counting to be reconciled away — restarts is the container's true total, and the OOM events carry the per-kill detail (memory limit, exit code, restart count at the time) that a window aggregate cannot.

restart_counters is a reading, and MUST NOT be added to the windows (ADR 0034). It is a superseding snapshot of the kubelet's restart counter for every collected container that has ever restarted, keyed by the payload kind and carrying the same captured_at as the workload metadata of that flush.

• restarts is the container's total since its pod was created, including restarts that happened before the agent was installed. The windows of container_restarts are a subset of it. A backend that sums the two reports a cluster restarting roughly twice as often as it does.
• restarts_before_observation is how much of that total the agent never watched, and observed_since is when it started watching. Those restarts are in no window and never will be: Kubernetes does not timestamp them.
• restarts - restarts_before_observation is never negative. Both are read from one observation, so the subtraction this section prescribes is always defined, and no defensive clamp is needed (ADR 0043).
• observed_since and pod_created_at bound the interval those restarts are spread over, and nothing narrows it further. The backend MAY derive a rate over that interval; it MUST NOT place them at a point inside it.
• container_started_at is how long the current incarnation has been running, and is absent when the container is not running. A large restarts with a long-running current incarnation is a workload that recovered, which the total alone does not distinguish from one still failing.
• last_termination is the single restart Kubernetes does date, with the reason as the kubelet wrote it. Unlike the window's reasons keys this is a free value from a set the backend does not control, so it MUST be tolerated rather than matched against a fixed list.

A container whose counter is zero has no record: the payload is a snapshot, so absence means zero rather than unknown. An empty record list means no collected container has ever restarted, and MUST NOT be read as a failed collection.

unavailable_sources disqualifies the payload's absences, and says nothing about why (ADR 0033, ADR 0035). workload_policy and cluster_policy each declare the sources they could not read at that capture. The field names resource classes, is omitted when every source was read, and each payload declares only its own sources.

• The rule it exists for: in these payloads an absent record is a positive statement — "nothing constrains this workload". When a source is listed, that statement is void for anything that source would have supplied, including workloads that vanish from the payload entirely rather than appearing with a field missing. The backend MUST NOT treat the payload as a complete picture of what it declares unreadable.
• One list means one thing: not available at this capture. It does not say whether the permission was never granted, was taken away from the running agent, or was lost to a webhook or an unreachable API server, and the backend MUST NOT infer a cause from it.
• It is about that capture and no other. A source listed once and not again was readable again; a source that is readable again may still be listed for a capture or two, because the agent errs toward declaring a gap. Neither is a claim that the source is permanently gone, and the backend MUST NOT latch it.
• A missing grant the agent cannot work without produces no payload at all — the agent exits (ADR 0035). Absence of deliveries and a declared source are therefore different signals, and the backend SHOULD alert on the first through whatever it uses for a silent cluster.

pod_disruptions is what the cluster took away, not what failed (ADR 0021). One record per pod per hour-aligned window: preempted, evicted under node pressure, drained by a taint, or removed through the eviction API. A pod whose own container exited non-zero is not here — that is container_restarts and oom_kill. The backend MUST NOT read an absence of disruption records as a healthy cluster; it means no pod was taken away, which is the normal state.

• disrupted_at is Kubernetes' own timestamp, not the observation instant, so a disruption that happened before the agent started still lands in the window where it occurred. Records may therefore arrive for a window already delivered; the payload supersedes by (window start, window length), so the backend MUST upsert and MUST NOT append.
• Records are keyed by (namespace, pod) within a window and are idempotent: the same pod reported again after an agent restart rewrites its record rather than duplicating it.
• node is the node the pod was taken from. For a node-pressure eviction it names the node that was under pressure — joined against node_metadata it is what turns "a pod was evicted" into "this instance type ran out".
• reason is drawn from a fixed set (TerminationByKubelet, PreemptionByScheduler, DeletionByTaintManager, DeletionByPodGC, EvictionByEvictionAPI, Evicted) plus other. The backend MUST tolerate other and MUST NOT assume the set never grows.

node_lifecycle is the only payload that describes a node that no longer exists (ADR 0064). One record per (node, event) per hour-aligned window, event being joined or left. It supersedes by (window start, window length), so the backend MUST upsert and MUST NOT append.

• Each record repeats the node's instance type, capacity type, zone, capacity and devices. The backend MUST NOT resolve a left record against node_metadata — the node is gone from that snapshot by definition, which is why the size travels on the event.
• at carries two provenances and at_observed says which. Absent, at is the node object's creation timestamp; true, it is the instant the agent noticed the node was gone, because a deleted object carries no deletion time. The backend MUST NOT treat a departure's at as exact.
• Arrivals are reported only when the agent can prove them. A node already present when the agent started produces no joined record, so the backend MUST NOT compute fleet size by summing arrivals and departures — that is node_metadata's job. This payload is churn, not inventory.
• A window in which the fleet did not change produces no file. A node that cycles twice within one window produces one record per event type, not two.

node_metadata.nodes[].conditions is present in full or not at all (ADR 0064). All five allow-listed condition types are reported when the node reports them, healthy or not, so the backend MUST NOT read a missing condition as healthy — it means the node never reported one. devices is allow-listed by hardware vendor: an absent devices list means no recognized accelerator, not no extended resource. Both, and taints, are omitted when empty.

workload_metadata.workload and .pod are two different scopes, and neither is per record (ADR 0014). A pod's containers each produce a record repeating both blocks, so the backend MUST NOT sum either across one workload's records. workload.update_strategy is present only for workload_kind of Deployment, StatefulSet or DaemonSet — the kinds the agent reads. For every other kind it is absent, and the backend MUST NOT read that absence as "this workload has no update strategy": an Argo Rollout has one and the agent holds no RBAC for it. Absence says only that the agent did not read one.

workload_metadata.pod.unscheduled explains the replica shortfall, it does not restate it. The gap between replicas and the sum of nodes has always shown how many replicas are unplaced; unscheduled counts those by the scheduler's reason. The backend MUST NOT treat these as two independent counts — they describe the same pods, and summing or cross-checking them as separate signals will produce nonsense during the moments when a pod is between states.

The reasons are not interchangeable and MUST NOT be merged: Unschedulable means nothing in the cluster fits the pod and is a capacity fact; SchedulingGated means the pod is deliberately held by a scheduling gate and is not waiting on capacity; SchedulerError is the scheduler failing on the pod's own spec. Rendering a gated pod as a capacity shortage invents a shortage that does not exist.

Structural snapshots say when they were taken, and the Go inventory says how complete it is (ADR 0017). Every structural payload carries captured_at — the instant the snapshot was assembled, which is not a window and not the age of the facts in it. go_inventory adds a coverage block whose counters are cumulative from the since instant carried with them:

• nodes_reported — how many nodes currently in the cluster have delivered at least one scan. Read against the node count in node_metadata, a shortfall means the inventory is incomplete for reasons no record can express.
• facts_received / facts_joined / facts_unjoined / facts_undigested — what arrived and what could not be attributed. The backend MUST NOT read a missing workload as "not a Go workload" when these say otherwise.

Node-fed records carry asserted_at, and the backend MUST read it rather than captured_at (ADR 0067). In go_inventory, process_peaks, process_counters and listening_ports a node's contribution stands until that node pushes again, so a node that stops pushing keeps contributing what it last said into payloads whose captured_at is always current. asserted_at is when a node last stated the record — the oldest contributing report where several nodes merge — and collection_coverage.scan.oldest_asserted_at bounds that block's sums likewise. Two payloads a minute apart can therefore carry the same hour-old measurement. go_inventory.coverage.nodes names each node and its latest instant; it is the only place a node is named, and an instant that stops advancing is a node gone quiet, never "nothing changed there". collection_coverage.intake_rejections says whether the controller is the one turning that node away — too_large is a node holding more than the channel carries, and re-sending will not help — while no rejections at all means it went quiet for a reason the controller never saw. How stale is too stale is the backend's judgement: the agent ships the instant and draws no conclusion.

A go_inventory record exists only while its workload does (ADR 0018). The agent drops records whose workload container has left its filtered pod index — deleted, or opted out by annotation — so a record's disappearance between two snapshots means "no longer collected", not "not observed this time". The backend MUST treat a superseding snapshot as the complete current truth and MUST NOT carry forward records absent from it. A short-lived workload may appear in one snapshot and not the next; accumulating history across them is the backend's job, and captured_at is what dates them.

workload_revisions.revision is a Deployment's number and nobody else's (ADR 0049). A record of any other workload.kind carries none — that controller numbers revisions under a key the agent does not read — so the backend MUST NOT read the absence as missing data, and MUST order those revisions by created_at.

go_build is write-once and keyed by image digest. What a build is made of and how it was built — the Go version, the main module, the dependency module paths, and the allow-listed build settings — never changes, so the payload is immutable given its key and the agent sends it once per distinct build. The backend MUST upsert it by image digest and MUST NOT expect it on every flush or treat a redelivery (after an agent restart) as a change. It carries no captured_at for the same reason. Join it to workloads through go_inventory.records[].image_digest. Each entry of modules is an object with path and the version the toolchain recorded; one flagged replaced was redirected by a replace directive, so its version names what the build required rather than what is linked, and MUST NOT be presented as the running version (ADR 0048).

go_build.settings is a bounded allow-list, and its keys are optional (ADR 0019). The agent keeps only the settings enumerated in security.md §8 and discards the rest on the node, so the backend MUST NOT expect any particular key to be present and MUST NOT infer anything from an absent one. In particular vcs.revision, vcs.time and vcs.modified are recorded by the Go toolchain only when the build had a VCS working tree, which containerized builds frequently do not: their absence is the common case and MUST NOT be surfaced as missing data or as a coverage gap. The whole settings object is omitted when nothing was kept.

go_build.godebug is read together with go_build.go_version, never alone (ADR 0050, ADR 0081). It carries at most fourteen names, each 0 or 1, and only those the build deviates from its toolchain in; the object is omitted when it deviates in nothing. An absent name MUST NOT be read as 0: for the eleven whose default changed with a Go release it means the build takes its toolchain's own, and which that is follows from go_version; for tarinsecurepath, zipinsecurepath and execerrdot no release supplies a default, so absence means only that no //go:debug line set them. Which of the two values is the weakening one differs by name, and the backend MUST NOT assume 1 means weak — tls10server=1 and rsa1024min=0 each hold a security default at its pre-tightening value. A GODEBUG variable in workload_metadata.runtime_env overrides a compiled default at startup; it is a fact about a workload, this is a fact about a build, and the backend MUST NOT merge them into one value.

collection_coverage is how the backend knows an empty report is empty rather than broken (ADR 0054). It arrives on every flush, including one where nothing else did, and supersedes its predecessor. A stale or absent captured_at is therefore a statement about the agent, not about the cluster, and the backend SHOULD treat it as one: no other kind's silence means anything.

Its counters are cumulative from since, which is when that process started counting; a restart moves the base, and a drop across a change in since MUST NOT be read as a decline. agent.config carries the shape of the customer's configuration — entry counts and switches — and never a name from it, and never a hash of it; agent.config.since is when that configuration took effect, and it is what separates "the data changed" from "the filters changed". sources[] reports what the agent's reads actually did, so a source that is synced: false or failing: true bounds every finding resting on it — the backend MUST NOT present such a finding as complete. failing_since is when the run of failures that left a cache unfed began, and failing is the same fact as a boolean, derived from it, deprecated and retained for the window of section 6. An absent instant means not failing now, never "never failed": a source that failed and recovered between two flushes states nothing at all, so the backend MUST NOT read a clean payload as evidence that these reads were healthy throughout the period since since (ADR 0087). The scan block is a fleet aggregate over the latest pass of each reporting node and MUST NOT be summed across flushes.

The filter counters count decisions, not pods. A customer who annotates a running pod changes the answer about it, so a pod already counted in pods_observed then moves an exclusion counter, and removing the annotation counts it observed again. The backend MUST NOT read pods_observed and the exclusion counters as a partition of the pods seen, and MUST NOT recover a pod count by adding them.

filter.excluded_profiling_annotation counts pods the customer excluded from profiling alone (ADR 0071): they are still collected and still ship every other kind, so the backend MUST NOT treat them as absent, and MUST NOT present a workload with no profile as one the agent failed on.

No field of this payload names an object the customer excluded, and the backend MUST NOT ask for one.

agent_states is the same two states as time, and MUST NOT be added to them (ADR 0088). One record per (window, state, subject), superseding by (window start, window length): the backend MUST upsert and MUST NOT append. collection_coverage says which states hold now; this says how much of each hour they held and how many episodes that was. The relationship is container_restarts to restart_counters, and the same rule applies — one is a reading, the other a window.

• occupied_nanos and entries are independent and neither derives the other: one long episode and several short ones can share a duration. Summing entries across windows counts episodes, because an episode continuing from the previous window contributes occupancy here and no entry.
• observed_nanos bounds both, and the backend MUST NOT read occupancy without it. Zero occupancy over a full window is a healthy hour; zero over zero observation is an hour nobody watched. A restart's gap is in neither number, which is what makes the second one the honest denominator.
• A restart that crosses a window boundary may add one entry to a continuing episode. It is detectable: collection_coverage.since moves at the same time.
• subject is a resource class the agent watches or the agent's own shipping. No object in the customer's cluster appears here at any window length, and the backend MUST NOT ask for one.

agent_counters is the same counters as change, and MUST NOT be added to them (ADR 0089). One record per (window, counter), superseding by (window start, window length) like the kind above. counter is the dotted path that number has in the payload reporting it cumulatively — filter.excluded_pod_annotation, pprof_pull.refused — so the name is the join and nothing here restates what the number means.

• delta is the rise within the window. A rise within one pass of a boundary may land in the later window, so the backend MUST NOT align a delta to a sub-window instant.
• observed_nanos bounds it, and a window the agent was absent for part of has a smaller delta for reasons about the agent. What rose while the agent was down is counted by nobody: its counters restart at zero, so no reading spans the outage, and the backend MUST NOT treat a gap as quiet.
• A counter present in one window and absent from the next was not being counted, which is not the same as counting zero. The two are distinguished by the record existing at all.
• Deltas count decisions, not objects — the rule collection_coverage's filter counters already carry — so the backend MUST NOT recover an object count by summing them.

network_window says how much moved, never where it went (ADR 0053). The counters are read at the pod's own interfaces, so the payload carries no destination, no peer and no address, and the backend MUST NOT derive a cross-zone or cross-region transfer figure from it. What it supports is ranking: which workloads move enough traffic for a routing or placement finding to be worth acting on.

Records are keyed by workload, not by container, because every container in a pod shares one network namespace — so these MUST NOT be joined to a usage record's container, and a sidecar's traffic is not separable from its application's. A record whose host_network is true carries the node's interface counters, which describe the machine and everything else on it; the backend MUST NOT sum those with the rest, and MUST NOT present them as that workload's own traffic. samples separates a quiet workload from a cluster whose runtime does not report these counters at all: zero bytes with samples above zero is silence, zero bytes with samples at zero is ignorance. Only closed windows are sent, so the current hour is absent by design and MUST NOT be read as a drop to zero.

process_peaks.records[].peak_rss_bytes is a floor, and the evidence it gives is one-sided (ADR 0052). It is the largest VmHWM among a container's Go processes: the resident memory of one process, while the memory limit is enforced on the cgroup, which also holds page cache, every other process in the container, and kernel structures. A peak near the limit is therefore strong evidence — one process alone nearly fills the cgroup — and the backend MAY act on it. A peak far below the limit is weak evidence for lowering that limit, because what the measurement omits is exactly what such a recommendation would omit, and the backend MUST NOT present it as a headroom figure. processes says how many processes the number was taken over; a record is absent, never zero, when none were read. The record is keyed by image digest as well as container: a peak belongs to the build that reached it, and MUST NOT be carried across a rollout.

process_counters records are differences, and observed_nanos is what they rest on (ADR 0062). Each is the change since the reporting node's previous pass, summed over processes and nodes, so it MUST NOT be summed across flushes: a node's next report replaces its interval rather than extending it. observed_nanos sums with the deltas, so it is process-time and not a duration — two processes watched for a minute each contribute two minutes — and it is the denominator every rate MUST be formed against, never the flush interval or the wall clock. on_cpu_nanos beside run_delay_nanos is CPU starvation; the backend forms the ratio, because the agent ships numerators and denominators and never the division. A processes count lower than the same container's in process_peaks is expected for one interval after a pod starts: a process seen once has a footprint and no difference yet, and MUST NOT be read as processes having exited.

The footprint fields beside it are samples, and the field names say so (ADR 0061). peak_rss_bytes is a mark the kernel maintains between readings; every *_max beside it is the highest value the agent happened to observe, so the two MUST NOT be averaged or charted as one series. rss_anon_bytes_max is the program's own pages and rss_file_bytes_max the file-backed ones; both count against a memory limit, and a recommendation to lower one MUST rest on the first rather than on their sum. pss_bytes_max charges shared pages to their sharers in equal parts, so it — never resident memory — is what may be summed across replicas of one image. open_files_max against open_files_limit_min is the most descriptors any replica held against the lowest ceiling any replica was given; an absent ceiling means unlimited and MUST NOT be read as zero. Any of these fields may be absent on a kernel that does not expose its source, and absent never means zero.

A pprof_profile is the workload's own runtime sampling itself, and dropped bounds what may be concluded from it (ADR 0058). It is a ten-second capture at the rate that runtime chose, not a window the agent measured, so it MUST NOT be summed with ebpf_profile captures of the same workload as if they were one series. Frames outside the allow-list are the placeholder [filtered], and dropped.third_party_frames says how many became one; a flame graph rendered from this MUST show the placeholder as redacted rather than as a function. No mapping, no build ID, no sample label and no source path is present, so nothing here locates the binary or the request it ran under. capture_start/capture_end bound the capture and the payload accumulates per window like ebpf_profile.

A goroutine_counts sample is a gauge, and summing is the way to get it wrong (ADR 0080). Each record is one process — pod, container and build — within one window, and each sample is that process's goroutine count at the instant named beside it. Samples MUST NOT be added to one another, and records of successive windows MUST NOT be added either: sixty readings of four thousand goroutines are four thousand goroutines, not two hundred and forty thousand. What a report may draw from the kind is the shape of the series.

One replica of each workload is read, so a count MUST NOT be multiplied by a replica count or presented as a workload total. A record under a pod name not seen before is a different process: its low count is a new runtime starting, never the previous pod's count falling, and the two MUST NOT be joined into one series. A non-zero samples_dropped means the series is thinner than the cadence implies, and a gap in a record that carries one MUST NOT be read as a gap in the process. No stack and no function name travels with a count, so the kind says that a process is accumulating goroutines and never which code is.

A profile in which the runtime dominates is a finding, not a defect (ADR 0063). Both profile kinds ship whenever one frame of the workload's own code appears anywhere in them, so a capture whose heaviest functions are all runtime.gc* is one the agent judged worth sending: it says the workload spends that share of its CPU collecting. The backend MUST NOT discard such a profile as unsymbolized or as noise, and MUST NOT read the runtime's share as agent overhead — the profiler samples the workload, not itself.

collection_coverage.pprof_pull.refused is a fact about the customer's setup, not a failure. It counts attempts a process declined, which in Go means its own CPU profiler holds the single slot available. The backend MUST NOT present such a workload as unprofilable or as an agent error; the honest rendering is that the workload profiles itself. invalid counts profiles that arrived and were not shippable, and shipped counts what became a payload.

collection_coverage.ebpf says why a node is not profiling, and its counters have a different base from the block above them (ADR 0060). states maps a profiling state — supported, disabled, program_load_failed, capture_stopped, kernel_too_old, btf_absent, kernel_unknown — to how many nodes are in it: a refused kernel is not a broken agent and MUST NOT be presented as one, while capture_stopped is one and SHOULD be surfaced. Unlike scan, these counters are cumulative since each node started, not one pass, so they MUST NOT be summed across flushes and a fall across a node restart MUST NOT be read as a decline. windows_no_scope is the agent failing closed and windows_no_samples is a profiler that captured nothing on a node where something ran; profiles_invalid with a rising third_party_dropped means the symbol allow-list stopped matching the workload's own modules. The block is absent until a node reports; absent does not mean zero. No node is named.

collection_coverage.pprof counts endpoint targets, not workloads, and the block is absent when discovery is off (ADR 0057). A target is one (image digest, port), so a workload with several bound ports contributes several. confirmed served the pprof index; absent answered and did not, which for a build whose pprof_endpoint is true means that port is not the one serving it — the backend MUST NOT read absent as the package being missing. unreachable is a statement about the network, not the endpoint, and is retried, so it MUST NOT be presented as a workload that cannot be profiled. excluded_by_annotation is targets the customer removed with rebuildstack.co/profile, and unlike the counters elsewhere in this payload it is a current count rather than a total: it falls when an annotation is removed, so it MUST NOT be summed across flushes. An absent block means discovery is switched off or no node role is deployed; it does not mean zero.

listening_ports is what the processes bind, and go_build.pprof_endpoint is what the build contains; neither implies the other (ADR 0056). The flag says net/http/pprof is linked into the image — false is conclusive, true is not: a service may import the package without serving it. The ports are read from the processes' own sockets, so they are the ports actually bound and not the containerPorts of the spec; the two disagreeing is a finding, not an error. A port marked loopback is reachable only from inside the pod and MUST NOT be presented as an exposed one. No address is carried, so no endpoint may be addressed from this payload. Records are keyed by image digest as well as container and MUST NOT be carried across a rollout; a port absent from the latest report is a port no process still binds.

A port MAY carry pprof, holding confirmed, absent or unreachable — what the agent's one request to that port answered (ADR 0082). An absent pprof field means the port was not asked about and MUST NOT be read as absent: loopback ports are never asked, nor is any port of a workload the profiling opt-out excludes. Only confirmed on a port not marked loopback states that the pprof index answered where another pod could reach it. The value is the latest answer and carries no history.

workload_policy Service endpoint counts are per address family and MUST NOT be summed across them (ADR 0051). A dual-stack Service lists each pod once in an IPv4 slice and once in an IPv6 slice, so adding the families doubles every zone. zones counts ready endpoints by zone; unzoned counts those on a node carrying no zone label and MUST NOT be folded into a zone named "". hinted is read together with traffic_distribution (or topology_mode): a hinted of 0 where neither is set means nothing, and where one is set means the cluster declined to arrange the routing that was asked for — which is the finding. The endpoints array is absent when the agent has not seen the Service's slices, which is not the same as a Service with no ready endpoints, and MUST NOT be rendered as zero.

node_metadata.nodes[].architecture is what go_build.settings.GOARCH is compared against. A build's target architecture and microarchitecture level mean nothing on their own; the comparison is the finding. The field is the kubelet's own value (amd64, arm64), and is omitted if the kubelet does not report one.

Delivery semantics

• At-least-once. The agent retransmits anything not acknowledged. The backend MUST deduplicate by natural keys (cluster, workload, container, window start, window length; profile identity) so that resends are harmless. Ingest MUST be idempotent.
• Open-window snapshots supersede. While a rollup window is open, the agent periodically ships a snapshot of its accumulating record so the data is fresh during the window, not an hour later. The backend MUST upsert snapshots by the natural key above: a newer snapshot of a window replaces an older one, and the final closed-window record replaces every snapshot. Last write wins, and no payload carries an ordering field (ADR 0027): the agent's spool holds exactly one version of a natural key at a time, atomically replaced, so it never has two versions to offer. In exchange the agent MUST keep at most one request per natural key in flight — a resend of an unacknowledged payload re-reads the current version and is therefore the same or newer, never older. Snapshots have the same aggregate shape as any rollup; they do not relax §9 (no raw time series).
• A windowed journal payload says whether it is finished. container_restarts, pod_disruptions, node_lifecycle, job_runs and goroutine_counts have one payload shape open or closed and no separate closed-window kind, so each carries captured_at, the instant the agent wrote it. captured_at at or after window_start + window_seconds means the window is final; earlier means it is a slice of a window still open and a later delivery will replace it (ADR 0078). The backend MUST take finality from this field, MUST NOT infer it from its own clock, and MUST read an absent captured_at as "still open". It MUST NOT derive an order from it: supersession stays last write wins under the natural key (ADR 0027).
• Acknowledgment is a durability promise. The backend MUST NOT acknowledge data it can still lose. The agent trims its local buffers only after acknowledgment.
• Late and out-of-order data is normal. The agent buffers through outages — in memory and in its spool, which is ephemeral (ADR 0026) — and uploads backlog on reconnection. The backend MUST accept historical timestamps and batch catch-up uploads, and SHOULD tolerate moderate agent clock skew — data is keyed by agent-side timestamps.
• Gaps are normal too. The agent's spool is ephemeral by decision, not by configuration (ADR 0007, ADR 0026): an agent restarted during an outage may reconnect with nothing older than its fresh data. Routine loss is bounded by the snapshot cadence. The backend MUST NOT treat a gap as an error and MUST NOT expect agents to re-supply history they no longer hold.
• Declared limits. Payload size and request-rate limits MUST be explicit in the protocol rather than discovered by failing. Their purpose is not to let the agent chunk: a superseding payload is the complete state under its key and the agent never splits one (ADR 0027, ADR 0073). A backend that needs a bound MUST derive it from the cadence below and the per-record sizes, and MUST accept what a cluster of the size it admits will produce. Chunking, if it is ever required, is a protocol change on both sides. Until a limit is stated, the agent applies a conservative ceiling of its own and refuses to offer a payload past it, so a backend that accepts more than the agent will send does not see everything a cluster produces (ADR 0075).

Cadence

Every kind has a declared cadence, and it is a constant of the agent's protocol version (ADR 0073). The backend MUST NOT set, request, or influence it — §1 leaves no channel that could. The numbers travel with the published agent version the backend already records (§6), not in the payloads.

A cadence is a floor (the shortest gap between two deliveries of one natural key), a ceiling (the longest, unconditional), and what decides the deliveries between them.

Cadence	Kinds
every minute, unconditionally	collection_coverage, usage_snapshot, container_restarts, pod_disruptions, job_runs, node_lifecycle, process_counters, goroutine_counts
on change; floor 1 min, ceiling 15 min	node_metadata
on change; floor 5 min, ceiling 15 min	workload_metadata, workload_revisions, workload_policy, cluster_policy, restart_counters, go_inventory, process_peaks, listening_ports
when the event happens	usage_window and network_window (their window closed), oom_kill, go_build, ebpf_profile, pprof_profile

• A superseding payload is current until the next delivery of its kind, not only at its captured_at. A change-triggered kind is re-sent when its contents change and not otherwise, so an unchanged payload's absence states that the payload still holds. captured_at is when the state was last observed, never the last moment it was true. Two payloads that are joined to each other — workload_metadata, workload_revisions, workload_policy, restart_counters and node_metadata — MAY therefore carry different capture instants, and the backend MUST NOT treat the older of two as stale on that ground.
• Nothing longer than a kind's ceiling is silence about the cluster. Past it, the absence of a delivery is a fact about the agent — the same reading collection_coverage supports every minute — and the backend SHOULD alert on it through whatever it uses for a silent cluster.
• The effective cadence of a fleet is its oldest installed agent. Agents stay installed for a long time (§6) and cannot be told to change, so the backend MUST tolerate every cadence any supported agent version declares, and MUST NOT assume the newest table applies to a cluster it has not seen the version of.

5. Error semantics

The agent's failure behavior branches on the class of error, so the backend MUST make the classes distinguishable:

Class	Examples	Agent reaction the backend must support
Transient	unavailability, rate limiting, timeouts	Back off, buffer locally, retry. Backend MUST tolerate the resulting catch-up bursts
Identity	expired / revoked certificate, unknown cluster	Stop sending, attempt re-enroll if a valid token is present, otherwise degrade to local-only. MUST be clearly distinct from transient errors — misclassification causes retry storms or false local-only degradation
Permanent	malformed payload, unsupported schema version	Drop-and-log or hold-and-alert; never blind retry

The agent maps these onto status codes as follows, and a backend that means one class MUST NOT answer with the code of another (ADR 0075). 429 and 5xx are transient and retried. 400 and 413 are permanent for those bytes: the payload is held, not offered again, and lost when its local retention expires — so a backend MUST NOT use either to mean "try later". 401 and 403 are identity: the agent stops shipping entirely and does not resume without a restart, while continuing to collect. Any other status is treated as transient, which is what keeps a misconfigured address from emptying an agent's buffer.

6. Compatibility

Enterprise agents upgrade slowly and stay installed for a long time.

• Ingest payloads are versioned. The backend MUST accept payloads from prior agent versions across a published support window (target: at least N-2 minor versions) and MUST NOT require lockstep upgrades.
• Protocol changes are backward-compatible by default; a breaking change requires a deprecation period during which both forms are accepted.
• The backend SHOULD record each agent's reported version, making "which installs are behind" answerable centrally.

7. Control plane (operator-facing, agent-adjacent)

Not used by the agent itself, but required for its lifecycle at fleet scale (see the Terraform/GitOps onboarding decision):

• Org-level API keys: long-lived, scoped, revocable independently of running agents; they authorize automation, never enter clusters.
• Cluster registry: create/read/update/delete with names and labels; idempotent creation by name (safe under terraform apply re-runs); token issuance MUST be a separate explicit action, not a side effect of reads.
• Operations: issue/rotate enrollment token, revoke certificate, re-enroll — all per cluster, all automatable.
• The control plane speaks REST/JSON with an OpenAPI definition — its consumers are humans and automation (Terraform provider, CLI, UI, later MCP), not the agent.
• Fleet visibility. The backend MUST expose, per cluster: expected vs. reporting, last-seen time, ingest lag, agent version, and fingerprint conflicts. "Created 50, reporting 47 — and which three" must be answerable without touching any cluster.

8. Isolation and retention

• Data MUST be partitioned by organization and cluster; nothing crosses org boundaries in any view, export, or API.
• The backend retains full history (it is the source of truth); retention policy is a product decision but MUST NOT rely on agents re-supplying old data — agents can't.
• Deleting a cluster record MUST have defined, documented consequences for its stored data (and MUST NOT be triggerable by the agent itself).

9. Explicit non-goals of this contract

The backend MUST NOT expect or attempt:

• pushing configuration, filters, or collection targets to agents — all agent behavior is defined in-cluster via Helm values
• addressing or connecting to anything inside a customer cluster
• receiving raw time series, raw stack samples, unfiltered profiles, secrets, or pod env/args/command — the agent never sends them, and their arrival indicates an agent bug, which the backend SHOULD flag rather than store
• coupling ingest availability to analysis availability: analysis being down MUST NOT cause data loss for connected agents