feat: the counters are subtracted on the node (ADR 0062)

cmd/agent/main.go

@@ -420,6 +420,12 @@ func run(ctx context.Context, logger *slog.Logger, clientset kubernetes.Interfac
 		if err := spool.WriteProcessPeaks(time.Now(), goStore.PeakSnapshot()); err != nil {
 			logger.Error("spooling process peaks", "error", err)
 		}
+		// What those processes did since each node's previous pass. Differences
+		// rather than readings, so this kind is the one whose records mean
+		// nothing without the interval they carry (ADR 0062).
+		if err := spool.WriteProcessCounters(time.Now(), goStore.CounterSnapshot()); err != nil {
+			logger.Error("spooling process counters", "error", err)
+		}
 		// Where those processes accept connections. Structural like the
 		// inventory, and separate from it because it changes without the build
 		// changing (ADR 0056 §3).

docs/adr/0052-the-peak-the-kernel-remembers.md

@@ -4,7 +4,7 @@ Date: 2026-08-29
 
 Status: Accepted
 
-Amended by: 0061
+Amended by: 0061, 0062
 
 Amends: 0009, 0006 §5
 

docs/adr/0061-what-the-process-holds-beside-its-peak.md

@@ -4,6 +4,8 @@ Date: 2026-08-31
 
 Status: Accepted
 
+Amended by: 0062
+
 Amends: 0052
 
 Adds seven fields to `process_peaks`, read from three files under

docs/adr/0062-the-counters-and-the-window-they-cover.md

@@ -0,0 +1,112 @@
+# 0062. The counters are subtracted on the node, and never leave without their interval
+
+Date: 2026-08-31
+
+Status: Accepted
+
+Amends: 0052, 0061
+
+Adds the payload kind `process_counters`, and one piece of node-held state: the
+previous pass's counter reading per process. Reads three more files under
+`/proc/<pid>/` and two more rows of one already open. No new privilege.
+
+## Context
+
+Everything ADR 0052 and ADR 0061 collect is a value: a peak, a footprint, a
+count of threads. The rows beside them are a different kind of thing — counters
+that only rise, since the process started. `run_delay` says how long a process
+has waited for a CPU in its whole life, and a life is not a question anybody
+asks. What a finding needs is the change over an interval.
+
+That subtraction has to happen somewhere, and where it happens is the decision.
+Shipping raw counters and subtracting on the controller was the first idea and it
+fails on two counts. A PID is reused, so two readings under one number can belong
+to two processes, and the controller has no way to tell — it never sees a start
+time, and holding per-PID state for a fleet's processes would grow the store with
+the cluster rather than with the workloads. And a process that exits between two
+flushes has its last interval lost, because the controller's previous reading is
+the only thing that would have closed it.
+
+The node has both halves: it reads the counters and the start time from the same
+file at the same moment, and it already keeps a cache across passes for the
+binary marker (ADR 0056).
+
+## Decision
+
+**1. The node subtracts, and the start time keys the subtraction.** A cache holds
+one reading per PID between passes — the counters, the instant they were read,
+and the process's start time. A pass compares against the previous reading only
+when the start time matches; otherwise the PID belongs to a different process and
+there is nothing to subtract from.
+
+Three cases produce no delta rather than a wrong one: the first pass that sees a
+process, a PID that has been reused, and a counter that fell. The last should be
+impossible when the start time matches, and it drops the whole delta rather than
+clamping one field, because a counter that fell means the reading is not of the
+process the last one described, whatever the start time says.
+
+Field two of `/proc/<pid>/stat` is the executable's own name, and it may contain
+spaces and parentheses. Fields are counted from the *last* `)` rather than by
+splitting the line — which is also why the name never reaches a variable here:
+it is behind the cut, not filtered after it (the discipline ADR 0052 set for
+`status`).
+
+**2. A delta never travels without the interval it covers.** Each carries
+`observed_nanos`, the wall-clock time between the two readings. A delta alone is
+a number no consumer can turn into a rate, and the interval is not a constant:
+a pass can be late, a report can be lost, and a node restarts.
+
+**3. The payload carries sums, never ratios.** `on_cpu_nanos` beside
+`run_delay_nanos` is CPU starvation stated as the two numbers it is made of. The
+agent could divide them, and does not: a payload that ships the division cannot
+be re-divided by a consumer asking a different question, which is the shape
+ADR 0004 settled for money and holds for every derived figure.
+
+`observed_nanos` sums across processes, so it is process-time and not a duration.
+Two processes observed for a minute each contribute two minutes. That makes it
+the denominator of a rate and not a window, and it is named for what it is.
+
+**4. A node's next report replaces its interval, it does not extend it.** The
+same wholesale replacement ADR 0052 §3 uses for peaks, for a different reason:
+adding successive intervals would turn a rate into a total over a window nobody
+chose, drifting further from any wall-clock boundary with every pass. The payload
+is therefore what the fleet's processes did over the most recent interval each
+node measured — ragged across nodes by construction, and honest about it, because
+each record carries the process-time it rests on.
+
+**5. Two rows come from a file already open.** The context-switch counters are in
+`/proc/<pid>/status`, which the scanner reads for the footprint. They are
+monotonic like the rest, so they join the reading rather than the footprint.
+
+## Consequences
+
+**Easier.** CPU starvation becomes measurable without depending on CFS
+throttling being visible: `run_delay` counts the wait even when the quota was
+never hit, and the non-voluntary switch count is the same fact from the
+scheduler's side. The user/system split says whether a service burns its own
+code or the kernel acting for it. Major faults separate a working set that does
+not fit from one that is merely large. Disk bytes per workload exist at all,
+which they do not in any kubelet reading.
+
+**Harder / given up.** The node now holds state between passes that it did not
+before: one reading per live process. It is bounded by the process table and
+dropped for every PID a pass does not see, and losing it costs one interval and
+never a total (ADR 0003). But it is state, and a node that restarts every minute
+would produce no counters at all — which is visible, because the record's
+`processes` count would stay at zero.
+
+The interval between a process's last observed pass and its exit is lost. There
+is no way to close it without the node observing the exit, which would mean
+watching every process rather than sampling the table.
+
+`processes` in this record is lower than in `process_peaks` for one interval
+after a pod starts, because a process seen once has a footprint and no delta.
+The two counts answer different questions and are not the same number.
+
+**Not changed.** No new capability — `/proc/<pid>/io` sits behind the same ptrace
+check the node already passes for `/proc/<pid>/exe` (ADR 0037). No identity: the
+executable name in `stat` is behind the field cut, and what a byte was read from
+is not in any of these files.
+
+This ADR records decisions implemented in the same pull request, per the process
+in [`README.md`](README.md).

docs/adr/README.md

@@ -119,3 +119,4 @@ line to paste.
 - [0059. The node knows whose code it profiles, so the allow-list stops being a prerequisite](0059-the-node-knows-whose-code-it-profiles.md)
 - [0060. The node says what its profiler did, on the channel that fires when it did nothing](0060-the-node-says-what-its-profiler-did.md)
 - [0061. What the process holds, beside the peak it once reached](0061-what-the-process-holds-beside-its-peak.md)
+- [0062. The counters are subtracted on the node, and never leave without their interval](0062-the-counters-and-the-window-they-cover.md)

docs/backend-requirements.md

@@ -413,6 +413,20 @@ a record is absent, never zero, when none were read. The record is keyed by
 image digest as well as container: a peak belongs to the build that reached it,
 and MUST NOT be carried across a rollout.
 
+**`process_counters` records are differences, and `observed_nanos` is what they
+rest on** ([ADR 0062](adr/0062-the-counters-and-the-window-they-cover.md)). Each
+is the change since the reporting node's previous pass, summed over processes and
+nodes, so it MUST NOT be summed across flushes: a node's next report replaces its
+interval rather than extending it. `observed_nanos` sums with the deltas, so it
+is process-time and not a duration — two processes watched for a minute each
+contribute two minutes — and it is the denominator every rate MUST be formed
+against, never the flush interval or the wall clock. `on_cpu_nanos` beside
+`run_delay_nanos` is CPU starvation; the backend forms the ratio, because the
+agent ships numerators and denominators and never the division. A `processes`
+count lower than the same container's in `process_peaks` is expected for one
+interval after a pod starts: a process seen once has a footprint and no
+difference yet, and MUST NOT be read as processes having exited.
+
 **The footprint fields beside it are samples, and the field names say so**
 ([ADR 0061](adr/0061-what-the-process-holds-beside-its-peak.md)). `peak_rss_bytes`
 is a mark the kernel maintains between readings; every `*_max` beside it is the

docs/prose_test.go

@@ -35,7 +35,7 @@ var ceilings = []struct {
 	section int
 }{
 	{"security.md", 1036, 360},
-	{"backend-requirements.md", 610, 467},
+	{"backend-requirements.md", 624, 481},
 }
 
 func TestNoCommentRunIsLongerThanAPointerToAnADR(t *testing.T) {

docs/security.md

@@ -524,6 +524,7 @@ test fails if a kind ships without appearing here (ADR 0022).
 | `go_inventory` | Per (namespace, workload, container): Go version, main module, image digest, PGO flag, plus a fleet-coverage block | no |
 | `process_peaks` | Per collected workload container and build: the largest `VmHWM` among its Go processes, how many processes that was taken over, and the range of permitted CPU counts. A **floor** under the container's peak, never the figure the OOM killer compares against — the cgroup also holds page cache and every other process in it ([ADR 0052](adr/0052-the-peak-the-kernel-remembers.md)). Beside it, the largest sample taken of each process's footprint: resident memory split into the program's own pages and file-backed ones, the memory it shares with its replicas and the memory nothing else maps, its OS threads, and how many file descriptors it held against the ceiling in force ([ADR 0061](adr/0061-what-the-process-holds-beside-its-peak.md)). **What each descriptor points at is never read** — they are counted, not followed | no |
 | `go_build` | Per image digest, written once: Go version, main module, each dependency module's **path and version**, allow-listed build settings, two allow-listed GODEBUG defaults, and whether `net/http/pprof` is linked into the build. This is a bill of materials for the build — see [§10.4](#104-the-build-inventory-is-a-bill-of-materials) | no |
+| `process_counters` | Per collected workload container and build: what its processes did since each node's previous scan pass — time on CPU and time waiting for one, that CPU time split between your code and the kernel acting for it, page faults that reached storage, context switches, and bytes that moved to and from block storage. Each record carries the process-time the numbers cover, because a difference without its interval is not a rate ([ADR 0062](adr/0062-the-counters-and-the-window-they-cover.md)). Sums only — the agent never ships the division | no |
 | `listening_ports` | Per collected workload container and build: the TCP ports its Go processes accept connections on, each with whether it is bound to loopback. Read from the processes' own sockets, so a port nothing declared is here and a declared port nothing binds is not ([ADR 0056](adr/0056-a-pprof-endpoint-is-proved-not-probed.md)). No address, and no connection the workload has open | no |
 | `ebpf_profile` | One capture: allow-list-filtered symbolized pprof bytes, keyed by workload, image digest and capture window | no |
 | `pprof_profile` | One ten-second CPU profile fetched from the workload's own `/debug/pprof` endpoint, allow-list-filtered, under the same key ([ADR 0058](adr/0058-the-pull-starts-a-profiler-and-the-binary-says-whose-code-it-is.md)). The mappings — the executable's path on disk and its build ID — the sample labels, and the full source paths are removed with the redacted frames, so what remains is function names your allow-list admits, base file names and line numbers. Beside it travel the counts of what was redacted, never what | no |