feat: the counters are subtracted on the node (ADR 0062)

[kfeofantov]

The rows beside the footprint are a different kind of thing: counters that only
rise since the process started. run_delay says how long a process has waited
for a CPU in its whole life, and a life is not a question anybody asks. What a
finding needs is the change over an interval.

Where the subtraction happens is the decision

Shipping raw counters and subtracting on the controller was the obvious design
and it fails twice:

• A PID is reused. Two readings under one number can belong to two processes,
  and the controller has no way to tell — it never sees a start time. Holding
  per-PID state for a fleet would grow the store with the cluster rather than
  with the workloads.
• A process that exits between flushes loses its last interval, because the
  previous reading is the only thing that would have closed it.

The node reads the counters and the start time from the same file at the same
moment, and it already keeps a cache across passes for the binary marker.

What the cache does

One reading per PID between passes — counters, the instant, and the start time.
Three cases produce no delta rather than a wrong one: the first pass that
sees a process, a reused PID, and a counter that fell. The last drops the whole
delta rather than clamping one field: a counter that fell means the reading is
not of the process the last one described, whatever the start time says.

State is bounded by the process table, dropped for every PID a pass does not
see, and losing it costs one interval and never a total.

What ships

"processes": 2, "observed_nanos": 120000000000,
"on_cpu_nanos": 9000000000, "run_delay_nanos": 4000000000,
"cpu_user_nanos": 7000000000, "cpu_system_nanos": 2000000000,
"major_faults": 0, "voluntary_switches": 48210, "nonvoluntary_switches": 3122,
"read_bytes": 0, "write_bytes": 4096

observed_nanos sums with the deltas, so it is process-time, not a duration
— two processes watched for a minute each contribute two minutes. It is the
denominator every rate must be formed against, and it is named for that.

Sums, never the division. on_cpu_nanos beside run_delay_nanos is CPU
starvation stated as the two numbers it is made of. A payload that ships the
ratio cannot be re-divided by a consumer asking a different question — the shape
ADR 0004 settled for money and which holds for any derived figure.

A node's next report replaces its interval rather than extending it, the same
wholesale replacement peaks use, because adding successive intervals would turn a
rate into a total over a window nobody chose.

Filtering

Field two of /proc/<pid>/stat is the executable's own name and may contain
spaces and parentheses, so fields are counted from the last ) rather than
by splitting the line — which is also what keeps the name behind the cut rather
than filtered after it. rchar/wchar sit directly above the two I/O counters
that are read and are not taken: a finding about storage rests on bytes that
moved, not bytes that were asked for.

No new capability: /proc/<pid>/io sits behind the same ptrace check the node
already passes for /proc/<pid>/exe.

Named in the ADR, not hidden

The interval between a process's last observed pass and its exit is lost; closing
it would mean watching every process rather than sampling the table. And
processes here is lower than in process_peaks for one interval after a pod
starts — a process seen once has a footprint and no difference yet.

Evidence

go test ./... -race and golangci-lint run ./... on host and GOOS=linux:
0 issues. The inventory e2e now asserts that a delta exists at all, which proves
a second pass compared against the first for the same process, and that on-CPU
time cannot exceed the process-time observed. The e2e suites run against a kind
cluster and are not part of CI, so this has not been exercised on one.

🤖 Generated with Claude Code

https://claude.ai/code/session_011YRKuL97KRndaopfpixrhp