feat: what the process holds, beside the peak it once reached (ADR 0061)

docs/adr/0052-the-peak-the-kernel-remembers.md

@@ -4,6 +4,8 @@ Date: 2026-08-29
 
 Status: Accepted
 
+Amended by: 0061
+
 Amends: 0009, 0006 §5
 
 Adds the payload kind `process_peaks`. Widens the node scanner's reads from

docs/adr/0061-what-the-process-holds-beside-its-peak.md

@@ -0,0 +1,115 @@
+# 0061. What the process holds, beside the peak it once reached
+
+Date: 2026-08-31
+
+Status: Accepted
+
+Amends: 0052
+
+Adds seven fields to `process_peaks`, read from three files under
+`/proc/<pid>/`. No new privilege, no new payload kind, no new RBAC, and no
+identity of anything.
+
+## Context
+
+ADR 0052 established the node's second read: `/proc/<pid>/status`, for `VmHWM`
+and `Cpus_allowed_list`. That file has about forty rows and the scanner takes
+two of them, because the allow-list was drawn for one finding.
+
+Three questions the agent cannot answer today are answered by rows in that same
+already-open file, or in two small neighbours:
+
+- **How much of a container's memory is the program's own.** A memory limit
+  counts anonymous pages and file-backed pages alike, so a service whose 800 MB
+  is 300 MB of heap and 500 MB of page cache looks identical to one that
+  allocated all of it. Advice about limits rests on the difference, and the
+  agent has been giving the total.
+- **What one more replica actually costs.** Resident memory counts a shared page
+  once per process mapping it. Replicas of one image share their text, so
+  summing RSS over replicas overstates what they occupy, and the figure that
+  does not is `Pss`.
+- **How close a process is to a ceiling it will hit without warning.** Descriptor
+  exhaustion is not visible in any metric until connections start failing.
+
+None of these needs a mechanism the agent does not have. The scanner already
+walks the process, already opens `status`, already passes the kernel's ptrace
+check to do it.
+
+## Decision
+
+**1. The status allow-list widens by three rows.** `RssAnon` and `RssFile` split
+resident memory into the program's own pages and the file-backed ones; `Threads`
+is the process's OS threads, which for a Go program is a consequence rather than
+a setting — the runtime creates one per blocking syscall it cannot park, so a
+count far above `GOMAXPROCS` is the signature of blocking or cgo work.
+
+The list stays an allow-list. `Name` is still line one of every status file and
+still has nowhere in the struct to go, which is the property the parser's test
+exists to keep true as the struct grows.
+
+**2. `smaps_rollup` is read for `Pss` and `Private_Dirty`.** The rollup is the
+kernel's own aggregate over every mapping, so this is one small file rather than
+a walk of `smaps`, which on a large address space runs to thousands of lines.
+Keys are matched exactly: `Pss_Dirty` sits directly under `Pss` and
+`Private_Clean` directly above `Private_Dirty`, so a prefix match would be wrong
+by a plausible amount, which is the worst kind of wrong.
+
+This is the one read in the pass with a cost the others do not have: the kernel
+walks the page tables to compute `Pss`. That cost is not measured, and by
+ADR 0039 it is therefore marked rather than claimed — see Consequences.
+
+**3. Descriptors are counted against the ceiling in force.** The entries of
+`/proc/<pid>/fd` are counted without being followed: what each points at is a
+path, a socket or a pipe, which is an identity. The soft `RLIMIT_NOFILE` comes
+from `/proc/<pid>/limits` in the same pass, so the headroom between the two is a
+fact about one process at one moment rather than a subtraction across two
+readings (ADR 0043's discipline). An unlimited ceiling reports as absent, never
+as a very large number that would read as a real one.
+
+**4. The new fields merge as maxima, except the ceiling, which merges as a
+minimum.** Everything else in this record is the largest value seen across the
+processes of one build on every node still running it. The descriptor limit is
+the exception because it is not a measurement but a bound: the honest pairing is
+the most descriptors any replica held against the lowest ceiling any replica was
+given.
+
+The field names say which kind of claim each is. `peak_rss_bytes` is a mark the
+kernel maintains between readings; every `*_max` beside it is the highest value
+the agent happened to sample. Naming them alike would have made the payload
+claim a continuity it does not have.
+
+**5. A process that measured nothing contributes nothing.** Unchanged from
+ADR 0052 and worth restating because it now covers more fields: a zero is not
+folded in, and such a process is not counted in `processes`, which says how many
+processes stood behind the numbers rather than how many ran.
+
+## Consequences
+
+**Easier.** Memory advice can distinguish the heap from the page cache, which is
+the distinction that makes a limit safe to cut. The marginal cost of a replica
+becomes a number rather than an assumption. Descriptor exhaustion becomes a
+prediction instead of a post-mortem.
+
+Three of the seven fields cost no new file: they are rows of a file the scanner
+already opens and parses.
+
+**Harder / given up.** `smaps_rollup` is read per kept process per pass and its
+cost scales with the process's address space. It is bounded by the same loop
+that bounds everything else in the pass, and a missing file — a kernel before
+4.14 — is the zero value rather than an error. But the cost itself is unmeasured
+on a real node, and this ADR states that rather than implying a measurement that
+was not taken. The way to close it is the inventory end-to-end run on a cluster
+with large processes, timing the pass with the read and without.
+
+The record now mixes two kinds of memory claim under one key. The names carry
+the distinction; a reader who ignores them will average a kernel-maintained mark
+with a sample.
+
+**Not changed.** No new capability: the node already runs with `hostPID` and
+`CAP_SYS_PTRACE` to open `/proc/<pid>/exe` (ADR 0037), and every file here sits
+behind the same check. No identity leaves — the descriptor entries are counted
+and not followed, the rollup's mapping paths are not read, and the process name
+still has nowhere to go.
+
+This ADR records decisions implemented in the same pull request, per the process
+in [`README.md`](README.md).

docs/adr/README.md

@@ -118,3 +118,4 @@ line to paste.
 - [0058. The pull starts a profiler, and the binary says whose code it is](0058-the-pull-starts-a-profiler-and-the-binary-says-whose-code-it-is.md)
 - [0059. The node knows whose code it profiles, so the allow-list stops being a prerequisite](0059-the-node-knows-whose-code-it-profiles.md)
 - [0060. The node says what its profiler did, on the channel that fires when it did nothing](0060-the-node-says-what-its-profiler-did.md)
+- [0061. What the process holds, beside the peak it once reached](0061-what-the-process-holds-beside-its-peak.md)

docs/backend-requirements.md

@@ -413,6 +413,20 @@ a record is absent, never zero, when none were read. The record is keyed by
 image digest as well as container: a peak belongs to the build that reached it,
 and MUST NOT be carried across a rollout.
 
+**The footprint fields beside it are samples, and the field names say so**
+([ADR 0061](adr/0061-what-the-process-holds-beside-its-peak.md)). `peak_rss_bytes`
+is a mark the kernel maintains between readings; every `*_max` beside it is the
+highest value the agent happened to observe, so the two MUST NOT be averaged or
+charted as one series. `rss_anon_bytes_max` is the program's own pages and
+`rss_file_bytes_max` the file-backed ones; both count against a memory limit, and
+a recommendation to lower one MUST rest on the first rather than on their sum.
+`pss_bytes_max` charges shared pages to their sharers in equal parts, so it —
+never resident memory — is what may be summed across replicas of one image.
+`open_files_max` against `open_files_limit_min` is the most descriptors any
+replica held against the lowest ceiling any replica was given; an absent ceiling
+means unlimited and MUST NOT be read as zero. Any of these fields may be absent
+on a kernel that does not expose its source, and absent never means zero.
+
 **A `pprof_profile` is the workload's own runtime sampling itself, and `dropped`
 bounds what may be concluded from it**
 ([ADR 0058](adr/0058-the-pull-starts-a-profiler-and-the-binary-says-whose-code-it-is.md)).

docs/prose_test.go

@@ -35,7 +35,7 @@ var ceilings = []struct {
 	section int
 }{
 	{"security.md", 1036, 360},
-	{"backend-requirements.md", 596, 455},
+	{"backend-requirements.md", 610, 467},
 }
 
 func TestNoCommentRunIsLongerThanAPointerToAnADR(t *testing.T) {

docs/security.md

@@ -522,7 +522,7 @@ test fails if a kind ships without appearing here (ADR 0022).
 | `cluster_policy` | Per collected namespace, its LimitRanges and ResourceQuotas; plus the cluster's PriorityClasses and StorageClasses, which workloads reference by name (ADR 0032), and `unavailable_sources` for any of those the agent could not read — never granted, or no longer being fed (ADR 0033, ADR 0035). StorageClass `parameters` are never read | no |
 | `node_metadata` | Per node: name, size, instance and capacity type, zone, region, kernel version, CPU architecture. The **name** is your cluster's, and on EKS and on GKE's legacy naming it encodes the node's private address — `ip-10-42-13-201.eu-west-1.compute.internal`. The agent reads no address field; the name it does read may be one (ADR 0039) | no |
 | `go_inventory` | Per (namespace, workload, container): Go version, main module, image digest, PGO flag, plus a fleet-coverage block | no |
-| `process_peaks` | Per collected workload container and build: the largest `VmHWM` among its Go processes, how many processes that was taken over, and the range of permitted CPU counts. A **floor** under the container's peak, never the figure the OOM killer compares against — the cgroup also holds page cache and every other process in it ([ADR 0052](adr/0052-the-peak-the-kernel-remembers.md)) | no |
+| `process_peaks` | Per collected workload container and build: the largest `VmHWM` among its Go processes, how many processes that was taken over, and the range of permitted CPU counts. A **floor** under the container's peak, never the figure the OOM killer compares against — the cgroup also holds page cache and every other process in it ([ADR 0052](adr/0052-the-peak-the-kernel-remembers.md)). Beside it, the largest sample taken of each process's footprint: resident memory split into the program's own pages and file-backed ones, the memory it shares with its replicas and the memory nothing else maps, its OS threads, and how many file descriptors it held against the ceiling in force ([ADR 0061](adr/0061-what-the-process-holds-beside-its-peak.md)). **What each descriptor points at is never read** — they are counted, not followed | no |
 | `go_build` | Per image digest, written once: Go version, main module, each dependency module's **path and version**, allow-listed build settings, two allow-listed GODEBUG defaults, and whether `net/http/pprof` is linked into the build. This is a bill of materials for the build — see [§10.4](#104-the-build-inventory-is-a-bill-of-materials) | no |
 | `listening_ports` | Per collected workload container and build: the TCP ports its Go processes accept connections on, each with whether it is bound to loopback. Read from the processes' own sockets, so a port nothing declared is here and a declared port nothing binds is not ([ADR 0056](adr/0056-a-pprof-endpoint-is-proved-not-probed.md)). No address, and no connection the workload has open | no |
 | `ebpf_profile` | One capture: allow-list-filtered symbolized pprof bytes, keyed by workload, image digest and capture window | no |

internal/inventory/store.go

@@ -75,6 +75,21 @@ type PeakRecord struct {
 	// mean the replicas sit on different machines, or one of them is pinned.
 	CPUsAllowedMin int `json:"cpus_allowed_min,omitempty"`
 	CPUsAllowedMax int `json:"cpus_allowed_max,omitempty"`
+	// The largest sample the agent took of each footprint field, over every
+	// process of this build the fleet still runs. Named apart from
+	// PeakRSSBytes above because they are a different kind of claim: that one
+	// is a mark the kernel maintains between readings, these are the highest
+	// value the agent happened to see (ADR 0061 §4).
+	RSSAnonBytesMax      int64 `json:"rss_anon_bytes_max,omitempty"`
+	RSSFileBytesMax      int64 `json:"rss_file_bytes_max,omitempty"`
+	PSSBytesMax          int64 `json:"pss_bytes_max,omitempty"`
+	PrivateDirtyBytesMax int64 `json:"private_dirty_bytes_max,omitempty"`
+	ThreadsMax           int   `json:"threads_max,omitempty"`
+	// OpenFilesMax against OpenFilesLimitMin is the tightest headroom any
+	// replica of this build has: the most descriptors one held, against the
+	// lowest ceiling another was given.
+	OpenFilesMax      int   `json:"open_files_max,omitempty"`
+	OpenFilesLimitMin int64 `json:"open_files_limit_min,omitempty"`
 }
 
 // nodePeaks is one node's contribution to a PeakRecord: what its latest report
@@ -83,6 +98,8 @@ type nodePeaks struct {
 	peakRSSBytes     int64
 	processes        int
 	cpusMin, cpusMax int
+	footprint        nodescan.Footprint // each field the largest this node saw
+	filesLimitMin    int64
 }
 
 // PortKey extends Key with the build, for the reason PeakKey does: a rollout
@@ -289,7 +306,7 @@ func (s *Store) Ingest(report nodescan.Report, resolver ContainerResolver) {
 //
 // Callers hold s.mu.
 func (s *Store) ingestPeaks(b nodescan.BinaryInfo, key Key, digest, node string, seen map[PeakKey]bool) {
-	if b.PeakRSSBytes == 0 && b.CPUsAllowed == 0 {
+	if b.PeakRSSBytes == 0 && b.CPUsAllowed == 0 && b.Footprint == (nodescan.Footprint{}) {
 		return
 	}
 	pk := PeakKey{Key: key, ImageDigest: digest}
@@ -315,6 +332,18 @@ func (s *Store) ingestPeaks(b nodescan.BinaryInfo, key Key, digest, node string,
 		}
 		n.cpusMax = max(n.cpusMax, c)
 	}
+	f := b.Footprint
+	n.footprint.RSSAnonBytes = max(n.footprint.RSSAnonBytes, f.RSSAnonBytes)
+	n.footprint.RSSFileBytes = max(n.footprint.RSSFileBytes, f.RSSFileBytes)
+	n.footprint.PSSBytes = max(n.footprint.PSSBytes, f.PSSBytes)
+	n.footprint.PrivateDirtyBytes = max(n.footprint.PrivateDirtyBytes, f.PrivateDirtyBytes)
+	n.footprint.Threads = max(n.footprint.Threads, f.Threads)
+	n.footprint.OpenFiles = max(n.footprint.OpenFiles, f.OpenFiles)
+	// The lowest ceiling, not the highest: headroom is what the tightest
+	// replica has, and a limit of zero is one that was not read.
+	if l := f.OpenFilesLimit; l > 0 && (n.filesLimitMin == 0 || l < n.filesLimitMin) {
+		n.filesLimitMin = l
+	}
 	byNode[node] = n
 }
 
@@ -599,6 +628,15 @@ func (s *Store) PeakSnapshot() []PeakRecord {
 				rec.CPUsAllowedMin = n.cpusMin
 			}
 			rec.CPUsAllowedMax = max(rec.CPUsAllowedMax, n.cpusMax)
+			rec.RSSAnonBytesMax = max(rec.RSSAnonBytesMax, n.footprint.RSSAnonBytes)
+			rec.RSSFileBytesMax = max(rec.RSSFileBytesMax, n.footprint.RSSFileBytes)
+			rec.PSSBytesMax = max(rec.PSSBytesMax, n.footprint.PSSBytes)
+			rec.PrivateDirtyBytesMax = max(rec.PrivateDirtyBytesMax, n.footprint.PrivateDirtyBytes)
+			rec.ThreadsMax = max(rec.ThreadsMax, n.footprint.Threads)
+			rec.OpenFilesMax = max(rec.OpenFilesMax, n.footprint.OpenFiles)
+			if l := n.filesLimitMin; l > 0 && (rec.OpenFilesLimitMin == 0 || l < rec.OpenFilesLimitMin) {
+				rec.OpenFilesLimitMin = l
+			}
 		}
 		if rec.Processes == 0 {
 			continue

internal/inventory/store_test.go

@@ -821,3 +821,74 @@ func TestProfileCoverageFollowsTheNodesThatAreLeft(t *testing.T) {
 		t.Errorf("after the node left: nodes = %d, states = %v", got.Nodes, got.States)
 	}
 }
+
+// withFootprint attaches what one process currently holds, beside the mark
+// withPeak attaches.
+func withFootprint(b nodescan.BinaryInfo, f nodescan.Footprint) nodescan.BinaryInfo {
+	b.Footprint = f
+	return b
+}
+
+// Each footprint field merges as the largest sample seen — except the
+// descriptor ceiling, which merges as the lowest: headroom is what the tightest
+// replica has (ADR 0061 §4).
+func TestFootprintTakesTheLargestSampleAndTheLowestCeiling(t *testing.T) {
+	s := NewStore(testSince)
+	s.Ingest(nodescan.Report{Node: "node-1", Binaries: []nodescan.BinaryInfo{
+		withFootprint(binary("pod-web-1", "cid-web", "go1.26.1", "github.com/acme/web", false), nodescan.Footprint{
+			RSSAnonBytes: 300 << 20, RSSFileBytes: 60 << 20, PSSBytes: 340 << 20,
+			PrivateDirtyBytes: 290 << 20, Threads: 17, OpenFiles: 900, OpenFilesLimit: 65536,
+		}),
+	}}, webResolver())
+	s.Ingest(nodescan.Report{Node: "node-2", Binaries: []nodescan.BinaryInfo{
+		withFootprint(binary("pod-web-2", "cid-web", "go1.26.1", "github.com/acme/web", false), nodescan.Footprint{
+			RSSAnonBytes: 420 << 20, RSSFileBytes: 55 << 20, PSSBytes: 470 << 20,
+			PrivateDirtyBytes: 410 << 20, Threads: 9, OpenFiles: 61000, OpenFilesLimit: 1024,
+		}),
+	}}, webResolver())
+
+	peaks := s.PeakSnapshot()
+	if len(peaks) != 1 {
+		t.Fatalf("peaks = %+v, want one record", peaks)
+	}
+	p := peaks[0]
+	if p.RSSAnonBytesMax != 420<<20 || p.RSSFileBytesMax != 60<<20 {
+		t.Errorf("rss anon/file = %d/%d, want the larger of each", p.RSSAnonBytesMax, p.RSSFileBytesMax)
+	}
+	if p.PSSBytesMax != 470<<20 || p.PrivateDirtyBytesMax != 410<<20 {
+		t.Errorf("pss/private dirty = %d/%d, want the larger of each", p.PSSBytesMax, p.PrivateDirtyBytesMax)
+	}
+	if p.ThreadsMax != 17 {
+		t.Errorf("threads = %d, want 17", p.ThreadsMax)
+	}
+	if p.OpenFilesMax != 61000 || p.OpenFilesLimitMin != 1024 {
+		t.Errorf("descriptors = %d of %d, want the most held against the lowest ceiling",
+			p.OpenFilesMax, p.OpenFilesLimitMin)
+	}
+}
+
+// A process whose footprint could not be read contributes nothing rather than a
+// zero, which would drag the descriptor ceiling to "no ceiling" and claim a
+// container reached nothing.
+func TestAnUnreadFootprintContributesNothing(t *testing.T) {
+	s := NewStore(testSince)
+	s.Ingest(nodescan.Report{Node: "node-1", Binaries: []nodescan.BinaryInfo{
+		withFootprint(binary("pod-web-1", "cid-web", "go1.26.1", "github.com/acme/web", false), nodescan.Footprint{
+			Threads: 17, OpenFiles: 900, OpenFilesLimit: 65536,
+		}),
+		binary("pod-web-2", "cid-web", "go1.26.1", "github.com/acme/web", false), // nothing read
+	}}, webResolver())
+
+	p := s.PeakSnapshot()[0]
+	if p.OpenFilesLimitMin != 65536 {
+		t.Errorf("ceiling = %d; an unread limit was treated as a limit", p.OpenFilesLimitMin)
+	}
+	if p.ThreadsMax != 17 {
+		t.Errorf("threads = %d, want 17", p.ThreadsMax)
+	}
+	// And it is not counted among the processes the numbers were taken over:
+	// `processes` says how many stood behind the figures, not how many ran.
+	if p.Processes != 1 {
+		t.Errorf("processes = %d, want 1: the process that measured nothing stands behind nothing", p.Processes)
+	}
+}