Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,13 @@ jobs:
- uses: golangci/golangci-lint-action@v9
with:
version: v2.11.4
# No analysis cache. On 2026-09-16 a restored one reported six
# SA5011 findings in files the branch did not touch, twice, on a
# tree byte-identical to a passing main; a cold run found none.
# Seeing that `t.Fatal` does not return takes knowledge of the
# testing package that a partial cache drops. 24s a run buys the
# whole class of false red.
skip-cache: true

proto:
name: Payload schema (buf)
Expand Down
10 changes: 8 additions & 2 deletions cmd/agent/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -541,8 +541,14 @@ func run(ctx context.Context, logger *slog.Logger, clientset kubernetes.Interfac
}
// Where those processes accept connections. Structural like the
// inventory, and separate from it because it changes without the build
// changing (ADR 0056 §3).
if err := spool.WriteListeningPorts(time.Now(), goStore.PortSnapshot()); err != nil {
// changing (ADR 0056 §3). The prober stamps each port with what its one
// request answered, which is the only part of this payload the
// controller learned rather than received (ADR 0082).
ports := goStore.PortSnapshot()
if prober != nil {
ports = prober.Stamp(ports)
}
if err := spool.WriteListeningPorts(time.Now(), ports); err != nil {
logger.Error("spooling listening ports", "error", err)
}
// Build facts are keyed by image digest and immutable for it, so each is
Expand Down
116 changes: 116 additions & 0 deletions docs/adr/0082-a-confirmed-endpoint-is-named-on-its-port.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
# 0082. A confirmed endpoint is named on the port it answered on

Date: 2026-09-16

Status: Accepted

Adds a `pprof` field to each port of `listening_ports`, carrying what the
prober's one request to that port answered.

## Context

Three facts about a workload's debug surface already ship, and the fourth — the
one that makes the other three a finding rather than a lead — does not.

| Fact | Payload | Identity |
|---|---|---|
| the binary links `net/http/pprof` | `go_build.pprof_endpoint` | image digest |
| the process listens on a port, loopback or not | `listening_ports` | workload, container, digest |
| six targets confirmed, eleven absent, two unreachable | `collection_coverage.pprof` | **none — counts only** |
| *which* port of *which* workload answered | — | **nowhere** |

The prober knows. It asks about a `Target{ImageDigest, Port}` and files the
answer under that pair (`internal/pprofprobe/probe.go`). What leaves the cluster
is `Snapshot()`, four integers.

The gap cannot be closed by joining what ships. Take the `listening_ports`
golden: `shop/web` links pprof, listens on 8080 and on 6060 bound to loopback.
Which of the two serves the mux? The payloads cannot say, and the difference is
the whole finding — a pprof mux on a routable port serves
`/debug/pprof/heap`, the contents of the process's memory, to anything that can
reach the pod, and `/debug/pprof/cmdline`, the argv that CLAUDE.md invariant 4
refuses to collect. On a loopback port it serves them to nothing.

An inference from "links pprof" and "has a routable port" is right most of the
time and wrong silently. A report that presents it as a finding spends the
credibility the report exists to build.

## Decision

**1. The answer is stamped onto the port, in the payload that already names
it.** `listening_ports` records carry `{namespace, workload, container, digest}`
and a list of ports; the prober's key is `{digest, port}`. The join is a lookup,
so no new payload, no new kind, and no new identity: the workload was already
named in this payload, on this record, one field to the left.

Stamping happens where the payload is assembled, and the prober does it —
`Prober.Stamp` — because the package that owns an answer should be the one that
writes it down. The store stays a pure join of what nodes reported.

**2. The value is a word, and its absence is not a fourth word.** `confirmed`,
`absent`, `unreachable` — the prober's own three states. A port nothing asked
about carries no field at all, and every loopback port is such a port:
`Candidates` drops them before the funnel, because nothing outside the pod can
open them.

A boolean would have collapsed "asked, and it is not pprof" into "not asked",
and the second is the common case. Proof of absence and absence of proof are
different claims and this payload makes both, so it must say which it is making.

**3. The controller's type is not the node's.** `inventory.Port` is new and
carries the node's observation plus the controller's answer;
`nodescan.ListeningPort` keeps carrying only what a node can know, which is what
crosses the node→controller channel. The alternative — one field on the node's
type that the node never fills — is one line shorter and blurs the line the
channel exists to draw (ADR 0009, ADR 0052's shape).

**4. Why this names a workload where the coverage payload does not.**
[ADR 0054](0054-coverage-says-how-much-was-hidden-never-what.md) reports counts
and no identities, and CLAUDE.md invariant 6 forbids the identity of a
*filtered-out* object from leaving the cluster. Neither is in the way here, and
saying why matters more than the field does.

The invariant protects the customer's refusal: they said do not look, and we do
not report on whom. This record is the opposite — an admitted workload, under
collection, and the fact is about that workload's own configuration, published
to that workload's own operator. A workload the profiling opt-out excluded is
never probed at all (ADR 0071), so it has no answer to stamp, and its silence
here is the same silence as everywhere else.

**5. The objection, named.** The payload becomes a list of debug endpoints
reachable inside the customer's cluster. Weak — the ports are already in this
payload, the mux answers any pod on the network whether or not we write it down,
and a reader of the spool is already inside — but weak is not absent, and an
unnamed objection is a hole in the argument (ADR 0039's discipline). It is
accepted because the alternative is not telling the operator that their memory
is readable.

## Consequences

**Easier.** The finding a report can make becomes provable instead of inferred:
this workload, this port, answered the pprof index. The loopback case, which is
the correct configuration and the common one, is legible as correct rather than
indistinguishable from the dangerous one.

The three coverage counters keep their meaning and are not replaced. They count
targets, including targets of workloads that have since gone; this field
describes ports that exist now.

**Harder / given up.** A second type where there was one, and a conversion in
`PortSnapshot` to go with it. The cost is real and it buys the channel's
boundary: a reviewer asking "can a node claim an endpoint is confirmed?" reads
the type and sees that it cannot.

The field describes the latest answer, not a history. A port that was confirmed
and is now unreachable reads as unreachable, and the payload does not say it
ever answered — the prober's own retry rule (`unreachableRetry`) decides how
long that takes to correct.

**Not changed.** No new request, no new read, no new connection: the answer
being published was already established by the probe ADR 0057 authorised, and
this ADR moves nothing but where it is written. `go_build.pprof_endpoint` still
answers the question about the image, and this field the question about the
address.

This ADR records decisions implemented in the same pull request, per the process
in [`README.md`](README.md).
1 change: 1 addition & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,3 +139,4 @@ line to paste.
- [0079. The last pass writes, and then ships what it wrote](0079-the-last-pass-writes-then-ships.md)
- [0080. The index page already counts the goroutines](0080-the-index-page-already-counts-the-goroutines.md)
- [0081. A weakened security default is a build fact like any other](0081-a-weakened-default-is-a-build-fact.md)
- [0082. A confirmed endpoint is named on the port it answered on](0082-a-confirmed-endpoint-is-named-on-its-port.md)
9 changes: 9 additions & 0 deletions docs/backend-requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -614,6 +614,15 @@ addressed from this payload. Records are keyed by image digest as well as
container and MUST NOT be carried across a rollout; a port absent from the
latest report is a port no process still binds.

A port MAY carry `pprof`, holding `confirmed`, `absent` or `unreachable` — what
the agent's one request to that port answered
([ADR 0082](adr/0082-a-confirmed-endpoint-is-named-on-its-port.md)). **An absent
`pprof` field means the port was not asked about and MUST NOT be read as
`absent`**: loopback ports are never asked, nor is any port of a workload the
profiling opt-out excludes. Only `confirmed` on a port not marked `loopback`
states that the pprof index answered where another pod could reach it. The value
is the latest answer and carries no history.

**`workload_policy` Service endpoint counts are per address family and MUST NOT
be summed across them** ([ADR 0051](adr/0051-topology-routing-is-asked-and-not-granted.md)).
A dual-stack Service lists each pod once in an IPv4 slice and once in an IPv6
Expand Down
17 changes: 9 additions & 8 deletions docs/prose_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,15 @@ var ceilings = []struct {
// GODEBUG names are read out of the build, on a second axis from the two
// the promise named before (ADR 0081).
{"security.md", 1116, 361},
// Each raise names its obligation here too. The request body is encoded, so
// the implementer must decode it and stop reading a payload's size off the
// request (ADR 0076); a journal window states whether it is finished
// (ADR 0078); a new kind arrives whose samples are gauges, where the obvious
// ingest — adding them up — is the wrong one (ADR 0080); `godebug` carries
// names where neither value is safe by convention (ADR 0081); and the filter
// counters count decisions, so adding them no longer recovers a pod count.
{"backend-requirements.md", 806, 635},
// Each raise names its obligation here too. The body is encoded, so a
// payload's size is no longer the request's (ADR 0076); a journal window
// states whether it is finished (ADR 0078); gauge samples arrive, where
// adding them up is the wrong ingest (ADR 0080); `godebug` carries names
// where neither value is safe by convention (ADR 0081); the filter counters
// count decisions, so adding them recovers no pod count; and a port's
// `pprof` is absent when nothing asked, which is not the claim `absent`
// makes (ADR 0082).
{"backend-requirements.md", 815, 644},
}

func TestNoCommentRunIsLongerThanAPointerToAnADR(t *testing.T) {
Expand Down
2 changes: 1 addition & 1 deletion docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -535,7 +535,7 @@ raise them, and nothing can ask for data sooner.
| `process_peaks` | Per collected workload container and build: the largest `VmHWM` among its Go processes, how many processes that was taken over, and the range of permitted CPU counts. A **floor** under the container's peak, never the figure the OOM killer compares against — the cgroup also holds page cache and every other process in it ([ADR 0052](adr/0052-the-peak-the-kernel-remembers.md)). Beside it, the largest sample taken of each process's footprint: resident memory split into the program's own pages and file-backed ones, the memory it shares with its replicas and the memory nothing else maps, its OS threads, and how many file descriptors it held against the ceiling in force ([ADR 0061](adr/0061-what-the-process-holds-beside-its-peak.md)). **What each descriptor points at is never read** — they are counted, not followed | no |
| `go_build` | Per image digest, written once: Go version, main module, each dependency module's **path and version**, allow-listed build settings, two allow-listed GODEBUG defaults, and whether `net/http/pprof` is linked into the build. This is a bill of materials for the build — see [§10.4](#104-the-build-inventory-is-a-bill-of-materials) | no |
| `process_counters` | Per collected workload container and build: what its processes did since each node's previous scan pass — time on CPU and time waiting for one, that CPU time split between your code and the kernel acting for it, page faults that reached storage, context switches, and bytes that moved to and from block storage. Each record carries the process-time the numbers cover, because a difference without its interval is not a rate ([ADR 0062](adr/0062-the-counters-and-the-window-they-cover.md)). Sums only — the agent never ships the division | no |
| `listening_ports` | Per collected workload container and build: the TCP ports its Go processes accept connections on, each with whether it is bound to loopback. Read from the processes' own sockets, so a port nothing declared is here and a declared port nothing binds is not ([ADR 0056](adr/0056-a-pprof-endpoint-is-proved-not-probed.md)). No address, and no connection the workload has open | no |
| `listening_ports` | Per collected workload container and build: the TCP ports its Go processes accept connections on, each with whether it is bound to loopback, and — for a port the agent asked — whether the Go pprof index answered there ([ADR 0082](adr/0082-a-confirmed-endpoint-is-named-on-its-port.md)). Read from the processes' own sockets, so a port nothing declared is here and a declared port nothing binds is not ([ADR 0056](adr/0056-a-pprof-endpoint-is-proved-not-probed.md)). No address, and no connection the workload has open | no |
| `ebpf_profile` | One capture: allow-list-filtered symbolized pprof bytes, keyed by workload, image digest and capture window | no |
| `pprof_profile` | One ten-second CPU profile fetched from the workload's own `/debug/pprof` endpoint, allow-list-filtered, under the same key ([ADR 0058](adr/0058-the-pull-starts-a-profiler-and-the-binary-says-whose-code-it-is.md)). The mappings — the executable's path on disk and its build ID — the sample labels, and the full source paths are removed with the redacted frames, so what remains is function names your allow-list admits, base file names and line numbers. Beside it travel the counts of what was redacted, never what | no |
| `goroutine_counts` | Per sampled process and window: how many goroutines it was running at each reading, and the pod, container and build it was read from ([ADR 0080](adr/0080-the-index-page-already-counts-the-goroutines.md)). A count and a timestamp. No stack, no function name and nothing about what those goroutines were doing | no |
Expand Down
25 changes: 23 additions & 2 deletions internal/inventory/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,20 @@ type PortKey struct {
ImageDigest string `json:"image_digest,omitempty"`
}

// Port is one port of one build, as the controller publishes it: what the node
// observed, plus what the controller established about it afterwards. Distinct
// from nodescan.ListeningPort, which is what crossed the channel and carries
// only what a node can know (ADR 0082).
type Port struct {
Port int `json:"port"`
Loopback bool `json:"loopback,omitempty"`
// Pprof is what the prober's one request to this port answered —
// "confirmed", "absent" or "unreachable" — and is empty for a port nothing
// asked about, which every loopback port is. Empty is "not asked", never
// "not pprof" (ADR 0082 §2).
Pprof string `json:"pprof,omitempty"`
}

// PortRecord is what the processes of one workload container, of one build,
// accept connections on — merged across every replica the agent can see.
//
Expand All @@ -185,7 +199,7 @@ type PortKey struct {
type PortRecord struct {
PortKey
// Ports is the union of what the replicas listen on, in port order.
Ports []nodescan.ListeningPort `json:"ports"`
Ports []Port `json:"ports"`
// AssertedAt is the oldest of the contributing nodes' latest reports — the
// bound on how current this record is. A node that stopped reporting keeps
// contributing what it last stated, so without this the record would be
Expand Down Expand Up @@ -915,7 +929,14 @@ func (s *Store) PortSnapshot() []PortRecord {
if len(merged) == 0 {
continue
}
out = append(out, PortRecord{PortKey: pk, Ports: merged, AssertedAt: asserted})
// Merging stays on the node's own type — it is the node's claim being
// reconciled — and the controller's type begins here, where the record
// leaves the store.
ports := make([]Port, len(merged))
for i, m := range merged {
ports[i] = Port{Port: m.Port, Loopback: m.Loopback}
}
out = append(out, PortRecord{PortKey: pk, Ports: ports, AssertedAt: asserted})
}
sort.Slice(out, func(i, j int) bool {
if out[i].Key != out[j].Key {
Expand Down
4 changes: 2 additions & 2 deletions internal/inventory/store_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -684,7 +684,7 @@ func TestPortsMergeAcrossReplicasAndNodes(t *testing.T) {
if len(ports) != 1 {
t.Fatalf("ports = %+v, want one record", ports)
}
want := []nodescan.ListeningPort{{Port: 6060}, {Port: 8080}, {Port: 9090}}
want := []Port{{Port: 6060}, {Port: 8080}, {Port: 9090}}
if !reflect.DeepEqual(ports[0].Ports, want) {
t.Errorf("ports = %+v, want %+v (6060 is reachable on node-2)", ports[0].Ports, want)
}
Expand All @@ -710,7 +710,7 @@ func TestAPortDoesNotOutliveTheProcessThatBoundIt(t *testing.T) {
if len(ports) != 1 {
t.Fatalf("ports = %+v, want one record", ports)
}
if want := []nodescan.ListeningPort{{Port: 8080}}; !reflect.DeepEqual(ports[0].Ports, want) {
if want := []Port{{Port: 8080}}; !reflect.DeepEqual(ports[0].Ports, want) {
t.Errorf("ports = %+v, want %+v (6060 was not in the latest report)", ports[0].Ports, want)
}
}
Expand Down
23 changes: 23 additions & 0 deletions internal/pprofprobe/probe.go
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,29 @@ func (p *Prober) Confirmed(candidates []Candidate) []Candidate {
return out
}

// Stamp writes what the prober established onto each port of each record, and
// returns the records. The answer is about a build's port — the same
// {digest, port} pair the prober asked about — so the join is a lookup, not an
// inference (ADR 0082 §1).
//
// A port nothing asked about is left empty rather than marked absent: Candidates
// drops loopback ports, so "no answer" is the common case and must not read as
// "not pprof".
func (p *Prober) Stamp(records []inventory.PortRecord) []inventory.PortRecord {
p.mu.Lock()
defer p.mu.Unlock()
for _, rec := range records {
for i, port := range rec.Ports {
a, ok := p.answers[Target{ImageDigest: rec.ImageDigest, Port: port.Port}]
if !ok || a.state == StateUnknown {
continue
}
rec.Ports[i].Pprof = a.state.String()
}
}
return records
}

// Coverage is what the prober has established, for the collection-coverage
// payload. Counts only: which workload was asked about is already in the
// inventory beside it, and which was refused is not a name this reports
Expand Down
Loading
Loading