Repository navigation
fix: the node scans only pods the controller says passed the filters (ADR 0015) - #41
Merged
Merged
Conversation
…(ADR 0015) docs/security.md §10.2 promises customers that node-level samples from processes whose module path *or namespace* is not allow-listed are discarded on the node, before transport. For the eBPF profiler that was true. For the Go-binary scanner it was not: it walked every PID, dropped only infra module paths, and shipped every remaining binary's module path, dependencies, pod UID and container ID to the controller, where the namespace filters were applied for the first time — at the join. A customer filtering the agent to two namespaces still had every Go process on every node leave the node. The node cannot fix this alone: zero API access by construction (ADR 0009), and a cgroup yields a pod UID, never a namespace. So it asks. A new /v1/node-scope endpoint answers the querying node with the UIDs of the pods on it that passed the filters — PodWatcher's admitted index itself, so the controller cannot name an excluded pod, it is answering from the filtered set rather than filtering at answer time. Three properties, each tested: - Fail closed. No endpoint, or an unreachable controller, means the node scans nothing. Scanner.Scan now requires a Scope and the zero Scope admits nothing, so no call site can scan unscoped by omission. A lost pass costs nothing; the next re-scans (loss-harmless, ADR 0003). - The cgroup is read before the executable. An out-of-scope process has its binary never opened and its module path never extracted — not collected, as opposed to collected and dropped (invariant 4). Only an aggregate filtered_scope counter moves (invariant 6). - A process in no pod — kubelet, a systemd unit — is never in scope, because no namespace filter can permit what is in no namespace. Those were previously reported with an empty pod UID and dropped at the join. Not a control channel (invariant 1): the external backend is uninvolved, the reply is identifiers derived from the operator's own Helm-values filters, and it can only narrow what the node does, never widen it. The node e2e could no longer assert "the scanner finds this Go workload" without a controller, so it became the fail-closed e2e — the security property proven against the real image and manifest, with a real Go workload running and still not reported. The positive path is TestGoInventoryEndToEnd, which deploys a controller and asserts on the payload. Also fixes a compile break the e2e package had carried since #39: the tagged tests are never built by `go test ./...`, so a changed signature rotted there unnoticed. `make test` and CI now run `go vet -tags e2e ./...`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The discrepancy
docs/security.md§10.2 promises customers, in public:For the eBPF profiler that was true (targeting is deny-by-default and controller-scoped, ADR 0011 §3). For the Go-binary scanner it was not. It walked every PID on the node, dropped only main modules on the infrastructure deny-list, and shipped every remaining binary's module path, dependency list, pod UID and container ID to the controller — where the namespace filters were applied for the first time, at the join.
A customer who filtered the agent down to two namespaces still had the module paths of every Go process on every node leave the node. No payload was wrong; the controller discarded them. But the promise was about the node, and the identities had already left it. Invariant 4 inverted: collected and dropped, where the document said not collected.
Why the node cannot fix this alone
Zero Kubernetes API access by construction (ADR 0009: no RBAC, default token not mounted), and its only attribution source is the process cgroup — which yields a pod UID and a container ID, never a namespace. "This process is in a namespace you allow-listed" is a claim the node has no way to evaluate.
The fix
A new node→controller endpoint,
/v1/node-scope, answers the querying node with the UIDs of the pods on it that passed the customer's filters. Same port, same projected-token audience, same NetworkPolicy, node-initiated like every other endpoint.The answer is
PodWatcher's admitted index itself — the set the namespace filters and opt-out annotations already produce. The controller cannot name an excluded pod, because it is answering from the filtered set rather than applying a filter at answer time.Three properties, each with a test:
Scanner.Scannow requires aScopeand the zeroScopeadmits nothing, so no call site can scan unscoped by omission. A lost pass costs nothing: the next one re-scans (loss-harmless, ADR 0003). Failing open would mean a controller outage silently reverting the agent to scanning the whole node — the exact behavior this removes.filtered_scopecounter moves, never an identity. The e2e assertsunreadable == 0andfiltered_infra == 0under a denying scope — observable proof that no executable was touched.Invariant 1 — not a control channel
Same reasoning as the targets query (ADR 0011 §3), stated explicitly in ADR 0015: the external backend is uninvolved at every point; the reply is pod identifiers derived from the operator's own Helm-values filters applied to the live cluster, never configuration or a command; and the reply can only narrow what the node does. A rogue or broken controller can withhold pods — the node then scans less. It cannot name an excluded pod, and it cannot make the node scan anything it would not otherwise scan.
The e2e had to change, and became better for it
The node role no longer works standalone: without a controller it scans nothing. That is the honest consequence of the promise, but it means
TestNodeScannerAgainstRealCluster— "deploy the DaemonSet, assert it finds this Go workload from its log" — could not pass as written.It is now
TestNodeScannerFailsClosedWithoutScope: same real image, same shipped manifest, same Go workload actually running on the node, asserting it is not reported. The security property proven end to end, parallel to the existing eBPF gate-refusal e2e.The positive path was never uniquely covered there anyway —
TestGoInventoryEndToEnddeploys a controller and asserts the module path, Go version, image digest and workload chain in the spooled payload, which is strictly stronger than a log line.Drive-by: the e2e package had been uncompilable since #39
The e2e tests sit behind a build tag, so
go test ./...never compiles them, and slice B'sWriteUsageSnapshotsignature change rotted there unnoticed on main. Fixed, and prevented from recurring:make testand CI now rungo vet -tags e2e ./....Docs
docs/security.md§10.2 — the promise is now true as written, with how the node learns the namespaces, the fail-closed behavior, and the no-pod case stated plainly.docs/security.md§5 and §8 — the scope query in the network table, and scanning scope as the first bullet of on-node binary scanning.Tests
internal/nodescan— classification under a scope; a denying scope keepsunreadableat zero even with a non-executable staged, proving the exe was never opened;Scopesemantics including the empty pod UID.internal/nodeintake— the handler returns the querying node's admitted pods, empty for a node with none, and rejects bad tokens / missing node / unknown fields / GET.internal/collector— an opted-out pod never appears in the scope, and the scope is per node.cmd/agent— the client round-trip; and three wiring tests throughrunNode: no endpoint scans nothing and says why, the controller's scope reaches the scanner, and a failing query fails closed.make test(go test -race ./...plus the new e2e type-check) andmake lintgreen.🤖 Generated with Claude Code