Skip to content

fix: the node scans only pods the controller says passed the filters (ADR 0015) - #41

Merged
kfeofantov merged 1 commit into
mainfrom
fix/nodescan-namespace-scope
Aug 20, 2026
Merged

kfeofantov merged 1 commit into
mainfrom
fix/nodescan-namespace-scope

Conversation

@kfeofantov

Copy link
Copy Markdown
Contributor

The discrepancy

docs/security.md §10.2 promises customers, in public:

samples from processes whose module path or namespace is not allow-listed are discarded on the node, before transport to the controller

For the eBPF profiler that was true (targeting is deny-by-default and controller-scoped, ADR 0011 §3). For the Go-binary scanner it was not. It walked every PID on the node, dropped only main modules on the infrastructure deny-list, and shipped every remaining binary's module path, dependency list, pod UID and container ID to the controller — where the namespace filters were applied for the first time, at the join.

A customer who filtered the agent down to two namespaces still had the module paths of every Go process on every node leave the node. No payload was wrong; the controller discarded them. But the promise was about the node, and the identities had already left it. Invariant 4 inverted: collected and dropped, where the document said not collected.

Why the node cannot fix this alone

Zero Kubernetes API access by construction (ADR 0009: no RBAC, default token not mounted), and its only attribution source is the process cgroup — which yields a pod UID and a container ID, never a namespace. "This process is in a namespace you allow-listed" is a claim the node has no way to evaluate.

The fix

A new node→controller endpoint, /v1/node-scope, answers the querying node with the UIDs of the pods on it that passed the customer's filters. Same port, same projected-token audience, same NetworkPolicy, node-initiated like every other endpoint.

The answer is PodWatcher's admitted index itself — the set the namespace filters and opt-out annotations already produce. The controller cannot name an excluded pod, because it is answering from the filtered set rather than applying a filter at answer time.

Three properties, each with a test:

  1. Fail closed. No endpoint configured, or a controller that cannot be reached → the node scans nothing. Scanner.Scan now requires a Scope and the zero Scope admits nothing, so no call site can scan unscoped by omission. A lost pass costs nothing: the next one re-scans (loss-harmless, ADR 0003). Failing open would mean a controller outage silently reverting the agent to scanning the whole node — the exact behavior this removes.
  2. The cgroup is read before the executable. An out-of-scope process has its binary never opened and its module path never extracted. Only an aggregate filtered_scope counter moves, never an identity. The e2e asserts unreadable == 0 and filtered_infra == 0 under a denying scope — observable proof that no executable was touched.
  3. A process in no pod is never in scope. The kubelet, a systemd unit, anything outside the kubepods hierarchy: no namespace filter can permit what is in no namespace. These were previously reported with an empty pod UID and discarded by the controller at the join.

Invariant 1 — not a control channel

Same reasoning as the targets query (ADR 0011 §3), stated explicitly in ADR 0015: the external backend is uninvolved at every point; the reply is pod identifiers derived from the operator's own Helm-values filters applied to the live cluster, never configuration or a command; and the reply can only narrow what the node does. A rogue or broken controller can withhold pods — the node then scans less. It cannot name an excluded pod, and it cannot make the node scan anything it would not otherwise scan.

The e2e had to change, and became better for it

The node role no longer works standalone: without a controller it scans nothing. That is the honest consequence of the promise, but it means TestNodeScannerAgainstRealCluster — "deploy the DaemonSet, assert it finds this Go workload from its log" — could not pass as written.

It is now TestNodeScannerFailsClosedWithoutScope: same real image, same shipped manifest, same Go workload actually running on the node, asserting it is not reported. The security property proven end to end, parallel to the existing eBPF gate-refusal e2e.

The positive path was never uniquely covered there anyway — TestGoInventoryEndToEnd deploys a controller and asserts the module path, Go version, image digest and workload chain in the spooled payload, which is strictly stronger than a log line.

Drive-by: the e2e package had been uncompilable since #39

The e2e tests sit behind a build tag, so go test ./... never compiles them, and slice B's WriteUsageSnapshot signature change rotted there unnoticed on main. Fixed, and prevented from recurring: make test and CI now run go vet -tags e2e ./....

Docs

  • docs/security.md §10.2 — the promise is now true as written, with how the node learns the namespaces, the fail-closed behavior, and the no-pod case stated plainly.
  • docs/security.md §5 and §8 — the scope query in the network table, and scanning scope as the first bullet of on-node binary scanning.
  • ADR 0015 — written from the implementation, including what was given up (the node role no longer works standalone) and why failing open was rejected.

Tests

  • internal/nodescan — classification under a scope; a denying scope keeps unreadable at zero even with a non-executable staged, proving the exe was never opened; Scope semantics including the empty pod UID.
  • internal/nodeintake — the handler returns the querying node's admitted pods, empty for a node with none, and rejects bad tokens / missing node / unknown fields / GET.
  • internal/collector — an opted-out pod never appears in the scope, and the scope is per node.
  • cmd/agent — the client round-trip; and three wiring tests through runNode: no endpoint scans nothing and says why, the controller's scope reaches the scanner, and a failing query fails closed.

make test (go test -race ./... plus the new e2e type-check) and make lint green.

🤖 Generated with Claude Code

…(ADR 0015)

docs/security.md §10.2 promises customers that node-level samples from
processes whose module path *or namespace* is not allow-listed are discarded
on the node, before transport. For the eBPF profiler that was true. For the
Go-binary scanner it was not: it walked every PID, dropped only infra module
paths, and shipped every remaining binary's module path, dependencies, pod
UID and container ID to the controller, where the namespace filters were
applied for the first time — at the join. A customer filtering the agent to
two namespaces still had every Go process on every node leave the node.

The node cannot fix this alone: zero API access by construction (ADR 0009),
and a cgroup yields a pod UID, never a namespace. So it asks. A new
/v1/node-scope endpoint answers the querying node with the UIDs of the pods
on it that passed the filters — PodWatcher's admitted index itself, so the
controller cannot name an excluded pod, it is answering from the filtered
set rather than filtering at answer time.

Three properties, each tested:

- Fail closed. No endpoint, or an unreachable controller, means the node
  scans nothing. Scanner.Scan now requires a Scope and the zero Scope admits
  nothing, so no call site can scan unscoped by omission. A lost pass costs
  nothing; the next re-scans (loss-harmless, ADR 0003).
- The cgroup is read before the executable. An out-of-scope process has its
  binary never opened and its module path never extracted — not collected,
  as opposed to collected and dropped (invariant 4). Only an aggregate
  filtered_scope counter moves (invariant 6).
- A process in no pod — kubelet, a systemd unit — is never in scope, because
  no namespace filter can permit what is in no namespace. Those were
  previously reported with an empty pod UID and dropped at the join.

Not a control channel (invariant 1): the external backend is uninvolved, the
reply is identifiers derived from the operator's own Helm-values filters, and
it can only narrow what the node does, never widen it.

The node e2e could no longer assert "the scanner finds this Go workload"
without a controller, so it became the fail-closed e2e — the security
property proven against the real image and manifest, with a real Go workload
running and still not reported. The positive path is TestGoInventoryEndToEnd,
which deploys a controller and asserts on the payload.

Also fixes a compile break the e2e package had carried since #39: the tagged
tests are never built by `go test ./...`, so a changed signature rotted there
unnoticed. `make test` and CI now run `go vet -tags e2e ./...`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kfeofantov
kfeofantov merged commit 64acd4d into main Aug 20, 2026
3 checks passed
@kfeofantov
kfeofantov deleted the fix/nodescan-namespace-scope branch August 20, 2026 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant