Skip to content

feat: the pull starts a profiler, and the binary says whose code it is (ADR 0058) - #93

Merged
kfeofantov merged 1 commit into
mainfrom
feat/the-pull-starts-a-profiler-and-says-so
Aug 30, 2026
Merged

kfeofantov merged 1 commit into
mainfrom
feat/the-pull-starts-a-profiler-and-says-so

Conversation

@kfeofantov

Copy link
Copy Markdown
Contributor

Slices 4 and 5 of the pprof plan, in one PR — pulling without filtering cannot ship anything, and filtering without pulling has nothing to filter.

The defect this found

The symbol filter classifies a frame by module path: no domain means stdlib or unpublished code and is kept, an allow-listed prefix is kept, everything else is third-party and redacted. A customer's own service lives under github.com/acme/web, which has a domain and is not on an empty allow-list — the chart's default.

So the shipped behaviour would have been: pull a profile, redact the entire service layer, keep main and the standard library, throw the result away as unshippable. Cost to the customer, nothing in return.

The fix is that the binary already says which code is theirs. go_build carries each build's main module and marks every dependency a replace directive redirected — the first is the service, the second is what a monorepo builds from source rather than consumes. Both are read from the binary, where no configured list can be wrong about them, and both are now admitted without configuration. The configured list adds to that, for shared internal modules a build merely requires.

This is not a widening of what may leave: those frames are the customer's own code, which the allow-list exists to keep. It just had no way to know them.

What the cluster pays

Bounded by construction rather than by a knob: 10 seconds per capture, one at a time cluster-wide, at most ten per five-minute round. Workloads are visited least-recently-profiled first — round-robin, not ranked by consumption, because a ranking profiles the same few services forever and the finding a report most often needs is about one nobody suspected.

Colliding with the customer's own profiler

Go allows one CPU profile per process. A non-200 is nearly always the process declining to start a second one, which means the customer runs their own. The target is held for six hours and never fought over.

I dropped the proactive deny-list of known profiler modules I'd proposed: it guesses at what one harmless attempt establishes, and the hold already converges after a single failure. The uncovered case is narrow — their profiler idle exactly when we asked — and it is accepted in the ADR rather than papered over.

What the filter removes beyond frames

A pulled profile carries fields a captured one never had: mappings (the executable's path on disk and its build ID), sample labels (arbitrary strings the profiled service attached), comments, and full source paths. All go with the redacted frames, and the output is rebuilt from what survived — otherwise a redacted name would remain in a function table nothing references.

One thing I want visible

Filtering moves into the controller for this path, and that is a real weakening. For a captured profile ADR 0011 §4's guarantee is physical: raw stacks exist only in the node's memory, and the filter runs before anything crosses. A pulled profile arrives already assembled, so the controller holds unreduced stacks for the milliseconds between read and filter. Nothing unreduced leaves the cluster either way, but "not collected" became "collected and dropped" for one path, and security.md §10.3 says so rather than hiding it.

Known gap, stated in the ADR

The same allow-list defect is still open for eBPF captures: the node filters against the configured list alone, so a cluster running five services with one prefix configured redacts four of them. The chart's refusal to install ebpf with an empty list bounds the worst case but does not close it. The node knows each binary's modules from its own scan — wiring that in is separate work.

Evidence

  • TestAPulledProfileCarriesNoIdentityOutsideTheAllowList filters a real profile captured from the test process and walks the decoded output: no function, filename, mapping or label outside the allow-list survives.
  • TestWithoutTheBuildsModulesThereIsNothingToShip is the defect above, pinned.
  • go test ./... and golangci-lint run ./... pass on host and GOOS=linux, 0 issues.
  • e2e not run locally (Docker is yours). checkProfilePulled asserts a filtered profile reaches the spool with the chart's allow-list left empty, so the frames that survive can only have survived via the binary's own modules.

Ceilings

security.md 996 → 1030 and backend-requirements.md 575 → 595, for a new payload kind and the first action the agent takes that a workload can notice.

Note: ADR 0048's body cites security.md §10.4, so the new material was folded into §10.3 rather than inserted as a section, to avoid making a shipped ADR's reference stale.

🤖 Generated with Claude Code

https://claude.ai/code/session_011YRKuL97KRndaopfpixrhp

…s (ADR 0058)

Adds the payload kind pprof_profile: a ten-second CPU profile fetched from an
endpoint #92 confirmed, reduced to allowed frames in the controller.

The defect found while building it, and the reason slices 4 and 5 are one PR:
the symbol filter classifies a frame by module path, so a customer's own
service under github.com/acme/web is "third-party" against an empty allow-list
— the chart's default. The shipped behaviour would have been to pull a
profile, redact the entire service layer, and throw the result away. Now the
filter also admits the modules the build states it was compiled from: its main
module, and every dependency a `replace` redirects, both read from the binary
where no configured list can be wrong about them. A default install produces a
usable profile.

- Bounded by construction: 10s per capture, one at a time, ≤10 per 5-minute
  round. Round-robin over least-recently-profiled, not ranked by consumption —
  a ranking never sees the tail.
- A non-200 is the process declining to start a second profiler, which means
  the customer runs their own. Held for 6h, never fought. No proactive list of
  profiler modules: it guesses at what one harmless attempt establishes.
- Mappings (executable path, build ID), sample labels, comments and full source
  paths are removed with the redacted frames; the output is rebuilt from what
  survived, so no redacted name is left in an unreferenced table.
- Filtering moves into the controller for this path. That is a real weakening
  of ADR 0011 §4's physical guarantee and security.md §10.3 says so.

go test ./... and golangci-lint run ./... (host and GOOS=linux) — pass,
0 issues. e2e not run locally; checkProfilePulled asserts a filtered profile
reaches the spool with the chart's allow-list left empty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011YRKuL97KRndaopfpixrhp
@kfeofantov
kfeofantov merged commit 1a64d97 into main Aug 30, 2026
3 checks passed
@kfeofantov
kfeofantov deleted the feat/the-pull-starts-a-profiler-and-says-so branch August 30, 2026 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant