Repository navigation
feat: the index page already counts the goroutines (ADR 0080) - #117
Merged
Merged
Conversation
Adds `goroutine_counts`: a series of goroutine counts per sampled process per window, read off the `/debug/pprof` index page the prober already fetches. It is what makes a goroutine leak visible, which the agent had no sensor for. The shape follows a measurement. Serving `/debug/pprof/goroutine` allocates ~1.4 KiB of transient heap per goroutine inside the customer's process — 726 MiB at a million goroutines — so a pull can OOM-kill the workload most likely to be leaking. The index page carries `runtime.NumGoroutine()` already, and reading it costs 134 µs, zero allocations and zero stop-the-world pauses at the same million. A leak is a trend, so the cheap reading carries the finding and the stacks are a separate slice (docs/plans/goroutine-leak-sensor.md). - No new path, privilege or annotation. `/debug/pprof/goroutine` is never fetched, in any debug mode. - Rides on `pprof.enabled`, not `pprof.pull`: it starts no profiler. - Fifth journal-shaped window kind — supersedes within its window, resumes at startup (ADR 0077), states finality with `captured_at` (ADR 0078). ADR 0077 predicted a fifth kind would be a case in a switch; it was. - The sample names the pod, the one place the pprof funnel does: a series that could not would read a pod replacement as a leak that cured itself. - A failed reading rests its target for five minutes, so dead endpoints cannot spend the round and starve the tail of the list. Evidence: `make test` (race), `make lint` (3 gosec findings pre-existing on main), `make proto-lint`, `make chart-lint` — all clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
goroutine_counts: a series of goroutine counts per sampled process perwindow, read off the
/debug/pprofindex page the prober already fetches.A goroutine leak had no sensor at all; this is the one that carries the finding.
Why this shape
Measured first, on Go 1.26 against a process parked at 10⁶ goroutines:
/debug/pprof/goroutineServing the profile allocates a record per goroutine inside the customer's
process, ~1.4 KiB each, so a pull can OOM-kill the workload most likely to be
leaking. The index page already renders
runtime.NumGoroutine()and costs avariable read.
A leak is a trend, not a value: the count answers whether, stacks answer
where, and only the first is needed to have a finding. So the stacks are a
separate slice with its own preconditions, written down in
docs/plans/goroutine-leak-sensor.md.What changed
/debug/pprof/goroutineis neverfetched, in any
debugmode —debug=2dumps every goroutine with itsfunction arguments and is in the class of
/debug/pprof/cmdline.profiling.pprof.enabled, notpprof.pull: it repeats the requestthat switch already describes and starts no profiler. What does change is
frequency — once per image and port becomes once a minute per process — and
security.md§10.3 states it rather than folding it in.startup (ADR 0077), states finality with
captured_at(ADR 0078). ADR 0077'sconsequences predicted a fifth kind would be a case in a switch; it was.
could not would read a pod replacement — a new process starting near zero — as
a leak that cured itself.
the front of a deterministic order cannot spend the round and starve the tail.
collection_coveragegains a block separatingunreachablefromunreadable; metrics gainpprof_count_readings_total.adding them up is the wrong ingest.
Evidence
make test(race),make lint,make proto-lint,make chart-lint— clean;the three gosec findings pre-exist on
main. The committed tree was re-extractedwith
git archive HEADand built and tested from scratch.The parser is tested against the page Go itself renders, and makes its claim by
moving the number: 2 000 goroutines are parked between two readings and the count
has to rise by them.
Not done: the e2e suite confirms a
pprof_profilefor its Go sample butasserts nothing about
goroutine_counts, though the sample serves the same indexpage. Noted in the plan file.
🤖 Generated with Claude Code
https://claude.ai/code/session_01KeZECJXTDHL6Nts95eKMJA