Skip to content

feat: the index page already counts the goroutines (ADR 0080) - #117

Merged
kfeofantov merged 1 commit into
mainfrom
feat/goroutine-counts
Sep 12, 2026
Merged

kfeofantov merged 1 commit into
mainfrom
feat/goroutine-counts

Conversation

@kfeofantov

Copy link
Copy Markdown
Contributor

Adds goroutine_counts: a series of goroutine counts per sampled process per
window, read off the /debug/pprof index page the prober already fetches.
A goroutine leak had no sensor at all; this is the one that carries the finding.

Why this shape

Measured first, on Go 1.26 against a process parked at 10⁶ goroutines:

index page /debug/pprof/goroutine
CPU in the workload 117 µs 7.90 s
heap allocated in the workload 0 726 MiB
stop-the-world pauses 0 2, ~87 µs
response 2 604 B 2 489 B

Serving the profile allocates a record per goroutine inside the customer's
process
, ~1.4 KiB each, so a pull can OOM-kill the workload most likely to be
leaking. The index page already renders runtime.NumGoroutine() and costs a
variable read.

A leak is a trend, not a value: the count answers whether, stacks answer
where, and only the first is needed to have a finding. So the stacks are a
separate slice with its own preconditions, written down in
docs/plans/goroutine-leak-sensor.md.

What changed

  • No new path, privilege or annotation. /debug/pprof/goroutine is never
    fetched, in any debug mode — debug=2 dumps every goroutine with its
    function arguments and is in the class of /debug/pprof/cmdline.
  • Rides on profiling.pprof.enabled, not pprof.pull: it repeats the request
    that switch already describes and starts no profiler. What does change is
    frequency — once per image and port becomes once a minute per process — and
    security.md §10.3 states it rather than folding it in.
  • A fifth journal-shaped window kind: supersedes within its window, resumes at
    startup (ADR 0077), states finality with captured_at (ADR 0078). ADR 0077's
    consequences predicted a fifth kind would be a case in a switch; it was.
  • The sample names the pod, the one place the pprof funnel does. A series that
    could not would read a pod replacement — a new process starting near zero — as
    a leak that cured itself.
  • A failed reading rests its target for five minutes, so a few dead endpoints at
    the front of a deterministic order cannot spend the round and starve the tail.
  • collection_coverage gains a block separating unreachable from
    unreadable; metrics gain pprof_count_readings_total.
  • The contract gains the obligation that matters: these samples are gauges, and
    adding them up is the wrong ingest.

Evidence

make test (race), make lint, make proto-lint, make chart-lint — clean;
the three gosec findings pre-exist on main. The committed tree was re-extracted
with git archive HEAD and built and tested from scratch.

The parser is tested against the page Go itself renders, and makes its claim by
moving the number: 2 000 goroutines are parked between two readings and the count
has to rise by them.

Not done: the e2e suite confirms a pprof_profile for its Go sample but
asserts nothing about goroutine_counts, though the sample serves the same index
page. Noted in the plan file.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KeZECJXTDHL6Nts95eKMJA

Adds `goroutine_counts`: a series of goroutine counts per sampled process per
window, read off the `/debug/pprof` index page the prober already fetches. It
is what makes a goroutine leak visible, which the agent had no sensor for.

The shape follows a measurement. Serving `/debug/pprof/goroutine` allocates
~1.4 KiB of transient heap per goroutine inside the customer's process — 726
MiB at a million goroutines — so a pull can OOM-kill the workload most likely
to be leaking. The index page carries `runtime.NumGoroutine()` already, and
reading it costs 134 µs, zero allocations and zero stop-the-world pauses at the
same million. A leak is a trend, so the cheap reading carries the finding and
the stacks are a separate slice (docs/plans/goroutine-leak-sensor.md).

- No new path, privilege or annotation. `/debug/pprof/goroutine` is never
  fetched, in any debug mode.
- Rides on `pprof.enabled`, not `pprof.pull`: it starts no profiler.
- Fifth journal-shaped window kind — supersedes within its window, resumes at
  startup (ADR 0077), states finality with `captured_at` (ADR 0078). ADR 0077
  predicted a fifth kind would be a case in a switch; it was.
- The sample names the pod, the one place the pprof funnel does: a series that
  could not would read a pod replacement as a leak that cured itself.
- A failed reading rests its target for five minutes, so dead endpoints cannot
  spend the round and starve the tail of the list.

Evidence: `make test` (race), `make lint` (3 gosec findings pre-existing on
main), `make proto-lint`, `make chart-lint` — all clean.
@kfeofantov
kfeofantov merged commit 7362f01 into main Sep 12, 2026
4 checks passed
@kfeofantov
kfeofantov deleted the feat/goroutine-counts branch September 12, 2026 08:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant