Skip to content

test: the e2e suite asserts the goroutine series, not just the count (ADR 0080) - #118

Merged
kfeofantov merged 1 commit into
mainfrom
test/goroutine-counts-e2e
Sep 12, 2026
Merged

kfeofantov merged 1 commit into
mainfrom
test/goroutine-counts-e2e

Conversation

@kfeofantov

Copy link
Copy Markdown
Contributor

The inventory e2e confirmed a pprof_profile for its Go sample but asserted
nothing about goroutine_counts, though the sample serves the very index page
the reading comes from. Noted as owed in
docs/plans/goroutine-leak-sensor.md; this
pays it.

What it asserts

Two samples is the assertion that matters. One reading proves the parser;
only a second proves what the kind exists for, since no single count is a
finding. It also shows the reading repeats — the part of the promise in
security.md §10.3 that a single fetch cannot demonstrate.

Beside it: the window's kind, source and length, a non-zero captured_at, the
pod name (the one place this funnel names a process rather than a build), the
image digest, and the coverage block reporting the readings that produced the
series with no unreadable answers.

Verified in kind against the merged agent:

goroutine series for goworkload-5bb85d86f8-z6f87: 2 samples, 5 then 8 goroutines
goroutine count coverage: {Sampled:6 Unreachable:0 Unreadable:0}

A pre-existing failure this found

The counters assertion required observed_nanos >= 20s while the test installs
scanInterval: 15s, so it could only pass when a scan pass drifted a third past
its own interval. It fails on main with the same agent image: observed_nanos = 15004472048.

observed_nanos is the wall-clock gap between two readings of one process
(now.Sub(prev.at)), and the two readings sit at different points inside their
scan passes, so the gap jitters either side of the interval — 14.981s and
15.002s, both measured here. The floor is now the installed interval less a
second of slack, from one constant, because two numbers that merely agree drift
(ADR 0073 §5). The target's -timeout goes to 20m to match the test's own 20m
context, which it was silently cutting short.

What is still red, and is not from here

TestGoInventoryEndToEnd also fails at checkOptOutRemovesTheRecord: the
opted-out workload stays in the go_inventory payload past the check's deadline.
This reproduces on main with the same agent image, so it is neither caused
nor unmasked by this PR:

run test tree counters assertion opt-out assertion
this PR branch pass fail
control origin/main fail (20s vs 15s) fail

It wants its own investigation rather than a wider deadline. The two halves are
unit-tested — Retain in internal/inventory/store_test.go, pod-level exclusion
in internal/collector — and UpdateFunc does call dropPod when admission is
withdrawn, so what is uncovered is the composition: pod update → drop from the
index → Retain → the cadence gate → the spool file. Whether the record never
leaves or merely leaves late decides whether this touches the promise in
security.md §11, and that needs controller logs from a run, which I have not
taken. The check's own message still says "after 4 minutes" while its deadline is
nine, so it has been widened once before (#111).

Evidence

make test (race), make lint, make proto-lint, make chart-lint clean; the
three gosec findings pre-exist on main. e2e run in kind on a freshly recreated
cluster, log in test/e2e/logs/.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KeZECJXTDHL6Nts95eKMJA

…(ADR 0080)

The inventory e2e confirmed a `pprof_profile` for its Go sample but asserted
nothing about `goroutine_counts`, though the sample serves the very index page
the reading comes from.

Two samples is the assertion that matters. One reading proves the parser; only a
second proves the thing the kind exists for, since no single count is a finding.
It also proves the reading repeats, which is the part of the promise in
security.md §10.3 that a single fetch cannot show. Beside it: the window's
kind/source/length, the pod name — the one place this funnel names a process —
the image digest, and the coverage block reporting the readings that produced the
series with no unreadable answers.

Verified in kind: two windows, 5 then 8 goroutines, `{Sampled:6 Unreachable:0
Unreadable:0}`.

Also fixes a pre-existing failure of the counters assertion, found by running
it. It required `observed_nanos >= 20s` while the test installs
`scanInterval: 15s`, so it could pass only when a scan pass drifted a third past
its own interval. `observed_nanos` is the wall-clock gap between two readings of
one process, and the readings sit at different points inside their passes, so the
gap jitters either side of the interval — 14.981s and 15.002s, both measured
here. The floor is now the installed interval less a second of slack, from one
constant, because two numbers that merely agree drift (ADR 0073 §5). The target's
timeout goes to 20m to match the test's own context.
@kfeofantov
kfeofantov merged commit 4aa1018 into main Sep 12, 2026
4 checks passed
@kfeofantov
kfeofantov deleted the test/goroutine-counts-e2e branch September 12, 2026 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant