Repository navigation
fix: a kubelet read is bounded, and one node's silence costs one node (ADR 0045) - #78
Merged
Merged
Conversation
… (ADR 0045) The two kubelet reads had no deadline and no size limit, and the nodes were read one after another, so a kubelet that accepted the connection and then went quiet held the whole poll — and the snapshot flush behind it. - 10 s deadline per read; 8 MiB for /stats/summary (buffered) and 64 MiB for /metrics/cadvisor (streamed). Passing a limit is a failed read, never a short one: a truncated exposition decodes as a node running fewer containers. - Nodes are read eight at a time. Only the reading — ingestion stays in the polling goroutine, so the accumulator keeps one owner and no mutex. - The API client's rate limit is stated (50 QPS / burst 100) instead of taking client-go's 5, which one poll of a 75-node cluster already exceeds and which would have consumed the deadline above. Measured on kind 1.36.1, one node, 9 → 49 pods: /stats/summary 44,367 → 167,654 B, /metrics/cadvisor 687,903 → 3,076,138 B. Negative checks, each failing with its own message: without the deadline the poll still hangs at 10 s; read serially the six-node poll takes 2.41 s against a 1.2 s bound; an io.LimitReader on a cut exposition returns 1 sample and nil. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The two kubelet reads (
/stats/summary,/metrics/cadvisor) had no deadlineand no size limit, and the nodes were read one after another in the goroutine
that also runs the snapshot flush. A kubelet that accepts the connection and
then goes quiet held the entire poll: client-go bounds the dial and the TLS
handshake, not the response, and the API server proxies silence as readily as
data.
already counted independently.
8 MiB for the summary (unmarshalled from a buffer, up to 8 in flight) and
64 MiB for the exposition (decoded off the wire). A truncated response is a
valid smaller response — a node that appears to run fewer containers.
goroutine, so the accumulator and the counter baselines keep one owner and no
mutex.
default of 5 QPS is already less than one poll of a 75-node cluster needs, and
the limiter waits inside the request's own context — the deadline above would
have fired on our own throttling.
Measured on kind 1.36.1, one node, by adding 40 pause pods to a 9-pod node:
/stats/summary/metrics/cadvisor~3.1 KB and ~60 KB per pod; at the kubelet's 110-pod ceiling that is ~355 KB and
~6.7 MB. The limits are ~23× and ~10× those.
Each guard was checked by making it fail:
Nothing collected, stored or transmitted changes, so
docs/security.mdisuntouched; the reasoning is in ADR 0045, which amends ADR 0006 §1.
go build ./...,go test -race -count=1 ./...,make lint(0 issues),go vet -tags e2e ./...,gofmt— all green.🤖 Generated with Claude Code