Conversation
A request entering the router incremented state.inFlight before ctrl.Instance() returned, so requests waiting for an instance counted as serving load. Under slow controller-side assignment, queue-inflated heartbeats reported false demand, the controller scaled up further, and the resulting contention slowed assignment more -- a feedback loop that turned a small slow-path event into a cluster-wide 5xx storm. The counter now reflects only requests being proxied to an instance, removing the loop's amplifier. An inFlightTracker plumbed via context idempotently marks the request once Instance() succeeds and is released in ServeHTTP's defer. The requests_in_flight metric's Help string is updated to match. Tests cover the queued-request invariant, retry idempotency, and the per-request touch goroutine's keepalive role across the heartbeat GC threshold.
Contributor
Author
|
This change is part of the following stack: Change managed by git-spice. |
This was referenced May 14, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The router's
requests_in_flightcounter incremented at the start ofServeHTTP, beforectrl.Instance()had returned an instance. Requests waiting for assignment counted as serving load. When controller-side assignment slowed (e.g. cohort transition during a deploy), queue-inflated heartbeats reported false demand, the controller scaled up further, and the resulting contention slowed assignment more -- a positive feedback loop that turned isolated stalls into a cluster-wide 5xx storm.What
The counter now reflects only requests being proxied to an instance. An
inFlightTrackerplumbed via context marks the request onceInstance()succeeds and is released inServeHTTP's defer. Mark and release are idempotent (atomic CAS) so retries do not double-count. Therequests_in_flightmetric's Help string is updated to match.What this does NOT fix
The underlying bottleneck -- multiple controllers racing on K8s JSON-patch operations against the shared unassigned-pod pool, which stalls
GetInstancecalls under contention -- remains. This change removes the amplifier that turns isolated stalls into cluster-wide events; a follow-up branch will address the contention directly.Tests
TestHeartbeatInFlightExcludesRequestsWaitingForInstancepins the queued-not-served invariant.TestHeartbeatInFlightIdempotentAcrossRetriescovers the CAS guard acrossInstance()retries.TestHeartbeatStateSurvivesLongInFlightRequestexercises the per-request touch goroutine across the heartbeat GC threshold.