Partition unassigned pod pool by hashring

[scott-rc]

Eliminates the inter-controller race in assignPod that drove the May 12 production incident -- bimodal GetInstance p99 (0.06ms p50, 3-7 min p99) and a 40:1 patch-failure ratio under cohort-transition contention.

Design

Each unassigned pod hashes (by name) to one controller in the existing hashring, so getUnassignedPods returns only pods this controller owns. Non-overlapping partitions mean no two controllers attempt to patch the same K8s object under steady state. An empty-partition fallback returns the unfiltered pool when this controller has zero owned candidates (cohort-transition window, unlucky hash distribution, deployment pool smaller than the controller fleet) so the function-responsible controller doesn't starve.

assignPod now uses server-side apply with a controller-IP-suffixed field manager. A real apiserver Conflict during a ring-rebalance window means a peer's field manager already owns the pod; assignPod treats it as a retry-pick signal and loops back to a different candidate after a short jittered backoff. ApplyOptions.Force stays unset -- forcing would steal the peer's assignment and defeat the partition guarantee, and retrying the same apply without it is futile, so a loop-back onto another pod is the only correct resolution. The JSON-patch test-op + goto GET_UNASSIGNED_POD retry loop is gone.

Two concurrent assignPod calls inside the same controller can pick the same pod via getUnassignedPod's random selection. An in-process per-pod reservation (xsync.Map[string, struct{}] keyed by namespace/podName) serializes the SSA path -- the second caller loops back to getUnassignedPod instead of running a same-manager SSA that would silently rewrite the first caller's work. The reservation release defers before the on-error deletePod, so the K8s side is clean before another goroutine can pick the same pod again.

Oneshot getReadyInstance forwards to the function-responsible controller when this controller is not ring.Get(fn), mirroring scale's forwarding pattern. Without forwarding, the entry-point controller's partition is empty for the oneshot's function on every call and the fallback runs every time.

Bench

The multi-controller BenchmarkAssignPodContention (added in #144) gains an applies/success metric that counts SSA Apply actions per successful assign. 1.000 proves each assign runs exactly one SSA -- no retryPick loop-back and no conflict-driven outer retry on this fake. The bench pins GOMAXPROCS to numControllers so b.RunParallel uses one goroutine per controller, matching the production-shaped contention scenario.

The contention-removal benefit itself cannot be measured on fake.NewClientset: all patches serialize through the fake's single tracker mutex regardless of design, and managedFieldObjectTracker (k8s 1.30+) makes per-call SSA cost higher than per-call JSON-patch cost on the fake. The May 12 production data remains the real evidence per the design doc.

Tests

Single-controller correctness:

• TestPodHashKey_StableHash, TestPodOwnedByMe_*, TestPodPartition_StableDeterministicMapping -- partition correctness.
• TestGetUnassignedPods_FiltersToOwned, TestGetUnassignedPods_FallsBackWhenPartitionEmpty -- filter behavior including the empty-partition fallback.
• TestAssignPod_IntraControllerReservation -- 10 goroutines, 10 pods, asserts every pod is assigned exactly once. Verified red without the reservation by toggling its LoadOrStore check.
• TestAssignPod_RecoversFromSSAConflict -- a reactor injects synthetic field-manager Conflict responses; asserts assignPod loops onto a different candidate instead of exhausting a futile same-apply retry budget.
• TestAssignPodRejectsAlreadyAssigned updated to assert the unassigned-pool filter excludes already-assigned pods (no global selector swap, parallelizes cleanly).

Multi-controller correctness:

• TestMultiControllerContention -- N controllers race on a shared unassigned pool, asserts distinct pods, no errors, and applies/success == 1 (one SSA per success, no retries).
• TestRingRebalanceReclaim -- a controller leaves the ring from the survivor's perspective, the survivor's partition grows, and the survivor can assign one of the orphan-share pods with no double-assign.
• TestOneshotForwarding -- a non-responsible controller forwards the oneshot GetInstance to the responsible peer instead of running assignPod locally.

Stacked on #144 (multi-controller bench harness), which is stacked on #143 (router in-flight counter fix).

[scott-rc]

This change is part of the following stack:

• Decouple router in-flight counter from queued requests #143
  • Add multi-controller assignPod benchmark and fixture #144
    • Partition unassigned pod pool by hashring #145 ◀

_{Change managed by git-spice.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2f3e095. Configure here.}

[cursor]

Oneshot panics empty hash ring

High Severity

The new ring.Get call in oneshot getReadyInstance panics on an empty ring. This prevents assignPod from using its "own everything" fallback during boot or ring transitions, leading to crashes instead of local assignment.

 

^{Reviewed by Cursor Bugbot for commit 2f3e095. Configure here.}

[cursor]

SSA Invalid no longer retries

Medium Severity

The change to server-side apply altered pod assignment retry logic. Previously, invalid patches or failed tests would prompt picking another pod. Now, only API server conflicts cause a retry with a new candidate. Other failures, such as a missing ReplicaSet owner or an invalid apply, are now terminal, potentially blocking function scale-up if a problematic pod is chosen.

Additional Locations (1)

• internal/controller/pod.go#L133-L136

 

^{Reviewed by Cursor Bugbot for commit 2f3e095. Configure here.}