fix(chart): replace the networks pod instead of rolling it - #8
Merged
Conversation
Upgrading the service deadlocked. It binds a durable NATS consumer and NATS permits one binding at a time, so the rolling update's extra pod started, failed with subscribe group deleted events: consumer is already bound to a subscription and never became ready — which meant the old pod was never retired, so the new one could never succeed. The rollout sat there indefinitely with the new pod in CrashLoopBackOff. Recreate takes the old pod down first. That costs a few seconds of downtime on upgrade, which is the honest trade for a service that cannot run two copies at once. Confirmed against a live cluster: with RollingUpdate the new pod crash-looped through five restarts and the rollout never finished; patching the deployment to Recreate let it complete immediately. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upgrading this service deadlocks. It binds a durable NATS consumer, and NATS permits exactly one binding at a time. A rolling update starts the new pod alongside the old one, so the new pod dies with
subscribe group deleted events: consumer is already bound to a subscription, never becomes ready, and therefore the old pod is never retired — leaving the rollout stuck with the new pod in CrashLoopBackOff.Recreatetakes the old pod down first. That costs a few seconds of downtime per upgrade, which is the honest trade for a service that cannot run two copies concurrently. (replicaCountis already 1; the defaultmaxSurge: 25%rounds up to one extra pod, which is enough to trigger it.)Confirmed against a live cluster: with RollingUpdate the new pod crash-looped through five restarts and the rollout never completed; patching the deployment to
Recreatelet it finish immediately, and the service now runs the published multi-arch0.1.1image and serves.