wip! Transient Leader/Primary Divergence - #1707
Draft
pniedzielski wants to merge 1 commit into
Draft
Conversation
pniedzielski
force-pushed
the
leader-primary-transient-divergence
branch
2 times, most recently
from
August 12, 2026 21:08
f99649e to
5a72651
Compare
A leader gets cut off from enough peers to lose quorum. If one of the peers it can no longer reach becomes the new leader, it never learns of it. Should that node then fall silent and the original be elected again, its Partition FSM never left PRIMARY_HEALED, so nothing refreshes the lease id; every request it subsequently sends carries the stale value, which replicas reject. This patch adds an integration test for that sequence, next to the existing test for the case where the old primary does observe the takeover and shuts itself down. The two differ only in whether the old primary keeps a link to its replacement, so they belong side by side. The shape is taken from a chaos run that hit this on a seven-node cluster. There the old leader kept two live peers throughout and still never learned who had replaced it, and the replacement was paused rather than killed -- it was that pause, not a crash, that let the rest of the cluster move on without it. The test asserts that the replacement aborts on waking, which was the sole uninjected event of that run and is what distinguishes this scenario from the adjacent one. Each election outcome is forced by quorum steering rather than left to timing. The old primary keeps a majority quorum while it is cut off: lowering it there would let it self-elect and produce a genuine two-leader split brain instead of the scenario under test. The two followers never meant to lead are given a quorum higher than the cluster size, so neither can win regardless of which links happen to still be up between them and the old primary -- without that, a bystander that keeps a live connection to the old primary can end up winning the race meant for the intended replacement, as it did the first time this test was run. The test is expected to fail until the Partition FSM learns to handle being told it is primary again while it still believes it is primary.
pniedzielski
force-pushed
the
leader-primary-transient-divergence
branch
from
August 12, 2026 21:59
5a72651 to
754d46e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.