Skip to content

fix(engine): hold new joins while workers wait an epoch for recovery - #1204

Merged
CassOnMars merged 1 commit into
QuilibriumNetwork:v2.1.0.25from
blacks1ne:fix/joins-wait-recovery-backlog
Nov 4, 2026
Merged

CassOnMars merged 1 commit into
QuilibriumNetwork:v2.1.0.25from
blacks1ne:fix/joins-wait-recovery-backlog

Conversation

@blacks1ne

Copy link
Copy Markdown
Contributor

A seat must host its committee session within an epoch or this node withholds its renewal. When a node's workers already wait over an epoch for archive recovery turns, a new seat without its shard's state lapses before it first votes, is left, and its worker joins another shard. Each cycle costs join, confirm and leave messages, adds a pending join to every other prover's plan, and puts another install in the queue it could not get through. Two regular nodes sat in this cycle with one or no worker of 20 and 55 serving.

Propose no new automatic join while a worker has waited an epoch for a recovery turn. The hold ends as waiting seats recover or lapse; landed joins are still decided and manual joins are not held. One line marks each change.

Tests: 22 focused tests pass; the new hold test fails under both mutation controls (never hold, always hold); fmt, check, warning census clean; CI command 6587/6587 on fedd60cd.

A seat must host its session within an epoch or its renewal is
withheld, so behind a recovery queue that long a new seat without its
shard's state lapses before it first votes, is left, and its worker
joins another shard. Each cycle costs join, confirm and leave messages
and a pending join in other provers' plans, and adds an install to the
queue the seat could not get through. Two regular nodes sat in this
cycle with up to 53 and 18 workers waiting up to 6.4 and 4.0 h for two
turns, and none or one of 55 and 20 workers serving.

Propose no new auto join while a worker has waited an epoch for a turn.
It cannot refuse forever: a waiting seat gets its turns or lapses and
leaves the queue, whose size the seats already held bound. Landed joins
are still decided. One line marks each change.
@blacks1ne
blacks1ne marked this pull request as ready for review November 4, 2026 11:45
@CassOnMars
CassOnMars added this pull request to the merge queue Nov 4, 2026
Merged via the queue into QuilibriumNetwork:v2.1.0.25 with commit f32d214 Nov 4, 2026
12 checks passed
@blacks1ne
blacks1ne deleted the fix/joins-wait-recovery-backlog branch November 4, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants