Skip to content

SOLR-17971: Fix REBALANCELEADERS wait when the Overseer is disabled - #5127

Open
sarkaramrit1993 wants to merge 2 commits into
apache:mainfrom
sarkaramrit1993:SOLR-17971-rebalance-leaders-distributed
Open

sarkaramrit1993 wants to merge 2 commits into
apache:mainfrom
sarkaramrit1993:SOLR-17971-rebalance-leaders-distributed

Conversation

@sarkaramrit1993

Copy link
Copy Markdown
Contributor

https://issues.apache.org/jira/browse/SOLR-17971

With the Overseer disabled, REBALANCELEADERS always waited the full maxWaitSeconds (60s by default) and then logged a timeout. The leaders still moved. waitAsyncRequests() only looked for sub-command results in the Overseer completed/failure maps, but in distributed mode those results go to DistributedApiAsyncTracker. Because of that, the ids were also never cleared and piled up under /distributedapi/async/persistent.

Changes:

  • waitAsyncRequests() now checks the store for the current mode, same branching as REQUESTSTATUS in ClusterCommands. In distributed mode, deleteSingleAsyncId() both tells us the task is done and removes it.
  • Overseer path behaves the same, just drops the extra contains() before remove().

Not changed: I went with the tracker directly instead of issuing a REQUESTSTATUS call per id as the JIRA suggests. It's the same lookup without the extra request.

Tests: new testRebalanceLeadersClearsAsyncIds in TestRebalanceLeaders asserts no rebalanceleaders_ ids are left in either store and the call returns before the timeout. Without the fix and the Overseer disabled, it fails with 29 leftover ids. Ran the full class with -Ptests.nightly=true in both modes, plus gradlew check -x test.

With the Overseer disabled, sub-command results go to DistributedApiAsyncTracker,
not the Overseer maps, so the wait loop always ran to maxWaitSeconds and never
cleared the async ids.
@dsmiley

dsmiley commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Looks good!

The fix branches on the mode the same way REQUESTSTATUS does in ClusterCommands

Then maybe the logic should be centralized? (didn't check how elegant this is)

@sarkaramrit1993

Copy link
Copy Markdown
Contributor Author

Good call, done. Both REBALANCELEADERS and DELETESTATUS now go through ZkController.removeStoredAsyncResponse(asyncId). The REQUESTSTATUS lookup and delete-all paths still do their own branching. Nothing else duplicates them, so I left those alone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants