Skip to content

SOLR-17820: Fix CLUSTERSTATUS health reporting GREEN with down replicas - #5046

Merged
epugh merged 4 commits into
apache:mainfrom
epugh:SOLR-17820
Oct 8, 2026
Merged

epugh merged 4 commits into
apache:mainfrom
epugh:SOLR-17820

Conversation

@epugh

@epugh epugh commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • health was computed from docCollection before crossCheckReplicaStateWithLiveNodes corrected replicas whose state.json entry was still "active" despite their node actually being dead - so a shard could report health=GREEN while the very same response showed one of its replicas as "down".
  • Moved the cross-check to run before health is computed, so health reflects the same (corrected) replica states shown elsewhere in the response.
  • Added ClusterStatusTest reproducing the dead-node scenario directly against crossCheckReplicaStateWithLiveNodes + postProcessCollectionJSON, asserting ORANGE (not GREEN) once one of two replicas is down.

Test plan

  • New unit test ClusterStatusTest#testHealthReflectsReplicaOnDeadNode passes, and fails with expected:<ORANGE> but was:<GREEN> if the call order is reverted.
  • Reproduced manually against a live 3-node cluster: killing a node showed the affected shard/collection health flip from GREEN to the correct degraded color.

https://issues.apache.org/jira/browse/SOLR-17820

… a replica on a dead node

health was computed from docCollection before crossCheckReplicaStateWithLiveNodes
corrected replicas whose state.json entry was still "active" despite their node
being dead - so a shard could report health=GREEN while the same response showed
one of its replicas as "down". Moved the cross-check to run before health is
computed, and added a unit test reproducing the dead-node scenario directly.
@github-actions github-actions Bot added the tests label Oct 6, 2026
@epugh epugh added this to the 10.x milestone Oct 6, 2026
@epugh

epugh commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

I also testing this by hand....

import org.junit.Test;

/** Tests that cluster health reflects replica states corrected against live nodes. */
public class ClusterStatusTest extends SolrTestCaseJ4 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SolrTestCase ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you! I also updated our Agents.md with a stronger phraseing.

@epugh
epugh merged commit 7e3b175 into apache:main Oct 8, 2026
4 of 5 checks passed
dsmiley pushed a commit that referenced this pull request Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants