SOLR-18531: Reserve a stopped test runner's port until its next start

[nick-boss-tech]

🤖 AI text below 🤖 (posted on behalf of Nick Shanin)

https://issues.apache.org/jira/browse/SOLR-18531

What happens today

A stopped test node can lose its port to another process before it restarts.

JettySolrRunner gives up its port when it stops and holds nothing in its place. If another process takes that port during the gap, the restart fails with java.net.BindException: Address already in use. The retry only tries the same port, and it stops after 60 seconds. LeaderElectionIntegrationTest.testSimpleSliceLeaderElection hit this once in CI, on an unrelated pull request. No assertion ran.

What this change does

A stopped runner holds its port until it starts again, is closed, or its cluster shuts down while the runner is still in the cluster.

When a runner stops, it binds a socket to its port on 127.0.0.1 and keeps it. The next start on that port closes the socket before Jetty binds the port. The sockets live in a static map, so a new runner for the same port sees them. MiniSolrCloudCluster.shutdown() closes the sockets for the runners that were in its list when shutdown began, after the stops finish, and a runner's close() releases the reservation on its current port, whichever runner holds it (see Limits).

The reservation socket binds with address reuse set. A node that served traffic can leave its port in TIME_WAIT when it stops; without reuse, the reservation bind fails there and the port stays unprotected apart from a log warning, while the held socket still refuses every later bind, with or without reuse, on Linux.

This is a test-framework change only. Behavior changes: a stopped runner keeps its port bound, so nothing else can bind that port on Linux until the node starts on it again, the runner is closed (a runner's close() releases its port), or, for runners still in the cluster's list, the cluster shuts down.

Proof

New tests pin the reservation down, and the existing restart suites still pass at this head.

New tests in TestJettySolrRunner: testStoppedRunnerKeepsItsPortUntilRestart and testStoppedRunnerThatServedTrafficKeepsItsPortUntilRestart. The first checks that a stopped runner's port stays reserved until a restart on it, and that close() gives the port back. The second serves a real request before the stop, then checks that outside binds fail with address reuse off and on, and that a restart on the same port serves again.

With the two framework source files reverted to the base commit and the new tests kept, both reservation tests fail on their reservation checks: 4 tests run, 2 failures, each the reservation assertion (a foreign socket binds the port that a stopped runner released). At this head, TestJettySolrRunner passes 4 of 4. Verified 2026-10-10 at c8a67c7.

Restart regression runs at the same head:

• LeaderElectionIntegrationTest: 1 test, 0 failures.
• TestCoordinatorRole: 9 tests, 0 failures.
• TestPullReplica: 39 tests, 1 skipped, 0 failures, run with nightly tests enabled (the class is marked @Nightly, so a plain run skips it; the one skip is an @Ignored method).

These runs show the existing restart tests still pass; they do not show the flake is gone. It was seen once, so one passing run says little about it.

A choice to check

Hold the port in the framework, or restart the one affected test on a fresh port.

This change holds the port in the framework, so restarts keep the same port, node names and test code. Or the one affected test could restart its node on a fresh port. That needs no reservation, but node names then change with the port, and the test compares node names. The cost of this change is one bound socket per stopped node. Was holding the port the right call?

Limits

Only the Jetty port is reserved, and some stopped runners hold it for a long time.

• Runners with a socket proxy in front reserve only their Jetty port. The shared cloud test base uses such runners. The proxy port closes on stop and rebinds on start, with no reservation. A failed rebind is logged at debug level and not raised.
• A runner stopped with stopJettySolrRunner and never restarted is removed from the cluster's list, so cluster shutdown does not release its port. Closing the runner releases it. Otherwise the port stays reserved until a later start on that port or the JVM exits.
• A restart on a fresh port leaves the old port reserved: closing the runner releases only the port it currently holds, so the old port stays reserved until a later start on that port or the JVM exits.
• Standalone runners that are stopped and never restarted or closed also keep their port until a later start on that port or the JVM exits.
• The reservation's refusal of later binds is verified on Linux. On Windows, address-reuse bind semantics differ, and the same exclusion is not verified there.
• A failed start leaves the port unreserved. start() releases the reservation before it binds the port, and a failure after that release does not put the reservation back.
• The reservations are held in one map keyed by port, not by runner. Closing a runner releases the reservation on its port even when a different runner has taken that port over in the meantime and holds the current reservation.

Changelog: changelog/unreleased/SOLR-18531-jetty-runner-port-reservation.yml

AI assistance

AI agents assisted with research, implementation, review, and drafting. Nick Shanin directed the work and takes responsibility for this contribution.