Replication: enable() clears "disabled" after starting the session, where disable() sets it before stopping one

[vharseko]

LDAPReplicationDomain.disable() sets disabled = true before disableService(), so a replay thread reads the flag of a domain which is going away before its session is stopped. enable() does not mirror it: it calls enableService() and clears disabled two statements later.

// disable()
disabled = true;
disableService();
...
// enable()
enableService();
sessionGeneration++;

disabled = false;

enableService() ends with startListenService(), so the listener can take a delivery, list it and hand it to a replay thread while disabled is still true. That thread reads the guard at the top of the replay loop, gives up on the change - and abandonReplay() returns without asking for it again, because a domain which is going away owns its session. The change stays listed, uncommitted and owned by nobody, so this domain's ServerState is held back until something else restarts the session.

The window is two statements wide and nothing in the protocol can be made to land inside it, so this is an invariant to hold rather than a bug to reproduce. Clearing the flag before enableService() closes it, and putting it back if enableService() throws keeps the other half: a domain whose session could not be started owns it the way a disabled one does.

Worth doing with the session generation of #922 rather than alone: a replay thread which survived disable() reads that flag too, and neither order of these two statements is right for it (#908).

Found while reviewing #892.

[vharseko]

Two CI failures on 2026-09-09 look like this window being hit in practice, which the
description above did not expect - "nothing in the protocol can be made to land inside it".

Both are replication tests which wait for a change to be replayed after a disable() /
enable() pair, and both show the same signature: the domain performs no operation at all
for the whole 30 s of the wait, and nothing is written to any log.

The two sightings

ReSyncTest.testResyncAfterRestore - run 34318115721, build-maven (ubuntu-latest, 25),
on the head of #980.

java.lang.AssertionError: Expected entry with dn "dc=fooUniqueName1,dc=example,dc=com" would exist expected [true] but found [false]
	at org.opends.server.replication.ReplicationTestCase.getEntry(ReplicationTestCase.java:759)
	at org.opends.server.replication.ReSyncTest.testResyncAfterRestore(ReSyncTest.java:167)

The method took 30.27 s, which is the whole getEntry(entryDN, 30000, true) budget. In the
access log of the failure dump the last records are at 11:10:35, and there is not one
request between 11:10:36 and 11:11:05. The error log ends at 11:10:35 on
"Archived backup file: tasks.ldif" and says nothing more until teardown.

DependencyTest.addModDelDependencyTest - run 34316279431, build-maven (ubuntu-latest, 25),
on the head #978 had at the time.

java.lang.AssertionError: Expected entry with dn "dc=dependency1,o=test" would exist expected [false] but found [true]
	at org.opends.server.replication.ReplicationTestCase.getEntry(ReplicationTestCase.java:759)
	at org.opends.server.replication.DependencyTest.addModDelDependencyTest(DependencyTest.java:196)

The method took 35.38 s. The last access-log record is MODIFY dn="o=test" at 06:45:06 - the
ServerState write - and no DELETE was ever attempted: nothing at all between 06:45:07 and
the failure at 06:45:36. The error log is silent over the same window.

Why these two tests

Both hand the replica a backlog at the exact moment the session comes back up, which is the
shape that can land in the window - not one message arriving at some random later time, but a
queue drained the instant startListenService() returns, with the two remaining statements of
enable() still to run.

• ReSyncTest.testResyncAfterRestore - the restore task goes through
  MultimasterReplication.processRestoreBegin() -> domain.disable() and
  processRestoreEnd() -> domain.enable(). The entry added between the backup and the
  restore is waiting at the replication server and is sent again as soon as the listener
  starts. This one is an operational sequence rather than something only a test does.
• DependencyTest.addModDelDependencyTest - the test disables the domain on purpose,
  publishes 30 dependent DELs to the replication server while it is down and enables it back,
  "so that when we enable it it receives the delete operation in bulk".

Why it leaves nothing in the log

abandonReplay() returns before it logs anything while the flag is still set
(LDAPReplicationDomain.java:3142):

private void abandonReplay(CSN csn)
{
  remotePendingChanges.replayFailed(csn);
  if (shutdown.get() || disabled)
  {
    // The domain owns its session, and it forgets its pending changes on its way down.
    return;
  }
  logger.info(NOTE_REPLAY_ABANDONED_CHANGE, csn, getBaseDN());
  sessionRestartRequested.set(true);
  runRequestedSessionRestarts(false);
}

replayFailed() un-owns the change but leaves it listed and uncommitted, so it becomes the
barrier this issue describes - and no line is written to say so, which is why both failures
look like the domain simply going quiet.

In DependencyTest that also explains why not a single DELETE reached the backend rather than
29 out of 30: the change which is lost is the oldest DEL of the chain, and
checkDependencies(DeleteOperation) - which runs before the replay loop - parks each of the
other 29 behind it, so none of them ever becomes an operation.

What was ruled out

For both failures the change under review is not a plausible cause:

• [#950] Announce a ReplicaOfflineMsg before it is published, not after it may have been forwarded #978 only moves where DSRSShutdownSync.replicaOfflineMsgs is written. The readers of that
  map are ReplicationServer.shutdown() and the non-DS branch of ServerWriter, and neither
  runs before the assertion in DependencyTest. Ten local runs of DependencyTest, five with
  that patch and five without, were all green at 5.36-5.48 s per run against a 30 s budget.
• [#953] Refuse a server-error-result-code which does not report a failure #980 only changes how the result of an operation which already ran is read, and adds
  validation of server-error-result-code. No operation ran at all.

On the rate: seven full-suite runs on the same base commit (f85d2d2) were green on
2026-09-09. Over 2026-09-08 and 2026-09-09 there were three failed Build runs in total - these
two, and ReplicationServerShutdownSyncTest on #947, which has a cause of its own and was
addressed by a follow-up commit on that branch. Both sightings landed on the
ubuntu-latest, 25 leg, which may well be chance.

None of this catches the two statements in the act, and the path is silent by construction, so
it is not a proof. But this issue is the only cause I could find which accounts for all three
observations at once: no operation attempted, no log line, and the domain never recovering on
its own.