Fix concurrency and equals-contract CodeQL alerts - #785
Conversation
DirectoryServer synchronized on the directoryServer field in seven places, including getNewInstance() which reassigns that very field. After an in-core restart the blocks would lock different monitors and provide no mutual exclusion at all. They all guard the same thing - the server life cycle flags and the singleton wiring - so they now share a dedicated static lock, and the field is volatile since it is read without any lock throughout the class. DebugLogPublisher had synchronized setters but unsynchronized readers over plain HashMap/TreeMap instances created lazily, which both published the maps unsafely and allowed a put concurrent with a get. The fields are now volatile and the maps concurrent, removeTraceSettings is synchronized like the other mutators, and the readers stay lock-free because they sit on the tracing hot path. The nested map does not rely on ordering, so ConcurrentHashMap replaces TreeMap. Both PointAdder classes busy-waited in a synchronized stop() while sleeping, and swallowed the caller's interruption in a catch (Throwable) whose parameter shadowed the thread field. The lock guarded nothing - start() is not synchronized - so it is gone, the handshake flags are volatile, and InterruptedException restores the interrupt status. The flag based wait is kept rather than Thread.join(), because Application.PointAdder sets the flag before a final listener notification that stop() must not wait for. ScheduleType.equals() accepted any object with a matching toString(), making a plain String compare equal to a schedule asymmetrically, and ProfileStack/ProfileStackFrame cast without an instanceof check and relied on catching ClassCastException, logging a stack trace on every comparison against another type. All three now check the type. The loggers in the profiler classes became unused and were removed. SuffixDescriptor overrode hashCode() and compareTo() by its id but left equals() as identity; it now compares ids as well. Note that this changes HashSet semantics for suffix descriptors in the installer from identity to value equality.
maximthomas
left a comment
There was a problem hiding this comment.
Five well-scoped fixes, accurately diagnosed. Verified the risky parts: no wait()/notify() anywhere in DirectoryServer, so swapping the monitor can't break a pairing; all seven sites converted; startServer() → initializeDefaultEntryCache() → setEntryCache() re-enters LIFECYCLE_LOCK on the same thread, so it stays correct.
The SuffixDescriptor change you flagged is safe: opendj-server-legacy/src/main/java/org/opends/quicksetup/installer/ui/SuffixesToReplicatePanel.java:74,249,121 already keys checkboxes by getId(), so equal-id descriptors were already indistinguishable in the UI, and opendj-server-legacy/src/main/java/org/opends/server/tools/dsreplication/ReplicationCliMain.java:5233-5234 uses natural-ordering TreeSets that already deduplicated via compareTo → getId(). Dropping the catch (Exception) in ProfileStack.equals is safe too: decode()'s only caller (opendj-server-legacy/src/main/java/org/opends/server/plugins/profiler/ProfileViewer.java:237) NPEs earlier in ProfileStackFrame.hashCode() (:254) on a malformed capture, identically before and after this PR.
One blocker.
PointAdder.stop() NPEs when the adder was never started (high)
Moving t.interrupt() above the try exposes a live start-less path in opendj-server-legacy/src/main/java/org/opends/server/tools/dsreplication/ReplicationCliMain.java:
8701 PointAdder pointAdder = new PointAdder(this);
8702 try {
8735 throw new ReplicationCliException(ERR_REPLICATION_USER_CANCELLED.get(), USER_CANCELLED, null);
8747 pointAdder.start();
8867 } finally { pointAdder.stop(); }Declining the confirmation prompt throws before start(), so stop() runs with t == null. Today the NPE is swallowed and the loop never reaches Thread.sleep — an infinite busy-spin, so dsreplication hangs at 100% CPU on a plain user cancel. After this PR the NPE escapes the finally and replaces the USER_CANCELLED exception. createTopologyCache(adsCtx1, cnx) at :8706 reaches the same finally.
Guard both classes:
public void stop()
{
stopPointAdder = true;
final Thread thread = t;
if (thread == null)
{
return; // never started
}
while (!pointAdderStopped)
{
thread.interrupt();
...Application.PointAdder has no such path today (all three Installer call sites pair start()/stop() under the same guard), but should match for symmetry.
PointAdder.t left non-volatile (medium)
t is written by start() and dereferenced by stop() — the same rationale used for the two flags. Leaving it plain also turns the issue above from "deterministically null" into "possibly stale null" across threads. Make it volatile; the local copy above then makes the read safe.
SuffixDescriptor.equals keys off a non-canonical, mutable id (low–medium)
getId() iterates getReplicas() (new HashSet<>(replicas)) and ReplicaDescriptor overrides neither equals nor hashCode, so ids are concatenated in identity-hash order — two descriptors with the same DN and servers but different replica instances can produce different ids. Separately, opendj-server-legacy/src/main/java/org/opends/admin/ads/TopologyCache.java:135 calls addReplica() on descriptors already in the HashSet at :69/:146, so the id changes after insertion.
Both pre-date this PR, but the new equals makes the id load-bearing for set membership. Sorting makes it canonical and fixes equals, hashCode and compareTo at once:
Set<String> ids = new TreeSet<>();
for (ReplicaDescriptor replica : getReplicas()) { ids.add(replica.getServer().getId()); }LIFECYCLE_LOCK javadoc overstates what is guarded (low)
The flags it names are plain non-volatile fields, still read unlocked in isRunning() (opendj-server-legacy/src/main/java/org/opends/server/core/DirectoryServer.java:1123), isShuttingDown() (:4708) and setEnvironmentConfig() (:1096), and written unlocked in destroy() (:4339-4344). A volatile reference doesn't publish later writes to those fields. Either mark the three flags volatile — cheap, and it genuinely closes java/unsynchronized-getter here — or soften the comment to "the singleton wiring and the guarded transitions".
ConcurrentHashMap is null-hostile (low)
addTraceSettings is public on a @PublicAPI(mayExtend=true) class. addTraceSettings(scope, null), getClassSettings(null) and hasTraceSettings(null) were tolerated by HashMap/TreeMap and now throw NPE. No in-tree caller does this (opendj-server-legacy/src/main/java/org/opends/server/loggers/TextDebugLogPublisher.java:81-89 guards both args), but an external subclass could. A @throws NullPointerException note would document the narrowed contract.
Nits
- Redundant reassignment:
opendj-server-legacy/src/main/java/org/opends/server/core/DirectoryServer.java:4331doesdirectoryServer = getNewInstance(envConfig);, butgetNewInstance()already assigns under the lock — the outer write repeats it outside it. Just call the method. - Leftover blank lines: removing the
loggerfields left four blank lines after{inopendj-server-legacy/src/main/java/org/opends/server/plugins/profiler/ProfileStack.java:31-34and.../ProfileStackFrame.java:37-40, plus a gap atProfileStackFrame.java:24-26. - Same alert class elsewhere (follow-up):
synchronized (directoryServer.authenticationPolicies)(DirectoryServer.java:2478,2509,2531,2548),.connectionHandlers(:3436,3455),.establishedConnections(:4448,4484) all lock non-final fields reassigned at:1155,:1162,:1217— the same bug just fixed fordirectoryServer. (supportedControls/supportedFeaturesarefinal, so those are fine.) Better as a follow-up issue. - Hot-path rationale:
getClassSettings/getMethodSettingsare only called fromDebugTracer.PublisherSettings's constructor (opendj-server-legacy/src/main/java/org/opends/server/loggers/DebugTracer.java:57-58) andupdateSettings()(:212); results are cached, so a monitor there wouldn't be paid per traced call. Change is still right, the justification is just stronger than needed. - No tests: nothing in the repo references any of the five touched classes. One case pinning
SuffixDescriptorid-equality plusHashSetcollapse, and one for theScheduleType-vs-Stringasymmetry, would document the only behavioural change here.
…e, canonical suffix id * PointAdder.stop() no longer NPEs when start() was never called. In ReplicationCliMain.mergeRegistries() the adder is created at :8701 but started only at :8747, so declining the merge confirmation (:8735) or a failure in createTopologyCache() (:8706) reaches the finally block with a null thread. Both PointAdder classes now return early, and the thread field became volatile like the two handshake flags. * DirectoryServer: isBootstrapped, isRunning, shuttingDown and serverLocked are volatile. LIFECYCLE_LOCK makes the guarded transitions atomic but does not publish the flags to isRunning(), isShuttingDown() and setEnvironmentConfig(), which read them unlocked, nor to destroy(), which writes them unlocked. * DirectoryServer.shutDown() no longer repeats the directoryServer assignment outside the lock: getNewInstance() already publishes the new instance. * authenticationPolicies, connectionHandlers and establishedConnections are final and cleared instead of reassigned during bootstrap. Eight synchronized blocks lock those fields, so reassigning them had the same java/unsafe-sync-on-field defect just fixed for the singleton itself. * SuffixDescriptor.getId() sorts the server ids. getReplicas() hands out a HashSet of ReplicaDescriptors that override neither equals() nor hashCode(), so the id -- now load-bearing for the new equals() -- otherwise depended on the identity hash iteration order. * DebugLogPublisher: corrected the field comment (DebugTracer caches the settings, so the getters are not on the per-call tracing path) and documented that addTraceSettings() now rejects null settings. * Removed the blank lines left behind by the deleted logger fields. * Added SuffixDescriptorTest and ScheduleTypeTest covering the id-based equality, the HashSet collapse and the ScheduleType-vs-String asymmetry.
|
Thanks — all eight points are addressed in 9bcd4ad, including the two you offered to defer.
|
Clears the remaining small concurrency and
equals-contract alerts:java/unsafe-sync-on-field(#650),java/unsynchronized-getter(#651, #652),java/sleep-with-lock-held(#621, #622),java/unchecked-cast-in-equals(#599, #600, #601) andjava/inconsistent-equals-and-hashcode(#578).1.
DirectoryServersynchronizes on fields it reassignsgetNewInstance()did:The problem is wider than that one method:
synchronized (directoryServer)appears in seven places —getNewInstance(),bootstrapClient(),bootstrapServer(), the end of bootstrapping,startServer(),setEntryCache()andshutDown(). Once the reference is swapped during an in-core restart, those blocks lock different monitors, so they provide no mutual exclusion at all.All seven guard the same thing — the server life cycle flags (
isRunning,isBootstrapped,shuttingDown) and the singleton wiring — so they now share a dedicatedLIFECYCLE_LOCK. ThedirectoryServerfield is also madevolatile, since it is read without any lock bygetInstance(),getEnvironmentConfig()and many other accessors.The lock only makes the guarded transitions atomic; it does not publish the flags to the code that reads them without holding it —
isRunning(),isShuttingDown()andsetEnvironmentConfig()read them unlocked anddestroy()writes them unlocked. SoisBootstrapped,isRunningandshuttingDownarevolatileas well, along withserverLocked, which is written outside the lock on shutdown.shutDown()also diddirectoryServer = getNewInstance(envConfig);, repeating outside the lock a writegetNewInstance()already performs inside it. It now just calls the method, likereinitialize()does.Three more fields were locked while being reassignable:
authenticationPolicies,connectionHandlersandestablishedConnectionsare the target of eightsynchronizedblocks but were replaced with fresh collections during bootstrap — the same defect as above. They are nowfinaland cleared instead. (supportedControlsandsupportedFeatureswere alreadyfinal.)2.
DebugLogPublisherreaders race with its synchronized writerssetClassSettings()/setMethodSettings()weresynchronized, butgetClassSettings(),getMethodSettings()andhasTraceSettings()were not, over plainHashMap/TreeMapinstances created lazily. Two real hazards: the map reference is published unsafely, and aputcan run concurrently with ageton a non-concurrent map.The fields become
volatileand the mapsConcurrentHashMap;removeTraceSettings()becomessynchronizedlike the other mutators. The readers stay lock-free: they are cheap, andDebugTracercaches their results inPublisherSettings, so there is nothing to gain from a monitor there. The nested map does not rely on ordering (onlyget/containsKey/remove), andDebugTracerkeeps the map returned bygetMethodSettings()and may iterate it while the publisher is reconfigured — which the oldTreeMapdid not survive — soConcurrentHashMapreplaces it.One narrowed contract:
addTraceSettings(scope, null)used to store a null value in aHashMapand now throwsNullPointerException. No in-tree caller does this (TextDebugLogPublisher:81-89guards both arguments) and every other accessor is package-private, but the javadoc now documents it.3. Both
PointAdderclasses sleep holding a lockorg.opends.server.util.cli.PointAdderandApplication.PointAdderare near-identical:There is no deadlock here — the worker thread never takes the monitor — but the lock guards nothing (
start()is not synchronized), the handshake flags are non-volatile, andcatch (Throwable t)both swallows the caller's interruption and shadows thetthread field.synchronizedis dropped, the flags and the thread field becomevolatile, andInterruptedExceptionis caught explicitly with the interrupt status restored.The
catch (Throwable)was also hiding a live bug.ReplicationCliMain.mergeRegistries()creates the adder at:8701but starts it at:8747, and both a failure increateTopologyCache()(:8706) and a declined confirmation (:8735) reach thefinally { pointAdder.stop(); }. Witht == nullthe NPE was swallowed,pointAdderStoppednever becametrueand the loop never reachedThread.sleep: declining the merge leftdsreplicationspinning at 100% CPU.stop()now returns early when the adder was never started.The flag-based wait is kept rather than switching to
Thread.join()on purpose:Application.PointAdder.run()setspointAdderStoppedbefore a finallistenerDelegate.notifyListeners(...)call, so joining would makestop()wait for that notification too — a deadlock risk whenstop()is called from the EDT.4.
equals()implementations that do not check the argument typeScheduleType.equals()waso != null && toString().equals(o.toString()), so a plainStringwith matching text compared equal to a schedule — and asymmetrically, sinceString.equalswould return false the other way. Now guarded byinstanceof ScheduleType.ProfileStackandProfileStackFramecast without aninstanceofcheck and relied oncatch (Exception)to handle the resultingClassCastException, logging a stack trace viatraceExceptionon every comparison against another type. Both now check the type up front. Theirloggerfields became unused and were removed along with the imports.5.
SuffixDescriptorhashes and compares by id but compares equal by identityThe class overrides
hashCode()andcompareTo()in terms ofgetId(), but never overrodeequals(), so equality stayed reference-based. It now compares ids.getId()was not canonical: it iteratedgetReplicas(), a freshHashSet<ReplicaDescriptor>, andReplicaDescriptoroverrides neitherequalsnorhashCode, so two descriptors with the same DN and the same servers could produce different ids depending on identity-hash iteration order. Now that the id is load-bearing for equality, the server ids are sorted, which makesequals,hashCodeandcompareTowell-defined at once.Behaviour note:
SuffixDescriptoris stored inHashSets in the installer (UserData,SuffixesToReplicateOptions,SuffixesToReplicatePanel), so two descriptors with the same id now collapse into one. An equal id means the same DN and the same set of replica servers, so they are genuine duplicates, and they were already indistinguishable elsewhere:SuffixesToReplicatePanelkeys its checkboxes bygetId(), andReplicationCliMain:5233-5234uses natural-orderingTreeSets that deduplicated viacompareTo→getId()already.Testing
mvn -pl opendj-server-legacy test-compile— BUILD SUCCESS.SuffixDescriptorTest(id-based equality,HashSetcollapse, id independent of replica order) andScheduleTypeTest(including theScheduleType-vs-Stringasymmetry) — 11 tests, all passing.DirectoryServerchanges touch bootstrap and shutdown:UnbindOperationTestCase(41 tests) andPluginConfigManagerTestCase(34 tests) pass, plus a scratch test forcing two in-core restarts throughshutDown()→getNewInstance()→ re-bootstrap.