Repository navigation
Fix thread pool shutdown race that segfaults mcp_tests - #66
Merged
Merged
Conversation
Detached workers dropped active_threads_ and then locked the pool again to wake the destructor. The destructor frees the pool as soon as the count hits zero, so a second exiting worker locked a destroyed mutex. Release builds then died with SIGSEGV or a glibc owner assertion while stdio tests created and destroyed servers back-to-back. Publish the exit and notify under the same lock, and do not touch the pool after that unlock. Co-authored-by: Zihong Zhang <hkr04@users.noreply.github.com>
hkr04
marked this pull request as ready for review
October 21, 2026 04:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI failure
Post-merge push to
mainfailed:https://github.com/hkr04/cpp-mcp/actions/runs/39539546999
cmakeconfigure and the Release build succeeded.ctestreportedmcp_testsSEGFAULT (~13s). The last gtest line wasStdioTransportTest.ValidIdentifiedPingsAcceptOptionalObjectParams, after oneProcessing method call: pinglog. The #65 pull-request check itself had passed. The trees differ only byci.yml; the crash is a race, not a bad squash.Root cause
Not the new JSON depth/size limits or the
paramsmove. Those paths run, and the nesting tests pass. The process dies while short-livedserverobjects are created and destroyed.Every
serverowns athread_poolof detached workers. On exit a worker decrementedactive_threads_and dropped the mutex, then locked the pool again tonotifythe destructor. The destructor returns as soon as the count hits zero and destroys the mutex and condition variable. A second worker still between those two critical sections then locks freed memory.That matches both failures:
SIGSEGVduringValidIdentifiedPingsAcceptOptionalObjectParams, right afterInvalidEnvelopeshad constructed dozens of servers.Fatal glibc error: pthread_mutex_lock.c:94 ... mutex->__data.__owner == 0at the start ofOversizedLineIsRejectedAndServerKeepsServing.ThreadSanitizer on a create/destroy loop reports the same race:
worker_looplockingqueue_mutex_/ notifyingshutdown_cv_after~thread_pool()(the next pool's constructor reuses that stack slot). #65 made this likely by adding more back-to-back stdio servers; the PR run simply won the race.Fix
In
worker_loop, publish the exit withfetch_subandshutdown_cv_.notify_all()before releasing the mutex, then return. The destructor'swaitcannot observe zero and destroy the pool until that unlock happens-before its relock, and the worker does not touch the pool after that.ThreadPool.DestructionDoesNotRaceDetachedWorkerschurns 400 pools (4 workers each, one submitted task) to hold the regression.Verification
Release, same commands as CI (
cmake -B build -DMCP_BUILD_TESTS=ON && cmake --build build --config Release && ctest --test-dir build --output-on-failure):ctestpassed, the next aborted inmcp_testswith the glibc mutex assertion.ctestruns passed (31 tests, ~28s each).