Skip to content

Fix thread pool shutdown race that segfaults mcp_tests - #66

Merged
hkr04 merged 1 commit into
mainfrom
cursor/fix-thread-pool-shutdown-race-5fe3
Oct 21, 2026
Merged

hkr04 merged 1 commit into
mainfrom
cursor/fix-thread-pool-shutdown-race-5fe3

Conversation

@hkr04

@hkr04 hkr04 commented Oct 21, 2026

Copy link
Copy Markdown
Owner

CI failure

Post-merge push to main failed:

https://github.com/hkr04/cpp-mcp/actions/runs/39539546999

cmake configure and the Release build succeeded. ctest reported mcp_tests SEGFAULT (~13s). The last gtest line was StdioTransportTest.ValidIdentifiedPingsAcceptOptionalObjectParams, after one Processing method call: ping log. The #65 pull-request check itself had passed. The trees differ only by ci.yml; the crash is a race, not a bad squash.

Root cause

Not the new JSON depth/size limits or the params move. Those paths run, and the nesting tests pass. The process dies while short-lived server objects are created and destroyed.

Every server owns a thread_pool of detached workers. On exit a worker decremented active_threads_ and dropped the mutex, then locked the pool again to notify the destructor. The destructor returns as soon as the count hits zero and destroys the mutex and condition variable. A second worker still between those two critical sections then locks freed memory.

That matches both failures:

  • GitHub: SIGSEGV during ValidIdentifiedPingsAcceptOptionalObjectParams, right after InvalidEnvelopes had constructed dozens of servers.
  • Local Release, second run: Fatal glibc error: pthread_mutex_lock.c:94 ... mutex->__data.__owner == 0 at the start of OversizedLineIsRejectedAndServerKeepsServing.

ThreadSanitizer on a create/destroy loop reports the same race: worker_loop locking queue_mutex_ / notifying shutdown_cv_ after ~thread_pool() (the next pool's constructor reuses that stack slot). #65 made this likely by adding more back-to-back stdio servers; the PR run simply won the race.

Fix

In worker_loop, publish the exit with fetch_sub and shutdown_cv_.notify_all() before releasing the mutex, then return. The destructor's wait cannot observe zero and destroy the pool until that unlock happens-before its relock, and the worker does not touch the pool after that.

ThreadPool.DestructionDoesNotRaceDetachedWorkers churns 400 pools (4 workers each, one submitted task) to hold the regression.

Verification

Release, same commands as CI (cmake -B build -DMCP_BUILD_TESTS=ON && cmake --build build --config Release && ctest --test-dir build --output-on-failure):

  • Before the fix: first ctest passed, the next aborted in mcp_tests with the glibc mutex assertion.
  • After the fix: three consecutive ctest runs passed (31 tests, ~28s each).
  • ThreadSanitizer on 2000 pool create/destroy iterations is clean. Before the fix it reported the use-after-free on the first iterations.
Open in Web Open in Cursor 

Detached workers dropped active_threads_ and then locked the pool again
to wake the destructor. The destructor frees the pool as soon as the
count hits zero, so a second exiting worker locked a destroyed mutex.
Release builds then died with SIGSEGV or a glibc owner assertion while
stdio tests created and destroyed servers back-to-back.

Publish the exit and notify under the same lock, and do not touch the
pool after that unlock.

Co-authored-by: Zihong Zhang <hkr04@users.noreply.github.com>
@hkr04
hkr04 marked this pull request as ready for review October 21, 2026 04:03
@hkr04
hkr04 merged commit 4ab9897 into main Oct 21, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants