Skip to content

[rocprofiler-sdk] Fix signal-handler re-entrancy deadlock and enable app-abort testsadd a thread re-entrancy guard - #6875

Closed
minseobshin11 wants to merge 1 commit into
developfrom
users/mishin/enable-app-abort-tests
Closed

[rocprofiler-sdk] Fix signal-handler re-entrancy deadlock and enable app-abort testsadd a thread re-entrancy guard#6875
minseobshin11 wants to merge 1 commit into
developfrom
users/mishin/enable-app-abort-tests

Conversation

@minseobshin11

Copy link
Copy Markdown
Contributor

Motivation

The rocprofv3-test-app-abort integration tests (4 tests) were disabled unconditionally as "currently unstable". The instability was a real deadlock in rocprofv3's abort signal handler that caused the profiled process to hang on every abort. This PR fixes that deadlock and re-enables the tests.

Technical Details

Root cause: rocprofv3 installs its signal handler before HSA does, so when HSA later calls sigaction() it receives rocprofv3's handler as its "previous" handler. On an abort signal, rocprofv3_error_signal_handler finalizes output and invokes the chained HSA handler, which calls back the previous handler (rocprofv3's), re-entering the handler on the same thread while still inside std::call_once. Recursive use of std::once_flag deadlocks, so the process hangs and CTest times out. (SA_RESETHAND doesn't help — the re-entry is a direct call, not a fresh signal delivery.)

Fix:

  • source/lib/rocprofiler-sdk-tool/tool.cpp: add a thread_local re-entrancy guard at handler entry that breaks the chain loop and terminates instead of recursing into call_once. Preserves cross-thread call_once semantics and is async-signal-safe.
  • tests/rocprofv3/aborted-app/CMakeLists.txt: remove the unconditional IS_DISABLED ON (still disabled under ThreadSanitizer).

JIRA ID

Resolves AIROCVAL-48

Test Plan

Built for gfx942 (MI300X, ROCm 7.2) and ran the app-abort tests under multiple conditions:

  • ctest -R app-abort (execute + 3 validate steps).
  • ctest --repeat until-fail:50 under heavy CPU contention (load ~205).
  • ~470+ abort invocations total: serial loops, 192-way concurrent, and under CPU/memory contention, with hang detection.

Test Result

100% pass; the execute step completes in ~1s (previously a 45s hang). Zero hangs across all runs and no leftover/spinning processes. The guard also correctly handled an induced SIGABRT loop-back during stress, terminating cleanly instead of deadlocking.

Submission Checklist

@minseobshin11
minseobshin11 force-pushed the users/mishin/enable-app-abort-tests branch from a1d28f9 to 299ce49 Compare June 10, 2026 15:08
@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage Report

Code Coverage Report

Tests Only

code coverage tests.png

Samples Only

code coverage samples.png

Tests + Samples

code coverage all.png

@mcao59

mcao59 commented Jun 10, 2026

Copy link
Copy Markdown
Member

I see this is still in draft, but just wanted to flag that #6717 takes a broader approach to redesigning rocprofv3 signal handling.
May be worth coordinating the direction here so we don’t end up with overlapping or conflicting changes :)

@minseobshin11

Copy link
Copy Markdown
Contributor Author

I see this is still in draft, but just wanted to flag that #6717 takes a broader approach to redesigning rocprofv3 signal handling. May be worth coordinating the direction here so we don’t end up with overlapping or conflicting changes :)

Thanks for the heads up! I agree there's overlap. #6717 reworks the whole handler to run finalization on a worker thread, which would address the same re-entrancy root cause and make this thread_local guard unnecessary. My PR was intended as a minimal fix :) I believe if #6717 can be merged, it will fix the app-abort flakiness as well!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants