type(fix): [rocprofiler-sdk] rework signal handling - #6717
Conversation
56dacbb to
e2664a5
Compare
e2664a5 to
a63f2d7
Compare
meserve-amd
left a comment
There was a problem hiding this comment.
Overall, this change seems good. Mostly minor feedback.
7ae29e3 to
1044356
Compare
ff72464 to
9adeeb2
Compare
|
Tests are a lil unstable, working on them. |
9adeeb2 to
97eb72b
Compare
The rocprofv3-test-app-abort tests (execute + 3 validators) were disabled unconditionally as "currently unstable". The underlying flakiness was the signal-handler re-entrancy deadlock that the worker-thread rework in #6717 fixes, so re-enable them here (still disabled under ThreadSanitizer). Enabling required adapting the test to the reworked handler's contract: - Drop ROCPROF_INTERNAL_TEST_SIGNAL_HANDLER_VIA_EXIT=1. With the new handler this quick_exit()s before the worker thread finalizes, so no output would be flushed and the validators would have no data. Removing it routes the abort through the real worker-thread finalize/flush path. - Wrap the command in `bash -c "... || exit $?"`. The handler now re-raises SIGINT, terminating the process by signal; CTest treats that as an exception that WILL_FAIL does not invert (and SIGINT interrupts the run). The wrapper turns it into a normal non-zero exit (128+signo) that WILL_FAIL inverts to a pass, while a genuine deadlock still blocks and is caught by the ctest TIMEOUT (timeouts are not inverted by WILL_FAIL). `|| exit $?` (not `; exit $?`) keeps it a single bash argument so CMake does not split on the semicolon. - Compare sorted(kernels) against the expected kernel list. The kernel_dispatch buffer records are not guaranteed to be in alphabetical order, and the abort path can flush the two records in either order, so the trace check must be order-independent. The comment wrapping conforms to the project .cmake-format.yaml (line_width 90). Test plan: built for gfx942 (MI300X, ROCm 7.2.3); ctest -R app-abort passes; ctest --repeat until-fail:500 (2000 sub-tests) 0 failures; 1920 concurrent abort cycles (192-way, all 8 GPUs) 0 hangs and all output flushed. Co-authored-by: Cursor <cursoragent@cursor.com>
97eb72b to
0ad8866
Compare
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
This comment was marked as resolved.
This comment was marked as resolved.
0ad8866 to
b0d7325
Compare
b0d7325 to
a3b6144
Compare
|
|
||
| if(ndangling > 0) | ||
| { | ||
| auto retirement_ctxs = get_active_contexts([](const context* ctx) { | ||
| return (ctx->buffered_tracer && | ||
| (ctx->buffered_tracer->domains( | ||
| ROCPROFILER_BUFFER_TRACING_CORRELATION_ID_RETIREMENT))); | ||
| }); | ||
|
|
||
| if(!retirement_ctxs.empty()) | ||
| { | ||
| ROCP_CI_LOG_IF(INFO, ndangling > 0) | ||
| << "retired dangling correlation IDs: " << ndangling; | ||
| } | ||
| else | ||
| { | ||
| ROCP_INFO << "retired dangling correlation IDs: " << ndangling | ||
| << " (no retirement consumer active)"; | ||
| } | ||
| } |
There was a problem hiding this comment.
Needed for cleaner CI checks for tests added in this PR, otherwise in-flight IDs abort new tests with this PR.
Should we use an env-level toggle instead, so that other tests that rely on this failing in CI fail loudly correctly?
|
correlation ID update is racy with signal-handling drive finalization, leading to some CI flakiness and crashes. Working on a fix. |
234b2d2 to
4f906d6
Compare
meserve-amd
left a comment
There was a problem hiding this comment.
Some minor comments but the approach seems solid. I think the correlation ID stuff might need a second look beyond this PR to make sure we're handling corner cases correctly.
0106b42 to
cab9877
Compare
| ROCP_WARNING << "signal worker: close(eventfd) after fork failed: " << strerror(errno); | ||
| } | ||
|
|
||
| std::memset(static_cast<void*>(&sw), 0, sizeof(sw)); |
There was a problem hiding this comment.
| std::memset(static_cast<void*>(&sw), 0, sizeof(sw)); | |
| sw.~signal_worker_state(); |
Call the destructor, don't just overwrite the memory. You have a thread in signal_worker_state. Unless you are detaching, you need to join the thread before destroying it.
| if(sw.eventfd >= 0) | ||
| { | ||
| // Write to unblock the worker's read() — closing the fd is UB while read() blocks | ||
| uint64_t val = 1; | ||
| if(write(sw.eventfd, &val, sizeof(val)) < 0) | ||
| { | ||
| ROCP_WARNING << "failed to write to signal worker eventfd: " << strerror(errno); | ||
| } | ||
| } | ||
| sw.thread.join(); | ||
| if(sw.eventfd >= 0) | ||
| { | ||
| if(close(sw.eventfd) < 0) | ||
| { | ||
| ROCP_WARNING << "failed to close signal worker eventfd: " << strerror(errno); | ||
| } | ||
| sw.eventfd = -1; | ||
| } |
There was a problem hiding this comment.
This looks like something you might want to put in a join member function of signal_worker_state so it can be called here and from the destructor of signal_worker_state.
| else | ||
|
|
||
| // Join the worker thread if called from a non-worker context (atexit / rocprofv3_main). | ||
| if(sw.thread.joinable() && sw.thread.get_id() != std::this_thread::get_id()) |
There was a problem hiding this comment.
This part would also be part of the join member function.
| if(!atfork_registered) | ||
| { | ||
| atfork_registered = true; | ||
| pthread_atfork(nullptr, nullptr, []() { | ||
| // Child handler: reset stale state and spawn a fresh worker. | ||
| auto& child_sw = get_signal_worker(true); | ||
| child_sw.eventfd = eventfd(0, EFD_CLOEXEC); | ||
| if(child_sw.eventfd >= 0) child_sw.thread = std::thread{signal_finalization_worker}; | ||
| }); | ||
| } | ||
| } |
There was a problem hiding this comment.
I guess I see why you used memset instead of calling the destructor... but I think I would prefer that to be here, not in get_signal_worker.
There was a problem hiding this comment.
Moved it out and added a join member function.
| if(!retirement_ctxs.empty()) | ||
| { | ||
| ROCP_CI_LOG_IF(INFO, ndangling > 0) | ||
| << "retired dangling correlation IDs: " << ndangling; |
| { | ||
| char range_name[128]; | ||
| snprintf(range_name, sizeof(range_name), "%s_pid_%d", label, getpid()); | ||
| roctxRangePush(range_name); |
There was a problem hiding this comment.
| roctxRangePush(range_name); | |
| roctxRangePush(join(label, "_pid_", getpid()).c_str()); |
| char iter_name[128]; | ||
| snprintf(iter_name, sizeof(iter_name), "%s_iter_%d", label, iter); | ||
|
|
||
| roctxRangePush(iter_name); |
There was a problem hiding this comment.
| char iter_name[128]; | |
| snprintf(iter_name, sizeof(iter_name), "%s_iter_%d", label, iter); | |
| roctxRangePush(iter_name); | |
| roctxRangePush(join(label, "_iter_", iter).c_str()); |
| rocprofiler_add_integration_execute_test( | ||
| ${WITH_HANDLING_NAME} | ||
| COMMAND | ||
| ${SH_EXECUTABLE} -c | ||
| "rm -rf '${BASE_OUT_DIR}/good-${PROC_TYPE}' && exec \"$@\"" ${SH_EXECUTABLE} | ||
| ${TIMEOUT_CMD} $<TARGET_FILE:rocprofiler-sdk::rocprofv3> --hip-trace | ||
| --marker-trace --output-format json --disable-signal-handlers -d | ||
| ${BASE_OUT_DIR}/good-${PROC_TYPE} -o out_%pid% -- | ||
| $<TARGET_FILE:signal-handler-test> --${PROC_TYPE} --app-signal-handler | ||
| DEPENDS signal-handler-test | ||
| TIMEOUT ${CTEST_TIMEOUT} | ||
| LABELS "integration-tests;signal-handler" | ||
| PRELOAD "${PRELOAD_ENV}" | ||
| PASS_REGULAR_EXPRESSION "clean exit" | ||
| FIXTURES_SETUP ${WITH_HANDLING_NAME}) |
There was a problem hiding this comment.
Why is it necessary to rm -rf?? I really think I would (A) prefer using python script wrapper with subprocess that times out instead of sh and (B) this test have a FIXTURES_REQUIRED on a "clean up" test to ensure the clean-up operation happens before the test runs, e.g. adding a rocprofiler_add_integration_cleanup_test function.
There was a problem hiding this comment.
Why is it necessary to rm -rf??
Pre-existing files conflict with the test, and wasn't sure of a clean way to run the validate only on newer files. Removing files from older runs was less messy, at the cost of rm -rf. Added a function and remove the loop. Not sure about rm -rf. Kept it for now but can replace it with something safer.
| # Generate all 8 test combos: {good, bad} x {single-process, fork, fork-exec, spawn} | ||
| # =========================================================================== | ||
|
|
||
| foreach(PROC_TYPE ${PROCESS_TYPES}) |
There was a problem hiding this comment.
Make this a function and invoke it with fork, fork-exec, and spawn.
There was a problem hiding this comment.
Done. Removed the loop and added explicit functions and formatting config.
| @@ -0,0 +1,2 @@ | |||
| [pytest] | |||
| addopts = -rA --tb=short | |||
There was a problem hiding this comment.
Why are you deviating from the standard pytest.ini contents here??
There was a problem hiding this comment.
Sorry slipped through my review, thanks for catching this! Fixed.
d65c70e to
2fdff05
Compare
Replace async-signal-unsafe rocprofv3_error_signal_handler with a worker thread architecture. Signal handler is now fully async-signal-safe: writes eventfd, waits on futex with bounded timeout, restores original handler, re-raises. Worker thread performs finalization in normal context where all APIs are safe to call. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
- use atomic<uint32_t> - add error checking for pipe - re-add previous removed INFO call for ignored finalization - re-init on fork after memset
The handler used to wake the finalization worker and then block the signaled thread until the flush finished, re-raising on that same thread. If the signal interrupted that thread mid-malloc (holding the arena lock), the worker's flush would block on the same lock -> deadlock; it also stalled the app's own coordinated-shutdown handlers. Now the handler hands off and gets out of the way: - Async signals (SIGINT/SIGQUIT/SIGTERM): record signo, wake the worker via eventfd, and return immediately. Returning lets the interrupted thread resume and release any lock it holds, so the worker's (non-async-signal- safe) flush cannot deadlock against it. - The worker owns termination: after finalize + reaping children it restores the wrapped disposition (the app's chained handler if one was saved, else SIG_DFL) and re-raises process-directed via kill(getpid(), signo) so it lands on an application thread (the worker itself blocks these signals). With no chained handler it unblocks first so SIG_DFL is guaranteed to exit. - SIGABRT stays synchronous: abort() resets to SIG_DFL and re-raises the instant the handler returns, so we must wait on the finalize_done futex for the flush before returning. This is the one path that can still block on a held lock -- acceptable since the process is already aborting. - Re-entry escalates instead of swallowing: if a chained handler (e.g. LLVM/comgr) restores our handler and re-raises into us, force SIG_DFL, unblock, and re-raise to guarantee termination rather than hang.
Build on the non-blocking handler + worker so signal re-delivery is safe and the profiler stays transparent to the app's own signal disposition: - Remove `SA_RESETHAND` so a second delivery of the same signal re-enters our guard instead of resetting to `SIG_DFL` and terminating mid-flush (was silent data loss). - Gate re-entry escalation on finalize_done: swallow repeats while the flush is in progress, force `SIG_DFL` + re-raise only after it completes -- a routine double `Ctrl+C` no longer truncates the profile, and a stuck flush stays killable. - Honor `SIG_IGN` symmetrically for `signal()` and `sigaction()` (`sigaction(SIG_IGN)` was dropped, wrongly terminating apps that deliberately ignore `Ctrl+C`). - Drop the unused signal-handler timeout knob; correctness relies on lock release and the gated escalation, not a clock. - signal-handler tests: each execute wipes its own output dir first (`sh` shim, `sh` located via `find_program`) so a truncated file from a prior local run can't fail validation; CI already starts clean.
… id at finalization The non-blocking signal handler finalizes on a worker thread, so application threads can keep calling instrumented APIs while finalization runs. `correlation_service::construct()` returns nullptr once finalization has begun, but most interceptors dereferenced it unguarded and crashed (`SIGSEGV` in roctx range push and HIP under signal-triggered shutdown), surfacing intermittently in CI. Fall back to the untraced path (run the underlying API and return) when `construct()` yields nullptr, via a new `RETURN_UNTRACED_ON_NULL_CORRELATION_ID` macro. Applied in hip, marker, range_marker, ompt, rccl, rocshmem, hipfile, rocjpeg and rocdecode; the HSA paths already handled this.
2fdff05 to
61a4370
Compare



Motivation
Fix signal handler deadlocks and blocking behavior in the rocprofv3 tool when signals are delivered during profiling.
Why handle signals
Some applications run as a tree of processes, often one worker per GPU or device, and manage their own shutdown. Profiling data is flushed at application "end" after
mainreturns in the usual case.If the app has no signal handlers, we want to flush profiling data when a signal (say
Ctrl+Cfrom the terminal) is delivered. For such a case, rocprofv3 helpfully installs signal handlers by default (unless--disable-signal-handlersis specified), flushes the data, and exits.Well-behaved apps that handle the signal, coordinate their children, and shut down cleanly should use
--disable-signal-handlers. With this flag, rocprofv3 installs no handler and finalizes from anatexithook on the normal exit path.Everything in this PR is for the cases where
--disable-signal-handlersdoes not apply: the app installs no signal handlers (in the parent or in its descendants), or it does not coordinate with its children. In such a case, a signal is fatal. It kills the process through the default action, and a process killed by a signal never runs itsatexithooks, so all profiling data is lost.By default, rocprofv3 handles this by installing its own signal handlers. Because it intercepts signal registration through
LD_PRELOAD, its handler is the one actually installed (the app's handler is saved), so it runs first when a signal arrives.Technical Details
Issues with previous implementation
The previous implementation of the signal handler finalized inline inside the signal handler, had re-entrancy issues, and generally had signal-unsafe code.
Re-entry deadlock: Some libraries install their own signal handler over ours through the intercepted
sigaction, then re-raise the signal with it unblocked (LLVM and Triton do this, vialibamd_comgr.soandlibtriton.so; see [rocprofiler-sdk] Fix signal handler hang in rocprofv3 tool #4358). The re-raised signal re-enters our handler on the same thread while the first call is still insidestd::call_once.std::call_onceis not re-entrant, so the second entry deadlocks on its futex.Inline finalize can deadlock: If the signal arrives while the app is inside
malloc, the handler's own logging and formatting allocate again. That re-enters glibc's arena lock, which the interrupted thread already holds, and deadlocks. In addition, because the handler might be called while a thread is holding some GPU resource, the finalization could immediately deadlock if it needs the same lock to flush the data.Blocked multi-process shutdown: The handler flushed inline and blocked the signalled thread, so the app's own handler never ran. Apps that coordinate a multi-process shutdown (the shape above) would hang.
Improvement decisions
These details of how signals are delivered / handled shape the current redesign.
Some apps have the descendants talk to the parent to coordinate shutdown, over pipes or
signals. As an example, sglang worker processes send the parent aSIGQUITto say they are exiting, and if a worker is slow, the parent force-kills it withSIGKILL.Because of that coordination and the group delivery, a single process can receive more than one signal in quick succession while it is still writing its profile.
The redesigned handler should not flush inline, and must be handled asynchronously.
Implementation
On tool init, we create a worker thread that waits on an
eventfdfor a notification delivered from thesignal-ed thread. The signal handler does not flush inline. It wakes that worker, which begins the flush, and thesignal-ed thread returns to normal execution. Once the flush is complete, the worker restores the app's original handler and callskill(getpid())to re-deliver the signal, so the app's own handler runs next if it has one.fork-ed children get a worker thread even though it's technically illegal for them to use GPU resources. It is still useful for, say, roctx markers, but flush in such a situation is best-effort, not guaranteed.fork+exec/spawncreate the worker thread during re-init of the library viaLD_PRELOAD.The basic flow above (wake the worker, flush, re-raise) already flushes a process that receives one signal. Two more things make it reliable under a multi-process, multi-signal teardown.
Each process defers its own shutdown until its flush is done. The handler re-raises into the app's handler only after the flush completes. So a parent does not begin the teardown that kills its workers until its own flush has finished. Every worker also starts flushing in parallel the moment the group signal arrives. Together that usually gives the workers enough time to finish before the parent gets around to killing them.
Repeat signals are swallowed while a flush is in progress. If a second signal arrives before the flush finishes (a coordination signal from the parent, or a second
Ctrl+C), the handler returns without acting on it. Only once the flush completes does the worker re-raise. Without this, that second signal would end the process mid-write and truncate the output. We saw exactly that when we tried passing the second signal through immediately.For a concrete flow diagram, refer to this comment.
Where this breaks down is the final
SIGKILL.SIGKILLcannot be caught, deferred, or chained and is a hard kill from the kernel. If a parent's teardown reaches a worker withSIGKILLbefore that worker's flush finishes, that worker's data is lost. The only levers are outside our control, such as the app delaying its own teardown or an opt-in cross-process sync. In practice, the flush wins the race because the process that issues theSIGKILLis held back by its own flush first.Test Plan
Test Results
* With a web server like
sglang/uvicorn, a singleCtrl+Cstarts a graceful shutdown that drains the in-flight request first (uvicornprintsWaiting for connections to close), so the server stays up until that request finishes. Aborting thecurl(or a secondCtrl+Cto force quit) ends it. While this looks similar to a deadlock, this is the server's own shutdown behavior, notrocprofv3.Jira ID
JIRA ID
AIROCVAL-48
AIPROFSDK-912
AIPROFSDK-970
AIPROFSDK-999 (some of it)
Related
#4358 (first iteration of this PR), #5551, #6875, #8429
Submission Checklist