Skip to content

type(fix): [rocprofiler-sdk] rework signal handling - #6717

Open
MythreyaK wants to merge 18 commits into
developfrom
users/mkuriche/rocprofv3-signal-handler-fix
Open

type(fix): [rocprofiler-sdk] rework signal handling#6717
MythreyaK wants to merge 18 commits into
developfrom
users/mkuriche/rocprofv3-signal-handler-fix

Conversation

@MythreyaK

@MythreyaK MythreyaK commented Jun 2, 2026

Copy link
Copy Markdown
Member

Motivation

Fix signal handler deadlocks and blocking behavior in the rocprofv3 tool when signals are delivered during profiling.

Why handle signals

Some applications run as a tree of processes, often one worker per GPU or device, and manage their own shutdown. Profiling data is flushed at application "end" after main returns in the usual case.

If the app has no signal handlers, we want to flush profiling data when a signal (say Ctrl+C from the terminal) is delivered. For such a case, rocprofv3 helpfully installs signal handlers by default (unless --disable-signal-handlers is specified), flushes the data, and exits.

Well-behaved apps that handle the signal, coordinate their children, and shut down cleanly should use --disable-signal-handlers. With this flag, rocprofv3 installs no handler and finalizes from an atexit hook on the normal exit path.

Everything in this PR is for the cases where --disable-signal-handlers does not apply: the app installs no signal handlers (in the parent or in its descendants), or it does not coordinate with its children. In such a case, a signal is fatal. It kills the process through the default action, and a process killed by a signal never runs its atexit hooks, so all profiling data is lost.

By default, rocprofv3 handles this by installing its own signal handlers. Because it intercepts signal registration through LD_PRELOAD, its handler is the one actually installed (the app's handler is saved), so it runs first when a signal arrives.


Technical Details

Issues with previous implementation

The previous implementation of the signal handler finalized inline inside the signal handler, had re-entrancy issues, and generally had signal-unsafe code.

  • Re-entry deadlock: Some libraries install their own signal handler over ours through the intercepted sigaction, then re-raise the signal with it unblocked (LLVM and Triton do this, via libamd_comgr.so and libtriton.so; see [rocprofiler-sdk] Fix signal handler hang in rocprofv3 tool #4358). The re-raised signal re-enters our handler on the same thread while the first call is still inside std::call_once. std::call_once is not re-entrant, so the second entry deadlocks on its futex.

  • Inline finalize can deadlock: If the signal arrives while the app is inside malloc, the handler's own logging and formatting allocate again. That re-enters glibc's arena lock, which the interrupted thread already holds, and deadlocks. In addition, because the handler might be called while a thread is holding some GPU resource, the finalization could immediately deadlock if it needs the same lock to flush the data.

  • Blocked multi-process shutdown: The handler flushed inline and blocked the signalled thread, so the app's own handler never ran. Apps that coordinate a multi-process shutdown (the shape above) would hang.

Improvement decisions

These details of how signals are delivered / handled shape the current redesign.

  • Some apps have the descendants talk to the parent to coordinate shutdown, over pipes or signals. As an example, sglang worker processes send the parent a SIGQUIT to say they are exiting, and if a worker is slow, the parent force-kills it with SIGKILL.

  • Because of that coordination and the group delivery, a single process can receive more than one signal in quick succession while it is still writing its profile.

  • The redesigned handler should not flush inline, and must be handled asynchronously.

Implementation

On tool init, we create a worker thread that waits on an eventfd for a notification delivered from the signal-ed thread. The signal handler does not flush inline. It wakes that worker, which begins the flush, and the signal-ed thread returns to normal execution. Once the flush is complete, the worker restores the app's original handler and calls kill(getpid()) to re-deliver the signal, so the app's own handler runs next if it has one.

fork-ed children get a worker thread even though it's technically illegal for them to use GPU resources. It is still useful for, say, roctx markers, but flush in such a situation is best-effort, not guaranteed. fork+exec/spawn create the worker thread during re-init of the library via LD_PRELOAD.

The basic flow above (wake the worker, flush, re-raise) already flushes a process that receives one signal. Two more things make it reliable under a multi-process, multi-signal teardown.

  • Each process defers its own shutdown until its flush is done. The handler re-raises into the app's handler only after the flush completes. So a parent does not begin the teardown that kills its workers until its own flush has finished. Every worker also starts flushing in parallel the moment the group signal arrives. Together that usually gives the workers enough time to finish before the parent gets around to killing them.

  • Repeat signals are swallowed while a flush is in progress. If a second signal arrives before the flush finishes (a coordination signal from the parent, or a second Ctrl+C), the handler returns without acting on it. Only once the flush completes does the worker re-raise. Without this, that second signal would end the process mid-write and truncate the output. We saw exactly that when we tried passing the second signal through immediately.

For a concrete flow diagram, refer to this comment.

Where this breaks down is the final SIGKILL. SIGKILL cannot be caught, deferred, or chained and is a hard kill from the kernel. If a parent's teardown reaches a worker with SIGKILL before that worker's flush finishes, that worker's data is lost. The only levers are outside our control, such as the app delaying its own teardown or an opt-in cross-process sync. In practice, the flush wins the race because the process that issues the SIGKILL is held back by its own flush first.

Test Plan

  • Normal exit: rocprofv3 completes, data flushed, no hang
  • SIGINT during profiling: worker finalizes, data flushed, process exits cleanly (exit 130)
  • App with custom SIGINT handler: app's handler fires after profiler finalization
  • Double Ctrl+C during flush: swallowed until finalization completes (no mid-flush data loss)
  • LLVM re-entry scenario (comgr loaded): no deadlock on Ctrl+C
  • Existing CTest signal handler tests pass
  • Signal during malloc stress test: no deadlock
  • Multi-process (sglang-like) coordinated shutdown: no hang, data flushed*
  • vllm does not hang with Ctrl+C, rocprofv3 flushes data

Test Results

  • Signal-handler ctests: 54/54 pass ({good,bad} × {single, fork, fork-exec, spawn} + coordinated-fork).
  • Real sglang (8-way tensor parallel, Qwen3-32B, gfx942): Ctrl+C flushes ~358 MB and the process exits*.

* With a web server like sglang/uvicorn, a single Ctrl+C starts a graceful shutdown that drains the in-flight request first (uvicorn prints Waiting for connections to close), so the server stays up until that request finishes. Aborting the curl (or a second Ctrl+C to force quit) ends it. While this looks similar to a deadlock, this is the server's own shutdown behavior, not rocprofv3.

Jira ID

JIRA ID

AIROCVAL-48
AIPROFSDK-912
AIPROFSDK-970
AIPROFSDK-999 (some of it)

Related

#4358 (first iteration of this PR), #5551, #6875, #8429

Submission Checklist

@MythreyaK MythreyaK self-assigned this Jun 2, 2026
@MythreyaK
MythreyaK force-pushed the users/mkuriche/rocprofv3-signal-handler-fix branch 2 times, most recently from 56dacbb to e2664a5 Compare June 6, 2026 01:22
@MythreyaK
MythreyaK force-pushed the users/mkuriche/rocprofv3-signal-handler-fix branch from e2664a5 to a63f2d7 Compare June 12, 2026 19:43
@MythreyaK
MythreyaK marked this pull request as ready for review June 12, 2026 22:26
@MythreyaK
MythreyaK requested review from a team as code owners June 12, 2026 22:26
@github-actions

github-actions Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Report

Code Coverage Report

Tests Only

code coverage tests.png

Samples Only

code coverage samples.png

Tests + Samples

code coverage all.png

@meserve-amd meserve-amd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this change seems good. Mostly minor feedback.

Comment thread projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/tool.cpp Outdated
Comment thread projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/tool.cpp Outdated
Comment thread projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/tool.cpp Outdated
Comment thread projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/tool.cpp Outdated
Comment thread projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/tool.cpp
Comment thread projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/tool.cpp Outdated
Comment thread projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/tool.cpp Outdated
Comment thread projects/rocprofiler-sdk/tests/bin/signal-handler/signal-handler-test.cpp Outdated
Comment thread projects/rocprofiler-sdk/tests/rocprofv3/signal-handler/conftest.py
Comment thread projects/rocprofiler-sdk/tests/bin/signal-handler/signal-handler-test.cpp Outdated
@MythreyaK
MythreyaK force-pushed the users/mkuriche/rocprofv3-signal-handler-fix branch from 7ae29e3 to 1044356 Compare June 16, 2026 20:46
@MythreyaK
MythreyaK requested a review from bwelton June 16, 2026 21:30
@MythreyaK
MythreyaK force-pushed the users/mkuriche/rocprofv3-signal-handler-fix branch from ff72464 to 9adeeb2 Compare June 23, 2026 17:28
@MythreyaK

Copy link
Copy Markdown
Member Author

Tests are a lil unstable, working on them.

@MythreyaK
MythreyaK force-pushed the users/mkuriche/rocprofv3-signal-handler-fix branch from 9adeeb2 to 97eb72b Compare June 24, 2026 18:20
minseobshin11 added a commit that referenced this pull request Jun 24, 2026
The rocprofv3-test-app-abort tests (execute + 3 validators) were disabled
unconditionally as "currently unstable". The underlying flakiness was the
signal-handler re-entrancy deadlock that the worker-thread rework in #6717
fixes, so re-enable them here (still disabled under ThreadSanitizer).

Enabling required adapting the test to the reworked handler's contract:

- Drop ROCPROF_INTERNAL_TEST_SIGNAL_HANDLER_VIA_EXIT=1. With the new handler
  this quick_exit()s before the worker thread finalizes, so no output would be
  flushed and the validators would have no data. Removing it routes the abort
  through the real worker-thread finalize/flush path.

- Wrap the command in `bash -c "... || exit $?"`. The handler now re-raises
  SIGINT, terminating the process by signal; CTest treats that as an exception
  that WILL_FAIL does not invert (and SIGINT interrupts the run). The wrapper
  turns it into a normal non-zero exit (128+signo) that WILL_FAIL inverts to a
  pass, while a genuine deadlock still blocks and is caught by the ctest TIMEOUT
  (timeouts are not inverted by WILL_FAIL). `|| exit $?` (not `; exit $?`) keeps
  it a single bash argument so CMake does not split on the semicolon.

- Compare sorted(kernels) against the expected kernel list. The kernel_dispatch
  buffer records are not guaranteed to be in alphabetical order, and the abort
  path can flush the two records in either order, so the trace check must be
  order-independent.

The comment wrapping conforms to the project .cmake-format.yaml (line_width 90).

Test plan: built for gfx942 (MI300X, ROCm 7.2.3); ctest -R app-abort passes;
ctest --repeat until-fail:500 (2000 sub-tests) 0 failures; 1920 concurrent
abort cycles (192-way, all 8 GPUs) 0 hangs and all output flushed.

Co-authored-by: Cursor <cursoragent@cursor.com>
@MythreyaK
MythreyaK force-pushed the users/mkuriche/rocprofv3-signal-handler-fix branch from 97eb72b to 0ad8866 Compare June 29, 2026 15:46
@therock-pr-bot

therock-pr-bot Bot commented Jun 29, 2026

Copy link
Copy Markdown

✅ All Policy Checks Passed

Check Status Details
📝 PR Description ✅ Pass
Forbidden Files ✅ Pass
🧪 Unit Test ⚠️ Warning Error: Source/code files changed without an accompanying unit test.
Expected: add at least one test file named like test_<name>.py / test_<name>.cpp (or <name>_test.*).
Current: code file(s) changed: projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/tool.cpp, projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/context/correlation_id.cpp, projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hip/hip.cpp, projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hipfile/hipfile.cpp, projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/marker/marker.cpp (+10 more); no test file found
🚫 Draft PR 🔜 To Be Enabled
🚩 Feature Flag 🔜 To Be Enabled
📊 Code Coverage 🔜 To Be Enabled

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

@therock-pr-bot

This comment was marked as resolved.

@MythreyaK MythreyaK changed the title [rocprofiler-sdk] rework signal handling type(fix): [rocprofiler-sdk] rework signal handling Jun 29, 2026
@MythreyaK
MythreyaK force-pushed the users/mkuriche/rocprofv3-signal-handler-fix branch from 0ad8866 to b0d7325 Compare July 6, 2026 19:20
@MythreyaK
MythreyaK force-pushed the users/mkuriche/rocprofv3-signal-handler-fix branch from b0d7325 to a3b6144 Compare July 10, 2026 20:29
Comment on lines +248 to +267

if(ndangling > 0)
{
auto retirement_ctxs = get_active_contexts([](const context* ctx) {
return (ctx->buffered_tracer &&
(ctx->buffered_tracer->domains(
ROCPROFILER_BUFFER_TRACING_CORRELATION_ID_RETIREMENT)));
});

if(!retirement_ctxs.empty())
{
ROCP_CI_LOG_IF(INFO, ndangling > 0)
<< "retired dangling correlation IDs: " << ndangling;
}
else
{
ROCP_INFO << "retired dangling correlation IDs: " << ndangling
<< " (no retirement consumer active)";
}
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed for cleaner CI checks for tests added in this PR, otherwise in-flight IDs abort new tests with this PR.

Should we use an env-level toggle instead, so that other tests that rely on this failing in CI fail loudly correctly?

@MythreyaK

Copy link
Copy Markdown
Member Author

correlation ID update is racy with signal-handling drive finalization, leading to some CI flakiness and crashes. Working on a fix.

@MythreyaK
MythreyaK force-pushed the users/mkuriche/rocprofv3-signal-handler-fix branch from 234b2d2 to 4f906d6 Compare August 16, 2026 23:43

@meserve-amd meserve-amd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor comments but the approach seems solid. I think the correlation ID stuff might need a second look beyond this PR to make sure we're handling corner cases correctly.

Comment thread projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/tool.cpp Outdated
Comment thread projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/tool.cpp Outdated
Comment thread projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/tool.cpp Outdated
@MythreyaK
MythreyaK force-pushed the users/mkuriche/rocprofv3-signal-handler-fix branch from 0106b42 to cab9877 Compare August 24, 2026 23:19
ROCP_WARNING << "signal worker: close(eventfd) after fork failed: " << strerror(errno);
}

std::memset(static_cast<void*>(&sw), 0, sizeof(sw));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
std::memset(static_cast<void*>(&sw), 0, sizeof(sw));
sw.~signal_worker_state();

Call the destructor, don't just overwrite the memory. You have a thread in signal_worker_state. Unless you are detaching, you need to join the thread before destroying it.

Comment on lines +2252 to +2269
if(sw.eventfd >= 0)
{
// Write to unblock the worker's read() — closing the fd is UB while read() blocks
uint64_t val = 1;
if(write(sw.eventfd, &val, sizeof(val)) < 0)
{
ROCP_WARNING << "failed to write to signal worker eventfd: " << strerror(errno);
}
}
sw.thread.join();
if(sw.eventfd >= 0)
{
if(close(sw.eventfd) < 0)
{
ROCP_WARNING << "failed to close signal worker eventfd: " << strerror(errno);
}
sw.eventfd = -1;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like something you might want to put in a join member function of signal_worker_state so it can be called here and from the destructor of signal_worker_state.

else

// Join the worker thread if called from a non-worker context (atexit / rocprofv3_main).
if(sw.thread.joinable() && sw.thread.get_id() != std::this_thread::get_id())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part would also be part of the join member function.

Comment on lines +4654 to +4664
if(!atfork_registered)
{
atfork_registered = true;
pthread_atfork(nullptr, nullptr, []() {
// Child handler: reset stale state and spawn a fresh worker.
auto& child_sw = get_signal_worker(true);
child_sw.eventfd = eventfd(0, EFD_CLOEXEC);
if(child_sw.eventfd >= 0) child_sw.thread = std::thread{signal_finalization_worker};
});
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I see why you used memset instead of calling the destructor... but I think I would prefer that to be here, not in get_signal_worker.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved it out and added a join member function.

if(!retirement_ctxs.empty())
{
ROCP_CI_LOG_IF(INFO, ndangling > 0)
<< "retired dangling correlation IDs: " << ndangling;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use fmt::format

{
char range_name[128];
snprintf(range_name, sizeof(range_name), "%s_pid_%d", label, getpid());
roctxRangePush(range_name);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
roctxRangePush(range_name);
roctxRangePush(join(label, "_pid_", getpid()).c_str());

Comment on lines +125 to +128
char iter_name[128];
snprintf(iter_name, sizeof(iter_name), "%s_iter_%d", label, iter);

roctxRangePush(iter_name);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
char iter_name[128];
snprintf(iter_name, sizeof(iter_name), "%s_iter_%d", label, iter);
roctxRangePush(iter_name);
roctxRangePush(join(label, "_iter_", iter).c_str());

Comment on lines +41 to +55
rocprofiler_add_integration_execute_test(
${WITH_HANDLING_NAME}
COMMAND
${SH_EXECUTABLE} -c
"rm -rf '${BASE_OUT_DIR}/good-${PROC_TYPE}' && exec \"$@\"" ${SH_EXECUTABLE}
${TIMEOUT_CMD} $<TARGET_FILE:rocprofiler-sdk::rocprofv3> --hip-trace
--marker-trace --output-format json --disable-signal-handlers -d
${BASE_OUT_DIR}/good-${PROC_TYPE} -o out_%pid% --
$<TARGET_FILE:signal-handler-test> --${PROC_TYPE} --app-signal-handler
DEPENDS signal-handler-test
TIMEOUT ${CTEST_TIMEOUT}
LABELS "integration-tests;signal-handler"
PRELOAD "${PRELOAD_ENV}"
PASS_REGULAR_EXPRESSION "clean exit"
FIXTURES_SETUP ${WITH_HANDLING_NAME})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it necessary to rm -rf?? I really think I would (A) prefer using python script wrapper with subprocess that times out instead of sh and (B) this test have a FIXTURES_REQUIRED on a "clean up" test to ensure the clean-up operation happens before the test runs, e.g. adding a rocprofiler_add_integration_cleanup_test function.

@MythreyaK MythreyaK Aug 28, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it necessary to rm -rf??

Pre-existing files conflict with the test, and wasn't sure of a clean way to run the validate only on newer files. Removing files from older runs was less messy, at the cost of rm -rf. Added a function and remove the loop. Not sure about rm -rf. Kept it for now but can replace it with something safer.

# Generate all 8 test combos: {good, bad} x {single-process, fork, fork-exec, spawn}
# ===========================================================================

foreach(PROC_TYPE ${PROCESS_TYPES})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make this a function and invoke it with fork, fork-exec, and spawn.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Removed the loop and added explicit functions and formatting config.

@@ -0,0 +1,2 @@
[pytest]
addopts = -rA --tb=short

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you deviating from the standard pytest.ini contents here??

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry slipped through my review, thanks for catching this! Fixed.

@MythreyaK
MythreyaK force-pushed the users/mkuriche/rocprofv3-signal-handler-fix branch from d65c70e to 2fdff05 Compare August 28, 2026 01:50
MythreyaK and others added 18 commits August 29, 2026 15:18
Replace async-signal-unsafe rocprofv3_error_signal_handler with a
worker thread architecture. Signal handler is now fully
async-signal-safe: writes eventfd, waits on futex with bounded
timeout, restores original handler, re-raises.

Worker thread performs finalization in normal context where all
APIs are safe to call.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Mark Meserve <mark.meserve@amd.com>
- use atomic<uint32_t>
- add error checking for pipe
- re-add previous removed INFO call for ignored finalization
- re-init on fork after memset
The handler used to wake the finalization worker and then block the
signaled thread until the flush finished, re-raising on that same thread.
If the signal interrupted that thread mid-malloc (holding the arena lock),
the worker's flush would block on the same lock -> deadlock; it also stalled
the app's own coordinated-shutdown handlers.

Now the handler hands off and gets out of the way:

- Async signals (SIGINT/SIGQUIT/SIGTERM): record signo, wake the worker via
  eventfd, and return immediately. Returning lets the interrupted thread
  resume and release any lock it holds, so the worker's (non-async-signal-
  safe) flush cannot deadlock against it.
- The worker owns termination: after finalize + reaping children it restores
  the wrapped disposition (the app's chained handler if one was saved, else
  SIG_DFL) and re-raises process-directed via kill(getpid(), signo) so it
  lands on an application thread (the worker itself blocks these signals).
  With no chained handler it unblocks first so SIG_DFL is guaranteed to exit.
- SIGABRT stays synchronous: abort() resets to SIG_DFL and re-raises the
  instant the handler returns, so we must wait on the finalize_done futex for
  the flush before returning. This is the one path that can still block on a
  held lock -- acceptable since the process is already aborting.
- Re-entry escalates instead of swallowing: if a chained handler (e.g.
  LLVM/comgr) restores our handler and re-raises into us, force SIG_DFL,
  unblock, and re-raise to guarantee termination rather than hang.
Build on the non-blocking handler + worker so signal re-delivery is safe and the
profiler stays transparent to the app's own signal disposition:

- Remove `SA_RESETHAND` so a second delivery of the same signal re-enters our guard
  instead of resetting to `SIG_DFL` and terminating mid-flush (was silent data loss).
- Gate re-entry escalation on finalize_done: swallow repeats while the flush is in
  progress, force `SIG_DFL` + re-raise only after it completes -- a routine double
  `Ctrl+C` no longer truncates the profile, and a stuck flush stays killable.
- Honor `SIG_IGN` symmetrically for `signal()` and `sigaction()` (`sigaction(SIG_IGN)` was
  dropped, wrongly terminating apps that deliberately ignore `Ctrl+C`).
- Drop the unused signal-handler timeout knob; correctness relies on lock release
  and the gated escalation, not a clock.
- signal-handler tests: each execute wipes its own output dir first (`sh` shim, `sh`
  located via `find_program`) so a truncated file from a prior local run can't fail
  validation; CI already starts clean.
… id at finalization

The non-blocking signal handler finalizes on a worker thread, so application
threads can keep calling instrumented APIs while finalization runs.
`correlation_service::construct()` returns nullptr once finalization has begun,
but most interceptors dereferenced it unguarded and crashed (`SIGSEGV` in roctx
range push and HIP under signal-triggered shutdown), surfacing intermittently
in CI.

Fall back to the untraced path (run the underlying API and return) when
`construct()` yields nullptr, via a new `RETURN_UNTRACED_ON_NULL_CORRELATION_ID`
macro. Applied in hip, marker, range_marker, ompt, rccl, rocshmem, hipfile,
rocjpeg and rocdecode; the HSA paths already handled this.
@MythreyaK
MythreyaK force-pushed the users/mkuriche/rocprofv3-signal-handler-fix branch from 2fdff05 to 61a4370 Compare August 29, 2026 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants