Skip to content

fix(clr): [ROCM-30371] cherry-pick #10579 to BKC 10.1 - #10939

Draft
astyrrian1 wants to merge 1 commit into
release/bkc/therock-10.1-20260825from
users/astyrrian1/ROCM-30371-bkc-10.1-20260825-cp-10579
Draft

fix(clr): [ROCM-30371] cherry-pick #10579 to BKC 10.1#10939
astyrrian1 wants to merge 1 commit into
release/bkc/therock-10.1-20260825from
users/astyrrian1/ROCM-30371-bkc-10.1-20260825-cp-10579

Conversation

@astyrrian1

@astyrrian1 astyrrian1 commented Aug 29, 2026

Copy link
Copy Markdown

Motivation

AccumulateCommand paired graph kernel names with GPU timestamps positionally: names were appended as each AQL packet was written, timings were appended as each completion signal was drained, and ReportActivity zipped the two vectors. That is correct only if both sequences have identical membership and identical order, and on the segmented graph path neither was guaranteed — so graph traces reported kernels with other kernels' durations, which makes per-kernel graph profiling unusable.

Technical Details

Three effects perturbed the pairing:

  • Registration order — pre-patched segment completion signals bypass ActiveSignal and were registered in bulk in submitAccumulate after every batch had been dispatched, so they landed on the Timestamp that updateCommandsState drains first and their timings arrived ahead of the per-batch dispatch timings.
  • Barrier misclassification — when every node in a batch is disabled the completion signal is relocated onto a standalone barrier, but the signal was classified from the original packet, so a barrier still counted as a kernel dispatch and contributed a timing with no name behind it.
  • Drain order — signals are not necessarily drained in packet order at all. Queue-pool reuse makes ActiveSignal wait on a recycled signal, which extracts that signal's timing immediately, ahead of signals for packets dispatched earlier.

The third cannot be fixed by reordering, since it is the signal pool rather than this code that decides when a timing is extracted. The pairing is made order-independent instead: addKernelDispatch records the kernel name and the queue it was dispatched on during the packet walk and returns that record's slot index, the packet's ProfilingSignal carries the slot in dispatch_slot_, and ExtractSignalTiming writes the timing back through setDispatchTiming rather than appending. Pairing is therefore fixed at the moment the packet is written, and no drain order can disturb it.

That also lets some machinery go away. Holding a dispatch slot replaces the isPacketDispatch_ flag and the packet-type sniffing in ApplyHwEventPatches, so a completion signal relocated onto a barrier is no longer mistaken for a dispatch. The queue a kernel ran on now comes from the dispatching vGPU during the packet walk, removing HwEventPatch::segment_id, HwEventPatch::queue_index, the launch-time loop resolving one into the other, and ProfilingSignal::queue_index_. Pre-patched signals are still registered with the Timestamp from their own packet, found by HSA handle, with the handle read from the host-side flat buffer because the ring slot is write-combining and cannot be read back reliably on the NT path. A slot whose signal never produced valid timing is skipped by ReportActivity instead of shifting every later kernel onto the wrong timestamp.

Separately, __hipUnregisterFatBinary called only SyncAllStreams before RemoveFatBinary. That waits for the GPU to finish and for the host to observe the completion signals, but not for the HSA async-handler thread to leave its callback — and that callback reports kernel names pointing into the Kernel objects RemoveFatBinary is about to destroy, so the final launch's names could reach the trace as freed memory. The handlers are now drained as well.

Issue Tracking

JIRA ID: ROCM-30371

Test Plan

A ten-node fan-out/join HIP graph in which every node runs a uniquely named kernel with a distinct GPU duration (100us through 1000us, spaced 50us apart). A name/timestamp mis-pairing then shows up directly as a kernel reporting a different node's duration, and a dangling kernel name shows up as unparseable bytes in the trace. Run across:

  • the default graph configuration plus two node-disable patterns, which exercise the relocated completion signal;
  • segment scheduling modes 0, 1 and 2;
  • both the NT and MOVDIR64B packet publication paths;
  • forced signal-pool reuse, pool size 2 against 4096, to provoke out-of-order drain.

Test Result

15 traces across the default configuration and the two node-disable patterns pair correctly on every node, with the four parallel paths attributed to four distinct queues. Forcing signal reuse keeps all 2800 dispatches present with their medians within 1.67%.

Submission Checklist

…10579)

## Motivation

`AccumulateCommand` paired graph kernel names with GPU timestamps
positionally: names were appended as each AQL packet was written,
timings were appended as each completion signal was drained, and
`ReportActivity` zipped the two vectors. That is correct only if both
sequences have identical membership and identical order, and on the
segmented graph path neither was guaranteed — so graph traces reported
kernels with other kernels' durations, which makes per-kernel graph
profiling unusable.

## Technical Details

Three effects perturbed the pairing:

- **Registration order** — pre-patched segment completion signals bypass
`ActiveSignal` and were registered in bulk in `submitAccumulate` after
every batch had been dispatched, so they landed on the `Timestamp` that
`updateCommandsState` drains first and their timings arrived ahead of
the per-batch dispatch timings.
- **Barrier misclassification** — when every node in a batch is disabled
the completion signal is relocated onto a standalone barrier, but the
signal was classified from the original packet, so a barrier still
counted as a kernel dispatch and contributed a timing with no name
behind it.
- **Drain order** — signals are not necessarily drained in packet order
at all. Queue-pool reuse makes `ActiveSignal` wait on a recycled signal,
which extracts that signal's timing immediately, ahead of signals for
packets dispatched earlier.

The third cannot be fixed by reordering, since it is the signal pool
rather than this code that decides when a timing is extracted. The
pairing is made order-independent instead: `addKernelDispatch` records
the kernel name and the queue it was dispatched on during the packet
walk and returns that record's slot index, the packet's
`ProfilingSignal` carries the slot in `dispatch_slot_`, and
`ExtractSignalTiming` writes the timing back through `setDispatchTiming`
rather than appending. Pairing is therefore fixed at the moment the
packet is written, and no drain order can disturb it.

That also lets some machinery go away. Holding a dispatch slot replaces
the `isPacketDispatch_` flag and the packet-type sniffing in
`ApplyHwEventPatches`, so a completion signal relocated onto a barrier
is no longer mistaken for a dispatch. The queue a kernel ran on now
comes from the dispatching vGPU during the packet walk, removing
`HwEventPatch::segment_id`, `HwEventPatch::queue_index`, the launch-time
loop resolving one into the other, and `ProfilingSignal::queue_index_`.
Pre-patched signals are still registered with the `Timestamp` from their
own packet, found by HSA handle, with the handle read from the host-side
flat buffer because the ring slot is write-combining and cannot be read
back reliably on the NT path. A slot whose signal never produced valid
timing is skipped by `ReportActivity` instead of shifting every later
kernel onto the wrong timestamp.

Separately, `__hipUnregisterFatBinary` called only `SyncAllStreams`
before `RemoveFatBinary`. That waits for the GPU to finish and for the
host to observe the completion signals, but not for the HSA
async-handler thread to leave its callback — and that callback reports
kernel names pointing into the `Kernel` objects `RemoveFatBinary` is
about to destroy, so the final launch's names could reach the trace as
freed memory. The handlers are now drained as well.

## Issue Tracking

JIRA ID: AIRUNTIME-N

## Test Plan

A ten-node fan-out/join HIP graph in which every node runs a uniquely
named kernel with a distinct GPU duration (100us through 1000us, spaced
50us apart). A name/timestamp mis-pairing then shows up directly as a
kernel reporting a different node's duration, and a dangling kernel name
shows up as unparseable bytes in the trace. Run across:

- the default graph configuration plus two node-disable patterns, which
exercise the relocated completion signal;
- segment scheduling modes 0, 1 and 2;
- both the NT and MOVDIR64B packet publication paths;
- forced signal-pool reuse, pool size 2 against 4096, to provoke
out-of-order drain.

## Test Result

15 traces across the default configuration and the two node-disable
patterns pair correctly on every node, with the four parallel paths
attributed to four distinct queues. Forcing signal reuse keeps all 2800
dispatches present with their medians within 1.67%.

## Submission Checklist

- [x] Look over the contributing guidelines at
https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

(cherry picked from commit 5aa428a)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants