fix(clr): [ROCM-30371] cherry-pick #10579 to BKC 10.1 - #10939
Draft
astyrrian1 wants to merge 1 commit into
Draft
Conversation
…10579) ## Motivation `AccumulateCommand` paired graph kernel names with GPU timestamps positionally: names were appended as each AQL packet was written, timings were appended as each completion signal was drained, and `ReportActivity` zipped the two vectors. That is correct only if both sequences have identical membership and identical order, and on the segmented graph path neither was guaranteed — so graph traces reported kernels with other kernels' durations, which makes per-kernel graph profiling unusable. ## Technical Details Three effects perturbed the pairing: - **Registration order** — pre-patched segment completion signals bypass `ActiveSignal` and were registered in bulk in `submitAccumulate` after every batch had been dispatched, so they landed on the `Timestamp` that `updateCommandsState` drains first and their timings arrived ahead of the per-batch dispatch timings. - **Barrier misclassification** — when every node in a batch is disabled the completion signal is relocated onto a standalone barrier, but the signal was classified from the original packet, so a barrier still counted as a kernel dispatch and contributed a timing with no name behind it. - **Drain order** — signals are not necessarily drained in packet order at all. Queue-pool reuse makes `ActiveSignal` wait on a recycled signal, which extracts that signal's timing immediately, ahead of signals for packets dispatched earlier. The third cannot be fixed by reordering, since it is the signal pool rather than this code that decides when a timing is extracted. The pairing is made order-independent instead: `addKernelDispatch` records the kernel name and the queue it was dispatched on during the packet walk and returns that record's slot index, the packet's `ProfilingSignal` carries the slot in `dispatch_slot_`, and `ExtractSignalTiming` writes the timing back through `setDispatchTiming` rather than appending. Pairing is therefore fixed at the moment the packet is written, and no drain order can disturb it. That also lets some machinery go away. Holding a dispatch slot replaces the `isPacketDispatch_` flag and the packet-type sniffing in `ApplyHwEventPatches`, so a completion signal relocated onto a barrier is no longer mistaken for a dispatch. The queue a kernel ran on now comes from the dispatching vGPU during the packet walk, removing `HwEventPatch::segment_id`, `HwEventPatch::queue_index`, the launch-time loop resolving one into the other, and `ProfilingSignal::queue_index_`. Pre-patched signals are still registered with the `Timestamp` from their own packet, found by HSA handle, with the handle read from the host-side flat buffer because the ring slot is write-combining and cannot be read back reliably on the NT path. A slot whose signal never produced valid timing is skipped by `ReportActivity` instead of shifting every later kernel onto the wrong timestamp. Separately, `__hipUnregisterFatBinary` called only `SyncAllStreams` before `RemoveFatBinary`. That waits for the GPU to finish and for the host to observe the completion signals, but not for the HSA async-handler thread to leave its callback — and that callback reports kernel names pointing into the `Kernel` objects `RemoveFatBinary` is about to destroy, so the final launch's names could reach the trace as freed memory. The handlers are now drained as well. ## Issue Tracking JIRA ID: AIRUNTIME-N ## Test Plan A ten-node fan-out/join HIP graph in which every node runs a uniquely named kernel with a distinct GPU duration (100us through 1000us, spaced 50us apart). A name/timestamp mis-pairing then shows up directly as a kernel reporting a different node's duration, and a dangling kernel name shows up as unparseable bytes in the trace. Run across: - the default graph configuration plus two node-disable patterns, which exercise the relocated completion signal; - segment scheduling modes 0, 1 and 2; - both the NT and MOVDIR64B packet publication paths; - forced signal-pool reuse, pool size 2 against 4096, to provoke out-of-order drain. ## Test Result 15 traces across the default configuration and the two node-disable patterns pair correctly on every node, with the four parallel paths attributed to four distinct queues. Forcing signal reuse keeps all 2800 dispatches present with their medians within 1.67%. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. (cherry picked from commit 5aa428a)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
AccumulateCommandpaired graph kernel names with GPU timestamps positionally: names were appended as each AQL packet was written, timings were appended as each completion signal was drained, andReportActivityzipped the two vectors. That is correct only if both sequences have identical membership and identical order, and on the segmented graph path neither was guaranteed — so graph traces reported kernels with other kernels' durations, which makes per-kernel graph profiling unusable.Technical Details
Three effects perturbed the pairing:
ActiveSignaland were registered in bulk insubmitAccumulateafter every batch had been dispatched, so they landed on theTimestampthatupdateCommandsStatedrains first and their timings arrived ahead of the per-batch dispatch timings.ActiveSignalwait on a recycled signal, which extracts that signal's timing immediately, ahead of signals for packets dispatched earlier.The third cannot be fixed by reordering, since it is the signal pool rather than this code that decides when a timing is extracted. The pairing is made order-independent instead:
addKernelDispatchrecords the kernel name and the queue it was dispatched on during the packet walk and returns that record's slot index, the packet'sProfilingSignalcarries the slot indispatch_slot_, andExtractSignalTimingwrites the timing back throughsetDispatchTimingrather than appending. Pairing is therefore fixed at the moment the packet is written, and no drain order can disturb it.That also lets some machinery go away. Holding a dispatch slot replaces the
isPacketDispatch_flag and the packet-type sniffing inApplyHwEventPatches, so a completion signal relocated onto a barrier is no longer mistaken for a dispatch. The queue a kernel ran on now comes from the dispatching vGPU during the packet walk, removingHwEventPatch::segment_id,HwEventPatch::queue_index, the launch-time loop resolving one into the other, andProfilingSignal::queue_index_. Pre-patched signals are still registered with theTimestampfrom their own packet, found by HSA handle, with the handle read from the host-side flat buffer because the ring slot is write-combining and cannot be read back reliably on the NT path. A slot whose signal never produced valid timing is skipped byReportActivityinstead of shifting every later kernel onto the wrong timestamp.Separately,
__hipUnregisterFatBinarycalled onlySyncAllStreamsbeforeRemoveFatBinary. That waits for the GPU to finish and for the host to observe the completion signals, but not for the HSA async-handler thread to leave its callback — and that callback reports kernel names pointing into theKernelobjectsRemoveFatBinaryis about to destroy, so the final launch's names could reach the trace as freed memory. The handlers are now drained as well.Issue Tracking
JIRA ID: ROCM-30371
Test Plan
A ten-node fan-out/join HIP graph in which every node runs a uniquely named kernel with a distinct GPU duration (100us through 1000us, spaced 50us apart). A name/timestamp mis-pairing then shows up directly as a kernel reporting a different node's duration, and a dangling kernel name shows up as unparseable bytes in the trace. Run across:
Test Result
15 traces across the default configuration and the two node-disable patterns pair correctly on every node, with the four parallel paths attributed to four distinct queues. Forcing signal reuse keeps all 2800 dispatches present with their medians within 1.67%.
Submission Checklist