Using semaphore to sync with all peer processes in finalization stage by systems-assistant[bot] · Pull Request #169 · ROCm/rocm-systems

systems-assistant · 2025-08-07T19:18:11Z

[rocprofv3] Implement synchronization using POSIX semaphore in finalization

PR Details

Associated Jira Ticket Number/Link

Previously in https://github.com/AMD-ROCm-Internal/rocprofiler-sdk-internal/pull/500

What type of PR is this? (check all applicable)

Technical details

This PR is to fix https://ontrack-internal.amd.com/browse/SWDEV-539022. In some workload, during the exiting stage, the framework uses a fast-fail pattern: if one process finishes, it kills all peer processes. E.g. in vLLM:
vllm/executor/multiproc_worker_utils.py:127:

This fix is using a POSIX semaphore to sync with peer processes, wait all peers finish and keep executing.

Added/updated tests?

Yes
No, Does not apply to this PR.

Updated CHANGELOG?

Yes
No, Does not apply to this PR.

Added/Updated documentation?

Yes
No, Does not apply to this PR.

🔁 Imported from ROCm/rocprofiler-sdk#114
🧑‍💻 Originally authored by @rocm-devops

* Generate codecoverage comment as collapsible summary * Tweak markdown formatting [ROCm/rocprofiler-sdk commit: 58ecbd8]

For compatibility with recent rocprofiler-sdk change. [ROCm/rocprofiler-systems commit: 2680ccc]

Update job_stats_sample.rst Co-authored-by: srawat <120587655+SwRaw@users.noreply.github.com> [ROCm/rdc commit: 4a230f0]

ammarwa · 2025-08-15T16:17:57Z

Code Coverage Report

Tests Only

Samples Only

Tests + Samples

[ROCm/hip-tests commit: 6e2c871]

bwelton

I had reviewed and signed off on the original PR in the other repo. Signing off here.

[rocprofv3] Implement synchronization using POSIX semaphore in finalization

Apply suggestion by Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

finalization stage (#169) * Using semaphore to sync with all peer processes in finalization stage [rocprofv3] Implement synchronization using POSIX semaphore in finalization * clang format code * clang 11 format code * Add process sync option for rocprofv3 * Default value of process sync is false * Update source/lib/rocprofiler-sdk-tool/tool.cpp Apply suggestion by Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * update according to comments * add new line to helper.hpp [rocm-systems] ROCm/rocm-systems#169 (commit b645010)

Bumps [requests](https://github.com/psf/requests) from 2.32.3 to 2.32.4. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.32.3...v2.32.4) --- updated-dependencies: - dependency-name: requests dependency-version: 2.32.4 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [requests](https://github.com/psf/requests) from 2.32.3 to 2.32.4. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.32.3...v2.32.4) --- updated-dependencies: - dependency-name: requests dependency-version: 2.32.4 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> [ROCm/rocshmem commit: 49f7f1b]

…ocs/sphinx (#169) Bumps [rocm-docs-core[api_reference]](https://github.com/ROCm/rocm-docs-core) from 1.21.1 to 1.22.0. - [Release notes](https://github.com/ROCm/rocm-docs-core/releases) - [Changelog](https://github.com/ROCm/rocm-docs-core/blob/develop/CHANGELOG.md) - [Commits](ROCm/rocm-docs-core@v1.21.1...v1.22.0) --- updated-dependencies: - dependency-name: rocm-docs-core[api_reference] dependency-version: 1.22.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ocs/sphinx (#169) Bumps [rocm-docs-core[api_reference]](https://github.com/ROCm/rocm-docs-core) from 1.21.1 to 1.22.0. - [Release notes](https://github.com/ROCm/rocm-docs-core/releases) - [Changelog](https://github.com/ROCm/rocm-docs-core/blob/develop/CHANGELOG.md) - [Commits](ROCm/rocm-docs-core@v1.21.1...v1.22.0) --- updated-dependencies: - dependency-name: rocm-docs-core[api_reference] dependency-version: 1.22.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> [ROCm/rocjpeg commit: a43e7f1]

Adds instructions on installing hipFile using the nightly packages and building aiscp.

Adds instructions on installing hipFile using the nightly packages and building aiscp. [ROCm/hipFile commit: 77d9fd5]

## Motivation Re-enable the ROCpd `matrix-exponential` validation test. It appears I may have disabled it due to a failure that was unique to my system. - Said failure involves `OMP_PROC_BIND` being on spread and running it when it could detect enough CPUs (a wide enough `OMP_PLACES` list) that the small OpenMP teams get a large spread stride, which tripped some local `LLVM/ROCm libomp` schedule(dynamic) dispatch bug, which silently drops loop iterations so fewer than the expected 171 rocblas_dgemm dispatches/markers get recorded and the ROCpd equals 171 check fails. - I say "local" as I was unable to reproduce this anywhere else but on my work system. So I am assuming this is a problem with my PC. It is disabled on TheRock as the binary is not built, but that's a different issue. **Note**: #7664 will fix the test regex in gfx950CI workflow. ## Technical Details - Enable ROCpd validation test. - Remove `matrix-exponential` from gfx950 exclusion. - Remove `pass_regex` for the test. The kernel may or may not be printed in stdout. Instead, validation is left to perfetto and rocpd, which both the `gfx950` and TheRock will validate against. ## JIRA ID Jira ID : AIPROFSYST-418 ## Test Plan gfx950 workflow (cannot be ran on non-gpu workflows and cannot be ran on TheRock CI either). ## Test Result tests pass: https://github.com/ROCm/rocm-systems/actions/runs/28790692837/job/85368413055?pr=7662 ``` Start 168: matrix-exponential-binary-rewrite 5/125 Test #168: matrix-exponential-binary-rewrite .............................. Passed 10.49 sec Start 169: matrix-exponential-sys-run 6/125 Test #169: matrix-exponential-sys-run ..................................... Passed 5.91 sec ``` ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

systems-assistant Bot added the imported pr label Aug 7, 2025

systems-assistant Bot requested review from SrirakshaNag and bgopesh as code owners August 7, 2025 19:18

systems-assistant Bot added the project: rocprofiler-sdk label Aug 7, 2025

jayhawk-commits assigned huanrwan-amd Aug 7, 2025

jayhawk-commits added the organization: ROCm label Aug 7, 2025

ammallya pushed a commit that referenced this pull request Aug 7, 2025

Generate code coverage comment as collapsible summary (#169)

33a480d

* Generate codecoverage comment as collapsible summary * Tweak markdown formatting [ROCm/rocprofiler-sdk commit: 58ecbd8]

jayhawk-commits added ias-review ready for 2nd review labels Aug 7, 2025

ywang103-amd pushed a commit to ywang103-amd/rocm-systems that referenced this pull request Aug 7, 2025

Fix rocprofiler-sdk includes (ROCm#169)

83cc60f

For compatibility with recent rocprofiler-sdk change. [ROCm/rocprofiler-systems commit: 2680ccc]

jayhawk-commits pushed a commit that referenced this pull request Aug 8, 2025

fix broken link (#169)

f9ceb0e

Update job_stats_sample.rst Co-authored-by: srawat <120587655+SwRaw@users.noreply.github.com> [ROCm/rdc commit: 4a230f0]

systems-assistant Bot pushed a commit that referenced this pull request Aug 10, 2025

Test cases for Fp4 and fp6 coversion apis (#169)

6e2c871

huanrwan-amd force-pushed the import/develop/ROCm_rocprofiler-sdk/huanran_finalization_processes_sync branch from 4fc07a5 to a381bc7 Compare August 14, 2025 20:55

github-actions Bot added the documentation Improvements or additions to documentation label Aug 14, 2025

subbul-amd reviewed Aug 15, 2025

View reviewed changes

Comment thread projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/helper.hpp Outdated

Comment thread projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/tool.cpp Outdated

huanrwan-amd force-pushed the import/develop/ROCm_rocprofiler-sdk/huanran_finalization_processes_sync branch 2 times, most recently from 1983ccb to cb60831 Compare August 18, 2025 17:03

jayhawk-commits pushed a commit that referenced this pull request Aug 18, 2025

Test cases for Fp4 and fp6 coversion apis (#169)

6735af9

[ROCm/hip-tests commit: 6e2c871]

bwelton self-assigned this Aug 19, 2025

huanrwan-amd force-pushed the import/develop/ROCm_rocprofiler-sdk/huanran_finalization_processes_sync branch from cb60831 to cff2b99 Compare August 19, 2025 18:19

bwelton approved these changes Aug 19, 2025

View reviewed changes

jrmadsen approved these changes Aug 19, 2025

View reviewed changes

huanrwan-amd force-pushed the import/develop/ROCm_rocprofiler-sdk/huanran_finalization_processes_sync branch from cff2b99 to b0ecb7d Compare August 22, 2025 15:10

huanrwan-amd and others added 6 commits August 25, 2025 08:14

Using semaphore to sync with all peer processes in finalization stage

4a73405

[rocprofv3] Implement synchronization using POSIX semaphore in finalization

clang format code

e479eb3

clang 11 format code

9849f18

Add process sync option for rocprofv3

34dace6

Default value of process sync is false

7d0eea3

Update source/lib/rocprofiler-sdk-tool/tool.cpp

96e4841

Apply suggestion by Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

huanrwan-amd added 2 commits August 25, 2025 08:14

update according to comments

9f2ddd8

add new line to helper.hpp

33a8dab

ammarwa force-pushed the import/develop/ROCm_rocprofiler-sdk/huanran_finalization_processes_sync branch from b0ecb7d to 33a8dab Compare August 25, 2025 13:14

ammarwa merged commit b645010 into develop Aug 25, 2025
33 of 37 checks passed

ammarwa deleted the import/develop/ROCm_rocprofiler-sdk/huanran_finalization_processes_sync branch August 25, 2025 13:57

SrirakshaNag added the ready for peer review label Sep 9, 2025

ammallya pushed a commit that referenced this pull request Apr 9, 2026

Add quick install section to INSTALL.md (#169)

77d9fd5

Adds instructions on installing hipFile using the nightly packages and building aiscp.

ammallya pushed a commit that referenced this pull request Apr 9, 2026

Add quick install section to INSTALL.md (#169)

28b9c98

Adds instructions on installing hipFile using the nightly packages and building aiscp. [ROCm/hipFile commit: 77d9fd5]

Navigation Menu

Uh oh!

Using semaphore to sync with all peer processes in finalization stage - #169

Using semaphore to sync with all peer processes in finalization stage#169
ammarwa merged 8 commits into
developfrom
import/develop/ROCm_rocprofiler-sdk/huanran_finalization_processes_sync

systems-assistant Bot commented Aug 7, 2025 •

edited by huanrwan-amd

Loading

Uh oh!

Uh oh!

Uh oh!

ammarwa commented Aug 15, 2025 •

edited

Loading

Code Coverage Report

Code Coverage Report

Tests Only

Samples Only

Tests + Samples

Uh oh!

bwelton left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

systems-assistant Bot commented Aug 7, 2025 • edited by huanrwan-amd Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Details

Associated Jira Ticket Number/Link

What type of PR is this? (check all applicable)

Technical details

Added/updated tests?

Updated CHANGELOG?

Added/Updated documentation?

Uh oh!

Uh oh!

Uh oh!

ammarwa commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Report

Code Coverage Report

Tests Only

Samples Only

Tests + Samples

Uh oh!

bwelton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

systems-assistant Bot commented Aug 7, 2025 •

edited by huanrwan-amd

Loading

ammarwa commented Aug 15, 2025 •

edited

Loading