f(rccl/gin): fix inverted system-scope fence guard on HIP backends by dlamd1dai · Pull Request #10675 · ROCm/rocm-systems

dlamd1dai · 2026-08-24T23:20:13Z

Supersedes dlamd1dai/rocm-systems#3 (fork PR retargeted upstream). Follow-up to merged #10672.

Summary

The GIN Put/PutValue device templates gate the pre-put system fence on (required==system && given>required). On HIP, hip_compat.h orders thread_scope_system as the max value, so given>required is never true when required==system — the __threadfence_system() branch is dead code, and a caller that only guaranteed a weaker scope (e.g. block) silently skips the system fence it requested.
Flip the guard to (required==system && given<required) on the two HIP backends (anvil_sdma, rocshmem_gda), routed through a new NCCL_GIN_THREADFENCE_SYSTEM() seam in gin_device_common.h (expands to __threadfence_system() in production; unit tests can override it to observe the branch).
gdaki intentionally left as given>required: it is CUDA-only (NCCL_GIN_GDAKI_ENABLE=0 on HIP) and libcu++ orders thread_scope_system as the lowest value — the opposite of HIP — so given>required is correct there. Its comment is expanded to document this.
Tests: install a counting override of the fence seam and replace the former Put_WeakerGivenScopeStillPuts with Put_WeakerGivenScopeFencesAndPuts (required=system, given=block → asserts the fence ran exactly once and the put landed), plus Put_EqualScopeTakesNoFence (required=given=system → asserts no fence and the put landed).

JIRA ID: AICOMRCCL-1527

Base

Rebased onto develop after #10672 merged (834c295). Two commits; diff vs develop: 5 files (+75/−19).

Test plan

Rebased onto develop; clean 2-commit diff (5 files).
Fence-counting GDA template tests: Put_WeakerGivenScopeFencesAndPuts and Put_EqualScopeTakesNoFence.
MI300X (dell300x-ccs-aus-k13-41, gfx942): 34/34 PASS on pre-rebase stack (13 GDA + 12 SDMA + 9 IPC); log ~/rocm-systems/ddai-artifacts/logs/pr3-fixtures-full-20260824-231854.log.
MI355 (smci355-ccs-aus-m03-17, gfx950): 34/34 PASS including fence-counting tests; log ~/rocm-systems/ddai-artifacts/logs/mi355-fixtures-20260827-171030.log.
gdaki path unchanged (CUDA-only; not built/tested in this AMD/HIP tree).

Made with Cursor

Document cherry-picked commits 38f46cd and a2899ae onto the NCCL 2.30.7 compatibility line after a clean apply of the HIP scope guard fix. Co-authored-by: Cursor <cursoragent@cursor.com>

Note 33/34 fixture pass including new G7/G7b fence-counting tests; IPC DetailHelpers gfx950 gap unchanged from PR ROCm#10672. Co-authored-by: Cursor <cursoragent@cursor.com>

Document cherry-picked commits 38f46cd and a2899ae onto the NCCL 2.30.7 compatibility line after a clean apply of the HIP scope guard fix. Co-authored-by: Cursor <cursoragent@cursor.com>

…ixture Update backport-manifest.md with PR ROCm#10672/ROCm#10675 MI355 fixture results and set signal_remote_addrs in DetailHelpers_ChannelAndDirty so useSdmaFusedSignal() passes on gfx950 (34/34 fixtures on MI355). Co-authored-by: Cursor <cursoragent@cursor.com>

therock-pr-bot · 2026-08-27T16:30:12Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

Flip the pre-put system fence guard to (required==system && given<required) on anvil_sdma and rocshmem_gda. Add NCCL_GIN_THREADFENCE_SYSTEM() seam in gin_device_common.h. Expand gdaki comment documenting opposite libcu++ ordering. Test updates for GinRocshmemGdaTemplate_test.cpp remain in the gin-gda-a2a-unittests PR. Co-authored-by: Cursor <cursoragent@cursor.com>

Count NCCL_GIN_THREADFENCE_SYSTEM invocations in GDA template tests so G7 validates the corrected given<required guard fires once when required is system and given is block, and does not fire when scopes match. Co-authored-by: Cursor <cursoragent@cursor.com>

dlamd1dai · 2026-08-27T22:55:31Z

CI triage (infra reds — not PR regressions)

All failing checks appear infra / environment / unrelated tests. None exercised the fence-guard diff in a way that indicates a code problem. Local MI355 fixtures 34/34 PASS (including fence-counting tests).

GIN / Single node gin testing — infra (runner deps)

Failed step: Resolve TheRock artifact run (18s; tests never ran)
Cause: ModuleNotFoundError: No module named 'botocore' on self-hosted runner when resolving TheRock artifacts
Action: Re-run triggered. If it fails again → RCCL CI to install botocore on runner or pin ARTIFACT_RUN_ID from green TheRock CI

device-API / Single node device-api testing — infra (GIN plugin not loaded)

Failed step: SLURM job 29848 (sbatch rc=1)
Cause: gin-d3/d4:alltoall_perf rc=6 — "GIN support is not enabled for this communicator"
Not related: PR only changes device-header fence guards, not GIN plugin loading
Action: Re-run triggered

mci/rocm-libraries/precheckin(rccl) — likely flaky / unrelated

Failed test: AllGather.InPlace on gfx950 (failed original + rerun)
Not related: PR touches GIN device headers only; develop precheckin passes AllGather.InPlace
Action: Re-triggering precheckin

therock-pr-bot — infra (check-wait timeout)

First run timed out waiting for long-running checks; policy checks themselves passed
Re-run in progress after MI355 validation comment

Re-runs: RCCL cluster CI (GIN + device-API) + MCI precheckin re-triggered from this comment.

Empty commit to refresh Jenkins/GitHub checks after infra-only CI failures.

Mirror the GDA fence-counter override in GinAnvilSdmaTemplate_test.cpp so Put_ThreadScopeFence (required=system, given=block) asserts the corrected given<required guard fires exactly once via NCCL_GIN_THREADFENCE_SYSTEM().

…ixture Update backport-manifest.md with PR ROCm#10672/ROCm#10675 MI355 fixture results and set signal_remote_addrs in DetailHelpers_ChannelAndDirty so useSdmaFusedSignal() passes on gfx950 (34/34 fixtures on MI355). Co-authored-by: Cursor <cursoragent@cursor.com>

systems-assistant · 2026-08-31T14:20:04Z

TheRock Submodule Bump Activity

Newest first

2026-08-31 — Included in TheRock via ROCm/TheRock#7760.

dlamd1dai requested a review from a team as a code owner August 24, 2026 23:20

dlamd1dai mentioned this pull request Aug 24, 2026

rccl/gin: fix inverted system-scope fence guard on HIP backends dlamd1dai/rocm-systems#3

Closed

5 tasks

github-actions Bot added the project: rccl label Aug 24, 2026

dlamd1dai changed the title ~~rccl/gin: fix inverted system-scope fence guard on HIP backends~~ f(rccl/gin): fix inverted system-scope fence guard on HIP backends Aug 24, 2026

systems-assistant Bot added the organization: ROCm label Aug 24, 2026

dlamd1dai mentioned this pull request Aug 25, 2026

test(rccl): GIN AllToAll device-template unit tests (gin-gda + gin-sdma shadow fix) #10672

Merged

3 tasks

dlamd1dai mentioned this pull request Aug 25, 2026

feat(rccl/gin-sdma): GIN SDMA AllGather Implementation (-D3 hybrid xGMI-SDMA) dlamd1dai/rocm-systems#4

Open

dlamd1dai force-pushed the users/dondai/gin-gda-a2a-unittests branch from 2dbe02b to 71b597c Compare August 25, 2026 23:19

dlamd1dai force-pushed the users/dondai/gin-scope-fence-guard-fix branch 2 times, most recently from 29e0dbd to 5d8068e Compare August 27, 2026 06:30

Base automatically changed from users/dondai/gin-gda-a2a-unittests to develop August 27, 2026 16:29

dlamd1dai requested a review from a team August 27, 2026 16:29

dlamd1dai and others added 2 commits August 27, 2026 13:46

dlamd1dai force-pushed the users/dondai/gin-scope-fence-guard-fix branch from 5d8068e to dca0117 Compare August 27, 2026 17:47

ci: retrigger MCI precheckin and cluster CI

dd04845

Empty commit to refresh Jenkins/GitHub checks after infra-only CI failures.

pvallem reviewed Aug 28, 2026

View reviewed changes

Comment thread projects/rccl/src/include/nccl_device/gin/anvil_sdma/gin_anvil_sdma.h

pvallem approved these changes Aug 28, 2026

View reviewed changes

amd-jiali approved these changes Aug 28, 2026

View reviewed changes

test(rccl/gin): assert SDMA fence seam in Put_ThreadScopeFence

d17ac7a

Mirror the GDA fence-counter override in GinAnvilSdmaTemplate_test.cpp so Put_ThreadScopeFence (required=system, given=block) asserts the corrected given<required guard fires exactly once via NCCL_GIN_THREADFENCE_SYSTEM().

dlamd1dai assigned dlamd1dai and unassigned dlamd1dai Aug 28, 2026

Merge branch 'develop' into users/dondai/gin-scope-fence-guard-fix

88fcdfc

dlamd1dai merged commit 37ef552 into develop Aug 28, 2026
31 of 36 checks passed

dlamd1dai deleted the users/dondai/gin-scope-fence-guard-fix branch August 28, 2026 23:22

dlamd1dai mentioned this pull request Aug 28, 2026

feat(rccl/gin-sdma): GIN SDMA AllGather Implementation (-D3 hybrid xGMI-SDMA) #10930

Open

Navigation Menu

Uh oh!

f(rccl/gin): fix inverted system-scope fence guard on HIP backends - #10675

f(rccl/gin): fix inverted system-scope fence guard on HIP backends#10675
dlamd1dai merged 5 commits into
developfrom
users/dondai/gin-scope-fence-guard-fix

dlamd1dai commented Aug 24, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Aug 27, 2026

Uh oh!

dlamd1dai commented Aug 27, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

systems-assistant Bot commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

dlamd1dai commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

JIRA ID: AICOMRCCL-1527

Base

Test plan

Uh oh!

therock-pr-bot Bot commented Aug 27, 2026

✅ All Policy Checks Passed

Uh oh!

dlamd1dai commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI triage (infra reds — not PR regressions)

GIN / Single node gin testing — infra (runner deps)

device-API / Single node device-api testing — infra (GIN plugin not loaded)

mci/rocm-libraries/precheckin(rccl) — likely flaky / unrelated

therock-pr-bot — infra (check-wait timeout)

Uh oh!

Uh oh!

Uh oh!

systems-assistant Bot commented Aug 31, 2026

TheRock Submodule Bump Activity

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dlamd1dai commented Aug 24, 2026 •

edited

Loading

dlamd1dai commented Aug 27, 2026 •

edited

Loading