test(rccl): GIN AllToAll device-template unit tests (gin-gda + gin-sdma shadow fix) by dlamd1dai · Pull Request #10672 · ROCm/rocm-systems

dlamd1dai · 2026-08-24T22:58:13Z

Supersedes dlamd1dai/rocm-systems#2 (fork PR retargeted to upstream develop).

Summary

gin-gda: Add GinRocshmemGdaTemplate_test.cpp — 12 unit tests for the GIN rocSHMEM-GDA AllToAll device template (NCCL_NET_DEVICE_GIN_ROCSHMEM_GDA): put_nbi data delivery, indexed/inc signals, counter quiet paths, thread-scope fences, inline PutValue, Flush, and signal/counter get+reset. Self-contained (supplies its own rocshmem::QueuePair device stubs in-TU), registered under an independent ENABLE_ROCSHMEM_GIN guard — no librocshmem device link and no header shadow.
gin-sdma: Fix the Anvil-SDMA unit-test header shadow so the Fixtures build compiles/runs with GIN_ANVIL_UNIT_TESTS=ON:
- test/device/sdma/anvil_device.hpp: rename the stub namespace gin_anvil::sdma → sdma_anvil to match the real Anvil header and the SDMA template test (fixes no member named 'sdma_anvil' in the global namespace).
- test/device/GinAnvilIpcDevice_test.cpp: update the stale 3-arg fenceBeforeSignal(...) call (now takes a leading context arg), fix DetailHelpers_ChannelAndDirty (blockId=2 for effectiveChannel()), and set signal_remote_addrs so useSdmaFusedSignal() passes on gfx950.
CI: Wire GIN Anvil unit tests into the gin CI pipeline (gin.sbatch, run-gin-ci.sh, gin-tests.json). Two-phase build: install.sh --rocshmem-gin (no -t) then reconfigure BUILD_TESTS=ON + GIN_ANVIL_UNIT_TESTS=ON and make rccl-UnitTestsFixtures only.

JIRA ID: AICOMRCCL-1527

Stacking

Base branch: develop. Depends on #10420 (128 MiB Anvil-SDMA put cap), merged. Related (not a hard dependency): #10658 (devtime A2A timing), #10675 (HIP scope fence guard).

Branch tip: 3c23bf7c15 — test + CI files under projects/rccl/test/ and projects/rccl/tools/ci/ only.

Test plan

Built rccl-UnitTestsFixtures with GIN_ANVIL_UNIT_TESTS=ON; 0 residual sdma_anvil namespace errors.
MI300X (gfx942): GDA 12/12, SDMA 12/12, IPC 9/9 — 33/33 PASS
MI355 (gfx950): full GIN fixture filter — 34/34 PASS

CI status (post-rebase)

Green: device-API, static-analysis, TheRock RCCL build (gfx942 + gfx950), Multi-Arch CI Summary, HIP NVIDIA CI, TheRock Test rccl (gfx94X).
GIN CI: build phase green (two-phase gin.sbatch fix); 3/4 perf tests passed on first full run. Fixtures failed at load with undefined symbol: rocshmem::envvar::log_flags because librccl.so is built without host librocshmem on the --rocshmem-gin path. Fixed in 3c23bf7c15: link librocshmem.a into rccl-UnitTestsFixtures when GIN_ANVIL_UNIT_TESTS=ON and ENABLE_ROCSHMEM=OFF.
Likely unrelated reds: Host unit tests (InitMicrotest.DevCommSetup_HappyPath), TheRock single-node gfx94X (SendRecv.UserBufferRegister), mci/precheckin/extra — appear to reproduce on infra/base, not introduced by this diff.

Review updates (addressed on fork PR #2)

@pvallem

GDA stub uses QueuePair::atomic_nofetch matching current queue_pair_device.h.
Tests use ncclGinOffsetPtr from GetSignalPtr/GetCounterPtr; Flush passes descriptor args.
IPC test replaces removed anvilSignalPtrOrDummy with direct signal-slot checks.
CMake: GDA test gated under ENABLE_ROCSHMEM_GIN AND (ENABLE_ROCSHMEM OR GIN_ANVIL_UNIT_TESTS).
G7 fence behavior corrected in #10675.
IPC fenceBeforeSignal compile/launch smoke acknowledged.
gin.sbatch two-phase Fixtures build + run-gin-ci.sh / gin-tests.json gtest filter for GDA/SDMA/IPC device-template tests.

Made with Cursor

therock-pr-bot · 2026-08-24T22:58:56Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

…L 2.30.7 wip Port the GDA/SDMA/IPC device-template fixture fixes and CI fixtures gate from ROCm#10672: updated Flush and ncclGinOffsetPtr call sites, IPC loadConst signal checks, CMake wiring for GDA under GIN_ANVIL_UNIT_TESTS, and gin CI fixtures runner support. Keeps NCCL 2.30.7-specific gin.sbatch rccl-tests link sed patches. Co-authored-by: Cursor <cursoragent@cursor.com>

Document commit 162bf43 and MI355 fixture results (32/33 pass) so the backport manifest reflects what was verified on the NCCL 2.30.7 wip line. Co-authored-by: Cursor <cursoragent@cursor.com>

Note 33/34 fixture pass including new G7/G7b fence-counting tests; IPC DetailHelpers gfx950 gap unchanged from PR ROCm#10672. Co-authored-by: Cursor <cursoragent@cursor.com>

…ixture Update backport-manifest.md with PR ROCm#10672/ROCm#10675 MI355 fixture results and set signal_remote_addrs in DetailHelpers_ChannelAndDirty so useSdmaFusedSignal() passes on gfx950 (34/34 fixtures on MI355). Co-authored-by: Cursor <cursoragent@cursor.com>

Add GinRocshmemGdaTemplate_test.cpp covering the GIN rocSHMEM-GDA specializations of the device Put/PutValue/Flush/Signal/Counter template (NCCL_NET_DEVICE_GIN_ROCSHMEM_GDA): put_nbi data delivery, indexed/inc signals, counter quiet paths, thread-scope fences, inline PutValue, flush, and signal/counter get+reset. The test is self-contained -- it supplies its own rocshmem::QueuePair device stubs in-TU, so it needs no librocshmem device link and no header shadow, and is registered under an independent ENABLE_ROCSHMEM_GIN guard (decoupled from the Anvil-SDMA shadow mechanism). Validated on gfx942: 12/12 GinRocshmemGdaTemplateTest cases pass. Co-authored-by: Cursor <cursoragent@cursor.com>

Enable the Anvil-SDMA device template tests to build and run under the header shadow (GIN_ANVIL_UNIT_TESTS=ON): - test/device/sdma/anvil_device.hpp: the shadow stub declared namespace gin_anvil::sdma, but the real Anvil header and the SDMA template test expect global-scope sdma_anvil. With the shadow enabled this broke every TU that includes gin_anvil_sdma.h ("no member named 'sdma_anvil' in the global namespace"). Rename the stub namespace to sdma_anvil. - test/device/GinAnvilIpcDevice_test.cpp: fix two stale spots surfaced once the shadow build compiled -- fenceBeforeSignal() gained a leading context arg (update the call site), and DetailHelpers_ChannelAndDirty asserted channel 2 via an obsolete bytes/64 formula while effectiveChannel() now uses blockId (pass blockId=2 to keep the intended non-zero rotated-channel check). Validated on gfx942 with the shadow enabled: GinAnvilSdmaTemplateTest, GinAnvilIpcDeviceTest, and GinRocshmemGdaTemplateTest (33 tests) all pass. Co-authored-by: Cursor <cursoragent@cursor.com>

Address PR review on the rocSHMEM-GDA AllToAll device-template unit tests: - Stub was QueuePair::atomic_add, but after the rebase onto the device-time work the template signal path calls QueuePair::atomic_nofetch(void*, uint32_t, int64_t, int64_t cond, ActiveWFInfo&) (gin_rocshmem_gda.h:54,101). The stale stub left atomic_nofetch undefined at link and defined a non-member, so the suite could not compile/link. Rename the stub to atomic_nofetch with the real 5-arg signature and fix the accompanying comments. - The suite body is guarded by NCCL_GIN_ROCSHMEM_GDA_ENABLE, which is 1 only when ENABLE_ROCSHMEM is defined. The test file was listed under ENABLE_ROCSHMEM_GIN alone, so a pure GIN build compiled it out to zero tests and only the ENABLE_ROCSHMEM/GIN_ANVIL_UNIT_TESTS config activated it (where it then hit the stub error) -- no build actually ran these tests. Move the file into the same ENABLE_ROCSHMEM_GIN AND (ENABLE_ROCSHMEM OR GIN_ANVIL_UNIT_TESTS) block as the Anvil-SDMA template test (where ENABLE_ROCSHMEM is defined for the target) and correct the misleading comment. - G7 claimed to exercise the thread-scope fence branch, but the template guard (required==system && given>required) can never be true when required==system, so no fence runs. Rename to Put_WeakerGivenScopeStillPuts and re-comment to reflect it only asserts data landing; the inverted guard is a pre-existing, repo-wide convention (anvil_sdma, gdaki, rocshmem_gda) tracked for a separate coordinated fix. Verified: rccl-UnitTestsFixtures built with GIN_ANVIL_UNIT_TESTS=ON (gfx942); all 12 GinRocshmemGdaTemplateTest cases pass. Co-authored-by: Cursor <cursoragent@cursor.com>

Align GDA QueuePair stub with atomic_add, use ncclGinOffsetPtr from GetSignalPtr/GetCounterPtr, and replace removed anvilSignalPtrOrDummy checks. Co-authored-by: Cursor <cursoragent@cursor.com>

Pass hasDescriptor=false and a null descriptor smem pointer before the memory-order and abort-flag arguments expected by current GIN headers. Co-authored-by: Cursor <cursoragent@cursor.com>

Build rccl-UnitTestsFixtures with GIN_ANVIL_UNIT_TESTS=ON during the RCCL stage and run the 33 GDA/SDMA/IPC device-template cases via run-gin-ci.sh. Co-authored-by: Cursor <cursoragent@cursor.com>

Set signal_remote_addrs in DetailHelpers_ChannelAndDirty so useSdmaFusedSignal() predicate matches oss7 on gfx950; signals alone was insufficient. MI355 fixtures 34/34 pass. Co-authored-by: Cursor <cursoragent@cursor.com>

install.sh -t pulled in rccl-UnitTestsGinAnvilPlugin, which could compile before hipify headers under parallel -j and abort before librccl.so installed. Split into librccl install then a targeted Fixtures build. Co-authored-by: Cursor <cursoragent@cursor.com>

dlamd1dai · 2026-08-26T02:59:38Z

For reviewers — status summary @ROCm/rccl-reviewers @ROCm/rccl-leads

This PR adds GIN AllToAll device-template unit tests (rocSHMEM-GDA + Anvil-SDMA) and fixes the Anvil unit-test header shadow so rccl-UnitTestsFixtures builds/runs with GIN_ANVIL_UNIT_TESTS=ON. It also wires those fixtures into the GIN CI pipeline.

Branch tip: 71b597c4b8 (rebased onto current develop)

Scope (9 files, tests + CI only)

gin-gda: new GinRocshmemGdaTemplate_test.cpp — 12 self-contained GDA template tests (in-TU QueuePair stubs, no librocshmem device link)
gin-sdma: shadow namespace fix (sdma_anvil), stale IPC/SDMA API call sites updated
CI: gin.sbatch / run-gin-ci.sh / gin-tests.json — runs GDA + SDMA + IPC fixture filter in GIN CI

No production RCCL source changes.

Local validation

Platform	Result
MI300X (gfx942)	GDA 12/12, SDMA 12/12, IPC 9/9 — 33/33 PASS
MI355 (gfx950)	full GIN fixture filter — 34/34 PASS

CI fixes landed on this branch

GIN CI build (71b597c4b8): replaced install.sh -t with a two-phase build — install librccl first, then make rccl-UnitTestsFixtures only. The old -t path pulled in rccl-UnitTestsGinAnvilPlugin, which could compile before hipify headers under parallel -j and abort before librccl.so was installed.
Rebase onto develop: branch is up to date with latest develop (includes DDA header move under src/algorithms/dda/).

CI status (post-rebase)

Green: policy bot, static-analysis, device-API, Multi-Arch CI Summary, CodeQL, HIP NVIDIA CI, TheRock RCCL build (gfx942 + gfx950)

Pending: GIN / Single node gin testing — has been queued for an extended period (cluster backlog); this is the check that exercises our new fixture gtests. Will update once it completes.

Red — appear unrelated to this diff:

Check	Failure	Notes
Host unit tests (CPU-only)	`InitMicrotest.DevCommSetup_HappyPath`	CUDA host alloc on fake/CPU-only runner; not in our changed files
mci/precheckin	same `InitMicrotest` failure	same root cause
TheRock single-node	`SendRecv.UserBufferRegister` mismatch	existing MPI unit test; not touched by this PR
mci/extra	`topo_expl` on `ubuntu24-therock && gfx90a` — `algorithms/dda/dda_init_detail.h` not found	topo_expl include-path issue on therock leg; header exists at `src/algorithms/dda/` on rebased branch
therock-pr-bot	timed out waiting for long/queued jobs	expected while GIN CI was still queued

Happy to re-run failed jobs and/or check whether the InitMicrotest / SendRecv.UserBufferRegister / topo_expl failures reproduce on develop. Please let me know if any of these are gating for merge.

Review focus

GDA template test coverage and stub correctness (GinRocshmemGdaTemplate_test.cpp)
SDMA shadow fix (anvil_device.hpp namespace, IPC gfx950 fused-signal predicate)
CMake gating: ENABLE_ROCSHMEM_GIN AND (ENABLE_ROCSHMEM OR GIN_ANVIL_UNIT_TESTS)
GIN CI wiring: two-phase Fixtures build in gin.sbatch, fixtures kind in run-gin-ci.sh

Prior fork review feedback from @pvallem is addressed (see PR description). Related but not blocking: #10675 (HIP scope fence guard).

Requesting review when convenient. Thanks!

When librccl.so is built with ENABLE_ROCSHMEM_GIN but without host librocshmem (GIN CI --rocshmem-gin path), rccl-UnitTestsFixtures fails at load with undefined rocshmem::envvar::log_flags. Link librocshmem.a into the Fixtures binary so the executable satisfies those symbols for librccl.so at runtime. Co-authored-by: Cursor <cursoragent@cursor.com>

…DA stub Use ASCII unit separator in parse_gin_config so empty env fields no longer break bash tab parsing; add parser regression test and NCCL_DEBUG=WARN on the fixtures matrix row. Stop defining ENABLE_ROCSHMEM on Fixtures when only GIN_ANVIL_UNIT_TESTS is set (match librccl.so ABI on the GIN CI path). Count put_nbi stub invocations so Put_ZeroByteSkipsDataStillSignals asserts the production zero-byte path never calls put_nbi.

…ates Reconcile PR #4 with merged upstream ROCm#10658 (A2A devtime host helpers, BenchTime skip/augment/check path) and ROCm#10672 (GDA/SDMA device template tests + GIN CI wiring). AllGather devtime now buffers via devtimeAugmentLine to match the A2A augment pattern; AllGather-specific unit test targets remain. Co-authored-by: Cursor <cursoragent@cursor.com>

systems-assistant · 2026-08-28T13:17:39Z

TheRock Submodule Bump Activity

Newest first

2026-08-28 — Included in TheRock via ROCm/TheRock#7710.

…ates Reconcile PR #4 with merged upstream ROCm#10658 (A2A devtime host helpers, BenchTime skip/augment/check path) and ROCm#10672 (GDA/SDMA device template tests + GIN CI wiring). AllGather devtime now buffers via devtimeAugmentLine to match the A2A augment pattern; AllGather-specific unit test targets remain. Co-authored-by: Cursor <cursoragent@cursor.com>

…ixture Update backport-manifest.md with PR ROCm#10672/ROCm#10675 MI355 fixture results and set signal_remote_addrs in DetailHelpers_ChannelAndDirty so useSdmaFusedSignal() passes on gfx950 (34/34 fixtures on MI355). Co-authored-by: Cursor <cursoragent@cursor.com>

…ates Reconcile PR #4 with merged upstream ROCm#10658 (A2A devtime host helpers, BenchTime skip/augment/check path) and ROCm#10672 (GDA/SDMA device template tests + GIN CI wiring). AllGather devtime now buffers via devtimeAugmentLine to match the A2A augment pattern; AllGather-specific unit test targets remain. Co-authored-by: Cursor <cursoragent@cursor.com>

dlamd1dai requested review from a team August 24, 2026 22:58

dlamd1dai mentioned this pull request Aug 24, 2026

rccl: GIN AllToAll device-template unit tests (gin-gda + gin-sdma shadow fix) dlamd1dai/rocm-systems#2

Closed

5 tasks

github-actions Bot added the project: rccl label Aug 24, 2026

dlamd1dai self-assigned this Aug 24, 2026

dlamd1dai changed the title ~~rccl: GIN AllToAll device-template unit tests (gin-gda + gin-sdma shadow fix)~~ test(rccl): GIN AllToAll device-template unit tests (gin-gda + gin-sdma shadow fix) Aug 24, 2026

systems-assistant Bot added the organization: ROCm label Aug 24, 2026

This was referenced Aug 24, 2026

f(rccl/gin): fix inverted system-scope fence guard on HIP backends #10675

Merged

rccl/gin: fix inverted system-scope fence guard on HIP backends dlamd1dai/rocm-systems#3

Closed

dlamd1dai mentioned this pull request Aug 25, 2026

feat(rccl/gin-sdma): GIN SDMA AllGather Implementation (-D3 hybrid xGMI-SDMA) dlamd1dai/rocm-systems#4

Open

dlamd1dai and others added 8 commits August 25, 2026 19:18

Fix GIN fixture tests for current device header API.

8e08c1c

Align GDA QueuePair stub with atomic_add, use ncclGinOffsetPtr from GetSignalPtr/GetCounterPtr, and replace removed anvilSignalPtrOrDummy checks. Co-authored-by: Cursor <cursoragent@cursor.com>

Fix Flush call sites for extended ncclGinApi_Flush signature.

a90ccde

Pass hasDescriptor=false and a null descriptor smem pointer before the memory-order and abort-flag arguments expected by current GIN headers. Co-authored-by: Cursor <cursoragent@cursor.com>

Wire GIN Anvil unit tests into gin CI sbatch pipeline.

dedfc4a

Build rccl-UnitTestsFixtures with GIN_ANVIL_UNIT_TESTS=ON during the RCCL stage and run the 33 GDA/SDMA/IPC device-template cases via run-gin-ci.sh. Co-authored-by: Cursor <cursoragent@cursor.com>

rccl/gin-sdma: fix IPC fused-signal fixture on MI355 (gfx950)

c2fd2e5

Set signal_remote_addrs in DetailHelpers_ChannelAndDirty so useSdmaFusedSignal() predicate matches oss7 on gfx950; signals alone was insufficient. MI355 fixtures 34/34 pass. Co-authored-by: Cursor <cursoragent@cursor.com>

dlamd1dai force-pushed the users/dondai/gin-gda-a2a-unittests branch from 2dbe02b to 71b597c Compare August 25, 2026 23:19

amd-jiali approved these changes Aug 26, 2026

View reviewed changes