Skip to content

[WIP] fix(rocjitsu): model finite CDNA wait counters in Waitcheck - #10926

Draft
newling wants to merge 1 commit into
users/newling/rocm28430-waitcheck-consan-investigationfrom
users/newling/rocjitsu-waitcheck-counter-capacity
Draft

[WIP] fix(rocjitsu): model finite CDNA wait counters in Waitcheck#10926
newling wants to merge 1 commit into
users/newling/rocm28430-waitcheck-consan-investigationfrom
users/newling/rocjitsu-waitcheck-counter-capacity

Conversation

@newling

@newling newling commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Model the finite CDNA3/CDNA4 LGKMCNT and VMCNT capacities in Waitcheck.
  • Retire a dependency once enough younger operations in the same guaranteed
    completion-order class have issued that hardware must have completed it.
  • Apply capacity backpressure before checking the overflow-triggering
    instruction's operands.
  • Keep LDS, VMEM, scalar-memory, generic-FLAT, and GDS ordering distinctions
    conservative across straight-line analysis and CFG merges.
  • Cover both sides of the 15-operation LDS and 63-operation VMEM boundaries,
    including stores and direct-to-LDS visibility.

Problem

On CDNA, LGKMCNT is four bits and VMCNT is six bits. Their all-ones operands
are no-wait values:

counter maximum outstanding no-wait operand largest effective explicit wait
LGKMCNT 15 15 14
VMCNT 63 63 62

Hardware prevents these counters from overflowing by stalling an instruction
whose issue would exceed the limit. Waitcheck already treated the all-ones
operands as no-ops, but retained an unbounded dependency history. It could
therefore report a producer as pending after hardware had necessarily
completed it.

One production-generated gfx950 example was reported as:

==> ds_read_b32 v51, v16
    ... 123 younger ds_read operations ...
    s_waitcnt vmcnt(63) expcnt(7) lgkmcnt(15)
==> v_perm_b32 v18, v51, v50, s77

lgkmcnt(15) does not synchronize the producer. Nevertheless, an older LDS
operation cannot remain outstanding after 15 younger ordered LDS operations
have issued, much less 123. A second production case had 17 younger LDS reads
before its first MFMA use and is safe for the same reason.

The 15-entry LGKMCNT limit belongs to the complete counter, which also accounts
for GDS, scalar-memory, and message events. Sharing a counter is not the same
as sharing a completion order. Fifteen later LDS operations are sufficient to
prove completion of an older LDS operation; fifteen arbitrary LGKM-counted
operations are not sufficient to identify which event completed.

This deliberately does not change hipBLASLt's MaxLgkmcnt from 15 to 14.
Fifteen is the correct capacity/no-wait encoding. Clamping it globally to 14
inserts explicit stalls even where issue backpressure has already made the
dependency safe.

Implementation

Each pending Waitcheck event now carries a separate ordered-counter domain:

  • Lds for native local-DS operations;
  • Vmem for CDNA non-FLAT VMEM operations, including stores and
    direct-to-LDS; and
  • None for scalar memory, generic FLAT, GDS, and cases without a usable
    ordering guarantee.

Waitcheck tracks the minimum number of later operations in the same ordered
class along every feasible path. Once that lower bound reaches 15 for LDS or
63 for VMEM, the event is retired. Backpressure is applied before dependency
checks, so the instruction that would overflow the counter may safely consume
the oldest completed result. CFG merges retain the minimum proven ordered age.

Mixed-class counter pressure remains deliberately conservative. It can cause
hardware progress, but Waitcheck does not guess whether an LDS, scalar-memory,
GDS, or generic-FLAT event was the one that completed.

Architecture and compiler cross-checks

  • The CDNA3/CDNA4 machine-readable ISA defines four LGKM bits and six VM bits
    in S_WAITCNT; all ones is documented as “do not wait.”
  • LLVM's pre-GFX12 event table assigns SMEM, LDS, GDS, and message events to
    LGKMCNT, and folds stores into VMCNT on targets without VSCNT.
  • LLVM treats scalar memory and mixed event kinds as potentially out of order,
    while pre-VSCNT VMEM loads/stores form one ordered stream.
  • LLVM's waitcnt-overflow.mir regression test and
    SIInsertWaitcnts.cpp::determineWaitForScore conservatively cap an
    overflow-age dependency at limit - 1, producing lgkmcnt(14) and
    vmcnt(62) for gfx9.

Relevant upstream references:

Evidence

Native MI355X probes distinguish no-wait operands from capacity-forced
progress:

sequence checked stale
one LDS read, lgkmcnt(15) 26,214,400 26,213,424
producer + 15 younger LDS reads, no wait 26,214,400 0
producer + 10 younger VMEM loads, vmcnt(63) 1,310,720 1,310,720
producer + 63 younger VMEM loads, no wait 1,310,720 0
producer + 20 younger VMEM stores, vmcnt(63) 1,310,720 1,310,720
producer + 63 younger VMEM stores, no wait 1,310,720 0

At the LDS boundary, the old analyzer reported:

.text+0xd4: missing s_waitcnt lgkmcnt(14) before use of v3
  producer .text+0x54: ds_read_b32 v3, v2
  consumer .text+0xd4: global_store_dword v[0:1], v3

The corrected analyzer reports zero diagnostics after 15 younger LDS reads.
The 14-younger-read control still reports the dependency at .text+0xd0.

Full scans of the frozen rocm-libraries fa8f393b corpus produced:

corpus kernels before after removed kernels reduced hazard to clean
gfx950 generic 69,492 2,151,862 695,231 1,456,631 11,525 32
gfx950 ID75a3 8,830 189,454 68,162 121,292 1,348 2
gfx942 144,013 11,603,472 3,698,274 7,905,198 24,689 12,089
total 222,335 13,944,788 4,461,667 9,483,121 37,562 12,123

No kernel gained diagnostics and no new producer/consumer instruction pair
appeared. The gfx950 runs contain only explained refinements at existing sites:
some conservative vmcnt(0) recommendations became precise partial counts,
and some multi-register loads acquired a corrected overlapping-register label.
The gfx942 run had no new diagnostic identity and no changed requirement.

Testing

  • 423/423 WaitcheckTest.* tests passed.
  • The complete Waitcheck selection passed 100 consecutive repetitions
    (42,300 test executions).
  • Formatting hooks and git diff --check passed for every changed file.

Remaining work before marking ready

  • Decide how to land or transplant this commit after its unmerged
    sanitizer/Waitcheck prerequisite branch.
  • Let normal CI exercise the repository's other supported configurations.
  • Continue triage of the remaining within-capacity diagnostics separately;
    this patch removes impossible reports but does not classify every survivor.
  • Consider accepting top-level compressed CCOB inputs directly. Current
    TensileCreateLibrary emits them by default, so this audit had to unbundle
    them before scanning.

Issue Tracking

Related: ROCM-28430

@github-actions github-actions Bot added documentation Improvements or additions to documentation project: rocjitsu labels Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation organization: ROCm project: rocjitsu

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant