perf(hip): narrowest stream columns at RDNA3.5's one- and two-row launches by AbrahamCete · Pull Request #1855 · ashhart/TensorFold

AbrahamCete · 2026-11-01T16:42:26Z

What this changes

The stream tile gives a lane the widest column count (CB 8, 4 or 2) that still reaches streams_wanted = 3000 waves. On gfx1151 that leaves one-row decode launches short of loads in flight. Qwen3.6-35B-A3B's GDN in_proj (12,352 columns) takes 4 columns a lane, 772 blocks, and the 248,320-column head takes 8. With 2 columns a lane, on rocprofv3 kernel traces of 64 decode steps:

in_proj: 98 → 74 µs
head: 1,558 → 1,212 µs

Two-row launches (MTP verify and drafts) gain the same way. Four-row launches measured slower narrow: 35B decode at four streams dropped 5-6%. So they keep the wave target.

The changes, one commit each:

perf(hip): narrowest stream columns at RDNA3.5's one- and two-row launches
- The column-count rule moves out of streamWith into launches/stream_pick.zig, as pick.
- On .rdna3_5, one- and two-row launches take the narrowest columns their registers allow.
- Every other generation, and RDNA3.5 at three rows or more, gets exactly the previous choice.
- The fused relu² pair launch (affine_pair.zig) and its contract case had their own copies of the rule. Both now call pick.
refactor(hip): the stream tile's row tile and block width from stream_pick
- rowTile(m) is the one row mapping.
- pick returns the block width with the column index, so callers no longer recompute it.
- A pair flag replaces the old divisor.
- When no width fits a lane's registers, pick falls back to the narrowest tile and its block width, as the inline rule did.
refactor(hip): assert at comptime that every stream tile width divides 8 (from review)
- A comptime block in stream_pick.zig asserts that each width in affine.stream_cbs divides 8, so a width that would change a column's start round fails the build.

A column's sum order does not depend on the column count. The start-round rotation uses col0 >> 3, and every column count divides 8 (tiles/stream.hpp, asserted in stream_pick.zig). The lanes a column (fixed by K), the chunk order and the shuffle tree are also unchanged, so the outputs are byte-identical.

Overlap with open pull requests:

fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830 (mine): this branch is rebased onto it and lands after it. Until fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830 lands, the diff here also shows its three commits.
perf(hip): wide stream rounds reuse cached weights, and a chunk's code pairs are made once #1694 (mine) changes the launch line in streamWith, next to this PR's table[b][ri][tile.at] line. Whichever lands second rebases; the two changes are independent.
hip: a 64-row routed prompt tile for many-expert plans on gfx1151 #1658 adds an import beside this PR's stream_pick import in affine.zig.
hip: load a shared expert wider than the routed experts beside the stack #1612, hip: decode router over the bf16 rows the checkpoint holds #1614 and hip flashnext: WMMA sparse attention for prompt chunks (sparse=wmma) #1622 touch launches.zig, where this PR adds one export line.

Receipt

Environment: Ryzen AI Max+ 395 (gfx1151), Ubuntu 26.04, Zig 0.17.0, hipcc with HIP 7.1.52801 and Ubuntu clang 21.1.8 (ROCm 7.13 from TheRock for the contract limits and rocprofv3); checkpoints Qwen3.6-35B-A3B-MLX-4bit-MTP and Qwen3.8-27B-MLX-4bit.
Base: the branch is three commits on fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830 (493704a, itself on main ba7b2c4). The served A/B below ran on main 5718638 with the first two commits; none of the 34 commits from there to 9e8c368 touches zig/src/hip. On fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830, against fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830:
- 64-step decode logits, and the lanes replies with drafts on, are byte-identical to fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830's on both checkpoints. check --only invariants matches as well.
- --parallel 2, tools/bench_concurrent.py --levels 1,2 --tokens 256 --reps 3 --temperatures 0 --alone --serial, two servers a side. 35B chat ×1: 78.2 and 81.6 on fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830, 92.2 and 92.7 here. Code ×1: 113.3 and 113.1, then 112.2 and 112.3. ×2 is within 2%. 27B: +5% at ×1 and +4% at ×2 on both prompts. Every server has the same token hashes, with 0 unequal and 0 failed.
- The same kernel contract SUMMARY as fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830. hip-host-test, test-qwen35-hip-host and check-native-hip-host exit 0.

Served A/B: one fresh server a run (--context 16384 --prompt-cache-gib 0), alternated B1 C1 B2 C2, with B = main and C = this PR.

Decode speed, tools/bench_openai.py --tokens 256 --reps 3 --temperatures 0, median tok/s:

checkpoint, prompt	B1	B2	C1	C2
35B-A3B MTP, code	113.0	112.8	112.3	112.5
35B-A3B MTP, chat	79.8	76.5	92.4	92.4 (+18%)
27B, code	13.3	13.3	14.0	14.0 (+5%)
27B, chat	13.6	13.6	14.3	14.3 (+5%)

The 35B gain depends on draft acceptance. Code accepts about 3 tokens a round, so its verify rounds run four-row tiles, which are unchanged. Chat accepts about 1.4, so its rounds mostly run one and two rows.

Concurrency, tools/bench_concurrent.py --tokens 256 --reps 3 --temperatures 0 --alone --serial:
- 35B (--parallel 4, levels 1 and 4): code ×1 112.9/112.7 vs 113.9/112.5 and chat ×1 77.7/82.4 vs 92.2/92.6. At ×4, code is 226.4/227.6 vs 228.5/211.3 and chat 191.7/191.4 vs 190.8/193.4. The C2 code ×4 cell is the per-process slow mode of lanes: on HIP, Qwen3.6-35B-A3B MTP at four streams falls into a slower, deeper-drafting mode that holds per server process #1695, fixed by fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830, which also hits main.
- 27B (--parallel 16, levels 1, 4, 8, 16): ×1 code 13.3 vs 14.0 and chat 13.6 vs 14.3. ×4, ×8 and ×16 are within 0.2 tok/s on both prompts.
tf-qwen35-test check --only speed, 35B one stream with drafts off: 71.2 and 70.4 on main, 76.3 and 76.3 with this PR.

Exactness:
- Every server of every run has the same token hashes. 35B: code 9f2923ac14d0, chat a8c28b8297b4, cold 8,192-token prompt 393096f974da. 27B: code 2b28aa2108de, chat a06fab348fbe, cold prompt 1ebfd15d62c7.
- --alone and --serial: 0 unequal and 0 failed, out of 136 checks on 35B and 728 on 27B.
- tf-qwen35-test logits --decode over 64 steps is byte-identical to main on both checkpoints.
- tf-qwen35-test check --only invariants: 27B 42/0/5 and 35B 46/0/1, the same as main.
Prompt speed: the cold 8,192-token ttft_s from tools/bench_cache.py --require-cold:
- 35B: 5.373 and 5.385 s on main, 5.351 and 5.449 s with this PR.
- 27B: 22.364 and 22.399 s on main, 22.197 and 22.458 s with this PR.
- A prompt only reaches this change through a one- or two-row last chunk.
Kernel contract: the same SUMMARY on main and on this PR.
- gfx1151 52 kernels: 52 pass gates 1-4, 0 fail, 2 known findings; 5 controls: 5 red as planted, 0 missed; 130 limit problems.
- The 2 known findings are the prompt attention tile's, from perf(hip): prompt attention tile keeps its probabilities in registers and fits 6 waves, same bits #1788.
- hip-kernel-contract-limits with ROCm 7.13: 220 problems on both.
- The relu² pair case passes its byte check against the two separate launches.
Tests run:
- New host tests in stream_pick.zig, through the public pick and rowTile:
  - "the widest columns that fill the card, and RDNA3.5's narrowest at one and two rows"
  - "three rows and more keep the widest that fills the card on every generation"
  - "a routed pair counts its gate and up as one column, over every item"
  - "a width no column count fits falls back to the narrowest tile and its block width"
  - "the row tile takes 1, 2, then 4 rows a block"
- With the RDNA3.5 rule disabled, the first and third fail ("expected 0, found 1" and "expected 0, found 2"). The previous choice on RDNA3 at 1, 2 and 4 rows is pinned in the first two. With the fallback's block width left at 0, the fourth fails ("expected 8, found 0").
- With stream_cbs set to 2, 4 and 16, the build fails at the comptime assert.
- hip-kernel-test: 114 passed, 1 skipped, 0 failed; main has 109/1/0.
- hip-host-test, test-qwen35-hip-host and check-native-hip-host exit 0.
- zig build test: 465/465 steps, with 6,168 of 6,213 tests passed and 45 skipped, the same as fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830. Both ran on trees exported with git archive.
- tools/zig/lean_check.py: 424 problems, the same as main. zig fmt --check is clean.
Not run, and why:
- gfx1150, gfx1152 and gfx1153 take the same rule as gfx1151, but only gfx1151 has been measured; I have none of them. They share gfx1151's instruction set, but their bandwidth and CU counts differ.
- gfx1030, gfx1100 and gfx1201: I have no card. Their choice is unchanged, and pinned by the tests above.
- Nemotron and Flash-Next: I have no HIP checkpoint locally. Their launches take the same rule on RDNA3.5; the contract's relu² pair case covers the fused Nemotron launch's bytes.
- Metal and CUDA: untouched.

Checklist

Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
No precision traded for speed. If the bits change, the description says which sums change and why.
A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt.
Prompt processing is no slower than the last release.
The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
New tests fail before the change, pass after it, and skip cleanly without their dependency.
Comments and docstrings are one line. No measurements or history in the source.
No personal data, machine names, internal hosts or local paths. No AI attribution lines.

ashhart · 2026-11-02T03:31:50Z

Thanks @AbrahamCete. We read this through, and it looks right. stream_pick.pick is the old rule exactly: the same skip and wave tests, the fallback to the narrowest tile and its block width, and the pair halving. The narrowest tile exists for every bit width and row tile. It stays exact because each column's start round, (col0 >> 3) % rounds, is the same for 2, 4 and 8 columns, since each divides 8, and the fold and shuffle tree doesn't depend on the width. A comptime assert that the width divides 8 would keep it that way.

We'd like to land it right after #1830, which it should follow. We don't have an RDNA3.5 machine, though, so could you post these from your gfx1151 on this branch rebased onto #1830?

the HIP kernel contract SUMMARY line;
64-step logits, byte-identical against fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830, on both checkpoints with drafts on;
--alone --serial decode at levels 1 and 2, alternating chat and code, against fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830.

The rule also covers gfx1150 to 1153, but only gfx1151 has been measured. That's fine to say in the body. The overlaps with #1694, #1658 and #1612/1614/1622 are one-line textual ones, so whichever lands second rebases.

AbrahamCete · 2026-11-02T11:53:08Z

Thanks @ashhart. The branch is rebased onto #1830 (493704a) with no conflicts, and it has a third commit, 92b35dd refactor(hip): assert at comptime that every stream tile width divides 8. A comptime block in stream_pick.zig asserts @mod(8, cb) == 0 for every width in affine.stream_cbs. With the widths set to 2, 4 and 16, the build fails there.

From gfx1151, #1830 at 493704a against this branch on top of it at 92b35dd, both built from git archive trees:

Kernel contract: the same SUMMARY on both, gfx1151 52 kernels: 52 pass gates 1-4, 0 fail, 2 known findings; 5 controls: 5 red as planted, 0 missed; 130 limit problems. hip-kernel-contract-limits with ROCm 7.13 reports 220 problems on both. hip-kernel-test: 109/1/0 on fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830, 114/1/0 here.
64-step logits: tf-qwen35-test logits --decode over 64 steps is byte-identical to fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830 on both checkpoints. That command runs one verify row a step, so it has no drafts. With drafts on (MTP), the tf-qwen35-test lanes replies over zig/tests/qwen35/prompts.json, 256 tokens, are byte-identical to fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830 on both checkpoints. check --only invariants gives 27B 42/0/5 and 35B 46/0/1 on both.
Served decode: --parallel 2 --context 4096 --prompt-cache-gib 0, one fresh server a run, alternated B1 C1 B2 C2, with B = fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830 and C = this PR. Each server ran bench_concurrent.py --levels 1,2 --tokens 256 --reps 3 --temperatures 0 --alone --serial twice: once with its code and chat cells, and once with --prompts alternating chat and code (at ×1 that cell sends chat). Aggregate tok/s:

checkpoint, cell	B1	B2	C1	C2
35B-A3B MTP, code ×1	113.3	113.1	112.2	112.3
35B-A3B MTP, code ×2	172.5	174.7	174.4	175.4
35B-A3B MTP, chat ×1	78.2	81.6	92.2	92.7
35B-A3B MTP, chat ×2	146.2	146.6	147.4	146.8
35B-A3B MTP, chat and code ×1	76.0	82.8	92.8	90.9
35B-A3B MTP, chat and code ×2	116.2	115.6	118.5	118.7
27B, code ×1	13.3	13.2	13.9	13.9
27B, code ×2	25.7	25.7	26.8	26.6
27B, chat ×1	13.6	13.6	14.3	14.3
27B, chat ×2	25.9	25.9	26.8	26.9
27B, chat and code ×1	13.6	13.6	14.3	14.2
27B, chat and code ×2	25.4	25.5	26.6	26.5

--alone and --serial: 0 unequal and 0 failed on every server, out of 54 alone and 16 serial checks an arm on each checkpoint. Every cell has the same token hashes on all four servers: 35B code 9f2923ac14d0 and chat a8c28b8297b4, 27B code 2b28aa2108de and chat a06fab348fbe.
hip-host-test, test-qwen35-hip-host and check-native-hip-host exit 0. zig build test gives 465/465 steps, with 6,168 of 6,213 tests passed and 45 skipped, on both. lean_check shows 424 problems, the same as main.

The body now says that gfx1150, 1152 and 1153 take the same rule and only gfx1151 has been measured.

…nches The stream tile gives a lane the widest column count that still reaches 3000 waves. On gfx1151 that leaves one-row launches short of loads in flight: Qwen3.6-35B-A3B's GDN in_proj takes 4 columns a lane (772 blocks) and the 248320-column head 8. With 2 columns a lane, the in_proj drops from 98 to 74 us and the head from 1558 to 1212 us. Two-row launches (MTP verify and drafts) gain the same way. Four rows keep the wave target, which measured slower narrow. A column's sum order does not depend on the column count, so outputs are byte-identical. The rule moves to launches/stream_pick.zig with unit tests; the fused relu^2 pair launch and its contract case call it instead of keeping their own copies. Other generations keep the current choice.

…_pick The callers of stream_pick derived the row tile again and recomputed the block width from the returned column index. rowTile is now the one row mapping, pick returns the block width with the index, and its pair flag names what the old divisor meant: a lane's columns are the gate and up halves of one output. When no width fits a lane's registers, pick falls back to the narrowest tile and its block width, as before. Launch geometry is unchanged.

A column's start round is (col0 >> 3) % rounds, the same for each width only while the width divides 8, which keeps stream_pick's choice of width out of the bits. A width that does not divide 8 now fails the build.

ashhart · 2026-11-02T18:43:04Z

Thank you @AbrahamCete. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

AbrahamCete force-pushed the perf/hip-decode-35b branch from 25920f3 to dd2947c Compare November 1, 2026 17:06

ashhart mentioned this pull request Nov 2, 2026

fix(qwen35/hip): time shared rounds at the served streams and their row buckets #1830

Merged

9 tasks done

AbrahamCete force-pushed the perf/hip-decode-35b branch from dd2947c to 92b35dd Compare November 2, 2026 11:53

AbrahamCete mentioned this pull request Nov 2, 2026

perf(hip): the planned decode score takes four keys a warp on long spans, two on short ones #1932

Open

9 tasks done

AbrahamCete and others added 4 commits November 2, 2026 18:42

changelog: ashhart#1855 under Unreleased

6e3619e

ashhart force-pushed the perf/hip-decode-35b branch from 92b35dd to 6e3619e Compare November 2, 2026 18:42

ashhart merged commit 089a18d into ashhart:main Nov 2, 2026
8 of 11 checks passed

Navigation Menu

Repository navigation

perf(hip): narrowest stream columns at RDNA3.5's one- and two-row launches - #1855

perf(hip): narrowest stream columns at RDNA3.5's one- and two-row launches#1855
ashhart merged 4 commits into
ashhart:mainfrom
AbrahamCete:perf/hip-decode-35b

AbrahamCete commented Nov 1, 2026 •

edited

Loading

Uh oh!

ashhart commented Nov 2, 2026

Uh oh!

AbrahamCete commented Nov 2, 2026

Uh oh!

Uh oh!

ashhart commented Nov 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

AbrahamCete commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this changes

Receipt

Checklist

Uh oh!

ashhart commented Nov 2, 2026

Uh oh!

AbrahamCete commented Nov 2, 2026

Uh oh!

Uh oh!

ashhart commented Nov 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AbrahamCete commented Nov 1, 2026 •

edited

Loading