Repository navigation
fix(hip): zero masked V per row in the wide prompt tile - #2357
Conversation
|
Thanks @laudney. The per-row mask matches what #2317 settled on, and the logits you posted show gfx1201 One more, on speed: the mask changes no bits on gfx1201, so gfx1201 shouldn't pay 0.8 to 0.9% for it. Could gfx1201 |
|
@laudney, here are the gfx1151 runs. They ran on a Ryzen AI Max+ 395 (Radeon 8060S, gfx1151), Ubuntu 26.04, Zig 0.17.0 and ROCm 10.1's hipcc (HIP 7.16.26385), in clean
If the head changes for the review asks, I can run these again. |
The per-row V mask changes no bits on gfx1201 but cost 0.8 to 0.9% of its prompt speed (ashhart#2357 review). The tile's body moves to prefill.hpp with MASKED and EDGE instances: the served tile is masked only where dot2_signed_zero holds and is main's code elsewhere, the same instructions on gfx1201. There a new edge launch follows each tile: a block scans the V rows its rows are masked from, page ids read once so it does not spill, and reruns masked only when one is non-finite. A kernel test checks float64's classes, the tile's own bytes in every unplanted dim, and chunk starts, paged and flat.
The prompt tile's gate-1 plant mixed an infinity past a row's causal edge with one under fp32's least weight, which still fails, so the contract stayed green with no mask or edge pass at all (ashhart#2357 review). Four entries, flat and paged in both cache types, plant +inf, -inf and NaN only past rows' edges over a span a page past the last row and expect a clean gate 1. Each has a planted control, main's tile without the edge pass, which must go red at that gate.
The ashhart#2357 review asked for an Unreleased line that states what users see: RDNA 3.5's prompt rows no longer depend on their chunk start, so its prompt bits change; other chips keep main's tile, bits and, on gfx1201, prompt speed within noise; and a non-finite value past a row's causal edge adds nothing to that row on every chip.
f3c4835 to
a465f1f
Compare
|
Thanks @ashhart. I took the arch-gated path; the description has the full receipt. Ask by ask, on head
gfx1100 and gfx1030 take the edge pass too, as their dot2 behaviour is unmeasured; I have no hardware for either. @AbrahamCete, a |
|
@laudney, these runs are on a465f1f against its base b3933ec, with the same gfx1151 setup as above: Ryzen AI Max+ 395, ROCm 10.1's hipcc and clean
|
|
Thanks @laudney. This covers everything: gfx1201 keeps main's tile, with disassembly and logits equal to main and ABBA medians within noise; the edge pass runs the masked tile only when its scan finds a non-finite value; and the four edge entries are clean on the head, red on main, each with a planted control. Thanks @AbrahamCete for the gfx1151 runs on both heads. We'll fold the gfx1151 prefill cost (under 0.7% at 2k and 8k) into the CHANGELOG line, and re-pin the record count to main's at landing. |
a465f1f to
f87a21d
Compare
The per-row V mask changes no bits on gfx1201 but cost 0.8 to 0.9% of its prompt speed (ashhart#2357 review). The tile's body moves to prefill.hpp with MASKED and EDGE instances: the served tile is masked only where dot2_signed_zero holds and is main's code elsewhere, the same instructions on gfx1201. There a new edge launch follows each tile: a block scans the V rows its rows are masked from, page ids read once so it does not spill, and reruns masked only when one is non-finite. A kernel test checks float64's classes, the tile's own bytes in every unplanted dim, and chunk starts, paged and flat.
The prompt tile's gate-1 plant mixed an infinity past a row's causal edge with one under fp32's least weight, which still fails, so the contract stayed green with no mask or edge pass at all (ashhart#2357 review). Four entries, flat and paged in both cache types, plant +inf, -inf and NaN only past rows' edges over a span a page past the last row and expect a clean gate 1. Each has a planted control, main's tile without the edge pass, which must go red at that gate.
|
As said above, I've rebased this onto today's main and pushed it to your branch (head |
On RDNA 3.5, v_dot2_f32_bf16 and v_dot2_f32_f16 round a sum differently when one lane's product is -0 instead of +0. The 64-row prompt tile multiplies P by V two keys at a time and zeroed V only past the span, so when a row saw an odd number of keys in its last tile, the partner key fed +0 in a run whose chunk ended there but p = 0 times its real V, a -0 for a negative value, in a run whose chunk ended later, and a row's bytes hung on where its chunk began (ashhart#2317). Each row now zeroes every value past its own causal edge, so all runs feed the same dot2 lanes on every chip. This also drops the 0 x inf NaN past the causal edge from the prefill contract's known finding, adds cut 1 to the chunk-start test, moves the gfx1151 pin to the new bits (gfx1152 reproduced the old pin exactly), and refreshes the prompt and image tiles' VGPR records on every chip.
RDNA 3.5's dot2 rounds a sum with a -0 lane apart from the same sum with +0, which made ashhart#2317's prompt rows depend on their chunk start, while gfx1201's does not. A caps field, dot2_signed_zero, records that fact per generation and reaches the kernels as TF_DOT2_SIGNED_ZERO, so the prompt tile can keep main's code on the chips that do not need the per-row mask.
The per-row V mask changes no bits on gfx1201 but cost 0.8 to 0.9% of its prompt speed (ashhart#2357 review). The tile's body moves to prefill.hpp with MASKED and EDGE instances: the served tile is masked only where dot2_signed_zero holds and is main's code elsewhere, the same instructions on gfx1201. There a new edge launch follows each tile: a block scans the V rows its rows are masked from, page ids read once so it does not spill, and reruns masked only when one is non-finite. A kernel test checks float64's classes, the tile's own bytes in every unplanted dim, and chunk starts, paged and flat.
The prompt tile's gate-1 plant mixed an infinity past a row's causal edge with one under fp32's least weight, which still fails, so the contract stayed green with no mask or edge pass at all (ashhart#2357 review). Four entries, flat and paged in both cache types, plant +inf, -inf and NaN only past rows' edges over a span a page past the last row and expect a clean gate 1. Each has a planted control, main's tile without the edge pass, which must go red at that gate.
f87a21d to
4bea033
Compare
The per-row V mask changes no bits on gfx1201 but cost 0.8 to 0.9% of its prompt speed (#2357 review). The tile's body moves to prefill.hpp with MASKED and EDGE instances: the served tile is masked only where dot2_signed_zero holds and is main's code elsewhere, the same instructions on gfx1201. There a new edge launch follows each tile: a block scans the V rows its rows are masked from, page ids read once so it does not spill, and reruns masked only when one is non-finite. A kernel test checks float64's classes, the tile's own bytes in every unplanted dim, and chunk starts, paged and flat.
The prompt tile's gate-1 plant mixed an infinity past a row's causal edge with one under fp32's least weight, which still fails, so the contract stayed green with no mask or edge pass at all (#2357 review). Four entries, flat and paged in both cache types, plant +inf, -inf and NaN only past rows' edges over a span a page past the last row and expect a clean gate 1. Each has a planted control, main's tile without the edge pass, which must go red at that gate.
|
Thank you @laudney. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes. |
What this changes
5e43279595b357985e43279595b35798(byte-identical)25ffc4e18acaac5225ffc4e18acaac52(byte-identical)Before: the 64-row prompt tile (
fa_dot2) multiplies P by V two keys at a time and zeroes V only past the chunk's span. A key inside the span but past a row's causal edge feedsp = 0times its real V, which is −0 for a negative value, while a run whose chunk ends at that key feeds +0. RDNA 3.5'sv_dot2_f32_{bf16,f16}round a −0 lane apart from a +0 lane (the probe in #2317), so a row's bytes depended on where its chunk started. On every chip, an infinite or NaN V at such a key also gave NaN in rows that must not see it (0 × inf).After:
Caps.dot2_signed_zerois true for RDNA 3.5 (gfx1150 to gfx1153) and reaches the kernels asTF_DOT2_SIGNED_ZERO.tf_fa_{paged,wide}_*) ANDs each V pair with the row's own visible-key mask, so every key past a row's edge enters as +0 in every run. This is the first commit's fix; the gfx1151 pin moves to0xe9e35ddbb2a6ad10.s_code_endpadding differs). After it, an edge pass (tf_fa_{paged,wide}_edge_*, same grid and arguments) scans, per block, only the V rows its live rows are masked from: keysq_pos0 + q0 + 1up to the end of the last 64-key tile read, at most 126 keys of one KV head. A block whose keys are all finite returns at once, so a finite prompt keeps main's bytes; a block with a non-finite value reruns the masked tile and writes its rows. On RDNA 3.5 the edge symbols are trap bodies and are never launched. The scan reads its (at most three) page ids once before its loop; a per-threadtable[]read in the loop pushed the paged edge kernel to 256 VGPRs with 44 bytes of scratch.tf_fa_full_bf16, non-causal): main's code and records, unchanged.tf_ctl_fa_{paged,wide}_unmasked_*: main's tile without the edge pass, built from the sameprefill.hpp, which must go red at gate 1.Which sums change: on RDNA 3.5, the P × V sum of a row whose causal edge falls inside a key pair (the partner lane now enters as +0, not −0). Elsewhere no finite sum changes; a row past a non-finite key's edge now gets that key's +0 instead of NaN.
atc overlap --json(2026-11-06T16:41Z, baseb3933ecd3): 647 overlaps, 4 clean, 637 unknown (the run hitotherPullRequestsLimitandpullFilesLimit), 6 conflicts, all inCHANGELOG.mdonly (fabric tp roce: a kernel without transparent hugepages keeps the 4 KiB property without the NOHUGEPAGE advice #2380, Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074, DeepSeek-V4.1 on CUDA: refuse a malformed TF_DSV41_* setting by name before loading, and register the ones that change outputs #2379, server: a refused API key file logs the path and the mode it read #2365, cli: tensorfold models --json, the listed checkpoints as one document #2364, qwen27 metal: a keep that fails part way ends its encoder, so the error returns instead of aborting the process #2376: each adds a line at the top of Unreleased).Checks on this head
a465f1f86onb3933ecd3, five commits: the original fix, the caps fact, the edge pass, the contract entries, the changelog line. The GPU runs used the same commits on6433d511a; the upstream commits since touch no HIP or Qwen3.5/3.6 code.python3 tools/zig/lean_check.py, on the base and on this head: 252 and 252, 0 new (lean_new.py), all inherited.zig build hip-kernel-test -Dhip=true -Dhip-arch=gfx1201: 138 pass, 3 skip, 0 fail. With the edge launch turned off locally (not committed), only the new edge test fails ("head 0 row 0 dim 7: -nan where float64 gives finite").zig build hip-kernel-test -Dhip=true -Dhip-arch=gfx1152: 135 pass, 6 skip, 0 fail.HSA_OVERRIDE_GFX_VERSION=11.5.1 zig build hip-kernel-test -Dhip=true -Dhip-arch=gfx1151on the same gfx1152: 137 pass, 4 skip, 0 fail (the pin, the chunk-start test and the new edge test included).attention_edge.zig: for both cache types, paged and flat, atq_pos087, 150 rows (a partial last row block) and a span of 256 (past the last row), every output has float64's class; every dim with no planted value is byte-equal to the tile alone in every row; and planted chunks cut at 1 and 87 give the uncut bytes, paged and flat.zig build hip-kernel-contract-limits -Dhip=true: every prompt, image and edge record current on gfx1030, gfx1100, gfx1151 and gfx1201; it exits 1 for the 5 stale records main has too (tf_moe_combine_gateon gfx1030, gfx1100 and gfx1151; the WMMA 4-bit 64-row item tile on gfx1100 and gfx1151).zig build test -Dhip=true: 612 of 612 steps pass.zig fmt --check build.zig zigandgit diff --check: pass.a465f1f86(7 of 7: zig, zig-windows, zig-macos-dev-tools, fmt, release-paths, benchmark-clients on Ubuntu and Windows).tools/bench_concurrent.pyandtools/bench_openai.py: see the Receipt.Receipt
5daafaf18against this branch's commits on6433d511a; Zig 0.17.0; ROCm hipcc HIP 7.16.26385 (AMD clang 24); Radeon AI PRO R9700 (gfx1201) and Radeon 860M (gfx1152), Linux;mlx-community/Qwen3.6-35B-A3B-4bitat38740b847e4cb78f352aba30aa41c76e08e6eb46.pagedCausal(served) and the 64-row branch oftf_causal, on every HIP chip with the wide tile. Ran gfx1201 and gfx1152, and gfx1151 code objects on gfx1152. Could not run gfx1151, gfx1100 or gfx1030 hardware.tools/bench_concurrent.py --alone --serialortoken_shaagainst"draft": false: not run. On gfx1201 the served tile is main's code (same instructions) and the edge pass writes nothing for a finite prompt;tf-qwen35-test logitsgives byte-identical prompt logits to main at 2100 and 4097 tokens. On RDNA 3.5 the bits change by design (the pin), and the chunk-start test shows they no longer depend on the chunk start.hip-kernel-testand your prompt timings on your gfx1151? It is the one chip I reach only through gfx1151 code objects on a gfx1152.tools/bench_openai.py: n/a, the 64-row tile and the edge pass run only in prompt attention.tf-qwen35-test prefill(median of 3 cold prefills after a warm-up, fresh caches), 8 pairs on gfx1201, 4 main-first and 4 branch-first. The medians are within 0.15% at 2k, 8k and 32k, and the branch is faster in some pairs and slower in others. The first run of each session is fast whichever binary it is, which is why the medians, and not the means, are given; the raw runs are below.Raw prefill runs (tok/s, run order)
Runs 1-4 were main then branch; runs 5-8 branch then main.
Kernel contract, only when a GPU kernel is new or changed
tf_fa_paged_{f16,bf16},tf_fa_wide_{f16,bf16}change in place on RDNA 3.5 and are main's elsewhere; the newtf_fa_{paged,wide}_edge_{f16,bf16}run after them wheredot2_signed_zerois false. The new entries test the served dispatch, tile and edge pass together.tf_ctl_fa_{paged,wide}_unmasked_{f16,bf16}, red at gate 1 on both chips below.SUMMARYline and exit status:zig build hip-kernel-contract -Dhip=true -Dhip-arch=gfx1201on this branch:SUMMARY gfx1201 111 kernels: 111 pass gates 1-4, 0 fail, 2 known findings; 19 controls: 19 red as planted, 0 missed; 0 limit problems; gates 1464632 ms, limits 3000 ms, exit 0.5daafaf18with the four new entries added (no controls, as main has noprefill.hpp):SUMMARY gfx1201 111 kernels: 107 pass gates 1-4, 4 fail, 2 known findings; 15 controls: 15 red as planted, 0 missed; 0 limit problems; gates 1455760 ms, limits 3517 ms, exit 1: the 4 failures are the new entries, each "126 of 49152 outputs differ in class; first at head 0 element 7: nan where float64 gives finite".HSA_OVERRIDE_GFX_VERSION=11.5.1 zig build hip-kernel-contract -Dhip=true -Dhip-arch=gfx1151on a gfx1152:SUMMARY gfx1151 109 kernels: 109 pass gates 1-4, 0 fail, 2 known findings; 18 controls: 18 red as planted, 0 missed; 2 limit problems; gates 298650 ms, limits 1667 ms, exit 1. The 2 limit problems are the inherited stale records oftf_moe_combine_gateand the WMMA 4-bit item tile.Checklist
The title and description match the head commit, including the receipt and the stack.
New tests fail before the change, pass after it, and skip cleanly without their dependency.
Comments and docstrings are one line. No measurements or history in the source.
No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
Drafted output equals
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Not run end to end; the chunk-start tests cover a resumed prompt against a fresh one at the kernel, finite and planted.)No precision traded for speed. If the bits change, the description says which sums change and why.
Decode and prompt processing are no slower than main at one stream and the widest admitted; with
--drafter, no slower than without it, greedy and sampled. (Measured: prompt processing at one stream on gfx1201, medians within 0.15%; decode is not reached. Not measured: the widest admitted width and drafts.)