Skip to content

perf(gfx1201): split packed-Q8 verifier KV scans — +22.6% DFlash E2E at 21.5K context - #760

Open
HUSRCF wants to merge 1 commit into
warpfront:fa2-prefill-gfx1201from
HUSRCF:perf/gfx1201-fa2-splitkv-verifier
Open

HUSRCF wants to merge 1 commit into
warpfront:fa2-prefill-gfx1201from
HUSRCF:perf/gfx1201-fa2-splitkv-verifier

Conversation

@HUSRCF

@HUSRCF HUSRCF commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Stacked follow-up to #758. The base is intentionally fa2-prefill-gfx1201: #758 supplies Kaden's packed-Q8 GQA/LDS front end, while this PR only adds the split-KV verifier back end and product admission.

For small DFlash verifier batches at long context, partition the live causal KV prefix across 2/4/8 CTAs. Each split emits normalized Oacc plus LSE, then a small merge kernel combines the records with a stable max-LSE reduction. This restores parallelism without discarding #758's shared packed-KV loads inside each tile.

The route is opt-in through HIPFIRE_GFX12_FA2_SPLIT_VERIFY=1 (HIPFIRE_GFX12_FA2_SPLIT_COUNT={2,4,8}, default 8). It remains fail-closed to the existing multi-row admission plus exact gfx1201, H24/KV4/D256, Q8 KV, eager sequential verification. Tree, independent-batch, graph-capture, and retained/PM4 recording routes remain on the established path. Plain AR (n=1) cannot enter this route.

Performance

Radeon AI PRO R9700, gfx1201, Q8 VMM KV, Qwen3.8-27B MQ4XT, B16, context 20,676:

attention path time relative to split S8
incumbent WMMA 6.8854 ms 16.79x slower
#758 direct packed FA2 2.5556 ms 6.23x slower
packed FA2 + split-KV S8 0.4100 ms 1.00x

Fresh-process product DFlash A/B at 21,550 actual prompt tokens, 200 output tokens, HIPFIRE_VERIFY_GRAPH=0:

route samples (tok/s) median
existing R4/R8 verifier 39.3, 39.3, 39.3 39.3
split-KV S8 48.3, 48.2, 48.2 48.2 (+22.6%)

All six samples retained tau 1.80 and 71 cycles. A plain-AR smoke was unchanged within noise (32.6 tok/s with the env arm off, 32.5 with it on), consistent with the n=1 exclusion.

Fixture identities:

  • prompt MD5: b4d0b63cddcac872648ddf3cdd92cac2
  • target SHA-256: 9f91556f7e0431a077d03756a7102d0154108757289e6e5fe9a2d204c0c9eeb7
  • DFlash draft SHA-256: d0a74a232a0e2166d889f823e91e0fbf778d21dd9668d7de055cdecb065401bc
  • measured daemon SHA-256: 36576b9fc40345bbce6fa96c90906f56db81f2955c042c254432bc64d01437c1

Correctness and long-run E2E

The CPU-f64 oracle covered S8 tail/ragged/sentinel shapes. Worst relative L2 was 2.486e-4; minimum cosine similarity was 0.999999821. The partial kernel compiles at 243 VGPR / 27 SGPR and the merge at 18 VGPR / 10 SGPR, with zero spill and zero private scratch.

LongBench hard30, thinking enabled (temperature=1, top_p=.95, xhigh), Q8 VMM KV, max_seq=131072, output allowance 65,536:

  • 30/30 completed, 0 errors, all natural stop
  • prompt range 20,462–30,258 tokens
  • 98,989 generated tokens total; median 1,568, maximum 21,876
  • 20/30 scored correct (66.7%; the previous same-fixture checkpoint recorded 17/30)
  • median decode 72.9 tok/s
  • no empty output, open-think failure, replacement character, NUL, or repeated-line attractor across the 30 transcripts
  • dataset SHA-256: 839a19be0b3b1c801a0ca58d388996faff34704e34dd196d54346adf17a1dce9

Five no-think long-decode fixtures (KV Cache article, SGLang article, story, Snake, Tetris):

  • 5/5 completed, 0 errors, 15,954 generated tokens total
  • three natural stop; the two 1,536-token technical articles reached their fixture-specific length cap
  • longest output: Tetris, 8,178 tokens / 24,036 characters, natural stop
  • generated Snake (290 lines) and Tetris (656 lines) Python code both pass AST parsing
  • no repeated prose attractor or malformed output observed

The E2E binary was the equivalent current-beta port used to develop this stacked delta. The kernel source in this PR and that measured binary is byte-identical (SHA-256 7a006296173579377e69ec2a8a3947d3af40017922fb35d0d69eee91add2e394).

Memory

At S8/B16/H24/D256 the split records require about 3 MiB, but they reuse the existing flash_partials allocation. There is no new persistent VRAM allocation.

Static validation on this exact stacked commit

  • changed-file rustfmt: pass
  • cargo check -p rdna-compute -p hipfire-arch-qwen35: pass
  • cargo test -p rdna-compute --lib: 246/246 pass
  • cargo test -p hipfire-arch-qwen35 --lib: 196/196 pass, 17 hardware-fixture tests ignored by design

This improves the verifier cost only. It does not claim to repair low draft/target acceptance on prompts where tau itself collapses.

Partition each live causal prefix across 2/4/8 CTAs, emit normalized output plus LSE per split, and merge with a stable max-LSE reduction. The exact gfx1201 H24/KV4/D256 eager multi-row route is opt-in and reuses the existing flash-partials allocation, so it adds no persistent VRAM.\n\nR9700 (gfx1201) evidence: B16/ctx20676 S8 attention 0.410 ms versus 2.556 ms for the direct packed FA2 path; fresh-process DFlash median 48.2 versus 39.3 tok/s (+22.6%). CPU-f64 oracle worst rel-L2 2.486e-4, minimum cosine 0.999999821. LongBench hard30 completed 30/30 with 0 errors; five long-decode fixtures completed 5/5.
@HUSRCF
HUSRCF marked this pull request as ready for review September 14, 2026 13:21
@HUSRCF
HUSRCF marked this pull request as draft September 14, 2026 13:28
@HUSRCF
HUSRCF marked this pull request as ready for review September 14, 2026 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant