Dev/huizzhan/flydsl gdn k6 - #1
Closed
huizzhan wants to merge 2 commits into
Closed
Conversation
The separate K5+K6 pipeline still runs the Triton output stage, which grows from 12% of the pipeline at batch*head=8 to 49% at 512 -- the main reason the separate path loses to the fused kernel at high batch*head. This adds a FlyDSL K6 that is signature-compatible with chunk_fwd_o_opt_vk so the two can be swapped at the call site. Two structural differences from the K6 half of the fused kernel: - Chunk-parallel grid. The fused kernel walks chunks serially because K5's state recurrence forces it to; K6 carries no cross-chunk state, so the chunk index becomes a grid axis and low batch*head stops being occupancy-starved. - Triton's pair gate exp(g_i - g_j) rather than the fused kernel's telescoping exp(g_i - g_last). The fused form is only cheap because K5 hands it a pre-gated v_new for free; here v_new arrives ungated from HBM, so that form would cost a column gate over v AND put a >= 1 factor on the critical path that overflows f32 once a chunk's decay exceeds ~88. The pair form is evaluated only under the causal mask, where g_i <= g_j bounds it by 1. Correctness only at this point: given identical h / v_new, the two kernels agree to an RMSE ratio of 0..1.8e-5 across varlen and dense, tail chunks, GQA, g and gk gates, and both exp/exp2 paths. Performance is 0.28-0.42x Triton -- the kernel is occupancy-bound at 88 KiB LDS (1 CTA/CU), which the tiling and swizzle passes still have to address. Co-authored-by: Cursor <cursoragent@cursor.com>
Three changes, each aimed at a different cap on the kernel: Take q out of LDS. It is the A operand of both GEMM3 and GEMM4a, and the M=BT axis is already split across waves, so a wave only ever reads its own 16 rows -- there was nothing to share. Its 4 lanes-worth of elements run along K, which is contiguous in HBM, so an A-fragment load is one 64b buffer read per K-tile and both GEMMs reuse the fragment. LDS drops 88 -> 72 KiB, which is what gets a second CTA onto a CU. 4720 -> 3340us at BV=128. Swizzle the LDS tiles. This was the real cost. The B-operand read of a [rows, 256B] tile has its 16 lanes exactly 256 B apart, which is 0 mod 32 banks -- every lane on bank 0, a 16-way conflict on the hottest read in the kernel. An XOR fold of the row into the group index spreads them over all 32 banks. BV=64 goes 4538 -> 1597us, and the optimum moves from BV=128 to BV=64 because once conflicts are gone the tile that fits 3 CTAs/CU beats the one that fits 2. Widen the CTA along N (NR_SPLIT). At BV=128 the LDS footprint is structural -- h, k, v and A all have to be shared -- so CTAs/CU cannot be improved. Splitting the N axis over a second set of waves doubles the resident waves at the same footprint instead. 1919 -> 1591us, enough for BV=128 to overtake BV=64 in the middle of the range, so it becomes the gfx950 default. BV=64 is now within 12% of the roofline (4.56 TB/s of ~5.2), so the remaining gap at high batch*head is traffic, not scheduling. Test matrix extended to BV=128 to cover the split path. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A standalone FlyDSL K6 (output projection), signature-compatible with the Triton
chunk_fwd_o_opt_vkso the two can be swapped at the call site. Motivation: theseparate K5+K6 pipeline still runs the Triton output stage, which grows from 12%
of the pipeline at batchhead=8 to 49% at 512 -- the main reason the separate
path loses to the fused kernel at high batchhead.
Two structural differences from the K6 half of the fused kernel:
state recurrence forces it to. K6 carries no cross-chunk state, so the chunk
index becomes a grid axis and low batch*head stops being occupancy-starved.
exp(g_i - g_j)rather than the fused kernel'stelescoping
exp(g_i - g_last). The fused form is only cheap because K5 handsit a pre-gated
v_newfor free; herev_newarrives ungated from HBM, so thatform would cost a column gate over v and put a
>= 1factor on the criticalpath that overflows f32 once a chunk's decay exceeds ~88. The pair form is
evaluated only under the causal mask, where
g_i <= g_jbounds it by 1.Performance (gfx950, seqlen=8192, K=V=128)
Landed at 0.87–1.19x Triton, up from 0.30x at first correct version:
Three optimizations got it there, each aimed at a different cap:
qout of LDS. It is the A operand of both GEMM3 and GEMM4a and the M=BTaxis is already split across waves, so a wave only reads its own 16 rows --
nothing to share. LDS 88 -> 72 KiB, which fits a second CTA per CU.
[rows, 256B]tile has its 16 lanes exactly 256 B apart, i.e. 0 mod 32 banks-- every lane on bank 0, a 16-way conflict on the hottest read in the kernel.
BV=64 went 4538 -> 1597us.
NR_SPLIT, widening the CTA along N. At BV=128 the LDS footprint isstructural (h, k, v and A must all be shared), so CTAs/CU cannot improve;
splitting N over a second set of waves doubles resident waves at the same
footprint. 1919 -> 1591us, making BV=128 the gfx950 default.
BV=64 now runs at 4.56 TB/s, ~88% of roofline, so the residual gap at high
batch*head is traffic rather than scheduling.
Test plan
op_tests/test_flydsl_gdn_o.py-- 75 cases, all passing. Given identicalh/v_new, FlyDSL and Triton agree to an RMSE ratio of 0..1.8e-5 acrossvarlen and dense, tail chunks, GQA,
gandgkgates, both exp/exp2paths, and BV in {32, 64, 128} (128 covers the
NR_SPLIT=2path).the gate-form choice above.
op_tests/flydsl_tests/bench_k6_flydsl_vs_triton.pyfor the table above.chunk.py-- the call site still uses Triton K6, sothis lands dormant. Enabling it is a follow-up.