Skip to content

Dev/huizzhan/flydsl gdn k6 - #1

Closed
huizzhan wants to merge 2 commits into
dev/huizzhan/flydsl_prefill_gdn_blockfrom
dev/huizzhan/flydsl_gdn_k6
Closed

Dev/huizzhan/flydsl gdn k6#1
huizzhan wants to merge 2 commits into
dev/huizzhan/flydsl_prefill_gdn_blockfrom
dev/huizzhan/flydsl_gdn_k6

Conversation

@huizzhan

@huizzhan huizzhan commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Summary

A standalone FlyDSL K6 (output projection), signature-compatible with the Triton
chunk_fwd_o_opt_vk so the two can be swapped at the call site. Motivation: the
separate K5+K6 pipeline still runs the Triton output stage, which grows from 12%
of the pipeline at batchhead=8 to 49% at 512 -- the main reason the separate
path loses to the fused kernel at high batch
head.

Two structural differences from the K6 half of the fused kernel:

  • Chunk-parallel grid. The fused kernel walks chunks serially because K5's
    state recurrence forces it to. K6 carries no cross-chunk state, so the chunk
    index becomes a grid axis and low batch*head stops being occupancy-starved.
  • Triton's pair gate exp(g_i - g_j) rather than the fused kernel's
    telescoping exp(g_i - g_last). The fused form is only cheap because K5 hands
    it a pre-gated v_new for free; here v_new arrives ungated from HBM, so that
    form would cost a column gate over v and put a >= 1 factor on the critical
    path that overflows f32 once a chunk's decay exceeds ~88. The pair form is
    evaluated only under the causal mask, where g_i <= g_j bounds it by 1.

Performance (gfx950, seqlen=8192, K=V=128)

Landed at 0.87–1.19x Triton, up from 0.30x at first correct version:

batch*head Triton (us) FlyDSL (us) speedup
8 33.5 28.2 1.19x
64 165.5 162.1 1.02x
512 1428.1 1603.2 0.89x

Three optimizations got it there, each aimed at a different cap:

  1. q out of LDS. It is the A operand of both GEMM3 and GEMM4a and the M=BT
    axis is already split across waves, so a wave only reads its own 16 rows --
    nothing to share. LDS 88 -> 72 KiB, which fits a second CTA per CU.
  2. XOR swizzle on the LDS tiles. The dominant cost. A B-operand read of a
    [rows, 256B] tile has its 16 lanes exactly 256 B apart, i.e. 0 mod 32 banks
    -- every lane on bank 0, a 16-way conflict on the hottest read in the kernel.
    BV=64 went 4538 -> 1597us.
  3. NR_SPLIT, widening the CTA along N. At BV=128 the LDS footprint is
    structural (h, k, v and A must all be shared), so CTAs/CU cannot improve;
    splitting N over a second set of waves doubles resident waves at the same
    footprint. 1919 -> 1591us, making BV=128 the gfx950 default.

BV=64 now runs at 4.56 TB/s, ~88% of roofline, so the residual gap at high
batch*head is traffic rather than scheduling.

Test plan

  • op_tests/test_flydsl_gdn_o.py -- 75 cases, all passing. Given identical
    h/v_new, FlyDSL and Triton agree to an RMSE ratio of 0..1.8e-5 across
    varlen and dense, tail chunks, GQA, g and gk gates, both exp/exp2
    paths, and BV in {32, 64, 128} (128 covers the NR_SPLIT=2 path).
  • Strong-decay regression (-5/token, ~-320 cumulative per chunk) guarding
    the gate-form choice above.
  • op_tests/flydsl_tests/bench_k6_flydsl_vs_triton.py for the table above.
  • Not yet wired into chunk.py -- the call site still uses Triton K6, so
    this lands dormant. Enabling it is a follow-up.

root and others added 2 commits September 1, 2026 01:34
The separate K5+K6 pipeline still runs the Triton output stage, which grows
from 12% of the pipeline at batch*head=8 to 49% at 512 -- the main reason the
separate path loses to the fused kernel at high batch*head. This adds a FlyDSL
K6 that is signature-compatible with chunk_fwd_o_opt_vk so the two can be
swapped at the call site.

Two structural differences from the K6 half of the fused kernel:

- Chunk-parallel grid. The fused kernel walks chunks serially because K5's
  state recurrence forces it to; K6 carries no cross-chunk state, so the chunk
  index becomes a grid axis and low batch*head stops being occupancy-starved.

- Triton's pair gate exp(g_i - g_j) rather than the fused kernel's telescoping
  exp(g_i - g_last). The fused form is only cheap because K5 hands it a
  pre-gated v_new for free; here v_new arrives ungated from HBM, so that form
  would cost a column gate over v AND put a >= 1 factor on the critical path
  that overflows f32 once a chunk's decay exceeds ~88. The pair form is
  evaluated only under the causal mask, where g_i <= g_j bounds it by 1.

Correctness only at this point: given identical h / v_new, the two kernels
agree to an RMSE ratio of 0..1.8e-5 across varlen and dense, tail chunks, GQA,
g and gk gates, and both exp/exp2 paths. Performance is 0.28-0.42x Triton --
the kernel is occupancy-bound at 88 KiB LDS (1 CTA/CU), which the tiling and
swizzle passes still have to address.

Co-authored-by: Cursor <cursoragent@cursor.com>
Three changes, each aimed at a different cap on the kernel:

Take q out of LDS. It is the A operand of both GEMM3 and GEMM4a, and the
M=BT axis is already split across waves, so a wave only ever reads its own 16
rows -- there was nothing to share. Its 4 lanes-worth of elements run along K,
which is contiguous in HBM, so an A-fragment load is one 64b buffer read per
K-tile and both GEMMs reuse the fragment. LDS drops 88 -> 72 KiB, which is what
gets a second CTA onto a CU. 4720 -> 3340us at BV=128.

Swizzle the LDS tiles. This was the real cost. The B-operand read of a
[rows, 256B] tile has its 16 lanes exactly 256 B apart, which is 0 mod 32 banks
-- every lane on bank 0, a 16-way conflict on the hottest read in the kernel.
An XOR fold of the row into the group index spreads them over all 32 banks.
BV=64 goes 4538 -> 1597us, and the optimum moves from BV=128 to BV=64 because
once conflicts are gone the tile that fits 3 CTAs/CU beats the one that fits 2.

Widen the CTA along N (NR_SPLIT). At BV=128 the LDS footprint is structural --
h, k, v and A all have to be shared -- so CTAs/CU cannot be improved. Splitting
the N axis over a second set of waves doubles the resident waves at the same
footprint instead. 1919 -> 1591us, enough for BV=128 to overtake BV=64 in the
middle of the range, so it becomes the gfx950 default.

BV=64 is now within 12% of the roofline (4.56 TB/s of ~5.2), so the remaining
gap at high batch*head is traffic, not scheduling. Test matrix extended to
BV=128 to cover the split path.

Co-authored-by: Cursor <cursoragent@cursor.com>
@huizzhan huizzhan closed this Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant