glm: GLM-5.3-Flash on CUDA, prompts 267.6 → 447.9 tok/s and decode 19.9 → 25.1 tok/s, every reply unchanged (#593) by justincasey · Pull Request #839 · ashhart/TensorFold

cuda: EXL3 linear and grouped routed experts, every codebook and width, a width per expert

justincasey

committed

224406d

glm: GLM-5.3-Flash on CUDA at TP=2 from the unsplit EXL3 checkpoint, run by tf-cuda-test glm-model

justincasey

committed

daf0a3f

glm: decode steps in the dense window replay a captured graph, one per KDA parity

justincasey

committed

778ac41

glm: hc_partial runs each group of four mixing dots in its own block (six times the blocks, the same sums)

justincasey

committed

24640f3

glm: bf16 matmul weights load in mma fragment order, two 16-byte loads a lane and K step

justincasey

committed

6193780

glm: a one-row Scratch writes q|k|v, f_a, g_a, b, gate|up and the shared slot in place, no copies

justincasey

committed

fd37b24

cuda: EXL3: the split-K counters are zeroed once, not before every call (each launch leaves them zero)

justincasey

committed

96d9b9d

glm: decode steps past the dense limit replay a graph per pool bucket (sparse kernels read the device position)

justincasey

committed

61d9956

glm: latent attention stops at a chunk's first tile past the last visible key (that tile and later ones change nothing)

justincasey

committed

73cd3c4

cuda: EXL3 experts: even widths in decode windows (16 rows at most) read trellis words as staged 16-byte no-allocate loads, two steps in flight

justincasey

committed

19c9736

glm: a KDA layer's f_a, g_a and b run as one stacked bf16 matmul (the same split, the same sums)

justincasey

committed

7b03095

cuda: EXL3 linear: even widths read a K step as staged 16-byte no-allocate loads, two steps in flight

justincasey

committed

30ed802

glm: the MoE router and the DSA indexer gate take bf16 weights in mma fragment order too

justincasey

committed

1044d26

glm: pool scores run one 64-pool block a program (a block's sums are its own, so the same bits)

justincasey

committed

fa1d428

glm: latent attention stages its block's query heads in shared memory once, not from global every K step

justincasey

committed

c70dcfb

glm: latent attention reads a thread's fp8 key bytes as four 16-byte loads

justincasey

committed

a841c4d

glm: a layer's strided row copies are one cuMemcpy2DAsync, not a copy a row

justincasey

committed

8f00f69

cuda: EXL3 experts: a grouped program walks its expert's member tiles in order to the first empty one, not a block a tile

justincasey

committed

d47816b

cuda: EXL3 experts: grouping is a stable counting sort over warp slices, not every expert scanning every pick

justincasey

committed

f37e715

glm: the fp32 MoE combine loads a thread's nine slots as 16-byte loads before its FMA chains

justincasey

committed

e3d6a81

cuda: EXL3 linear: prompt blocks of 64 rows decode each weight tile once, with x staged in shared memory

justincasey

committed

5e307e5

docs: GLM-5.3-Flash on CUDA's decode graphs and speeds, and the EXL3 prompt kernel

justincasey

committed

9a0484c

Navigation Menu

Repository navigation

glm: GLM-5.3-Flash on CUDA, prompts 267.6 → 447.9 tok/s and decode 19.9 → 25.1 tok/s, every reply unchanged (#593) - #839

glm: GLM-5.3-Flash on CUDA, prompts 267.6 → 447.9 tok/s and decode 19.9 → 25.1 tok/s, every reply unchanged (#593)#839
justincasey wants to merge 22 commits into
ashhart:mainfrom
justincasey:up/glm-cuda-perf

Commits on Oct 14, 2026

cuda: EXL3 linear and grouped routed experts, every codebook and width, a width per expert

glm: GLM-5.3-Flash on CUDA at TP=2 from the unsplit EXL3 checkpoint, run by tf-cuda-test glm-model

glm: decode steps in the dense window replay a captured graph, one per KDA parity

glm: hc_partial runs each group of four mixing dots in its own block (six times the blocks, the same sums)

glm: bf16 matmul weights load in mma fragment order, two 16-byte loads a lane and K step

glm: a one-row Scratch writes q|k|v, f_a, g_a, b, gate|up and the shared slot in place, no copies

cuda: EXL3: the split-K counters are zeroed once, not before every call (each launch leaves them zero)

glm: decode steps past the dense limit replay a graph per pool bucket (sparse kernels read the device position)

glm: latent attention stops at a chunk's first tile past the last visible key (that tile and later ones change nothing)

cuda: EXL3 experts: even widths in decode windows (16 rows at most) read trellis words as staged 16-byte no-allocate loads, two steps in flight

glm: a KDA layer's f_a, g_a and b run as one stacked bf16 matmul (the same split, the same sums)

cuda: EXL3 linear: even widths read a K step as staged 16-byte no-allocate loads, two steps in flight

glm: the MoE router and the DSA indexer gate take bf16 weights in mma fragment order too

glm: pool scores run one 64-pool block a program (a block's sums are its own, so the same bits)

glm: latent attention stages its block's query heads in shared memory once, not from global every K step

glm: latent attention reads a thread's fp8 key bytes as four 16-byte loads

glm: a layer's strided row copies are one cuMemcpy2DAsync, not a copy a row

cuda: EXL3 experts: a grouped program walks its expert's member tiles in order to the first empty one, not a block a tile

cuda: EXL3 experts: grouping is a stable counting sort over warp slices, not every expert scanning every pick

glm: the fp32 MoE combine loads a thread's nine slots as 16-byte loads before its FMA chains

cuda: EXL3 linear: prompt blocks of 64 rows decode each weight tile once, with x staged in shared memory

docs: GLM-5.3-Flash on CUDA's decode graphs and speeds, and the EXL3 prompt kernel