Repository navigation
perf(qwen35-cuda): decode GEMV at 223–504 GB/s vs llama.cpp MMVQ 776–894 — Q5_K 3.5×, Q4_K/Q6_K 1.75× slower (79% of the 4B decode gap) #5221
Description
Follow-up from the #4220 sweep (issuecomment-6127623356). Qwen3.5 decode on CUDA runs at 0.47–0.56× ollama on every size, 0.8B through 27B. The ratio does not improve at 27B, so launch overhead is not the main cause; the cause is kernel bandwidth.
Measured: nsys kernel sums, Qwen3.5-4B Q4_K_M, RTX 4090, 255 decode steps on each engine
- apr: 0.71.1 (03d4af1),
apr run --max-tokens 256 --temperature 0. The trace showsBackend: GPU. - llama.cpp:
llama-bench -n 255 -ngl 99at 60b06ab9a, measuring 187 tok/s. ollama 0.40.3 measured 200 tok/s in the CRUX parity spike: apr serve vs ollama on Qwen3.5 (0.8B/2B/4B/9B/27B) — measure gap, name techniques, file fixes #4220 sweep. - Tool:
nsys profile -t cuda --cuda-graph-trace=node, thennsys stats -r cuda_gpu_kern_sum. - Bandwidth figure: (bytes of that type's tensors in the file) × (decode steps) ÷ (kernel time). Every weight tensor is read once per step; call counts ÷ tensor counts = 255 on all four types.
| weight type (bytes in file) | apr kernel | apr ms | apr GB/s | llama.cpp kernel | llama.cpp ms | llama.cpp GB/s | apr slower by |
|---|---|---|---|---|---|---|---|
| Q4_K (1.361 GB; 131 tensors) | mwv_q4k_gemv |
760.4 | 456 | mul_mat_vec_q<12> |
436.1 | 799 | 1.74× |
Q5_K (0.519 GB; 48 × attn_qkv) |
q5k_gemv_warp_reduce |
593.9 | 223 | mul_mat_vec_q<13> |
171.3 | 776 | 3.47× |
Q6_K (0.842 GB; 16 × ffn_down, 5 × attn_v, tied LM head) |
mwv_q6k_gemv |
425.8 | 504 | mul_mat_vec_q<14> |
241.1 | 894 | 1.77× |
Q8_0 (4 MB; ssm_alpha/ssm_beta) |
q8_0_gemv_warp_reduce |
87.1 | — (7.1 µs/call) | mul_mat_vec_q<8> |
23.4 | — (1.9 µs/call) | 3.7× |
Totals:
- All GPU kernels: apr 2423 ms, about 9.5 ms per token; llama.cpp 1160 ms, about 4.5 ms per token.
- GEMV kernels make up 996 ms (79%) of the 1263 ms gap.
- The Q5_K kernel alone is 423 ms of it, the largest single line. It is also the only weight type still on the old
warp_reducekernel; Q4_K and Q6_K run onmwv.
What this ticket is
Raise the decode GEMV kernels toward llama.cpp's MMVQ bandwidth on the Qwen3.5 decode path, in this order:
- Q5_K: move to the multi-warp (
mwv) shape, or to a DP4A shape. - Q4_K / Q6_K: close the 1.75× gap.
- Q8_0: the small-matrix calls are latency-bound; fuse or batch the alpha/beta projections.
Acceptance:
- A before/after nsys kernel table like the one above, from the pinned binary (version + sha).
apr servedecode tok/s against ollama, measured withscripts/crux_ollama_parity.sh(onspike/4220-crux-ollama-parity).- A correctness falsifier: greedy text identical, or a stated tolerance.
Not in scope:
- perf(serve): decode GEMV precision on Qwen3.5 (RCA-SRV-001 fix 3) #4485 (owner la-0.71) is the DP4A precision decision. Qwen3.5 still pins the float GEMV; see
Qwen35CudaModel::pin_float_gemv. Its last receipt showed +4% from DP4A, so precision alone is not the lever. - Decode attention (Qwen3.5 CUDA long-context DECODE is slow: the decode attention kernel runs 16 blocks — split-K needed for consumer use (RAH ~10 tok/s at 16k) #3725) also stays out.
Other findings from the same trace:
- The Gated DeltaNet recurrence is the other 20% of the gap.
gdn_delta_rule_recurrencetakes 273.7 ms (44.7 µs/call); llama.cpp'sgated_delta_net_cuda<128>takes 15.6 ms (2.5 µs/call), so apr is 17.5× slower there. Tracked separately. apr profilerefuses qwen35. It goes through the dense loader and fails with "Load it withapr run", so this profile used nsys.
Activity
noahgift commentedon Oct 12, 2026
Receipt: multi-warp Q5_K GEMV, branch
perf/5221-qwen35-decode-gemv-bw@ 848961b (branch-only, no PR)Change. On the decode path,
q5k_gemv_intonow dispatches a newmwv_q5k_gemvkernel (MultiWarpQ5KGemvKernel, 3 warps by default,MWV_WARPSoverride) whenk % 256 == 0. Before this, the single-warpq5k_gemv_warp_reducewas the slowest per-byte GEMV in the Qwen3.5 decode step. The old kernel still handles any other K.Kernel time. Qwen3.5-4B-Q4_K_M, RTX 4090, greedy, 256 tokens, measured with
nsys -t cuda --cuda-graph-trace=nodeandcuda_gpu_kern_sum:q5k_gemv_warp_reduce, about 223 GB/s)mwv_q5k_gemv, about 494 GB/s), 2.21×MWV_WARPS sweep (all kernels, ms over the run).
The default stays at 3. Warp count is not the lever. The remaining gap to llama.cpp's Q5_K (about 776 GB/s) is load granularity: the kernel loads one byte at a time where the Q4_K
mwvvectorized kernel issues u32 loads. That vectorized Q5_K version is the next step.End to end (
crux_ollama_parity.sh, the same GGUF on the same host, 3 × 128-token decode):That is +15.5% decode tok/s. The greedy 256-token text is byte-identical before and after; only the timing line differs.
Tests.
aprender-gpu --features cuda: themwv_q5ktests pass 6/6. They cover the build at 4B shapes, the parameters, the shared-memory size, barrier safety for every warp count, the kernel name and the layout offsets.aprender-serve --features cuda --lib -- q5k gemv_entry_name: 122/122 pass.q5k_gemv_intoon the multi-warp path against gguf-py values: a one-hot test over super-blocks 0, 5 and 10, and a ramp dot-product test.The doc comment in
q5k/mod.rsgives the wrong Q5_K layout (qs at 16, qh at 144). The ggml layout is d, dmin, scales[12], qh[32] at 16, qs[128] at 48; the code and the new kernel follow ggml.perf(cuda): u32 qs/qh loads in mwv_q5k_gemv — 21.9 → 16.7 µs per call…
perf(cuda): mwv_q6k_gemv loads each quant byte once — 78.6 → 61.4 µs …
noahgift commentedon Oct 14, 2026
merged into #5305 (master ticket for epic #3598; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5320 (Qwen3.5 CUDA decode speed: close the GEMV and DeltaNet gaps to llama.cpp), split out of #5305 by theme.