Repository navigation
qwen35 batched prefill: flash attention for the full-attention layers — 32k prefill 3.2k tok/s vs llama.cpp 10.1k (0.26× ollama) #4442
Description
Child of #4220 (CRUX ollama, part 2). Found by measurement on lambda (RTX 4090), same GGUF (Qwen3.5-4B-Q4_K_M.gguf, sha256 00fe7986…), apr 0.69.5-rc.1 (5660b0877).
Observed. apr's batched prefill rate falls off with prompt length. llama.cpp's does not.
| prompt tokens | apr batched prefill (serve trace line) | ollama v0.34.4, server prompt_eval | llama-bench b11050 -fa 1 pp |
|---|---|---|---|
| 856–863 | 5389 tok/s (159 ms) | 5765 | 10267 ± 1319 (pp850) |
| 4075–4082 | 7798 tok/s (523 ms) | 10166 | 11490 ± 326 (pp4096) |
| 32727–32734 | 3178 tok/s (10300 ms) | 12365 | 10099 ± 55 (pp32768) |
At 32k apr is 0.26× ollama. The trace line reads attention cuBLAS f32: the 8 full-attention layers materialise the score matrix per chunk, so attention is quadratic in memory traffic. llama.cpp runs these layers with flash attention (-fa 1; ollama's default is --flash-attn auto).
This is not #3787: 10300 ms of the 10428 ms TTFT is inside the prefill trace line.
Ask. A fused flash-attention prefill kernel (online softmax, no score matrix) for the qwen35 full-attention layers in the batched prefill path. Done when the serve trace line at 32k is at least 0.8× llama-bench pp32768 on the same box and GGUF, and the temp-0 output matches the current path's.
Receipts: /mnt/nvme-raid0/tmp/crux4220/p2/out-4b-r2/ (serve.log, llama-bench.json, ollama.jsonl).
Activity
noahgift commentedon Sep 25, 2026
A/B:
APR_QWEN35_PREFILL_ATTENTION=f32vsflash: flash is slower at every length, and 1.5× slower at 32k.Binary
apr 0.69.5-rc.1 (5660b0877)sha2564bfdacd2…eb49014, RTX 4090 (driver 580.119.02), Qwen3.5-4B Q4_K_M,serve run --gpu-layers all --context-length 40960 --trace. Arms interleaved f32→flash, 2 replicates each.attention flash (f16 inputs, f32 accumulation)prefill lines and no cuBLAS lines. Each f32 log has 8attention cuBLAS f32lines and 0 flash lines.a78db9251af1…over the 3 non-warmup completions).Conclusion: the flash prefill path as shipped does not close the 32k gap (f32 32k ≈ 3.9k tok/s → flash ≈ 2.5k tok/s, vs llama.cpp 10.1k). f32 should stay the default. The flash kernel needs profiling (
apr profile/ nsys on the 32k chunk) before it is a candidate.docs(findings): stage/0.70.1 prefill projections lost the f16 tensor-…
noahgift commentedon Oct 14, 2026
merged into #5305 (master ticket for epic #3598; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.