Repository navigation
0.70.x: prefill kernel parity — llama-bench pp ≥0.8x llama.cpp (today 0.19–0.20x) #4376
Description
Cop ruling 2026-09-25 10:36Z: the 0.69.5 gate is G3 serve-TTFT (#4313 as written). The kernel-level number is a separate target filed here.
Measured (aprender-98): apr batched prefill on 1006 tokens takes ~430 ms. llama-bench pp1006 at d1d3c3396 is 11,915 tok/s. Ratio 0.19–0.20x, Qwen3.5-4B Q4_K_M, RTX 4090.
Target: pp ≥0.8x at the pin, quiesced, n=3.
Carries: the MMQ int8 activation matmul (the unfinished half of #4313) and #3513 (DP4A through the GDN recurrence).
Reported in the 0.69.5 rc notes. It must not be called 'prefill parity' until it passes.
🤖 Generated with Claude Code
Activity
feat(ptx): int8 mma.sync m16n8k32 s32.s8.s8.s32 — MMQ building block …
feat(prefill): mmq leg — tensor-core Q4K×Q8_1 GEMM, opt-in (Refs #4376)
docs(4376): mmq v1 speed receipt — 1.6–1.9x over dp4a, 0.6–0.7x of f1…
perf(prefill): mmq v2 — smem-tiled 64x128 blocks, magic-seeded mma, 6…
perf(prefill): mmq leg runs non-Q4K projections on f16 — 0.86-0.94x o…
perf(prefill): mmq v3a — ldmatrix fragments, v4 scale loads, cp.async…
noahgift commentedon Oct 14, 2026
merged into #5305 (master ticket for epic #3598; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.