Repository navigation
Class: the shipped CPU path runs a narrower kernel than the host supports — dispatch returns before the wider ISA, kernels absent, module DCE'd, prefill run as decode #2880
Description
Mechanism. The numbers are correct and only the speed is wrong, so no correctness gate can see it: the AVX-512 VNNI Q4_K kernels are compiled, linked and structurally unreachable because the avx2 && fma fast path returns first; Q5_K/Q6_K have no AVX-512 kernel at all so the largest tensor (the Q6_K lm_head) runs on 256-bit lanes; the entire trueno Q4_K GEMV module is dead-code-eliminated from the shipped apr; the aarch64 matmul_q4k_f32_parallel called the scalar path (parallel half fixed, NEON half open); CPU prefill loops forward_single_with_cache per prompt token — decode run once per token, 6.5× behind llama.cpp on the same box. Every instance was found by perf probes or perf counters on the published artifact, never by a test.
The one structural fix. A per-ISA kernel-reachability gate on the built artifact: for each (host ISA, qtype, shape class) the dispatch path TAKEN is recorded (perf probe / counter, the compute_class idea one level down) and compared against the widest kernel the host supports; a narrower path is a REPORTED regression with an owner, never silence. Then the kernels: CPU batched prefill (GEMM, not per-token matvec), AVX-512 Q5_K/Q6_K, the VNNI gate before the AVX2 return, NEON Q4_K/Q6_K.
Lane. 0.67 by D-1 (kernel work). PP-066 §6 carries W-B kernel (Q4_K GEMV family) and W-G (GB10 NEON Q4_K GEMV, batched prefill on sm_121) but no CPU-x86 kernel row and no CPU-prefill row — that gap is flagged for the epic session by this class.
Instances (one line each):
- AVX-512 VNNI Q4_K kernels are compiled, linked and structurally unreachable — the
avx2 && fmafast path returns before the VNNI gate is ever read #2604 AVX-512 VNNI Q4_K kernels compiled, linked, never executed (0 probe hits on two hosts). - Q5_K and Q6_K have no AVX-512 kernel at all, so an AVX-512 host runs its largest tensor on 256-bit lanes #2605 no AVX-512 Q5_K/Q6_K kernel; the Q6_K lm_head runs on 256-bit lanes.
- The entire trueno Q4_K GEMV module (AVX-512 branch included) is dead-code-eliminated from the shipped
aprbinary #2611 the trueno Q4_K GEMV module (AVX-512 branch included) is DCE'd from the shippedapr. - Q4_K GEMV has no aarch64 SIMD and its "parallel" variant calls the scalar path #2567 Q4_K GEMV has no aarch64 SIMD (parallel half fixed in PR perf(compute): the aarch64 Q4_K "parallel" path is now actually parallel — 1.21x measured on GB10 (#2567) #2689, 1.21× on GB10; NEON half open → W-G).
- apr prefills at decode rate on CPU: no batched prefill outside src/cuda/, 6.5x slower than llama.cpp on the same box #2787 no batched CPU prefill outside
src/cuda/; prefill-to-decode ratio 1.07× on intel, 1.02× on mini.
Rows. §6 W-B kernel, W-G; the CPU-x86 kernel row is a §6 gap.
Members (parented here): #2567, #2604, #2605, #2611, #2787
PP-066 triage 2026-09-05 (issue #2873); ledger docs/audits/pp-066-triage-2026-09-05.md.
Activity
avx2 && fmafast path returns before the VNNI gate is ever read #2604aprbinary #2611noahgift commentedon Sep 23, 2026
triage 2026-09-23 (aprender-dd, standing triage owner): milestoned to 0.73.0, theme llama.cpp Parity (performance, kernels, parity benches); it moved out of
backlog(the 2026-09-17 overflow bucket, not a release).noahgift commentedon Sep 26, 2026
Measured (aprender-1c, E6): the "VNNI gate before the AVX2 return" item is falsified on the fleet's AVX-512 host. It stays as it is.
fused_q4k_q8k_parallel_matvec_into(quantize/q5k_q6k_matvec.rs, the compiled copy), the lean AVX2 path does return before the VNNI 4-row kernel. So VNNI is unreachable on x86, as this issue says.1c/2880-vnni-reachable: a dispatch comment citing the measurement, plus an#[ignore]re-measure test (vnni_reach_bench_tests.rs). Re-measure before reordering on a host with native 512-bit datapaths.quantize/fused_q.rsis not compiled (nothinginclude!s it). Three falsify tests still cite it as "the CPU side".Still open in this class: AVX-512 Q5_K/Q6_K kernels (none exist), NEON Q4_K/Q6_K, CPU batched prefill (#2801), and the per-ISA reachability gate.
🤖 Generated with Claude Code
14 remaining items
docs(audit): quorum-PMAT-2880 3/3 PASS through slice 6
shifulegend commentedon Sep 26, 2026
That Zen 4 VNNI measurement (slower than lean AVX2) is a really interesting edge case. I hit something basically identical when writing the SIMD dispatch for Project Zero (a zero-dependency C99 inference engine I built). We had AVX-512 VBMI LUT kernels that would do 36.25 tok/s on an older Xeon, but on some consumer chips the latency of the wider instructions meant the AVX2 fallback was actually faster in practice.
Are you planning to add any microarchitecture-aware dynamic dispatch to handle the Zen 4 VNNI case, or just letting the AVX2 path win out universally where measured?
noahgift commentedon Oct 14, 2026
P0 to P2 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, and its milestone is 0.73.0.
noahgift commentedon Oct 14, 2026
merged into #5313 (master ticket for epic #3999; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5338 (Backend portability: correct kernels and config on every CPU, wgpu and CUDA target), split out of #5313 by theme.