Repository navigation
CUDA Q5_K gemv/gemm read a non-ggml block layout (qh bit order; gemm also qh/qs offsets and scale unpacking) #3111
Description
Found while fixing the CPU readers in #3110 (PMAT-1101). I read the CUDA Q5_K kernels in crates/aprender-gpu/src/kernels/quantize/q5k/ against ggml's block_q5_K (d, dmin, scales[12], qh[32] at offset 16, qs[128] at offset 48; value l of sub-block s takes its fifth bit from bit s of qh[l]; sub-blocks 2c/2c+1 share qs[32c..32c+32], low nibbles then high). Neither kernel matches.
gemv.rs (Q5KGemvKernel, the M=1 decode path; WeightQuantType::Q5K => GemvKernel::Q5K)
qsoffset and nibble selection match ggml.- The fifth bit is read from
qh[val_idx / 8] >> (val_idx % 8), a sequential bitmask. ggml puts it atqh[val_idx % 32] >> (val_idx / 32), so it differs for most values.
gemm.rs (Q5KKernel)
- Assumes
qssits at 16..144 andqhat 144..176. ggml hasqhat 16..48 andqsat 48..176. - Takes sub-block
s'sqsbytes from16 + s*16 + lane/2, and the bit from(s*32 + lane)as a sequential bitmask. - Unpacks the scales as a 12-bit stride (
bit_offset = s*12) and normalises by1/63. That is not ggml'sget_scale_min_k4.
Not measured yet. This comes from reading the code. The plan is to reuse the gguf-py golden super-block from FALSIFY-QDOT-009 (#3110) as a GPU-side test: gemv against the gguf-py dot, and gemm against the reference matmul. It will run on lambda (sm_89) and gx10 (sm_121), then the kernels get fixed.
Impact, if confirmed: --gpu on any model with Q5_K tensors (every Q5_K_M/Q5_K_S file, plus the Q5_K tensors inside Q4_K_M files) computes wrong projections on CUDA, and the CUDA release assets ship that path.
Activity
noahgift commentedon Sep 11, 2026
Measured on an RTX 4090 (lambda), 2026-09-11. This is no longer just a reading of the code. The GEMV was run on the FALSIFY-QDOT-009 golden super-block from #3110, which is real bytes from a llama.cpp-quantized GGUF:
GEMV fix → #3113 (PMAT-1104). It is stacked on #3110 and has auto-merge on.
Q5KKernel(the GEMM) still has its own wrong layout: qh/qs offsets and the scale unpack. Inference never dispatches it; onlyq5k_matvecin the layer-trace path and tests call it. This issue stays open for it.noahgift commentedon Sep 12, 2026
Train #3127 landed (squash cb829fc, 2026-09-12T09:12Z) with #3110 (CPU +
apr importQ5_K readers, oracle = gguf-py on a real block) and #3113 (CUDA Q5_K GEMV: the fifth bit read from a sequential bitmask). This issue also names the GEMM's qh/qs offsets — not claimed by either row, so it stays open until a GEMM-path receipt against the same oracle exists. Milestone follows the freeze.noahgift commentedon Sep 12, 2026
slipped_from: 0.67.0 — §4.1 freeze (06x-release-schedule), 2026-09-12: the train landed #3110 + #3113 (CPU, apr import, CUDA GEMV); the GEMM qh/qs half has no receipt against the gguf-py oracle yet.
release(0.67): train A — 17 row PRs of the 0.67.0 train in one build …
merge origin/main — the Q5_K conflicts are main's own renames (FALSIF…
noahgift commentedon Sep 25, 2026
Parked by APR-EPIC-001: over E3 budget (rank 10/79). Reopen = pull. Applied by aprender-d1 under a 3-lane quorum receipt (handoff/balance-quorum-rev4.json).
11 remaining items
noahgift commentedon Oct 12, 2026
GEMM half re-cut on current main: branch
fix/3111-q5k-gemm-r2@ 2695cd0. This is a cherry-pick of 232c198 onto main 6bd89b0. It applied cleanly: main has not touched any of the 5 files since the old base. No PR is open; the branch waits for a slot.Re-measured on an RTX 4090 (sm_89) at 2695cd0:
cargo test -p aprender-gpu --features cuda --lib q5k_: 35/35 pass, includingq5k_gemm_matches_the_cpu_matmulandq5k_dequant_matches_the_cpu_reader.cargo test -p aprender-serve --features cuda --lib q5k_matvec: 2/2 pass, includingtest_3111_q5k_matvec_matches_cpu_dequant.gemm.rs:147, qh offset 16→144):q5k_gemm_matches_the_cpu_matmulFAILS with gpu 1259.49 vs cpu 1052.54, the same numbers as the 09-26 receipt. The file was restored afterwards.cargo fmt --all -- --check: clean.The GEMV half landed in #3113. Once this branch merges, both halves named in this issue have a receipt against the oracle.
noahgift commentedon Oct 14, 2026
P0 to P2 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, and its milestone is 0.72.0.
noahgift commentedon Oct 14, 2026
merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5323 (CUDA quantized kernels: correct block layouts, cache keys and precision), split out of #5312 by theme.