CUDA quantized kernels: correct block layouts, cache keys and precision #5323

Scope: Correctness of the CUDA quantized kernels: Q5_K block layout, FP8 GEMM stale activations and FP16 overflow, RMSNorm epsilon in the cache key, stale Q8_1 activation caches, unknown quant types decoded as Q4_K, IQ kernels on Blackwell, the DP4A path, and a registry that refuses any unregistered kernel.

Why it matters: Each of these makes the GPU answer differ from the CPU answer or from llama.cpp, silently. A wrong-but-fast kernel is worse than a slow one.

Done when: Every (qtype, layout, arch) kernel in use is in the registry, and the GPU-vs-CPU parity test passes on lambda (sm_89) and gx10 (Blackwell) for every quant type in the model ladder.

Epic: #3994. Folded under triage rule 7b (2026-10-14): each row below was its own issue, closed as not planned and kept here as a sub-ticket. Reopen an issue when its work starts or its PR opens.

Sub-tickets

• CUDA Q5_K gemv/gemm read a non-ggml block layout (qh bit order; gemm also qh/qs offsets and scale unpacking) #3111 CUDA Q5_K gemv/gemm read a non-ggml block layout (qh bit order; gemm also qh/qs offsets and scale unpacking) (milestone 0.72.0); branch fix/3111-q5k-gemm-r2
  • accept: the outcome in the title, with a receipt
• DP4A Q4K/Q6K GEMV profile is catastrophic through the Gated DeltaNet recurrence — Qwen3.5 on sm_89 needs the float MWV variant (PMAT-806 class) #3513 DP4A Q4K/Q6K GEMV profile is catastrophic through the Gated DeltaNet recurrence — Qwen3.5 on sm_89 needs the float MWV variant (PMAT-806 class) (milestone 0.71.0); branch fix/3513-on-0701
  • accept: the outcome in the title, with a receipt
• apr run --gpu is 2-3x SLOWER than --no-gpu on the dense path: the GPU result fails correctness (cosine 0.4153) and falls back to CPU after the user has paid for it #3602 apr run --gpu is 2-3x SLOWER than --no-gpu on the dense path: the GPU result fails correctness (cosine 0.4153) and falls back to CPU after the user has paid for it (milestone 0.71.0); branch w71-e3a/3602-validate-ms-on-main
  • accept: the outcome in the title, with a receipt
• FP8 batched GEMM multiplies a STALE activation on mixed-quant models: the PMAT-084 cache key (ptr, count) aliases across layers — #3602's cosine 0.4153 #3727 FP8 batched GEMM multiplies a STALE activation on mixed-quant models: the PMAT-084 cache key (ptr, count) aliases across layers — apr run --gpu is 2-3x SLOWER than --no-gpu on the dense path: the GPU result fails correctness (cosine 0.4153) and falls back to CPU after the user has paid for it #3602's cosine 0.4153 (milestone 0.73.0)
  • accept: the outcome in the title, with a receipt
• FP8 prefill GEMM overflows its FP16 output: D = true × 448/act_absmax exceeds 65,504 on qwen2.5-coder-7b (cosine 0.5668, lambda) #3728 FP8 prefill GEMM overflows its FP16 output: D = true × 448/act_absmax exceeds 65,504 on qwen2.5-coder-7b (cosine 0.5668, lambda) (milestone 0.73.0)
  • accept: the outcome in the title, with a receipt
• GPU RMSNorm runs with ε = 1e-5, not the model's 1e-6: the kernel cache key omits ε and preload bakes 1e-5 — special-token positions come out 0.431× (qwen2.5-coder-7b, F2 pos 13/44) #3759 GPU RMSNorm runs with ε = 1e-5, not the model's 1e-6: the kernel cache key omits ε and preload bakes 1e-5 — special-token positions come out 0.431× (qwen2.5-coder-7b, F2 pos 13/44) (milestone 0.71.0); branch fix/3759-module-key-lab
  • accept: the outcome in the title, with a receipt
• resolve_qtype silently decodes an UNKNOWN ggml quant as Q4_K, and the whitelist that would refuse it is not on the apr qa GPU path (#3821, second instance) #3850 resolve_qtype silently decodes an UNKNOWN ggml quant as Q4_K, and the whitelist that would refuse it is not on the apr qa GPU path (Verdict-lattice obligation: two surfaces decide GPU-correct(model, host, apr) and disagree silently — apr run --gpu refuses what apr qa ships #3821, second instance) (milestone 0.71.0); branch la-71/3850-fold
  • accept: the outcome in the title, with a receipt
• P0: IQ3_S/IQ2_S GEMV kernels fail to load on cc>=12 (GB10/Blackwell) — GH-480 PTX patch injects .reg inside a global array initializer #4096 P0: IQ3_S/IQ2_S GEMV kernels fail to load on cc>=12 (GB10/Blackwell) — apr-cli: gate realizar/CUDA imports behind feature flags for aarch64 builds #480 PTX patch injects .reg inside a global array initializer (milestone 0.71.0); branch la-71/4096-move-on-main
  • accept: the outcome in the title, with a receipt
• PMAT-027 stale Q8_1 activation cache: batched fallback, batched decode, and Qwen3-MoE forward reuse the previous GEMV's input under DP4A #4378 PMAT-027 stale Q8_1 activation cache: batched fallback, batched decode, and Qwen3-MoE forward reuse the previous GEMV's input under DP4A (milestone backlog); branch fix/4378-q8-cache-invalidate
  • accept: the outcome in the title, with a receipt
• KREG-001: SHACL kernel registry — loader refuses any unregistered (qtype, layout, arch, shape, precision) kernel #4539 KREG-001: SHACL kernel registry — loader refuses any unregistered (qtype, layout, arch, shape, precision) kernel (milestone 0.71.1); branch kreg/4539-parity-receipts
  • accept: the outcome in the title, with a receipt

Folded in by the 0.73–0.76 triage (2026-10-31)

These were open tickets; to keep the repo at 100 open or fewer they now live here as checkboxes. Reopen a ticket when this epic schedules it.

• apr run --gpu is 2-3x SLOWER than --no-gpu on the dense path: the GPU result fails correctness (cosine 0.4153) and falls back to CPU after the user has paid for it #3602 (was 0.73.0): apr run --gpu is 2-3x SLOWER than --no-gpu on the dense path: the GPU result fails correct
• GPU RMSNorm runs with ε = 1e-5, not the model's 1e-6: the kernel cache key omits ε and preload bakes 1e-5 — special-token positions come out 0.431× (qwen2.5-coder-7b, F2 pos 13/44) #3759 (was 0.73.0): GPU RMSNorm runs with ε = 1e-5, not the model's 1e-6: the kernel cache key omits ε and pre
• resolve_qtype silently decodes an UNKNOWN ggml quant as Q4_K, and the whitelist that would refuse it is not on the apr qa GPU path (#3821, second instance) #3850 (was 0.73.0): resolve_qtype silently decodes an UNKNOWN ggml quant as Q4_K, and the whitelist that would
• P0: IQ3_S/IQ2_S GEMV kernels fail to load on cc>=12 (GB10/Blackwell) — GH-480 PTX patch injects .reg inside a global array initializer #4096 (was 0.73.0): P0: IQ3_S/IQ2_S GEMV kernels fail to load on cc>=12 (GB10/Blackwell) — apr-cli: gate realizar/CUDA imports behind feature flags for aarch64 builds #480 PTX patch in