Repository navigation
DP4A Q4K/Q6K GEMV profile is catastrophic through the Gated DeltaNet recurrence — Qwen3.5 on sm_89 needs the float MWV variant (PMAT-806 class) #3513
Description
Measured 2026-09-19 on lambda-vector (RTX 4090, sm_89) while building the Qwen3.5 GPU forward (#3090, branch PMAT-3477-gdn-gpu, Qwen35CudaModel):
- A single Q4_K DP4A GEMV is 3.67e-3 (relative L∞) from an exact f64 dequantized dot — fine in isolation.
- The DeltaNet-only path (zero attention layers) with the production default
Q4kVariant::HwDp4aends 1.656 apart from the same path with the floatMwvGEMV from a zeroed state; float-vs-float repeats at exactly 0.000. - Full 6-token e2e with
HwDp4aon q4k alone: relative L∞ 1.17…2.84 vs the CPU, argmax 86320/139962/44791/53206/9077/3094 (garbage); q6k alone: 1.08…1.38, argmax 6067/103736/…; float GEMV: argmax identical to the CPU at every position (13,198,11,198,0,908).
So the DP4A Q8_1 activation quantization, harmless per op, compounds through the recurrent state (the same mechanism PMAT-806 found for massive activations on Blackwell). Mitigation shipped in 0.68.2: Qwen35CudaModel pins the float GEMV variants (pin_reference_gemv) for qwen35 regardless of the profile's default. This ticket: characterise which projections drive it (attn_qkv into the conv/recurrence vs the FFN), and either a DP4A variant with a per-row activation scale that survives the recurrence or a documented per-architecture profile rule. Falsifier: the e2e argmax/cosine test in forward_qwen35_cuda_tests.rs under HwDp4a.
Activity
contracts(qwen35-gpu): QHF-GPU-009 floor is the MEASURED one — 0.996 …
noahgift commentedon Sep 23, 2026
milestone → 0.73.0 per docs/roadmaps/milestone-triage-2026-09-23.md (#4024), operator approval 2026-09-23 (relayed by aprender-cf). Reason: performance/decode-prefill path: llama.cpp Parity (#3999)
docs(roadmap): 10 placement corrections pending approval + 6 duplicat…
docs(roadmap): record the 9 applied corrections, the #3513 withdrawal…
docs(roadmap): quorum round 12 -- NOT-applied paragraph matches the t…
test(#3513): state the mutation that was actually proven, not 'any on…
8 remaining items
noahgift commentedon Oct 14, 2026
merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5323 (CUDA quantized kernels: correct block layouts, cache keys and precision), split out of #5312 by theme.
noahgift commentedon Oct 15, 2026
Work started (aprender-4936, branch fix/3513-dp4a-q8-cache off main 8f601ca). The stale-Q8_1 cause is already fixed on main (#4258, cbf623e), so the 09-25 branch is not ported. What remains is the throughput call on the float pin: an interleaved float-vs-DP4A eager decode A/B on the 4090 for 0.8B and 4B, each checked against CPU greedy tokens. The receipt goes here.
noahgift commentedon Oct 15, 2026
Evidence, branch
fix/3513-dp4a-q8-cache@ 2813463 (off main 8f601ca). No PR yet.The ticket's premise is inverted. With #4258 (stale Q8_1 activation) fixed on main, DP4A is faster than the float MWV pair on sm_89 through the Gated DeltaNet recurrence, and it holds CPU parity. So the change lifts the constructor's float pin instead of choosing it.
RTX 4090, release, eager decode, 5 rounds x 64 tokens interleaved (
qwen35_cuda_gemv_pair_ab_*):GPU vs CPU, teacher-forced, 4 prompts x 49 steps (
qwen35_cuda_both_gemv_pairs_match_cpu_teacher_forced_*):Full
qwen35_cudamodule (release,--features cuda): 41 passed, 0 failed, 1 ignored. That includes the graph token-identity, parity-floor and prefill tests; the prefill parity helper pins the float pair explicitly, because its budgets are written against it. Clippy-D warningsis clean.Agent: aprender-4936
test(qwen35-cuda): launch witness + the default pair measured, not al…
noahgift commentedon Oct 15, 2026
Correction and follow-up, branch
fix/3513-dp4a-q8-cache@ 6a8087c.My comment above says the lifted pin hands Qwen3.5 the DP4A pair. It does not.
GpuProfile::auto_q4kkeeps Q4_K on float MWV on every GPU, so the sm_89 default is float Q4_K + half-warp DP4A Q6_K. The pin only ever changed Q6_K. A launch witness exposed this; that pair is now measured as well.RTX 4090, release, eager, 5×64 interleaved, ms/tok:
qwen35_cuda_default_decode_launches_the_profiles_gemvsreads the executor's compiled-module cache after a default decode. It holdsmwv_q4k_gemv_*andhw_dp4a_q6k_gemv_*, and nomwv_q6k_gemv_*.pin_float_gemvcall turns this test andqwen35_cuda_a_fresh_model_keeps_the_profiles_gemv_variantsRED.qwen35_cudamodule passes, 42 passed, 0 failed, 1 ignored. Clippy is clean.Moving Q4_K to DP4A would gain more (4B at 0.792x), but that is the global
auto_q4kpolicy: it is float because of a qwen2.5-coder-1.5B parity failure. That is out of scope for this ticket.Agent: aprender-4936
noahgift commentedon Oct 22, 2026
Tracked in #4001 (triage 2026-10-22).