Repository navigation
PMAT-027 stale Q8_1 activation cache: batched fallback, batched decode, and Qwen3-MoE forward reuse the previous GEMV's input under DP4A #4378
Description
Found while root-causing #3513. That bug was a stale PMAT-027 Q8_1 activation cache in Qwen35CudaModel; DP4A was not "catastrophic through the recurrence".
q8_activation_valid is keyed on nothing. The DP4A GEMVs (q4k_mwv_gemv.rs :125/:235/:467/:628, weight.rs :262/:352) quantize only when the flag is false, then set it true. Any caller that feeds a NEW input without clearing the flag gets the PREVIOUS input's quantized activation, with no error.
A read-only audit found three more paths at risk (not yet measured):
- Per-sequence fallback,
cuda/executor/layers/batched_qkv.rs:31-52(USE_BATCHED_GEMV=false): sequences 1..m reuse sequence 0's activation. It is reached from:cublas_prefill/attention.rs:1182layers/par-121.rs:284(batched LM head)prefill.rs:868-936graph_dispatch.rs:90
- Batched decode, 2<=m<=8, Q4K, HwDp4a:
batched_ffn.rs:74(output projection) and:153(FFN down), andbatched_forward.rs:412(LM head), with no clear after the fused QKV and gate/up. gguf/cuda/forward_qwen3_moe_resident.rshas no invalidations at all, the same shape as DP4A Q4K/Q6K GEMV profile is catastrophic through the Gated DeltaNet recurrence — Qwen3.5 on sm_89 needs the float MWV variant (PMAT-806 class) #3513. The affected sites are:- the output projection at
:976 - experts at
:1045 - the Q/router GEMVs at
:903and:1001 - the LM head at
:861
- the output projection at
Suggested fix: clear the flag inside the per-sequence loop and at the start of batched_gemv_or_gemm, and before each GEMV in the MoE forward. Then add a parity test for each path, and prove it by deleting the clear and watching it go red. #3513 adds CudaExecutor::invalidate_q8_activation() (branch fix/3513-dp4a-groups).
Activity
test(cuda): #4378 unit test ran on zero weights; the DP4A recurrence …
noahgift commentedon Sep 25, 2026
GPU evidence, fix vs mutant. Branch
fix/4378-q8-cache-invalidate@383bbb2cc, RTX 4090, gpu-q, private target dir. The mutant reducesq8_cache_hitfromvalid && cached == srctovalid, which is the pre-fix behaviour.test_hw_dp4a_new_input_requantizes_without_a_clearqwen35_cuda_dp4a_gemv_matches_the_cpu_through_the_recurrence(real Qwen3.5-0.8B, 6 positions)FIX_RC=0, MUT_RC=101.
The finding: the "DP4A is catastrophic through the Qwen3.5 recurrence" result behind the float-GEMV pin (PMAT-3477 / #3090, measured at "1.656 relative") was this stale-cache bug, not int8 activation error. With the cache keyed on (pointer, length), DP4A matches the CPU. With the old flag-only key, it reproduces the 1.6-relative collapse exactly.
383bbb2ccalso fixes the new unit test, which ran on the harness's all-zero weights and proved nothing (both builds tripped its ownassert_ne!). The falsifier is now inverted into a parity regression. The float pin itself is unchanged in this branch; removing it needs a tok/s A/B and its own ticket (proposed to the cop).noahgift commentedon Oct 14, 2026
P0 to P2 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, and its milestone is backlog.
noahgift commentedon Oct 14, 2026
merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5323 (CUDA quantized kernels: correct block layouts, cache keys and precision), split out of #5312 by theme.