Repository navigation
FP8 batched prefill fails CPU parity at the post-prompt decode step — catastrophic on per-head-QK-norm models (Qwen3-8B cos −0.10), marginal on Qwen2.5 (0.92) #3483
Description
Found by the #3413 C guard (F2 now validates the batched prefill path generation uses), branch PMAT-3477-qwen3-cuda-qk-norm, RTX 4090 sm_89, 2026-09-18.
Measured (apr run --gpu, CPU-vs-GPU per-position probe incl. one decode step after the prompt):
| model | FP8 batched prefill | FP16 (HGEMM) batched prefill | serial prefill |
|---|---|---|---|
| Qwen3-8B-Q4_K_M | cos −0.0972 at the decode step, garbage (graph on or off) | "Paris" | "Paris" |
| Qwen3-1.7B-Q4_K_M | 2/4 prompts reject, cos 0.8718 / 0.8759 (one with the SAME argmax) | 4/4 pass | pass |
| qwen2.5-1.5b (no QK-norm) | pos 27 cos 0.9187 same argmax on one run, ≥0.95 on another | pass | pass |
Prefill positions 0..S-1 pass on every path; only the decode step that attends to the FP8-produced K/V misses. So the defect is in the FP8 GEMM output feeding the KV cache (per-tensor absmax scaling on K/V projections? the QK-norm then amplifies it), not in attention.
Shipped in 0.68.2 as mitigation (#3477): GpuProfile::disable_fp8_for_qk_norm turns FP8 prefill+decode off for has_qk_norm models (FP8_PREFILL=1 forces it back for A/B), and the F2 guard re-measures a non-catastrophic FP8 miss on the FP16 prefill before falling back (PMAT-798 pattern), printing both notes. Floors unchanged.
This ticket: root-cause the FP8 K/V precision loss, fix or scope FP8 per projection, and delete the class gate once apr qa Golden Output GPU passes on Qwen3-8B under FP8. Falsifier: the table above re-measured with FP8 on.
Activity
PMAT-3477: QK-norm models take the serial prefill with FP8 off — the …
noahgift commentedon Sep 20, 2026
Reopening: this was closed COMPLETED on 2026-09-18, and the mechanism model in the body is contradicted by a measurement on today's HEAD. The mitigation that shipped does not cover the model below, and the retry that should have caught it is gated out of firing.
Measured with
apr 0.68.2 (e6f77c98c), RTX 4090 sm_89, 631 MiB idle at start,qwen2.5-coder-0.5b-instruct-q4_k_m,--gpu --max-tokens 1.1. The divergence is at a PREFILL position, not the decode step
The body states: "Prefill positions 0..S-1 pass on every path; only the decode step that attends to the FP8-produced K/V misses."
HiWhat is 2+2?Write one sentence about the ocean.Explain in detail how a binary search tree maintains its invariant during insertion, and why the worst case is linear.For the last prompt the probe is far longer than two tokens (
gpu_probetakes the last 64 prompt tokens), so position 1 cannot be the post-prompt decode step — it is the second position of the prefill. The quoted claim is false for this model.2. The cosine is identical to four decimals across four prompts that differ from the first word
That is not the signature of a precision defect; a precision defect varies with content. Two hypotheses remain consistent with it, and they are worth separating because they imply different fixes:
Either way it is not the "per-tensor absmax scaling on K/V projections amplified by QK-norm" story in the body.
3. It is FP8 prefill, established by a 2×2 rather than a single knob
Both flags named explicitly in every cell, because
detect_fp8_decodefalls through tofp8_prefillwhenFP8_DECODEis unset and a single-knob bisect cannot separate them:FP8_PREFILLFP8_DECODEused_gpuThe divergence tracks
FP8_PREFILLexactly and is independent ofFP8_DECODE.4. Why nothing caught it
GpuProfile::disable_fp8_for_qk_normkeys onhas_qk_norm. Qwen2.5-Coder-0.5B has no per-head QK-norm, so FP8 stays on for it.f2_should_retry_without_fp8requiresmin_cosine_real >= F2_CATASTROPHIC_COSINE(0.90). At 0.4153 it never fires, and itsre-measuring on the FP16 prefill pathnote is absent from stderr. Rows 3–4 above show the retry would have succeeded — FP16 is clean — so the band skips a case where the precision fallback works and costs a 2× wall-clock regression (34.3 s → 16.9 s) by pushing the model to CPU instead.Recommendation
Drop the
min_cosine_real/first_bad_cosineconjuncts fromf2_should_retry_without_fp8and retry whenever!accepted && fp8_prefill_onon a matching path. The band guards against "hiding a real divergence behind a second measurement", but the retry re-measures against the sameF2_GATE_COSINE_MINfloor in a different configuration that then persists — a failed retry still rejects and the floors never move. All the band buys is one skipped probe on hopeless cases.I am not making that change from here; this is your call and your issue. Context: #3602, where this surfaced as "
--gpuis slower than--no-gpu" — which it is, and this is why.noahgift commentedon Sep 20, 2026
Milestone: this should move to 0.70. No PR of mine will close it.
I reopened this with the measurement that contradicts its mechanism model, but the fix is FP8 prefill arithmetic and that is not work I am carrying. #3638 (#3602) only makes the consequence visible — it stops
apr run --gpureporting a CPU fallback as success — and explicitly does not touchf2_should_retry_without_fp8's band, which is the change recommended here and belongs to whoever owns this issue.So at freeze: move to 0.70 rather than slip 0.69 waiting on it.
The one thing worth not losing in the move:
FP8_PREFILL=0is a working mitigation today on this model class, with the GPU still used and the run faster than CPU (16.9 s vs 20.8 s), while the shipped default is 34.3 s on CPU after a rejected GPU attempt. Whether that should change the default for non-QK-norm Qwen2.5 before the arithmetic is fixed is a smaller, separable decision than the root cause, and it could ship in 0.69 on its own if someone wants it.noahgift commentedon Sep 21, 2026
Milestone → 0.70.0 by the cop (05:30Z), on the owner's own note above: no PR closes this for 0.69; #3638 makes the consequence visible (the refusal is recorded instead of a silent CPU fallback) and the FP8-prefill arithmetic fix is 0.70 work. The separable decision to change the shipped FP8-prefill default for non-QK-norm Qwen2.5 can still ride 0.69 if someone takes it before the 09-22 18:00Z freeze.
noahgift commentedon Sep 23, 2026
milestone → 0.71.0 per docs/roadmaps/milestone-triage-2026-09-23.md (#4024), operator approval 2026-09-23 (relayed by aprender-cf). Reason: FP8 prefill fails CPU parity: GPU correctness underneath the matrix (#3994)
docs(roadmap): correct the triage after the quorum -- 3 moves reverte…
12 remaining items
noahgift commentedon Oct 12, 2026
Re-measure of the class-gate deletion on current main (la-71 look-ahead, branch
la-71/3483-fp8-gate-delete@0ef14fe7c5= cherry-pick -x of 4e893a3 onto main 6bd89b0). Binaryapr 0.71.1 (0ef14fe7c), built with--features cuda, RTX 4090 sm_89, no env overrides.Mechanism delta (same model, Qwen3-8B Q4_K_M):
[#3413] architecture 'qwen3' uses per-head QK-norm: FP8 prefill off,prefill GEMM precision: FP16,[PREFILL-PATH] serialprefill GEMM precision: FP8 (E4M3),[PREFILL-PATH] batched cc=89, FP8 weight cache 7217.5 MBapr run --gpu --max-tokens 32, 4 prompts × 4 models: 16/16 rc=0, all coherentapr qaon the branch binary: Qwen3-8B and Qwen3-1.7B both ALL GATES PASSED — Golden Output 3/3 (GPU leg on device 0), GPU State Isolation 3/3, PTX Parity 6/6, Perf Regression none. Throughput 101.6 / 316.9 tok/s.Reading: with the gate gone, the QK-norm class runs FP8 batched prefill and stays coherent; the per-load F2 floor still catches the borderline case (Qwen3-8B sat at 0.9499 on one prompt, one ten-thousandth under the floor), so the guard, not the class gate, is what holds parity. That margin is thin on 8B and worth a note before the gate deletion lands. Nothing merged or closed by this lane.
noahgift commentedon Oct 14, 2026
merged into #5317 (master ticket for epic #4381; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.