Repository navigation
FP8 prefill GEMM overflows its FP16 output: D = true × 448/act_absmax exceeds 65,504 on qwen2.5-coder-7b (cosine 0.5668, lambda) #3728
Description
qwen2.5-coder-7b Q4_K_M on lambda CUDA: the F2 gate reports "diverges at position 1 (argmax 474 != 198, cosine 0.5668), validated via batched prefill" and the run falls back to CPU (0.69.1 row, #3710). This is a different defect from the stale-cache one: the PMAT-084 trace for this model shows exactly the intended 84 hits.
cublas_prefill_fp8_gemm quantizes activations with quant_scale = 448/act_absmax and has cuBLASLt write D = (w_absmax/448)·FP8(W)·FP8(X) in FP16. That makes D = true_result × 448/act_absmax, and act_absmax/448 is applied only afterwards, in the FP16→FP32 conversion. When a GEMM's input absmax is small relative to its output, D exceeds FP16's 65,504 and saturates.
The CPU per-op dumps (apr parity --per-op, the 0.69.0 binary, prompt "What is 2+2?") give the measurement. qwen2.5-coder-7b layer 1 QKV has input absmax 0.0602 and pre-bias output up to 11.54, so D peaks at 85,807 on rows 1–3, which are the positions where divergence starts. qwen2.5-1.5b (green) peaks at 6,822, and qwen2.5-coder-0.5b's FP8 GEMMs peak at 1,436.
done_when
- The FP8 prefill GEMM's intermediate can't overflow: D in BF16 (or FP32), or
act_absmax/448applied before the 16-bit store. Its range comes from the output dtype, not from the activation scale. - A GPU test with a small-absmax activation and outputs whose
true × 448/absmaxexceeds 65,504 (the 7b layer-1 QKV numbers). The FP8 result must match the f32 reference within the FP8 tolerance. The pre-fix FP16 D goes RED (inf/NaN or cosine collapse). - qwen2.5-coder-7b Q4_K_M: GPU backend with no fallback,
apr runcorrect,apr qagolden 3/3, on lambda AND on gx10 withBATCHED_PREFILL=1.
Activity
noahgift commentedon Sep 23, 2026
milestone → 0.73.0 per docs/roadmaps/milestone-triage-2026-09-23.md (#4024), operator approval 2026-09-23 (relayed by aprender-cf). Reason: performance/decode-prefill path: llama.cpp Parity (#3999)
docs(roadmap): quorum round 10 -- 18 applied rows annotated as verify…
ci(cuda-unit): run every cuda-gated aprender-serve lib test (1,337, n…
noahgift commentedon Oct 14, 2026
P0 to P2 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, and its milestone is 0.73.0.
noahgift commentedon Oct 14, 2026
merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5323 (CUDA quantized kernels: correct block layouts, cache keys and precision), split out of #5312 by theme.
noahgift commentedon Oct 14, 2026
Measured on
main@ 3064c77 (the BF16-D fix reached main through the v0.69.3 merge-back, #4338:gemm_fp8_e4m3_to_bf16_cached,fp8_gemm_range_tests_3728.rs). Built with--features cuda.GPU test (done_when 2).
an_fp8_gemm_whose_activation_gain_overflows_fp16_still_matches_f32passes on an RTX 4090.qwen2.5-coder-7b Q4_K_M (done_when 3).
apr run --gpu,apr run --format json,apr qa --json:apr runBATCHED_PREFILL=1 FP8_PREFILL=1On sm_121,
BATCHED_PREFILL=1alone now prefills in FP16 (#3785 default), so the FP8 GEMM never ran there. That leg was rerun withFP8_PREFILL=1so the FP8 path is actually exercised.Every done_when item now has a measurement. This comment does not close the issue; closing is the triage call.