Repository navigation
apr run --gpu is 2-3x SLOWER than --no-gpu on the dense path: the GPU result fails correctness (cosine 0.4153) and falls back to CPU after the user has paid for it #3602
Description
On the dense path, apr run --gpu is 2–3× SLOWER than --no-gpu, because the GPU result fails its own correctness check and the run falls back to CPU after paying for the GPU attempt. The user pays twice and the warning is on stderr.
Measured by the instrument lane (#3598 row 1) on lambda 4090, GPU idle at start, --max-tokens 1, prompt Hi:
| model | GPU attempt | --no-gpu |
|---|---|---|
| qwen2.5-coder-0.5b | 32.0 s · 35.8 s | 17.5 s |
| qwen2.5-coder-1.5b | 10.4 s | 3.7 s |
The dense GPU validation fails on correctness — cosine 0.4153 at position 1, via batched prefill — and the run then falls back through wgpu to CPU. So asking for the GPU costs 15–17 s for a rejected attempt and then the full CPU run on top.
Why this is a 0.69 row and not a performance row
0.69's claim is verbs that are true. Two things here are untrue in the sense that matters:
--gpumakes the command slower, and nothing in the product says so. A user who reads "use--gpufor speed" measures the opposite and has no way to learn why. Time is the only signal they get and it points the wrong way.- A correctness failure is being discharged by a fallback.
cosine 0.4153at position 1 is not a near-miss; the dense GPU path is producing wrong logits and the fallback converts that into a latency complaint. The fallback is the right behaviour — better slow than wrong — but it currently hides the defect rather than reporting it, which is the fail-open shape this release is supposed to eliminate. Compare apr chat hard-skips CUDA for every qwen35 model and prints a banner citing #3090 as unimplemented — #3090 shipped a day earlier, and the corrected replacement has zero callers #3595, where a true capability was advertised as absent; this is the mirror, a real defect presented as slowness.
Likely the batched-prefill FP8 class (#3483), on the evidence that the failure arrives via batched prefill. Not asserted — nobody has bisected it, and the instrument lane deliberately did not touch it.
done_when
- The correctness failure is surfaced as a correctness failure, not absorbed into wall clock:
apr runreports that the GPU result was rejected, the measured cosine, and the position — on a stream the user reads, once, not buried beside a fallback notice on stderr. - The cost of a rejected GPU attempt is attributable — it shows up in the EPIC E2 0.73: Verbs Are Fast #3598 timing fields (a
validate_ms/rejected_msstage), so "GPU made it slower" is readable from the receipt instead of inferred from a stopwatch. - Root cause named for the dense path — whether
cosine 0.4153is FP8 batched prefill fails CPU parity at the post-prompt decode step — catastrophic on per-head-QK-norm models (Qwen3-8B cos −0.10), marginal on Qwen2.5 (0.92) #3483's FP8 batched-prefill class or something else. One bisect, stated either way. - A decision, recorded, on whether
--gpushould fail rather than fall back when the GPU result is rejected on correctness. Silent-and-slow and loud-and-failed are both defensible; silent-and-slow-while-a-correctness-check-failed is not. Whichever is chosen, the refusal or the fallback names the reason. - Falsifier: a planted correctness failure must produce the same user-visible report, and a healthy GPU run must not.
Scope
Dense path only — OwnedQuantizedModel / forward_gpu_resident. The hybrid (qwen35) path is a separate finding: its pre-generation cost is f2_validate_qwen35 at 67 % of a 14 s TTFT (#3596), and its decode is ~370 tok/s against llama.cpp's 179–189 on the same box. These two must not be conflated — one is a guard that is too expensive, the other is a GPU result that is wrong.
Found while measuring something else, which is why nothing was looking for it: the fallback works, so no test fails.
Activity
noahgift commentedon Sep 20, 2026
Operator ruling 2026-09-20: stays P0 on 0.69, and the 0.69 fix is the reporting, not the kernel
0.69 scope — the fallback becomes loud and receipted:
backend: cpu (gpu rejected: cosine 0.4153@1)in--json— the rejection, its reason and its position are fields, not prose.0.70 — the batched-prefill correctness bug underneath, unless the bisect turns out to be quick. Carries
removed_by: v0.70on the refusal either way, so it is a named refusal in the 0.69 notes rather than an omission.That split is the release doctrine applied literally: 0.69 is verbs that are true; 0.70 is verbs that are fast. A GPU path that produces
cosine 0.4153and quietly hands the work to the CPU is not slow — it is lying by omission, and the lie is fixable this train even though the kernel defect is not.Do not conflate this with #3604. Two different pre-generation costs on two different paths:
qwen35f2_validate_qwen35, 67 % of a 14 s TTFTOwnedQuantizedModelOne is an expensive right answer; the other is a wrong answer that costs the user twice. Fixing the receipt cache does nothing for this issue, and fixing this does nothing for TTFT on the hybrid path.
Refs #3604, #3596, #3598, #3483.
noahgift commentedon Sep 20, 2026
done_when 3 — the bisect, stated either way
It is FP8 prefill. Not the FP8 weight cache, and not batched/HGEMM prefill. One knob clears it.
Lambda 4090, GPU idle at 563 MiB recorded at start,
qwen2.5-coder-0.5b-instruct-q4_k_m, promptHi, CUDA build fromPMAT-3602-gpu-rejection-visible:cosine 0.4153REALIZR_NO_FP8_CACHE=1cosine 0.4153HGEMM_PREFILL=0cosine 0.4153FP8_PREFILL=0So the FP8 weight cache is not implicated — disabling it changes nothing — and neither is the batched/HGEMM prefill path, which is the hypothesis the stderr line's "validated via batched prefill" invites. The divergence survives
HGEMM_PREFILL=0and dies withFP8_PREFILL=0.This confirms #3483's FP8 class and narrows it: the defect is in the FP8 prefill arithmetic, and it is reachable on the non-HGEMM path too.
And with FP8 prefill off, the GPU is used and the command gets faster
used_gpu--gpubaseline--gpu FP8_PREFILL=0--no-gpuThe issue's headline reproduces —
--gpuat 38.3 s against--no-gpuat 20.8 s — and inverts under one flag: 18.8 s with the GPU actually running, now faster than CPU rather than ~2× slower.This is a measurement, not a proposed fix.
FP8_PREFILL=0is a bisect knob; whether the shipped default should change is a separate call that belongs with whoever owns #3483, because the FP8 prefill path presumably exists for a reason on models where it is correct. What the bisect settles is where to look.done_when 2 is already satisfied by #3606, not owed here
validate_msin #3606 instrumentsvalidate_gpu_first_token— which is exactly where a rejected GPU attempt's cost lives on the dense path. Measured there at 15.1 s on this model. So "GPU made it slower" is already readable from the receipt rather than inferred from a stopwatch, and this row does not need to re-add the field.Remaining here: item 1 (surface the rejection where a user reads it, and in
--json), item 4 (the recorded fail-vs-fall-back decision), item 5 (the falsifier both ways).Off the GPU.
noahgift commentedon Sep 20, 2026
Correction to the comment above — two of its three rows were inert
I read the knobs' names and not their bodies. Before anything else, what was wrong with my own bisect:
HGEMM_PREFILL=0did not clear it ⇒ "not the batched/HGEMM path"warmup_fp16_cache_if_enabledonly — and that function already returns early whengpu_profile.fp8_prefillis true (crates/aprender-serve/src/gguf/cuda/mod.rs:346)REALIZR_NO_FP8_CACHE=1did not clear it ⇒ "not the FP8 weight cache"…/gguf/cuda/mod.rs:366). FP8 GEMMs still run against a lazily-populated cacheFP8_PREFILL=0cleared it ⇒ "FP8 prefill specifically"detect_fp8_decodefalls through tofp8_prefillwheneverFP8_DECODEis unset (crates/aprender-serve/src/cuda/gpu_profile.rs)A gate short-circuited by the very setting under test, a warmup switch read as a cache switch, and a fallthrough coupling two flags. The generative rule: a bisect over env knobs proves nothing until each knob's body is read for what it gates under the baseline's other settings.
The 2×2 that does separate them
Every cell names both flags explicitly, so no cell depends on the fallthrough.
apr 0.68.2 (e6f77c98c)— the binary built from this branch's HEAD — lambda RTX 4090, 631 MiB idle at start, same model and prompt,--max-tokens 1 --gpu --json:FP8_PREFILLFP8_DECODEused_gpuposition 1, cosine 0.4153position 1, cosine 0.4153The divergence tracks
FP8_PREFILLexactly and is completely independent ofFP8_DECODE. Row 2 holds FP8 decode off and the divergence is unchanged to four decimals; row 3 holds FP8 decode on with prefill off and it is clean. Row 3 is the discriminating cell, and it is the one my first matrix never ran.So the original headline was right and is now established rather than asserted: it is FP8 prefill. This confirms #3483's class.
The mechanism that should have caught this already declined to fire
f2_should_retry_without_fp8(crates/aprender-serve/src/infer/inference_result.rs:1078) exists for exactly this: on an FP8 miss it re-measures on the FP16 prefill path, andf2_remeasure_without_fp8persistsfp8_prefill = falsefor the run when FP16 passes. That is a precision fallback instead of a backend fallback — the right design, already built.It did not fire here, and its
note: … re-measuring on the FP16 prefill pathline is absent from all of stderr. One conjunct is sufficient to explain that:The band's stated purpose is that "anything below the catastrophic band is never retried", justified by Qwen3-8B under FP8 at −0.0972. But it lumps 0.4153 in with −0.0972, and rows 3–4 show 0.4153 is a case where FP16 is clean and the retry would have succeeded — keeping the model on the GPU at half the wall time (16.9 s vs 34.3 s) instead of pushing it to CPU.
Recommendation, for whoever owns #3483: drop the
min_cosine_real/first_bad_cosineconjuncts and retry whenever!accepted && fp8_prefill_on(and the path matches). The fear the band guards against — "hiding a real divergence behind a second measurement" — does not apply, because the retry re-measures against the sameF2_GATE_COSINE_MINfloor on a different configuration that then persists. A failed retry still rejects; the floors never move. All the band buys is one skipped probe on hopeless cases, and it costs a 2× regression on this one.I am not making that change here — it is #3483's call and its owner should have the table. Posting it there.
Still owed on this issue
Item 3 is now answered. Item 2 remains satisfied by #3606's
validate_ms(15.1 s on this model). Items 1, 4 and 5 are mine and in progress.Off the GPU.
chore(roadmap): mint PMAT-3602 as a work item so the AD-04 quorum can…
fix(run): quorum round 1 found the --json surface skipped and a branc…
fix(run): quorum round 2 — a --json --benchmark leak, a comment claim…
refactor(run): extract reconcile_and_emit — run() hit cognitive 27 ag…
PMAT-3577: dispatch_runtime_commands back to main's complexity — #3604'…
noahgift commentedon Sep 21, 2026
slipped_from: 0.69.0
§4.1 freeze (06x-release-schedule.md §1: "scope that is not on the platform rides the next train"). Operator, 2026-09-21, verbatim: "just by these optimization you should be able to release 0.69 today, ensure this happens". The 0.69 platform is #3669 plus the bump.
Why this row rides 0.70: item 1 lands in #3669 (#3638: a CPU fallback is reported as a refusal). Item 2 is owed to #3606, which is not on the platform.
— aprender-04 (cop)
noahgift commentedon Sep 21, 2026
Root causes (aprender-37, measured 2026-09-21), two DIFFERENT defects in the FP8 BATCHED prefill GEMM. "lambda-only" was a path artifact: gx10 defaults to serial prefill (
select_prefill_path: cc≥120 → Serial; cc 89 → Batched). Forcing batched on gx10 reproduces the same failure (cos 0.4159).ptr=0x71aec6600000 count=238336) and reuse layer 0's activation.Knobs:
FP8_PREFILL=0orBATCHED_PREFILL=0turn both green.REALIZR_NO_FP8_CACHE=1is NOT an off switch, andAPR_FP8_PAD_ZERO=1doesn't help (so cublas_prefill_fp8_gemm reads 16 rows of FP8 scratch where 1 was written: the padded tail is the previous GEMM's activations #2765's padded tail isn't the cause).Ruling (cop): fix both for 0.69.1: the cache key must be un-aliasable across layers, and D goes in BF16/FP32 or is scaled before the FP16 store. Each gets a red-turning test and is proven on lambda AND gx10 with batched forced. No default-off workaround.
38 remaining items
noahgift commentedon Oct 14, 2026
P0 to P1 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, so it stays on the 0.71.0 milestone at P1. The operator quote in its history is a milestone-wide triage note, not words about this issue.
noahgift commentedon Oct 14, 2026
merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5323 (CUDA quantized kernels: correct block layouts, cache keys and precision), split out of #5312 by theme.