Repository navigation
0.70.1 P0: Qwen3 on GB10 (sm_121) is correct but ~30x slow — ship sm_121 SASS, re-test batched prefill #4590
Closed as not planned
Description
Known issue in 0.70: on GB10 (sm_121), Qwen3 (the non-3.5 models) runs correctly but about 27–34× slower than Qwen3.5. At 8k think-on, Qwen3-1.7B took 967s, Qwen3-8B 2163s and Qwen3.5-9B 63–80s (#3715 pilots, apr 0.70.0-dev 8d021f6).
Two causes, both measured:
- There is no native sm_121 device code.
cuobjdump --list-elfshows no fatbin: the kernels ship as PTX (sm_70/75/90), and the driver JIT-compiles them for sm_121. - At cc>=120, prefill runs one token at a time by design (
select_prefill_path, sm12x default). On GB10, batched prefill gave wrong output on 2 of 4 prompts (FALSIFY-CPU-GPU-009 / PMAT-810).
Operator ruling (2026-09-28): 0.70 de-claims Qwen3 on GB10. apr warns, the release notes list it as a known issue, and only the Qwen3×GB10 cells leave #3715's required set.
0.70.1 P0:
- Ship sm_121 SASS (KTEST falsifier F-4).
- With native code in place, re-test batched prefill on GB10. The wrong answers may be the PTX-JIT wrong-arch class; if so, one fix solves both.
- Put Qwen3×GB10 back in the pv SHACL release-readiness shape: the tag is REFUSED unless every (Q4_K model in host inventory × {lambda,gx10} × {run,chat,serve,code} × consumer-context) cell has a Pass receipt — a missing cell is a violation, never a skip #3715 required set.
Refs #3715
Activity
noahgift commentedon Sep 28, 2026
Operator ruling 2026-09-28 (0.70.1 P0; the de-claim stands for 0.70). Root-cause hypotheses, tested in parallel, receipts only, ≤1 h each:
CUDA_FORCE_PTX_JIT=1on lambda → is batched prefill wrong? + build with sm_121 SASS → fixed?Owner: aprender-6b (+ L1 aprender-85 for KTEST F-4 sm_121 SASS). Deliverable: which hypothesis the evidence confirms, then fix + a planted falsifier.
noahgift commentedon Sep 28, 2026
Scope added (operator 12:50): GB10 decode is slower than the bandwidth ceiling. Receipt (aprender-85, gx10:~/kreg-4590/): Qwen3-1.7B Q4_K_M, cold 69.4s vs warm 68.5s → runtime JIT ruled out as the cause of slowness (H1 stays open for wrong answers only). Decode ~85 ms/step vs ~4 ms floor for 1.1G weights (~20x). H2–H5 continue (owner aprender-6b).
noahgift commentedon Sep 28, 2026
#4590 H1–H5 receipts (aprender-6b, 2026-09-28 10:30–11:20Z)
Verdict: none of H1–H5 is confirmed, because the defect they explain no longer reproduces. With the current binary, GB10 batched prefill is correct. It matches serial token-for-token and is bit-identical to the RTX 4090. The 30× slowdown is the serial-prefill policy itself: two stacked rules keep Qwen3 on GB10 serial.
Subject:
apr 0.70.0 (8d021f61e)on both hosts. It is the same binary as the #3715 pilots. gx10: GB10 cc 12.1, driver 590.48.01, CUDA 13.1. lambda: RTX 4090 cc 8.9, CUDA 13.0. Both dlopen libcublas 13.6.0.2. Greedy,--temperature 0, and only the generated text is compared (timing lines stripped).The premise (PMAT-810 / FALSIFY-CPU-GPU-009), re-measured on gx10
prefill_extract_first_token, 0.1 GB free VRAM on the shared 4090), the other 3 matchSpeed on the same needle (wall, includes load): lambda ~8k tokens serial 148.7 s vs batched 7.0 s. gx10 ~2k tokens 323.6 s vs 21.1 s; ~8k tokens >1200 s (timeout) vs 54.7 s. gx10 was shared with the :8091 shadow serve and the #3715 pilots, so these wall times are indicative, not a bench.
Hypotheses
sm_121.cubinon gx10. So "native sm_121 code" already runs, and batched is correct under it.CUDA_FORCE_PTX_JIT=1on lambda is not measurable:cublasCreate_v2fails withCUBLAS_STATUS_ALLOC_FAILED, the batched F2 probe errors, and the run falls back. The SASS arm (KTEST F-4) still matters for JIT cold-start cost (85's cold/warm split), not for correctness.apr tensors --stats --json(310 tensors: host dequant mean/std/min/max/nan) is byte-identical between gx10 (aarch64) and lambda (x86_64). The token count (24) and trace output are identical, and all GPU outputs are bit-identical across hosts. The one cross-host difference is the CPU path on one prompt (haiku,--no-gpu): x86 and aarch64 CPU disagree with each other, and both GPUs agree with each other. That is a CPU SIMD near-tie, not a GB10 GPU defect.compute-sanitizer2026.2.1,BATCHED_PREFILL=1([PREFILL-PATH] batched cc=121), qwen2.5-coder-0.5b: synccheck 0 errors, memcheck 0 errors, racecheck 0 hazards. Racecheck was filtered toregex=(scatter|rope|attn|attention|prefill|batch|rmsnorm|kv), which matchesbatched_*,kv_cache_scatter,flash_attention*andbatched_rope_neox_*in the gx10 cubin cache. Unfiltered racecheck is OOM-killed at 57.9G, and again under a 24G cap, on unified memory. cuBLAS GEMMs were not in racecheck scope. The output under every tool was correct ("4")./usr/local/cuda-13{,.0,.3}only). apr's device code is PTX JIT'd by the driver, independent of the toolkit, and lambda runs the same CUDA 13 cuBLAS 13.6 and is correct. There is no CUDA-13-only discrepancy to explain.[#3785] … FP16 (cc=121)), and manual decode graph ON (trueno#243 ✓ Manual graph). WithBATCHED_PREFILL=1, Qwen3 on lambda resolves to that same set: FP8 is off via the QK-norm rule, the prefill is FP16, and the graph is on. Its outputs are bit-identical to gx10's.Why Qwen3 × GB10 is slow: two serial rules, both still in force at
origin/mainselect_prefill_path(crates/aprender-serve/src/cuda/gpu_profile.rs:509):cc >= 120gives Serial ("sm12x default"). This is a correctness guard for FALSIFY-CPU-GPU-009, which does not reproduce above. That defect was measured 2026-06-16 on a v0.49-era build.e237ea751c(PMAT-886a),9750e9179band75b178a93fall landed since and touch this path. I have not bisected which one fixed it.GpuProfile::disable_fp8_for_qk_norm(gpu_profile.rs:1135): a QK-norm model (Qwen3) goes Batched to Serial on every GPU. Its own doc says the reason is VRAM: the FP16 HGEMM cache OOMs Qwen3-8B plus the CPU twin on 24 GB. FP16 batched is correct for Qwen3; only FP8 is broken (FP8 batched prefill fails CPU parity at the post-prompt decode step — catastrophic on per-head-QK-norm models (Qwen3-8B cos −0.10), marginal on Qwen2.5 (0.92) #3483). GB10's 128 GB unified memory does not have that constraint.Lifting rule 1 alone does not help Qwen3, because rule 2 flips it straight back to serial.
Proposed fix + planted falsifier (0.70.1)
BATCHED_PREFILL=0stays the opt-out.apr runwith no env vars on Qwen3-1.7B and qwen2.5-coder-1.5b, using the 4 PMAT-810 prompts plus the ~8k needle.[PREFILL-PATH] batched, matchBATCHED_PREFILL=0token-for-token, and finish the ~8k needle ≥10× faster than serial.cc >= 120 → Serialarm must turn it RED (on the path line and the speed ratio).select_prefill_path_table's(121, None) → Serialrows flip, and the planted revert fails them.Harness and receipts:
lambda:/mnt/nvme-raid0/tmp/6b-4590/andgx10:/mnt/nvme-raid0/tmp/6b-4590/. Files:repro.sh,cmp.sh,long.sh,h2.sh,h3d.sh, plus per-runsummary.txtand*.err.docs(#3715): release-note warning — qwen3moe on GB10 is not claimed f…
feat(cells): de-claim Qwen3 on GB10 for 0.70 -- only those cells leav…
feat(serve): warn [KNOWN-ISSUE #4590] for Qwen3 on sm_12x; release-no…
fix(#3715): de-claim Qwen3 on gx10 through DATA, not a pv change (#4590)
test(ont4c5): follow #4590's de-claim — B names qwen3-8b@lambda, not …
test(ont4c5): B follows the G2 de-claim of qwen2-1.5b-q4km@gx10
fix(prerun): SEC010 release_readiness selftest mkdirs via mk; PERF-01…
noahgift commentedon Oct 14, 2026
Folded into #5311 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it. Pushed branch:
6b/4590-gb10-batched-prefill,6b/4590-on-integ.Agent: aprender-clerk
noahgift commentedon Oct 14, 2026
Reopened: this ticket carries the operator's own words, and TRIAGE-100 keeps those open unless their owner agrees to the fold. The checklist row on #5311 is withdrawn; this ticket stays the record. Re-linked under #5327.
Agent: aprender-clerk
noahgift commentedon Oct 14, 2026
#4590 re-receipt on GB10: rebased fix, hand-off (aprender-wmainred, 2026-10-14)
Hand-off: the cop moved this row to unowned (#5327) at 14:43Z. The branch below is local and unpushed; this comment is the full state.
Subject. I took aprender-6b's
bca4987d21(2026-09-28) and cherry-picked it onto main8c3b19c8c1. The rebase drops main'sknown_issue_warningsm12x notice, because the fix makes it false. It leaves the capability-ladder de-claim of Qwen3 on gx10 untouched, because restoring that is a gate change and needs its own receipt. The result is commit6e0a3ffa9e, tree0f317227ac. I built it on gx10 (GB10 cc 12.1) with--features cudaand gotapr 0.71.3 (497fde05d). The baseline is mainapr 0.71.2 (98a370669). All runs are greedy with 16 tokens, and I compare only the generated text.cargo test --release -p aprender-serve --features cuda --lib -- gpu_profile: pass.BATCHED_PREFILL=0(brs), main default (main).Findings
The capital of France is Paris.and serial givesThe capital of France is **Paris**.The divergence is at the markdown-bold token, which looks like a near-tie. It is still a FALSIFY-CPU-GPU-009 miss on one of 16 prompts. 6b measured 4/4 on 0.70.0, so a re-bisect is needed before the default flips. Per CLAUDE.md rule 6, the next step is to vary the neighbouring prompts and compare with lambda's 4090 on the same commit.cudaMemGetInfoplus reclaimable), not raw free.PELICAN-42) on all three. All three ran serial, so this measures nothing about batched.Next for the owner: (a) the fix for finding 2, then re-run the needle and Qwen3-8B; (b) the p3 near-tie bisect. Patch: rebased
6e0a3ffa9eon top of8c3b19c8c1. 6b's original fix isbca4987d21, which is reachable from the repo.noahgift commentedon Oct 14, 2026
Folded into #5327 as a checklist row (TRIAGE-100, 2026-10-14). Under the cop's ruling, tickets with a P0 label or the operator's own words that are sub-rows of an existing master are folded, with the operator's text quoted on the master exactly as written. Priority and owner carry over on the row.
Owner la-0.71: if you object, say so here or in the inbox and this ticket is reopened. Work on it continues as before; a PR for it reopens it.
Agent: aprender-clerk