perf(serve): Qwen3.5 tensor-core prefill projections (RCA-SRV-001 fix 1) #4483

Activity

1. noahgift

   added this to the 0.71.0 milestone on Sep 26, 2026
2. noahgift

   added a parent issue on Oct 4, 2026

   • Qwen3.5 CUDA long-context DECODE is slow: the decode attention kernel runs 16 blocks — split-K needed for consumer use (RAH ~10 tok/s at 16k) #3725
3. noahgift

   added

   P1High priorityHigh priority

   size:MK̂ 3-4 hK̂ 3-4 h

   owner:aprender-wprodmodelsowning sessionowning session

   on Oct 4, 2026
4. noahgift commented on Oct 9, 2026

   

   noahgift

   on Oct 9, 2026

   MemberAuthor

   More actions

   Acceptance receipt on the RTX 4090 (sm_89), on a main binary: apr 0.70.3, 3824eba424 = main acd599e52a + docs, sha256 eb417ad9…8944. PRM-S1 v2 set a4910a65…, 200 items, Qwen3.5-4B-Q4_K_M, llama.cpp d1d3c3396, identical prompt ids on every leg.

   Receipt: docs/audits/prm-001/s1-v2/4483/lambda-4090/RECEIPT.md on branch wprod/4483-tc-prefill (commit 13f57be8b1). The branch also moves the gx10 receipt in from the stale branch. No PR yet: the PR cap is full.

   leg	prefill p50 ms	vs llama.cpp	wall p50 ms	vs llama.cpp
   f16 (default)	1030	1.68×	3801	1.74×
   f32 (same binary)	2473	4.04×	5265	2.40×
   llama.cpp	611	1	2189	1

   • f32/f16 prefill, paired per item: p50 2.42 (min 1.69, max 3.50). The f16 gap to llama.cpp grows with context, from 1.24× at 2k to 2.40× at 32k.
   • Quality falsifier, with the tolerance stated as "verdict identical on every item": met, 200/200 (39 Fail / 161 Pass on both legs). Output token count matches on 177/200, so greedy text is not bit-identical. A second f16 pass matches the first on 200/200, so the differences come from precision, not noise.
   • Engaged: the f16 log has fp16 prefill weights prewarmed: 6807 MiB; the f32 log has no prewarm line.
   • apr prefill time comes from the server's batched prefill log lines, because main has no SRV-TIM-001 timings on the Qwen3.5 route (APR-OBS-001: dogfood observability (OBS-00 + row checklist) #4487). The pairing rule and per-row TSVs are in the receipt.

   Not covered by this fix: at ctx 36864 on 24 GB, the 27B fp16 cache does not fit (46445 MiB needed, 7223 MiB free). Its prefill stays on f32, with no tensor core. The 9B fits (13196 MiB). A per-call fp16 dequant leg would cover the 27B; that is a candidate follow-up, not filed.

   Agent: aprender-wprodmodels
5. noahgift commented on Oct 14, 2026

   

   noahgift

   on Oct 14, 2026

   MemberAuthor

   More actions

   merged into #5305 (master ticket for epic #3598; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
6. noahgift

   removed a parent issue on Oct 14, 2026

   • Qwen3.5 CUDA long-context DECODE is slow: the decode attention kernel runs 16 blocks — split-K needed for consumer use (RAH ~10 tok/s at 16k) #3725
7. noahgift

   closed this as not plannedon Oct 14, 2026
8. noahgift

   reopened this on Oct 23, 2026
9. noahgift

   added a parent issue on Oct 23, 2026

   • Qwen3.5 speed: prefill and decode parity with llama.cpp, and the harness that measures it #5305
10. noahgift

    modified the milestones: 0.71.0, 0.73.0 on Oct 27, 2026