perf(serve): CUDA-graph decode step + alloc-free token loop (RCA-SRV-001 fix 4) #4486

Activity

1. noahgift

   added this to the 0.71.0 milestone on Sep 26, 2026
2. noahgift commented on Sep 26, 2026

   

   noahgift

   on Sep 26, 2026

   MemberAuthor

   More actions

   #4486 measured: split-indirect decode graph (commit 49fb4bb)

   Binaries. new = apr 0.70.0 (49fb4bb4cf), control = car apr 0.70.0 (1ac56f258). llama.cpp 0.5.0-dev build 11201 (2145525a4). All on the RTX 4090. Every run held flock /tmp/apr-gpu.lock around the binary only.

   Set. The PRM-S1 v2 replay set (review-replay-v1) is not in the tree yet; PRM-001 v3 lists it as todo. So this is a proxy: one 2241-token prompt (mid.txt), Qwen3.5-4B-Q4_K_M, greedy, n=1025, 3 reps per mode.

   mode	"Completed in" r1 / r2 / r3 (s)	decode ms/tok (r1, r3)
   new, eager (QWEN35_CUDA_GRAPH=0)	16.08 / 34.74* / 16.29	9.81, 9.95
   new, graph	15.97 / 33.89* / 15.93	9.71, 9.69
   car, graph (control)	71.59 / 33.09* / 18.62	unstable (r1 is 4.5× slower)
   llama.cpp (eval time, 2 reps)	—	4.77, 4.81

   * r2 was contended across every mode, so it is excluded.
   ms/tok = (Completed_1025 − 6.0 s)/1024. The 6.0 s is the clean n=1 wall (prefill ≈ 0.74 s + load); the eager n=1 runs were noisy (21–28 s).

   Falsifier (greedy text identical). PASS. All 9 apr outputs have the same md5 b73c617f (532 words), across eager, new graph and car graph.

   Verdict.

   • The new graph is correct: it is bit-identical to the direct path in the unit test gdn_decode_attention_split_indirect_is_bit_identical_to_direct and gives identical e2e text.
   • It is stable (634 kernels captured at pos 2241), unlike the car graph.
   • It is only about 1–2% faster than eager.
   • Against llama.cpp, decode is still about 2.0× slower (9.7 vs 4.8 ms/tok).
   • So the graph is not the bottleneck: launch overhead is not where the time goes. The next lever is per-kernel time (nsys on the eager path), not more graphing.
   • Branch perf/4486-qwen35-decode-graph is ready for 59 to fold into B1r.
3. noahgift

   added a parent issue on Oct 4, 2026

   • Qwen3.5 CUDA long-context DECODE is slow: the decode attention kernel runs 16 blocks — split-K needed for consumer use (RAH ~10 tok/s at 16k) #3725
4. noahgift

   removed a parent issue on Oct 4, 2026

   • Qwen3.5 CUDA long-context DECODE is slow: the decode attention kernel runs 16 blocks — split-K needed for consumer use (RAH ~10 tok/s at 16k) #3725
5. noahgift

   added a parent issue on Oct 4, 2026

   • apr run --gpu is 2-3x SLOWER than --no-gpu on the dense path: the GPU result fails correctness (cosine 0.4153) and falls back to CPU after the user has paid for it #3602
6. noahgift

   added

   P0Critical priorityCritical priority

   size:MK̂ 3-4 hK̂ 3-4 h

   owner:la-0.71Owned by the 0.71 look-ahead slotOwned by the 0.71 look-ahead slot

   on Oct 4, 2026
7. noahgift

   added

   P1High priorityHigh priority

   and removed

   P0Critical priorityCritical priority

   on Oct 14, 2026
8. noahgift commented on Oct 14, 2026

   

   noahgift

   on Oct 14, 2026

   MemberAuthor

   More actions

   P0 to P1 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, so it stays on the 0.71.0 milestone at P1.
9. noahgift commented on Oct 14, 2026

   

   noahgift

   on Oct 14, 2026

   MemberAuthor

   More actions

   merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
10. noahgift

    removed a parent issue on Oct 14, 2026

    • apr run --gpu is 2-3x SLOWER than --no-gpu on the dense path: the GPU result fails correctness (cosine 0.4153) and falls back to CPU after the user has paid for it #3602
11. noahgift

    closed this as not plannedon Oct 14, 2026
12. noahgift

    reopened this on Oct 23, 2026
13. noahgift

    added a parent issue on Oct 23, 2026

    • apr serve on CUDA: batch concurrent Qwen3.5 requests correctly #5312
14. noahgift

    modified the milestones: 0.71.0, 0.73.0, 0.74.0 on Oct 27, 2026
15. noahgift commented on Oct 31, 2026

    

    noahgift

    on Oct 31, 2026

    MemberAuthor

    More actions

    Folded into #5312 as a checkbox by the 0.73–0.76 triage (2026-10-31): open tickets capped at 100, 0.74.0 capped by releases left. Not abandoned: reopen when #5312 schedules it. Icebox policy: #6521.
16. noahgift

    closed this as not plannedon Oct 31, 2026