Repository navigation
apr serve on CUDA: batch concurrent Qwen3.5 requests correctly #5312
Description
Scope: Concurrent serving on CUDA: a batch scheduler over a BatchEngine trait, a multi-slot greedy step, batched GEMV that is row-equal to the single-row kernels, CUDA-graph decode, and the fix for garbage output when m>1.
Why it matters: apr serve runs Qwen3.5 one request at a time, so the serving-shape (V3) bar cannot pass, and the old batched path produced garbage.
Done when: At c=4/8/16 on Qwen3.5-4B, apr serve forms batches (m_formed>0), every reply equals the single-request reply token for token, and aggregate tok/s rises with c, shown by a perf041 witness on lambda.
Epic: #3994. Folded under triage rule 7b (2026-10-14): each row below was its own issue, closed as not planned and kept here as a sub-ticket. Reopen an issue when its work starts or its PR opens.
Sub-tickets
- serve: Qwen3.5 on CUDA never batches concurrent requests (m_formed=0 at c=4/8/16) #5279 serve: Qwen3.5 on CUDA never batches concurrent requests (m_formed=0 at c=4/8/16) (milestone none)
- accept: the outcome in the title, with a receipt
- serve: CUDA batch scheduler over a BatchEngine trait (#5279 P0) #5280 serve: CUDA batch scheduler over a BatchEngine trait (serve: Qwen3.5 on CUDA never batches concurrent requests (m_formed=0 at c=4/8/16) #5279 P0) (milestone none)
- accept: Done when:
- serve: Qwen3.5 CUDA multi-slot greedy step, looped (#5279 P1) #5289 serve: Qwen3.5 CUDA multi-slot greedy step, looped (serve: Qwen3.5 on CUDA never batches concurrent requests (m_formed=0 at c=4/8/16) #5279 P1) (milestone none)
- accept: Done when.
- serve: Qwen3.5 CUDA batch engine and serve wiring (#5279 P2) #5293 serve: Qwen3.5 CUDA batch engine and serve wiring (serve: Qwen3.5 on CUDA never batches concurrent requests (m_formed=0 at c=4/8/16) #5279 P2) (milestone none)
- accept: Done when.
- realizar: batched GEMV row-equal to the single-row kernels for Qwen3.5 (#5279 P3) #5303 realizar: batched GEMV row-equal to the single-row kernels for Qwen3.5 (serve: Qwen3.5 on CUDA never batches concurrent requests (m_formed=0 at c=4/8/16) #5279 P3) (milestone none)
- accept: Done when.
- V3 bar vs code gap: apr serve runs Qwen3.5 one request at a time, so S3 cannot pass #5046 V3 bar vs code gap: apr serve runs Qwen3.5 one request at a time, so S3 cannot pass (milestone 0.71.1)
- accept: the outcome in the title, with a receipt
- V3 serving-shape parity: lambda perf041 witness for Qwen3.5-4B at c in {1,4,8,16} #4971 V3 serving-shape parity: lambda perf041 witness for Qwen3.5-4B at c in {1,4,8,16} (milestone 0.71.1); branch
la-711a/4971-v3a-margin-m3- accept: Done when: one lambda perf041 witness for Qwen3.5-4B (sha256 recorded) measures c in {1,4,8,16} with m_formed == c, divergence_at >= declared_min vs m=1 or top2
- P0: batched CUDA decode emits garbage for every m>1 — PERF-001's 3.32x aggregate is throughput of garbage tokens #2753 P0: batched CUDA decode emits garbage for every m>1 — PERF-001's 3.32x aggregate is throughput of garbage tokens (milestone 0.73.0)
- accept: the outcome in the title, with a receipt
- perf(serve): CUDA-graph decode step + alloc-free token loop (RCA-SRV-001 fix 4) #4486 perf(serve): CUDA-graph decode step + alloc-free token loop (RCA-SRV-001 fix 4) (milestone 0.71.0); branch
perf/4486-qwen35-decode-graph- accept: the outcome in the title, with a receipt
Activity
[-]MASTER E3 0.71: folded sub-tickets (epic #3994)[/-][+]apr serve on CUDA: batch concurrent Qwen3.5 requests correctly[/+]noahgift commentedon Oct 14, 2026
Merged into #5329 (apr serve request path: BPE tokenization, graceful drain, no 500 on oversize prompts): this ticket and its sub-ticket rows are now listed there under "Merged in". Closed as not planned under triage rule 7b; reopen it when its own work starts or a PR opens.
noahgift commentedon Oct 14, 2026
0.72 floor row, closed in error
fix(serve): re-cite the four fusion call sites, format the include!d …
noahgift commentedon Oct 15, 2026
Acceptance check against main @ 8f601ca. #5364 (merge 6f1c82d) is an ancestor of it. This is evidence only; the issue stays open.
Done when: at c=4/8/16 on Qwen3.5-4B,
apr serveforms batches (m_formed>0), every reply equals the single-request reply token for token, and aggregate tok/s rises with c, shown by a perf041 witness on the release host.BatchEnginetraitapi/cuda_batch_scheduler.rs.cuda_batch_scheduler_engine_tests.rs::concurrent_requests_form_one_batch_of_c_over_the_traitruns host-only, in CIqwen35_batch_engine.rs,forward_qwen35_cuda_batch*.rs,serve/server.rstests_batched_mwv.rs. The PTX of each single-row kernel was compared at base and head (seat-1 review)forward_qwen35_cuda_batch_graph.rs.the_captured_batched_step_is_bitwise_the_eager_onefour_concurrent_requests_batch_and_match_their_solo_turns,batched_greedy_streams_equal_each_slot_decoded_aloneand siblings. They need the 0.8B file and a CUDA device and printSKIPotherwise, so CI's lib run exercises none of them. The commit bodies cite GB10 runs; no review re-ran them on a GPUevidence/perf041/directory. The files there (witness.json2026-10-07,marker.json,witness-v3.0-rule.json) are all qwen2.5-coder-1.5b and predate #5364. The device tests usemax_batch: 4on 0.8BMissing: one perf041 witness on the release host for Qwen3.5-4B (model sha256 recorded) at c in {1,4,8,16}, from an
aprbuilt at a commit that contains 6f1c82d, showing m_formed, per-request token equality against c=1, and aggregate tok/s. The code items are on main; the outcome in the title is not yet measured on the release host.noahgift commentedon Oct 15, 2026
Qwen3.5-4B measurement (the remaining acceptance item): measured, ADMISSIBLE ×3.
Setup:
apr serve run Qwen3.5-4B-Q4_K_M.gguf --gpu-layers allon an RTX 4090,CUDA_BATCH_WINDOW_MS=200, thenapr test llm parity --ladder 1,4,8,16andapr test llm shape-check. The binary isapr 0.71.3 (8f601caff2), built with--features cuda, sha25626073455f173…. The model's sha256 is00fe7986ff5f….The server log shows the CUDA backend for the 4090 (qwen35 hybrid forward),
CONTINUOUS BATCHING: max_batch=16, window=200ms, andBatch m=N donelines.The witnesses are committed on branch
PMAT-5312-qwen35-4b-witness(7831932) aswitness-qwen35-4b-batched-run{1,2,3}.jsonbeside the existing perf041 4090 witnesses. No PR is open yet. I'm leaving the issue open.Disclosure from review: in run 2's c=8 band, two slots (i=6 and i=7) have bit-identical
decode_tok_svalues (27.574169291500333). The per-slot rate is 127 ÷ (last − first client-side token arrival) at nanosecond resolution, and every slot comes from its own HTTP stream; I found no code path that copies a rate between slots. All 8 slots in that band agree to within 1.8×10⁻⁵, which fits lockstep batched decode, but an exact match is unlikely, and I have no proven cause. It is the only duplicate across the 3 runs × 4 bands. Nothing reported here uses slots 6 and 7 on their own (the band value is the median), and the summed figures above include them as measured.evidence(perf041): Qwen3.5-4B run4 re-run + run2 duplicate-slot note …
noahgift commentedon Oct 15, 2026
Folded into #5329 by the cop batch at 10:30Z (triage cap 100; clerk proposal 1015 row). Tracked as a checklist row there; reopen if it needs its own owner.
noahgift commentedon Oct 15, 2026
Reopened by the cop: this is a protected 0.72 floor row (handoff/protected-072-floor.txt), so it must not be folded. The 10:30Z fold was the cop's error. Its master link stays as a cross-ref.
noahgift commentedon Oct 22, 2026
Tracked in #5348 (triage 2026-10-22).