serve: Qwen3.5 on CUDA never batches concurrent requests (m_formed=0 at c=4/8/16) #5279

Activity

1. noahgift

   added a parent issue on Oct 14, 2026

   • V3 serving-shape parity: lambda perf041 witness for Qwen3.5-4B at c in {1,4,8,16} #4971
2. noahgift commented on Oct 14, 2026

   

   noahgift

   on Oct 14, 2026

   MemberAuthor

   More actions

   Where concurrency is lost (read on main d97936f)

   Why no batch forms

   • Startup returns early. crates/apr-cli/src/commands/serve/handler_gpu_completion.rs:405-406 checks hybrid_forward_handles(arch) and then calls start_qwen35_server. That happens before build_serve_model, so no OwnedQuantizedModelCuda is built and spawn_cuda_batch_scheduler (crates/aprender-serve/src/api/cuda_batch_scheduler.rs:204, typed to the dense model) is never spawned.
   • One mutex serializes requests. crates/aprender-serve/src/api/mod_app_state_qwen35.rs:9-10 holds the session as Mutex<Qwen35Session> ("One request at a time: the hybrid is a single-stream model"). It also sets cuda_batch_tx: None. qwen35_chat_backend.rs takes session.lock() for a whole turn, so concurrent requests queue on the lock.

   No batched decode exists. Qwen35CudaState (gguf/cuda/forward_qwen35_cuda.rs:99-118) holds state for one sequence. Decode runs through forward_single (:1726) one token at a time, and the decode CUDA graph is also single-sequence. The DeltaNet kernels in cuda/executor/gdn_ops.rs have no sequence dimension. "Batched prefill" (#3596, #4228) means the prompt rows of one sequence.

   What done needs

   1. An m-sequence forward: tokens[m], states[m], positions[m].
   2. Q4K GEMM, not GEMV, for the projections and the FFN.
   3. DeltaNet conv, gate and delta-rule kernels that take a sequence dimension, with per-sequence conv and SSM state.
   4. Attention KV caches per sequence, each with its own length.
   5. A scheduler for the hybrid model. The existing one assumes the dense BatchedDecodeState.
   6. The early return and the mutex replaced, with per-slot prefix reuse and checkpoints.
   7. A contract that batched output matches single-sequence output token for token, like the parity gates in continuous-batching-v1.

   Size: L. A round-robin over N single-sequence states would print m=N but would not batch, so it does not meet the done-when.
3. noahgift commented on Oct 14, 2026

   

   noahgift

   on Oct 14, 2026

   MemberAuthor

   More actions

   Phase plan (read on main d97936f). One PR per phase, each with its own falsifier

   What already holds

   • State is separate from weights. Per-sequence state is Qwen35CudaState (forward_qwen35_cuda.rs:102-119), and the weights live in Qwen35CudaModel (:254-264). forward_single, prefill and the checkpoint calls all take the state from outside. What is single-row is the shared scratch (:182-269).
   • The decode graph is keyed by the state's buffer addresses (forward_qwen35_cuda_graph.rs:64). So the batch path stays eager at first.
   • Problem: no existing batched matmul is bitwise equal to the single-sequence path.
     • The prefill row helper qwen35_project_rows uses cuBLAS F16, so it is not equal to gemv_dispatch.
     • The dense batched Q4K/Q6K GEMVs are switched off (USE_BATCHED_GEMV = false, layers/batched_qkv.rs:22).
     • dp4a quantizes the activations.

   Phases

   • P0 — Scheduler over a trait. cuda_batch_scheduler becomes generic over a BatchEngine trait, with seven methods: single request, setup+prefill, decode step, add slot, recycle slot, cleanup, reset. The dense impl is plain delegation. Test: a mock engine forms Batch m=c for c ∈ {1,4,8,16}.
   • P1 — Looped multi-slot step on Qwen35CudaModel. forward_batch_greedy(tokens, states, positions), with a looped forward per slot. Test: greedy streams are bitwise equal to forward_single, for m=1..4 with ragged prompts.
   • P2 — Qwen3.5 engine plus serve wiring.
     • Removes the Mutex<Qwen35Session> and the startup early return. m=1 keeps the session turn (prefix reuse and logprobs unchanged); m>1 uses a pool of per-slot states.
     • Tests: c=4 batched output equals serial output, and an apr-cli smoke test asserts Batch m=N.
     • After P2, m_formed == c, but decode is still serial. The S7 decode-timing rule will likely still reject the witness.
   • P3 — Row-twin GEMV kernels (Q4K, Q6K, Q8_0, F32). Each weight block is loaded once and dotted against m rows, with the same accumulation order per row as the single-row kernel. Test: bitwise equal to gemv_dispatch for m ∈ {1,2,4,8,16}. This is the risky phase: the same idea was switched off once before (realizr#220).
   • P4 — Batched matmuls. Every projection, the FFN and lm_head run at m rows through P3, plus batched_gpu_argmax. GDN and attention still loop per slot over row views. Test: P4 steps are bitwise equal to P1 steps. The first phase that should satisfy the V3 timing rule.
   • P5 — GDN conv1d and the delta rule gain a slot dimension. grid.y = m, with a device u64[m] array of per-layer state pointers. Test: bitwise equal to the per-slot result.
   • P6 — Decode attention (head_dim 256, output gate) gains a slot dimension. Pointer arrays plus seq_lens[m]. Test: ragged lengths {1, 17, 300, 4095} equal the per-slot result.
   • P7 — Per-slot prefix reuse. Longest-prefix admission. Test: shared-prefix prefill rows are skipped and the output is identical.
   • P8 (optional) — Batched CUDA graph keyed by the slot-pointer arrays. Test: graph output equals eager output.

   Fallback if P3 cannot reach bitwise equality: the parity criterion drops to the prefill tests' standard: argmax equal and cosine at or above the budget (prefill_tests.rs:101).
4. noahgift commented on Oct 14, 2026

   

   noahgift

   on Oct 14, 2026

   MemberAuthor

   More actions

   merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
5. noahgift

   removed a parent issue on Oct 14, 2026

   • V3 serving-shape parity: lambda perf041 witness for Qwen3.5-4B at c in {1,4,8,16} #4971
6. noahgift

   closed this as not plannedon Oct 14, 2026