serve: Qwen3.5 CUDA batch engine and serve wiring (#5279 P2) #5293
Copy link
Copy link
Closed as not planned
Description
Phase P2 of #5279. Builds on #5280 (scheduler over a BatchEngine trait) and #5289 (Qwen35CudaModel::forward_batch_greedy).
apr serve on a Qwen3.5 GGUF with CUDA answers every chat request under one Mutex<Qwen35Session>, so c concurrent requests decode one after another and never form a batch (m = 1 for every c).
Change.
- A
Qwen35BatchEngineimplementsBatchEngine.- A single request (m = 1) is one turn of the resident session, exactly as today.
- For m > 1 it borrows the session's device model and keeps one
Qwen35CudaStateper slot, so the session's own state and prefix reuse are untouched. It prefills each prompt, then takes oneforward_batch_greedystep per decode step for the greedy slots. A slot with temperature > 0 is sampled from its own logits. - The F2 CPU-parity guard runs before the first batch, as it does before the session's first turn.
start_qwen35_serverspawns the CUDA batch scheduler over that engine when the session is on the GPU.try_qwen35_backendsends chat requests to it. Requests that ask for logprobs or a trace keep the session path, which records them.
Done when.
- c concurrent requests to the engine form one batch of m = c (scheduler log
Batch m=). - Each greedy slot's tokens equal the same prompt decoded alone through
forward_single_greedy. - The m = 1 route still goes through the session.
Not in this phase. At m > 1 a slot ignores seed and repeat_penalty, as the dense batched path does today. Decode is still one forward per slot. Batched kernels are later phases, so this phase is not expected to meet the c = 4 decode-timing rule.
Metadata
Metadata
Assignees
Labels
No labels
Activity
noahgift commentedon Oct 14, 2026
merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.