Skip to content

serve: Qwen3.5 CUDA batch engine and serve wiring (#5279 P2) #5293

Closed as not planned

Description

@noahgift

Phase P2 of #5279. Builds on #5280 (scheduler over a BatchEngine trait) and #5289 (Qwen35CudaModel::forward_batch_greedy).

apr serve on a Qwen3.5 GGUF with CUDA answers every chat request under one Mutex<Qwen35Session>, so c concurrent requests decode one after another and never form a batch (m = 1 for every c).

Change.

  • A Qwen35BatchEngine implements BatchEngine.
    • A single request (m = 1) is one turn of the resident session, exactly as today.
    • For m > 1 it borrows the session's device model and keeps one Qwen35CudaState per slot, so the session's own state and prefix reuse are untouched. It prefills each prompt, then takes one forward_batch_greedy step per decode step for the greedy slots. A slot with temperature > 0 is sampled from its own logits.
    • The F2 CPU-parity guard runs before the first batch, as it does before the session's first turn.
  • start_qwen35_server spawns the CUDA batch scheduler over that engine when the session is on the GPU.
  • try_qwen35_backend sends chat requests to it. Requests that ask for logprobs or a trace keep the session path, which records them.

Done when.

  • c concurrent requests to the engine form one batch of m = c (scheduler log Batch m=).
  • Each greedy slot's tokens equal the same prompt decoded alone through forward_single_greedy.
  • The m = 1 route still goes through the session.

Not in this phase. At m > 1 a slot ignores seed and repeat_penalty, as the dense batched path does today. Decode is still one forward per slot. Batched kernels are later phases, so this phase is not expected to meet the c = 4 decode-timing rule.

Activity

  1. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions