Skip to content

serve: Qwen3.5 CUDA multi-slot greedy step, looped (#5279 P1) #5289

Closed as not planned

Description

@noahgift

Phase P1 of #5279.

Qwen35CudaModel decodes one sequence at a time (forward_single / forward_single_greedy take one Qwen35CudaState). The serve batch scheduler (P0, #5280) needs an engine that takes one step for m sequences at once.

Change. Add Qwen35CudaModel::forward_batch_greedy(tokens, states, positions) -> Vec<u32>. It takes one greedy step for each of m slots by looping forward_single_greedy per slot. The shape is checked before any device work: the three lengths must be equal and non-zero. Later phases (P4 onward) replace the loop with batched kernels behind the same signature.

Done when.

  • On the real Qwen3.5-0.8B file, for m = 1..4 with ragged prompt lengths, every slot's greedy stream from the interleaved batched steps is bitwise equal to the same prompt decoded alone with forward_single_greedy. This shows that the model's shared scratch buffers do not leak between slots.
  • A shape mismatch (unequal lengths, m = 0) is refused by name before any device work.

Activity

  1. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions