Skip to content

serve: CUDA batch scheduler over a BatchEngine trait (#5279 P0) #5280

Closed as not planned

Description

@noahgift

Part P0 of #5279, from the phase plan in #5279 (comment).

cuda_batch_scheduler becomes generic over a BatchEngine trait with seven methods: single request, setup+prefill, decode step, add slot, recycle slot, cleanup, reset. The dense OwnedQuantizedModelCuda implements it by plain delegation, so its behaviour does not change. This lets a Qwen3.5 engine plug into the same scheduler later (P2).

Done when:

  • the scheduler builds over the trait;
  • the existing scheduler tests pass unchanged;
  • a new mock-engine test shows the scheduler forms Batch m=c for c in {1,4,8,16} concurrent requests.

Falsifier: the mock-engine test fails if setup ever sees m != c.

Activity

  1. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions