You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
serve: CUDA batch scheduler over a BatchEngine trait (#5279 P0) #5280
cuda_batch_scheduler becomes generic over a BatchEngine trait with seven methods: single request, setup+prefill, decode step, add slot, recycle slot, cleanup, reset. The dense OwnedQuantizedModelCuda implements it by plain delegation, so its behaviour does not change. This lets a Qwen3.5 engine plug into the same scheduler later (P2).
Done when:
the scheduler builds over the trait;
the existing scheduler tests pass unchanged;
a new mock-engine test shows the scheduler forms Batch m=c for c in {1,4,8,16} concurrent requests.
Falsifier: the mock-engine test fails if setup ever sees m != c.
merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
Activity
noahgift commentedon Oct 14, 2026
merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.