You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
realizar: batched GEMV row-equal to the single-row kernels for Qwen3.5 (#5279 P3) #5303
Phase P3 of #5279. Builds on #5289 (Qwen35CudaModel::forward_batch_greedy) and #5293 (the serve batch engine).
At m > 1, forward_batch_greedy still runs one full forward per slot, so every projection streams its weights m times per step. On gx10 (Qwen3.5-4B Q4_K_M) four concurrent requests each decode at 24.9% of a lone request's rate. The V3 shape rule needs at least 50%.
Change.
Measure, on the real 0.8B file, whether the existing batched GEMV kernels (batched_hw_dp4a_q4k_gemv_into, batched_q4k_gemv_into, batched_q6k_gemv_into) give each row bitwise the same output as gemv_dispatch on that row alone. This is the single-row path a lone request takes.
Use a batched kernel for a qtype only where each row is bitwise equal. Otherwise add a batched kernel that keeps the single-row kernel's per-row accumulation order: it loads each weight block once and dots it against m rows.
The Q8_1 activation buffer grows to hold m rows, once, when the model is built.
Done when.
A test on the real file holds each batched GEMV used by the Qwen3.5 forward bitwise equal to gemv_dispatch row by row, for m in {1, 2, 4, 8, 16} and for every qtype the file uses (Q4_K, Q5_K, Q6_K, Q8_0).
A qtype that cannot reach bitwise equality is named, with its measured max difference, and stays on the per-row loop.
Not in this phase. Wiring the kernels into the forward (P4).
merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
Activity
noahgift commentedon Oct 14, 2026
merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.