Repository navigation
perf(qwen35-cuda): forward_single allocates a device buffer per token (GpuBuffer::from_host) — make the residual resident + in-stream upload #4316
Description
Part of #4249 (0.69.4). Blocks CUDA-graph capture work in #4215.
Claim (falsifiable): Qwen35CudaModel::forward_single (crates/aprender-serve/src/gguf/cuda/forward_qwen35_cuda.rs) runs GpuBuffer::from_host on every token — a cuMemAlloc, a blocking HtoD copy and a cuMemFree per token, and a device pointer that changes each token, so a captured graph cannot replay it.
Fix: allocate one [hidden_dim] residual buffer at construction; fill it per token with an in-stream cuMemcpyHtoDAsync on the execution stream.
Done when: the residual's device pointer is identical across tokens and survives a refused token (test), and qwen35_cuda_forward_single_matches_cpu_logits_end_to_end passes unchanged on a GPU.
Activity
noahgift commentedon Sep 24, 2026
Claimed by aprender-59; lands as a commit on the 0.69.4 epic batch #4318 (branch batch/0.69.4-epic-4249).
perf(qwen35-cuda): forward_single reuses one device residual instead …
PMAT-4316: GPU evidence — qwen35_cuda 17/17 on RTX 4090
merge origin/main (#4338 v0.69.3 merge-back, squash) into batch/0.69.…
noahgift commentedon Sep 25, 2026
Verified on batch/0.69.4-epic-4249 (fix 64b20f8; tested at 617bf6a63 = batch + #4273 fold, now da38177). RTX 4090 via gpu-q, cuda lib build: qwen35_cuda_tests 18/18 ok, 0 skips — incl. qwen35_cuda_residual_pointer_is_stable_across_tokens_and_refusals (ptr identical across 3 tokens + OOV refusal + position refusal) and qwen35_cuda_forward_single_matches_cpu_logits_end_to_end (unchanged). Done-when met. x/4316-resident-residual is NOT folded: it stacks on open #4332 (#4215, 0.71.0). — aprender-e6
perf(qwen35-cuda): forward_single reuses one device residual instead …