Repository navigation
perf(qwen35-cuda): gdn_delta_rule_recurrence 44.7 µs/call vs llama.cpp 2.5 µs (17.5×) — ~20% of the 4B decode gap #5222
Description
Found in the same nsys trace as the Qwen3.5 decode GEMV ticket. Setup: Qwen3.5-4B Q4_K_M, RTX 4090, 255 decode steps on each engine, apr 0.71.1 (03d4af1) against llama.cpp llama-bench at 60b06ab9a.
| kernel | calls | total | per call | |
|---|---|---|---|---|
| apr | gdn_delta_rule_recurrence |
6120 (24 layers × 255) | 273.7 ms | 44.7 µs |
| llama.cpp | gated_delta_net_cuda<128> |
6144 (24 × 256) | 15.6 ms | 2.5 µs |
- apr is 17.5× slower here. That is about 1.07 ms of apr's 9.5 ms per token, and about 20% of the 4B decode gap to llama.cpp. The GEMV kernels account for most of the rest.
- Every DeltaNet layer runs this kernel once per decode step, so the cost scales with model depth on every Qwen3.5 size.
Acceptance:
- A before/after nsys row from the pinned binary.
- The existing Qwen3.5 CUDA-vs-CPU parity tests stay green.
- Greedy decode text identical.
Activity
noahgift commentedon Oct 14, 2026
merged into #5305 (master ticket for epic #3598; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5320 (Qwen3.5 CUDA decode speed: close the GEMV and DeltaNet gaps to llama.cpp), split out of #5305 by theme.