Repository navigation
perf(serve): Qwen3.5 GPU decode at 0.15-0.27x of ollama on 8 GB unified-memory Orin; batched prefill falls back to one token #5301
Description
Measured on the 8 GB unified-memory Orin (runner label jetson), model Qwen3.5-2B-Q4_K_M.gguf (sha256 aaf42c8b7c3c…), apr 0.71.2 built from #5292 head cebe27dc, --features cuda.
| apr | ollama 0.17.6 (same file) | |
|---|---|---|
GPU decode, apr bench --fast (64 tok, 5 iters) |
7.1 tok/s | — |
GPU decode, CRUX run cells, engine-reported, median of 19 |
4.0 tok/s (1.4–6.6) | 26.7 tok/s (26.4–28.8) |
TTFT, apr bench |
897 ms | not measured (prompt rate 418 tok/s median) |
apr is at 0.15–0.27× of ollama on GPU decode. apr bench logs, on every iteration: batched prefill: N tokens ... would not fit (... needs 561 MiB ...; 0 MiB free); prefilling one token at a time on the GPU, so prefill falls back to one token at a time. Peak RSS of the same bench is 6857 MB (#5258).
Done when the edge nightly's GPU decode ratio row reads ≥ 1.0 on this device.
Refs #5254.
Activity
noahgift commentedon Oct 14, 2026
merged into #5305 (master ticket for epic #3598; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5321 (apr memory footprint: serve Qwen3.5 within 8 GB unified-memory ARM64 CUDA (Orin)), split out of #5305 by theme.