Repository navigation
perf(serve): GPU decode on 8 GB unified-memory ARM64 CUDA runs 0.27-0.39x ollama on the same Qwen3.5 GGUF #5259
Description
Child of #5254.
GPU decode on an 8 GB unified-memory ARM64 CUDA edge device (Orin) runs at 0.27–0.39× ollama 0.17.6 on the same GGUF file:
| model (Q4_K_M) | sha256 | apr bench --fast tok/s | ollama eval tok/s | ratio |
|---|---|---|---|---|
| Qwen3.5-2B | aaf42c8b7c3c… | 7.1 (TTFT 897 ms) | 26.5 | 0.27× |
| Qwen3.5-0.8B | bd258782e35f… | 14.6 | 37.3 | 0.39× |
apr: apr bench <gguf> --fast --warmup 2 --iterations 5 --max-tokens 64 --json, backend line GPU (CUDA, Orin), apr 0.71.2 @42b9873da plus the unified-headroom fix (otherwise every GPU plan is refused). ollama: /api/generate raw, num_predict 64, temperature 0, 3 runs (26.4/26.5/26.5 and 37.1/37.3/37.4).
Batched prefill also falls back on every prompt: "batched prefill: N tokens … would not fit (cuBLAS f32 at 2048 rows needs ~561 MiB …)".
Falsifier: apr decode tok/s ≥ 0.9 × ollama on both rows, on the same device and files.
Activity
noahgift commentedon Oct 14, 2026
merged into #5305 (master ticket for epic #3598; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5321 (apr memory footprint: serve Qwen3.5 within 8 GB unified-memory ARM64 CUDA (Orin)), split out of #5305 by theme.