perf(serve): CPU decode and TTFT parity vs ollama on 8 GB unified-memory ARM64 CUDA (Orin) #5276

Child of #5254. GPU decode is tracked in #5259; this ticket covers the rest of the speed-parity ask on the same device.

Problem

On the 8 GB unified-memory ARM64 CUDA edge device (Orin), the ollama comparison is measured only for GPU decode (#5259: 7.1 vs 26.5 tok/s on Qwen3.5-2B Q4_K_M). Three things are not measured yet:

1. CPU decode: apr --fast on CPU vs ollama with num_gpu 0, same GGUF files.
2. TTFT / prefill: apr TTFT is 897 ms on 2B. Batched prefill falls back on every prompt ("would not fit (cuBLAS f32 at 2048 rows needs ~561 MiB …)"), and there is no ollama prompt_eval figure next to it.
3. Where the time goes: no apr profile or apr trace breakdown for the gap on this device.

Plan

1. Baseline: the first nightly ollama rows from ci(lab): nightly fast-verbs CRUX smoke + tok/s on the ARM64 CUDA edge runner, ranked gap list #5254's edge job (CPU and GPU, decode and prompt_eval, run id and sha).
2. Profile the biggest gap first with apr profile (roofline) and apr trace --trace-level layer, using a binary built from the measured sha.
3. One fix PR per root cause, each with a before/after row from the same device and files.

Falsifier

On both Qwen3.5 rows (2B, 0.8B), same device and files:

• CPU decode tok/s ≥ 0.9 × ollama num_gpu 0 eval tok/s.
• TTFT ≤ 1.1 × ollama's (load excluded; prompt_eval_duration for the same prompt).
• GPU decode is covered by perf(serve): GPU decode on 8 GB unified-memory ARM64 CUDA runs 0.27-0.39x ollama on the same Qwen3.5 GGUF #5259's falsifier.

🤖 Generated with Claude Code