apr memory footprint: serve Qwen3.5 within 8 GB unified-memory ARM64 CUDA (Orin) #5321

Scope: Memory use and speed of apr serve/run on small unified-memory CUDA devices (Jetson Orin 8 GB): weights held twice, a fixed 16 GiB headroom that refuses every plan, KV caches committed in full at construction, whole-file reads for header-only checks, and decode/TTFT parity with ollama there.

Why it matters: Edge devices are a target platform. Today apr refuses or runs at 0.15-0.39x ollama on the same model on the Orin, so the edge story does not hold.

Done when: On the Orin runner, apr serve loads Qwen3.5 0.8B and 2B on GPU without refusal, decode is at least 0.8x ollama on the same GGUF, and the nightly verb run drives every apr verb there, all with receipts.

Epic: #3598. Folded under triage rule 7b (2026-10-14): each row below was its own issue, closed as not planned and kept here as a sub-ticket. Reopen an issue when its work starts or its PR opens.

Sub-tickets

• perf(serve): Qwen3.5 host build double-holds weights on unified memory (f32 embed + fp16 prefill prewarm), so the 4B Q4_K_M cannot fit 8 GB #5258 perf(serve): Qwen3.5 host build double-holds weights on unified memory (f32 embed + fp16 prefill prewarm), so the 4B Q4_K_M cannot fit 8 GB (milestone none)
  • accept: the outcome in the title, with a receipt
• perf(serve): GPU decode on 8 GB unified-memory ARM64 CUDA runs 0.27-0.39x ollama on the same Qwen3.5 GGUF #5259 perf(serve): GPU decode on 8 GB unified-memory ARM64 CUDA runs 0.27-0.39x ollama on the same Qwen3.5 GGUF (milestone none)
  • accept: the outcome in the title, with a receipt
• fix(serve): fixed 16 GiB unified-memory headroom refuses every GPU plan on an 8 GB device #5260 fix(serve): fixed 16 GiB unified-memory headroom refuses every GPU plan on an 8 GB device (milestone none)
  • accept: the outcome in the title, with a receipt
• perf(serve): CPU decode and TTFT parity vs ollama on 8 GB unified-memory ARM64 CUDA (Orin) #5276 perf(serve): CPU decode and TTFT parity vs ollama on 8 GB unified-memory ARM64 CUDA (Orin) (milestone none); branch perf/5276-bench-no-gpu
  • accept: the outcome in the title, with a receipt
• perf(serve): Qwen3.5 GPU decode at 0.15-0.27x of ollama on 8 GB unified-memory Orin; batched prefill falls back to one token #5301 perf(serve): Qwen3.5 GPU decode at 0.15-0.27x of ollama on 8 GB unified-memory Orin; batched prefill falls back to one token (milestone none)
  • accept: Done when the edge nightly's GPU decode ratio row reads ≥ 1.0 on this device.
• fix(qa): ollama_parity cannot measure on unified memory, ollama stays resident while apr plans its GPU load #5302 fix(qa): ollama_parity cannot measure on unified memory, ollama stays resident while apr plans its GPU load (milestone none)
  • accept: Done when the edge nightly's ollama parity row is measured on this device.
• CI: Jetson nightly stage S4 builds apr with CUDA on the Orin and drives every apr verb against one tiny model #4885 CI: Jetson nightly stage S4 builds apr with CUDA on the Orin and drives every apr verb against one tiny model (milestone backlog)
  • accept: the outcome in the title, with a receipt
• serve: KVCache::new and OptimizedKVCache::new commit num_layers-1 full layers at construction (the form #4778 removes from StreamingKVCache) #4783 serve: KVCache::new and OptimizedKVCache::new commit num_layers-1 full layers at construction (the form fix(serve): KV caches committed every byte at construction; re-enable the three FLAKE-0 memory-bound tests (#4769) #4778 removes from StreamingKVCache) (milestone backlog)
  • accept: the outcome in the title, with a receipt
• #3750 PR B: the 17 header-only whole-file model reads outside apr qa — magic/header/metadata readers in their format crates #3761 apr qa's capability predicates std::fs::read the WHOLE model (twice, plus a to_vec copy) — RSS swings past 30 GB on Qwen3-30B-A3B, an OOM risk on GB10's unified memory #3750 PR B: the 17 header-only whole-file model reads outside apr qa — magic/header/metadata readers in their format crates (milestone backlog); branch fold/rc1-S1-3761
  • accept: the outcome in the title, with a receipt