You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf(serve): Qwen3.5 host build double-holds weights on unified memory (f32 embed + fp16 prefill prewarm), so the 4B Q4_K_M cannot fit 8 GB #5258
On an 8 GB unified-memory ARM64 CUDA edge device (Orin, 7.6 GB MemTotal), the CRUX release model Qwen3.5-4B Q4_K_M (sha256 00fe7986ff5f…) cannot run on the GPU, and apr chat is refused by the host-RAM check, because the host build holds a second, expanded copy of the weights:
the token embedding is expanded to f32: 248,320 × 2,560 × 4 B = 2.4 GiB (host_capacity.rs: need = 1.10 × file + 0.60 × f32 embed = 4,330 MiB for a 2,613 MiB file)
measured: apr serve --gpu RSS 6.2 GB for the 2.7 GB 4B file; after the host load, MemAvailable drops by ~3.2 GB, then the GPU plan needs 3,368 MiB more
Measured on apr 0.71.2 @42b9873da, built with --features cuda.
Ask: keep the embedding quantized (dequantize rows on lookup) and do not keep an fp16 copy of the weights resident on a unified-memory device, so the 4B fits in 8 GB as it does in llama.cpp/ollama.
Falsifier: on the 8 GB device, apr run and apr chat on Qwen3.5-4B Q4_K_M answer on the GPU with no capacity refusal, and peak RSS is ≤ 1.5 × file size.
merged into #5305 (master ticket for epic #3598; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
Moved: this sub-ticket is now a row in #5321 (apr memory footprint: serve Qwen3.5 within 8 GB unified-memory ARM64 CUDA (Orin)), split out of #5305 by theme.
Activity
noahgift commentedon Oct 14, 2026
merged into #5305 (master ticket for epic #3598; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5321 (apr memory footprint: serve Qwen3.5 within 8 GB unified-memory ARM64 CUDA (Orin)), split out of #5305 by theme.