Skip to content

perf(serve): Qwen3.5 host build double-holds weights on unified memory (f32 embed + fp16 prefill prewarm), so the 4B Q4_K_M cannot fit 8 GB #5258

Closed as not planned

Description

@noahgift

Child of #5254.

On an 8 GB unified-memory ARM64 CUDA edge device (Orin, 7.6 GB MemTotal), the CRUX release model Qwen3.5-4B Q4_K_M (sha256 00fe7986ff5f…) cannot run on the GPU, and apr chat is refused by the host-RAM check, because the host build holds a second, expanded copy of the weights:

  • the token embedding is expanded to f32: 248,320 × 2,560 × 4 B = 2.4 GiB (host_capacity.rs: need = 1.10 × file + 0.60 × f32 embed = 4,330 MiB for a 2,613 MiB file)
  • the qwen35 fp16 prefill prewarm holds 2,618 MiB for the 1.28 GB Qwen3.5-2B file (perf(prefill): MMQ int8 prefill matmul + LM head on the last token only (0.69.5) #4313)
  • measured: apr serve --gpu RSS 6.2 GB for the 2.7 GB 4B file; after the host load, MemAvailable drops by ~3.2 GB, then the GPU plan needs 3,368 MiB more

Measured on apr 0.71.2 @42b9873da, built with --features cuda.

Ask: keep the embedding quantized (dequantize rows on lookup) and do not keep an fp16 copy of the weights resident on a unified-memory device, so the 4B fits in 8 GB as it does in llama.cpp/ollama.

Falsifier: on the 8 GB device, apr run and apr chat on Qwen3.5-4B Q4_K_M answer on the GPU with no capacity refusal, and peak RSS is ≤ 1.5 × file size.

Activity

  1. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    merged into #5305 (master ticket for epic #3598; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.

  2. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    Moved: this sub-ticket is now a row in #5321 (apr memory footprint: serve Qwen3.5 within 8 GB unified-memory ARM64 CUDA (Orin)), split out of #5305 by theme.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions