Repository navigation
Small unified-memory CUDA devices: 16 GiB headroom refuses every model, no refusal before OOM, wrong pull hint #4947
Description
Parent: #3994 (E3) · Target: 0.71
Measured with the v0.70.3 aarch64 CUDA release asset (apr 0.70.3 (95f64ba14), tarball sha256 matches its .sha256) on an Orin Nano class device: 7.6 GB unified memory, models fetched with apr pull.
Rows
- F1 — the 16 GiB unified-memory headroom refuses every model on a small device.
UNIFIED_HEADROOM_BYTES = 16 GiB(crates/aprender-serve/src/capacity.rs) is subtracted fromMemAvailable. On 7.6 GB that leaves 0, soapr runrefuses Qwen3.5-0.8B (rc 8):weights 497 MiB + KV 0 MiB … = 1093 MiB, against 0 MiB free of 0 MiB (unified memory: MemAvailable 3910 MiB less a 16384 MiB headroom; cuMemGetInfo said 3722 MiB free). The same constant makesapr chatprefill one token at a time (~1 tok/s).- Regression check: NOT new since 0.70.2. The v0.70.2 asset (
apr 0.70.2 (89e261cda)) refuses with the same message on the same device. The constant is identical in v0.70.1, v0.70.2 and v0.70.3;git blameputs it at 5fa13fe (merge-back of v0.69.3, release: merge back v0.69.3 into main #4338, 2026-09-25). - Fix on a branch (not yet a PR):
unified_headroom(total) = min(16 GiB, total / 4), with a red/green test at this device's measured MemTotal/MemAvailable.
- Regression check: NOT new since 0.70.2. The v0.70.2 asset (
- F2 — no refusal before the OOM kill. Qwen3.5-4B Q4_K_M (2.7 GB file) is OOM-killed on every route, including
apr chat --no-gpu, on "What is 2+2?". Nothing is printed: RSS reached 3.2–3.6 GB and the process was killed by the system. Expected: a capacity refusal naming the shortfall, as F1's path prints for the GPU. - F3 — the
apr pullnext-step hint is wrong.apr pullprintsapr serve <path>; that fails withunrecognized subcommand. The command isapr serve run <path>(crates/apr-cli/src/commands/pull.rs, three sites). - F4 (LAB) — nightly e2e on the release-asset path for this device class. Install the published aarch64 CUDA asset (sha256-checked),
apr pulla small qwen35, then run / chat /serve run --gpusmoke. LAB per the gate rule: never on a PR, merge-queue or release job; a red updates this issue.
Not reproduced (for the record)
A report of qwen35 path rejected … decode state for 32768 positions … CUDA_ERROR_OUT_OF_MEMORY followed by a CPU fallback that ignored Ctrl-C did not reproduce on this device. The decode state starts at 4096 positions and doubles, so 32768 is reached only by a turn needing more than 16384 positions; an 18,013-token turn allocated it without error, also with 3.5 GB of memory pre-filled. SIGINT exited in 0.21–0.51 s on the GPU, CPU and serve paths, and the CPU route produced tokens.
🤖 Generated with Claude Code
Activity
fix(serve): recalibrate the host-RAM guard to the post-#4991 build (R…
fix(serve): the host-RAM guard reads memory before the map, not after…
fix(tests): the real-0.8B guard test uses the recalibrated need (Refs #…
fix(serve): a cgroup's reclaimable page cache is host-build headroom …
fix(serve): drop the uncited Q4_K dequant speedup numbers (Refs #4947)
noahgift commentedon Oct 12, 2026
F4 (LAB), first run, 2026-10-12 06:36–06:41Z. Jetson Orin (7607 MiB unified memory), rebooted at ~06:28Z.
Asset:
apr-v0.70.3-aarch64-unknown-linux-gnu-cuda.tar.gzfrom the v0.70.3 release. The sha256 matches the published.sha256(99a28cc6…604f9f). The binary reportsapr 0.70.3 (95f64ba14).Model:
apr pull hf://unsloth/Qwen3.5-0.8B-GGUF/Qwen3.5-0.8B-Q4_K_M.gguf, which returned rc 0. The aliasesqwen3.5:0.8bandqwen35:0.8bboth returnartifact not found.apr run <m> --gpuGPU capacity refused … 1093 MiB, against 0 MiB free of 0 MiB (unified memory: MemAvailable 4843 MiB less a 16384 MiB headroom; cuMemGetInfo said 4621 MiB free)apr chat <m> --gpuBackend: GPU (CUDA, Orin), answered2 + 2 = 4apr serve run <m> --gpu+POST /v1/chat/completions"content":"$2 + 2 = 4$."The red is F1, the 16 GiB headroom, as shipped in v0.70.3. The fix is #5146, merged at fbe921e, so it is on main but in no release yet. The red on this path clears with the first release asset cut after fbe921e, and the next F4 run on that asset checks it. Not yet done: running this every night. The e2e script for that is
~/f4-4947/f4.shon the Jetson.Note: the Jetson answers at DHCP 192.168.1.209. The
jetsonssh stanza still points at 192.168.50.53, which times out (infra#1562).Agent: aprender-w71-e3b
fix(serve): Qwen3.5 host-RAM guard + 1.27x host footprint, recalibrat…
noahgift commentedon Oct 12, 2026
F4 (LAB), v0.71.1 run, 2026-10-12 10:31:30–10:31:50Z. Same Jetson Orin (7607 MiB unified, 5995 MiB available at start).
Asset:
apr-v0.71.1-aarch64-unknown-linux-gnu-cuda.tar.gz, sha256a61645ba9650…matches the published.sha256, binaryapr 0.71.1 (e89fb30c4). Model: the samehf://unsloth/Qwen3.5-0.8B-GGUF/Qwen3.5-0.8B-Q4_K_M.gguf(theqwen3.5:0.8b/qwen35:0.8bshort refs both give rc 10, artifact not found).run --gpu1093 MiB against 0 MiB free of 0 MiB(16 GiB unified headroom)chat --gpu2 + 2 = 4, at ~1 tok/s (F1: prefill one token at a time)serve run --gpu+/v1/chat/completions2 + 2 = 4; serve logsbatched prefill ... would not fit (538 MiB)(same headroom)F1 still reproduces on the 0.71.1 release asset; F2/F3 not exercised by this smoke. Red stays on this issue (LAB).
noahgift commentedon Oct 14, 2026
merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5325 (apr on GPU: refuse by name before OOM, never fall back to CPU silently), split out of #5312 by theme.
noahgift commentedon Oct 30, 2026
Closing as not planned (iceboxed): this ticket is outside milestones 0.72.x-0.76.0 in the 2026-10-30 date-driven release plan, and the open-ticket cap needs it off the board. Tracked in the icebox epic #6521. Reopen it and give it a milestone to bring it back. (release planning worker, PMAT-5306, operator rulings 11:17Z)