Repository navigation
qwen35 session: #4274 headroom doubles a 32k turn to 65536 positions — 27B OOMs and silently falls back to CPU #4443
Description
Regression in #4318 (0.69.4 batch), introduced by 90f8ec7 (#4274 fix). Found by the CRUX #4223 cells.
grown_capacity doubles MIN_CAPACITY until the turn fits, then clamps to self.context_length. That clamp is the GGUF's {arch}.context_length, which is 262144 for Qwen3.5, not the server's --context-length, so it never applies. A 32.8k-token turn now asks for 65536 positions, where the pre-fix code asked for exactly 32782.
Qwen3.5-27B Q4_K_M on an RTX 4090 (apr serve run --context-length 34816 --backend cuda --gpu-layers all, apr 0.69.4 90f8ec7, exe sha 3f67ab9d1d09):
[qwen35] batched prefill: 4106 tokens in 7032 ms (584 tok/s, chunk 512 rows, attention cuBLAS f32, from position 0)
warning: GPU (CUDA) qwen35 path rejected, falling back to CPU: a decode state for 65536 positions would not allocate: ... GPU memory allocation failed: CUDA driver error
After that the p32k request ran on the CPU and hit the 900 s client timeout. llamafile 0.10.6 finishes the same prompt in 15.6 s on the same GPU.
Two defects
- Headroom is an optimisation, yet failing to allocate it demotes the whole session to the CPU. The fix is to retry at exactly
positionsbefore falling back. - The session clamps to the GGUF context rather than the server's
--context-length. That is secondary, and plumbing the served context through fixes it.
Falsifier: 27B at p32k on the RC binary must print batched prefill: ~32.8k tokens ... from position 0 with no falling back to CPU line.
Activity
test(serve): drive #4443 through ensure_capacity, not just its helpers
quorum(#4318): 3/3 PASS at c13a233 — the #4443 ensure_capacity findin…
noahgift commentedon Sep 25, 2026
e2e receipt — fix at c13a233 (in #4318 head dded9c7, quorum 3/3 PASS)
Binary
apr 0.69.5 (d1bb4d9f4), bin sha256f0dcd183…. c13a233 changes only#[cfg(test)]code on top of d1bb4d9, so the release build is identical. Model: Qwen3.5-27B-Q4_K_M (gguf84b5f7f1…) on an RTX 4090,gpu-layers resolved=64/64 (backend=cuda). Two runs, the second on a clean GPU (936 MiB foreign).[qwen35] decode state: 65536 positions would not allocate, sized to the turn (32782) instead (#4443), and there are 0 "falling back" lines. GPU decode is 22.7 tok/s (129 tokens ×3, spread 0.2%).cpu_a_turn_past_min_capacity_is_allocated_with_headroomdrivesensure_capacityon a real CPU session. A mutant that reverts thegrown_capacitycall site topositionsis RED (left 20, right 32).gpu_headroom_that_will_not_allocate_keeps_the_turn_on_the_gpu(cuda) uses a test-only allocation ceiling. It passes on the 4090 and prints the qwen35 session: #4274 headroom doubles a 32k turn to 65536 positions — 27B OOMs and silently falls back to CPU #4443 notice. A mutant that drops the retry is RED at the stays-on-GPU assert. As a control, a ceiling below the turn does move the turn to the CPU.32780 tokens at position 0 would not fit (… flash at 512 rows needs 1144 MiB; 440 MiB free); prefilling one token at a time on the GPU. The request then times out at 300 s. The 27B weights leave 5.2 GB free, and the exact 32782-position state takes about 4.8 GB of it. 0.69.4 also timed out on 27B p32k (CRUX parity spike: apr serve vs llamafile on Qwen3.5 (0.8B/2B/4B/9B/27B) — measure gap, name techniques, file fixes #4223 table). My issue body's "batched prefill" falsifier overreached, and this belongs to perf(serve): qwen35 prefill falls to one token at a time when the 512-row workspace misses — 9B 32k TTFT 733 s, and it sticks (#4220 child) #4450.fix(serve): #4274 headroom that would not allocate demoted Qwen3.5 to…
test(serve): pin the exact allocation attempts, so a no-headroom retr…
test(serve): drive #4443 through ensure_capacity, not just its helpers
apr codeon Qwen3.5 Q4_K_M (4B, 9B) on CUDA: does it complete a scripted edit-and-verify task on lambda and gx10? UNMEASURED, and its driver runs throughapr serve --gpu(#3571) #3719noahgift commentedon Oct 14, 2026
merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
apr codeon Qwen3.5 Q4_K_M (4B, 9B) on CUDA: does it complete a scripted edit-and-verify task on lambda and gx10? UNMEASURED, and its driver runs throughapr serve --gpu(#3571) #3719noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5325 (apr on GPU: refuse by name before OOM, never fall back to CPU silently), split out of #5312 by theme.