Repository navigation
apr serve GGUF CPU: POST /v1/completions runs past stop on 13/13 CRUX prompts (llama-server stops) #4339
Closed as not planned
Description
Binary: apr v0.69.3-rc.2 x86_64 cpu asset (sha256 faa473e339f81744…). Model: qwen2.5-coder-1.5b-instruct-q4_k_m.gguf (sha cc324af070c2). Found by the #3797 CRUX serve run #2 on lambda (rows under /mnt/nvme-raid0/tmp/tmp.ZtvuBEgtRL/cc324af070c2/off/serve/).
On all 13 prompts, in both stream and nonstream modes, apr keeps generating past the answer. It continues with "\nHuman: …" turns, while llama-server on the same prompt stops at <answer>…</answer>. /v1/chat/completions is unaffected. This looks like the GGUF counterpart of #4334 (SafeTensors stop_tokens=[]).
done_when: the /v1/completions GGUF path stops at EOS or the turn-end ids. A RED→GREEN test pins it, and a CRUX serve rerun shows 0/13 overruns.
Activity
noahgift commentedon Sep 24, 2026
Claimed by aprender-60 (cop assignment, 23:50Z). Branch off batch/0.70.0.
test(serve): gate completions_eog_stop_4339 on gpu like its neighbour (…
fix(serve): the batch scheduler answering /v1/completions ran past EO…
noahgift commentedon Sep 25, 2026
Fixed on
fix/4339-gguf-completions-stop@ace8997f5(branched offbatch/0.70.0@ 79f968b), ready for aprender-59 to fold.Cause. Every CPU GGUF
/v1/completionsbackend built its config withstop_tokens: Vec::new():try_quantized_completions,try_cached_completions, and the continuous-batch scheduler (batch_processing.rs, both of its configs). A single EOS would not have been enough either. The Qwen instruct GGUF declares<|im_end|>(151645) as EOS, but a raw completion ends with<|endoftext|>(151643).Fix.
completion_stop_tokens(tokenizer, model_eos)returns the EOS plus every end-of-generation marker the vocabulary actually contains (<|im_end|>,<|endoftext|>,<|eot_id|>,<|end_of_text|>,<end_of_turn>), each once.ContinuousBatchRequestgainsstop_tokens, so the batch scheduler gets the same set.Tests (
api/tests/completions_eog_stop_4339.rs): 7/7 pass.Vec::new(): 3 tests fail.batch_processing.rsput back toVec::new(): the batch test fails, and the response reads"model":"batch-q4k-1", which shows the batch path ran.cargo test -p aprender-serve --lib api::gives 1920 passed. clippy-D warningsand fmt are clean.Real model (qwen2.5-coder-1.5b-instruct Q4_K_M, apr built from the fix, CRUX
/v1/completionssweep against llama-server): 0 of 38 rows ran past the answer (finishlengthor\nHuman:). The rc.2 run had 28 of 28. Of the 19 non-streaming rows, 13 match llama's text exactly. The other 6:<|im_end|>on later turns of multi-turn prompts. That is apr mis-tokenizes <|im_end|> after sentence punctuation: 4 junk tokens are fed to the model instead of the turn delimiter #2815: the tokenizer does not split<|im_end|>after punctuation, so the prompt the model sees is off.Quorum on 79f968b...ace8997: sonnet-5 PASS, haiku-4-5 PASS, sonnet-5 PASS. Degraded same-family (no agy; the author is opus-5.5). Round 2's haiku FAIL on the batch scheduler is what
ace8997f5fixes.Out of scope:
try_gpu_completions(wgpu) and the CUDA paths, filed as #4345.🤖 Generated with Claude Code
noahgift commentedon Oct 14, 2026
merged into #5313 (master ticket for epic #3999; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.