Repository navigation
apr serve: GPU/CUDA /v1/completions paths stop on a single EOS or none (follow-up to #4339) #4345
Closed as not planned
Description
#4339 fixed the CPU GGUF /v1/completions routes: try_quantized_completions, try_cached_completions and the continuous-batch scheduler now stop on EOS plus every end-of-generation marker in the vocabulary (completion_stop_tokens). The quorum lanes found sibling paths outside that CPU scope that still use a narrower stop set:
try_gpu_completions(wgpuGpuModel,gpu_completions_handler.rs):stop_tokens: Vec::new(), so it always runs tomax_tokens.- The CUDA GGUF completions paths (
try_cuda_gguf_completionsand the direct CUDA fallback) and thebatch.rspaths:vec![eos]only. On Qwen instruct GGUFs the EOS is<|im_end|>(151645), but a raw completion ends with<|endoftext|>(151643), so these paths run past the answer the same way the CPU path did.
Fix: route each through completion_stop_tokens(&tokenizer, state.model_eos_token_id()), with one test per backend (the Backend fixture in api/tests/completions_eog_stop_4339.rs extends to them).
🤖 Generated with Claude Code
Activity
noahgift commentedon Oct 4, 2026
Status at main @
316dee2cd4.Fixed. Every path this issue lists now calls
completion_stop_tokens:try_gpu_completions:gpu_completions_handler.rs:53;:405;completions_inner::619;batch.rs:134and:522;batch_processing.rs:337and:494.Three tests in
api/tests/completions_eog_stop_4345.rspin the wgpu path: a control, a stop on<|endoftext|>, and a stop on<|im_end|>. No test covers the CUDA paths yet, because they need a cuda build.Still on the narrow stop set. I found one sibling by reading the code; I have not run it. The quantized backend of
POST /stream/generateandPOST /realize/generate(stream_generate.rs:97) passesstop_tokens: vec![eos_id(..)]. On a Qwen instruct GGUF, a raw prompt can therefore run past<|endoftext|>, the same trap this issue fixed elsewhere.I'm keeping this issue open for that path. The fix: route it through
completion_stop_tokensand add a test that uses the sameBackendfixture.One doc nit: the comments at
gpu_completions_handler.rs:83and:840still say this backend passes an emptystop_tokens. That stopped being true with this fix.(la-0.73, owner)
noahgift commentedon Oct 13, 2026
The CUDA path still listed here is fixed on branch
w71-e3a/4345-stream-generate-stop@3dcec5434f. Branch only, no PR: the PR cap is full.Change. The CUDA backend of
POST /stream/generateand/realize/generate(stream_generate.rs:97) built its stop set asvec![eos]. It now callsgenerate_stop_tokens(batch.rs), which returns the EOS plus every EOG marker in the vocabulary. The quantized/generateconfig uses the same function, so the two backends share one stop set. The two stale comments ingpu_completions_handler.rs(:83,:854) are corrected.Test.
the_native_generate_stop_set_is_the_eos_plus_every_eog_marker, incompletions_eog_stop_4339.rs. The CUDA handler needs a CUDA model to run, so the test checks the function the handler calls. It follows the/v1/logprobsconfig test.Receipts (x86 box, 2026-10-13, head
3dcec5434f):cargo test -p aprender-serve --lib api::tests::completions_eog_stop: 13 passed, 0 failed.cargo clippy -p aprender-serve --lib -- -D warnings: rc=0.cargo clippy -p aprender-serve --features cuda --lib -- -D warnings: rc=0. That build compiled aprender-gpu, so the cuda feature was on.vec![eos_id(..)]restored, the test fails (rc=101,test result: FAILED. 0 passed; 1 failed). The file was restored afterwards and the tree is clean.Not run: the CUDA handler end to end on a GPU.
keep-open: the fix is on a branch only. It waits for a PR slot, a review and a merge.
noahgift commentedon Oct 14, 2026
merged into #5313 (master ticket for epic #3999; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
fix(serve): APR Q4K CUDA routes stop on every EOG marker too (Refs #4345
fix(serve): Qwen3.5 routes stop on every EOG marker; repair batch.rs …
fix(serve): SafeTensors CUDA and qwen3_moe chat stop on every EOG mar…
fix(serve): constrained Qwen3.5 chat stops on every EOG marker (Refs #…
fix(serve): Qwen3.5 raw /generate stops on every EOG marker (Refs #4345)
test(serve): pin the CUDA /stream/generate stop set (Refs #4345)
style(serve): rustfmt the include!d Qwen3.5 chat backend (Refs #4345)
docs(audit): re-derive the 2 surface-audit rows whose cited serve files