You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Chat routes stop only on the declared EOS; completions stop on every end-of-generation marker (#4661) #4810
Sub-ticket of #4661 (one of its two causes; the other is #4804).
The chat routes (/v1/chat/completions, /api/chat) build their stop set from the declared EOS id alone, while /v1/completions uses completion_stop_tokens: the EOS plus every end-of-generation marker in the vocabulary (<|im_end|>, <|endoftext|>, <|eot_id|>, <|end_of_text|>, <end_of_turn>). A Qwen instruct GGUF declares <|im_end|> as EOS. When a turn ends on <|endoftext|>, chat decodes on to max_tokens while completions stops.
Measured A/B (same tokenizer, /v1/chat/completions, 5 prompts x seeds 1/5/42, CUDA serve):
Qwen3-Coder-30B-A3B Q4_K_M, "What is 2+2?": base runs to the 256-token limit repeating the prompt (3/3 seeds); with the shared stop set it stops at 8 tokens, "2 + 2 = 4" (3/3). Haiku prompt: 99 -> 24 tokens. The other 3 prompts are unchanged.
chat_stop_tokens returns completion_stop_tokens(tokenizer, Some(eos)), so every chat path (quantized CPU, CUDA dense, CUDA MoE, Qwen3.5, cached, GPU) stops on the same set as /v1/completions; ignore_eos still empties it.
Unit test chat_stops_on_every_eog_marker_not_just_eos goes through chat_quantized_config (signature unchanged), is RED on the parent commit and GREEN on the fix.
The 30B "What is 2+2?" chat request stops at 8 tokens for seeds 1, 5 and 42 on the CUDA serve path.
try_cuda_backend passes the pmat complexity gate (cognitive <= 25) through a behaviour-neutral extraction; cargo test -p aprender-serve --lib is green.
P0 to P2 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, and its milestone is backlog.
Folded into #4701 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it. Pushed branch: fix/4810-chat-eog-stop.
Activity
docs(audits): PMAT-4810 receipt, AC 3 measured A/B cited (Refs #4810)
refactor(serve): every chat path calls chat_stop_tokens (Refs #4810)
docs(audits): PMAT-4810 receipt, call-site table and complete lib run…
fix(serve): APR Q4K, SafeTensors CUDA and APR transformer chat stop o…
docs(audits): PMAT-4810 receipt and AC 1 cover APR Q4K, SafeTensors C…
noahgift commentedon Oct 14, 2026
P0 to P2 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, and its milestone is backlog.
noahgift commentedon Oct 14, 2026
Folded into #4701 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it. Pushed branch:
fix/4810-chat-eog-stop.Agent: aprender-clerk