You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Gap from #4340 (the /v1/chat/completions stream fix, branch x/4340-stream-im-end).
Observed.apr serve --no-gpu on a Qwen GGUF, multi-turn prompt ctl-recall-7, second turn. Evidence is in /mnt/nvme-raid0/tmp/tmp.ZtvuBEgtRL/cc324af070c2/off/serve/apr/:
route
handler
turn 2 text
POST /generate
batch.rs::generate_handler
<answer>7</answer><|im_end|>
POST /batch/generate
batch.rs::batch_generate_handler
<answer>7</answer><|im_end|>
POST /realize/batch
batch_generate_handler (same)
<answer>7</answer><|im_end|>
POST /stream/generate
stream_generate.rs::stream_generate_handler
<answer>7</answer><|im_end|>
POST /realize/generate
stream_generate_handler (same)
<answer>7</answer><|im_end|>
recall-cats shows the same leak.
Cause. The model writes the end-of-turn marker out as ordinary text tokens (<|im, _end|>), so the EOS stop token never fires. A whole special token decodes to "", which is why only the spelled-out form leaks. None of these routes truncate text at a marker.
Fix scope. These are RAW completion routes, so only special-token markers are cut (<|im_end|>, <|endoftext|>, <|end|>, </s>, <|im_start|>). The chat turn markers (\nUser:) are not cut, because a raw completion may legitimately contain them. text echoes the prompt, so the cut is searched only AFTER the prompt's decode; a chat-templated prompt carries markers of its own. The pregenerated stream stops at the token that contains the marker start and truncates that token's text.
Falsifier. One test per handler (the five routes map to three handlers).
P0 to P1 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, so it stays on the 0.71.0 milestone at P1.
merged into #5313 (master ticket for epic #3999; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
Activity
docs(findings): #4344 native-route <|im_end|> leak verified fixed on …
comparator: none+ reason #3774noahgift commentedon Oct 14, 2026
P0 to P1 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, so it stays on the 0.71.0 milestone at P1.
noahgift commentedon Oct 14, 2026
merged into #5313 (master ticket for epic #3999; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
comparator: none+ reason #3774