Skip to content

apr serve native routes leak <|im_end|> into text: /generate, /batch/generate, /stream/generate, /realize/generate, /realize/batch #4344

Closed as not planned

Description

@noahgift

Gap from #4340 (the /v1/chat/completions stream fix, branch x/4340-stream-im-end).

Observed. apr serve --no-gpu on a Qwen GGUF, multi-turn prompt ctl-recall-7, second turn. Evidence is in /mnt/nvme-raid0/tmp/tmp.ZtvuBEgtRL/cc324af070c2/off/serve/apr/:

route handler turn 2 text
POST /generate batch.rs::generate_handler <answer>7</answer><|im_end|>
POST /batch/generate batch.rs::batch_generate_handler <answer>7</answer><|im_end|>
POST /realize/batch batch_generate_handler (same) <answer>7</answer><|im_end|>
POST /stream/generate stream_generate.rs::stream_generate_handler <answer>7</answer><|im_end|>
POST /realize/generate stream_generate_handler (same) <answer>7</answer><|im_end|>

recall-cats shows the same leak.

Cause. The model writes the end-of-turn marker out as ordinary text tokens (<|im, _end|>), so the EOS stop token never fires. A whole special token decodes to "", which is why only the spelled-out form leaks. None of these routes truncate text at a marker.

Fix scope. These are RAW completion routes, so only special-token markers are cut (<|im_end|>, <|endoftext|>, <|end|>, </s>, <|im_start|>). The chat turn markers (\nUser:) are not cut, because a raw completion may legitimately contain them. text echoes the prompt, so the cut is searched only AFTER the prompt's decode; a chat-templated prompt carries markers of its own. The pregenerated stream stops at the token that contains the marker start and truncates that token's text.

Falsifier. One test per handler (the five routes map to three handlers).

Activity

  1. added this to the 0.70.0 milestone on Sep 25, 2026
  2. added a commit that references this issue on Sep 25, 2026
  3. modified the milestones: 0.70.0, 0.70.1 on Sep 26, 2026
  4. modified the milestones: 0.70.1, 0.71.0 on Oct 4, 2026
  5. added
    P1High priority
    and removed
    P0Critical priority
    on Oct 14, 2026
  6. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    P0 to P1 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, so it stays on the 0.71.0 milestone at P1.

  7. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    merged into #5313 (master ticket for epic #3999; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions