Skip to content

Chat routes stop only on the declared EOS; completions stop on every end-of-generation marker (#4661) #4810

Closed as not planned

Description

@noahgift

Sub-ticket of #4661 (one of its two causes; the other is #4804).

The chat routes (/v1/chat/completions, /api/chat) build their stop set from the declared EOS id alone, while /v1/completions uses completion_stop_tokens: the EOS plus every end-of-generation marker in the vocabulary (<|im_end|>, <|endoftext|>, <|eot_id|>, <|end_of_text|>, <end_of_turn>). A Qwen instruct GGUF declares <|im_end|> as EOS. When a turn ends on <|endoftext|>, chat decodes on to max_tokens while completions stops.

Measured A/B (same tokenizer, /v1/chat/completions, 5 prompts x seeds 1/5/42, CUDA serve):

Acceptance:

  • chat_stop_tokens returns completion_stop_tokens(tokenizer, Some(eos)), so every chat path (quantized CPU, CUDA dense, CUDA MoE, Qwen3.5, cached, GPU) stops on the same set as /v1/completions; ignore_eos still empties it.
  • Unit test chat_stops_on_every_eog_marker_not_just_eos goes through chat_quantized_config (signature unchanged), is RED on the parent commit and GREEN on the fix.
  • The 30B "What is 2+2?" chat request stops at 8 tokens for seeds 1, 5 and 42 on the CUDA serve path.
  • try_cuda_backend passes the pmat complexity gate (cognitive <= 25) through a behaviour-neutral extraction; cargo test -p aprender-serve --lib is green.

Activity

  1. added
    bugSomething isn't working
    P0Critical priority
    on Oct 5, 2026
  2. added this to the backlog milestone on Oct 8, 2026
  3. added
    P2Medium priority
    and removed
    P0Critical priority
    on Oct 14, 2026
  4. noahgift commented on Oct 14, 2026

    @noahgift
    ContributorAuthor

    P0 to P2 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, and its milestone is backlog.

  5. noahgift commented on Oct 14, 2026

    @noahgift
    ContributorAuthor

    Folded into #4701 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it. Pushed branch: fix/4810-chat-eog-stop.

    Agent: aprender-clerk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium prioritybugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions