0.70.2 P0: Qwen2.5-0.5B-Instruct-f16.gguf cuda serve /api/chat: gibberish output (known failure in 0.70.1)

Known failure shipped in 0.70.1 (aprender 0.70.1, candidate cc4463f) under C272 5b; P0 for 0.70.2. Not a regression: it fails the same way on v0.69.5-rc.1 (5660b08).

Model: Qwen2.5-0.5B-Instruct-f16.gguf
Host(s): lambda
Check: cuda serve /api/chat
0.70.1 result: FAIL: gibberish (fragment ' | |' repeats), stream=false and stream=true
v0.69.5-rc.1 result: FAIL: identical
Logs: 0.70.1: /mnt/nvme-raid0/tmp/evid-0701b/lambda-ladder.log; rc.1: /mnt/nvme-raid0/tmp/screen-0701/logs/inv_Qwen2.5-0.5B-Instruct-f16.gguf.log, /mnt/nvme-raid0/tmp/screen-0701/receipts-0695/
Reproduce: scripts/model_ladder.sh --host <lambda|gx10> --only inv:Qwen2.5-0.5B-Instruct-f16.gguf

Done when: the ladder row is OK on lambda and gx10.

Root-cause evidence (aprender-df): the forward pass is correct; the run-on is the model's own sampled tail

Binary apr 0.70.0 (316dee2cd4) built from origin/main, CPU (--no-gpu); the same symptom appears on CUDA. Prompt = apr's rendered ChatML, 42 tokens (default Qwen system turn plus "What is 2+2?").

Logits match llama.cpp at every step. apr's sampled-path log-softmax (debug print in session.rs::advance_and_choose) was compared with llama-server /completion (n_probs 5) on identical token ids at positions 42–121. The argmax agrees at every position and |Δ logprob| ≤ 0.02.
Where it goes wrong: pos 109, right after "feel free to ask!". <|im_end|> p = 0.55 (apr 0.592 nats, llama 0.592 nats), and ' |' (760) is the runner-up. At temp 0.7, seed 42 draws ' |'. From then on, both apr and llama put p ≈ 0.96–0.99 on ' |', so the loop is the model's own prediction.
llama.cpp shows the same failure at the same sampler settings (temp 0.7, top_k 40, top_p 1, min_p 0, no repeat penalty, 256 tokens). Seeds 0–29: llama.cpp stopped on 28 and ran on into ' | | | |' on 2 (seeds 18 and 28). apr on seeds 0–29: 30 stop, 0 run-on.
Why the ladder always fails: the /api/chat probe sends no options. apr then uses temp 0.7 and the fixed default seed 42, which deterministically lands in this tail. With temp 0 the output is "2+2 equals 4." and finish stop. Temp 0.7 with seeds 7 or 0–29 also stops. This is the same in 0.69.5-rc.1 and 0.70.1 because neither the sampler nor the model changed.

Conclusion: there is no numeric, sampler or stop-token defect in apr for this row. Making the probe deterministic, or otherwise changing how it samples, changes what the gate measures, so it needs operator sign-off and is not part of this fix.

Correction: my earlier "the model's sampled tail, not apr" comment is withdrawn.

That comparison fed both engines apr's own prompt token ids, and those ids were already wrong. The defect is in apr.

Root cause. Serve builds GGUF tokenizers with BPETokenizer::new, which does greedy longest-match with no special-token handling. Qwen vocabularies hold ?< (75414) and .< (15757), so …?<|im_end|> encodes as ?<, |, i, m, _end, |, >Ċ. The model never sees a turn end. <|im_start|> survives only where a text segment begins, which is why it looked fine.

The same defect is behind #4662: there the model emits <|endoftext|> (outside the stop set) and runs on.

Fix. Greedy encode now matches <|…|> vocab entries whole before running longest-match. The regression test fails on main with exactly the split shown above. The PR is coming, linked from #4662. The contract comment at c14794d is not in any PR and is superseded.

Agent: aprender-b3

fix(serve): APR Q4K, SafeTensors CUDA and APR transformer chat stop o…

0.70.2 ships on a CRUX-smoke emergency scope (operator ruling C322, A1 and A2). This issue is refused for 0.70.2 and moves to milestone 0.71.0, with no per-issue contract edit. The 0.70.2 release notes name it as a known failure.

P0 to P1 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, so it stays on the 0.71.0 milestone at P1. Its 0.70.2 label belonged to a release that has shipped.

Re-parent: former parent #4689 is closed. Moved under open epic #3998 (same 0.71 train).

Re-parent: grouped under #4701 (the model-ladder ticket, open, under E1 #3998) with the other ladder known-fail rows #4661-#4664; same 0.71 train.

Folded into #4701 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it. Pushed branch: fix/4661-chat-eog-stop-set, fix/4661-cuda-serve-gibberish.

Agent: aprender-clerk

Navigation Menu

Repository navigation

0.70.2 P0: Qwen2.5-0.5B-Instruct-f16.gguf cuda serve /api/chat: gibberish output (known failure in 0.70.1) #4661

Root-cause evidence (aprender-df): the forward pass is correct; the run-on is the model's own sampled tail

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

0.70.2 P0: Qwen2.5-0.5B-Instruct-f16.gguf cuda serve /api/chat: gibberish output (known failure in 0.70.1) #4661

Description

Activity

noahgift commented on Oct 4, 2026

Root-cause evidence (aprender-df): the forward pass is correct; the run-on is the model's own sampled tail

noahgift commented on Oct 4, 2026

noahgift commented on Oct 7, 2026

noahgift commented on Oct 14, 2026

noahgift commented on Oct 14, 2026

noahgift commented on Oct 14, 2026

noahgift commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions