Repository navigation
serve tokenizes GGUF prompts by greedy longest-match, not BPE merges: serve/code answers differ from run #4979
Description
Symptom
On Qwen3-30B-A3B-Instruct-2507 Q4_K_M (qwen3moe), apr serve and apr code (which drives
serve) answer a needle prompt wrongly, while apr run and apr chat on the same file answer
it correctly. The needle is TANGERINE-4417.
| path | prompt tokens | answer |
|---|---|---|
apr run --gpu |
45 | TANGERINE-4417 |
apr serve run --gpu, /v1/chat/completions |
49 | TA NGERINE-4417Human: The passphrase is TA NGERINE-4417. Remember it.… (runs to the token cap) |
The serve answer is the same on 0.70.3 and on a current branch. It does not change with
temperature 0, with frequency_penalty/presence_penalty 0 and repeat_penalty 1.0, or with
chat_template_kwargs removed. With a long (~5k-token) prompt it gives
TA NGERINE-4417\n\nWait, let me correct that…. Ladder cells: serve and code fail at the
4k, 8k and 20k rungs (0/6). run and chat pass at all three rungs (6/6).
Cause
Serve tokenizes the prompt by greedy longest-match, not by BPE merges. POST /tokenize
on The passphrase is TANGERINE-4417. returns
[785, 75493, 374, 38891, 6140, 640, 3981, 12, 19, 19, 16, 22, 13]
The Ġpassphrase Ġis ĠTA NG ER INE - 4 4 1 7 .
Rank-ordered BPE over the same vocabulary and merges gives ĠT ANGER INE (350, 92368, 3981).
The model has never seen ĠTA NG ER INE in training, and it copies the odd split back as
"TA NGERINE".
Code path (current main):
BPETokenizer::encode(crates/aprender-serve/src/tokenizer.rs) uses the merge-rank BPE
path only whenmerge_rulesis non-empty. Otherwise it falls back to greedy
longest-match. The comment there says greedy is "sufficient for GGUF vocabularies"; this
measurement says it is not.apr serve run <file>.ggufbuilds its tokenizer from the vocabulary alone.
extract_gguf_vocabreads onlymodel.vocabulary().
AppState::with_quantized_model_and_vocaband the qwen3moe--gpuroute both call
BPETokenizer::new(vocab, vec![], unk), andBPETokenizer::newignores its_merges
argument anyway.- The other
AppState::with_*_and_vocabconstructors inapi/mod_app_state_gpu.rs,
mod_app_state_new.rsandmod_app_state_qwen35.rsbuild their tokenizer the same way.
serve/chat.rspasses the tokenizer.json merges toBPETokenizer::new, which drops them.
Onlywith_cuda_model_and_bpekeeps merges. apr runis correct becauseprepare_tokens_ggufcallsGGUFModel::encode, which goes
throughByteLevelBpe::from_gguf: special tokens, the model's pre-tokenizer, and ranked
merges, as in llama.cpp (apr's byte-level BPE tokenizer is non-canonical: every non-ASCII byte → id 0, ASCII segmented unlike llama.cpp (2 of 10 ids agree on a path) — corrupts copy tasks and every non-ASCII user; apr parity is blind (shared tokenizer) #3726).
Fix direction
Serve should encode with the same tokenizer as apr run: GGUFModel::encode /
ByteLevelBpe for GGUF, or merge_rules() passed through with_merges, checked against
ByteLevelBpe on the pre-tokenizer. Add a falsifier: serve /tokenize and apr run
produce the same ids on a prompt with a word that greedy matching splits differently
(TANGERINE is one).
Scope not yet measured
- Only one file was measured (above). Qwen3-Coder-30B-A3B-Instruct passed the same serve
cells, but it most likely gets the same greedy ids and is just more robust to them. - Of serve's +4 prompt tokens, 1 is accounted for (
TANGERINEsplit into 4 tokens instead
of 3). The other 3 are not attributed yet.
🤖 Generated with Claude Code
Activity
noahgift commentedon Oct 9, 2026
Fix on branch
fix/4979-serve-gguf-byte-level-bpe@ ae31e17 (no PR yet: PR cap).BPETokenizercarries the file'sByteLevelBpe;encode()uses it when attached.GGUFModel::byte_level_bpe()returns it for a gpt2/bpe file with an implemented pre-tokenizer and merges, andNoneotherwise.AppState::with_mapped_gguf_modelattaches it, which covers the CPU, qwen3moe/dense CUDA and batch routes.with_qwen35_sessionattaches it too.tokenizer.ggml.modelcheck, the attach call) each turn their rows RED.End to end on an RTX 4090 with Qwen3-30B-A3B-Instruct-2507 Q4_K_M (CUDA route confirmed in the serve log):
/tokenize " TANGERINE"apr run: 45)TA NGERINE-4417Human: …TANGERINE-4417apr run: 5243)TA NGERINE-4417TANGERINE-4417Not covered: the release tokenizer parity gate measures
apr tokenize encode, not serve's/tokenize.Agent: apr-embed
noahgift commentedon Oct 9, 2026
Fix is on branch
fix/4979-serve-gguf-byte-level-bpe@ ae31e17. No PR yet (open-PR cap); it opens at its slot.Evidence, Qwen3-30B-A3B GGUFs (Q4_K_M), temperature 0:
/tokenize " TANGERINE"now gives ĠT|ANGER|INE (was ĠTA|NG|ER|INE). On the 4k needle prompt, serve prompt_tokens went 5247 → 5243, equal toapr run, and the answer went "TA NGERINE-4417" → "TANGERINE-4417". The server log shows the qwen3moe MoE CUDA dispatch.aprsha256 8ff8d61a…, built from the qwen3moe (Qwen3-30B-A3B, Qwen3-Coder-30B-A3B, Q4_K_M) has NO working CUDA path: apr run --gpu falls back to CPU (rc 14), apr qa emits no gates — blocks 0.69.1 under the all-Q4_K-CUDA rule #3714 gap-2 tree plus this fix (the fix's 8 files hash-match ae31e17).apr codereports no backend, so their backend is not established.encode()delegation, thetokenizer.ggml.modelcheck or the attach call each turns them RED.Not covered: the release tokenizer parity check measures
apr tokenize encode, not serve's/tokenize. Extending it is a separate gate change.Refs #3714
noahgift commentedon Oct 14, 2026
merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5329 (apr serve request path: BPE tokenization, graceful drain, no 500 on oversize prompts), split out of #5312 by theme.