serve: GGUF prompts tokenized greedily, <|im_end|> after text splits into 6 ids (+3 per turn vs llama.cpp) #4804

Sub-issue of #4662 (B6). Refs #4662.

Defect. apr serve run <model.gguf> tokenizes prompts with a greedy longest-match BPETokenizer that has no merges and no special-token split, so <|im_end|> straight after text is not one token. .<|im_end|>\n encodes as .< |i m _end | >\n (6 ids) where llama.cpp gives . <|im_end|> \n (3 ids). The model sees a turn end it never saw in training, which feeds the degenerate tail in #4662.

Where. Both GGUF serve states build the tokenizer as BPETokenizer::new(vocab, vec![], unk):

• AppState::with_quantized_model_and_vocab (crates/aprender-serve/src/api/mod_app_state_gpu.rs, CPU serve via crates/apr-cli/src/commands/serve/server.rs)
• AppState::with_cuda_model_and_vocab (same file, CUDA serve when the GGUF has no merges handed over)

With empty merges, BPETokenizer::encode (crates/aprender-serve/src/tokenizer.rs) takes the greedy loop, which never consults special tokens. GGUFModel::encode (crates/aprender-serve/src/gguf/token.rs, canonical byte-level BPE with tokenizer_st_partition, #3726/#3993) is not used by these routes.

Measured (main build 6192aa9, CPU serve vs llama.cpp server, same GGUF, llama.cpp's /apply-template text fed to both /tokenize):

/v1/chat/completions usage.prompt_tokens moves by the same amount (0.5B: 42/36, 42/36, 45/39, 39/33). <|im_start|> after a newline happens to survive (greedy longest match from a line start). Prompt 2 on the 30B also shows ordinary text off the merges (2 ids differ in "haiku about rust."), the #3726 class.

Done when. The GGUF serve routes encode with the same tokenizer as GGUFModel::encode, and a test asserts the ids of a ChatML prompt with <|im_end|> right after punctuation equal llama.cpp's (e.g. ?<|im_end|>\n → 30 151645 198 on Qwen2.5).

Owner aprender-a7n