fix(preprocessor): table-driven Unicode classes, file-driven added tokens, parse_special and neutralize policy, Qwen3 byte-split decode - #1
Open
Aharrypotter wants to merge 5 commits into
Conversation
…rocess locale
The hand-written pre-tokenizer matchers implemented \p{L}, \p{N}, and \s
with std::iswalpha, std::iswdigit, and std::iswspace, which depend on the
process-global locale that initLocal() installed. Two consequences:
- correctness varied by platform: on H20 the container has no en_US.UTF-8,
initLocal() fell back to C.UTF-8 with a warning on stdout, and the
result depended on glibc's wctype tables rather than on the tokenizer
- std::iswdigit accepts only ASCII digits, while Hugging Face's \p{N}
matches every Nd, Nl, and No code point. MiniCPM5's \p{N}{1,3} chunking
therefore split "编号①②③，价格٣٤٥元，第Ⅳ章，面积²" differently from the
official tokenizer (parity case unicode_numerals_and_spaces, mllm token
ids wrong before this change, exact after)
isLetter, isDigit, the new isMark, isWhitespace, and toLower now read the
vendored llama.cpp Unicode tables that mllm already ships under
preprocessor/tokenizers/llama_cpp_unicode. Eight tokenizers switch from
std::iswspace and std::towlower to the table-backed helpers. initLocal()
no longer touches the global locale; it only prepares std::wcout for the
examples that print wide strings, silently.
Mllm-Test-UnicodeClass pins the behavior under the classic C locale, and
the parity gate gains the numeral-and-space case for all five models.
…eir attributes Special tokens were a hand-written list in every tokenizer. The lists drifted from the checkpoints (the Qwen3 family lacked six added tokens, and lacked the tool markers until the previous fix), and MiniCPM5 kept a private linear scan over its 510 added tokens instead. - BPE and BPEUTF8 now keep every added_tokens entry with id, content, special, lstrip, rstrip, single_word, and normalized (AddedToken.hpp) - AutoTokenizer::registerAddedTokens feeds them into the special-token trie with match-time attributes; Qwen3, Qwen3-MoE, Qwen3.5, and Qwen Ascend drop their hand lists, MiniCPM5 drops its scan, and Qwen NPU and MiniCPM4 (no local checkpoint) keep their lists and register the file's entries on top - the tries gain splitSegments(): each segment says whether it is a matched token, lstrip/rstrip absorb adjacent white space into the match (the white space is dropped, only the token id is emitted), single_word requires word boundaries, and SplitOptions::parse_special=false leaves control tokens in the text. Semantics were pinned against Hugging Face tokenizers 0.22.2 before writing the tests - the trie automaton is replaced by a plain leftmost-longest walk. The ported Python automaton mis-split "</think>" once "/think" was also registered, which the MiniCPM5 parity gate caught immediately All five model parity gates pass on both builds; MiniCPM5 now tokenizes through the same trie as the rest at the same token ids.
…ize()
Add TokenizeOptions{parse_special} to AutoTokenizer and AutoTokenizerUTF8.
With parse_special=false the checkpoint's control tokens are not
recognized in the text, so "<|im_start|>" inside user content tokenizes
as ordinary characters. Added tokens that are not control tokens
(<think>, <tool_call> in the Qwen3 family) are still matched, which is
exactly Hugging Face's split_special_tokens=True. The migrated tokenizers
(Qwen3, Qwen3-MoE, Qwen3.5, Qwen Ascend, Qwen NPU, MiniCPM5, MiniCPM4)
implement the overload through the trie's SplitOptions; other tokenizers
keep their behavior through the default overload.
BPE::attrOf(id) reports kControl / kUserDefined / kNormal, the subset of
llama.cpp's token attributes mllm needs.
The parity script gains stage C: parse_special=false output equals
tokenizer.encode(text, split_special_tokens=True) and contains no control
token id, for a forged turn boundary, a BOS in running text, and plain
text, on all five models. The probe tool exposes --no_parse_special.
…nce-tagged prompts Add control_token_policy to the migrated model configurations. The default stays 'reject'. With 'neutralize' the official template is rendered with provenance (the jinja.cpp fork's Template::render_parts marks every string under messages and tools), ChatPreprocessor::renderSpans returns origin-tagged spans, and tokenizePromptSpans tokenizes input-origin spans with parse_special=false. A forged turn boundary inside user content is then ordinary text: the prompt decodes to exactly the rendered bytes and carries no control token that the template did not emit. - ChatPreprocessor::render keeps rejecting under both policies, since a flat string cannot carry provenance; only renderSpans honours neutralize - neutralize requires jinja_required and fails at construction on the legacy backend, which cannot tell template text from message content - Qwen3, Qwen3-MoE, Qwen3.5, Qwen Ascend, Qwen NPU, MiniCPM5, and MiniCPM4 gain requestFor / renderPromptSpans / tokenizePrompt; Qwen3.5 expands its image and video placeholders only in template-origin spans - the jinja.cpp submodule moves to the fork commit that adds render_parts - parity stage D: neutralize yields the reject-path token ids for every benign product case, and accepts the forged prompt with the same number of control ids as a benign one, on all five models
Qwen3, Qwen3-MoE, and Qwen Ascend converted every token to a wide string on its own. Byte-level BPE splits a multi-byte character across tokens, so a single token is often not valid UTF-8, and the conversion dropped the incomplete tail: the Qwen3 service, the probing service, and the two example runners printed mojibake or missing characters for text such as "日本語テキスト 🇨🇳". This is the TODO left on UbiquitousLearning#701 for the next PR. Add detokenizeBytes() to the three tokenizers, the same contract Qwen3.5, MiniCPM5, and LFM2.5 already use, and stream the bytes through StreamingUtf8Decoder in the service callback, the probing callback, and the examples, flushing the decoder at EOS. detokenize() is kept for callers that want a wide string of one token and is now implemented on top of the byte form. Qwen3DecodeTest shows the fixture splits at least one character across tokens, that the per-token wide path loses it, and that the byte path round-trips exactly. The parity gate gains stage E: streaming decode of Transformers' ids equals Transformers' decode for CJK, emoji, combining marks, and regional-indicator flags on all five models; the probe tool exposes --decode.
Aharrypotter
marked this pull request as ready for review
September 7, 2026 02:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to UbiquitousLearning#707. It makes the tokenizer layer deterministic and data-driven, and turns the injection guard from a refusal into a choice. Two tracks, reviewed together because the second depends on the attributes the first records:
tokenizer.jsonwith their attributes instead of per-model hand-written lists. Both are behavior fixes with a Transformers oracle.tokenize()gains aparse_specialgate equal to Hugging Facesplit_special_tokens=True, and a per-modelcontrol_token_policyaddsneutralize, which renders the official template with provenance so a control token inside message content becomes ordinary text instead of an error.It also closes the follow-up item recorded on UbiquitousLearning#701: the Qwen3 family decoded each token as its own wide string, which dropped the bytes of characters that byte-level BPE splits across tokens.
Every migrated runner still produces byte-identical prompts and identical token ids under the default configuration; the parity gate proves it on five models.
Reviewer focus
isLetter/isDigit/iswspacewerestd::iswalphaand friends on a global locale thatinitLocal()installed. On the H20 container that locale does not exist, so results depended on glibc'sC.UTF-8tables; andstd::iswdigitonly accepts ASCII while\p{N}is every Nd/Nl/No code point. The new parity caseunicode_numerals_and_spacesfailed on MiniCPM5 before commit44e26651and passes after. The tables are the ones mllm already vendored from llama.cpp underpreprocessor/tokenizers/llama_cpp_unicode.added_tokens(special,lstrip,rstrip,single_word) and match leftmost-longest. The ported Python automaton mis-split</think>once MiniCPM5's/thinkwas registered; the parity gate caught it and the walk was replaced.lstrip/rstrip/single_wordsemantics were pinned againsttokenizers0.22.2 before the tests were written; no local checkpoint uses them, so they are covered by unit tests only.added_tokens); MiniCPM5 drops its private 510-token linear scan. Qwen NPU and MiniCPM4 have no local checkpoint, so they keep their lists and register the file's entries on top.ChatPreprocessor::render()refuses under both policies, because a flat string cannot carry provenance; onlyrenderSpans()returns origin-tagged spans, and only thejinja_requiredbackend can produce them (the engine fork'sTemplate::render_parts).neutralizeon the legacy backend fails at construction. Token merges do not cross span boundaries; stage D shows benign prompts tokenize identically under both policies for the Qwen and MiniCPM templates, whose literals end at a control token or newline.detokenizeBytes(), the contract Qwen3.5, MiniCPM5, and LFM2.5 already use; the Qwen3 service, the probing service, and the two examples stream those bytes throughStreamingUtf8Decoderand flush at EOS.Qwen3DecodeTestshows the fixture splits a character across tokens and that only the byte path round-trips it; parity stage E compares streaming decode of Transformers' ids with Transformers' decode for CJK, emoji, combining marks, and flags on all five models.Aharrypotter/jinja.cpp(a04ea9b): reserved code points U+E000/U+E001 wrap input strings, comparison and search operations ignore them, slicing operations keep per-byte provenance,tojson/repr wrap the literal,render()strips them, and a context that already contains them is refused. Upstream's 414-case suite passes with and without the opt-in macros.Review map
mllm/preprocessor/tokenizers/Unicode.hppandtests/preprocessor/UnicodeClassTest.cpp— table-backed predicates,initLocal()no longer touches the global locale.mllm/preprocessor/tokenizers/AddedToken.hpp,BPE*.{hpp,cpp},AutoTokenizer.{hpp,cpp}— added-token attributes,registerAddedTokens,splitSegments,TokenizeOptions,attrOf.mllm/preprocessor/chat_template/ChatTemplate.{hpp,cpp},PromptTokenization.hpp—ControlTokenPolicy,renderSpans,tokenizePromptSpans.third_party/jinja.cpp(submodule bump toa04ea9b) —render_parts.requestFor/renderPromptSpans/tokenizePrompt,control_token_policy.mllm/models/qwen3/modeling_qwen3_service.hpp,modeling_qwen3_probing_service.hpp,examples/qwen3*/main.cpp— streaming decode.tests/preprocessor/—AddedTokenTest.cpp,Qwen3DecodeTest.cpp, newChatTemplateTestcases, probe flags--no_parse_special/--policy, parity stages C and D.Validation
Source identity: branch
feat/tokenizer-unicode-attributesatad6b8554, five commits on top of UbiquitousLearning#707's320515fe. Every gate below ran at that exact head; the evidence boundary is that no row makes a claim about any other tree. macOS with Jinja ON and OFF: 32 / 19 chat-template tests, 12 Unicode, added-token, and decode tests, MiniCPM5 and Qwen3.5 tokenizer A/B, and the parity gate for qwen3_5, minicpm5, qwen3, qwen3_moe, qwen_ascend with stages A (bytes, HF ids, mllm ids), B (product path legacy vs jinja vs HF), C (parse_special=falsevssplit_special_tokens=True), D (neutralizevsrejecton benign prompts, forged prompt accepted with no extra control ids and decoded text equal to the rendered prompt), and E (streaming decode equals Transformers decode on byte-split text). H20 repeated the Linux ON/OFF suites and cross-compiled eleven Android arm64 artifacts with NDK r28b. The OnePlus 13T ran the bundle: unit tests, injection refusal, neutralize checks,parse_specialgate, probe A/B, and 16-token greedy runner A/B across legacy,jinja_required, andneutralizeconfigurations for MiniCPM5-1B and Qwen3.5-0.8B with identical generated ids. Upstream CI and human review are pending.Validation matrix — exact location, identity, and permitted conclusion
ad6b8554320515feunicode_numerals_and_spaces44e26651a04ea9bad6b8554ad6b8554050b095eartifacts, models hash-matchedSupported scope and limits
control_token_policydefaults toreject; nothing changes without an explicit configuration opt-in.neutralizerequiresjinja_required. Underneutralize,render()still refuses; callers must userenderSpans().lstrip/rstrip/single_wordare implemented per Hugging Face semantics but no supported checkpoint uses them; coverage is unit-test only.neutralizethan underreject. None of the five validated templates does.🤖 Generated with Claude Code