From e5b2e75410f4f3af6da07baeb5388dba47ae8572 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 10:47:37 +0200 Subject: [PATCH] =?UTF-8?q?fix(tokenizer):=20the=20unknown=20token=20is=20?= =?UTF-8?q?the=20model's,=20not=20BPE's=20=E2=80=94=20Qwen3.5=20serves;=20?= =?UTF-8?q?a=20model=20with=20no=20vocabulary=20refuses=20by=20name=20(PMA?= =?UTF-8?q?T-3609)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BPETokenizer::new required `` in every vocabulary. Qwen3.5 has none: it declares tokenizer.ggml.eos_token_id and no unknown_token_id key at all. So `apr serve` refused a real, complete vocabulary, while a GGUF/APR with NO vocabulary loaded, because GH-226's placeholder synthesised `token{i}` with `` in slot 0. The less-specified input was the one accepted. This follows the operator's three constraints (#3609, 2026-09-20) and the cop's rulings (2026-09-21): - The unknown token is Option: `new(vocab, merges, impl Into>)`. A named token must exist. With no unknown token, CONSTRUCTION refuses, naming the first byte that has neither a `<0xNN>` token nor a byte-level glyph token (with_merges also requires every glyph), so encode never meets an unencodable byte: nothing is dropped and nothing is synthesised. With an unknown token present, encode is unchanged. - No vocabulary refuses by name at all 13 placeholder sites (apr-cli serve x7, aprender-serve cli x6). - apr serve's tokenizer.json path uses the DECLARED model.unk_token. The vocabulary-only AppState constructors use `vocabulary_unk_token` (the vocabulary's own ``, else none), the by-name interim #3675 replaces. - Prose: the GH-226 comment, the book's api-server example and examples/model_cache.rs no longer teach `` as a requirement. - Fixtures: REAL Qwen3.5-0.8B and TinyLlama GGUF headers, every key and scalar verbatim, only the per-token arrays sliced (generate.py, MANIFEST.json with each source header's sha256). They use the `.gguf-header` extension because .gitignore ignores `*.gguf`. Verified: aprender-serve lib 15892 passed (tokenizer 215, 9 new); apr-cli lib 7283 passed; clippy -D warnings clean on both; cargo check with cuda; contracts lib 1666; deny ok. MUTANT: restoring the unconditional `get("").ok_or_else(..)` turns the real-Qwen3.5 row RED with the original error "Unknown token '' not in vocabulary". Found and filed separately: #3677 (greedy encode reads Latin-1 characters as byte-level glyph tokens, so `é` decodes to U+FFFD). Closes #3609 Co-Authored-By: Claude Opus 5 (1M context) --- crates/apr-cli/src/commands/serve/chat.rs | 10 +- .../commands/serve/handler_gpu_completion.rs | 43 ++--- crates/apr-cli/src/commands/serve/handlers.rs | 27 +-- .../src/commands/serve/handlers_include_01.rs | 49 ++---- .../book/src/examples/api-server.md | 5 +- crates/aprender-serve/examples/model_cache.rs | 5 +- .../src/api/mod_app_state_gpu.rs | 24 ++- .../src/api/mod_app_state_new.rs | 15 +- .../src/cli/mod_server_commands.rs | 56 +++--- crates/aprender-serve/src/tokenizer.rs | 134 ++++++++++++--- crates/aprender-serve/src/tokenizer_04.rs | 4 + .../src/tokenizer_tests_unk_3609.rs | 161 ++++++++++++++++++ .../fixtures/gguf-header-slices/MANIFEST.json | 56 ++++++ .../fixtures/gguf-header-slices/generate.py | 135 +++++++++++++++ .../qwen3.5-0.8b.gguf-header | Bin 0 -> 39063 bytes .../tinyllama-1.1b-chat.gguf-header | Bin 0 -> 18724 bytes docs/roadmaps/entries/PMAT-3609.yaml | 22 +++ docs/roadmaps/roadmap.yaml | 22 +++ 18 files changed, 624 insertions(+), 144 deletions(-) create mode 100644 crates/aprender-serve/src/tokenizer_tests_unk_3609.rs create mode 100644 crates/aprender-serve/tests/fixtures/gguf-header-slices/MANIFEST.json create mode 100644 crates/aprender-serve/tests/fixtures/gguf-header-slices/generate.py create mode 100644 crates/aprender-serve/tests/fixtures/gguf-header-slices/qwen3.5-0.8b.gguf-header create mode 100644 crates/aprender-serve/tests/fixtures/gguf-header-slices/tinyllama-1.1b-chat.gguf-header create mode 100644 docs/roadmaps/entries/PMAT-3609.yaml diff --git a/crates/apr-cli/src/commands/serve/chat.rs b/crates/apr-cli/src/commands/serve/chat.rs index f7596598be..f8190817df 100644 --- a/crates/apr-cli/src/commands/serve/chat.rs +++ b/crates/apr-cli/src/commands/serve/chat.rs @@ -74,8 +74,14 @@ pub(crate) fn load_safetensors_tokenizer(path: &Path) -> Option").ok()?; + // Create BPE tokenizer with vocab and merge rules. #3609: the unknown token is the one + // tokenizer.json DECLARES (`model.unk_token`, null for byte-level models like Qwen), + // not a literal "" every model is assumed to have. + let unk_token = json + .get("model") + .and_then(|m| m.get("unk_token")) + .and_then(|v| v.as_str()); + let tokenizer = realizar::tokenizer::BPETokenizer::new(vocab.clone(), merges, unk_token).ok()?; Some(SafeTensorsTokenizerInfo { tokenizer: std::sync::Arc::new(tokenizer), diff --git a/crates/apr-cli/src/commands/serve/handler_gpu_completion.rs b/crates/apr-cli/src/commands/serve/handler_gpu_completion.rs index 910ceafcb6..07e390a622 100644 --- a/crates/apr-cli/src/commands/serve/handler_gpu_completion.rs +++ b/crates/apr-cli/src/commands/serve/handler_gpu_completion.rs @@ -418,7 +418,7 @@ fn start_gguf_server(model_path: &Path, config: &ServerConfig) -> Result<()> { .green() ); - let vocab = extract_gguf_vocab(&mapped_model, quantized_model.config().vocab_size); + let vocab = extract_gguf_vocab(&mapped_model)?; // PERF-021 / N4 / I-2: a request has a RESOLUTION, and it is reported. // @@ -463,26 +463,27 @@ fn start_gguf_server(model_path: &Path, config: &ServerConfig) -> Result<()> { run_cpu_server(quantized_model, vocab, Some(mapped_model), config, Some(offload)) } -/// Extract vocabulary from GGUF model, falling back to placeholder tokens. +/// Extract the vocabulary from a GGUF model, or refuse to serve it (#3609). /// -/// GH-226: When the GGUF lacks `tokenizer.ggml.tokens` metadata, the placeholder -/// vocabulary must include `` so that `BPETokenizer::new` can find it. -/// Without this, the serve path fails with "Unknown token '' not in vocabulary" -/// and the server exits before binding — causing "Server failed to become ready". -fn extract_gguf_vocab( - mapped_model: &realizar::gguf::MappedGGUFModel, - vocab_size: usize, -) -> Vec { - mapped_model.model.vocabulary().unwrap_or_else(|| { - eprintln!("Warning: No vocabulary in GGUF, using placeholder tokens"); - let mut vocab: Vec = (0..vocab_size).map(|i| format!("token{i}")).collect(); - // GH-226: Ensure is present for BPETokenizer compatibility. - // Use slot 0 (standard convention for unknown token). - if !vocab.is_empty() { - vocab[0] = "".to_string(); - } - vocab - }) +/// GH-226 once filled a missing `tokenizer.ggml.tokens` with `token0..tokenN` and put +/// `` in slot 0, because `BPETokenizer::new` required one. So a model with NO +/// vocabulary loaded, while a model with a real vocabulary that lacked `` (Qwen3.5) +/// was refused: the less-specified input was the one accepted. The unknown token is +/// now optional, and a model with no vocabulary refuses by name. Placeholder tokens +/// are not a tokenizer. +fn extract_gguf_vocab(mapped_model: &realizar::gguf::MappedGGUFModel) -> Result> { + mapped_model + .model + .vocabulary() + .ok_or_else(|| no_vocabulary("the GGUF (no tokenizer.ggml.tokens)")) +} + +/// #3609: the refusal every serve path gives for a model with no vocabulary. +fn no_vocabulary(what: &str) -> CliError { + CliError::ModelLoadFailed(format!( + "{what} has no vocabulary to tokenize with, so there is nothing to serve; \ + refusing to substitute placeholder tokens (#3609)" + )) } /// #2762: resolve the KV-cache context length for the GGUF + CUDA serve path. @@ -651,7 +652,7 @@ fn start_gguf_server_cuda( let quantized_model = OwnedQuantizedModel::from_mapped(&mapped_model).map_err(|e| { CliError::ModelLoadFailed(format!("Failed to rebuild quantized model: {e}")) })?; - let vocab = extract_gguf_vocab(&mapped_model, quantized_model.config().vocab_size); + let vocab = extract_gguf_vocab(&mapped_model)?; // CUDA init failed and this process fell back to CPU. The offload // report travels with it UNCHANGED, so `/v1/effective-config` shows // `gpu_layers_resolved` beside `backend_loaded: ["cpu"]` — which is diff --git a/crates/apr-cli/src/commands/serve/handlers.rs b/crates/apr-cli/src/commands/serve/handlers.rs index c6b7d51fe2..ded665a5b4 100644 --- a/crates/apr-cli/src/commands/serve/handlers.rs +++ b/crates/apr-cli/src/commands/serve/handlers.rs @@ -594,16 +594,13 @@ fn wgpu_smoke_test( } /// PMAT-340: Extract the real vocab from the GGUF for tokenize/detokenize. +/// #3609: a GGUF with no vocabulary refuses by name (see `extract_gguf_vocab`). #[cfg(feature = "wgpu")] -fn wgpu_vocabulary(mapped: &realizar::gguf::MappedGGUFModel, vocab_size: usize) -> Vec { - mapped.model.vocabulary().unwrap_or_else(|| { - eprintln!("Warning: No vocabulary in GGUF, using placeholder"); - let mut v: Vec = (0..vocab_size).map(|i| format!("token{i}")).collect(); - if !v.is_empty() { - v[0] = "".to_string(); - } - v - }) +fn wgpu_vocabulary(mapped: &realizar::gguf::MappedGGUFModel) -> Result> { + mapped + .model + .vocabulary() + .ok_or_else(|| no_vocabulary("the GGUF (no tokenizer.ggml.tokens)")) } /// PMAT-341: Build a BPE tokenizer from the GGUF merge rules. @@ -822,7 +819,7 @@ fn serve_wgpu_backend( println!("{}", "Starting WGPU inference server...".cyan()); // PMAT-340: Extract real vocab from GGUF for tokenization/detokenization - let vocab = wgpu_vocabulary(mapped, vocab_size); + let vocab = wgpu_vocabulary(mapped)?; // PMAT-341: Extract BPE merge rules for proper tokenization let merges = mapped.model.merge_rules().unwrap_or_default(); println!( @@ -1443,15 +1440,7 @@ fn try_apr_quantized_cpu(model_path: &Path, config: &ServerConfig) -> Result<()> let vocab = mapped .metadata .get_embedded_vocabulary() - .unwrap_or_else(|| { - let vocab_size = mapped.metadata.vocab_size.unwrap_or(32000); - eprintln!("Warning: No embedded vocabulary in APR, using placeholder tokens"); - let mut v: Vec = (0..vocab_size).map(|i| format!("token{i}")).collect(); - if !v.is_empty() { - v[0] = "".to_string(); - } - v - }); + .ok_or_else(|| no_vocabulary("the APR file (no embedded vocabulary)"))?; println!("{}", "Q4K CPU inference ready".green()); diff --git a/crates/apr-cli/src/commands/serve/handlers_include_01.rs b/crates/apr-cli/src/commands/serve/handlers_include_01.rs index 537efcdb9b..c03979450b 100644 --- a/crates/apr-cli/src/commands/serve/handlers_include_01.rs +++ b/crates/apr-cli/src/commands/serve/handlers_include_01.rs @@ -50,15 +50,10 @@ fn start_apr_server_gpu( ); // Extract vocabulary from embedded APR metadata - let vocab = mapped.metadata.get_embedded_vocabulary().unwrap_or_else(|| { - let vocab_size = mapped.metadata.vocab_size.unwrap_or(32000); - eprintln!("Warning: No embedded vocabulary in APR, using placeholder tokens"); - let mut v: Vec = (0..vocab_size).map(|i| format!("token{i}")).collect(); - if !v.is_empty() { - v[0] = "".to_string(); - } - v - }); + let vocab = mapped + .metadata + .get_embedded_vocabulary() + .ok_or_else(|| no_vocabulary("the APR file (no embedded vocabulary)"))?; // GH-88: Extract merge rules for proper BPE tokenization (HuggingFace models) let merges = mapped.metadata.get_embedded_merges(); @@ -126,13 +121,9 @@ fn start_apr_q4k_server_gpu( .and_then(|m| m.load_embedded_tokenizer()) .map(|t| (t.id_to_token.clone(), None)) }) - .unwrap_or_else(|| { - println!( - "{}", - "Warning: No vocabulary found, using placeholder tokens".yellow() - ); - ((0..151936).map(|i| format!("token{i}")).collect(), None) - }); + .ok_or_else(|| { + no_vocabulary("the APR model (no sibling tokenizer.json, no embedded tokenizer)") + })?; println!(" Vocab: {} tokens", vocab.len()); if let Some(eos) = eos_id { @@ -200,15 +191,10 @@ fn start_safetensors_server_gpu( ); // Extract vocabulary from embedded APR metadata - let vocab = mapped.metadata.get_embedded_vocabulary().unwrap_or_else(|| { - let vocab_size = mapped.metadata.vocab_size.unwrap_or(32000); - eprintln!("Warning: No embedded vocabulary in APR, using placeholder tokens"); - let mut v: Vec = (0..vocab_size).map(|i| format!("token{i}")).collect(); - if !v.is_empty() { - v[0] = "".to_string(); - } - v - }); + let vocab = mapped + .metadata + .get_embedded_vocabulary() + .ok_or_else(|| no_vocabulary("the APR file (no embedded vocabulary)"))?; // GH-88: Extract merge rules for proper BPE tokenization (HuggingFace models) let merges = mapped.metadata.get_embedded_merges(); @@ -281,15 +267,10 @@ fn start_safetensors_server_cpu_quantized( ); // Extract vocabulary from embedded APR metadata - let vocab = mapped.metadata.get_embedded_vocabulary().unwrap_or_else(|| { - let vocab_size = mapped.metadata.vocab_size.unwrap_or(32000); - eprintln!("Warning: No embedded vocabulary in APR, using placeholder tokens"); - let mut v: Vec = (0..vocab_size).map(|i| format!("token{i}")).collect(); - if !v.is_empty() { - v[0] = "".to_string(); - } - v - }); + let vocab = mapped + .metadata + .get_embedded_vocabulary() + .ok_or_else(|| no_vocabulary("the APR file (no embedded vocabulary)"))?; let _ = std::fs::remove_file(&tmp_apr); diff --git a/crates/aprender-serve/book/src/examples/api-server.md b/crates/aprender-serve/book/src/examples/api-server.md index ac4bc3f8bc..d62940cca5 100644 --- a/crates/aprender-serve/book/src/examples/api-server.md +++ b/crates/aprender-serve/book/src/examples/api-server.md @@ -31,7 +31,10 @@ let config = ModelConfig { eps: 1e-5, }; let model = Model::new(config)?; -let tokenizer = BPETokenizer::new(vocab, merges, "")?; +// The unknown token is the model's own, if it has one: LLaMA-style vocabularies declare +// ``, byte-level ones (GPT-2, Qwen) have none. BPETokenizer requires neither (#3609). +let unk = realizar::tokenizer::vocabulary_unk_token(&vocab); // Some("") or None +let tokenizer = BPETokenizer::new(vocab, merges, unk)?; // Create server let state = AppState::new(model, tokenizer); diff --git a/crates/aprender-serve/examples/model_cache.rs b/crates/aprender-serve/examples/model_cache.rs index d97ff30163..e5c29d0125 100644 --- a/crates/aprender-serve/examples/model_cache.rs +++ b/crates/aprender-serve/examples/model_cache.rs @@ -35,7 +35,10 @@ fn create_demo_model(vocab_size: usize) -> (Model, BPETokenizer) { } }) .collect(); - let tokenizer = BPETokenizer::new(vocab, vec![], "").expect("Failed to create tokenizer"); + // This demo vocabulary declares `` at id 0, so it passes it. The unknown token is + // the model's own: a byte-level vocabulary (GPT-2, Qwen) has none and passes `None` (#3609). + let tokenizer = + BPETokenizer::new(vocab, vec![], Some("")).expect("Failed to create tokenizer"); (model, tokenizer) } diff --git a/crates/aprender-serve/src/api/mod_app_state_gpu.rs b/crates/aprender-serve/src/api/mod_app_state_gpu.rs index acee3111ff..233849b7ca 100644 --- a/crates/aprender-serve/src/api/mod_app_state_gpu.rs +++ b/crates/aprender-serve/src/api/mod_app_state_gpu.rs @@ -28,7 +28,8 @@ impl AppState { } }) .collect(); - let tokenizer = BPETokenizer::new(vocab, vec![], "")?; + let unk = crate::tokenizer::vocabulary_unk_token(&vocab); // #3609: never a literal + let tokenizer = BPETokenizer::new(vocab, vec![], unk)?; let (audit_logger, audit_sink) = create_audit_state(); Ok(Self { @@ -85,7 +86,8 @@ impl AppState { cached_model: crate::gguf::OwnedQuantizedModelCachedSync, vocab: Vec, ) -> Result { - let tokenizer = BPETokenizer::new(vocab, vec![], "")?; + let unk = crate::tokenizer::vocabulary_unk_token(&vocab); // #3609: never a literal + let tokenizer = BPETokenizer::new(vocab, vec![], unk)?; let (audit_logger, audit_sink) = create_audit_state(); Ok(Self { @@ -140,7 +142,8 @@ impl AppState { quantized_model: crate::gguf::OwnedQuantizedModel, vocab: Vec, ) -> Result { - let tokenizer = BPETokenizer::new(vocab, vec![], "")?; + let unk = crate::tokenizer::vocabulary_unk_token(&vocab); // #3609: never a literal + let tokenizer = BPETokenizer::new(vocab, vec![], unk)?; // PMAT-181: Cache architecture for chat template auto-detection. // Qwen3 models get Qwen3NoThinkTemplate (disables thinking mode). @@ -208,7 +211,8 @@ impl AppState { cuda_model: crate::gguf::OwnedQuantizedModelCuda, vocab: Vec, ) -> Result { - let tokenizer = BPETokenizer::new(vocab, vec![], "")?; + let unk = crate::tokenizer::vocabulary_unk_token(&vocab); // #3609: never a literal + let tokenizer = BPETokenizer::new(vocab, vec![], unk)?; // PMAT-073: Cache architecture at construction to avoid RwLock in hot path. // model_architecture() was blocking HTTP handlers for ~2s due to read lock // contention with the batch scheduler's write lock. @@ -265,7 +269,8 @@ impl AppState { vocab: Vec, merges: Vec<(String, String)>, ) -> Result { - let tokenizer = BPETokenizer::with_merges(vocab, merges, "")?; + let unk = crate::tokenizer::vocabulary_unk_token(&vocab); // #3609: never a literal + let tokenizer = BPETokenizer::with_merges(vocab, merges, unk)?; let arch = Some(cuda_model.model().config.architecture.clone()); let eos = cuda_model.model().config.eos_token_id; @@ -328,7 +333,8 @@ impl AppState { transformer: crate::apr_transformer::AprTransformer, vocab: Vec, ) -> Result { - let tokenizer = BPETokenizer::new(vocab, vec![], "")?; + let unk = crate::tokenizer::vocabulary_unk_token(&vocab); // #3609: never a literal + let tokenizer = BPETokenizer::new(vocab, vec![], unk)?; let (audit_logger, audit_sink) = create_audit_state(); Ok(Self { @@ -555,7 +561,8 @@ impl AppState { vocab: Vec, eos_id: Option, ) -> Result { - let tokenizer = BPETokenizer::new(vocab, vec![], "")?; + let unk = crate::tokenizer::vocabulary_unk_token(&vocab); // #3609: never a literal + let tokenizer = BPETokenizer::new(vocab, vec![], unk)?; let (audit_logger, audit_sink) = create_audit_state(); Ok(Self { model: None, @@ -601,7 +608,8 @@ impl AppState { model: crate::safetensors_cuda::SafeTensorsCudaModel, vocab: Vec, ) -> Result { - let tokenizer = BPETokenizer::new(vocab, vec![], "")?; + let unk = crate::tokenizer::vocabulary_unk_token(&vocab); // #3609: never a literal + let tokenizer = BPETokenizer::new(vocab, vec![], unk)?; let metrics = Arc::new(MetricsCollector::new()); let (audit_logger, audit_sink) = create_audit_state(); diff --git a/crates/aprender-serve/src/api/mod_app_state_new.rs b/crates/aprender-serve/src/api/mod_app_state_new.rs index d5984e9c8a..18b11b0b8c 100644 --- a/crates/aprender-serve/src/api/mod_app_state_new.rs +++ b/crates/aprender-serve/src/api/mod_app_state_new.rs @@ -227,8 +227,9 @@ impl AppState { } }) .collect(); + let unk = crate::tokenizer::vocabulary_unk_token(&vocab); // #3609: never a literal let tokenizer = - BPETokenizer::new(vocab, vec![], "").expect("Failed to create tokenizer"); + BPETokenizer::new(vocab, vec![], unk).expect("Failed to create tokenizer"); let (audit_logger, audit_sink) = create_audit_state(); Self { @@ -298,7 +299,8 @@ impl AppState { } }) .collect(); - let tokenizer = BPETokenizer::new(vocab, vec![], "")?; + let unk = crate::tokenizer::vocabulary_unk_token(&vocab); // #3609: never a literal + let tokenizer = BPETokenizer::new(vocab, vec![], unk)?; // Create demo APR model (real inference, not mock) // Simple model: sum of inputs with bias @@ -424,7 +426,8 @@ impl AppState { } }) .collect(); - let tokenizer = BPETokenizer::new(vocab, vec![], "")?; + let unk = crate::tokenizer::vocabulary_unk_token(&vocab); // #3609: never a literal + let tokenizer = BPETokenizer::new(vocab, vec![], unk)?; let (audit_logger, audit_sink) = create_audit_state(); Ok(Self { @@ -480,7 +483,8 @@ impl AppState { gpu_model: crate::gpu::GpuModel, vocab: Vec, ) -> Result { - let tokenizer = BPETokenizer::new(vocab, vec![], "")?; + let unk = crate::tokenizer::vocabulary_unk_token(&vocab); // #3609: never a literal + let tokenizer = BPETokenizer::new(vocab, vec![], unk)?; let (audit_logger, audit_sink) = create_audit_state(); Ok(Self { @@ -545,7 +549,8 @@ impl AppState { } }) .collect(); - let tokenizer = BPETokenizer::new(vocab, vec![], "")?; + let unk = crate::tokenizer::vocabulary_unk_token(&vocab); // #3609: never a literal + let tokenizer = BPETokenizer::new(vocab, vec![], unk)?; let (audit_logger, audit_sink) = create_audit_state(); Ok(Self { diff --git a/crates/aprender-serve/src/cli/mod_server_commands.rs b/crates/aprender-serve/src/cli/mod_server_commands.rs index 81d5d99c7e..a4e0fcf4d8 100644 --- a/crates/aprender-serve/src/cli/mod_server_commands.rs +++ b/crates/aprender-serve/src/cli/mod_server_commands.rs @@ -6,6 +6,20 @@ mod server_commands { use super::Result; + /// #3609: the refusal every serve path gives for a model with no vocabulary. + /// + /// These paths used to substitute `token0..tokenN` and serve that. Placeholder tokens are + /// not a tokenizer: a model with no vocabulary refuses by name, it does not load. + fn no_vocabulary(what: &str) -> crate::error::RealizarError { + crate::error::RealizarError::UnsupportedOperation { + operation: "load_vocabulary".to_string(), + reason: format!( + "{what} has no vocabulary to tokenize with, so there is nothing to serve; \ + refusing to substitute placeholder tokens (#3609)" + ), + } + } + /// Result of preparing server state (returned by `prepare_serve_state`) pub struct PreparedServer { /// The prepared AppState for the server @@ -95,12 +109,10 @@ mod server_commands { println!(" Layers: {}", quantized_model.layers.len()); // Extract vocabulary from GGUF for proper token decoding - let vocab = mapped.model.vocabulary().unwrap_or_else(|| { - eprintln!(" Warning: No vocabulary in GGUF, using placeholder tokens"); - (0..quantized_model.config.vocab_size) - .map(|i| format!("token{i}")) - .collect() - }); + let vocab = mapped + .model + .vocabulary() + .ok_or_else(|| no_vocabulary("the GGUF (no tokenizer.ggml.tokens)"))?; println!(" Vocab loaded: {} tokens", vocab.len()); println!(); @@ -336,14 +348,9 @@ mod server_commands { } })?; - let vocab_size = cuda_model.config().vocab_size; - #[allow(clippy::map_unwrap_or)] let vocab = crate::apr::AprV2Model::load_tokenizer_from_sibling(model_path_obj) .map(|(v, _, _)| v) - .unwrap_or_else(|| { - println!(" Warning: No tokenizer.json found, using simple vocabulary"); - (0..vocab_size).map(|i| format!("token{i}")).collect() - }); + .ok_or_else(|| no_vocabulary("the SafeTensors model (no sibling tokenizer.json)"))?; println!(" Vocab size: {}", vocab.len()); @@ -377,15 +384,9 @@ mod server_commands { println!(" Layers: {}", transformer.config.num_layers); println!(" Hidden: {}", transformer.config.hidden_dim); - #[allow(clippy::map_unwrap_or)] let vocab = crate::apr::AprV2Model::load_tokenizer_from_sibling(model_path_obj) .map(|(v, _, _)| v) - .unwrap_or_else(|| { - println!(" Warning: No tokenizer.json found, using simple vocabulary"); - (0..transformer.config.vocab_size) - .map(|i| format!("token{i}")) - .collect() - }); + .ok_or_else(|| no_vocabulary("the SafeTensors model (no sibling tokenizer.json)"))?; println!(" Vocab size: {}", vocab.len()); println!(" Mode: CPU (F32 inference)"); @@ -449,12 +450,7 @@ mod server_commands { .and_then(|m| m.load_embedded_tokenizer()) .map(|t| t.id_to_token.clone()) }) - .unwrap_or_else(|| { - println!(" Warning: No vocabulary found, using simple vocabulary"); - (0..model.config.vocab_size) - .map(|i| format!("token{i}")) - .collect() - }); + .ok_or_else(|| no_vocabulary("the APR model (no sibling tokenizer.json, no embedded tokenizer)"))?; println!(" Vocab size: {}", vocab.len()); println!(" Mode: CPU (per-tensor scratch dequant)"); @@ -494,10 +490,7 @@ mod server_commands { let vocab = apr_v2.as_ref() .and_then(|m| m.metadata().get_embedded_vocabulary()) .or_else(|| crate::apr::AprV2Model::load_tokenizer_from_sibling(path).map(|(v, _, _)| v)) - .unwrap_or_else(|| { - println!(" Warning: No tokenizer found"); - (0..model.config.vocab_size).map(|i| format!("token{i}")).collect() - }); + .ok_or_else(|| no_vocabulary("the APR model (no embedded vocabulary, no sibling tokenizer.json)"))?; println!(" Vocab size: {}", vocab.len()); // PMAT-785: fail-closed quant gate at GPU-resident construction. An APR @@ -556,10 +549,7 @@ mod server_commands { .and_then(|m| m.load_embedded_tokenizer()) .map(|t| (t.id_to_token.clone(), None)) }) - .unwrap_or_else(|| { - println!(" Warning: No vocabulary found, using simple vocabulary"); - ((0..151936).map(|i| format!("token{i}")).collect(), None) - }); + .ok_or_else(|| no_vocabulary("the APR model (no sibling tokenizer.json, no embedded tokenizer)"))?; println!(" Vocab: {} tokens", vocab.len()); if let Some(eos) = eos_id { diff --git a/crates/aprender-serve/src/tokenizer.rs b/crates/aprender-serve/src/tokenizer.rs index 2124a316a3..5ef9de3bcb 100644 --- a/crates/aprender-serve/src/tokenizer.rs +++ b/crates/aprender-serve/src/tokenizer.rs @@ -130,8 +130,13 @@ pub struct BPETokenizer { token_to_id: HashMap, /// ID to token mapping id_to_token: HashMap, - /// Unknown token ID - unk_token_id: u32, + /// The model's unknown token, or `None` when it has none (#3609). + /// + /// An unknown token belongs to a model family, not to BPE. LLaMA-style vocabularies + /// declare ``; byte-level ones (GPT-2, Qwen) need none, because every byte has a + /// token, and Qwen3.5 declares none. When this is `None`, construction has proved + /// that every byte has a token, so `encode` never needs a stand-in. + unk_token_id: Option, /// Cached vocabulary size (PMAT-805: vLLM pattern) vocab_size: usize, /// Cached maximum token ID (PMAT-805: vLLM pattern) @@ -151,15 +156,21 @@ impl BPETokenizer { /// /// * `vocab` - List of tokens (index = token ID) /// * `merges` - List of merge pairs in priority order - /// * `unk_token` - Unknown token string + /// * `unk_token` - The model's unknown token if it has one (`""` or + /// `Some("")`), or `None` if it has none. Having one is a property of the + /// model, not a requirement of this type (#3609). /// /// # Errors /// - /// Returns error if vocabulary is empty or unknown token not found - pub fn new( + /// Returns an error if the vocabulary is empty, or if `unk_token` names a token the + /// vocabulary does not contain. With no unknown token, also returns an error naming + /// the first byte that has neither a `<0xNN>` token nor a byte-level glyph token, + /// because `encode` would have nothing to emit for it. Absent means absent: no + /// stand-in is synthesised and no input is dropped. + pub fn new<'a>( vocab: Vec, _merges: Vec<(String, String)>, - unk_token: &str, + unk_token: impl Into>, ) -> Result { if vocab.is_empty() { return Err(RealizarError::UnsupportedOperation { @@ -181,12 +192,31 @@ impl BPETokenizer { } let unk_token_id = - *token_to_id - .get(unk_token) - .ok_or_else(|| RealizarError::UnsupportedOperation { - operation: "create_bpe_tokenizer".to_string(), - reason: format!("Unknown token '{unk_token}' not in vocabulary"), - })?; + match unk_token.into() { + Some(unk) => Some(*token_to_id.get(unk).ok_or_else(|| { + RealizarError::UnsupportedOperation { + operation: "create_bpe_tokenizer".to_string(), + reason: format!( + "unknown token '{unk}' was named, but the vocabulary does not contain it" + ), + } + })?), + None => { + if let Some(byte) = + (0u8..=255).find(|&b| byte_token_id(&token_to_id, b).is_none()) + { + return Err(RealizarError::UnsupportedOperation { + operation: "create_bpe_tokenizer".to_string(), + reason: format!( + "the model has no unknown token, and byte 0x{byte:02X} has neither a \ + '<0x{byte:02X}>' token nor a byte-level glyph token, so encode would \ + have nothing to emit for it" + ), + }); + } + None + }, + }; // PMAT-805: Cache vocabulary properties at construction (vLLM pattern) // This avoids repeated HashMap operations during inference @@ -209,12 +239,28 @@ impl BPETokenizer { /// HuggingFace vocabularies require merge-based BPE (not greedy longest-match). /// When merge rules are provided, `encode()` delegates to `bpe_encode()` from /// `apr::tokenizer` which handles special tokens atomically and applies merges. - pub fn with_merges( + pub fn with_merges<'a>( vocab: Vec, merges: Vec<(String, String)>, - unk_token: &str, + unk_token: impl Into>, ) -> Result { let mut tokenizer = Self::new(vocab, vec![], unk_token)?; + // #3609: the merge path (`bpe_encode`) maps each byte to its byte-level glyph and + // nothing else, so with no unknown token every glyph must be in the vocabulary, + // or that byte would be dropped from the input. + if tokenizer.unk_token_id.is_none() { + if let Some(byte) = + (0u8..=255).find(|&b| byte_glyph_id(&tokenizer.token_to_id, b).is_none()) + { + return Err(RealizarError::UnsupportedOperation { + operation: "create_bpe_tokenizer".to_string(), + reason: format!( + "the model has no unknown token, and byte 0x{byte:02X} has no byte-level \ + glyph token, which merge-based encoding needs" + ), + }); + } + } // Extract special tokens from vocabulary for atomic tokenization let special_tokens: HashMap = tokenizer .token_to_id @@ -317,12 +363,7 @@ impl BPETokenizer { let ch_len = ch.len_utf8(); for byte in remaining[..ch_len].bytes() { - let byte_token = format!("<0x{byte:02X}>"); - if let Some(&id) = self.token_to_id.get(&byte_token) { - tokens.push(id); - } else { - tokens.push(self.unk_token_id); - } + tokens.push(self.byte_fallback_id(byte)); } remaining = &remaining[ch_len..]; } @@ -433,6 +474,27 @@ impl BPETokenizer { self.token_to_id.get(token).copied() } + /// The unknown token's ID, or `None` when the model has none (#3609). + #[must_use] + pub fn unk_token_id(&self) -> Option { + self.unk_token_id + } + + /// The token for one byte that no vocabulary entry covers: its `<0xNN>` token, else the + /// model's unknown token, else its byte-level glyph (#3609). With no unknown token, + /// `new` proved that every byte has a `<0xNN>` or a glyph token, so this cannot miss. + fn byte_fallback_id(&self, byte: u8) -> u32 { + if let Some(&id) = self.token_to_id.get(&format!("<0x{byte:02X}>")) { + return id; + } + if let Some(unk) = self.unk_token_id { + return unk; + } + byte_glyph_id(&self.token_to_id, byte).expect( + "BPETokenizer::new proved every byte encodable without an unknown token (#3609)", + ) + } + /// Get token for a token ID #[must_use] pub fn get_token(&self, id: u32) -> Option<&str> { @@ -450,6 +512,38 @@ impl BPETokenizer { } } +/// The token for one raw byte that no vocabulary token covers: SentencePiece's `<0xNN>` +/// byte-fallback token, else the GPT-2 byte-level glyph (#3609). +fn byte_token_id(token_to_id: &HashMap, byte: u8) -> Option { + token_to_id + .get(&format!("<0x{byte:02X}>")) + .copied() + .or_else(|| byte_glyph_id(token_to_id, byte)) +} + +/// The GPT-2 byte-level glyph token for one byte (`é`'s 0xC3 → `Ã`), if the vocabulary has it. +fn byte_glyph_id(token_to_id: &HashMap, byte: u8) -> Option { + let glyph = crate::gguf::utils::gpt2_byte_to_unicode(byte); + token_to_id + .get(glyph.encode_utf8(&mut [0u8; 4]) as &str) + .copied() +} + +/// The unknown token to pass to [`BPETokenizer::new`] when the caller has only a token list. +/// +/// That is the vocabulary's own `` entry if it has one, and `None` if it does not. +/// Nothing is synthesised: a vocabulary without `` (Qwen3.5, GPT-2) gets no unknown +/// token, and `BPETokenizer::new` then proves that every byte has a token (#3609). +/// +/// This finds the token by NAME. It is the interim for callers that receive a `Vec` +/// and nothing else. The model's DECLARATION (`tokenizer.ggml.unknown_token_id`, or +/// `tokenizer.json`'s `unk_token`) is the design, and threading it through every loader +/// is #3675. +#[must_use] +pub fn vocabulary_unk_token(vocab: &[String]) -> Option<&'static str> { + vocab.iter().any(|t| t == "").then_some("") +} + /// Viterbi algorithm state: `(best_score, best_token)` at each position type ViterbiState = (Vec, Vec>); diff --git a/crates/aprender-serve/src/tokenizer_04.rs b/crates/aprender-serve/src/tokenizer_04.rs index 1771c11c29..edbac1fdbd 100644 --- a/crates/aprender-serve/src/tokenizer_04.rs +++ b/crates/aprender-serve/src/tokenizer_04.rs @@ -6,3 +6,7 @@ mod tokenizer_tests_bpe_encode; #[cfg(test)] #[path = "tokenizer_tests_sentencepiece_viterbi.rs"] mod tokenizer_tests_sentencepiece_viterbi; + +#[cfg(test)] +#[path = "tokenizer_tests_unk_3609.rs"] +mod tokenizer_tests_unk_3609; diff --git a/crates/aprender-serve/src/tokenizer_tests_unk_3609.rs b/crates/aprender-serve/src/tokenizer_tests_unk_3609.rs new file mode 100644 index 0000000000..8bed72618b --- /dev/null +++ b/crates/aprender-serve/src/tokenizer_tests_unk_3609.rs @@ -0,0 +1,161 @@ +//! #3609: an unknown token is a property of a model family, not of BPE. +//! +//! LLaMA-style vocabularies declare ``. Byte-level ones (GPT-2, Qwen) have no need +//! for one, because every byte already has a token, and Qwen3.5 declares none at all. +//! These rows pin both directions of the fix: a model WITH an unknown token still +//! resolves it (the fix is not "ignore unk"), and a model WITHOUT one tokenizes instead +//! of being refused. With no unknown token, nothing is synthesised and nothing is +//! dropped: a byte with no token is refused by name at construction. +//! +//! The two `*.gguf-header` fixtures are REAL headers, cut down by +//! `tests/fixtures/gguf-header-slices/generate.py`: every key and scalar value verbatim, +//! only the per-token arrays sliced (see that directory's MANIFEST.json). + +use crate::gguf::{GGUFModel, GGUFValue}; +use crate::tokenizer::{vocabulary_unk_token, BPETokenizer}; + +const QWEN35_HEADER: &[u8] = + include_bytes!("../tests/fixtures/gguf-header-slices/qwen3.5-0.8b.gguf-header"); +const TINYLLAMA_HEADER: &[u8] = + include_bytes!("../tests/fixtures/gguf-header-slices/tinyllama-1.1b-chat.gguf-header"); + +const UNK_ID_KEY: &str = "tokenizer.ggml.unknown_token_id"; +const EOS_ID_KEY: &str = "tokenizer.ggml.eos_token_id"; + +fn header(bytes: &[u8]) -> GGUFModel { + GGUFModel::from_bytes(bytes).expect("a real GGUF header parses") +} + +fn u32_key(model: &GGUFModel, key: &str) -> Option { + match model.metadata.get(key) { + Some(GGUFValue::UInt32(v)) => Some(*v), + _ => None, + } +} + +/// All 256 GPT-2 byte-level glyphs: the first 256 tokens of a GPT-2/Qwen vocabulary. +fn byte_level_vocab() -> Vec { + (0u8..=255) + .map(|b| crate::gguf::utils::gpt2_byte_to_unicode(b).to_string()) + .collect() +} + +fn id_of(tok: &BPETokenizer, token: &str) -> u32 { + tok.get_token_id(token).expect("token in vocabulary") +} + +// --------------------------------------------------------------------------- +// The real Qwen3.5 header: declares EOS, declares no unknown token, has no ``. +// --------------------------------------------------------------------------- + +#[test] +fn qwen35_real_header_declares_eos_and_no_unknown_token() { + let model = header(QWEN35_HEADER); + assert_eq!(u32_key(&model, EOS_ID_KEY), Some(248_046), "the real EOS id, verbatim"); + assert!( + !model.metadata.contains_key(UNK_ID_KEY), + "Qwen3.5 declares no tokenizer.ggml.unknown_token_id at all" + ); + let vocab = model.vocabulary().expect("the header carries a vocabulary"); + assert!(!vocab.iter().any(|t| t == ""), "and has no token"); +} + +/// THE row #3609 is about. Mutation: restore the unconditional +/// `token_to_id.get("").ok_or_else(..)` in `BPETokenizer::new` and this goes RED. +#[test] +fn qwen35_real_vocabulary_builds_a_tokenizer_with_no_unknown_token() { + let vocab = header(QWEN35_HEADER).vocabulary().expect("vocabulary"); + let unk = vocabulary_unk_token(&vocab); + assert_eq!(unk, None, "nothing named , so nothing is passed"); + let tok = BPETokenizer::new(vocab, vec![], unk) + .expect("a real vocabulary without is not refused (#3609)"); + assert_eq!(tok.unk_token_id(), None, "absent means absent: no synthesised "); + // With no unknown token, a character the vocabulary lacks (CJK here) goes through + // the per-byte glyph fallback and round-trips. `é` is NOT a row: the greedy path + // matches it as the byte-0xE9 glyph token before any fallback runs. That is a + // pre-existing greedy-path defect, independent of the unknown token, filed as #3677. + for text in ["Hi there", "世界", "a\nb"] { + let ids = tok.encode(text); + assert!(!ids.is_empty(), "{text:?} encodes to something"); + assert_eq!(tok.decode(&ids).expect("decode"), text, "{text:?} round-trips"); + } +} + +// --------------------------------------------------------------------------- +// The real TinyLlama header: declares unknown_token_id = 0 = ``. It must still +// resolve (the other direction: the fix is not "ignore unk"). +// --------------------------------------------------------------------------- + +#[test] +fn tinyllama_real_header_declares_unk_and_it_still_resolves() { + let model = header(TINYLLAMA_HEADER); + assert_eq!(u32_key(&model, UNK_ID_KEY), Some(0), "the real declaration, verbatim"); + let vocab = model.vocabulary().expect("vocabulary"); + assert_eq!(vocab[0], "", "id 0 is the declared "); + let unk = vocabulary_unk_token(&vocab); + assert_eq!(unk, Some("")); + let tok = BPETokenizer::new(vocab, vec![], unk).expect("tokenizer"); + assert_eq!(tok.unk_token_id(), Some(0), "the declared unknown token still resolves"); +} + +// --------------------------------------------------------------------------- +// The case table on small vocabularies. +// --------------------------------------------------------------------------- + +#[test] +fn declared_unknown_token_is_still_emitted_for_an_unencodable_byte() { + let tok = BPETokenizer::new(vec!["".into(), "a".into()], vec![], "") + .expect("tokenizer"); + assert_eq!(tok.encode("é"), vec![0, 0], "unchanged: two bytes, two "); +} + +#[test] +fn a_named_unknown_token_missing_from_the_vocabulary_is_refused() { + let err = BPETokenizer::new(vec!["a".into(), "b".into()], vec![], "") + .expect_err("naming an unknown token the vocabulary lacks is an error"); + assert!(err.to_string().contains("''"), "names the token: {err}"); +} + +#[test] +fn no_unknown_token_and_byte_level_glyphs_encode_every_byte() { + let tok = BPETokenizer::new(byte_level_vocab(), vec![], None).expect("tokenizer"); + // '世' is E4 B8 96: not a token, so each byte takes its glyph. + let glyphs: Vec = [0xE4u8, 0xB8, 0x96] + .iter() + .map(|&b| id_of(&tok, &crate::gguf::utils::gpt2_byte_to_unicode(b).to_string())) + .collect(); + assert_eq!(tok.encode("世"), glyphs); + assert_eq!(tok.decode(&tok.encode("世")).expect("decode"), "世"); +} + +#[test] +fn no_unknown_token_and_byte_fallback_tokens_encode_every_byte() { + let vocab: Vec = (0u8..=255).map(|b| format!("<0x{b:02X}>")).collect(); + let tok = BPETokenizer::new(vocab, vec![], None).expect("tokenizer"); + assert_eq!( + tok.encode("é"), + vec![id_of(&tok, "<0xC3>"), id_of(&tok, "<0xA9>")] + ); +} + +/// The cop's row: absent means absent at encode time too. No unknown token, and one byte +/// with neither a `<0xNN>` token nor a glyph token, is refused by name. It is never +/// dropped and never emitted as nothing. +#[test] +fn no_unknown_token_and_a_byte_with_no_token_is_refused_by_name() { + let glyph_a9 = crate::gguf::utils::gpt2_byte_to_unicode(0xA9).to_string(); + let vocab: Vec = byte_level_vocab() + .into_iter() + .filter(|t| *t != glyph_a9) + .collect(); + let err = BPETokenizer::new(vocab, vec![], None) + .expect_err("a byte nothing can encode must refuse the tokenizer"); + assert!(err.to_string().contains("0xA9"), "names the byte: {err}"); +} + +#[test] +fn vocabulary_unk_token_names_only_an_unk_that_is_present() { + assert_eq!(vocabulary_unk_token(&["".into(), "a".into()]), Some("")); + assert_eq!(vocabulary_unk_token(&["a".into(), "b".into()]), None); + assert_eq!(vocabulary_unk_token(&[]), None); +} diff --git a/crates/aprender-serve/tests/fixtures/gguf-header-slices/MANIFEST.json b/crates/aprender-serve/tests/fixtures/gguf-header-slices/MANIFEST.json new file mode 100644 index 0000000000..889f7ab19f --- /dev/null +++ b/crates/aprender-serve/tests/fixtures/gguf-header-slices/MANIFEST.json @@ -0,0 +1,56 @@ +{ + "qwen3.5-0.8b.gguf-header": { + "keep": 1024, + "keys": 46, + "output_bytes": 39063, + "output_sha256": "0855f96190fcfced331768da28b0bd59117c1a825175220b9de1dddac38cb082", + "sliced_arrays": { + "tokenizer.ggml.merges": { + "kept": 1024, + "source_len": 247587 + }, + "tokenizer.ggml.token_type": { + "kept": 1024, + "source_len": 248320 + }, + "tokenizer.ggml.tokens": { + "kept": 1024, + "source_len": 248320 + } + }, + "source_file": "Qwen3.5-0.8B-IQ4_XS.gguf", + "source_gguf_version": 3, + "source_header_bytes": 10943082, + "source_header_sha256": "2eb5e86ecd53b7ce44da7d43c5b20cdd8962ef87444b9e4f963d0f0a0fdf5d36", + "source_tensor_count": 320 + }, + "tinyllama-1.1b-chat.gguf-header": { + "keep": 512, + "keys": 23, + "output_bytes": 18724, + "output_sha256": "e0e4c5887f9b5de6837620b3caa9681e0db92724aaa7a3fa849ee81a1fde71e9", + "sliced_arrays": { + "tokenizer.ggml.merges": { + "kept": 512, + "source_len": 61249 + }, + "tokenizer.ggml.scores": { + "kept": 512, + "source_len": 32000 + }, + "tokenizer.ggml.token_type": { + "kept": 512, + "source_len": 32000 + }, + "tokenizer.ggml.tokens": { + "kept": 512, + "source_len": 32000 + } + }, + "source_file": "tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf", + "source_gguf_version": 3, + "source_header_bytes": 1697530, + "source_header_sha256": "2d44149d9173954e4dfb8b1eca58823d0951e88ac35de4e56abe71f6454b39a9", + "source_tensor_count": 201 + } +} diff --git a/crates/aprender-serve/tests/fixtures/gguf-header-slices/generate.py b/crates/aprender-serve/tests/fixtures/gguf-header-slices/generate.py new file mode 100644 index 0000000000..dace075a29 --- /dev/null +++ b/crates/aprender-serve/tests/fixtures/gguf-header-slices/generate.py @@ -0,0 +1,135 @@ +#!/usr/bin/env python3 +"""Cut a REAL GGUF header down to a committable fixture (#3609). + +A GGUF's metadata section carries its tokenizer: the vocabulary, the merges, and +the keys that say which special tokens the model declares. The real Qwen3.5-0.8B +header is 10.9 MB (248,320 tokens, 247,587 merges), too large to commit. What +the #3609 tests need from it is the real KEY SET and real values: Qwen3.5 +declares `tokenizer.ggml.eos_token_id` and has NO `tokenizer.ggml.unknown_token_id` +key at all, while TinyLlama declares `unknown_token_id = 0` (``). + +So the output is the source header with: + * every key, and every scalar and string value, copied byte-for-byte; + * the tokenizer's per-token arrays (tokens, token_type, scores, merges) cut + to their first --keep entries, which is the ONLY change; + * tensor_count = 0 and no tensor data (a header, not a model). + +The truncation is named here and in MANIFEST.json, never hidden. MANIFEST.json +records the source file's name, the sha256 of its header bytes, and the output's +sha256, so a reader can check that the fixture came from what it claims to. + +usage: generate.py SOURCE.gguf OUT.gguf-header --keep N +""" +import argparse +import hashlib +import json +import os +import struct +import sys + +# The per-token arrays: the only values this script changes. +SLICED = { + "tokenizer.ggml.tokens", + "tokenizer.ggml.token_type", + "tokenizer.ggml.scores", + "tokenizer.ggml.merges", +} +SCALAR_SIZE = {0: 1, 1: 1, 2: 2, 3: 2, 4: 4, 5: 4, 6: 4, 7: 1, 10: 8, 11: 8, 12: 8} +STRING, ARRAY = 8, 9 + + +def read_exact(f, n): + b = f.read(n) + if len(b) != n: + sys.exit(f"truncated source: wanted {n} bytes, got {len(b)}") + return b + + +def read_string_raw(f): + """The string's length prefix and bytes, verbatim.""" + head = read_exact(f, 8) + (n,) = struct.unpack("!i6nE%}`21S!sS>ALH%5l*hPnzPl;@)qb-@kuY z3_kb#?^tR*J{rbWziEEGU-yb?xzQl+$40#VRYR3MzxizTg%^L%62jelRD4|fWpeLN z^1--#ujqAlkBr>kGROD&U{uxPgWtD=euSo-D&H+CYy9K$yGF?GUTRsD-C{6`bpOQs zRzA$T2gT-RvuBs7m^aR|dei(6XzC3nE>MdD21)v zs@~n|=jCAQ#`UY$Zr-`JESIJj)}2Wc$^H``k9{&@{WJ4coLG}=C(aK@m<-0o{BOX& zKXQ7zXR}kLBACRU__&UBv!4&j_wsRB4^mMZ>p%H%9aBcv=M6ut7^z@nd7ST$ej9KY z0miC*s1ap9-!C@D#l!LDxQ>5$XQt^HrlfnXBT<<+|6s&$=FGfBw;ovkPE`!{#|JZI z`tdV=c(GB@-z|DQOn%x7ZU5+Ev%LZ;yjM4edDBZ<{;o4;{@7y6d^|3o^7SA)DDs{Y zXiQQw<^-;4+PU|@DE*BmTQ~Kv$a-ZTY7gw45o`edMgiLLp7D3)%x?#5|F|lagc7mr zHO2joDV$l4umAFAzOshD_BlaVl^+#NXHYl&PSYQCis7iN>Vb9pwa@?D^QW?YuQ*Dj zgvq5+ofzRkUQMv(eU8xh4@F``&S=!fnixFj+$)+vQ5mQ9Z;3N&H5%vRqGMLX8hjo9 zR-oE%>dDY2#+EV{*_=85_pQh(*)KCtP|KEaGt<31TGNRB#;r13 zDFV+E_$dOPC-4OVf11EW0zXaQ5`k?3mkE54zzYPf5O|TmRRS*&xJKY*0z}(BBaB zHwFE&puZ*PZwvZ6g8r_czbEML3;G9w{-L0MBxou~T6Zk1JC@cROY4rMb;r`WV`<&7 zwC-41cPyyH18D3JagOaG3gf5+0lW9i?q^zT^ucP#xomi`?}|Bj`9$I`!J z>EE&R?^ybGEd4u{{vAvITIpXa{cELvt@N*z{0c}TYo&j!^sklvwbH*<`qxVTTIpXa z{cELvt@N*z{0c}TYo&j!^sklvwbH*<`qxVTejxq(f%NYO(!U>kgJEDhZKoC%+Q(q~ zkL#`+u-ed0$INEgh=19*Xjo=gL{cY6ayD)XLiw1`NKi50wYqUSnVx~pkJ^u%vMw7} zgnKXg=kY+O(L;x;dqBcylnLxIfxvMI7gHZ4ZH*iH`# zB0ban9-*?!+wFAMLESILezEUhEn$7=Mxv>SHJ4C|ir2al&!DH?({$nwh#wg%dzC$2 zY^NeW?=z2Dlmx#y`mqx_bxqjZbfoWluMkdx3+wS^MNEdkqV5G zSy!*(q0@4QNb`j2v%J_nAT)9Dh~)F<9nS}BScs3h3M-X|BJ;oOJJxixQz=&I&;vH? zYK7Ga*$+q#QecAZbcdaCAZ_Ici}#o;7T<*S;D=_t|5>+Cg=hD;>iNizBsHmtQM z+UXiE7Rkj+oXS{|VvF?+UT&wTzVLFcg;O_9w^j@4nND}9l`)jq#=_K%nY~Ta@weJ< zxm+Zoauh3Z)?yPMQjppcp{;61@Tya=D}fZv$S>Ce=MB{jV$O$zCgQT4zQ&q@tIaBi z7o=BXi zV2g-)(y(6TY!NG}w*;HQXH#r#Z@YIIAC!rR59d`sbARO?k+jp72tFtpf+#l6Vpyr@NcJ*ajl7iMRN?fE~D&6I_`<$rgp12 z-P$4%P%DD%^ai`F%fQ{AaD)mP<4hf5#@M=ikC0DjJH5${?KIcQuU&QJWB1*kl~uAV zb5WL$a}rs*$7o7wSUVjywZa&9Kl_h8P)Z|+>O8@A`lf@|98tIK3lY3is)2c>v_S`<~k;%5K zU7$p;dqwF%(HT)4;-xQvZ0HR}3Y@ShNEG=4fkJH%c z5bBE2bLV^vclg)=V$+R`qqWl)2$sbm=W=GNI5IK=57SH0Z!U{{+v&tx@A#OdPbl^% zIefWGD0dH8%cf&IN%38{4L5zmXui~0LU0Yr6PH~|A=>HoWs*Aafbjk#QRT!%Z)|Xg zwS>ca#E=hk^~kk!GKe7si8Y^UA9jKx_eq&(#SZ0gYPX|#88=yMt#9MQezs}qQih>( z+*-Bm^l4tjrZT}^`G6fkaUr`|RF$TDdXTL6qEV7tPuxslLx?Dcsf+$H<<*gPK;+wO zTkb1Lypx!vp}0hdESiQMY1wu%G52IEaf*PEvp?3)qoP=dvp(GZR28M{T&qNhZ; zlCauok<8kta6gFzN>a)jS)X<)N4WM5>6j#oqZm*?$h!`Bg5%@d+{t^yJJu@iXl+lr z$*iY2kdl}z3AftmIU>ssNf=hA)=a(2t2ePCDC2ynDDLi>D+ugAThz@Fua{<7i6tMn z^)2$GWMmuhZ1f;4HRQf(4iWT_-0L^Fd3jPk>@kbph_#(w+9KRTKuJuu#n#Y+uCmgc zp)QoriI7Hd`1+INv7I7mA;OI$0y@YQGT&M7Dx;(ZGLba%hmNsVC@4b}NXoe{d!1Jv zVcfb&$qzl+ExQMfZr9g3B$m|ATiqIAV$1Iv7GQTHpk245t zyM+(i=J%wv*;O`xaj8AI+nd{Wc4?&nX+~N6fedv}12f#Q?jo6s|YOhTdm7=ga zJG_sG=>_vv_?)yHKim}x^Y%4DXD^DR%1cpWt?xpj#OW7OH0dQ8?M@pWSvClHMKX=P z!be5Jsou5ljk5y_p42M#QUlgU0?lTzj;2)#tD6M=h&b6S*2cBMvpXcboxVkI;-<)B ztI~t++NYwUNhzfdC8mmuAeVQyd^K?kh{;9kIeeTvGv^dM)J&$m(8}Gb9eoo0;Y5*N z?|B*U%8nzqP|&}P8_#!eK!^NSg}BNRvpq<{6%@XxCYiLc>v>S#DC(_h6=JW+BOp%n z5<}?R0}ZP$trBUSV4#9$K;uPWHJl4iDtO zJG+z^m_kbIjrzVt)x?vqMyq8d_TdI>?X;37J*%l2^a_Pe`!Z+PLDJeBTGBus_R4+Y z!^ysWLXbzF9qG3nbJT>q#E$Ls>VXrV?;=V|@bolUFd7pn0JbsG{Qvq(lrf?_6odI~ zu~jC4BS3UN+et#4`^gZC!C+Sw>&`VN-FNQXj!tH00!=Y?P*S6;MxG%i6*ZiMbZox2 zb47XAJ4tQgi)>nU(_PV?*Ip(`!B_iZ`BCa7k}BVt!ucE}L!X3rw2^cpI?<#h%L00! zkSgm@;AZGNNxi|oDurF1$X>ZY43iNY8-)*FBOeFEN`Z^&h+;s=rfw9dAr4be zS`J)0pwY4o=yBkAXNGu{r>xU>hmsphF83!WrjblknQIlTs=^A_BzlO*gW}6avb=Yb z@_Z5>WZKd|KcYZ@qlgrXjOx-Y7V2{G05I>BgXl|$A%}JODO(r#q-=e9SeV9MGuqMa zP&xMOuqnm6B^Tsdr;b9vpuYSPT5fsBqOh7Z}?@8*B>M?lf|>nmMKEQ^C&MB z74{|^qE-lrr%|K_Tjxnm4^M_+XwyJOGvSg;d`oAukSmce5kc&r1Db>OiM zJl287I`CKr9_zqk9eAt*k9FX&4m{R@|G#zMln1!`MYCU=cz~N#jdmK^sc)OaukDm& zl`kO0A`1&mDUnw26|3q)IeCSiq8;$qG6iQHP}d75u1H81aoI z6x3W|tn-wEpbg_9s4xs_wIf#sFI^WwrN}ZQrPZKAJm{$=MNknFmsLattO9WRVen(r zP0-$a?$RoP_CUKXxIswvzCxL5i3+O4z}uH0sTQU}@AFk0lo0>UCFOcLPRO1H-+SB) zn-nKld*3_M=>%!lsSLi;^vy-ZS*Nm=&hu8-@&YE*Mbe$PzKj+$MB*yK!-Ozxs^<%$ zILk4+LEEB}RBlqT^Fif7@c9Az6+x`kp?YGrK+jN${5zK>zxoQtI!cI?GM28FWza)K zU0<#B0vCE_Bp)xxLfhcEwW5QIyHjsIep|uW`ULOYDssjlF2cK3rT_YrI_rKL*vnz?1f&bGjY2-DSer zl!^Qg6DKzfr8W&9XzhSM2)^-@?>qTu^CE{ka8FP?Oou+ZFmx^+&%@ZH*5)>u2Ug!2 zE0W2>jhD$aVENM8=Qm%FG+YeZ7s0m|?>PreotWxr*xF)y(?r4;>jv3e53;8BgU--b z0na-)pNC*TmbMJz@PHGXbNeANz_MyrD_?ZEZ?X+2zvk#5r@qkPQKw+fI9AFKw~sQ z5-MH{I);Z;Dualzp5a&)Av$$c(22=%Klt-}>tT)fVYg+hCzZ=)lvw2~WK@N-;R3~i z4aeXQTpR^IwJ}TsN${wk4wSyTHnSQs8;SWKnEvIQnB&3IP|jy4(lFx|39MXn80{`E z*CCle1TG1lD()y6Rd&u3{=_{>=B8^cT%gFXpf%~(-*=h8s!PGMbigFYxfHaXbJ(y1 z=e`RTL3y&5Y7T;8u)}Df{SMjc_~3A1QtUGLfe9$KUr$D?la&u^<;UNq-LUvXl(Q9&Q#;{uhMT>RN$;M9TkY$LE;JI+l2TQ^e z7$F7J8zJMk;Qj@|C6)~*-OM3uB8kAU7ua@=Q3f#--6#YsvAwF2;7$loS>EI3&ch@03zP=qUb40|!QWQ9}WF@4r z#75AW#WnY-;95tBuukP`j}i@^46q~+-ja4Z904~JD_?2 z!Zd&01J+$029cvYB1Zo*XRYJPgom$!PKm*LC>ywwT0fY2GHZ!}TnV&6HWF0$G4`I@ zXZWyWZY`W38DN3z)UXuJX)yc80kMkwRAo1{x97taQC$nS310WGq)LL)j%x`fRf`)7 zkg~_aqajb_^#-Yfu$3jkT9Gn{bfWf%=i~s9awLs#)n~H*HO^Z;lnozIVf#PZ%7MlpT&rhYX=}Soy>S3(*y> zBIHRG4`;?)&tbt_gW+6VArV&l*~shR=I=j6rjQ{!%>z=>fXBCj4qNp?@gGy9?TTzC&FKYAAq1B$;w(#i*z zUx@{GN|2B(PbtPHAim?P5wpzE7NcHF+92-{4pCdl&#R(+9Y|>6)ojLT- zg9M;?#LJzq=Q4mKWl1q*9>rXD2ZNR7t7hP(C0KvYyeA&f7Kl!`O6@r?Xe~?smKE41 zrUwR)@C7?zQ==j?xK6Xiqzq;GE(M74uJ}&s3iV-dD+RA16-LzLStdNIA{>ybnDS28 zA$|+|uAE|&Qmbz8lA&L|?bXxU;Tk99Y`92K;H^(y8XUV}E?mO*3U^uXP03P7q6&)_ zWZ^8qN+;Sa29#CLEbIn*N{n-dr^Est6Qc!DXvT5hoJs z0}xNPvv7y#A-A^(V?N{*lK`ngNGrVXarNk4~N zr)OG>rE_Y zsk?xPoB;UPq?HsH_DnO0h)2J*1`sPFbyvBW2+V+oOpmhcM%LvK|f(Jjm#TUCPob zUwG=wW!cAM5j9*rX|rd}kfM2#h6C=N)*)Eeqt5e;mSiIc)c)wCACh-0Sbfi>4f^o> z86Ah^mG+4VrU6bMqE9^yXXPz>1dRkE&vuNqmeSXZz^$|0S)a@SA{|Zgz8g(Ei011Y z8jMHL$kJXY^_*yCQ;`rC{Iwz-Pfid#dmNC3mwdvmh6ASxd-mKAtvw<&* zHI?h(C7)3!w}-b&4Kpt?BIr#5fKD2lh5N}+OYtp|I*Rd5xaOR|JM(&CwAEtVQCOIT9>=lg){@Nrde~w2YaZiUY{tv$?s{Ni zsU0)zrZl@)-Mj-{@cPXd1{Ws8V97J3hj0o?d128CRj(m2KtWMOC#Fa?XUwN&bEx-XMPGnwym zS!W_L#a2PWmMd*Q7x{w~*+>o^aSLNv2nZlP!`<)W_#Ez?2#jU0|%~Yc-$>K&1vYT%WoHXN3!&jLP_6i z!!Wn(3}J9h;i)%x57`!I%~PZd?m}z9CO!$lynnWri6pIz+_JUr$j8i)#0q))3HL(@W^Ez7ln=ou1 ziiW?Rvb|44ZtWc#i%t{g>4H1U=mr!MGkD$0HW77$*aY=W4t=i734*>~zDfwy={ay(-+4*(% zJXS;)?0fee%SkB+xHsi;op6WKVlfr;-HQ*#e4_8~!ajOVGPOc0!U7^ic~iCvJ4NFK z+Tlgm&MY?NZhlrX2d1=iFDCUf>_un_Gi)idGqRbkN`D1>tGg+yjb7q_aKf_XR-_X$ zAV_MUD31iqAvX0!5*Pfowr`MQE2Tg%k1;t6Ozzv9hVdD~vdf(0z)f~JYKEa?1-Ic%pBHAa5f!6^sH z!2%E|Ff>iC=WVBmZ4#B-4S07ixw{*Z(@q0KCdpjDe^OgxETBymm{m`X*26lHWsdAt zh;L?cPR;1}OM;`jCvs?F@`^zylZ1ehnu{)03E0O1W`8p*UU&jqNyRaCy`32xeuH1e zyI?3jR8f5ep_518$SXrq<9X9JD{$1ui%cNms~mKA;L=B8qPS@-IhE55BNvE;2+nEd z2evlCPm@9nRwS%osqAwZqGG@J@|!#1j?2p83fXD1zfu!a5#Bzd((4>7v28CgBE&~oh9(eYaBDwo`~8Q@a)Ly zDZFDYJt-u>-Rb~}$(NG@7bfjM(N7Ktjz(L-o$xBv#GApC(C#Tx7$eUPRn{=Z%GL( zT)Jlf(-GJ~I8AB5ylGDdIjM{WOqwV_gpwRiz}xu~w|za_x+z>(Hi_CnU6bj+&VT_EMk&?egt5SZ-fdd9b%uP;^!c^-ZhW;TCVy z75sWq0F?9WpvFpu@X|skOA@X_74V{P;+TD-B#;#)#tNGqzEyhHx7^o#<74c8&(g4D zx?e+$iG$z;_z0^=QB0bBR|lzuvIXp#d!I$e5IP;o7u6cy3`4}@T`ZdCFlMar+C;L3 zq(mW+v{JnDj`+xn|86MSMYC6QrL1F9`n*Sd(=YtyF zynB$3JFsiR3R$x0<342|8V*^g|7?YPWF_<7!9J-(-D``L}~)CWJo zwh!=n>-tl)?dM&D@pw@bTnd`wcQgm%_4(fmp;vbvl<}ug<8NkOKp5B=#cg4>-HvZK zi$e!|Y$44}H8* zcNr^C;9bpTa!n!CBN;djJ|l1~%5u#b#t!H>yJSB&yl~lWXZC&=6!+(C)>3XFOUBg< z@d*K~Y8wfG6q>xm4-^*<=dW#AUz@#O6(zO!KFWYMHXFAlP5ctWZ2m{t+D4d*jblfA zCc;4vGF;-AH*sfaI#;!1>m_~5qK-l95*?ghdnP=$3Q?TWPoM`&gZhP(a0T6w8VG9b zHN0#QFR%}5;cF{M_*z)Ih+99jR1EW_M{^Cw@7y-yN0%=>nz4v=YhUI!A=yWlPmCb7 zU5m@@WAtWze+7#_0=>v!KgJS3HX1jH8cD_oYvS+X}i*_ zQVwY7%mAU~vVPCBZ&a*WIA&u{SeMra#Y21dT!Ec;vy#!Oxv5-XSXM6L`9(9fbCPG2 zESm2@jieAw5?6p1eieHNix!r;QM#s;wTh=t;)Np^I|&a9w9miHyd9{v@1QsOl9{m` ziJ4WRyNQQq*bXmUw#n|z$2u_%@Q&-c!r2Hot{DWk1&O;m)72HuEDFS|5Pm5;TBMo7 zeS6?S0)=PKg&iB@?ZMlFw~(FMFC-vB>w-v(X&VaghE-J`nw#Vc!bcaa`K37e<+Ic$ zW56cUE}KJ~5tnGAQFQk0fjZDC#}1Fs{$0FJGsOmLnQF(Z`=zbradCak2)e!(@S1p3 z>uS7A(x}H9H4*2FY-9~L-a?Ertd1gmi@f1Bm z@eRK>UqeTP?d`!BF`&8qfXEJ#>7#&Jf>=B#G$=6FOCxWQ2O}ejm+a#E^;ciJb$jQ^ z%^lplUG$My77gUG8D5^BB#dGng;!s{voqI~Na6@4qTDO7Ndl7mJvEv8Xg9|as zvdfozcoawQUc}Hx0T>4u0%$cOV{-z1iuyJqEG07h%&RxTINt|bo8fj5U!IxBVon&a zi6Coey57gG^n4K_cebPgrBl3E6j~Uy9lC5kJK*{g^$C;=as&}}G7-mgCWxiOi`QPh zb^F?EwWLzH5b*kcsMAi!Eh)5tp`^A;SJ+Gru&a_bN0N=-MX_M@0)#cfYZ@cU_*(#= z4zq^?Y6FvIHB;0`UU&9_)^Kl7AL4!3Ajb?w50CaE<{RVpPa+a~pK^$A;$0jwYJ<^oUgLxU`OomrBF@YNQ-#dGLWn{C1sv<06dR8a;P+`f z89u@W5@TXWXgK1kUklHuO%k~+m{X!#2yERz#;9y=7-K)2bRi}79aN;`|ET3)_HDHx zcg~-gH%VDJ!S#Yn?2FCVV6>0~MpFToQ4Eo4*44Jn%>{enYTSVyHb)&Z01vk*k4IH} zIc(Sr&na&-voybNc9u==Uc9*Z+&igUV{%~VK*2hjEnwUXRb3CWIOelyS4_!Zlr0e( z6@QLL+IrSB|1A4bCZ9%E@%rnS$IkWiR1L*CW2&1)Upq33wZ1kpSlGY92F0kjjC?Ce zK{L!0+s#aF(olJ@k!v5HtP?zM&sI^&;8Ks}$+Pq;NtIIYgEiV z+Xx~pCN}%5WPd3or;CG|Hh;cb?87y}>m-3^LYt5%#FJ+m|2yySSB+an$&HRPNDMwKM9|)v7m1pVCBWrB!5B(r}_5OxL-Dd ze!^5OiQ0YkJgHf&)0bufoGJGSYG!)PX|T*8T?5N8#46?Z;aAo&*5!)Uk_|G~8$;b$ z_D5AtE?X{;rh==RtjG&(M(txGnsF7*#2RP1sC0JLY4-m)v1*hliUn0Ga`WZBrZ?&g zYarMWEzP@OSgD&tJtygOqrB8`zF1BuCY$kZR5)e3Op82$G0k!EEd?~~LaL-dnwa=E zvav#`c;Eic)LeKiEu)CHf1M*PD;$UWBy#2gNRqE#KQW(X2E}xSU#Xecq0pH0uBgY4K=O-x%c{)K) zLYyilkHRS5Lqdro`o{i@bgV?m42yhy@y-Zovi&Ss3`iGmnZ=m-+6`=_z%eGbG0=%! zn3Uy6HEd(jC9!*}TEguu)*?QB*(}t9QhXn;St2 z0a>zM#wCM`WIkybma`XD@FH1QQ{YYUvibL@bdfjIk;lgEr{-;0${Zr@`9+j*fTcx# z`Yfq`Y8_`&w-^!{ijS*<&%H0U&8ttU9WN$Pwso~%pePm}_3NM<)yFG6<_tLD^Jqtb zb}~aoE<{`mtZvkafsa+M5>ITpwBoiULLA9e$RnruRG%geq_;6Lj?Qr&Va{Tyv?B$Z zYTqB9AEcl0U1j$tNFB=eW)i^3C)+5toqzjT4nC*Ew)SHs@n<%yf$eMa^_p`eMTTuR z7^#PuwY1&ur}57|Up*`apUa-#d^Y>Si<_^!^7_kL^Ol{-U{uxPgKU3)viF%;<1;7F z*>By!8XLwZ=WXWwEGQ^<=M$KYl8qmZ=UeY5@xie*WidjEgI@qPW_}g_esYfa#OU3F M$>3hn_%$^AzeZg=s{jB1 literal 0 HcmV?d00001 diff --git a/crates/aprender-serve/tests/fixtures/gguf-header-slices/tinyllama-1.1b-chat.gguf-header b/crates/aprender-serve/tests/fixtures/gguf-header-slices/tinyllama-1.1b-chat.gguf-header new file mode 100644 index 0000000000000000000000000000000000000000..b98c27845f358fc84d86dc54d7a5c6f34b199d8d GIT binary patch literal 18724 zcmeI4+m0m1afa7N*|Y(}faRkMC%B@AD`{r8&okC8p*~X78PZa^kc?4wcTIO`s;heI zFtfAN0-_$m7k&m`5Ht)321H%ye)bjoB#!K!i2VOC7xsnkI1LUrqbe&izR1YPtf=aK z_ucnrzjrTw{_3Xq7dQW|k~}GWcI3;&I<1pM-IU26$+I%)uI(If3{SL~09 zESG7%x>f7Xsn$Fz7U!pnqRH#fOlab7sKQy2EKkph^3s<}Y)NYVEmbUwZF00sH%VTl zMIM{{n$+T&zOIwJmP$wK#4lq_CpG_G{=#R^@M$$q&o6?e;qMJY{OI|kM$|6OyJR%}{C) zUyJx!#MdIe7V))+uSI+<;%gCKi}+f^*CM_a@wJGrMSLydYZG6a_}awRCcZZDwTZ7y zd~M=u6JMM7+Qio;zBci-iLXt3ZQ|k?m=_`1Z`CB824b&0P_d|l$}5?`12y2RHdzAo`~iLXn1UE=EzUyt~D z#MdLf9`W^vuSa}6;_DG#kNA4T*CW0j@%4zWM|?fv>l0s}`1-`xC%!)M^@*=fe0}2U z6JMYB`oz~KzCQ8wiLXz5ec~Gs-+=fA#5W+m0r3rpZ$Nwl;u{d(fcOT)Hz2+N@ePP? zKzsw@J0`wk;yWh3W8ym|zGLD$Ccb0hJ0`wk;yWh3W8ym|zGLD$Ccb0hJ0ZRk;yWR} z6XH7|z7yg*A-)shJ0ZRk;yWR}6XH7|z7yg*A-)sh8xr4;_=dzcB)%c>4T*0^d_&?J z65o*chQv1{z9I1qiEl`JL*g3|--!4|#5W?o5%G6)!zA^EQiEm7NW8#|---P%k#5WDMtrl# z_h0|=q5P+JhEl9=ZX)j`rIykh&U-zYJ*LpaqG-EWf>e}S64t>^s)EH;KFmN-a1fJzONhfw|rl=mr^ zz1BX*0eY$(7igRdJ(}9E7UqQBjN2fB4PRo{H0EL{_W31i-L5G;pq2>;(WRf66Bg*C z-!{jnP@OC7q%wo)b)CfLg^rkySBc#YNh3YfyYjF-2dQ%31s|}Ag#`GqUyLbKXD|%Yh zT9oPVxiFvQBX+7KHH_WhIDZkey02OlaWde{{gIwun@=vp6QG56O@Z1RB&pDz-_aUL z4CdP{YFB}9lV;sxY|8?mF2xF}&pz3DLwg>3ut2-DA$jmsy(lX}cJ}5X40bfg7S$yW z51>DVO5?&}L+Z72PXRsiaP0>GnAnoK2onO&JU2z-Xw0FRpV@g^svA`C8>&jWSVOx@ zT)n;vCt@T@FZ9x)j4oB84>hRj{rfO3^*RQ(bDVc;w85eE(BMk`bd5nhFR;tpJSxp_ z8OkgQEK+T?gw;5=^A8`Y$%?HSGFheh8_^GBEJGA~6px^1{I6v-=R7p4UA+46)h|^s zUzty)FI1ZcM_MXY+Jba-{TmR95Jt%kQ^QYmtcYSp?g{cUTB+5RfDUq{e>}Q=|Wo)yZ#CWYXM>1^jyT4 z@!%`3dFWFuiZa^3qt8;0;t*uR9ou#vd?yR*Ta4}3O#;CE#H-~HX$;LZm=J3jE5?7}7~SNrV3r@c}t znGt8Ho5We(B6XPfbC&K*oIUd@9Om_Fhe>l+TN);Dn8RYho-_EUL=lznti(dIGpVbz zSUSthqz-d7qE7-ToO!V_!+f=SPmh<*s-!N1YYTSbI z1uOWrR`G6wh95&iay3Y9Nj~rcyb2#mxgnnU2@2?G;=}le#)~b%EuB2T!7HDtx&@oo z67^~Ht0K-B9|x#me)EzfUf4v3PuVxWq}`{^E-ak~>&Scq$(1O{w8V3%P10v^#uPlM zzO4?Oh|gHr(Nk?_&%>%X%onk^h{ZG(8;%{{Lo1II(?`sP4gN%{NZc^JWbCXg!>Z*% z9LUPzK3c9No?6)@rK+$AlQ0T0J56x%OhS7uI95*dcp7JcuN^#Hl}D`-E_4XOgV#8_ zCBRBe@VqlulqgEr1<}c4T2(3p1`@sn6|Y>;peVr+L?_R2RUc_)&dxKp5S+OyHPi(d zO$ON(B=0O%*3^uIedxl`3nO5FKoQ^9D4jiPXxPkeLSvV_rd_T9Z6U3pC|!Z** z@9WSiM@RAXIIv{qDK=nVI9j-KXiK6CND@u7Em{7wm_#S{(ax#VP@9VhDH4C*PQa{` z*=;j4L@})s0^DwAv0p=^Szn{Pb|gzz9tTX8P$HlO^^0hN`wCQGw4g zEuza~?6}0s5XLjXKi)MaaWN~q+R758>>0Q0YiMKh9laot7-?mRkgC(xZGlvlYLJ{q zN#Y!yt8hKpI(j<3mBkYMPM)2E@h@dMtWgf54zddm*Q0)BreKV7V<-{2w5Yy~>(2xC zRc$3BeZ<5YP*27JMVTeG>+zdtHCd}km7hWrV{UPLt3`RPtD2AgfDsHwWOWrcR?vxpFwiZY=4`&-52X*X5TByXhK;lqB-HX?4)sO%My&u zHYnY9_wn5Q`{0%0qiKcm+L0_FBe4&{CXT5y@gl7&kZpoCio2?jQy zFA@J#-JXSU7tjCt1PfUsjAX|C2E4@)2*7NhLEfulbNnSUTjSTBxNO&v(s4 zDfD2H3olNn&%gT6z51o{c!;qQBrV*e&YsRca7PhgrEP|3as3;R#7#ls{m!LCTo|iM z=?O$9FJ8ckBdf9{S;pm&6X^TOOiqW_KI%LPy;Di|(1-;)&mA-`?4c$8(r6u)v9ptuJxp zSO#Z3bcblZ>yJU2yDN~~-1hoe^l;hbT8NXOu&v6gUx>{VRJ_fGLiU+f_7ESv#7C{~ zVb;kW1ZSPZ0MzNJT_Yy5p%+`)wmpp3OB}rS;dh@$N3Ky>nMi~2+z?HrbK zZ9J^>TIGTJ3LwA7%fNd!86-m4E>h-*l@k~mNn`B3JblA{CcB>vu%3m8N3y5{vOAOd z#{7x9{#bRCtb`s9!}ntq{9HUY3Xl&`kH)QFX+eBHj~BM%yq=utSmbda~%msR+hc4-BrpRj+Mna zR!0oCgx+Me2Z^@=bKWS-WeWWmO8n z%4%R`St+zE;~sqg{6VtA<_~g@WBw3U*G~Yt;=75;a

7`hch?17tJeZTCP!Y88gr z1a}!TJf*WQgQVhA+NR|6Os8Pqf6@r?2xbTfhH(8B`>+u0N3z zTMs{Db6O{xZRYFU?&V*D!t(jxt&n^avsIN}C9+@EV&&d?`G%ZL&-BQT56U7-4*uCa zdEyS5Dk%?kRX%VBkAEJ-pTG0Qf%|f-z59)vyp{{T?92x4<*wBvdriC5RV8Xd%liZW zFKx?P;__t<`3eCvZ^w2${GZj&m!iKJC@YHXQRXXIe)FaH*79`!vo#?%aVj~Y&9;uF vf*udl8qi#lFT;Sm_3{h*0lKHZfb*;o&C?gVZ}^2K>JPB{op! literal 0 HcmV?d00001 diff --git a/docs/roadmaps/entries/PMAT-3609.yaml b/docs/roadmaps/entries/PMAT-3609.yaml new file mode 100644 index 0000000000..08e7536c3f --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3609.yaml @@ -0,0 +1,22 @@ +- id: PMAT-3609 + github_issue: 3609 + item_type: task + title: 'The BPE tokenizer requires a '''' token that Qwen3.5 does not have — a universal unknown-token is an assumption, not a fact' + status: in_progress + priority: high + assigned_to: aprender-62 + created: 2026-09-21T09:00:00Z + updated: 2026-09-21T09:00:00Z + spec: null + acceptance_criteria: + - 'Operator ruling on #3609 (2026-09-20), constraint 1, verbatim: "`unk` becomes `Option`; absent means absent. No synthesis in either direction. That is stronger than the original framing and deliberately so: the `vocab[0] = ""` placeholder for an absent vocabulary goes too. A GGUF with no vocabulary refuses by name — it does not load. Today it loads, which is the inversion this ticket is about; removing only the refusal on the present-but-lacking side would leave the permissive fallback intact and simply move the asymmetry. The two-direction falsifier already proposed covers both: a model with an unknown token still resolves it (so the fix is not \"ignore unk\"), and a model with no vocabulary now refuses (so the fix is not \"synthesise one\")."' + - 'Constraint 2, verbatim: "Prose changes in the same PR, or the belief is reproduced. `book/src/examples/api-server.md` and `examples/model_cache.rs` both repeat \"vocabulary must include ``\" — a property of our constructor stated as a property of the data. Run `pmat query` for `` and `unknown_token`; every hit is either fixed or listed in the PR body as intentionally kept. A hit that is neither is the defect''s next copy."' + - 'Constraint 3, verbatim: "Mutation: reintroduce the `ok_or_else` → the Qwen3.5 fixture goes RED. A fix with no red-turning mutation is a claim. The fixture must be a real Qwen3.5 GGUF header — the family declares `tokenizer.ggml.eos_token_id` and no `unknown_token_id` key at all, which is the actual input, not a synthetic one."' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'Taken by aprender-62, sequenced ahead by the cop (aprender-04). Two interpretation decisions, sent to the cop before implementation: (1) THE FIXTURE. The real Qwen3.5-0.8B header is 10.9 MB (248,320 tokens, 247,587 merges), too large to commit. The fixture is a header-only GGUF generated from the real file: every one of its 46 keys and every scalar value verbatim (eos 248046, padding 248055, model gpt2, pre qwen35, the chat template; no unknown_token_id and no bos key, exactly as in the real file), with only the three arrays (tokens, token_type, merges) cut to a leading slice. Qwen''s first 256 tokens are its byte-level glyphs, so the slice still covers every byte. The generator is committed next to it and records the source header''s sha256. (2) WHERE THE UNKNOWN TOKEN COMES FROM at the vocabulary-only constructors, which receive a Vec and nothing else: the vocabulary''s own `` entry when it has one, otherwise none. Nothing is synthesised, and every model that loads today keeps its unknown token. Threading `tokenizer.ggml.unknown_token_id` through every loader is a filed follow-up, not a silent gap. + + The cop ruled both decisions YES on 2026-09-21. It also added a guard, verbatim: "When the glyph for a byte is ALSO absent, encode must refuse with an error naming the byte. It must never drop the input or emit nothing. Put a row for it in the case table (a vocab without that glyph and without unk → error). Absent means absent at encode time too." How it is met: `encode` returns an infallible Vec and has dozens of callers, so the refusal happens at CONSTRUCTION. With no unknown token, `BPETokenizer::new` refuses, naming the first byte that has neither a `<0xNN>` token nor a byte-level glyph token. `with_merges` also requires every glyph, because the merge path maps bytes to glyphs only. `encode` therefore cannot meet an unencodable byte, and the no-drop guarantee holds on every path. Case row: `no_unknown_token_and_a_byte_with_no_token_is_refused_by_name` (a 256-glyph vocabulary minus byte 0xA9''s glyph → error naming 0xA9). With the unknown token present, encode is unchanged: `<0xNN>`, then the unknown token. The glyph fallback applies only when there is no unknown token. Follow-up filed before the PR: #3675 (thread the DECLARED unknown token through every loader; by-name `vocabulary_unk_token` is the interim, and `apr serve`''s chat path already uses tokenizer.json''s declared `model.unk_token`). Found while testing, filed separately because it is independent of the unknown token: #3677 (the greedy path matches Latin-1 characters such as `é` as byte-level glyph tokens, so `é` decodes to U+FFFD). The Qwen round-trip rows therefore use ASCII, CJK and newline, not `é`. Same class but outside the query''s hits, so not in this PR: `apr showcase`''s pipeline substitutes token ids [1,2,3,4,5] when a model has no vocabulary (commands/showcase/pipeline.rs:165).' diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index 84e48c530f..61df6c6fea 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -18973,6 +18973,28 @@ roadmap: labels: - kind:code notes: 'ACCEPTANCE (hand-entered from issue #3587 "What lands"; minted on the PR branch so the AD-04 quorum can run -- pmat work status refuses a missing item). Two halves: the explicit-command runner (#3616, merged) and the nextest [profile.ci] key (#3626). Lands: 1. **--no-fail-fast on the shard.** The cost is runner minutes on an already-red shard; the benefit is that a red run still measures everything else it was going to measure — which is exactly when you most want the data. 2. **The reducer reports ran / skipped / total** in the shard receipt. The skipped count is a **number**, never an absence. 3. **A receipt with skipped > 0 is Unknown, not Fail and not Pass** — it names what was not measured, per doctrine 4. ' +- id: PMAT-3609 + github_issue: 3609 + item_type: task + title: 'The BPE tokenizer requires a '''' token that Qwen3.5 does not have — a universal unknown-token is an assumption, not a fact' + status: in_progress + priority: high + assigned_to: aprender-62 + created: 2026-09-21T09:00:00Z + updated: 2026-09-21T09:00:00Z + spec: null + acceptance_criteria: + - 'Operator ruling on #3609 (2026-09-20), constraint 1, verbatim: "`unk` becomes `Option`; absent means absent. No synthesis in either direction. That is stronger than the original framing and deliberately so: the `vocab[0] = ""` placeholder for an absent vocabulary goes too. A GGUF with no vocabulary refuses by name — it does not load. Today it loads, which is the inversion this ticket is about; removing only the refusal on the present-but-lacking side would leave the permissive fallback intact and simply move the asymmetry. The two-direction falsifier already proposed covers both: a model with an unknown token still resolves it (so the fix is not \"ignore unk\"), and a model with no vocabulary now refuses (so the fix is not \"synthesise one\")."' + - 'Constraint 2, verbatim: "Prose changes in the same PR, or the belief is reproduced. `book/src/examples/api-server.md` and `examples/model_cache.rs` both repeat \"vocabulary must include ``\" — a property of our constructor stated as a property of the data. Run `pmat query` for `` and `unknown_token`; every hit is either fixed or listed in the PR body as intentionally kept. A hit that is neither is the defect''s next copy."' + - 'Constraint 3, verbatim: "Mutation: reintroduce the `ok_or_else` → the Qwen3.5 fixture goes RED. A fix with no red-turning mutation is a claim. The fixture must be a real Qwen3.5 GGUF header — the family declares `tokenizer.ggml.eos_token_id` and no `unknown_token_id` key at all, which is the actual input, not a synthetic one."' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'Taken by aprender-62, sequenced ahead by the cop (aprender-04). Two interpretation decisions, sent to the cop before implementation: (1) THE FIXTURE. The real Qwen3.5-0.8B header is 10.9 MB (248,320 tokens, 247,587 merges), too large to commit. The fixture is a header-only GGUF generated from the real file: every one of its 46 keys and every scalar value verbatim (eos 248046, padding 248055, model gpt2, pre qwen35, the chat template; no unknown_token_id and no bos key, exactly as in the real file), with only the three arrays (tokens, token_type, merges) cut to a leading slice. Qwen''s first 256 tokens are its byte-level glyphs, so the slice still covers every byte. The generator is committed next to it and records the source header''s sha256. (2) WHERE THE UNKNOWN TOKEN COMES FROM at the vocabulary-only constructors, which receive a Vec and nothing else: the vocabulary''s own `` entry when it has one, otherwise none. Nothing is synthesised, and every model that loads today keeps its unknown token. Threading `tokenizer.ggml.unknown_token_id` through every loader is a filed follow-up, not a silent gap. + + The cop ruled both decisions YES on 2026-09-21. It also added a guard, verbatim: "When the glyph for a byte is ALSO absent, encode must refuse with an error naming the byte. It must never drop the input or emit nothing. Put a row for it in the case table (a vocab without that glyph and without unk → error). Absent means absent at encode time too." How it is met: `encode` returns an infallible Vec and has dozens of callers, so the refusal happens at CONSTRUCTION. With no unknown token, `BPETokenizer::new` refuses, naming the first byte that has neither a `<0xNN>` token nor a byte-level glyph token. `with_merges` also requires every glyph, because the merge path maps bytes to glyphs only. `encode` therefore cannot meet an unencodable byte, and the no-drop guarantee holds on every path. Case row: `no_unknown_token_and_a_byte_with_no_token_is_refused_by_name` (a 256-glyph vocabulary minus byte 0xA9''s glyph → error naming 0xA9). With the unknown token present, encode is unchanged: `<0xNN>`, then the unknown token. The glyph fallback applies only when there is no unknown token. Follow-up filed before the PR: #3675 (thread the DECLARED unknown token through every loader; by-name `vocabulary_unk_token` is the interim, and `apr serve`''s chat path already uses tokenizer.json''s declared `model.unk_token`). Found while testing, filed separately because it is independent of the unknown token: #3677 (the greedy path matches Latin-1 characters such as `é` as byte-level glyph tokens, so `é` decodes to U+FFFD). The Qwen round-trip rows therefore use ASCII, CJK and newline, not `é`. Same class but outside the query''s hits, so not in this PR: `apr showcase`''s pipeline substitutes token ids [1,2,3,4,5] when a model has no vocabulary (commands/showcase/pipeline.rs:165).' - id: PMAT-3627 github_issue: 3627 item_type: task