§4.4.6's FATAL tokenization rule has no implementation: perf_gate.sh:79 checks only that method is a non-empty string, and the gate's own OK fixture uses an illegal value #2816

§4.4.6 is the spec's answer to SGLang RFC #9808 — "a throughput ratio between two servers that count tokens differently is not a ratio of anything". It is specified as FATAL. Nothing implements it.

This is this repo's most common defect class: an assertion that exists, is believed by readers, and cannot fire.

What the rule says

docs/specifications/APR-PERF-GATE-001-v2.2.md §4.4.6:

tokenization:
  method: server_usage | client_tokenizer   # REQUIRED, no default
  tokenizer_sha256: "<REQUIRED when method = client_tokenizer>"
  counts_special_tokens: true | false        # REQUIRED
  counts_prompt_echo: true | false           # REQUIRED

Any mismatch in the tokenization block between the measured lane and its comparator baseline is FATAL — the ratio is refused, not annotated.

What is implemented

1. The only check is that method is a non-empty string. scripts/perf_gate.sh:79-80:

if not (r.get("tokenization") or {}).get("method"):
    bad.append(f"tokenization.method absent{why('tokenization.method')}")

Any truthy string passes. The two legal values are never enumerated.

2. The gate's own OK fixture carries an illegal value. scripts/perf_gate.sh:427:

"tokenization":{"method":"hf-tokenizers"},

hf-tokenizers is neither server_usage nor client_tokenizer. The selftest's negative case (line 574) only blanks the field to "" — it tests emptiness, which is the only thing the check tests. The fixture and the check agree with each other and both disagree with the spec.

3. counts_special_tokens and counts_prompt_echo are read by nothing. They are plumbed from CLI args (extended_commands.rs:1822/1825) through test_llm_band.rs into the receipt writer (perf_gate/receipt.rs:339-340) and serialized. Verified over origin/main: every other occurrence is a struct field, a constructor, or a test literal. No assertion anywhere consumes either value.

4. There is no comparator-vs-measured comparison anywhere in the gate. The word FATAL in §4.4.6 describes a comparison that does not exist in any file. Arm C compares completed/requested; nothing compares two tokenization blocks.

5. tokenizer_sha256 is validated for shape, never computed. perf_gate/receipt.rs:286-294 checks 64 lowercase hex characters. Nothing hashes a tokenizer file. Verified: no tokenizers dependency in crates/aprender-test-lib/Cargo.toml, and zero Tokenizer::from_file or .encode( anywhere in src/llm/ or src/perf_gate/ (both greps rc=1). The declared canonical method — client_tokenizer — has no implementation to be canonical of.

6. TokenizationBlock::require_counter(has_client_counter: bool) takes the answer as a parameter. receipt.rs:307. Its only callers are its own tests (protocol.rs:281-289), which pass the bool literally. A poka-yoke that is told what it is checking is a function, not a guard.

The sharp point: as SPECIFIED, this rule would still have been blind

This is the part that matters for the fix, and it is not obvious from reading §4.4.6.

Measured on W1, same corpus, same model, same client, deterministic (min == median == max), n=32 llama / n=16 apr:

Constant delta −21, ratio 0.9607 — 4.09% more prefill work on the comparator, on identical input.

Cause: llama-server applies the GGUF's embedded jinja chat template, which injects Qwen's default system message. apr uses its own hardcoded ChatML builder (crates/aprender-serve/src/api/realize_handlers.rs:200 format_chat_messages), which injects no system message and adds a constant 8-token wrapper. Reconstructing each with the canonical tokenizer reproduces both figures exactly.

Now write the §4.4.6 block for both lanes as the spec defines it:

tokenization: {method: server_usage, counts_special_tokens: <..>, counts_prompt_echo: false}

They are identical. Both servers report their own usage; the block has no field that can express which template produced the ids being counted. So the FATAL rule, even fully implemented exactly as written, would have declared these two lanes conformant while they differed by 4.09% on the quantity the ratio divides by.

The divergence is also not correctable by an offset. Shape-varied, apr vs llama:

Not a constant offset, not a constant ratio, and the sign flips. There is no post-hoc correction factor.

Suggested

Implementing §4.4.6 as literally specified would close items 1–6 and still leave the gate blind to the thing it exists to catch. The rule needs new join-key fields before it is worth implementing:

1. template_identity — a digest of the resolved chat template actually applied (the GGUF's embedded jinja on the comparator lane, the hardcoded builder on apr's), plus whether a default system message was injected. This is the field that separates 513 from 534.
2. prefill_tokens_measured — the observed prompt length per lane, so a 4.09% divergence is a visible number in the receipt rather than an inference from two receipts nobody diffed.
3. Then make the FATAL comparison real, over the block including those two fields.
4. Enumerate method against {server_usage, client_tokenizer} and fix the OK fixture, which is currently asserting the illegal value is fine.
5. Mutation-verify each one: flip the fixture's method to a third string, observe RED. Give the two lanes different template_identity, observe RED. Per this repo's ratchet rule, a guard that has not been reverted-to-RED is not yet a guard.

Also worth deciding what counts_special_tokens is for now that it is known the two servers sit on opposite sides of it: llama-server counts the stop token in completion_tokens and apr does not — exactly +1 on llama for every naturally-terminated request, 0 when truncated at max_tokens. That is precisely the axis §4.4.6 named, with the two lanes on opposite sides of it, and the field that records it is read by nothing.

Related

Same investigation as #2754 (a streaming receipt declares server_usage while the tokens are client-counted — the mirror of item 5 here: the method is a caller assertion, never cross-checked) and #2743 (llama_pin.toml declares a threshold nothing reads). Epic #2706.

Sibling findings from this investigation

One tokenization/counting investigation against the perf-gate comparator lane produced five findings. Each is self-contained; together they are why no Arm B ratio is currently trustworthy.

Shared setup: apr 0.64.0 (745fa8588) CPU build · llama-server 7746 (39173bcac) CUDA, 29/29 layers offloaded · model qwen2.5-coder-7b-instruct-q4_k_m.gguf sha256 509287f78cb4d4cf6b3843734733b914b2c158e43e22a7f4bf5e963800894d3c · canonical tokenizer sha256 c0382117ea329cdf097041132f6d735924b697924d6f6fc3945713e96ce87539 (7,031,645 bytes). All measurements reproduced twice.

Epic: #2706.