BLOCKER: no W1 cell is producible on the comparator lane in either mode — streaming reports prompt_tokens=0, non-streaming reports 534 against a 512 ± 8 band #2817

No W1 cell can be produced on the comparator lane. Not in streaming mode, not in non-streaming mode. Both exits were taken with the real producer, and both are exit 4.

This is the campaign blocker for #2706. Arm B needs a comparator ratio; the comparator lane cannot currently produce a conformant W1 band by either route.

Both modes, verbatim

Real producer against llama-server build 7746 (39173bcac), CUDA, 29/29 layers offloaded, model qwen2.5-coder-7b-instruct-q4_k_m.gguf sha256 509287f78cb4d4cf6b3843734733b914b2c158e43e22a7f4bf5e963800894d3c:

Streaming (--stream) — exit 4:

all 74 completed requests reported prompt_tokens = 0

Non-streaming — exit 4:

64 of 64 OUTSIDE the band — 534 tokens ... the band measured under it is not W1

Streaming is the mode §4.5 requires, because itl_ms and decode_tok_s need per-token arrival times that a blocking response does not carry.

Why streaming fails: neither server reports usage in SSE

• apr emits no usage object in its SSE stream at all, even when the request sets stream_options: {"include_usage": true}.
• llama-server emits it only when that flag is set.
• The flag is never sent. git grep stream_options origin/main → rc=1, zero hits.

So crates/aprender-test-lib/src/llm/client.rs:494-508 never populates final_usage, and band.rs:136 usage_counts() takes its fallback arm:

fn usage_counts(usage: Option<&Usage>, fallback_tokens: u32) -> (u32, u32) {
    usage.map_or((fallback_tokens, 0), |u| {
        (u.completion_tokens, u.prompt_tokens)
    })
}

prompt_tokens is substituted with 0 — which is what the exit-4 message reports. Completion tokens fall back to a count of non-empty SSE content deltas.

There is a third fallback in the tree that would be no better: loadtest.rs:1481 estimate_prompt_tokens still uses words × 1.3.

Why non-streaming fails: the same corpus is W1 on one lane and not-W1 on the other

Deterministic across all runs (min == median == max), n=32 llama / n=16 apr:

Constant delta −21, ratio 0.9607 — 4.09% more prefill work on the comparator for identical input.

This is the uncomfortable part: prompts-w1.jsonl satisfies §4.3.1's 512 ± 8 on the apr lane and violates it on the comparator lane. "This corpus is W1" is not a property of the corpus. It is a property of the corpus and the lane, and §4.3.1 does not say so.

The spec half-anticipates this — §4.3.1 already notes that it "does not say whether the 512 is counted before or after the chat template wrapper", and requires the receipt to declare which side of the boundary its count was taken on. What it does not anticipate is that the two lanes wrap differently: llama-server applies the GGUF's embedded jinja template (which injects Qwen's default system message), apr applies its own hardcoded ChatML builder (realize_handlers.rs:200 format_chat_messages, no system message, constant 8-token wrapper). Reconstructing both with the canonical tokenizer reproduces 513 and 534 exactly. So no declaration of "which side" makes the two counts comparable — they are wrapped by different templates.

Not correctable by an offset either — shape-varied, apr vs llama: short 25 vs 43 (−18), W1 513 vs 534 (−21), special-token text 76 vs 57 (+19), natural 16 vs 34 (−18). The sign flips.

What is NOT the problem

Vocabulary. On raw text all three agree exactly — apr 14/505, llama 14/505, client 14/505 — and the GGUF-embedded vocab and the canonical tokenizer.json (sha256 c0382117ea329cdf097041132f6d735924b697924d6f6fc3945713e96ce87539) give byte-identical id sequences. This is entirely a chat-template and usage-reporting problem.

Options, none free

1. Send stream_options: {"include_usage": true} and make apr honour it. Unblocks llama's streaming usage immediately; requires an apr serve change. Still leaves the 513-vs-534 template divergence, so it fixes the streaming exit and not the band exit.
2. Count client-side with the model's own tokenizer — which §4.4.6 already calls "the canonical method". This is lane-independent by construction and is the only option that makes 512 ± 8 mean one thing. It needs real work: there is no tokenizers dependency in crates/aprender-test-lib/Cargo.toml and zero Tokenizer::from_file / .encode( in src/llm/ or src/perf_gate/ (both greps rc=1).
3. Drive the comparator with a raw-completion endpoint and apply one template client-side to both lanes. Changes what Arm B is comparing.
4. Re-target the corpus at the comparator's count. Rejected — it just moves the violation to the apr lane, and I-9 makes the spent runs unrecoverable.

Option 2 looks right and is the largest; option 1 is worth doing regardless because a client that cannot ask for usage is a defect independent of which counting scheme wins.

Do not measure a cell until this is decided. I-9 makes a mis-configured cell a spent run, and #2778 makes the same point about ordering: an assertion that has to exist before the first replicate, not after the first surprise.

Related

• P0: every streaming receipt declares tokenization.method server_usage while the tokens are client-counted #2754 — the apr-side mirror: streaming receipts declare server_usage while the tokens are client-counted, with all samples carrying prompt_tokens: 0. Same root fallback, this issue adds the comparator lane and the non-streaming exit.
• nothing asserts W1's 512 ± 8 prompt-length invariant that the corpus _meta promises, and I-9 forbids re-running a band to green #2778 — nothing asserts the 512 ± 8 invariant. This issue shows that even once something does, the assertion's verdict depends on which lane it runs against.
• the protocol's hard 120s per-request timeout makes mini/W1 unmeasurable at any budget, and intel's first unwarmed request clears it by 1.2% #2788 — the 120 s per-request timeout makes other cells unmeasurable; same class of "this cell cannot be produced".
• The §4.4.6 rule that should have caught the 4.09% divergence is itself unimplemented — filed separately.
• Epic Epic: APR-PERF-GATE-001 — the receipt rule: a performance number is evidence only if something can prove how it was measured #2706.

Sibling findings from this investigation

One tokenization/counting investigation against the perf-gate comparator lane produced five findings. Each is self-contained; together they are why no Arm B ratio is currently trustworthy.

Shared setup: apr 0.64.0 (745fa8588) CPU build · llama-server 7746 (39173bcac) CUDA, 29/29 layers offloaded · model qwen2.5-coder-7b-instruct-q4_k_m.gguf sha256 509287f78cb4d4cf6b3843734733b914b2c158e43e22a7f4bf5e963800894d3c · canonical tokenizer sha256 c0382117ea329cdf097041132f6d735924b697924d6f6fc3945713e96ce87539 (7,031,645 bytes). All measurements reproduced twice.

Epic: #2706.