[imD-5]

Effect

• One agent turn of a 94K-token chat: tokenizer work 2.67 M allocations / 694 MiB to 4.2 K / 4.2 MiB.
• Rank 0 growth over 30 such turns: 14.2 MiB a turn to 4.7-8.3 MiB.
• Replies unchanged: greedy token_sha identical between the two images, and --alone --serial equal in every cell.

Reproduce

python3 tools/bench_concurrent.py http://<head>:8100 <MODEL> --alone --serial   # concurrent equals solo, drafted equals serial
python3 -m pytest -q tests/test_prompt_pieces.py                                 # ids equal the tokenizer's

• Two images, one session, same pair: a token_sha per reply, the same conversation re-sent with a new last turn each time.

Change

• Adds server/prompt_pieces.py and one call in cuda/server.py: the rendered prompt is cut after each added token and only unseen pieces are encoded.

Why

• The tokenizer splits its input at the added tokens it matches before normalization, so a piece that ends in one encodes alone to the ids it gets inside the whole text.

Type

• type: fix: a bug fix, or a fix after review findings

Tests

• 9 new tests, failing before the change; the suite's 41 pre-existing failures are unchanged.

Checklist

• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

AI disclosure

AI: Claude Opus 5.5 via Hermes Agent
Checked: by author