[imD-5]

Effect

• Cold 64K prefill on two GB10 (TP2, DeepSeek-V4.1-Flash-1017 EXL3 2.9 bpw): 512 rows 48.09 s, 2048 rows 30.87 s, 4096 rows 28.14 s.
• 32K: 24.27 / 15.49 / 14.40 s. 8K: 6.10 / 4.06 / 3.94 s. Decode unchanged: 60.4 / 59.9 / 64.5 / 69.2 tok/s.
• 8192 rows does not run: its MoE step needs about 1.3 GiB more than the allocator ceiling leaves.

Reproduce

# one round per chunk size: fresh prompts, spill off, restart between rounds, median of 3 reps a length
TF_DS_PREFILL_CHUNK=4096 python3 tools/prefill_cold.py build <MODEL_DIR> prompts-4096.json
TF_DS_PREFILL_CHUNK=4096 python3 tools/prefill_cold.py run http://<head>:8100 <MODEL> prompts-4096.json out.json
python3 tools/bench_openai.py http://<head>:8100 <MODEL> --output openai.json

• Engine: commit 852e6b8 on deepseek-v41-tp2, torch from nvcr.io/nvidia/pytorch:26.07-py3, checkpoint EXL3 2.9 bpw on LibertAI @ afc7aa0a.
• Spill off (TF_DS_SPILL_GIB=0) and a restart per round, or a kept prompt resumes instead of prefilling.

Change

• engine.PREFILL_CHUNK: default 512 to 4096, one line plus its comment.

Why

• A chunk is the rows one forward processes, so a bigger chunk reads the expert weights once for more of them; 512 pays that read every 512 rows.
• 4096 is the top that runs here, because 8192 crosses the ceiling inside the MoE.

Caveats

• A chunk size is not bit-neutral: 4096 gives other (reproducible) tokens than 2048 or 512.
• Measured on two GB10 only; a single card has less headroom and deserves its own check.

Type

• type: feature: new code (a feature, tool or add-on)

Tests

• No test covers a default; compileall passes and the engine serves at 4096 (verify.sh 5/5).

Checklist

• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

AI disclosure

AI: Claude Opus 5.5 via Hermes Agent
Checked: by author