Repository navigation
Flash Next CUDA: prompt chunk size against decode waits, how other engines handle it, and options #1554
Description
Follow-up to question 1 on #1143 ("should the default move, or stay an option?"), which stayed open. #1553 documents the trade-off as it stands. This issue asks which way the engine itself should go.
What we measured
On one GB10, nvidia/Qwen3.8-Flash-Next-FP4-Local at b9ce18a6, main a05d62003, --parallel 3 --no-drafts --prompt-cache-gib 0. The four servers ran alternately (1,536, 8,192, 8,192, 1,536), and each cell is the median of 3.
| 1,536 (default) | 8,192 | |
|---|---|---|
| First token, 1,922 / 8,361 / 31,901 tokens | 1.06 / 4.49 / 17.8 s | 0.93 / 3.94 / 15.0 s |
| Longest wait between a decoding stream's tokens while an 8k prompt runs | 0.90 s | 3.78 s |
| Memory available after load | 44 GB | 38 GB |
- Where the gain comes from: all of it is in the two NVFP4 expert kernels. Every pass reads the weights of each expert its rows route to: about 350 of 512 at 1,536 rows, 1.84 MB of gate and up weights each. Fewer passes therefore mean fewer reads.
- Where the cost comes from: a pass runs to its end, so a stream that is decoding gets one token per pass.
- Exactness: the bits do not depend on the chunk. All 272 token-hash comparisons in the Flash Next CUDA: NVFP4 experts on prompt tiles with the decode form's sums, bit for bit #1143 sweep were identical across 1,024 to 8,192 rows: greedy and seeded, 1 and 4 streams, cold, warm and resumed. The cuts sit on the 128-row grid and the kernels are row-invariant.
How other engines handle it
None of them interrupts a pass. Each one caps the prompt work per step and lets decodes run in between. The trade-off is a setting:
- vLLM V1: chunked prefill is on by default. The scheduler "batches all pending decode requests before scheduling any prefill operations", then fills the rest of
max_num_batched_tokenswith prefill, chunking a prompt that does not fit. In the docs' words, "smaller values (e.g., 2048) achieve better ITL because there are fewer prefills slowing down decodes", while "higher values achieve better time to first token". So decodes and a prompt chunk share one forward pass. - SGLang:
--chunked-prefill-sizecaps a chunk.--enable-mixed-chunk(off by default) mixes "prefill and decode in a batch when using chunked prefill". - llama.cpp server: continuous batching is on by default. One batch takes every slot's next token and fills up with prompt tokens to
-b(default 2048 logical, 512 physical).
TF already interleaves at pass granularity: the lane host's prompt share from #1188, and Flash Next's prefill_step runs one chunk. What it does not do is put decode rows into the prompt pass.
Options
| What | For | Against | |
|---|---|---|---|
| A | Keep 1,536 and document it (#1553) | No code change | Single-stream users leave 12–16 % of prompt time unused unless they find the variable |
| B | Raise the fixed default, for example to 4,096 | About −8 to −11 % prompt time for everyone (#1143 data) | Decode waits about 3 s instead of about 1 s with several streams; about 2.3 GiB more scratch |
| C | Choose each pass's rows when it starts: the large chunk when no other stream is decoding, the small one otherwise | Single-stream gets the large-chunk TTFT, and the waits under load stay at today's levels. Exact, because bits do not depend on the chunk. Small change in the lane host and Flash Next's prefill_step |
A request that arrives during a large pass waits for that pass (up to about 3.8 s at 8,192). Scratch is sized for the large chunk all the time (+6 GB at 8,192) |
| D | Mixed passes, as vLLM and llama.cpp do: the decoding streams' rows ride in the prompt pass | The standard approach. On an MoE the decode rows reuse the expert weights the pass reads anyway, so they cost almost nothing | A large change: plans and kernels take both kinds of row in one pass, and MTP drafting complicates it. Exactness has to be re-proven for mixed passes. Waits are still one pass long |
| E | Pause a pass at a layer boundary and run decode rounds in between | Bounds the waits at any chunk size. Exact, because the same kernels run on the same data | TF-specific. The paused pass's activations need their own buffer, kept apart from the decode rounds' scratch. More moving parts than C |
Activity
ashhart commentedon Oct 28, 2026
/atc claim
ashhart commentedon Oct 28, 2026
/atc release
ashhart commentedon Oct 30, 2026
Thanks @jschmied for the measurements and the survey; the table of options makes the choice easy to see.
We'll go with C: choose a pass's rows when it starts, the large chunk when no other stream is decoding and the small
one otherwise. The #1143 sweep shows the bits don't depend on the chunk, so replies stay exact. It starts as an
opt-in, and it becomes the default only if the receipts show nothing slower. Those receipts are your table's rows for
both settings, plus a request that arrives during a large pass, which is the cost C adds.
Memory counts as a cost too. +6 GB held all the time at 8,192 rows is a lot on a 128 GB box. Please report the memory
after load with C on, and try a smaller large chunk if 4,096 rows takes most of the gain.
D (mixed passes) and E (pausing a pass) are not taken now. D means proving exactness again for every family on the
lane host, and MTP complicates it. E has more moving parts than C for the same bound. #1553's docs stay.