[BlivionIaG]

Part of #1419.

Draft, slice 3 of 7 in the chain of #1419. Stacked on slice 2: this branch contains the earlier slices, review only the last commit. Numbers marked "pending" are being measured and will be filled in before this leaves draft.

What

Deep prefill for the offloaded pack: long prompts run through a tiled prefill path that overlaps the expert copies with compute (ring of staged expert lines, paired head, window/writeback, expert-parallel prefill), instead of the batched pass of slice 2.

Exactness

The deep path accumulates in a different order than the batched pass, so it is not bit-identical to slice 2 on long prompts: the docs output hash changes from 1882ea29ebf6 (slice 2) to e6834625432c; the 40-token and Zig prompts are unchanged (90e6c65ca5ba, 6dfc89b1abb6). Both paths are self-consistent: drafted equals plain, a resumed prompt-cache request equals a fresh one, and reruns are equal. This is a reordered accumulation inside the prefill kernels; FP64 per-class receipt: pending.

Notes on the measurements

Prefill is ~5x faster on long prompts; short prompts and decode are unchanged. The deep path reserves 727 MB more device memory than slice 2.

Tests run

On this tree: zig build native -Dnvcc=nvcc -Dsm=120, test-flashnext-exl3-host (111 pass), compile-cuda-tests, test-kernel-contract on sm_120; zig fmt --check; python3 tools/zig/lean_check.py at 428 problems, the same set as upstream main; no file we touch over 600 lines. Linux zig build test and the HIP build/test were run on slices 1, 2 and 7 with no failure and the same skips as main; the CI-style runs for this exact commit are pending. Not run: other architectures, test-golden on a Mac for this slice (run on slice 1).

Notes

sm_120 only; the loader gate from slice 1 refuses other architectures. No tool or AI trailers; no block comments.

Receipts (2x RTX 5060 Ti 16 GB sm_120, PCIe 4.0 x8, EPYC 7452, 251 GB RAM, driver 615.71, CUDA 13.4, Zig 0.17.0)

Measured on this slice's tree before the rebase onto current main (same code); a smoke run on the rebased heads of slices 1, 2 and 7 reproduces every output hash above and drafted equals plain, with plain timings within 5%. Open item: with drafts on, cold TTFT was higher on that run (slice 7 Zig 19.2 s against 10.9 s plain, docs 21.0 s against 10.7 s); the earlier run above shows 11.5 s and 11.2 s, so it is being re-checked. 192 output tokens, temperature 0; cold is the first request after server start, warm the identical repeat; the bench has no prompt cache. Peak memory and the prompt-cache check are in the receipt files.

bench2.py: three prompts (short = 40 tokens, zig = 16,022 tokens of source files, docs = 12,780 tokens of docs), 192 new tokens, temperature 0, ignore_eos, thinking off, streamed over the OpenAI chat API, two repetitions each. Cold = the first request of that prompt after server start (includes first-touch and graph capture), warm = the identical repeat. The prompt cache is off (--prompt-cache-gib 0 or none), so the warm repeat prefills again from scratch. Prefill tok/s = prompt tokens / TTFT; decode tok/s = (gen-1) / (last-first chunk span); mean ITL = that span / (gen-1).

Plain decode (no drafts)

prompt	prompt tok	gen tok	TTFT cold s	TTFT warm s	prefill cold tok/s	prefill warm tok/s	decode cold tok/s	decode warm tok/s	mean ITL cold ms	mean ITL warm ms	text hash
short	40	192	0.62	0.18	65.0	221.2	40.17	42.29	24.90	23.65	90e6c65ca5ba
zig	16022	192	10.46	10.21	1531.9	1568.7	33.44	34.12	29.90	29.31	6dfc89b1abb6
docs	12780	192	10.28	10.02	1242.9	1275.3	34.75	36.50	28.78	27.39	e6834625432c

Drafted decode (MTP head loaded, drafts enabled)

prompt	prompt tok	gen tok	TTFT cold s	TTFT warm s	prefill cold tok/s	prefill warm tok/s	decode cold tok/s	decode warm tok/s	mean ITL cold ms	mean ITL warm ms	text hash
short	40	192	0.69	0.27	57.6	146.1	54.37	58.84	18.39	17.00	90e6c65ca5ba
zig	16022	192	11.04	10.77	1451.5	1487.5	101.67	102.52	9.84	9.75	6dfc89b1abb6
docs	12780	192	10.92	10.69	1170.5	1195.9	51.59	55.70	19.38	17.95	e6834625432c

drafted == plain

prompt	plain hash	drafted hash	drafted == plain (all reps)	plain decode cold	drafted decode cold	ratio	plain ITL	drafted ITL	plain decode warm	drafted decode warm
short	90e6c65ca5ba	90e6c65ca5ba	yes	40.17	54.37	1.35x	24.90	18.39	42.29	58.84
zig	6dfc89b1abb6	6dfc89b1abb6	yes	33.44	101.67	3.04x	29.90	9.84	34.12	102.52
docs	e6834625432c	e6834625432c	yes	34.75	51.59	1.48x	28.78	19.38	36.50	55.70

bench_openai 256-token greedy token hashes, plain vs drafted: fibonacci-raw: plain ['97f8951316ab'] drafted ['97f8951316ab'] -> equal; gpu-chat-no-think: plain ['de2cc2940523'] drafted ['de2cc2940523'] -> equal

Repo-client receipts

bench_openai decode, 256 tokens x3 greedy (plain)

prompt	decode tok/s median	3 reps	TTFT s	prefill tok/s	mean ITL ms	ITL p50	ITL p95	token sha
fibonacci-raw	36.11	35.40/36.11/36.54	0.102	137.9	27.69	26.2	35.0	97f8951316ab
gpu-chat-no-think	39.31	38.70/39.31/39.67	0.192	161.2	25.44	24.9	29.3	de2cc2940523

bench_cache cold prefill, exact token-id prompts, --require-cold (plain)

requested tok	prompt tok	cold prefill TTFT s	prefill tok/s	decode tok/s	cache state	client verdict
2048	2048	2.31	886.3	28.73	cold	qualified=True; repeats cold TTFT 2.2/2.2
8192	8192	5.26	1556.1	29.09	cold	qualified=True; repeats cold TTFT 5.2/5.2
30000	30000	20.49	1464.2	27.94	cold	qualified=True; repeats cold TTFT 20.3/20.3

bench_openai decode, 256 tokens x3 greedy (drafted)

prompt	decode tok/s median	3 reps	TTFT s	prefill tok/s	mean ITL ms	ITL p50	ITL p95	token sha
fibonacci-raw	60.83	59.57/60.83/61.66	0.158	88.8	16.44	0.0	85.1	97f8951316ab
gpu-chat-no-think	59.08	58.01/59.08/59.77	0.264	117.2	16.93	0.0	63.8	de2cc2940523

bench_cache cold prefill, exact token-id prompts, --require-cold (drafted)

requested tok	prompt tok	cold prefill TTFT s	prefill tok/s	decode tok/s	cache state	client verdict
2048	2048	2.43	844.1	34.05	cold	qualified=True; repeats cold TTFT 2.3/2.4
8192	8192	5.55	1474.8	34.72	cold	qualified=True; repeats cold TTFT 5.5/5.4
30000	30000	21.52	1394.2	36.83	cold	qualified=True; repeats cold TTFT 21.4/21.3

bench_concurrent, 1 and 2 streams, --alone --serial (PARALLEL=2 server)

prompt	streams	aggregate tok/s	per-stream tok/s	TTFT max s	mean ITL ms	ITL p95 ms	end-to-end tok/s	failed	equal-to-alone	equal-to-serial(no draft)
code	1	37.3	37.3	0.09	26.78	31.5	37.0	0	3 eq / 0 neq / 0 fail	1 eq / 0 neq / 0 fail
code	2	70.8	35.5	0.14	28.18	32.9	70.1	0	6 eq / 0 neq / 0 fail	1 eq / 0 neq / 0 fail
chat	1	37.3	37.3	0.22	26.82	31.4	36.3	0	3 eq / 0 neq / 0 fail	1 eq / 0 neq / 0 fail
chat	2	71.6	36.0	0.29	27.79	31.7	69.8	0	6 eq / 0 neq / 0 fail	1 eq / 0 neq / 0 fail

[ashhart]

The prefill speedup is impressive: the 16k-token prompt goes from 56 s to 10.5 s. The blocker is arithmetic.

The deep path is on by default for segments of 512 rows or more (exl3/prefill/cuda_exl3_prefill_state.zig:22,28;
cuda_exl3_prefill.zig:211-212). It runs dense projections of 64 rows or more on #806's candidate tiles
(cuda_exl3_prefill_tiles.zig:10-14) and experts on the new 64-member tiles, while decode and shorter segments use
Linear.matmul and runSlots. The docs prompt's hash moving from 1882ea29ebf6 to e6834625432c shows the
arithmetic changed.

Keeping cache states on the step grid (snap_grid) is a careful way to make resumes match fresh runs. Even so, a
prompt's tokens now depend on the step size and on whether the deep buffers could be reserved. When they can't, at
startup (cuda_exl3_prefill.zig:79) or at request time (:267), the batched pass takes over. Prompt rows also no
longer match serial decode, which is our oracle.

To land:

• The prompt kernels need to produce the decode kernels' bytes for every row, whatever the chunk or segment length.
  Main's NVFP4 Flash Next prompt tiles do this; see the coverage section of docs/kernel-contract.md.
• Register tf_exl3pf_gu and tf_exl3pf_down with replaces against the decode kernels, so the contract checks the
  bytes.
• Add a test that shows the same tokens with deep prefill on and off, and at two step sizes.

Once prompt rows match decode, the grid is no longer needed for exactness.

The ring, overlap and prefetch work can stay as it is.