EXL3 Qwen3.8-Flash-Next on CUDA sm_120: expert offload, TP=2, graphs and MTP (pilot; the chain that replaces draft #1113) #1419

What

Serve the Qwen3.8-Flash-Next EXL3 3.05 bpw checkpoint (85 GB) on 16 GB RTX 5060 Ti cards with the native CUDA engine: one card, or two with expert-parallel TP=2. The routed experts live in pinned host RAM with a per-layer device cache, decode and prefill run under CUDA graphs, KV pages come from the core pool (a host-RAM tier for it is part of the chain), and MTP drafts are guarded per stream by measured tokens per millisecond.

The origin draft was #1113 (closed). This issue is the plan to land it as focused PRs, and the place where each one is linked.

Scope: a pilot on sm_120

• sm_120 only (RTX 50-series, RTX PRO 6000 Blackwell). It is the only architecture we can test, so it is the only one this work claims.
• Offload to host RAM does not make sense on sm_121 (GB10/Spark: unified memory), and other architectures are untested. The loader refuses EXL3 Flash Next on any other compute capability, with a message naming the gate. Another architecture gets enabled by a PR from someone who can run its gates, with its receipts.
• No change to NVFP4 or the other families' bytes.

The chain (draft PRs, each stacked on the one before)

Closed: #1113 (the origin draft) and #1447 (the DeltaNet head split, closed because it is not bit-identical to TP=1; not pursued unless it can be made so).

Boundaries are confirmed as each draft lands. Each PR carries its tests, kernel-contract records, receipts and the lean checker at zero new problems over main's baseline. We run atc overlap before each PR.

How expert offload works

• What stays on the card: the dense, attention and DeltaNet weights (about 12.9 GiB at one rank), the KV pages and the per-layer expert cache.
• What lives in host RAM: the routed experts of every layer, in a pinned bank (about 42 GiB at one rank; with TP=2 each rank holds its half, about 22 GiB). A forward step reads only the experts the router picked.
• The device cache: each layer keeps a pool of expert "lines" of 1.76 MiB each (95 lines per layer for one user on one rank, 80 to 88 at TP=2). A host planner keeps a frequency profile and decides which experts stay resident; a miss is copied over PCIe on demand. Hit and miss telemetry is reported per layer.
• What a miss costs: PCIe 4.0 x8 bounds it. The first extra verify row of a drafted round costs about 18 ms when its experts miss. That is why drafts are guarded: they only pay where the draft lands often, so the lane engine measures tokens per millisecond for plain and drafted rounds per stream and runs the cheaper one.
• TP=2: experts are split between the ranks (expert-parallel), each rank with its own cache and bank. The per-slot expert outputs are summed in the same order the one-rank path uses, so the two configurations give the same tokens (review requirement on EXL3 Flash Next: expert-parallel TP=2 (same tokens as TP=1) and MTP drafts (2/8) #1442).
• Prefill streams the experts a chunk needs; deep prefill overlaps those copies with compute. One-rank prefill is bound by that streaming (about 130 tok/s); two ranks with batched prefill about 300; deep prefill reaches 1.2k to 1.5k tok/s.
• Memory on one 16 GB card: about 12.9 GiB of weights plus about 8 GiB of expert cache, 14.7 GiB peak.

Full TTFT, prefill, decode and mean ITL tables (cold and warm) are in each PR body.

Questions (answered by review where noted)

1. Is a sm_120-only pilot with a refusing gate acceptable, to be widened by later PRs with their own receipts? (Reviewers called the shape a good one; no objection so far.)
2. Reduction order: answered in review. The bar is bit-identical to TP=1, not an FP64 receipt. TP=2 (EXL3 Flash Next: expert-parallel TP=2 (same tokens as TP=1) and MTP drafts (2/8) #1442) is being reworked to match, and the DeltaNet head split was closed for the same reason.
3. Kernel-contract records: sm_120 only for the new kernels, with other architectures listed as not run?
4. The repo's bench clients will be used for the receipts, with cold-prefill arrays at 2k, 8k and 30k tokens (the served window is 32k).

Overlap

We run atc overlap before each PR and rebase over whatever lands first rather than duplicate it. Shared CUDA primitives: where they land first, our kernels will use them.

We are doing this work.