[jschmied]

Gemma 4 and Qwen 3.5 on CUDA now read their checkpoints with direct reads. Cold load is 28–31 % faster with the same weights and replies. The loader is shared core code that Flash Next now uses too.

What changes

• core/checkpoint_direct.zig (new): a layer loader's tensors are read past the page cache into reused buffers, and the next layer is read while the current one packs. Checkpoint.get() hands out the copies, so each family's change is a few lines in its cuda_weights.zig. TF_LOAD_DIRECT=0 keeps core: TF_LOAD_PREFETCH; Gemma 4 and Qwen 3.5 CUDA read each layer ahead while it packs #1928's read-ahead.
• core/direct_gather.zig: Pair reads the next views on a thread while the last read's are used. Flash Next's own copy of it from flashnext cuda: the direct expert load gathers the next chunk while the current one packs, and tensor() stops reopening shards #1960 now uses this one.
• Nemotron already loads with direct reads (zig Nemotron CUDA load: direct I/O model loading #458), and that loader is left as it is. GLM has its own loader and isn't touched.
• Host memory during load grows by about two layers' buffers (~0.6 GB).

Measured (one GB10, page cache dropped before every start, A-B-B-A)

model	main cold load (s)	branch (s)	replies
Gemma 4 31B 4-bit	11.55 / 11.29	7.79 / 8.04 (−31 %)	identical
Qwen3.8-27B NVFP4	15.30 / 15.05	11.29 / 10.80 (−28 %)	identical
Qwen3.8 Flash Next FP4	44.11 / 43.86	44.11 / 43.61 (level)	identical

Checks

• zig build test and test-cuda-host pass, and so does the no-GPU CI set.
• Lean finds 0 new problems (425 on base and head). zig fmt --check and git diff --check are clean.
• The branch merges cleanly into today's main.

[ashhart]

Thanks @jschmied. One shared direct loader with the next layer read while this one packs is a clean way to give Gemma 4 and Qwen 3.5 what Nemotron already has, and 28–31% faster cold loads with identical replies is a good result. Since this changes the default load path, two things before it queues:

1. A CHANGELOG line under Unreleased: direct reads are the default for Gemma 4 and Qwen 3.5 on CUDA, about 0.6 GB more host memory during load, and TF_LOAD_DIRECT=0 for the old read-ahead.
2. One cold load off the GB10, if you have one: a discrete card on an ordinary NVMe host, main against branch, ABBA, page cache dropped, so the new default is shown on both kinds of memory.

The Flash Next part reads correctly to us; its speed is reviewed separately.

[ashhart]

Thanks @jschmied. The Flash Next side is fine.

Pair keeps what Next did. The two arenas still alternate, so a chunk packs from one while the next fills the other, and the gather mark still runs on the reading thread before its read. A failed gather still comes back through the same wait. The only change is that chunk 0 now lands in the second arena rather than the first, which doesn't matter: the globals' reads are finished by then.

Your level cold load on FP4 does exercise this path, since that checkpoint is ModelOpt and loads through cuda_modelopt_threads.loadMode. So nothing more is needed from the Flash Next side.

[jschmied]

Thanks @ashhart. The CHANGELOG line is in: b84620b.

We have no discrete card here, only the GB10, so I can't do the second one myself.

@BobClawblaw, could you run this on your RTX 5090, if that host loads from an ordinary NVMe drive? One cold load, main against this branch, A-B-B-A, any Gemma 4 or Qwen 3.5 checkpoint that fits the card. Please drop the page cache before every start (sync; echo 3 > /proc/sys/vm/drop_caches) and post the load seconds for all four, plus the checkpoint, the drive and its filesystem. A reply to one prompt in each arm would show the weights match. Direct reads depend on the filesystem, so a Linux run is the useful one.

On the GB10 the same comparison gave −28 % on Qwen3.8-27B NVFP4. On a discrete card the time goes to the disk and to the upload to the card, so the gain may well be smaller. That is what we'd like to see.

[ashhart]

Thanks @jschmied. The CHANGELOG line covers it: direct reads as the default, the extra host memory and
TF_LOAD_DIRECT=0. The second load was only if you had a discrete card, so it doesn't hold this up. If a discrete-card
number arrives here, it'll go beside the GB10's. The byte test against the mapping, read ahead and out of order,
is what we need for a default change that must load the same weights. It goes in the landing queue once our own cells on this head are in: main against it, with tokens equal and speed level.

[ashhart]

/atc exchange 1816bcede6594bf1f83f3eef2a5a40c988a1bb32958fcfdbf10d8b7d29bb9482

[ashhart]

Our cells on this head (b84620b163) are in, main against it, runs ordered base, new, new, base, with the first run dropped. Replies equal main's in each, and speeds are level:

• Qwen3.8-27B MLX 4-bit on a GB10: -0.7% to +0.2%.
• Qwen3.8 Flash Next FP4 on a GB10: -0.8% to +0.7%.
• Gemma 4 12B 4-bit on a GB10: -0.8% to +0.6%.

It's in the landing queue now.