[jschmied]

Gemma 4 and Qwen 3.5 on CUDA load their layers one at a time: fault the mapped pages in, upload, pack, synchronize. The page faults run on the loading thread at about 1.4 GB/s buffered, so a cold load mostly waits on the disk. GLM already avoids this with TF_GLM_LOAD_PREFETCH.

This PR moves that knob into core and uses it for both families:

• core/checkpoint_prefetch.zig: readerCount / configured() read TF_LOAD_PREFETCH. Unset now means on, with 16 readers (GLM's default). 0 is exactly main's path: no readers and no per-layer release, so pages are released once at cache.close. Anything up to max_readers sets the count. Any other value now fails the load with error.BadLoadPrefetch, where main ignored the variable. A test covers all four cases. core.checkpoint.loadReaders re-exports it.
• gemma4/cuda_weights.zig and qwen3_5/cuda_weights.zig: every layer's Checkpoint.prefetch(prefix, readers) reads it into the page cache on its own threads while the loop uploads and packs it. When a layer was read ahead (readers > 0), release() drops its pages once it is on the device, so the page cache never holds more than about one layer.

Cold load on a GB10 (page cache dropped before every start, NVMe; order main, branch, branch, main; the reply to a fixed greedy prompt is byte-identical in every run):

model	main (s)	this PR (s)	warm page cache (s)
Qwen3.8-27B NVFP4 (22.0 GB)	25, 22	15, 15	9
Gemma 4 31B 4-bit (18.4 GB)	18, 18	11, 11	7

Checks: lean (0 new), zig build test, the CI set and diff-check all pass on 2737e5f.

A question before a follow-up: starting layer i+1's read while layer i packs (one-layer lookahead) takes this further, to 13/12 s for the 27B and 8/8 s for Gemma, close to the warm floor. Same replies. The catch is that the page release then can't be per layer: release() drops a whole file, and the next layer's pages are already in it. So the cache would grow to the model's size until cache.close. A clean fix is a range release in core (fadvise/madvise on one layer's byte span). Do you want that as a follow-up PR?

[ashhart]

Thanks @jschmied. We reviewed this independently, and it's safe. Readers only pread the layer's header spans into private 8 MiB buffers, the loader's bytes still come from their own offsets with the same checks, nothing reads past a shard, the join runs before the release on every exit, and it composes with #1868 and #1901. One change: neither setting is main's path. Unset means 16 readers plus a whole-checkpoint ck.release() after every layer, and even TF_LOAD_PREFETCH=0 keeps that per-layer release (madvise and fadvise DONTNEED on every shard), where main releases once at cache.close. Could you release only when read-ahead ran (readers > 0), so 0 is exactly main? And please say plainly in the body that unset now means on (16 readers). A bad value now fails the load with error.BadLoadPrefetch, while main ignored the variable, which is fine, but worth a line in the body.

[jschmied]

Thanks @ashhart. Done in 2737e5f: the per-layer release() now runs only when read-ahead ran (readers > 0), so TF_LOAD_PREFETCH=0 is exactly main's path, with one release at cache.close. The body now says plainly that unset means on (16 readers), and that a bad value fails the load with error.BadLoadPrefetch where main ignored the variable. Lean (0 new), zig build test, the CI set and diff-check pass on 2737e5f.

[ashhart]

Thanks @jschmied: all three asks are answered in 2737e5f. The per-layer release is deferred while readers are open, TF_LOAD_PREFETCH=0 starts no reader and keeps main's single read, and the body now says plainly that unset means on (16 readers). Since that changes the default load for Gemma 4 and Qwen 3.5 on CUDA, we're running main against this head on two GB10s (Qwen3.8-27B and Gemma 4 12B) for tokens and load time. It lands when those are green.

[ashhart]

Thanks @jschmied. GB10 A/B of 2737e5fbc against main:

• Qwen3.8-27B: loads in 12.3 s and 12.3 s against 18.3 s and 18.9 s on main (about 34% faster). 0 of 20 prompt rows differ, and decode is level at every stream count (one stream within ±0.3%, runs overlapping).
• Gemma 4 12B: GREEN as well.

With the earlier re-review's LAND, this is queued to land.

[ashhart]

Thank you @jschmied. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ashhart]

Thanks @jschmied! This is in TensorFold 1.2.2, out now: https://github.com/ashhart/TensorFold/releases/tag/v1.2.2