Nemotron-Labs Dory 1.0 4B on native CUDA: a looped nGPT family (dory) #1233

Nemotron-Labs Dory 1.0 4B (nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-NVFP4, OpenMDW-1.1) is a looped nGPT model: 16 input sublayers, 40 recurrent sublayers run L times (L = 1..4, default 2), then 16 output sublayers. Attention and MLP sublayers alternate. Attention is 16/4 heads at head dim 256, sliding (window 1024) and full in an S m S m S m F m pattern, with an output gate. There are no norm weights: every residual is an L2-normalized lerp ("slerp"), and q and k are L2-normalized after RoPE. The head and embeddings are bf16, 131072 × 2560, untied. NVIDIA's NVFP4 build is ModelOpt W4A16, 3.2 GB; there are MXFP8 and BF16 builds too.

We'd like to build it as a native CUDA family (dory), and would welcome help with any part.

Why it fits TF now. #1123 and #1124 (window and global attention at head dim 256 and 512) have landed, so the attention is covered. The projections go through the existing ModelOpt NVFP4 dense lane (as GLM FP4-Local), the head through dense_rows, sampling and lanes unchanged. All of the new math is rows kernels; there is no new mixer.

References. None has merged: vLLM #60149 (NVIDIA, Apache-2.0), llama.cpp #30306 (MIT), mlx-lm #1988 (MIT). We'll pin their SHAs and cross-check a Zig f64 reference against vLLM's model code run in float64 on tiny weights.

KV policy. per_loop (one cache per recurrent attention sublayer and loop, vLLM's default) is the only mode that can pass drafted == plain and resumed == fresh: in last_loop, rows inside one chunk or verify window see the current loop's KV and earlier chunks see last-loop KV, so the output depends on chunk boundaries (confirmed in the #60149 thread). We'd ship per_loop only. At L=2 it is 56 KiB of KV a token (7 GiB at 128K).

Steps (small PRs, kernels first):

1. Family skeleton, host only: config, weight map, the virtual-layer schedule for L = 1..4, memory estimate.
2. f64 reference of the whole forward, per_loop chunk-invariance asserted.
3. Kernels, each with contract gates and planted controls that drop one rounding point:
   • norms: L2 rows (+add) and the slerp tail;
   • qk_rows: RoPE first, then L2, q post-scale vector, v raw;
   • act_rows: scaled SwiGLU + row L2 over 7168, scaled sigmoid gate + row L2 over 4096;
   • dense_rows: a per-column fp32 logit-scale epilogue.
4. CUDA forward, one stream, NVFP4: per-sublayer taps vs the reference, greedy tokens vs vLLM #60149 at L = 1, 2, 4.
5. Serving on tensorfold-native (graphs per row bucket and L, chat template, <function= tools, <think>), then prompt cache and snapshot, then MXFP8/BF16, then paged KV.

Expected decode at short context on a GB10, NVFP4, from bytes per token: about 87 / 62 / 39 tok/s at L = 1 / 2 / 4.

Questions:

1. Loop count: a load option (--loops, default 2), or per request (for example through reasoning_effort)? Per request costs max(L) per shared round and a graph set per L.
2. per_loop only, or also last_loop as an opt-in mode outside the exactness gates?
3. Rounding points: vLLM's bf16 steps, llama.cpp's fp32 activations, or TF's own (fp32 inside each rows kernel, bf16 at stores)?
4. The kernel extensions as separate PRs ahead of the family, as with cuda norms: bf16 residual tails with a branch norm, a layer scalar and MLX's rounded rows #1056 and cuda act_rows: GeGLU and SwiGLU in MLX's bf16 steps #1068-cuda qk_rows: head dims to 512, a weightless V norm and MLX's rounded rows #1071?
5. Paged KV on core kv_pool from the start, or contiguous first?