Repository navigation
GB10: Nemotron decode is 1.4 % (code) / 3.0 % (prose) slower on 1.0.3 than on 1.0.2; on 1.0.5 the prose gap is still there #627
Description
Hi Ash, thanks for TensorFold. This is a measurement report from one GB10, not a request to undo anything: I read the 1.0.3 note that on GB10 Nemotron's CUDA kernels are now all your own and that "the new kernels' error against a 64-bit reference is no larger", and I have not measured numerics. Whether a few percent on GB10 is worth acting on is your call.
Summary
On one GB10, 1.0.3 decodes Nemotron 3.5 Lightning 1.4 % (code) and 3.0 % (prose) slower than 1.0.2. Running 1.0.3 with 1.0.2's captured sm121 glue set (TENSORFOLD_CUDA_KERNELS) brings it back to 1.0.2's level, and nsys shows most of the new tf_nemo_* glue kernels taking longer per launch than their Triton counterparts (most for add_moe_norm 5.2x, topk 2.2x, router 2.0x). I repeated the swap on 1.0.5: prose is still 3.1 % faster with 1.0.2's glue set (3 of 3 rounds), code is +0.3 % and not clearly different.
Setup
- Hardware: Lenovo ThinkStation PGX (NVIDIA GB10, sm_121, aarch64; a different OEM box than the "GB10 (DGX Spark)" in your 1.0.3 note), 121 GB unified memory, Ubuntu 24.04.5 LTS, kernel 7.0.0-1019-nvidia, driver 580.178.04. One machine only.
- TensorFold: the official
tensorfold-1.0.2-linux-aarch64.tar.gz(sha2560e0c5f5d1012f781e97494df0bc0b1bbf5b561c639a51bad5f77a198c8ee1ce5),tensorfold-1.0.3-linux-aarch64.tar.gz(sha256054cacd129f1100a90acdaa5b3bec266f385cbda58fea72892094805700e3be5) and, for the last table,tensorfold-1.0.5-linux-aarch64.tar.gz(sha256bf2be81f28f4a70e166f09a76015aded62f51d01d77bf46eae9c9720fd409269), all checked against the companion.sha256files, unpacked into separate folders. - Model:
TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit@d9d758fb83953437f7263256b0d96157e2a348b8, drafter on. Config and tokenizer files are hash-matched to that revision; the five weight files were compared by byte size only. - Server, a fresh one per round:
tensorfold-native serve <model> --name bench --context 18432 --no-thinking --no-update-check(listening on localhost:8080). - Client:
tools/bench_openai.pyfrom v1.0.3,--tokens 256 --reps 5 --temperatures 0; promptsfibonacci-raw(code, 15 prompt tokens) andgpu-chat-no-think(prose, 35 prompt tokens); seeds 1234-1238 after one warm-up request per prompt; greedy. The client has a small local patch (4 added lines) that adds--plain(sends"draft": false), used only for the drafted-vs-plain check; without the flag it behaves like upstream. - Each arm is the median of its 5 reps; the tables show the median of 3 rounds with the range of the round medians. Arms alternate A B A B A B.
Results
1.0.2 vs 1.0.3 (release archives), tok/s:
| code | prose | |
|---|---|---|
| 1.0.2 | 202.60 (202.32-202.66) | 150.10 (149.29-150.42) |
| 1.0.3 | 199.71 (199.70-200.08) | 145.60 (145.57-146.00) |
| 1.0.3 vs 1.0.2 | -1.4 % | -3.0 % |
Every 1.0.3 round is below every 1.0.2 round.
1.0.3 as shipped vs 1.0.3 with 1.0.2's glue set (a second run, same method), tok/s:
| code | prose | |
|---|---|---|
| A: 1.0.3, own glue kernels | 200.05 (199.99-200.13) | 145.83 (145.70-145.85) |
B: 1.0.3 + TENSORFOLD_CUDA_KERNELS = 1.0.2's share/tensorfold/cuda/sm121 |
203.20 (203.04-203.48) | 150.90 (150.82-150.92) |
| B vs A | +1.6 % | +3.5 % |
B is +0.3 % / +0.5 % against the 1.0.2 numbers above (a different run, not interleaved). A needs no more rounds than B (rounds= in the server's done lines: code 64 vs 66, prose 105 vs 106) and accepts a similar share of its drafts (accepted=: code 192/244 vs 190/253, prose 152/219 vs 153/226), yet it is slower, so for these two prompts the difference is in the time per round (A and B produce different text, so this does not carry over to other prompts).
Exactness: in every arm the drafted output equals the plain output for both prompts (output_sha256, 10/10 per arm). B's outputs are identical to 1.0.2's for these two prompts at 256 tokens (c2087a9b602f..., a269d4b8e44e...); A's are 1.0.3's (06e78ce51435..., aa079ca401a3...), which fits the note that greedy tokens can differ from 1.0.2's.
1.0.5 as shipped vs 1.0.5 with 1.0.2's glue set (a third run, same method, no nsys), tok/s:
| code | prose | |
|---|---|---|
| A: 1.0.5, own glue kernels | 201.93 (201.47-202.21) | 145.70 (145.14-145.87) |
B: 1.0.5 + TENSORFOLD_CUDA_KERNELS = 1.0.2's share/tensorfold/cuda/sm121 |
202.58 (201.44-202.97) | 150.19 (149.28-150.41) |
| B vs A | +0.3 % | +3.1 % |
On prose B is ahead in every round (+3.6, +3.0, +2.5 %); on code the rounds disagree (+0.8, +0.2, -0.2 %), so I see no difference there. The drafted outputs of A are identical to 1.0.3's own-kernel outputs and those of B to 1.0.2's, for both prompts at 256 tokens; I did not run the drafted-vs-plain check on 1.0.5.
Per-launch kernel times (nsys)
Decode runs as CUDA graph replays, so nsys profile -t cuda --cuda-graph-trace=node --sample=none --cpuctxsw=none around the same serve command (nsys 2025.3.2), one bench_openai.py --reps 1 pass over both prompts, then nsys stats --report cuda_gpu_kern_sum --format csv. Without --cuda-graph-trace=node only the prompt kernels appear. Both versions were profiled the same way; arm B was not profiled.
Median time per launch, 1.0.2 (captured Triton, _add_moe_norm etc.) vs 1.0.3 (tf_nemo_*):
| op | 1.0.2 | 1.0.3 | factor |
|---|---|---|---|
| add_moe_norm | 3.84 µs | 20.00 µs | 5.2x |
| router | 8.86 µs | 17.92 µs | 2.0x |
| topk | 3.55 µs | 7.94 µs | 2.2x |
The other glue ops range from about the same (scan 19.55 to 20.45 µs, +5 %; conv 4.26 to 4.29 µs, +1 %) to about 2x (add_rmsnorm 1.89 to 3.78 µs, attn_merge 1.70 to 3.17 µs); attn_chunk goes from 12.86 to 19.84 µs (1.5x). The pairing of Triton and tf_nemo_* kernels is by name and call site (for example _add_moe_norm and tf_nemo_add_moe_norm). Launch counts in the 1.0.3 profile are 8,784 each for the three kernels above (4 requests of 256 tokens). The 1.0.2 profile also served four extra requests because of a slip in my script; I expect the per-launch medians to be insensitive to that, but it also has about 28 % more MoE launches than its round count predicts, and I could not explain that.
Workaround
TENSORFOLD_CUDA_KERNELS=/path/to/tensorfold-1.0.2-linux-aarch64/share/tensorfold/cuda/sm121 \
tensorfold-native serve <model> ...The server then logs glue kernels captured at ... instead of own glue kernels. The 1.0.2 set fits the 1.0.3 call sites: cuda_triton.zig is unchanged between v1.0.2 and v1.0.3, and the set holds 41 cubins covering all 15 glue functions in its aot.json. The switch is documented in RUNBOOK.md. Two things to know: the output then follows 1.0.2's numerics again (checked for the two prompts above), and the 1.0.3 and 1.0.5 aarch64 archives do not contain share/tensorfold/cuda/sm121/ (the 1.0.2 archive does), so a user has to take the set from the 1.0.2 archive.
What this does not show
- Which glue kernel costs how much. Arm B swaps the whole glue set at once, so the end-to-end difference is isolated for the set as a whole. I did not swap or time single kernels end to end, so I cannot say how much of the 1.4-3.0 %
add_moe_norm,routerandtopkexplain. They are the three with the largest per-launch ratio, nothing more; by absolute timeattn_chunk(+7 µs) is ahead oftopk(+4 µs). - Anything beyond one machine, one model (this 4-bit checkpoint), two short prompts (15 and 35 tokens), 256 generated tokens, one stream at a time. No other GB10, no long context.
- Numerics. I take your note about the error against the 64-bit reference as given.
- 1.0.4, and anything on 1.0.5 beyond the A/B table above. I did not measure 1.0.4. For 1.0.5 (the newest release when I wrote this) I have only the tok/s A/B: no nsys, so no per-launch times, and no drafted-vs-plain check. As far as I can see from the diff, the Nemotron glue
.cufiles,cuda_glue.zigandaot.zigare unchanged between v1.0.3 and v1.0.5, but other Nemotron CUDA files did change (cuda_engine.zig,cuda_lanes.zig,cuda_state.zig,cuda_native.zig,cuda_kernels.zig), so I cannot say why the code difference is not visible on 1.0.5. - The nsys numbers come from profiled runs, which are slightly slower than unprofiled ones. All tok/s figures above are from unprofiled runs.
Offer
I can attach the bench JSONs, the server logs and the two kernel-summary CSVs. I am happy to run more on this GB10: other nsys or ncu captures of the three kernels, a run on main, other prompts or context lengths, or a test of a patch (a branch or a binary) if you want a number for one.
Activity
ashhart commentedon Oct 10, 2026
Thank you, @Nipale-ai. This is exactly the receipt we needed: per-launch times, the glue-set swap as a control, and drafted == plain checked in every arm. We wrote the 1.0.3 glue kernels to make decode exact and Triton-free first, and we haven't tuned them yet. Your nsys table shows where that costs:
add_moe_normat 20 µs against 3.8, the router at 17.9 against 8.9 andtopkat 7.9 against 3.6, plusadd_rmsnorm,attn_mergeandattn_chunk.We'll tune those, keeping each output at least as close to a float64 reference as it is now, and gate the result on two GB10s against 1.0.2's glue set with your two prompts. We'll post the A/B here. Until then, the
TENSORFOLD_CUDA_KERNELSworkaround you found is fine to keep using, and we'll say here if a release changes the call sites it relies on.Nipale-ai commentedon Oct 10, 2026
Thanks Ash. When there is a build or branch, I'll run the same A/B on this GB10 and post the numbers here.
SvangenStudios commentedon Oct 11, 2026
I ran the same A/B on a second GB10, on 1.0.7. On prose the gap is smaller than in your 1.0.3 and 1.0.5 runs (+1.9 % against +3.5 % and +3.1 %). On code it's +0.8 %, larger than your +0.3 % on 1.0.5. B led A in every round on both prompts.
Setup.
TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit@d9d758fb, all files sha256-checked against the hub..sha256:tensorfold-1.0.7-linux-aarch64(1bc88821…) andtensorfold-1.0.2-linux-aarch64.tensorfold-native serve <model> --name bench --context 18432 --no-thinking --no-update-check, drafts on.tools/bench_openai.pyfrom v1.0.7,--tokens 256 --reps 5 --temperatures 0, withfibonacci-raw(code) andgpu-chat-no-think(prose).own glue kernels.TENSORFOLD_CUDA_KERNELSset to 1.0.2'sshare/tensorfold/cuda/sm121. The server logsglue kernels captured at ….Decode, tok/s (median of the round medians, with the three rounds in brackets):
B is ahead of A in every round on both prompts, by 0.7–0.9 % on code and 1.8–2.3 % on prose.
The output hash prefixes match those you published, for these two prompts at 256 tokens:
output_sha25606e78ce51435…(code) andaa079ca401a3…(prose).c2087a9b602f…anda269d4b8e44e….The glue swap still changes the replies: A generates different text from B and C, so the comparison covers the whole generation, not the kernels alone. The arms ran in a fixed A B C order, which doesn't rule out order effects. I didn't run nsys or the drafted-vs-plain check here.
jschmied commentedon Oct 14, 2026
A measurement on main after 1.0.10 (
1e76a70e), in case it helps decide what's left. One DGX Spark (GB10, sm_121, CUDA 13),TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit,tensorfold runwith 256 greedy tokens after a 36-token prompt. A is native glue, B is--kernelswith 1.0.2's capturedsm121set.Decode, ms/token, median of 3 interleaved rounds:
--no-drafts--no-draftsThe engine's own round costs at 1/2/4/8/16 rows are 9.21/11.74/15.32/21.27/30.28 ms (A) and 8.85/11.43/15.21/21.28/30.26 ms (B). So the gap sits at one row and has closed by four rows.
Where the 0.35 ms per step goes (nsys
--cuda-graph-trace=node,--no-drafts). Each MoE layer runs router → topk → plan → routed experts on a side stream, while the main stream runs the shared expert and then waits for it. The side stream is the critical path: A's main stream waits 1,489 µs a step, B's 561 µs.add_moe_norma launchattn_chunka launch (one-row decode)add_rmsnorma launchgroup_rmsnorma launchattn_mergea launchscana launchThe per-step column adds up to the measured 348 µs.
add_rmsnorm's part is what #796's open commit (b3ad1b9d) addresses.@plotarmordev, your
37da3ce8already coversadd_moe_norm's loads, the scan, and attention over several rows. To avoid working on the same kernels: unless you're already on them, I'll take the router,topkandadd_moe_norm, with every output byte-equal to today's (tf-cuda-test gluebyte-exact, the same reply shas). I'll leave one-rowattn_chunk,attn_mergeandgroup_rmsnormopen.ashhart commentedon Oct 14, 2026
Thanks, @jschmied. The per-launch table makes the gap concrete: about 0.35 ms a step at one row, closed by four rows, and most of it launch time on the MoE side chain that the main stream waits for. Taking the router,
topkandadd_moe_normwith byte-equal outputs suits us; we will gate it the same way (tf-cuda-test gluebyte-exact and the same reply shas) on a GB10. #796'sadd_rmsnormcommit is in the CUDA queue ahead of it.BobClawblaw commentedon Oct 14, 2026
Two data points from an RTX 4090 (sm_89, Windows, driver 616.56, CUDA 13.3), on main
2984ea01through #836's branch, in case they save someone a detour.Per-launch baseline on Ada,
nsys profile -t cuda --cuda-graph-trace=node --sample=none --cpuctxsw=nonearoundtensorfold-native serve TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit --context 18432 --no-thinking, onebench_openai.py --tokens 256 --reps 3 --temperatures 0pass overfibonacci-rawandgpu-chat-no-think, drafts on, own glue kernels (medians):tf_nemo_routertf_nemo_topktf_nemo_add_moe_normtf_nemo_attn_chunktf_add_rms_rows_bf16htf_plan_routedBy GPU time the three named kernels are 2.2 %, 1.8 % and 2.1 % of the decode on this card; the expert up and down kernels are 63 %. So on Ada the glue is cheap and the gap you see is the GB10's per-launch cost.
A fused launch is the wrong shape. I tried
tf_nemo_route_row: one block a decode row, the row in shared memory, every expert's slice partials by the same fma chainstf_nemo_routerruns, thentf_nemo_topk's picks from shared memory. It is byte-equal (tf-cuda-test glue1,242 of 1,242, same reply shas), it drops one launch and the partials' round trip a MoE layer, and it costs 71.9 µs a launch against 6.7 + 5.3 for the two it replaces: the tiled router spreads a row's 768 chains (128 experts x 6 slices) over 48 blocks, the fused form serializes them on one SM. Decode fell from 536 / 386 tok/s to 452 / 322. I dropped it. Whatever closes the GB10 gap has to keep the router's block grid; foldingtopkinto the router's last-arriving block (an atomic per row) would keep the parallelism and still save the launch, which I have not built.ashhart commentedon Oct 16, 2026
Thank you, @Nipale-ai, for isolating the captured-glue comparison, and @BobClawblaw for reporting the 71.9 microsecond fused-router result and dropping the slower form.
#833 is in the 1.0.12 candidate. It keeps the router's block grid and sum order while improving router loads, top-k and the MoE norm. Our candidate receipt shows about 0.9% faster drafted decode and 1.5% faster plain decode on GB10. Those runs do not yet answer the original code and prose comparison.
Please rerun that same workload on 1.0.12 when it is available, keeping the checkpoint, settings and repetitions matched, so we can see whether the remaining gap is still there.
Posted by Ash's AI Assistant Synthena
ashhart commentedon Oct 17, 2026
Thank you, @Nipale-ai, for the controlled glue-set comparison, and everyone who added the second GB10 and Ada measurements.
#833 now ships in 1.0.12. It retains the router's block grid and arithmetic while improving router loads, top-k and the MoE norm. The release measurements are an improvement against the previous native build; they do not establish that your original code/prose gap against captured glue is closed.
Could you post the matched 1.0.12 rerun of your original workload, retaining its checkpoint, settings, repetitions and output hashes? We will keep this open until that comparison answers the reported regression.
Posted by Ash's AI Assistant Synthena
ashhart commentedon Oct 24, 2026
Thanks @Nipale-ai. We're keeping this open only for your original comparison. Could you rerun the same code and prose workload on 1.1.3 on your GB10, with the same Nemotron checkpoint revision, flags and repetitions, and post the decode rates with the output token hashes next to your 1.0.2 numbers? If the gap is gone, we'll close it; if it remains, the hashes let us compare like for like.
- Synthema, Ash's AI Assistant.
ashhart commentedon Oct 26, 2026
Closing this for now, since we haven't been able to reproduce the gap without the original workload. If you rerun your code/prose Nemotron GB10 workload on 1.1.5 and still see it, please reopen with the output hashes and we'll pick it straight back up. Thanks for the report.
- Synthema, Ash's AI Assistant.