[plotarmordev]

What this changes

On CUDA, Nemotron 3.5 Super VL's lane prefill (Cuda.prefillFn in zig/src/families/nemotron/cuda_lanes.zig) calls
Vision.begin twice for a request with images. The merge of #897 and #938 (7f6f940) put one call where Bolt's branch
set up its image rows, after the kept-prefix restore, and kept the one Super VL's images brought (e144e45) after the
prompt-rows setup. begin ends any pass in progress, so the second call plans the same spans and runs every image
through the tower and projector again before the first chunk. The rows it writes are the same, so replies do not
change; each image just costs a second tower pass (about 0.9 s for a 2048 x 1664 image on a GB10). This removes the
second call and its defer.

No host test comes with this. Tower.encode needs the ViT kernels and cuBLASLt, so no host test can run
Vision.begin past it without a GPU (the existing host tests stop at its error.RowShape shape check, where the
first begin would already fail). The proof is the GPU receipt below: tower kernel launches halve and replies stay
byte-identical. The smallest seam for a behavioural host test would be an encode hook on Vision, as Flash Next's
encoder has (a ctx and an encode function), so a test could count encodes per request; I can add it as a
follow-up if you want it.

Bug report

• Native version: 1.1.0 (76aafcf) and main d46d17a, CUDA backend. Main has not changed zig/ since 1.1.0.
• Model: nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 at revision 82d93699acd3be9e4617c53360fe5b3f5dbeb1b3.
• GPU/OS: one DGX Spark (GB10, sm_121), driver 580.173.02, CUDA 13.0, Linux aarch64.
• Command: tensorfold-native serve MODEL --vision --vision-image-tokens 13312 --vision-max-images 4 --context 18432 --parallel 1 --prompt-cache-gib 0.
• Request: a chat completion with one image, or four copies of it (a generated 2048 x 1664 PNG of three shapes; the
  prompts are 3,356 and 13,366 tokens), greedy, thinking off, "draft": false, 32 tokens.

	1.1.0	this fix
tf_vit_position_add launches, 8 requests carrying 20 images (nsys)	40	20
tf_vit_unshuffle launches, same requests	40	20
1 image token_sha256, 4 runs	541bca2d…9735ec	541bca2d…9735ec
4 images token_sha256, 4 runs	ef22b44e…d444c4	ef22b44e…d444c4
1 image TTFT, median of 3	3.170 s, 3.188 s	2.285 s (−28%)
4 images TTFT, median of 3, less the heat wait logged inside each request	10.666 s, 10.678 s	7.034 s (−34%)

Full hashes: 541bca2d52751d8bb9c44b2af53ce8ff2dfb93745d8cb6bdb740442dad9735ec,
ef22b44ea1d731737c2d2a757d671d03b1269563d6bb505944e58608ecd444c4. Both kernels run once per image the tower encodes.

How to reproduce

nsys profile --trace=cuda --sample=none --cpuctxsw=none --output=tower \
  tensorfold-native serve MODEL --port 8897 --vision --vision-image-tokens 13312 --vision-max-images 4 \
  --context 18432 --parallel 1 --prompt-cache-gib 0
API_URL=http://127.0.0.1:8897 python3 bench.py image-ttft   # receipts' scripts/, next to prompts.py
# stop the server, then:
nsys stats --report cuda_gpu_kern_sum --format csv --output tower tower.nsys-rep
grep -E 'tf_vit_(position_add|unshuffle)' tower_cuda_gpu_kern_sum.csv

bench.py image-ttft sends 4 one-image and 4 four-image requests (20 images) and prints each TTFT and
token_sha256. The launch count column read 40 on 1.1.0 (main's source) and 20 with this fix.

Receipt

Raw files: https://github.com/plotarmordev/TensorFold/tree/nemotron-receipts/receipts/nemotron/vision-tower-once (README.md maps every file; SHA256SUMS).

• Environment, with the TensorFold commit, MLX or PyTorch version, machine and GPU, checkpoint and revision: this
  branch at e2357b5 on main d46d17a. The GPU runs compare 1.1.0 (76aafcf) with 1.1.0 plus this fix (b4495aa3);
  git diff --stat 76aafcf2 d46d17a7 -- zig build.zig build.zig.zon tools is empty, and this branch's diff against
  main is byte-identical to git diff 76aafcf2 b4495aa3. Each built from a clean checkout with zig build fatbins -Dsm=121 -Dnvcc=… and zig build native -Doptimize=fast -Dstrip=true -Dversion=1.1.0 -Dtarget=aarch64-linux-gnu.2.28 -Dcpu=baseline
  (zig 0.17.0, nvcc 13.0.r13.0); tensorfold-native sha256 1.1.0 04167a17…74015088, fix 52b3990e…a15853fd
  (full in the receipts). One DGX Spark (GB10), driver 580.173.02, CUDA 13.0,
  nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4@82d93699acd3be9e4617c53360fe5b3f5dbeb1b3, served as in the bug
  report on loopback with TF_HEAT_HIGH=85 TF_HEAT_LOW=75, a fresh server and HOME each hold.
• Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: every image
  request is "draft": false, and each build's 4 runs at 1 and at 4 images give one hash, the same on both builds
  (above), in the profiled and the unprofiled holds. Drafted image replies and bench_concurrent.py: not run.
• Decode speed before and after, from tools/bench_openai.py: not measured. The change removes work from a prompt
  pass with images only; decode rounds and text-only prompts run the same code as before.
• Prompt speed before and after: tools/bench_cache.py sends no images, so the TTFT above is the server's
  time_to_first_token from bench.py image-ttft, cooled to 75 °C before every request. Unprofiled holds in the
  order 1.1.0, fix, 1.1.0, each started at a zone of 60 °C or less. One image: 3.170 / 2.285 / 3.188 s, no heat waits.
  Four images: 20.666 / 15.034 / 18.678 s raw; every four-image request on every build paused at the heat gate inside
  its prompt pass (10 / 8 / 8 s, logged as heat_wait_s), so the table nets them out. Both counts save about 0.9 s an
  image, one tower pass.
• Tests run, with pass, fail and skip counts (CPU only, a DGX Spark, -j1, a fresh cache each step), on this branch:
  zig build test 152/152 steps, 2413 of 2422 tests passed, 9 skipped, 0 failed (main: the same counts);
  zig build test-kernel-contract 43 of 43; zig build test-golden 306/319 equal to the frozen answers, 0
  unexpected differences; zig build fuzz-cortheon 8 of 8; zig fmt --check zig build.zig clean;
  tools/zig/lean_check.py 503 problems, 503 on main, 0 new (tools/zig/lean_new.py); zig build fatbins -Dsm=121
  and zig build native -Doptimize=fast -Dstrip=true succeed.
• Not run, and why:
  • Metal: the change is CUDA-only, and there is no Mac here.
  • Tensor parallelism: image rows are filled on rank 0 and broadcast; a TP=2 server with images was not run.
  • Bolt with images, video, and a request resumed from a kept prefix that ends inside an image: not run on a GPU.
  • tf-cuda-test kernel-contract: no kernel changed.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (The change only drops a repeated tower pass whose rows the first pass already wrote; image replies are byte-equal before and after with "draft": false. Drafted, resumed and concurrent image requests were not run.)
• No precision traded for speed. If the bits change, the description says which sums change and why. (No sum changes; the hashes are equal.)
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt. (No kernel changed.)
• Prompt processing is no slower than the last release. (Image prompts are faster; text prompts run unchanged code.)
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (No new test: a host test cannot run the tower, see above; the GPU receipt shows the launch counts and equal replies.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[ashhart]

Thanks @plotarmordev, that's a clean find. The second Vision.begin planned the same spans again and ran every image back through the tower, and your nsys counts (40 tower launches down to 20 for 20 images) show what it cost. The call that stays takes the same from, so nothing else moves.

We've carried it onto the 1.1.2 line under your name and measured it on a GB10 against the line without it. Super VL NVFP4, one stream, prompt cache off, ABBA, 20 requests a cell: cold TTFT fell 20.3% with one image (1.558 to 1.241 s) and 21.8% with two (2.677 to 2.093 s), with identical token hashes, drafted equal to plain, greedy and seeded. That's a little under your 28%, with our own fixtures. A repeated image that hits the prompt cache skips the tower anyway, so the gain is on cold requests. An encode hook on Vision, so a host test can count encodes per request, would be welcome as a follow-up PR.

Posted by Ash's AI Assistant Synthena

[plotarmordev]

The encode hook on Vision you suggested is in #1104, with a host test that counts tower passes through a lane prefill.

[ashhart]

The Nemotron CUDA image-prompt fix is included in 1.1.2 (https://github.com/ashhart/TensorFold/releases/tag/v1.1.2), with your original authorship retained. The prompt path starts the RADIO vision work once instead of starting it again. Closing this as carried, and thank you.

It landed as 6bdcd38.

- Synthema, Ash's AI Assistant.