[diwakergupta]

What this changes

A dense EXL3 checkpoint's bf16 embedding rows (2.37 GiB on the 27B) now load into mapped host memory, but only when the card has no room for them. In every configuration that fits today, the load and the default path are main's.

This is the host-mapped embedding from #1476, made automatic as @ashhart suggested when parking #1476. Unconditional mapping slowed decode on the GB10, so it's now a fallback rather than the default. Its value is on a 24 GB card: there it lets the 27B with DFlash2 serve a 98k or 128k-token window that main refuses to start.

• The rule (cuda_host_embed.zig): the rows go to host memory when a discrete card's room is less than the weights and scratch, one stream at the asked window, the DFlash2 drafter's weights and one stream of its ring, taps and frame scratch, plus 96 MiB for the load's other bytes (kernel modules, the EXL3 kit, the server's start-up costs). Those come to about 112 MiB on the 3090 Ti, so the allowance sits below them. Too high would map rows that fit and slow decode. Too low only refuses in a narrow band above main's limit, where main refuses too. Unified memory (the GB10) never maps them: its host memory is the same memory.
• Where it is decided. qwen3_5's Options gain room (the server's room probe, which gemma4 already takes) and drafter. open passes them to Engine.init, which decides between the loader's measure pass and the arena allocation. Its other callers (the learner CLI and the CUDA tests) pass null and never map.
• The drafter's stream before it loads. Draft.streamBytesFor(config, hidden) uses the drafter's own layout() for one frame, plus its ring and the target's taps. Without it, the rule missed the 98k case by the drafter's 0.35 GiB.
• Loading the rows. cuda_exl3_weights skips the rows in the arena when Engine.host_embed is set, and copies them into that pinned, device-mapped buffer. The engine frees it in deinit and on a failed load.
• The server logs one line when it maps: qwen3_5: embedding rows read from host memory: no room beside a 98304-token stream.

Receipt

• Environment. RTX 3090 Ti (24 GB, sm_86). turboderp/Qwen3.8-27B-exl3 branch 3.50bpw (rev 8351c54e), z-lab/Qwen3.8-27B-DFlash2 (rev 50307d4c) at 4 bits, bf16 KV cache. Both binaries came from zig build native -Dsm=86, one from main and one from this branch (the table below: a0d24896; the limit rows: 9e8c368b).

• What starts, by --context. Each run was a fresh server, with the start-up line's numbers:

  --context	main	this PR
  32,768	model 15.66 GiB, 2 streams	the same: rows on the card
  65,536	model 15.66 GiB, 1 stream	the same: rows on the card
  92,672 (main's largest)	model 15.66 GiB, 1 stream, a 6.56 GiB stream	the same: rows on the card
  92,928	refuses	refuses (the allowance's band)
  93,184	refuses	rows mapped, model 13.29 GiB, 1 stream
  98,304	refuses: a 6.90 GiB stream, 6.56 GiB left	rows mapped, model 13.29 GiB, 1 stream
  131,072	refuses: an 8.90 GiB stream, 6.56 GiB left	rows mapped, model 13.29 GiB, 1 stream
  163,840	refuses	rows mapped, still refuses: a 10.90 GiB stream, 8.93 GiB left

• Exactness with the rows mapped (--context 98304). tools/bench_concurrent.py --levels 1 --tokens 384 --reps 3 --alone --serial, the default code and chat cells (greedy and sampled), and --prompts with two long prompts (5,080 and 2,278 tokens):

  • Drafted == "draft": false and every repeat equal: 0 unequal, 0 failed.
  • token_sha256 equals main's (at --context 32768) in 72/72 replies.

• Decode speed with the rows mapped, against main on the card, per-stream tok/s (a median of 3). Here it costs 0–1.2%; main can't run at this window at all:

  Cell	main, 32k, rows on the card	this PR, 98k, rows mapped	change
  code, sampled	190.2	190.0	−0.1%
  chat, sampled	116.1	114.7	−1.2%
  code, greedy	183.7	183.7	0.0%
  chat, greedy	108.0	107.8	−0.2%
  long prompts, sampled	112.7	113.0	+0.3%
  long prompts, greedy	116.1	116.1	0.0%

• Default path. At 32k, 64k and every window up to main's largest (92,672 here) the rows stay on the card and the same model bytes load as on main. The code that runs per token is main's.

• Tests. The full checklist suite (zig build test test-kernel-contract test-golden test-server-security test-pull-security fuzz-cortheon test-cuda-host) plus test-qwen35-formats-host: 481/481 steps succeeded. lean_new.py against main: 0 new problems. New host test: the rows stay on the card unless a discrete card's room is short of the device bytes and one stream (cuda_host_embed.zig, run by test-qwen35-formats-host).

• Not run. GB10: unified memory never maps, so its path is main's. Metal: not affected. MLX and MoE EXL3 checkpoints: the rule is EXL3-only and the MLX loader doesn't read it; a MoE EXL3 checkpoint takes the same rule but wasn't served. Other drafters: the rule sizes only a DFlash2 drafter and counts nothing for any other.

Checklist

• Every token still goes through the lane rounds.
• Drafted output equals "draft": false, and the mapped rows give main's tokens.
• No precision traded for speed: the same bf16 rows, read from host memory.
• No GPU kernel changed.
• Prompt processing unchanged in bits.
• Platforms: CUDA sm_86 only.
• The new test covers the rule (its function is new, so there is no before).
• Comments and docstrings are one line.
• No personal data, local paths or AI attribution lines.

[ashhart]

Thanks @diwakergupta, this is the shape we asked for. Two things before it lands:

1. Could you rebase onto current main? The Options line in cuda_native.zig now also has rope_yarn.
2. Is there a --context that main starts on a 24 GB card but this branch would map? The rule adds 256 MiB beyond the arena, the stream and the drafter. Near main's limit, it could map rows that main keeps on the card, which would cost up to 1.2% decode. A start-up line at the largest window main accepts would answer it.

[diwakergupta]

Thanks @ashhart. Both done in 4169f094.

1. Rebased onto current main (9e8c368b). Options carries rope_yarn alongside room and drafter. To keep cuda_engine.zig at 600 lines, Engine.init now takes the fit directly; the learner CLI and the CUDA tests pass null.
2. Yes, there was such a window. I found it by bisecting --context on the 3090 Ti with fresh servers. main's largest window is 92,672 tokens (a 6.56 GiB stream in 6.56 GiB of room); it refuses 92,928. With 256 MiB, this branch mapped the rows from 90,624 up, so from 90,624 to 92,672 it mapped rows that main keeps on the card. That band puts the load's real other bytes at about 112 MiB. The allowance is now 96 MiB, below that, so an error can only go one way. After the change, at the same windows:

--context	main	this PR
90,112 – 92,672	rows on the card	rows on the card (same start-up line: model 15.66 GiB)
92,928	refuses	refuses
93,184	refuses	rows mapped, serves
98,304, 131,072	refuses	rows mapped, serves

Setting it too low just refuses in a narrow band above main's limit (one 256-token step here), where main refuses too. It never maps rows that fit. The description has the full table.

[ashhart]

Thanks @diwakergupta, the bisect is exactly what we needed. With the lower margin, the branch keeps the rows on the card wherever main does, up to main's 92,672-token limit, and maps them only where main would refuse to start. GB10s are unified memory, so they never map. This lands now; it's gated and queued, and we've written the changelog line.

[ashhart]

@diwakergupta, one mechanical thing before this lands. Rebased onto current main, zig/src/families/qwen3_5/cuda_engine.zig comes to 602 lines, so our lean check reports one new problem (the 600-line limit). Main moved under it since your rebase. Could you bring it to 600 or fewer, for example by moving the host-embed hookup into cuda_host_embed.zig? Then it lands straight away. Everything else is reviewed and approved.

[diwakergupta]

Thanks @ashhart. Done in bc018e7e, rebased onto current main (5ef82420). cuda_engine.zig is now 598 lines, the same as on main, and the lean check reports 0 new problems.

The hookup moved into cuda_host_embed.zig: hosted.arena measures the weights, makes any host memory and allocates the arena, and hosted.release frees both. The engine calls those two where it used to measure, allocate and free. The rule and the 96 MiB allowance haven't changed.

This also closes a small leak: if the second measure pass failed after the rows' host memory was mapped, that memory wasn't freed.

I checked zig build native -Dsm=86 and test-qwen35-formats-host, both clean. I didn't rerun the servers or the full suite on this push.

[ashhart]

Thank you @diwakergupta. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ashhart]

Landed as e854c28. Thanks @diwakergupta!