qwen3_5_moe on native CUDA with 3-bit routed experts (a 35B-A3B on a 24 GB card): where it lives, and its gate #803
Description
A coordination question before I build it, per CONTRIBUTING's new-family rule. It overlaps @jkuepker's plan on #491 and #785, so I'd like to agree on the split before either of us gets far.
What I want to run
Ornith 1.5 35B-A3B (MIT), a qwen3_5_moe checkpoint with Qwen3.6-35B-A3B's shapes, on one RTX 3090 (sm_86, 24 GB). It is stored as MLX affine in groups of 64: routed experts at 3 bits, everything else at 4, routers and the shared-expert gate at 8, plus its MTP layer as mtp-4bit.safetensors. That is 15 GB of weights, which leaves the 3090 room for a long window.
It serves today on a fork of the Python 0.6 line with added 3-bit expert kernels. On the 3090 it decodes 416-535 tok/s drafted. KL against bf16 is 0.229, where bf16's kernel-noise floor is about 0.04. On 19 Terminal-Bench 2 tasks, 3 attempts each, it solves 10.0 a run against 10.3 for the Q8 GGUF. #286 froze that line, so I'd like it on the native engine.
What exists on native
- Layers: Qwen3.8-27B on CUDA: the qwen27 family's engine for NVIDIA GPUs, every stream drafted #608's qwen27 CUDA engine already runs the Gated DeltaNet and gated attention layers this model shares with the 27B, with the 0.6.6 bytes. Qwen3.8-27B on CUDA: RTX 3090 (sm_86) support, the scratch sized by --parallel machinegenieorg/TensorFold#3 and docs: show model speeds on TensorFold hero #4 add sm_86 to it.
- Routed experts:
cuda/experts.zig'sGroupedhasAffine4Experts; there is no 3-bit or general-bit affine form. - MTP drafts: Nemotron's
cuda_mtp.zigand the lane core's drafted path. - Loading: feat(qwen3_5): the GPU backends' checkpoint loader, on the host (slice 11) #626's loader reads
qwen3_5_moegeometry, the MTP layer and MLX affine at 2 to 8 bits.
Proposal and questions
-
Where the engine lives. One option is to grow Qwen3.8-27B on CUDA: the qwen27 family's engine for NVIDIA GPUs, every stream drafted #608's CUDA engine to the MoE shapes: routed experts in place of the dense MLP, and the MTP layer as the drafter in place of DFlash2. The other is a separate CUDA backend for the qwen3_5 family on feat(qwen3_5): the GPU backends' checkpoint loader, on the host (slice 11) #626's loader, as @jkuepker proposed on Zig ROCM: RDNA2/3/3.5/4 and more integration #491. I'm happy to do either, or to take a slice of @jkuepker's backend: the routed experts, the MTP drafter, or sm_86. Which do you want?
-
Affine experts at other widths. I'd add 3-bit (and if wanted 2- to 8-bit) affine experts beside
Affine4ExpertsinGrouped. Each would have:- its own kernel and the 4-bit form's accumulation order;
- an FP64 reference per projection class, with worst-column and aggregate error;
- decode and prompt rows checked separately.
Is that the gate, or does a new expert width count as an arithmetic change with more to agree?
-
The exactness reference. 0.6.6 reads only 4-bit group-64 checkpoints for this family. My proposal:
- 4-bit: the engine is byte-equal to 0.6.6 on every logit row, using
TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTPor Ornith's MLX 4-bit conversion. Drafted equals serial, concurrent equals solo, and resumed equals fresh. - 3-bit: the same contracts, plus the FP64 references above. Add a teacher-forced fidelity run against bf16 at 1K, 8K and 32K, recording all top-token differences and margins.
Does that work, or would you rather the 3-bit form wait until the 4-bit engine has landed?
- 4-bit: the engine is byte-equal to 0.6.6 on every logit row, using
Hardware I can test on: an RTX 3090 locally, and rented A6000, A100 and H100 cards. I have no DGX Spark or RTX PRO 6000.
Activity
R1s on the Zig engine: upstream coordination issue filed (ashhart/Ten…
ashhart commentedon Oct 14, 2026
Hello @CanadaApollo6,
Thank you for asking where it lives before building it.
The native binary has no qwen3_5_moe family, so pull and info refuse the checkpoint (docs/recipes/qwen3.6-moe.md:9). CUDA on main serves Nemotron only (zig/src/native/cuda.zig:13). The grouped-expert kernel on CUDA is Affine4Experts (zig/src/cuda/experts.zig:91). There is no 3-bit form beside it. The Python line is frozen at v0.6.6, which is why the fork cannot move onto main as it stands.
Where the engine lives, and whether 3-bit waits until a 4-bit engine matches 0.6.6 byte for byte, are the maintainer's call. The 4-bit contract you wrote is the gate that helps a first PR: drafted equals serial, concurrent equals solo, resumed equals fresh, on TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP. The RTX 3090 receipts belong with that PR.
Thank you again.
BobClawblaw commentedon Oct 14, 2026
Taking the 4-bit gate as a first PR, so nobody builds it twice:
qwen3_5_moeon the Qwen3.5 CUDA engine from #836 (the one that serves the 2B and the 27B), on an RTX 4090 (sm_89, 24 GB). The routing math is the frozen engine'stensorfold/cuda/moe.py(fp32 logits over the experts and the shared gate row, top-k by logit with ties to the lower id, weightsexp(l - l_top)over the k, the shared expert as a slot weighted bysigmoid), the experts through the existing grouped affine-4 kernel with its SwiGLU epilogue instantiated, and the MTP layer as the drafter after plain decode matches. Receipts onTensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP: drafted equals serial, concurrent equals solo, resumed equals fresh, one-row steps equal windows and trees bit for bit. 3-bit experts are not in it.BobClawblaw commentedon Oct 14, 2026
Progress on the 4-bit gate, on an RTX 4090 (24 GB, sm_89, Windows, CUDA 13.3),
TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTPatf84b054c. The engine is the Qwen3.5 CUDA engine from #836 with routed experts in place of the dense MLP; it servesmodel_type: qwen3_5_moebeside the 2B and the 27B, whose bits are unchanged.What is in it. The routing is the frozen
tensorfold/cuda/moe.py's: fp32 logits over the 256 experts and the shared expert's gate row, top-8 by logit with ties to the lower id, weightsexp(l - l_top)over the eight rounded to bf16, the shared expert as a ninth slot weighted bybf16(sigmoid(bf16(gate))), the slots' fma sum rounded once to bf16. The experts run through the existing grouped affine-4 kernel with its SwiGLU epilogue instantiated (gate and up interleaved per group; the shared expert last in the same tables), decode rows to fp32 sums and prompt chunks past 16 rows through the prefill form. The 8-bit router rows are dequantized to bf16 at load.tf-cuda-test moechecks each piece against a host reference: router and combine byte-equal, picks equal (a tie, a flat row and a NaN included) with weights within a bf16 step, the pair experts within 1e-6 of fp64 on random tables.Exactness so far.
tf-qwen35-cudaover 300 ids: windows of 1 to 16 rows and tree windows equal one-row steps byte for byte (300 of 300 rows each).--parallel 2 --context 8192:tools/bench_concurrent.py --levels 1,2 --tokens 128 --reps 2 --alone --serial, code and chat prompts at T=1 seeded and T=0: every alone and serial check equal, 0 unequal, 0 failed.Speed, one card. Decode
tools/bench_openai.py --tokens 128 --reps 2, median tok/s: 96 / 95 at T=1 (code / chat), 101 / 98 greedy; 2 streams aggregate 195 on code. Plain decode, no drafts yet. Load 8 s from the page cache; 18.6 GiB on the device, 2.8 GiB left for streams at this window.Next. The MTP layer as the drafter (the
mtp-4bit.safetensorshead: fc over [norm(embedding) | norm(hidden)], one attention-plus-experts layer, the target's head), so that drafted equals serial means something; thenresumed equals fresh, and the pull request on top of #836 (it needs that engine). The 3-bit form is not in this; acheckrefuses any width the kernels do not read.BobClawblaw commentedon Oct 14, 2026
The 4-bit gate is up as #856: Qwen3.6-35B-A3B on the Qwen3.5 CUDA engine with its MTP layer drafting chains, stacked on #836. Receipt in the pull request: drafted equals serial and lone runs (96 of 96 checks at two and four streams), windows and trees equal one-row steps byte for byte over 300 ids, cold 2,571-token prompt 0.72 s, decode 240 / 206 tok/s greedy (code / chat) at two streams and up to 689 aggregate at four, on an RTX 4090. The 3-bit expert form stays yours; the loader refuses any width the kernels do not read.
ashhart commentedon Oct 15, 2026
Thank you, @CanadaApollo6, for proposing separate prompt and decode checks against an FP64 reference for each expert projection.
We'll use #856 on #836 for the 4 bit baseline, then add 3 bit experts as a separate width PR. Please build that increment beside the existing grouped affine experts, keeping the loader's accepted widths aligned with the kernels. We'll agree the fidelity gate before enabling the new width. Include your proposed projection references and RTX 3090 receipts for drafted versus plain, resumed versus fresh and concurrent versus solo replies, plus windows and trees against individual row steps. That qualifies the requested width on your target card.
Posted by Ash's AI Assistant Synthena
CanadaApollo6 commentedon Oct 16, 2026
Thank you, that settles it. I'll build the 3-bit width as its own pull request on #856 (on #836), beside
Affine4ExpertsinGrouped, with the loader accepting 3-bit experts only where a kernel reads them.Proposed gate, to agree before the width is turned on:
#836 and #856 haven't run on Ampere yet, so my first step is #856's 4-bit receipts on the 3090, which I'll post on #856.
CanadaApollo6 commentedon Oct 16, 2026
The 3-bit width is up as #972, stacked on #856. It adds
Affine3ExpertsbesideAffine4ExpertsinGrouped, and the loader accepts 3-bitswitch_mlponly where these kernels read it.On the gate, one change to what I proposed above. For the engine-level check, the reference is the same checkpoint with its 3-bit words rewritten as 4-bit words, run through #856's affine-4 path. Step, window and prompt logits are byte-equal to it over 300 ids. So the engine's distance to any fp32 reference of this checkpoint is the affine-4 engine's. Each projection matches affine-4's fp64 error on the same codes, decode and prompt separately. The RTX 3090 contracts all hold (96 of 96 drafted vs alone/serial, windows and trees equal steps, resumed equals fresh). If you'd still like the teacher-forced fp32 KL at 1K/8K/32K on top, I'll add it to the pull request.
ashhart commentedon Oct 17, 2026
Thank you, @CanadaApollo6, widening the same 3-bit codes into the affine-4 representation removes a quantisation difference from the engine comparison.
We checked #972's separate affine3 dispatch and your pinned follow-up. The 300-ID byte-equal step/window/prompt control, per-projection FP64 checks and RTX3090 lane receipts are useful, while the original checkpoint's quantisation loss stays a separate measurement.
We will carry the width after the chosen #836/#856 base and owned-image composition, retaining unsupported-format refusals. Please keep the long-window same-checkpoint quality and image/cache cases with that final candidate. We will confirm the remaining teacher-forced reporting with the shared gate rather than treat the earlier 4-bit KL figure as a new tolerance.
Posted by Ash's AI Assistant Synthena
ashhart commentedon Oct 18, 2026
Thank you, @CanadaApollo6, for the 300-ID same-code control and the 3-bit routed experts beside a 4-bit shared expert. 1.1.0 carries #972 on #856/#836, with its separate affine3 dispatch, validated per-projection recipes and retained native image/cache paths. The final same-checkpoint width, FP64, long-window, image and draft/cache/concurrency gates must pass before closing this width proposal. Original quantisation loss stays distinct from same-code engine equality; unsupported recipes still refuse by name.
GB10 MoE serving passed 17/17 and chat 12/12; the specific mixed-width 300-ID/oracle/reference panel is not separately recorded.
Posted by Ash's AI Assistant Synthena