Qwen3.6-35B-A3B (qwen3_5_moe) on the native Metal engine: proposal and gate

Qwen3.6-35B-A3B (model_type qwen3_5_moe) has no native family yet. #856 brings it to CUDA, on #836. I'd
like to bring it to the native Metal engine. CONTRIBUTING asks for an issue before a new family, so this is to
agree on where it lives and on its gate before I open pull requests.

I brought this model to Macs on the Python engine in #138 (0.6.1), and I serve it every day on an M4 Pro.

What it is

The 35B-A3B uses the 27B's layers (Gated DeltaNet, with gated full attention on every fourth layer). Routed
experts replace the dense MLP: 256 experts, 8 picked per token, and a shared expert. The checkpoint
TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP also ships its MTP layer as mtp-4bit.safetensors.

What I'd build, in slices, each on the previous one

The family, plain decode. Reuse the 27B's Metal layers and add the routed experts. The routing is Qwen3.6-35B-A3B on CUDA: routed experts and its MTP layer drafting chains (on #836) #856's,
so that both backends pick the same experts:
- fp32 logits over the experts and the shared expert's gate;
- top-k by logit, with ties going to the lower id;
- weights exp(l - l_top) over the k picks, rounded to bf16;
- the shared expert is one more slot, weighted bf16(sigmoid(bf16(gate)));
- the weighted sum is rounded once to bf16.
The experts are MLX affine 4-bit in groups of 64, with the routers at 8 bits. Bit widths are checked on the
Metal backend only, as you asked on qwen3_5_moe: the CUDA 4-bit/group-64 check gates Metal serving, refusing 8-bit checkpoints the Mac kernels read #683.
The lane host and the prompt cache. Windows of 1 to 16 rows equal one-row steps. A resumed prompt equals a
fresh one. Two and four streams equal solo.
MTP drafts. The checkpoint's own MTP layer drafts with the contract Qwen3.6-35B-A3B on CUDA: routed experts and its MTP layer drafting chains (on #836) #856 uses on CUDA. Drafts only propose,
and every reply equals its "draft": false run.
8-bit experts. This is the qwen3_5_moe: the CUDA 4-bit/group-64 check gates Metal serving, refusing 8-bit checkpoints the Mac kernels read #683 case: mlx-community/Qwen3.6-35B-A3B-8bit, about 35 GiB resident.
(If you want it) the z-lab DFlash drafter z-lab/Qwen3.6-35B-A3B-DFlash, on the 27B's dflash/ path. In
Qwen3.6-35B-A3B on Macs: row decoder MoE + DFlash (v1) drafts #138 it was the fast path on Macs for code prompts.

Vision stays out. It is coordinated on #652, #856 and #885.

Gate (receipts I'll post on each pull request)

An FP64 reference for each projection class whose reduction order changes: router logits, the expert gate/up
pair, expert down and the combine. Both prompt rows and decode rows are checked.
The router picks and weights equal a host reference over a fixed set of ids, as tf-cuda-test moe checks them
on Qwen3.6-35B-A3B on CUDA: routed experts and its MTP layer drafting chains (on #836) #856.
Drafted equals "draft": false, greedy and sampled, with thinking on and off. That includes prompts past the
first 4,096-key chunk.
Resumed equals fresh. Two and four streams equal solo. A chat with the cache on equals the cache off at every
step.
The 2B's and the 27B's bits stay unchanged.
Speed is measured against mlx_lm and against the 0.6.6 Python engine on the same Mac. No precision is traded
for speed.
zig build test passes. Files stay under about 600 lines.

Hardware

I have a Mac mini M4 Pro with 64 GB. The 4-bit and 8-bit checkpoints both fit. The M4 Pro has no tensor units,
so I can't produce M5 or Ultra receipts. Could you run those, or should I find someone with an Ultra?

Questions

Should it live in the qwen3_5 family (where Qwen3.6-35B-A3B on CUDA: routed experts and its MTP layer drafting chains (on #836) #856 puts the CUDA routed experts), or in qwen27 (where the
27B's Metal engine is)? Or should it be a separate qwen3_5_moe family like lfm2_moe?
Should the expert matmuls and routing reuse core/moe_route and the core affine kernels that LFM2 uses, or
are GLM's metal/glm/moe_* the better base for 256 experts?
For drafts, should it be MTP first, as Qwen3.6-35B-A3B on CUDA: routed experts and its MTP layer drafting chains (on #836) #856 does it, and DFlash later or not at all?
Should I open one pull request per slice, as perf(core): tile_qmm, prompt tiles for Macs without tensor units, and their decode twin #821 to feat(lfm2): prompt reuse through the lane host's store #823 did?

@cshintov

Thank you, @cshintov, the routing, cache and draft comparisons give the Metal proposal a useful starting point.

We checked 833bbfc and there is no Metal qwen3_5_moe offer. Shared Qwen layers and core MoE/affine operations are the right pieces to reuse, preserving Qwen's routing rather than GLM's different rule. We will come back on exact family placement and the M5/Ultra validation window.

Please add the checkpoint's owned native image path and image/draft/cache cases to the proposal. Operators can arrive in small slices, but a complete image-capable family cannot stop at the text-only slice. MTP-first work can use the checkpoint head; any DFlash extension should keep its own direct and full-token checks.

Posted by Ash's AI Assistant Synthena

@ashhart

Thanks, @ashhart. Here is the proposal with the image path and the image, draft and cache cases added.

Images. The checkpoint's own tower runs on the 27B's native image path. The two towers have the same shape:
27 blocks, 1,152 wide, 16 heads, patch 16, spatial merge 2, no deepstack. Only the last projection differs,
2,048 wide here against the 27B's 5,120. The tower's weights ship in the text checkpoint
(mlx-community/Qwen3.6-35B-A3B-4bit has 333 vision_tower tensors).

Slices. Each one is a small PR. Images come right after decode, so the family is image-capable early.

Plain decode: routed experts in the 27B's engine, as one layer kind beside the dense MLP.
Images, on the checkpoint's tower.
Prompt cache and lanes.
MTP drafts from the checkpoint's own head (mtp-4bit.safetensors in TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP),
with Qwen3.6-35B-A3B on CUDA: routed experts and its MTP layer drafting chains (on #836) #856's contract.
8-bit experts.
(Optional) the z-lab DFlash drafter, with its own direct and full-token checks.

Cases added to the receipts.

Cache: a reused prompt gives the same states and logits as a fresh pass, byte for byte. That covers keep, reset,
slot reuse and lanes against serial.
Drafts: drafted equals "draft": false, greedy and sampled, with thinking on and off. That includes prompts past
the 128-row chunk, and resets and slot reuse mid-reply.
Images: one image and several images, image rows across a chunk boundary, an image prompt reused from the cache,
and drafted equals "draft": false with an image in the prompt.

Where slice 1 stands, on a Mac mini M4 Pro with 64 GB, against mlx_lm 0.31.3:

A 24-token prompt: 60 of 60 greedy tokens equal, and teacher-forced top-1 is 60 of 60.
A 378-token prompt: teacher-forced top-1 differs at 4 of 72 positions. Two are exact bf16 ties, and mlx_lm ties at
the first one too. Two are one bf16 step apart.
Prompt and decode states are byte-equal between prompt chunks of 128 and 16 rows, at every 32-row seam.
On TensorFold/Qwen3.8-27B-MLX-4bit, main and this branch give the same tokens, state hashes and top-2 logits at
every step in plain decode. Lanes, drafted rounds and prompt reuse are being compared now.
The 35B serves through tensorfold-native serve with 8 streams and the prompt cache, and its reply equals mlx_lm's.
zig build test passes.
Still open before the PR: the new kernels' kernel-contract registration, and the bench receipts.

Two questions:

Is the 27B's family the placement you want? Or should we wait for your call before opening slice 1?
Can slice 1 open as a PR now, text only, with --vision refused until slice 2?

@cshintov

Thanks @cshintov for the 60/60 short-prompt control and the added image/cache cases. Reuse the dense engine's operations, but put the MoE family adapter, config and loader under the same Qwen3.5 family as CUDA.

#1009 is now open and reviewed, so there is no need to wait to submit that development slice. Its explicit vision refusal should remain until the tower exists. Landing qualification still needs the model's MTP through shared lanes, native images, registered attention specializations and exact staged layer references, with the long-prompt mismatch explained. The current text-only slice is not approval to ship the multimodal family without those pieces.

Posted by Ash's AI Assistant Synthena

@cshintov

Thanks @cshintov. The family placement and acceptance gate are agreed, and implementation review continues in #1009 and its follow-up slices. We'll close this proposal discussion and keep that work open. Shared-family placement, native images, checkpoint MTP through the lanes, registered attention cases and the long-prompt reference comparison remain required before the Metal family ships.

Posted by Ash's AI Assistant Synthena

qwen27: Qwen3.6-35B-A3B on Metal, a layer's MLP dense or routed exper…

Navigation Menu

Repository navigation

Qwen3.6-35B-A3B (qwen3_5_moe) on the native Metal engine: proposal and gate #931

What it is

What I'd build, in slices, each on the previous one

Gate (receipts I'll post on each pull request)

Hardware

Questions

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Qwen3.6-35B-A3B (qwen3_5_moe) on the native Metal engine: proposal and gate #931

Description

What it is

What I'd build, in slices, each on the previous one

Gate (receipts I'll post on each pull request)

Hardware

Questions

Activity

ashhart commented on Oct 17, 2026

cshintov commented on Oct 17, 2026

ashhart commented on Oct 17, 2026

ashhart commented on Oct 19, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions