Sage-A1.5 (qwen3_5 MoE 35B-A3B) on the XPU backend: decode, serve, vision and MTP #1329

Proposal

The qwen3_5 family runs on CUDA. I propose the XPU path for Sage-A1.5 (35B total, 3B active, 40 layers: 30 Gated DeltaNet and 10 attention, 256 experts with top-8) on one Intel Arc Pro B70 (Xe2, 32 GB):

• Loaders for the MLX 4-bit (affine 4/64) export and the W4A16 AutoRound export.
• Xe2 OpenCL kernels: MoE decode and grouped MoE, XMX prompt chunks, sliced decode attention, the GDN step in one kernel, and an int8 K/V cache.
• A runner with up to 8 slots, and tensorfold-native serve on the XPU engine.
• The Qwen3.5 vision tower on Xe2 (PNG images).
• MTP speculative decode from the model's own draft head, loaded from the AutoRound checkpoint. Greedy tokens stay bit-identical to plain decode.

The branch is ready on v1.1.5 (44 commits) and merges clean onto main. I open the pull request after we agree on the gate.

Measured (one B70, greedy, int8 K/V)

2-hour soak at 4 streams with mixed prompts and images: 2524 of 2524 requests ok.

Proposed gate

1. Greedy tokens with MTP equal plain decode: one stream (k = 1 to 3), and each of 4 and 8 streams against the same stream alone.
2. 8 greedy chat requests at once through serve equal each request alone.
3. The greedy eval set with images, at 1 and 8 slots, with and without the draft head.
4. No change to Metal or CUDA code.

Limits to agree on

• Sampled streams take a one-row path without drafts. Drafts are for greedy streams.
• Plain decode with several streams uses a grouped MoE. Its sum order differs from one stream alone. MTP decode uses the per-row kernels and does not have this difference.
• One card. PNG only. 8 slots with the draft head fit up to about 115k context.

Files

New: zig/src/families/qwen3_5/sage_config.zig, sage_load.zig and a config fixture, zig/src/native/xpu_sage.zig, zig/src/xpu/sage.zig, sage_main.zig, vision.zig, and the Sage and vision kernels under zig/kernels/xpu/. Changed: zig/src/native/xpu.zig, zig/build/xpu.zig, zig/src/xpu/dev.zig, forward.zig and kernels.zig, and the qwen27 XPU kernels take their sizes from a new shape.h. 24 files.

Overlap

No file is shared with #1125 (Qwen3.5-MoE on Metal through qwen27), #1137 (qwen3_5 CUDA scoring), #1113, #1295, #1281 or #728. git merge-tree against main is clean.

Question

CONTRIBUTING asks for one change per pull request. The branch has 44 commits in four groups: kernels and loaders, runner, serve and vision, MTP. One pull request, or four in sequence? I can split it either way.

Credits

ashhart for the XPU backend's device layer, GEMMs and qwen27 kernels. The Intel AutoRound team for the W4A16 checkpoint, which carries the MTP head.