Repository navigation
New family: MiniCPM-V 4.7 35B-A3B (Qwen3.6 MoE fine-tune + vision tower) #785
Description
MiniCPM-V 4.7 35B-A3B (openbmb/MiniCPM-V-4.7-35B-A3B, Apache-2.0) is a vision-language MoE whose language model is a fine-tune of Qwen3.6-35B-A3B (text_config.model_type qwen3_5_moe_text). CONTRIBUTING asks for a new family to start as an issue, and #286 says the Python engine takes no more changes, so this is not a pull request. I'm offering what I have as a reference and asking whether you want it on the native engine, and where it would fit.
What exists today
The family works on the 0.6.6 Python engine, on a fork:
- Tested branch: jkuepker/TensorFold
minicpmv47-066, on the v0.6.6 tag. Every number and check below ran on it. - Rebased branch:
minicpmv47-py06is the same work as 4 commits onpython-0.6. Only the CPU suite has run on it; it is not GPU-tested. - Text: runs on the Qwen3.6 MoE CUDA engine unchanged. Drafts use Qwen3.6's MTP layer (
TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP), since MiniCPM ships none; 71 of 75 drafted tokens were accepted on code. - Images:
--visionruns MiniCPM's own BF16 tower and merger in plain PyTorch. Slicing and the canvas 3-D rotary positions are ported from transformers and vLLM (both Apache-2.0, listed in THIRD_PARTY_NOTICES). Positions equal transformers'get_rope_indexexactly on 6 layouts. - Engine change: the MoE engine takes an MTP side file, a tensor skip and an image lane. One fix applies to stock Qwen3.6 too: the MoE CUDA-graph refresh dropped
rope_delta. - Checkpoint: kuepker/MiniCPM-V-4.7-35B-A3B-MLX-4bit-TensorFold, converted with
mlx_lm.convertto the same layout asmlx-community/Qwen3.6-35B-A3B-4bit, plus a BF16 vision shard.
Measurements (one DGX Spark, GB10, greedy, thinking off)
| Decode, tok/s | 1k | 4k | 16k |
|---|---|---|---|
| TensorFold 0.6.6 + branch, drafted | 153.2 | 147.6 | 142.1 |
TensorFold 0.6.6 + branch, "draft": false |
89.7 | 87.5 | 79.6 |
| vLLM 0.31.1rc1, NVFP4 W4A16 | 102.5 | 101.3 | 93.1 |
- Quality: MMLU 83.6 on the MLX 4-bit checkpoint, against 84.1 for BF16 on vLLM.
- Exactness: drafted replies equal
"draft": falsetoken for token, on text and image prompts. - Tests:
tests/cudapassed 1297 on both v0.6.6 andminicpmv47-066. The CPU suite onminicpmv47-py06fails the same 7 tests aspython-0.6itself.
Limits, stated up front
- Lanes:
--visionrefuses--parallelabove 1. Text-only serving keeps the engine's concurrency. This sits against house rule 1, which is one reason I'm asking rather than proposing a design. - Hardware: CUDA only, GB10 only. There is no Metal path for this family.
- Images: multi-image prompts and requests over 16,384 patches are verified on CPU only.
Questions
- Does the native engine want MiniCPM-V 4.7 once Qwen3.6 MoE runs natively on CUDA and native image input (Native image and video inputs in the server (Python frontend's preparation, data and HTTPS URLs), before any family consumes them #565, Native image input: images from a request's image parts, prepared before admission; GLM-5.3-Flash reads them on Metal #598) lands?
- If yes, should its tower follow the GLM-5.3-Flash image path from Native image input: images from a request's image parts, prepared before admission; GLM-5.3-Flash reads them on Metal #598, or wait for a shared vision interface?
- Would the
rope_deltagraph-refresh fix be useful as a note for the native Qwen3.6 MoE CUDA work?
Activity
jkuepker commentedon Oct 13, 2026
Update: native CUDA serves only Nemotron, and the Qwen3.5 / 3.6 family is coming through #491 (HIP first). I'll start a CUDA backend for that family against the latest release, built on #626's loader so it doesn't fork a second family. MiniCPM-V 4.7 text would follow as a thin wrapper, and images after #598. I've asked on #491 how to stay aligned. If you'd rather this went another way, say so before it gets large.
ashhart commentedon Oct 14, 2026
Hello @jkuepker,
Thank you for the reference, and for opening an issue before a family.
1.0.10 serves Qwen3.5-2B and Qwen3.8-27B on Metal, both as
model_typeqwen3_5. A dense checkpoint uses the 27B engine. The pinned 2B stays on the 2B engine. On CUDA the native engine serves Nemotron. Qwen3.6-35B-A3B is still the Python 0.6.6 engine (docs/recipes/qwen3.6-moe.md). The README still lists it under qualification for 1.0.x. Aqwen3_5_moe_textcheckpoint is refused: the native engine has no backend for that type, andtensorfold pullsays the same.Native image input is open (#565). #598 is GLM-5.3-Flash on Metal. #726 is the 27B's tower on CUDA. Each of those towers is that family's. MiniCPM's is its own: BF16, with slicing and canvas 3-D positions. The shared piece for those two paths is one JPEG decoder. There is no shared tower, and MiniCPM's tower is new code on either choice.
A port follows
docs/recipes/adding-a-zig-family.md. The family supplies its config, weight map and layer graph, and kernels only for what is its own. It uses the engine's scheduler. A new family does not add another one (CONTRIBUTING.md). Drafted tokens match"draft": false, a resumed prompt matches a fresh run, and each concurrent reply matches its solo run. MiniCPM ships no MTP layer. Your drafts use Qwen3.6's MTP side file. The fork also adds a tensor skip and an image lane.Your
--visionpath refuses--parallelabove 1. Text-only keeps the concurrency the engine already has. A port sends image requests through that same scheduler. The reference you measured is CUDA on a GB10, and there is no Metal path in it.The
rope_deltamiss is in the Python MoE CUDA-graph refresh. 1.0.10 has no Qwen3.6 MoE graph and norope_deltasymbol. Your 0.6.6 fork and the converted checkpoint are the reference. Python 0.6.6 takes no further family changes (#286).jkuepker commentedon Oct 14, 2026
Thanks @ashhart. The native port is at jkuepker/TensorFold
minicpmv47-native(CUDA, GB10, text only). How it lines up with what you listed:qwen3_5on CUDA runs on coreLaneHostandcore/lanesas a lane backend, like Nemotron'scuda_lanes.zig. It adds no scheduler of its own. One overlap is left: a batched-admit helper (admitAll,core/lanes/admit.zig) that I'm reconciling with 1.0.10'slane_host_admit.zigandlane_host_fill.zig."draft": false(mtp_check, serve_check--draft-off).text_configtower and skips the image tower's tensors. The MTP side file is optional. It changes speed only, never tokens. No image lane yet, and--parallelworks as for Qwen3.6.qwen3_5?lean_check, add FP64 references for the new kernels, and postbench_openai.py/bench_concurrent.pyreceipts from a GB10.ashhart commentedon Oct 14, 2026
Hello @jkuepker,
Thanks for the native port, the GB10 checks, and the 137-line wrapper that reads the text tower and skips the image tensors. Drafted output equals "draft": false, a resumed prompt equals a fresh run, each concurrent reply equals its solo run, and the tokens equal the Python 0.6.6 engine.
Rebase once #608 lands. The 27B CUDA engine is being built on #608's design. A second copy of its pieces in qwen3_5 is a second path for the same engine.
One admit path. zig/src/core/lane_host_admit.zig:7 picks the queued request that fits when a lane is free, and zig/src/core/lane_host_fill.zig:5 picks the next prompt chunk, foreground first, then the fewest tokens left. Reuse those. Leave admitAll out.
Your list before a pull request is right. Add the bench_openai.py and bench_concurrent.py receipts on the same GB10 against 1.0.10.
A new family in the native engine is the maintainer's call. It is not decided yet.
ashhart commentedon Oct 15, 2026
Thank you, @jkuepker, for the 137 line native wrapper and the GB10 checks covering drafts, resumed prompts and concurrent requests.
We'll coordinate one native port on #836 and #856, including MiniCPM's own image tower. Please rebase the wrapper onto that stack and propose the tower, slicing and canvas position work in the port PR, with the source of the implementation clearly documented. Reuse the shared decoder and admission path. We'll review those boundaries and qualify the family with native image kernels, including image prompts in the draft, resume and concurrency checks. Your existing wrapper gives that work a small starting point.
Posted by Ash's AI Assistant Synthena
ashhart commentedon Oct 17, 2026
Thank you, @jkuepker, for the small native wrapper and the GB10 draft, resume and concurrency receipts.
MiniCPM is not registered in 1.0.12. The next step remains one native port on the chosen Qwen CUDA base #836 and MoE stack #856/#972, with #885's image ownership and the shared decoder and admission path.
Please bring the rebased wrapper PR here, including the plan for MiniCPM's own tower, slicing and canvas positions, with implementation provenance. Its image path must retain native lanes and cover images in the draft, resume and concurrency checks. The earlier Python tower remains a reference, rather than the serving implementation.
Posted by Ash's AI Assistant Synthena
ashhart commentedon Oct 24, 2026
Closing for now, since the rebased PR we discussed hasn't arrived. When it's ready, please open the PR (or reopen this issue) and we'll review and measure it. Vision and MoE families are welcome when they're off by default and leave the existing engine byte-identical.
- Synthema, Ash's AI Assistant.