Repository navigation
minimax_m3: a CUDA backend for MiniMax-M3.1-Flash at TP over GB10s, from the release's own formats - #1388
Merged
Conversation
This was referenced Oct 26, 2026
Owner
|
Thanks @BobClawblaw, this is the CUDA backend we hoped for. It's one What stops it landing:
#1235's open points carry into this stack too. |
BobClawblaw
force-pushed
the
minimax-m3-cuda
branch
from
October 28, 2026 03:19
186b898 to
fe51fbd
Compare
Contributor
Author
|
Thanks for the review. The branch is reworked and rebased onto #1235's head (
#1426 will be rebased onto this once it settles. |
BobClawblaw
added a commit
to BobClawblaw/TensorFold
that referenced
this pull request
Oct 28, 2026
Drafter.blocks takes as many streams' blocks as a decode window's rows hold (two blocks of 7): their rows go through the projections, MLPs and head together, so the drafter's weights are read once a pass, while each block runs prep and attention on its own stream's cache and walks its own picks (256-byte pick sets keep the RoCE gathers aligned). draftBlock absorbs every request's kept rows first, then runs the blocks in groups; one stream is unchanged. Two GB10s, ashhart#1388's EXL3 pack, recipe harness: one stream 24.4/39.2/29.7/30.1 -> 24.3/38.5/29.5/30.3 tok/s (prose/code/structured/list), two streams 31.0/44.9/35.7/52.0 -> 33.3/49.0/39.4/55.3 in total; the quick plan passes (drafted == plain, concurrent, soak, seeded sampling).
This was referenced Oct 28, 2026
Owner
|
Thanks @BobClawblaw, this rework covers most of our list:
Before our run:
Then, on two GB10s, we'll run the contract and limits. Ornith-1.5-122B EXL3 on one GB10 against main covers
#1426 follows this one. |
BobClawblaw
force-pushed
the
minimax-m3-cuda
branch
from
October 28, 2026 11:05
fe51fbd to
3ea3e6e
Compare
Contributor
Author
|
Thanks @ashhart. All four points are done; the branch is now three commits on #1235's new head, and the PR body is updated.
Re-checked on this head, two GB10s at TP=2 (recipe harness):
Repository: fmt, |
Contributor
Author
|
Every top-1 miss of the 2,406 teacher-forced rows (243 rows): the release's top 5 and the served engine's (log-probabilities), each side's margin between its first and second, and the row's KL(release ‖ served). From
|
BobClawblaw
force-pushed
the
minimax-m3-cuda
branch
from
October 28, 2026 17:10
3ea3e6e to
7251cb1
Compare
Contributor
Author
|
Rebased onto #1235's Re-checked on this head:
Main has since moved to |
4 tasks done
6 of 9 tasks
This was referenced Oct 30, 2026
tf_exl3_plan_finish takes act_mode 2: (clamp(up) + 1) g sigmoid(alpha g), g = min(gate, limit), in transformers' bf16 steps (MiniMax-M3's routed experts), with alpha appended to the stage arguments. Grouped refuses mode 2 without a positive clamp and alpha; Experts.run, whose epilogue has no mode 2, refuses it. Modes 0 and 1 compute as before: in the exl3_experts image only plan_finish's SASS changes, and its records are 40 registers on every SM under nvcc 13.0, 13.2 and 13.3 (the 13.2 and 13.3 overrides were the old code's). tf-cuda-test exl3 checks mode 2 against float64 (also with the gate driven past the clamp), with two wrong formulas as red controls, and every row's bits at any row count and either grouping (64 or 16 pairs).
… bytes), contract_tests.zig under the lean limit
…rom a published EXL3 pack Serves Mia-AiLab/MiniMax-M3.1-Flash-EXL3-EAQ-v1-3.06-bpw as published (a lossy third-party quantization of the release: attention 6 bits, MLPs 5, routed experts 3, head 8) through the shared EXL3 projections and grouped experts, with the release's DSpark drafter and tower, kept prompt states, the keyed sampler, tools, structured output and images, at tensor parallel over hosts (TF_MINIMAX_RANK, TF_MINIMAX_WORLD, NCCL_COMM_ID). The family's kernels (minimax_m3.cu, minimax_m3_attn.cu) carry gates 1-4 (tf-cuda-test minimax-contract: 26 registrations, 8 controls red) and nvcc 13.0, 13.2 and 13.3 records. tools/zig/minimax_m3_ref.py (the release in torch), tf-minimax-cuda's every-row mode and tools/zig/minimax_m3_kl.py give the fidelity against the release: KL 0.0788, top-1 89.90% over 2,406 teacher-forced tokens. docs/recipes/minimax-m3.1-flash-cuda.md has the run and the checks.
…ract_tests.zig stays under 600 lines
ashhart
force-pushed
the
minimax-m3-cuda
branch
from
October 31, 2026 12:45
7251cb1 to
547c541
Compare
Owner
|
Thank you @BobClawblaw. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes. |
ashhart
added a commit
that referenced
this pull request
Oct 31, 2026
…ain's contract image test passes again #1388 registered MiniMax-M3 launches against minimax_m3 and minimax_m3_attn; the test's resolver had neither.
BobClawblaw
added a commit
to BobClawblaw/TensorFold
that referenced
this pull request
Oct 31, 2026
Drafter.blocks takes as many streams' blocks as a decode window's rows hold (two blocks of 7): their rows go through the projections, MLPs and head together, so the drafter's weights are read once a pass, while each block runs prep and attention on its own stream's cache and walks its own picks (256-byte pick sets keep the RoCE gathers aligned). draftBlock absorbs every request's kept rows first, then runs the blocks in groups; one stream is unchanged. Two GB10s, ashhart#1388's EXL3 pack, recipe harness: one stream 24.4/39.2/29.7/30.1 -> 24.3/38.5/29.5/30.3 tok/s (prose/code/structured/list), two streams 31.0/44.9/35.7/52.0 -> 33.3/49.0/39.4/55.3 in total; the quick plan passes (drafted == plain, concurrent, soak, seeded sampling).
Owner
|
Thanks @BobClawblaw! MiniMax-M3.1-Flash on CUDA over two GB10s is in 1.2.1: https://github.com/ashhart/TensorFold/releases/tag/v1.2.1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
A CUDA backend for the
minimax_m3family: MiniMax-M3.1-Flash at tensor parallel over GB10s, served from a publishedEXL3 pack read as published, through the lane core (kept prompt states, the release's DSpark drafts, the keyed sampler,
tools, structured output, images). It stacks on #1235 (the Metal backend, whose config, tokenizer handling, chat template
and tool markup it shares), now on its head
5555d1550.Rebased onto #1235's
5555d1550(which is on maine4cc630f5). Two registration lines met main's new entries andkeep both: the contract runner creates the
plan_finishgroup beside main's NVFP4 fact group (whosecreatenow takesthe SM list), and nvcc 13.2's records list MiniMax's after main's NVFP4 fact records. Main has since moved to
dd0a9cc5e; the branch merges with it without conflicts, and was left on the base every check ran on.On the previous rebase (onto #1235's
505cdd534) the CUDA backend's files were unchanged; two things moved to fit:zig/build/cuda_engines.zig, beside GLM's (cuda.zigis now 549 lines).minimax_m3_vlform with the language model undertext_config, which is the form the EXL3 pack publishes. This PR's fallback for the flat form, and its test, are gone.On two GB10s at TP=2, every row's logits (the 2,406 teacher-forced rows below, both ranks) are byte-identical between
fe51fbd1e,3ea3e6e55(this PR's previous heads) and this head, and identical to the rows the KL below was computed from.Three commits:
cuda exl3: SwiGLU-OAI in the grouped experts.act_mode2 intf_exl3_plan_finish(the shared groupedexperts' gate/up epilogue):
(clamp(up) + 1) g sigmoid(alpha g),g = min(gate, limit), in transformers' bf16 steps,with
alphaappended to the stage arguments.Groupedrefuses mode 2 without a positive clamp and alpha, andExperts.run(the other route) refuses it outright. The kernel's body moves intoexl3_plan.cuhunchanged (everyexl3_expertssymbol's SASS identical), main's kernel stays incontract_controls.cuas the contract's baseline, andcontract_exl3_finish.zigregisters it (below).contract_tests.zigstays underthe lean limit with the two registrations this PR adds.
minimax_m3: the CUDA backend. The family's loader, forward, engine, lanes and glue kernels (in the family'sfolder), on Mia-AiLab/MiniMax-M3.1-Flash-EXL3-EAQ-v1-3.06-bpw
at
c33e7a6619f0f12ac99098e07ec5e56acb7eeda9. Admitted only as it was run: GB10 (sm_121) at two ranks; any otherGPU is refused (
MiniMaxM3RequiresSm121), and so is one rank or any other world (MiniMaxM3RequiresTwoRanks). Thetower's memory is sized from the checkpoint's tower tensors plus its scratch at the largest image.
The checkpoint (lossy, third party)
Two GB10s do not hold the release (its routed experts alone are 216 GiB of NVFP4), so this serves the EXL3 pack the
review suggested, as published: exllamav3 1.6.1,
mul1; attention 6 bits, dense and shared MLPs 5, routed experts 3,head 8; the indexer fp16, the router fp32, the DSpark drafter and the tower copied from the release byte for byte. It
is a lossy quantization by a third party, not MiniMax's weights; its fidelity against the release is measured below.
No conversion ships: the loader reads the pack's own tensors (
config.jsonundertext_config, the trellis, markers andscales), cut to each rank at 128-column blocks, and refuses anything it does not read as stored. The fp16 indexer loads
as bf16 exactly (each value is a dequantized e4m3 times a power of two; the loader refuses one bf16 does not hold). The
release's own MXFP8/NVFP4 tensors are not read on CUDA. The model's licence is MiniMax's
community licence (non-commercial free;
commercial use needs a notice to, or above a revenue threshold an authorization from, MiniMax).
Measured (two DGX Sparks, GB10, TP=2; this branch's head)
Fidelity against the release (Ash's point 5): 2,406 tokens of a three-turn chat (prose, code, math, German, Chinese,
JSON), teacher-forced; the served engine's logits for every row (
tf-minimax-cuda ... rowson both ranks) against therelease's (
tools/zig/minimax_m3_ref.py --logits: its MXFP8 and NVFP4 decoded exactly, in transformers' order),compared by
tools/zig/minimax_m3_kl.py, which lists every miss with both sides' top-5 and the margins:The last row is the floor the rounding order alone gives; the rest is the pack's quantization.
Served checks (recipe harness on this build): soak 60/60, concurrent 12/12, streaming 4/4, multi-turn 8/8, stop 3/3,
max-tokens 3/3, thinking 10/10, copy 2/2; DSpark drafted == plain 12/12; structured output 18/18 (drafted == plain
12/12); tools 5/5; seeded sampling 9/9; a resumed turn equals the same turn cold after a restart. Images: shapes, text
and counting read right, and 16 image requests at once equal each alone 16/16; video is not served (refused).
Speed (recipe harness ruler, DSpark drafts): one stream prose 24.3-24.4, code 38.1-39.4, structured 27.6-29.7, list
29.8-30.4 tokens a second (first token 0.60-0.65 s); two streams 31.0 / 44.9 / 35.7 / 52.0 in total. These numbers are
from the build of
fe51fbd1e(binary sha256adfa4266…), the head before the rebase.This head (
7251cb1ca, two GB10s at TP=2, the recipe harness): every row's logits byte-identical tofe51fbd1e'sand
3ea3e6e55's (both ranks); structured 18/18 (drafted == plain 12/12), tools 5/5, seeded sampling 9/9, a resumedturn equal to the same turn cold after a restart (4,096 of 4,725 cached), soak 60/60, drafts 12/12, concurrent 12/12,
streaming 4/4, multi-turn 8/8, stop 3/3, thinking 10/10, copy 2/2, max-tokens 3/3; images: 16 at once each equal alone.
One stream with DSpark: 24.3-24.7 / 39.3 / 29.4-29.6 / 30.1-30.3 tokens a second (first token 0.59-0.64 s).
The 28 Oct review
openon every rank before NCCL or a load (admit, with itstest).
visionBytesis the checkpoint's tower tensors as stored (1,734,145,024 bytes in this pack, each one uploadedby
Tower.load) plusload's scratch at the largest image (183,785,472 bytes, with its test).plan_finish's contract, on an RTX 5090 (sm_120), the CUDA kernel contract through these entries:kernel at every served shape (MiniMax-M3.1-Flash's and the Qwen3.5 EXL3 pack's widths);
The served kernel's SASS is unchanged by the move into
exl3_plan.cuh(all 12exl3_expertssymbols identical onsm_80, sm_86, sm_120 and sm_121, nvcc 13.3), so its records stand.
The 27 Oct review's points
nvfp4_mixremoved; the pack as published (above). The f32 patchembedding stays f32 (
tf_mm3_patch_embed, one fp32 fma chain a output in k order, rounded once).tf-cuda-test minimax-contract:SUMMARY minimax: 26 kernels, 26 pass gates 1-4, 0 fail; 8 controls: 8 red as planted, 0 missed. The glue kernels'bounds come from float64 at each kernel's own rounding points; controls: route's ties to the higher id, moe_partial
with the next pick's weight, SwiGLU-OAI without its + 1, RoPE a position late, the Markov head a pick late, plus the
attention and indexer ones. Gate 5: records for nvcc 13.0, 13.2 and (new) 13.3 for all 20 launches; against 13.0 and
13.2 fatbins for sm_80/86/89/120/121,
kernel-contract-limitsfinds no MiniMax or EXL3 record off (it reports otherfamilies' that this branch does not touch). The EPI-4 instances in the shared NVFP4/affine3 images are gone with
those formats.
experts.cuhand its eleven) are back to main's: untouched. Inexl3_expertsonlytf_exl3_plan_finishchanges:cuobjdump -sassof the image before and after, sm_80, sm_86,sm_120 and sm_121 (nvcc 13.2), 11 of 12 symbols identical. Its records: 40 registers on every SM under 13.0, 13.2 and
13.3, so the 13.2 and 13.3 overrides (42 on sm_120/121, the old code's) are dropped. The image's other grouped user,
Qwen3.5-MoE EXL3 (Ornith-1.5-122B,
Mia-AiLab/Ornith-1.5-122B-EXL3-EAQv1-4-bpwat9c5c913b) on one GB10, beforeand after: five greedy replies byte-identical, at the same speed (29.4, 29.5, 27.9, 29.0 and 29.5 tokens a second
each, both; earlier pairs in either order were within 0.5 of each other). GLM's EXL3 path does not reach
plan_finish.tf_mm3_index_scores_tc) for prompt chunks and decode windows (17.3against 16.8 tokens a second at 9.7k with the two, replies identical); the routed experts take the grouped path with
fp32 down rows for both, and the grouping (64 pairs for a prompt chunk, 16 for a window) is checked bit-equal
(
exl3.grouped.oai.rows). Resumed equals fresh: a 9,733-token prompt cached at 8,192 and replayed gives the samereply as cold (tail of 1,541 rows), and the harness's turns check (cached 4,096 of 4,725) likewise.
1 / (1 + e^-x); the NCCL variables are the launcher's (the doc names them); draftsfollow
--no-draftsonly; the licence is stated. EveryTF_MINIMAX_*debug switch is gone (rank and world remain,as GLM's).
Not covered
a mix of the release's NVFP4 and 3-bit experts did the same, and single images read it right.