MiMo-V2.6-Flash-MOPD on one RTX PRO 6000

Decode 8–18% faster: 77 tok/s on coding and on a held-out mix of tool calls, coding and prose, and 71 tok/s after a 131K-token prompt. Speed v6 (2026-10-06) shortens the GPU and CPU work around each decode step's expert copies. It also stores the BF16 projections exactly in 12.5 bits, spends the freed VRAM on cached experts, and stops each draft at its first unsure token. Prompts got 2–4% faster as a side effect. The recipe now serves Xiaomi's MiMo-V2.6-Flash-MOPD, the release that fixes the RL model's repeated tool calls.

This is the reproducible recipe for Xiaomi's MiMo-V2.6-Flash-MOPD (310B total, 15B active parameters). It uses the exact NVFP4 weights, a 524,288-token context, scaled E4M3 FP8 KV, native thinking, tool calls and a target-verified DFlash draft. It needs one GPU plus about 96 GiB of pinned system RAM: the 4,096 most-used experts are pinned in VRAM, the rest live in host RAM behind a GPU expert cache.

MiMo-V2.6-Flash-MOPD, projector loaded, same day	Speed v5.3	Speed v6
Coding, seed 17 (4 tasks, up to 2,048 tokens)	69.9 tok/s	77.0 tok/s
Tool-calling agent, held-out prompts	59.8 tok/s	70.7 tok/s
Coding, held-out prompts	73.1 tok/s	78.8 tok/s
Prose, held-out prompts	64.8 tok/s	75.8 tok/s
Decode after a cold 131,000-token prompt	64.5 tok/s	71.2 tok/s
4,096-token prompt	1.75 s	1.70 s
Cold 131K-token prompt (retrieval check, 5/5 on both)	28.7 s	28.3 s

Decode rates include reasoning tokens and exclude prompt processing. "Held-out" prompts were not used to choose the pinned experts. Both runtimes are this repo's builds, measured one after the other on fresh servers. On MOPD, v5.3 decodes more slowly than it did on RL (69.9 against 72.3 tok/s on the seed-17 coding tasks). MOPD reuses RL's DFlash draft, which accepts fewer of MOPD's tokens (0.572 against 0.607). v6 raises acceptance to 0.69 and more than makes up for it. A clean clone of this repo, built with scripts/build.sh, returned the same response bytes as the measured v6 build on all five matched-benchmark requests. Measurements.

v6 (2026-10-06) speeds up the decode step:

• Less time outside the copies.
  • The full-vocabulary top-p runs on 8 threads (350 → 170 µs per token), with the same probabilities, cut and order.
  • The next layer's expert prediction moved into the router's own matmul: one two-channel matmul and a fused top-k replace about 27 µs of kernels per layer. Its prefetch decision now runs after the layer's expert kernels are launched.
  • The f8s128 QKV projection fits three blocks per SM and runs in one wave.
  • With the same drafts, a verify step takes 38.8 ms instead of 42.3.
• More cached experts.
  • The BF16 attention-output projections and output head are stored as BF16P12: 12.5 bits per weight, decoded into the same BF16 values inside the kernels (4.35 → 3.40 GiB).
  • The image encoder's 937 MiB compute buffer now exists only while an image is encoded.
  • The expert cache grows from 2,294 to 2,418 slots. Free VRAM while decoding stays above 2 GiB.
• Shorter drafts when the draft is unsure. A draft stops at the first token it gives less than 50% (--spec-draft-p-min 0.5). Acceptance rises from 0.57–0.61 to 0.69–0.72, and speculative sampling keeps the target's distribution exact.

Every v6 change but two is exact.

• The fused prediction matches the runtime with expert prediction switched off. v5.3's unfused prediction is what perturbed 13–20-token batches.
• The draft cutoff keeps the target's distribution, but it changes which tokens are verified together, so sampled text differs from v5.3's.

With the draft model removed, v6's greedy logprobs equal v5.3's at every position of 15 of 16 test prompts. The 16th is a 20-token prompt, inside that range. patches/0011, how.

MOPD (from v6).

• The target is now the exact NVFP4 conversion of XiaomiMiMo/MiMo-V2.6-Flash-MOPD, published at ProCreations/MiMo-V2.6-Flash-MOPD-NVFP4, with the same tensor layout and format as RL's.
• The Q8_0 draft and the f8s128 overlay are rebuilt from it. The image/audio projector is unchanged: MOPD's encoder tensors are RL's, byte for byte.
• patches/0010 fixes the audio encoder's skip connection; the MOPD profile was qualified with it.
• The API model ID is mimo-v2.6-flash-mopd; mimo-v2.6-flash-rl is still accepted.
• To stay on RL, use commit 8eb2b30 (speed v5.3). See upgrading.

Earlier versions (measured on MiMo-V2.6-Flash-RL)

Faster prompts at every length: a 512-token prompt in 1.14 s instead of 1.56 s, 1K–4K-token prompts in about 1.7 s instead of 1.9–2.1 s, and a cold 131K-token prompt in 29 s instead of 34 s. Speed v5.3 (2026-09-24) removes three stalls around prefill and runs v5.2's FP8 activations from 129-token batches instead of 512. Decode is unchanged. Speed v5.2 had already cut a 16K-token prompt from 7.1 to 3.0 s.

Image and audio input, and 71 tok/s decode at 131K context, with the creator's sampling, the exact NVFP4 weights and the full 524,288-token context. Speed v5 adds MiMo's vision and audio encoders and a decode attention kernel for the FP8 KV cache: at 131K tokens of context decode went from 55.6 to 70.7 tok/s, and near 520K from 34.4 to 45.1 tok/s. Short-context decode is unchanged, about 74 tok/s on coding. The previous public MiMo NVFP4 runtime ran the same coding benchmark at 27.0 tok/s on the same machine.

Runtime, same machine and settings	Coding, seed 17 (up to 2,048 tokens)	Coding, held-out prompts	Tool-calling agent	Prose
Previous public runtime (45 patches)	27.0 tok/s
Speed v2	41.0 tok/s
Speed v3	56.5 tok/s	59.2 tok/s	45.5 tok/s	62.7 tok/s
Speed v4 / v4.1	74.5 tok/s	74.4 tok/s	61.7 tok/s	65.0 tok/s
Speed v5, no projector	73.0 tok/s	76.6 tok/s	64.4 tok/s*	66.3 tok/s

Decode at long context	Speed v4.1	Speed v5
131,000-token prompt (no projector)	55.6 tok/s	70.7 tok/s
519,882-token prompt (projector loaded)	34.4 tok/s	45.1 tok/s

Prompt processing, projector loaded	Speed v5.1	Speed v5.2	Speed v5.3
256 new tokens	1.14 s	1.14 s	0.96 s
512 new tokens	1.56 s	1.56 s	1.14 s
1,024 new tokens	1.93 s	1.94 s	1.69 s
4,096 new tokens	2.39 s	2.07 s	1.78 s
16,384 new tokens	7.09 s	2.84 s	2.47 s
16,384 tokens appended to an 8K cached prompt	7.34 s	3.01 s	2.62 s
cold 131K-token prompt	68.6 s	33.5 s	28.9 s
16,384 tokens appended to a 141K-token cached prompt	10.9 s	6.09 s	5.71 s
cold 262K-token prompt	172.5 s	95.8 s	82.4 s
cold 520K-token prompt	495 s (v5)	305 s	264 s

v5.2 and v5.3 were measured on the same day, one after the other. The v5.1 column comes from the v5.1/v5.2 comparison earlier that day; v5.2 then measured 1.14, 1.56, 1.94, 2.14, 3.04, 3.02, 33.4 and 6.0 s. The 262K and 520K rows are the cold retrieval checks of each version's qualification.

Decode rates include reasoning tokens and exclude prompt processing. Speed v5.1 and v5.2 do not change decode: the seed-17 coding requests return the same bytes as v5, at 70.3 tok/s on both with the projector loaded. v5.3 leaves the decode arithmetic alone too. Three of those four prompts have 150–156 tokens, though, and now run with FP8 activations. They sample different text: 72.2 tok/s at 0.607 draft acceptance, 39.0 against 38.1 ms per verify step. "Held-out" prompts were not used to choose the pinned experts. The same-day v4.1 run of the first table measured 74.0, 74.2, 61.5 and 64.9 tok/s. *27 of 28 requests: the 28th repeated a tool call until its token cap (with it, 73.3 tok/s). With the image/audio projector loaded, the default, short-context decode is about 1% slower, because the projector takes 6 of 80 expert-cache slots. At long context part of the gain is draft acceptance on different text; per verify step, 131K is 16% faster.

Speed v2 and v3 generate byte-identical output to the previous runtime. Speed v4 does not: it uses speculative rejection sampling, which keeps the target's exact sampling distribution but draws different samples, and its expert renumbering changes floating-point summation order about as much as a batch-size change does (KL divergence 0.006). Speed v5's attention kernel changes summation order by a similar amount (0.005). See how and measurements.

v5.3 (2026-09-24) removes the stalls around prefill:

• Cached experts are copied into the prefetch ring on their own stream, so the PCIe copies of the host experts run back to back. Every prompt of 600+ tokens is 12–17% faster, and 1K–4K-token prompts take about 1.7 s, the time the ~84 GB of host experts need over PCIe.
• The KV cache grows device to device: 0.04–0.14 s per 32K step, instead of up to 1.4 s.
• The DFlash draft reads the target's hidden states in place on the GPU.

These three are exact. Prompts of about 140–516 tokens now use v5.2's FP8-activation GEMMs, with their experts read through the expert cache, instead of the exact grouped kernel. That makes them 10–27% faster, at the same small distance from the exact path as v5.2's longer prompts. The patch also has an FP8 Q·K attention kernel for the global layers, 1.3–1.5× faster per layer, which stays off. Over 30 seeds of a 7.8K-token code-reading task it answered 22 correctly, against 28 without it. patches/0009, how.

v5.2 (2026-09-24) runs prefill with FP8 activations. Expert GEMMs use MXFP8 activations on Blackwell's block-scaled FP8 × FP4 tensor cores, which take the stored NVFP4 weights as they are. The FP8 dense layers use E4M3 activations with a scale per token and 128 values, through the runtime's own kernel, and the BF16 attention output uses BF16 activations. Sliding-window attention skips masked tiles, and the expert prefetch ring no longer stalls. Prompts under about 2K tokens are unchanged, because copying the host experts over PCIe sets their time. Prefill logits differ from the F32 path by a KL divergence of 0.0143, where changing only the batch size gives 0.0091. patches/0008, how.

v5.1 (2026-09-24) keeps a prompt checkpoint every 32K tokens. When a client rewrites old history (OpenCode 2 drops old images from long conversations), the server resumes from the nearest checkpoint instead of re-processing the whole prompt: a 120K-token prompt edited at token 100K took 22 s instead of 63 s. patches/0007.

v5 (2026-09-23) adds image and audio input through MiMo's own encoders (--mmproj, built locally from the upstream checkpoint). It fixes llama.cpp's MiMo vision attention, which had distorted image features (cosine similarity to Xiaomi's reference: 0.69 before, 0.99999 after), and its audio segmentation and padding. Photos, screenshots, charts, several images per request and speech transcription work; video input does not. patches/0006 also adds the attention kernel and grows the KV cache in 32K steps instead of doubling.

v4.1 (2026-09-23) fixes a crash in v4: a conversation that crossed 65,536, 131,072 or 262,144 tokens while generating failed with "failed to allocate buffer for kv cache", and the next request crashed the server. patches/0005 frees the cached experts before the KV cache grows. Speed and output are unchanged.

What you need

The reference machine is an RTX PRO 6000 Blackwell Workstation Edition 96GB with a Ryzen 9 9950X3D and 128GB DDR5 (4 DIMMs), on PCIe Gen5 x16. It runs Ubuntu 24.04.4, NVIDIA driver 615.71.09, CUDA 13.3, GCC 13.3 and CMake 3.28.

• 128GB RAM is the minimum. 95.9 GiB of experts are pinned. Preflight requires at least 104 GiB available at start, and a guard stops the server if usable RAM falls below 1.5 GiB. Close other memory-heavy work.
• Speed scales with host memory and PCIe bandwidth. Every expert-cache miss is copied from RAM, and every prompt of more than a few hundred tokens touches almost every expert. The reference host reads about 50 GB/s (four DIMMs), and H2D copies reach 49.8 GB/s. Slower PCIe links will hurt.
• Reserve the whole GPU. There is one inference slot, and the profile uses nearly all 96 GB.
• About 205GB of free disk: 192 GB of weights, a 4.2 GB draft, the 2.2 GB and 4.0 GB locally generated files, and for image/audio input a 2.7 GB projector built from 2 GB of fetched encoder tensors.
• Linux with cgroup v2 and a working systemctl --user. No Docker is involved.

Build and run

Install the NVIDIA driver and the CUDA 13.3 toolkit (NVIDIA's guide). The scripts don't change drivers or install system packages.

sudo apt-get install build-essential cmake ninja-build git curl python3.12-venv
git clone https://github.com/SSHdotCodes/mimo-v2.6-flash-pro6000.git
cd mimo-v2.6-flash-pro6000

# Pinned llama.cpp + eleven cumulative patches, verified by SHA-256; full CUDA build.
CUDA_HOME=/usr/local/cuda-13.3 JOBS=12 scripts/build.sh

# CPU-only Python helpers for the overlay conversion and the benchmark client,
# plus llama.cpp's converter requirements (CPU PyTorch) for the image/audio projector.
python3.12 -m venv .venv && .venv/bin/pip install -r requirements-tools.txt \
    -r engine/requirements/requirements-convert_hf_to_gguf.txt

# 10 NVFP4 shards + BF16 draft at a pinned revision; SHA-256 checked, resumable.
python3 scripts/download.py
# Q8_0 draft, the exact dense-weight overlay and the image/audio projector,
# all checked against the tested bytes.
.venv/bin/python scripts/prepare_models.py

python3 scripts/run.py command   # exact environment and arguments
python3 scripts/run.py start     # systemd user unit, 116G cap, no swap; waits for readiness

The server listens on http://127.0.0.1:18085/v1 (loopback only), with model ID mimo-v2.6-flash-mopd (mimo-v2.6-flash-rl is still accepted). It serves both OpenAI chat completions and Anthropic-style /v1/messages, including tools, and takes images (image_url) and audio (input_audio) in OpenAI chat messages. Use temperature 1.0, top-p 0.95, top-k 0, min-p 0 and "chat_template_kwargs": {"enable_thinking": true}. Keep cache_prompt: true for agents. To run without the projector, set "vision": false in config.local.json. See operations for clients, image and audio requests, logs, your own expert profile and troubleshooting.

What makes it fast

A decode step verifies up to four tokens: one sampled token plus up to three drafted ones. Most of its time is expert copies over PCIe for cache misses; speed v6 works on what surrounds them:

• Sampling on 8 threads. The creator's top-p cuts a 152K-token distribution for every verified token. It now runs in parallel with the reference's exact cut and order: 350 → 170 µs per token.
• The next-layer prediction inside the router's matmul. The router of each MoE layer gets a second channel holding the next layer's router, with its input norm folded in. One matmul and a fused top-k give the next layer's likely experts for the prefetch: about 3 µs per layer instead of 27.5. The host now decides that prefetch after launching the layer's expert kernels.
• BF16P12. The BF16 attention-output projections and output head store each weight as a sign-and-mantissa byte plus a 4-bit exponent offset from its block of 128. The kernels decode them into the same BF16 values. 0.95 GiB more for cached experts.
• The image encoder's compute buffer only while encoding. 937 MiB more for cached experts between images.
• Drafts stop at their first unsure token. A drafted token the draft gives less than 50% is usually rejected, and verifying it widens the batch.

A prompt of more than 512 tokens runs as prefill: whole-layer GEMMs over every expert while the host experts stream in over PCIe. Speed v5.2 cut its GPU work by more than half: FP8 × FP4 tensor-core GEMMs for the experts (1,110 TFLOPS peak for FP8 on this GPU, against 278 for the TF32 it replaces), an FP8 kernel for the dense layers, masked-tile skipping in sliding-window attention, and high-priority copies that keep the prefetch ring ahead of compute. A 16K-token prompt then spent 2.25 s in kernels and 1.96 s copying 94 GB of experts, overlapped.

Speed v5.3 removes what still stalled a prompt:

• Cached-expert copies on their own stream. The ring's device-to-device copies of cached experts used to sit between the host copies on one stream, and PCIe idled about 0.8 ms per layer. Now PCIe stays 97% busy at 49.4 GB/s, and a prompt of 1K–4K tokens takes about as long as its ~84 GB of host experts take to arrive: 1.7 s.
• KV growth on the device. Growing the cache by 32K cells went through a temporary file and took up to 1.4 s at 131K. The new cache is now filled by device copies in 0.05–0.10 s.
• The draft's input stays on the GPU. The DFlash draft reads the target's hidden states at 5 layers. They went through pageable host memory, about 0.4 s per 16K-token prompt; the draft now reads them in place.
• FP8 activations from 129 tokens. Batches of up to 512 tokens keep the expert cache and copy only the experts they route to. v5.2 gave them the exact grouped kernel; they now use its FP8-activation GEMMs. A 512-token prompt went from 1.56 to 1.14 s.
• Not enabled: FP8 Q·K for global prefill attention. The 9 global layers attend over the whole context, so they dominate long prompts. A kernel that multiplies E4M3 queries directly with the cached FP8 keys is 1.3–1.5× faster per layer, but rounding the queries cost accuracy on a code-reading task, so the profile leaves it off.

A decode step verifies four tokens. Step time is mostly expert copies over PCIe plus GPU kernels, and the draft decides how many tokens each step keeps. At long context attention becomes the largest kernel. Speed v5 adds:

• A decode attention kernel for the FP8 KV cache. The key range is split over all SMs, sliding-window layers skip the ring-buffer tiles no query can see, and each tile's FP8-to-F16 expansion overlaps the previous tile's tensor-core math. One global-attention layer at 131K context went from 445 to 146 µs, and a sliding-window layer from 57 to about 10 µs.
• KV growth in 32K steps. The cache used to double (131K → 262K), and every cell of cache costs expert-cache slots. Now a conversation just past 131K keeps 3 more slots.

Speed v4 worked on all three parts of the step:

• Per-expert residency. Routing is skewed inside every layer. Instead of keeping 16 whole layers on the GPU, each of the 47 MoE layers pins its own most-used experts: 4,096 in total, picked from decode traces of coding, a tool-calling agent and prose (profiles/hot-experts.txt). A 2,480-slot LFU cache, shared out per layer, holds the rest on demand. Experts are renumbered at load and the router is permuted to match, so every token selects the same experts. Replayed cache misses per step fall by 14–35%.
• Speculative sampling. The target used to accept a draft token only when its own sample was the same token. Standard rejection sampling (accept with min(1, p/q), else resample from the residual) keeps each token's distribution exactly the target's, and accepts more draft tokens.
• Prefill that keeps the expert cache. Prompts used to drop the whole decode cache for their buffers. Now they release only what they need (typically 19–27 GiB of experts stay), copy cached experts device to device, and hand the cache back warm.
• An exact grouped kernel for short prompts (9–512 tokens) replaces dequantize-to-F32 plus thin GEMMs, using the decode kernel's arithmetic.
• From earlier releases: lossless compact encodings of the dense weights (f8s128) and the experts (nvfp4c, 4.125 instead of 4.5 bits, same values), next-layer expert prefetch, in-kernel waits for each expert's copy, and prefill that streams whole layers ahead of compute.

All switches are opt-in environment variables in patches/; scripts/run.py sets the measured combination. The engine notes also list what did not help.

Quantization, context and quality

• Weights. The target is the exact NVFP4 transcode of MiMo-V2.6-Flash-MOPD's released MXFP4 experts, published at ProCreations/MiMo-V2.6-Flash-MOPD-NVFP4 (revision da654d8e). Up to v5.3 this repo used the RL model's (ProCreations/MiMo-V2.6-Flash-RL-NVFP4, revision 34cebc5d). The target is never requantized. The dense-weight overlay, the in-memory expert re-encoding and the BF16P12 packing of the BF16 projections are lossless, verified bit-exact.
• Activations. Decode and prompts under 133 tokens use the original F32 activations. Prompt batches of 129+ tokens use FP8 (v5.2 from 512 tokens, v5.3 from 129): E4M3 with a power-of-two scale per 32 values for the expert GEMMs (MXFP8), E4M3 with an F32 scale per token and 128 values for the FP8 dense weights, and BF16 for the BF16 attention output projection. The weights are never converted. Their logits differ from the F32 path by a KL divergence of 0.0143, with the same perplexity (2.713 against 2.722); changing only the batch size gives 0.0091. On 140–520-token prompts, v5.3's next-token KL divergence against the exact path is 0.009; changing only the micro-batch size gives 0.005, and v5.2's FP8 path on 600–1,000-token prompts 0.006. The top token was the same on all 104 test prompts. Attention keeps F16 queries. v5.3's optional FP8 Q·K kernel (GGML_CUDA_MIMO_FA_PREFILL) is off, because it answered a 7.8K-token code-reading task correctly in 22 of 30 seeds against 28 of 30 without it. The other large GEMMs allow TF32. Native FP4 activation kernels are disabled.
• KV cache. Target KV is scaled E4M3 FP8, as in the previous runtime, which measured a +0.27% perplexity ratio against BF16 KV. Draft KV is BF16, and the draft's weights are Q8_0. The target verifies every token.
• Sampling. Temperature 1.0, top-p 0.95, top-k 0, min-p 0, applied by the target to every token. Speculative sampling reproduces that distribution exactly; test-spec-sampling checks it with chi-square tests. Since v6 a draft stops at its first token below 0.5 draft probability. That changes only what is proposed: the target still verifies every token.
• Output equivalence. Against speed v3 over the same 8,192 tokens, the KL divergence of v4's logits is 0.0062 and perplexity is unchanged (1.4491 vs 1.4543 ± 0.02). Changing only v3's batch size gives 0.0058. The 16K-budget coding tasks grade as before (three of four pass unit tests; the fourth runs into the cap in both). These are runtime checks, not a new accuracy evaluation of the model. Speed v5.2 returns the same bytes as v5 for prompts under 512 tokens. Speed v5.3 returns the same bytes as v5.2 for prompts under 133 tokens and for every batch over 512 tokens. Its long-context retrieval answers are token for token v5.2's. With its fused prediction off and the draft cutoff at 0, speed v6 returns v5.3's logprobs bit for bit. The fused prediction returns what v5.3 returns with expert prediction switched off: v5.3's unfused prediction changed which CUDA fusions ran on 13–20-token batches. See output checks.
• Context. The full 524,288-token capacity works (a ~520K-token request prefills in 211 s, 2,460 tok/s, and then decodes at about 45–50 tok/s). The synthetic near-capacity retrieval test failed on every runtime before v5.3; v5.3 passed it once, in a single sample. Treat 512K as a capacity, not a retrieval guarantee. Input and output share the context, and a full 131K-token generation has not been tested.
• Images and audio. The projector is a BF16 conversion of the upstream vision and audio encoder weights, with no further quantization. It is built from the RL checkpoint, whose encoder tensors MOPD shares byte for byte. Images are limited to 2,048 tokens each.

Reproduce the measurements

python3 scripts/run.py start
.venv/bin/python scripts/benchmark.py --workload --prompt-sweep --output results/benchmark.json
python3 scripts/run.py stop

By default the benchmark sends one cold prefill of about 8K tokens from the pinned llama.cpp source, then four fixed-seed 2,048-token coding requests with native thinking. It records server timings, draft acceptance and a SHA-256 of every response. On identical hardware and drivers the responses should match ours: a clean clone did. v6's answers differ from v5.3's. Its drafts stop earlier, so the speculative sampler draws differently from the same distribution. --workload adds held-out tool-calling agent sessions, coding requests and prose answers over two seeds; --prompt-sweep measures prompt time from 16 to 16,384 tokens, fresh and appended to an 8K cached prefix. scripts/long_bench.py --context 131000 measures decode from a long cached prompt. GitHub CI checks source download, patch application and the Python harness; CUDA and model checks need the GPU.

Credits and licenses

Thanks to Xiaomi MiMo for the model and its DFlash draft, the ggml-org / llama.cpp contributors, and memoriaru for the persistent expert-pool design the runtime builds on.

Repo code is MIT; inherited source keeps its upstream license. The MiMo weights, tokenizer and draft are MIT-licensed upstream. No weights or converted derivatives are redistributed here: the scripts fetch the pinned public files and build the three derived files locally. See THIRD_PARTY.md.