feat(native-mtp): prototype Qwen3-Next native MTP path

[ntny]

This is an early prototype to discuss the native Qwen3-Next MTP direction with the community.

The goal is to keep the MTP block inside the target model, load its weights with the target checkpoint, and reuse the existing EAGLE v2 orchestration with minimal changes.

Motivation

Prototype native Qwen3-Next MTP: keep the MTP block inside the target model and load its weights with the target checkpoint.

This avoids a separate draft model loading path and makes MTP initialization/debugging simpler. Draft KV stays separate to avoid corrupting target KV.

Modifications

• Added --enable-native-mtp.
• Qwen3-Next initializes an embedded MTP block inside the target model when the flag is enabled.
• MTP checkpoint weights are loaded through the target model weight-loading path.
• Added a native MTP speculative path that reuses EAGLE v2 orchestration and overrides only draft runner construction.
• Draft execution calls the embedded target-model MTP block.
• Draft KV remains separate from target KV.

Prototype Limitation

Draft CUDA graph capture is intentionally disabled at this prototype stage. Native MTP currently runs draft decode/extend through eager execution.

So current performance numbers should not be treated as final.

The main risk with enabling CUDA graph capture is correctness around static graph buffers and KV writes: the existing EAGLE graph path was built around a separate draft runner, while this prototype uses an embedded MTP block exposed through a lightweight runner
facade. Before enabling graphs, we need to verify that draft decode/extend writes only to draft KV and produces logits matching the separate draft model path.

Startup Check

Both the baseline separate-draft flow and the native MTP prototype start successfully and handle a /generate request.

From the startup logs:

• Baseline loads the target model first:
  • Qwen3NextForCausalLM
  • memory usage: 74.31 GB
• Baseline then loads a separate draft model:
  • Qwen3NextForCausalLMMTP
  • additional memory usage: 2.12 GB
• Native MTP loads only the target model:
  • Qwen3NextForCausalLM
  • memory usage: 75.84 GB

So on tp=2, native MTP avoids the second draft model load and saves about: 74.31 GB + 2.12 GB - 75.84 GB = 0.59 GB

This is expected because native MTP shares target embed_tokens and lm_head instead of keeping separate draft copies.

Accuracy Tests

In progress. This is currently a prototype for validating the native MTP direction.

Speed Tests and Profiling

In progress. Performance numbers will be collected after the prototype path is validated.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Run #32876597305
Latest PR Test (Extra): ❌ Run #32876597058
Latest PR Test (AMD ROCm 7.2): ❌ Run #32876597273