Skip to content

feat(moe): Ling 3.0 (bailingmoe3) support - #162

Merged
Helldez merged 2 commits into
mainfrom
feat/ling3-flash
Aug 17, 2026
Merged

feat(moe): Ling 3.0 (bailingmoe3) support#162
Helldez merged 2 commits into
mainfrom
feat/ling3-flash

Conversation

@Helldez

@Helldez Helldez commented Aug 17, 2026

Copy link
Copy Markdown
Owner

What

Ling 3.0 support (bailingmoe3), e.g. Ling-3.0-flash: 127B total, ~5B active, 512 routed experts. Upstream merged BailingMoE3 today (ggml-org/llama.cpp#26608); this PR bumps the submodule to that base and adds the one registry row the streamer needs.

How

  • Submodule bump to fork branch bmoe/expert-ready-hook-ling3: the same single expert-ready-hook commit, cherry-picked clean onto upstream 3733366. The previous fork branch stays untouched so the old pin remains reachable. The jump crosses 496 upstream commits.
  • Recipe row. bailingmoe3 combines patterns the registry already carries: split expert suffixes, biased top-k (the lfm2moe pattern), a resident shared expert, leading dense blocks, and a NextN/MTP block llama.cpp does not load by default.
  • Three adaptations the new base forces, all on our side of the seam:
    • llama_model_params.use_mmap became the load_mode enum. The streamer's required layout is now pinned with LLAMA_LOAD_MODE_MMAP.
    • The nextn/MTP tensors are now skipped at load unless load_mtp is set, and it defaults to off, so --mtp would have built its second context over a block whose tensors were never loaded. It fails badly rather than loudly: n_layer_nextn comes from the gguf metadata and stays non-zero either way, so the "no trained MTP head" guard would have passed and the trouble surfaced further in. The block is now requested exactly when speculation asks for it.
    • gpt-oss/harmony now declares its thinking tags, so the think-control probe decides by mechanism instead: a prefill that moves past the closed span into further structure of the format itself is binding, while one that just ends at the closing tag is only a suggestion (LFM2.5 stays reported as uncontrollable).

Evidence

Byte-identity gates pass on the new base (qwen3moe, gemma4, 4-shard split): streamed == resident.

PC smoke on Qwen3.6-35B-A3B, every cell of the recipe that still applies, against the recorded baseline:

cell tok/s read/token hit signal
plain 2.14 270.5 MiB 39.6% coherent text (baseline 2.11 / 272 / 36%)
--ngram --draft 3 2.14 270.5 MiB 39.6% drafted on 0 of 24 steps, identical to plain, which is the designed property
--route-ahead 2 2.68 264.3 MiB 69.0% 2984/2984 speculated experts useful, stall 0.30 down to 0.04
--mtp --draft 3 2.25 (2.09 effective) 313.4 MiB 23.9% 17/19 drafts accepted, 3.43 tokens per verify

(--odirect-zero-copy is not in main, it lives on the PR #143 branch, so that cell does not apply here.)

think_control covers the new probe logic: harmony as prefill, LFM2.5 as none, non-reasoning LFM variants as template.

Not covered

No device run yet, and no load of gpt-oss, lfm2moe or deepseek4 on the new base. Given the size of the upstream jump, a device validation is owed before this is released.

App version 0.20.0 (versionCode 35). A catalog entry for Ling follows once a reputable quantized upload exists; the URL field and file picker already take any bailingmoe3 gguf.

@Helldez
Helldez force-pushed the feat/ling3-flash branch 2 times, most recently from fec79c8 to 6a5bc4c Compare August 17, 2026 14:15
The submodule moves from fork branch bmoe/expert-ready-hook (expert-ready
hook on a July base) to bmoe/expert-ready-hook-ling3: the same single
commit, cherry-picked clean onto upstream 3733366 ("model : BailingMoE3
Support"). The old branch stays on the fork untouched, so the commit the
previous pin names remains reachable. The jump crosses 496 upstream
commits.

Three adaptations the new base forces, all on our side of the seam:

- llama_model_params lost use_mmap for the load_mode enum. The streamer's
  required layout is now pinned with LLAMA_LOAD_MODE_MMAP.
- The nextn/MTP tensors are now skipped at load unless load_mtp is set,
  and it defaults to off, so --mtp would have built its second context
  over a block whose tensors were never loaded. It fails badly rather
  than loudly: n_layer_nextn comes from the gguf metadata and stays
  non-zero either way, so the "no trained MTP head" guard would have
  passed and the trouble surfaced further in. The block is now requested
  exactly when speculation asks for it.
- gpt-oss/harmony now declares its thinking tags, upstream having grown a
  reasoning-budget sampler, so the think-control probe can no longer read
  "declares no tags" as "reasoning is structural". The probe now decides by
  mechanism: if the prefilled render moves PAST the closed span into further
  structure of the format itself (harmony closes analysis and opens the
  final channel), the prefill is binding; if the render just ends at the
  closing tag, the close is only a suggestion to a model that owns its span
  (LFM2.5, issue #82) and the probe still says so.

Byte-identity gates pass on the new base (qwen3moe, gemma4, 4-shard split).
The PC smoke on Qwen3.6-35B-A3B reads like the recorded baseline across
every cell that still applies: 2.14 tok/s at 270.5 MiB/token and 39.6% hit
plain; --ngram identical to it with 0 of 24 steps drafted, as designed;
--route-ahead 2 at 2.68 tok/s, 69.0% hit, stall 0.30 down to 0.04, with
2984/2984 speculated experts useful; --mtp at 17/19 drafts accepted and
3.43 tokens per verify decode.
Ling-3.0-flash (127B total, ~5B active) routes over 512 experts and
combines patterns the registry already carries, so streaming is one row:
standard split expert suffixes, biased top-k (the lfm2moe pattern), an
always-on shared expert that stays resident, leading dense blocks that
never bind, and a trailing NextN/MTP block that llama.cpp does not load
by default. The hybrid KDA/MLA attention stack is dense-side llama.cpp
machinery, invisible to the seam.

App version 0.20.0 (versionCode 35). Device validation and a catalog
entry follow once a reputable quantized upload exists; the URL field
and file picker already take any bailingmoe3 gguf.
@Helldez
Helldez merged commit 2d4f2b1 into main Aug 17, 2026
5 checks passed
@Helldez
Helldez deleted the feat/ling3-flash branch August 17, 2026 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant