feat(moe): Ling 3.0 (bailingmoe3) support - #162
Merged
Merged
Conversation
Helldez
force-pushed
the
feat/ling3-flash
branch
2 times, most recently
from
August 17, 2026 14:15
fec79c8 to
6a5bc4c
Compare
The submodule moves from fork branch bmoe/expert-ready-hook (expert-ready
hook on a July base) to bmoe/expert-ready-hook-ling3: the same single
commit, cherry-picked clean onto upstream 3733366 ("model : BailingMoE3
Support"). The old branch stays on the fork untouched, so the commit the
previous pin names remains reachable. The jump crosses 496 upstream
commits.
Three adaptations the new base forces, all on our side of the seam:
- llama_model_params lost use_mmap for the load_mode enum. The streamer's
required layout is now pinned with LLAMA_LOAD_MODE_MMAP.
- The nextn/MTP tensors are now skipped at load unless load_mtp is set,
and it defaults to off, so --mtp would have built its second context
over a block whose tensors were never loaded. It fails badly rather
than loudly: n_layer_nextn comes from the gguf metadata and stays
non-zero either way, so the "no trained MTP head" guard would have
passed and the trouble surfaced further in. The block is now requested
exactly when speculation asks for it.
- gpt-oss/harmony now declares its thinking tags, upstream having grown a
reasoning-budget sampler, so the think-control probe can no longer read
"declares no tags" as "reasoning is structural". The probe now decides by
mechanism: if the prefilled render moves PAST the closed span into further
structure of the format itself (harmony closes analysis and opens the
final channel), the prefill is binding; if the render just ends at the
closing tag, the close is only a suggestion to a model that owns its span
(LFM2.5, issue #82) and the probe still says so.
Byte-identity gates pass on the new base (qwen3moe, gemma4, 4-shard split).
The PC smoke on Qwen3.6-35B-A3B reads like the recorded baseline across
every cell that still applies: 2.14 tok/s at 270.5 MiB/token and 39.6% hit
plain; --ngram identical to it with 0 of 24 steps drafted, as designed;
--route-ahead 2 at 2.68 tok/s, 69.0% hit, stall 0.30 down to 0.04, with
2984/2984 speculated experts useful; --mtp at 17/19 drafts accepted and
3.43 tokens per verify decode.
Ling-3.0-flash (127B total, ~5B active) routes over 512 experts and combines patterns the registry already carries, so streaming is one row: standard split expert suffixes, biased top-k (the lfm2moe pattern), an always-on shared expert that stays resident, leading dense blocks that never bind, and a trailing NextN/MTP block that llama.cpp does not load by default. The hybrid KDA/MLA attention stack is dense-side llama.cpp machinery, invisible to the seam. App version 0.20.0 (versionCode 35). Device validation and a catalog entry follow once a reputable quantized upload exists; the URL field and file picker already take any bailingmoe3 gguf.
Helldez
force-pushed
the
feat/ling3-flash
branch
from
August 17, 2026 14:24
6a5bc4c to
8dbb96c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ling 3.0 support (
bailingmoe3), e.g. Ling-3.0-flash: 127B total, ~5B active, 512 routed experts. Upstream merged BailingMoE3 today (ggml-org/llama.cpp#26608); this PR bumps the submodule to that base and adds the one registry row the streamer needs.How
bmoe/expert-ready-hook-ling3: the same single expert-ready-hook commit, cherry-picked clean onto upstream3733366. The previous fork branch stays untouched so the old pin remains reachable. The jump crosses 496 upstream commits.bailingmoe3combines patterns the registry already carries: split expert suffixes, biased top-k (thelfm2moepattern), a resident shared expert, leading dense blocks, and a NextN/MTP block llama.cpp does not load by default.llama_model_params.use_mmapbecame theload_modeenum. The streamer's required layout is now pinned withLLAMA_LOAD_MODE_MMAP.load_mtpis set, and it defaults to off, so--mtpwould have built its second context over a block whose tensors were never loaded. It fails badly rather than loudly:n_layer_nextncomes from the gguf metadata and stays non-zero either way, so the "no trained MTP head" guard would have passed and the trouble surfaced further in. The block is now requested exactly when speculation asks for it.Evidence
Byte-identity gates pass on the new base (
qwen3moe,gemma4, 4-shard split): streamed == resident.PC smoke on Qwen3.6-35B-A3B, every cell of the recipe that still applies, against the recorded baseline:
--ngram --draft 3--route-ahead 2--mtp --draft 3(
--odirect-zero-copyis not inmain, it lives on the PR #143 branch, so that cell does not apply here.)think_controlcovers the new probe logic: harmony as prefill, LFM2.5 as none, non-reasoning LFM variants as template.Not covered
No device run yet, and no load of
gpt-oss,lfm2moeordeepseek4on the new base. Given the size of the upstream jump, a device validation is owed before this is released.App version 0.20.0 (versionCode 35). A catalog entry for Ling follows once a reputable quantized upload exists; the URL field and file picker already take any
bailingmoe3gguf.