fix(sdk): align llama_cpp MTP speculative decode with llama.cpp - #1376
Merged
Conversation
…ugin The target context was built without the speculative config, so it never requested recurrent-state rollback snapshots and capped itself at one logits row per sequence while the verification batch asks for n_max + 1. The draft context was pinned to a 64-token batch instead of inheriting the target's geometry. Follow common_base_params_to_speculative / server_output_limits instead. On gemma-4-26B this is a faithfulness fix rather than a speedup: llama.cpp requests n_rs_seq = n_max too and llama_context clamps it to 0 for architectures without partial rollback, and nothing attaches a backend sampler that would enforce the per-sequence output limit. Signed-off-by: Mengsheng Wu <mengshen@qti.qualcomm.com>
build_threadpool_params strict-pinned the 6 HTP-side threads to cores 2-7 on Linux. On Snapdragon X2 Elite that costs decode throughput (19.9 -> 21.7 tok/s on gemma-4-26B MTP), and driving llama-server with the same mask reproduces it. llama.cpp never pins on its own Snapdragon Linux path; Windows already opts out and Android keeps its mask. Signed-off-by: Mengsheng Wu <mengshen@qti.qualcomm.com>
decode_speculative called llama_memory_seq_rm on both the target and draft memory every step, even when the target accepted the whole draft and the call was a no-op. The server guards the same rollback behind n_rollback > 0; seq_rm is not free on the HTP backend. Signed-off-by: Mengsheng Wu <mengshen@qti.qualcomm.com>
GGML_OPENMP defaulted to ON, so ggml-cpu ran its graphs in OpenMP parallel regions and ignored the threadpool the plugin attaches -- the n_threads and poll tuning in build_threadpool_params never took effect. It is the dominant cost in MTP decode on Snapdragon X2 Elite: 21.7 -> 30.3 tok/s on gemma-4-26B with a 4267-token prompt. llama.cpp's own Snapdragon presets set it OFF on every platform. Signed-off-by: Mengsheng Wu <mengshen@qti.qualcomm.com>
Mengsheng Wu (mengshengwu)
requested review from
David Qian (Davidqian123),
AlexCHEN (alexchen4ai),
Paul Zhu (vinovo) and
Perry Cheng (zhycheng614)
as code owners
August 24, 2026 09:59
Mengsheng Wu (mengshengwu)
requested review from
RemiliaForever (RemiliaForever) and
Zack Li (zhiyuan8)
as code owners
August 24, 2026 09:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
--spec-type draft-mtpdecoded well belowllama-serverbuilt from the same pinnedthird-party/llama.cpp. Per-phase timing insidedecode_speculativeshowed no CPU-side overhead — the whole verification step was the target'sllama_decode, and only batched forwards were slow (the single-token forward was already faster than llama.cpp's). That pointed at the build config rather than the plugin logic.GGML_OPENMPdefaulted to ON, so ggml-cpu ran its graphs in OpenMP parallel regions and ignored the threadpool the plugin attaches;build_threadpool_params'n_threads=6/poll=1000never applied. Dominant cost: 21.7 → 30.3 tok/s.llama-serverwith the same mask reproduces the slowdown.n_rs_seqand the per-draft logits rows now come from the parsed spec config, and the draft context inherits the target's batch geometry instead of a hardcoded 64. Faithfulness fix; no measurable delta on gemma-4-26B.n_rollback > 0.GGML_OPENMP=OFFlands on the sharedsnapdragonandcpu-onlypresets, matching llama.cpp on all three platforms; the thread-pinning change is Linux/NPU only.Test plan
gemma-4-26B-A4B-itQ4_0 + its MTP head on Snapdragon X2 Elite (Debian 13, 4x HTP), greedy, thinking on, againstllama-serverat the same submodule revision.llama-server(25.7); prefill (638 → 632) and draft acceptance (94.1%) unchanged, so the win is entirely per-verification-step costGGML_OPENMP=OFFreaches them through the shared preset