fix(sdk): default llama_cpp thread counts to llama.cpp's own value - #1367
Draft
Mengsheng Wu (mengshengwu) wants to merge 1 commit into
Draft
fix(sdk): default llama_cpp thread counts to llama.cpp's own value#1367Mengsheng Wu (mengshengwu) wants to merge 1 commit into
Mengsheng Wu (mengshengwu) wants to merge 1 commit into
Conversation
The NPU/GPU device paths hardcoded n_threads/n_threads_batch to 6 on every platform, an unfulfilled compute_configs.py placeholder from the original per-(platform, device) params refactor. llama.cpp's own tools never vary thread count by device and default to common_cpu_get_num_math() for both decode and batch threads, so geniex-bench's NPU prefill throughput lagged llama-bench's by a wide margin on multi-core hosts. Mirror that default instead. Raising the thread count also exposed a latent bounds bug in the CPU pinning logic: reserved_cores=2 assumed n_threads always left slack below the core count, which no longer holds once n_threads can reach hardware_concurrency(). Scale reserved_cores down instead of pinning past the real core count or oversubscribing it. Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
Mengsheng Wu (mengshengwu)
marked this pull request as draft
August 20, 2026 14:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
geniex-bench's NPU prefill throughput lagged llama.cpp's own
llama-benchby a wide margin on the same model/device. Root-caused to a thread-count mismatch, not NPU compute:sdk/plugins/llama_cpp/src/params.cpphardcodedn_threads/n_threads_batchto 6 for the NPU/GPU device paths on every platform, an unfulfilledcompute_configs.pyplaceholder from the original per-(platform, device) params refactor.llama-bench,llama-cli) never vary thread count by backend — they default tocommon_cpu_get_num_math()for both decode and batch threads.This change makes
resolve_n_threadsmirror that upstream default for every device/platform, removing the placeholder table entirely.Raising the thread count also exposed a latent bounds bug in
build_threadpool_params's CPU pinning:reserved_cores=2assumedn_threadsalways left slack below the real core count, which no longer holds oncen_threadscan reachhardware_concurrency().reserved_coresnow scales down instead of pinning past the real core count or oversubscribing it.Test plan
On-device verification (Qualcomm IQ-9075 EVK, QCS9075, 8-core CPU, Hexagon v73 NPU), same
gemma-4-E4B-it-Q4_0.ggufmodel,-p 16256 -c 16384:[Optimise] context paramslog now showsn_threads=8, n_threads_batch=8(was hardcoded to 6) — confirmscommon_cpu_get_num_math()is picked up.llama-bench's own default-thread baseline (103.3 tps) on the same device/model.reserved_coresfix, raising the thread count produced 2 recurringfailed to set affinity mask ... Invalid argumentwarnings per run (pinning past the real 8-core count) and silently capped effective parallelism back down; after the fix, zero affinity warnings and prefill throughput unaffected.