Skip to content

fix(sdk): default llama_cpp thread counts to llama.cpp's own value - #1367

Draft
Mengsheng Wu (mengshengwu) wants to merge 1 commit into
mainfrom
fix/npu-thread-count-default
Draft

fix(sdk): default llama_cpp thread counts to llama.cpp's own value#1367
Mengsheng Wu (mengshengwu) wants to merge 1 commit into
mainfrom
fix/npu-thread-count-default

Conversation

@mengshengwu

Copy link
Copy Markdown
Contributor

Summary

geniex-bench's NPU prefill throughput lagged llama.cpp's own llama-bench by a wide margin on the same model/device. Root-caused to a thread-count mismatch, not NPU compute:

  • sdk/plugins/llama_cpp/src/params.cpp hardcoded n_threads/n_threads_batch to 6 for the NPU/GPU device paths on every platform, an unfulfilled compute_configs.py placeholder from the original per-(platform, device) params refactor.
  • llama.cpp's own tools (llama-bench, llama-cli) never vary thread count by backend — they default to common_cpu_get_num_math() for both decode and batch threads.

This change makes resolve_n_threads mirror that upstream default for every device/platform, removing the placeholder table entirely.

Raising the thread count also exposed a latent bounds bug in build_threadpool_params's CPU pinning: reserved_cores=2 assumed n_threads always left slack below the real core count, which no longer holds once n_threads can reach hardware_concurrency(). reserved_cores now scales down instead of pinning past the real core count or oversubscribing it.

Test plan

On-device verification (Qualcomm IQ-9075 EVK, QCS9075, 8-core CPU, Hexagon v73 NPU), same gemma-4-E4B-it-Q4_0.gguf model, -p 16256 -c 16384:

  • [Optimise] context params log now shows n_threads=8, n_threads_batch=8 (was hardcoded to 6) — confirms common_cpu_get_num_math() is picked up.
  • NPU prefill throughput: 80.7 tps → 98.2 tps, closing to within ~5% of llama-bench's own default-thread baseline (103.3 tps) on the same device/model.
  • No regression in decode throughput.
  • CPU-pinning fix verified: before the reserved_cores fix, raising the thread count produced 2 recurring failed to set affinity mask ... Invalid argument warnings per run (pinning past the real 8-core count) and silently capped effective parallelism back down; after the fix, zero affinity warnings and prefill throughput unaffected.
  • GPU path — not independently perf-validated on-device (the QDC device session ended mid-verification); same reasoning applies (the "6" was never device-tuned) but flagging as untested for GPU/Windows/Android.

The NPU/GPU device paths hardcoded n_threads/n_threads_batch to 6 on
every platform, an unfulfilled compute_configs.py placeholder from the
original per-(platform, device) params refactor. llama.cpp's own tools
never vary thread count by device and default to
common_cpu_get_num_math() for both decode and batch threads, so
geniex-bench's NPU prefill throughput lagged llama-bench's by a wide
margin on multi-core hosts. Mirror that default instead.

Raising the thread count also exposed a latent bounds bug in the CPU
pinning logic: reserved_cores=2 assumed n_threads always left slack
below the core count, which no longer holds once n_threads can reach
hardware_concurrency(). Scale reserved_cores down instead of pinning
past the real core count or oversubscribing it.

Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant