Skip to content

feat(qairt): enable HTP multicore for QAIRT bundles - #1313

Open
MAN$I VERMA (mansiverma897993) wants to merge 2 commits into
qualcomm:mainfrom
mansiverma897993:feat/qairt-htp-multicore
Open

feat(qairt): enable HTP multicore for QAIRT bundles#1313
MAN$I VERMA (mansiverma897993) wants to merge 2 commits into
qualcomm:mainfrom
mansiverma897993:feat/qairt-htp-multicore

Conversation

@mansiverma897993

@mansiverma897993 MAN$I VERMA (mansiverma897993) commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Bumps third-party/geniex-qairt to pick up HTP multicore support (feat(core): request HTP multicore execution via NUM_CORES graph config geniex-qairt-plugin#31): ModelConfig::num_cores auto-derives from the bundle's htp_backend_ext_config.json devices[].cores list and is applied to every loaded graph via QNN_HTP_GRAPH_CONFIG_OPTION_NUM_CORES. All load paths honor the bundle knob — GenieX CLI/SDK (QairtLlm::create via modelConfigFromDirectory), plugin example executables, and VLM vision encoders — with no FFI changes.
  • The runtime now logs the device-reported and effective core counts at init, warns and clamps when a bundle requests more cores than the device exposes, and warns (without failing init) when the driver rejects the config (Multicore support is unavailable).
  • Documents the multicore story (load-time vs context-binary-generation-time, SoC support) in notes/run.md and the plugin's docs/README.md.

Depends on qualcomm/geniex-qairt-plugin#31 — the submodule pointer currently references that PR's head and should be retargeted to the upstream merge commit once it lands. A typed geniex_LlmConfig field + CLI flag can follow as a separate FFI-touching PR (per the sliding-window precedent in a10cdc7) if maintainers want a programmatic override on top of the bundle knob.

Test plan

  • Modified plugin TUs compile clean (MSVC /std:c++20) against the vendored QNN headers
  • New llm_spec_loader_test unit suite (10 cases) passes locally — two runs, including after the final revision
  • Behavioral probe of Model::applyHtpNumCores (4 cases: JSON fallback, explicit override, no-clamp, default) passes against the link-time QnnApi stub
  • On-device validation on Snapdragon/QDC hardware per the instructions in the test report
Test report (local verification + on-device instructions for hardware testers)

Test report — HTP multicore support (#1312)

What changed

  • ModelConfig::num_cores (new): HTP cores to request per graph. 0 (default) = auto-derive from the bundle's htp_backend_ext_config.json devices[].cores list (parseHtpCoreCount) — via modelConfigFromDirectory or, for hand-built configs (example executables, embedders), at init from htp_config_path. 1 = force single core. Any explicit value overrides the JSON. The GenieX SDK path (QairtLlm::create) picks it up with no SDK/FFI changes, and every load path (CLI, SDK, plugin examples, VLM vision encoders) honors the same bundle knob.
  • Model::initializeapplyHtpNumCores: logs the device-reported NSP core count, clamps requests above it with a warning, applies QNN_HTP_GRAPH_CONFIG_OPTION_NUM_CORES to every loaded graph via QnnGraph_setConfig (between context creation and first execute — the only valid window for graphs retrieved from prebuilt context binaries), and warns without failing init when the backend rejects the config.
  • QnnApi::getHtpDeviceNumCores / QnnApi::setHtpNumCores (new QNN wrapper methods).
  • Docs: docs/README.md § "HTP Backend Config and Multicore Execution" (JSON schema, load-time vs generation-time split, the Error code 1000 … key = 304 diagnostic).

Verified locally (no NPU required)

Host: Windows 11 x64, MSVC 19.29 (VS2019 BuildTools), /std:c++20, vendored QNN headers.

Check Result
Compile core/src/llm/llm_spec_loader.cpp ✅ clean (exit 0)
Compile core/src/model_init.cpp ✅ clean (exit 0)
Compile qnn-api/src/QnnApi.cpp ✅ clean (exit 0, only pre-existing size_t warnings)
New unit suite llm_spec_loader_test (10 cases) ✅ 10/10 passed (two runs, incl. after final rev)
Behavioral probe of Model::applyHtpNumCores against the link-time QnnApi stub ✅ 4/4 cases passed
clang-format on changed core/ + tests/ files ✅ no diff

Unit cases: missing file / malformed JSON / missing devices / non-array devices / device without cores → 0; single-core → 1; dual-core → 2; max across multiple devices → 3; modelConfigFromDirectory wires num_cores from the bundle JSON; defaults to 0 without the JSON.

[==========] 10 tests from 1 test suite ran. (90 ms total)
[  PASSED  ] 10 tests.

Behavioral probe (drives the exact init path hardware will hit, with the stub reporting device cores = unknown and rejecting the graph config):

  1. num_cores=0 + 2-core JSON → init-time fallback parses the JSON, attempts multicore, logs the reject warning, init continues — ✅
  2. num_cores=1 explicit + 2-core JSON → forces single core, never touches the backend — ✅
  3. num_cores=5 explicit, device count unknown → no clamp, reject warning names 5 — ✅
  4. no JSON, num_cores=0 → single-core default log only — ✅

Design note: the knob is deliberately bundle-JSON + typed ModelConfig only — no env var. GenieX previously shipped GENIEX_SLIDING_WINDOW as an env var and maintainers replaced it with typed config (a10cdc70 fix(sdk)!: move sliding-window opt-in from env var to typed config); a typed geniex_LlmConfig field + CLI flag can follow as a separate FFI-touching PR if desired.

Needs on-device validation (Snapdragon / QDC)

Not verifiable on x64 hosts — please run on real HTP hardware (e.g. via QDC):

  1. Baseline (unchanged behavior). Run any QAIRT bundle whose JSON declares one core, with GENIEX_LOG=info (CLI: --log info). Expect:
    HTP device reports 1 NSP core(s)
    HTP graphs will execute on 1 core (default; ...)
    
    Generation output and tok/s should match the previous build.
  2. Multicore request on a single-NSP SoC (clamp path). Edit the bundle's htp_backend_ext_config.json to list a second entry in devices[0].cores ({"core_id": 1, "perf_profile": "burst", "rpc_control_latency": 100}). Expect a clamp warning (Requested 2 HTP cores but the device exposes 1; clamping to 1) and a normal single-core run — no crash, no init failure. This works identically through the GenieX CLI (geniex infer), the SDK, and the plugin's example executables — all load paths honor the bundle JSON.
  3. Multicore on a multi-NSP SoC (target path). Same edit on a device whose platform info reports numCores > 1 (certain automotive/IoT parts). Expect HTP multicore enabled: graphs will execute on 2 cores; verify output correctness and compare prefill/decode tok/s vs single core.
  4. Driver-reject path. If the QNN verbose trace shows Error code 1000 ... key = 304 / Multicore support is unavailable, init must still succeed with the new warning (backend rejected QNN_HTP_GRAPH_CONFIG_OPTION_NUM_CORES; continuing on a single core).
  5. Qwen3-VL-4B-Instruct repro from the issue. Re-run the original scenario; the difference is the core counts are now visible at info level without a verbose QNN trace.

Note: context binaries compiled without a multicore graph config may not speed up even when the config is accepted — see the new docs section for the generation-time vs load-time split.

Closes #1312

Bumps the geniex-qairt submodule to the commit that plumbs a
core-count knob (ModelConfig::num_cores, derived from the bundle
htp_backend_ext_config.json devices[].cores list) through to
QNN_HTP_GRAPH_CONFIG_OPTION_NUM_CORES on every loaded graph, logs
device-reported and effective core counts at init, and warns on
clamped or rejected requests instead of staying silent.

Points at the upstream merge commit of qualcomm/geniex-qairt-plugin#31
(7aedc3e) on geniex-qairt-plugin main.

Fixes qualcomm#1312

Signed-off-by: mansiverma897993 <vmansi756@gmail.com>
Covers the htp_backend_ext_config.json cores knob, what is settable
at load time versus fixed at context-binary generation time, the
init-time core-count logging, and the multicore-unavailable driver
diagnostic.

Signed-off-by: mansiverma897993 <vmansi756@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(qairt): expose HTP multicore support in geniex-qairt-plugin

1 participant