Skip to content

Add golden store publish scripts + nightly CI wiring - #703

Open
yuhuchua-qti wants to merge 7 commits into
mainfrom
dev/yuhuchua/qnn-ut-golden-publish
Open

yuhuchua-qti wants to merge 7 commits into
mainfrom
dev/yuhuchua/qnn-ut-golden-publish

Conversation

@yuhuchua-qti

@yuhuchua-qti yuhuchua-qti commented Aug 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This PR adds the producer side of the QNN EP snapshot golden pipeline.

It regenerates snapshot/session-snapshot goldens, verifies numerical correctness through the accuracy tier, packages only accuracy-passing op groups, stamps the package with QAIRT/ORT version metadata, and wires a nightly internal workflow to publish the golden package to Artifactory.

This PR is additive to the current test/CI pipeline and does not change per-PR gating behavior. The per-PR golden consumer, routing gate, and broader snapshot non-gating rollout remain follow-up PRs.

What's In It

Snapshot Golden Publisher

  • Adds qcom/scripts/linux/publish_goldens.sh.
  • Runs run_snapshot_accuracy.sh --generate-goldens to generate snapshot/session-snapshot goldens and run accuracy verification.
  • Reads snapshot_accuracy_results/accuracy_results.json and packages only op groups whose QnnAcc_<Group>_Accuracy[_<Variant>]Test cases pass.
  • Continues after an accuracy-regression exit code and publishes only the passing subset; setup/tooling failures still abort publishing.
  • Writes a version-stamped manifest.json with QAIRT version, ORT version, git SHA, generated time, passing groups, and golden count.
  • Defaults to dry-run. --publish is required before the script uploads to Artifactory.

Version Resolver

  • Adds qcom/scripts/linux/resolve_tool_versions.sh as the shared QAIRT/ORT version resolver for this producer and the future consumer.
  • Resolves QAIRT from <bin_dir>/CMakeCache.txt -> onnxruntime_QNN_HOME -> <QNN_HOME>/sdk.yaml.
  • Resolves ORT from <bin_dir>/CMakeCache.txt -> onnxruntime_ORT_HOME -> VERSION_NUMBER / VERSION for prebuilt ORT builds.
  • Falls back to <bin_dir>/_deps/ort_core-src/VERSION_NUMBER when onnxruntime_ORT_HOME is absent, which covers the current FetchContent CI path.
  • Returns a distinct failure code when versions are undeterminable so the publisher refuses to stamp an unversioned package.

Nightly CI Producer

  • Adds .github/workflows/qualcomm-internal-snapshot-golden-publish.yml as the reusable/manual workflow.
  • Updates .github/workflows/qualcomm-internal-release-nightly.yml to call the snapshot golden producer nightly with publishing enabled.
  • Keeps workflow_dispatch safe by defaulting dry_run to true; an operator may explicitly select dry_run: false for a recovery publish.
  • Adds coverage_build_linux_x86_64 in qcom/build_and_test.py so the producer can build the coverage-instrumented test binary without generating the full coverage report.

Artifactory Store and Retention

The golden store is CI-managed validation data under ci/, not a QA release artifact:

<BUILD_ARTIFACTORY_REPO>/ci/qnn-ep-test-store/snapshot-goldens/latest/goldens.zip
<BUILD_ARTIFACTORY_REPO>/ci/qnn-ep-test-store/snapshot-goldens/archive/<date>-<sha>/goldens.zip

Archive and pointer behavior:

  • The dated/SHA archive is published first with a conditional create-only PUT. An existing archive causes the job to fail before latest is touched.
  • latest/goldens.zip is the only mutable pointer and is updated only after archive creation succeeds.
  • A future recovery mode that promotes an existing archive to latest is out of scope for this producer PR.

Cleanup behavior:

  • General ci/* artifacts still expire after 1 day.
  • ci/qnn-ep-test-store/* is excluded from the general ci/* cleanup rule.
  • ci/qnn-ep-test-store/*/archive/* is pruned after 7 days.
  • latest/* is not matched by the golden archive cleanup rule, so CI keeps a current golden pointer after the first successful publish.

Test Plan

  • Focused filter_pass_groups coverage for passing, failure, SKIPPED, NOTRUN, mixed-variant, and unrelated-suite cases using the tier-first QnnAcc naming.
  • publish_goldens.sh dry-run against a real coverage build: generated goldens, ran accuracy, staged passing groups, wrote manifest.json, packaged goldens.zip, and printed publish commands without updating the formal store.
  • Verified generated goldens.zip structure: root manifest.json, snapshot/, and session_snapshot/; 18 Clip golden JSON files; JSON parse OK; Qualcomm copyright string verified.
  • Pre-merge Artifactory round-trip validation using an isolated ci/qnn-ep-test-store/snapshot-goldens-pr703-test path.
  • Validate conditional archive creation by rerunning the same isolated publish path and confirming the existing archive fails without updating latest.
  • Live nightly publish to the formal ci/qnn-ep-test-store/snapshot-goldens store after merge.

@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/qnn-ut-golden-publish branch 2 times, most recently from e434afc to e45fe62 Compare August 18, 2026 05:55
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/migrate-clip-ut branch from e65be32 to 7c7bbb7 Compare August 19, 2026 09:13
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/qnn-ut-golden-publish branch from b0154fa to a8010de Compare August 20, 2026 05:11
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/migrate-clip-ut branch 3 times, most recently from 0407fe9 to 7717d38 Compare August 31, 2026 07:30
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/migrate-clip-ut branch 3 times, most recently from 8db7faf to 096e681 Compare September 8, 2026 04:54
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/qnn-ut-golden-publish branch 3 times, most recently from 6ba666b to 79cec5c Compare September 14, 2026 06:07
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/migrate-clip-ut branch 3 times, most recently from d937314 to 84b714b Compare September 16, 2026 08:24
@yuhuchua-qti
yuhuchua-qti changed the base branch from dev/yuhuchua/migrate-clip-ut to dev/yuhuchua/phase3-pr3-clip-migration September 18, 2026 08:16
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/phase3-pr3-clip-migration branch from 4d0c4ef to 88fd91d Compare September 22, 2026 02:55
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/qnn-ut-golden-publish branch from 5453b6b to 6a7e4dd Compare September 22, 2026 07:23
Base automatically changed from dev/yuhuchua/phase3-pr3-clip-migration to main September 22, 2026 07:56
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/qnn-ut-golden-publish branch 5 times, most recently from eec4d9f to e52ac25 Compare September 23, 2026 03:49
@yuhuchua-qti
yuhuchua-qti marked this pull request as ready for review September 23, 2026 04:01
Comment thread .github/workflows/qualcomm-internal-snapshot-golden-publish.yml Outdated
Comment thread .github/workflows/qualcomm-internal-snapshot-golden-publish.yml Outdated
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/qnn-ut-golden-publish branch from 455929c to e7f9c2a Compare September 23, 2026 06:31
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/qnn-ut-golden-publish branch 4 times, most recently from 8d37144 to a0606a9 Compare September 23, 2026 08:07
Comment thread qcom/scripts/linux/publish_goldens.sh Outdated
Comment thread qcom/scripts/linux/filter_accuracy_pass_groups.py
@yuhuchua-qti
yuhuchua-qti marked this pull request as draft September 24, 2026 01:58
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/qnn-ut-golden-publish branch 2 times, most recently from 82a0b26 to 8f521e1 Compare September 24, 2026 04:10
@yuhuchua-qti
yuhuchua-qti marked this pull request as ready for review September 24, 2026 04:19
Comment thread .github/workflows/qualcomm-internal-release-nightly.yml
Comment thread .github/workflows/qualcomm-internal-snapshot-golden-publish.yml Outdated
Comment thread .github/workflows/qualcomm-internal-snapshot-golden-publish.yml
Comment thread qcom/scripts/artifactory/aql/expired_ci_artifacts.json Outdated
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/qnn-ut-golden-publish branch from f6f3585 to e43c19f Compare September 29, 2026 10:05

@kuanyul-qti kuanyul-qti left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add CI testing for the tests under qcom/scripts/linux/tests? We can track it as a follow-up. Others LGTM.

@yuhuchua-qti

Copy link
Copy Markdown
Collaborator Author

Should we add CI testing for the tests under qcom/scripts/linux/tests? We can track it as a follow-up. Others LGTM.

Thanks for the suggestion. The tests under qcom/scripts/linux/tests are not currently invoked by CI. We will track adding a lightweight Linux CI step to run python3 -m pytest qcom/scripts/linux/tests -q as a follow-up, so it can be scoped independently from this publisher change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants