Skip to content

Add a utility for testing Omega PRs on the supported machines - #830

Draft
xylar wants to merge 66 commits into
E3SM-Project:mainfrom
xylar:add-omega-testing-workflow
Draft

Add a utility for testing Omega PRs on the supported machines#830
xylar wants to merge 66 commits into
E3SM-Project:mainfrom
xylar:add-omega-testing-workflow

Conversation

@xylar

@xylar xylar commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

This PR adds utils/omega/pr_testing, a utility that does the testing the Omega PR template asks for (CTests and omega_pr on each supported machine, plus linting and the docs build) and posts each result as a Testing comment. An initiator on any machine pins the commits to test in a branch on the requester's fork; a tester on each machine builds, runs and reports its row. Agents follow utils/omega/pr_testing/AGENTS.md, and the design is in docs/design_docs/omega_pr_testing.md.

For reviewers to weigh in on:

  • which steps are scripted and which are left to agent instructions (see Decisions in the design doc)
  • an existing baseline is reused only if it was run with exactly the same Polaris commit

Supporting changes:

  • the PR summaries from polaris suite and the CTest utility now give the Polaris and Omega commits, with recent history, for both the baseline and the PR (building on Record the commit a model was built from in provenance #829)
  • polaris.baselines finds existing baseline runs whose provenance matches
  • polaris.job can submit job scripts with Slurm or PBS dependencies

The first trial is E3SM-Project/Omega#553.

Checklist

  • Developer's Guide has been updated
  • API documentation in the Developer's Guide (api.md) has any new or modified class, method and/or functions listed
  • Testing comment in the PR documents testing used to verify the changes

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

🤖 Generated with Claude Code

@xylar xylar added enhancement New feature or request framework Changes relating to the polaris framework as opposed to individual tests or analysis utility Utility script(s) in the utils directory AI Related to AI interactions labels Sep 29, 2026
@xylar xylar self-assigned this Sep 29, 2026
@xylar
xylar requested a review from sbrus89 September 29, 2026 14:12
@xylar

xylar commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

@sbrus89, I'm still testing this out. Feel free to have a look at the design and description if you want to weigh in early.

@xylar
xylar force-pushed the add-omega-testing-workflow branch 4 times, most recently from c9b05a6 to ea30a87 Compare September 29, 2026 19:57
@xylar

xylar commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Testing

I used this branch to test E3SM-Project/Omega#553 on all six rows of the Omega PR template, with an agent on each of Chrysalis, Frontier, Perlmutter and Aurora. Every row passed and posted its own Testing comment on the PR:

row comment
chrysalis, intel link
frontier, craygnu link
frontier, craygnu-mphipcc link
pm-cpu, gnu link
pm-gpu, gnugpu link
aurora, oneapi-ifx link

The trial led to the Frontier (normal QOS) and Aurora (capacity queue, chained jobs) submission changes, and to E3SM-Project/Omega#581, fixed by E3SM-Project/Omega#582. The unit tests (tests/omega_pr_testing, test_baselines.py, test_job_submit.py, test_output_for_pr.py, test_provenance.py) pass.

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

@xylar
xylar force-pushed the add-omega-testing-workflow branch 3 times, most recently from 3aa3e69 to b77c9fa Compare October 1, 2026 13:06
xylar and others added 11 commits October 1, 2026 08:07
@xylar @claude
Add a design doc for Omega pull request testing
19dc591 
The design turns the one-off agent handoff used to test Omega#481 into a
Polaris utility, utils/omega/pr_testing, that any developer can use.  An
initiator on any supported machine pins the commits under test in a branch
on the requester's fork; a tester on each machine builds and runs both
sides and posts its row's Testing comment, with provenance, verbatim
results and new compiler warnings.  Scripts cover pinning, baseline
matching and reporting; agent instructions cover setup and failures.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Give the Omega PR testing baseline its own Polaris commit
d8be199 
Omega#524 needs Polaris changes (polaris#731) that the baseline Omega cannot run, so the PR is tested with a Polaris test merge and the baseline runs from a second checkout without it.  The manifest records the baseline's Polaris commit, the polaris_commit by default, and its submodule pin is the default baseline.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Make the Omega PR tester's prompt the whole handoff
2f56c66 
init's prompt names the Polaris branches and fork when the PR needs Polaris changes, and the utility's AGENTS.md shows its template, so nothing is ever added to a handoff by hand.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Give every Omega PR tester the same prompt
dff04d7 
The prompt has no row in it.  Each tester finds its rows with status, and Perlmutter's two rows go to one tester.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Never require gh for Omega PR testing
bb2b842 
Developers without gh logged in can still test.  Queries fall back to GitHub's REST API, the requester comes from the fork, and only posting needs gh; without it, comments are posted by hand.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Prepare each Omega worktree once in Omega PR testing
78af784 
Rows set up at the same time share Omega worktrees, and each build initialized the tree's submodules, so concurrent setups on Perlmutter collided.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Give Omega PR testing jobs on Frontier an hour
c99204c 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Add a reader for provenance files and record the model
d58ce81 
polaris.provenance.read() parses the entries before the list of tasks,
gathering the git entries for Polaris and the component (with their logs
and whether they were marked "(at setup, not build)") into dictionaries.
The PR summary and the baseline matcher for Omega PR testing both need
these entries, which only polaris.provenance knows how to write.

provenance also records the ocean model, so a reader can name the
component.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
List the Polaris and component commits in the PR summary
a5988a0 
<suite>_output_for_pr.md now names the Polaris, baseline and PR component
commits, with separate baseline and PR Polaris lines when the two runs
used different commits, and ends with a collapsed block of each side's
recent commits.  It reads both provenance files with
polaris.provenance.read(), which replaces _parse_baseline_build().

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Write the CTest summary to ctest_output_for_pr.md
92c53c7 
The CTest job now writes the summary it prints for Omega PRs to
ctest_output_for_pr.md in the build directory as well, and adds the build
directory and the Omega commit from the build's omega_source.txt.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Add tests for provenance reading and the PR summaries
093a4a3 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xylar and others added 28 commits October 1, 2026 08:07
@xylar @claude
Test extra submission options and refused submissions
a56cfca 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Add SubmissionError to the API documentation
f386a62 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Chain each row's Omega PR testing jobs on Aurora
b4ba88a 
Aurora limits how many jobs a user may have in the 'Q' state, and a row queued its baseline and CTest jobs at once.  On Aurora, setup now makes each job wait for the one before, since a job held on a dependency does not count toward the limit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Test that Omega PR testing chains jobs on Aurora
908d8e5 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Tell Omega PR testers what to do when a job is refused
11df23a 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Note that Omega#582 fixes the build race in the PR testing instructions
f4a5103 
Builds of Omega trees that include Omega#582 keep their CIME case output in their own build directory.  The one-setup-at-a-time rule stays until Polaris' Omega submodule includes the fix, since baselines are built from it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Submit Omega PR testing jobs to Aurora's capacity queue
f262b55 
The omega_pr suite asks for Aurora's debug queue, which is limited to a job at a time, and prod only takes jobs of 256 nodes or more.  capacity, for 1 to 16 nodes, is the queue for jobs this size, so setup now adds -q capacity on Aurora.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Test that Omega PR testing uses Aurora's capacity queue
e887813 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Tell Omega PR testers not to choose another queue themselves
08d0532 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Let Omega PR testers run setups on one machine at the same time
bca32af 
Polaris' Omega submodule now includes Omega#582, so baselines and PR builds both keep their CIME case output in their own build directory and no longer race.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Record the baseline's Polaris commit in the Omega PR testing manifest
62192e8 
init --baseline-polaris-ref names the Polaris commit the baseline runs with, and its Omega submodule becomes the default baseline.  It is the --polaris-ref commit unless an Omega PR needs Polaris changes that the baseline Omega cannot run.  The tester prompts say when the baseline needs a second checkout.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Set up an Omega PR testing baseline from a second Polaris checkout
ea90ce2 
When the manifest's baseline Polaris commit is not its Polaris commit, setup --baseline-load-script names the load script of a checkout at the baseline commit.  setup checks that the checkout is clean, is for the row, contains the baseline commit but not the PR's Polaris commit, and pins the baseline Omega.  The matcher looks for baselines from that checkout's commit, and a new baseline is set up in a clean login shell with that load script sourced, so its job script loads that checkout too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Say in Omega PR test reports when the baseline used another Polaris
226b2ff 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Test the Omega PR testing baseline from a second Polaris checkout
b7ac42a 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Document testing an Omega PR that needs Polaris changes
d19c0a1 
The initiator passes a Polaris test merge and the baseline's Polaris commit to init, and each tester passes setup the load script of a second checkout for the baseline.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Have Omega PR testers check their config with the requester
dea412f 
The requester cannot keep ~/.config/omega_pr_test.cfg up to date by hand on every machine, so the tester checks it, drafts any change and writes it once the requester agrees.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Name the Polaris branches in Omega PR testers' prompts
76b6932 
--polaris-ref and --baseline-polaris-ref now name branches on the requester's Polaris fork, which the new polaris_fork setting gives.  When either is used, each tester's prompt says which checkouts the PR and baseline run from, so the prompt is the whole handoff.  --baseline-polaris-ref needs --polaris-ref.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Give the Omega PR testing handoff a fixed form
d745265 
AGENTS.md shows the prompt init prints, which is the whole handoff, and tells testers how to get the PR's and baseline's Polaris checkouts from it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Have Omega PR testers make missing Polaris checkouts themselves
1ba4e96 
Only deploying a checkout is left to the requester.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Have Omega PR testers show the requester things in their replies
0e6c461 
The requester often cannot see an agent's command output, so a config file or summary the agent asks about goes in its message as a code block.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Print one Omega PR testing prompt and name each machine's rows
2e86ba5 
init prints a single prompt for every machine.  status, given the test branch, names the manifest's rows for the machine it runs on, with pm-cpu and pm-gpu together since they share login nodes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Have Omega PR testers work out their own rows
b534bd1 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Read GitHub without gh in the Omega PR testing utility
ad16067 
When gh is not installed or not logged in, pull requests, comments and CI checks come from GitHub's REST API, with GH_TOKEN or GITHUB_TOKEN used if set.  The requester is the owner of the fork in the config.  --post still needs gh, and without it stops and says to post the comment by hand.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Tell Omega PR testers that gh is optional
c0c9f94 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Let agents run ./deploy.py
a9b54ff 
Creating or refreshing pixi-env/ and the load scripts no longer needs a developer.  Agents still ask before --deploy-spack or --spack-path, which build or replace Spack environments that may be shared.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Have Omega PR testers deploy missing load scripts themselves
8302158 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Initialize an Omega worktree's submodules once, under a lock
7b46578 
Rows set up at the same time on one machine share Omega worktrees keyed by commit, and each polaris suite --build ran git submodule update in the shared tree.  On Perlmutter, pm-cpu's baseline build failed with "Unable to find current revision in submodule path 'externals/scorpio'" while pm-gpu updated the same tree.  setup now makes each worktree and initializes the submodules build_omega.template needs under a mkdir lock, which works on GPFS and Lustre, and leaves a .ready marker, so the build's own update finds nothing to do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar @claude
Submit Omega PR testing jobs on Frontier with an hour of wall time
bb41df9 
On the normal QOS, the omega_pr suite's 30-minute wall time, chosen to fit debug queues, is only a limit the job script sets.  Frontier's GPU row ran its baseline past it on Omega#524, and the tester had to resubmit it.  The suite and other people's Frontier runs are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar
xylar force-pushed the add-omega-testing-workflow branch from b77c9fa to bb41df9 Compare October 1, 2026 13:19
@xylar

xylar commented Oct 1, 2026

Copy link
Copy Markdown
Collaborator Author

Testing

Second trial: Omega#524 on all six template rows, using a Polaris test merge with #731 for the PR and a second checkout without it for the baseline. All six rows passed with no new warnings. The reports are on Omega#524 and the five comments after it.

pytest tests on Chrysalis, on bb41df948: 1192 passed.

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sbrus89 sbrus89 Awaiting requested review from sbrus89

At least 1 approving review is required to merge this pull request.

Assignees

@xylar xylar

Labels

AI Related to AI interactions enhancement New feature or request framework Changes relating to the polaris framework as opposed to individual tests or analysis utility Utility script(s) in the utils directory

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@xylar