Conversation
forsyth2
added
Documentation
Collaborator
Author
|
Updating expected results for the 8/12 test run Set up branchgit checkout partial-update-expectations
git fetch upstream main
git rebase upstream/main
git log --oneline | head -n 4
# b5d41f34 Add auto-generated update files
# ef72638c Change term from diags to tasks
# fffd2338 Support partial updates of weekly/campaign expected test results
# 4735ef4d Merge pull request #854 from E3SM-Project/update-pre-commit-deps
lcrc_conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-pr852-20260814
conda activate zppy-pr852-20260814
pre-commit run --all-files
python -m pip install .
cd docs
make html
cp -r _build/ /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_docs_pr852
chmod -R 755 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_docs_pr852# First, let's copy over the old expected results #############################
cp -r /lcrc/group/e3sm/public_html/zppy_test_resources /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20260814 # < 30 min
# Second, update the expected results #########################################
# Let's update the simpler tests' results first:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
git status
# You might have changed branches since you ran the tests.
# Good, on correct branch: test_zppy_20260812_run1
conda activate test-zppy-main-20260812_run1
# Good, now on correct env
# Make sure the update script permissions are set up
chmod 755 tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_campaign_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_defaults_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Specific to partial updates:
# Update the task list
emacs tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# This doesn't have the necessary lines yet because PR #852 didn't exist when this was run!
git add -A
git commit -m "Changes made during testing"
# Remove the old templates and replace with the files from PR #852
rm tests/integration/template_update_campaign_expected_files.sh
rm tests/integration/template_update_weekly_expected_files.sh
emacs tests/integration/template_update_campaign_expected_files.sh
emacs tests/integration/template_update_weekly_expected_files.sh
git diff # Looks ok
git add -A
git commit -m "Copy changes from PR 852"
# Now, we need to re-render the generated files
python tests/integration/utils.py
git diff # Looks ok
chmod 755 tests/integration/generated/update_campaign_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Double check we have a record of the original expected results
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20260814
# Update the task list now that it's available:
emacs tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# These are the 3 tasks we know we can update the expected results for:
# tasks=("e3sm_diags" "global_time_series" "pcmdi_diags")
git add -A
git commit -m "Update the update scripts"
# These scripts update the expected results and re-run the tests:
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# 1 passed in 2.84s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 6 passed in 2.43s
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed in 0.72s
# This script only updates the expected results
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh # ~30 minSkipped because ls /lcrc/group/e3sm/public_html/zppy_test_resources
# Confirmed there are expected results subdirs and image lists for each cfg
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Confirmed everything was updated today 8/14
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
pytest tests/integration/test_bundles.py
# 2 passed in 1.53s
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
bash
source ~/.bashrc
lcrc_conda
conda activate test-zppy-main-20260812_run1
pytest tests/integration/test_images.py # ~40 min
exit # Exit bash shell
exit # Exit compute noteOutputOutput:
Failing image check tests:
Good, this table matches the |
Collaborator
Author
|
I think this PR may need to be merged independently of the environment tracking noted here . The expected-results-updater script just chooses which output directories to copy over. To actually populate those dirs with environment notes, we'd need to update the test script itself. And that's what #871 is doing. Specifically:
|
forsyth2
force-pushed
the
partial-update-expectations
branch
from
September 15, 2026 00:54
14c63dc to
03ce4d2
Compare
Previously, update_weekly_expected_files_*.sh and
update_campaign_expected_files.sh always overwrote every diag
subdirectory (e.g. e3sm_diags, mpas_analysis, ilamb, ...) or every
campaign in one pass, making it impossible to accept new results for
one diagnostic while leaving another pinned to its old expected output.
- template_update_weekly_expected_files.sh: add a `diags` selector
array. Empty (default) preserves the old full-wipe-and-copy
behavior. Populated (e.g. diags=("e3sm_diags")), only those
subdirectories are refreshed; "bundle_files" is selectable the
same way for the bundle*.bash copy step. The image list is still
generated by walking the full directory, so it reflects both
updated and untouched diags.
- template_update_campaign_expected_files.sh: hardcoded campaign
list replaced with a `campaigns` array for the same kind of subset
updates.
- template_update_bash_generation_expected_files.sh and
template_update_defaults_expected_files.sh: unchanged -- each
produces a single expected-files dir with nothing to partially select.
- docs/source/dev_guide/tests/update_expected_results.rst: document
the new selectors and note the full backup step should still
always run first.
This diff was generated by Claude (Anthropic).
Add the `update_` files that `python tests/integration/utils.py` creates.
forsyth2
force-pushed
the
partial-update-expectations
branch
from
September 15, 2026 00:55
03ce4d2 to
7cfba4b
Compare
Collaborator
Author
|
Updating expected results for the 9/4 test run. We got confirmation that Set up branchcd ~/ez/zppy
git status
# nothing to commit, working tree clean
git checkout partial-update-expectations
git log --oneline | head -n 4
# b5d41f34 Add auto-generated update files
# ef72638c Change term from diags to tasks
# fffd2338 Support partial updates of weekly/campaign expected test results
# 4735ef4d Merge pull request #854 from E3SM-Project/update-pre-commit-deps
# Good, commit titles match https://github.com/E3SM-Project/zppy/pull/852/commits
# Hashes don't match though. We need to update the GitHub branch since we rebased
# on an earlier results update.
git push -f upstream partial-update-expectations
git fetch upstream main
git rebase upstream/main
git log --oneline | head -n 4
# 7cfba4b1 Add auto-generated update files
# 1ecb557d Change term from diags to tasks
# 9c6c5185 Support partial updates of weekly/campaign expected test results
# 5232fe30 Rank image check failures by severity (#865)
# Push the latest commit hashes
git push -f upstream partial-update-expectations
# Good, https://github.com/E3SM-Project/zppy/pull/852/commits is updated.
lcrc_conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-pr852-20260914
conda activate zppy-pr852-20260904
pre-commit run --all-files
python -m pip install .Copy over the old expected resultsls -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Updated 9/4
# To know when each specific task had its expected results updated, check:
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources/expected_comprehensive_v3/
# Sep 4 mpas_analysis
# Sep 4 e3sm_diags
# Aug 14 pcmdi_diags
# Aug 14 global_time_series
# May 20 livvkit
# May 20 ilamb
cp -r /lcrc/group/e3sm/public_html/zppy_test_resources /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20260914 # <30 minutesUpdate the expected results# Let's update the simpler tests' results first:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
git status
# Good, on correct branch: On branch test_zppy_20260904_run1
# Some changed files exist
git add -A
git commit -m "Testing" --no-verify
git log --oneline | head -n 20
# fdca17c7 Testing
# b72f1d03 Ignore hairlines when looking for changed values
# 4992994e Rank the cosmetic sample by how much actually differs
# 7fd3dbc6 Separate images that did not change from ones that look cosmetic
# a3eec64a Testing
# 35fb99ba Describe how images differ, not why
# ec3fe89f Do not claim a panel changed when only labels did
# da893007 Settle unchanged images from their file bytes
# ee8b0b03 Do not fail when every image is unchanged
# 3fb3d241 Testing
# 7775d42f Name the likely cause on each diff grid row
# 8588ffb2 Hide unused axes on the last page of the diff grid
# e13ee3e0 Label and order the diff grid PDF by severity
# 2dd4d83c Ignore scattered spots when looking for changed values
# ce369bd5 Document how to review image check results
# 3bbacefd Rank image check failures by severity instead of listing them flat
# 31f3d9cd Detect small isolated changes such as a changed number
# 5c346f72 Add image severity scorer
# d0aa0826 Testing
# 77f4ab66 Append try# suffix to image_check_failures dir instead of overwriting (#861)
# Let's drop these extra commits from testing https://github.com/E3SM-Project/zppy/pull/865
git checkout -b test_zppy_20260904_run1_backup20260914
git checkout test_zppy_20260904_run1
git rebase -i 77f4ab66 # Drop later commits
git log --oneline | head -n 2
# d0aa0826 Testing
# 77f4ab66 Append try# suffix to image_check_failures dir instead of overwriting (#861)
conda activate test-zppy-main-20260828_run1
# Good, now on correct env
# Make sure the update script permissions are set up
chmod 755 tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_campaign_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_defaults_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Specific to partial updates:
# Update the task list
emacs tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# This doesn't have the necessary lines yet because PR #852 didn't exist when this was run!
# Remove the old templates and replace with the files from PR #852
rm tests/integration/template_update_campaign_expected_files.sh
rm tests/integration/template_update_weekly_expected_files.sh
# Recall ~/ez/zppy is on the branch for PR #852
cp ~/ez/zppy/tests/integration/template_update_campaign_expected_files.sh tests/integration/template_update_campaign_expected_files.sh
cp ~/ez/zppy/tests/integration/template_update_weekly_expected_files.sh tests/integration/template_update_weekly_expected_files.sh
git diff # Looks ok
git add -A
git commit -m "Copy changes from PR 852"
# Now, we need to re-render the generated files
emacs tests/integration/utils.py # Confirm ALL cfgs_to_run and tasks_to_run are set to run
python tests/integration/utils.py
git status
# modified: tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# modified: tests/integration/generated/update_weekly_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_campaign_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Double check we have a record of the original expected results
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20260914
# Update the task list now that it's available:
emacs tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# What we know we can update the expected results for:
# tasks=("pcmdi_diags")
git add -A
git commit -m "Update the update scripts"
# All the non-image-checker integration tests passed, so we can skip these steps:
# These scripts update the expected results and re-run the tests:
# ./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# ./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# ./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# This script only updates the expected results
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh # <10 minSkipped because ls /lcrc/group/e3sm/public_html/zppy_test_resources
# Confirmed there are expected results subdirs and image lists for each cfg
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Confirmed the two pcmdi-diags-containing dirs were updated today 9/14.
# All image lists were updated today 9/14.
# Everything else was last updated 9/4.Rerun the tests against the new expected resultscd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
git status
# On branch test_zppy_20260904_run1
# nothing to commit, working tree clean
pytest tests/integration/test_bundles.py
# 2 passed in 1.74s
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
bash
source ~/.bashrc
lcrc_conda
conda activate test-zppy-main-20260828_run1
python -m pip install .
pytest tests/integration/test_images.py
exit # Exit bash shell
exit # Exit compute noteOutput:
The test output reduces correctly to the following, which correctly matches the 9/4 test run minus the
|
Collaborator
Author
|
We probably want to update the expected results for all tasks except failed tasks we explicitly don't have confirmation for yet. That is, we should update the expected results even if the image checker tests pass for a task. For example: ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources/expected_comprehensive_v3currently shows: because |
Collaborator
Author
|
Promoting the 9/28 test run results to be the new expected results baseline. There, we saw that all diffs were to be expected. That means we can do an umbrella/blanket-update of the expected results. Set up branchcd ~/ez/zppy
git status
# On branch issue-869-improve-test-automation
# nothing to commit, working tree clean
git checkout partial-update-expectations
git log --oneline | head -n 4
# 7cfba4b1 Add auto-generated update files
# 1ecb557d Change term from diags to tasks
# 9c6c5185 Support partial updates of weekly/campaign expected test results
# 5232fe30 Rank image check failures by severity (#865)
# Good, has the 3 commits of https://github.com/E3SM-Project/zppy/pull/852/commits
# Titles and hashes match.
lcrc_conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-pr852-20260930
conda activate zppy-pr852-20260930
pre-commit run --all-files
python -m pip install .Copy over the old expected resultsls -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Updated 9/4, 9/14
# To know when each specific task had its expected results updated, check:
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources/expected_comprehensive_v3/
# Sep 4 mpas_analysis
# Sep 4 e3sm_diags
# Aug 14 pcmdi_diags
# Aug 14 global_time_series
# May 20 livvkit
# May 20 ilamb
cp -r /lcrc/group/e3sm/public_html/zppy_test_resources /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20260930 # <30 minutesUpdate the expected results# Let's update the simpler tests' results first:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
git status
# Good, on correct branch: On branch test_zppy_20260928_run6
# Some changed files exist
git add -A
git commit -m "Testing" --no-verify
git log --oneline | head -n 36
# db567d5c Testing
# c7dbac38 Guard apply_conda_install_line's conda install with set +u/-u
# ...
# 0b1dc8b8 Automate image checker, add per-task env descriptions, and auto-generate test report
# 64bffc9f Merge pull request #815 from E3SM-Project/zppy_pcmdi_enhancement
# Recall ~/ez/zppy is on the branch for PR #852
cp ~/ez/zppy/tests/integration/template_update_campaign_expected_files.sh tests/integration/template_update_campaign_expected_files.sh
cp ~/ez/zppy/tests/integration/template_update_weekly_expected_files.sh tests/integration/template_update_weekly_expected_files.sh
git diff # Looks ok
git add -A
git commit -m "Copy changes from PR 852"
# Now, we need to re-render the generated files
emacs tests/integration/utils.py # Confirm ALL cfgs_to_run and tasks_to_run are set to run
python tests/integration/utils.py
git status
# modified: tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# modified: tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Make sure the update script permissions are set up
chmod 755 tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_campaign_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_defaults_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Double check we have a record of the original expected results
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20260930
# Update the task list now that it's available:
emacs tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# What we know we can update the expected results for:
# tasks=()
# We can actually leave it empty so that everything gets updated. 9/28 was a fully passing run (after accounting for expected diffs)
git add -A
git commit -m "Update the update scripts"
# All the non-image-checker integration tests passed,
# so the following steps are only necessary to update the modified-date.
# BUT we have to run each command twice,
# because the tests auto-delete output directories on passing runs, so initially there's nothing to transfer.
# These scripts update the expected results and re-run the tests:
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# 1 passed in 0.61s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 1 passed in 0.59s
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed in 0.68s
# This script only updates the expected results
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh # <10 minMissing the legacy cfgs we deleted, of course. ls /lcrc/group/e3sm/public_html/zppy_test_resources
# Confirmed there are expected results subdirs and image lists for each cfg
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# The only thing not updated today 9/30 arte the image lists for the legacy cfgs we deleted.Rerun the tests against the new expected resultscd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
git status
# On branch test_zppy_20260928_run6
# nothing to commit, working tree clean
pytest tests/integration/test_bundles.py
# 2 passed in 4.16s
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
bash
source ~/.bashrc
lcrc_conda
conda activate test-zppy-issue-869-improve-test-automation-20260928_run6
python -m pip install .
pytest tests/integration/test_images.py
# 1 passed in 397.69s (0:06:37)
exit # Exit bash shell
exit # Exit compute noteGood, everything passes, as expected. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Objectives:
Select one: This pull request is...
Small Change