Improve test automation

[forsyth2]

Summary

Improve test automation by updating the test automation script tests/main_branch_testing/run_integration_test.bash to:

1. Write the Markdown report.
2. When producing results, include an environment description for each task. For example, global_time_series/env_description.txt would include the commit hash of zppy-interfaces it was using, and a printout of the conda environment package versions.
3. Run the tests of the tests (e.g., tests/images/test_image_checker.py, tests/images/test_image_severity.py)
4. Auto-launch the image checker test rather than having that be a manual step.

Additionally, updates docs/source/dev_guide/tests/automated_test.rst to explain how to auto-launch this test script via a weekly cron job.

Issue resolution:

• Closes [Feature]: Automate execution of complete run test #869
• Extension of Add main branch testing automation #774

Select one: This pull request is...

• a bug fix: increment the patch version
• a small improvement: increment the minor version
• a new feature: increment the minor version
• an incompatible (non-backwards compatible) API change: increment the major version

Big Change

• To merge, I will use "Create a merge commit". That is, this change is large enough to require multiple units of work (i.e., it should be multiple commits).

1. Does this do what we want it to do?

Required:

• Product Management: I have confirmed with the stakeholders that the objectives above are correct and complete.
• Testing: I have added or modified at least one "min-case" configuration file to test this change. Every objective above is represented in at least one cfg.
• Testing: I have considered likely and/or severe edge cases and have included them in testing.

If applicable:

• Testing: this pull request introduces an important feature or bug fix that we must test often. I have updated the weekly-test configuration files, not just a "min-case" one.
• Testing: this pull request adds at least one new possible parameter to the cfg. I have tested using this parameter with and without any other parameter that may interact with it.

2. Are the implementation details accurate & efficient?

Required:

• Logic: I have visually inspected the entire pull request myself.
• Logic: I have left GitHub comments highlighting important pieces of code logic. I have had these code blocks reviewed by at least one other team member.

If applicable:

• Dependencies: This pull request introduces a new dependency. I have discussed this requirement with at least one other team member. The dependency is noted in zppy/conda, not just an import statement.

3. Is this well documented?

Required:

• Documentation: by looking at the docs, a new user could easily understand the functionality introduced by this pull request.

4. Is this code clean?

Required:

• Readability: The code is as simple as possible and well-commented, such that a new team member could understand what's happening.
• Pre-commit checks: All the pre-commits checks have passed.

If applicable:

• Software architecture: I have discussed relevant trade-offs in design decisions with at least one other team member. It is unlikely that this pull request will increase tech debt.

[Copilot]

🟡 Changes recommended

Critical and moderate issues remain in image-test handling, reporting accuracy, provenance, and automation reliability.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This pull request expands integration-test automation with environment provenance, Markdown reporting, automatic image checking, and weekly cron documentation.

Changes:

• Adds report configuration and per-task environment capture.
• Automates image-checker execution and report generation.
• Documents weekly cron scheduling.

File summaries

File	Summary
tests/main_branch_testing/zppy_test.cfg	Adds report configuration options.
tests/main_branch_testing/run_integration_test.bash	Implements environment capture, reporting, and automated image checking.
docs/source/dev_guide/tests/automated_test.rst	Documents reports and cron-based execution.

Review details

Suppressed comments (8)

docs/source/dev_guide/tests/automated_test.rst:91

• The new workflow leaves tests/main_branch_testing/README.md inconsistent with this documentation: its Phase 3 section still says the script only prints instructions to run test_images.py manually, while this change launches it automatically. Please update that guide in the same change so users do not get contradictory instructions.

* Step 8: Run all Python tests, including the image checker (``pytest tests/integration/test_images.py``), which is now launched automatically on a compute node -- no manual step required.

docs/source/dev_guide/tests/automated_test.rst:337

• The weekly driver copies the runner from $HOME/ez/zppy, but the runner itself leaves that checkout on test_zppy_<TAG> during Phase 1. On the next cron run this cp can therefore pick up the previous test branch's older runner instead of the latest main version, so automation updates to the script are not reliably adopted. Use a separate stable checkout for the driver source or explicitly update that source checkout before copying.

       cp "$HOME/ez/zppy/tests/main_branch_testing/run_integration_test.bash" .

tests/main_branch_testing/run_integration_test.bash:1208

• The || true converts any git log error (missing upstream/<branch>, invalid date, or repository failure) into an empty result, which is then rendered as None. The report can therefore claim that there were no changes when it actually could not inspect the repository; emit an unavailable/error row instead of conflating the two cases.

    local commits
    commits=$(git -C "$repo_dir" log "upstream/${branch}" --since="${EXPECTED_RESULTS_UPDATED_DATE}" --oneline 2>/dev/null || true)

    if [[ -z "$commits" ]]; then
        report_append "| [${label}](${repo_url}/commits/${branch}) | None |"

tests/main_branch_testing/run_integration_test.bash:487

• Every CFGS_ARRAY entry already starts with weekly_ (for example, weekly_comprehensive_v3), while the generated output root is zppy_weekly_comprehensive_v3_www. This creates zppy_weekly_weekly_comprehensive_v3_www and puts env_description.txt in a new, unused tree instead of alongside the diagnostics.

            target_dir="${OUTPUT_WORKSPACE}/zppy_weekly_${cfg}_www/${UNIQUE_ID}/${case}/${task}"

tests/main_branch_testing/run_integration_test.bash:533

• This waits on every job owned by USER, not the submitted image-checker job. An unrelated long-running job can consume the two-hour timeout, and an early squeue snapshot can miss a newly submitted image job. Poll the submitted job's state directly and handle its terminal result.

    wait_for_slurm_jobs 120 7200

tests/main_branch_testing/run_integration_test.bash:1175

• The generated table has eight columns, but this separator row has only four cells. Markdown renderers will treat the failing-only table as malformed; emit eight separators to match the header and data rows.

        print("| --- | --- | --- | --- |")

tests/main_branch_testing/run_integration_test.bash:1171

• by_task preserves the first task occurrence from the cfg-major summary order, which is not sorted by task (the configured task order is e3sm_diags, mpas_analysis, global_time_series, ...). The section is explicitly titled "sorted by task", so iterate over sorted(by_task.items()) or otherwise sort the task keys before rendering.

    for task, flines in by_task.items():

tests/main_branch_testing/run_integration_test.bash:129

• The manual Perlmutter allocation above includes a one-hour walltime, but this batch directive omits --time. The automatically submitted checker therefore uses the site's default batch limit instead of the documented limit and may be terminated before the image test completes; keep the batch request consistent with the manual allocation.

        SBATCH_DIRECTIVES=$'#SBATCH --qos=interactive\n#SBATCH --time=01:00:00\n#SBATCH --constraint=cpu\n#SBATCH --account=e3sm'

• Files reviewed: 3/3 changed files
• Comments generated: 5
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Critical and moderate issues remain in image-test handling, reporting accuracy, provenance, and automation reliability....

Addressed in 2d55a48.

[forsyth2]

Did a high-level read-through.

[forsyth2]

Remaining action items:

• On next test run for weekly testing, use this branch and confirm it behaves as expected.
• If so, try running via a cron job. (The branch may need to be merged to main first).

[Copilot]

🟡 Changes recommended

Unresolved critical and moderate issues affect environment capture, portability, report accuracy, and automated result status.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (7)

tests/integration/image_summary_report.py:53

• The real image-check summary rows are keyed like comprehensive_v2_e3sm_diags (image_checker.py builds <cfg>_<task>), so this split treats the entire string as one token and never matches e3sm_diags or global_time_series. As a result, every failing row is grouped under other in the generated report; match the task suffix as well.

    tokens = {token for token in re.split(r"[^A-Za-z0-9_]+", name) if token}
    for task in tasks:
        if task in tokens:

tests/main_branch_testing/run_integration_test.bash:607

• run_image_checker is called as the condition of if ! run_image_checker, so Bash's errexit is suppressed inside the function. If sbatch fails, this assignment leaves an empty job ID and the function proceeds to poll that ID for up to two hours instead of reporting submission failure. Check the submission status explicitly before waiting.

    IMAGE_CHECKER_JOB_ID=$(sbatch --parsable "$IMAGE_CHECKER_SBATCH")
    log "Image checker job submitted: ${IMAGE_CHECKER_JOB_ID}"

tests/main_branch_testing/run_integration_test.bash:600

• These lines bypass the machine-specific initialization used elsewhere in this script (source ~/.bashrc plus CONDA_ACTIVATION_CMD). On Compy/Perlmutter, or any setup where CONDA_PROFILE is not the active installation, the batch job can fail at conda activate, so the automatic image check is not portable across the supported machines. Initialize the batch shell with the same sequence before activating ZPPY_ENV.

source ${CONDA_PROFILE}
conda activate ${ZPPY_ENV}

tests/main_branch_testing/run_integration_test.bash:631

• test_images.py writes early_test_images_summary.md before re-raising a worker exception (see tests/integration/test_images.py:294-305), but this path only checks the final summary filename. When a worker fails, the generated report loses the partial table and only says the summary is missing; please fall back to the early summary (and clear any stale early file before submission) while retaining the failed status.

    if [[ -f "${ZPPY_DIR}/test_images_summary.md" ]]; then
        cp "${ZPPY_DIR}/test_images_summary.md" "${SCRIPT_RUN_DIR}/test_images_summary_${TAG}.md"
        log_success "Copied test_images_summary.md -> ${SCRIPT_RUN_DIR}/test_images_summary_${TAG}.md"
    else
        log_warning "test_images_summary.md not found in ${ZPPY_DIR}"

tests/main_branch_testing/run_integration_test.bash:1175

• These bullets are unconditional and do not record any result: the integration invocations above discard exit codes with || log_warning, and the unit/status outcomes are not stored here. A passing and failing run therefore produce the same automated-results section, which makes the generated report insufficient for unattended review. Capture the outcomes and render the actual pass/fail status (or failure details) in this section.

    report_append "* zppy-interfaces unit tests: see script log for \`Running zppy-interfaces unit tests...\` / \`zppy-interfaces unit tests passed\`"
    report_append "* zppy unit tests: see script log for \`Running zppy unit tests...\` / \`zppy unit tests passed\`"
    report_append "* Image-checker/report unit tests (tests of the tests): \`tests/images/test_image_checker.py\`, \`tests/images/test_image_severity.py\`, \`tests/test_image_summary_report.py\`"
    report_append "* Output directory status files: checked automatically; see \`Checking all status files...\` in the script log"
    report_append "* Integration tests run: \`test_last_year.py\`, \`test_bash_generation.py\`, \`test_campaign.py\`, \`test_defaults.py\`, \`test_bundles.py\`"

tests/main_branch_testing/run_integration_test.bash:499

• The generated cfgs place diagnostics under the machine's user_www root (for example, /lcrc/group/e3sm/public_html/diagnostic_output/$USER/...), while OUTPUT_WORKSPACE is the separate job-output root. This creates env_description.txt under a new .../$USER/zppy_*_www tree instead of alongside the diagnostics that test_images.py reads, so the promised per-task files are not visible with the results. Derive this target from the same web root used by the generated cfgs.

            target_dir="${OUTPUT_WORKSPACE}/zppy_${cfg#test_}_www/${UNIQUE_ID}/${case}/${task}"
            mkdir -p "$target_dir" 2>/dev/null || {
                log_warning "Could not create ${target_dir}; skipping env description for ${cfg}/${task}"
                continue
            }
            cp "$desc_file" "${target_dir}/env_description.txt"

tests/main_branch_testing/run_integration_test.bash:600

• When Phase 3 is resumed directly with ZPPY_EXISTING_ENV set, the override at lines 820-822 is never executed, so ZPPY_ENV still has the auto-generated name. The new batch job then tries to activate a nonexistent environment instead of the configured existing one; use the existing-env fallback when generating this command.

conda activate ${ZPPY_ENV}

• Files reviewed: 6/6 changed files
• Comments generated: 3
• Review effort level: Lite

[Copilot]

Updated the docs in 633ed2b so this section now matches the script output more precisely.

[Copilot]

🟡 Changes recommended

Unresolved critical and moderate issues affect environment capture, portability, report accuracy, and automated result status....

Addressed in f346516.

[forsyth2]

I have used this branch to run yesterday's weekly test (9/15 test run. I've identified several edits to make to the test script, which I outlined in that post.

[forsyth2]

The second-to-last commit, fb2babb, addresses my comments here.

The last commit, 5c20e89 addresses my comments here.