Reduce CI footprint: skeleton on branch pushes, full suite on PRs, ci-<token> opt-in

.claude/skills/ci-triage/SKILL.md

@@ -0,0 +1,94 @@
+---
+name: ci-triage
+description: Triage failing GitHub Actions runs on pywatershed PRs or branches using the public API - identify failing jobs, read annotations when logs are locked, distinguish stale runs from real failures, and set up local reproduction. Use when the user reports CI failures or invokes /ci-triage, optionally with a PR number or run URL.
+---
+
+# Triaging pywatershed CI failures
+
+You diagnose; the human clicks. You run read-only API queries (curl,
+no auth) and local reproductions; the human re-runs workflows, pushes
+fixes, and merges. Unauthenticated API calls are limited to 60/hour
+per IP — batch queries and cache responses to /tmp files.
+
+## 1. Identify the target
+
+- PR number → head sha: `curl -s "https://api.github.com/repos/DOI-USGS/pywatershed/pulls?state=open"`
+  (confirm which PR the user means — numbers are easy to confuse).
+- All checks for a sha: `curl -s ".../commits/<sha>/check-runs?per_page=100"`.
+  Group by `conclusion`; a full run is ~30+ checks across four workflows
+  (CI, CI example notebooks, Documentation Build, Safety Check — the
+  notebook check names are bare OS names like `ubuntu-latest py3.13`).
+
+## 2. First question: is the failure current?
+
+Before reading any error text, compare timestamps. Check `started_at`
+on the check runs and `created_at`/`run_attempt` on
+`.../actions/runs/<run_id>`. Passing and failing checks on the same sha
+with different start times mean someone re-ran one workflow but not the
+others — **re-runs are per workflow**; re-running "CI" does not refresh
+docs/notebooks/safety.
+
+Correlate failure times with external events before debugging code:
+dependency releases on PyPI
+(`curl -s https://pypi.org/pypi/<pkg>/json` → `releases[v][0]['upload_time']`),
+mf6 nightly builds, flopy releases. A failure that predates the fix for
+its own cause needs a re-run, not a diagnosis. The human has re-run
+permission on run pages ("Re-run all jobs") even where the
+workflow-dispatch button is missing upstream.
+
+## 3. Getting failure details
+
+Job logs (`.../actions/jobs/<job_id>/logs`) return 403 "Must have admin
+rights" without auth. Use the public annotations instead:
+`curl -s ".../check-runs/<check_run_id>/annotations"` (check-run id ==
+job id). Annotations usually contain the terminal error; step-level
+detail beyond that requires the human to open the job page.
+
+## 4. Known failure signatures
+
+- **`micromamba ... exit code 1` + ENOENT `micromamba-shell`**: env
+  creation failed, almost always the pip section of the env file being
+  unresolvable (a version floor not yet published, a new release of a
+  transitive dep). Note which env file the workflow uses: CI and docs
+  use `environment.yml`; notebooks and safety use
+  `environment_w_jupyter.yml`.
+- **flopy rejects valid-looking mf6 input, or mf6 rejects data CI used
+  to accept** (e.g. "Error converting to an integer"): mf6 develop
+  changed a definition and flopy's bundled dfns lag. Both
+  `generate_classes` calls in `ci.yaml` need `--ref develop`, and
+  `autotest/ci_local.sh` must stay in sync with `ci.yaml`.
+- **A test fails only in CI via `--error-for-skips`**: a conditional
+  skip; either `--ignore` it in the broad steps or give it a dedicated
+  step whose `--control_pattern` avoids the skip.
+- **`check_version.yaml` exit codes**: version files must match the
+  release branch name; exit 7 = missing entry in `doc/index.rst`
+  (major releases).
+- **Domain jobs absent from a branch-push run**: not a failure — the
+  skeleton/full split gates them (DEVELOPER.md "CI"); tokens or a draft
+  PR bring them back.
+
+## 5. Local reproduction
+
+Reproduce with the job's exact pytest line (visible in `ci.yaml`),
+from `autotest/`, in the project conda env. Environment parity
+checklist, in order:
+
+1. mf6 binary: delete the stale one so the nightly build is
+   reinstalled to match CI.
+2. `python -m flopy.mf6.utils.generate_classes --ref develop`
+   (run from a modflow6 clone's autotest dir, as CI does).
+3. Test data: regenerate if stale (`python generate_test_data.py
+   -n=auto` in `autotest/`; a version guard fails tests when data
+   predates the current version).
+4. Run the job's pytest command with its `--domain` and
+   `--control_pattern`. Domainless quirk: the conftest exemption is an
+   exact string match on `-m domainless` — pass exactly that markexpr,
+   no `--domain` needed.
+
+## 6. Report format
+
+Lead with the verdict (real failure vs stale run vs infra). When
+staleness is involved, show a short timeline table (run times vs the
+external event). List the re-run links per failed workflow — the human
+clicks them. If code changes are needed, hand over a diagnosis and the
+failing test invocation before drafting fixes.

.github/workflows/ci.yaml

@@ -1,5 +1,14 @@
 name: CI
 on:
+  # Pushes to any branch (here or on forks) run a skeleton: installs,
+  # linting, and domainless tests. The full suite (the domain test jobs)
+  # runs for pull requests, pushes to develop/main, and workflow_dispatch.
+  # To ALSO run domain jobs on branch pushes, put a ci-<token> anywhere in
+  # the branch name or in the pushed (head) commit message. Tokens:
+  # ci-all, ci-sagehen, ci-hru1, ci-drb, ci-ucb, ci-fgr. For example, a
+  # branch named my_feature_ci-fgr runs both fgr jobs on every push, with
+  # no workflow edits to clean up before merging; a commit-message token
+  # applies to its push only. Details in DEVELOPER.md under "CI".
   push:
     branches:
       - "*"
@@ -17,6 +26,12 @@ on:
         required: false
         default: false
 
+# Cancel in-flight runs superseded by a newer push on the same ref, but
+# let every run on the mainline branches finish
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: ${{ github.ref_name != 'develop' && github.ref_name != 'main' }}
+
 jobs:
   pws_install:
     name: standard installation
@@ -233,6 +248,16 @@ jobs:
 
   test_sagehen_5yr:
     name: sagehen_5yr ${{ matrix.os }} py${{ matrix.python-version }}
+    # domain jobs skip plain branch pushes; see the ci-<token> note
+    # in the trigger comment at the top of this file
+    if: >-
+      github.event_name != 'push' ||
+      github.ref_name == 'develop' ||
+      github.ref_name == 'main' ||
+      contains(github.ref_name, 'ci-all') ||
+      contains(github.event.head_commit.message, 'ci-all') ||
+      contains(github.ref_name, 'ci-sagehen') ||
+      contains(github.event.head_commit.message, 'ci-sagehen')
     runs-on: ${{ matrix.os }}
     defaults:
       run:
@@ -348,6 +373,16 @@ jobs:
 
   test_hru_1:
     name: hru_1 ${{ matrix.os }} py${{ matrix.python-version }}
+    # domain jobs skip plain branch pushes; see the ci-<token> note
+    # in the trigger comment at the top of this file
+    if: >-
+      github.event_name != 'push' ||
+      github.ref_name == 'develop' ||
+      github.ref_name == 'main' ||
+      contains(github.ref_name, 'ci-all') ||
+      contains(github.event.head_commit.message, 'ci-all') ||
+      contains(github.ref_name, 'ci-hru1') ||
+      contains(github.event.head_commit.message, 'ci-hru1')
     runs-on: ${{ matrix.os }}
     defaults:
       run:
@@ -476,6 +511,16 @@ jobs:
 
   test_drb_2yr:
     name: drb_2yr ${{ matrix.os }} py${{ matrix.python-version }}
+    # domain jobs skip plain branch pushes; see the ci-<token> note
+    # in the trigger comment at the top of this file
+    if: >-
+      github.event_name != 'push' ||
+      github.ref_name == 'develop' ||
+      github.ref_name == 'main' ||
+      contains(github.ref_name, 'ci-all') ||
+      contains(github.event.head_commit.message, 'ci-all') ||
+      contains(github.ref_name, 'ci-drb') ||
+      contains(github.event.head_commit.message, 'ci-drb')
     runs-on: ${{ matrix.os }}
     defaults:
       run:
@@ -712,6 +757,16 @@ jobs:
 
   test_ucb_2yr_nhm:
     name: ucb_2yr_nhm ${{ matrix.os }} py${{ matrix.python-version }}
+    # domain jobs skip plain branch pushes; see the ci-<token> note
+    # in the trigger comment at the top of this file
+    if: >-
+      github.event_name != 'push' ||
+      github.ref_name == 'develop' ||
+      github.ref_name == 'main' ||
+      contains(github.ref_name, 'ci-all') ||
+      contains(github.event.head_commit.message, 'ci-all') ||
+      contains(github.ref_name, 'ci-ucb') ||
+      contains(github.event.head_commit.message, 'ci-ucb')
     runs-on: ${{ matrix.os }}
     defaults:
       run:
@@ -834,6 +889,16 @@ jobs:
 
   test_ucb_2yr_slow_tests:
     name: ucb_2yr_slow_tests ${{ matrix.os }} py${{ matrix.python-version }}
+    # domain jobs skip plain branch pushes; see the ci-<token> note
+    # in the trigger comment at the top of this file
+    if: >-
+      github.event_name != 'push' ||
+      github.ref_name == 'develop' ||
+      github.ref_name == 'main' ||
+      contains(github.ref_name, 'ci-all') ||
+      contains(github.event.head_commit.message, 'ci-all') ||
+      contains(github.ref_name, 'ci-ucb') ||
+      contains(github.event.head_commit.message, 'ci-ucb')
     runs-on: ${{ matrix.os }}
     defaults:
       run:
@@ -944,6 +1009,16 @@ jobs:
 
   test_fgr_ag_2yr_subset1:
     name: fgr_ag_2yr_subset1 ${{ matrix.os }} py${{ matrix.python-version }}
+    # domain jobs skip plain branch pushes; see the ci-<token> note
+    # in the trigger comment at the top of this file
+    if: >-
+      github.event_name != 'push' ||
+      github.ref_name == 'develop' ||
+      github.ref_name == 'main' ||
+      contains(github.ref_name, 'ci-all') ||
+      contains(github.event.head_commit.message, 'ci-all') ||
+      contains(github.ref_name, 'ci-fgr') ||
+      contains(github.event.head_commit.message, 'ci-fgr')
     runs-on: ${{ matrix.os }}
     defaults:
       run:
@@ -1132,6 +1207,16 @@ jobs:
 
   test_fgr_ag_2yr_subset2:
     name: fgr_ag_2yr_subset2 ${{ matrix.os }} py${{ matrix.python-version }}
+    # domain jobs skip plain branch pushes; see the ci-<token> note
+    # in the trigger comment at the top of this file
+    if: >-
+      github.event_name != 'push' ||
+      github.ref_name == 'develop' ||
+      github.ref_name == 'main' ||
+      contains(github.ref_name, 'ci-all') ||
+      contains(github.event.head_commit.message, 'ci-all') ||
+      contains(github.ref_name, 'ci-fgr') ||
+      contains(github.event.head_commit.message, 'ci-fgr')
     runs-on: ${{ matrix.os }}
     defaults:
       run:

.github/workflows/ci_docs.yaml

@@ -5,6 +5,12 @@ on:
   pull_request:
   workflow_dispatch:
 
+# Cancel in-flight runs superseded by a newer push on the same ref, but
+# let every run on the mainline branches finish
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: ${{ github.ref_name != 'develop' && github.ref_name != 'main' }}
+
 jobs:
   build_docs:
     name: Build Documentation

.github/workflows/ci_examples.yaml

@@ -1,6 +1,9 @@
 name: CI example notebooks
 
 on:
+  # Pushes to any branch (here or on forks) run the notebooks on ubuntu
+  # only; all platforms run for pull requests and develop/main pushes.
+  # See DEVELOPER.md under "CI" for the skeleton/full CI strategy.
   push:
     branches:
       - "*"
@@ -10,6 +13,12 @@ on:
       - "*"
       - "!v[0-9]+.[0-9]+.[0-9]+*"
 
+# Cancel in-flight runs superseded by a newer push on the same ref, but
+# let every run on the mainline branches finish
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: ${{ github.ref_name != 'develop' && github.ref_name != 'main' }}
+
 jobs:
   test:
     name: ${{ matrix.os}} py${{ matrix.python-version }}
@@ -27,7 +36,9 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: ["ubuntu-latest", "macos-latest", "windows-latest"]
+        # skeleton/full split: all platforms for pull requests and
+        # develop/main pushes; ubuntu only for other pushes
+        os: ${{ (github.event_name != 'push' || github.ref_name == 'develop' || github.ref_name == 'main') && fromJSON('["ubuntu-latest", "macos-latest", "windows-latest"]') || fromJSON('["ubuntu-latest"]') }}
         python-version: ["3.13"]
 
     steps:

CLAUDE.md

@@ -29,6 +29,12 @@ reimplementation of PRMS process representations (see README.md).
 - CI uses `--error-for-skips`: a test that skips conditionally must be
   either `--ignore`d in the broad CI steps or run in a dedicated step
   whose `--control_pattern` avoids the skip.
+- Every domain test job in `ci.yaml` carries an `if:` gate (the
+  skeleton/full split). A new domain job must copy the gate from an
+  existing domain job, with an appropriate `ci-<token>` — an ungated job
+  runs on every push to every branch. Watch for this especially when
+  merging develop into branches that predate the gates. See DEVELOPER.md
+  under "CI" for the strategy and token semantics.
 
 ## Conventions
 
@@ -57,6 +63,9 @@ skills:
 - `/release` — assists a pywatershed release step by step, following
   `.github/RELEASE.md`: drafts the commands for the human to run,
   runs the checks, and verifies each stage (PyPI, tag, release assets).
+- `/ci-triage` — triages failing GitHub Actions runs on a PR or branch
+  via the public API: stale-run vs real-failure verdict, annotations
+  when logs are locked, known failure signatures, local repro steps.
 
 Claude: in your first reply of a session, briefly show the user this
 bullet list of available skills (many users don't know skills exist).

DEVELOPER.md

@@ -138,6 +138,63 @@ The automated practices of installing, linting, and testing described below are
 all formally encoded in `.github/workflows/ci.yaml` and
 `.github/workflows/ci_examples.yaml` files.
 
+CI runs at two levels. Pushes to any branch — in this repository or on a fork —
+run a skeleton: installs, linting, domainless tests, the documentation build,
+and the example notebooks on ubuntu only. The full suite (the domain test jobs
+on all three platforms) runs for pull requests (including drafts), pushes to
+`develop`/`main`, and `workflow_dispatch`.
+
+To also run domain test jobs on branch pushes, put a `ci-<token>` anywhere in
+the branch name or in the pushed (head) commit message. The tokens are
+`ci-all`, `ci-sagehen`, `ci-hru1`, `ci-drb`, `ci-ucb`, and `ci-fgr`.
+
+The two variants complement each other:
+
+- **Branch name — sticky.** A branch named `my_feature_ci-fgr` runs both fgr
+  domain jobs on every push, with nothing to clean up before merging since
+  the opt-in lives in the branch name.
+- **Commit message — one shot.** The gate is evaluated fresh on each push
+  with no memory of earlier pushes: a push whose head commit message contains
+  `ci-drb` runs the drb job for that push only, even on a branch previously
+  pushed without any token, and later pushes without a token drop back to the
+  skeleton. Only the *head* (most recent) commit of a push is checked — when
+  pushing several commits at once, the token must be in the last one. To
+  trigger a domain run on the current state without changing any code, push
+  an empty commit:
+
+  ```shell
+  git commit --allow-empty -m "trigger ci-fgr"
+  git push <fork-remote> my_branch
+  ```
+
+Alternatively, open a draft PR or use the workflow dispatch button (on your
+fork) to get the full suite on any branch.
+
+Token matching is a plain substring test — there is no parsing of token
+lists, separators, or suffixes. Consequences:
+
+- Tokens can only add jobs, never subtract them: appending to a token
+  (`ci-sagehen_gridded`) does not narrow the selection, it matches the
+  `ci-sagehen` token and triggers every job in that family.
+- A commit message that merely *mentions* a token triggers it — easy to do
+  accidentally in a commit message about the CI configuration itself. Write
+  `ci-<token>` (as in this file) rather than a literal token when referring
+  to the mechanism.
+- Tokens are matched anywhere in the branch name, so avoid naming a branch
+  with a literal token unless the sticky behavior is wanted.
+
+Notes for maintaining the gates in `ci.yaml`:
+
+- New domain test jobs must copy the `if:` gate from an existing domain job
+  (with an appropriate token) — an ungated job runs on every push to every
+  branch, silently defeating the skeleton/full split.
+- When adding finer-grained tokens, no token may be a substring of another:
+  a bare `ci-sagehen` cannot coexist with a distinct `ci-sagehen-gridded`
+  switch, because writing the longer token always matches the shorter one.
+  To split a family, replace the family token with mutually non-substring
+  tokens, e.g. `ci-sagehen-5yr`, `ci-sagehen-gridded`, and `ci-sagehen-all`
+  for the whole family.
+
 
 ## Testing
 Once the dependencies are available, we want to verify the software by running

doc/whats-new.rst

@@ -27,6 +27,16 @@ Bug fixes
 
 Internal changes
 ~~~~~~~~~~~~~~~~
+- Reduce CI footprint with a skeleton/full split: pushes to any branch (in
+  this repository or on forks) run a skeleton — installs, linting, domainless
+  tests, the docs build, and example notebooks on ubuntu only — while the
+  full suite (domain test jobs, all platforms) runs for pull requests
+  (including drafts), pushes to ``develop``/``main``, and
+  ``workflow_dispatch``. Domain jobs can be opted in on branch pushes with a
+  ``ci-<token>`` (e.g. ``ci-fgr``, ``ci-all``) in the branch name or head
+  commit message — see DEVELOPER.md. ``concurrency`` groups cancel in-flight
+  runs superseded by a newer push on the same non-mainline ref.
+  (:pull:`408`) By `James McCreight <https://github.com/jmccreight>`_.
 - Require pyPRMS >=0.10.0 and remove the temporary ``packaging <26.3`` pin it
   supersedes (pyPRMS 0.9.10 crashed on import of metadata with packaging >=26.3).
   Also remove calls to pyPRMS methods deprecated in 0.10.0: