Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 31 additions & 12 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,13 @@ jobs:
pip -V
pip list

- name: Compile PRMS 5.2.1
run: |
which gfortran
gfortran --version
which make
python -c "import pywatershed as pws; pws.utils.compile_prms('5.2.1')"

- name: sagehen_5yr_no_cascades - generate and manage test data domain, run PRMS and convert csv output to NetCDF
working-directory: autotest
run: |
Expand Down Expand Up @@ -424,6 +431,13 @@ jobs:
pip -V
pip list

- name: Compile PRMS 5.2.1
run: |
which gfortran
gfortran --version
which make
python -c "import pywatershed as pws; pws.utils.compile_prms('5.2.1')"

- name: hru_1 all configs - generate and manage test data, run PRMS and convert csv output to NetCDF
working-directory: autotest
run: |
Expand Down Expand Up @@ -604,22 +618,13 @@ jobs:
run: |
python pywatershed/utils/gis_files.py

- name: Compile PRMS 5.2.1.1
working-directory: prms_src/prms5.2.1.1
- name: Compile PRMS 5.2.1 and 5.2.1.1
run: |
which gfortran
gfortran --version
which make
MAKE_PATH=$(which make)
$MAKE_PATH clean MAKE="$MAKE_PATH"
$MAKE_PATH DBL_PREC=true FC=gfortran CC=gcc MAKE="$MAKE_PATH"
if [ "$RUNNER_OS" == "Windows" ]; then
cp bin/prms ../../bin/prms_5.2.1.1_gfort_win_dbl_prec.exe
elif [ "$RUNNER_OS" == "macOS" ]; then
cp bin/prms ../../bin/prms_5.2.1.1_gfortran_apple_silicon_dbl_prec
elif [ "$RUNNER_OS" == "Linux" ]; then
cp bin/prms ../../bin/prms_5.2.1.1_gfort_linux_dbl_prec
fi
python -c "import pywatershed as pws; pws.utils.compile_prms('5.2.1')"
python -c "import pywatershed as pws; pws.utils.compile_prms('5.2.1.1')"

- name: drb_2yr all domains - generate and manage test data
working-directory: autotest
Expand Down Expand Up @@ -824,6 +829,13 @@ jobs:
pip -V
pip list

- name: Compile PRMS 5.2.1
run: |
which gfortran
gfortran --version
which make
python -c "import pywatershed as pws; pws.utils.compile_prms('5.2.1')"

- name: ucb_2yr_nhm - generate and manage test data
working-directory: autotest
run: |
Expand Down Expand Up @@ -940,6 +952,13 @@ jobs:
pip -V
pip list

- name: Compile PRMS 5.2.1
run: |
which gfortran
gfortran --version
which make
python -c "import pywatershed as pws; pws.utils.compile_prms('5.2.1')"

- name: ucb_2yr_slow_tests - generate and manage test data
working-directory: autotest
run: |
Expand Down
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,11 @@ bin/libmf6.dylib
bin/mf5to6
bin/mf6
bin/zbud6
# PRMS binaries are compiled from prms_src/ on demand rather than checked
# in; only the GSFLOW binaries (whose source is not in this repo) are
# tracked in bin/.
bin/prms_*
bin/.compile_prms_*.lock

# prms output files
*.out
Expand Down Expand Up @@ -116,3 +121,5 @@ starfit_minimal/

# claude code: personal settings stay untracked
.claude/settings.local.json

.codegraph/
70 changes: 36 additions & 34 deletions DEVELOPER.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,19 +40,23 @@ is a good source of information.

### Compilers

C and Fortran compilers are required. We are currently using gnu (gcc,
gfortran) 11 and 12 as well as intel (icc, ifort) 2021 on Windows, Linux, and
MacOS (including Apple Silicon). Both of these are freely obtainable but the
installation process varies widely. We are looking for a conda-based approach
to obtaining compilers, but currently do not have a solution. Compilers are
needed for two applications:

1. Compiling and running C/Fortran PRMS code to generate testing/verification
data 2. Compiling (installing) and running fortran backends/kernels for some
hydrological process representations in pywatershed

On Apple Silicon, the PRMS source code is only currently known to compile with
intel while the fortran kernels in pywatershed only compile with gnu.
C and Fortran compilers are required. We use gnu (gcc, gfortran)
exclusively, on Windows, Linux, and Apple Silicon MacOS. The mamba/conda
environment described below supplies them (`gfortran>=15.2.0` in
`environment.yml`), so no separate compiler installation is needed.
Compilers are needed for compiling and running C/Fortran PRMS code to generate testing/verification data.

The PRMS binaries used to generate test data are not kept in the
repository. They are compiled from `prms_src/` the first time they are
needed, by `pywatershed.utils.compile_prms()`, and written to `bin/`
(which is gitignored for these). `autotest/ci_local.sh` and `ci.yaml` call
the same function, so there is one build path everywhere.

Intel compilers (icc, ifort) are no longer used or supported, and Intel
MacOS is no longer detected. The one remaining intel-built artifact is
`bin/gsflow_2.4.0_ifort_apple_silicon_dbl_prec`: GSFLOW source is not part
of this repository, so that binary is checked in rather than compiled. It
is an x86_64 build and therefore needs Rosetta 2 on Apple Silicon.

### Python

Expand Down Expand Up @@ -84,7 +88,7 @@ To install all dependencies with `pip`:
pip install ".[all]"
```

Several dependency groups are defined in `pyproject.toml` and can be selected
Several dependency groups are defined in `pyproject.toml` and can be selected
instead of `all` for a more lightweight environment:

- `lint`
Expand All @@ -100,8 +104,7 @@ in the root, `source venv/bin/activate`), `pywatershed` can be installed in
mode](https://setuptools.pypa.io/en/latest/userguide/development_mode.html)
with:

``` pip install -e . ```

`pip install -e . `

#### F2PY

Expand All @@ -111,7 +114,7 @@ numpy](https://numpy.org/doc/stable/f2py/index.html). This repository is
configured NOT to compile on install by default. Currently, we have not
established this compilation procedure for Windows. On Linux and MacOS,
compilation of fortran kernels on package installation is achieved by setting
several environent variables before installing the `pywatershed` module. For
several environent variables before installing the `pywatershed` module. For
instance, from the project root:

```
Expand All @@ -124,25 +127,25 @@ pip install -e .

Note that an editable (`-e` above) is required to compile the fotran code.


## Branching model

This project uses the [git
flow](https://nvie.com/posts/a-successful-git-branching-model/): development
occurs on the `develop` branch, while `main` is reserved for the state of the
latest release. Development PRs are typically squashed to `develop`, to avoid
merge commits. At release time, release branches are merged to `main`, and then
`main` is merged back into `develop`.


## Maintenance ledger

Maintenance todos that are blocked on external events (dependency
releases, cross-repo work like conda-forge feedstocks) are tracked in
[`MAINTENANCE.md`](MAINTENANCE.md) at the repo root, with mechanically
checkable unblock conditions. The `/maintenance` Claude skill checks
them live and reports what is actionable.


## CI

The automated practices of installing, linting, and testing described below are
all formally encoded in `.github/workflows/ci.yaml` and
`.github/workflows/ci_examples.yaml` files.
Expand All @@ -166,7 +169,7 @@ The two variants complement each other:
with no memory of earlier pushes: a push whose head commit message contains
`ci-drb` runs the drb job for that push only, even on a branch previously
pushed without any token, and later pushes without a token drop back to the
skeleton. Only the *head* (most recent) commit of a push is checked — when
skeleton. Only the _head_ (most recent) commit of a push is checked — when
pushing several commits at once, the token must be in the last one. To
trigger a domain run on the current state without changing any code, push
an empty commit:
Expand All @@ -185,7 +188,7 @@ lists, separators, or suffixes. Consequences:
- Tokens can only add jobs, never subtract them: appending to a token
(`ci-sagehen_gridded`) does not narrow the selection, it matches the
`ci-sagehen` token and triggers every job in that family.
- A commit message that merely *mentions* a token triggers it — easy to do
- A commit message that merely _mentions_ a token triggers it — easy to do
accidentally in a commit message about the CI configuration itself. Write
`ci-<token>` (as in this file) rather than a literal token when referring
to the mechanism.
Expand All @@ -204,8 +207,8 @@ Notes for maintaining the gates in `ci.yaml`:
tokens, e.g. `ci-sagehen-5yr`, `ci-sagehen-gridded`, and `ci-sagehen-all`
for the whole family.


## Testing

Once the dependencies are available, we want to verify the software by running
its test suite. However, we first need to generate the test data. This consists
of running binaries (PRMS) and then converting the output to netcdf files used
Expand All @@ -231,22 +234,23 @@ machine.

For more details on the autotests, see [`autotest/README.md`](autotest/README.md).


## Linting

Automated linting procedures are performed in CI and enforced, these are

```shell
ruff check .
ruff format .
```

And you'll need to run these locally to pass CI checks.


## Committing Jupyter Notebooks

All outputs are required to be stripped from jupyter notebooks prior to
committing. To facilitate this we have
[pre-commit hooks](https://pre-commit.com/) which will strip
outputs and metadata from jupyter notebooks. When a `git commit` is attempted,
outputs and metadata from jupyter notebooks. When a `git commit` is attempted,
the hook will check all staged `*.ipynb` files. If the file is modified after
running the hook (which runs
[nbstripout](https://github.com/kynan/nbstripout)), then the
Expand All @@ -260,6 +264,7 @@ time to keep very large diffs out of the repository history. If you are using
The maximal amount of metadata can be stripped from Jupyter notebooks by following the example configuration found in the [nbstripout section on stripping metadata](https://github.com/kynan/nbstripout#stripping-metadata).

## pre-commit hooks

Pre-commit hooks apply actionas at commit-time. These are available when
`pre-commit` is installed in the environment, as in the `environment.yml`
supplied. To install yourself
Expand All @@ -271,24 +276,21 @@ pre-commit install
As specified in `.pre-commit-config.yaml`, we adopt the following pre-commit
hooks

* [nbstripout](https://github.com/kynan/nbstripout):
- [nbstripout](https://github.com/kynan/nbstripout):
strip outputs from jupyter notebooks
* [blackdoc](https://github.com/keewis/blackdoc):
- [blackdoc](https://github.com/keewis/blackdoc):
apply black within documentation
* [doctoc](https://github.com/thlorenz/doctoc): auto generate tables of
- [doctoc](https://github.com/thlorenz/doctoc): auto generate tables of
contents in markdown docs


## Documentation

[Google-style docstrings](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings)
are used for documenting source code. (Though numpy style is supposedly handled
as well by Napolean, preference is for google-style.)



## Miscellaneous


### Locating the root

Python scripts often need to reference files elsewhere in the project. To allow
Expand All @@ -299,4 +301,4 @@ for file access, rather than using relative paths (e.g., `../some/path`).
For a script in a subdirectory of the root, for instance, the conventional
approach would be:

```Python project_root_path = Path(__file__).parent.parent ```
`Python project_root_path = Path(__file__).parent.parent `
43 changes: 32 additions & 11 deletions MAINTENANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
- [Maintenance ledger](#maintenance-ledger)
- [Open](#open)
- [3.0.1 hotfix release (flopy pin unwind + pyPRMS floor)](#301-hotfix-release-flopy-pin-unwind--pyprms-floor)
- [conda-forge feedstock: pyprms floor replaces packaging pin](#conda-forge-feedstock-pyprms-floor-replaces-packaging-pin)
- [pyPRMS upstream: close PR #64, port its regression test](#pyprms-upstream-close-pr-64-port-its-regression-test)
- [Reconcile check_version.yaml with release_preflight.sh](#reconcile-check_versionyaml-with-release_preflightsh)
- [Retire the GSFLOW ifort binary](#retire-the-gsflow-ifort-binary)
- [Deliver CI-usage findings to org admins](#deliver-ci-usage-findings-to-org-admins)
- [Done](#done)

Expand Down Expand Up @@ -51,16 +51,6 @@ check it), **Action** (what to do once unblocked), and optional
released flopy until then; flopy imports lazily so this is not an
import-time problem.

### conda-forge feedstock: pyprms floor replaces packaging pin

- **Blocked on:** feedstock PR #14 (pyprms >=0.10.0 floor, build 1)
MERGED 2026-08-19; awaiting the main-branch CI upload. Check:
`https://api.anaconda.org/package/conda-forge/pywatershed` shows a
3.0.0 build with build number 1.
- **Action:** verify the new build's run deps carry `pyprms >=0.10.0`
and no `packaging <26.3`, then move to Done. This was the last
`packaging <26.3` pin anywhere.

### pyPRMS upstream: close PR #64, port its regression test

- **Blocked on:** nothing — #64 was closed unmerged 2026-08-17; the
Expand All @@ -78,6 +68,32 @@ check it), **Action** (what to do once unblocked), and optional
retire `check_version.yaml` in favor of the preflight + release.yaml
checks.

### Retire the GSFLOW ifort binary

- **Blocked on:** nothing external -- this is the follow-up half of the
ifort retirement. Check: `git ls-files bin/` still lists
`gsflow_2.4.0_ifort_apple_silicon_dbl_prec`.
- **Action:** decide what replaces it. GSFLOW source is not in this
repository (`gsflow_src/` is gitignored), so unlike PRMS it cannot be
compiled on demand; it is the last intel-built artifact here, and it is
an **x86_64** binary, so on Apple Silicon it runs under Rosetta 2 --
including on the `macos-latest` runners for the three
`test_fgr_ag_2yr_*` jobs, which are the only consumers. Options:
vendor/point at a GSFLOW source and build it with gfortran the way PRMS
now is; ship a gfortran-built arm64 GSFLOW binary; or drop the binary
and remove `macos-latest` from the `test_fgr_ag_2yr_*` matrices.
- **Notes:** `bin/gsflow_2.4.0_ifort_apple_silicon_dbl_prec_og` is
byte-identical to the non-`_og` file (verified with `cmp`) and is
referenced nowhere -- 11.9 MB of pure duplicate to delete whichever way
the decision goes. The PRMS half of this work (binaries compiled on
demand from `prms_src/`, ifort and Intel MacOS dropped, `m1` -> `mac_arm`)
landed on branch `feat_drop_ifort`. That compile-on-demand scheme is an
accepted intermediate step: the intended end state is to move the
PRMS/GSFLOW sources into a separate "oracle" repository that publishes
binaries for pywatershed to download, which would settle this item too.
Prefer a cheap stopgap here over a design that assumes the sources stay
in this repository.

### Deliver CI-usage findings to org admins

- **Blocked on:** nothing; conversational.
Expand All @@ -92,6 +108,11 @@ check it), **Action** (what to do once unblocked), and optional

## Done

- 2026-08-19: conda-forge pywatershed 3.0.0 build 1 published (feedstock
PR #14: `pyprms >=0.10.0` replaces `packaging <26.3`; verified on
anaconda.org, uploaded 19:35 UTC). The last `packaging <26.3` pin
anywhere is gone. Also: PR #407's sagehen ubuntu OOM resolved by
serial pytest (`-n=1`) on Linux.
- 2026-08-19: gh-pages v3.0.0 extended release notes are live upstream
(`doi-usgs.github.io/pywatershed`, 2026-07-13 post) and all 3.0.0
release-body links resolve — step 9 done, the 3.0.0 release is
Expand Down
Loading
Loading