Feat cascades port - #407
Open
jmccreight wants to merge 14 commits into
Open
Conversation
- PRMSRunoffCascadesNoDprst / PRMSSoilzoneCascadesNoDprst and preprocess_cascades/preprocess_gridded utilities - HruMixin: active-HRU masks and nan-masking of inactive HRUs; applied to atmosphere, solar geometry, canopy, snow, runoff, soilzone and groundwater (active-HRU loops via hru_route_order / wh_active_hrus) - Budget/_set_budget gain an additive active_mask (ignore_nans kept) - cascade_flag/cascadegw_flag control options; cascade variable metadata; PRMS param file reading of scalar/monthly-scalar params and extra dims (nrain, ntemp, nlake, ...) - Process.subset keeps process dims (keep_dims); Parameters .from_dataset_dict; name-guards so subclasses can pre-set names - prms_src: G0 precision CBH output format (binaries NOT rebuilt)
- above/below_snow: sagehen_no_gw_cascades + sagehen_gridded_cascades configs; comparisons subset to active HRUs; below_snow now runs with imbalance_behavior='error' (strict budgets) - isolated soilzone/groundwater tests skip cascade/cascadegw configs (coupled below_snow is the designated check) - utils_compare: active-HRU subsetting, compare_model_in_memory, assert_dicts_equal - generation: CBH-generation runs for gridded domains, PRMS hru_hortn_cascflow -> hru_horton_cascflow rename, tolerate absent streamflow_module; test collection ignores *make_cbh_only* controls
- sagehen_no_gw_cascades control + testing yamls; per-process cascade parameter files (preprocessed, incl. derived cascade params) - sagehen_gridded_5yr domain: myparam/sagehen.data/controls/yamls and per-process + dis parameter files regenerated from current myparam (previous dis had stale hru_type) - restore prms 5.2.1.1 apple-silicon binary (drb stream_temp data gen) - cascading_flow example notebook (untested against develop APIs) Caveat: gridded CBH inputs (prcp/tmax/tmin nc, .day) are generated files (gitignored); regenerating them requires binaries rebuilt with the G0 write_climate_hru patch.
- prms_exe_utils: PRMS 5.2.1 (repo source w/ cascades + G0 CBH output) is now a compilable variant like 5.2.1.1; test-data generation uses get_or_compile_prms_exe so missing compilable binaries are built automatically. Compiled binaries are gitignored (untrack 5.2.1.1 mac binary accordingly). - gridded controls: executable_desc PRMS 5.2.1; make_cbh_only needs parameter_check_flag=0 because WRITE_CLIMATE mode treats all HRUs as active and errors fatally on inactive cells' zero soil params (flag=0 clamps and warns instead) - makelist_double_precision: -std=gnu99 for the K&R-era mmf C code (rejected by gcc >= 15 defaults) - generation-only make_cbh_only controls skipped in soltab and final variable conversions Verified: freshly compiled binary reproduces the prior CBH files to ~1e-14 and all sagehen_gridded_5yr above/below_snow tests pass on data generated entirely from source.
- ignore test_preprocess_cascades in the broad hru_1_nhm, drb_2yr_nhm and ucb_2yr_nhm steps: it skips when cascade_flag is absent and --error-for-skips makes that an error (it still runs in the sagehen steps, whose controls have cascade_flag) - PRMSRunoffAg: basin_init (inherited) sets active-HRU info on demand for subclasses whose inits do not call _set_active_hrus, and ag's kernel passes the (inactive) cascade arguments to the shared compute_infil Verified locally: fgr_ag_2yr runoff/soilzone ag tests 28 passed after domain regeneration; sagehen cascade tests still green.
- sagehen_5yr job gains sagehen_no_gw_cascades steps (generate + tests):
explicit file list validated locally under --error-for-skips
(preprocess_cascades, above/below_snow, atmosphere, canopy,
groundwater, snow, solar_geom, self_drive)
- new sagehen_gridded_5yr job on all three platforms: compiles PRMS
5.2.1 from prms_src, generates CBH forcings with PRMS
(make_cbh_only), generates the main run, tests above/below_snow +
groundwater/snow/solar_geom. (Gridded canopy/atmosphere isolated
tests need active-HRU masking in compare_netcdfs; follow-up.)
- ci_local.sh mirrors both ('c' flag disables the gridded domain)
- generate conftest: domains with a make_cbh_only control are
self-contained (no committed CBH required)
- compile_prms: prefer the active environment's compilers by
prepending sys.prefix/bin to the subprocess PATH (keeps gcc/gfortran
names, which the makelists key their flags on)
- test_preprocess_cascades skips non-sagehen_5yr domains (answers are
hard-coded for that domain)
Verified locally via a clean-clone rehearsal on macOS: with no
compiled binary and no CBH files present, the gridded job sequence
auto-compiles, regenerates CBH bit-identically, and passes all tests
(14 passed, 4 xfailed). The sagehen cascades step passes 22/4 xfail
under --error-for-skips.
The sagehen_gridded_5yr CI job removes the large PRMS .day CBH files after test data generation; the tests read the netcdf CBH files. The self-contained-domain check now also accepts prcp.nc.
Skips plain branch pushes; runs for PRs, pushes to develop/main, and via the opt-in tokens described in the trigger comment at the top of the workflow.
Three attempts on the previous head died in the sagehen_gridded_5yr ubuntu pytest step (exit 143 / runner killed) while macOS and Windows passed: 4 xdist workers x the gridded domain's memory footprint exhausts the tightest-RAM runner. The ci-sagehen token in this message runs the sagehen family on this push to verify.
jmccreight
added a commit
to jmccreight/pywatershed
that referenced
this pull request
Aug 20, 2026
Hardening from the skill's first real run (PR DOI-USGS#407): - Preflight step: REST-first diff acquisition via the public GitHub API (follow the EC-USGS -> DOI-USGS 301); gh is optional and its GraphQL paths are SAML-blocked for this org, so gh pr diff does not work. - Consolidate blocking questions into one poll and forbid proceeding on a documented default for an unanswered question. - Split the ground rules: the report file is the only permitted write, while read-only verification by execution is encouraged; pass both down to the built-in code-review sub-agents. - Permit read-only git under /review alongside the REST preference, and note that a diff vs upstream/develop is only as fresh as the last fetch. - Document fuse mode: a blind second pass plus a self-contained fused report with per-finding provenance and explicit reconciliation. - Sharpen Layer A from real misses: compare pytest flags in ci_local.sh and not just file lists, verify doc/api autosummary targets exist, and read a skip guard's presence-vs-value logic before calling a collected test a CI failure. CLAUDE.md: note the second-pass and fused report modes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
whats-new.rstapi.rstor it's sub rsts?Docs
Look for this PR number in our read-the-docs builds where you can browse on-line.
You can alternatively get a CI-build of the docs by entering the PR number for the
###in https://github.com/DOI-USGS/pywatershed/pull/###/checks then clicking onDocumentation Buildand finally looking for thedocumentation-htmlartifact which will download as a zip file.