[mxkpp]

The NWM CONUS retrospective (historical) forcing processor had the following inefficiencies, leading to slow run times:

1. Disk cache was not being leveraged
2. Variables were being sliced individually and then merged into one dataset, rather than lazy-merging once and then slicing once.
3. The sliced dataset remained lazy, causing large remote zarr chunks to be fetched repeatedly.

This resolves these issues by leveraging the cache, merging the variables before slicing, and loading the sliced data into memory once. The potential memory impact is mitigated by loading variables sequentially after the lazy merge & slice.

Speed improvement for one small test case

Command ran from nmw-rte via run.sh, a 2-day realization with 3 total iterations (1 initial/baseline iteration, plus 2 calibration iterations), for the small test basin of the USGS gage "01123000":

docker_run python -um "ngen_rte.run_calibration" -n 2 -fconfig "nwm" -start "2013-07-25 00:00:00" -dur 2 --gage_id ${TEST_GAGE} --hydrofab_file "${TEST_HYDROFAB_FILE}"

Before: 19.5 minutes.

After: 0.9 minutes.

The disk cache files were deleted between those 2 calls for apples-to-apples benchmark.

For a deterministic comparison to confirm that outputs are the same before and after this change, the following "default" realization was ran both ways as well, and ngen outputs were inspected:

docker_run python -um "ngen_rte.run_default" -n 2 -fconfig "nwm" -dt "2013-07-25 00:00:00" -dur 2 -rname "default_nwm" --output_format CSV NetCDF --gage_id ${TEST_GAGE} --hydrofab_file "${TEST_HYDROFAB_FILE}"

Additions

• None

Removals

• None

Changes

• See above

Testing

1. See above

Screenshots

Notes

Todos

Checklist

• PR has an informative and human-readable title
• Changes are limited to a single goal (no scope creep)
• Code can be automatically merged (no conflicts)
• Code follows project standards (link if applicable)
• Passes all existing automated tests
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• Visually tested in supported browsers and devices (see checklist below 👇)
• Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
• Reviewers requested with the Reviewers tool ➡️

Testing checklist

Target Environment support

• Linux

[mxkpp]

Test team conducted tests: calibration for the default gage, and "default" workflow for the VPU 03S.