Skip to content

feat: Add WPC QPF reaper - #12

Merged
sray014 merged 13 commits into
mainfrom
feature/wpc-reaper
Oct 8, 2026
Merged

sray014 merged 13 commits into
mainfrom
feature/wpc-reaper

Conversation

@sray014

@sray014 sray014 commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

Adds WPCQPFReaper, which fetches NOAA WPC 2.5 km CONUS 6-hour QPF forecasts (GRIB2) from https://ftp-wpc.ncep.noaa.gov/2p5km_qpf/.

How it works

  • Reads the server's directory listing to find the files for the requested issuance. init_time="latest" picks the newest issuance that has every requested forecast hour.
  • Checks that init_time is 00, 06, 12 or 18 UTC and that forecast_hours are multiples of 6 within that cycle's limit (168 h for 00Z/12Z, 174 h for 06Z/18Z).
  • Raises DataNotFoundError if any requested hour is missing, so summed tp can't silently come out too low.
  • Downloads the files, converts longitudes to −180–180, applies any transformations to each file, then combines them along step.

@sray014
sray014 requested a review from cheginit September 23, 2026 19:46

@cheginit cheginit left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cosecha mainly runs as scheduled AWS Lambda functions, where peak memory sets the configured size and the cost of every run. Please treat memory footprint as a design constraint for each reaper. Some rules of thumb:

  1. Measure peak RSS. ds.nbytes undercounts. A full WPC issuance is 438 MB but peaks at ~1.25 GB. Check with resource.getrusage(resource.RUSAGE_SELF).ru_maxrss.
  2. Reduce each file as it loads. Subset, select, and convert before combining. Subsetting after concat saves nothing.
  3. Slices are views. A small isel/sel subset keeps its full parent array alive, so deep-copy it before holding on to it.
  4. Don't hold inputs and output together. A list plus xr.concat peaks at ~2x the result, with or without a generator. Don't stack with xr.merge: it aligns every piece to the full index first (15x the peak of concat in a test).
  5. Lazy-load heavy packages. import cosecha costs 263 MB against ~190 MB for what WPC needs, since pyiceberg (~54 MB), s3fs (~41 MB), and icechunk (~27 MB) load on every cold start. For a follow-up PR: a lazy public API like this template, and moving the icechunk/pyiceberg/pyarrow imports in reaping/base.py into the sow_to_* methods.

Requested changes (details inline):

  1. Apply transformations per file (bbox peak drops from ~1.25 GB to ~550 MB).
  2. Raise when explicitly requested hours are missing.
  3. Bound forecast hours by the issuance's own cycle.
  4. Keep the DataNotFoundError/APIError types.

Please also add a PR description and a README entry.

Comment thread src/cosecha/reaping/wpc.py Outdated
Comment thread src/cosecha/reaping/wpc.py Outdated
Comment thread src/cosecha/reaping/wpc.py Outdated
Comment thread src/cosecha/reaping/wpc.py Outdated
Comment thread src/cosecha/reaping/wpc.py Outdated
Comment thread src/cosecha/reaping/wpc.py Outdated
Comment thread src/cosecha/reaping/wpc.py Outdated
@sray014
sray014 requested a review from cheginit October 6, 2026 15:28
@sray014
sray014 merged commit d1e1263 into main Oct 8, 2026
16 checks passed
@sray014
sray014 deleted the feature/wpc-reaper branch October 8, 2026 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants