Repository navigation
feat: Add WPC QPF reaper - #12
Merged
Merged
Conversation
cheginit
requested changes
Oct 5, 2026
cheginit
left a comment
Collaborator
There was a problem hiding this comment.
Cosecha mainly runs as scheduled AWS Lambda functions, where peak memory sets the configured size and the cost of every run. Please treat memory footprint as a design constraint for each reaper. Some rules of thumb:
- Measure peak RSS.
ds.nbytesundercounts. A full WPC issuance is 438 MB but peaks at ~1.25 GB. Check withresource.getrusage(resource.RUSAGE_SELF).ru_maxrss. - Reduce each file as it loads. Subset, select, and convert before combining. Subsetting after
concatsaves nothing. - Slices are views. A small
isel/selsubset keeps its full parent array alive, so deep-copy it before holding on to it. - Don't hold inputs and output together. A list plus
xr.concatpeaks at ~2x the result, with or without a generator. Don't stack withxr.merge: it aligns every piece to the full index first (15x the peak ofconcatin a test). - Lazy-load heavy packages.
import cosechacosts 263 MB against ~190 MB for what WPC needs, since pyiceberg (~54 MB), s3fs (~41 MB), and icechunk (~27 MB) load on every cold start. For a follow-up PR: a lazy public API like this template, and moving theicechunk/pyiceberg/pyarrowimports inreaping/base.pyinto thesow_to_*methods.
Requested changes (details inline):
- Apply transformations per file (bbox peak drops from ~1.25 GB to ~550 MB).
- Raise when explicitly requested hours are missing.
- Bound forecast hours by the issuance's own cycle.
- Keep the
DataNotFoundError/APIErrortypes.
Please also add a PR description and a README entry.
cheginit
approved these changes
Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
WPCQPFReaper, which fetches NOAA WPC 2.5 km CONUS 6-hour QPF forecasts (GRIB2) from https://ftp-wpc.ncep.noaa.gov/2p5km_qpf/.How it works
init_time="latest"picks the newest issuance that has every requested forecast hour.init_timeis 00, 06, 12 or 18 UTC and thatforecast_hoursare multiples of 6 within that cycle's limit (168 h for 00Z/12Z, 174 h for 06Z/18Z).DataNotFoundErrorif any requested hour is missing, so summedtpcan't silently come out too low.step.