WDFN APIs will break `nwis_client`

[jarq6c]

USGS is currently in the process of migrating access to water data from the current suite of NWIS APIs to a completely new set of APIs. See the blog post here:

https://waterdata.usgs.gov/blog/api-whats-new-wdfn-apis/

These new APIs are incompatible with nwis-client. The existing APIs may start going offline next calendar year (perhaps as early as Q1). This issue is meant to raise awareness about these new APIs.

This raises some development concerns for hydrotools. We could redevelop nwis_client to work with the new APIs. However, another option is to contribute to the official NWIS python client: https://github.com/DOI-USGS/dataretrieval-python

It looks like dataretrieval uses the requests package. Assuming USGS is open to collaboration, our experience with asynchronous REST clients could help improve the efficiency and robustness of the official client. I also think there's merit in supporting development of the official client, given USGS' importance to the broader hydrological community.

@aaraney @hellkite500 @christophertubbs @jameshalgren @frsalas-noaa

[jameshalgren]

• @JoshCu

[jarq6c]

@mgdenno @samlamont @GregoryPetrochenkov-NOAA @MonicaStone-NOAA

[aaraney]

I also think there's merit in supporting development of the official client, given USGS' importance to the broader hydrological community.

I agree in principle @jarq6c. I'm not sure the USGS will have an appetite for a client that sends multiple concurrent requests to their service. I hope i'm wrong though.

[jarq6c]

Feel free to use this space to document/list your use-cases and data requirements with respect to USGS observations. For example, common use-cases for Evaluations have included:

1. Validation of the National Water Model 40-year Retrospective Simulation at 8500 streamflow gauges across CONUS, Alaska, Hawaii, and Puerto Rico.
2. 90-day Routine Operational Evaluation of hourly streamflow forecasts from the National Water Model at 8500 streamflow gauges.
3. Post-event forecast evaluations of flood-inducing streamflow at all available streamflow gauges within a city, county, state, HUC region, Weather Forecast Office boundary, or River Forecast Center boundary for a 10 day period.
4. Derivation of long-term streamflow statistics, quantiles, percentiles, annual exceedance probabilities, or recurrence intervals on an annual, daily, day-of-year, or hourly basis.

[christophertubbs]

5. Ad-hoc evaluations over HUC-12 or RFC spans of locations over a range of times, maybe 48-72 hours in length, from a cold start, i.e. no preexisting observations from USGS, without a means for long term storage of requested observations

[jarq6c]

Some reference code.

Latest continuous API

import geopandas as gpd

# Get all active streamflow gages as a POINT layer
url = (
  "https://api.waterdata.usgs.gov/ogcapi/v0/collections/latest-continuous/items"
  "?f=json&parameter_code=00060&limit=10000"
)
gdf = gpd.read_file(url)
print(gdf.head())

# Save to file
gdf.to_file("USGS_streamflow_sites.gpkg", layer="gages", driver="GPKG")

                        id                       monitoring_location_name state_code  ...  basin_code time_zone_abbreviation                    geometry
0            USGS-02339210                 WEHADKEE CREEK NEAR PITTMAN AL         01  ...        None                    CST  POINT (-85.31995 33.22512)
1            USGS-02339215              WEHADKEE CREEK NEAR ROCK MILLS AL         01  ...        None                    CST  POINT (-85.28189 33.14734)
2            USGS-02339225            WEHADKEE CREEK BELOW ROCK MILLS, AL         01  ...        None                    CST  POINT (-85.24911 33.12234)
3            USGS-02339430            KELLEM HILL CREEK AT LAFAYETTE, ALA         01  ...        None                    CST  POINT (-85.39106 32.88179)
4            USGS-02339455                  ALLEN CREEK NR LAFAYETTE, ALA         01  ...        None                    CST  POINT (-85.36606 32.91096)

Pros: Super nice not to have to write a parser.
Gotchas: If there are more than 10,000 active locations you'll need to use the cursor parameter in the response to incrementally retrieve all sites.

"links":[
        {
            "rel":"next",
            "href":"https://api.waterdata.usgs.gov/ogcapi/v0/collections/latest-continuous/items?cursor=MDE0ZGVhOTEtYjYyMS00YjUwLWI4NmUtMTM3MDdkYjgxMjAz&parameter_code=00060&limit=50",
            "type":"application/geo+json",
            "title":"Items (next)"
        },
    ],
    "timeStamp":"2025-08-26T14:47:59.093299Z"

Monitoring location API

I was able to get all streamflow sites in Alabama using this URL:

url = (
  "https://api.waterdata.usgs.gov/ogcapi/v0/collections/monitoring-locations/items"
  "?f=json&state_code=01&site_type_code=ST&limit=10000"
)

This returns 2600+ locations. There does not appear to be a status parameter that lets you just get active gages. It also requires a numerical state ID drawn from this table: https://www2.census.gov/geo/docs/reference/state.txt

Time series metadata API

This API seems to only allow spatial queries by bbox (instead of state/HUC/etc). The date based parameters (begin, end, begin_utc, and end_utc) appear to currently be broken, as well. The docs say that these will default to UTC in the future. It appears this URL maxes out the return limit.

url = (
  "https://api.waterdata.usgs.gov/ogcapi/v0/collections/time-series-metadata/items"
  "?f=json&parameter_code=00060&limit=10000"
)

Real-time flood impacts API

Includes flood stage thresholds and embeded crosswalk from USGS site codes to NWS LIDs. See file referenced above for state codes.

import pandas as pd

url = "https://api.waterdata.usgs.gov/rtfi-api/referencepoints/state/01"
df = pd.read_json(url, orient="records")
print(df.head())

       id                                          site_name                                        description nws_id  ... unit  gage_height  is_flooding  active
0     804                  BIG ESCAMBIA CREEK AT FLOMATON AL  Flood Impact Statement from National Weather S...  FLMA1  ...   ft         6.93        False    True
1     172    CHATTAHOOCHEE RIVER AT 14TH ST, AT COLUMBUS, GA  BM1 - Left bank river walk in front of Courtya...  CBUG1  ...   ft        16.60        False    True
2     778              VALLEY CREEK AT AVE W NEAR ENSLEY, AL  Flood Impact Statement from National Weather S...  VAWA1  ...   ft         3.84        False    True
3     796  TALLAPOOSA RIVER NR NEW SITE, AL.(HORSESHOE BEND)  Flood Impact Statement from National Weather S...  HBPA1  ...   ft         3.49        False    True
4     771            STYX R NR LOXLEY, AT SMITHS SIDING, ALA  Flood Impact Statement from National Weather S...  SXLA1  ...   ft         4.17        False    True

Water data statistics API

Looks like this one is still under development. Getting annual peaks requires some additional processing compared to the other APIs.

import json
import geopandas as gpd
import pandas as pd

# Get monthly and annual stats for a single site
url =(
  "https://api.waterdata.usgs.gov/statistics/v0/observationIntervals"
  "approval_status=approved&computation_type=maximum&country_code=US&
  "mime_type=application%2Fjson&monitoring_location_id=USGS-01432160&page_size=1000"
)
gdf = gpd.read_file(url)

# Extract json
data = json.loads(gdf["data"].iloc[0])

# Parse
df = pd.DataFrame.from_records(data[0]["values"])
print(df.head())

    start_date    end_date interval_type    value  sample_count approval_status                        computation_id computation
0   2018-10-01  2018-10-31         month  16700.0            31        approved  e54db94d-5f75-400d-99a9-aba1a9c3793b     maximum
1   2018-11-01  2018-11-30         month  24600.0            30        approved  adc4ac2f-53c1-4e47-892e-75df93f227d4     maximum
2   2018-12-01  2018-12-31         month  23200.0            31        approved  53abd2f7-fed6-4117-ac97-76fab739e1db     maximum
3   2019-01-01  2019-01-31         month  26300.0            31        approved  d6b1f6ca-2436-4ef2-a7a8-8db55eb78c67     maximum
4   2019-02-01  2019-02-28         month  18300.0            28        approved  0e4057b2-03ef-496b-989b-a9079fb9f9fc     maximum

[jarq6c]

Additional use-case:

6. Evaluation of hourly streamflow predictions over multiple decades for specific gage sets including: CAMELS, GAGES-II, and arbitrary lists (100s to 1000s of gages).

[samlamont]

Just adding this edge case we encountered for awareness. It seems in some cases there can be multiple "types" of discharge returned for parameter 00060 at a single site. An example is site 08025360 (reservoir): https://waterdata.usgs.gov/monitoring-location/USGS-08025360/#dataTypeId=continuous-00060--315947573&period=P7D&showMedian=true

We couldn't find a clean way to discern between data types and just allowed users to pass in a string describing the value they want (ie, "Discharge, cubic feet per second [Total Spillway Releases]"), along with the site number as a dictionary. Not necessarily recommending this approach :) Could be something to think about when/if building a new client.

More notes are here: RTIInternational/teehr#498

[christophertubbs]

This will have to be a post-response task, but the discriminatory factors will come from value.timeSeries[*].values[*].method[0]. You'd have to ask USGS to see if they ever have more than 1 item in the method collection or if it's just the schema (that's where my money's at).

Another way to look at time series selection here that a lot of us do is something along the lines of value.timeSeries[*].values[method[0].methodDescription=""] (forgive me if the syntax is wrong - it's just a rough wag at the jmespath). In this case, there is no methodDescription="", so you have three options:

1. Go with the first time series (what a lot of code defaults to)
2. Have the user specify what method to run with if there are methodDescriptions
3. Go from returning a singular series of observations to a collection of observations and require/encourage integrators to operate on the product of modeled values and the entire set of observations

Option 1 is likely too naive and option 2 will result in a lot of edge cases where the method of interest for location a might be "[Some Method 1]", but the method of interest for the second location might be "[SOME METHOD 1]" or "[Totally Different Method 2]" because that location just doesn't have "[Some Method 1]" and you can't make a fallback list because what if one location has "[Some Method 1]" and "[Totally Different Method 2]", but where "[Some Method 1]"'s data is undesirable. Option 3 seems more rigourous from a less scientifically literate perspective but will undoubtably produce a lot more noise that would be hard to explain to target audiences.

[jarq6c]

Just adding this edge case we encountered for awareness. It seems in some cases there can be multiple "types" of discharge returned for parameter 00060 at a single site. An example is site 08025360 (reservoir): https://waterdata.usgs.gov/monitoring-location/USGS-08025360/#dataTypeId=continuous-00060--315947573&period=P7D&showMedian=true

We couldn't find a clean way to discern between data types and just allowed users to pass in a string describing the value they want (ie, "Discharge, cubic feet per second [Total Spillway Releases]"), along with the site number as a dictionary. Not necessarily recommending this approach :) Could be something to think about when/if building a new client.

More notes are here: RTIInternational/teehr#498

@samlamont A bit late, but I pulled these data using the alpha version of the new hydrotools client and it appears that the API returned three time series each with a unique time_series_id.

[jarq6c]

A new hydrotools.waterdata_client is under development. This issue will probably close after the beta.