mean_top_downward_shortwave_radiation_flux is all nans

[djgagne]

I was working on calculating some normalization values from WeatherBench2 ERA5 and noticed that mean_top_downward_shortwave_radiation_flux has nan values everywhere on Earth and potentially throughout the whole dataset. I spot checked in xr.open_zarr('gs://weatherbench2/datasets/era5/1959-2023_01_10-wb13-6h-1440x721_with_derived_variables.zarr') and saw the same issue in the 240x121 version of the dataset. Other variables are fine when I pull them down, so I don't think the issue is on my end. I'm kinda surprised this one hasn't been caught before, but maybe people are using their own solar calculations or sin(day of year) instead of what is in weatherbench. We have implemented our own solar calculations in CREDIT, so fixing this is not super urgent but would be nice to have for future intercomparisons.

[raspstephan]

Hey, thanks for flagging this. I don't really know why this is the case but I would suggest switching to https://github.com/google-research/arco-era5 which is our up to date ERA5 version.

[langmore]

@djgagne FYI the arco-era5 pipeline does occasionally have failures and need to be restarted. When this happens, the metadata will say it's "up to date", but in fact recent dates will be missing (and read as NaN). You can monitor Github issues, and avoid downloading recent dates with problems. For example, at this moment there are missing recent dates, and this issue (google-research/arco-era5#136) explains what is going on.