Who is requesting this?
@benjwadams
What is being requested?
Plot functions run in production on Lambda/serverless infrastructure every six hours. There is logic to avoid replotting completed and old deployments, but currently active deployments have their CTD timeseries data fetched each plot run. This seems to be correlated with noticeably increased server load during every six hour interval, possibly implicating the plotting process in causing this increased server load.
The plot functions are presently in the Glider DAC Status repo here but should be ported over to this repo instead.
An approach should be taken to cache the previous timeseries data and time extent and then only fetch ERDDAP data which resides outside of these temporal bounds to ameliorate the server load needed for requesting data from ERDDAP. Cache busting approaches need to be considered if data providers decide to reupload data, possibly marking the cache as stale in cases where a deployment file is overwritten or is determined to lie within the start and end time bounds of the cached data. Some possible approaches to caching include either putting it in Redis or persisting NumPy data export formats to a location such as Amazon S3.
What is the requested deadline and why?
No response
What is the current status quo (i.e., what happens if this does not get done)?
Higher server load averages around every 6 hour mark in UTC, possible slowdown or rejection of HTTP requests pursuant to #583, especially with respect to ERDDAP data requests.
What indicates this is done (i.e., how do we know this is complete)?
No response
Provide a description or any other important information.
No response
Who is requesting this?
@benjwadams
What is being requested?
Plot functions run in production on Lambda/serverless infrastructure every six hours. There is logic to avoid replotting completed and old deployments, but currently active deployments have their CTD timeseries data fetched each plot run. This seems to be correlated with noticeably increased server load during every six hour interval, possibly implicating the plotting process in causing this increased server load.
The plot functions are presently in the Glider DAC Status repo here but should be ported over to this repo instead.
An approach should be taken to cache the previous timeseries data and time extent and then only fetch ERDDAP data which resides outside of these temporal bounds to ameliorate the server load needed for requesting data from ERDDAP. Cache busting approaches need to be considered if data providers decide to reupload data, possibly marking the cache as stale in cases where a deployment file is overwritten or is determined to lie within the start and end time bounds of the cached data. Some possible approaches to caching include either putting it in Redis or persisting NumPy data export formats to a location such as Amazon S3.
What is the requested deadline and why?
No response
What is the current status quo (i.e., what happens if this does not get done)?
Higher server load averages around every 6 hour mark in UTC, possible slowdown or rejection of HTTP requests pursuant to #583, especially with respect to ERDDAP data requests.
What indicates this is done (i.e., how do we know this is complete)?
No response
Provide a description or any other important information.
No response