Repository navigation
Archiving ERDDAP datasets at NOAA's NCEI and avoiding fix fixity issue for netCDF files #367
Replies: 5 comments 6 replies
|
I think an important thing to discuss as well is who uses ArchiveADataset and does it have a place in a new workflow (assuming we provide something like the proposed .ncNCEI or find another alternative) or can we deprecate/stop supporting ArchiveADataset? I do think can cachefromurl cover the use case is a very important question. If not, is there an improvement we can make so it could cover the use case? |
|
I personally haven't heard of people using the ArchiveADataset for this type of workflow. I had a chat with @relphj (NOAA NCEI) at the ESIP July meeting about this topic. Is it possible to have a new endpoint that would be the .nc format, without the |
|
We developed the https://github.com/axiom-data-science/bagitify tool to archive NCEI ready (e.g. bagitified) datasets from ERDDAP (edit: which I now see is mentioned in the original post 😁)
I think this should be revised to ".nc format with the history attribute unaltered", i.e. preserve any existing history attribute but prevent ERDDAP from appending "Download from..." etc. That should be fairly easy to implement, but I think it would be better to control the history appending with a query parameter instead of a new netCDF download format. That way all of the existing netCDF formats could respect that
This part might be trickier. Do you want the file checksum as a response header? etag? In all cases ERDDAP has to read all relevant data and calculate the checksum for each request, so clients checking for changed checksums within a specified timespan would double the work for ERDDAP vs just downloading the file and comparing checksums locally. At least that's my understanding. |
|
@srstsavage @ChrisJohnNOAA Curious how your tool compares with the existing tool in ERDDAP™ which I don't think has been updated in a while. If memory serves it is in the WEB-INF directory |
|
It's been long enough that I don't exhaustively remember all the specifics, but some of the reasons we created a separate external tool:
|
|
@srstsavage If the idea is to add something as an ERDDAP™ endpoint I would think the existing code would be a better starting point since it is Java. Or am I misunderstanding this? As I said, I don't think the existing code has been updated for years, but I imagine it could be updated and made into a service without a huge amount of work (which of course is easy for me to say since I will not be doing it) |
|
I think the main proposal is to improve ERDDAP's capability to produce archive-friendly dataset downloads, mainly concerning the indeterminate handling of the |
|
If a parameter (could be added to the UI as well) to not alter history would solve the problem we can add that. We could support that on all appropriate file downloads. As Shane said checksum is slightly more complicated because we need to communicate that somehow and if it's a separate request that is likely putting much more strain on the server. One question I have is what would be needed to replace usage of the ArchiveADataset tool and/or the bagitify tool. I thought there were some groups that used ArchiveADataset as part of a workflow. I wonder if a REST endpoint that supports exporting the data would be better or worse for those flows. |
|
I think this could be a useful conversation to have at the next Technical Board meeting. Is the next one scheduled? We can invite @relphj and have a discussion about what NCEI needs and what can actually be done. I'm not sure who uses ArchiveADataset. Since it's an internal tool, which an ERDDAP Admin has to run, it doesn't help NCEI much. But maybe folks are using it to generate their archive packages? |
|
The Technical Board meetings are scheduled quarterly. The next one is: Wednesday, October 28 · 12:00 – 1:00pm |
Uh oh!
There was an error while loading. Please reload this page.
Mainly described in this conversation ioos/ioos-atn-data#28. Is there a way to use ERDDAP as a platform to submit (or generate) packages for NCEI long term archival?
Problem
When downloading a netCDF file from ERDDAP we run into a file fixity problem. Essentially every time a user goes to an ERDDAP endpoint and downloads a netCDF file (preferred format for archival at NCEI), the resultant file will always have a different checksum. Even if the data contents are the same.
It's been stated that this is because some of the metadata ERDDAP included in the resultant file is dynamically generated by ERDDAP (e.g.
:historyattribute containing information about when the file was downloaded) and thus gets changed every time someone downloads the dataset, resulting in a different checksum. Even though the data don't actually change.For NCEI to understand when files change, they need some certainty that a change in checksum is an actual change to data or some other part of the data that requires an update to the archive.
Discussion
The proposal discussed in ioos/ioos-atn-data#28 suggests that NCEI downloads a temporary version of the file without the metadata (.jsonl) and computes checksums on that, then pulls nc files if there are changes. That's a lot of overhead for NCEI to manage.
Axiom has built a packager for building ready to archive packages for NCEI from ERDDAP datasets. However, it uses the .ncCFMA endpoint. So, we will run into the file fixity issue.
Would it be possible to add another output format, lets say .ncNCEI, that outputs the file without updating the
:historyglobal attribute?Is there another approach we are not considering here? (thinking about ERDDAP's capability to copy data with
cachefromurl, maybe NCEI stands up an ERDDAP and registers approved datasets to archive in there? Then, the files copied could be archived. Just thinking out loud.)cc: @relphj, @apkrelling, @ksauby, @ChrisJohnNOAA
All reactions