Strategies for improvement robustness with JLD2

[odunbar]

Issue

It seems like using JLD2 can be quite brittle with updating of EKP/Julia/JLD versions. This can be frustrating for users who have for example the priors / data saved in this format.

It would be good to investigate the most brittle parts of this, and improve/overload the saving functionality, so that rebuilding is more robust. Or at least rebuilding by default could fall back to some flattened objects that can be used to extract the data more easily.

This isnt a small piece of work but could be quite helpful. I think particularly as the use-case is often that EKP is used often for longer term storage relative to it's version evolution.

Possible solution

How this could work:

1. Define an EKP state (containing the internals of current EKP) for the current version V1 this can rebuild EKP. (Also do this for ParameterDistribution for priors). It should be in a more robust format than the full EKP object.
2. Define a converter between the EKP and the state, for the current EKP.
3. Whenever the state is changed, define a new state and a migration pattern from one version to the next V1 -> V2 -> V3 as the state is changed over time. (e.g. replacing missing states with default values etc.)
4. Have JLD2 work with only the state and some metadata (e.g. could just be the state version 1,2,3, ... )
5. Define new methods for the JLD2 writeas, rconvert, wconvert etc. So that users do not see the destructure-restructure patterns and can use @save and @load