Have you e.g. considered reading/writing from/to S3?
In the resave.py script we are working on for the challenge, options like these:
time ./resave.py \
zarr/v0.4/idr0001A/2551.zarr \
--input-bucket=idr \
--input-endpoint=https://uk1s3.embassy.ebi.ac.uk \
--input-anon \
...
prevent the need to download the data locally.
I'm currently working on using zarrs_reencode but generating a script:
./resave.py zarr/v0.4/idr0001A/2551.zarr --output-script ...
which produces a script per Zarr array of the form:
zarrs_reencode --chunk-shape 1,1,1040,1376 --shard-shape 2,16,1040,1376 --dimension-names c,z,y,x --validate \
zarr/v0.4/idr0001A/2551.zarr/C/3/0 OUTPUT/C/3/0
but this of course won't work when the source or target are on S3.
Have you e.g. considered reading/writing from/to S3?
In the resave.py script we are working on for the challenge, options like these:
prevent the need to download the data locally.
I'm currently working on using
zarrs_reencodebut generating a script:which produces a script per Zarr array of the form:
but this of course won't work when the source or target are on S3.