Repository navigation
fix(examples): move the IDR microscopy source to livingobjects.ebi.ac.uk - #98
Merged
Merged
Conversation
EMBL-EBI is retiring the uk1s3.embassy.ebi.ac.uk S3 endpoint. The same idr bucket is served from livingobjects.ebi.ac.uk with the same key layout and anonymous path-style access, so only the endpoint constant changes. Adds a remote test that streams a batch off the real image and checks it against a direct zarr read, so the --source idr path is exercised at all. Closes #97 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
emfdavid
commented
Sep 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
EMBL-EBI is retiring the
uk1s3.embassy.ebi.ac.ukS3 endpoint that the microscopy example's--source idrreads from. The IDR image now lives atlivingobjects.ebi.ac.uk, which has the sameidrbucket, the same key layout, the same anonymous path-style S3 access, and byte-identical.zarraymetadata. The only code change isIDR_ENDPOINTinexamples/microscopy/data.py.Nothing tested the real-IDR path before this PR, so this adds
test_idr_store_streams_the_real_image(markedremote). It checks the geometry (236 planes, Z-chunks of 1 and 30), streams one batch, and compares it with a direct zarr read.Closes #97
For reviewers
The test is only as good as the proof that it reads the live host, so it was run three ways:
https://nonexistent.invalid): fails with an obstore error, so the test really goes through the endpoint rather than a cache.python -m examples.microscopy.train_torch --source idr --epochs 1 --sample-range 0,120also runs end to end against the new host.CI never passes
--remote, so CI will skip this test. #100 proposes a weekly scheduled--remotejob. The Zarr v3 (OME-NGFF v0.5) version of this image, which the issue mentions, is tracked separately in #99: its layout differs (4-D, sharded, different label path), so it isn't just a URL swap.Author attestation
have verified the claims made in this description.
Checklist
uv run ruff check src tests bench examples,uv run ruff format --check src tests bench examples,uv run mypy src bench examplesanduv run pytest -qare greenlocally (668 passed, 9 skipped;
test_tf.pyskipped because TensorFlow isn't installed in this env), plusuv run --extra docs mkdocs build --strict## UnreleasedinCHANGELOG.md🤖 Generated with Claude Code