Skip to content

fix(examples): move the IDR microscopy source to livingobjects.ebi.ac.uk - #98

Merged
emfdavid merged 1 commit into
mainfrom
fix/idr-livingobjects-endpoint
Sep 28, 2026
Merged

emfdavid merged 1 commit into
mainfrom
fix/idr-livingobjects-endpoint

Conversation

@emfdavid

@emfdavid emfdavid commented Sep 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

EMBL-EBI is retiring the uk1s3.embassy.ebi.ac.uk S3 endpoint that the microscopy example's --source idr reads from. The IDR image now lives at livingobjects.ebi.ac.uk, which has the same idr bucket, the same key layout, the same anonymous path-style S3 access, and byte-identical .zarray metadata. The only code change is IDR_ENDPOINT in examples/microscopy/data.py.

Nothing tested the real-IDR path before this PR, so this adds test_idr_store_streams_the_real_image (marked remote). It checks the geometry (236 planes, Z-chunks of 1 and 30), streams one batch, and compares it with a direct zarr read.

Closes #97

For reviewers

The test is only as good as the proof that it reads the live host, so it was run three ways:

  • Old endpoint: passes, which shows the test works.
  • Nonexistent endpoint (https://nonexistent.invalid): fails with an obstore error, so the test really goes through the endpoint rather than a cache.
  • New endpoint: passes.

python -m examples.microscopy.train_torch --source idr --epochs 1 --sample-range 0,120 also runs end to end against the new host.

CI never passes --remote, so CI will skip this test. #100 proposes a weekly scheduled --remote job. The Zarr v3 (OME-NGFF v0.5) version of this image, which the issue mentions, is tracked separately in #99: its layout differs (4-D, sharded, different label path), so it isn't just a URL swap.

Author attestation

  • I have reviewed every change in this PR, I can explain why each one is correct, and I
    have verified the claims made in this description.

Checklist

  • Tests added or updated. The old endpoint still works, so no test could fail before this change; the bogus-endpoint run above stands in for that failing-first run.
  • uv run ruff check src tests bench examples, uv run ruff format --check src tests bench examples, uv run mypy src bench examples and uv run pytest -q are green
    locally (668 passed, 9 skipped; test_tf.py skipped because TensorFlow isn't installed in this env), plus uv run --extra docs mkdocs build --strict
  • A bullet added under ## Unreleased in CHANGELOG.md
  • No load-bearing invariant is broken

🤖 Generated with Claude Code

EMBL-EBI is retiring the uk1s3.embassy.ebi.ac.uk S3 endpoint. The same idr
bucket is served from livingobjects.ebi.ac.uk with the same key layout and
anonymous path-style access, so only the endpoint constant changes. Adds a
remote test that streams a batch off the real image and checks it against a
direct zarr read, so the --source idr path is exercised at all.

Closes #97

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@emfdavid emfdavid left a comment •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @will-moore

@emfdavid
emfdavid merged commit 92be2e8 into main Sep 28, 2026
9 checks passed
@emfdavid
emfdavid deleted the fix/idr-livingobjects-endpoint branch September 28, 2026 03:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

uk1s3.embassy.ebi.ac.uk migration to livingobjects storage

1 participant