Issue:
As we migrate to PyTorch/MONAI and support larger Zarr/BIDS datasets, there's a risk of silent failures or deep-stack crashes if the data doesn't perfectly match what nobrainer.dataset.get_dataset() and MONAI's CacheDataset expect. Currently get_dataset() fails late — during training — when data has broken DataLad symlinks, inconsistent spacing, mismatched label classes, corrupt Zarr shards, or orientation mismatches.
Example: many neuroimaging datasets (including neuronets/trained-models) are DataLad/git-annex repositories where large files are symlinks to annex objects. Before datalad get is run, these symlinks exist on the filesystem but resolve to nothing. Attempting to read a NIfTI header from an un-fetched annex pointer produces a cryptic read error rather than a helpful message telling the user which files need to be downloaded.
Proposal:
Add nobrainer/data/spec.py with a DataSpec dataclass capturing the implicit contract that get_dataset() and predict() already assume:
Image/label path pairs exist and are readable (NIfTI or Zarr)
DataLad/git-annex awareness: detect symlinks pointing to annex objects and verify the content is locally present before attempting header reads. On failure, report the exact paths and suggest datalad get <path>
Spatial dimensions of image and label match per subject
Voxel spacing is within a declared range and consistent across subjects
Orientation codes are uniform (or explicitly declared mixed)
Label values are a subset of a declared class set
For Zarr: chunk shape, shard presence, and pyramid level count are valid
Add nobrainer validate <manifest> — exits non-zero on any violation, prints actionable errors
Add nobrainer inspect <manifest> — outputs human-readable summary + JSON
Wire preflight validation into train and predict entry points (skippable with --skip-validate)
No new external dependencies. Uses nibabel (already present) for NIfTI header reads and zarr/fsspec (already present) for Zarr inspection.
Issue:
As we migrate to PyTorch/MONAI and support larger Zarr/BIDS datasets, there's a risk of silent failures or deep-stack crashes if the data doesn't perfectly match what
nobrainer.dataset.get_dataset()and MONAI'sCacheDatasetexpect. Currentlyget_dataset()fails late — during training — when data has broken DataLad symlinks, inconsistent spacing, mismatched label classes, corrupt Zarr shards, or orientation mismatches.Example: many neuroimaging datasets (including neuronets/trained-models) are DataLad/git-annex repositories where large files are symlinks to annex objects. Before
datalad getis run, these symlinks exist on the filesystem but resolve to nothing. Attempting to read a NIfTI header from an un-fetched annex pointer produces a cryptic read error rather than a helpful message telling the user which files need to be downloaded.Proposal:
Add nobrainer/data/spec.py with a
DataSpecdataclass capturing the implicit contract thatget_dataset()andpredict()already assume:Image/label path pairs exist and are readable (NIfTI or Zarr)
DataLad/git-annex awareness: detect symlinks pointing to annex objects and verify the content is locally present before attempting header reads. On failure, report the exact paths and suggest
datalad get <path>Spatial dimensions of image and label match per subject
Voxel spacing is within a declared range and consistent across subjects
Orientation codes are uniform (or explicitly declared mixed)
Label values are a subset of a declared class set
For Zarr: chunk shape, shard presence, and pyramid level count are valid
Add
nobrainer validate <manifest>— exits non-zero on any violation, prints actionable errorsAdd
nobrainer inspect <manifest>— outputs human-readable summary + JSONWire preflight validation into train and predict entry points (skippable with --skip-validate)
No new external dependencies. Uses nibabel (already present) for NIfTI header reads and zarr/fsspec (already present) for Zarr inspection.