Skip to content

feat(data): add BIDS discovery and reshaping - #382

Open
dhritimandas wants to merge 7 commits into
neuronets:alphafrom
dhritimandas:feat/bids-support
Open

feat(data): add BIDS discovery and reshaping#382
dhritimandas wants to merge 7 commits into
neuronets:alphafrom
dhritimandas:feat/bids-support

Conversation

@dhritimandas

@dhritimandas dhritimandas commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

What: We add BIDS support to nobrainer.data. DataSpec.from_bids() reads a BIDS dataset. to_bids() writes one.
Why: OpenNeuro ships BIDS datasets. Users had to convert them to manifests by hand. No reverse path existed.
This PR depends on #376 and is stacked on it until it merges.

Discovery (scan_bids / DataSpec.from_bids)

  • entity grammar, directory rules, and derivative naming pinned to BIDS 1.11.1
  • two backends, one output: zero-dependency walker (default fallback, what CI exercises) and optional pybids
  • three pybids traps guarded: bare filenames drop subject, run coerced to int (kills zero-padding), long entity names != schema keys
  • never crashes on a bad file — every skip recorded with a reason; fatal only on missing root or zero matches (raises with a skip histogram)
  • entries never a partial mix: all labelled or none (require_labels), protecting the parallel-list construction in get_dataset()
  • no file-presence I/O of its own — reuses nobrainer.data.spec.validate(), so git-annex/DataLad checks apply unchanged

Reshaper (to_bids)

  • symlinks only, never copies
  • labels under derivatives/nobrainer/ as _dseg — dseg/probseg/mask are not valid raw-BIDS suffixes
  • dataset_description.json at both levels; dseg.tsv lookup table when labels named

CLI + deps

  • data group: nobrainer data from-bids / to-bids
  • new optional extra bids = ["pybids >= 0.19"] — only for backend="pybids", core paths need nothing

Test plan

  • 394 unit tests pass, no regressions
  • 46 new BIDS tests (43 + 3 skipped without pybids; parity test proves both backends emit identical entries)
  • from_bids() output feeds get_dataset() unmodified (real DataLoader batch)
  • annex-placeholder symlink included by scan, reported by validate() with the existing datalad get message
  • pre-commit clean
  • CI

dhritimandas and others added 7 commits April 10, 2026 16:52
@dhritimandas
feat(data): add DataSpec validation and inspection system
b607eda 
Add versioned dataset contract with DataLad-aware preflight validation:
- DataSpec dataclass with JSON serialization
- validate() with symlink-aware file checks, spacing/orientation/label validation
- inspect_entry() for NIfTI/Zarr metadata inspection
- CLI commands: validate, inspect, --skip-validate on predict/convert-to-zarr
- 33 unit tests covering all branches
@dhritimandas
fix(croissant): use official Croissant 1.0 JSON-LD context
a2d8c33 
- Replace inline @context dicts with CROISSANT_CONTEXT constant
- Change @type from cr:Dataset to sc:Dataset per Croissant 1.0 spec
- Add conformsTo and sha256 fields required by mlcroissant validation
- Update test assertions to match corrected @type
@dhritimandas
fix(tests): disable MPS when Conv3D unsupported
e8281aa
@dhritimandas
docs: add development guidelines to CLAUDE.md
f5b7580
@dhritimandas
chore: update .gitignore
fd69f6b
@satra
Merge branch 'alpha' into alpha
b85a1ca
@dhritimandas
feat(data): add BIDS discovery and reshaping
12806b3 
DataSpec (added in neuronets#376) could only be built from a hand-written JSON
manifest, so every BIDS dataset -- what OpenNeuro ships and what most
collaborators already have -- had to be converted by hand first. The
reverse was also missing: nobrainer's own outputs and ad-hoc file
collections couldn't be published as BIDS.

- nobrainer/data/bids.py: scan_bids() discovers image/label pairs from
  a BIDS(-Derivatives) tree; to_bids() reshapes arbitrary entries into
  a conformant tree. Entity grammar, directory rules, and derivative
  naming are pinned to BIDS 1.11.1.
- DataSpec.from_bids() classmethod, mirroring from_json().
- CLI: `nobrainer data from-bids` / `nobrainer data to-bids`.
- pyproject: new [bids] extra (pybids) -- optional, the internal
  walker needs no new dependencies and is what CI actually exercises
  (the workflow does not install pybids).

Design notes:
- Two discovery backends, one output schema. The zero-dependency
  walker is the default fallback and is fully tested on its own;
  pybids is a convenience layer with three documented traps guarded
  explicitly (it drops `subject` for a bare filename, coerces `run` to
  int destroying zero-padding, and uses different entity names than
  the BIDS schema).
- from_bids() never crashes on a single bad file -- every unusable
  file is recorded with a reason and the scan continues. It only
  raises if the root doesn't exist or nothing at all was found.
- Image/label entries are never a partial mix: either every returned
  entry has a label or none do, matching how the parallel image/label
  lists are built downstream.
- from_bids() does no file-presence I/O itself; it reuses
  nobrainer.data.spec.validate() for that, so git-annex/DataLad
  symlink checks apply to BIDS-sourced specs exactly as they do to
  JSON-manifest ones, with no duplicated logic.
- to_bids() writes symlinks only, never copies. Labels are written to
  a derivatives/ subtree, since dseg/probseg/mask are not valid raw
  BIDS suffixes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dhritimandas @satra