Skip to content

use shards when writing ome zarr datasets#176

Merged
akhanf merged 8 commits into
mainfrom
shards
Jun 11, 2026
Merged

use shards when writing ome zarr datasets#176
akhanf merged 8 commits into
mainfrom
shards

Conversation

@akhanf

@akhanf akhanf commented Jun 10, 2026

Copy link
Copy Markdown
Member

instead of ozx

Also increase mem and resources for jobs

@akhanf akhanf requested a review from Copilot June 10, 2026 16:48
@akhanf akhanf marked this pull request as ready for review June 10, 2026 16:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates SPIMquant’s Snakemake workflow to write segmentation-derived datasets as directory-based OME-Zarr stores (rather than .ozx zipstores), with shared writer kwargs intended to enable chunking/sharding, and adjusts resource settings for several jobs. It also expands input discovery to include Imaris .ims files.

Changes:

  • Switch many mask/dist outputs from *.ozx to *.ome.zarr and mark most as directory(...) outputs in rules.
  • Standardize OME-Zarr writing in multiple scripts via match_scale_factors_from=... and **config["zarrnii_out_kwargs"].
  • Increase memory/resources for some rules and add .ims as an allowed SPIM input extension.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
spimquant/workflow/scripts/vesselfm.py Writes vessel masks to .ome.zarr with scale-factor matching and shared writer kwargs.
spimquant/workflow/scripts/threshold.py Writes threshold masks to .ome.zarr with scale-factor matching and shared writer kwargs.
spimquant/workflow/scripts/skeletonize_vessels_mask.py Writes skeleton masks to .ome.zarr with scale-factor matching and shared writer kwargs.
spimquant/workflow/scripts/signed_distance_transform.py Writes SDT outputs to .ome.zarr with scale-factor matching and shared writer kwargs.
spimquant/workflow/scripts/n4_biasfield.py Adds .ims-specific execution behavior for N4 processing.
spimquant/workflow/scripts/gmmthresh.py Writes GMM threshold masks with shared writer kwargs.
spimquant/workflow/scripts/clean_segmentation.py Writes cleaned masks with scale-factor matching and shared writer kwargs.
spimquant/workflow/rules/vessels.smk Converts vessel intermediates to .ome.zarr directories and adjusts threads/mem/runtime.
spimquant/workflow/rules/templatereg.smk Updates some template-registration outputs to .ome.zarr and increases memory for selected rules.
spimquant/workflow/rules/segmentation.smk Converts segmentation outputs to .ome.zarr directories.
spimquant/workflow/rules/regionprops.smk Updates inputs to consume .ome.zarr masks/dist outputs.
spimquant/workflow/rules/qc.smk Updates QC rules to consume .ome.zarr masks.
spimquant/workflow/rules/patches.smk Updates patch generation inputs to consume .ome.zarr masks.
spimquant/workflow/rules/heatmaps.smk Increases memory for heatmap generation rules.
spimquant/workflow/rules/fieldfrac.smk Updates inputs to consume .ome.zarr masks and increases memory for mapping step.
spimquant/workflow/rules/common.smk Minor formatting-only change.
spimquant/config/snakebids.yml Adds .ims input extension support and introduces zarrnii_out_kwargs for writer configuration.

Comment thread spimquant/workflow/rules/templatereg.smk Outdated
Comment thread spimquant/workflow/rules/templatereg.smk Outdated
Comment thread spimquant/workflow/scripts/n4_biasfield.py Outdated
Comment thread spimquant/workflow/scripts/n4_biasfield.py Outdated
Comment thread spimquant/workflow/scripts/n4_biasfield.py
akhanf and others added 5 commits June 11, 2026 09:45
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Added additional parameters to the to_ome_zarr function call.
@akhanf akhanf merged commit f7079e9 into main Jun 11, 2026
2 checks passed
@akhanf akhanf deleted the shards branch June 11, 2026 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants