🚀 This package is part of
BIOMERO 2.0 — For complete deployment and FAIR infrastructure setup, start with the NL-BIOMERO Documentation 📖
OMERO scripts for exporting image data, running containerized workflows on Slurm, and importing results with recorded provenance. They work with the BIOMERO library and the OMERO.biomero web interface.
The scripts reference is published with BIOMERO, whose release series these scripts follow. Deployment and administration are documented in NL-BIOMERO.
- Scripts reference: installation, workflow execution, result options and logging.
- NL-BIOMERO deployment: containers, worker configuration and shared storage.
- Detached workflows and remote shallowing.
- Metadata administration: previewing and refreshing existing workflow annotations.
- Metadata developer reference: event-store views and the scripts persistence adapter.
| Directory | Entry points | Purpose |
|---|---|---|
__workflows/ |
SLURM_Run_Workflow.py, SLURM_Run_Workflow_Batched.py |
Run an analysis and retrieve results; optionally split inputs into batches. |
_data/ |
Image/File Transfer, Remote Conversion, Get Update | Export inputs, convert formats and monitor jobs. |
_data/ |
SLURM_Get_Results.py, SLURM_Import_Results.py |
Upload results through OMERO or import in place with BIOMERO.importer. |
admin/ |
Slurm Init, Check Setup, Shallow Storage Migration, Cownary, Tail Logs | Initialize and inspect the cluster, migrate managed storage metadata, maintain workflow metadata and diagnose execution. |
CellPose Segmentation and Example Minimal Slurm Script are examples, not the standard workflow entry points; NL-BIOMERO does not install them by default.
Use the released NL-BIOMERO containers for a coordinated installation. For a custom deployment, follow the scripts installation guide and install the scripts on the OMERO server and detached worker.
Deploy matching core and scripts releases together. This release requires core's
biomero.provenance and biomero.maintenance APIs even when optional features
are disabled. Prerelease tag counters can differ between repositories; use the
component references supplied by the NL-BIOMERO release.
Detached execution and shallow storage are opt-in for existing deployments:
missing or false feature flags leave them disabled. Within enabled shallow
storage, remote normalization is preferred; set BIOMERO_REMOTE_SHALLOW_ZARR=false
to use importer-side normalization. NL-BIOMERO's demo enables these features.
Result file attachments use the existing on-disk paths. Workflow-qualified attachment names are stored in OMERO metadata, so in-place image registrations and attachment links remain valid. Directory-backed Zarr stores are available through result registration or the bulk results ZIP; individual file attachments skip those directories and report the skipped count.
Shallow-result provenance uses the shared schema-2 manifest parser from
biomero-schema>=0.3.0b1,<0.4 (also required by the matching BIOMERO core).
Plate annotations collect the source identities from the manifest's image
bindings; Image annotations select their own image node. Both v2 and v3
interchange profiles are read without converting the store. Unsupported
manifest versions, inconsistent bindings or ambiguous references produce a
metadata warning rather than guessed provenance.
The import records storage evidence before rendering workflow annotations.
If this step or annotation persistence fails, the script's Message reports
WARNING with the complete/reduced/incomplete counts and points to the activity
log. The imported data remains available; a completed import does not imply
that all its metadata was written.
For an existing result whose storage evidence is missing, an administrator can
use backfill_import_storage_provenance() from the installed import script.
Run this in the worker's Python environment, with an authenticated administrator
conn in the result's group and the normal workflow event-store configuration:
import json
import runpy
from pathlib import Path
from types import SimpleNamespace
from biomero import WorkflowTracker
scripts_root = Path("/opt/omero/server/OMERO.server/lib/scripts/biomero")
adapter = runpy.run_path(str(scripts_root / "_data/SLURM_Import_Results.py"))
backfill = adapter["backfill_import_storage_provenance"]
maintenance = runpy.run_path(str(scripts_root / "admin/SLURM_Init_environment.py"))
refresh = maintenance["refresh_workflow_metadata"]
# Select the existing result and the workflow that imported it.
result_type, result_id, workflow_id = "Plate", 123, "your-workflow-uuid"
with WorkflowTracker() as tracker:
client = SimpleNamespace(workflowTracker=tracker)
preview = backfill(conn, client, result_type, result_id, workflow_id)
print(json.dumps(preview, indent=2))
# After reviewing the preview, append its evidence:
# backfill(conn, client, result_type, result_id, workflow_id, dry_run=False)
# After recording, preview the corresponding OMERO annotation changes:
# plan = refresh(conn, tracker, result_type, result_id, workflow_id,
# refresh_storage=True)
# print(json.dumps(plan, indent=2))
# Apply with a new backup filename after reviewing that plan:
# refresh(conn, tracker, result_type, result_id, workflow_id,
# refresh_storage=True, dry_run=False,
# backup_path="/data/biomero-metadata-backups/result-before-repair.json")The default is a read-only preview. Apply validates the matching workflow annotation, managed storage evidence and unique import task, then appends the observed evidence to the event store. Identical evidence is not recorded twice. Missing files, mismatched workflows, conflicting recorded evidence and ambiguous results must be resolved before applying. Source and result pixels are never read, rewritten or renormalized by this operation.
The explicit refresh_storage=True option requires a coordinated core build
whose plan_metadata_refresh() supports that option. It advances only the
import-task snapshot needed to expose newly recorded storage evidence, including
its paired job maps. Other task snapshots remain pinned. Normal metadata
refresh, including the Slurm Init UI, preserves the old snapshots and therefore
does not expose later evidence by itself. See
metadata administration
for backup and refresh controls. Recording evidence and refreshing maps are two
separate steps. If refresh fails, retry it; repeating the backfill is also safe.
Tests live on the separate test-suite branch, so they are not exposed as scripts by OMERO. CI runs that harness against the pull request's source revision. See its README for local test instructions.
See LICENSE. These scripts build on OME's copyleft-licensed scripts.
This section provides machine-readable information about your scripts. It will be used to help generate a landing page and links for your work. Please modify all values on each branch to describe your scripts.
BIOMERO.scripts repository
5.6
5.6
T.T. Luik
Amsterdam UMC
https://nl-bioimaging.github.io/biomero/
These scripts are to be used with the BIOMERO library.
They show how to use the library to run workflows directly from OMERO on a Slurm cluster.