Doc software

docs/fair.md

@@ -11,6 +11,7 @@ Practical recommendations include:
 * Assign persistent identifiers to datasets and published outputs.
 * Maintain a consistent project structure so that raw data, processed data, analyses, and documentation can be located easily.
 * Include metadata describing relationships between datasets, workflows, and derived results.
+* Use NGFF OME-Zarr format which includes standardized metadata, also supporting efficient metadata exploration for large datasets.
 * Use RO-Crate metadata as a single entry point to datasets, workflows, software, and documentation.
 
 Relevant resources:

docs/green.md

@@ -1 +1,36 @@
-# Green computing
+# Green computing
+
+As part of efficient use of computing resources, and accountability for the environmental impact of computing, it is possible to measure the carbon footprint of heavy compute operations.
+General recommendations on this are provided by the Green Algorithms initiative, see the link below for more information.
+Although there are currently many tools to measure carbon footprint of compute, we explore the tools provided by the Cambridge Sustainable group.
+
+We use the Green Algorithms for High Performance Computing (**GA4HPC**) software, as this is easily deployable on HPC clusters, supports SLURM, and can be retrospectively used without the need for pre-installed software or other components running on the cluster.
+
+As a test case, we use [muvis-align](muvis-align.md), software for multi-view registration of microscopy images, and we measure the carbon footprint of the registration of a large dataset.
+
+Specific details on operation and dataset:
+* Operation: X/Y stitching of 2D tiles: registration and fusion including export of registration information and fused image data in OME-Zarr format
+* Source data: 64 tiles (OME-Zarr format) of 6400 x 6400 x 16 bit = ~5GB
+* Resulting fused data: 50438 x 50502 x 16 bit = ~5GB
+
+HPC hardware setup:
+* CPU nodes: Dell PowerEdge R6525
+* Two 2nd/3rd Generation AMD EPYCTM Processors with 64 cores per processor
+* 280W for the CPU nodes processors (for 64 cores)
+
+Power consumption / carbon factors:
+* Power Usage Effectiveness (PUE):
+* Average carbon intensity: 137 gCO2e/kWh
+
+Results from GA4HPC:
+* Energy used: 0.28 kWh
+* CPU: 0 days 01:00:19 (1 hours)
+* Total wallclock time: 0 days 00:41:13
+* 38 gCO2e
+
+
+Relevant resources:
+
+* Green Algorithms Initiative, Towards environmentally sustainable computational science: https://www.green-algorithms.org/
+* GA4HPC tool: [github.com/Cambridge-Sustainable-Computing-Lab/GreenAlgorithms4HPC](https://github.com/Cambridge-Sustainable-Computing-Lab/GreenAlgorithms4HPC)
+* Green DiSC https://www.software.ac.uk/GreenDiSC, https://github.com/Cambridge-Sustainable-Computing-Lab/greenDiSC

docs/index.md

@@ -1,5 +1,5 @@
 # CCP-volumeEM guide
 
-Community good practice guide
+Community guide of recommended best practices
 
 ![logo](images/logo.png)

docs/muvis-align.md

@@ -128,8 +128,59 @@ Rather than only documenting FAIR principles, `muvis-align` implements concrete
 
 Source: [`src/muvis_align/ui/project_template.yaml`](https://github.com/ccp-volume-em/muvis-align/blob/main/src/muvis_align/ui/project_template.yaml)
 
-The project_template.yaml file is written in the Bilayers format and serves as the declarative description of the muvis-align workflow. The file is used to dynamically generate the [magicgui](https://pyapp-kit.github.io/magicgui/)-based [napari](https://napari.org/stable/) user interface, allowing workflow parameters, input definitions, and processing options to be defined in a structured configuration file rather than being hard-coded in the application. This approach simplifies interface maintenance, improves reproducibility, and enables the same workflow definition to be reused across the [napari](https://napari.org/stable/) plugin, command-line execution, and containerized deployments.
+The project_template.yaml file is written in the Bilayers format and serves as the declarative description of the muvis-align workflow.
+The file is used to dynamically generate the [magicgui](https://pyapp-kit.github.io/magicgui/)-based [napari](https://napari.org/stable/) user interface, allowing workflow parameters, input definitions, and processing options to be defined in a structured configuration file rather than being hard-coded in the application.
+This approach simplifies interface maintenance, improves reproducibility, and enables the same workflow definition to be reused across the [napari](https://napari.org/stable/) plugin, command-line execution, and containerized deployments.
+Bilayers schemas are represented as a [Pydantic](https://docs.pydantic.dev/) model, which adds typed validation and structured serialization for this workflow definition layer.
 
+Source: [`src/muvis_align/ui/bilayers_util.py`](https://github.com/ccp-volume-em/muvis-align/blob/main/src/muvis_align/ui/bilayers_util.py)
+
+```python
+from pydantic import BaseModel, ConfigDict, Field, field_validator
+
+
+class BilayersField(BaseModel):
+    model_config = ConfigDict(extra='allow')
+
+    name: str
+    type: str
+    label: str | None = None
+    default: Any = None
+    description: str | None = None
+    section_id: str | None = None
+    section_key: str | None = None
+    cli_order: int = 0
+
+    @field_validator('type', mode='before')
+    @classmethod
+    def _normalize_type(cls, value):
+        return str(value).lower()
+
+    @field_validator('cli_order', mode='before')
+    @classmethod
+    def _normalize_cli_order(cls, value):
+        if value is None or value == '':
+            return 0
+        return int(value)
+
+
+def get_section_dict(template, keys=None):
+    sections = {}
+    if keys is None:
+        keys = [None]
+
+    for key in keys:
+        items = template.get(key, []) if key is not None else template
+        for raw_item in items:
+            item = BilayersField.model_validate({**raw_item, 'section_key': key})
+            section_id = item.section_id or key
+            sections.setdefault(section_id, []).append(item)
+
+    return {
+        section_id: [item.model_dump() for item in sorted(section_items, key=lambda item: item.cli_order)]
+        for section_id, section_items in sections.items()
+    }
+```
 
 The package creates RO-Crates for outputs and adds acquisition/workflow metadata.
 
@@ -183,6 +234,28 @@ def create_zarr_ro_crate(dest_path):
 
 Input metadata is read from OME-TIFF or OME-Zarr and converted into physical units, positions, rotations, channels, scales, and transforms.
 
+### Interoperable NGFF RFC 6 transforms
+
+Registration transforms are written in the [OME-NGFF RFC 6 coordinate-transform format](https://github.com/ome/ngff-spec/blob/main/index.md#coordinateTransformations-metadata) rather than as application-specific matrix arrays. Each affine identifies its image path and names its input (`source_metadata`) and output (`registered`) coordinate systems. Using the `ome-zarr-models` v0.6 model for both serialization and validation makes the saved transforms portable to other tools that implement the NGFF coordinate-system and transformation model.
+
+Source: [`src/muvis_align/file/transforms.py`](https://github.com/ccp-volume-em/muvis-align/blob/main/src/muvis_align/file/transforms.py)
+
+```python
+def metadata_models(path, transform, source_key='source_metadata', transform_key='registered'):
+    affine = Affine(
+        path=path,
+        affine=transform,
+        input=CoordinateSystemIdentifier(name=source_key),
+        output=CoordinateSystemIdentifier(name=transform_key),
+    )
+    return affine.model_dump(exclude_none=True, exclude_defaults=True, exclude_unset=True)
+
+
+def read_transforms(filename):
+    transforms = import_json(filename)
+    return {label: Affine.model_validate(transform).affine for label, transform in transforms.items()}
+```
+
 Source: [`src/muvis_align/image/ZarrDaskSource.py`](https://github.com/ccp-volume-em/muvis-align/blob/main/src/muvis_align/image/ZarrDaskSource.py)
 
 ```python
@@ -491,7 +564,8 @@ CMD echo "Launching napari on Xpra. Connect via http://localhost:$XPRA_PORT or $
 | NGFF                    | OME-Zarr default extension, NGFF v0.5, `write_sim_to_ome_zarr`              |
 | Larger-than-memory data | Dask-backed TIFF and Zarr sources, chunked fusion, direct Zarr output       |
 | FAIR implementation     | package metadata, YAML configuration, RO-Crate generation, OME-Zarr outputs |
-| Persistent metadata     | OME metadata parsing, `xarray`/`MSIM` spatial transforms, positions, scales, channels |
+| Persistent metadata     | OME metadata parsing, `xarray`/`MSIM` spatial transforms, positions, scales, channels; interoperable NGFF RFC 6 affine transforms |
 | Performance metrics     | NCC, SSIM, ONMI, MSE-derived score, match metrics, timing logs              |
 | Resumable workflow      | state machine, pair mappings, global mappings, metrics JSON                 |
 | Xpra interface          | Docker stage exposing browser-based napari over Xpra                        |
+

docs/software.md

@@ -47,9 +47,16 @@ We also recommend to make this as efficient as possible considering limited reso
 There are tools for automated dependency updates, such as [Dependabot](https://github.com/dependabot) and [Renovate](https://docs.renovatebot.com/), including taking care of security vulnerabilities.
 
 ## Support
-Besides Github Issues, for further user support we recomment connecting to a single suitable community forum (e.g. [image.sc](https://forum.image.sc/) for image analysis related software) and providing a link to this forum in the documentation.
+Besides Github Issues, for further user support we recommend connecting to a single suitable community forum (e.g. [image.sc](https://forum.image.sc/) for image analysis related software) and providing a link to this forum in the documentation.
 This also allows the community to support each other and share knowledge.
 
+## AI use
+
+Modern software development increasingly involves generative-AI tools, which can significantly speed up development cycles. Their use must be weighed against risks including poor code quality, increased technical debt and reduced code comprehension. 
+When used, AI-generated code should be reviewed and tested — ideally by independently and human written tests — to ensure it is correct and maintainable, and the use of generative-AI should be acknowledged in the documentation as well as relevant commits.
+
+We suggest reading the Institue for Research Software's [Responsible use of AI in research software](https://www.software.ac.uk/guide/responsible-use-ai-research-software) before using generative-AI in your projects.
+
 ## Further resources
 - A collection of resources for python tooling: [ARC python tooling](https://github-pages.arc.ucl.ac.uk/python-tooling/)
 - A comprehensive guide to reproducible data science, including software development and documentation: [Turing way book](https://book.the-turing-way.org)
@@ -58,4 +65,3 @@ This also allows the community to support each other and share knowledge.
 - Documentation on using Docker for containerization: [Docker documentation](https://docs.docker.com/)
 
 - CCP-volumeEM: Good practices for image analysis software with Dr Kimberly Meechan [YouTube video](https://www.youtube.com/watch?v=HHJJZ79KC4Y)
-