From ccbfdd8ce30e431e39097b20a182cf9f8d99a65e Mon Sep 17 00:00:00 2001 From: Joost de Folter Date: Wed, 29 Jul 2026 13:48:00 +0200 Subject: [PATCH 01/13] Added software publishing --- docs/software_publishing.md | 206 ++++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 207 insertions(+) create mode 100644 docs/software_publishing.md diff --git a/docs/software_publishing.md b/docs/software_publishing.md new file mode 100644 index 0000000..b1b4433 --- /dev/null +++ b/docs/software_publishing.md @@ -0,0 +1,206 @@ +# Making software public + +Publishing research software is more than making a repository visible. A useful +public release should be installable, documented, licensed, citable and free of +information that should remain private. The recommendations below assume that +the software has already been tested, reviewed and versioned as described in +[Software Development](software.md). + +## Before publishing + +Review the complete repository history as well as the current files. Removing a +secret from the latest commit does not remove it from earlier commits. + +Before the first public release: + +* Confirm that all contributors and, where relevant, their employers or funders + agree that the software can be released. +* Remove credentials, access tokens, private URLs, personal data, patient data, + confidential issue discussions and data that you do not have permission to + redistribute. Revoke and replace any credential that has ever been committed. +* Check the licences of source code, models, example data, fonts, icons and + other third-party material. Record their copyright and attribution notices. +* Add a `README`, a licence, installation and usage instructions, contributor + guidance, a citation file and a way to report problems. +* Test installation in a clean environment and run the automated test suite. + Test the built package, not only the source checkout. +* Choose a release version and prepare release notes describing important + changes, known limitations and any compatibility breaks. + +Avoid committing large datasets, model weights and generated build files to the +source repository. Deposit large research outputs in an appropriate public repository +and link to them using a persistent identifier. + +## GitHub + +A public [GitHub repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/about-repositories) +makes the source and its history discoverable and gives users a place to report +issues and contribute changes. Before changing a private repository to public, +inspect branches, tags, pull requests, Actions logs and release assets: some of +these also become visible. GitHub documents the +[consequences of changing repository visibility](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/managing-repository-settings/setting-repository-visibility). + +Suggested contents for the repository: + +* `README.md`: purpose, status, supported platforms, installation, a minimal + example, links to full documentation, support and citation instructions. +* `LICENSE` or `LICENSE.md`: the full licence text. +* `CITATION.cff`: title, authors, ORCID identifiers, version, release date and + DOI, when available. GitHub uses this file to display a **Cite this + repository** link; see its + [citation-file guidance](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-citation-files). +* `CONTRIBUTING.md`: how to set up a development environment, run tests and + submit a change. +* `CODE_OF_CONDUCT.md`: expected behaviour and a route for confidential + reporting. +* `SECURITY.md`: supported versions and how to report a vulnerability without + first disclosing it publicly. +* A dependency or environment specification, with direct dependencies and + supported language versions declared accurately. + +Use an organisation-owned repository for collaborative projects when possible, +so that stewardship does not depend on one person's account. Give at least two +maintainers the permissions needed to make a release and enable multi-factor +authentication and branch protection. + +### Create a release + +Create a release from a reviewed commit on the default branch: + +1. Update the version, changelog, citation metadata and documentation. +2. Run tests and build the documentation and distribution artefacts in a clean + environment. +3. Create an annotated Git tag using the project's version convention, for + example `v1.2.0`. +4. Create a GitHub Release from that tag and add release notes. +5. Publish packages and container images from the same tag, then test that + users can install them using the documented commands. + +[GitHub Releases](https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases) +associate a fixed Git tag with release notes and downloadable assets. Do not +move or reuse a published tag. If a release is faulty, publish a new patch +release and explain the problem. + +For long-term preservation and citation, connect the repository to a research +archive such as [Zenodo](https://help.zenodo.org/docs/github/). Archive each +release and add the resulting version-specific DOI to `CITATION.cff`. A concept +DOI can refer to the software as a whole, while a version DOI identifies the +exact software used in an analysis; prefer the version DOI when reproducibility +requires a particular release. + +## PyPI package + +Publishing a Python library or application on the +[Python Package Index (PyPI)](https://pypi.org/) allows users to install it with +`pip`. Follow the maintained +[Python Packaging User Guide](https://packaging.python.org/en/latest/tutorials/packaging-projects/) +rather than invoking `setup.py` directly. + +Define the build system and package metadata in a single definition using `pyproject.toml`. Metadata should +include: + +* a unique distribution name and version; +* a short description and `README` as the long description; +* the licence expression and licence files; +* required Python version and runtime dependencies; +* authors or maintainers; +* project URLs for source, documentation, issues and changelog; +* classifiers and optional dependencies where useful. + +Build both a source distribution and a wheel, then check their metadata: + +```console +python -m pip install --upgrade build twine +python -m build +python -m twine check --strict dist/* +``` + +Inspect the archive contents and install the wheel into a new virtual +environment. If the project has compiled extensions, provide wheels for the +supported combinations of operating system, architecture and Python version, +and test each build. + +Use [TestPyPI](https://packaging.python.org/en/latest/guides/using-testpypi/) +to exercise a new packaging workflow before the production release. PyPI does +not permit a published file or version to be replaced, so corrections require a +new version. + +For automated releases, prefer +[PyPI Trusted Publishing](https://docs.pypi.org/trusted-publishers/). It uses +short-lived identity tokens from the CI service instead of storing a long-lived +PyPI token in repository secrets. Restrict the publishing job to protected tags +or a protected release environment, and require review where appropriate. +Build once, test those exact artefacts, and promote the same files to the +production index. + +After publishing, verify the package page and test the normal user workflow: + +```console +python -m pip install your-package +python -c "import your_package" +``` + +## Licence + +Copyright exists even when no licence file is present. A public repository +without a licence can be viewed and forked under the hosting platform's terms, +but that does not generally give others permission to use, modify or distribute +the software. See [Choose a License](https://choosealicense.com) +for an accessible explanation. + +Choose the licence before accepting external contributions, and obtain agreement +from all copyright holders. Consider: + +* compatibility with the licences of dependencies and incorporated code; +* whether derived works may be distributed under other terms (a permissive + licence) or must remain open under compatible terms (a copyleft licence); +* requirements imposed by an employer, funder, consortium or target ecosystem; +* patent provisions, attribution requirements and the intended commercial use. + +Common choices include BSD-3-Clause, MIT and Apache-2.0 as permissive licences, +and GPL-3.0 as a strong copyleft licence. This list is not legal advice; consult +your institution's research software or legal team when ownership or licence +compatibility is unclear. Use a standard +[OSI-approved licence](https://opensource.org/licenses) without changing its +text, and include its exact SPDX identifier in package metadata. + +A repository may need separate licences for software, documentation, data and +model weights. State clearly which files each licence covers. Creative Commons +licences may suit documentation and data but are +[not recommended for software](https://choosealicense.com/non-software/). +Retain third-party notices and make sure example data can legally be published. + +## Statements about AI use + +Generative-AI tools may influence source code, tests, documentation, images or +translations. The human contributors and maintainers remain responsible for +everything released: AI systems should not be listed as authors or copyright +holders. + +Follow the policies of your institution, funder, publisher and software +community. When AI use was substantive, add a short statement to the repository, +release notes or associated paper. Record enough information to make the role of +the tool understandable, without publishing confidential prompts or sensitive +data. A statement can identify: + +* the tool and model, including the version or access date where known; +* which parts of the work it assisted with; +* whether generated output was modified; +* how a human reviewed, tested and validated the result. + +For example: + +> Generative AI was used to suggest docstrings and unit-test cases. All +> suggestions were reviewed and edited by the maintainers, and the resulting +> software passed the project's automated tests. No confidential source code or +> research data was provided to the service. + +If no generative-AI tool contributed to the release, a statement is normally +unnecessary unless a relevant policy explicitly requires one. + +Do not submit personal, confidential, security-sensitive or unpublished +material to an external AI service unless its data handling terms and your +project approvals permit this. Treat generated code like any other external +contribution: understand it, review it for security and bias, test it, and check +for licence or attribution concerns before release. AI output can be plausible +but incorrect, and an AI-use statement does not replace quality assurance. diff --git a/mkdocs.yml b/mkdocs.yml index cac52bb..f89b753 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -20,6 +20,7 @@ nav: - fair.md - green.md - software.md + - software_publishing.md - plugins.md - plugin_testing.md - container_testing.md From c4446011f5a762b58622dc5a1f09c15d6fea8c06 Mon Sep 17 00:00:00 2001 From: Joost de Folter Date: Wed, 29 Jul 2026 14:06:54 +0200 Subject: [PATCH 02/13] including 2 other docs --- docs/muvis-align.md | 1 + docs/software.md | 1 - 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/muvis-align.md b/docs/muvis-align.md index cd2991d..0d3b8a9 100644 --- a/docs/muvis-align.md +++ b/docs/muvis-align.md @@ -495,3 +495,4 @@ CMD echo "Launching napari on Xpra. Connect via http://localhost:$XPRA_PORT or $ | Performance metrics | NCC, SSIM, ONMI, MSE-derived score, match metrics, timing logs | | Resumable workflow | state machine, pair mappings, global mappings, metrics JSON | | Xpra interface | Docker stage exposing browser-based napari over Xpra | + diff --git a/docs/software.md b/docs/software.md index 75b3210..3f61a2c 100644 --- a/docs/software.md +++ b/docs/software.md @@ -58,4 +58,3 @@ This also allows the community to support each other and share knowledge. - Documentation on using Docker for containerization: [Docker documentation](https://docs.docker.com/) - CCP-volumeEM: Good practices for image analysis software with Dr Kimberly Meechan [YouTube video](https://www.youtube.com/watch?v=HHJJZ79KC4Y) - From b650d110239e83e3a4d99e46c2ce49929a62ed97 Mon Sep 17 00:00:00 2001 From: Joost de Folter Date: Wed, 12 Aug 2026 15:43:46 +0200 Subject: [PATCH 03/13] Added containerisation, reshuffled pypi --- docs/software_publishing.md | 264 +++++++++++++++++++++++++++++------- 1 file changed, 212 insertions(+), 52 deletions(-) diff --git a/docs/software_publishing.md b/docs/software_publishing.md index b1b4433..8a53e8c 100644 --- a/docs/software_publishing.md +++ b/docs/software_publishing.md @@ -88,58 +88,6 @@ DOI can refer to the software as a whole, while a version DOI identifies the exact software used in an analysis; prefer the version DOI when reproducibility requires a particular release. -## PyPI package - -Publishing a Python library or application on the -[Python Package Index (PyPI)](https://pypi.org/) allows users to install it with -`pip`. Follow the maintained -[Python Packaging User Guide](https://packaging.python.org/en/latest/tutorials/packaging-projects/) -rather than invoking `setup.py` directly. - -Define the build system and package metadata in a single definition using `pyproject.toml`. Metadata should -include: - -* a unique distribution name and version; -* a short description and `README` as the long description; -* the licence expression and licence files; -* required Python version and runtime dependencies; -* authors or maintainers; -* project URLs for source, documentation, issues and changelog; -* classifiers and optional dependencies where useful. - -Build both a source distribution and a wheel, then check their metadata: - -```console -python -m pip install --upgrade build twine -python -m build -python -m twine check --strict dist/* -``` - -Inspect the archive contents and install the wheel into a new virtual -environment. If the project has compiled extensions, provide wheels for the -supported combinations of operating system, architecture and Python version, -and test each build. - -Use [TestPyPI](https://packaging.python.org/en/latest/guides/using-testpypi/) -to exercise a new packaging workflow before the production release. PyPI does -not permit a published file or version to be replaced, so corrections require a -new version. - -For automated releases, prefer -[PyPI Trusted Publishing](https://docs.pypi.org/trusted-publishers/). It uses -short-lived identity tokens from the CI service instead of storing a long-lived -PyPI token in repository secrets. Restrict the publishing job to protected tags -or a protected release environment, and require review where appropriate. -Build once, test those exact artefacts, and promote the same files to the -production index. - -After publishing, verify the package page and test the normal user workflow: - -```console -python -m pip install your-package -python -c "import your_package" -``` - ## Licence Copyright exists even when no licence file is present. A public repository @@ -204,3 +152,215 @@ project approvals permit this. Treat generated code like any other external contribution: understand it, review it for security and bias, test it, and check for licence or attribution concerns before release. AI output can be plausible but incorrect, and an AI-use statement does not replace quality assurance. + + +## PyPI package + +Publishing a Python library or application on the +[Python Package Index (PyPI)](https://pypi.org/) allows users to install it with +`pip`. Follow the maintained +[Python Packaging User Guide](https://packaging.python.org/en/latest/tutorials/packaging-projects/) +rather than invoking `setup.py` directly. + +Define the build system and package metadata in a single definition using `pyproject.toml`. Metadata should +include: + +* a unique distribution name and version; +* a short description and `README` as the long description; +* the licence expression and licence files; +* required Python version and runtime dependencies; +* authors or maintainers; +* project URLs for source, documentation, issues and changelog; +* classifiers and optional dependencies where useful. + +Build both a source distribution and a wheel, then check their metadata: + +```console +python -m pip install --upgrade build twine +python -m build +python -m twine check --strict dist/* +``` + +Inspect the archive contents and install the wheel into a new virtual +environment. If the project has compiled extensions, provide wheels for the +supported combinations of operating system, architecture and Python version, +and test each build. + +Use [TestPyPI](https://packaging.python.org/en/latest/guides/using-testpypi/) +to exercise a new packaging workflow before the production release. PyPI does +not permit a published file or version to be replaced, so corrections require a +new version. + +For automated releases, prefer +[PyPI Trusted Publishing](https://docs.pypi.org/trusted-publishers/). It uses +short-lived identity tokens from the CI service instead of storing a long-lived +PyPI token in repository secrets. Restrict the publishing job to protected tags +or a protected release environment, and require review where appropriate. +Build once, test those exact artefacts, and promote the same files to the +production index. + +After publishing, verify the package page and test the normal user workflow: + +```console +python -m pip install your-package +python -c "import your_package" +``` + +## Containerisation + +For maximal reproducibility, a key [FAIR](fair.md) principle, consider creating a +container definition that installs the software and its dependencies. A container +image bundles the interpreter, libraries, system packages and configuration that +the software needs, so users obtain the same environment as the developers. This +is particularly valuable for volume EM tools, which often combine Python +packages, compiled libraries, GPU frameworks and command-line utilities that are +difficult to install consistently across operating systems. + +Containerisation complements, rather than replaces, a conventional package +release. Publish a [PyPI package](#pypi-package) or conda package for users who +want to work inside their own environment, and a container image for users who +want a fixed, ready-to-run environment or who work on shared infrastructure. + +For an introduction to containers in this context, see +[Using Containers](https://rosalindfranklininstitute.github.io/volume-em-container-documentation/intro/containers/). + +### Write the container definition + +Keep the container definition, for example a `Dockerfile`, in the source +repository so that it is versioned and reviewed alongside the code. Build the +image from a released artefact or a specific commit rather than from uncommitted +local files. + +Recommendations for the definition: + +* Start from a minimal, actively maintained base image and record why it was + chosen. Reuse an established base image, such as an official Python or CUDA + image, instead of building an environment from scratch. +* Pin what you depend on: the base image tag, and ideally its digest; the + system packages; and the versions of the software and its dependencies. An + unpinned build produces a different environment every time it runs. +* Install the software as a package (`pip install .` or an equivalent) rather + than copying source files into an arbitrary directory, so that version + metadata and entry points are available inside the image. +* Use multi-stage builds to keep build tools out of the final image, and remove + package-manager caches in the same layer that creates them. +* Run the application as a non-root user, and document any writable directories + it needs. +* Define an `ENTRYPOINT` that runs the application, so that + `docker run --help` behaves as users expect. +* Add [OCI image labels](https://github.com/opencontainers/image-spec/blob/main/annotations.md) + such as `org.opencontainers.image.source`, `.version`, `.revision`, + `.licenses` and `.description`. These make an image traceable back to the + exact commit that produced it. +* Keep large data, model weights and test datasets out of the image where + possible; download or mount them at run time and document how. + +Graphical applications, such as [napari](https://napari.org/) plugins, need +additional consideration: the container must be able to reach a display server, +either through the host's X11 socket or through a browser-based server such as +Xpra. See [`muvis-align`](muvis-align.md#interactive-interface-with-xpra-container) +for a worked example of a napari image with an Xpra-enabled variant. + +### Build and test in continuous integration + +A container image can be built and tested in a continuous integration workflow, +which ensures the definition keeps working as dependencies change. At minimum, +build the image on pull requests and on release tags, and check that it starts +and reports its version without error. + +For containerised research software we recommend: + +* Building on every change to the container definition or dependency + specification, and on a schedule to detect breakage caused by upstream + updates. +* Running the test suite inside the built image, using a virtual display server + such as `Xvfb` where a GUI is involved. +* Adding start-up or installation checks to the entrypoint, which warn the user + when the runtime environment lacks something the application needs, such as a + display, GPU access or a writable bind mount. +* Building for the architectures you intend to support, for example `amd64` and + `arm64`, and testing each one. + +See [Container Testing](container_testing.md) for a fuller discussion and +examples, including the start-up tests written for the napari Empanada plugin. + +### Tag and publish + +A container image can be built, properly tagged, and published to a public +registry such as [Docker Hub](https://hub.docker.com/), +[Quay.io](https://quay.io/) or the +[GitHub Container Registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry). +Consider where your user community already looks for images, whether the +registry applies pull-rate limits or retention policies, and whether it will +still be available in several years. + +The tags follow the same guidelines as release versioning. Likewise, a specific +tag can be referenced for reproducibility. A practical tagging scheme is: + +* an immutable version tag matching the Git tag, for example `v1.2.0`, published + once and never moved; +* optional moving tags such as `v1.2`, `v1` or `latest` for convenience; +* variant suffixes where several images are built from the same release, for + example `v1.2.0-cuda` or `v1.2.0-xpra`. + +Publish the image from the same tag as the corresponding +[GitHub Release](#create-a-release), and record the version in the image labels. +Where the registry supports it, generate build provenance and a software bill of +materials, so users can verify what an image contains and where it came from. + +Automate publishing from CI using the registry's OpenID Connect or short-lived +credentials where available, rather than storing long-lived registry passwords in +repository secrets, and restrict the publishing job to protected tags or a +protected release environment. + +### Document how to run the image + +Include the container definition in the repository and provide instructions for +users to run it. Give a complete command that a new user can copy, and explain +each option that is not obvious: + +```console +docker pull ghcr.io/your-org/your-tool:1.2.0 +docker run --rm -v /path/to/data:/data ghcr.io/your-org/your-tool:1.2.0 --help +``` + +Document, as applicable: + +* how to mount input and output data, and which paths the image expects; +* how to request GPU access, for example `--gpus all`, and which driver and + CUDA versions are required; +* how to run a graphical interface, including any `DISPLAY` or X11 socket + settings; +* the user and file-permission behaviour, for example `--user $(id -u):$(id -g)` + when written files should be owned by the invoking user; +* resource expectations, such as memory, shared memory or disk space for large + volumes. + +### Shared infrastructure and archiving + +Many users run analyses on HPC clusters, where Docker is usually unavailable +because it requires elevated privileges. Container runtimes such as +[Apptainer](https://apptainer.org/) (formerly Singularity) are commonly +installed instead, and can convert and run a published OCI image directly: + +```console +apptainer pull your-tool_1.2.0.sif docker://ghcr.io/your-org/your-tool:1.2.0 +apptainer run --bind /path/to/data:/data your-tool_1.2.0.sif --help +``` + +Test this route if you expect cluster use, since bind-mount behaviour, the home +directory, environment variables and the container user differ from Docker. + +For work that must remain reproducible in the long term, reference images by +digest rather than by tag, since a digest identifies exact image content: + +```console +docker run --rm your-tool@sha256: +``` + +Registries are not archives. Where a published analysis depends on a specific +image, record its digest in the methods or workflow description, and consider +depositing the container definition, the resolved dependency versions and, if +size permits, an exported image archive alongside the release in a research +archive such as [Zenodo](https://zenodo.org/). A container definition that can +be rebuilt from pinned inputs is often more durable than a stored image. From ba8964b6b12ab3fcab791076839501ea2c488437 Mon Sep 17 00:00:00 2001 From: Joost de Folter Date: Mon, 24 Aug 2026 13:39:37 +0200 Subject: [PATCH 04/13] Added docs based on suggestions from @piperfw , added on pydantic model use for bilayers --- docs/fair.md | 1 + docs/muvis-align.md | 53 ++++++++++++++++++++++++++++++++++++++++++++- docs/software.md | 5 +++++ 3 files changed, 58 insertions(+), 1 deletion(-) diff --git a/docs/fair.md b/docs/fair.md index cbb7dd7..4076a53 100644 --- a/docs/fair.md +++ b/docs/fair.md @@ -11,6 +11,7 @@ Practical recommendations include: * Assign persistent identifiers to datasets and published outputs. * Maintain a consistent project structure so that raw data, processed data, analyses, and documentation can be located easily. * Include metadata describing relationships between datasets, workflows, and derived results. +* Use NGFF OME-Zarr format which includes standardized metadata, also supporting efficient metadata exploration for large datasets. * Use RO-Crate metadata as a single entry point to datasets, workflows, software, and documentation. Relevant resources: diff --git a/docs/muvis-align.md b/docs/muvis-align.md index 0d3b8a9..62081d4 100644 --- a/docs/muvis-align.md +++ b/docs/muvis-align.md @@ -128,8 +128,59 @@ Rather than only documenting FAIR principles, `muvis-align` implements concrete Source: [`src/muvis_align/ui/project_template.yaml`](https://github.com/ccp-volume-em/muvis-align/blob/main/src/muvis_align/ui/project_template.yaml) -The project_template.yaml file is written in the Bilayers format and serves as the declarative description of the muvis-align workflow. The file is used to dynamically generate the [magicgui](https://pyapp-kit.github.io/magicgui/)-based [napari](https://napari.org/stable/) user interface, allowing workflow parameters, input definitions, and processing options to be defined in a structured configuration file rather than being hard-coded in the application. This approach simplifies interface maintenance, improves reproducibility, and enables the same workflow definition to be reused across the [napari](https://napari.org/stable/) plugin, command-line execution, and containerized deployments. +The project_template.yaml file is written in the Bilayers format and serves as the declarative description of the muvis-align workflow. +The file is used to dynamically generate the [magicgui](https://pyapp-kit.github.io/magicgui/)-based [napari](https://napari.org/stable/) user interface, allowing workflow parameters, input definitions, and processing options to be defined in a structured configuration file rather than being hard-coded in the application. +This approach simplifies interface maintenance, improves reproducibility, and enables the same workflow definition to be reused across the [napari](https://napari.org/stable/) plugin, command-line execution, and containerized deployments. +Bilayers schemas are represented as a [Pydantic](https://docs.pydantic.dev/) model, which adds typed validation and structured serialization for this workflow definition layer. +Source: [`src/muvis_align/ui/bilayers_util.py`](https://github.com/ccp-volume-em/muvis-align/blob/main/src/muvis_align/ui/bilayers_util.py) + +```python +from pydantic import BaseModel, ConfigDict, Field, field_validator + + +class BilayersField(BaseModel): + model_config = ConfigDict(extra='allow') + + name: str + type: str + label: str | None = None + default: Any = None + description: str | None = None + section_id: str | None = None + section_key: str | None = None + cli_order: int = 0 + + @field_validator('type', mode='before') + @classmethod + def _normalize_type(cls, value): + return str(value).lower() + + @field_validator('cli_order', mode='before') + @classmethod + def _normalize_cli_order(cls, value): + if value is None or value == '': + return 0 + return int(value) + + +def get_section_dict(template, keys=None): + sections = {} + if keys is None: + keys = [None] + + for key in keys: + items = template.get(key, []) if key is not None else template + for raw_item in items: + item = BilayersField.model_validate({**raw_item, 'section_key': key}) + section_id = item.section_id or key + sections.setdefault(section_id, []).append(item) + + return { + section_id: [item.model_dump() for item in sorted(section_items, key=lambda item: item.cli_order)] + for section_id, section_items in sections.items() + } +``` The package creates RO-Crates for outputs and adds acquisition/workflow metadata. diff --git a/docs/software.md b/docs/software.md index 3f61a2c..1a6b741 100644 --- a/docs/software.md +++ b/docs/software.md @@ -50,6 +50,11 @@ There are tools for automated dependency updates, such as [Dependabot](https://g Besides Github Issues, for further user support we recomment connecting to a single suitable community forum (e.g. [image.sc](https://forum.image.sc/) for image analysis related software) and providing a link to this forum in the documentation. This also allows the community to support each other and share knowledge. +## AI use + +Modern software development currently often includes the use of generative-AI tools, where benefits should be weighed against maintainability and potential technical debt. +Where used, the code should be reviewed and tested to ensure it is correct and maintainable, and it should be acknowledged in the documentation. + ## Further resources - A collection of resources for python tooling: [ARC python tooling](https://github-pages.arc.ucl.ac.uk/python-tooling/) - A comprehensive guide to reproducible data science, including software development and documentation: [Turing way book](https://book.the-turing-way.org) From 76cbaa365de0c174ec1465a13b3fac54c8531c79 Mon Sep 17 00:00:00 2001 From: Joost de Folter Date: Wed, 26 Aug 2026 09:49:07 +0200 Subject: [PATCH 05/13] Update muvis-align.md --- docs/muvis-align.md | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/docs/muvis-align.md b/docs/muvis-align.md index 62081d4..1fd941b 100644 --- a/docs/muvis-align.md +++ b/docs/muvis-align.md @@ -234,6 +234,28 @@ def create_zarr_ro_crate(dest_path): Input metadata is read from OME-TIFF or OME-Zarr and converted into physical units, positions, rotations, channels, scales, and transforms. +### Interoperable NGFF RFC 6 transforms + +Registration transforms are written in the [OME-NGFF RFC 6 coordinate-transform format](https://github.com/ome/ngff-spec/blob/main/index.md#coordinateTransformations-metadata) rather than as application-specific matrix arrays. Each affine identifies its image path and names its input (`source_metadata`) and output (`registered`) coordinate systems. Using the `ome-zarr-models` v0.6 model for both serialization and validation makes the saved transforms portable to other tools that implement the NGFF coordinate-system and transformation model. + +Source: [`src/muvis_align/file/transforms.py`](https://github.com/ccp-volume-em/muvis-align/blob/main/src/muvis_align/file/transforms.py) + +```python +def metadata_models(path, transform, source_key='source_metadata', transform_key='registered'): + affine = Affine( + path=path, + affine=transform, + input=CoordinateSystemIdentifier(name=source_key), + output=CoordinateSystemIdentifier(name=transform_key), + ) + return affine.model_dump(exclude_none=True, exclude_defaults=True, exclude_unset=True) + + +def read_transforms(filename): + transforms = import_json(filename) + return {label: Affine.model_validate(transform).affine for label, transform in transforms.items()} +``` + Source: [`src/muvis_align/image/ZarrDaskSource.py`](https://github.com/ccp-volume-em/muvis-align/blob/main/src/muvis_align/image/ZarrDaskSource.py) ```python @@ -542,7 +564,7 @@ CMD echo "Launching napari on Xpra. Connect via http://localhost:$XPRA_PORT or $ | NGFF | OME-Zarr default extension, NGFF v0.5, `write_sim_to_ome_zarr` | | Larger-than-memory data | Dask-backed TIFF and Zarr sources, chunked fusion, direct Zarr output | | FAIR implementation | package metadata, YAML configuration, RO-Crate generation, OME-Zarr outputs | -| Persistent metadata | OME metadata parsing, `xarray`/`MSIM` spatial transforms, positions, scales, channels | +| Persistent metadata | OME metadata parsing, `xarray`/`MSIM` spatial transforms, positions, scales, channels; interoperable NGFF RFC 6 affine transforms | | Performance metrics | NCC, SSIM, ONMI, MSE-derived score, match metrics, timing logs | | Resumable workflow | state machine, pair mappings, global mappings, metrics JSON | | Xpra interface | Docker stage exposing browser-based napari over Xpra | From b3d8c853411d72085ccce8c13ebdab475caa4598 Mon Sep 17 00:00:00 2001 From: Joost de Folter Date: Mon, 31 Aug 2026 19:13:26 +0200 Subject: [PATCH 06/13] Added green computing test case --- docs/green.md | 37 ++++++++++++++++++++++++++++++++++++- docs/index.md | 2 +- mkdocs.yml | 2 +- readme.md | 2 +- 4 files changed, 39 insertions(+), 4 deletions(-) diff --git a/docs/green.md b/docs/green.md index 71b21fe..58c6786 100644 --- a/docs/green.md +++ b/docs/green.md @@ -1 +1,36 @@ -# Green computing \ No newline at end of file +# Green computing + +As part of efficient use of computing resources, and accountability for the environmental impact of computing, it is possible to measure the carbon footprint of heavy compute operations. +General recommendations on this are provided by the Green Algorithms initiative, see the link below for more information. +Although there are currently many tools to measure carbon footprint of compute, we explore the tools provided by the Cambridge Sustainable group. + +We use the Green Algorithms for High Performance Computing (**GA4HPC**) software, as this is easily deployable on HPC clusters, supports SLURM, and can be retrospectively used without the need for pre-installed software or other components running on the cluster. + +As a test case, we use [muvis-align](muvis-align.md), software for multi-view registration of microscopy images, and we measure the carbon footprint of the registration of a large dataset. + +Specific details on operation and dataset: +* Operation: X/Y stitching of 2D tiles: registration and fusion including export of registration information and fused image data in OME-Zarr format +* Source data: 64 tiles (OME-Zarr format) of 6400 x 6400 x 16 bit = ~5GB +* Resulting fused data: 50438 x 50502 x 16 bit = ~5GB + +HPC hardware setup: +* CPU nodes: Dell PowerEdge R6525 +* Two 2nd/3rd Generation AMD EPYCTM Processors with 64 cores per processor +* 280W for the CPU nodes processors (for 64 cores) + +Power consumption / carbon factors: +* Power Usage Effectiveness (PUE): +* Average carbon intensity: 137 gCO2e/kWh + +Results from GA4HPC: +* Energy used: 0.28 kWh +* CPU: 0 days 01:00:19 (1 hours) +* Total wallclock time: 0 days 00:41:13 +* 38 gCO2e + + +Relevant resources: + +* Green Algorithms Initiative, Towards environmentally sustainable computational science: https://www.green-algorithms.org/ +* GA4HPC tool: [github.com/Cambridge-Sustainable-Computing-Lab/GreenAlgorithms4HPC](https://github.com/Cambridge-Sustainable-Computing-Lab/GreenAlgorithms4HPC) +* Green DiSC https://www.software.ac.uk/GreenDiSC, https://github.com/Cambridge-Sustainable-Computing-Lab/greenDiSC diff --git a/docs/index.md b/docs/index.md index f7b63e2..f05e963 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,5 +1,5 @@ # CCP-volumeEM guide -Community good practice guide +Community guide of recommended best practices ![logo](images/logo.png) diff --git a/mkdocs.yml b/mkdocs.yml index f89b753..db9cfb1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,6 +1,6 @@ site_name: ccp-volume-em-guide site_url: https://ccp-volume-em.github.io/guide/ -site_description: Community good practice guide +site_description: Community best practice guide repo_name: guide repo_url: https://github.com/ccp-volume-em/guide diff --git a/readme.md b/readme.md index 827966e..1f04a7b 100644 --- a/readme.md +++ b/readme.md @@ -2,4 +2,4 @@ ![logo](docs/images/logo.png) -[Community good practice guide](https://ccp-volume-em.github.io/guide/) +[Community best practice guide](https://ccp-volume-em.github.io/guide/) From 14fc449abddeb4208952cfeb451100accfc0a7eb Mon Sep 17 00:00:00 2001 From: Joost de Folter Date: Wed, 2 Sep 2026 10:53:33 +0200 Subject: [PATCH 07/13] Update docs/software.md Co-authored-by: Piper Fowler-Wright --- docs/software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/software.md b/docs/software.md index 1a6b741..64f660d 100644 --- a/docs/software.md +++ b/docs/software.md @@ -52,7 +52,7 @@ This also allows the community to support each other and share knowledge. ## AI use -Modern software development currently often includes the use of generative-AI tools, where benefits should be weighed against maintainability and potential technical debt. +Modern software development increasingly involves generative-AI tools, which can significantly speed up development cycles. Their use must be weighed against risks including poor code quality, increased technical debt and reduced code comprehension. Where used, the code should be reviewed and tested to ensure it is correct and maintainable, and it should be acknowledged in the documentation. ## Further resources From 125acdbefb0a9d0d4303766b9987e0dbfd6b6fd0 Mon Sep 17 00:00:00 2001 From: Joost de Folter Date: Wed, 2 Sep 2026 10:54:00 +0200 Subject: [PATCH 08/13] Update docs/software.md Co-authored-by: Piper Fowler-Wright --- docs/software.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/software.md b/docs/software.md index 64f660d..1b8c364 100644 --- a/docs/software.md +++ b/docs/software.md @@ -53,7 +53,9 @@ This also allows the community to support each other and share knowledge. ## AI use Modern software development increasingly involves generative-AI tools, which can significantly speed up development cycles. Their use must be weighed against risks including poor code quality, increased technical debt and reduced code comprehension. -Where used, the code should be reviewed and tested to ensure it is correct and maintainable, and it should be acknowledged in the documentation. +When used, AI-generated code should be reviewed and tested — ideally by independently and human written tests — to ensure it is correct and maintainable, and the use of generative-AI should be acknowledged in the documentation as well as relevant commits. + +We suggest reading the Institue for Research Software's [Responsible use of AI in research software](https://www.software.ac.uk/guide/responsible-use-ai-research-software) before using generative-AI in your projects. ## Further resources - A collection of resources for python tooling: [ARC python tooling](https://github-pages.arc.ucl.ac.uk/python-tooling/) From a29bf440e8768c9dbe03cf6e66b0f043c66eaf33 Mon Sep 17 00:00:00 2001 From: Joost de Folter Date: Wed, 2 Sep 2026 10:55:56 +0200 Subject: [PATCH 09/13] Update docs/software_publishing.md Co-authored-by: Piper Fowler-Wright --- docs/software_publishing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/software_publishing.md b/docs/software_publishing.md index 8a53e8c..f3f7ae3 100644 --- a/docs/software_publishing.md +++ b/docs/software_publishing.md @@ -258,7 +258,7 @@ Recommendations for the definition: Graphical applications, such as [napari](https://napari.org/) plugins, need additional consideration: the container must be able to reach a display server, either through the host's X11 socket or through a browser-based server such as -Xpra. See [`muvis-align`](muvis-align.md#interactive-interface-with-xpra-container) +Xpra. For X server access with Docker, make sure the DISPLAY environmental variable is set and host display drivers are mounted at runtime — example docker commands can be found in the [RFI vEM container documentation](https://rosalindfranklininstitute.github.io/volume-em-container-documentation/). See [`muvis-align`](muvis-align.md#interactive-interface-with-xpra-container) for a worked example of a napari image with an Xpra-enabled variant. ### Build and test in continuous integration From e86d99fa2fbbc7fdba6742700881c39433a64d8d Mon Sep 17 00:00:00 2001 From: Joost de Folter Date: Wed, 2 Sep 2026 10:56:11 +0200 Subject: [PATCH 10/13] Update docs/software.md Co-authored-by: Piper Fowler-Wright --- docs/software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/software.md b/docs/software.md index 1b8c364..6143b2f 100644 --- a/docs/software.md +++ b/docs/software.md @@ -47,7 +47,7 @@ We also recommend to make this as efficient as possible considering limited reso There are tools for automated dependency updates, such as [Dependabot](https://github.com/dependabot) and [Renovate](https://docs.renovatebot.com/), including taking care of security vulnerabilities. ## Support -Besides Github Issues, for further user support we recomment connecting to a single suitable community forum (e.g. [image.sc](https://forum.image.sc/) for image analysis related software) and providing a link to this forum in the documentation. +Besides Github Issues, for further user support we recommend connecting to a single suitable community forum (e.g. [image.sc](https://forum.image.sc/) for image analysis related software) and providing a link to this forum in the documentation. This also allows the community to support each other and share knowledge. ## AI use From ab304f8d66287c39b2ef4912ea55984d0a249822 Mon Sep 17 00:00:00 2001 From: Joost de Folter Date: Wed, 2 Sep 2026 11:21:47 +0200 Subject: [PATCH 11/13] Update docs/software_publishing.md Co-authored-by: Piper Fowler-Wright --- docs/software_publishing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/software_publishing.md b/docs/software_publishing.md index f3f7ae3..f714f9f 100644 --- a/docs/software_publishing.md +++ b/docs/software_publishing.md @@ -236,7 +236,7 @@ Recommendations for the definition: * Start from a minimal, actively maintained base image and record why it was chosen. Reuse an established base image, such as an official Python or CUDA image, instead of building an environment from scratch. -* Pin what you depend on: the base image tag, and ideally its digest; the +* Pinning dependecy versions confers reproducibility, but can overtime can lead to security vulnerabilities and build failures due to upstream availability. We recommend a balanced approach whereby you fix the versions of dependencies your application depends on immediately, leaving auxiliary system and build packages unpinned. If you need to guarantee binary reproducibility, consider minting an image in a public registry like Zenodo. system packages; and the versions of the software and its dependencies. An unpinned build produces a different environment every time it runs. * Install the software as a package (`pip install .` or an equivalent) rather From 02d01c5389ad597dcd44d49ad6c415ce6aae7c19 Mon Sep 17 00:00:00 2001 From: Joost de Folter Date: Wed, 2 Sep 2026 11:22:02 +0200 Subject: [PATCH 12/13] Update docs/software_publishing.md Co-authored-by: Piper Fowler-Wright --- docs/software_publishing.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/software_publishing.md b/docs/software_publishing.md index f714f9f..31e6932 100644 --- a/docs/software_publishing.md +++ b/docs/software_publishing.md @@ -238,7 +238,6 @@ Recommendations for the definition: image, instead of building an environment from scratch. * Pinning dependecy versions confers reproducibility, but can overtime can lead to security vulnerabilities and build failures due to upstream availability. We recommend a balanced approach whereby you fix the versions of dependencies your application depends on immediately, leaving auxiliary system and build packages unpinned. If you need to guarantee binary reproducibility, consider minting an image in a public registry like Zenodo. system packages; and the versions of the software and its dependencies. An - unpinned build produces a different environment every time it runs. * Install the software as a package (`pip install .` or an equivalent) rather than copying source files into an arbitrary directory, so that version metadata and entry points are available inside the image. From 453fcd93e67d054f384801184f3fdc18ad7b5231 Mon Sep 17 00:00:00 2001 From: Joost de Folter Date: Wed, 2 Sep 2026 11:22:21 +0200 Subject: [PATCH 13/13] Update docs/software_publishing.md Co-authored-by: Piper Fowler-Wright --- docs/software_publishing.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/software_publishing.md b/docs/software_publishing.md index 31e6932..cce4ac3 100644 --- a/docs/software_publishing.md +++ b/docs/software_publishing.md @@ -237,7 +237,6 @@ Recommendations for the definition: chosen. Reuse an established base image, such as an official Python or CUDA image, instead of building an environment from scratch. * Pinning dependecy versions confers reproducibility, but can overtime can lead to security vulnerabilities and build failures due to upstream availability. We recommend a balanced approach whereby you fix the versions of dependencies your application depends on immediately, leaving auxiliary system and build packages unpinned. If you need to guarantee binary reproducibility, consider minting an image in a public registry like Zenodo. - system packages; and the versions of the software and its dependencies. An * Install the software as a package (`pip install .` or an equivalent) rather than copying source files into an arbitrary directory, so that version metadata and entry points are available inside the image.