tsio: A command line utility for extracting images from microscopy-related files

A Command Line Interface (CLI) application for extracting images from microscopy files to common image file formats, such as JPEG, PNG, and/or TIFF. Supported microscopy files include:

• dcm (DICOM) - medical image file format standard
• dm3/dm4 (DigitalMicrograph) - (S)TEM image file format by GATAN
• emd (Velox) - (S)TEM image file format by ThermoFisher

1. Prerequisites
   1. Python
      1. Ubuntu
      2. macOS
   2. pipx
      1. Ubuntu
      2. macOS
2. Installation
   1. pipx
   2. Source
3. Upgrade
   1. pipx
   2. Source
4. Usage
5. Contributing
6. License

Prerequisites

All of the prerequisites may already be installed and configured by the superuser, a.k.a. root, of the computer. The prerequisites only need to be installed and configured once per machine.

Python

The Python programming language is needed to run the tsio Command Line Interface (CLI) application and/or use in other Python scripts. Both macOS and Ubuntu Linux have the Python programming language installed, but it is generally reserved for the operating system (OS) to use and is an older version. It is best practice to install a newer version that is separate from the system-provided Python version.

Ubuntu

1. Add the "deadsnakes" Ubuntu Personal Package Archives (PPA).

   sudo add-apt-repository ppa:deadsnakes/ppa

2. Obtain the latest packages from the PPA.

   sudo update

3. Install Python v3.11 or newer.

   sudo apt install python3.11

4. Install the venv package.

   sudo apt install python3.11-venv

macOS

1. Install Homebrew if it is not already installed.

   /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

2. Install Python v3.11 or newer.

   brew install python@3.11

pipx

The pipx utility enables distribution of Python-based CLI applications, like tsio, to be installed for all users with all of the appropriate dependencies within an isolated environment. It is the recommended installation for the tsio application.

Ubuntu

1. Create a virtual environment for pipx and Python v3.11 or newer.

   sudo python3.11 -m venv --upgrade-deps /opt/pipx

2. Install pipx for all users.

   sudo /opt/pipx/bin/pip install pipx

3. Ensure the pipx command is available to all users.

   sudo ln -s /opt/pipx/bin/pipx /usr/local/bin/pipx

4. Add pipx to the PATH environment variable.

   pipx ensurepath

5. Add pipx for all users.

   sudo pipx ensurepath --global

Post-installation, the pipx application can be upgraded with the following command:

sudo /opt/pipx/bin/pip install --upgrade pipx

macOS

1. Install pipx using Homebrew.

   brew install pipx

2. Add pipx to the PATH environment variable.

   pipx ensurepath

3. Add pipx for all users.

   sudo pipx ensurepath --global

Post-installation, the pipx application can be upgraded with the following command:

brew update && brew upgrade pipx

Installation

The tsio code includes a Python application with a Command Line Interface (CLI). It can be installed as a standalone application.

pipx (recommended)

1. Obtain a package key from Theia Scientific personnel. Save it for Step 3.

2. Ensure pipx is installed. See the Prerequisites.

   $ pipx --version
   1.9.0

3. Install tsio command globally for all users.

   sudo pipx install --global --python python3.11 "tsio"

4. Verify tsio command is available.

   $ tsio --version
   tsio 0.1.0

Source

1. Clone the tsio repository.

   git clone https://github.com/Theia-Scientific/tsio.git && cd tsio

2. Create a virtual environment.

   python3.11 -m venv .venv

3. Activate the virtual environment.

   source .venv/bin/activate

   or if direnv is installed:

   cp .envrc.example .envrc

   followed by:

   direnv allow

4. Upgrade pip to the latest version.

   pip install --upgrade pip

5. Locally install the package, utility, and its dependencies. This will create the tsio command within the virtual environment. This also installs the Slack feature for sending notifications to a Slack channel.

   pip install -e .

Upgrade

pipx (recommended)

1. Upgrade the tsio application via pipx.

   sudo pipx upgrade --global tsio

2. Verify new version.

   $ tsio --version
   tsio 0.1.0

Source

1. Navigate to the root of the source tree.

   cd ~/Code/tsio

2. Activate the virtual environment.

   source .venv/bin/activate

   or if direnv is installed, the virtual environment will automatically be activated.

3. Pull the latest changes on main.

   git pull

4. Upgrade the tsio application within the virtual environment.

   pip install --upgrade -e .

5. Verify new version.

   $ tsio --version
   tsio 0.1.0

Usage

Convert a single page TIFF to a JPEG. Creating a JPEG image file is the default because this is useful for Machine Learning (ML) tools, supported by all web browsers, and it creates the smallest file, making this the quickest for obtaining a thumbnail of the microscopy image file.

$ tsio image.tif
$ ls
image.jpg image.tif

Convert a single page TIFF to a PNG. All of the following commands are equivalent.

# Use the long option.
$ tsio --to png image.tif
$ ls
image.png image.tif

# Use the `=` syntax for specifying an option value.
$ tsio --to=png image.tif
$ ls
image.png image.tif

# Use the short option.
$ tsio -t png image.tif
$ ls
image.png image.tif

# Use the short option with the `=` syntax.
$ tsio -t=png image.tif
$ ls
image.png image.tif

Extract all the "frames" from a multi-page TIFF.

$ tsio multi-page.tif
$ ls
multi-page/ multi-page.tif
$ ls multi-page/
0.jpg 1.jpg 2.jpg 3.jpg 4.jpg

# Use the `-o,--output` option to extract to different location.
$ tsio --output /path/to/directory tiff png multi-page.tif
$ ls
multi-page.tif
$ ls /path/to/directory
0.jpg 1.jpg 2.jpg 3.jpg 4.jpg

Extract an image from a DM3 or DM4 file.

$ tsio ./00001.dm3
$ ls
00001.dm3  00001.jpg

$ tsio ./00001.dm4
$ ls
00001.dm4  00001.jpg

Extract all the "frames" from a EMD file.

$ tsio multiple-frames.emd
$ ls
multiple-frames/ multiple-frames.emd
$ ls multiple-frames/
0.jpg 1.jpg 2.jpg 3.jpg 4.jpg

Contributing

1. Clone this repository.

   git clone https://github.com/Theia-Scientific/tsio && cd tsio

2. Create a virtual environment.

   python3.11 -m venv .venv

3. Activate the virtual environment.

   source .venv/bin/activate

   or if direnv is installed, the virtual environment will automatically be activated.

4. Upgrade pip.

   pip install --upgrade pip

5. Install all the dependencies.

   pip install -e ".[dev]"

6. Create a local branch.

   git checkout -b feature-awesome-new-feature

7. Modify the code.

8. Run the tests.

   pytest --color=yes --cov=tsio --cov-report=term-missing

9. Commit changes to your local branch.

   git add -A && git commit -m "Add new feature"

10. Push your local branch to GitHub to create a Pull Request (PR).

git push origin feature-awesome-new-feature

11. Create a Pull Request (PR) in GitHub.

12. Wait for CI to complete.

13. Add comment to PR that it is ready to review.

14. Wait for review from a maintainer.

15. Address any comments from the reviewer by modifying your local files and pushing to the remote branch/PR.

    git push origin feature-awesome-new-feature

16. Once the PR is approved, then it will be "Squash and Merge". Congratulations on contributing to an open source project, and Thank you!

License

The tsio project is licensed under the GPL-3.0 license. See the LICENSE.txt file for more information about licensing and copyright.

Acknowledgments

This material is based upon work supported by the U.S. Department of Energy, Office of Basic Science and Office of Nuclear Energy under Award Number DE-SC0021529 and DE-SC0021936, respectively.