MIST: Merging Instance Segmentation Tiler

A Command Line Interface (CLI) application and Python package for running the Merging Instance Segmentation Tiler (MIST) with Machine Learning (ML) computer vision models. MIST creates tiles from a large image, runs inference on each tile, and combines, or merges, instances of the same class together using the instance segmentation results from inference. Non-maximum suppression (NMS) is not used to determine overlap. Instead, each instance of a class is logically "anded" into a binary mask. The individual instances within a class are identified as contours through OpenCV's connectivity algorithm. In this manner, large instances that span multiple tiles are combined, or merged, into a single instance and small instances within the overlap region between two or more tiles are automatically filtered and reduced to a single instance. Only a single "pass" per class is required.

MIST is inspired by the Slicing Aided Hyper Inference (SAHI), YOLO Patch-Based Inference (YPBI), and dask_relabeling packages. The SAHI tiler does not merge large instances spanning multiple tiles and uses NMS for instance reduction in overlap regions. The YPBI tiler does instance segmentations but does not merge large instances, and it performs multiple NMS iterations for both bounding boxes and segmentations. The dask_relabeling package does instance segmentations and merging large instances, but it does not work with YOLO models and GPU-powered inference.

1. Prerequisites
   1. Python
      1. Ubuntu
      2. macOS
   2. pipx
      1. Ubuntu
      2. macOS
2. Installation
   1. pipx (recommended)
   2. Source
3. Upgrade
   1. pipx
   2. Source
4. Usage
   1. Terminal
   2. Python
5. Contributing
6. License

Prerequisites

All of the prerequisites may already be installed and configured by the superuser, a.k.a. root, of the computer. The prerequisites only need to be installed and configured once per machine.

Python

The Python programming language is needed to run the mist Command Line Interface (CLI) application and/or use the mist package in other Python scripts or Jupyter notebooks. Both macOS and Linux have the Python programming language installed, but it is generally reserved for the operating system (OS) to use and is an older version. It is best practice to install a newer version that is separate from the system-provided Python version.

Ubuntu

MIST was developed and tested on Ubuntu 22.04 Linux. The following steps are for Ubuntu Linux, but any Linux distribution can be used. The commands will be similar but different for other Linux distributions.

1. Add the "deadsnakes" Ubuntu Personal Package Archives (PPA).

   sudo add-apt-repository ppa:deadsnakes/ppa

2. Obtain the latest packages from the PPA.

   sudo apt update

3. Install Python v3.11 or newer.

   sudo apt install -y python3.11

4. Install the venv package.

   sudo apt install -y python3.11-venv

macOS

MIST has been deployed and tested on a Macbook Pro laptop with a M3 Apple Silicon processor.

1. Install Homebrew if it is not already installed.

   /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

2. Install Python v3.11 or newer.

   brew install python@3.11

pipx

The pipx utility enables distribution of Python-based CLI applications, like mist, to be installed for all users with all of the appropriate dependencies within an isolated environment. It is the recommended installation for the mist application.

Ubuntu

1. Create a virtual environment for pipx and Python v3.11 or newer.

   sudo python3.11 -m venv --upgrade-deps /opt/pipx

2. Install pipx for all users.

   sudo /opt/pipx/bin/pip install pipx

3. Ensure the pipx command is available to all users.

   sudo ln -s /opt/pipx/bin/pipx /usr/local/bin/pipx

4. Add pipx to the PATH environment variable.

   pipx ensurepath

5. Add pipx for all users.

   sudo pipx ensurepath --global

Post-installation, the pipx application can be upgraded with the following command:

sudo /opt/pipx/bin/pip install --upgrade pipx

macOS

1. Install pipx using Homebrew.

   brew install pipx

2. Add pipx to the PATH environment variable.

   pipx ensurepath

3. Add pipx for all users.

   sudo pipx ensurepath --global

Post-installation, the pipx application can be upgraded with the following command:

brew update && brew upgrade pipx

Installation

pipx (recommended)

1. Ensure pipx is installed. See the Prerequisites.

   $ pipx --version
   1.7.1

2. Install mist command globally for all users.

   sudo pipx install --global --python python3.11 "tsmist[cli]"

3. Verify mist command is available.

   $ mist --version
   mist 0.1.0

Source

1. Clone this repository.

   git clone https://github.com/Theia-Scientific/mist.git && cd mist

2. Create a virtual environment.

   python3 -m venv .venv

3. Activate the virtual environment.

   source .venv/bin/activate

   or if direnv is installed:

   cp .envrc.example .envrc

   followed by:

   direnv allow

4. Upgrade pip to the latest version.

   python3 -m pip install --upgrade pip

5. Locally install the package, utility, and its dependencies. This will create the mist command within the virtual environment.

   python3 -m pip install -e ".[cli]"

Upgrade

pipx (recommended)

1. Upgrade the mist application via pipx.

   sudo pipx install --global --python python3.11 --force tsmist

2. Verify new version.

   $ mist --version
   mist 0.1.0

Source

1. Navigate to the root of the source tree.

   cd ~/Code/mist

2. Activate the virtual environment.

   source .venv/bin/activate

   or if direnv is installed, the virtual environment will automatically be activated.

3. Pull the latest changes on main.

   git pull

4. Upgrade the mist application within the virtual environment.

   python -m pip install --upgrade -e .

5. Verify new version.

   $ mist --version
   mist 0.1.0

Usage

Terminal

Using the Ultralytics YOLOv8 model:

mist yolov8n-seg.pt example1.jpg example2.jpg /path/to/images/dir

For running on macOS:

mist --device=mps yolov8n-seg.pt example1.jpg example2.jpg /path/to/images/dir

Python

Using an Ultralytics YOLO segmentation model and defaults.

import numpy as np
import numpy.typing as npt
import supervision as sv

from mist import detecting
from pathlib import Path
from ultralytics.models import YOLO

model = YOLO("yolo26n-seg.pt")


def predict(image: npt.NDArray[np.uint8]) -> sv.Detections:
    return sv.Detections.from_ultralytics(
        list(
            model(
                image,
                agnostic_nms=False,
                device="cuda:0",
                classes=None,
                conf=0.35,
                imgsz=640,
                iou=0.7,
                max_det=1000,
                retina_masks=True,
                verbose=False
            )
        )[0]
    )


result = detecting.run(
    np.zeros((3, 4096, 4096), dtype=np.uint8),
    predict,
    class_names=[name for _, name in sorted(model.names.items())],
)
print(results)

Using a custom model.

import numpy as np
import numpy.typing as npt
import supervision as sv

from mist import detecting
from pathlib import Path

model = CustomModel(weights_file)


def predict(image: npt.NDArray[np.uint8]) -> sv.Detections:
    return sv.Detections.from_inference(model(image))


results = detecting.run(
    np.zeros((3, 4096, 4096), dtype=np.uint8),
    predict,
    class_names=["object"]
)
print(results)

Contributing

1. Clone this repository.

   git clone https://github.com/Theia-Scientific/mist.git && cd mist

2. Create a virtual environment.

   python3 -m venv .venv

3. Activate the virtual environment.

   source .venv/bin/activate

   or if direnv is installed:

   cp .envrc.example .envrc

   followed by:

   direnv allow

4. Install the package, utility, the required dependencies, and the development dependencies.

   pip install -e ".[cli,dev]"

5. Create a local branch.

   git checkout -b feature-awesome-new-feature

6. Modify the code.

7. Run the tests.

   pytest --color=yes

8. Commit changes to your local branch.

   git add -A && git commit -m "Add new feature"

9. Push your local branch to GitHub to create a Pull Request (PR).

   git push origin feature-awesome-new-feature

10. Create a Pull Request (PR) in GitHub.

11. Wait for CI to complete.

12. Add comment to PR that it is ready to review.

License

• LICENSE.

Acknowledgments

This material is based upon work supported by the U.S. Department of Energy, Office of Science and Office of Nuclear Energy under Awards: DE-SC0021529 and DE-SC0021936, respectively.