Skull-stripping benchmark

This repository compares five brain-extraction tools on a small, heterogeneous set of NIfTI images:

• SynthStrip
• MindGrab
• niimath CPU MindGrab
• FSL BET
• AFNI 3dSkullStrip

The benchmark is intentionally simple. It uses one standard command per tool and does not tune parameters for individual images. The outputs are useful for qualitative and performance comparisons, but they are not a gold-standard segmentation dataset. Some images are intentionally edge cases. For example, T1w_ARC2017 is from the Aphasia Recovery Cohort and includes a lesion. This repository is neither a validation nor a training dataset, and it does not provide ground truth. The SynthStrip team shares a large and diverse training dataset.

Repository layout

inputs/                  Input NIfTI images
outputs/
  synthstrip/            SynthStrip results
  mindgrab/              MindGrab results
  niimath/               niimath CPU MindGrab results
  bet/                   FSL BET results
  3dSkullStrip/          AFNI 3dSkullStrip results
benchmark.py             Reproducible benchmark runner

Each output uses the input filename, for example inputs/T2w.nii.gz becomes outputs/synthstrip/T2w.nii.gz.

Requirements

Install the five tools separately and make these commands available on PATH:

mri_synthstrip
brainchop
niimath
bet
3dSkullStrip

FreeSurfer users may need to load its environment before running the benchmark. On the machine used for the results below, the fs shell alias performs this setup.

The runner has no third-party Python dependencies. It uses /usr/bin/time for peak resident-memory measurements on macOS and Linux. On other platforms it still records wall time, but peak RAM may be unavailable.

Run the benchmark

Run all methods on all files in inputs/:

python3 benchmark.py

Existing outputs are skipped by default. Use --force to replace them:

python3 benchmark.py --force

Run selected methods or inputs:

python3 benchmark.py --methods synthstrip bet
python3 benchmark.py --inputs T1w1mm.nii.gz T2w.nii.gz

The runner executes jobs serially so elapsed time and peak-memory measurements are easier to compare. It writes:

• tool logs to outputs/logs/<method>-<input>.log
• elapsed time, peak RSS, status, and command to outputs/metrics.csv

A failed job is recorded and does not stop the remaining jobs.

Runtime results

Wall time in seconds (lower is faster), measured serially on this machine:

Input image	BET	3dSkullStrip	MindGrab	niimath	SynthStrip (CPU)
FLAIR_MICCAI2017.nii.gz	1.576	51.508	6.961	7.202	308.018
PD.nii.gz	1.530	21.340	6.402	7.136	25.015
T1w1mm.nii.gz	6.482	77.853	6.796	8.011	196.916
T1w2mm.nii.gz	2.953	48.640	6.512	7.539	182.901
T1w_ARC2017.nii.gz	2.445	100.154	6.694	8.156	23.682
T1w_MICCAI2017.nii.gz	9.513	62.928	6.779	8.241	*
T2w.nii.gz	1.610	16.690	6.541	7.907	128.964
dwi.nii.gz	0.970	14.211	6.453	7.856	23.949
fmri.nii.gz	1.024	17.157	6.463	8.034	114.257

Here, MindGrab is the BrainChop implementation and niimath is its CPU -mindgrab implementation. SynthStrip ran without its -g GPU option after loading the FreeSurfer 8.1.0 environment. The complete measurements, including status and peak RSS, are in outputs/metrics.csv.

* FreeSurfer 8.1.0 stopped before inference with RuntimeError: quantile() input tensor is too large. SynthStrip can process the image after cropping it with robustfov.

Commands

The script runs the equivalent of:

mri_synthstrip -i inputs/T1w1mm.nii.gz \
  -o outputs/synthstrip/T1w1mm.nii.gz

brainchop -m mindgrab \
  -o outputs/mindgrab/T1w1mm.nii.gz \
  inputs/T1w1mm.nii.gz

niimath inputs/T1w1mm.nii.gz -mindgrab outputs/niimath/T1w1mm

bet inputs/T1w1mm.nii.gz outputs/bet/T1w1mm -R

3dSkullStrip -input inputs/T1w1mm.nii.gz \
  -prefix outputs/3dSkullStrip/T1w1mm.nii.gz

BET uses robust brain-centre estimation (-R) with its default fractional intensity threshold. AFNI's default output intensities may differ from the input intensities; add -orig_vol if preservation of in-brain values is required. Parameter tuning or preprocessing such as robustfov may improve particular datasets, but is outside this baseline comparison.

Methods and references

• SynthStrip introduced a robust, contrast-agnostic extraction method trained with synthetic data: Hoopes et al., NeuroImage 260 (2022), 119474. doi:10.1016/j.neuroimage.2022.119474
• MindGrab is a lightweight BrainChop model trained using SynthStrip-derived data and based on the efficient MeshNet architecture.
• BET is a widely used brain-extraction method: Smith, Human Brain Mapping 17 (2002), 143–155. doi:10.1002/hbm.10062
• AFNI 3dSkullStrip is a modified spherical-surface method inspired by BET.

Interpretation

Runtime and memory depend on image dimensions, CPU, software versions, thread settings, and filesystem caching. Record the tool versions and machine details when publishing comparisons. Visual inspection is also essential: a fast method is not useful if it removes brain tissue or retains substantial non-brain tissue.