Evaluate command

Description

Evaluate the performance of a trained segmentation model using the intersection-over-union metric and give error analysis data.

Overview diagram

Command arguments

Argument	Type	Description	Required?
--segmenter_fullfname	full file name	Full file name (with path) to the segmenter pickle file that was obtained from the train command (*.pkl).	yes
--preproc_volume_fullfname	full file name	Full file name (with path) to the preprocessed volume data file (*.hdf).	yes
--subvolume_dir	directory	Directory of the subvolume that was labelled.	yes
--label_dirs	list of directories	Space separated list of directories, each of which represents a label and contains the slices in subvolume_dir with the regions that are not part of the label being blacked out.	yes
--results_dir	directory	Directory to the folder that will contain the results.	yes
--confusion_map_with_input_slice	yes/no	Whether to use the input slice as a background for the confusion map. If false then the label colour of the label in question will be used. Default is yes.	no
--checkpoint_fullfname	full file name	Full path to file that is used to let the process save its progress and continue from where it left off in case of interruption (*.json). If left out then the process will run from beginning to end without saving any checkpoints.	no
--checkpoint_namespace	string	Unique name for the group of checkpoints used by this command.	no
--reset_checkpoint	yes/no	Whether to clear the checkpoints about this command from the checkpoint file and start afresh or not (default is no).	no
--log_file_fullfname	full file name	Full path to file that is used to store a log of what is displayed on screen (*.txt).	no
--max_processes_featuriser	whole number	Maximum number of parallel processes to use concurrently whilst featurising (-1 to use maximum, default).	no
--max_processes_classifier	whole number	Maximum number of parallel processes to use concurrently whilst classifying (-1 to use maximum, default).	no
--max_batch_memory	fractional number	Maximum amount of GB to allow for processing the volume in batches (-1 to use maximum, default).	no
--use_gpu	yes/no	Whether to use the GPU for computing features (default is no).	no
--print_output	yes/no	Whether to output to the screen (default is yes).	no
--debug_mode	yes/no	Whether to give full error messages (default is no).	no

Command line example

python ASEMI-segmenter/Python/asemi_segmenter/bin/evaluate.py \
    --segmenter_fullfname "output/train/segmenter.pkl" \
    --preproc_volume_fullfname "output/preprocess/volume.hdf" \
    --subvolume_dir "testing_set/subvolume" \
    --label_dirs \
        "testing_set/labels/air" \
        "testing_set/labels/tissues" \
        "testing_set/labels/bones" \
    --results_dir "output/evaluate" \
    --confusion_map_with_input_slice "yes" \
    --checkpoint_fullfname "output/checkpoint.json" \
    --log_file_fullfname "output/log.txt" \
    --max_processes_featuriser 4 \
    --max_processes_classifier 4 \
    --max_batch_memory 1.0 \
    --use_gpu "no"

Stages

#	Log message	Description
1	Loading data	Load input data, validate it, and initialise the checkpoint and hash function.
2	Hashing subvolume slices	Create hash vectors for each subvolume slice.
3	Constructing labels dataset	Load all the labels from the given dataset.
4	Evaluating	Create the results file and evaluate each slice in the dataset.

Checkpoints

#	Checkpoint name	In stage	Description
1	create_results_file	4	Empty results file has been created and should not be overwritten with a new one.
2	evaluation_prog	4	The number of slices of the subvolume that have been evaluated.
3	conclude	4	The concluding row has been added to the results file.
4	global_confusion_matrix	4	The global confusion matrix has been computed and saved.
5	overall	4	The process has been completed and does not need to be repeated.

Further information

• Evaluation result format