Preprocess command

Description

Preprocess a directory of slice images into a single volume file that can be used by the other commands.

Overview diagram

Command arguments

Argument	Type	Description	Required?
--volume_dir	directory	Directory of full volume of image slices.	yes
--config_fullfname	full file name	Full path to configuration file specifying how to preprocess the volume (*.json).	yes
--result_data_fullfname	full file name	Full path of data file to store the preprocessed volume.	yes
--checkpoint_fullfname	full file name	Full path to file that is used to let the process save its progress and continue from where it left off in case of interruption (*.json). If left out then the process will run from beginning to end without saving any checkpoints.	no
--checkpoint_namespace	string	Unique name for the group of checkpoints used by this command.	no
--reset_checkpoint	yes/no	Whether to clear the checkpoints about this command from the checkpoint file and start afresh or not (default is no).	no
--log_file_fullfname	full file name	Full path to file that is used to store a log of what is displayed on screen (*.txt).	no
--max_processes	whole number	Maximum number of parallel processes to use (-1 to use maximum, default).	no
--max_batch_memory	fractional number	Maximum amount of GB to allow for processing the volume in batches (-1 to use maximum, default).	no
--print_output	yes/no	Whether to output to the screen (default is yes).	no
--debug_mode	yes/no	Whether to give full error messages (default is no).	no

Command line example

python ASEMI-segmenter/Python/asemi_segmenter/bin/preprocess.py \
    --volume_dir "volume" \
    --config_fullfname "configs/preprocess_config.json" \
    --result_data_fullfname "output/preprocess/volume.hdf" \
    --checkpoint_fullfname "output/checkpoint.json" \
    --log_file_fullfname "output/log.txt" \
    --max_processes 4 \
    --max_batch_memory 1.0

Stages

#	Log message	Description
1	Loading data	Load input data, validate it, and initialise the checkpoint and hash function.
2	Creating empty data file	Create an empty HDF file (empty in terms of data, not bytes) to reserve the disk space needed be able to store the preprocessed volume.
3	Dumping slices into data file	Create a copy of the original volume slices.
4	Downscaling volume	Create all the resized versions of the original volume.
5	Hashing volume slices	Create hash vectors for each slice in the original volume.

Checkpoints

#	Checkpoint name	In stage	Description
1	empty_data_file	2	Empty preprocessed file has been created and should not be overwritten with a new one.
2	dump_slices	3	Slices have been dumped into scale 0 of the preprocessed file.
3	downscale	4	All downscaled volumes have been created and added to the preprocessed volume.
4	downscale_prog	4	The number of downscales of the volume that have been created and added to the preprocessed volume.
5	hashing_slices	5	All slices in scale 0 have been hashed.
6	overall	-	The process has been completed and does not need to be repeated.

Further information

• Volume format
• Preprocess configuration
• Preprocessed volume format (technical)