perch-runner

A Docker container that generates Perch audio embeddings for folders of audio files.

Quick Start

docker run --rm \
  -v /path/to/audio:/mnt/input \
  -v /path/to/output:/mnt/output \
  qutecoacoustics/perchrunner:latest analyze --embed

This processes all audio files in /path/to/audio and writes Parquet embedding files to /path/to/output/.

Usage

docker run --rm \
  -v <source>:/mnt/input \
  -v <output>:/mnt/output \
  [-v <config_dir>:/mnt/config] \
  qutecoacoustics/perchrunner:latest <subcommand> [options]

Subcommands

Subcommand	Description
analyze	Run embedding/classification pipeline
version	Print perch-runner, perch-hoplite, and model versions
config	Print default resolved config as JSON

Analyze Options

Flag	Description	Default
--embed	Enable embedding export (boolean flag). Use --embed with no value to enable	false
--embeddings_table_format	Table layout: serialized or columns	serialized
--embeddings_table_filetype	File format for the embedding table: parquet or csv	parquet
--embeddings_output_path_template	Output path template for embedding files. Tokens: {parents}, {filestem}, {ext}, {embeddings_table_format}, {analysis}	{analysis}{ext}
--embeddings_output_path_type	Preset output layout for embeddings: flat_filestem, nested_filestem, nested, flat	flat
--classify	Enable Perch global classification output (boolean flag). Use --classify with no value to enable	false
--classify_filetype	File format for classification output: parquet or csv	csv
--classify_species_list	Legacy classify species-list path option (currently not used by Perch global classify output)	None
--perch_species_list	Species filter for Perch classify output. Accepts comma/newline text, list values, or a text-file path	None
--perch_max_detections_per_window	Maximum number of classify detections kept per window	10
--classify_require_species_list	Require perch_species_list when classify is enabled	false
--classify_output_path_template	Output path template for classification files	{analysis}{ext}
--classify_output_path_type	Preset output layout for classification files: flat_filestem, nested_filestem, nested, flat	flat
--recognizers	Path to a recognizers JSON file. Runs embeddings through linear classifiers and writes per-recognizer result files	None
--recognizer_results_filetype	File format for recognizer results: parquet or csv	csv
--recognizer_output_path_template	Output path template for recognizer result files. Tokens: {classifier_name}, {parents}, {filestem}, {ext}, {analysis}	{analysis}{ext}
--recognizer_output_path_type	Preset output layout for recognizer results: flat_filestem, nested_filestem, nested, flat	flat
--model_choice	Model to use: perch_v2 or perch_8	perch_v2
--output_path_type	Preset output layout applied to both embeddings and recognizer results (overridden by more specific keys): flat_filestem, nested_filestem, nested, flat	flat
--dataset_name	Dataset name used in runner configuration	search_set
--db_path	Database path; relative paths resolve under output	db
--sourcemap_name	Optional sourcemap preset used to rewrite the output source value	None
--file_metadata	Optional JSON object of token values used to fill sourcemap template placeholders	None
--sourcemap_template	Optional sourcemap destination template, e.g. https://.../audio_recordings/{audio_recording_id}/original	None
--file_metadata_pattern	Optional regex (or preset name) used to extract named token values from basename (including extension)	None
--save_db	Persist the hoplite embedding database. Use --save_db with no value to enable (default: false)	false
--file_glob	Glob pattern for audio files, e.g. */*, */*/*	Auto-detected
--workers	Worker count or auto	auto
--log_level	App log level: DEBUG, INFO, WARNING, ERROR, CRITICAL	INFO
--hoplite_log_level	Library/root log level	WARNING
--tf_log_level	TensorFlow C++ log level	WARNING
--log_file	Optional log file path	None
--config_file	Path to a YAML config file	None
--source	Override source path (default: /mnt/input)	/mnt/input
--output	Override output path (default: /mnt/output)	/mnt/output

Analyze Option Details

--embed

Enables embedding export. This is a boolean flag.

• --embed or --embed true: enable embedding export.
• --embed false: explicitly disable (overrides a config file that enables it).
• Use --embeddings_table_format and --embeddings_table_filetype to control the output format.
• Setting any --embeddings_* key (e.g. --embeddings_table_format columns) implicitly enables embed unless --embed false is also set.

--embeddings_table_format

Controls table layout for embed outputs that do not explicitly include a table format.

• Allowed values: serialized, columns
• Multiple values allowed, comma-separated.

What they mean:

• serialized: one embeddings column containing a serialized vector per row.
• columns: one column per embedding dimension (f0000, f0001, ...).

--file_glob

Selects which audio files under source directory are embedded.

• If provided, it is used directly (examples: *, */*, */*/*).
• If omitted or false-like, it is auto-detected.
• recursive globbing is not possible, due to how perch-hoplite works
• This is ignored if the source points to a single file

Auto-detection behavior:

• The runner scans audio files recursively.
• It chooses a glob depth based on the shallowest audio file found:
  • top-level audio file -> *
  • one level deep -> */*
  • two levels deep -> */*/*
• Deeper files than the chosen depth are skipped.
• A warning is logged when deeper files are skipped.

Special case:

• If --source points to a single audio file, only that file is embedded (internally using the filename as the glob).

--embeddings_output_path_template

For the tabular (e.g. parquet) outputting of embeddings, you can specify where they are saved within <output>/ using a template.

• Supported tokens:
  • {parents} the parent directories of the audio file, relative to the source directory
  • {filestem} the filename of the source audio file, without extension
  • {ext} the extension of the output format, e.g. .parquet, .csv
  • {embeddings_table_format} the table format e.g. serialized or columns
  • {analysis} the output type — for embeddings this is always embeddings
• Must be a relative path.
• Must not contain .. path traversal.

Examples:

• {analysis}{ext} (default — all recordings are merged into a single file, e.g. embeddings.parquet)
• {parents}/{filestem}/{embeddings_table_format}/embeddings{ext}

If exporting both parquet table formats, include {embeddings_table_format} in the template to avoid path collisions.

If more than one source audio files map to the same output file, they will all be included in the same output file.

--embeddings_output_path_type

Preset output paths (mutually exclusive with --embeddings_output_path_template):

• flat_filestem -> {filestem}{ext}
• nested_filestem -> {parents}/{filestem}{ext}
• nested -> {parents}/{analysis}{ext}
• flat -> {analysis}{ext} (all recordings in a single file, e.g. embeddings.parquet)

--recognizers

Runs embeddings through one or more linear classifiers (embeddings-classifier) and writes per-classifier result files.

• Accepts a path to a JSON file containing a recognizers list, or a bare list directly.
• By default, recognizer outputs are merged into a single file ({analysis}{ext}), unless you include {classifier_name} and/or source tokens in a custom template.
• Output path controlled by --recognizer_output_path_template or --recognizer_output_path_type.
• When recognizers are configured, model_choice is derived automatically from the recognizer metadata unless explicitly set.

--recognizer_output_path_template

Output path template for recognizer result files. Controls where results are written within <output>/.

• Supported tokens:
  • {classifier_name} the recognizer's name
  • {parents} parent directories of the source audio file
  • {filestem} filestem of the source audio file, without extension
  • {ext} output file extension, e.g. .csv, .parquet
  • {analysis} the output type — for recognizer results this is always recognizer_results
• Must be a relative path.
• Must not contain .. path traversal.
• Default: {analysis}{ext}

Examples:

• {classifier_name}/{parents}/{filestem}/{analysis}{ext} (one directory per recognizer, mirroring source structure)
• {classifier_name}/{analysis}{ext} (one flat file per recognizer, all recordings merged)
• {analysis}{ext} (single file, all recognizers and recordings merged — use with care)

--recognizer_output_path_type

Preset output paths for recognizer results (mutually exclusive with --recognizer_output_path_template):

• flat_filestem -> {filestem}{ext}
• nested_filestem -> {parents}/{filestem}{ext}
• nested -> {parents}/{analysis}{ext}
• flat -> {analysis}{ext} (all recordings for all recognizers in a single file)

Note: these presets do not include {classifier_name}, so results from multiple recognizers will be merged into the same file. Add a --recognizer_output_path_template with {classifier_name} if you need per-recognizer separation.

--output_path_type

Applies a preset layout to both embeddings and recognizer results at once. The more specific embeddings_output_path_type and recognizer_output_path_type take priority if also set.

• Accepted values: flat_filestem, nested_filestem, nested, flat
• Equivalent to setting both --embeddings_output_path_type and --recognizer_output_path_type to the same value.
• If unset, it resolves to flat.

--db_path

Location for the internal embedding database.

• Relative paths are resolved under --output.
• Default is db, which resolves to <output>/db.

--sourcemap_name

Selects a hardcoded sourcemap preset template.

• If unset (and no other sourcemap options are set), source paths are written unchanged.
• Current presets: a2o_original, baw_original, ecosounds_original

--sourcemap_template

Defines the output template directly. Use {token} placeholders for values from --file_metadata and/or named groups extracted by --file_metadata_pattern.

Examples:

• Constant mapping for every row:
  • analyze --embed --sourcemap_template 'https://api.ecosounds.org/audio_recordings/1234/original'
• Pattern + values mapping:
  • analyze --embed --sourcemap_template '{domain}/audio_recordings/{audio_recording_id}/original' --file_metadata_pattern canonical_filename --file_metadata '{"domain":"https://api.ecosounds.org"}'

--file_metadata_pattern

Optional pattern used to extract named token values from each basename (including extension).

• Can be a raw regex with named groups, e.g. (?P<audio_recording_id>\d+).
• Can also be the preset name canonical_filename.
  • Intended for canonical filenames like 20210428T100000Z_Five-Rivers-Dry-A_909057.flac.
  • Extracts named groups: timestamp, site_name, audio_recording_id, extension.
• If a required token cannot be resolved for a file, sourcemap rendering raises an error.

--file_metadata

JSON object (CLI string or config file object) used to provide static template token values.

• Keys must be simple token names (letters, numbers, underscore).
• Values are merged with pattern-extracted tokens per file.
• Pattern-extracted tokens take precedence for the current file.

--save_db

Controls whether the hoplite embedding database is saved after processing.

• Behavior:
  • --save_db true (or --save_db with no value): Database is saved at the location specified by --db_path.
  • --save_db false (default): If the database folder specified by --db_path already exists, it is preserved and used. If the folder does not exist, a temporary database is created, used for exports, and then deleted.
• Validation:
  • At least one of --embed, --classify, --save_db, or --recognizers must be specified.
  • You can specify --save_db true without --embed to create and save only the database.

Usage examples:

• DB only: analyze --save_db (creates and saves the database, no embeddings exported)
• Embeddings only: analyze --embed (creates embeddings, database is deleted after)
• Both: analyze --embed --save_db (creates embeddings, saves database)

--workers

Controls embedding worker count passed to perch-hoplite

• auto (default): computed from available RAM.
• Integer value: explicit worker count.

--classify

Enable Perch classify output.

• Use --classify or --classify true to enable.
• Use --classify false to explicitly disable.
• File format is controlled by --classify_filetype (csv or parquet).
• If --perch_species_list is provided, classify rows are filtered to that species set.
• Current app behavior: classify export runs in the embedding pipeline path, so in practice you must also set at least one of --embed, --save_db, or --recognizers.

--perch_species_list

Model-specific species filter for Perch global classify output.

• Supported input forms:
  • Comma/newline-separated inline string (for example: "koala, emu").
  • Repeated/list values from config files.
  • Path to a text file (one or many species entries, comma/newline-separated).
  • Preset key (currently: australian_birds_01).
• Resolution rules:
  • If the value is a path to an existing file, that file is loaded.
  • For relative paths from config files, resolution is relative to the config file directory.
  • Otherwise the value is treated as inline species text.
• Validation rules:
  • Entries are validated against the model-specific final label set used by classify export:
    • src/species_lists/perch_8.txt
    • src/species_lists/perch_v2.txt
  • Matching is case-insensitive.
  • Accepted values are canonicalized to the exact label text from those files.
  • Duplicate species are removed after canonicalization.
  • Invalid entries raise an error before processing starts.
  • Empty lists are rejected.

--classify_require_species_list

When set to true, classify mode requires --perch_species_list.

• If classify is enabled and no species list is provided, config loading raises an error.

--perch_max_detections_per_window

Limits classify output density per audio window.

• Must be an integer greater than 0.
• Defaults to 10.
• After thresholding/species filtering, only the top N detections per window are kept.

--classify_species_list

Legacy option currently retained for compatibility.

• It is accepted by config parsing, but is not used by the current Perch global classify export path.

--model_choice

Embedding model preset.

• Allowed values: perch_v2, perch_8

--dataset_name

Dataset name stored in runner configuration.

• Default: search_set
• This is currently a config value/CLI flag only; it does not change output path rendering unless some downstream consumer uses it.

--source and --output

Input and output roots.

• Both paths must already exist and mounted into the container.
• Source can be a directory or a single audio file.

--config_file

Path to config file (.yml, .yaml, or .json).

• CLI flags override values loaded from config file.

--log_level, --hoplite_log_level, --tf_log_level, --log_file

Logging controls.

• --log_level: perch-runner logs.
• --hoplite_log_level: library/root logs.
• --tf_log_level: TensorFlow C++ logs.
• --log_file: optional file output in addition to console.

Examples

Single file, recognizer output AND classify output, BAW sourcemap, add audio_recording_id column

docker run --rm \
  -v $(pwd)/tests/files/audio/gympie_np_1192_333354_20151010_152034_30_0.wav:/mnt/input/audio_file.wav \
  -v $(pwd)/tests/files/configs/koala.json:/mnt/config/recognizers.json \
  -v /path/to/output:/mnt/output \
  qutecoacoustics/perchrunner:latest analyze \
  --source /mnt/input/audio_file.wav \
  --recognizers /mnt/config/recognizers.json \
  --sourcemap_name baw_original \
  --file_metadata '{"domain":"https://api.acousticsobservatory.org.au","audio_recording_id":"1234"}' \
  --classify

Notes:

• audio_recording_id is added to classify/recognizer results if it's supplied (like in this example)
• We don't specify a perch version. Because this example recognizer has a model config indicating perch_8, that is what is used
• This particular example provides --recognizers AND --classify, therefore it will save a results file for the koala recognizer and the perch global recognizer.
• the perch global recognizer will use perch_8 as well. You can't ask for different embedding models for different outputs.

Single file, Perch classify output

docker run --rm \
  -v $(pwd)/tests/files/audio/gympie_np_1192_333354_20151010_152034_30_0.wav:/mnt/input/audio_file.wav \
  -v /path/to/output:/mnt/output \
  qutecoacoustics/perchrunner:latest analyze \
  --source /mnt/input/audio_file.wav \
  --classify \
  --save_db \
  --classify_filetype csv

Notes:

• --save_db means that the perch-hoplite database won't be removed at the end
• we don't specify --embedding_model, so it will default to perch_v2
• --classify is specified, so we expect a csv of perch_v2 classification results

Single file, embeddings parquet flat to embeddings.parquet, BAW sourcemap

docker run --rm \
  -v /path/to/audio_file.wav:/mnt/input/audio_file.wav \
  -v /path/to/output:/mnt/output \
  qutecoacoustics/perchrunner:latest analyze \
  --source /mnt/input/audio_file.wav \
  --embed \
  --embeddings_table_filetype parquet \
  --embeddings_output_path_type flat \
  --sourcemap_name baw_original \
  --file_metadata '{"domain":"https://api.acousticsobservatory.org.au","audio_recording_id":"1234"}'

Supported Audio Formats

.wav, .flac, .mp3, .ogg

Output Structure

With defaults (output_path_type=flat), output is merged by analysis into a single file per analysis type:

/mnt/output/
  embeddings.parquet
  recognizer_results.csv

If you set nested or nested_filestem, output can mirror source subdirectories.

Each Parquet file contains one row per 5-second window with columns: source, channel, offset, embeddings (serialized numpy array).

With --embeddings_table_format columns, the embeddings column is replaced by individual dimension columns (f0000, f0001, ...).

Config File

Instead of CLI flags, you can mount a YAML config file:

source: /mnt/input
output: /mnt/output
embed: true
model_choice: perch_v2
embeddings_table_format: serialized
file_glob: "*/*"

docker run --rm \
  -v /path/to/audio:/mnt/input \
  -v /path/to/output:/mnt/output \
  -v /path/to/config:/mnt/config \
  qutecoacoustics/perchrunner:latest analyze --config_file /mnt/config/config.yml

Models

Model	Embedding Dimensions	Description
perch_v2	1536	Default. Google Perch v2 bird embedding model
perch_8	1280	Google Perch v8

Models are cached in the Docker image at build time — no internet access is required at runtime.

Regenerating model species lists

Use the helper script to regenerate the model-aligned species label files under src/species_lists:

# regenerate both default models
python -m src.generate_species_lists

# regenerate one model
python -m src.generate_species_lists --models perch_8

# custom output directory
python -m src.generate_species_lists --output-dir src/species_lists

Building

# Local build (current architecture)
./build.sh

# Local build with explicit tag/version (for example, dev)
./build.sh dev

# Build and push to Docker Hub (amd64 + arm64)
./build.sh --push

The image build resolves/downloads models and embeds them in the image cache. Tests are run after build in CI and during local development.

Testing

Inside the dev container (development)

# Run all tests (network is blocked; models must be cached)
pytest

From the host against a built image

# run the full suite of tests in the container from the host
./run_tests_in_container.sh

or just paste:

IMAGE="${IMAGE:-qutecoacoustics/perchrunner:latest}"
docker run --rm --network=none --entrypoint /app/tests/run_tests "$IMAGE"

end-to-end tests from host

# run end-to-end tests only, on the host
./run_tests.sh

This runs host end-to-end tests in tests/end_to_end_tests, which execute the built container via docker run and validate produced outputs on the host. Requires pytest and other python libraries to be installed on the host environment (see requirements-host.txt).
It must be run from the root of the repo for test discovery and accessing the test files.

License

Apache 2.0