PYA-1149: benchmark on speaker attributed ASR

[clement-pages]

No description provided.

[Copilot]

Pull request overview

This PR adds benchmarking capabilities for speaker-attributed automatic speech recognition (ASR) by extending the existing diarization benchmark command to support transcription pipelines and metrics.

Key Changes

• Added transcription metrics (WER and TCP-WER) for both word-level and turn-level transcription
• Extended the benchmark command to detect and handle transcription pipelines alongside diarization pipelines
• Added STM file format support for saving transcription outputs

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 11 comments.

File	Description
src/pyannote/audio/main.py	Extended benchmark functionality with transcription support, added Granularity enum, get_transcription() and write_stm() functions, and reorganized output directory structure to separate diarization and transcription results
pyproject.toml	Added [transcription] extra to pyannote-metrics dependency and configured git source for development branch

Comments suppressed due to low confidence (2)

src/pyannote/audio/main.py:885

• The speaker count confusion matrix is computed using speaker_diarization which may not be defined for transcription-only pipelines. This code should be guarded to only execute for speaker diarization pipelines:

# increment speaker count confusion matrix
if is_sd_pipeline or is_streaming_sd_pipeline:
    pred_num_speakers: int = len(speaker_diarization.labels())
    true_num_speakers: int = len(file["annotation"].labels())
    speaker_count.setdefault(true_num_speakers, dict()).setdefault(
        pred_num_speakers, 0
    )
    speaker_count[true_num_speakers][pred_num_speakers] += 1

        # increment speaker count confusion matrix
        pred_num_speakers: int = len(speaker_diarization.labels())
        true_num_speakers: int = len(file["annotation"].labels())
        speaker_count.setdefault(true_num_speakers, dict()).setdefault(
            pred_num_speakers, 0
        )
        speaker_count[true_num_speakers][pred_num_speakers] += 1

src/pyannote/audio/main.py:636

• The function docstring states "Benchmark a pretrained diarization PIPELINE" but the function now supports both diarization and transcription pipelines. The docstring should be updated to reflect this:

"""
Benchmark a pretrained diarization or transcription PIPELINE

This will run the pipeline on all files in the specified protocol and subset,
save the results in RTTM format (for diarization) or STM format (for transcription),
and compute relevant metrics (DER for diarization, WER/TCP-WER for transcription).
If `--optimize` is used for diarization pipelines, it will also post-process predictions
by filling short within speaker gaps and save the results in a separate file.
"""

    """
    Benchmark a pretrained diarization PIPELINE

    This will run the pipeline on all files in the specified protocol and subset,
    save the results in RTTM format, and compute the Diarization Error Rate (DER)
    for each file. If `--optimize` is used, it will also post-process predictions
    by filling short within speaker gaps and save the results in a separate file.
    """

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[hbredin]

I'd rather not require the transcription extra by default.

Maybe group all dependencies only needed by the pyannote.audio ... CLI in the cli extra.

Something like this should do the trick.

uv add --optional cli pyannote-metrics --extra transcription

[hbredin]

Make sure it does not break if meeteval is not installed.
Also, since meeteval is imported directly here, it should be added to the cli extra (even if it is a dependency of pyannote.metrics[transcription]).

Generally speaking, any directly imported dependencies should be in project.toml.

[hbredin]

Let's make it work also when pyannote.metrics.transcription fails at import time.

[hbredin]

Make it work even if SegLST could not be imported. Use "SegLST" maybe?

[hbredin]

Same remark about SegLST possibly not imported.

[hbredin]

In retrospect, I think it would be more readable to always compute every metrics (and simply use "empty" predictions when it does not exist in the output). What do you think?

Same for the reference -- if not there, use empty reference.

[hbredin]

Or you could add two flags --diarization and --transcription and fail when, for instance, we ask for transcription and there is no reference for it. What do you think?

[clement-pages]

Well, I think the second option with two flags is a good idea, as it's a more explicit way to compute metrics (compute transcription metrics only if user request it, whereas current behavior is to compute those metrics if the pipeline support transcription, which is more an implicit behavior). So let's do this !

[Copilot]

Pull request overview

Copilot reviewed 2 out of 3 changed files in this pull request and generated 12 comments.

Comments suppressed due to low confidence (1)

src/pyannote/audio/main.py:705

• The docstring on lines 698-705 is outdated and doesn't reflect the changes made in this PR. It still says "Benchmark a pretrained diarization PIPELINE" and only mentions diarization-specific functionality (RTTM format, DER), but now the function also supports transcription benchmarking with STM format and WER metrics. The docstring should be updated to reflect both capabilities.

    """
    Benchmark a pretrained diarization PIPELINE

    This will run the pipeline on all files in the specified protocol and subset,
    save the results in RTTM format, and compute the Diarization Error Rate (DER)
    for each file. If `--optimize` is used, it will also post-process predictions
    by filling short within speaker gaps and save the results in a separate file.
    """

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.