multi-speaker ?

[lumpidu]

Hi @mush42 , thanks for your hard work !

Is it possible to train multiple speaker embeddings at the same time ? I'd like to experiment with ConvNeXt

[mush42]

@lumpidu Initial support for multi-speaker/lingual models is there, but it has not been tested.

[lumpidu]

How could I try to test it ? How does the datasets need to be prepared and how is the embedding integrated into the model ?

[mush42]

@lumpidu

From this section in the readme:

The metadata.csv file can contain 2, 3 or 4 columns delimited by | (bar character) in one of the following formats:
• 2 columns: file_id|text
• 3 columns: file_id|speaker_id|text
• 4 columns: file_id|speaker_id|language_id|text

We use simple randomly initialized embedding table indexed by speaker_id.

[lumpidu]

Thx ! So this means by simply adding the columns, training is automatically done on those additional embeddings. That is quite nice!
Do you have an estimate on how much these additional embeddings increase model size ?

How do I infer with these additional embeddings ?

[EyalcoAudc]

Thx ! So this means by simply adding the columns, training is automatically done on those additional embeddings. That is quite nice! Do you have an estimate on how much these additional embeddings increase model size ?

I didn't see any increase in the model size (Default backbone)

How do I infer with these additional embeddings ?

You need to change the infer script, when calling model.prepare_input you need to set the speaker (speaker=...)