Running models on seperate devices

[Sens612]

Hi,

I was wondering if it it possible to run 2 models on seperate devices. For example if I want to run

all-MiniLM-L6-v2 on CPU with torch engine
cross-encoder/ms-marco-MiniLM-L-6-v2 on GPU with torch engine

Is that possible to do with the CLI? If so what would the CLI command look like

infinity_emb v2 ...

[michaelfeil]

@Sens612 There is a section in the Readme on CLI usage.

e.g.

infinity_emb v2 \
--model-id all-MiniLM-L6-v2 --engine optimum --device cpu --batch-size 8 \
--model-id cross-encoder/ms-marco-MiniLM-L-6-v2  --engine torch --device cuda --batch-size 16 

[Sens612]

Aha, I had no clue you could do it that way, I thought majority of the arguments were forced on both models. Looking in the README I don't think this is shown clearly with an example. Either way thank you, and great work on the repo.

[michaelfeil]

infinity-emb-py3.11(base) michael@laptop:~/infinity$ infinity_emb v2 --help
                                                                                                                     
 Usage: infinity_emb v2 [OPTIONS]                                                                                    
                                                                                                                     
 Infinity API ♾️  cli v2. MIT License. Copyright (c) 2023-now Michael Feil                                            
 Multiple Model CLI Playbook:                                                                                        
 - 1. cli options can be overloaded i.e. `v2 --model-id model/id1 --model-id/id2 --batch-size 8 --batch-size 4` @Sens612      
 - 2. or adapt the defaults by setting ENV Variables separated by `;`: INFINITY_MODEL_ID="model/id1;model/id2;" &&   
 INFINITY_BATCH_SIZE="8;4;"                                                                                          
 - 3. single items are broadcasted to `--model-id` length, making `v2 --model-id model/id1 --model-id/id2            
 --batch-size 8` both models have batch-size 8. 

[michaelfeil]

Maybe I should advertise the feature more prominently, but yeah!