Loading model NOT from Huggingface?

[molntamas]

A have a fine tuned text classification model. (based on: intfloat/multilingual-e5-large-instruct)

I was thinking about hosting inference with infinity, as I understand this should be possible.
If I run a docker based image, e.g. for CPU specialized one, can I specifiy a local file path or a custom URL to load the model from?

Thank you! :)

[wirthual]

Yes this is possible.

You can mount your local model from the host machine into the container and then specify the path inside the container when launching the infinity container.

Check out the example from this issue:

cd /tmp
git install lfs 
mkdir models && cd models && git clone https://huggingface.co/BAAI/bge-m3 && cd ..
docker run -it   -v /tmp/models:/models  -p 8081:8081  michaelf34/infinity:0.0.70 v2  --model-id "/models/bge-m3"  --served-model-name bge-m3  --port 8081