LiteLLM Proxy

[kao73]

Currently I'm using Aider with LiteLLM as Proxy configured as described in the documentation: https://docs.litellm.ai/docs/providers/litellm_proxy
I'd like to try AiderDesk using the same provider, but I can't find if the AiderDesk can support LiteLLM in proxy mode.
Is it possible?

[wladimiiir]

AiderDesk has direct support for LiteLLM - you should be able to set it up via the Model Library. However, I personally do not use this provider, so I am not really sure if that's something really supported. The integration with the provider has been submitted by @jutaz, so maybe he will be able to answer your question.

[kao73]

Thanks! Yes, I tried the LiteLLM provider, but it doesn't work for me.
I'm using the litellm_proxy/<model_id> prefix in Aider. The AiderDesk is using the litellm/<model_id> no matter what I put into the prefix field of the LiteLLM provider settings.

[wladimiiir]

Hm, I see, so this is a specific provider different from LiteLLM, I guess. Can you try to set it up as an OpenAI Compatible provider to see it that will work?

Change the Base URL and set the API Key, if needed.

[kao73]

Thank works! Thank you for support!
The only issue: it can't extract a list of available models. Not a big deal.

[wladimiiir]

As the LiteLLM Proxy provider does seem to have /models endpoint, it's not possible to get the models automatically, so you will need to add them manually via the Model Library. I guess you have some config.yaml file for the provider you are using. To have it fully supported, we will need to add direct support for this provider to be able to parse the models as well. I will add that to the backlog.

[jutaz]

As the LiteLLM Proxy provider does seem to have /models endpoint

We use a different endpoint which provides info on models, as OpenAI compatible /models endpoint does not include context sizes and cost.

This sounds like a misconfiguration issue on LiteLLM - I'm running litellm proxy no problem, getting models fetched automatically, i.e.:

[MatthewDaniels]

Hey all, in case anyone else is still facing the issue where LiteLLM Proxy is not working as expected, these are the things I did to make it work (including listing the models):

• Use the LiteLLM provider in Aider Desk
• Use the prefix litellm_proxy in the Aider Desk provider config - this is not needed, I tested with just litellm and it worked also
• Give the API key in LiteLLM the MANAGEMENT permission also (the Default permission usually includes this)

The last step is what allowed Aider Desk to list the models, prior to that I was getting an error.

Hope this helps someone.

---

Note to debug my setup, I hovered over the little exclamation mark on the provider in Aider Desk and it showed the error. I then used the LiteLLM logs (I use Docker for mine - so either docker logs on the command line or just use docker desktop) and this let me know, that the API key - which was set to ONLY be an LLM key - did not have the permissions to call the models/info route in LiteLLM.