Skip to content

Feat/add litellm provider - #53

Open
prodmanpd wants to merge 4 commits into
xunbu:mainfrom
prodmanpd:feat/add-litellm-provider
Open

Feat/add litellm provider#53
prodmanpd wants to merge 4 commits into
xunbu:mainfrom
prodmanpd:feat/add-litellm-provider

Conversation

@prodmanpd

@prodmanpd prodmanpd commented Jul 17, 2026

Copy link
Copy Markdown

Summary

Add LiteLLM as a provider with a native SDK code path, giving users direct access to 100+ LLM providers (Anthropic, Google, Azure, Bedrock, Groq, Ollama, and more) for document translation without needing an OpenAI-compatible proxy.

Changes

  • docutranslate/agents/agent.py - Added _call_litellm_sync() and _call_litellm_async() methods that use litellm.completion() / litellm.acompletion() instead of raw httpx POST. All 4 HTTP call sites patched (send, send_async, _continue_fetch, _continue_fetch_async) to route through litellm when provider == "litellm"
  • docutranslate/agents/provider/provider.py - Added "litellm" to ProviderType
  • docutranslate/agents/thinking/thinking_factory.py - Added thinking mode delegation by model name for litellm
  • tests/test_litellm_provider.py - 8 unit tests

Implementation details

When --provider litellm is set, the Agent bypasses raw httpx and calls litellm's SDK directly:

def _call_litellm_sync(self, data: dict) -> dict:
    import litellm
    kwargs = {
        "model": data["model"],
        "messages": data["messages"],
        "temperature": data.get("temperature"),
        "top_p": data.get("top_p"),
        "stream": False,
        "drop_params": True,
    }
    if self.key and self.key != "xx":
        kwargs["api_key"] = self.key
    if data.get("response_format"):
        kwargs["response_format"] = data["response_format"]
    kwargs = {k: v for k, v in kwargs.items() if v is not None}
    response = litellm.completion(**kwargs)
    return response.model_dump()
  • drop_params=True silently handles cross-provider parameter incompatibilities (e.g. frequency_penalty rejected by Anthropic, response_format rejected by some providers)
  • litellm.acompletion() used for async path so concurrent translation batches work
  • API key forwarded when set, otherwise litellm reads provider env vars (ANTHROPIC_API_KEY, OPENAI_API_KEY, etc.)
  • Thinking mode resolved by model name since litellm proxies multiple providers (e.g. qwen-plus gets Qwen's enable_thinking, gemini-2.5-flash gets Google's reasoning_effort)
  • Response is model_dump()'d to dict so existing token counting and response parsing works unchanged

Tests

8 unit tests covering provider detection, thinking mode delegation, and type system.

Full regression: 64 passed (56 existing + 8 new), 0 failures.

Example usage

# Direct SDK usage - litellm routes to the right provider
pip install litellm
export ANTHROPIC_API_KEY=sk-ant-...

docutranslate translate \
  --base-url http://localhost:4000/v1 \
  --model anthropic/claude-sonnet-4-6 \
  --provider litellm \
  input.pdf
from docutranslate.agents.agent import Agent, AgentConfig

agent = Agent(AgentConfig(
    base_url="unused-for-litellm",
    model_id="anthropic/claude-sonnet-4-6",
    provider="litellm",
))
# Agent uses litellm.completion() instead of raw httpx
# No OpenAI-compatible proxy needed

Risk / Compatibility

  • Existing providers untouched - litellm path only activates when provider == "litellm"
  • Raw httpx path unchanged for all other providers
  • litellm is imported lazily inside the method bodies

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant