Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion docs/agents/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,21 @@ The `model_args` parameter allows you to pass additional arguments to the model,

The `api_base` parameter allows you to specify a custom API endpoint. This is useful when:

- Using a local model server (e.g., Ollama, llama.cpp, llamafile)
- Using a local model server (e.g., Ollama, llmman, llama.cpp, llamafile)
- Routing through a proxy
- Using a self-hosted model endpoint

For example, [llmman](https://github.com/llmmanorg/llmman) serves the Ollama API on port `17434` instead of `11434`, so it can be used through the `ollama/` provider by pointing `api_base` at it:

```python
from any_agent import AgentConfig

config = AgentConfig(
model_id="ollama/gemma4",
api_base="http://localhost:17434",
)
```

The `api_key` parameter allows you to explicitly specify an API key for authentication. By default, `any-llm` will automatically search for common environment variables (like `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, etc.).

See the [AnyLLM Provider Documentation](https://docs.mozilla.ai/any-llm/providers/) for the complete list of supported providers.
3 changes: 3 additions & 0 deletions docs/cookbook/agent-with-local-llm.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ we will use [Ollama](https://ollama.com/) to run our LLM locally!

First, install Ollama by following their instructions: https://ollama.com/download

>**Note**: [llmman](https://github.com/llmmanorg/llmman) serves the same Ollama API on port `17434`.
> To use it instead, keep the `ollama/` prefix in `model_id` and add `api_base="http://localhost:17434"` to your `AgentConfig`.

### Picking an LLM

Pick a model that you can run locally based on your hardware and download it from your terminal. For example:
Expand Down
3 changes: 3 additions & 0 deletions docs/cookbook/agent_with_local_llm.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,9 @@
"\n",
"First, install Ollama by following their instructions: https://ollama.com/download\n",
"\n",
">**Note**: [llmman](https://github.com/llmmanorg/llmman) serves the same Ollama API on port `17434`.\n",
"> To use it instead, keep the `ollama/` prefix in `model_id` and add `api_base=\"http://localhost:17434\"` to your `AgentConfig`.\n",
"\n",
"### Picking an LLM\n",
"\n",
"Pick a model that you can run locally based on your hardware and download it from your terminal. For example:\n",
Expand Down
4 changes: 2 additions & 2 deletions src/any_agent/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,8 @@ class AgentConfig(BaseModel):
api_base: str | None = None
"""Custom API endpoint URL for the model provider.

Use this to specify custom endpoints for local models (Ollama, llama.cpp, etc.) or proxy services.
For example: `http://localhost:11434/v1` for Ollama.
Use this to specify custom endpoints for local models (Ollama, llmman, llama.cpp, etc.) or proxy services.
For example: `http://localhost:11434/v1` for Ollama, or `http://localhost:17434` for llmman.
"""

api_key: str | None = None
Expand Down