Skip to content

feat: add LiteLLM as an AI provider - #2757

Open
prodmanpd wants to merge 1 commit into
OtterMind:mainfrom
prodmanpd:feat/add-litellm-provider
Open

feat: add LiteLLM as an AI provider#2757
prodmanpd wants to merge 1 commit into
OtterMind:mainfrom
prodmanpd:feat/add-litellm-provider

Conversation

@prodmanpd

Copy link
Copy Markdown

Summary

Adds LiteLLM as a first class AI provider, alongside OpenAI, Claude, Gemini, and MiniMax.

LiteLLM exposes an OpenAI compatible endpoint (the self hosted LiteLLM proxy), so a single "LiteLLM" provider entry lets a user reach 100+ models across every provider LiteLLM supports (OpenAI, Anthropic, Gemini, Bedrock, Vertex, Azure, Groq, Mistral, DeepSeek, local models, and more) through one configuration, without needing a Vertex or GCP project for Gemini or a separate entry per vendor.

Because the LiteLLM proxy speaks the OpenAI protocol, the provider reuses the existing OpenAiChatModel client pointed at the user's configured base URL.

Affected surfaces

  • Frontend / Web
  • Backend / API / Storage
  • Database plugin / Driver
  • JCEF / Desktop packaging
  • CI / Build / Release
  • Documentation only

Verification

  • Commands and results:
    • Backend compiles (JDK 21): mvn -pl chat2db-community-web -am compile gives BUILD SUCCESS (0 errors; AiProviderEnum, AiModelFactory, AiModelConfigServiceImpl all compiled).
    • Frontend unit test (repo's own runner): tsx src/blocks/AI/components/AIModelConfigModal/modelConfigDefaults.test.ts prints AI model config default tests passed. (extended with the LiteLLM base URL default and provider switch cases).
  • Manual verification:
    • Provider LITELLM routes through AiModelFactory.openAiClient(...) (the OpenAI compatible path). Wire compatibility is the standard LiteLLM proxy OpenAI endpoint (POST <baseUrl>/v1/chat/completions), which is the same endpoint the existing OpenAI provider already targets and the repo already ships and tests.
  • UI evidence: added { label: 'LiteLLM', value: 'LITELLM' } to the provider dropdown and the api key required rule; no layout change (reuses the existing base URL and API key fields). Screenshot can be added on request.

Risk and compatibility

  • Backward compatibility: additive. OpenAI, Claude, Gemini, and MiniMax behavior is untouched.

Reviewer map

  • Start here: AiModelFactory.java (the LITELLM to openAiClient branch) and AiModelConfigServiceImpl.java (test, normalize, env fallback, and validate branches).
  • Failure condition: a LITELLM config with no reachable base_url fails exactly like an OpenAI config with a bad base URL (surfaced by the existing connection test).
  • Rollback or disable path: remove the LITELLM enum value and its branches; no data migration needed.

Contributor declaration

  • I linked the Issue that defines this change.
  • I tested the affected behavior and reported the actual results above.
  • I did not include credentials, private data, or generated build output.
  • I disclosed substantial AI assistance below, or this PR contains no substantial AI-generated code.

AI assistance: This provider integration was implemented with AI assisted coding and reviewed before submission.

@auenger

auenger commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

整体 Review 结论:本次 LiteLLM Provider 的改动方向和实现逻辑没有发现阻塞问题。

从 PR 描述看,提交者已经完成:

  • JDK 21 下 chat2db-community-web 及其依赖模块的编译;
  • 前端 modelConfigDefaults 相关测试。

不过,目前描述中没有看到 Java 17 环境下的完整 Community 打包、实际服务启动,以及连接真实 LiteLLM Proxy 完成模型调用的验证记录。请提交者确认是否做过上述启动和联调测试;如已完成,建议补充测试命令与结果。

我基于当前 PR HEAD(cab7ecd153654edbbcb8fba633ec2deeacbb409f)补充进行了以下验证:

  • Java 17.0.6 下受影响后端模块编译:通过(6 个 Reactor 模块,测试跳过);
  • 前端 yarn run test:ai-model-config:通过;
  • Java 17.0.6 下 Community 后端完整打包:通过(48 个 Reactor 模块,打包命令显式跳过测试);
  • 使用隔离的 user.home,以 Community / OFFLINE 模式绑定 127.0.0.1:18025 进行启动冒烟:通过,Tomcat 正常监听,Application 在约 4.3 秒内启动;验证后临时服务已停止。

代码 Review 方面:

  • Provider 枚举、前后端类型、客户端工厂和默认配置链路完整;
  • 复用 OpenAI-compatible 客户端的方式合理;
  • 对 Base URL 末尾 /v1 的归一化处理符合 LiteLLM 接口形式;
  • 未发现对现有 Provider、数据库结构或依赖产生破坏性影响。

剩余验证缺口是当前环境没有配置可用的真实 LiteLLM Proxy,因此未执行端到端模型请求。另请注意,默认地址 http://localhost:4000 对本机部署友好,但 Docker 场景通常需要配置为容器可访问的地址。

综上,本 PR 整体没有发现阻塞性问题;提交者确认实际 LiteLLM 联调情况后即可推进合并。

@auenger auenger left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review 通过。

本次 LiteLLM Provider 的前后端类型、默认配置、客户端工厂和 OpenAI-compatible 调用链路完整,未发现阻塞性问题。已在当前 PR HEAD 上补充完成 Java 17 后端模块编译、Community 完整打包、前端专项测试和本地服务启动冒烟,结果均通过。

真实 LiteLLM Proxy 的端到端模型调用当前未执行,建议合并前由提交者确认联调结果;该项作为非阻塞验证建议。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Review

Development

Successfully merging this pull request may close these issues.

3 participants