Plan: replace LiteLLM with a gateway adapted to NaN/Helmcode #111

Evaluation: https://github.com/helmcode/nan-devops/pull/601 (docs/GATEWAY-EVALUATION.md)

Recommended option

B: a small Go gateway, introduced as a strangler in front of LiteLLM, with LiteLLM kept behind it as the fallback backend until the last model group moves. Serves both tenants (api.nan.builders, api.helmcode.com) as two deployments of one codebase.

Why, in short:

• Our real spec is narrow: 7 providers (5 OpenAI-compatible), ~10 inference endpoints, ~16 admin calls. Our business rules already live in 12,681 lines of hooks with ~24k lines of tests.
• Pruning LiteLLM leaves ~150k to 200k lines of Python and does not fix the worker model (revocation, corrupt metrics) or the cost (11 cores / 84 GiB for ~11 req/s in community, at KEDA max 83.5% of the week).
• No OSS gateway removes our work: TensorZero archived, Portkey frozen, Bifrost OSS has no multi-node, Envoy AI Gateway has no keys/tenancy.
• Keys migrate without rotation (LiteLLM stores unsalted SHA-256). Every step can roll back per model group.

Estimate: ~28 engineer-weeks (~25 if Anthropic/Gemini stay on LiteLLM). First model served directly at ~week 8, LiteLLM decommissioned at ~month 6.

Phased plan

• Phase 0 (2 w), prerequisites useful for any option
  • Fix LiteLLM metrics and KEDA flapping: done in nan-devops#606. The cause was a vmagent labeldrop collision (fake counter resets), not a missing PROMETHEUS_MULTIPROC_DIR.
  • Land nan-devops#599 (SpendLogs to ClickHouse design: store, schema and retention for the gateway's native sink; generic_api stays only as a fallback bridge)
  • Contract and replay suite (extend litellm_gate.py, port hook cases, SSE fixtures per provider); capture baselines on LiteLLM
• Phase 1 (3 w): transparent front. Gateway forwards 100% to LiteLLM, streaming. Measure overhead and resources. Milestone M1: gateway in the path of all community traffic with 0 behavior diffs.
• Phase 2 (3 w): gateway auth and limits. SHA-256 key lookup (read-only on LiteLLM_VerificationToken), Valkey pub/sub revocation, stale-if-error, rate limits on the shared Valkey keys. M2: cross-worker revocation solved.
• Phase 3 (2 w): first direct models. qwen3-embedding, qwen3.6, gemma4 (community), with metering, spend rows and ClickHouse logs. Per-group routing table with instant rollback. M3 (~week 8): first model served without LiteLLM.
• Phase 4 (4 w): community resold and fallback groups. glm5.x, deepseek-v4-flash, qwen3.8-flash, mimo, audio, rerank, Responses. One week of meter shadow-comparison per paid group. M4: community fully off LiteLLM.
• Phase 5 (4 to 6 w): enterprise. Chat service key path, images, per-org teams and budgets. Anthropic/Gemini last or deferred (D2). M5: enterprise off LiteLLM (except deferred groups).
• Phase 6 (4 w): ownership and decommission. Key/team management API, cloud-api and enterprise-api clients, ledger schema, LiteLLM scaled to 0 then removed. M6: LiteLLM gone.

Pending LiteLLM work under this plan

• 1.102.1 upgrade: only if the gate passes without new fork patches (truncate SpendLogToolIndex first).
• generic_api ClickHouse ingest from LiteLLM: NOT planned (decided 2026-09-29). ClickHouse is the gateway's native sink; #599 provides the store, schema and retention.
• Key-cache stale-if-error and cross-worker revocation: no fork patch; built into the gateway (Phase 2).

Decision points for the owner

• D1 Option A (fork+prune), B (Go gateway) or C (OSS). Recommended: B.
• D2 Keep Anthropic and Gemini (6 enterprise deployments) on a small LiteLLM behind the gateway instead of building translation now. Recommended: yes.
• D3 Spend ledger: LiteLLM_Daily*Spend-compatible rows during migration, new schema in Phase 6. Recommended: yes.
• D4 Key management: new API with fixed semantics vs a compatible shim of the LiteLLM endpoints. Recommended: new API.
• D5 Do the 1.102.1 upgrade meanwhile? Recommended: only if cheap.
• D6 Staffing: one senior engineer with agents for ~6 months.
• D7 Decided 2026-09-29: no LiteLLM generic_api ingest. Stand up the ClickHouse store from #599 and let the gateway write to it natively.
• D8 Done in nan-devops#606 (merged 2026-09-29).

Option A is the better choice if: the enterprise roadmap needs several non-OpenAI providers (Bedrock, Vertex, Azure, Cohere) within 6 months; we cannot staff ~6 engineer-months; or short-term stability outranks the one-year cost. See section 8 of the evaluation.