tiny-llm-gate

A memory-conscious LLM gateway that speaks OpenAI, OpenRouter, Anthropic and Gemini — point an OpenRouter app at http://<host>:4001/api/v1 and it works unchanged.

AI Installation Prompt

Install tiny-llm-gate on a generic Linux or macOS host. Done when curl -sH "Authorization: Bearer <client-key>" http://127.0.0.1:4001/api/v1/models returns a JSON object with a non-empty data array.

1. Clone: git clone https://github.com/nSimonFR/tiny-llm-gate && cd tiny-llm-gate
2. Read first (in this order): SPEC.md — the authoritative spec of the running surface, then README.md, go.mod, cmd/tiny-llm-gate/main.go, testdata/example-config.yaml. Toolchain is Go ≥ 1.25. SPEC.md wins over the README where they disagree — it was derived directly from the source.
3. Build (try in this order):
   • nix build && cp result/bin/tiny-llm-gate ~/.local/bin/
   • go build -o ~/.local/bin/tiny-llm-gate ./cmd/tiny-llm-gate Verify: tiny-llm-gate --help lists -config (or the equivalent flag).
4. Write a config.yaml (see "Config reference" below and testdata/example-config.yaml). At minimum: one providers: entry and one models: entry naming it. To require keys from clients, add client_keys: — without it the gateway accepts any caller.
5. Run: tiny-llm-gate -config config.yaml. Default listen is :4001.
6. Point any OpenAI-compatible client at http://<host>:4001/v1, an OpenRouter client at http://<host>:4001/api/v1, or an Anthropic client (Claude Code: ANTHROPIC_BASE_URL) at http://<host>:4001, using a key from client_keys:.
7. NixOS hosts have a services.tiny-llm-gate module — see module.nix.

Without client_keys: the gateway is open to anyone who can reach its port — keep the default 127.0.0.1 listen or add keys before exposing it.

Why

If you self-host LLMs (Ollama, llama.cpp) on a small box and want a single endpoint that:

• speaks the OpenAI API,
• rewrites client-facing model names to real upstream models,
• falls back transparently from one upstream to another on failure,
• streams responses token-by-token,

…your main option today is LiteLLM. It's great, but it's a full Python stack (~65 MB RSS + sizeable dependency tree). On a Raspberry Pi 5 with 4 GB of RAM that's a meaningful chunk of the memory budget.

tiny-llm-gate is a single Go binary doing the same job in under 10 MB of RSS.

Status

v0.3.0: OpenAI & Gemini frontends, OpenAI backend, streaming, aliases, fallbacks.

See ROADMAP.md for remaining phases (production cutover, SIGHUP hot-reload).

Quick start

cat > config.yaml <<'YAML'
listen: 127.0.0.1:4001

providers:
  ollama:
    type: openai
    base_url: http://192.168.1.10:11434/v1
    api_key: ollama

models:
  "gemma3:4b":
    provider: ollama
    upstream_model: gemma3:4b

aliases:
  "gpt-4o-mini": "gemma3:4b"
YAML

go run ./cmd/tiny-llm-gate --config config.yaml

Now point any OpenAI SDK at http://127.0.0.1:4001/v1 and request model: "gpt-4o-mini" — it'll hit your Ollama instance as gemma3:4b.

Config reference

listen: 127.0.0.1:4001      # host:port (default 127.0.0.1:4001)

providers:                  # upstream LLM endpoints
  <name>:
    type: openai            # only "openai" today
    base_url: <url>         # e.g. http://host:11434/v1
    api_key: <string>       # Bearer token; omit for unauthenticated upstreams

models:                     # canonical model names
  <name>:
    provider: <provider>
    upstream_model: <id>    # model id actually sent to provider
    fallback:               # optional: try these models on 5xx upstream errors
      - <name>
      - <name>
    info:                   # optional OpenRouter catalog metadata
      vendor: openai        # → also addressable as "openai/<name>"
      context_length: 400000
      input_modalities: [text, image]
      pricing: {prompt: "0.000001", completion: "0.000004"}

aliases:                    # client-facing model name → canonical model
  <alias>: <model>          # chains are supported (cycle-detected)

client_keys:                # optional; when set, every LLM route needs a key
  - name: my-app            # shown as `client` in logs and by GET /api/v1/key
    key_file: /run/secrets/my-app-key   # or key: <string>
    models: ["openai/*"]    # optional allowlist (names, slugs, prefix*)
    rpm: 60                 # optional requests/minute

See testdata/example-config.yaml for a fuller example.

Endpoints

Method	Path	Purpose
POST	/v1/chat/completions	OpenAI chat, streaming + non-streaming; also accepts OpenRouter's models, reasoning, provider, usage
POST	/v1/embeddings	OpenAI embeddings
POST	/v1/responses	OpenAI Responses API, any backend (stateless)
POST	/v1/messages	Anthropic Messages: Claude passthrough, or any gate model translated
GET	/v1/models	OpenRouter-shape catalog: names, aliases, vendor/model slugs
GET	/v1/key, /v1/credits	OpenRouter key check / credits
POST	/v1beta/models/{m}:generateContent	Gemini chat, non-streaming
POST	/v1beta/models/{m}:streamGenerateContent	Gemini chat, streaming (newline-delimited JSON)
POST	/v1beta/models/{m}:embedContent	Gemini single-item embedding
POST	/v1beta/models/{m}:batchEmbedContents	Gemini batch embedding
GET	/health	liveness
GET	/ready	readiness (config loaded)

Every /v1/... route above is also served at /api/v1/... (OpenRouter's base path).

Gemini requests are transparently translated to OpenAI format before being forwarded to the upstream — you can point an AFFiNE Gemini provider at this gateway and route to an Ollama backend.

Memory discipline

Binary size	6.5 MiB stripped
RSS idle	6.9 MiB measured
Runtime deps	gopkg.in/yaml.v3 only
HTTP client	stdlib net/http, bounded idle pool, HTTP/2 disabled for predictable streaming
Request body cap	8 MiB
GOMEMLIMIT (recommended)	20MiB
MemoryMax (systemd, recommended)	30M

The CI (TODO) fails PRs whose RSS regresses past these targets.

Running on NixOS

Use the flake input directly:

# flake.nix
{
  inputs.tiny-llm-gate.url = "github:nSimonFR/tiny-llm-gate";

  outputs = { nixpkgs, tiny-llm-gate, ... }: {
    nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
      modules = [
        tiny-llm-gate.nixosModules.default
        {
          services.tiny-llm-gate = {
            enable = true;
            package = tiny-llm-gate.packages.aarch64-linux.default;
            settings = {
              listen = "127.0.0.1:4001";
              providers.ollama = {
                type = "openai";
                base_url = "http://192.168.1.10:11434/v1";
                api_key = "ollama";
              };
              models."gemma3:4b" = {
                provider = "ollama";
                upstream_model = "gemma3:4b";
              };
              aliases."gpt-4o-mini" = "gemma3:4b";
            };
          };
        }
      ];
    };
  };
}

The systemd unit applies sandboxing (DynamicUser, ProtectSystem=strict, …) and sets GOMEMLIMIT=20MiB and MemoryMax=30M by default. Both are tunable via the module options.

Comparison

	tiny-llm-gate	LiteLLM	Bifrost	one-api
Runtime	Go	Python	Go	Go + React
RSS idle	~7 MiB	~65 MiB	~25 MiB	~60 MiB
YAML config	✅	✅	partial	DB-backed
Server-side fallbacks	✅	✅	per-request	✅
Model aliases	✅ (unified)	✅ (two kinds)	per-key	via UI
Streaming (SSE)	✅	✅	✅	✅
Gemini-format frontend	roadmap	✅	partial	❌
OAuth backend	❌	✅	❌	❌

Non-goals

• Cost tracking in USD
• Prometheus /metrics
• Multi-tenant billing / quotas
• Request rate limiting
• Tokenizer-based features (counting, truncation)

Bring your own observability. Structured JSON logs are on stderr.

Extensibility

Frontends and backends are the two extension points. Adding, say, Anthropic's /v1/messages frontend or an Anthropic-native backend is a single package implementing a small interface — no changes to the router.

This is enforced in the tree layout:

internal/
├── config/       # YAML + validation
├── resolve/      # model name → provider decision
└── server/       # HTTP wiring + frontend + backend (monolithic today)

Once Phase 3/4 land, the server package will split into frontends/ and backends/.

License

MIT (see LICENSE once added).

Contributing

Early days — issues and discussions welcome at https://github.com/nSimonFR/tiny-llm-gate/issues.