llm-lab

Benchmark harness for local LLMs (Darwin, Gemma4, etc.) over OpenAI-compatible endpoints.

Install

cd /home/jooyu/llm-lab
uv venv && source .venv/bin/activate
uv pip install -e '.[dev]'

Use

# Make sure the local LLM proxy is reachable. WSL→Windows host:
HOST=$(ip route show default | awk '{print $3}')   # e.g. 172.24.0.1
curl -s "http://$HOST:4000/v1/models"

# Run the default suite against Darwin
llm-lab run --model darwin --base-url "http://$HOST:4000/v1"

# Swap the loaded model on llama-server, then run Gemma4
llm-lab run --model gemma4 --base-url "http://$HOST:4000/v1"

# Compare two runs
llm-lab compare runs/*-darwin.json runs/*-gemma4.json

# Inspect one run's responses
llm-lab show runs/20260414-120000-darwin.json

Layout

src/llm_lab/
  models.py    # Prompt, PromptSuite, PromptResult, RunResult
  runner.py    # async runner against /v1/chat/completions
  cli.py       # typer CLI: run | compare | show | version
prompts/
  default.json # 10 prompts: reasoning, code, korean, summarization, instruction, general
runs/          # output JSON (gitignored)

Suite format

{
  "name": "default",
  "prompts": [
    {"id": "...", "category": "reasoning|code|korean|summarization|instruction|general", "text": "..."}
  ]
}

Categories drive aggregation in the comparison report. expect is optional and not currently scored automatically — leave for future LLM-as-judge.

Notes

• The runner targets OpenAI-compatible /chat/completions. Works with the local llm-proxy.py (port 4000) or any compatible endpoint.
• Token counts depend on whether the endpoint returns a usage block. llama-server does.
• Sequential by design — local single-slot inference (-np 1).