codenames

Building AI agents for Codenames. Three phases of work in this repo:

1. LLM baselines — Anthropic API players (Claude Sonnet / Opus).
2. Embedding baselines — sentence-transformer codemasters with several scoring variants. Useful for cross-pair experiments where an embedding codemaster is paired with a Claude guesser.
3. RL training — fine-tune a small open model (Gemma 3) with GRPO, using an unmodified copy of the same model as guesser. Work in progress.

The headline metric is net_mean = mean(Red revealed) − P(enemy) − 5·P(assassin). See AGENTS.md for the rationale.

Setup

Uses uv for environment + deps.

uv sync                       # install everything
echo "ANTHROPIC_API_KEY=..." > .env   # for Claude-based players
echo "VOYAGE_API_KEY=..."   >> .env   # only needed for the Voyage embedding study

Running a game

uv run python main.py --red claude --blue embedding-margin2 --seed 7

Available agent types (see main.py): human, random, claude, embedding, embedding-prefix, embedding-margin, embedding-margin2, cheat.

Running tournaments

# 50 games of Claude self-play, first-turn mode
uv run python tournament.py --mode first-turn --red claude --blue claude \
    -n 50 --workers 2 --log-dir logs/first-turn-sonnet-low

# Cross-pair: embedding codemaster + Claude guesser
uv run python tournament.py --mode first-turn \
    --red-cm embedding-margin2 --red-gr claude \
    -n 50 --workers 2 --log-dir logs/first-turn-embedding-margin2-cm-claude-gr

Aggregated benchmark results live in logs/RESULTS.md, and visualizations in studies/results.ipynb.

Project layout

src/codenames/        Game class, player ABCs, player implementations
main.py               CLI to play one game
tournament.py         Batch runner with --mode {full, single-team, first-turn}
first_turn.py         Single-team first-turn helper CLI
studies/              Research scripts and JSONL datasets
  embedding_alignment.py        Logistic regression: Claude vs embedding sim
  results.ipynb                 Matplotlib companion to RESULTS.md
logs/                 Generated game records (gitignored except RESULTS.md)
references/           Cloned reference repos (gitignored, read-only)
AGENTS.md             Notes for AI coding assistants

Players

All players subclass Codemaster or Guesser from src/codenames/players/base.py. Each player exposes a config() method returning a JSON-serializable dict, which is logged per game for tournament analysis.

Heavy agents (LLM, embedding, local HF) are not auto-imported by from codenames.players import *. Import explicitly:

from codenames.players.llm import ClaudeCodemaster, ClaudeGuesser
from codenames.players.local_lm import LocalLMCodemaster, LocalLMGuesser

Selected findings

• Net-mean ranking on first-turn (n=50, self-play): Opus 4.7 high (+1.28) > Opus 4.7 low (+1.22) > Sonnet 4.6 low (+1.18) > Sonnet 4.6 high (+0.90)
• Embedding-vs-Claude alignment (500-triple forced-choice study): MiniLM, mpnet, BGE, and Voyage all give ~17-20% pseudo-R² on Claude's choice; gap dominates absolute similarity level. See studies/results.ipynb.
• Best embedding codemaster paired with Claude guesser: Margin2EmbeddingCodemaster (absolute-buffer scoring) at net_mean = +0.76, still well below LLM self-play.