feat: mmbench — agent-vs-agent multimodal eval system

[nwaughachukwuma]

Summary

Introduces mmbench/, a self-contained agent-vs-agent benchmark system that measures whether giving an AI agent the mm CLI makes it more capable and faster at real-world multimodal directory tasks.

The core thesis is simple: run every agent twice per task — once with only its native tools (without_mm), once with mm on PATH plus a one-page primer (with_mm) — and score both on correctness and speed. The headline numbers are lift (correctness delta) and speedup (wall-clock ratio).

---

Table of contents

1. Overview
2. Data source and datasets
3. Cases, graders, sandboxes, and artifacts
4. Harness, SQLite store, token usage tracking, and mm shims
5. Assistants, profiles, LLM judge, and custom judge setup
6. Primer, without_mm vs with_mm, preflights, and agent_output
7. FastAPI app and Svelte frontend
8. How to try it

---

1. Overview

mmbench/ adds ~9,000 lines across 52 files. The directory is structured as:

mmbench/
├── DESIGN.md              # Full design doc: thesis, decisions, scoring, pitfalls
├── README.md              # Usage guide, flags, case list
├── __init__.py
├── harness/               # Execution engine (Python)
│   ├── cases.py           # Typed case model + JSONL loader
│   ├── sandbox.py         # Per-run disposable working copies
│   ├── assistants.py      # Agent-CLI adapters + PATH shims
│   ├── grader.py          # Deterministic checks + LLM judge scoring
│   ├── store.py           # SQLite results store (sessions → runs → case_results)
│   ├── run.py             # Orchestrator + CLI entry point
│   ├── preflight.py       # Pre-run validation (agents, profiles, judge, fixture)
│   ├── profiles.py        # Named and on-the-fly mm profile specs
│   ├── agent_output.py    # Per-agent stdout parser (text + token usage)
│   └── primer.md          # One-page mm usage guide given to the with_mm arm
├── app/                   # FastAPI JSON API + prebuilt SPA
│   ├── app.py             # API routes + static mount
│   ├── db.py              # Read-side aggregations (leaderboard, drilldown, artifacts)
│   └── static/            # Built Svelte SPA (served at /)
└── frontend/              # Svelte 5 + Tailwind + Vite + Chart.js dashboard source
    └── src/
        ├── App.svelte     # Hash router (home ↔ cell detail)
        ├── api.js         # Fetch wrappers for /api/* endpoints
        ├── pages/
        │   ├── Leaderboard.svelte    # Ranked table + charts + filters
        │   └── CellDetail.svelte     # Session drilldown + transcript/artifact viewer
        └── components/
            ├── Chart.svelte          # Chart.js wrapper
            ├── InfoTip.svelte        # Tooltip on column headers
            └── ArtifactView.svelte   # Multi-format artifact renderer

---

2. Data source and datasets

Dataset: a frozen, nested, multimodal file fixture (mmbench-agent/) with ~180 files (images, video, audio, PDFs, docs) spread across subfolders, plus cases.jsonl (the 20 case definitions). Hosted on Hugging Face Hub:

https://huggingface.co/datasets/vlm-run/mmbench (private)

The harness auto-downloads on first run via ensure_dataset() → huggingface_hub.snapshot_download() into mmbench/data/ (gitignored). Subsequent runs reuse the local copy. HF auth (hf auth login) is required.

# run.py
HF_DATASET = "vlm-run/mmbench"
DATASETS_ROOT = Path(__file__).resolve().parents[1] / "data"

def ensure_dataset():
    if (DATASETS_ROOT / "cases.jsonl").exists() and (DATASETS_ROOT / "mmbench-agent").is_dir():
        return
    snapshot_download(repo_id=HF_DATASET, repo_type="dataset", local_dir=str(DATASETS_ROOT))

The fixture is purpose-built from three reproducible sources. It includes a 15-page paper (deep PDF QA), invoices in PDF and image form, a video only readable with mm, OCR targets (one container photo among 150+), floor-plan classification sets, and structured extraction targets.

---

3. Cases, graders, sandboxes, and artifacts

Cases (harness/cases.py)

20 declarative cases in cases.jsonl (data, not code). Each is a frozen EvalCase dataclass:

@dataclass(frozen=True)
class EvalCase:
    id: str                    # stable slug, e.g. "find-floor-plans"
    title: str                 # human-readable one-liner
    archetype: str             # "retrieval" | "organization" | "artifact"
    modality: list[str]        # ["image", "video", "audio", "pdf", "doc", "mixed"]
    dataset: str               # subtree path under datasets root
    mm_commands: list[str]     # mm surfaces the with_mm arm should exercise
    difficulty: str            # "easy" | "medium" | "hard"
    prompt: str                # given verbatim to both arms
    ground_truth: dict         # frozen, checkable answer
    checks: list[Check]        # deterministic partial-credit checks
    judge_objective: str       # rubric for the LLM judge (0-5)
    timeout_s: int             # 360-600s per case

Composition: ~10 retrieval, ~6 artifact, ~4 organization. Every mm command (find, peek, wc, sql, grep, cat) is primary in ≥3 cases. Every modality is load-bearing in ≥3 cases. 14 hard / 6 medium.

Grader (harness/grader.py)

Correctness is a 50/50 blend of two signals:

1. Deterministic checks (CHECK_SPECS): partial-credit, weighted checks run against three surfaces:

   • Answer checks (retrieval): names_file, contains_number, contains_text — matched against the agent's final text output.
   • Filesystem checks (organization): path_exists, path_absent — matched against the sandbox's final directory state.
   • Artifact checks (artifact-creation): artifact_exists, artifact_contains, artifact_row_count — matched against a file the agent wrote, parsed and compared to ground truth.

2. LLM judge: a single 0-5 score from the judge model against the case's judge_objective, grounded in its ground_truth. Retried up to 3 times; on persistent failure the run is voided (rows deleted) so no run mixes judged and checks-only cells.

correctness = 0.5 * (checkpoint_score * 100) + 0.5 * (judge_score / 5 * 100)

Additional signals: task_completion (binary: agent produced output and didn't timeout), failure_mode (timeout | tool_error).

Sandboxes (harness/sandbox.py)

Every (assistant, profile, case, arm, run_index) runs in its own disposable copy of the dataset:

class SandboxManager:
    def materialize(self, source, *, assistant, profile, case_id, arm, run_index, keep=False) -> Sandbox:
        # shutil.copytree into mmbench/data/_sandboxes/<tag>
        # tag = "claude__gateway__find-floor-plans__with_mm__r0"

• Organization and artifact tasks mutate the filesystem; the grader inspects the sandbox's final state.
• Sandboxes are disposed after grading by default (--keep-sandboxes retains them).
• mm's own state is isolated per run via temp MM_DATA_DIR and MM_CACHE_DIR, so neither the DB nor the cache leaks across runs.

Artifacts

Agent-written files (named by artifact_* checks) are copied out of the sandbox before disposal into mmbench/data/_artifacts/<session>/<case>/<arm>/ for dashboard review. The FastAPI app serves them via /api/artifact-file with path-traversal protection; the Svelte ArtifactView component renders them inline (image, video, audio, PDF, or text) with download links.

---

4. Harness, SQLite store, token usage tracking, and mm shims

Orchestrator (harness/run.py)

The unit of work is an (assistant, profile) cell. The orchestrator takes the cartesian product of --assistants and --profiles:

--assistants claude,gemini --profiles gateway,my-profile → 4 cells

Each cell is one session; its inner loop: for case → for arm in [without_mm, with_mm] → run → grade → persist. Sessions accumulate for trend analysis. Supports --resume (reuse latest session, skip completed cells), --stream (tee agent stdout live), and --runs N (variance control; dashboard shows mean±std).

SQLite store (harness/store.py)

Three-table schema modeling the result hierarchy:

sessions(session_id PK, assistant, profile_name, base_url, model, started_at, ended_at, status)
  → runs(run_id PK, session_id FK, run_index, started_at, ended_at, elapsed_s)
    → case_results(run_id FK, session_id FK, case_id, arm,
                   correctness, checkpoint_score, judge_score,
                   speed_s, task_completion, mm_used,
                   mm_commands_used_json, failure_mode,
                   final_output, stderr, mm_log,
                   token_total, token_usage_json,
                   mm_token_total, mm_token_usage_json,
                   PK(run_id, case_id, arm))

Indexed on session_id, case_id, arm, (assistant, profile_name), started_at. Supports void_run() to atomically discard a run when the judge fails mid-run.

Token usage tracking

Two layers of token tracking per case result:

1. Agent token usage (token_total, token_usage_json): the agent CLI's own LLM consumption (input, output, cache read/write, cost). Parsed from each agent's JSON output format by AgentOutputParser (see section 6).

2. mm token usage (mm_token_total, mm_token_usage_json): mm's own LLM token spend (prompt + completion across mm cat -m accurate calls). Read post-run from mm's run-isolated SQLite cache (MM_DATA_DIR/mm.db → extractions.metadata.verbose_suffix), so it adds zero runtime overhead:

# assistants.py
def _read_mm_token_usage(db_path: Path) -> TokenUsage | None:
    # Regex-parse "N→M tokens" from mm's verbose pipeline suffix
    # stored in the per-run mm.db (isolated via MM_DATA_DIR)

mm PATH shims (_mm_shim)

The core isolation mechanism. A context manager creates a temp bin dir whose mm entry is:

• with_mm arm: a logging shim that runs the real mm, passes stdout/stderr/exit code through, and appends <exit_code>\t<duration_s>\t<args> to $MMBENCH_MM_LOG. mm-grounding is read from this log — reliable, agent-agnostic, no transcript parsing.
• without_mm arm: a stub that exits 127 ("mm: command not found"), so the agent genuinely has no mm.

The shim dir is prepended to PATH. Additionally, MM_DATA_DIR and MM_CACHE_DIR are set to per-run temp paths so mm's global DB and cache never leak across runs or sessions.

---

5. Assistants, profiles, LLM judge, and custom judge setup

Assistants (harness/assistants.py)

Eight agent CLIs in the registry, each with its autonomy flag and output format:

_REGISTRY = {
    "claude":   ["claude", "--dangerously-skip-permissions", "--output-format", "json", "-p"],
    "codex":    ["codex", "exec", "--dangerously-bypass-approvals-and-sandbox", "--json"],
    "gemini":   ["gemini", "--yolo", "-o", "json", "-p"],
    "qwen":     ["qwen", "--yolo", "-o", "json", "-p"],
    "opencode": ["opencode", "run", "--dangerously-skip-permissions", "--format", "json"],
    "openclaw": ["openclaw", "agent", "--local", "--json"],
    "hermes":   ["hermes", "--yolo", "-z"],
    "pi":       ["pi", "--no-session", "--mode", "json", "-p"],
}

Each Assistant.run() builds the prompt (with_mm prepends the primer), creates the PATH shim environment, invokes the agent subprocess (buffered or streaming), reads the mm log, reads mm's token usage from the run's isolated DB, and parses the agent's structured output.

Profiles (harness/profiles.py)

Two types of mm backend specs:

• Named profiles: point at a profile in the user's mm profile list (e.g. gateway). Resolved at runtime from the user's mm config.
• On-the-fly (ad-hoc) profiles: --profile.model / --profile.base-url / --profile.api-key materialize a throwaway mm.toml in a temp dir, wired via MM_CONFIG_DIR. The user's global config is untouched.

# profiles.py
def materialize_adhoc(*, model, base_url, api_key="") -> ProfileSpec:
    # Writes mm.toml with one profile, returns ProfileSpec(config_dir=tmpdir)

Profiles only affect the with_mm arm (they set mm's backend); the without_mm arm never touches mm.

LLM judge and custom judge setup

Default judge: google/gemini-3.1-flash-lite via OpenRouter (https://openrouter.ai/api/v1), key from MMBENCH_JUDGE_API_KEY or OPENROUTER_API_KEY.

Override per-run with all three flags:

--judge.model openai/gpt-5 \
--judge.base-url https://openrouter.ai/api/v1 \
--judge.api-key sk-...

Or via environment: MMBENCH_JUDGE_MODEL, MMBENCH_JUDGE_BASE_URL, MMBENCH_JUDGE_API_KEY.

The judge is a zero-temperature, structured-output call:

# System: "Score 0-5. Respond with ONLY a single integer."
# User: evaluation objective + ground truth + model response

Retried up to 3 times. On persistent failure, JudgeError is raised and the orchestrator voids the run (deletes its rows) and halts — no run ever mixes judged and checks-only cells. Fix the judge endpoint and --resume.

---

6. Primer, without_mm vs with_mm, preflights, and agent_output

Primer (harness/primer.md)

A one-page reference (141 lines) given only to the with_mm arm. It documents:

• When to reach for mm (files an LLM can't read natively, scale).
• Command-by-intent lookup: retrieval → find/sql/grep; read binary → cat -m accurate; metadata → peek; budget → wc.
• Each command's synopsis, key flags, and concrete examples.
• Recipes by task archetype (retrieval, organization, artifact).

The same task prompt goes to both arms verbatim; the only variable is mm availability.

without_mm vs with_mm

Aspect	without_mm	with_mm
mm on PATH	stub → exit 127	logging shim → real mm
Prompt	task only	primer + task
mm profile	n/a	MM_PROFILE=<profile>
mm state	n/a	isolated MM_DATA_DIR + MM_CACHE_DIR
mm grounding	n/a	shim log → mm_commands_used
mm token usage	n/a	post-run read from mm.db

Preflights (harness/preflight.py)

Run before any session. Four checks, in order:

1. Fixture: every selected case's dataset subtree exists on disk.
2. Assistants: each is installed on PATH AND passes a live autonomy probe (must non-interactively execute cat <file> and echo a sentinel).
3. Profiles: each mm profile resolves and its chat endpoint answers a 1-token ping.
4. Judge: if enabled, the judge endpoint answers a 1-token ping.

No silent fallbacks. A failure aborts with a precise reason. --check runs only preflight and exits (for CI/setup validation). --skip-preflight bypasses (not advised).

agent_output (harness/agent_output.py)

Normalizes each agent CLI's stdout into AgentOutput(final_output, token_usage):

• claude: JSON object → result (text) + usage.{input_tokens, output_tokens, cache_read_input_tokens, cache_creation_input_tokens}. Total recomputed to include cache.
• codex: NDJSON stream → item.completed events (text) + turn.completed (usage).
• gemini: JSON object → response (text) + stats.models.*.tokens.{input, candidates, total, cached}.
• qwen: JSON array/NDJSON → assistant events with content[].text + usage.
• opencode: NDJSON → text events (text) + step_finish (per-step tokens with cache).
• openclaw: JSON object → meta.finalAssistantVisibleText (text) + meta.agentMeta.usage.
• hermes: plain-text stdout (no JSON mode); token usage fetched post-run via hermes sessions export.
• pi: NDJSON → text_end events (text) + message.usage.

---

7. FastAPI app and Svelte frontend

FastAPI app (app/app.py, app/db.py)

Serves the prebuilt SPA and a JSON API at http://localhost:9095:

Endpoint	Returns
GET /api/leaderboard	Ranked rows, one per (assistant, profile) cell, averaged over all sessions. Each row: without_mm/with_mm arm stats, lift, speedup, n_sessions, n_runs.
GET /api/sessions	Per-(cell, session) with_mm correctness over time (trend chart data).
GET /api/case-breakdown	Per-(cell, case) mean correctness for both arms (cross-case comparison).
GET /api/cell?assistant=&profile=	One cell's overall scores + per-session breakdown + per-case breakdown.
GET /api/session/{id}	One session's runs + per-case results with full arm detail.
GET /api/transcript?session=&case=	Stored stdout, stderr, mm log, agent tokens, mm tokens for one (session, case).
GET /api/case-spec?case=	Raw case definition (prompt, ground truth, checks) from cases.jsonl.
GET /api/artifacts?session=&case=&arm=	List of artifact file paths the agent wrote.
GET /api/artifact-file?...	Serve an artifact file (path-traversal guarded).

db.py contains the read-side aggregation queries. Hero metric: _lift_speedup(without_stats, with_stats). Pass threshold: correctness ≥ 60.

Svelte frontend (frontend/)

Built with Svelte 5 (runes), Tailwind CSS, Chart.js, svelte-multiselect. Hash-based routing (#/ → Leaderboard, #/cell/<assistant>/<profile> → CellDetail).

Leaderboard page (Leaderboard.svelte):

• Filters: svelte-multiselect chips for assistants and mm profiles. Selections persisted in localStorage.
• Ranked table: sortable columns — Without %, With %, Lift (color-coded green/red), Speedup, Pass rate (mm), Tokens (wo/w, dimmed/bright comparison), Sessions, Runs. Each row is an (assistant, profile) cell; click to drill down.
• Correctness bar chart: grouped bars (without mm gray, with mm blue) per cell.
• Session trend chart: line chart of with_mm correctness over sessions per cell (regression tracking).
• Token usage chart: horizontal overlaid bars (faded = without mm, solid = with mm), ranked least-utilized first.
• Per-case comparison chart: grouped bars per selected case, with a multiselect case filter. Faded = without mm, solid = with mm.
• Per-case table: all 20 cases × all cells, showing without/with correctness pairs.
• "How it works" dialog: explains the methodology in four steps.

CellDetail page (CellDetail.svelte):

• Header cards: Without mm, With mm, Lift, Speedup, Tokens (wo/w).
• Sessions table: rows per session with Without %, With %, Lift, Tokens. Click to expand per-case breakdown inline (slide transition).
• Per-case table (expanded session): case_id, archetype, difficulty, Without %, With %, Tokens, mm commands used, with a "view" button per case.
• Transcript dialog (modal): four tabs — Result (agent's final output), Logs (token usage cards + mm commands used log with exit codes/durations + stderr), Artifact (inline renderer via ArtifactView: images, video, audio, PDF, text with download), Case spec (prompt + full case definition JSON).

Components:

• Chart.svelte: reactive Chart.js wrapper (auto-creates or updates on data change).
• InfoTip.svelte: click-toggle tooltip for column header explanations.
• ArtifactView.svelte: auto-detects file type by extension and renders inline (img, video, audio, PDF iframe, or text pre-block) with download link.

---

8. How to try it

cd <repo-root>

# 0. Preflight — verify agents, profiles, and judge are reachable:
uv run python -m mmbench.harness.run --assistants claude --check

# 1. Run the benchmark (dataset auto-downloads from HF on first run):
uv run python -m mmbench.harness.run --assistants claude --profiles gateway

# 2. View the dashboard:
uv run python -m mmbench.app.app        # → http://localhost:9095

# With multiple agents and custom backends:
uv run python -m mmbench.harness.run \
    --assistants claude,gemini \
    --profiles gateway \
    --runs 3

# On-the-fly backend + custom judge:
uv run python -m mmbench.harness.run --assistants claude \
    --profile.model google/gemini-3.1-flash-lite \
    --profile.base-url https://openrouter.ai/api/v1 \
    --profile.api-key sk-... \
    --judge.model openai/gpt-5 \
    --judge.base-url https://openrouter.ai/api/v1 \
    --judge.api-key sk-...

# Rebuild the frontend after editing mmbench/frontend/:
cd mmbench/frontend && npm install && npm run build

Prerequisites: an agent CLI installed and authed, an mm profile for the with_mm arm, MMBENCH_JUDGE_API_KEY (or OPENROUTER_API_KEY) for the judge, and hf auth login for the private dataset.

---

Other changes

• Added fastapi>=0.110, uvicorn>=0.29, huggingface-hub>=0.25 to [dev] dependencies.
• Minor type annotation fix in python/mm/config.py (update_config_key backend assignment).
• Added [tool.ty.analysis].allowed-unresolved-imports for optional extras (mlx, faster_whisper, etc.).

Link to Devin session: https://app.devin.ai/sessions/0c4184b797aa4b6ca6699a179dad9300
Requested by: @nwaughachukwuma

[gemini-code-assist]

Code Review

This pull request refactors the benchmark suite by reorganizing files into a more structured directory layout, updating script names, and improving path resolution logic for data and results directories. Specifically, it standardizes the calculation of DATA_DIR across several shell scripts and updates Python helper scripts to use absolute path resolution. A suggestion was made to further simplify the DATA_DIR construction in bench_universal.sh to avoid relative path segments.

[gemini-code-assist]

The logic for determining DATA_DIR is robust for absolute paths, but it relies on cd succeeding to return a canonical path. If the directory does not yet exist, it falls back to a path containing ... While functional, consider using a more direct absolute path construction if the goal is to avoid relative path segments in the variable.

Suggested change

	DATA_DIR="$(cd "${SCRIPT_DIR}/../data" && pwd 2>/dev/null || echo "${SCRIPT_DIR}/../data")"
	DATA_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)/data"

[devin-ai-integration]

Devin Review found 4 potential issues.

View 4 additional findings in Devin Review.

[devin-ai-integration]

🔴 Opencode assistant probe uses wrong flag format, so it always fails and is silently skipped

The opencode reachability probe uses a single-dash flag (opencode -prompt 'hi' at benchmarks/bench_universal/bench_universal_tiny_metadata.sh:233) instead of the correct double-dash --prompt, while the same script's actual benchmark commands use --prompt (benchmarks/bench_universal/bench_universal_tiny_metadata.sh:279), so the probe always fails and opencode is incorrectly skipped from every metadata benchmark run.

Impact: Opencode is silently excluded from all metadata benchmark results even when it is properly installed and authenticated.

Inconsistency between probe and usage within the same script and across peer scripts

The check_assistant function at line 233 uses opencode -prompt 'hi' (single dash). In contrast:

• The same file's assistant_cmd at line 279 uses opencode --prompt '${prompt}' (double dash)
• The sister script bench_universal_tiny_extractions.sh:235 correctly uses opencode --prompt 'hi' (double dash)
• The mmbench harness mmbench/harness/assistants.py:55 also uses the double-dash form

Since -prompt is not a valid flag (the CLI expects --prompt), the probe returns a non-zero exit code, and the probe_assistants loop at line 253-258 marks opencode as "unreachable (skipping)".

Suggested change

	opencode) opencode -prompt 'hi' </dev/null >/dev/null 2>&1 ;;
	opencode) opencode --prompt 'hi' </dev/null >/dev/null 2>&1 ;;

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🔴 Codex assistant probe only checks installation, not authentication, so unauthenticated runs proceed and fail

The codex reachability probe checks codex --version (benchmarks/bench_universal/bench_universal_tiny_metadata.sh:230) instead of running an actual query like codex -q 'hi', so the probe passes for any installed-but-unauthenticated codex, and the benchmark runs will then fail on every actual codex -q '...' command.

Impact: An unauthenticated codex is accepted during preflight and then fails on every benchmark task, wasting benchmark time and producing error results.

Inconsistency with all other scripts that probe codex

Every other script uses codex -q 'hi' to probe codex reachability:

• bench_universal_tiny_extractions.sh:232: codex -q 'hi'
• bench_universal.sh:132: codex -q 'hi'
• helpers/lib_common.sh:144: codex -q 'hi'

The metadata script's own assistant_cmd at bench_universal_tiny_metadata.sh:276 generates commands with codex -q, confirming the probe is testing a different code path than actual usage.

Suggested change

	codex) codex --version </dev/null >/dev/null 2>&1 ;;
	codex) codex -q 'hi' </dev/null >/dev/null 2>&1 ;;

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🚩 Unreferenced invoices.csv committed to repo root

A new file invoices.csv was added at the repository root with two sample invoice rows. No code in the diff or the broader repo references this file (confirmed via grep). It appears to be a test artifact or example data that was accidentally committed. While not a code bug, it adds clutter to the repo root and may confuse contributors.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🚩 stat -f '%z' fallback for total_size_bytes is Linux-incompatible

Both bench_universal_tiny_metadata.sh:582 and bench_universal_tiny_extractions.sh:613 compute total_size_bytes using stat -f '%z' (macOS) with a fallback to stat -c... (Linux). However, the diff shows the Linux fallback as truncated (-exec stat -c...). If the fallback is incomplete or malformed, the YAML header's total_size_bytes field will be wrong on Linux. Since this is cosmetic metadata in the YAML header (not used for grading or comparison), it's non-critical, but may produce invalid YAML on Linux.

---

Was this helpful? React with 👍 or 👎 to provide feedback.