FinanceBenchmark

Can AI actually do finance work — or does it just talk about it well?

An open, independent benchmark testing frontier AI models on real finance-industry tasks: Basel III capital calculations, derivative pricing, credit underwriting, LBO returns, portfolio attribution. Not textbook multiple choice — the work itself.

What we found (July 2026)

AI knows finance. 25 of 26 frontier models score a perfect 100% on CFA-style knowledge questions. Multiple choice is finished as an AI eval.

AI can't yet reliably do finance. Ask the same models to implement the pricing model in Python and the average drops to 50% — the best manages 80%, and it's not the flagship: Claude Haiku 4.5, one of the cheapest models on the board, tops the quant category outright. The gap between knowing and doing is the finding — and so is who closes it.

Most leaderboard gaps are noise. Every score here ships with a 95% confidence interval, and 22 of 26 models' intervals overlap the top score — one statistical cluster, not a ranking. A $0.02-per-eval model (DeepSeek V4 Flash) is in a dead heat with the $1.14 flagship (Claude Opus 4.7) at 93.2%.

Explore without touching code

• Live leaderboard — scores, confidence intervals, cost per evaluation
• Task catalog — read the actual questions
• How scoring works — plain-language methodology
• Get your model evaluated — labs: API access in, verified run out within 48h (process)

Everything below this line is for people who want to run or extend the benchmark themselves.

---

The task sets

v1 core set (45 tasks):

• 15 multiple-choice questions (knowledge)
• 15 number questions (analysis)
• 15 Python coding questions (quant / derivative pricing)

v2 hard set (28 tasks) — modeled on real industry workflows, with seeded random parameters and gold reference implementations so answers cannot be memorized:

• portfolio_mgmt — CFA L3-style: Brinson-Fachler attribution, GIPS composites, currency hedging, tax-loss harvesting, endowment IPS
• credit_underwriting — mortgage desk: self-employed income add-backs + DTI, max purchase price, ARM qualifying rate, PMI tiers, prepayment amortization
• banking_regulation — bank treasury: Basel III standardised RWA + CET1, LCR with Level 2 caps, CECL lifetime ECL, repricing-gap NII
• derivatives_structuring — structuring desk: autocallable note Monte Carlo, variance swap replication (VIX method), swap value + DV01, CVA, FX-hedged yield with cross-currency basis
• corporate_finance — advisory: LBO equity IRR, merger accretion/dilution, bottom-up beta WACC, FCFF with traps, mid-year DCF

v3 extension (36 tasks) — structured output, coding/data pipelines, compliance guardrails, sentiment, chronology, cross-table reasoning, vendor risk, and behavioral communications. Each v3 task carries business_function and modality metadata for two-axis reporting:

• reporting_formatting — JSON risk reports, markdown tables
• coding_data — Python data wrangling, SQLite SQL
• advanced_sentiment — central-bank / earnings tone with evidence
• compliance_guardrails — refusal boundaries and safe alternatives
• chronological_reasoning — ordered timelines from disclosures
• cross_table_reasoning — covenant and leverage across statements
• third_party_risk — vendor control gaps and remediation
• privacy_cyber_compliance — classification, breach notification
• behavioral_reputation — de-escalation and regulated tone

Verify v3 without API keys: python scripts/verify_v3_tasks.py.

v2.1 extension (10 tasks) — extended quant set with headroom: implied vol, digital options, Merton jump-diffusion, Vasicek bonds, Black-76, FX options, par swap rates, Cornish-Fisher VaR, portfolio DV01, drawdown/Calmar. All seeded with gold reference implementations. --task-set v2.1 runs v2 + these; the frozen v2 set is unchanged so published results stay comparable.

See TASKS.md for a human-readable list of every task. Verify the hard set without API keys: python scripts/verify_hard_tasks.py.

New here? Read docs/README.md — short guides written for easy understanding.

AI assistant? Read AGENTS.md first.

Quick start

python -m venv .venv
source .venv/bin/activate
pip install -e .

# Test without API keys
finbench run --model mock --dry-run
finbench run --model mock --dry-run --task-set v3
finbench list-tasks
pytest

Run a real model

finbench run --model anthropic/claude-sonnet-4-6 --tasks all --runs 3
finbench run --model google/gemini-2.5-flash --tasks knowledge --runs 1
finbench run --model xai/grok-3 --workers 2
finbench run --model openai/gpt-4o --task-set v3 --runs 3

Supported providers: OpenAI (chat + Responses/Codex), Anthropic, Google Gemini, xAI, DeepSeek, Z.ai, Moonshot, Mistral, Alibaba DashScope.

Set API keys in .env (see .env.example):

Provider	Env var
OpenAI	OPENAI_API_KEY
Anthropic	ANTHROPIC_API_KEY
Google	GOOGLE_API_KEY
xAI	XAI_API_KEY
DeepSeek	DEEPSEEK_API_KEY
Kimi	KIMI_API_KEY

Use --workers 2 (default) to stay under provider rate limits during parallel runs.

Results are checkpointed to the output JSON every few attempts. To continue after a crash or quota failure:

finbench run --model openai/gpt-5.5-pro --output results/gpt-5.5-pro_partial.json --resume

To list or repair failed API attempts in existing result files (updates only those slots):

finbench scan-api-errors
finbench retry-api-errors --dry-run
finbench retry-api-errors results/gpt-5.5-pro_20260715_133635.json

Run all models at once

# See catalog and which providers have keys configured
finbench list-models

# Edit models.json to choose models, or run a tier from models/registry.json:
finbench run-all --tier frontier --publish

# Full active set (models.json):
finbench run-all --publish --workers 2

# Kimi K3 + K2.7 Code + GPT-5.6 Luna (publishes to Supabase).
# Entry-tier Kimi is 1 concurrency / 3 RPM — script defaults to WORKERS=1,
# 20s pacing, and FINBENCH_VERBOSE=1 (heartbeat every 15s while waiting on API).
./scripts/run_kimi_luna.sh

run-all evaluates every model in models.json, saves one results file per model, and (with --publish) uploads each run to the Supabase leaderboard. Models whose API key is missing are skipped and reported in the summary.

By default, run-all only evaluates new models — ones without a successful run for the same task set and run count. Use --force to re-run everything.

finbench eval-status          # show evaluated vs pending models (v2 by default)
finbench eval-status --task-set v3
finbench run-all --publish    # only new models (v2 task set)
finbench run-all --task-set v3 --publish
finbench run-all --all        # evaluate every model in models.json

The full model catalog lives in models/registry.json. models.json is the active evaluation subset.

Wired providers: OpenAI (chat + Responses API for gpt-5.5-pro / Codex), Anthropic, Google, xAI, DeepSeek, Z.ai (GLM), Moonshot (Kimi), Mistral, Alibaba DashScope (Qwen).

Current API IDs in the active set include: GPT-5.6 Sol/Terra, GPT-5.5-Pro, GPT-5.3-Codex, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5 / 4, DeepSeek V4 Pro/Flash, Kimi K3 (moonshot/kimi-k3), plus GLM / Mistral / Qwen when their keys are set.

Set optional keys in .env (see .env.example): ZAI_API_KEY, KIMI_API_KEY (Kimi Open Platform; MOONSHOT_API_KEY still accepted; optional KIMI_REASONING_EFFORT=low|high|max for K3), MISTRAL_API_KEY, DASHSCOPE_API_KEY. Missing keys are skipped by finbench run-all.

Project layout

tasks/          # 119 questions in JSON (17 categories: v1 + v2 + v2.1 + v3)
harness/        # Python runner (finbench)
results/        # Saved score files (includes per-attempt drill-down)
logs/           # Eval run logs + one-shot model lists (models_*.json)
web/            # Leaderboard website (Next.js)
supabase/       # Database schema
docs/           # Plain-language documentation
models.json     # Active evaluation subset (default for finbench run-all)

Statistical rigor, calibration, and failure analysis

Every run now reports more than a point estimate:

Confidence intervals. Each results file carries a 95% bootstrap CI on Pass@1 (resampling tasks). The leaderboard shows the interval under each score; overlapping intervals are statistical ties. Compare models properly:

finbench stats --task-set v2            # CIs + pairwise significance tests
finbench stats --task-set v2 --backfill # write CI fields into old results files

Calibration (protocol 0.2.0). By default, finbench run asks models for a CONFIDENCE: 0-100 line and allows FINAL_ANSWER: ABSTAIN. Results include ECE, Brier, overconfidence gap, selective accuracy at 50/70/90% confidence, abstention rate, and a risk-adjusted score where confident-wrong answers are penalized in proportion to expressed confidence — in finance, wrong-but-confident is the failure mode that matters. Disable with --no-calibration for protocol-0.1 comparability.

Failure taxonomy. Failed attempts are classified (api_error, code_runtime_error, format_error, refusal, near_miss, wrong_method, wrong_choice, structure_mismatch...) — near_miss vs wrong_method separates convention slips from broken methodology. Counts land in run metadata; full transcript reports:

finbench failures results/<file>.json --report results/failures/<model>.md

Human baseline. Collect professional baselines with the same grading rules and timing (see docs/HUMAN_BASELINE.md):

finbench human --name analyst-1 --task-set v2.1 --shuffle-seed 42

Saturated categories. knowledge and analysis (~100% for all frontier models) are still scored and displayed but excluded from the Finance Index composite so the index retains discriminating power.

Public/private split. --split public|private|all plus a private held-out directory (FINBENCH_PRIVATE_TASKS_DIR, gitignored tasks_private/) that never ships in this repo. Model submissions: docs/SUBMISSIONS.md.

How scoring works

Category	Model output	Pass rule
knowledge	FINAL_ANSWER: B	Exact letter match
analysis	FINAL_ANSWER: 0.24	Within 1% of correct number
quant	Python prints JSON/number	Run code, compare to gold implementation
json_schema / markdown_table	Strict JSON or markdown table	Schema/table match (v3)
sql / code (v3)	SQL query or Python	Execute against fixture, compare result
policy / rubric	Structured text or JSON	Refusal boundaries or rubric checklist

Pass@1 = percent of tasks where the model passed at least once.

Token usage and eval cost

Every evaluation records per-attempt input/output tokens and latency from the provider’s usage fields (including reasoning tokens billed as output on models like gpt-5.5-pro). Character-length estimates are only a fallback. The results file metadata includes:

• total_input_tokens / total_output_tokens
• eval_cost_usd — list-price estimate from harness/finbench/pricing.py (mirrored in web/src/lib/pricing.ts; rates snapshot 2026-07-16 from OpenAI / Anthropic / Google / xAI / DeepSeek docs; standard / cache-miss tier). Example: openai/gpt-5.5-pro is $30 / $180 per 1M input/output tokens.
• duration_seconds — wall-clock time for the full model eval
• median_latency_ms / median_output_tps

Progress lines also show elapsed and ETA while a run is in flight.

On finbench publish (or run-all --publish), that metadata is stored on the Supabase runs.metadata JSON column. The leaderboard view exposes it so the site can show Eval cost and latency without needing local results/*.json files. Apply migration 003_leaderboard_metadata.sql in production.

Web leaderboard

cd web
cp .env.local.example .env.local
npm install
npm run dev

The site at finbenchmark.ai is mobile-first (no page-wide horizontal scroll on phones) and includes:

• Leaderboard — Tasks passed, Finance Index, consistency; card list on mobile, compact table on larger screens; domain columns behind a toggle
• Time · Cost · Rating — rotating 3D scatter of eval wall-clock time, estimated cost ($), and Finance Index (extreme cost outliers such as gpt-5.5-pro are omitted from the plot and called out in text so the scale stays readable)
• 2D tradeoff — pairwise scatter with a toggle for Time vs Cost, Cost vs Finance Index, or Finance Index vs Time (same outlier exclusion as the 3D plot)
• Collaborate / How Scoring Works — homepage CTAs after the tradeoff charts
• Scores by domain — one interactive chart (category / business function / modality); defaults to the domain with the largest score entropy across top models
• Model detail (/models/<provider>--<model>) — per-task drill-down with raw responses, difficulty breakdown, run provenance, JSON download
• Compare (/compare) — head-to-head across categories and difficulty
• Tasks — public catalog (v2 + v3); private held-out variants with rotated seeds live outside this repo (see docs/SUBMISSIONS.md)
• Methodology — metric glossary, scoring rules, held-out policy, reproduction steps
• Collaborate (/contact) — LinkedIn, feedback form, and how to evaluate and publish your model

The site reads scores from:

1. Supabase (if configured)
2. Local results/*.json files
3. Demo data (fallback)

Apply Supabase migrations (001_initial.sql, 002_leaderboard_v2_categories.sql, 003_leaderboard_metadata.sql) for full v2 category columns plus token/cost metadata on the leaderboard view.

Contact form (optional, for /contact): set in web/.env.local:

Variable	Purpose
RESEND_API_KEY	Resend API key
RESEND_FROM_EMAIL	Verified sender (e.g. onboarding@resend.dev for testing)
CONTACT_TO_EMAIL	Recipient inbox for /contact (required)

Publish results online

# Apply supabase/migrations/*.sql first (through 003_leaderboard_metadata.sql)
finbench publish results/<file>.json

Publish uploads scores and run metadata (token totals, estimated eval cost, latency). Re-publishing a result file backfills eval_cost_usd when pricing is known for that model.

To replace old v1 Supabase rows with a single v2 run per model (and optionally re-run models that lack a full 219-attempt local file):

python scripts/migrate_v1_models_to_v2.py --dry-run
python scripts/migrate_v1_models_to_v2.py --run-missing

Deploy to finbenchmark.ai

See docs/DEPLOY.md for Railway + Supabase + Namecheap DNS setup.

Railway Root Directory is web/. Build uses Node 20+ and npm ci --include=dev (see web/nixpacks.toml). Do not set NODE_ENV=production as a Railway variable — it skips build tooling and breaks next build. Refresh the task catalog with cd web && npm run sync-tasks and commit web/src/data/ before deploy.

Code map (harness)

Each file does one job:

• runner.py — main loop
• models/ — provider adapters (OpenAI chat + Responses, Anthropic, Gemini, xAI, DeepSeek, Z.ai, Moonshot, Mistral, Alibaba, mock)
• prompt_builder.py — build prompts
• answer_parsers.py — read model answers
• reference_answers.py — compute correct answers
• scoring.py — pass/fail
• sandbox.py — run model Python code

See docs/02-file-map.md for the full list.

Versions

• Harness: 0.2.0 — adds bootstrap CIs, calibration protocol (CONFIDENCE + abstention), failure taxonomy, human baseline mode, public/private splits
• Task sets: v2 (73 tasks, frozen), v2.1 (83 tasks — v2 + extended quant), v3 (36 tasks — structured output, compliance, reasoning), or all (119 combined)