yaml-vs-md-benchmark

A reproducible benchmark comparing five serialization formats — JSON, YAML, Markdown, plain text, and TOON (a compact custom notation) — as context-window transports for an LLM. Eight task categories, three context sizes each, automated scoring against ground truth.

The question: does the format you use to pass structured data to Claude affect extraction accuracy or token cost?

Article with the full narrative and discussion: YAML vs Markdown vs JSON: какой формат лучше для промптов (link will go live once the article is published).

TL;DR

• TOON uses 62% fewer tokens than JSON for the same data, with no measurable accuracy loss on Haiku.
• Markdown and TXT sit in the middle: ~55% fewer tokens than JSON.
• YAML saves ~32% vs JSON but still carries significant syntax overhead.
• JSON is the most expensive format by a wide margin — curly braces, quotes, and commas add up fast at scale.
• The accuracy gap between formats is small; the token gap is not.

Token efficiency (avg across all cases, excl. case 8)

Format	Avg tokens	vs JSON
JSON	3,710	baseline
YAML	2,516	−32%
Markdown	1,723	−54%
TXT	1,583	−57%
TOON	1,394	−62%

Full token counts per case and size in results/token_counts.json.

Formats compared

ID	What it is
json	Standard JSON with 2-space indentation. The "safe default" most people reach for.
yaml	PyYAML default output. Less punctuation than JSON, but verbose on nested structures.
md	Markdown tables, headings, and lists. Familiar to LLMs from training data.
txt	Plain text with whitespace alignment. No formal structure, relies on layout.
toon	A minimal custom notation: header declares schema, rows are CSV-like. Maximally compact.

What is TOON?

A compact notation designed to minimize tokens while remaining parseable by LLMs:

products[3]{id,name,price,category}:
1,Mouse Pro,29.99,peripherals
2,Keyboard Ultra,89.99,peripherals
3,USB-C Hub Slim,45.50,accessories

The header line declares the array name, length, and field schema. Data rows are positional CSV. No quotes, no braces, no repeated keys.

Test cases

#	Case	What it tests
1	case1-instructions	Following structured rules (system prompt style)
2	case2-products	Looking up values in a product catalog
3	case3-tasks	Extracting info from a task/ticket list
4	case4-rules	Reasoning over business rules and policies
5	case5-fewshot	Learning from few-shot examples
6	case6-hierarchy	Navigating hierarchical org/category data
7	case7-api-docs	Understanding API documentation
8	case8-output	Generating structured output in a requested format

Each case has three sizes — S, M, L — to test how formats scale with context length. Data is deterministic (random.seed(42)).

Methodology

Model and runtime

• Model: claude-haiku-4-5-20251001 via direct Anthropic API (cheapest, good for high-volume iteration; swap to Sonnet/Opus with --model).
• Agent loop: single-turn messages.create, no agent SDK, no tool use — pure context-in, answer-out.
• Temperature: 0 (deterministic).
• Scoring: automated against hand-crafted ground truth via exact match, numeric tolerance, set overlap, and boolean checks (see scorer.py).

Metrics captured per run

• input_tokens, output_tokens
• response text
• ground_truth expected value
• Scored accuracy (exact, numeric, set_match, boolean)
• Format compliance (case 8)

Running the benchmark yourself

# 1. Clone and set up
git clone https://github.com/webmaster-ramos/yaml-vs-md-benchmark
cd yaml-vs-md-benchmark
python -m venv .venv
source .venv/bin/activate
pip install anthropic pyyaml

# 2. Set your API key
export ANTHROPIC_API_KEY="your-key-here"

# 3. Generate test data (all 8 cases × 3 sizes × 5 formats)
python generate_data.py

# 4. Count tokens for every generated file
python tokenizer.py

# 5. Run the benchmark
python runner.py --model claude-haiku-4-5-20251001

# 6. Score results
python scorer.py results/accuracy_haiku-4-5_*.json --by-case

Individual runs and ablations:

# Specific cases and sizes
python runner.py --model claude-haiku-4-5-20251001 --cases case2-products --sizes s,m

# Only JSON vs TOON, 3 runs for statistical robustness
python runner.py --model claude-haiku-4-5-20251001 --formats json,toon --runs 3

# Dry run — show prompts without calling the API
python runner.py --dry-run --cases case2-products --sizes s --formats json,md

# Score with full breakdown
python scorer.py results/accuracy_haiku-4-5_20260413-115221.json --by-case

Repo layout

yaml-vs-md-benchmark/
├── README.md                 # this file
├── generate_data.py          # deterministic data generator (all formats, all sizes)
├── tokenizer.py              # token counter using Anthropic's tokenizer
├── runner.py                 # benchmark runner: prompts → Claude API → results
├── scorer.py                 # automated scoring against ground truth
├── data/                     # generated input files
│   ├── case1-instructions/   #   input-{s,m,l}.{json,yaml,md,txt,toon}
│   ├── case2-products/
│   ├── case3-tasks/
│   ├── case4-rules/
│   ├── case5-fewshot/
│   ├── case6-hierarchy/
│   ├── case7-api-docs/
│   └── case8-output/
├── questions/                # questions + ground truth per case
│   ├── case1-instructions.json
│   ├── ...
│   └── ground_truth.json
└── results/                  # token counts and accuracy results
    └── token_counts.json

License

MIT