mcp-vs-cli-aws-benchmark

A reproducible benchmark comparing raw aws CLI against the official awslabs.aws-api-mcp-server as tool transports for an LLM agent, on five read-only tasks against a real AWS account. 250 runs total, clean Anthropic API (no claude-agent-sdk context pollution), automated verification against ground truth snapshotted from boto3.

Article with the full narrative and discussion: MCP vs CLI для AI-агентов: реальный бенчмарк на AWS (link will go live once the article is published).

TL;DR

• A well-engineered CLI tool (cli-full in this repo) beats awslabs.aws-api-mcp-server by 43–60% on input tokens on every one of the five tasks in the benchmark, at parity on success rate.
• But getting there takes about half a day of engineering per service: rich tool description, batch input schema, runtime context in the system prompt, and a whitelist that covers the optimal commands instead of only the naive ones.
• The real question isn't "MCP or CLI?" but "do you have enough QPS to justify that engineering work over the 30-second uvx install of the MCP server?"
• Three effects explain the whole gap: HTTP response metadata (the MCP server passes AWS response headers back to the model, including Date), batch calling (cli_command accepts a list), and a broader allowlist. None are properties of the MCP protocol itself.

Full numbers in docs/findings.md.

Final results (median of n=10 per cell)

Task	cli-full input	mcp input	Δ input	cli-full ok%	mcp ok%
ec2_running	3,053	5,368	−43%	90%*	100%
s3_bucket_policy	2,975	5,425	−45%	100%	100%
s3_bucket_regions	5,801	14,317	−60%	100%	100%
iam_admin_roles	2,934	5,213	−44%	100%	100%
ec2_cpu_last_hour	5,345	9,461	−44%	100%	100%

* single Anthropic API 529 Overloaded, not a transport failure.

Full summary (with p25/p75, wall clock, tool call counts) in results/scrubbed/final_summary.json.

Methodology

Transports compared

ID	What it is
cli	Plain aws_cli tool: subprocess wrapper, ~500-char description.
cli-ctx	Same tool, but the system prompt injects current UTC time, default region, and identity ARN. Four lines.
cli-v2	Rich tool description (~2,700 chars, mirrors awslabs structure), cli_command accepts string or list of strings, parallel execution via asyncio.gather, batch results joined with indexed headers.
cli-full	cli-v2 tool spec + cli-ctx system prompt. The complete "purpose-built CLI" recommended by the article.
mcp	Official awslabs.aws-api-mcp-server launched via uvx stdio; the mcp python library handles the handshake. READ_OPERATIONS_ONLY=true.

Tasks (read-only, all five)

ID	Category	What it tests
ec2_running	simple	single call + filtering
s3_bucket_policy	edge	optional resource (NoSuchBucketPolicy)
s3_bucket_regions	chained	list + per-item lookup, batch-friendly
iam_admin_roles	filter	pagination + content filtering (ground truth: empty list)
ec2_cpu_last_hour	chained	composition, CloudWatch time windows

Task prompts are verbatim in src/tasks.py. Ground truth is fetched at benchmark time via boto3 (src/ground_truth.py), so account drift is detected automatically.

Model and runtime

• Model: claude-sonnet-4-6 via direct Anthropic API.
• Agent loop: hand-rolled in src/agent_loop.py, ~150 lines. No claude-agent-sdk, no Claude Code — both leak the parent ~/.claude.json into the context (figma, pencil, PubMed and friends). That would contaminate every measurement and did: we lost 1.5 days to this before rewriting.
• CLI transport: subprocess.run(['aws', ...]) behind a strict whitelist in src/safety.py.
• MCP transport: mcp python library spawns awslabs.aws-api-mcp-server via uvx stdio and drives the real MCP handshake; tool descriptions are read live from the server, not hardcoded.
• n = 10 per (task, transport) cell for the main series, plus ablation runs for the hypothesis investigation.

Metrics captured per run

• input_tokens, cache_creation_input_tokens, cache_read_input_tokens
• output_tokens
• tool_call_count, full tool call list with inputs
• wall_clock_ms, num_turns
• stop_reason, error (if any)
• Verdict against ground truth (ok: bool, reason: str)

See src/runner.py and src/aggregate.py.

Safety

Two layers of defence keep the agent from doing anything destructive:

1. IAM ReadOnlyAccess policy on a dedicated benchmark user (mcp-benchmark). Write operations fail at the AWS edge regardless of what the agent or the runner code does. This is the real guarantee.
2. Whitelist in src/safety.py, covering a curated set of service + action pairs. This is the second line of defence and also catches obvious errors before they reach AWS.

The whitelist caused one of the most interesting findings in the benchmark: our early version blocked iam list-entities-for-policy, which is the optimal command for the iam_admin_roles task. The model tried it first on every run, got rejected by our own code, and fell back to a naive 36-call loop. We were measuring our whitelist, not MCP vs CLI. Article has the full story; the fix is one line in safety.ALLOWED.

Running the benchmark yourself

# 1. Clone and set up
git clone https://github.com/webmaster-ramos/mcp-vs-cli-aws-benchmark
cd mcp-vs-cli-aws-benchmark
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# 2. Create a read-only IAM user and put its credentials under the
#    profile "mcp-benchmark" (see docs/setup.md for the aws iam
#    commands). Then:
cp .env.example .env
$EDITOR .env   # AWS_PROFILE, AWS_REGION, ANTHROPIC_API_KEY

# 3. Confirm the IAM user works and sees your account
./scripts/discover.sh

# 4. Snapshot ground truth for your account
python -m src.ground_truth

# 5. Run the full series (5 tasks x 5 transports x n=10 ≈ 15-20 min)
python -m src.runner --n 10

# 6. Summarise
python -m src.aggregate results/raw/latest.jsonl

Individual smoke tests and ablation variants are in src/dry_run.py:

python -m src.dry_run cli ec2_running
python -m src.dry_run cli-ctx ec2_cpu_last_hour
python -m src.dry_run cli-v2 iam_admin_roles
python -m src.dry_run mcp s3_bucket_regions
python -m src.dry_run cli-full ec2_cpu_last_hour

Repo layout

mcp-vs-cli-aws-benchmark/
├── README.md                    # this file
├── pyproject.toml
├── .env.example
├── src/
│   ├── agent_loop.py            # hand-rolled Anthropic agent loop
│   ├── safety.py                # whitelist + subprocess wrapper
│   ├── tasks.py                 # 5 task definitions
│   ├── ground_truth.py          # boto3 ground truth fetcher
│   ├── verify.py                # verdict vs ground truth
│   ├── tools_cli.py             # plain CLI transport
│   ├── tools_cli_v2.py          # batch + rich description CLI
│   ├── tools_cli_rich.py        # description ablation (unused in final)
│   ├── tools_cli_renamed.py     # naming ablation (unused in final)
│   ├── tools_cli_with_fake_suggest.py  # scaffolding ablation (unused in final)
│   ├── tools_mcp.py             # mcp python lib wrapper for awslabs server
│   ├── dry_run.py               # single-task smoke tests
│   ├── runner.py                # full benchmark loop, JSONL output
│   └── aggregate.py             # median + IQR + success rate table
├── scripts/
│   ├── discover.sh              # read-only AWS account probe
│   └── scrub.py                 # account ID / ARN redaction
├── tests/
│   └── test_safety.py           # 24 whitelist tests
├── results/
│   ├── raw/                     # JSONL per run (gitignored, contains
│   │                            # real account / instance / bucket names)
│   └── scrubbed/
│       ├── final_summary.json   # aggregated n=10 summary, publishable
│       ├── sample_runs.jsonl    # 8 hand-curated runs, 3 mechanisms
│       └── sample_runs.README.md
└── docs/
    └── findings.md              # analytical writeup (basis of the article)

License

MIT.

Privacy / data handling

• Real account ID, ARNs, bucket names and instance IDs appear in results/raw/ and are gitignored.
• scripts/scrub.py maps them to stable placeholders (<ACCOUNT_ID>, bucket-N, i-instance0N) before anything moves to results/scrubbed/.
• Only scrubbed artefacts are committed or published.