PowerAgentBench

PowerAgentBench is a benchmark suite for evaluating AI agents on power-system operation and planning tasks. The current release includes steady-state and dynamic-study tracks, covering contingency analysis, dynamic model-quality review, dynamic security-risk screening, scripted baselines, and LLM/tool-agent evaluation.

The benchmark is built around a public/hidden split. Agents see public case data, scenarios, action spaces, and tool APIs. Hidden evaluators recompute steady-state or dynamic validity and return discovery, evidence, safety, mitigation, efficiency, workflow, and reliability metrics.

Repository Structure

PowerAgentBench/
├── cases/                                      # Network case data in multiple formats
│   ├── case39/
│   │   ├── pypsa/case39.nc                     # PyPSA netCDF format
│   │   ├── matpower/case39.m                   # MATPOWER .m format
│   │   └── pandapower/case39.json              # PandaPower JSON format
│   └── solar_wecc/
│       └── psse/                               # WECC solar PV dynamic case (PSS/E)
│           ├── Solar.sav                       # Power-flow case
│           └── Solar.dyr                       # Dynamic model (corrupted REECAU1 gains)
├── benchmarks/                                 # Benchmark definitions and task configs
│   ├── steady/
│   │   ├── level_1/                            # N-1 steady-state audit and mitigation
│   │   │   ├── README.md                       # Full Level 1 benchmark specification
│   │   │   ├── actionspace.json                # Action contract and operating limits
│   │   │   ├── actioncost.json                 # Per-step action costs
│   │   │   ├── baseline_summary.json
│   │   │   └── solution_template.json
│   │   └── level_2/                            # Agentic N-2 search and mitigation
│   │       ├── README.md                       # Full Level 2 benchmark specification
│   │       ├── .env.example                    # Template for private model/API configuration
│   │       ├── .gitignore                      # Keeps local .env files out of git
│   │       └── prompts/
│   │           └── steady_n2_llm_prompt.json   # Shared LLM tool-use prompt template
│   └── dynamic/
│       └── level1/                             # Dynamic model-quality review (DMView + PSS/E)
│           ├── README.md                       # Full benchmark spec + install prerequisites
│           ├── actionspace.json                # REECAU1 gain action contract + test suite
│           ├── actioncost.json                 # Per-simulation cost and budget
│           ├── baseline_summary.json           # Corrupted-model 0/8 reference + good solution
│           ├── solution_template.json
│           └── harness/                        # Runnable harness (DMView automation + agent loop)
├── scripts/                                    # Runnable entry points
│   ├── build_case.py                           # Rebuild the stressed Level 1 scenario
│   ├── convert_case.py                         # Export case39 to MATPOWER and PandaPower
│   ├── evaluate_solution.py                    # Score a Level 1 solution
│   ├── run_steady_n2_baselines.py              # Run Level 2 scripted baselines
│   ├── run_steady_n2_ollama_eval.py            # Run Level 2 Ollama-hosted LLM agents
│   └── run_steady_n2_openai_eval.py            # Run Level 2 OpenAI/ChatGPT-style agents
└── poweragentbench/                            # Shared library code
    ├── benchmark_utils.py                      # Level 1 case construction and scoring
    ├── steady_state_agentic.py                 # Level 2 DC N-2 evaluator and baselines
    ├── llm_agent_adapter.py                    # Provider-agnostic JSON-command LLM adapter
    ├── ollama_client.py                        # Ollama generate/chat client
    └── openai_client.py                        # OpenAI Responses API client

Installation

pip install -e .

The package intentionally uses lightweight Python dependencies. Provider SDKs are not required for the built-in Ollama and OpenAI runners because both clients use standard-library HTTP calls.

Quick Start

Level 1: N-1 steady-state audit and mitigation

# Rebuild the benchmark case from source
python scripts/build_case.py

# Export to MATPOWER and PandaPower formats
python scripts/convert_case.py

# Evaluate a solution
python scripts/evaluate_solution.py \
  --solution benchmarks/steady/level_1/solution_template.json

Level 2: Agentic N-2 search and mitigation

Run scripted baselines on deterministic variants of the existing IEEE 39-bus case:

python scripts/run_steady_n2_baselines.py \
  --case-source case39 \
  --cases 8 \
  --budget 80 \
  --report-k 20

Run deployed Ollama LLM agents:

python scripts/run_steady_n2_ollama_eval.py \
  --case-source case39 \
  --cases 8 \
  --budget 80 \
  --report-k 20 \
  --max-turns 12 \
  --prompt-template benchmarks/steady/level_2/prompts/steady_n2_llm_prompt.json

Run an OpenAI/ChatGPT-style agent, for example GPT-5.5:

python scripts/run_steady_n2_openai_eval.py \
  --case-source case39 \
  --cases 8 \
  --budget 80 \
  --report-k 20 \
  --max-turns 12 \
  --prompt-template benchmarks/steady/level_2/prompts/steady_n2_llm_prompt.json

Outputs are written under results/steady_n2/ for Ollama runs and results/steady_n2_openai/ for OpenAI runs. Each run produces per-case CSVs, aggregate CSVs, tool logs, sanitized API debug files, and LaTeX table rows.

Case Formats

The IEEE 39-bus stressed scenario is provided in three formats so that agents and solvers are not tied to a single tool:

• PyPSA (cases/case39/pypsa/case39.nc): primary format used by the Level 1 evaluator and by the Level 2 case39 converter.
• PandaPower (cases/case39/pandapower/case39.json): for PandaPower-based tools.
• MATPOWER (cases/case39/matpower/case39.m): for MATPOWER or MATPOWER-compatible solvers.

Benchmarks

Steady Level 1

benchmarks/steady/level_1/ evaluates N-1 steady-state audit and mitigation on a stressed IEEE 39-bus case. The agent receives a case, a published contingency list, and a bounded action space. The evaluator checks base-case and contingency violations after the submitted actions.

See:

benchmarks/steady/level_1/README.md

Steady Level 2

benchmarks/steady/level_2/ evaluates agentic N-2 contingency search and optional mitigation. The agent must spend a limited validation budget, submit evidence-backed ranked contingencies, and optionally improve the hidden post-action violation score.

The default case source is the existing IEEE 39-bus case distributed in this repository. The runner converts it to a lightweight DC representation and creates deterministic operating-point variants from fixed seeds. A synthetic fallback is also available for development.

See:

benchmarks/steady/level_2/README.md

Dynamic Level 1

benchmarks/dynamic/level1/ evaluates dynamic model-quality review on a modified WECC solar PV model. The agent runs the DMView model-quality test suite (flat start, voltage/frequency steps, HVRT/LVRT, weak-grid SCR), diagnoses the failures, and repairs the model by adjusting only four allowed REECAU1 controller gains within a five-iteration budget.

Note: Unlike the steady-state benchmarks (open-source PyPSA), this dynamic benchmark requires licensed/external tooling that you must install first: PSS/E 36.2 (Siemens, with valid license and Python bindings) and the DMView 3.4 dynamic-model review tool (https://sites.google.com/view/dmview/home), running on Python 3.11. Set DMVIEW_ROOT and PY311 in benchmarks/dynamic/level1/harness/config.py for your install.

See:

benchmarks/dynamic/level1/README.md

Model and API Configuration

Private model endpoints and API keys should not be committed to the repository. Configure them through a local .env file:

cp benchmarks/steady/level_2/.env.example benchmarks/steady/level_2/.env

The local .env file is ignored by Git. You may also pass the same settings through command-line flags or process environment variables.

Ollama configuration

Example local Ollama settings:

POWERAGENTBENCH_OLLAMA_URL=http://localhost:11434/api/generate
POWERAGENTBENCH_OLLAMA_MODELS=qwen3.5:latest mistral-nemo:12b command-r:35b
POWERAGENTBENCH_OLLAMA_TEMPERATURE=0.0
POWERAGENTBENCH_OLLAMA_NUM_CTX=16384
POWERAGENTBENCH_OLLAMA_API_MODE=generate
POWERAGENTBENCH_OLLAMA_THINK=false
POWERAGENTBENCH_OLLAMA_SCHEMA_FORMAT=true

For internal deployments, replace POWERAGENTBENCH_OLLAMA_URL locally. Do not commit internal URLs.

Some Ollama models expose a thinking field when POWERAGENTBENCH_OLLAMA_THINK=true. PowerAgentBench treats this only as a generation option. Raw thinking traces are not parsed, scored, or required for benchmark results.

OpenAI/ChatGPT configuration

Example local OpenAI settings:

POWERAGENTBENCH_OPENAI_API_KEY=sk-your-private-token
POWERAGENTBENCH_OPENAI_MODELS=gpt-5.5
POWERAGENTBENCH_OPENAI_URL=https://api.openai.com/v1/responses
POWERAGENTBENCH_OPENAI_TEMPERATURE=none
POWERAGENTBENCH_OPENAI_MAX_OUTPUT_TOKENS=4096
POWERAGENTBENCH_OPENAI_STRUCTURED_OUTPUTS=true
POWERAGENTBENCH_OPENAI_REASONING_EFFORT=medium
POWERAGENTBENCH_OPENAI_REASONING_SUMMARY=none
POWERAGENTBENCH_OPENAI_TIMEOUT=300
POWERAGENTBENCH_OPENAI_MAX_RETRIES=3
POWERAGENTBENCH_OPENAI_RETRY_BACKOFF=2.0

Many reasoning models reject a temperature parameter. Use POWERAGENTBENCH_OPENAI_TEMPERATURE=none to omit it. The OpenAI runner uses sanitized API debug logs and does not store the API key, raw output text, or reasoning content.

If a run times out, increase the timeout:

python scripts/run_steady_n2_openai_eval.py \
  --case-source case39 \
  --cases 8 \
  --budget 80 \
  --report-k 20 \
  --max-turns 12 \
  --prompt-template benchmarks/steady/level_2/prompts/steady_n2_llm_prompt.json \
  --timeout 600

Metrics

PowerAgentBench returns per-case and aggregate metrics, including:

• submitted, evidence-backed, and found top-20 recall,
• evidence rate and unvalidated-claim rate,
• best severity capture and severity regret,
• false-safe rates and severity-weighted false negatives,
• post-action violation and violation reduction,
• action cost,
• invalid tool calls,
• schema repairs and type coercions,
• duplicate validation requests,
• explicit submission and auto-finalization indicators,
• validation budget use,
• completed and requested case counts.

These metrics distinguish answer quality, tool evidence, search quality, mitigation quality, safety behavior, and workflow compliance.

Result Files

Typical Level 2 baseline outputs:

results/steady_n2/baseline_per_case.csv
results/steady_n2/baseline_summary.csv

Typical Ollama outputs:

results/steady_n2/ollama_all_per_case.csv
results/steady_n2/ollama_all_summary.csv
results/steady_n2/<model>_per_case.csv
results/steady_n2/<model>_summary.csv
results/steady_n2/<model>_tool_logs.jsonl
results/steady_n2/<model>_api_debug.jsonl

Typical OpenAI outputs:

results/steady_n2_openai/openai_all_per_case.csv
results/steady_n2_openai/openai_all_summary.csv
results/steady_n2_openai/<model>-OpenAI_per_case.csv
results/steady_n2_openai/<model>-OpenAI_summary.csv
results/steady_n2_openai/<model>-OpenAI_tool_logs.jsonl
results/steady_n2_openai/<model>-OpenAI_api_debug.jsonl

If an OpenAI run stops early after a retry failure, partial outputs are preserved with _partial in the filename and errors are written to an errors JSONL file.

Development Notes

• Use Level 1 to test basic steady-state action submission and physical validation.
• Use Level 2 to test agentic behavior, tool use, validation-budget allocation, evidence-backed reporting, and LLM workflows.
• Keep hidden oracle quantities, private endpoint URLs, and API keys outside the public repository.
• Rotate any API key that is accidentally shared or committed.
• Regenerate results after modifying prompts, adapters, scoring rules, or case-generation settings.