This repository contains code for benchmarking optimization solvers on problems from the energy planning domain, and an interactive website for analyzing the results. The live website can be viewed at:
https://openenergybenchmark.org/
All our benchmark problems are open and available as LP/MPS files that can be downloaded in one click from our website's Benchmark Problem Set page. Some problems have been generated by us using open source energy modelling frameworks, and for these we have configuration files and instructions for reproducing the problems.
For more details on how to contribute benchmark problems, see the Benchmarks README.
The benchmark runner can run solvers listed in the Solvers README. We use the last released solver version in each calendar year. 2025 solvers will be updated at the end of the year.
Understanding the project layout to help you navigate and contribute:
solver-benchmark/
├── runner/ # Benchmark execution scripts
│ ├── benchmark_all.sh # Main entry point for running benchmarks
│ ├── run_benchmarks.py # Python script that orchestrates benchmark runs
│ ├── run_solver.py # Individual solver runner
│ ├── envs/ # Conda environment definitions for each solver year
│ └── benchmarks/ # Downloaded benchmark problem files
├── benchmarks/ # Benchmark problem definitions and metadata
│ ├── pypsa/ # PyPSA-generated energy models
│ ├── jump_highs_platform/ # JuMP/HiGHS benchmark metadata
│ └── *_metadata.yaml # Problem definitions and details
├── website/ # Next.js website for viewing results
├── infrastructure/ # GCP VM deployment scripts (for running benchmarks at scale)
├── results/ # Output directory for benchmark results
├── benchmark_results.csv # Main results file
└── metadata.yaml # Merged metadata of all problems on the websiteThe benchmark runner currently requires Linux as it uses systemd-run to enforce memory limits on solvers, which is not available on macOS or Windows.
Supported Linux distributions:
- Ubuntu 20.04 LTS or later
- Debian 11 or later
- Other systemd-based Linux distributions
macOS / Windows users: Use the provided Docker image to run benchmarks in a Linux container. See Running with Docker below.
Ensure you have the following installed:
- Python 3.12+
- Conda (install Miniconda)
- systemd (usually pre-installed on modern Linux distributions)
The benchmark runner script (runner/benchmark_all.sh) is the main entry point for running benchmarks. It takes a list of solvers and a list of years as arguments, and runs the benchmarks for each solver and year. It creates conda environments containing the solvers and other necessary prerequisites, so a virtual environment is not necessary just for running the benchmark runner. See README .
Quickstart:
- Run benchmarks
./runner/benchmark_all.sh -s "highs scip" -y "2025" infrastructure/benchmarks/sample_run/standard-00.yaml- View logs and results
tail results/benchmark_results.csv # will overwrite currently committed results
tail runner/logs/*- View and analyze results by running the website locally
The script will save the measured runtime and memory consumption into a CSV file in results/ that the website will then read and display. Running the website locally will allow you to view and analyze results in a user friendly way. It will use the results from results/benchmark_results.csv.
runner/benchmark_all.sh uses runner/run_benchmarks.py to run the benchmarks by year. If you wish to run benchmarks directly, you can set up the requisite conda env manually. See Documentation.
You can use Docker to run benchmarks in a Linux container that has all the required dependencies.
# Build the runner image
docker build -t solver-benchmark-runner -f runner/Dockerfile .
# Run all solvers across all years, results are written to host via a volume mount
docker run --rm \
-v $(pwd)/results:/solver-benchmark/results \
solver-benchmark-runner results/metadata.yamlThe container accepts the same flags as benchmark_all.sh (e.g. -s, -y). Memory limit enforcement via systemd-run is not available inside Docker and is skipped automatically. For more details on available options, conda env caching, and Gurobi licensing, see the runner Docker documentation.
We have cloud orchestration setup for running benchmarks on Google Cloud Platform. See Documentation.
Quickstart:
For cloud infrastructure setup, install:
gcloud auth application-default login
cd infrastructure
tofu init
tofu apply -var-file benchmarks/sample_run/run.tfvarsTo set up comprehensive benchmark campaigns, like the one available on the website:
- Use
notebooks/allocate-benchmarks-to-vms.ipynbto create the benchmark campaign. - Run
notebooks/run-and-observe-benchmarks.ipynbto observe the benchmark campaign progress.
Alternatively, the benchmark campaign generation can be performed from the command line using the tools described below.
This repository contains tooling to create benchmark campaigns from the benchmark metadata.
Campaigns can target either:
- cloud execution using Google Cloud Platform VMs (currently allowed to maintainers only);
- local execution using the existing benchmark runner workflow.
The main entry point is:
python benchmarks/create_benchmark_campaign.pyThe script prepares benchmark metadata, selects benchmark instances, and creates either:
- a cloud campaign under
infrastructure/benchmarks/<run-id>/; or - a local campaign under
infrastructure/local/benchmarks/<run-id>/.
The campaign generator supports two execution targets:
| Target | Description |
|---|---|
cloud |
Generates OpenTofu VM configuration files under infrastructure/benchmarks/<run-id>/. |
local |
Generates local benchmark files under infrastructure/local/benchmarks/<run-id>/. |
The default target is:
cloud
Override it with:
python benchmarks/create_benchmark_campaign.py \
--target localto launch a benchmark campaign on your local machine.
Create a campaign for all benchmark instances (please do this very carefully, especially if running on the cloud):
python benchmarks/create_benchmark_campaign.py \
--campaign my-test \
--allThis creates a run ID of the form:
YYYYMMDD-my-test
Campaigns can also be defined through a YAML configuration file.
A complete template is provided in:
benchmarks/config.campaign.default.yaml
Run a campaign directly from a configuration file:
python benchmarks/create_benchmark_campaign.py \
--configfile benchmarks/config.campaign.default.yamlCommand-line arguments always override values defined in the configuration file.
For example:
python benchmarks/create_benchmark_campaign.py \
--configfile benchmarks/config.campaign.default.yaml \
--campaign my-test \
--timeout-hours 6Select all instances of one benchmark:
python benchmarks/create_benchmark_campaign.py \
--campaign pypsa-eur-test \
--benchmark pypsa-eurSelect benchmark instances by size class:
python benchmarks/create_benchmark_campaign.py \
--campaign pypsa-eur-small-medium \
--benchmark pypsa-eur-elec \
--size S MSelect benchmark instances by metadata name, for example specific spatial or temporal resolutions (field Name in the metadata):
python benchmarks/create_benchmark_campaign.py \
--campaign pypsa-eur-resolution-test \
--benchmark pypsa-eur-elec \
--name 2-1h 3-2hCombine size and name filters:
python benchmarks/create_benchmark_campaign.py \
--campaign pypsa-eur-filtered \
--benchmark pypsa-eur-elec \
--size S M \
--name 2-1h 3-2hSelect mixed benchmark instances explicitly:
python benchmarks/create_benchmark_campaign.py \
--campaign mixed-test \
--instance pypsa-eur:2-1h \
--instance pypsa-earth:3-2hThe instance selector uses the format:
<benchmark-name>:<instance-name>
For example:
PyPSA-DE:10-1h
Include metadata entries marked as skipped (due to known timeout or memory issues):
python benchmarks/create_benchmark_campaign.py \
--campaign clean-test \
--all \
--do-not-skipThis section only applies to cloud campaigns.
By default, the script creates one VM per selected benchmark instance.
Use a custom number of VMs with:
python benchmarks/create_benchmark_campaign.py \
--campaign packed-test \
--all \
--num-vms 5By default, benchmark instances are assigned to VM profiles automatically based on their metadata size class:
| Size class | Machine profile | GCP machine type | Timeout |
|---|---|---|---|
| S, M | short | c4-standard-2 | 1 hour |
| L | long | c4-highmem-16 | 24 hours |
Default zone:
us-central1-a
When --machine-type is specified, all selected benchmark instances use the chosen profile regardless of their size classification.
If no timeout is provided, the script applies the default timeout policy:
S/M instances: 1 hour
L instances: 24 hours
Override the timeout:
python benchmarks/create_benchmark_campaign.py \
--campaign timeout-test \
--all \
--timeout-hours 6By default, the generated VM YAML files benchmark all the following solvers:
gurobi highs scip cbc glpk
The default solver year is:
2025
The year corresponds to the benchmark conda environment:
benchmark-<year>
For example:
2025 -> benchmark-2025
Run specific solvers for one or more years:
python benchmarks/create_benchmark_campaign.py \
--campaign year-test \
--all \
--solver cbc highs \
--years 2024 2025Every generated campaign includes a:
campaign_summary.csv
containing the benchmark selection, campaign configuration, solver selection, timeout settings, allocation decisions, and metadata used to create the campaign.
The script fails if the target campaign directory already exists.
Use a different campaign name, remove the existing directory, or overwrite it:
python benchmarks/create_benchmark_campaign.py \
--campaign my-test \
--all \
--forceAfter reviewing the generated files, launch the campaign from the infrastructure directory:
cd infrastructure
tofu apply \
-var-file benchmarks/<run-id>/run.tfvars \
-state=states/<run-id>.tfstateGenerate a local campaign:
python benchmarks/create_benchmark_campaign.py \
--target local \
--campaign my-local-run \
--benchmark pypsa-de-elecThis creates:
infrastructure/local/benchmarks/<run-id>/
├── campaign_summary.csv
├── local_benchmarks.yaml
└── run_local.sh
By default, local campaigns ask for confirmation before execution.
The generated run script can also be executed manually:
bash infrastructure/local/benchmarks/<run-id>/run_local.shLocal campaigns use the existing runner/benchmark_all.sh workflow and execute benchmark instances sequentially.
python benchmarks/create_benchmark_campaign.py \
--configfile benchmarks/config.campaign.default.yaml \
--campaign pypsa-de-scaling
cd infrastructure
tofu apply \
-var-file benchmarks/<run-id>/run.tfvars \
-state=states/<run-id>.tfstatepython benchmarks/create_benchmark_campaign.py \
--configfile benchmarks/config.campaign.default.yaml \
--target local \
--campaign pypsa-de-localThe campaign generator creates the local benchmark files and asks whether the benchmark run should be started immediately.
To run your own benchmark problems, either locally or on the cloud, follow the steps in the appropriate section above but using a benchmarks.yaml file of your own that gives the details (metadata) and URL/path of your benchmark problems.
Here is a small example:
benchmarks:
genx-3_three_zones_w_co2_capture-no_uc:
Sizes:
- Name: 3-1h
# Size classification
Size: M
# URL of the problem (needed for cloud runs)
URL: https://storage.googleapis.com/solver-benchmarks/genx-3_three_zones_w_co2_capture-no_uc.lp
# ALTERNATIVELY, for local runs, you can also give a local path
Path: tests/sample_benchmarks/sample_lp.lpYou can quickly try running your own problem locally on our supported set of solvers by following these instructions.
To run either our benchmarks, or your own (see the previous section), on a solver that we do not yet support, you need to install it into the active conda evironment and modify the run_solver.py appropriately. Please reach out to us (or open an issue) if you would like more details, or any help with this.
The website code is under website/. To run the website locally, you need a recent version of node and npm installed. Then, run the following commands:
cd website/
npm install
npm run build && npm run devOpen http://localhost:3000 with your browser to see the website.
To see the results from your runs, navigate to the results page.
We use the ruff code linter and formatter, and GitHub Actions runs various pre-commit checks to ensure code and files are clean.
You can install a git pre-commit that will ensure that your changes are formatted and no lint issues are detected before creating new commits:
pip install pre-commit
pre-commit installIf you want to skip these pre-commit steps for a particular commit, you can run:
git commit --no-verify