A comprehensive benchmarking framework for evaluating AI agents on science tasks. The system provides standardized environments, tools, and evaluation metrics to test agent performance across diverse materials science challenges.
The shortest useful Corral run does not need CorralRunner, trials, scoring,
or report generation. corral run executes one independent task using
the execute_task function in run.py.
You need Python 3.11 or newer for SampleMath, uv,
and a model API key such as OPENAI_API_KEY. Clone the
repository and install SampleMath together with the framework:
git clone https://github.com/lamalab-org/corral.git
cd corral/tasks/samplemath
uv syncRun one task with one agent and model:
uv run corral run \
--agent tool-calling \
--environment samplemath \
--task task1 \
--model openai/gpt-5.6This path skips evaluation: it prints the final status, submitted
answer, and commit hash, and writes the authored commit ledger to
.corral/run-commits.sqlite3. It does not create trials, invoke a scorer, or
generate a benchmark report. Because it executes exactly one task, it rejects a
task with upstream dependencies and points to corral bench instead.
Re-running with the same --execution-id and commit database restores the
latest persisted task state.
The equivalent Python API is below. Save this as quickstart.py in the current
tasks/samplemath directory:
import asyncio
from uuid import uuid4
from dotenv import load_dotenv
from corral import (
RuntimeRegistry,
RunTaskInput,
execute_task,
load_environment_group,
)
from corral.agents import ToolCallingAgent
from corral.persistence import SQLiteCommitStore
AGENT_ID = "tool-calling"
TASK_ID = "task1"
MODEL = "openai/gpt-5.6"
async def main():
load_dotenv()
environment = load_environment_group("samplemath")[TASK_ID]
store = SQLiteCommitStore(".corral/quickstart-commits.sqlite3")
registry = RuntimeRegistry(
agents={AGENT_ID: ToolCallingAgent(model=MODEL)},
environments={TASK_ID: environment},
)
try:
state = await execute_task(
state_store=store,
registry=registry,
task=RunTaskInput(
execution_id=f"samplemath-task1-{uuid4().hex}",
task_id=TASK_ID,
environment_id=TASK_ID,
agent_id=AGENT_ID,
model=MODEL,
max_iterations=10,
),
)
finally:
registry.close()
await store.aclose()
print(f"status: {state.status}")
print(f"answer: {state.submission}")
if __name__ == "__main__":
asyncio.run(main())Run it with the current SampleMath virtual environment:
uv run python quickstart.pyThe result is a StateRef pointing to the persisted final state, and the
example prints its status and submitted answer. No scorer,
aggregate metric, or benchmark report runs. task1 is independent; use
corral bench for a task such as task4 whose inputs come from earlier tasks.
The installed corral command can run every public concrete agent against a
registered environment.
Each task environment is a separate Python project because it has its own
dependencies. Change into that project once, then uv automatically uses its
environment and no --project option is needed:
cd tasks/samplemath
uv sync
uv run corral bench \
--agent react \
--environment samplemath \
--model openai/gpt-4o \
--task task4 \
--trials 3corral bench calls the same CorralRunner used by Python callers. The runner
schedules trials in the CLI process using asyncio.
Benchmarks run one Docker container per (run, task, trial) by default. Corral
builds corral-benchmark:latest from docker/benchmark.Dockerfile when that
image is missing, resolves it to an immutable image ID before scheduling any
trial, and applies these defaults: 2 CPUs, 4 GiB memory, 256 PIDs, a read-only
root filesystem, and Docker's bridge network so the container has outbound
network access. Pass --sandbox-network none for fully network-isolated trials.
Use --sandbox local only for local debugging; corral run remains
local and never contacts Docker.
By default, each invocation creates one descriptive, self-contained directory
below .corral/runs/. Its name records the UTC start time, agent, model,
environment, requested tasks, k/trial count, and sandbox mode. The exact task
and trial checkpoint layout is:
.corral/runs/<descriptive-run-name>/
├── report.json
├── run-metadata.json
└── task-<task-id>/
└── k-<1-based-trial>/
├── metadata.json
├── commits.sqlite3
├── artifacts/
├── workspace-snapshots/
│ ├── <revision>.json
│ └── latest.json
├── state-snapshots/
│ ├── <sequence>-<commit>.json
│ ├── latest.json
│ └── final.json
└── recovery/
Workspace snapshot manifests reference the content-addressed files in
artifacts/, so the combination is a complete, restorable workspace snapshot.
State projections are exported at the ledger interval and again at terminal
completion; commits.sqlite3 remains the authoritative, replayable history.
Docker runs also place request.json, result.json, and sandbox.json in the
trial directory.
The final report.json includes the resolved agent, model, environment, task
mapping, benchmark settings, concurrency, retry policy, sandbox configuration,
and output paths. Credential-like values in runtime option dictionaries are
redacted. --output-dir (also accepted as --state-dir) moves the runs root,
and --report overrides the default report location. If a task launch
fails, its retry discards the old container and volume, creates a clean
workspace, and restores the last committed workspace revision. Completed
actions are not repeated; filesystem changes from an action interrupted before
its completion commit are deliberately excluded. --keep-sandboxes on-failure
or always retains Docker resources for inspection without making them the
recovery source of truth.
The main isolation controls all have CLI defaults and can be overridden with
--sandbox-image, --sandbox-cpus, --sandbox-memory,
--sandbox-pids-limit, --sandbox-network, --keep-sandboxes, and repeated
--sandbox-env NAME arguments. Only allowlisted host variables are forwarded;
the built-in list covers common model-provider credentials. Environment stacks
that need dependencies beyond the base image should provide their own image
with --sandbox-image, optionally paired with an image-specific registry via
--sandbox-registry-module module:create_registry.
If that environment is already activated, uv run is optional and the command
is simply corral bench .... The --project form is only needed when invoking
uv from elsewhere in the monorepo.
The available agent names are ai-scientist, claude-code, codex,
llm-planner, openhands, react, reflexion, terminus, and
tool-calling. Class names and underscore spellings such as ReActAgent and
tool_calling are accepted as aliases. Claude Code, Codex, and OpenHands need
their corresponding optional package extra and report an installation hint if
it is missing.
Use --agent-kwargs for constructor-specific settings. Where the agent supports
them, explicit --model, --api-endpoint, and --temperature options take
precedence over the same keys in that JSON object:
uv run corral bench \
--agent codex --environment samplemath --task task1 \
--model gpt-5.4 \
--agent-kwargs '{"reasoning_effort": "high"}'ReflexionAgent wraps ToolCallingAgent by default. Select another actor with
{"actor": "react", "actor_kwargs": {...}}. For AIScientistAgent, a JSON
config object is validated as AIScientistConfig; add
"config_profile": "sakana" to use SakanaAIScientistConfig.
run_scripts/run_tool_calling.py delegates to
the same local benchmark execution function as corral bench; it does not maintain
a separate runner implementation. The runner infers BenchmarkTaskMetadata
and includes the selected task's dependencies automatically. Its existing
options, defaults, and invocation remain supported:
From the repository root:
uv sync --project tasks/samplemath
uv run --project tasks/samplemath python run_scripts/run_tool_calling.py \
--environment samplemath --list-tasks
uv run --project tasks/samplemath python run_scripts/run_tool_calling.py \
--environment samplemath --task task4 --model openai/gpt-4o \
--report .corral/samplemath-report.jsonThe built-in presets are afm, catalyst, corral_md, ml,
resistor_network, retrosynthesis, samplemath, spectra_elucidation, and
wetlab. These are fixed choices for --environment. Use --env-kwargs for
environment-specific configuration, including the common level, subtasks,
task_config, and work_dir keys. Environment dependencies and credentials
still need to be configured as described in each task package's README.
Omit --task to run every task returned by the environment. Pass --task
multiple times to select specific tasks; their required dependencies are
included automatically.
The runner applies no time limit to task execution or evaluation. Failed attempts
are retried up to --max-attempts times (default: 3).
For example, select SampleMath's subtask set with one environment argument:
python run_scripts/run_tool_calling.py --environment samplemath \
--env-kwargs '{"subtasks": true}'CorralRunner is the higher-level convenience layer for repeated trials,
evaluation, metric calculation, and reports. Use it after the direct execution
path above when those benchmark features are actually needed. Given the
store and model settings from the example above (keep the store open
until the benchmark finishes):
from corral import CorralRunner
environments = load_environment_group("samplemath")
registry = RuntimeRegistry(
agents={AGENT_ID: ToolCallingAgent(model=MODEL)},
environments=environments,
)
runner = CorralRunner(
registry,
environments=environments,
agent_id=AGENT_ID,
model=MODEL,
max_iterations=10,
state_store=store,
)
result = await runner.run(
"samplemath-run-2",
task_ids=["task1", "task2"],
trials_per_task=3,
k_values=[1, 2, 3],
max_parallel=4,
max_parallel_per_task=2,
)CorralRunner handles concurrency, task retries, and dependency readiness.
Task state remains persisted in the commit store. Tool verbosity is fixed to Corral's default (brief) on this path.
The framework includes several pre-built environments:
| Environment | Description |
|---|---|
samplemath |
Basic mathematical operations |
spectra_elucidation |
Spectroscopy/NMR spectra elucidation tasks |
corral_md |
LAMMPS molecular dynamics simulation setup |
catalyst |
Catalysis research and material design tasks |
afm |
Atomic force microscopy image analysis |
ml |
Machine learning model training and evaluation |
The framework includes several built-in agent types:
Uses progressive tree search to formulate hypotheses, run experiments, refine results, and verify conclusions. It is based on Sakana AI's original AI Scientist v2 implementation.
from corral.agents import AIScientistAgent
agent = AIScientistAgent(
model="gpt-4o",
evaluator_model="gpt-4o",
)Uses the ReAct (Reasoning and Acting) framework for step-by-step problem solving.
from corral.agents import ReActAgent
agent = ReActAgent(
model="gpt-4o", # or "claude-3-5-sonnet-20241022" or any other model litellm supports
temperature=0.1,
)Uses native function calling from LLM providers to solve tasks by leveraging built-in tool/function calling capabilities.
from corral.agents import ToolCallingAgent
agent = ToolCallingAgent(
model="gpt-4o", # or "claude-3-5-sonnet-20241022" or any other model LiteLLM supports
temperature=0.0,
)Uses hierarchical planning with high-level planning and low-level execution delegation to other agents.
from corral.agents import LLMPlanner
agent = LLMPlanner(model="gpt-4o", temperature=0.1)Implements the Reflexion architecture (paper) which adds self-reflection and learning from mistakes.
from corral.agents import ReActAgent, ReflexionAgent, ToolCallingAgent
# Create base agent (the "Actor")
base_agent = ToolCallingAgent(model="gpt-4o", temperature=0.1)
# Wrap with Reflexion capabilities
reflexion_agent = ReflexionAgent(
actor=base_agent,
reflection_model="gpt-4o", # Model for generating reflections
reflection_temperature=0.0, # Deterministic reflections
)Corral persists one small, immutable, typed commit for every durable event in a
SQLite ledger. parent_hash links each ordinary commit to the current branch
head, while based_on_hash records the projection the author actually saw.
ExecutionState is rebuilt from the ledger plus occasional replay snapshots;
agents receive an authorized AgentContext, not the complete execution trace.
Agent and tool authors are bound capabilities. Subagents share the same linear
branch and appear in the parent conversation as ordinary tool calls with
automatic result summaries. Their full conversations and state remain private
unless the parent chooses to inspect and import selected details. Parallel tool
completions are stored in real completion order and presented to the model in
declared action order. Shared environment and workspace effects carry revision
preconditions, so overlapping writes conflict instead of silently rebasing.
Explicit experiments use SQLiteCommitStore.create_branch(); spawning a
subagent never creates a branch.
Subagent inspection is opt-in per agent. Agents declaring
AgentSessionCapabilities(inspect_subagents=True) receive an
inspect_subagent tool that returns a bounded authorized child context through
the normal tool-call ledger path. AI Scientist enables it; other built-in
agents do not expose it.
-
Create environment directory
mkdir -p tasks/my_new_env/my_new_env cd tasks/my_new_env -
Create pyproject.toml
[project] name = "my_new_env" version = "0.1.0" dependencies = [ "corral", # Add your specific dependencies ]
-
Create tools
# tasks/my_new_env/my_new_env/tools.py from corral.core.tool import tool @tool def my_custom_tool(input_param: str) -> str: """Description of what the tool does. Args: input_param: Description of the parameter Returns: Description of the return value """ # Your tool implementation return f"Processed: {input_param}"
Note that the docstring has to be formatted correctly for the tool to be registered properly. This means it has to include a description of the parameters and return values as in the example above.
-
Define the task and environment
# tasks/my_new_env/my_new_env/env.py from corral.core.environment import Environment, Toolset from corral.core.task import TaskDefinition from .tools import my_custom_tool task = TaskDefinition( name="task_1", description="Solve this problem: Problem 1", tools=["my_custom_tool"], scoring_fn=lambda answer: float(answer == "Answer 1"), submission_format={"answer": "string"}, resolve_answer=False, ) environment = Environment( "task_1", task, toolset=Toolset(pool={"my_custom_tool": my_custom_tool}), )
-
Create agent file
# src/corral/agents/my_agent.py from corral.agents import AgentOutcome from corral.core import submit_answer_action class MyAgent: model = "gpt-4o" async def run_session(self, session): result = await session.execute(submit_answer_action("Your final answer")) if not result.success: return AgentOutcome( status="protocol_failure", error=f"submit_answer failed: {result.error}", ) return AgentOutcome(status="completed", answer="Your final answer")
-
Add to agent registry
# src/corral/agents/__init__.py from .my_agent import MyAgent __all__ = ["MyAgent", ...]
-
Register your agent for execution
from corral.agents.my_agent import MyAgent from corral import RuntimeRegistry registry = RuntimeRegistry( agents={"my-agent": MyAgent()}, environments={"my-environment": environment}, )
-
Install development dependencies
uv pip install -e . -
Install pre-commit hooks with commitizen commits
pre-commit install --hook-type commit-msg --hook-type pre-push
from corral.core.tool import tool
@tool
def calculate_molecular_weight(formula: str) -> float:
"""Calculate molecular weight from chemical formula.
Args:
formula: Chemical formula (e.g., 'H2O', 'CH4')
Returns:
Molecular weight in g/mol
"""
# Implementation here
passThe framework provides comprehensive evaluation metrics:
result = await runner.run(
"metrics-run-1",
trials_per_task=10,
k_values=[1, 3, 5],
)
metrics = result.calculate_metrics()
print(metrics["average_score"])
print(metrics["pass_at_1"])
# Per-task analysis
for task_id, task_result in result.task_results.items():
print(task_id, [trial.score for trial in task_result.trials])- Issues: Report bugs and request features on GitHub Issues
- Discussions: Join conversations on GitHub Discussions
- Contributing: See our Contributing Guide
This project is licensed under the MIT License - see the LICENSE file for details.
If you use Corral in your research, please consider citing:
@article{ríos-garcía2026ai,
title = {AI scientists produce results without reasoning scientifically},
author = {Martiño Ríos-García and Nawaf Alampara and Chandan Gupta and Indrajeet Mandal and Sajid Mannan and Ali Asghar Aghajani and N. M. Anoop Krishnan and Kevin Maik Jablonka},
year = {2026},
journal = {arXiv preprint arXiv: 2604.18805}
}