AHE++ — Causal and Multi-Objective Automatic Evolution of Coding-Agent Harnesses

AHE++ is a production-quality research prototype inspired by Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses (arXiv:2604.25850) and informed by the public AHE implementation. It does not clone the original repository and does not aim to be another generic coding agent.

The research question:

Can automatic harness evolution become more reliable if we explicitly measure the causal effect of each harness modification, identify which component caused the improvement, optimize performance and cost together, and test whether improvements generalize?

Current Phase

Phase 10 is implemented: Phase 1's sandboxed tool-using agent, Phase 2's 10-task benchmark runner, Phase 3's experience observability, Phase 4's harness evolution proposal workflow, Phase 5's causal control/treatment experiment engine, Phase 6's component-level attribution, Phase 7's experience memory retrieval, Phase 8's regression protection gates, Phase 9's cross-model generalization matrix, and Phase 10's React research dashboard.

Implemented now:

• Required tools: read_file, write_file, edit_file, search_code, terminal, run_tests
• Docker sandbox executor with repository mount, network policy, CPU/memory limits, timeout, stdout/stderr/exit-code/duration capture
• Deterministic Phase 1 coding-agent runtime
• 10 real local benchmark tasks under benchmark/tasks
• Benchmark runner that copies tasks into isolated temporary workspaces
• Aggregate metrics from actual stored trajectories
• Event extraction from raw trajectories
• Failure/inefficiency pattern detection
• Root-cause evidence and component-targeted recommendations
• Structured evolution proposals with falsifiable predictions
• Candidate harness version artifacts with base Git SHA, manifest, and Git-style patch
• Reversible candidate creation that does not mutate the active Harness v0 source tree
• Control/treatment/holdout experiment execution over the same benchmark task set
• Observed effect estimation and prediction-error calculation
• Component-level attribution experiments over every changed-component subset
• Interaction reporting for multi-component candidates
• PostgreSQL/pgvector experience memory model and repository
• Deterministic embedding model for reproducible local retrieval tests
• Retrieval-backed evolution proposals that explicitly include historical evidence
• Regression protection for overall, category, holdout, previously successful task, cost, and latency checks
• Configurable reject/warn/accept policy gate
• Cross-model generalization matrix over source and unseen model profiles
• Transfer classification: GENERAL, MODEL_SPECIFIC, NEGATIVE_TRANSFER, or NO_EFFECT
• React + TypeScript + Vite + Tailwind dashboard
• React Query artifact loading and Recharts visualizations
• JSON trajectory persistence
• SQLAlchemy table-boundary scaffold for the requested production database model
• FastAPI health endpoint
• Unit/integration tests for tools, trajectories, agent execution, Docker sandbox command construction, benchmark execution, and observability analysis

Deferred by design:

• Celery worker execution remains deferred

Reference Analysis

See docs/architecture.md and docs/phase-1-plan.md.

Phase reports:

• docs/phase-1-plan.md
• docs/phase-2-report.md
• docs/phase-3-report.md
• docs/phase-4-report.md
• docs/phase-5-report.md
• docs/phase-6-report.md
• docs/phase-7-report.md
• docs/phase-8-report.md
• docs/phase-9-report.md
• docs/phase-10-report.md

Conceptually reused from AHE:

• file-level harness components
• trace-first evaluation
• experience distillation
• prediction-backed edits
• Git-backed reversibility

Independently reimplemented for AHE++:

• Docker sandbox runtime
• tool execution layer
• trajectory schema
• benchmark task format
• production backend schema boundaries

Run Phase 1

Install dev dependencies:

cd path\to\ahe-plus-plus
pip install -e ".[dev]"

Run tests:

python -m pytest

Run the 10-task benchmark with Docker sandbox:

python scripts\run_benchmark.py --output-root experiments\results

For development-only verification without Docker:

python scripts\run_benchmark_local.py --output-root experiments\results

Latest verified local-dev benchmark:

experiments/results/benchmark-20260812T103243Z/summary.json

Observed Harness v0 metrics from that run:

{
  "task_count": 10,
  "pass_rate": 1.0,
  "total_tokens": 5959,
  "avg_latency_ms": 242.6,
  "avg_tool_calls": 5.1
}

Run experience observability on a benchmark run:

python scripts\analyze_experiences.py experiments\results\benchmark-20260812T103243Z

Latest generated experience reports:

experiments/results/benchmark-20260812T103243Z/experience_report.json
experiments/results/benchmark-20260812T103243Z/experience_report.md

Observed Phase 3 analysis:

{
  "trajectory_count": 10,
  "event_count": 132,
  "patterns_detected": 2
}

Create a Phase 4 harness evolution candidate:

python scripts\propose_evolution.py experiments\results\benchmark-20260812T103243Z\experience_report.json

Latest generated candidate:

experiments/candidates/harness-candidate-07c12f3f

Candidate prediction:

{
  "target_component": "middleware",
  "success_rate_delta": 0.0,
  "token_delta": -0.08,
  "tool_call_delta": -0.18,
  "latency_delta": -0.15
}

Run the Phase 5 causal experiment:

python scripts\run_causal_experiment.py experiments\candidates\harness-candidate-07c12f3f\manifest.json --local-dev

Latest generated experiment report:

experiments/results/experiment-harness-candidate-07c12f3f/experiment_report.json
experiments/results/experiment-harness-candidate-07c12f3f/experiment_report.md

Observed Phase 5 main effect:

{
  "control_pass_rate": 1.0,
  "treatment_pass_rate": 1.0,
  "pass_rate_difference": 0.0,
  "token_relative_delta": -0.6689529957635667,
  "tool_call_relative_delta": -0.1951219512195122,
  "latency_relative_delta": -0.525764192139738
}

The experiment decision remains PENDING; Phase 5 measures causal effect but does not accept harness changes into the active version.

Run Phase 6 component attribution:

python scripts\run_attribution.py experiments\candidates\harness-candidate-07c12f3f\manifest.json --local-dev

Latest generated attribution report:

experiments/results/attribution-harness-candidate-07c12f3f/attribution_report.json
experiments/results/attribution-harness-candidate-07c12f3f/attribution_report.md

Observed Phase 6 component contribution:

{
  "component": "middleware",
  "pass_rate_contribution": 0.0,
  "token_relative_delta": -0.6643732169827152,
  "tool_call_relative_delta": -0.19607843137254904,
  "latency_relative_delta": -0.47337526205450736
}

Run Phase 7 retrieval-backed proposal generation:

python scripts\propose_with_memory.py experiments\results\benchmark-20260812T103243Z\experience_report.json --attribution-report experiments\results\attribution-harness-candidate-07c12f3f\attribution_report.json --local-dev

Latest generated memory-backed proposal:

experiments/memory/memory-proposal-93ec0893/proposal.json
experiments/memory/memory-proposal-93ec0893/retrieved_experiences.json

Observed Phase 7 retrieval effect:

{
  "retrieved_top_experience": "test_after_edit_retry",
  "similarity": 0.7929695146252059,
  "prior_tool_call_delta": -0.19607843137254904,
  "new_prediction_tool_call_delta": -0.19607843137254904
}

Run Phase 8 regression gate:

python scripts\run_regression_gate.py experiments\results\experiment-harness-candidate-07c12f3f\experiment_report.json

Latest generated regression report:

experiments/results/regression-harness-candidate-07c12f3f/regression_report.json
experiments/results/regression-harness-candidate-07c12f3f/regression_report.md

Observed Phase 8 decision:

{
  "decision": "ACCEPT",
  "overall_pass_rate_delta": 0.0,
  "holdout_pass_rate_delta": 0.0,
  "latency_delta": -0.525764192139738,
  "reject_findings": 0,
  "warning_findings": 0
}

Run Phase 9 cross-model generalization:

python scripts\run_generalization.py experiments\candidates\harness-candidate-07c12f3f\manifest.json --local-dev

Latest generated generalization report:

experiments/results/generalization-harness-candidate-07c12f3f/generalization_report.json
experiments/results/generalization-harness-candidate-07c12f3f/generalization_report.md

Observed Phase 9 classification:

{
  "classification": "GENERAL",
  "source_model": "local-seed",
  "unseen_models": [
    "gpt-4o-mini-profile",
    "claude-haiku-profile",
    "ollama-llama-profile"
  ],
  "tool_call_delta_all_models": -0.19607843137254904,
  "pass_rate_delta_all_models": 0.0
}

Phase 9 currently uses deterministic local model profiles for OpenAI, Anthropic, and Ollama-style comparisons. It does not call live provider APIs yet.

Run the Phase 10 dashboard:

cd frontend
npm install
npm run dev

Then open:

http://127.0.0.1:5173

Build verification:

cd frontend
npm run build

The dashboard syncs real generated artifacts from experiments/ into frontend/public/data before dev/build.

Run the Phase 1 agent with Docker sandbox:

python -m agent.runtime.cli run-task benchmark\tasks\bugfix_math --output-dir experiments\results

Docker Desktop must be running. If docker info cannot connect to the Docker Desktop Linux engine, start Docker Desktop and rerun the command.

For development-only verification without Docker:

python scripts\run_phase1_local.py benchmark\tasks\bugfix_math --output-dir experiments\results

The local-dev command uses the same tools and trajectory schema, but it is not the production sandbox boundary.

Expected result:

{
  "task_id": "bugfix_math",
  "success": true,
  "trajectory_path": "experiments\\results\\bugfix_math-harness-v0.json"
}

The trajectory contains the required schema:

{
  "task_id": "...",
  "agent_id": "...",
  "harness_version": "...",
  "timestamp": "...",
  "tool_calls": [
    {
      "tool": "...",
      "input": "...",
      "output": "...",
      "duration_ms": 0,
      "tokens": 0
    }
  ],
  "final_result": {
    "success": true,
    "tests_passed": 1,
    "tests_failed": 0
  }
}

Run the FastAPI health check:

uvicorn backend.app.main:app --reload

Then visit:

http://127.0.0.1:8000/health

Infrastructure

PostgreSQL with pgvector and Redis are declared for later phases:

$env:AHEPP_POSTGRES_PASSWORD = "replace-with-a-long-random-local-password"
docker compose up -d postgres redis

The compose file binds Postgres and Redis to 127.0.0.1 only. Do not expose these services directly on a public interface.

Phase 1 and Phase 2 do not require them to run.

Security Notes Before Public Use

• DockerSandboxExecutor is the only executor intended for untrusted task code. The local executor requires allow_unsafe=True and is limited to tests/local development scripts because it runs commands on the host.
• Benchmark tests and ground-truth files are protected in two layers: file tools refuse writes to test paths, and the benchmark runner hashes tests before and after each task. A run is marked failed if tests are modified.
• Candidate harness files are allowlisted by target component and reject absolute paths or .. traversal.
• Dashboard data is generated from experiment artifacts with local paths, likely secrets, and long tool outputs scrubbed. Generated files under frontend/public/data/ are ignored and should be regenerated locally.

Next Phase

Next production-hardening work:

• Celery workers for long-running benchmark jobs
• live PostgreSQL/pgvector verification once Docker Desktop is running
• real OpenAI/Anthropic/Ollama provider execution
• Git accept/revert lifecycle for approved harness versions