Efficient ale_bench_eval - #35
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR focuses on improving the efficiency and robustness of ale_bench_eval runs by introducing configurable LLM concurrency limits, parallelizing repeated-sampling within a problem, and making evaluation artifacts safer to persist/resume. It also updates evaluation dependencies and expands supported model configs/pricing, alongside minor infra and documentation updates.
Changes:
- Add global/per-problem LLM concurrency controls (
max_concurrent_llm_calls,max_repeated_sampling_workers) and parallelize repeated sampling with a thread pool. - Make result/conversation persistence more robust via atomic JSON writes and per-iteration self-refine conversation files (with cleanup).
- Update eval dependencies and add new LLM configs + pricing entries; refresh docs/scripts and Terraform naming options.
Reviewed changes
Copilot reviewed 14 out of 15 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
uv.lock |
Updates locked dependency versions (notably pydantic-ai, anthropic, fastmcp, dev tools). |
pyproject.toml |
Bumps dev/eval dependency minimums; keeps tool.uv relative exclude-newer. |
src/ale_bench_eval/__main__.py |
Adds CLI params for LLM concurrency; wires semaphore/worker limits into evaluation orchestration. |
src/ale_bench_eval/scaffolds.py |
Parallel repeated-sampling generation; adds retry wrapper for judge/save; adjusts self-refine persistence strategy. |
src/ale_bench_eval/safe_generation.py |
Adds optional semaphore gating around LLM calls. |
src/ale_bench_eval/logger.py |
Introduces atomic JSON dumping and uses it for SaveInfo outputs. |
src/ale_bench_eval/calc_cost.py |
Adds pricing entries for minimax-m3 and step-3.7-flash. |
tests/test_utils.py |
Tightens typing in dict comparisons using cast(). |
scripts/run_eval.sh |
Adds flags for new concurrency controls and refactors CLI invocation into an args array. |
docs/evaluation.md |
Documents new CLI/script options and updates output file naming for self-refine conversations. |
llm_configs/step-3.7-flash-medium.json |
New model config. |
llm_configs/minimax-m3.json |
New model config. |
llm_configs/claude-4.8-opus-high.json |
New model config. |
cloud/variables.tf |
Adds name_prefix variable and normalizes formatting. |
cloud/main.tf |
Applies name_prefix to resource names/tags and normalizes formatting. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
457
to
+460
| except Exception as e: | ||
| save_info.logger.info("Code evaluation failed for refinement %s: %s", i, e) | ||
| save_info.logger.info("Code evaluation failed for refinement %s after retries: %s", i, e) | ||
| msg = f"Judge/save failed during self-refinement {i}: {e}" | ||
| raise ValueError(msg) from e |
|
|
||
| # Or directly run using uv | ||
| uv run -m ale_bench_eval --model_config_path llm_configs/gpt-5.json --n_repeated_sampling 15 --n_self_refine 16 --num_workers 10 --n_public_cases 50 --judge_version 202510 --code_language typescript --prompt_language en --max_parallel_problems 5 --problem_ids_type all --selection_method median | ||
| uv run -m ale_bench_eval --model_config_path llm_configs/gpt-5.json --n_repeated_sampling 15 --n_self_refine 16 --num_workers 10 --n_public_cases 50 --judge_version 202510 --code_language typescript --prompt_language en --max_parallel_problems 5 --max_concurrent_llm_calls 20 --max_repeated_sampling_workers 100 --problem_ids_type all --selection_method median |
Comment on lines
+474
to
+475
| with (exp_root / "results.json").open("w") as f: | ||
| json.dump(results, f) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.