Efficient docker lifecycle

README.md

@@ -25,6 +25,7 @@ https://github.com/user-attachments/assets/50a8de5a-b519-4aef-8e54-c60ac9dcbb90
 - [Evaluation](#evaluation)
 - [Documentation](#documentation)
 - [Development and Contributing](#development-and-contributing)
+- [License](#license)
 - [Citation](#citation)
 
 ## Setup
@@ -224,6 +225,13 @@ For more details about ALE-Bench, please refer to the [docs/](./docs/) directory
 ## Development and Contributing
 Please see the [CONTRIBUTING.md](./CONTRIBUTING.md) file.
 
+## License
+
+The code in this GitHub repository is licensed under the Apache License 2.0.
+See the [LICENSE](./LICENSE) file for details.
+
+The ALE-Bench dataset hosted on [Hugging Face](https://huggingface.co/datasets/SakanaAI/ALE-Bench) is licensed under the [Creative Commons Attribution-NoDerivatives 4.0 International License (CC BY-ND 4.0)](https://creativecommons.org/licenses/by-nd/4.0/).
+
 ## Citation
 
 Please cite ALE-Bench as follows:

docs/evaluation.md

@@ -120,6 +120,7 @@ bash scripts/run_eval.sh gpt-5 --max_concurrent_llm_calls 20 --max_repeated_samp
 | `n_repeated_sampling` | int | 1 | Number of repeated sampling iterations |
 | `n_self_refine` | int | 1 | Number of self-refinement iterations including repeated sampling process (`1` means no self-refinement) |
 | `num_workers` | int | 1 | Number of parallel case evaluation workers for each problem |
+| `reuse_containers` | bool | `False` | Reuse long-lived execution and tool containers across cases instead of creating per-case containers |
 | `n_public_cases` | int | `None` | Number of cases to use for public evaluation (`None` means using ALE-Bench default: 50 for `all`, 5 for `lite`) |
 | `judge_version` | str | `202301` | Judge toolchain version (`201907`, `202301`, `202510`) |
 | `code_language` | str | `cpp20` | Target programming language (`any`, `bash`, `cpp17`, `cpp20`, `cpp23`, `csharp`, `fish`, `fortran`, `go`, `haskell`, `javascript`, `julia`, `lean`, `ocaml`, `perl`, `pypy`, `python`, `rust`, `typescript`) |
@@ -135,6 +136,10 @@ bash scripts/run_eval.sh gpt-5 --max_concurrent_llm_calls 20 --max_repeated_samp
 
 > **Note**: Ensure that `num_workers` $\times$ `max_parallel_problems` does not exceed the number of physical CPU cores available on your machine to avoid resource contention and performance degradation.
 
+> **Note**: Near time-limit boundaries, ALE-Bench results can be sensitive to measurement granularity and host load. Execution time is derived from GNU `/usr/bin/time`. Values around the limit, especially within about 0.01 seconds, may flip between AC and TLE. A run that exceeds the time limit by less than 0.01 seconds may still be reported as AC.
+
+> **Note**: `reuse_containers=True` reduces Docker create/remove overhead by keeping up to `num_workers` execution/tool containers alive and dispatching cases to whichever worker becomes free. It is opt-in because writable container-layer state such as files under `/tmp` can persist between cases assigned to the same worker.
+
 > **Note**: `max_parallel_problems` controls problem-level concurrency. `max_repeated_sampling_workers` controls only repeated-sampling LLM generation within each problem. If it is `None`, it is resolved to `n_repeated_sampling`; otherwise it is capped at `n_repeated_sampling`. `max_concurrent_llm_calls` is a global cap shared by repeated sampling and self-refinement LLM calls. If it is `None`, it is resolved to `max_parallel_problems * effective_max_repeated_sampling_workers`. Judge execution remains bounded by `num_workers` for each active problem.
 
 > **Note**: `code_language` must be supported by the selected `judge_version`.

docs/session_object.md

@@ -86,10 +86,13 @@ Evaluates the provided code against the given input string(s). This method is in
 - `time_limit (float, optional)`: Custom time limit for execution in seconds. Defaults to `None` (uses problem-specific default).
 - `memory_limit (int | str, optional)`: Custom memory limit for execution (e.g., `256_000_000` for 256MB, or "256m"). Defaults to `None` (uses problem-specific default).
 - `skip_local_visualization (bool, optional)`: If `True`, skips generating local visualizations even if available. Defaults to `False`.
+- `reuse_containers (bool, optional)`: If `True`, reuses long-lived execution and tool containers for this call. Defaults to `False`.
 
 **Returns:**
 - `Result`: A `Result` object containing the evaluation details, including scores, execution time, and memory usage for each case.
 
+`reuse_containers=True` is also available on `case_gen_eval()`, `public_eval()`, and `private_eval()`. It avoids per-case execution/tool container create/remove overhead, but writable container-layer state such as files under `/tmp` may persist between cases assigned to the same worker.
+
 ---
 ### `case_gen_eval`
 A convenience method that first generates test case(s) using specified seeds and generation arguments, and then immediately evaluates the provided code against these newly generated cases.
@@ -103,6 +106,7 @@ A convenience method that first generates test case(s) using specified seeds and
 - `memory_limit (int | str, optional)`: Custom memory limit. Defaults to `None`.
 - `gen_kwargs (dict, optional)`: Arguments for the case generator. Defaults to an empty dictionary.
 - `skip_local_visualization (bool, optional)`: If `True`, skips local visualizations. Defaults to `False`.
+- `reuse_containers (bool, optional)`: If `True`, reuses long-lived execution and tool containers for this call. Defaults to `False`.
 
 **Returns:**
 - `Result`: A `Result` object with the evaluation outcome.
@@ -127,6 +131,7 @@ Evaluates the provided code against the predefined set of public test cases for
 - `code_language (CodeLanguage | str)`: The programming language of the code.
 - `judge_version (JudgeVersion | str, optional)`: The judge version. Defaults to `None` (`202301`).
 - `skip_local_visualization (bool, optional)`: If `True`, skips local visualizations. Defaults to `True` for public evaluations.
+- `reuse_containers (bool, optional)`: If `True`, reuses long-lived execution and tool containers for this call. Defaults to `False`.
 
 **Returns:**
 - `Result`: A `Result` object detailing the performance on public test cases.
@@ -139,6 +144,7 @@ Evaluates the provided code against the predefined set of private test cases. Th
 - `code (str)`: The source code to evaluate.
 - `code_language (CodeLanguage | str)`: The programming language of the code.
 - `judge_version (JudgeVersion | str, optional)`: The judge version. Defaults to `None` (`202301`).
+- `reuse_containers (bool, optional)`: If `True`, reuses long-lived execution and tool containers for this call. Defaults to `False`.
 
 **Returns:**
 - `Result`: A `Result` object detailing the performance on private test cases.

llm_configs/claude-fable-5-high.json

@@ -0,0 +1,12 @@
+{
+    "model_name": "claude-fable-5",
+    "provider": "anthropic",
+    "settings": {
+        "max_tokens": 128000,
+        "anthropic_thinking": {
+            "type": "adaptive",
+            "display": "summarized"
+        },
+        "anthropic_effort": "high"
+    }
+}

llm_configs/glm-5.2-high.json

@@ -0,0 +1,23 @@
+{
+    "model_name": "z-ai/glm-5.2",
+    "provider": "openrouter",
+    "settings": {
+        "temperature": 1.0,
+        "top_p": 0.95,
+        "extra_body": {
+            "provider": {
+                "allow_fallbacks": false,
+                "data_collection": "deny",
+                "order": ["z-ai"],
+                "require_parameters": true
+            },
+            "reasoning": {
+                "effort": "high",
+                "enabled": true
+            },
+            "usage": {
+                "include": true
+            }
+        }
+    }
+}

llm_configs/glm-5.2-max.json

@@ -0,0 +1,23 @@
+{
+    "model_name": "z-ai/glm-5.2",
+    "provider": "openrouter",
+    "settings": {
+        "temperature": 1.0,
+        "top_p": 0.95,
+        "extra_body": {
+            "provider": {
+                "allow_fallbacks": false,
+                "data_collection": "deny",
+                "order": ["z-ai"],
+                "require_parameters": true
+            },
+            "reasoning": {
+                "effort": "xhigh",
+                "enabled": true
+            },
+            "usage": {
+                "include": true
+            }
+        }
+    }
+}

llm_configs/kimi-k2.7-code.json

@@ -0,0 +1,20 @@
+{
+    "model_name": "moonshotai/kimi-k2.7-code",
+    "provider": "openrouter",
+    "settings": {
+        "extra_body": {
+            "provider": {
+                "allow_fallbacks": false,
+                "data_collection": "deny",
+                "order": ["moonshotai/int4"],
+                "require_parameters": true
+            },
+            "reasoning": {
+                "enabled": true
+            },
+            "usage": {
+                "include": true
+            }
+        }
+    }
+}

llm_configs/ling-2.6-1t.json

@@ -0,0 +1,17 @@
+{
+    "model_name": "inclusionai/ling-2.6-1t",
+    "provider": "openrouter",
+    "settings": {
+        "extra_body": {
+            "provider": {
+                "allow_fallbacks": false,
+                "data_collection": "deny",
+                "order": ["novita"],
+                "require_parameters": true
+            },
+            "usage": {
+                "include": true
+            }
+        }
+    }
+}

llm_configs/ring-2.6-1t-high.json

@@ -0,0 +1,21 @@
+{
+    "model_name": "inclusionai/ring-2.6-1t",
+    "provider": "openrouter",
+    "settings": {
+        "extra_body": {
+            "provider": {
+                "allow_fallbacks": false,
+                "data_collection": "deny",
+                "order": ["novita"],
+                "require_parameters": true
+            },
+            "reasoning": {
+                "effort": "high",
+                "enabled": true
+            },
+            "usage": {
+                "include": true
+            }
+        }
+    }
+}

src/ale_bench/session.py

@@ -285,6 +285,7 @@ def case_eval(
         time_limit: float | None = None,
         memory_limit: int | str | None = None,
         skip_local_visualization: bool = False,
+        reuse_containers: bool = False,
     ) -> Result:
         """Evaluate the code with the given input.
 
@@ -298,6 +299,8 @@ def case_eval(
             time_limit (float, optional): The time limit in seconds. Defaults to None.
             memory_limit (int | str, optional): The memory limit in bytes. Defaults to None.
             skip_local_visualization (bool, optional): Whether to skip local visualization. Defaults to False.
+            reuse_containers (bool, optional): Whether to reuse execution and tool containers across cases.
+                Defaults to False.
 
         Returns:
             Result: The result of the evaluation.
@@ -336,6 +339,7 @@ def case_eval(
             return_details=True,
             skip_local_visualization=skip_local_visualization,
             num_workers=self.num_workers,
+            reuse_containers=reuse_containers,
         )
 
         # Postprocessing
@@ -384,6 +388,7 @@ def case_gen_eval(
         memory_limit: int | str | None = None,
         gen_kwargs: dict[str, Any] | None = None,
         skip_local_visualization: bool = False,
+        reuse_containers: bool = False,
     ) -> Result:
         """Generate a case and evaluate the code with the given input.
 
@@ -396,6 +401,8 @@ def case_gen_eval(
             memory_limit (int | str, optional): The memory limit in bytes. Defaults to None.
             gen_kwargs (dict[str, Any]): The generation arguments. Defaults to an empty dictionary.
             skip_local_visualization (bool, optional): Whether to skip local visualization. Defaults to False.
+            reuse_containers (bool, optional): Whether to reuse execution and tool containers across cases.
+                Defaults to False.
 
         Returns:
             Result: The result of the evaluation.
@@ -421,7 +428,14 @@ def case_gen_eval(
         # Generation and evaluation (postprocessing is done in each function)
         input_str = self.case_gen(seed, gen_kwargs=gen_kwargs)
         result = self.case_eval(
-            input_str, code, code_language, judge_version, time_limit, memory_limit, skip_local_visualization
+            input_str=input_str,
+            code=code,
+            code_language=code_language,
+            judge_version=judge_version,
+            time_limit=time_limit,
+            memory_limit=memory_limit,
+            skip_local_visualization=skip_local_visualization,
+            reuse_containers=reuse_containers,
         )
         if not self._check_within_resource_usage_after(AleBenchFunction.CASE_GEN_EVAL):
             # NOTE: maybe this block is not reached because we check the resource usage in each function
@@ -485,6 +499,7 @@ def public_eval(
         code_language: CodeLanguage | str,
         judge_version: JudgeVersion | str | None = None,
         skip_local_visualization: bool = True,
+        reuse_containers: bool = False,
     ) -> Result:
         """Evaluate the public score of the submission.
 
@@ -493,6 +508,8 @@ def public_eval(
             code_language (CodeLanguage | str): The code language.
             judge_version (JudgeVersion | str, optional): The judge version. Defaults to None (202301).
             skip_local_visualization (bool, optional): Whether to skip local visualization. Defaults to True.
+            reuse_containers (bool, optional): Whether to reuse execution and tool containers across cases.
+                Defaults to False.
 
         Returns:
             Result: The result of the evaluation.
@@ -527,6 +544,7 @@ def public_eval(
             return_details=True,
             skip_local_visualization=skip_local_visualization,
             num_workers=self.num_workers,
+            reuse_containers=reuse_containers,
         )
 
         # Postprocessing
@@ -564,13 +582,16 @@ def private_eval(
         code: str,
         code_language: CodeLanguage | str,
         judge_version: JudgeVersion | str | None = None,
+        reuse_containers: bool = False,
     ) -> tuple[Result, int, int]:
         """Evaluate the private score of the submission.
 
         Args:
             code (str): The code to evaluate.
             code_language (CodeLanguage | str): The code language.
             judge_version (JudgeVersion | str, optional): The judge version. Defaults to None (202301).
+            reuse_containers (bool, optional): Whether to reuse execution and tool containers across cases.
+                Defaults to False.
 
         Returns:
             Result: The result of the evaluation.
@@ -607,6 +628,7 @@ def private_eval(
             return_details=False,
             skip_local_visualization=True,
             num_workers=self.num_workers,
+            reuse_containers=reuse_containers,
         )
 
         # Postprocessing