diff --git a/README.md b/README.md index a7cafa94..5b91557f 100644 --- a/README.md +++ b/README.md @@ -25,6 +25,7 @@ https://github.com/user-attachments/assets/50a8de5a-b519-4aef-8e54-c60ac9dcbb90 - [Evaluation](#evaluation) - [Documentation](#documentation) - [Development and Contributing](#development-and-contributing) +- [License](#license) - [Citation](#citation) ## Setup @@ -224,6 +225,13 @@ For more details about ALE-Bench, please refer to the [docs/](./docs/) directory ## Development and Contributing Please see the [CONTRIBUTING.md](./CONTRIBUTING.md) file. +## License + +The code in this GitHub repository is licensed under the Apache License 2.0. +See the [LICENSE](./LICENSE) file for details. + +The ALE-Bench dataset hosted on [Hugging Face](https://huggingface.co/datasets/SakanaAI/ALE-Bench) is licensed under the [Creative Commons Attribution-NoDerivatives 4.0 International License (CC BY-ND 4.0)](https://creativecommons.org/licenses/by-nd/4.0/). + ## Citation Please cite ALE-Bench as follows: diff --git a/docs/evaluation.md b/docs/evaluation.md index 03f41690..844b88b7 100644 --- a/docs/evaluation.md +++ b/docs/evaluation.md @@ -120,6 +120,7 @@ bash scripts/run_eval.sh gpt-5 --max_concurrent_llm_calls 20 --max_repeated_samp | `n_repeated_sampling` | int | 1 | Number of repeated sampling iterations | | `n_self_refine` | int | 1 | Number of self-refinement iterations including repeated sampling process (`1` means no self-refinement) | | `num_workers` | int | 1 | Number of parallel case evaluation workers for each problem | +| `reuse_containers` | bool | `False` | Reuse long-lived execution and tool containers across cases instead of creating per-case containers | | `n_public_cases` | int | `None` | Number of cases to use for public evaluation (`None` means using ALE-Bench default: 50 for `all`, 5 for `lite`) | | `judge_version` | str | `202301` | Judge toolchain version (`201907`, `202301`, `202510`) | | `code_language` | str | `cpp20` | Target programming language (`any`, `bash`, `cpp17`, `cpp20`, `cpp23`, `csharp`, `fish`, `fortran`, `go`, `haskell`, `javascript`, `julia`, `lean`, `ocaml`, `perl`, `pypy`, `python`, `rust`, `typescript`) | @@ -135,6 +136,10 @@ bash scripts/run_eval.sh gpt-5 --max_concurrent_llm_calls 20 --max_repeated_samp > **Note**: Ensure that `num_workers` $\times$ `max_parallel_problems` does not exceed the number of physical CPU cores available on your machine to avoid resource contention and performance degradation. +> **Note**: Near time-limit boundaries, ALE-Bench results can be sensitive to measurement granularity and host load. Execution time is derived from GNU `/usr/bin/time`. Values around the limit, especially within about 0.01 seconds, may flip between AC and TLE. A run that exceeds the time limit by less than 0.01 seconds may still be reported as AC. + +> **Note**: `reuse_containers=True` reduces Docker create/remove overhead by keeping up to `num_workers` execution/tool containers alive and dispatching cases to whichever worker becomes free. It is opt-in because writable container-layer state such as files under `/tmp` can persist between cases assigned to the same worker. + > **Note**: `max_parallel_problems` controls problem-level concurrency. `max_repeated_sampling_workers` controls only repeated-sampling LLM generation within each problem. If it is `None`, it is resolved to `n_repeated_sampling`; otherwise it is capped at `n_repeated_sampling`. `max_concurrent_llm_calls` is a global cap shared by repeated sampling and self-refinement LLM calls. If it is `None`, it is resolved to `max_parallel_problems * effective_max_repeated_sampling_workers`. Judge execution remains bounded by `num_workers` for each active problem. > **Note**: `code_language` must be supported by the selected `judge_version`. diff --git a/docs/session_object.md b/docs/session_object.md index cfad6f05..6b5594ee 100644 --- a/docs/session_object.md +++ b/docs/session_object.md @@ -86,10 +86,13 @@ Evaluates the provided code against the given input string(s). This method is in - `time_limit (float, optional)`: Custom time limit for execution in seconds. Defaults to `None` (uses problem-specific default). - `memory_limit (int | str, optional)`: Custom memory limit for execution (e.g., `256_000_000` for 256MB, or "256m"). Defaults to `None` (uses problem-specific default). - `skip_local_visualization (bool, optional)`: If `True`, skips generating local visualizations even if available. Defaults to `False`. +- `reuse_containers (bool, optional)`: If `True`, reuses long-lived execution and tool containers for this call. Defaults to `False`. **Returns:** - `Result`: A `Result` object containing the evaluation details, including scores, execution time, and memory usage for each case. +`reuse_containers=True` is also available on `case_gen_eval()`, `public_eval()`, and `private_eval()`. It avoids per-case execution/tool container create/remove overhead, but writable container-layer state such as files under `/tmp` may persist between cases assigned to the same worker. + --- ### `case_gen_eval` A convenience method that first generates test case(s) using specified seeds and generation arguments, and then immediately evaluates the provided code against these newly generated cases. @@ -103,6 +106,7 @@ A convenience method that first generates test case(s) using specified seeds and - `memory_limit (int | str, optional)`: Custom memory limit. Defaults to `None`. - `gen_kwargs (dict, optional)`: Arguments for the case generator. Defaults to an empty dictionary. - `skip_local_visualization (bool, optional)`: If `True`, skips local visualizations. Defaults to `False`. +- `reuse_containers (bool, optional)`: If `True`, reuses long-lived execution and tool containers for this call. Defaults to `False`. **Returns:** - `Result`: A `Result` object with the evaluation outcome. @@ -127,6 +131,7 @@ Evaluates the provided code against the predefined set of public test cases for - `code_language (CodeLanguage | str)`: The programming language of the code. - `judge_version (JudgeVersion | str, optional)`: The judge version. Defaults to `None` (`202301`). - `skip_local_visualization (bool, optional)`: If `True`, skips local visualizations. Defaults to `True` for public evaluations. +- `reuse_containers (bool, optional)`: If `True`, reuses long-lived execution and tool containers for this call. Defaults to `False`. **Returns:** - `Result`: A `Result` object detailing the performance on public test cases. @@ -139,6 +144,7 @@ Evaluates the provided code against the predefined set of private test cases. Th - `code (str)`: The source code to evaluate. - `code_language (CodeLanguage | str)`: The programming language of the code. - `judge_version (JudgeVersion | str, optional)`: The judge version. Defaults to `None` (`202301`). +- `reuse_containers (bool, optional)`: If `True`, reuses long-lived execution and tool containers for this call. Defaults to `False`. **Returns:** - `Result`: A `Result` object detailing the performance on private test cases. diff --git a/llm_configs/claude-fable-5-high.json b/llm_configs/claude-fable-5-high.json new file mode 100644 index 00000000..44658cd6 --- /dev/null +++ b/llm_configs/claude-fable-5-high.json @@ -0,0 +1,12 @@ +{ + "model_name": "claude-fable-5", + "provider": "anthropic", + "settings": { + "max_tokens": 128000, + "anthropic_thinking": { + "type": "adaptive", + "display": "summarized" + }, + "anthropic_effort": "high" + } +} diff --git a/llm_configs/glm-5.2-high.json b/llm_configs/glm-5.2-high.json new file mode 100644 index 00000000..35cb9c23 --- /dev/null +++ b/llm_configs/glm-5.2-high.json @@ -0,0 +1,23 @@ +{ + "model_name": "z-ai/glm-5.2", + "provider": "openrouter", + "settings": { + "temperature": 1.0, + "top_p": 0.95, + "extra_body": { + "provider": { + "allow_fallbacks": false, + "data_collection": "deny", + "order": ["z-ai"], + "require_parameters": true + }, + "reasoning": { + "effort": "high", + "enabled": true + }, + "usage": { + "include": true + } + } + } +} diff --git a/llm_configs/glm-5.2-max.json b/llm_configs/glm-5.2-max.json new file mode 100644 index 00000000..bacc5c30 --- /dev/null +++ b/llm_configs/glm-5.2-max.json @@ -0,0 +1,23 @@ +{ + "model_name": "z-ai/glm-5.2", + "provider": "openrouter", + "settings": { + "temperature": 1.0, + "top_p": 0.95, + "extra_body": { + "provider": { + "allow_fallbacks": false, + "data_collection": "deny", + "order": ["z-ai"], + "require_parameters": true + }, + "reasoning": { + "effort": "xhigh", + "enabled": true + }, + "usage": { + "include": true + } + } + } +} diff --git a/llm_configs/kimi-k2.7-code.json b/llm_configs/kimi-k2.7-code.json new file mode 100644 index 00000000..84628c3d --- /dev/null +++ b/llm_configs/kimi-k2.7-code.json @@ -0,0 +1,20 @@ +{ + "model_name": "moonshotai/kimi-k2.7-code", + "provider": "openrouter", + "settings": { + "extra_body": { + "provider": { + "allow_fallbacks": false, + "data_collection": "deny", + "order": ["moonshotai/int4"], + "require_parameters": true + }, + "reasoning": { + "enabled": true + }, + "usage": { + "include": true + } + } + } +} diff --git a/llm_configs/ling-2.6-1t.json b/llm_configs/ling-2.6-1t.json new file mode 100644 index 00000000..4750d9e6 --- /dev/null +++ b/llm_configs/ling-2.6-1t.json @@ -0,0 +1,17 @@ +{ + "model_name": "inclusionai/ling-2.6-1t", + "provider": "openrouter", + "settings": { + "extra_body": { + "provider": { + "allow_fallbacks": false, + "data_collection": "deny", + "order": ["novita"], + "require_parameters": true + }, + "usage": { + "include": true + } + } + } +} diff --git a/llm_configs/ring-2.6-1t-high.json b/llm_configs/ring-2.6-1t-high.json new file mode 100644 index 00000000..fb86c20d --- /dev/null +++ b/llm_configs/ring-2.6-1t-high.json @@ -0,0 +1,21 @@ +{ + "model_name": "inclusionai/ring-2.6-1t", + "provider": "openrouter", + "settings": { + "extra_body": { + "provider": { + "allow_fallbacks": false, + "data_collection": "deny", + "order": ["novita"], + "require_parameters": true + }, + "reasoning": { + "effort": "high", + "enabled": true + }, + "usage": { + "include": true + } + } + } +} diff --git a/src/ale_bench/session.py b/src/ale_bench/session.py index 998c9632..feea828f 100644 --- a/src/ale_bench/session.py +++ b/src/ale_bench/session.py @@ -285,6 +285,7 @@ def case_eval( time_limit: float | None = None, memory_limit: int | str | None = None, skip_local_visualization: bool = False, + reuse_containers: bool = False, ) -> Result: """Evaluate the code with the given input. @@ -298,6 +299,8 @@ def case_eval( time_limit (float, optional): The time limit in seconds. Defaults to None. memory_limit (int | str, optional): The memory limit in bytes. Defaults to None. skip_local_visualization (bool, optional): Whether to skip local visualization. Defaults to False. + reuse_containers (bool, optional): Whether to reuse execution and tool containers across cases. + Defaults to False. Returns: Result: The result of the evaluation. @@ -336,6 +339,7 @@ def case_eval( return_details=True, skip_local_visualization=skip_local_visualization, num_workers=self.num_workers, + reuse_containers=reuse_containers, ) # Postprocessing @@ -384,6 +388,7 @@ def case_gen_eval( memory_limit: int | str | None = None, gen_kwargs: dict[str, Any] | None = None, skip_local_visualization: bool = False, + reuse_containers: bool = False, ) -> Result: """Generate a case and evaluate the code with the given input. @@ -396,6 +401,8 @@ def case_gen_eval( memory_limit (int | str, optional): The memory limit in bytes. Defaults to None. gen_kwargs (dict[str, Any]): The generation arguments. Defaults to an empty dictionary. skip_local_visualization (bool, optional): Whether to skip local visualization. Defaults to False. + reuse_containers (bool, optional): Whether to reuse execution and tool containers across cases. + Defaults to False. Returns: Result: The result of the evaluation. @@ -421,7 +428,14 @@ def case_gen_eval( # Generation and evaluation (postprocessing is done in each function) input_str = self.case_gen(seed, gen_kwargs=gen_kwargs) result = self.case_eval( - input_str, code, code_language, judge_version, time_limit, memory_limit, skip_local_visualization + input_str=input_str, + code=code, + code_language=code_language, + judge_version=judge_version, + time_limit=time_limit, + memory_limit=memory_limit, + skip_local_visualization=skip_local_visualization, + reuse_containers=reuse_containers, ) if not self._check_within_resource_usage_after(AleBenchFunction.CASE_GEN_EVAL): # NOTE: maybe this block is not reached because we check the resource usage in each function @@ -485,6 +499,7 @@ def public_eval( code_language: CodeLanguage | str, judge_version: JudgeVersion | str | None = None, skip_local_visualization: bool = True, + reuse_containers: bool = False, ) -> Result: """Evaluate the public score of the submission. @@ -493,6 +508,8 @@ def public_eval( code_language (CodeLanguage | str): The code language. judge_version (JudgeVersion | str, optional): The judge version. Defaults to None (202301). skip_local_visualization (bool, optional): Whether to skip local visualization. Defaults to True. + reuse_containers (bool, optional): Whether to reuse execution and tool containers across cases. + Defaults to False. Returns: Result: The result of the evaluation. @@ -527,6 +544,7 @@ def public_eval( return_details=True, skip_local_visualization=skip_local_visualization, num_workers=self.num_workers, + reuse_containers=reuse_containers, ) # Postprocessing @@ -564,6 +582,7 @@ def private_eval( code: str, code_language: CodeLanguage | str, judge_version: JudgeVersion | str | None = None, + reuse_containers: bool = False, ) -> tuple[Result, int, int]: """Evaluate the private score of the submission. @@ -571,6 +590,8 @@ def private_eval( code (str): The code to evaluate. code_language (CodeLanguage | str): The code language. judge_version (JudgeVersion | str, optional): The judge version. Defaults to None (202301). + reuse_containers (bool, optional): Whether to reuse execution and tool containers across cases. + Defaults to False. Returns: Result: The result of the evaluation. @@ -607,6 +628,7 @@ def private_eval( return_details=False, skip_local_visualization=True, num_workers=self.num_workers, + reuse_containers=reuse_containers, ) # Postprocessing diff --git a/src/ale_bench/tool_wrappers/case_runner.py b/src/ale_bench/tool_wrappers/case_runner.py index 592ba6c3..cef49adb 100644 --- a/src/ale_bench/tool_wrappers/case_runner.py +++ b/src/ale_bench/tool_wrappers/case_runner.py @@ -4,9 +4,11 @@ import math import os import re +import shlex import tempfile import time from concurrent.futures import ThreadPoolExecutor, as_completed +from contextlib import nullcontext from pathlib import Path from pydantic import BaseModel, ConfigDict, Field @@ -24,8 +26,14 @@ ) from ale_bench.data import ProblemType from ale_bench.result import CaseResult, JudgeResult, Profiles +from ale_bench.tool_wrappers.reusable_container_pool import ( + ReusableSubmissionContainerPool, + ReusableToolContainerPool, +) from ale_bench.utils import docker_client, read_svg +TIMEOUT_EXIT_CODE = 124 + class HostPathsCompile(BaseModel): """Paths on the host for the compilation step of the submission.""" @@ -189,24 +197,34 @@ def get_batch_run_volumes(host_paths: HostPathsBatchRun, temp_dir: Path) -> dict } -def build_batch_run_command(code_language: CodeLanguage, judge_version: JudgeVersion, time_limit: float) -> str: +def build_batch_run_command( + code_language: CodeLanguage, + judge_version: JudgeVersion, + time_limit: float, + input_file: str = ale_bench.constants.INPUT_FILE, + output_file: str = ale_bench.constants.OUTPUT_FILE, + profiles_file: str = ale_bench.constants.PROFILES_FILE, +) -> str: """Build the run command for the given code language and judge version. Args: code_language (CodeLanguage): The code language. judge_version (JudgeVersion): The judge version. time_limit (float): The time limit in seconds. + input_file (str): The input file path in the container. + output_file (str): The output file path in the container. + profiles_file (str): The profiles file path in the container. Returns: str: The run command. """ run_command = get_run_command(code_language, judge_version) - run_command += f" < {ale_bench.constants.INPUT_FILE} > {ale_bench.constants.OUTPUT_FILE}" + run_command += f" < {input_file} > {output_file}" run_command = ( "/usr/bin/time " f'-f "{ale_bench.constants.TIME_OUTPUT_FORMAT}" ' - f"-o {ale_bench.constants.PROFILES_FILE} {run_command}" + f"-o {profiles_file} {run_command}" ) # NOTE: We use the GNU Time to measure the resource usage # NOTE: the profiles by GNU Time update every 1 sec (from observations while debugging) time_limit_ceil = math.ceil(time_limit + 0.1) @@ -266,14 +284,17 @@ def get_batch_judge_volumes(host_paths: HostPathsBatchJudge, tool_dir: Path) -> } -def build_batch_judge_command() -> str: +def build_batch_judge_command( + input_file: str = ale_bench.constants.INPUT_FILE, + output_file: str = ale_bench.constants.OUTPUT_FILE, +) -> str: """Build the judging command. Returns: str: The judging command. """ - return f"{ale_bench.constants.TESTER_BIN} {ale_bench.constants.INPUT_FILE} {ale_bench.constants.OUTPUT_FILE}" + return f"{ale_bench.constants.TESTER_BIN} {input_file} {output_file}" class HostPathsReactiveJudge(BaseModel): @@ -358,24 +379,34 @@ def get_reactive_judge_volumes( } -def build_reactive_judge_command(code_language: CodeLanguage, judge_version: JudgeVersion, time_limit: float) -> str: +def build_reactive_judge_command( + code_language: CodeLanguage, + judge_version: JudgeVersion, + time_limit: float, + input_file: str = ale_bench.constants.INPUT_FILE, + output_file: str = ale_bench.constants.OUTPUT_FILE, + profiles_file: str = ale_bench.constants.PROFILES_FILE, +) -> str: """Build the run command for the given code language and judge version. Args: code_language (CodeLanguage): The code language. judge_version (JudgeVersion): The judge version. time_limit (float): The time limit in seconds. + input_file (str): The input file path in the container. + output_file (str): The output file path in the container. + profiles_file (str): The profiles file path in the container. Returns: str: The run command. """ run_command = get_run_command(code_language, judge_version) - run_command += f" < {ale_bench.constants.INPUT_FILE} > {ale_bench.constants.OUTPUT_FILE}" + run_command += f" < {input_file} > {output_file}" run_command = ( f"{ale_bench.constants.TESTER_BIN} /usr/bin/time " f'-f "{ale_bench.constants.TIME_OUTPUT_FORMAT}" ' - f"-o {ale_bench.constants.PROFILES_FILE} {run_command}" + f"-o {profiles_file} {run_command}" ) # NOTE: We use the GNU Time to measure the resource usage # NOTE: the profiles by GNU Time update every 1 sec (from observations while debugging) time_limit_ceil = math.ceil(time_limit + 0.1) @@ -459,14 +490,17 @@ def get_vis_volumes(host_paths: HostPathsVis, tool_dir: Path) -> dict[str, dict[ } -def build_vis_command() -> str: +def build_vis_command( + input_file: str = ale_bench.constants.INPUT_FILE, + output_file: str = ale_bench.constants.OUTPUT_FILE, +) -> str: """Build the visualization command. Returns: str: The visualization command. """ - return f"{ale_bench.constants.VIS_BIN} {ale_bench.constants.INPUT_FILE} {ale_bench.constants.OUTPUT_FILE}" + return f"{ale_bench.constants.VIS_BIN} {input_file} {output_file}" def run_compile_container( @@ -636,6 +670,39 @@ def run_batch_run_container( return execution_time_host, stderr # Run succeeded, return the execution time and stderr +def run_batch_run_reusable_container( + reusable_submission_container_pool: ReusableSubmissionContainerPool, + time_limit: float, + run_command: str, + input_str: str | None, +) -> CaseResult | tuple[float, str]: + """Run the batch submission command in a reusable Docker container.""" + execution_time_host, exit_code, stderr = reusable_submission_container_pool.run(run_command) + if exit_code != 0: + if execution_time_host > time_limit: # Killed by `timeout` command + return CaseResult( + input_str=input_str, + output_str=None, + error_str=stderr if input_str is not None else None, + judge_result=JudgeResult.TIME_LIMIT_EXCEEDED, + message="Time limit exceeded.", + absolute_score=ale_bench.constants.REJECTED_ABSOLUTE_SCORE, + execution_time=min(execution_time_host, time_limit + 0.1), # NOTE: slight longer than time limit + memory_usage=0, + ) + return CaseResult( + input_str=input_str, + output_str=None, + error_str=stderr if input_str is not None else None, + judge_result=JudgeResult.RUNTIME_ERROR, + message="Runtime error.", + absolute_score=ale_bench.constants.REJECTED_ABSOLUTE_SCORE, + execution_time=execution_time_host, + memory_usage=0, + ) + return execution_time_host, stderr + + def run_batch_judge_container( judge_volumes: dict[str, dict[str, str]], judge_command: str, @@ -720,6 +787,55 @@ def run_batch_judge_container( return int(score_match.group(1)) +def run_batch_judge_reusable_container( + reusable_tool_container_pool: ReusableToolContainerPool, + judge_command: str, + execution_time_host: float, + input_str: str | None, + output_str: str | None, + error_str: str | None, +) -> CaseResult | int: + """Run the batch judge command in a reusable Docker tool container.""" + _execution_time_host_judge, exit_code, stderr = reusable_tool_container_pool.run(judge_command) + if exit_code != 0: + return CaseResult( + input_str=input_str, + output_str=output_str, + error_str=error_str, + judge_result=JudgeResult.WRONG_ANSWER, + message=f"Wrong answer.\nStandard error:\n{stderr}", + absolute_score=ale_bench.constants.REJECTED_ABSOLUTE_SCORE, + execution_time=execution_time_host, + memory_usage=0, + ) + if "wrong answer: " in stderr: + error_message = stderr.split("wrong answer: ")[1] + return CaseResult( + input_str=input_str, + output_str=output_str, + error_str=error_str, + judge_result=JudgeResult.WRONG_ANSWER, + message=f"Wrong answer.\n{error_message}", + absolute_score=ale_bench.constants.REJECTED_ABSOLUTE_SCORE, + execution_time=execution_time_host, + memory_usage=0, + ) + stderr_last_line = stderr.splitlines()[-1] + score_match = re.match(r"Score = (\d+)", stderr_last_line) + if score_match is None: + return CaseResult( + input_str=input_str, + output_str=output_str, + error_str=error_str, + judge_result=JudgeResult.WRONG_ANSWER, + message=f"Wrong answer.\nStandard error:\n{stderr}", + absolute_score=ale_bench.constants.REJECTED_ABSOLUTE_SCORE, + execution_time=execution_time_host, + memory_usage=0, + ) + return int(score_match.group(1)) + + def run_reactive_judge_container( code_language: CodeLanguage, judge_version: JudgeVersion, @@ -810,6 +926,54 @@ def run_reactive_judge_container( return (execution_time_host, score, stderr) # Run succeeded, return the execution time +def run_reactive_judge_reusable_container( + reusable_submission_container_pool: ReusableSubmissionContainerPool, + time_limit: float, + judge_command: str, + input_str: str | None, + output_file_path: Path | None, +) -> CaseResult | tuple[float, int, str]: + """Run the reactive judge command in a reusable Docker container.""" + execution_time_host, exit_code, stderr = reusable_submission_container_pool.run(judge_command) + if exit_code != 0 or stderr == "": + if execution_time_host > time_limit: # Killed by `timeout` command + return CaseResult( + input_str=input_str, + output_str=None, + error_str=stderr if input_str is not None else None, + judge_result=JudgeResult.TIME_LIMIT_EXCEEDED, + message="Time limit exceeded.", + absolute_score=ale_bench.constants.REJECTED_ABSOLUTE_SCORE, + execution_time=min(execution_time_host, time_limit + 0.1), # NOTE: slight longer than time limit + memory_usage=0, + ) + return CaseResult( + input_str=input_str, + output_str=None, + error_str=stderr if input_str is not None else None, + judge_result=JudgeResult.RUNTIME_ERROR, + message="Runtime error.", + absolute_score=ale_bench.constants.REJECTED_ABSOLUTE_SCORE, + execution_time=execution_time_host, + memory_usage=0, + ) + stderr_last_line = stderr.splitlines()[-1] + score_match = re.match(r"Score = (\d+)", stderr_last_line) + if score_match is None: + return CaseResult( + input_str=input_str, + output_str=output_file_path.read_text() if output_file_path else None, + error_str=stderr if input_str is not None else None, + judge_result=JudgeResult.WRONG_ANSWER, + message="Wrong answer.", # NOTE: exclude stderr because we don't want to be exploited by the user + absolute_score=ale_bench.constants.REJECTED_ABSOLUTE_SCORE, + execution_time=execution_time_host, + memory_usage=0, + ) + score = int(score_match.group(1)) + return (execution_time_host, score, stderr) + + def run_vis_container(vis_command: str, vis_volumes: dict[str, dict[str, str]]) -> None: """Run the visualization command in a Docker container. @@ -848,6 +1012,28 @@ def run_vis_container(vis_command: str, vis_volumes: dict[str, dict[str, str]]) raise RuntimeError(msg) +def run_vis_reusable_container( + reusable_tool_container_pool: ReusableToolContainerPool, + vis_command: str, + local_visualization_file: str, + generated_file_path: str, +) -> None: + """Run the visualization command in a reusable Docker tool container.""" + inner_command = ( + f"rm -f {shlex.quote(generated_file_path)}; " + f"{vis_command}; " + f"cp {shlex.quote(generated_file_path)} {shlex.quote(local_visualization_file)}" + ) + timed_command = f"timeout {ale_bench.constants.VISUALIZE_TIMEOUT} bash -c {shlex.quote(inner_command)}" + _execution_time_host, exit_code, _stderr = reusable_tool_container_pool.run(timed_command) + if exit_code == TIMEOUT_EXIT_CODE: + msg = "Timeout while running the visualization command. Something went wrong." + raise RuntimeError(msg) + if exit_code != 0: + msg = "Failed to run the visualization command. Something went wrong." + raise RuntimeError(msg) + + def parse_profiles( time_limit: float, memory_limit: int, @@ -1003,19 +1189,42 @@ def case_iter_func( batch_judge_command: str, reactive_judge_command: str, vis_command: str, + reusable_submission_container_pool: ReusableSubmissionContainerPool | None = None, + reusable_tool_container_pool: ReusableToolContainerPool | None = None, ) -> CaseResult: """Run a single case end-to-end and return its judge result.""" result_input_str = input_str if return_details else None host_paths_judge: HostPathsBatchJudge | HostPathsReactiveJudge execution_time_host = -1.0 + case_temp_dir = ( + reusable_submission_container_pool.scratch_dir if reusable_submission_container_pool is not None else temp_dir + ) if problem_type == ProblemType.BATCH: # Run the submission code and generate the output file - host_paths_run = setup_paths_batch_run(host_paths_compile, temp_dir, input_str, f"{problem_id}_{case_idx:06d}_") - run_volumes = get_batch_run_volumes(host_paths_run, temp_dir) - run_result = run_batch_run_container( - code_language, judge_version, time_limit, run_volumes, batch_run_command, result_input_str + host_paths_run = setup_paths_batch_run( + host_paths_compile, case_temp_dir, input_str, f"{problem_id}_{case_idx:06d}_" ) + if reusable_submission_container_pool is None: + run_volumes = get_batch_run_volumes(host_paths_run, temp_dir) + run_result = run_batch_run_container( + code_language, judge_version, time_limit, run_volumes, batch_run_command, result_input_str + ) + else: + reusable_batch_run_command = build_batch_run_command( + code_language, + judge_version, + time_limit, + input_file=reusable_submission_container_pool.container_path(host_paths_run.input_file), + output_file=reusable_submission_container_pool.container_path(host_paths_run.output_file), + profiles_file=reusable_submission_container_pool.container_path(host_paths_run.profiles_file), + ) + run_result = run_batch_run_reusable_container( + reusable_submission_container_pool, + time_limit, + reusable_batch_run_command, + result_input_str, + ) if isinstance(run_result, CaseResult): return run_result if not isinstance(run_result, tuple): @@ -1043,15 +1252,29 @@ def case_iter_func( execution_time, memory_usage = profiles_result # Calculate score by the input and output files host_paths_judge = setup_paths_batch_judge(host_paths_run) - judge_volumes = get_batch_judge_volumes(host_paths_judge, tool_dir) - batch_judge_result = run_batch_judge_container( - judge_volumes, - batch_judge_command, - execution_time_host, - result_input_str, - result_output_str, - result_error_str, - ) + if reusable_tool_container_pool is None: + judge_volumes = get_batch_judge_volumes(host_paths_judge, tool_dir) + batch_judge_result = run_batch_judge_container( + judge_volumes, + batch_judge_command, + execution_time_host, + result_input_str, + result_output_str, + result_error_str, + ) + else: + reusable_batch_judge_command = build_batch_judge_command( + input_file=reusable_tool_container_pool.container_path(host_paths_judge.input_file), + output_file=reusable_tool_container_pool.container_path(host_paths_judge.output_file), + ) + batch_judge_result = run_batch_judge_reusable_container( + reusable_tool_container_pool, + reusable_batch_judge_command, + execution_time_host, + result_input_str, + result_output_str, + result_error_str, + ) if isinstance(batch_judge_result, CaseResult): return batch_judge_result if not isinstance(batch_judge_result, int): @@ -1061,20 +1284,37 @@ def case_iter_func( elif problem_type == ProblemType.REACTIVE: host_paths_judge = setup_paths_reactive_judge( host_paths_compile, - temp_dir, + case_temp_dir, input_str, f"{problem_id}_{case_idx:06d}_", ) - judge_volumes = get_reactive_judge_volumes(host_paths_judge, temp_dir, tool_dir) - reactive_judge_result = run_reactive_judge_container( - code_language, - judge_version, - time_limit, - judge_volumes, - reactive_judge_command, - result_input_str, - host_paths_judge.output_file if return_details else None, - ) + if reusable_submission_container_pool is None: + judge_volumes = get_reactive_judge_volumes(host_paths_judge, temp_dir, tool_dir) + reactive_judge_result = run_reactive_judge_container( + code_language, + judge_version, + time_limit, + judge_volumes, + reactive_judge_command, + result_input_str, + host_paths_judge.output_file if return_details else None, + ) + else: + reusable_reactive_judge_command = build_reactive_judge_command( + code_language, + judge_version, + time_limit, + input_file=reusable_submission_container_pool.container_path(host_paths_judge.input_file), + output_file=reusable_submission_container_pool.container_path(host_paths_judge.output_file), + profiles_file=reusable_submission_container_pool.container_path(host_paths_judge.profiles_file), + ) + reactive_judge_result = run_reactive_judge_reusable_container( + reusable_submission_container_pool, + time_limit, + reusable_reactive_judge_command, + result_input_str, + host_paths_judge.output_file if return_details else None, + ) wo_profile_result = None if isinstance(reactive_judge_result, CaseResult): wo_profile_result = reactive_judge_result @@ -1124,9 +1364,27 @@ def case_iter_func( local_visualization = None if not skip_local_visualization and problem_id not in ale_bench.constants.NO_LOCAL_VIS: # Run the local visualization command in the Docker container - host_paths_vis = setup_paths_vis(host_paths_judge, temp_dir, problem_id, f"{problem_id}_{case_idx:06d}_") - vis_volumes = get_vis_volumes(host_paths_vis, tool_dir) - run_vis_container(vis_command, vis_volumes) + vis_temp_dir = case_temp_dir if reusable_tool_container_pool is not None else temp_dir + host_paths_vis = setup_paths_vis(host_paths_judge, vis_temp_dir, problem_id, f"{problem_id}_{case_idx:06d}_") + if reusable_tool_container_pool is None: + vis_volumes = get_vis_volumes(host_paths_vis, tool_dir) + run_vis_container(vis_command, vis_volumes) + else: + generated_file_path = ( + ale_bench.constants.LOCAL_VIS_SVG + if host_paths_vis.local_visualization_file.suffix == ".svg" + else ale_bench.constants.LOCAL_VIS_HTML + ) + reusable_vis_command = build_vis_command( + input_file=reusable_tool_container_pool.container_path(host_paths_vis.input_file), + output_file=reusable_tool_container_pool.container_path(host_paths_vis.output_file), + ) + run_vis_reusable_container( + reusable_tool_container_pool, + reusable_vis_command, + reusable_tool_container_pool.container_path(host_paths_vis.local_visualization_file), + generated_file_path, + ) # Read the local visualization SVG or HTML svg_text = host_paths_vis.local_visualization_file.read_text() svg_text = svg_text.replace("\n", "").removeprefix("").removesuffix("") @@ -1161,6 +1419,7 @@ def run_cases( return_details: bool, skip_local_visualization: bool, num_workers: int, + reuse_containers: bool = False, ) -> list[CaseResult]: """Run the cases for the given inputs and code. @@ -1177,6 +1436,7 @@ def run_cases( return_details (bool): Whether to return detailed results (input_str, output_str, error_str). skip_local_visualization (bool): Whether to skip local visualization. num_workers (int): The number of workers for running cases. + reuse_containers (bool): Whether to reuse long-lived execution containers. Returns: list[CaseResult]: The list of case results. @@ -1210,49 +1470,41 @@ def run_cases( # Run the code and calculate the score in the Docker container case_results: list[CaseResult] = [] - if len(inputs) == 1 or num_workers == 1: - for case_idx, input_str in enumerate(inputs): - case_result = case_iter_func( - problem_id, - time_limit, - memory_limit, - problem_type, - case_idx, - input_str, - code_language, - judge_version, - temp_dir, - tool_dir, - return_details, - skip_local_visualization, - host_paths_compile, - batch_run_command, - batch_judge_command, - reactive_judge_command, - vis_command, + reusable_scratch_dir = temp_dir / "reusable_case_files" + use_reusable_containers = reuse_containers and bool(inputs) + if use_reusable_containers: + reusable_scratch_dir.mkdir() + reusable_submission_pool_context = ReusableSubmissionContainerPool( + code_language=code_language, + judge_version=judge_version, + temp_dir=temp_dir, + scratch_dir=reusable_scratch_dir, + tool_dir=tool_dir, + problem_type=problem_type, + num_workers=max(1, min(num_workers, len(inputs))), + ) + use_reusable_tool_pool = problem_type == ProblemType.BATCH or ( + not skip_local_visualization and problem_id not in ale_bench.constants.NO_LOCAL_VIS + ) + if use_reusable_tool_pool: + reusable_tool_pool_context = ReusableToolContainerPool( + scratch_dir=reusable_scratch_dir, + tool_dir=tool_dir, + num_workers=max(1, min(num_workers, len(inputs))), ) - # Add the result - case_results.append(case_result) + else: + reusable_tool_pool_context = nullcontext(None) else: - case_results = [ - CaseResult( - input_str=input_str if return_details else None, - output_str=None, - error_str=None, - judge_result=JudgeResult.INTERNAL_ERROR, - message="Internal Error: Unexpected error occurred.", - absolute_score=ale_bench.constants.REJECTED_ABSOLUTE_SCORE, - execution_time=0.0, - memory_usage=0, - ) - for input_str in inputs - ] - # Use ThreadPoolExecutor to run the cases in parallel - with ThreadPoolExecutor(max_workers=num_workers) as executor: - future_to_case_idx = {} + reusable_submission_pool_context = nullcontext(None) + reusable_tool_pool_context = nullcontext(None) + + with ( + reusable_submission_pool_context as reusable_submission_container_pool, + reusable_tool_pool_context as reusable_tool_container_pool, + ): + if len(inputs) == 1 or num_workers == 1: for case_idx, input_str in enumerate(inputs): - future = executor.submit( - case_iter_func, + case_result = case_iter_func( problem_id, time_limit, memory_limit, @@ -1270,23 +1522,67 @@ def run_cases( batch_judge_command, reactive_judge_command, vis_command, + reusable_submission_container_pool, + reusable_tool_container_pool, + ) + # Add the result + case_results.append(case_result) + else: + case_results = [ + CaseResult( + input_str=input_str if return_details else None, + output_str=None, + error_str=None, + judge_result=JudgeResult.INTERNAL_ERROR, + message="Internal Error: Unexpected error occurred.", + absolute_score=ale_bench.constants.REJECTED_ABSOLUTE_SCORE, + execution_time=0.0, + memory_usage=0, ) - future_to_case_idx[future] = case_idx - for future in as_completed(future_to_case_idx): - case_idx = future_to_case_idx[future] - try: - case_result = future.result() - except Exception as e: - case_result = CaseResult( - input_str=inputs[case_idx] if return_details else None, - output_str=None, - error_str=None, - judge_result=JudgeResult.INTERNAL_ERROR, - message=f"Internal Error: {e}", - absolute_score=ale_bench.constants.REJECTED_ABSOLUTE_SCORE, - execution_time=0.0, - memory_usage=0, + for input_str in inputs + ] + # Use ThreadPoolExecutor to run the cases in parallel + with ThreadPoolExecutor(max_workers=num_workers) as executor: + future_to_case_idx = {} + for case_idx, input_str in enumerate(inputs): + future = executor.submit( + case_iter_func, + problem_id, + time_limit, + memory_limit, + problem_type, + case_idx, + input_str, + code_language, + judge_version, + temp_dir, + tool_dir, + return_details, + skip_local_visualization, + host_paths_compile, + batch_run_command, + batch_judge_command, + reactive_judge_command, + vis_command, + reusable_submission_container_pool, + reusable_tool_container_pool, ) - case_results[case_idx] = case_result + future_to_case_idx[future] = case_idx + for future in as_completed(future_to_case_idx): + case_idx = future_to_case_idx[future] + try: + case_result = future.result() + except Exception as e: + case_result = CaseResult( + input_str=inputs[case_idx] if return_details else None, + output_str=None, + error_str=None, + judge_result=JudgeResult.INTERNAL_ERROR, + message=f"Internal Error: {e}", + absolute_score=ale_bench.constants.REJECTED_ABSOLUTE_SCORE, + execution_time=0.0, + memory_usage=0, + ) + case_results[case_idx] = case_result return case_results diff --git a/src/ale_bench/tool_wrappers/reusable_container_pool.py b/src/ale_bench/tool_wrappers/reusable_container_pool.py new file mode 100644 index 00000000..20eb2392 --- /dev/null +++ b/src/ale_bench/tool_wrappers/reusable_container_pool.py @@ -0,0 +1,309 @@ +"""Reusable Docker container pools for case execution.""" + +from __future__ import annotations + +import os +import time +from contextlib import suppress +from queue import Queue +from typing import TYPE_CHECKING + +import ale_bench.constants +from ale_bench.code_language import CodeLanguage, JudgeVersion, get_docker_image_name +from ale_bench.data import ProblemType +from ale_bench.utils import docker_client + +if TYPE_CHECKING: + from pathlib import Path + from types import TracebackType + + from docker import DockerClient + from docker.models.containers import Container + +REUSABLE_SUBMISSION_TMP_DIR = f"{ale_bench.constants.TMP_DIR}/ale-bench-run" +REUSABLE_TOOL_TMP_DIR = f"{ale_bench.constants.TMP_DIR}/ale-bench-tool" + + +def get_reusable_tool_volumes( + scratch_dir: Path, + tool_dir: Path, +) -> dict[str, dict[str, str]]: + """Get volumes for a reusable judge/visualization tool container.""" + return { + str(scratch_dir): {"bind": REUSABLE_TOOL_TMP_DIR, "mode": "rw"}, + str(tool_dir / "tools" / "target" / "release" / "tester"): { + "bind": ale_bench.constants.TESTER_BIN, + "mode": "ro", + }, + str(tool_dir / "tools" / "target" / "release" / "vis"): { + "bind": ale_bench.constants.VIS_BIN, + "mode": "ro", + }, + } + + +def get_reusable_submission_volumes( + temp_dir: Path, + scratch_dir: Path, + tool_dir: Path, + problem_type: ProblemType, +) -> dict[str, dict[str, str]]: + """Get volumes for a reusable submission container. + + The compiled submission directory is mounted read-only at WORK_DIR. Per-case input, + output, and profile files live in a separate read-write scratch mount. + """ + volumes = { + str(temp_dir): {"bind": ale_bench.constants.WORK_DIR, "mode": "ro"}, + str(scratch_dir): {"bind": REUSABLE_SUBMISSION_TMP_DIR, "mode": "rw"}, + } + if problem_type == ProblemType.REACTIVE: + volumes[str(tool_dir / "tools" / "target" / "release" / "tester")] = { + "bind": ale_bench.constants.TESTER_BIN, + "mode": "ro", + } + return volumes + + +class ReusableSubmissionContainerPool: + """A fixed-size pool of long-lived submission execution containers.""" + + def __init__( + self, + code_language: CodeLanguage, + judge_version: JudgeVersion, + temp_dir: Path, + scratch_dir: Path, + tool_dir: Path, + problem_type: ProblemType, + num_workers: int, + ) -> None: + """Initialize the reusable container pool configuration.""" + self.code_language = code_language + self.judge_version = judge_version + self.temp_dir = temp_dir + self.scratch_dir = scratch_dir + self.tool_dir = tool_dir + self.problem_type = problem_type + self.num_workers = num_workers + self._client_context = docker_client() + self._client: DockerClient | None = None + self._containers: list[Container] = [] + self._available_containers: Queue[Container] = Queue() + + def __enter__(self) -> ReusableSubmissionContainerPool: # noqa: PYI034 + """Create and start all worker containers.""" + self._client = self._client_context.__enter__() + for _ in range(self.num_workers): + self._available_containers.put(self._create_container()) + return self + + def __exit__( + self, + exc_type: type[BaseException] | None, + exc_value: BaseException | None, + traceback: TracebackType | None, + ) -> None: + """Remove all worker containers and close the Docker client.""" + for container in self._containers: + self._remove_container(container) + self._containers.clear() + self._client_context.__exit__(exc_type, exc_value, traceback) + self._client = None + + def container_path(self, host_path: Path) -> str: + """Convert a scratch host path to its reusable-container path.""" + return f"{REUSABLE_SUBMISSION_TMP_DIR}/{host_path.relative_to(self.scratch_dir)}" + + def run(self, command: str) -> tuple[float, int, str]: + """Run one command via docker exec in an available reusable container.""" + container = self._available_containers.get() + should_replace = False + try: + start_at = time.perf_counter() + exec_result = container.exec_run( + cmd=["/bin/bash", "--noprofile", "--norc", "-c", command], + demux=True, + stderr=True, + stdout=True, + user=str(os.getuid()), + workdir=ale_bench.constants.WORK_DIR, + ) + end_at = time.perf_counter() + _stdout_bytes, stderr_bytes = exec_result.output or (b"", b"") + stderr = (stderr_bytes or b"").decode("utf-8", errors="replace").strip() + return end_at - start_at, exec_result.exit_code, stderr + except Exception: + should_replace = True + raise + finally: + self._release_container(container, should_replace) + + def _create_container(self) -> Container: + if self._client is None: + msg = "ReusableSubmissionContainerPool must be entered before creating containers." + raise RuntimeError(msg) + volumes = get_reusable_submission_volumes( + self.temp_dir, + self.scratch_dir, + self.tool_dir, + self.problem_type, + ) + container = self._client.containers.run( + image=get_docker_image_name(self.code_language, self.judge_version), + command=["/bin/bash", "--noprofile", "--norc", "-c", "sleep infinity"], + remove=False, + auto_remove=False, + cpu_period=100000, + cpu_quota=100000, # 1 CPU + detach=True, + group_add=[os.getgid()], + mem_limit=ale_bench.constants.MAX_MEMORY_LIMIT, + network_disabled=True, + user=os.getuid(), + volumes=volumes, + working_dir=ale_bench.constants.WORK_DIR, + ) + self._containers.append(container) + return container + + def _release_container(self, container: Container, should_replace: bool) -> None: + if should_replace or not self._is_container_running(container): + self._remove_container(container) + if container in self._containers: + self._containers.remove(container) + self._available_containers.put(self._create_container()) + else: + self._available_containers.put(container) + + def _is_container_running(self, container: Container) -> bool: + try: + container.reload() + attrs = container.attrs + if not isinstance(attrs, dict): + return False + state = attrs.get("State", {}) + if not isinstance(state, dict): + return False + return bool(state.get("Running", False)) + except Exception: + return False + + def _remove_container(self, container: Container) -> None: + with suppress(Exception): + container.remove(force=True) + + +class ReusableToolContainerPool: + """A fixed-size pool of long-lived judge/visualization tool containers.""" + + def __init__( + self, + scratch_dir: Path, + tool_dir: Path, + num_workers: int, + ) -> None: + """Initialize the reusable tool container pool configuration.""" + self.scratch_dir = scratch_dir + self.tool_dir = tool_dir + self.num_workers = num_workers + self._client_context = docker_client() + self._client: DockerClient | None = None + self._containers: list[Container] = [] + self._available_containers: Queue[Container] = Queue() + + def __enter__(self) -> ReusableToolContainerPool: # noqa: PYI034 + """Create and start all worker containers.""" + self._client = self._client_context.__enter__() + for _ in range(self.num_workers): + self._available_containers.put(self._create_container()) + return self + + def __exit__( + self, + exc_type: type[BaseException] | None, + exc_value: BaseException | None, + traceback: TracebackType | None, + ) -> None: + """Remove all worker containers and close the Docker client.""" + for container in self._containers: + self._remove_container(container) + self._containers.clear() + self._client_context.__exit__(exc_type, exc_value, traceback) + self._client = None + + def container_path(self, host_path: Path) -> str: + """Convert a scratch host path to its reusable-container path.""" + return f"{REUSABLE_TOOL_TMP_DIR}/{host_path.relative_to(self.scratch_dir)}" + + def run(self, command: str, *, workdir: str = ale_bench.constants.WORK_DIR) -> tuple[float, int, str]: + """Run one command via docker exec in an available reusable tool container.""" + container = self._available_containers.get() + should_replace = False + try: + start_at = time.perf_counter() + exec_result = container.exec_run( + cmd=["/bin/bash", "--noprofile", "--norc", "-c", command], + demux=True, + stderr=True, + stdout=True, + user=str(os.getuid()), + workdir=workdir, + ) + end_at = time.perf_counter() + _stdout_bytes, stderr_bytes = exec_result.output or (b"", b"") + stderr = (stderr_bytes or b"").decode("utf-8", errors="replace").strip() + return end_at - start_at, exec_result.exit_code, stderr + except Exception: + should_replace = True + raise + finally: + self._release_container(container, should_replace) + + def _create_container(self) -> Container: + if self._client is None: + msg = "ReusableToolContainerPool must be entered before creating containers." + raise RuntimeError(msg) + container = self._client.containers.run( + image=ale_bench.constants.RUST_TOOL_DOCKER_IMAGE, + command=["/bin/bash", "--noprofile", "--norc", "-c", "sleep infinity"], + remove=False, + auto_remove=False, + cpu_period=100000, + cpu_quota=100000, # 1 CPU + detach=True, + group_add=[os.getgid()], + mem_limit=ale_bench.constants.MAX_MEMORY_LIMIT, + network_disabled=True, + user=os.getuid(), + volumes=get_reusable_tool_volumes(self.scratch_dir, self.tool_dir), + working_dir=ale_bench.constants.WORK_DIR, + ) + self._containers.append(container) + return container + + def _release_container(self, container: Container, should_replace: bool) -> None: + if should_replace or not self._is_container_running(container): + self._remove_container(container) + if container in self._containers: + self._containers.remove(container) + self._available_containers.put(self._create_container()) + else: + self._available_containers.put(container) + + def _is_container_running(self, container: Container) -> bool: + try: + container.reload() + attrs = container.attrs + if not isinstance(attrs, dict): + return False + state = attrs.get("State", {}) + if not isinstance(state, dict): + return False + return bool(state.get("Running", False)) + except Exception: + return False + + def _remove_container(self, container: Container) -> None: + with suppress(Exception): + container.remove(force=True) diff --git a/src/ale_bench_eval/__main__.py b/src/ale_bench_eval/__main__.py index 7a40fd94..3327dd07 100644 --- a/src/ale_bench_eval/__main__.py +++ b/src/ale_bench_eval/__main__.py @@ -68,6 +68,7 @@ def evaluate_contest( problem_id: str, lite_version: bool, num_workers: int, + reuse_containers: bool, n_public_cases: int | None = None, selection_method: Literal["best", "median"] = "median", root_path: Path | None = None, @@ -83,6 +84,7 @@ def evaluate_contest( n_repeated_sampling=n_repeated_sampling, n_self_refine=n_self_refine, num_workers=num_workers, + reuse_containers=reuse_containers, n_public_cases=n_public_cases, prompt_args=prompt_args, problem_id=problem_id, @@ -249,6 +251,7 @@ def _run_evaluation_task( problem_id: str, lite_version: bool, num_workers: int, + reuse_containers: bool, n_public_cases: int | None, selection_method: Literal["best", "median"], root_path: Path, @@ -267,6 +270,7 @@ def _run_evaluation_task( problem_id=problem_id, lite_version=lite_version, num_workers=num_workers, + reuse_containers=reuse_containers, n_public_cases=n_public_cases, selection_method=selection_method, root_path=root_path, @@ -287,6 +291,7 @@ def main( n_repeated_sampling: int = 1, n_self_refine: int = 1, num_workers: int = 1, + reuse_containers: bool = False, n_public_cases: int | None = None, code_language: EvalCodeLanguage = "cpp20", judge_version: EvalJudgeVersion = "202301", @@ -383,6 +388,9 @@ def main( # NOTE: skip num_workers check to allow resuming with different num_workers # NOTE: skip max_concurrent_llm_calls check to allow resuming with different LLM concurrency # NOTE: skip max_repeated_sampling_workers check to allow resuming with different repeated sampling concurrency + if existing_settings.get("reuse_containers", False) != reuse_containers: + msg = "Experiment settings already exist with different reuse_containers" + raise ValueError(msg) if existing_settings["n_public_cases"] != n_public_cases: msg = "Experiment settings already exist with different n_public_cases" raise ValueError(msg) @@ -411,6 +419,7 @@ def main( "n_repeated_sampling": n_repeated_sampling, "n_self_refine": n_self_refine, "num_workers": num_workers, + "reuse_containers": reuse_containers, "n_public_cases": n_public_cases, "code_language": code_language, "judge_version": judge_version, @@ -432,6 +441,7 @@ def main( print( f"📊 Model: {model_name}, Repeated Sampling: {n_repeated_sampling}, Self-Refine: {n_self_refine}, " f"Code Language: {code_language}, Judge Version: {judge_version}, " + f"Reuse Containers: {reuse_containers}, " f"Max Concurrent LLM Calls: {max_concurrent_llm_calls}, " f"Max Repeated Sampling Workers: {max_repeated_sampling_workers}" ) @@ -459,6 +469,7 @@ def main( problem_id, lite_version, num_workers, + reuse_containers, n_public_cases, selection_method, exp_root, diff --git a/src/ale_bench_eval/calc_cost.py b/src/ale_bench_eval/calc_cost.py index 57a276e4..4440045b 100644 --- a/src/ale_bench_eval/calc_cost.py +++ b/src/ale_bench_eval/calc_cost.py @@ -141,6 +141,12 @@ cache_read_mtok=Decimal(5) / Decimal(10), output_mtok=Decimal(25), ), + "claude-fable-5": ModelPrice( + input_mtok=Decimal(10), + cache_write_mtok=Decimal(125) / Decimal(10), + cache_read_mtok=Decimal(1), + output_mtok=Decimal(50), + ), "grok-4.1-fast": ModelPrice( input_mtok=Decimal(2) / Decimal(10), cache_read_mtok=Decimal(5) / Decimal(100), @@ -244,6 +250,11 @@ output_mtok=Decimal(44) / Decimal(10), cache_read_mtok=Decimal(26) / Decimal(100), ), + "glm-5.2": ModelPrice( + input_mtok=Decimal(14) / Decimal(10), + output_mtok=Decimal(44) / Decimal(10), + cache_read_mtok=Decimal(26) / Decimal(100), + ), "glm-5-turbo": ModelPrice( input_mtok=Decimal(12) / Decimal(10), output_mtok=Decimal(4), @@ -283,6 +294,11 @@ output_mtok=Decimal(4), cache_read_mtok=Decimal(16) / Decimal(100), ), + "kimi-k2.7-code": ModelPrice( + input_mtok=Decimal(95) / Decimal(100), + output_mtok=Decimal(4), + cache_read_mtok=Decimal(19) / Decimal(100), + ), "mercury-2": ModelPrice( input_mtok=Decimal(25) / Decimal(100), cache_read_mtok=Decimal(25) / Decimal(1000), @@ -314,6 +330,16 @@ output_mtok=Decimal(5) / Decimal(10), cache_read_mtok=Decimal(1) / Decimal(10), ), + "ling-2.6-1t": ModelPrice( + input_mtok=Decimal(3) / Decimal(10), + cache_read_mtok=Decimal(6) / Decimal(100), + output_mtok=Decimal(25) / Decimal(10), + ), + "ring-2.6-1t": ModelPrice( + input_mtok=Decimal(3) / Decimal(10), + cache_read_mtok=Decimal(6) / Decimal(100), + output_mtok=Decimal(25) / Decimal(10), + ), "qwen3-235b-a22b-thinking-2507": ModelPrice( input_mtok=Decimal(3) / Decimal(10), output_mtok=Decimal(29) / Decimal(10) ), diff --git a/src/ale_bench_eval/data_types.py b/src/ale_bench_eval/data_types.py index a4f7246b..1bb0c804 100644 --- a/src/ale_bench_eval/data_types.py +++ b/src/ale_bench_eval/data_types.py @@ -21,6 +21,7 @@ class EvaluationConfig: n_repeated_sampling: int n_self_refine: int num_workers: int + reuse_containers: bool n_public_cases: int | None prompt_args: PromptArgs problem_id: str diff --git a/src/ale_bench_eval/evaluate.py b/src/ale_bench_eval/evaluate.py index 276b4d94..3a2e45fb 100644 --- a/src/ale_bench_eval/evaluate.py +++ b/src/ale_bench_eval/evaluate.py @@ -112,6 +112,7 @@ def run_private_evaluation( solution_code, code_language=solution_code_language, judge_version=config.prompt_args.judge_version, + reuse_containers=config.reuse_containers, ) if save_info is not None: diff --git a/src/ale_bench_eval/scaffolds.py b/src/ale_bench_eval/scaffolds.py index b702071b..10128905 100644 --- a/src/ale_bench_eval/scaffolds.py +++ b/src/ale_bench_eval/scaffolds.py @@ -72,12 +72,14 @@ def _evaluate_public_result_once( code_language, judge_version=config.prompt_args.judge_version, skip_local_visualization=True, + reuse_containers=config.reuse_containers, ) else: public_result = session.public_eval( code, code_language, judge_version=config.prompt_args.judge_version, + reuse_containers=config.reuse_containers, ) if public_result.overall_judge_result == JudgeResult.INTERNAL_ERROR: msg = "Judge returned INTERNAL_ERROR." diff --git a/tests/judge/codes/ac_cpp20_ahc001.cpp b/tests/judge/codes/ac_cpp20_ahc001.cpp new file mode 100644 index 00000000..20210543 --- /dev/null +++ b/tests/judge/codes/ac_cpp20_ahc001.cpp @@ -0,0 +1,2468 @@ +#include +using namespace std; + +struct FastRNG { + uint64_t x; + FastRNG(uint64_t seed = 88172645463325252ULL) { + x = seed ? seed : 88172645463325252ULL; + } + uint64_t nextU64() { + x ^= x >> 12; + x ^= x << 25; + x ^= x >> 27; + return x * 2685821657736338717ULL; + } + int nextInt(int n) { return (int)(nextU64() % (uint64_t)n); } + double nextDouble() { return (nextU64() >> 11) * (1.0 / 9007199254740992.0); } +}; + +struct Timer { + chrono::steady_clock::time_point st; + Timer() { reset(); } + void reset() { st = chrono::steady_clock::now(); } + double elapsed() const { + return chrono::duration(chrono::steady_clock::now() - st).count(); + } +}; + +struct Rect { + int a, b, c, d; +}; + +static uint64_t splitmix64_hash(uint64_t z) { + z += 0x9e3779b97f4a7c15ULL; + z = (z ^ (z >> 30)) * 0xbf58476d1ce4e5b9ULL; + z = (z ^ (z >> 27)) * 0x94d049bb133111ebULL; + return z ^ (z >> 31); +} + +class Solver { + static constexpr int SZ = 10000; + static constexpr double TIME_LIMIT = 4.86; + static constexpr double FINAL_RESERVE = 0.045; + + int n; + vector x, y; + vector r; + vector rd, invR; + FastRNG rng; + Timer timer; + + struct State { + vector rect; + vector val; + double total = 0.0; + }; + + struct Move { + bool ok = false; + int dir = 0; + int coord = 0; + double newScore = 0.0; + double delta = 0.0; + }; + + struct PairMove { + bool ok = false; + int orient = 0; + int i = -1, j = -1; + int coord = 0; + double newScoreI = 0.0; + double newScoreJ = 0.0; + double delta = 0.0; + }; + + struct LineComp { + int orient = 0; + int coord = -1; + vector A; + vector B; + }; + + struct LineMove { + bool ok = false; + int orient = 0; + int coord = -1; + int newCoord = 0; + vector A, B; + double delta = 0.0; + }; + + struct SplitCand { + int orient; + int k; + int t; + double cost; + }; + + long long rectArea(const Rect& rc) const { + return 1LL * (rc.c - rc.a) * (rc.d - rc.b); + } + + double scoreOne(int i, long long s) const { + double ratio; + if (s <= r[i]) ratio = (double)s * invR[i]; + else ratio = rd[i] / (double)s; + return 2.0 * ratio - ratio * ratio; + } + + static double scoreRatio(long long s, long long target) { + double ratio; + if (s <= target) ratio = (double)s / (double)target; + else ratio = (double)target / (double)s; + return 2.0 * ratio - ratio * ratio; + } + + State makeState(const vector& rects) const { + State st; + st.rect = rects; + st.val.assign(n, 0.0); + for (int i = 0; i < n; i++) { + st.val[i] = scoreOne(i, rectArea(st.rect[i])); + st.total += st.val[i]; + } + return st; + } + + int clampLL(long long v, int lo, int hi) const { + if (v < lo) return lo; + if (v > hi) return hi; + return (int)v; + } + + bool yOverlap(const Rect& p, const Rect& q) const { + return max(p.b, q.b) < min(p.d, q.d); + } + + bool xOverlap(const Rect& p, const Rect& q) const { + return max(p.a, q.a) < min(p.c, q.c); + } + + bool overlapRect(const Rect& p, const Rect& q) const { + return xOverlap(p, q) && yOverlap(p, q); + } + + void sortIds(vector& ord, int orient) const { + if (orient == 0) { + sort(ord.begin(), ord.end(), [&](int u, int v) { + if (x[u] != x[v]) return x[u] < x[v]; + return y[u] < y[v]; + }); + } else { + sort(ord.begin(), ord.end(), [&](int u, int v) { + if (y[u] != y[v]) return y[u] < y[v]; + return x[u] < x[v]; + }); + } + } + + vector generateSplitCands(const vector& ids, const Rect& box) const { + int m = (int)ids.size(); + vector cands; + if (m <= 1) return cands; + + long long totalR = 0; + for (int id : ids) totalR += r[id]; + + long long A = rectArea(box); + int W = box.c - box.a; + int H = box.d - box.b; + + cands.reserve(m * 40); + + for (int orient = 0; orient < 2; orient++) { + vector ord = ids; + sortIds(ord, orient); + + long long pref = 0; + for (int k = 1; k < m; k++) { + pref += r[ord[k - 1]]; + + int lo, hi; + if (orient == 0) { + int xl = x[ord[k - 1]]; + int xr = x[ord[k]]; + lo = max(box.a + 1, xl + 1); + hi = min(box.c - 1, xr); + } else { + int yl = y[ord[k - 1]]; + int yr = y[ord[k]]; + lo = max(box.b + 1, yl + 1); + hi = min(box.d - 1, yr); + } + + if (lo > hi) continue; + + vector ts; + ts.reserve(32); + + auto addT = [&](int t) { + if (t < lo || t > hi) return; + for (int u : ts) if (u == t) return; + ts.push_back(t); + }; + + auto addReal = [&](long double real) { + long long f = (long long)floorl(real); + for (long long v = f - 2; v <= f + 2; v++) { + addT(clampLL(v, lo, hi)); + } + long long rr = (long long)llroundl(real); + for (long long v = rr - 1; v <= rr + 1; v++) { + addT(clampLL(v, lo, hi)); + } + }; + + if (orient == 0) { + long double prop = box.a + ((long double)A * pref / totalR) / H; + long double exactL = box.a + (long double)pref / H; + long double exactR = box.a + ((long double)A - (long double)(totalR - pref)) / H; + addReal(prop); + addReal(exactL); + addReal(exactR); + } else { + long double prop = box.b + ((long double)A * pref / totalR) / W; + long double exactL = box.b + (long double)pref / W; + long double exactR = box.b + ((long double)A - (long double)(totalR - pref)) / W; + addReal(prop); + addReal(exactL); + addReal(exactR); + } + + addT(lo); + addT(hi); + + for (int t : ts) { + long long leftA; + if (orient == 0) leftA = 1LL * (t - box.a) * H; + else leftA = 1LL * W * (t - box.b); + + long long rightA = A - leftA; + if (leftA <= 0 || rightA <= 0) continue; + + double sc = + k * scoreRatio(leftA, pref) + + (m - k) * scoreRatio(rightA, totalR - pref); + + double loss = 1.0 - sc / m; + + auto aspectPenalty = [&](int ww, int hh, int cnt) -> double { + if (cnt <= 1) return 0.0; + double ar = max((double)ww / hh, (double)hh / ww); + return 0.00025 * (cnt / (double)m) * max(0.0, log(ar) - 2.0); + }; + + double cost = loss; + cost += 0.00020 * abs(m - 2 * k) / (double)m; + + if (orient == 0) { + cost += aspectPenalty(t - box.a, H, k); + cost += aspectPenalty(box.c - t, H, m - k); + } else { + cost += aspectPenalty(W, t - box.b, k); + cost += aspectPenalty(W, box.d - t, m - k); + } + + cands.push_back({orient, k, t, cost}); + } + } + } + + return cands; + } + + void buildRec(const vector& ids, const Rect& box, vector& out, double temp) { + int m = (int)ids.size(); + if (m == 1) { + out[ids[0]] = box; + return; + } + + vector cands = generateSplitCands(ids, box); + + if (cands.empty()) { + for (int id : ids) { + out[id] = {x[id], y[id], x[id] + 1, y[id] + 1}; + } + return; + } + + sort(cands.begin(), cands.end(), [](const SplitCand& p, const SplitCand& q) { + return p.cost < q.cost; + }); + + int chosen = 0; + if (temp > 1e-12) { + double bestCost = cands[0].cost; + double maxDiff = max(0.03, temp * 12.0); + + vector> cumulative; + double sum = 0.0; + for (int i = 0; i < (int)cands.size(); i++) { + double diff = cands[i].cost - bestCost; + if (diff > maxDiff) continue; + double w = exp(-diff / temp); + if (w < 1e-12) continue; + sum += w; + cumulative.push_back({i, sum}); + } + + if (sum > 0.0) { + double z = rng.nextDouble() * sum; + for (auto [idx, cum] : cumulative) { + if (z <= cum) { + chosen = idx; + break; + } + } + } + } + + SplitCand sp = cands[chosen]; + + vector ord = ids; + sortIds(ord, sp.orient); + + vector leftIds(ord.begin(), ord.begin() + sp.k); + vector rightIds(ord.begin() + sp.k, ord.end()); + + if (sp.orient == 0) { + Rect L{box.a, box.b, sp.t, box.d}; + Rect R{sp.t, box.b, box.c, box.d}; + buildRec(leftIds, L, out, temp); + buildRec(rightIds, R, out, temp); + } else { + Rect B{box.a, box.b, box.c, sp.t}; + Rect T{box.a, sp.t, box.c, box.d}; + buildRec(leftIds, B, out, temp); + buildRec(rightIds, T, out, temp); + } + } + + double scoreIds(const vector& rects, const vector& ids) const { + double s = 0.0; + for (int id : ids) s += scoreOne(id, rectArea(rects[id])); + return s; + } + + double buildRecBeam(const vector& ids, const Rect& box, vector& out, double stopTime) { + int m = (int)ids.size(); + if (m == 1) { + out[ids[0]] = box; + return scoreOne(ids[0], rectArea(box)); + } + + if (timer.elapsed() > stopTime) { + buildRec(ids, box, out, 0.0); + return scoreIds(out, ids); + } + + vector cands = generateSplitCands(ids, box); + if (cands.empty()) { + for (int id : ids) out[id] = {x[id], y[id], x[id] + 1, y[id] + 1}; + return scoreIds(out, ids); + } + + sort(cands.begin(), cands.end(), [](const SplitCand& p, const SplitCand& q) { + return p.cost < q.cost; + }); + + int width; + if (m <= 3) width = 10; + else if (m <= 5) width = 5; + else width = 3; + + width = min(width, (int)cands.size()); + + double best = -1e100; + vector bestOut = out; + + for (int ci = 0; ci < width; ci++) { + if (timer.elapsed() > stopTime) break; + + SplitCand sp = cands[ci]; + vector ord = ids; + sortIds(ord, sp.orient); + + vector leftIds(ord.begin(), ord.begin() + sp.k); + vector rightIds(ord.begin() + sp.k, ord.end()); + + vector tmp = out; + double sc = 0.0; + + if (sp.orient == 0) { + Rect L{box.a, box.b, sp.t, box.d}; + Rect R{sp.t, box.b, box.c, box.d}; + sc += buildRecBeam(leftIds, L, tmp, stopTime); + sc += buildRecBeam(rightIds, R, tmp, stopTime); + } else { + Rect B{box.a, box.b, box.c, sp.t}; + Rect T{box.a, sp.t, box.c, box.d}; + sc += buildRecBeam(leftIds, B, tmp, stopTime); + sc += buildRecBeam(rightIds, T, tmp, stopTime); + } + + if (sc > best) { + best = sc; + bestOut = std::move(tmp); + } + } + + if (best < -1e90) { + buildRec(ids, box, out, 0.0); + return scoreIds(out, ids); + } + + out = std::move(bestOut); + return best; + } + + vector buildRecursive(double temp) { + vector out(n); + vector ids(n); + iota(ids.begin(), ids.end(), 0); + buildRec(ids, Rect{0, 0, SZ, SZ}, out, temp); + return out; + } + + vector buildUnit() const { + vector rects(n); + for (int i = 0; i < n; i++) { + rects[i] = {x[i], y[i], x[i] + 1, y[i] + 1}; + } + return rects; + } + + pair legalInterval(const State& st, int i, int dir) const { + const Rect& ri = st.rect[i]; + + if (dir == 0) { + int lo = 0; + int hi = min(x[i], ri.c - 1); + for (int j = 0; j < n; j++) if (j != i) { + const Rect& rj = st.rect[j]; + if (yOverlap(ri, rj) && rj.a < ri.c) { + lo = max(lo, rj.c); + } + } + return {lo, hi}; + } + + if (dir == 1) { + int lo = max(x[i] + 1, ri.a + 1); + int hi = SZ; + for (int j = 0; j < n; j++) if (j != i) { + const Rect& rj = st.rect[j]; + if (yOverlap(ri, rj) && rj.c > ri.a) { + hi = min(hi, rj.a); + } + } + return {lo, hi}; + } + + if (dir == 2) { + int lo = 0; + int hi = min(y[i], ri.d - 1); + for (int j = 0; j < n; j++) if (j != i) { + const Rect& rj = st.rect[j]; + if (xOverlap(ri, rj) && rj.b < ri.d) { + lo = max(lo, rj.d); + } + } + return {lo, hi}; + } + + int lo = max(y[i] + 1, ri.b + 1); + int hi = SZ; + for (int j = 0; j < n; j++) if (j != i) { + const Rect& rj = st.rect[j]; + if (xOverlap(ri, rj) && rj.d > ri.b) { + hi = min(hi, rj.b); + } + } + return {lo, hi}; + } + + int getCoord(const Rect& rc, int dir) const { + if (dir == 0) return rc.a; + if (dir == 1) return rc.c; + if (dir == 2) return rc.b; + return rc.d; + } + + void setCoord(Rect& rc, int dir, int v) { + if (dir == 0) rc.a = v; + else if (dir == 1) rc.c = v; + else if (dir == 2) rc.b = v; + else rc.d = v; + } + + long long areaAfterCoord(const Rect& rc, int dir, int coord) const { + if (dir == 0) return 1LL * (rc.c - coord) * (rc.d - rc.b); + if (dir == 1) return 1LL * (coord - rc.a) * (rc.d - rc.b); + if (dir == 2) return 1LL * (rc.c - rc.a) * (rc.d - coord); + return 1LL * (rc.c - rc.a) * (coord - rc.b); + } + + Move bestEdgeDir(const State& st, int i, int dir) const { + Move mv; + auto [lo, hi] = legalInterval(st, i, dir); + if (lo > hi) return mv; + + const Rect& rc = st.rect[i]; + int cur = getCoord(rc, dir); + + mv.ok = true; + mv.dir = dir; + mv.coord = cur; + mv.newScore = st.val[i]; + mv.delta = 0.0; + + int cand[48]; + int cnt = 0; + + auto addCand = [&](int v) { + if (v < lo) v = lo; + if (v > hi) v = hi; + for (int k = 0; k < cnt; k++) if (cand[k] == v) return; + cand[cnt++] = v; + }; + + addCand(cur); + addCand(lo); + addCand(hi); + + long double real; + if (dir == 0 || dir == 1) { + int h = rc.d - rc.b; + long double wantW = (long double)r[i] / h; + if (dir == 0) real = rc.c - wantW; + else real = rc.a + wantW; + } else { + int w = rc.c - rc.a; + long double wantH = (long double)r[i] / w; + if (dir == 2) real = rc.d - wantH; + else real = rc.b + wantH; + } + + long long f = (long long)floorl(real); + for (long long v = f - 5; v <= f + 5; v++) { + addCand(clampLL(v, lo, hi)); + } + + double bestScore = st.val[i]; + int bestCoord = cur; + + for (int k = 0; k < cnt; k++) { + int v = cand[k]; + long long ar = areaAfterCoord(rc, dir, v); + double ns = scoreOne(i, ar); + if (ns > bestScore + 1e-15) { + bestScore = ns; + bestCoord = v; + } + } + + mv.coord = bestCoord; + mv.newScore = bestScore; + mv.delta = bestScore - st.val[i]; + return mv; + } + + Move bestMove(const State& st, int i) const { + Move best; + best.newScore = st.val[i]; + for (int dir = 0; dir < 4; dir++) { + Move mv = bestEdgeDir(st, i, dir); + if (!mv.ok) continue; + if (!best.ok || mv.delta > best.delta) best = mv; + } + return best; + } + + void applyCoord(State& st, int i, int dir, int coord, double newScore) { + setCoord(st.rect[i], dir, coord); + st.total += newScore - st.val[i]; + st.val[i] = newScore; + } + + void applyMove(State& st, int i, const Move& mv) { + applyCoord(st, i, mv.dir, mv.coord, mv.newScore); + } + + PairMove bestPairVertical(const State& st, int li, int ri) const { + PairMove pm; + pm.orient = 0; + pm.i = li; + pm.j = ri; + + if (li == ri) return pm; + const Rect& L = st.rect[li]; + const Rect& R = st.rect[ri]; + + if (L.c > R.a) return pm; + if (!yOverlap(L, R)) return pm; + + int lo1 = max(x[li] + 1, L.a + 1); + int hi1 = SZ; + for (int k = 0; k < n; k++) if (k != li && k != ri) { + const Rect& K = st.rect[k]; + if (yOverlap(L, K) && K.c > L.a) hi1 = min(hi1, K.a); + } + + int lo2 = 0; + int hi2 = min(x[ri], R.c - 1); + for (int k = 0; k < n; k++) if (k != li && k != ri) { + const Rect& K = st.rect[k]; + if (yOverlap(R, K) && K.a < R.c) lo2 = max(lo2, K.c); + } + + int lo = max(lo1, lo2); + int hi = min(hi1, hi2); + if (lo > hi) return pm; + + int hL = L.d - L.b; + int hR = R.d - R.b; + + auto eval = [&](int t) -> double { + long long a1 = 1LL * (t - L.a) * hL; + long long a2 = 1LL * (R.c - t) * hR; + return scoreOne(li, a1) + scoreOne(ri, a2); + }; + + double bestScore = -1e100; + int bestT = lo; + + auto test = [&](int t) { + if (t < lo || t > hi) return; + double sc = eval(t); + if (sc > bestScore + 1e-15) { + bestScore = sc; + bestT = t; + } + }; + + test(lo); + test(hi); + test(clampLL(L.c, lo, hi)); + test(clampLL(R.a, lo, hi)); + test(lo + (hi - lo) / 2); + test(lo + (hi - lo) / 4); + test(lo + 3 * (hi - lo) / 4); + + auto addReal = [&](long double z) { + long long f = (long long)floorl(z); + for (long long v = f - 6; v <= f + 6; v++) test(clampLL(v, lo, hi)); + }; + + addReal((long double)L.a + (long double)r[li] / hL); + addReal((long double)R.c - (long double)r[ri] / hR); + + int TL = lo, TR = hi; + while (TR - TL > 40) { + int m1 = TL + (TR - TL) / 3; + int m2 = TR - (TR - TL) / 3; + if (eval(m1) < eval(m2)) TL = m1; + else TR = m2; + } + for (int t = TL; t <= TR; t++) test(t); + + long long a1 = 1LL * (bestT - L.a) * hL; + long long a2 = 1LL * (R.c - bestT) * hR; + + pm.ok = true; + pm.coord = bestT; + pm.newScoreI = scoreOne(li, a1); + pm.newScoreJ = scoreOne(ri, a2); + pm.delta = pm.newScoreI + pm.newScoreJ - st.val[li] - st.val[ri]; + return pm; + } + + PairMove bestPairHorizontal(const State& st, int bi, int ti) const { + PairMove pm; + pm.orient = 1; + pm.i = bi; + pm.j = ti; + + if (bi == ti) return pm; + const Rect& B = st.rect[bi]; + const Rect& T = st.rect[ti]; + + if (B.d > T.b) return pm; + if (!xOverlap(B, T)) return pm; + + int lo1 = max(y[bi] + 1, B.b + 1); + int hi1 = SZ; + for (int k = 0; k < n; k++) if (k != bi && k != ti) { + const Rect& K = st.rect[k]; + if (xOverlap(B, K) && K.d > B.b) hi1 = min(hi1, K.b); + } + + int lo2 = 0; + int hi2 = min(y[ti], T.d - 1); + for (int k = 0; k < n; k++) if (k != bi && k != ti) { + const Rect& K = st.rect[k]; + if (xOverlap(T, K) && K.b < T.d) lo2 = max(lo2, K.d); + } + + int lo = max(lo1, lo2); + int hi = min(hi1, hi2); + if (lo > hi) return pm; + + int wB = B.c - B.a; + int wT = T.c - T.a; + + auto eval = [&](int t) -> double { + long long a1 = 1LL * wB * (t - B.b); + long long a2 = 1LL * wT * (T.d - t); + return scoreOne(bi, a1) + scoreOne(ti, a2); + }; + + double bestScore = -1e100; + int bestT = lo; + + auto test = [&](int t) { + if (t < lo || t > hi) return; + double sc = eval(t); + if (sc > bestScore + 1e-15) { + bestScore = sc; + bestT = t; + } + }; + + test(lo); + test(hi); + test(clampLL(B.d, lo, hi)); + test(clampLL(T.b, lo, hi)); + test(lo + (hi - lo) / 2); + test(lo + (hi - lo) / 4); + test(lo + 3 * (hi - lo) / 4); + + auto addReal = [&](long double z) { + long long f = (long long)floorl(z); + for (long long v = f - 6; v <= f + 6; v++) test(clampLL(v, lo, hi)); + }; + + addReal((long double)B.b + (long double)r[bi] / wB); + addReal((long double)T.d - (long double)r[ti] / wT); + + int TL = lo, TR = hi; + while (TR - TL > 40) { + int m1 = TL + (TR - TL) / 3; + int m2 = TR - (TR - TL) / 3; + if (eval(m1) < eval(m2)) TL = m1; + else TR = m2; + } + for (int t = TL; t <= TR; t++) test(t); + + long long a1 = 1LL * wB * (bestT - B.b); + long long a2 = 1LL * wT * (T.d - bestT); + + pm.ok = true; + pm.coord = bestT; + pm.newScoreI = scoreOne(bi, a1); + pm.newScoreJ = scoreOne(ti, a2); + pm.delta = pm.newScoreI + pm.newScoreJ - st.val[bi] - st.val[ti]; + return pm; + } + + void applyPair(State& st, const PairMove& pm) { + int i = pm.i; + int j = pm.j; + + if (pm.orient == 0) { + st.rect[i].c = pm.coord; + st.rect[j].a = pm.coord; + } else { + st.rect[i].d = pm.coord; + st.rect[j].b = pm.coord; + } + + st.total += pm.newScoreI - st.val[i]; + st.total += pm.newScoreJ - st.val[j]; + + st.val[i] = pm.newScoreI; + st.val[j] = pm.newScoreJ; + } + + void shuffleVector(vector& v) { + for (int i = (int)v.size() - 1; i > 0; i--) { + int j = rng.nextInt(i + 1); + swap(v[i], v[j]); + } + } + + int selectBad(const State& st) { + int best = rng.nextInt(n); + double bv = st.val[best]; + + int K = min(n, 6); + for (int k = 1; k < K; k++) { + int j = rng.nextInt(n); + if (st.val[j] < bv) { + bv = st.val[j]; + best = j; + } + } + return best; + } + + bool greedyPasses(State& st, int passes, double stopTime) { + vector ord(n); + iota(ord.begin(), ord.end(), 0); + + bool globalAny = false; + + for (int pass = 0; pass < passes; pass++) { + if (timer.elapsed() > stopTime) return globalAny; + shuffleVector(ord); + + bool any = false; + for (int id : ord) { + for (int rep = 0; rep < 4; rep++) { + Move mv = bestMove(st, id); + if (mv.ok && mv.delta > 1e-12) { + applyMove(st, id, mv); + any = true; + globalAny = true; + } else { + break; + } + } + } + if (!any) break; + } + + return globalAny; + } + + bool pairGreedyPasses(State& st, int passes, double stopTime) { + vector ord(n); + iota(ord.begin(), ord.end(), 0); + + bool globalAny = false; + + for (int pass = 0; pass < passes; pass++) { + shuffleVector(ord); + bool any = false; + int checks = 0; + + for (int ai = 0; ai < n; ai++) { + int i = ord[ai]; + for (int bj = ai + 1; bj < n; bj++) { + if ((checks++ & 255) == 0 && timer.elapsed() > stopTime) return globalAny; + + int j = ord[bj]; + const Rect& A = st.rect[i]; + const Rect& B = st.rect[j]; + + PairMove best; + best.delta = 0.0; + + if (yOverlap(A, B)) { + PairMove mv; + if (A.c <= B.a) mv = bestPairVertical(st, i, j); + else if (B.c <= A.a) mv = bestPairVertical(st, j, i); + if (mv.ok && (!best.ok || mv.delta > best.delta)) best = mv; + } + + if (xOverlap(A, B)) { + PairMove mv; + if (A.d <= B.b) mv = bestPairHorizontal(st, i, j); + else if (B.d <= A.b) mv = bestPairHorizontal(st, j, i); + if (mv.ok && (!best.ok || mv.delta > best.delta)) best = mv; + } + + if (best.ok && best.delta > 1e-12) { + applyPair(st, best); + any = true; + globalAny = true; + } + } + } + + if (!any) break; + } + + return globalAny; + } + + vector buildLineComponents(const State& st, int orient) const { + vector coords; + coords.reserve(2 * n); + + for (int i = 0; i < n; i++) { + const Rect& rc = st.rect[i]; + if (orient == 0) { + if (0 < rc.c && rc.c < SZ) coords.push_back(rc.c); + if (0 < rc.a && rc.a < SZ) coords.push_back(rc.a); + } else { + if (0 < rc.d && rc.d < SZ) coords.push_back(rc.d); + if (0 < rc.b && rc.b < SZ) coords.push_back(rc.b); + } + } + + sort(coords.begin(), coords.end()); + coords.erase(unique(coords.begin(), coords.end()), coords.end()); + + vector comps; + + for (int coord : coords) { + vector A, B; + for (int i = 0; i < n; i++) { + const Rect& rc = st.rect[i]; + if (orient == 0) { + if (rc.c == coord) A.push_back(i); + if (rc.a == coord) B.push_back(i); + } else { + if (rc.d == coord) A.push_back(i); + if (rc.b == coord) B.push_back(i); + } + } + + if (A.empty() || B.empty()) continue; + + int na = (int)A.size(); + int nb = (int)B.size(); + vector> adj(na + nb); + + for (int i = 0; i < na; i++) { + for (int j = 0; j < nb; j++) { + bool ov = (orient == 0) + ? yOverlap(st.rect[A[i]], st.rect[B[j]]) + : xOverlap(st.rect[A[i]], st.rect[B[j]]); + if (ov) { + adj[i].push_back(na + j); + adj[na + j].push_back(i); + } + } + } + + vector vis(na + nb, 0); + for (int s = 0; s < na + nb; s++) { + if (vis[s]) continue; + + vector q = {s}; + vis[s] = 1; + for (int qi = 0; qi < (int)q.size(); qi++) { + int v = q[qi]; + for (int to : adj[v]) { + if (!vis[to]) { + vis[to] = 1; + q.push_back(to); + } + } + } + + LineComp cp; + cp.orient = orient; + cp.coord = coord; + + for (int v : q) { + if (v < na) cp.A.push_back(A[v]); + else cp.B.push_back(B[v - na]); + } + + if (!cp.A.empty() && !cp.B.empty()) comps.push_back(std::move(cp)); + } + } + + return comps; + } + + bool componentInternalOK(const State& st, const LineComp& cp) const { + vector side(n, 0); + + for (int id : cp.A) { + if (side[id] & 1) return false; + side[id] |= 1; + } + for (int id : cp.B) { + if (side[id] & 2) return false; + if (side[id] & 1) return false; + side[id] |= 2; + } + + auto perpOverlap = [&](int u, int v) -> bool { + if (cp.orient == 0) return yOverlap(st.rect[u], st.rect[v]); + else return xOverlap(st.rect[u], st.rect[v]); + }; + + for (int i = 0; i < (int)cp.A.size(); i++) { + for (int j = i + 1; j < (int)cp.A.size(); j++) { + if (perpOverlap(cp.A[i], cp.A[j])) return false; + } + } + for (int i = 0; i < (int)cp.B.size(); i++) { + for (int j = i + 1; j < (int)cp.B.size(); j++) { + if (perpOverlap(cp.B[i], cp.B[j])) return false; + } + } + + return true; + } + + vector buildFrontierComponents(const State& st, int orient) const { + const int INF = 1e9; + vector bestA(n, INF), bestB(n, INF); + + auto canFace = [&](int i, int j, int& gap) -> bool { + if (i == j) return false; + const Rect& P = st.rect[i]; + const Rect& Q = st.rect[j]; + + if (orient == 0) { + if (!yOverlap(P, Q)) return false; + if (P.c <= Q.a) { + gap = Q.a - P.c; + return true; + } + } else { + if (!xOverlap(P, Q)) return false; + if (P.d <= Q.b) { + gap = Q.b - P.d; + return true; + } + } + return false; + }; + + for (int i = 0; i < n; i++) { + for (int j = 0; j < n; j++) { + int gap; + if (canFace(i, j, gap)) { + bestA[i] = min(bestA[i], gap); + bestB[j] = min(bestB[j], gap); + } + } + } + + vector> adj(2 * n); + + for (int i = 0; i < n; i++) { + for (int j = 0; j < n; j++) { + int gap; + if (canFace(i, j, gap)) { + if (gap == bestA[i] && gap == bestB[j]) { + adj[i].push_back(n + j); + adj[n + j].push_back(i); + } + } + } + } + + vector comps; + vector vis(2 * n, 0); + + for (int s = 0; s < 2 * n; s++) { + if (vis[s] || adj[s].empty()) continue; + + vector q = {s}; + vis[s] = 1; + + for (int qi = 0; qi < (int)q.size(); qi++) { + int v = q[qi]; + for (int to : adj[v]) { + if (!vis[to]) { + vis[to] = 1; + q.push_back(to); + } + } + } + + LineComp cp; + cp.orient = orient; + cp.coord = -1; + + for (int v : q) { + if (v < n) cp.A.push_back(v); + else cp.B.push_back(v - n); + } + + if (cp.A.empty() || cp.B.empty()) continue; + if (!componentInternalOK(st, cp)) continue; + + bool sameCoord = true; + int common = -1; + + auto feedCoord = [&](int z) { + if (common < 0) common = z; + else if (common != z) sameCoord = false; + }; + + if (orient == 0) { + for (int id : cp.A) feedCoord(st.rect[id].c); + for (int id : cp.B) feedCoord(st.rect[id].a); + } else { + for (int id : cp.A) feedCoord(st.rect[id].d); + for (int id : cp.B) feedCoord(st.rect[id].b); + } + + if (sameCoord) continue; + comps.push_back(std::move(cp)); + } + + return comps; + } + + LineMove bestBoundaryComponent(const State& st, const LineComp& cp, double stopTime) { + LineMove lm; + lm.orient = cp.orient; + lm.coord = cp.coord; + lm.A = cp.A; + lm.B = cp.B; + + if (cp.A.empty() || cp.B.empty()) return lm; + if (!componentInternalOK(st, cp)) return lm; + + if (cp.coord >= 0) { + if (cp.orient == 0) { + for (int id : cp.A) if (st.rect[id].c != cp.coord) return lm; + for (int id : cp.B) if (st.rect[id].a != cp.coord) return lm; + } else { + for (int id : cp.A) if (st.rect[id].d != cp.coord) return lm; + for (int id : cp.B) if (st.rect[id].b != cp.coord) return lm; + } + } + + vector moved(n, 0); + double curScore = 0.0; + + for (int id : cp.A) { + if (!moved[id]) { + moved[id] = 1; + curScore += st.val[id]; + } + } + for (int id : cp.B) { + if (!moved[id]) { + moved[id] = 1; + curScore += st.val[id]; + } + } + + int lo = 0; + int hi = SZ; + + if (cp.orient == 0) { + for (int id : cp.A) { + const Rect& ri = st.rect[id]; + lo = max(lo, max(x[id] + 1, ri.a + 1)); + + for (int k = 0; k < n; k++) if (!moved[k]) { + const Rect& rk = st.rect[k]; + if (yOverlap(ri, rk) && rk.c > ri.a) hi = min(hi, rk.a); + } + } + + for (int id : cp.B) { + const Rect& ri = st.rect[id]; + hi = min(hi, min(x[id], ri.c - 1)); + + for (int k = 0; k < n; k++) if (!moved[k]) { + const Rect& rk = st.rect[k]; + if (yOverlap(ri, rk) && rk.a < ri.c) lo = max(lo, rk.c); + } + } + } else { + for (int id : cp.A) { + const Rect& ri = st.rect[id]; + lo = max(lo, max(y[id] + 1, ri.b + 1)); + + for (int k = 0; k < n; k++) if (!moved[k]) { + const Rect& rk = st.rect[k]; + if (xOverlap(ri, rk) && rk.d > ri.b) hi = min(hi, rk.b); + } + } + + for (int id : cp.B) { + const Rect& ri = st.rect[id]; + hi = min(hi, min(y[id], ri.d - 1)); + + for (int k = 0; k < n; k++) if (!moved[k]) { + const Rect& rk = st.rect[k]; + if (xOverlap(ri, rk) && rk.b < ri.d) lo = max(lo, rk.d); + } + } + } + + if (lo > hi) return lm; + + auto eval = [&](int t) -> double { + double sc = 0.0; + + if (cp.orient == 0) { + for (int id : cp.A) { + const Rect& rc = st.rect[id]; + long long ar = 1LL * (t - rc.a) * (rc.d - rc.b); + sc += scoreOne(id, ar); + } + for (int id : cp.B) { + const Rect& rc = st.rect[id]; + long long ar = 1LL * (rc.c - t) * (rc.d - rc.b); + sc += scoreOne(id, ar); + } + } else { + for (int id : cp.A) { + const Rect& rc = st.rect[id]; + long long ar = 1LL * (rc.c - rc.a) * (t - rc.b); + sc += scoreOne(id, ar); + } + for (int id : cp.B) { + const Rect& rc = st.rect[id]; + long long ar = 1LL * (rc.c - rc.a) * (rc.d - t); + sc += scoreOne(id, ar); + } + } + + return sc; + }; + + double bestScore = curScore; + int bestT = (cp.coord >= 0 ? clampLL(cp.coord, lo, hi) : lo); + + for (int t = lo; t <= hi; t++) { + if (((t - lo) & 1023) == 0 && timer.elapsed() > stopTime) break; + double sc = eval(t); + if (sc > bestScore + 1e-15) { + bestScore = sc; + bestT = t; + } + } + + lm.ok = true; + lm.newCoord = bestT; + lm.delta = bestScore - curScore; + return lm; + } + + void applyLineMove(State& st, const LineMove& lm) { + int t = lm.newCoord; + + if (lm.orient == 0) { + for (int id : lm.A) st.rect[id].c = t; + for (int id : lm.B) st.rect[id].a = t; + } else { + for (int id : lm.A) st.rect[id].d = t; + for (int id : lm.B) st.rect[id].b = t; + } + + vector upd(n, 0); + for (int id : lm.A) { + if (!upd[id]) { + upd[id] = 1; + double ns = scoreOne(id, rectArea(st.rect[id])); + st.total += ns - st.val[id]; + st.val[id] = ns; + } + } + for (int id : lm.B) { + if (!upd[id]) { + upd[id] = 1; + double ns = scoreOne(id, rectArea(st.rect[id])); + st.total += ns - st.val[id]; + st.val[id] = ns; + } + } + } + + bool boundarySequentialPass(State& st, int orient, bool includeFrontier, double stopTime) { + vector comps = buildLineComponents(st, orient); + + if (includeFrontier) { + vector extra = buildFrontierComponents(st, orient); + for (auto& cp : extra) comps.push_back(std::move(cp)); + } + + sort(comps.begin(), comps.end(), [](const LineComp& p, const LineComp& q) { + int sp = (int)p.A.size() + (int)p.B.size(); + int sq = (int)q.A.size() + (int)q.B.size(); + if (sp != sq) return sp > sq; + bool lp = p.coord >= 0; + bool lq = q.coord >= 0; + if (lp != lq) return lp > lq; + return p.coord < q.coord; + }); + + bool any = false; + + for (const LineComp& cp : comps) { + if (timer.elapsed() > stopTime) break; + + LineMove mv = bestBoundaryComponent(st, cp, stopTime); + if (mv.ok && mv.delta > 1e-12) { + applyLineMove(st, mv); + any = true; + } + } + + return any; + } + + bool boundaryGreedyPasses(State& st, int passes, double stopTime, bool includeFrontier) { + bool globalAny = false; + + for (int pass = 0; pass < passes; pass++) { + if (timer.elapsed() > stopTime) return globalAny; + + bool any = false; + if (rng.nextInt(2) == 0) { + any |= boundarySequentialPass(st, 0, includeFrontier, stopTime); + any |= boundarySequentialPass(st, 1, includeFrontier, stopTime); + } else { + any |= boundarySequentialPass(st, 1, includeFrontier, stopTime); + any |= boundarySequentialPass(st, 0, includeFrontier, stopTime); + } + + if (!any) break; + globalAny = true; + } + + return globalAny; + } + + Rect groupBoundingBox(const State& st, const vector& ids) const { + Rect box{SZ, SZ, 0, 0}; + for (int id : ids) { + const Rect& rc = st.rect[id]; + box.a = min(box.a, rc.a); + box.b = min(box.b, rc.b); + box.c = max(box.c, rc.c); + box.d = max(box.d, rc.d); + } + return box; + } + + bool bboxClearForGroup(const State& st, const vector& ids, const Rect& box) const { + vector in(n, 0); + for (int id : ids) in[id] = 1; + for (int i = 0; i < n; i++) { + if (in[i]) continue; + if (overlapRect(st.rect[i], box)) return false; + } + return true; + } + + long long desiredSum(const vector& ids) const { + long long s = 0; + for (int id : ids) s += r[id]; + return s; + } + + bool boxUsableForGroup(const State& st, const vector& ids, const Rect& box) const { + if (!(0 <= box.a && box.a < box.c && box.c <= SZ)) return false; + if (!(0 <= box.b && box.b < box.d && box.d <= SZ)) return false; + + for (int id : ids) { + if (!(box.a <= x[id] && x[id] + 1 <= box.c)) return false; + if (!(box.b <= y[id] && y[id] + 1 <= box.d)) return false; + } + + return bboxClearForGroup(st, ids, box); + } + + vector alternativeRepackBoxes(const State& st, const vector& ids, const Rect& orig) const { + vector res; + + long long S = 0; + int minX = SZ, minY = SZ, maxX1 = 0, maxY1 = 0; + + for (int id : ids) { + S += r[id]; + minX = min(minX, x[id]); + minY = min(minY, y[id]); + maxX1 = max(maxX1, x[id] + 1); + maxY1 = max(maxY1, y[id] + 1); + } + + int pW = max(1, maxX1 - minX); + int pH = max(1, maxY1 - minY); + int oW = orig.c - orig.a; + int oH = orig.d - orig.b; + + auto sameBox = [](const Rect& u, const Rect& v) { + return u.a == v.a && u.b == v.b && u.c == v.c && u.d == v.d; + }; + + auto addRect = [&](const Rect& bx) { + if ((int)res.size() >= 4) return; + if (sameBox(bx, orig)) return; + if (!boxUsableForGroup(st, ids, bx)) return; + for (const Rect& e : res) if (sameBox(e, bx)) return; + res.push_back(bx); + }; + + auto ceilDiv = [](long long a, long long b) -> long long { + return (a + b - 1) / b; + }; + + auto trySize = [&](long long W0, long long H0) { + if ((int)res.size() >= 4) return; + if (W0 <= 0 || H0 <= 0) return; + + long long W = max(W0, pW); + long long H = max(H0, pH); + + if (W > SZ) return; + + long long needH = ceilDiv(S, W); + if (H < needH) H = needH; + + if (H > SZ) { + H = SZ; + long long needW = ceilDiv(S, H); + if (W < needW) W = needW; + } + + if (W > SZ || H > SZ) return; + + int w = (int)W; + int h = (int)H; + + auto placements = [&](bool inside) { + int alo = max(0, maxX1 - w); + int ahi = min(minX, SZ - w); + int blo = max(0, maxY1 - h); + int bhi = min(minY, SZ - h); + + if (inside) { + alo = max(alo, orig.a); + ahi = min(ahi, orig.c - w); + blo = max(blo, orig.b); + bhi = min(bhi, orig.d - h); + } + + if (alo > ahi || blo > bhi) return; + + vector As, Bs; + auto addA = [&](long long v) { + int z = clampLL(v, alo, ahi); + for (int q : As) if (q == z) return; + As.push_back(z); + }; + auto addB = [&](long long v) { + int z = clampLL(v, blo, bhi); + for (int q : Bs) if (q == z) return; + Bs.push_back(z); + }; + + addA(((long long)orig.a + orig.c - w) / 2); + addA(orig.a); + addA((long long)orig.c - w); + addA((long long)minX - w / 2); + + addB(((long long)orig.b + orig.d - h) / 2); + addB(orig.b); + addB((long long)orig.d - h); + addB((long long)minY - h / 2); + + for (int a : As) { + for (int b : Bs) { + addRect(Rect{a, b, a + w, b + h}); + if ((int)res.size() >= 4) return; + } + } + }; + + bool inside = (w <= oW && h <= oH); + placements(inside); + if (!inside) placements(false); + }; + + auto ceilDiv2 = ceilDiv; + + trySize(oW, ceilDiv2(S, max(1, oW))); + trySize(ceilDiv2(S, max(1, oH)), oH); + + { + long double asp = max(0.05L, min(20.0L, (long double)oW / max(1, oH))); + long long w = max(1LL, (long long)llroundl(sqrtl((long double)S * asp))); + trySize(w, ceilDiv2(S, w)); + } + + { + long long w = max(1LL, (long long)llroundl(sqrtl((long double)S))); + trySize(w, ceilDiv2(S, w)); + } + + { + long double asp = max(0.05L, min(20.0L, (long double)pW / max(1, pH))); + long long w = max(1LL, (long long)llroundl(sqrtl((long double)S * asp))); + trySize(w, ceilDiv2(S, w)); + } + + trySize(pW, ceilDiv2(S, max(1, pW))); + trySize(ceilDiv2(S, max(1, pH)), pH); + + return res; + } + + bool closureGroup(const State& st, vector ids, int maxK, vector& out) const { + if (ids.empty()) return false; + + sort(ids.begin(), ids.end()); + ids.erase(unique(ids.begin(), ids.end()), ids.end()); + + if ((int)ids.size() > maxK) return false; + + vector in(n, 0); + for (int id : ids) in[id] = 1; + + Rect box = groupBoundingBox(st, ids); + + bool changed = true; + while (changed) { + changed = false; + + for (int i = 0; i < n; i++) { + if (in[i]) continue; + + if (overlapRect(st.rect[i], box)) { + in[i] = 1; + ids.push_back(i); + + if ((int)ids.size() > maxK) return false; + + const Rect& rc = st.rect[i]; + box.a = min(box.a, rc.a); + box.b = min(box.b, rc.b); + box.c = max(box.c, rc.c); + box.d = max(box.d, rc.d); + + changed = true; + } + } + } + + sort(ids.begin(), ids.end()); + out = ids; + return true; + } + + bool validSubsetRects(const vector& v, const vector& ids) const { + vector in(n, 0); + for (int id : ids) in[id] = 1; + + for (int id : ids) { + const Rect& rc = v[id]; + if (!(0 <= rc.a && rc.a < rc.c && rc.c <= SZ)) return false; + if (!(0 <= rc.b && rc.b < rc.d && rc.d <= SZ)) return false; + if (!(rc.a <= x[id] && x[id] + 1 <= rc.c)) return false; + if (!(rc.b <= y[id] && y[id] + 1 <= rc.d)) return false; + } + + for (int i = 0; i < (int)ids.size(); i++) { + for (int j = i + 1; j < (int)ids.size(); j++) { + if (overlapRect(v[ids[i]], v[ids[j]])) return false; + } + } + + for (int id : ids) { + for (int j = 0; j < n; j++) { + if (in[j]) continue; + if (overlapRect(v[id], v[j])) return false; + } + } + + return true; + } + + void recomputeSubset(State& st, const vector& ids) { + for (int id : ids) { + double ns = scoreOne(id, rectArea(st.rect[id])); + st.total += ns - st.val[id]; + st.val[id] = ns; + } + } + + bool greedySubset(State& st, vector ids, int passes, double stopTime) { + bool globalAny = false; + + for (int pass = 0; pass < passes; pass++) { + if (timer.elapsed() > stopTime) return globalAny; + shuffleVector(ids); + + bool any = false; + for (int id : ids) { + for (int rep = 0; rep < 4; rep++) { + Move mv = bestMove(st, id); + if (mv.ok && mv.delta > 1e-12) { + applyMove(st, id, mv); + any = true; + globalAny = true; + } else { + break; + } + } + } + + if (!any) break; + } + + return globalAny; + } + + bool pairSubsetPass(State& st, vector ids, int passes, double stopTime) { + bool globalAny = false; + + for (int pass = 0; pass < passes; pass++) { + if (timer.elapsed() > stopTime) return globalAny; + shuffleVector(ids); + + bool any = false; + int checks = 0; + + for (int ai = 0; ai < (int)ids.size(); ai++) { + int i = ids[ai]; + for (int bj = ai + 1; bj < (int)ids.size(); bj++) { + if ((checks++ & 63) == 0 && timer.elapsed() > stopTime) return globalAny; + + int j = ids[bj]; + const Rect& A = st.rect[i]; + const Rect& B = st.rect[j]; + + PairMove best; + best.delta = 0.0; + + if (yOverlap(A, B)) { + PairMove mv; + if (A.c <= B.a) mv = bestPairVertical(st, i, j); + else if (B.c <= A.a) mv = bestPairVertical(st, j, i); + if (mv.ok && (!best.ok || mv.delta > best.delta)) best = mv; + } + + if (xOverlap(A, B)) { + PairMove mv; + if (A.d <= B.b) mv = bestPairHorizontal(st, i, j); + else if (B.d <= A.b) mv = bestPairHorizontal(st, j, i); + if (mv.ok && (!best.ok || mv.delta > best.delta)) best = mv; + } + + if (best.ok && best.delta > 1e-12) { + applyPair(st, best); + any = true; + globalAny = true; + } + } + } + + if (!any) break; + } + + return globalAny; + } + + bool tryRepackGroup(State& st, vector ids, double stopTime, int trials) { + if (timer.elapsed() > stopTime) return false; + + sort(ids.begin(), ids.end()); + ids.erase(unique(ids.begin(), ids.end()), ids.end()); + + if ((int)ids.size() < 2 || (int)ids.size() > 8) return false; + + Rect box = groupBoundingBox(st, ids); + if (!(box.a < box.c && box.b < box.d)) return false; + if (!bboxClearForGroup(st, ids, box)) return false; + + State bestLocal = st; + + double beforeScore = 0.0; + for (int id : ids) beforeScore += st.val[id]; + + long long desire = desiredSum(ids); + double mismatch = fabs((double)rectArea(box) - (double)desire) / max(1.0, (double)desire); + double loss = (double)ids.size() - beforeScore; + + auto consider = [&](const vector& fullRects) { + if (timer.elapsed() > stopTime) return; + if (!validSubsetRects(fullRects, ids)) return; + + State tr = st; + for (int id : ids) tr.rect[id] = fullRects[id]; + recomputeSubset(tr, ids); + + greedySubset(tr, ids, 2, stopTime); + pairSubsetPass(tr, ids, 1, stopTime); + greedySubset(tr, ids, 1, stopTime); + + if (tr.total > bestLocal.total + 1e-12) bestLocal = std::move(tr); + }; + + if ((int)ids.size() <= 6 && timer.elapsed() + 0.004 < stopTime) { + vector rects = st.rect; + buildRecBeam(ids, box, rects, stopTime); + consider(rects); + } + + { + vector rects = st.rect; + buildRec(ids, box, rects, 0.0); + consider(rects); + } + + if ((loss > 0.015 || mismatch > 0.08) && timer.elapsed() + 0.005 < stopTime) { + vector altBoxes = alternativeRepackBoxes(st, ids, box); + + int used = 0; + for (const Rect& bx : altBoxes) { + if (used >= 3) break; + if (timer.elapsed() + 0.003 > stopTime) break; + + { + vector rects = st.rect; + if ((int)ids.size() <= 5 && timer.elapsed() + 0.004 < stopTime) { + buildRecBeam(ids, bx, rects, stopTime); + } else { + buildRec(ids, bx, rects, 0.0); + } + consider(rects); + } + + if (trials >= 4 && (int)ids.size() <= 4 && timer.elapsed() + 0.004 < stopTime) { + vector rects = st.rect; + double temp = 0.0005 * pow(500.0, rng.nextDouble()); + buildRec(ids, bx, rects, temp); + consider(rects); + } + + used++; + } + } + + if (trials >= 3 && timer.elapsed() < stopTime) { + vector rects = st.rect; + for (int id : ids) rects[id] = {x[id], y[id], x[id] + 1, y[id] + 1}; + consider(rects); + } + + for (int t = 0; t < trials && timer.elapsed() < stopTime; t++) { + vector rects = st.rect; + double temp = 0.0005 * pow(500.0, rng.nextDouble()); + buildRec(ids, box, rects, temp); + consider(rects); + } + + if (bestLocal.total > st.total + 1e-12) { + st = std::move(bestLocal); + return true; + } + + return false; + } + + bool tryRepackGroupClosure(State& st, vector ids, double stopTime, int trials, int maxK = 8) { + vector cl; + if (!closureGroup(st, std::move(ids), maxK, cl)) return false; + return tryRepackGroup(st, cl, stopTime, trials); + } + + bool componentRepackPass(State& st, double stopTime, int maxGroups) { + struct GC { + double key; + vector ids; + }; + + vector groups; + + for (int orient = 0; orient < 2; orient++) { + vector comps = buildLineComponents(st, orient); + for (const LineComp& cp : comps) { + vector ids = cp.A; + for (int id : cp.B) ids.push_back(id); + + vector cl; + if (!closureGroup(st, ids, 8, cl)) continue; + + int k = (int)cl.size(); + if (k < 2 || k > 8) continue; + + double loss = 0.0; + for (int id : cl) loss += 1.0 - st.val[id]; + + double key = loss + 0.0015 * k + 1e-5 * rng.nextDouble(); + groups.push_back({key, cl}); + } + } + + sort(groups.begin(), groups.end(), [](const GC& p, const GC& q) { + return p.key > q.key; + }); + + bool any = false; + int tried = 0; + + for (auto& g : groups) { + if (timer.elapsed() > stopTime) break; + if (tried++ >= maxGroups) break; + + int k = (int)g.ids.size(); + int trials = (k <= 3 ? 5 : 3); + + if (tryRepackGroup(st, g.ids, stopTime, trials)) any = true; + } + + return any; + } + + int rectGap(const Rect& A, const Rect& B) const { + int dx = 0; + if (A.c < B.a) dx = B.a - A.c; + else if (B.c < A.a) dx = A.a - B.c; + + int dy = 0; + if (A.d < B.b) dy = B.b - A.d; + else if (B.d < A.b) dy = A.b - B.d; + + int penalty = (dx > 0 && dy > 0) ? 10000 : 0; + return dx + dy + penalty; + } + + bool pairBlockRepackPass(State& st, double stopTime, int maxGroups) { + struct PC { + double key; + int i, j; + }; + + vector cand; + bool stop = false; + + for (int i = 0; i < n && !stop; i++) { + for (int j = i + 1; j < n; j++) { + if (((int)cand.size() & 511) == 0 && timer.elapsed() > stopTime) { + stop = true; + break; + } + + int g = rectGap(st.rect[i], st.rect[j]); + double loss = (1.0 - st.val[i]) + (1.0 - st.val[j]); + + if (g > 8000 && loss < 0.05) continue; + + double key = loss - 0.0000015 * g + 0.0001 / (1.0 + g) + 1e-5 * rng.nextDouble(); + cand.push_back({key, i, j}); + } + } + + sort(cand.begin(), cand.end(), [](const PC& p, const PC& q) { + return p.key > q.key; + }); + + bool any = false; + int lim = min(maxGroups, (int)cand.size()); + + for (int t = 0; t < lim; t++) { + if (timer.elapsed() > stopTime) break; + if (tryRepackGroupClosure(st, vector{cand[t].i, cand[t].j}, stopTime, 3, 8)) any = true; + } + + return any; + } + + bool neighborClosureRepackPass(State& st, double stopTime, int maxBase) { + vector ord(n); + iota(ord.begin(), ord.end(), 0); + sort(ord.begin(), ord.end(), [&](int u, int v) { + return st.val[u] < st.val[v]; + }); + + bool any = false; + int bases = min(maxBase, n); + + for (int bi = 0; bi < bases && timer.elapsed() < stopTime; bi++) { + int base = ord[bi]; + + vector> near; + near.reserve(n - 1); + for (int j = 0; j < n; j++) { + if (j == base) continue; + near.push_back({rectGap(st.rect[base], st.rect[j]), j}); + } + sort(near.begin(), near.end()); + + int lim = min(4, (int)near.size()); + for (int t = 0; t < lim && timer.elapsed() < stopTime; t++) { + if (tryRepackGroupClosure(st, vector{base, near[t].second}, stopTime, 3, 8)) any = true; + } + + if (near.size() >= 2 && timer.elapsed() < stopTime) { + vector g{base, near[0].second, near[1].second}; + if (tryRepackGroupClosure(st, g, stopTime, 3, 8)) any = true; + } + + if (near.size() >= 3 && timer.elapsed() < stopTime) { + vector g{base, near[0].second, near[1].second, near[2].second}; + if (tryRepackGroupClosure(st, g, stopTime, 2, 8)) any = true; + } + } + + return any; + } + + bool randomBlockRepackAttempts(State& st, double stopTime, int attempts) { + bool any = false; + + for (int at = 0; at < attempts && timer.elapsed() < stopTime; at++) { + int base; + if (rng.nextDouble() < 0.75) base = selectBad(st); + else base = rng.nextInt(n); + + vector group{base}; + vector used(n, 0); + used[base] = 1; + + int target = 2 + rng.nextInt(4); + + for (int step = 1; step < target && timer.elapsed() < stopTime; step++) { + vector> cands; + + for (int j = 0; j < n; j++) { + if (used[j]) continue; + + int g = 1e9; + for (int id : group) { + g = min(g, rectGap(st.rect[id], st.rect[j])); + } + + double key = (double)g - 1800.0 * (1.0 - st.val[j]) + 20.0 * rng.nextDouble(); + cands.push_back({key, j}); + } + + if (cands.empty()) break; + sort(cands.begin(), cands.end()); + + int lim = min(6, (int)cands.size()); + int pick = 0; + if (rng.nextDouble() > 0.70) pick = rng.nextInt(lim); + + int add = cands[pick].second; + used[add] = 1; + group.push_back(add); + + if ((int)group.size() >= 2) { + if (tryRepackGroupClosure(st, group, stopTime, 3, 8)) { + any = true; + break; + } + } + } + } + + return any; + } + + void randomGreedyOps(State& st, int ops, double stopTime) { + for (int op = 0; op < ops; op++) { + if ((op & 255) == 0 && timer.elapsed() > stopTime) break; + + int i; + if (rng.nextDouble() < 0.85) i = selectBad(st); + else i = rng.nextInt(n); + + Move mv = bestMove(st, i); + if (mv.ok && mv.delta > 1e-12) applyMove(st, i, mv); + } + } + + bool randomPairMove(State& st) { + int i; + if (rng.nextDouble() < 0.80) i = selectBad(st); + else i = rng.nextInt(n); + + int dir = rng.nextInt(4); + + static constexpr int K = 8; + int cand[K]; + int gap[K]; + int cnt = 0; + + auto addCand = [&](int j, int g) { + if (cnt < K) { + cand[cnt] = j; + gap[cnt] = g; + cnt++; + } else { + int worst = 0; + for (int t = 1; t < K; t++) { + if (gap[t] > gap[worst]) worst = t; + } + if (g < gap[worst] || rng.nextDouble() < 0.02) { + cand[worst] = j; + gap[worst] = g; + } + } + }; + + const Rect& R = st.rect[i]; + + for (int j = 0; j < n; j++) if (j != i) { + const Rect& Q = st.rect[j]; + + if (dir == 0) { + if (Q.c <= R.a && yOverlap(Q, R)) addCand(j, R.a - Q.c); + } else if (dir == 1) { + if (R.c <= Q.a && yOverlap(R, Q)) addCand(j, Q.a - R.c); + } else if (dir == 2) { + if (Q.d <= R.b && xOverlap(Q, R)) addCand(j, R.b - Q.d); + } else { + if (R.d <= Q.b && xOverlap(R, Q)) addCand(j, Q.b - R.d); + } + } + + if (cnt == 0) return false; + + int pos = 0; + for (int t = 1; t < cnt; t++) if (gap[t] < gap[pos]) pos = t; + if (rng.nextDouble() < 0.35) pos = rng.nextInt(cnt); + + int j = cand[pos]; + + PairMove mv; + if (dir == 0) mv = bestPairVertical(st, j, i); + else if (dir == 1) mv = bestPairVertical(st, i, j); + else if (dir == 2) mv = bestPairHorizontal(st, j, i); + else mv = bestPairHorizontal(st, i, j); + + if (mv.ok && mv.delta > 1e-12) { + applyPair(st, mv); + return true; + } + return false; + } + + bool randomResize(State& st, double temp, double progress) { + int i; + if (rng.nextDouble() < 0.55) i = selectBad(st); + else i = rng.nextInt(n); + + int dir = rng.nextInt(4); + auto [lo, hi] = legalInterval(st, i, dir); + if (lo > hi) return false; + + const Rect& rc = st.rect[i]; + int cur = getCoord(rc, dir); + if (lo == hi) return false; + + int coord = cur; + double q = rng.nextDouble(); + + if (q < 0.25) { + Move mv = bestEdgeDir(st, i, dir); + if (!mv.ok) return false; + coord = mv.coord; + } else if (q < 0.80) { + long long ar = rectArea(rc); + bool under = ar < r[i]; + bool preferImprove = rng.nextDouble() < 0.72; + + int sign; + if (dir == 0 || dir == 2) sign = under ? -1 : +1; + else sign = under ? +1 : -1; + + if (!preferImprove) sign = -sign; + + int maxd = (sign < 0 ? cur - lo : hi - cur); + if (maxd <= 0) { + sign = -sign; + maxd = (sign < 0 ? cur - lo : hi - cur); + } + + if (maxd > 0) { + int step = 1 + (int)(3000.0 * (1.0 - progress) * (1.0 - progress)); + int d = 1 + rng.nextInt(min(maxd, step)); + coord = cur + sign * d; + } else { + coord = lo + rng.nextInt(hi - lo + 1); + } + } else { + coord = lo + rng.nextInt(hi - lo + 1); + } + + if (coord == cur) return false; + + long long newArea = areaAfterCoord(rc, dir, coord); + double ns = scoreOne(i, newArea); + double delta = ns - st.val[i]; + + if (delta >= 0.0 || rng.nextDouble() < exp(delta / max(temp, 1e-9))) { + applyCoord(st, i, dir, coord, ns); + return true; + } + return false; + } + + bool randomShift(State& st, double progress) { + int i; + if (rng.nextDouble() < 0.5) i = selectBad(st); + else i = rng.nextInt(n); + + int orient = rng.nextInt(2); + Rect& ri = st.rect[i]; + + int lo, hi; + + if (orient == 0) { + lo = max(-ri.a, x[i] + 1 - ri.c); + hi = min(SZ - ri.c, x[i] - ri.a); + + for (int j = 0; j < n; j++) if (j != i) { + const Rect& rj = st.rect[j]; + if (!yOverlap(ri, rj)) continue; + + if (rj.c <= ri.a) lo = max(lo, rj.c - ri.a); + else if (rj.a >= ri.c) hi = min(hi, rj.a - ri.c); + } + + if (lo > hi || (lo == 0 && hi == 0)) return false; + + int d = 0; + if (rng.nextDouble() < 0.70) { + int step = 1 + (int)(1200.0 * (1.0 - progress)); + int L = max(lo, -step); + int R = min(hi, step); + if (L <= R) d = L + rng.nextInt(R - L + 1); + } else { + d = lo + rng.nextInt(hi - lo + 1); + } + + if (d == 0) return false; + ri.a += d; + ri.c += d; + return true; + } else { + lo = max(-ri.b, y[i] + 1 - ri.d); + hi = min(SZ - ri.d, y[i] - ri.b); + + for (int j = 0; j < n; j++) if (j != i) { + const Rect& rj = st.rect[j]; + if (!xOverlap(ri, rj)) continue; + + if (rj.d <= ri.b) lo = max(lo, rj.d - ri.b); + else if (rj.b >= ri.d) hi = min(hi, rj.b - ri.d); + } + + if (lo > hi || (lo == 0 && hi == 0)) return false; + + int d = 0; + if (rng.nextDouble() < 0.70) { + int step = 1 + (int)(1200.0 * (1.0 - progress)); + int L = max(lo, -step); + int R = min(hi, step); + if (L <= R) d = L + rng.nextInt(R - L + 1); + } else { + d = lo + rng.nextInt(hi - lo + 1); + } + + if (d == 0) return false; + ri.b += d; + ri.d += d; + return true; + } + } + + bool randomReshape(State& st, double progress) { + for (int attempt = 0; attempt < 3; attempt++) { + int i; + if (rng.nextDouble() < 0.25) i = selectBad(st); + else i = rng.nextInt(n); + + Rect oldRect = st.rect[i]; + double oldVal = st.val[i]; + double oldTotal = st.total; + + int dir1 = rng.nextInt(4); + int dir2; + if (dir1 < 2) dir2 = 2 + rng.nextInt(2); + else dir2 = rng.nextInt(2); + + auto [lo, hi] = legalInterval(st, i, dir1); + if (lo > hi) continue; + + int cur = getCoord(st.rect[i], dir1); + if (lo == hi) continue; + + int coord = cur; + double q = rng.nextDouble(); + + if (q < 0.22) { + coord = (rng.nextInt(2) ? lo : hi); + } else { + int step = 1 + (int)(2600.0 * (1.0 - progress) + 80.0); + int L = max(lo, cur - step); + int R = min(hi, cur + step); + if (L > R) { + L = lo; + R = hi; + } + coord = L + rng.nextInt(R - L + 1); + } + + if (coord == cur) { + if (cur > lo) coord = cur - 1; + else if (cur < hi) coord = cur + 1; + else continue; + } + + long long ar1 = areaAfterCoord(st.rect[i], dir1, coord); + double ns1 = scoreOne(i, ar1); + applyCoord(st, i, dir1, coord, ns1); + + Move mv = bestEdgeDir(st, i, dir2); + if (mv.ok) applyMove(st, i, mv); + + bool changed = + st.rect[i].a != oldRect.a || st.rect[i].b != oldRect.b || + st.rect[i].c != oldRect.c || st.rect[i].d != oldRect.d; + + if (changed && st.val[i] + 1e-12 >= oldVal) return true; + + st.rect[i] = oldRect; + st.val[i] = oldVal; + st.total = oldTotal; + } + + return false; + } + + State initialSolution() { + constexpr double INIT_END = 0.80; + + State best; + bool hasBest = false; + + auto consider = [&](const State& st) { + if (!hasBest || st.total > best.total) { + best = st; + hasBest = true; + } + }; + + { + vector rects = buildRecursive(0.0); + State st = makeState(rects); + greedyPasses(st, 5, INIT_END); + pairGreedyPasses(st, 1, INIT_END); + boundaryGreedyPasses(st, 1, INIT_END, false); + randomGreedyOps(st, 5 * n, INIT_END); + consider(st); + } + + if (timer.elapsed() < INIT_END) { + vector rects = buildUnit(); + State st = makeState(rects); + greedyPasses(st, 7, INIT_END); + randomGreedyOps(st, 12 * n, INIT_END); + pairGreedyPasses(st, 1, INIT_END); + consider(st); + } + + while (timer.elapsed() < INIT_END) { + double temp = 0.00035 * pow(180.0, rng.nextDouble()); + vector rects = buildRecursive(temp); + State st = makeState(rects); + + greedyPasses(st, 2, INIT_END); + randomGreedyOps(st, 3 * n, INIT_END); + consider(st); + } + + return best; + } + + State improve(State bestState) { + State cur = bestState; + + double preEnd = min(TIME_LIMIT - 0.72, timer.elapsed() + 0.38); + boundaryGreedyPasses(cur, 2, preEnd, true); + pairGreedyPasses(cur, 1, preEnd); + greedyPasses(cur, 2, preEnd); + boundaryGreedyPasses(cur, 1, preEnd, true); + + if (cur.total > bestState.total + 1e-12) bestState = cur; + + double saStart = timer.elapsed(); + double saEnd = TIME_LIMIT - 0.50; + + int iter = 0; + double progress = 0.0; + double temp = 0.05; + + while (true) { + if ((iter & 1023) == 0) { + double now = timer.elapsed(); + if (now > saEnd) break; + + progress = (now - saStart) / max(1e-9, saEnd - saStart); + progress = min(1.0, max(0.0, progress)); + + double T0 = 0.050; + double T1 = 0.00001; + temp = T0 * pow(T1 / T0, progress); + } + + double q = rng.nextDouble(); + + if (q < 0.05) { + randomShift(cur, progress); + } else if (q < 0.12) { + randomReshape(cur, progress); + } else if (q < 0.34) { + int i; + if (rng.nextDouble() < 0.82) i = selectBad(cur); + else i = rng.nextInt(n); + + Move mv = bestMove(cur, i); + if (mv.ok && mv.delta > 1e-12) applyMove(cur, i, mv); + } else if (q < 0.62) { + randomPairMove(cur); + } else { + randomResize(cur, temp, progress); + } + + if (cur.total > bestState.total + 1e-12) bestState = cur; + + if ((iter & 65535) == 0) { + double threshold = max(1.0, 0.015 * n); + if (cur.total < bestState.total - threshold) cur = bestState; + } + + iter++; + } + + cur = bestState; + + int stagnant = 0; + const double finalLoopEnd = TIME_LIMIT - FINAL_RESERVE; + + while (timer.elapsed() < finalLoopEnd) { + bool imp = false; + + imp |= boundaryGreedyPasses(cur, 1, finalLoopEnd, true); + if (cur.total > bestState.total + 1e-12) bestState = cur; + + imp |= pairGreedyPasses(cur, 1, finalLoopEnd); + if (cur.total > bestState.total + 1e-12) bestState = cur; + + imp |= greedyPasses(cur, 1, finalLoopEnd); + if (cur.total > bestState.total + 1e-12) bestState = cur; + + double rem = finalLoopEnd - timer.elapsed(); + if (rem > 0.18) { + double blockStop = min(finalLoopEnd, timer.elapsed() + 0.13); + + imp |= componentRepackPass(cur, blockStop, 24); + if (cur.total > bestState.total + 1e-12) bestState = cur; + + imp |= neighborClosureRepackPass(cur, blockStop, 7); + if (cur.total > bestState.total + 1e-12) bestState = cur; + + imp |= pairBlockRepackPass(cur, blockStop, 20); + if (cur.total > bestState.total + 1e-12) bestState = cur; + + imp |= randomBlockRepackAttempts(cur, blockStop, 7); + if (cur.total > bestState.total + 1e-12) bestState = cur; + } + + for (int k = 0; k < 240 && timer.elapsed() < finalLoopEnd; k++) { + double q = rng.nextDouble(); + + if (q < 0.28) { + if (randomPairMove(cur)) imp = true; + } else if (q < 0.40) { + if (randomReshape(cur, 1.0)) imp = true; + } else { + int i; + if (rng.nextDouble() < 0.90) i = selectBad(cur); + else i = rng.nextInt(n); + + Move mv = bestMove(cur, i); + if (mv.ok && mv.delta > 1e-12) { + applyMove(cur, i, mv); + imp = true; + } + } + + if (cur.total > bestState.total + 1e-12) bestState = cur; + } + + if (!imp) { + stagnant++; + if (stagnant >= 4) break; + } else { + stagnant = 0; + } + } + + State polish = bestState; + greedyPasses(polish, 2, TIME_LIMIT); + pairGreedyPasses(polish, 1, TIME_LIMIT); + boundaryGreedyPasses(polish, 1, TIME_LIMIT, false); + greedyPasses(polish, 1, TIME_LIMIT); + if (polish.total > bestState.total + 1e-12) bestState = std::move(polish); + + return bestState; + } + + bool validateRects(const vector& v) const { + if ((int)v.size() != n) return false; + + for (int i = 0; i < n; i++) { + const Rect& rc = v[i]; + if (!(0 <= rc.a && rc.a < rc.c && rc.c <= SZ)) return false; + if (!(0 <= rc.b && rc.b < rc.d && rc.d <= SZ)) return false; + if (!(rc.a <= x[i] && x[i] + 1 <= rc.c)) return false; + if (!(rc.b <= y[i] && y[i] + 1 <= rc.d)) return false; + } + + for (int i = 0; i < n; i++) { + for (int j = i + 1; j < n; j++) { + if (xOverlap(v[i], v[j]) && yOverlap(v[i], v[j])) return false; + } + } + + return true; + } + +public: + void run() { + ios::sync_with_stdio(false); + cin.tie(nullptr); + + cin >> n; + x.resize(n); + y.resize(n); + r.resize(n); + rd.resize(n); + invR.resize(n); + + uint64_t seed = 1234567891234567ULL; + for (int i = 0; i < n; i++) { + cin >> x[i] >> y[i] >> r[i]; + rd[i] = (double)r[i]; + invR[i] = 1.0 / rd[i]; + + uint64_t z = ((uint64_t)x[i] << 32) ^ ((uint64_t)y[i] << 16) ^ (uint64_t)r[i]; + seed ^= splitmix64_hash(z + seed); + } + rng = FastRNG(seed); + + timer.reset(); + + State bestState = initialSolution(); + bestState = improve(bestState); + + if (!validateRects(bestState.rect)) { + bestState = makeState(buildUnit()); + } + + for (int i = 0; i < n; i++) { + const Rect& rc = bestState.rect[i]; + cout << rc.a << ' ' << rc.b << ' ' << rc.c << ' ' << rc.d << '\n'; + } + } +}; + +int main() { + Solver solver; + solver.run(); + return 0; +} diff --git a/tests/judge/codes/ac_cpp20_ahc003.cpp b/tests/judge/codes/ac_cpp20_ahc003.cpp new file mode 100644 index 00000000..10e64f44 --- /dev/null +++ b/tests/judge/codes/ac_cpp20_ahc003.cpp @@ -0,0 +1,1301 @@ +#include +using namespace std; + +static constexpr int N = 30; +static constexpr int LINE_VARS = 2 * N; + +static constexpr int BINS = 4; +static constexpr int HBIN_VARS = N * BINS; +static constexpr int BIN_VARS = 2 * N * BINS; + +static constexpr int SEG_VARS = 2 * LINE_VARS; + +static constexpr int FINE_BINS = 6; +static constexpr int HFINE_VARS = N * FINE_BINS; +static constexpr int FINE_VARS = 2 * N * FINE_BINS; + +static constexpr int H_EDGES = N * (N - 1); +static constexpr int EDGE_VARS = 2 * H_EDGES; + +static constexpr int EXPLORE_TURNS = 100; + +static inline double clampd(double x, double lo, double hi) { + return min(hi, max(lo, x)); +} + +static inline int popcnt(uint32_t x) { + return __builtin_popcount(x); +} + +static inline uint32_t lowMask(int x) { + if (x <= 0) return 0; + return (1u << x) - 1u; +} + +struct RidgeModel { + int n; + vector ata; + vector atb; + + RidgeModel(int n_ = 0) : n(n_), ata(n_ * n_, 0.0), atb(n_, 0.0) {} + + void addObservation(const vector>& counts, int steps, double result) { + if (steps <= 0) return; + + vector> f; + f.reserve(counts.size()); + + double inv_steps = 1.0 / steps; + for (auto [idx, cnt] : counts) { + if (cnt > 0) f.push_back({idx, cnt * inv_steps}); + } + + double target = result * inv_steps; + + for (auto [ia, va] : f) { + double* row = &ata[ia * n]; + atb[ia] += va * target; + for (auto [ib, vb] : f) { + row[ib] += va * vb; + } + } + } + + void solve(const vector& prior, double lambda, vector& out) const { + vector a = ata; + vector b(n); + + for (int i = 0; i < n; i++) { + a[i * n + i] += lambda; + b[i] = atb[i] + lambda * prior[i]; + } + + for (int i = 0; i < n; i++) { + for (int j = 0; j <= i; j++) { + double sum = a[i * n + j]; + for (int k = 0; k < j; k++) { + sum -= a[i * n + k] * a[j * n + k]; + } + + if (i == j) { + if (sum < 1e-9) sum = 1e-9; + a[i * n + j] = sqrt(sum); + } else { + a[i * n + j] = sum / a[j * n + j]; + } + } + } + + vector y(n); + for (int i = 0; i < n; i++) { + double sum = b[i]; + for (int k = 0; k < i; k++) sum -= a[i * n + k] * y[k]; + y[i] = sum / a[i * n + i]; + } + + out.assign(n, 0.0); + for (int i = n - 1; i >= 0; i--) { + double sum = y[i]; + for (int k = i + 1; k < n; k++) sum -= a[k * n + i] * out[k]; + out[i] = sum / a[i * n + i]; + out[i] = clampd(out[i], 1000.0, 9000.0); + } + } +}; + +struct Observation { + int result = 0; + int steps = 0; + array mask; + + Observation() { + mask.fill(0); + } +}; + +struct Solver { + RidgeModel lineModel; + RidgeModel binModel; + RidgeModel fineModel; + + vector lineEst; + vector binDelta; + vector segEst; + vector fineDelta; + vector edgeDelta; + + array lineSeen{}; + array binSeen{}; + array fineSeen{}; + array edgeSeen{}; + array edgeLearnSeen{}; + + array split{}; + array segSupport{}; + + vector observations; + + mt19937 rng; + int turn = 0; + + double segGlobalConf = 0.0; + + bool fineReady = false; + double fineValidationConf = 1.0; + deque> fineValidationWindow; + + double edgeValidationConf = 1.0; + deque> edgeValidationWindow; + + Solver() + : lineModel(LINE_VARS), + binModel(BIN_VARS), + fineModel(FINE_VARS), + lineEst(LINE_VARS, 5000.0), + binDelta(BIN_VARS, 0.0), + segEst(SEG_VARS, 5000.0), + fineDelta(FINE_VARS, 0.0), + edgeDelta(EDGE_VARS, 0.0), + rng(1234567) { + lineSeen.fill(0); + binSeen.fill(0); + fineSeen.fill(0); + edgeSeen.fill(0); + edgeLearnSeen.fill(0); + split.fill(14); + segSupport.fill(0); + } + + static int hVar(int i, int j) { + int q = j * BINS / (N - 1); + return i * BINS + q; + } + + static int vVar(int i, int j) { + int q = i * BINS / (N - 1); + return HBIN_VARS + j * BINS + q; + } + + static int fineHVar(int i, int j) { + int q = j * FINE_BINS / (N - 1); + return i * FINE_BINS + q; + } + + static int fineVVar(int i, int j) { + int q = i * FINE_BINS / (N - 1); + return HFINE_VARS + j * FINE_BINS + q; + } + + static int hEdge(int i, int j) { + return i * (N - 1) + j; + } + + static int vEdge(int i, int j) { + return H_EDGES + i * N + j; + } + + double globalWeight() const { + return clampd(turn / 300.0, 0.0, 1.0); + } + + double binWeight() const { + return clampd((turn - 80) / 420.0, 0.0, 1.0); + } + + double segmentWeight() const { + double t = clampd((turn - 110) / 390.0, 0.0, 1.0); + return t * segGlobalConf; + } + + double fineWeight() const { + double t = clampd((turn - 300) / 500.0, 0.0, 1.0); + return 0.40 * t * fineValidationConf; + } + + double edgeWeight() const { + double t = clampd((turn - 520) / 400.0, 0.0, 1.0); + return 0.15 * t * edgeValidationConf; + } + + double rawStructH(int i, int j) const { + int line = i; + int bv = hVar(i, j); + + double bw = binWeight(); + double base = lineEst[line] + bw * binDelta[bv]; + + int side = (j < split[line] ? 0 : 1); + int sv = 2 * line + side; + + double cov = clampd(segSupport[sv] / 25.0, 0.0, 1.0); + double sw = 0.90 * segmentWeight() * cov; + + double raw = (1.0 - sw) * base + sw * segEst[sv]; + return clampd(raw, 1000.0, 9000.0); + } + + double rawStructV(int i, int j) const { + int line = N + j; + int bv = vVar(i, j); + + double bw = binWeight(); + double base = lineEst[line] + bw * binDelta[bv]; + + int side = (i < split[line] ? 0 : 1); + int sv = 2 * line + side; + + double cov = clampd(segSupport[sv] / 25.0, 0.0, 1.0); + double sw = 0.90 * segmentWeight() * cov; + + double raw = (1.0 - sw) * base + sw * segEst[sv]; + return clampd(raw, 1000.0, 9000.0); + } + + double rawNoEdgeH(int i, int j) const { + double raw = rawStructH(i, j); + + if (fineReady) { + int fv = fineHVar(i, j); + double seenConf = clampd((fineSeen[fv] - 3.0) / 12.0, 0.0, 1.0); + double fw = fineWeight() * seenConf; + raw = clampd(raw + fw * fineDelta[fv], 1000.0, 9000.0); + } + + return raw; + } + + double rawNoEdgeV(int i, int j) const { + double raw = rawStructV(i, j); + + if (fineReady) { + int fv = fineVVar(i, j); + double seenConf = clampd((fineSeen[fv] - 3.0) / 12.0, 0.0, 1.0); + double fw = fineWeight() * seenConf; + raw = clampd(raw + fw * fineDelta[fv], 1000.0, 9000.0); + } + + return raw; + } + + double hCostNoFine(int i, int j) const { + double raw = rawStructH(i, j); + double g = globalWeight(); + return clampd(5000.0 + g * (raw - 5000.0), 1000.0, 9000.0); + } + + double vCostNoFine(int i, int j) const { + double raw = rawStructV(i, j); + double g = globalWeight(); + return clampd(5000.0 + g * (raw - 5000.0), 1000.0, 9000.0); + } + + double hCostNoEdge(int i, int j) const { + double raw = rawNoEdgeH(i, j); + double g = globalWeight(); + return clampd(5000.0 + g * (raw - 5000.0), 1000.0, 9000.0); + } + + double vCostNoEdge(int i, int j) const { + double raw = rawNoEdgeV(i, j); + double g = globalWeight(); + return clampd(5000.0 + g * (raw - 5000.0), 1000.0, 9000.0); + } + + double hCost(int i, int j) const { + double raw = rawNoEdgeH(i, j); + + int e = hEdge(i, j); + double seenConf = clampd((edgeLearnSeen[e] - 2.0) / 8.0, 0.0, 1.0); + double ew = edgeWeight() * seenConf; + raw = clampd(raw + ew * edgeDelta[e], 1000.0, 9000.0); + + double g = globalWeight(); + return clampd(5000.0 + g * (raw - 5000.0), 1000.0, 9000.0); + } + + double vCost(int i, int j) const { + double raw = rawNoEdgeV(i, j); + + int e = vEdge(i, j); + double seenConf = clampd((edgeLearnSeen[e] - 2.0) / 8.0, 0.0, 1.0); + double ew = edgeWeight() * seenConf; + raw = clampd(raw + ew * edgeDelta[e], 1000.0, 9000.0); + + double g = globalWeight(); + return clampd(5000.0 + g * (raw - 5000.0), 1000.0, 9000.0); + } + + static void appendVertical(string& s, int from, int to) { + if (to > from) s.append(to - from, 'D'); + else s.append(from - to, 'U'); + } + + static void appendHorizontal(string& s, int from, int to) { + if (to > from) s.append(to - from, 'R'); + else s.append(from - to, 'L'); + } + + string makeDirect(int si, int sj, int ti, int tj, bool horizontalFirst) const { + string s; + if (horizontalFirst) { + appendHorizontal(s, sj, tj); + appendVertical(s, si, ti); + } else { + appendVertical(s, si, ti); + appendHorizontal(s, sj, tj); + } + return s; + } + + string makeViaRow(int si, int sj, int ti, int tj, int r) const { + string s; + appendVertical(s, si, r); + appendHorizontal(s, sj, tj); + appendVertical(s, r, ti); + return s; + } + + string makeViaCol(int si, int sj, int ti, int tj, int c) const { + string s; + appendHorizontal(s, sj, c); + appendVertical(s, si, ti); + appendHorizontal(s, c, tj); + return s; + } + + bool validatePath(int si, int sj, int ti, int tj, const string& path) const { + array vis{}; + int r = si, c = sj; + vis[r * N + c] = 1; + + for (char ch : path) { + if (ch == 'U') r--; + else if (ch == 'D') r++; + else if (ch == 'L') c--; + else if (ch == 'R') c++; + else return false; + + if (r < 0 || r >= N || c < 0 || c >= N) return false; + + int id = r * N + c; + if (vis[id]) return false; + vis[id] = 1; + } + + return r == ti && c == tj; + } + + double estimatePathCost(int si, int sj, const string& path) const { + int r = si, c = sj; + double total = 0.0; + + for (char ch : path) { + if (ch == 'R') { + total += hCost(r, c); + c++; + } else if (ch == 'L') { + total += hCost(r, c - 1); + c--; + } else if (ch == 'D') { + total += vCost(r, c); + r++; + } else if (ch == 'U') { + total += vCost(r - 1, c); + r--; + } + } + + return total; + } + + double estimatePathCostNoEdge(int si, int sj, const string& path) const { + int r = si, c = sj; + double total = 0.0; + + for (char ch : path) { + if (ch == 'R') { + total += hCostNoEdge(r, c); + c++; + } else if (ch == 'L') { + total += hCostNoEdge(r, c - 1); + c--; + } else if (ch == 'D') { + total += vCostNoEdge(r, c); + r++; + } else if (ch == 'U') { + total += vCostNoEdge(r - 1, c); + r--; + } + } + + return total; + } + + double estimatePathCostNoFine(int si, int sj, const string& path) const { + int r = si, c = sj; + double total = 0.0; + + for (char ch : path) { + if (ch == 'R') { + total += hCostNoFine(r, c); + c++; + } else if (ch == 'L') { + total += hCostNoFine(r, c - 1); + c--; + } else if (ch == 'D') { + total += vCostNoFine(r, c); + r++; + } else if (ch == 'U') { + total += vCostNoFine(r - 1, c); + r--; + } + } + + return total; + } + + double explorationSeenScore(int si, int sj, const string& path) const { + array lu{}; + array bu{}; + + int r = si, c = sj; + + for (char ch : path) { + int line = -1, bv = -1; + + if (ch == 'R') { + line = r; + bv = hVar(r, c); + c++; + } else if (ch == 'L') { + c--; + line = r; + bv = hVar(r, c); + } else if (ch == 'D') { + line = N + c; + bv = vVar(r, c); + r++; + } else if (ch == 'U') { + r--; + line = N + c; + bv = vVar(r, c); + } + + if (line >= 0) lu[line] = 1; + if (bv >= 0) bu[bv] = 1; + } + + double score = 0.0; + for (int i = 0; i < LINE_VARS; i++) { + if (lu[i]) score += lineSeen[i]; + } + for (int i = 0; i < BIN_VARS; i++) { + if (bu[i]) score += 0.35 * binSeen[i]; + } + + return score; + } + + string chooseBestDirect(int si, int sj, int ti, int tj) const { + string p1 = makeDirect(si, sj, ti, tj, true); + string p2 = makeDirect(si, sj, ti, tj, false); + + double c1 = estimatePathCost(si, sj, p1); + double c2 = estimatePathCost(si, sj, p2); + + return (c1 <= c2 ? p1 : p2); + } + + string chooseExplorationPath(int si, int sj, int ti, int tj) { + string p1 = makeDirect(si, sj, ti, tj, true); + string p2 = makeDirect(si, sj, ti, tj, false); + + double c1 = estimatePathCost(si, sj, p1); + double c2 = estimatePathCost(si, sj, p2); + + double s1 = explorationSeenScore(si, sj, p1); + double s2 = explorationSeenScore(si, sj, p2); + + double bonus = 1500.0 * (1.0 - turn / double(EXPLORE_TURNS)); + + double v1 = c1 + bonus * s1; + double v2 = c2 + bonus * s2; + + if (abs(v1 - v2) < 1e-9) { + return (rng() & 1) ? p1 : p2; + } + return (v1 <= v2 ? p1 : p2); + } + + string bestCorridorPath(int si, int sj, int ti, int tj) const { + string best; + double bestCost = 1e100; + + auto consider = [&](const string& p) { + if (!validatePath(si, sj, ti, tj, p)) return; + + double c = estimatePathCost(si, sj, p); + if (c < bestCost - 1e-9 || + (abs(c - bestCost) < 1e-9 && (best.empty() || p.size() < best.size()))) { + bestCost = c; + best = p; + } + }; + + consider(makeDirect(si, sj, ti, tj, true)); + consider(makeDirect(si, sj, ti, tj, false)); + + for (int r = 0; r < N; r++) { + consider(makeViaRow(si, sj, ti, tj, r)); + } + for (int c = 0; c < N; c++) { + consider(makeViaCol(si, sj, ti, tj, c)); + } + + if (best.empty()) best = chooseBestDirect(si, sj, ti, tj); + return best; + } + + string dijkstra(int si, int sj, int ti, int tj, bool useFine, bool useEdge) const { + int S = si * N + sj; + int T = ti * N + tj; + + const double INF = 1e100; + vector dist(N * N, INF); + vector pre(N * N, -1); + vector pmove(N * N, 0); + + using P = pair; + priority_queue, greater

> pq; + + dist[S] = 0.0; + pre[S] = S; + pq.push({0.0, S}); + + while (!pq.empty()) { + auto [d, u] = pq.top(); + pq.pop(); + + if (d > dist[u] + 1e-9) continue; + if (u == T) break; + + int r = u / N; + int c = u % N; + + int dirs[4]; + bool used[4] = {}; + int m = 0; + + auto addDir = [&](int x) { + if (!used[x]) { + used[x] = true; + dirs[m++] = x; + } + }; + + if (ti < r) addDir(0); + if (ti > r) addDir(1); + if (tj < c) addDir(2); + if (tj > c) addDir(3); + for (int x = 0; x < 4; x++) addDir(x); + + for (int idx = 0; idx < 4; idx++) { + int dir = dirs[idx]; + + int nr = r, nc = c; + char mv = '?'; + double w = 0.0; + + if (dir == 0) { + if (r == 0) continue; + nr = r - 1; + mv = 'U'; + if (!useFine) w = vCostNoFine(r - 1, c); + else if (!useEdge) w = vCostNoEdge(r - 1, c); + else w = vCost(r - 1, c); + } else if (dir == 1) { + if (r == N - 1) continue; + nr = r + 1; + mv = 'D'; + if (!useFine) w = vCostNoFine(r, c); + else if (!useEdge) w = vCostNoEdge(r, c); + else w = vCost(r, c); + } else if (dir == 2) { + if (c == 0) continue; + nc = c - 1; + mv = 'L'; + if (!useFine) w = hCostNoFine(r, c - 1); + else if (!useEdge) w = hCostNoEdge(r, c - 1); + else w = hCost(r, c - 1); + } else { + if (c == N - 1) continue; + nc = c + 1; + mv = 'R'; + if (!useFine) w = hCostNoFine(r, c); + else if (!useEdge) w = hCostNoEdge(r, c); + else w = hCost(r, c); + } + + int v = nr * N + nc; + double nd = d + w; + + if (nd + 1e-9 < dist[v]) { + dist[v] = nd; + pre[v] = u; + pmove[v] = mv; + pq.push({nd, v}); + } + } + } + + if (pre[T] == -1) return ""; + + string path; + int cur = T; + while (cur != S) { + path.push_back(pmove[cur]); + cur = pre[cur]; + if (cur < 0) return ""; + } + + reverse(path.begin(), path.end()); + return path; + } + + string choosePath(int si, int sj, int ti, int tj) { + if (turn < EXPLORE_TURNS) { + return chooseExplorationPath(si, sj, ti, tj); + } + + string safe = bestCorridorPath(si, sj, ti, tj); + string p = dijkstra(si, sj, ti, tj, true, true); + + if (p.empty() || !validatePath(si, sj, ti, tj, p)) { + return safe; + } + + if (turn >= 620 && edgeWeight() > 1e-9) { + string pne = dijkstra(si, sj, ti, tj, true, false); + + if (!pne.empty() && validatePath(si, sj, ti, tj, pne)) { + double cfP = estimatePathCost(si, sj, p); + double ceP = estimatePathCostNoEdge(si, sj, p); + double cfN = estimatePathCost(si, sj, pne); + double ceN = estimatePathCostNoEdge(si, sj, pne); + + double trustEdge = clampd(0.65 + 0.25 * edgeValidationConf, 0.70, 0.90); + double scoreP = trustEdge * cfP + (1.0 - trustEdge) * ceP; + double scoreN = trustEdge * cfN + (1.0 - trustEdge) * ceN; + + if (scoreN < scoreP * 0.998 && (int)pne.size() <= (int)p.size() + 35) { + p = pne; + } + } + } + + if (fineReady && turn >= 380) { + string pn = dijkstra(si, sj, ti, tj, false, false); + + if (!pn.empty() && validatePath(si, sj, ti, tj, pn)) { + double cfP = estimatePathCost(si, sj, p); + double csP = estimatePathCostNoFine(si, sj, p); + double cfN = estimatePathCost(si, sj, pn); + double csN = estimatePathCostNoFine(si, sj, pn); + + double trustFine = clampd(0.70 + 0.22 * fineValidationConf, 0.80, 0.92); + + double scoreP = trustFine * cfP + (1.0 - trustFine) * csP; + double scoreN = trustFine * cfN + (1.0 - trustFine) * csN; + + if (scoreN < scoreP * 0.997 && (int)pn.size() <= (int)p.size() + 40) { + p = pn; + } + } + } + + int manhattan = abs(si - ti) + abs(sj - tj); + + int extraLimit; + if (turn < 200) extraLimit = 12; + else if (turn < 350) extraLimit = 25; + else if (turn < 500) extraLimit = 45; + else if (turn < 700) extraLimit = 65; + else extraLimit = 80 + int(20.0 * segGlobalConf); + + double cd = estimatePathCost(si, sj, p); + double cs = estimatePathCost(si, sj, safe); + + int extra = int(p.size()) - manhattan; + + if (extra > extraLimit) { + if ((int)p.size() > manhattan + 130) return safe; + if (cd > cs * 0.82) return safe; + } + + int lenDiff = int(p.size()) - int(safe.size()); + if (lenDiff > 30) { + double requiredRatio = 1.0 - min(0.08, 0.001 * lenDiff); + if (cd > cs * requiredRatio) return safe; + } + + return p; + } + + void updateFineValidation(int si, int sj, const string& path, int result) { + if (!fineReady || turn < 350 || result <= 0) return; + + double ps = estimatePathCostNoFine(si, sj, path); + double pf = estimatePathCostNoEdge(si, sj, path); + + double denom = max(1.0, double(result)); + double es = (ps - result) / denom; + double ef = (pf - result) / denom; + + double es2 = min(0.25, es * es); + double ef2 = min(0.25, ef * ef); + + fineValidationWindow.push_back({es2, ef2}); + if ((int)fineValidationWindow.size() > 220) fineValidationWindow.pop_front(); + + if ((int)fineValidationWindow.size() < 120) { + fineValidationConf = 1.0; + return; + } + + double ss = 0.0; + double sf = 0.0; + for (auto [a, b] : fineValidationWindow) { + ss += a; + sf += b; + } + + ss /= fineValidationWindow.size(); + sf /= fineValidationWindow.size(); + + double worse = sf - ss; + + if (worse <= 0.0008) { + fineValidationConf = 1.0; + } else { + fineValidationConf = clampd(1.0 - (worse - 0.0008) / 0.0030 * 0.45, 0.55, 1.0); + } + } + + void updateEdgeValidation(int si, int sj, const string& path, int result) { + if (turn < 600 || result <= 0) return; + + double ps = estimatePathCostNoEdge(si, sj, path); + double pe = estimatePathCost(si, sj, path); + + double denom = max(1.0, double(result)); + double es = (ps - result) / denom; + double ee = (pe - result) / denom; + + double es2 = min(0.25, es * es); + double ee2 = min(0.25, ee * ee); + + edgeValidationWindow.push_back({es2, ee2}); + if ((int)edgeValidationWindow.size() > 180) edgeValidationWindow.pop_front(); + + if ((int)edgeValidationWindow.size() < 90) { + edgeValidationConf = 1.0; + return; + } + + double ss = 0.0; + double se = 0.0; + for (auto [a, b] : edgeValidationWindow) { + ss += a; + se += b; + } + + ss /= edgeValidationWindow.size(); + se /= edgeValidationWindow.size(); + + double worse = se - ss; + + if (worse <= 0.0006) { + edgeValidationConf = 1.0; + } else { + edgeValidationConf = clampd(1.0 - (worse - 0.0006) / 0.0025 * 0.50, 0.50, 1.0); + } + } + + void updateEdgeResidual(int si, int sj, const string& path, int result, const vector& edgeList) { + if (turn < 300 || result <= 0 || path.empty()) return; + + double pred = estimatePathCostNoEdge(si, sj, path); + double residual = result - pred; + + double per = residual / double(path.size()); + per = clampd(per, -2200.0, 2200.0); + + double prog = clampd((turn - 300) / 500.0, 0.0, 1.0); + double lr = 0.10 + 0.08 * prog; + + for (int e : edgeList) { + edgeLearnSeen[e]++; + int c = max(1, edgeLearnSeen[e]); + double rate = lr / sqrt(1.0 + 0.04 * c); + edgeDelta[e] += rate * per; + edgeDelta[e] = clampd(edgeDelta[e], -1800.0, 1800.0); + } + } + + void updateModelsWithResult(int si, int sj, const string& path, int result) { + updateFineValidation(si, sj, path, result); + updateEdgeValidation(si, sj, path, result); + + Observation ob; + ob.result = result; + ob.steps = (int)path.size(); + + array lineCnt{}; + array binCnt{}; + array fineCnt{}; + + vector edgeList; + edgeList.reserve(path.size()); + + int r = si; + int c = sj; + + for (char ch : path) { + if (ch == 'R') { + int line = r; + int pos = c; + int bv = hVar(r, c); + int fv = fineHVar(r, c); + int ev = hEdge(r, c); + + lineCnt[line]++; + binCnt[bv]++; + fineCnt[fv]++; + edgeSeen[ev]++; + edgeList.push_back(ev); + + ob.mask[line] |= (1u << pos); + c++; + } else if (ch == 'L') { + c--; + + int line = r; + int pos = c; + int bv = hVar(r, c); + int fv = fineHVar(r, c); + int ev = hEdge(r, c); + + lineCnt[line]++; + binCnt[bv]++; + fineCnt[fv]++; + edgeSeen[ev]++; + edgeList.push_back(ev); + + ob.mask[line] |= (1u << pos); + } else if (ch == 'D') { + int line = N + c; + int pos = r; + int bv = vVar(r, c); + int fv = fineVVar(r, c); + int ev = vEdge(r, c); + + lineCnt[line]++; + binCnt[bv]++; + fineCnt[fv]++; + edgeSeen[ev]++; + edgeList.push_back(ev); + + ob.mask[line] |= (1u << pos); + r++; + } else if (ch == 'U') { + r--; + + int line = N + c; + int pos = r; + int bv = vVar(r, c); + int fv = fineVVar(r, c); + int ev = vEdge(r, c); + + lineCnt[line]++; + binCnt[bv]++; + fineCnt[fv]++; + edgeSeen[ev]++; + edgeList.push_back(ev); + + ob.mask[line] |= (1u << pos); + } + } + + vector> lf; + vector> bf; + vector> ff; + + for (int i = 0; i < LINE_VARS; i++) { + if (lineCnt[i] > 0) { + lf.push_back({i, lineCnt[i]}); + lineSeen[i]++; + } + } + + for (int i = 0; i < BIN_VARS; i++) { + if (binCnt[i] > 0) { + bf.push_back({i, binCnt[i]}); + binSeen[i]++; + } + } + + for (int i = 0; i < FINE_VARS; i++) { + if (fineCnt[i] > 0) { + ff.push_back({i, fineCnt[i]}); + fineSeen[i]++; + } + } + + lineModel.addObservation(lf, ob.steps, result); + binModel.addObservation(bf, ob.steps, result); + fineModel.addObservation(ff, ob.steps, result); + + observations.push_back(ob); + + updateEdgeResidual(si, sj, path, result, edgeList); + } + + double predictObsSegment(const Observation& ob) const { + double sum = 0.0; + + for (int line = 0; line < LINE_VARS; line++) { + uint32_t m = ob.mask[line]; + if (!m) continue; + + int total = popcnt(m); + int x = split[line]; + int cl = popcnt(m & lowMask(x)); + int cr = total - cl; + + sum += segEst[2 * line] * cl; + sum += segEst[2 * line + 1] * cr; + } + + return sum / ob.steps; + } + + void seedSplitsFromBin() { + static const int bounds[3] = {8, 15, 22}; + + for (int line = 0; line < LINE_VARS; line++) { + if (lineSeen[line] < 2) continue; + + double val[BINS]; + + for (int q = 0; q < BINS; q++) { + int idx; + if (line < N) idx = line * BINS + q; + else idx = HBIN_VARS + (line - N) * BINS + q; + + val[q] = lineEst[line] + binDelta[idx]; + } + + double bestDiff = 0.0; + int bestBoundary = split[line]; + + for (int q = 0; q + 1 < BINS; q++) { + double d = abs(val[q + 1] - val[q]); + if (d > bestDiff) { + bestDiff = d; + bestBoundary = bounds[q]; + } + } + + double segDiff = abs(segEst[2 * line] - segEst[2 * line + 1]); + + if (bestDiff > 450.0 && + (segDiff < 700.0 || observations.size() < 180)) { + split[line] = bestBoundary; + } + } + } + + void solveSegmentRidge(double lambda) { + RidgeModel model(SEG_VARS); + segSupport.fill(0); + + for (const auto& ob : observations) { + vector> f; + f.reserve(16); + + for (int line = 0; line < LINE_VARS; line++) { + uint32_t m = ob.mask[line]; + if (!m) continue; + + int total = popcnt(m); + int x = split[line]; + int cl = popcnt(m & lowMask(x)); + int cr = total - cl; + + if (cl > 0) { + f.push_back({2 * line, cl}); + segSupport[2 * line] += cl; + } + if (cr > 0) { + f.push_back({2 * line + 1, cr}); + segSupport[2 * line + 1] += cr; + } + } + + model.addObservation(f, ob.steps, ob.result); + } + + vector prior(SEG_VARS); + for (int i = 0; i < SEG_VARS; i++) { + prior[i] = lineEst[i / 2]; + } + + model.solve(prior, lambda, segEst); + } + + void refineSplits() { + int m = (int)observations.size(); + if (m < 60) return; + + vector pred(m); + vector target(m); + + for (int i = 0; i < m; i++) { + pred[i] = predictObsSegment(observations[i]); + target[i] = observations[i].result / double(observations[i].steps); + } + + for (int line = 0; line < LINE_VARS; line++) { + double v0 = segEst[2 * line]; + double v1 = segEst[2 * line + 1]; + + if (abs(v0 - v1) < 250.0) continue; + + int active = 0; + for (const auto& ob : observations) { + if (ob.mask[line]) active++; + } + + if (active < 8) continue; + + int oldX = split[line]; + uint32_t oldMask = lowMask(oldX); + + double bestSSE = 1e100; + double oldSSE = 1e100; + int bestX = oldX; + + for (int x = 1; x <= 28; x++) { + uint32_t xm = lowMask(x); + double sse = 0.0; + + for (int idx = 0; idx < m; idx++) { + const auto& ob = observations[idx]; + uint32_t mask = ob.mask[line]; + if (!mask) continue; + + int total = popcnt(mask); + + int oldLeft = popcnt(mask & oldMask); + double oldContr = (v0 * oldLeft + v1 * (total - oldLeft)) / ob.steps; + + double baseResidualTarget = target[idx] - pred[idx] + oldContr; + + int newLeft = popcnt(mask & xm); + double newContr = (v0 * newLeft + v1 * (total - newLeft)) / ob.steps; + + double res = baseResidualTarget - newContr; + sse += res * res; + } + + if (x == oldX) oldSSE = sse; + + if (sse < bestSSE) { + bestSSE = sse; + bestX = x; + } + } + + double threshold = max(20000.0, 0.001 * oldSSE); + + if (bestX != oldX && bestSSE + threshold < oldSSE) { + uint32_t newMask = lowMask(bestX); + + for (int idx = 0; idx < m; idx++) { + const auto& ob = observations[idx]; + uint32_t mask = ob.mask[line]; + if (!mask) continue; + + int total = popcnt(mask); + + int oldLeft = popcnt(mask & oldMask); + double oldContr = (v0 * oldLeft + v1 * (total - oldLeft)) / ob.steps; + + int newLeft = popcnt(mask & newMask); + double newContr = (v0 * newLeft + v1 * (total - newLeft)) / ob.steps; + + pred[idx] += newContr - oldContr; + } + + split[line] = bestX; + } + } + } + + void updateSegmentConfidence() { + int strong = 0; + int usable = 0; + double sumConf = 0.0; + + for (int line = 0; line < LINE_VARS; line++) { + if (lineSeen[line] < 3) continue; + if (segSupport[2 * line] < 6) continue; + if (segSupport[2 * line + 1] < 6) continue; + + usable++; + + double d = abs(segEst[2 * line] - segEst[2 * line + 1]); + + if (d > 900.0) strong++; + sumConf += clampd((d - 500.0) / 1500.0, 0.0, 1.0); + } + + double c1 = clampd((strong - 4) / 18.0, 0.0, 1.0); + + double c2 = 0.0; + if (usable > 0) { + double avg = sumConf / usable; + c2 = clampd((avg - 0.12) / 0.38, 0.0, 1.0); + } + + segGlobalConf = max(c1, 0.8 * c2); + } + + double priorFineValue(int line, int q) const { + double sum = 0.0; + int cnt = 0; + + for (int pos = 0; pos < N - 1; pos++) { + int qq = pos * FINE_BINS / (N - 1); + if (qq != q) continue; + + if (line < N) { + sum += rawStructH(line, pos); + } else { + int col = line - N; + sum += rawStructV(pos, col); + } + cnt++; + } + + if (cnt == 0) return lineEst[line]; + return sum / cnt; + } + + void solveFineModel(int observationsCount) { + vector prior(FINE_VARS, 5000.0); + + for (int line = 0; line < LINE_VARS; line++) { + for (int q = 0; q < FINE_BINS; q++) { + int idx; + if (line < N) idx = line * FINE_BINS + q; + else idx = HFINE_VARS + (line - N) * FINE_BINS + q; + + prior[idx] = priorFineValue(line, q); + } + } + + double prog = min(1.0, observationsCount / 900.0); + double lambdaFine = 1.05 - 0.45 * prog; + + vector absFine; + fineModel.solve(prior, lambdaFine, absFine); + + for (int i = 0; i < FINE_VARS; i++) { + fineDelta[i] = clampd(absFine[i] - prior[i], -1300.0, 1300.0); + } + + fineReady = true; + } + + void solveModels(int observationsCount) { + vector priorLine(LINE_VARS, 5000.0); + lineModel.solve(priorLine, 0.1, lineEst); + + int binPeriod = (observationsCount < 300 ? 10 : 20); + if (observationsCount % binPeriod == 0) { + vector priorBin(BIN_VARS); + + for (int i = 0; i < N; i++) { + for (int q = 0; q < BINS; q++) { + priorBin[i * BINS + q] = lineEst[i]; + } + } + + for (int j = 0; j < N; j++) { + for (int q = 0; q < BINS; q++) { + priorBin[HBIN_VARS + j * BINS + q] = lineEst[N + j]; + } + } + + double prog = min(1.0, observationsCount / 800.0); + double lambdaBin = 1.5 - prog; + + vector absBin; + binModel.solve(priorBin, lambdaBin, absBin); + + for (int i = 0; i < BIN_VARS; i++) { + binDelta[i] = absBin[i] - priorBin[i]; + } + } + + if (observationsCount >= 80) { + int segPeriod = (observationsCount < 300 ? 20 : 30); + + if (observationsCount % segPeriod == 0) { + seedSplitsFromBin(); + + double prog = min(1.0, observationsCount / 900.0); + double lambdaSeg = 0.9 - 0.55 * prog; + + solveSegmentRidge(lambdaSeg); + refineSplits(); + solveSegmentRidge(lambdaSeg); + updateSegmentConfidence(); + } + } + + if (observationsCount >= 280 && (observationsCount - 280) % 80 == 0) { + solveFineModel(observationsCount); + } + } +}; + +int main() { + ios::sync_with_stdio(false); + cin.tie(nullptr); + + Solver solver; + + for (int k = 0; k < 1000; k++) { + int si, sj, ti, tj; + if (!(cin >> si >> sj >> ti >> tj)) return 0; + + solver.turn = k; + + string path = solver.choosePath(si, sj, ti, tj); + + cout << path << '\n' << flush; + + int result; + if (!(cin >> result)) return 0; + + solver.updateModelsWithResult(si, sj, path, result); + + if (k + 1 < 1000) { + solver.solveModels(k + 1); + } + } + + return 0; +} diff --git a/tests/judge/test_reuse_containers.py b/tests/judge/test_reuse_containers.py new file mode 100644 index 00000000..acd583be --- /dev/null +++ b/tests/judge/test_reuse_containers.py @@ -0,0 +1,122 @@ +from collections.abc import Generator +from pathlib import Path + +import pytest + +import ale_bench +import ale_bench.constants +from ale_bench.code_language import CodeLanguage, JudgeVersion +from ale_bench.data import ProblemType +from ale_bench.result import CaseResult, JudgeResult +from ale_bench.session import Session +from ale_bench.tool_wrappers import run_cases + + +@pytest.mark.docker +class TestReuseContainers: + CODES_ROOT = Path(__file__).resolve().parent / "codes" + INPUTS_ROOT = Path(__file__).resolve().parent / "inputs" + CODE_LANGUAGE = CodeLanguage.CPP20 + JUDGE_VERSION = JudgeVersion.V202301 + SCORE_RELATIVE_TOLERANCE = 0.005 + + @pytest.fixture(scope="class") + def ahc001_session(self) -> Generator[Session, None, None]: + session = ale_bench.start("ahc001", lite_version=False) + try: + yield session + finally: + session.close() + + @pytest.fixture(scope="class") + def ahc003_session(self) -> Generator[Session, None, None]: + session = ale_bench.start("ahc003", lite_version=False) + try: + yield session + finally: + session.close() + + @pytest.fixture(scope="class") + def inputs(self) -> dict[str, str]: + return { + "ahc001": (self.INPUTS_ROOT / "ahc001.txt").read_text(), + "ahc003": (self.INPUTS_ROOT / "ahc003.txt").read_text(), + } + + @pytest.fixture(scope="class") + def ac_codes(self) -> dict[str, str]: + return { + "ahc001": (self.CODES_ROOT / "ac_cpp20_ahc001.cpp").read_text(), + "ahc003": (self.CODES_ROOT / "ac_cpp20_ahc003.cpp").read_text(), + } + + @classmethod + def assert_scores_close(cls, default_results: list[CaseResult], reused_results: list[CaseResult]) -> None: + for default_result, reused_result in zip(default_results, reused_results, strict=True): + assert default_result.absolute_score > ale_bench.constants.REJECTED_ABSOLUTE_SCORE + assert reused_result.absolute_score > ale_bench.constants.REJECTED_ABSOLUTE_SCORE + score_denominator = max(default_result.absolute_score, reused_result.absolute_score) + relative_diff = abs(default_result.absolute_score - reused_result.absolute_score) / score_denominator + assert relative_diff <= cls.SCORE_RELATIVE_TOLERANCE + + def test_reuse_containers_matches_default_batch( + self, + inputs: dict[str, str], + ac_codes: dict[str, str], + ahc001_session: Session, + ) -> None: + input_str = inputs["ahc001"] + common_kwargs = { + "inputs": [input_str, input_str], + "code": ac_codes["ahc001"], + "code_language": self.CODE_LANGUAGE, + "judge_version": self.JUDGE_VERSION, + "time_limit": 5.0, + "memory_limit": 256 * 1024 * 1024, + "problem_id": "ahc001", + "problem_type": ProblemType.BATCH, + "tool_dir": ahc001_session.tool_dir, + "return_details": True, + "skip_local_visualization": False, + "num_workers": 2, + } + + default_results = run_cases(**common_kwargs, reuse_containers=False) + reused_results = run_cases(**common_kwargs, reuse_containers=True) + + assert [result.judge_result for result in default_results] == [JudgeResult.ACCEPTED, JudgeResult.ACCEPTED] + assert [result.judge_result for result in reused_results] == [JudgeResult.ACCEPTED, JudgeResult.ACCEPTED] + self.assert_scores_close(default_results, reused_results) + assert all(result.local_visualization is not None for result in default_results) + assert all(result.local_visualization is not None for result in reused_results) + + def test_reuse_containers_matches_default_reactive( + self, + inputs: dict[str, str], + ac_codes: dict[str, str], + ahc003_session: Session, + ) -> None: + input_str = inputs["ahc003"] + common_kwargs = { + "inputs": [input_str, input_str], + "code": ac_codes["ahc003"], + "code_language": self.CODE_LANGUAGE, + "judge_version": self.JUDGE_VERSION, + "time_limit": 2.0, + "memory_limit": 256 * 1024 * 1024, + "problem_id": "ahc003", + "problem_type": ProblemType.REACTIVE, + "tool_dir": ahc003_session.tool_dir, + "return_details": False, + "skip_local_visualization": False, + "num_workers": 2, + } + + default_results = run_cases(**common_kwargs, reuse_containers=False) + reused_results = run_cases(**common_kwargs, reuse_containers=True) + + assert [result.judge_result for result in default_results] == [JudgeResult.ACCEPTED, JudgeResult.ACCEPTED] + assert [result.judge_result for result in reused_results] == [JudgeResult.ACCEPTED, JudgeResult.ACCEPTED] + self.assert_scores_close(default_results, reused_results) + assert all(result.local_visualization is not None for result in default_results) + assert all(result.local_visualization is not None for result in reused_results) diff --git a/tests/test_session.py b/tests/test_session.py index ffdfd008..af4e5588 100644 --- a/tests/test_session.py +++ b/tests/test_session.py @@ -193,6 +193,137 @@ def test_code_run( (utc_now - dummy_session.session_started_at).total_seconds() ) + def test_case_eval_forwards_reuse_containers( + self, + dummy_session: Session, + mocker: MockerFixture, + ) -> None: + run_cases_mock = mocker.patch( + "ale_bench.session.run_cases", + return_value=[ + CaseResult( + judge_result=JudgeResult.ACCEPTED, + message="", + absolute_score=100, + execution_time=1.0, + memory_usage=1024, + ) + ], + ) + + dummy_session.case_eval( + input_str="dummy input", + code="dummy code", + code_language="rust", + reuse_containers=True, + ) + + assert run_cases_mock.call_args.kwargs["reuse_containers"] is True + + def test_case_gen_eval_forwards_reuse_containers( + self, + dummy_session: Session, + mocker: MockerFixture, + ) -> None: + case_eval_mock = mocker.patch.object( + dummy_session, + "case_eval", + return_value=Result( + allow_score_non_ac=True, + resource_usage=ResourceUsage(), + case_results=[ + CaseResult( + judge_result=JudgeResult.ACCEPTED, + message="", + absolute_score=100, + execution_time=1.0, + memory_usage=1024, + ) + ], + ), + ) + + dummy_session.case_gen_eval( + code="dummy code", + code_language="rust", + seed=[0, 1, 2], + reuse_containers=True, + ) + + assert case_eval_mock.call_args.kwargs["reuse_containers"] is True + + def test_public_eval_forwards_reuse_containers( + self, + dummy_session: Session, + mocker: MockerFixture, + ) -> None: + run_cases_mock = mocker.patch( + "ale_bench.session.run_cases", + return_value=[ + CaseResult( + judge_result=JudgeResult.ACCEPTED, + message="", + absolute_score=100, + execution_time=1.0, + memory_usage=1024, + ), + CaseResult( + judge_result=JudgeResult.ACCEPTED, + message="", + absolute_score=100, + execution_time=1.0, + memory_usage=1024, + ), + CaseResult( + judge_result=JudgeResult.ACCEPTED, + message="", + absolute_score=100, + execution_time=1.0, + memory_usage=1024, + ), + ], + ) + + dummy_session.public_eval(code="dummy code", code_language="rust", reuse_containers=True) + + assert run_cases_mock.call_args.kwargs["reuse_containers"] is True + + def test_private_eval_forwards_reuse_containers( + self, + dummy_session: Session, + mocker: MockerFixture, + ) -> None: + run_cases_mock = mocker.patch( + "ale_bench.session.run_cases", + return_value=[ + CaseResult( + judge_result=JudgeResult.ACCEPTED, + message="", + absolute_score=100, + execution_time=1.0, + memory_usage=1024, + ), + CaseResult( + judge_result=JudgeResult.ACCEPTED, + message="", + absolute_score=100, + execution_time=1.0, + memory_usage=1024, + ), + CaseResult( + judge_result=JudgeResult.ACCEPTED, + message="", + absolute_score=100, + execution_time=1.0, + memory_usage=1024, + ), + ], + ) + + dummy_session.private_eval(code="dummy code", code_language="rust", reuse_containers=True) + + assert run_cases_mock.call_args.kwargs["reuse_containers"] is True + @pytest.mark.parametrize( ("current_resource_usage", "utc_now", "context"), [ diff --git a/tests/tool_wrappers/test_case_runner.py b/tests/tool_wrappers/test_case_runner.py index 217a6660..76603505 100644 --- a/tests/tool_wrappers/test_case_runner.py +++ b/tests/tool_wrappers/test_case_runner.py @@ -4,7 +4,9 @@ import pytest import ale_bench.constants +import ale_bench.tool_wrappers.case_runner as case_runner_module from ale_bench.code_language import CodeLanguage, JudgeVersion, get_compile_command +from ale_bench.data import ProblemType from ale_bench.result import CaseResult, JudgeResult from ale_bench.tool_wrappers.case_runner import ( HostPathsBatchJudge, @@ -23,12 +25,14 @@ get_reactive_judge_volumes, get_vis_volumes, parse_profiles, + run_cases, setup_paths_batch_judge, setup_paths_batch_run, setup_paths_compile, setup_paths_reactive_judge, setup_paths_vis, ) +from ale_bench.tool_wrappers.reusable_container_pool import REUSABLE_TOOL_TMP_DIR TMP_TEST_DIR = f"{ale_bench.constants.TMP_DIR}/test" TMP_CACHE_DIR = f"{ale_bench.constants.TMP_DIR}/cache" @@ -450,6 +454,23 @@ def test_build_batch_run_command( assert run_command == expected +def test_build_batch_run_command_custom_paths() -> None: + run_command = build_batch_run_command( + CodeLanguage.CPP20, + JudgeVersion.V202301, + 1.0, + input_file="/reuse/input.txt", + output_file="/reuse/output.txt", + profiles_file="/reuse/profiles.json", + ) + assert run_command == ( + "timeout 2.2 prlimit --cpu=2.1 " + f'/usr/bin/time -f "{ale_bench.constants.TIME_OUTPUT_FORMAT}" ' + "-o /reuse/profiles.json " + "./a.out < /reuse/input.txt > /reuse/output.txt; sync" + ) + + def test_setup_paths_batch_judge() -> None: host_paths_run = HostPathsBatchRun( code_file=Path(f"{TMP_TEST_DIR}/code.cpp"), @@ -491,6 +512,14 @@ def test_build_batch_judge_command() -> None: ) +def test_build_batch_judge_command_custom_paths() -> None: + judge_command = build_batch_judge_command( + input_file="/reuse/input.txt", + output_file="/reuse/output.txt", + ) + assert judge_command == f"{ale_bench.constants.TESTER_BIN} /reuse/input.txt /reuse/output.txt" + + @pytest.mark.parametrize( ("problem_id", "case_idx", "input_str", "input_file_name", "output_file_name", "profiles_file_name"), [ @@ -808,6 +837,24 @@ def test_build_reactive_judge_command( assert run_command == expected +def test_build_reactive_judge_command_custom_paths() -> None: + run_command = build_reactive_judge_command( + CodeLanguage.CPP20, + JudgeVersion.V202301, + 1.0, + input_file="/reuse/input.txt", + output_file="/reuse/output.txt", + profiles_file="/reuse/profiles.json", + ) + assert run_command == ( + "timeout 2.2 prlimit --cpu=2.1 " + f"{ale_bench.constants.TESTER_BIN} " + f'/usr/bin/time -f "{ale_bench.constants.TIME_OUTPUT_FORMAT}" ' + "-o /reuse/profiles.json " + "./a.out < /reuse/input.txt > /reuse/output.txt; sync" + ) + + @pytest.mark.parametrize( ("problem_id", "case_idx", "local_visualization_file_name"), [ @@ -973,6 +1020,152 @@ def test_build_vis_command() -> None: ) +def test_build_vis_command_custom_paths() -> None: + vis_command = build_vis_command(input_file="/reuse/input.txt", output_file="/reuse/output.txt") + assert vis_command == f"{ale_bench.constants.VIS_BIN} /reuse/input.txt /reuse/output.txt" + + +def test_run_batch_judge_reusable_container() -> None: + class DummyReusableToolContainerPool(case_runner_module.ReusableToolContainerPool): + def __init__(self) -> None: + self.commands: list[str] = [] + + def run(self, command: str, *, workdir: str = ale_bench.constants.WORK_DIR) -> tuple[float, int, str]: + assert workdir == ale_bench.constants.WORK_DIR + self.commands.append(command) + return 0.01, 0, "Score = 123" + + pool = DummyReusableToolContainerPool() + result = case_runner_module.run_batch_judge_reusable_container( + pool, + "tester input output", + execution_time_host=0.02, + input_str="input", + output_str="output", + error_str="", + ) + + assert result == 123 + assert pool.commands == ["tester input output"] + + +def test_run_vis_reusable_container() -> None: + class DummyReusableToolContainerPool(case_runner_module.ReusableToolContainerPool): + def __init__(self) -> None: + self.calls: list[tuple[str, str]] = [] + + def run(self, command: str, *, workdir: str = ale_bench.constants.WORK_DIR) -> tuple[float, int, str]: + self.calls.append((command, workdir)) + return 0.01, 0, "" + + pool = DummyReusableToolContainerPool() + reusable_input_file = f"{REUSABLE_TOOL_TMP_DIR}/input.txt" + reusable_output_file = f"{REUSABLE_TOOL_TMP_DIR}/output.txt" + reusable_local_visualization_file = f"{REUSABLE_TOOL_TMP_DIR}/ahc001_000000_local_visualization.html" + case_runner_module.run_vis_reusable_container( + pool, + f"vis {reusable_input_file} {reusable_output_file}", + reusable_local_visualization_file, + ale_bench.constants.LOCAL_VIS_HTML, + ) + + assert pool.calls == [ + ( + "timeout 10 bash -c 'rm -f /workdir/vis.html; " + f"vis {reusable_input_file} {reusable_output_file}; " + f"cp /workdir/vis.html {reusable_local_visualization_file}'", + ale_bench.constants.WORK_DIR, + ) + ] + + +def test_run_cases_uses_reusable_container_pools(monkeypatch: pytest.MonkeyPatch) -> None: + created_submission_pools = [] + created_tool_pools = [] + + class DummyReusableSubmissionContainerPool: + def __init__(self, **kwargs: object) -> None: + self.kwargs = kwargs + self.scratch_dir = kwargs["scratch_dir"] + created_submission_pools.append(self) + + def __enter__(self) -> "DummyReusableSubmissionContainerPool": # noqa: PYI034 + return self + + def __exit__(self, exc_type: object, exc_value: object, traceback: object) -> None: + return None + + class DummyReusableToolContainerPool: + def __init__(self, **kwargs: object) -> None: + self.kwargs = kwargs + self.scratch_dir = kwargs["scratch_dir"] + created_tool_pools.append(self) + + def __enter__(self) -> "DummyReusableToolContainerPool": # noqa: PYI034 + return self + + def __exit__(self, exc_type: object, exc_value: object, traceback: object) -> None: + return None + + def fake_case_iter_func(*args: object) -> CaseResult: + assert args[-2] is created_submission_pools[0] + assert args[-1] is created_tool_pools[0] + case_idx = args[4] + input_str = args[5] + assert isinstance(case_idx, int) + assert isinstance(input_str, str) + return CaseResult( + input_str=input_str, + output_str="", + error_str="", + judge_result=JudgeResult.ACCEPTED, + message="", + absolute_score=case_idx, + execution_time=0.0, + memory_usage=0, + ) + + def fake_run_compile_container(*args: object, **kwargs: object) -> None: + return None + + monkeypatch.setattr(case_runner_module, "run_compile_container", fake_run_compile_container) + monkeypatch.setattr( + case_runner_module, + "ReusableSubmissionContainerPool", + DummyReusableSubmissionContainerPool, + ) + monkeypatch.setattr( + case_runner_module, + "ReusableToolContainerPool", + DummyReusableToolContainerPool, + ) + monkeypatch.setattr(case_runner_module, "case_iter_func", fake_case_iter_func) + + case_results = run_cases( + inputs=["input 0", "input 1", "input 2"], + code="print(0)", + code_language=CodeLanguage.PYTHON, + judge_version=JudgeVersion.V202301, + time_limit=1.0, + memory_limit=1024 * 1024 * 1024, + problem_id="ahc001", + problem_type=ProblemType.BATCH, + tool_dir=Path(f"{TMP_CACHE_DIR}"), + return_details=True, + skip_local_visualization=True, + num_workers=5, + reuse_containers=True, + ) + + assert len(created_submission_pools) == 1 + assert len(created_tool_pools) == 1 + assert created_submission_pools[0].kwargs["num_workers"] == 3 + assert created_submission_pools[0].kwargs["problem_type"] == ProblemType.BATCH + assert created_tool_pools[0].kwargs["num_workers"] == 3 + assert created_tool_pools[0].kwargs["scratch_dir"] == created_submission_pools[0].kwargs["scratch_dir"] + assert [case_result.absolute_score for case_result in case_results] == [0, 1, 2] + + sample_profiles_content = """{{ "command": "dummy command", "exit_status": "{}", diff --git a/tests/tool_wrappers/test_reusable_container_pool.py b/tests/tool_wrappers/test_reusable_container_pool.py new file mode 100644 index 00000000..d82acac9 --- /dev/null +++ b/tests/tool_wrappers/test_reusable_container_pool.py @@ -0,0 +1,105 @@ +from pathlib import Path + +import ale_bench.constants +from ale_bench.code_language import CodeLanguage, JudgeVersion +from ale_bench.data import ProblemType +from ale_bench.tool_wrappers.reusable_container_pool import ( + REUSABLE_SUBMISSION_TMP_DIR, + REUSABLE_TOOL_TMP_DIR, + ReusableSubmissionContainerPool, + ReusableToolContainerPool, + get_reusable_submission_volumes, + get_reusable_tool_volumes, +) + +TMP_TEST_DIR = f"{ale_bench.constants.TMP_DIR}/test" +TMP_CACHE_DIR = f"{ale_bench.constants.TMP_DIR}/cache" + + +def test_get_reusable_submission_volumes_batch() -> None: + reusable_volumes = get_reusable_submission_volumes( + Path(TMP_TEST_DIR), + Path(f"{TMP_TEST_DIR}/case-files"), + Path(TMP_CACHE_DIR), + ProblemType.BATCH, + ) + assert reusable_volumes == { + TMP_TEST_DIR: {"bind": ale_bench.constants.WORK_DIR, "mode": "ro"}, + f"{TMP_TEST_DIR}/case-files": { + "bind": REUSABLE_SUBMISSION_TMP_DIR, + "mode": "rw", + }, + } + + +def test_get_reusable_submission_volumes_reactive() -> None: + reusable_volumes = get_reusable_submission_volumes( + Path(TMP_TEST_DIR), + Path(f"{TMP_TEST_DIR}/case-files"), + Path(TMP_CACHE_DIR), + ProblemType.REACTIVE, + ) + assert reusable_volumes == { + TMP_TEST_DIR: {"bind": ale_bench.constants.WORK_DIR, "mode": "ro"}, + f"{TMP_TEST_DIR}/case-files": { + "bind": REUSABLE_SUBMISSION_TMP_DIR, + "mode": "rw", + }, + f"{TMP_CACHE_DIR}/tools/target/release/tester": { + "bind": ale_bench.constants.TESTER_BIN, + "mode": "ro", + }, + } + + +def test_reusable_submission_container_pool_container_path() -> None: + scratch_dir = Path(f"{TMP_TEST_DIR}/case-files") + pool = ReusableSubmissionContainerPool( + code_language=CodeLanguage.PYTHON, + judge_version=JudgeVersion.V202301, + temp_dir=Path(TMP_TEST_DIR), + scratch_dir=scratch_dir, + tool_dir=Path(TMP_CACHE_DIR), + problem_type=ProblemType.BATCH, + num_workers=1, + ) + + assert ( + pool.container_path(scratch_dir / "ahc001_000000_input.txt") + == f"{REUSABLE_SUBMISSION_TMP_DIR}/ahc001_000000_input.txt" + ) + + +def test_get_reusable_tool_volumes() -> None: + reusable_volumes = get_reusable_tool_volumes( + Path(f"{TMP_TEST_DIR}/case-files"), + Path(TMP_CACHE_DIR), + ) + assert reusable_volumes == { + f"{TMP_TEST_DIR}/case-files": { + "bind": REUSABLE_TOOL_TMP_DIR, + "mode": "rw", + }, + f"{TMP_CACHE_DIR}/tools/target/release/tester": { + "bind": ale_bench.constants.TESTER_BIN, + "mode": "ro", + }, + f"{TMP_CACHE_DIR}/tools/target/release/vis": { + "bind": ale_bench.constants.VIS_BIN, + "mode": "ro", + }, + } + + +def test_reusable_tool_container_pool_container_path() -> None: + scratch_dir = Path(f"{TMP_TEST_DIR}/case-files") + pool = ReusableToolContainerPool( + scratch_dir=scratch_dir, + tool_dir=Path(TMP_CACHE_DIR), + num_workers=1, + ) + + assert ( + pool.container_path(scratch_dir / "ahc001_000000_output.txt") + == f"{REUSABLE_TOOL_TMP_DIR}/ahc001_000000_output.txt" + )