Runner refactoring 2/5: Characterization tests for runner/utils.py, run_solver.py, run_benchmarks.py

[eantonini]

This PR locks in current behavior of the runner scripts before the actual refactor splits them into a module package.

• Adds runner/tests/ (conftest.py + test_utils.py + test_run_solver.py + test_run_benchmarks.py), 106 tests covering: calculate_sgm, is_solved, compute_summary_results, allocate_vms_greedy, allocate_benchmarks, load_benchmark_metadata; HighsVariant, is_mip_problem, get_duality_gap, get_reported_runtime, calculate_integrality_violation, get_solver's seed-options wiring; csv_record, write_csv_headers/write_csv_row round-trip, parse_memory, get_conda_package_versions, download_benchmark_file, benchmark_solver's timeout/OOM/error branches.
• Note: several functions the original issue attributed to utils.py (csv_record, parse_memory, get_conda_package_versions, download_benchmark_file, benchmark_solver) actually live in run_benchmarks.py today — tested there instead, matching the real current layout.
• Wires pixi run -e runner pytest runner/tests -v into CI, right after metadata-schema validation.
• Extends the runner pixi feature with matplotlib, numpy, ipython, linopy — needed to import utils.py/run_solver.py/run_benchmarks.py at all (they pull these in at module level even for non-plotting/non-solving functions). None of this is needed by the CLI in production; it's exactly the coupling Refactor PR 3's module split removes, so expect this to shrink back out then.

Notable finding

Characterization testing surfaced a real pre-existing bug: benchmark_solver crashes with an uncaught IndexError (not ValueError) if the wrapped subprocess produces completely empty stderr, since parse_memory does output.splitlines()[-1] with no guard for empty input. Documented as a test locking in current behavior rather than fixed here — out of scope for a characterization-only PR.

Test plan

• pixi run -e runner pytest runner/tests -v — 106 passed
• No production-code diff introduced (only pre-existing bot-formatting noise already on the branch)
• ruff check/ruff format --check clean on new test files

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
solver-benchmark	Ready	Preview	Aug 6, 2026 4:12pm