Fix LiveCodeBench stdin tests for solutions that read sys.stdin.buffer

[iziz]

Problem

run_test_std in llm_evaluation/livecodebench_util.py replaces sys.stdin with StringIO(test_input). A StringIO has no buffer attribute, so a solution that reads its input through sys.stdin.buffer (the common fast-input idiom data = sys.stdin.buffer.read().split()) raises AttributeError before printing anything, and every stdin test fails even when the solution is correct. The failure is recorded as an ordinary wrong answer, so it does not stand out.

LiveCodeBench's own harness handles this case: since LiveCodeBench/LiveCodeBench@16de2a1 (May 2025), lcb_runner/evaluation/testing_util.py replaces stdin with a mock that has a buffer attribute (MockStdinWithBuffer).

Reproduction

from livecodebench_util import run_test_std

code = "import sys\ndata = sys.stdin.buffer.read().split()\nprint(int(data[0]) + int(data[1]))\n"
print(run_test_std(code, "2 3\n", "5"))
# before: AttributeError: '_io.StringIO' object has no attribute 'buffer'
# after:  (True, '5')

(run_test_std replaces sys.stdout; restore it before printing the result.) sys.stdin.buffer.readline() fails the same way. input(), sys.stdin.read(), sys.stdin.readline() and for line in sys.stdin pass before and after the change.

Impact

LiveCodeBench stdin rows whose last code block reads sys.stdin.buffer, re-scored inside the evaluation image with no network, with the current code and with this patch:

prediction file	rows	pass now	pass with this patch
AsiaInfo-Router	53	0	51
BARouter	50	0	44
lynkr	32	0	30
weave-router	25	0	4
sqwish-fvcta8	11	0	10
PSAI-Auto-Router	9	0	9
cross-router	8	0	3
vllm-sr	8	0	1
agentforge-router	4	0	0
sqwish-router	4	0	0
llm-router	3	0	3
orcarouter-adaptive	3	0	3
Paix2-router	2	0	0
total (main, 04c99fd)	212	0	158
KT-ModelRouter and the other prediction files	0

Open submissions: #228 (dalpha-llm-router) has 42 such rows (0 pass now, 40 with this patch), #232 (divyam-router) has 34 (0 now, 30 with this patch), and #229 (BizRouter) has none.

Regression check: every LiveCodeBench stdin row of KT-ModelRouter (202 rows, none of which reads sys.stdin.buffer) scores the same with and without the patch (178 pass in both), and no row that passes with the current code fails with the patch.

Disclosure: we submitted #229 and #230. #229 has no rows that read sys.stdin.buffer, so this change does not affect its score; it raises the scores of other submissions, including open ones. We found the bug while evaluating a model that reads stdin this way in most of its stdin answers. As with #230, whether and how to re-evaluate existing entries is up to the maintainers.

Fix

Give run_test_std a text stream over a bytes buffer (io.TextIOWrapper(io.BytesIO(...), encoding="utf-8", newline="\n")) instead of a StringIO. Text reads behave as before (newline="\n" keeps StringIO's newline handling), and sys.stdin.buffer reads the same input.

This is independent of #230: the two changes touch different lines of run_test_std and merge cleanly in either order.

Checked with the pinned pre-commit hooks (ruff v0.11.7, ruff-format, codespell v2.4.1, mypy).

[iziz]

Correction to the row counts in "Impact"

The counts in the Impact section included for_optimality rows: the other pool models' answers on the sub_10 questions, which llm_evaluation/run.py uses only for the optimality metrics and excludes from the RouterArena score. Counting scored rows separately (same prediction files at 04c99fd, same per-row results from the evaluation image; nothing was re-run):

prediction file	scored rows	scored rows passing with this patch	for_optimality rows (passing with this patch)
AsiaInfo-Router	53	51	0
BARouter	45	39	5 (5)
lynkr	32	30	0
weave-router	25	4	0
PSAI-Auto-Router	7	7	2 (2)
orcarouter-adaptive	3	3	0
sqwish-fvcta8	1	1	10 (9)
cross-router	0	0	8 (3)
vllm-sr	0	0	8 (1)
agentforge-router	0	0	4 (0)
sqwish-router	0	0	4 (0)
llm-router	0	0	3 (3)
Paix2-router	0	0	2 (0)
total (main, 04c99fd)	166	135	46 (23)

None of these rows passes with the current code.

Open submissions: #228 (dalpha-llm-router) has 30 scored rows, 28 of which pass with this patch; its other 12 rows (all passing) are for_optimality rows, so "42 rows, 40 with this patch" overstated the change to its score. #232 (divyam-router) is unchanged: all 34 rows are scored, and 30 pass with this patch. #229 has none.

The bug, the fix and the regression check are unchanged.