All issues

Search results

The Atla inference branch calls an argument vllm==0.13.0 does not accept

Status: Open.

#276 In allenai/reward-bench;

· sixijsu77-hub opened on Aug 15, 2026

rewardbench[vllm] cannot import run_generative_v2: google.genai and together are imported at module scope

Status: Open.

#274 In allenai/reward-bench;

· sixijsu77-hub opened on Aug 13, 2026

eval-set-scores: first ten Factuality ids are wrong in 179 of 188 files; scoring protocol is not recorded

Status: Open.

#273 In allenai/reward-bench;

· sixijsu77-hub opened on Aug 13, 2026

Generative scores are not reproducible run to run: candidate placement is unseeded

Status: Open.

#272 In allenai/reward-bench;

· sixijsu77-hub opened on Aug 13, 2026

Multiple-comparison correction for per-subset comparisons (best-of-23 inflates "leads on subset X")

Status: Open.

#270 In allenai/reward-bench;

· ipezygj opened on Jul 21, 2026

Per-category #1 rankings are within noise on their own samples (paired McNemar) — consider showing CIs / significance groups

Status: Open.

#269 In allenai/reward-bench;

· ipezygj opened on Jul 19, 2026

Focus subset is length-confounded: a content-blind "pick the shortest completion" baseline scores 52.7% (2.1× chance, z=14.2)

Status: Open.

#268 In allenai/reward-bench;

· ipezygj opened on Jul 17, 2026

alpacaeval-easy and alpacaeval-hard are length-degenerate: a 0-parameter "pick the longer answer" baseline scores 100%

Status: Open.

#267 In allenai/reward-bench;

· ipezygj opened on Jul 17, 2026

[BOUNTY CLAIM] Calibration diagnostics #263

Status: Open.

#265 In allenai/reward-bench;

· nikos1005 opened on Jun 7, 2026

Adding calibration diagnostics for reward model evaluation.

Status: Open.

#263 In allenai/reward-bench;

· akshan-main opened on Jun 3, 2026

Add Unsloth Fast Inference option

Status: Open.

#251 In allenai/reward-bench;

· natolambert opened on Jun 12, 2025