Skip to content

Pull requests: EleutherAI/lm-evaluation-harness

New pull request New
247 Open 1,802 Closed
247 Open 1,802 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Fix RACE doc_to_text keeping blank marker and dropping the question body
#3716 opened Apr 19, 2026 by Chessing234 Loading…
[fix] Disambiguate cache entries for repeated generate_until requests (#3046)
#3715 opened Apr 17, 2026 by FazeelUsmani Loading…
2 tasks done
1
1
Fix DummyLM.generate_until write_out printing context as gen_kwargs
#3714 opened Apr 17, 2026 by Chessing234 Loading…
Fix infinite loop in ruler qa generate_samples when used_docs cannot shrink
#3713 opened Apr 17, 2026 by Chessing234 Loading…
Fix DummyLM.generate_until printing context as gen_kwargs
#3711 opened Apr 16, 2026 by Chessing234 Loading…
fix: preserve first line in mbpp code extraction
#3710 opened Apr 15, 2026 by yu2001-s Loading…
1
Add diagnostic columns for answer-not-found and invalid-filter tracking
#3709 opened Apr 15, 2026 by fxmarty-amd Contributor Loading…
2 tasks
2
Fix MultiChoiceRegexFilter.find_match IndexError on all-empty capture groups
#3708 opened Apr 15, 2026 by Chessing234 Loading…
Add MolecularIQ chemistry reasoning benchmark task
#3707 opened Apr 15, 2026 by cbartmann Loading…
1
Add OpenSubtitles2024 multi40 task configs and documentation
#3706 opened Apr 15, 2026 by hengyu-luo Loading…
1
Add LICA-Bench: graphic design VLM evaluation (39 tasks, 7 domains)
#3705 opened Apr 15, 2026 by purvanshi Loading…
3 tasks
2
fix: predict all CoQA turn answers instead of only the last turn
#3704 opened Apr 14, 2026 by rahulraj-jhawar-devrev • Draft
1
1
[dev] 0.5 review
#3703 opened Apr 14, 2026 by fxmarty-amd Contributor Loading…
7
Fix BigBench multiple-choice crash on mixed-format tasks
#3702 opened Apr 13, 2026 by Chessing234 Loading…
1
refactor(vllm): remove deprecated vLLM V0 ray code path
#3701 opened Apr 13, 2026 by Anai-Guo Loading…
1
4
fix: don't pass task stop sequences to vLLM for reasoning models
#3700 opened Apr 12, 2026 by jwmacd Loading…
1
Adding Cruxeval
#3699 opened Apr 12, 2026 by ThomasHeap Loading…
5 of 6 tasks
Fix median aggregation returning arbitrary element instead of median
#3696 opened Apr 12, 2026 by Chessing234 Loading…
1 of 2 tasks
Fix acc_all_stderr grouping by question_id only (drops paragraph_id)
#3695 opened Apr 11, 2026 by Chessing234 Loading…
fix(vllm): add ray to vllm extras (#3688)
#3694 opened Apr 10, 2026 by kvr06-ai Loading…
1
4
Fix mmlu_pro fewshot answers leaking into user role under chat template
#3693 opened Apr 9, 2026 by kiwaku Loading…
1
1
[Feat] Add native Tensor Parallelism support for HF backend
#3692 opened Apr 9, 2026 by YangKai0616 Loading…
3
Fix GPQA preprocessing: remove bracket-stripping regex that corrupts answer text
#3691 opened Apr 8, 2026 by Robby955 Loading…
1
1
refactor(5.0)
#3690 opened Apr 8, 2026 by baberabb Contributor Loading…
fix(vllm): guard against None prefix_token_id in tok_encode
#3687 opened Apr 8, 2026 by Darcy-Lee Loading…
1
2
ProTip! Find all pull requests that aren't related to any open issues with -linked:issue.