Questions on the leaderboard

[BradKML]

1. Can the topic of scaffolding be evaluated? According to SWE-Bench, the top 5 scaffolds are TRAE, Refact, Moatless, OpenHands, and Augment (the next one is SWE-Agent, which is made by the same people as SWE-Bench) https://liveswebench.ai/
2. Most open-weight models (DeepSeek v3.1, GLM 4.5, Qwen3) have thinking modes. Would this feature boost ALE-Bench performance?
3. Also, would you consider their latest updates instead of the initial release (assuming we are going by "live" rules and only include those that came out before September 2025)?
4. Since Qwen3 80B-A3B came out, would the models with around 72B parameters be added to the benchmark? Others like LiveBench have done so https://livebench.ai/
5. How is ALE different from LiveCodeBench GSO or METR's RE-Bench? https://livecodebench.github.io/gso.html

[Yuki-Imajuku]

Thank you for your thoughtful questions and interest in our work. We are happy to provide more details on our design philosophy and future plans for ALE-Bench. Here are our responses to your points:

1. Evaluation of Scaffolding

We believe ALE-Bench is well-suited for evaluating both core model capabilities and the effectiveness of different scaffolding techniques. In our paper, we demonstrate this by experimenting with the OpenHands agent and also proposing a simpler, custom scaffolding tailored to the heuristic tasks in ALE-Bench. However, given the vast and rapidly evolving landscape of scaffolding techniques, we chose to standardize the primary leaderboard on a simple yet effective method (self-refinement) to ensure a controlled and fair comparison of the base models. We believe this approach provides a clear baseline, while still allowing for future tracks that focus specifically on scaffolding innovation.

2. Influence of "Thinking Mode"

We are fully aware of the significant impact of reasoning budget on performance. Our qualitative observations, supported by GPT-5 results (w/ reasoning minimal or high) on our leaderboard, indicate a strong correlation between performance and allocated computational budget. As you can likely see from our LLM configs, we generally allocate a substantial reasoning budget to most models to elicit their best possible performance.

3. Addition of New Problems and "Live" Nature

Through our collaboration with AtCoder, Inc., we plan to add new problems to the benchmark. However, we wish to be transparent about the constraints. New official contests are held approximately once or twice a month, which naturally limits the pace at which we can introduce new, high-quality problems.

4. Inclusion of ~72B Parameter Models

Technically, adding these models is straightforward, and we intend to include them. However, our initial attempts to run Qwen3 Next Thinking (80B A3B) via OpenRouter were met with significant stability issues, such as frequent empty responses or non-terminating generation loops. To maintain the integrity and reproducibility of our results, we had to exclude it from the current leaderboard. We expect that as the model's generation becomes more stable and reliable through common inference providers, it will naturally be included. We would be very grateful if you have any recommendations for more reliable inference providers or methods/parameters for this model.

5. How ALE-Bench Differs from LiveCodeBench GSO and RE-Bench

While all three benchmarks focus on enabling AI to handle more long-horizon, practical coding tasks, they target distinct capabilities:

• LiveCodeBench GSO focuses on improving software performance metrics (e.g., execution time, memory usage). The underlying tasks, as we understand them, are still correctness-based, meaning they are solved by passing a fixed set of test cases.
• RE-Bench shares a similar design philosophy with ALE-Bench in terms of tackling complex, multi-step problems. However, the domain expertise differs: RE-Bench is centered on often within the AI domain, while ALE-Bench is specialized in combinatorial optimization.
• ALE-Bench is distinct in that it addresses problems where no single "correct" answer exists, or where finding the optimal solution is computationally intractable in a realistic timeframe (e.g., variants of the Traveling Salesman Problem). Success is measured by the quality of a heuristic solution submitted to a live judge. This open-ended nature affords a high degree of freedom in algorithmic approach (e.g., simulated annealing, beam search, genetic algorithms), making the tasks more akin to applied research.

Furthermore, ALE-Bench offers two unique features:

• Human-Scale Task Duration: The problems are designed for extended human engagement, with contest durations ranging from four hours to two weeks.
• Direct Human-AI Comparison: Our platform allows for a direct and precise comparison of AI performance against a large cohort of human participants (often around 1,000 per contest), including world-class competitive programmers like Psyho (who famously outperformed OpenAI in a past contest). This provides a clear and intuitive measure of how close AI is to reaching, or exceeding, expert human performance on complex problem-solving.

We hope this clarifies our vision for ALE-Bench. We welcome further discussion and appreciate you pushing the community to think deeply about these important issues.

[BradKML]

Forgot some more questions:

1. How are you guys different from MLE-Bench? In fact, to make it more general, do you think algo competitions would likely lead to acceleration in AI design?
2. Since Qwen3-Next and other related open-weight models (for as both "thinking" and "non-thinking") are not yet for the leaderboard, it is safe to assume that the benchmark update would be coming later?

Side observation: I have started a survey on which models are the most prone to tool call issues, and overwhelmingly Qwen3-Coder is the one with the most blame (likely they made the same issue as GPT-OSS with the Harmony Response Format). Will need to look further if they managed to repeat the same issue with Qwen3-Next (or that maybe the new architecture is more floating-point sensitive?) https://github.com/RooCodeInc/Roo-Code/discussions/7590

[Yuki-Imajuku]

For the first question, please refer to the Related Works section in our paper.
For the second one and your side observation, thanks for sharing your thought and experience. I expect it will be irregular, but I plan to add models that catch my interest.

[BradKML]

As a side note @Yuki-Imajuku there are multiple servers getting "brown out" (both BF16 and FP8), the same thing applies to GMICloud, Weights and Biases, SiliconFlow, Parasail, Fireworks etc. at the same time for other models. It is likely some kind of systematic failure on cloud platforms rather than AI service provider, since even some of the direct AI services have the same outage patterns. I do not have a solution yet, other than recommending GPU provider services that avoid certain cloud or DNS providers for the time being. It's very odd.

[Yuki-Imajuku]

True, some providers didn't respond or run stably. So I did specify providers in the config (especially for open source models).
Thanks for detailed explanation!

[BradKML]

I looked into it again, and it is using DeepInfra, which has less of a chance for downtime. https://github.com/SakanaAI/ALE-Bench/blob/main/llm_configs/qwen3-next-thinking.json
I am looking at why: the penalty system is slightly different, and that the old repetition_penalty was probably not the best idea. One of the key concern that as long as there is a way for the model to form a high-confidence sentence loop, it will stick to it. Innovation like XTC (excluding top tokens for maximizing creativity but not effectiveness) and DRY (effective subtle loop prevention which are not implemented by OpenRouter AND not yet tested on coding agents) are developed after the invention of Min-P and the alternative penalties. https://www.reddit.com/r/LocalLLaMA/comments/1g383mq/repetition_penalties_are_terribly_implemented_a/
Since repetition_penalty set to 1.05 might cause paradoxical effects, and big names like DRY might not be universal, presence_penalty and frequency_penalty are often suggested as an alternative (usually set to 0.2 according to people online). https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct#best-practices