Add new models - #37
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds support for additional LLMs/variants in ALE-Bench eval by introducing new model configuration presets and extending fallback pricing, plus a temporary compatibility patch to preserve OpenAI cache-write token accounting.
Changes:
- Monkey-patch pydantic-ai’s OpenAI usage mapping to retain
cache_write_tokensfor newer OpenAI responses. - Extend
FALLBACK_DICTpricing for new models (GPT‑5.6 variants, Claude Sonnet 5, Grok 4.5, HY3). - Add new
llm_configs/*.jsonpresets for the added models/variants.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/ale_bench_eval/safe_generation.py | Adds an import-time monkey patch to preserve OpenAI cache-write usage tokens. |
| src/ale_bench_eval/calc_cost.py | Adds fallback price entries for new models (including tiered pricing and HY3). |
| llm_configs/hy3-high.json | Adds a vLLM Chat config preset for HY3 with “high” reasoning settings. |
| llm_configs/grok-4.5-high.json | Adds an OpenRouter config preset for Grok 4.5 with high reasoning enabled and usage included. |
| llm_configs/gpt-5.6-terra-max.json | Adds an OpenAI config preset for GPT‑5.6 Terra with max reasoning settings. |
| llm_configs/gpt-5.6-sol-max.json | Adds an OpenAI config preset for GPT‑5.6 Sol with max reasoning settings. |
| llm_configs/gpt-5.6-luna-max.json | Adds an OpenAI config preset for GPT‑5.6 Luna with max reasoning settings. |
| llm_configs/claude-sonnet-5-high.json | Adds an Anthropic config preset for Claude Sonnet 5 with high effort thinking. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+60
to
+65
| def patch_openai_cache_write_tokens() -> None: | ||
| """Preserve GPT-5.6 cache-write usage until genai-prices handles the new field.""" | ||
| original_map_usage = pydantic_openai._map_usage # noqa: SLF001 | ||
| if getattr(original_map_usage, _CACHE_WRITE_PATCH_MARKER, False): | ||
| return | ||
|
|
Comment on lines
+77
to
+83
| response_usage = getattr(response, "usage", None) | ||
| if response_usage is None: | ||
| return mapped_usage | ||
|
|
||
| usage_data = response_usage.model_dump(exclude_none=True) | ||
| token_details = usage_data.get("input_tokens_details") or usage_data.get("prompt_tokens_details") or {} | ||
| cache_write_tokens = token_details.get("cache_write_tokens", 0) |
Comment on lines
+9
to
+13
| "extra_body": { | ||
| "chat_template_kwargs": { | ||
| "reasoning_effort": "high" | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.