ci(eval): run Muse-Glimmer in nightly BFCL and tau2 - #2280
Draft
hello-alexmcc wants to merge 8 commits into
Draft
Conversation
The BFCL and tau2 harnesses are both A/B: they hold the model fixed, vary the frontend, and report a delta. That answers "did this change move the number", but not "is our serving of this model faithful at all". A delta is blind to a fault both arms share, and a model new enough that no other engine can serve it has no second arm to compare against. Add a single-arm harness that produces an absolute score and holds it against a number the model's publisher put in print, recording the reproduction protocol and known divergences alongside it. The first suite is AIME 2026, the only benchmark on the Muse-Glimmer card that is simultaneously run by the publisher (rather than sourced from a third party), graded deterministically, text-only, small, and scored near ceiling -- which is what makes it sharp enough to catch a broken serving path. Grading reads content only, never reasoning_content, so a reasoning parser that stops separating channels shows up here as a score drop rather than passing silently. Signed-off-by: Alex McC <319643551+hello-alexmcc@users.noreply.github.com>
github-actions
Bot
added
documentation
|
Caution Review failedAn error occurred during the review process. Please try again later. Warning Your free Security trial is over. An organization admin can activate billing to continue. Comment |
Two problems, both found by CI on the previous commit. reference.py and suites.py carried a `#!/usr/bin/env python3` line without the executable bit, which check-shebang-scripts-are-executable rejects. They are libraries, not entry points, so the shebang is what was wrong -- removed there, and run_eval.py is marked executable to match run_ab.py. Separately: nothing in CI ran scripts/tests/ at all. Those guards were dead weight, including test_register_bfcl_model.py, which was written to keep a 38-point BFCL mis-scoring bug from returning and had never once executed. They are pure Python and run in under a second, so the lint job is the right home for them -- no GPU lane needed. Signed-off-by: Alex McC <319643551+hello-alexmcc@users.noreply.github.com>
4 tasks
Signed-off-by: Alex McC <319643551+hello-alexmcc@users.noreply.github.com>
Signed-off-by: Alex McC <319643551+hello-alexmcc@users.noreply.github.com>
Signed-off-by: Alex McC <319643551+hello-alexmcc@users.noreply.github.com>
Signed-off-by: Alex McC <319643551+hello-alexmcc@users.noreply.github.com>
github-actions
Bot
added
tool-parser
hello-alexmcc
changed the title
Signed-off-by: Alex McC <319643551+hello-alexmcc@users.noreply.github.com>
github-actions
Bot
removed
tool-parser
Signed-off-by: Alex McC <319643551+hello-alexmcc@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Problem
Muse-Glimmer needs end-to-end accuracy validation through SMG, but the pinned vLLM cannot load the checkpoint. The first version of this PR added a standalone AIME workflow with separate serving orchestration. That duplicated the repository's established benchmark infrastructure and did not follow the nightly BFCL/tau2 lifecycle, reporting, or artifact conventions.
Solution
Add Muse-Glimmer as a targeted
smg_onlymatrix leg in the existing nightly BFCL and tau2 workflows. SGLang serves raw tokens over the SMG gRPC contract; SMG owns chat-template rendering and both Muse-Glimmer parsers. The existing workflow machinery launches, scores, reports, uploads artifacts, and tears the run down.Changes
muse-glimmerto the existing nightly BFCL and tau2 matrices as ansmg_only, SGLang-backed legscripts/tests/suite in PR CIWhy this is single-arm
The pinned vLLM cannot load
meta-models/Muse-Glimmer-30B. For this matrix leg, SGLang 0.5.18 serves raw tokens over the SMG gRPC contract and SMG owns chat-template rendering plus both Muse-Glimmer parsers. The result is an absolute end-to-end SMG score, not a vLLM-vs-SMG frontend parity claim.The single-arm gate fails on a missing category/domain, timeout, nonzero benchmark subprocess, partial task/trial cardinality, or zero processed samples. A genuine 0% result with nonzero samples remains a valid benchmark result, which is important for diagnosing weak parallel-tool behavior without mislabeling it as broken infrastructure.
Benchmark interpretation
Test Plan
pytest scripts/tests/ -q— 53 passedbash -n, actionlint, andgit diff --checkcargo +nightly fmt --all -- --checkcargo clippy --workspace --all-targets --all-features -- -D warningscargo testLive benchmark results
Targeted Muse-Glimmer BFCL and tau2 dispatches will be linked here after the stacked branch is pushed. The PR will remain draft until both runs settle and the exact scores/configuration are recorded.