Fix Benchmark CI: green without running benchmarks since #489 (asv 0.6.5 vs py-rattler 0.22) - #517
Open
glaziermag wants to merge 2 commits into
Open
Fix Benchmark CI: green without running benchmarks since #489 (asv 0.6.5 vs py-rattler 0.22)#517glaziermag wants to merge 2 commits into
glaziermag wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every
Benchmark / Linuxrun since Feb 4 has passed in ~20s without executing a single benchmark — 95 straight green runs (the first blind ones were #489's own checks). #489 bumped py-rattler to 0.22 inuv.lock, and asv 0.6.5's rattler plugin still callssolve(channels=...), which py-rattler >= 0.22 renamed tosources, so environment creation crashes immediately:The failure check that should turn this red (benchmarks.yml lines 55-61 at
d2857f3) never sees it: the traceback goes to stderr, bypassing the| sed | tee benchmarks.logpipe the grep reads, andteemasks asv's exit status (the step runs without pipefail). Example on main: run 30468384668 — success in 22s with the TypeError in its log. The last genuine runs are from Feb 4 morning, ~11-13 min each (e.g. 21679459393); every run from 21680736982 (Feb 4, 17:03 UTC) onward is blind.Two commits:
set -o pipefail+2>&1, so an asv crash fails the job (and tracebacks get the existing::error::annotation).uv lock -P asv==0.6.6: asv 0.6.6 handles the py-rattler rename, so benchmarks actually run again. (Dependabot's uv-group PRs bump py-rattler but not asv, so Bump the uv group across 1 directory with 8 updates #515 doesn't resolve this.)Verified on my fork:
BENCHMARKS NOT SIGNIFICANTLY CHANGED.I used Claude to find and bisect this; I re-ran the reproductions and read the logs and diffs myself.