Skip to content

Fix Benchmark CI: green without running benchmarks since #489 (asv 0.6.5 vs py-rattler 0.22) - #517

Open
glaziermag wants to merge 2 commits into
xarray-contrib:mainfrom
glaziermag:benchmark-gate-fix
Open

Fix Benchmark CI: green without running benchmarks since #489 (asv 0.6.5 vs py-rattler 0.22)#517
glaziermag wants to merge 2 commits into
xarray-contrib:mainfrom
glaziermag:benchmark-gate-fix

Conversation

@glaziermag

Copy link
Copy Markdown

Every Benchmark / Linux run since Feb 4 has passed in ~20s without executing a single benchmark — 95 straight green runs (the first blind ones were #489's own checks). #489 bumped py-rattler to 0.22 in uv.lock, and asv 0.6.5's rattler plugin still calls solve(channels=...), which py-rattler >= 0.22 renamed to sources, so environment creation crashes immediately:

TypeError: solve() got an unexpected keyword argument 'channels'
·· Failure creating environment for rattler-py3.12

The failure check that should turn this red (benchmarks.yml lines 55-61 at d2857f3) never sees it: the traceback goes to stderr, bypassing the | sed | tee benchmarks.log pipe the grep reads, and tee masks asv's exit status (the step runs without pipefail). Example on main: run 30468384668 — success in 22s with the TypeError in its log. The last genuine runs are from Feb 4 morning, ~11-13 min each (e.g. 21679459393); every run from 21680736982 (Feb 4, 17:03 UTC) onward is blind.

Two commits:

  1. benchmarks.ymlset -o pipefail + 2>&1, so an asv crash fails the job (and tracebacks get the existing ::error:: annotation).
  2. uv.lockuv lock -P asv==0.6.6: asv 0.6.6 handles the py-rattler rename, so benchmarks actually run again. (Dependabot's uv-group PRs bump py-rattler but not asv, so Bump the uv group across 1 directory with 8 updates #515 doesn't resolve this.)

Verified on my fork:

I used Claude to find and bisect this; I re-ran the reproductions and read the logs and diffs myself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant