Size CI matrices by event to cut GitHub Actions runner minutes

[aaronspring]

Description

A full CI run costs ~425 Linux-equivalent runner minutes across 15 jobs. Measured on run 35582791229 (the green run on main), weighted by GitHub's billing multipliers — macOS 10×, Windows 2×:

Platform	Jobs	Wall min	Weighted min	Share
macOS	2	21.8	218	51%
Windows	2	39.4	79	19%
Linux	11	128.2	128	30%
Total	15	189	425

Four jobs — the macOS and Windows ones — were ~70% of the cost, and they catch essentially nothing Linux does not.

So the matrices are now event-dependent:

• Pull requests run a lean matrix: Linux, Python 3.10 and 3.13.
• The weekly schedule and workflow_dispatch keep the full matrix, unchanged: Linux, macOS and Windows, Python 3.10 through 3.13.
• A pull request that needs the full matrix before merging opts in with the full-ci label. Labels come from the event payload, so label first, then re-run the workflow (or push again).

Measured on this PR's own run (35607142259), all nine jobs on ubuntu:

Job	min
Detect CI trigger (ubuntu-slim)	0.1
Doctests (3.10)	1.3
Notebooks (3.10)	4.9
Test minimum dependencies (3.13, conda)	7.1
Test minimum dependencies (3.10, conda)	11.7
Test minimum dependencies (3.13, pip)	13.9
Test optional dependencies (3.13, conda)	14.3
Test minimum dependencies (3.10, pip)	14.9
Test optional dependencies (3.10, conda)	17.6
Total	85.8

425 → 85.8 weighted minutes, a 80% reduction, and wall-clock 24.1 → 19.3 min, without changing what the weekly run covers.

Alongside that:

• Coverage once, not four times. All four maximum-test-conda entries ran --cov and uploaded to codecov. Codecov on this PR confirms the other three were redundant: 91.27%, 5977 hits / 6548 lines, +/- 0.00% — identical to the base commit's four-upload result. The saving is the three redundant uploads and their report processing; this run shows no measurable runtime difference from the instrumentation itself.
• Tutorial datasets are cached (keyed on tutorial.py) rather than re-downloaded in every job — 20–70 s per job, and one less flaky network step. setup-python now caches pip wheels too; the Windows install alone took 4.5 min. Both caches were cold on this first run, so the benefit lands from the next run onward.
• Every job has a timeout-minutes cap. Without one, a hung macOS job runs to the 6 h default — 3600 weighted minutes from a single hang.
• benchmarks.yml gained the concurrency group the other workflows already had, so superseded runs are cancelled.
• Dead matrix cruft removed in maximum-test-conda: an include: carrying an env value whose only consumer was an "Install bleeding edge" step that could never fire, and a hand-rolled cache-environment-key interpolating env.TODAY and env.CONDA_ENV_FILE, neither of which this workflow defines. setup-micromamba already appends a hash of the environment file, create-args, environment name and OS to the cache key, so the custom prefix contributed nothing.

Two bugs found on the way, both fixed here:

• testpaths pointed at the pre-src-layout climpred/tests. pytest found nothing there, warned, and silently rescanned the whole repository on every run: PytestConfigWarning: No files were found in testpaths; consider removing or adjusting your testpaths configuration. Searching recursively from the current directory instead. Now src/climpred/tests.
• The maximum-test-conda cache key above.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• Refactoring
• This change requires a documentation update

How Has This Been Tested?

• actionlint on all workflows: no new findings. The two it reports (ubuntu-slim unknown to actionlint 1.7.7, an untrusted-input expression in benchmarks.yml) are pre-existing and untouched.

• pre-commit run --all-files: all hooks pass, including check-jsonschema's GitHub workflow schema validation.

• The testpaths change was verified on a minimal reproduction of the layout — the old value does fall back to a full-repo rescan, the new one collects the tests, and src/climpred/conftest.py sits above the testpath so its fixtures are still collected.

• The matrix reduction is verified by this PR's own run: nine jobs, all Linux, all green.

• Tests added for pytest, if necessary. — not applicable, CI configuration only.

Checklist (while developing)

• I have commented my code, particularly in hard-to-understand areas.
• CHANGELOG is updated with reference to this PR.
• AGENTS.md gained a Continuous Integration section describing the lean/full matrix policy and the full-ci label.

Note for the reviewer

Branch protection needs updating before this can merge — see the comment below. CI is green and there is no conflict, but the PR reports mergeable_state: blocked.

One thing deliberately left alone: no conda cache key rotates on a timer, so an unchanged environment file keeps its original solve indefinitely. If the weekly run should re-solve to pick up new dependency releases, that wants a deliberate date-based key across all four conda jobs — a separate change, and one that costs minutes rather than saving them.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DVFyiToC1QjAJdDHecCfDX

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.27%. Comparing base (0569861) to head (8bb7258).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #931   +/-   ##
=======================================
  Coverage   91.27%   91.27%           
=======================================
  Files          59       59           
  Lines        6548     6548           
=======================================
  Hits         5977     5977           
  Misses        571      571           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aaronspring]

CI green; merge blocked by branch protection — needs a settings change

Run 35607142259 is green: 9/9 jobs, all ubuntu. pre-commit.ci ✅, Read the Docs ✅, codecov unchanged at 91.27%. No merge conflict.

The PR nonetheless reports mergeable_state: blocked, and this is almost certainly the branch-protection caveat in the description coming true rather than anything wrong with the diff.

What I can see: main is protected. This PR has zero reviews — but so did #930 when it merged earlier today, which suggests required approving reviews are not what is blocking here. The remaining explanation is required status checks that no longer report, because the job names they pin no longer exist on pull requests:

Required check name (if pinned)	Still runs on a PR?
Test minimum dependencies (Python3.10, macos, pip)	❌ schedule / full-ci only
Test minimum dependencies (Python3.13, macos, pip)	❌
Test minimum dependencies (Python3.10, windows, pip)	❌
Test minimum dependencies (Python3.13, windows, pip)	❌
Test optional dependencies (Python3.11, ubuntu, conda)	❌
Test optional dependencies (Python3.12, ubuntu, conda)	❌

GitHub holds such a check as Expected — waiting for status to be reported, which blocks the merge indefinitely.

What's needed (Settings → Branches → main): drop the platform- and version-specific entries from the required-checks list, keeping the ones that still run on every PR:

• Test minimum dependencies (Python3.10, ubuntu, pip)
• Test minimum dependencies (Python3.13, ubuntu, pip)
• Test minimum dependencies (Python3.10 ubuntu, conda)
• Test minimum dependencies (Python3.13 ubuntu, conda)
• Test optional dependencies (Python3.10, ubuntu, conda)
• Test optional dependencies (Python3.13, ubuntu, conda)
• Doctests (Python3.10, ubuntu)
• Notebooks (Python3.10, ubuntu)

I can't read the protection settings through my access, so I can't confirm which entries are actually pinned, and this is not something a push can fix — restoring those job names would undo the change this PR is for. If the block turns out to be something else entirely (a required review after all, or a rule I can't see), say so and I'll take it from there.

---

Generated by Claude Code