Repository navigation
CI: zig and zig-windows spend most of their time rerunning tests that can't have changed #2242
Description
Problem
zig and zig-windows take about 18 and 26 minutes on pull requests (p50, 63 runs on 5 Nov), and pull request jobs wait 5 to 10 minutes for a runner at the median and 34 to 39 at p90. Those two jobs are 44 of the 61 job-hours in that window, so they also set the wait for everything else, and landing waits on all of it.
Where the time goes
- Builds start from nothing.
mlugg/setup-zigempties its cache past 2 GiB and saves the empty folder, and both jobs have outgrown that. - Every test reruns. Running the tests takes 10 to 13 minutes even with nothing to compile, so a warm cache alone gains little on Linux. A PR that changes only docs pays the full cost.
- A few binaries dominate. In one cold run,
zig/qwen4_family.zigtook over 6 minutes on Linux and over 3 on Windows. The top 8 binaries were about 70% of test running time. Most test binaries are built in Debug. - Compile-only jobs lose their cache too.
zig-macos-dev-toolstakes 13 minutes and waits longest for a runner. Pull requests save caches that no other pull request can read.
Plan
- CI: zig and zig-windows keep main's build cache, and the quick checks run first #2140: main saves the build cache and pull requests restore it, with the quick checks first.
- The same cache for
release-pathsandzig-macos-dev-tools. - Build the heaviest test binaries in ReleaseSafe, where running them costs more than compiling them.
- Reuse a test's pass when its binary and every non-Zig file it could read are unchanged. In the 190 landings over the 4 days to 5 Nov, 92 touched the sources nearly every test imports, 76 changed other Zig and 22 changed no Zig.
Every check keeps running on main whenever the shared sources change, because main then starts from an empty cache. #2172 measures item 4 and traces which files tests read at run time. A changed fixture must never reuse an old pass.
Activity
test(ci): the heaviest host gates build in ReleaseSafe (ashhart#2242)
ashhart commentedon Nov 5, 2026
Thanks @devopsfortunato. Item 1 landed as #2140. If you're picking up one of items 2 to 4, please say which here, so the work doesn't overlap.
ashhart commentedon Nov 5, 2026
What the probes on #2172 found, with a correction.
Why a restored cache didn't help:
x86_64_v3hit 8 of 8, in 2-7 s instead of 108-263 s. The fix goes into CI: main's build cache fits every runner, and release-paths and macOS use it too #2252 after the 1.2.3 cut..zigsource with the same contents stays cached; a touched.csource recompiles. On Windows that's 13 test compiles and 8 small libraries, about 2.4 minutes of summed compile time, and it reruns those tests, includingqwen4_family. This one stays for now.Correction: I said on #2140 that the misses came from zig's global cache not being restored. That was wrong on CI:
setup-zigalready keeps it inside.zig-cache, so theZIG_GLOBAL_CACHE_DIRline #2140 added does nothing. #2252 removes it.Whole suite on the fixed CPU (run 42038532226):
zig build testThat's one run of each, and the Windows restore landed on a newer CPU than the save, so the ratio is a guide rather than a promise. A pull request that changes the shared sources still recompiles nearly everything.
test(ci): the heaviest host gates build in ReleaseSafe (ashhart#2242)
test(ci): the heaviest host gates build in ReleaseSafe (ashhart#2242)
test(ci): the heaviest host gates build in ReleaseSafe (#2242)
ashhart commentedon Nov 6, 2026
All four items are on main: the build cache on main with the quick checks first (#2140); the same cache for
release-pathsand the macOS dev tools, compiled for one CPU so it hits on every runner (#2252); the heaviest test gates in ReleaseSafe (#2249, thanks @devopsfortunato); and pull requests reusing main's test passes and compiles, C and C++ included (#2274). That last one also covers the C and C++ recompiles we said would stay. Main still checks every commit from scratch. #2311 is the one follow-up still open. If a pull request'szigorzig-windowsjob still reruns tests its diff can't reach, please open a new issue with the run link.ashhart commentedon Nov 8, 2026
Thanks, @devopsfortunato. All four items are on main: the build cache with the quick checks first (#2140), the same cache on every runner (#2252), the heaviest gates in ReleaseSafe (#2249, yours), and pull requests reusing main's test passes (#2274). The follow-up #2311 has landed too. Closing. If a job still reruns tests its diff can't reach, please open a new issue with the run link.