Repository navigation
CI: a pull request reuses main's test passes and C compiles, so it runs only what its diff changed - #2274
Merged
Conversation
4 tasks done
…ns only what its diff changed Zig's build cache could reuse a passing test, but on CI it never did. Zig 0.17's check misses on any stat change to a declared input even when the contents match, and a checkout or a restored cache gives every file a new id. Every test binary, fixture and C or C++ source looked changed. Zig's master has the fix (ziglang/zig#37238), but no release does yet, 0.17.1 included. tools/zig/zig_cache_restat.zig runs before the first build. For each file a manifest records whose contents still hash to the recorded digest, it records the file's id today, so Zig sees nothing to check. A file whose contents changed is left alone and misses as it should. If a Zig update changes the format, the step warns and the job rebuilds and reruns more than it needs to, and the regression test fails. A cached pass says nothing about a file a test opens while it runs, so a trace of every test run found those. Four tests read fixtures, now declared in zig/build/test_reads.zig so that a changed fixture reruns them. The goldens and the q35 launch check read whole folders, so they always run: a directory input hashes only its entries' names. The kernel symbols check always ran. Main builds from nothing and restores no cache, so every commit on main is still checked from scratch, and a pass that should not have been reused shows on the next main run. That also keeps the cache at one build, so the shared-source key is gone.
ashhart
force-pushed
the
lead/ci-reuse-tests
branch
from
November 5, 2026 19:25
e30e06b to
5d956ea
Compare
8 tasks done
This was referenced Nov 5, 2026
Merged
ashhart
added a commit
that referenced
this pull request
Nov 5, 2026
…on change, on main too #2274 skipped it on pull requests that leave the tool, this workflow and Zig's version alone, but main ran it on every push, about 2 minutes of each main zig job. Main now compares with the commit before the push, as the lean check does, and runs the test only when one of those files changed. A push whose previous commit isn't in the checkout, such as a new branch, still runs it.
ashhart
added a commit
that referenced
this pull request
Nov 5, 2026
…on change, on main too #2274 skipped it on pull requests that leave the tool, this workflow and Zig's version alone, but main ran it on every push, about 2 minutes of each main zig job. Main now compares with the commit before the push, as the lean check does, and runs the test only when one of those files changed. A push whose previous commit isn't in the checkout, such as a new branch, still runs it.
ashhart
added a commit
that referenced
this pull request
Nov 6, 2026
…an't see a name added since It walks zig, docs, tools and .github for names Windows can't check out. Those directory listings aren't inputs the build knows of, so since pull requests reuse main's test passes (#2274), a pull request that adds such a name would reuse the pass and stay green until main's own run. Found by tracing what every run in test-host opens (the review of #2095): of the runs the build can skip, this is the only one that reads tracked paths it doesn't declare; the others declare theirs in zig/build/test_reads.zig.
ashhart
added a commit
that referenced
this pull request
Nov 6, 2026
…an't see a name added since It walks zig, docs, tools and .github for names Windows can't check out. Those directory listings aren't inputs the build knows of, so since pull requests reuse main's test passes (#2274), a pull request that adds such a name would reuse the pass and stay green until main's own run. Found by tracing what every run in test-host opens (the review of #2095): of the runs the build can skip, this is the only one that reads tracked paths it doesn't declare; the others declare theirs in zig/build/test_reads.zig.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A pull request's
zigandzig-windowsjobs now reuse main's test passes and compiles, C and C++ included, so they run only what the diff changed. Main still builds and tests every commit from scratch.Why tests never reused a pass
Zig's build cache can reuse a passing test, but on CI it never did. Zig 0.17's check misses on any stat change to a declared input even when the contents match, and a checkout or a restored cache gives every file a new id. So every test binary, fixture and C or C++ source looked changed. The bug is ziglang/zig#37200. The fix (#37238) is on Zig's master, but no release has it yet, 0.17.1 included.
What changes
tools/zig/zig_cache_restat.zigruns before the first build. A file a manifest records gets its new id only if its contents still hash to the recorded digest. A file whose contents changed is left alone, so Zig misses as it should. Zig's own library isn't patched, and C and C++ compiles reuse as well. If a Zig update changes the cache format, the step warns, the job rebuilds and reruns more than it needs to, andtools/zig/zig_cache_restat_test.pyfails. That test runs on main, and on pull requests that change the tool, this workflow or Zig's version.zig/build/test_reads.zig, so a changed fixture reruns them.-Drerun-testsis removed. It made every run of a test a side effect, so main saved no test results for a pull request to reuse.Measurements
On a Mac, a full
test,test-goldenandfuzz-cortheonrun, one run of each, on a loaded machine:The 23 test runs are the steps that always run, and the goldens take 60 of those 86 seconds. On Linux, with an earlier version that still recompiled C and C++, a restored run with nothing changed took 316 s, against 701 s cold.
This PR's own CI restored main's first save from #2252. That save holds compiles but no test results, so every test ran once here. Test passes start reusing once this lands and main saves its first run. What it shows for the tool:
zig-windowszig build test13.9 min with every test run, against 24.6 coldzigzig build test12.4 min with every test run, against 11.7 cold on another runner. Linux time is mostly tests, and these logs carry no build summaryzig-macos-dev-toolszig build dev-tools18 s, and the whole job 1 min, against a median of about 13 before #2252release-pathsThe tool's own regression test ran here because this PR changes it: 2 minutes on Linux.
Not done here
release-pathsand the macOS dev tools run the tool too, but have no warm number yet.The approach of updating a restored cache's records came from cosmic-lua's zig-restat. This tool is written from Zig's own cache code.
Test plan
zig-windowscompiles nothing in C or C++ that main's save already holdscachedfor what its diff didn't touch