Skip to content

CI: a pull request reuses main's test passes and C compiles, so it runs only what its diff changed - #2274

Merged
ashhart merged 2 commits into
mainfrom
lead/ci-reuse-tests
Nov 5, 2026
Merged

ashhart merged 2 commits into
mainfrom
lead/ci-reuse-tests

Conversation

@ashhart

@ashhart ashhart commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

A pull request's zig and zig-windows jobs now reuse main's test passes and compiles, C and C++ included, so they run only what the diff changed. Main still builds and tests every commit from scratch.

Why tests never reused a pass

Zig's build cache can reuse a passing test, but on CI it never did. Zig 0.17's check misses on any stat change to a declared input even when the contents match, and a checkout or a restored cache gives every file a new id. So every test binary, fixture and C or C++ source looked changed. The bug is ziglang/zig#37200. The fix (#37238) is on Zig's master, but no release has it yet, 0.17.1 included.

What changes

  • tools/zig/zig_cache_restat.zig runs before the first build. A file a manifest records gets its new id only if its contents still hash to the recorded digest. A file whose contents changed is left alone, so Zig misses as it should. Zig's own library isn't patched, and C and C++ compiles reuse as well. If a Zig update changes the cache format, the step warns, the job rebuilds and reruns more than it needs to, and tools/zig/zig_cache_restat_test.py fails. That test runs on main, and on pull requests that change the tool, this workflow or Zig's version.
  • Tests that read files while they run. A cached pass says nothing about a file a test opens at run time. A trace of every test run on Linux found these:
    • Four tests read fixtures, now declared in zig/build/test_reads.zig, so a changed fixture reruns them.
    • The goldens and the q35 launch check read whole folders, so they always run, because a directory input hashes only its entries' names.
    • The kernel symbols check always ran.
  • Main stays cold. It restores no cache, so every commit on main is still checked from scratch. A pass that shouldn't have been reused shows on the next main run. That also keeps the cache at one build, so the shared-source part of the key is gone.
  • -Drerun-tests is removed. It made every run of a test a side effect, so main saved no test results for a pull request to reuse.

Measurements

On a Mac, a full test, test-golden and fuzz-cortheon run, one run of each, on a loaded machine:

Fresh checkout, main's cache restored Time Redone
Cold, for comparison 680 s everything
Nothing changed, without the tool 440 s 25 compiles, 316 test runs
Nothing changed, with the tool 86 s plus 7 s for the tool 0 compiles, 23 test runs
A core file changed 284 s 66 compiles, 82 test runs
A fixture changed 84 s its tests reran (and failed, as they should)

The 23 test runs are the steps that always run, and the goldens take 60 of those 86 seconds. On Linux, with an earlier version that still recompiled C and C++, a restored run with nothing changed took 316 s, against 701 s cold.

This PR's own CI restored main's first save from #2252. That save holds compiles but no test results, so every test ran once here. Test passes start reusing once this lands and main saves its first run. What it shows for the tool:

Job Records the tool updated Time
zig-windows 427,677 in 1,713 manifests, 18 s zig build test 13.9 min with every test run, against 24.6 cold
zig 305,485 in 429 manifests, 20 s zig build test 12.4 min with every test run, against 11.7 cold on another runner. Linux time is mostly tests, and these logs carry no build summary
zig-macos-dev-tools 166,349 in 221 manifests, 11 s zig build dev-tools 18 s, and the whole job 1 min, against a median of about 13 before #2252
release-paths 9,433 in 42 manifests the whole job 2 min

The tool's own regression test ran here because this PR changes it: 2 minutes on Linux.

Not done here

The approach of updating a restored cache's records came from cosmic-lua's zig-restat. This tool is written from Zig's own cache code.

Test plan

  • This PR's CI passes, and the tool's step logs updated records on Linux and Windows
  • zig-windows compiles nothing in C or C++ that main's save already holds
  • After landing: main saves, and the next pull request's test steps show cached for what its diff didn't touch

…ns only what its diff changed

Zig's build cache could reuse a passing test, but on CI it never did. Zig 0.17's check
misses on any stat change to a declared input even when the contents match, and a
checkout or a restored cache gives every file a new id. Every test binary, fixture and C
or C++ source looked changed. Zig's master has the fix (ziglang/zig#37238), but no
release does yet, 0.17.1 included.

tools/zig/zig_cache_restat.zig runs before the first build. For each file a manifest
records whose contents still hash to the recorded digest, it records the file's id
today, so Zig sees nothing to check. A file whose contents changed is left alone and
misses as it should. If a Zig update changes the format, the step warns and the job
rebuilds and reruns more than it needs to, and the regression test fails.

A cached pass says nothing about a file a test opens while it runs, so a trace of every
test run found those. Four tests read fixtures, now declared in zig/build/test_reads.zig
so that a changed fixture reruns them. The goldens and the q35 launch check read whole
folders, so they always run: a directory input hashes only its entries' names. The
kernel symbols check always ran.

Main builds from nothing and restores no cache, so every commit on main is still checked
from scratch, and a pass that should not have been reused shows on the next main run.
That also keeps the cache at one build, so the shared-source key is gone.
@ashhart
ashhart force-pushed the lead/ci-reuse-tests branch from e30e06b to 5d956ea Compare November 5, 2026 19:25
@ashhart
ashhart merged commit 09e2f7f into main Nov 5, 2026
7 checks passed
ashhart added a commit that referenced this pull request Nov 5, 2026
…on change, on main too

#2274 skipped it on pull requests that leave the tool, this workflow and Zig's version
alone, but main ran it on every push, about 2 minutes of each main zig job. Main now
compares with the commit before the push, as the lean check does, and runs the test
only when one of those files changed. A push whose previous commit isn't in the
checkout, such as a new branch, still runs it.
ashhart added a commit that referenced this pull request Nov 5, 2026
…on change, on main too

#2274 skipped it on pull requests that leave the tool, this workflow and Zig's version
alone, but main ran it on every push, about 2 minutes of each main zig job. Main now
compares with the commit before the push, as the lean check does, and runs the test
only when one of those files changed. A push whose previous commit isn't in the
checkout, such as a new branch, still runs it.
ashhart added a commit that referenced this pull request Nov 6, 2026
…an't see a name added since

It walks zig, docs, tools and .github for names Windows can't check out. Those directory listings aren't inputs the
build knows of, so since pull requests reuse main's test passes (#2274), a pull request that adds such a name would
reuse the pass and stay green until main's own run. Found by tracing what every run in test-host opens (the review of
#2095): of the runs the build can skip, this is the only one that reads tracked paths it doesn't declare; the others
declare theirs in zig/build/test_reads.zig.
ashhart added a commit that referenced this pull request Nov 6, 2026
…an't see a name added since

It walks zig, docs, tools and .github for names Windows can't check out. Those directory listings aren't inputs the
build knows of, so since pull requests reuse main's test passes (#2274), a pull request that adds such a name would
reuse the pass and stay green until main's own run. Found by tracing what every run in test-host opens (the review of
#2095): of the runs the build can skip, this is the only one that reads tracked paths it doesn't declare; the others
declare theirs in zig/build/test_reads.zig.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant