Background
Reassure Performance Tests runs on nearly every PR: three jobs on blacksmith-4vcpu-ubuntu-2404 measuring origin/main (baseline-perf-tests) against the PR head (branch-perf-tests), failing the PR if a render count moved at all (COUNT_DEVIATION: 0) or a duration moved more than 20%.
Over the 55 most recent successful runs (workflow id 58601416), a typical run makes a PR wait about 6 minutes (369s), and its three jobs together occupy about 12 minutes of runner time (696s). Two of those three jobs spend 314s each inside the one step that runs the benchmarks, which is 90% of the machine time a run costs. It is the last check to report on nearly every PR, so that wait is usually the last thing between a PR and merge. At ~252 runs a day it comes to ~48.7 runner-hours a day.
Nearly half of that step is not measurement. It is Jest booting and pushing ~7,000 files through Babel, made several times more expensive by Reassure's --max-opt=1, which disables TurboFan. Jest is already configured to cache that output in .jest-cache (jest.config.js:50). The perf jobs discard it after every job.
Problem
Every Reassure run re-transforms the whole source tree with Babel, even though the Reassure run before it already turned the same files into byte-identical output. And when a contributor pushes a fix, the Reassure run measuring the previous commit keeps going to the end, spending both its measure jobs producing numbers for a commit that will never be merged and that no one will look at.
Solution
Implemented in a draft PR: #100297. One new reusable workflow called from seedStickyDisks.yml, a test, and about sixty lines in reassurePerformanceTests.yml.
Problems to solve:
- Every run re-transforms a source tree the run before it already transformed identically. Solved by having both measure jobs restore a Jest transform cache, and never write it. The key covers what a Babel transform can depend on: the normalized lockfile,
patches/**, babel.config.js, config/babel/**, jest.config.js and .nvmrc.
- If the PR jobs only read the cache, something else has to fill it. Solved by
seedJestPerfCache.yml, a reusable workflow called as a job from seedStickyDisks.yml, which already seeds CI caches on every push to main. It probes the key first and exits in ~40s when the entry is present, so it repopulates only when the key actually rotates — 1.6 times a day over the last 90 days, since most pushes that touch a path in the key change nothing but the app version in package-lock.json, which setupNode strips before the hash. Probing every push rather than a filtered subset costs ~0.5 runner-hours a day, and means an evicted entry is rebuilt on the next merge.
- A run measuring an abandoned commit still runs to the end. Solved by a concurrency group on the perf workflow, so a new push cancels that PR's older run instead of paying for it to finish.
What this saves
before after change
PR waits for this check 6m 09s ~3m 55s -36%
CI machine time, one run 11m 36s ~7m 08s -38%
CI machine time, per day 48.7 h ~29.3 h -19.4 h, -40%
The before column is the measured p50 across the 55 most recent successful runs. The after column is that p50 scaled by the ratio a throwaway workflow measured on blacksmith-4vcpu-ubuntu-2404: 336s with an empty .jest-cache, 192s with one restored from a different job — 42.9%. The daily row nets out the seed job and the runs the concurrency group cancels.
Reported in Slack.
Background
Reassure Performance Testsruns on nearly every PR: three jobs onblacksmith-4vcpu-ubuntu-2404measuringorigin/main(baseline-perf-tests) against the PR head (branch-perf-tests), failing the PR if a render count moved at all (COUNT_DEVIATION: 0) or a duration moved more than 20%.Over the 55 most recent successful runs (workflow id 58601416), a typical run makes a PR wait about 6 minutes (369s), and its three jobs together occupy about 12 minutes of runner time (696s). Two of those three jobs spend 314s each inside the one step that runs the benchmarks, which is 90% of the machine time a run costs. It is the last check to report on nearly every PR, so that wait is usually the last thing between a PR and merge. At ~252 runs a day it comes to ~48.7 runner-hours a day.
Nearly half of that step is not measurement. It is Jest booting and pushing ~7,000 files through Babel, made several times more expensive by Reassure's
--max-opt=1, which disables TurboFan. Jest is already configured to cache that output in.jest-cache(jest.config.js:50). The perf jobs discard it after every job.Problem
Every Reassure run re-transforms the whole source tree with Babel, even though the Reassure run before it already turned the same files into byte-identical output. And when a contributor pushes a fix, the Reassure run measuring the previous commit keeps going to the end, spending both its measure jobs producing numbers for a commit that will never be merged and that no one will look at.
Solution
Implemented in a draft PR: #100297. One new reusable workflow called from
seedStickyDisks.yml, a test, and about sixty lines inreassurePerformanceTests.yml.Problems to solve:
patches/**,babel.config.js,config/babel/**,jest.config.jsand.nvmrc.seedJestPerfCache.yml, a reusable workflow called as a job fromseedStickyDisks.yml, which already seeds CI caches on every push tomain. It probes the key first and exits in ~40s when the entry is present, so it repopulates only when the key actually rotates — 1.6 times a day over the last 90 days, since most pushes that touch a path in the key change nothing but the app version inpackage-lock.json, whichsetupNodestrips before the hash. Probing every push rather than a filtered subset costs ~0.5 runner-hours a day, and means an evicted entry is rebuilt on the next merge.What this saves
The before column is the measured p50 across the 55 most recent successful runs. The after column is that p50 scaled by the ratio a throwaway workflow measured on
blacksmith-4vcpu-ubuntu-2404: 336s with an empty.jest-cache, 192s with one restored from a different job — 42.9%. The daily row nets out the seed job and the runs the concurrency group cancels.Reported in Slack.