Repository navigation
ci: run the accessibility contracts at the head that built their Storybook - #9971
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
AI review status for this pull request.
|
PR Analysis Report
Generated by PR Enrichment workflow | View current CI run |
4a02611 to
271cea5
Compare
There was a problem hiding this comment.
Requesting changes: the workflow change itself looks right, but this head fails a repository guard test, so the required test check fails.
- Miss: the new head checkout brings the contracts job under
.github/scripts/ci-head-checkout-scripts.test.mjs, which now fails (and with ittest-node (1/4)andtest) because it countstouched-components.mjsandlib/a11y-spec-files.mjsas unguarded head scripts even though both run on the merge checkout before the head checkout; teach the guard to skip steps that run after a plain merge checkout and before the head checkout, with a test case for it.
The CI impact: line now covers what the CI routing spec asks for, and the new contract assertions fail against the previous workflow and pass here.
[Automated review]
271cea5 to
a3f1ea5
Compare
There was a problem hiding this comment.
Approved pending CI: CI has not finished on this head yet.
The contracts job now tests a single commit, the head's specs against the Storybook that head built, while routing lines and recorded durations still come from the merge commit, which keeps routing on current policy and matches the CI routing spec. The head-checkout guard failure from the previous review is fixed: the new merge-checkout exemption covers exactly the two selection steps, and a test case pins it.
[Automated review]
a3f1ea5 to
b3e4962
Compare
There was a problem hiding this comment.
Approved pending CI: CI has not run on this head yet, because the pull request is a draft. Marking it ready for review starts the run, and the required checks need to pass before it merges.
The contracts job now tests one commit, the head's specs against the Storybook that head built, while the runner, routing lines and recorded durations still come from the merge commit; that matches the CI routing spec. The contract tests pin the head checkout's ref and step order, and the head-checkout guard's new exemption covers only the two selection steps.
[Automated review]
Maintainer impact
The "Component accessibility contracts" job tests a Storybook that
build-storybookbuilds from the pull request head, but it checked out the default merge commit. So it ran the merged specs frommainagainst a Storybook built from an older head. A pull request whose head predates a newer spec onmainfailed that spec for a reason it did not cause. Example: #9954 at head397395a9failed 8 of 8 tests inDropdownMenuTooltip.a11y.chromium.spec.ts. That spec arrived onmainwith #9280, and the head-built Storybook does not have the #9280 DropdownMenu change.mainitself passes all 8.Intended invariant
The lane tests one commit: the specs, their helpers, the lockfile, and the Storybook all come from the pull request head. Which contracts run is decided from the pull request's own changes (the merge-base-to-head file list) and the head's own analysis, under current routing policy.
Change
pr-component-a11y-contractsonly:The selection steps still read the merge commit (renamed "Check out the selection scripts"), as the touched-components job does. A head that predates ci: run a touched component's own accessibility contracts beside the tests, and block the merge on them #9707 does not have these scripts. They read the pull request's changed-file list from the scope action, which is the pull request's own diff (merge base to head), and the
pr-analysisartifact, whichbuild-storybookmakes at the head.New step "Check out the head that built the Storybook" checks out
github.event.pull_request.head.sha || github.sha, which is the same ref asbuild-storybook, before Node setup, the Storybook download,pnpm install, and the run.The run step loads the runner from the merge commit (
git show "$WORKFLOW_SHA:…"), aspr-a11y-specsalready does. So routing lines stay current policy, and the specs run at the head.playwright test --listruns at the head, so a spec that the head does not have is not listed and does not run.a11y-pr-contracts.mjsreads the recorded durations from the merge commit, as it reads routing lines and the audit config. Before this change the durations came from the checkout, which was the merge commit. Without this, a head from before ci: run a touched component's own accessibility contracts beside the tests, and block the merge on them #9707 would have no durations file.ci-head-checkout-scripts.test.mjs(the guard that a job which checks out the head runs only scripts that the head has) accepts a step that runs in the job's merge checkout before the head checkout replaces the workspace. That is how the two selection steps run. A unit case shows that the same step after the head checkout, or in a job with no merge checkout, is still flagged.build-storybookand every other job are unchanged. Interactions that appear only after the merge stay covered by post-merge CI onmainand byrelease-check, which run the full suites on the merged commit.How #9954 would now behave
At head
397395a9,DropdownMenuTooltip.a11y.chromium.spec.tsdoes not exist (git cat-file -e 397395a9:<spec>fails). The head checkout therefore cannot list it, and the lane does not run it. The lane runs only the head's own DropdownMenu-owned specs against the head's own Storybook. After the branch mergesmain, its head contains #9280 and the spec runs against a Storybook that has the fix.Evidence
touched-components.test.mjscontract asserts that the contracts job's head checkout ref equalsbuild-storybook's checkout ref, that it runs after the selection steps and before setup, install and run, that no other checkout moves the tree, and that the runner comes from$WORKFLOW_SHA. This PR's own CI does not reach the changed steps, because no component changed. So a temporary commit that touched one DropdownMenu spec forced one run of the lane. That commit was dropped before review. The result is in the Testing section.ubuntu-latestwith the same needs. The one new step is a depth-1 head checkout. It runs only when a contract applies. In the forced run it took about 3 seconds of a 3.3-minute job (job 15:41:34 to 15:44:51; head checkout 15:41:52 to 15:41:55). So job-minutes per update go from about 3.3 to about 3.35 when contracts run, and stay unchanged (0 added) when no contract applies. At 3 seconds a run, even 100 contract runs a day add about 5 runner-minutes a day. Time to green is unchanged, because the job still runs beside the unit-test jobs and ends before the slowest of them.Scope
Testing
pnpm exec vitest runontouched-components,a11y-pr-contracts,ci-test-join,ci-test-routing,ci-tooling-routing,ci-admission,interaction-legs, andci-head-checkout-scriptstests pass. The full.github/scriptssuite passes locally, exceptoffline-media(it cannot launch a local browser, and it fails the same way onmain) and onertl-audit-clitiming case, which passes when it runs alone, here and onmain.bash -non everyrun:script in the job.4a02611that changed one line inDropdownMenuTooltip.a11y.chromium.spec.ts, since dropped): job log. Every changed step ran. The selection checkout was the merge commit26240f7, and thengit checkout --force 4a02611…moved the tree to the head. The lane selected 10 spec files (the changed spec, the DropdownMenu folder specs, the menu pattern specs, and one ownedcore/DropdownMenuspec), loaded the runner from$WORKFLOW_SHA, and reported99 passed (2.2m).