Skip to content

ci: run the accessibility contracts at the head that built their Storybook - #9971

Merged
cixzhang merged 2 commits into
mainfrom
fix/a11y-contracts-head-ref
Nov 4, 2026
Merged

cixzhang merged 2 commits into
mainfrom
fix/a11y-contracts-head-ref

Conversation

@cixzhang

@cixzhang cixzhang commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Maintainer impact

The "Component accessibility contracts" job tests a Storybook that build-storybook builds from the pull request head, but it checked out the default merge commit. So it ran the merged specs from main against a Storybook built from an older head. A pull request whose head predates a newer spec on main failed that spec for a reason it did not cause. Example: #9954 at head 397395a9 failed 8 of 8 tests in DropdownMenuTooltip.a11y.chromium.spec.ts. That spec arrived on main with #9280, and the head-built Storybook does not have the #9280 DropdownMenu change. main itself passes all 8.

Intended invariant

The lane tests one commit: the specs, their helpers, the lockfile, and the Storybook all come from the pull request head. Which contracts run is decided from the pull request's own changes (the merge-base-to-head file list) and the head's own analysis, under current routing policy.

Change

pr-component-a11y-contracts only:

  • The selection steps still read the merge commit (renamed "Check out the selection scripts"), as the touched-components job does. A head that predates ci: run a touched component's own accessibility contracts beside the tests, and block the merge on them #9707 does not have these scripts. They read the pull request's changed-file list from the scope action, which is the pull request's own diff (merge base to head), and the pr-analysis artifact, which build-storybook makes at the head.

  • New step "Check out the head that built the Storybook" checks out github.event.pull_request.head.sha || github.sha, which is the same ref as build-storybook, before Node setup, the Storybook download, pnpm install, and the run.

  • The run step loads the runner from the merge commit (git show "$WORKFLOW_SHA:…"), as pr-a11y-specs already does. So routing lines stay current policy, and the specs run at the head. playwright test --list runs at the head, so a spec that the head does not have is not listed and does not run.

  • a11y-pr-contracts.mjs reads the recorded durations from the merge commit, as it reads routing lines and the audit config. Before this change the durations came from the checkout, which was the merge commit. Without this, a head from before ci: run a touched component's own accessibility contracts beside the tests, and block the merge on them #9707 would have no durations file.

  • ci-head-checkout-scripts.test.mjs (the guard that a job which checks out the head runs only scripts that the head has) accepts a step that runs in the job's merge checkout before the head checkout replaces the workspace. That is how the two selection steps run. A unit case shows that the same step after the head checkout, or in a job with no merge checkout, is still flagged.

build-storybook and every other job are unchanged. Interactions that appear only after the merge stay covered by post-merge CI on main and by release-check, which run the full suites on the merged commit.

How #9954 would now behave

At head 397395a9, DropdownMenuTooltip.a11y.chromium.spec.ts does not exist (git cat-file -e 397395a9:<spec> fails). The head checkout therefore cannot list it, and the lane does not run it. The lane runs only the head's own DropdownMenu-owned specs against the head's own Storybook. After the branch merges main, its head contains #9280 and the spec runs against a Storybook that has the fix.

Evidence

  • Failure before the change: the merge-commit checkout ran a spec that the head-built Storybook cannot satisfy (chore(core): split long comment sentences in some menu press sources and tests #9954 above).
  • Success after the change: the touched-components.test.mjs contract asserts that the contracts job's head checkout ref equals build-storybook's checkout ref, that it runs after the selection steps and before setup, install and run, that no other checkout moves the tree, and that the runner comes from $WORKFLOW_SHA. This PR's own CI does not reach the changed steps, because no component changed. So a temporary commit that touched one DropdownMenu spec forced one run of the lane. That commit was dropped before review. The result is in the Testing section.
  • Product/runtime behavior verified unchanged: CI-only; no package source changes.
  • CI impact: no new job, and the job stays on ubuntu-latest with the same needs. The one new step is a depth-1 head checkout. It runs only when a contract applies. In the forced run it took about 3 seconds of a 3.3-minute job (job 15:41:34 to 15:44:51; head checkout 15:41:52 to 15:41:55). So job-minutes per update go from about 3.3 to about 3.35 when contracts run, and stay unchanged (0 added) when no contract applies. At 3 seconds a run, even 100 contract runs a day add about 5 runner-minutes a day. Time to green is unchanged, because the job still runs beside the unit-test jobs and ends before the slowest of them.

Scope

  • No intended public API, product behavior, visual direction, or policy change.
  • Product changes discovered during the work were removed or split.
  • Public text and artifacts contain no internal Meta context.

Testing

  • pnpm exec vitest run on touched-components, a11y-pr-contracts, ci-test-join, ci-test-routing, ci-tooling-routing, ci-admission, interaction-legs, and ci-head-checkout-scripts tests pass. The full .github/scripts suite passes locally, except offline-media (it cannot launch a local browser, and it fails the same way on main) and one rtl-audit-cli timing case, which passes when it runs alone, here and on main.
  • bash -n on every run: script in the job.
  • Forced lane run (temporary commit 4a02611 that changed one line in DropdownMenuTooltip.a11y.chromium.spec.ts, since dropped): job log. Every changed step ran. The selection checkout was the merge commit 26240f7, and then git checkout --force 4a02611… moved the tree to the head. The lane selected 10 spec files (the changed spec, the DropdownMenu folder specs, the menu pattern specs, and one owned core/DropdownMenu spec), loaded the runner from $WORKFLOW_SHA, and reported 99 passed (2.2m).

@vercel

vercel Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
astryx Ready Ready Preview Nov 4, 2026 7:02pm UTC

Request Review

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Nov 4, 2026
@cixzhang
cixzhang marked this pull request as ready for review November 4, 2026 15:36
@astracat-bot

astracat-bot Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

AI review status for this pull request.

Review status Updated
🟢 Reviewed (for maintainers only) Nov 4, 2026, 7:19 PM UTC

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

PR Analysis Report

Current run: The current CI run concluded skipped. Current analysis is unavailable.


Generated by PR Enrichment workflow | View current CI run

@astracat-bot astracat-bot Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: the workflow change itself looks right, but this head fails a repository guard test, so the required test check fails.

  • Miss: the new head checkout brings the contracts job under .github/scripts/ci-head-checkout-scripts.test.mjs, which now fails (and with it test-node (1/4) and test) because it counts touched-components.mjs and lib/a11y-spec-files.mjs as unguarded head scripts even though both run on the merge checkout before the head checkout; teach the guard to skip steps that run after a plain merge checkout and before the head checkout, with a test case for it.

The CI impact: line now covers what the CI routing spec asks for, and the new contract assertions fail against the previous workflow and pass here.

[Automated review]

@cixzhang
cixzhang marked this pull request as draft November 4, 2026 17:19
@cixzhang
cixzhang force-pushed the fix/a11y-contracts-head-ref branch from 271cea5 to a3f1ea5 Compare November 4, 2026 17:20
@cixzhang
cixzhang marked this pull request as ready for review November 4, 2026 17:22

@astracat-bot astracat-bot Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved pending CI: CI has not finished on this head yet.

The contracts job now tests a single commit, the head's specs against the Storybook that head built, while routing lines and recorded durations still come from the merge commit, which keeps routing on current policy and matches the CI routing spec. The head-checkout guard failure from the previous review is fixed: the new merge-checkout exemption covers exactly the two selection steps, and a test case pins it.

[Automated review]

@cixzhang
cixzhang marked this pull request as draft November 4, 2026 18:54
@cixzhang
cixzhang force-pushed the fix/a11y-contracts-head-ref branch from a3f1ea5 to b3e4962 Compare November 4, 2026 18:54
@cixzhang
cixzhang marked this pull request as ready for review November 4, 2026 19:18

@astracat-bot astracat-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved pending CI: CI has not run on this head yet, because the pull request is a draft. Marking it ready for review starts the run, and the required checks need to pass before it merges.

The contracts job now tests one commit, the head's specs against the Storybook that head built, while the runner, routing lines and recorded durations still come from the merge commit; that matches the CI routing spec. The contract tests pin the head checkout's ref and step order, and the head-checkout guard's new exemption covers only the two selection steps.

[Automated review]

@cixzhang
cixzhang merged commit 54166ff into main Nov 4, 2026
101 checks passed
@github-actions
github-actions Bot deleted the fix/a11y-contracts-head-ref branch November 5, 2026 06:03

This branch was successfully deployed

1 active deployment
Preview — b3e49622 Deployed Nov 4, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant