Repository navigation
test: each --repeat-each copy of an audit keeps its own evidence folder - #9515
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
AI review status for this pull request.
|
PR Analysis Report🔎 Preview ·
|
| Component | Size | LoC | Visual · A11y · RTL |
|---|---|---|---|
| ChatComposer | 61.2 kB (0) · 10.6 kB CSS (0) | — | — · — · — |
| ChatComposerDrawer | 17.5 kB (0) · 4.7 kB CSS (0) | — | — · — · — |
| ChatComposerInput | 59.2 kB (0) · 10.0 kB CSS (0) | — | — · — · — |
42 more components
| Component | Size | LoC | Visual · A11y · RTL |
|---|---|---|---|
| ChatComposerTokenElement | 59.2 kB (0) · 10.0 kB CSS (0) | — | — · — · — |
| ChatDictationButton | 45.6 kB (0) · 8.8 kB CSS (0) | — | — · — · — |
| ChatLayout | 48.3 kB (0) · 9.3 kB CSS (0) | — | — · — · — |
| ChatLayoutScrollButton | 45.6 kB (0) · 8.9 kB CSS (0) | — | — · — · — |
| ChatMessage | 16.7 kB (0) · 3.7 kB CSS (0) | — | — · — · — |
| ChatMessageBubble | 7.1 kB (0) · 3.7 kB CSS (0) | — | — · — · — |
| ChatMessageList | 22.8 kB (0) · 6.9 kB CSS (0) | — | — · — · — |
| ChatMessageMetadata | 19.8 kB (0) · 4.1 kB CSS (0) | — | — · — · — |
| ChatSendButton | 45.5 kB (0) · 8.7 kB CSS (0) | — | — · — · — |
| ChatSystemMessage | 6.8 kB (0) · 3.8 kB CSS (0) | — | — · — · — |
| ChatTokenizedText | 7.1 kB (0) · 4.0 kB CSS (0) | — | — · — · — |
| ChatToolCalls | 25.9 kB (0) · 8.2 kB CSS (0) | — | — · — · — |
| CommandPalette | 51.2 kB (0) · 12.3 kB CSS (0) | — | — · — · — |
| CommandPaletteEmpty | 6.2 kB (0) · 3.5 kB CSS (0) | — | — · — · — |
| CommandPaletteFooter | 17.0 kB (0) · 3.7 kB CSS (0) | — | — · — · — |
| CommandPaletteGroup | 3.5 kB (0) · 3.5 kB CSS (0) | — | — · — · — |
| CommandPaletteInput | 26.9 kB (0) · 7.4 kB CSS (0) | — | — · — · — |
| CommandPaletteItem | 10.3 kB (0) · 3.6 kB CSS (0) | — | — · — · — |
| CommandPaletteList | 16.0 kB (0) · 3.4 kB CSS (0) | — | — · — · — |
| ComplexSelector · Storybook | 55.6 kB (0) · 10.4 kB CSS (0) | 695 (0) | — · — · — |
| ContextMenu · Storybook | 74.0 kB (0) · 15.5 kB CSS (0) | 693 (0) | — · — · — |
| ContextMenuItem | 20.1 kB (0) · 5.6 kB CSS (0) | — | — · — · — |
| Drawer | 37.8 kB (0) · 8.2 kB CSS (0) | — | — · — · — |
| DrawerHeader | 48.0 kB (0) · 11.6 kB CSS (0) | — | — · — · — |
| DropdownMenu | 72.5 kB (0) · 15.6 kB CSS (0) | — | — · — · — |
| DropdownMenuCheckboxItem | 18.6 kB (0) · 5.9 kB CSS (0) | — | — · — · — |
| DropdownMenuDivider | 5.2 kB (0) · 548 B CSS (0) | — | — · — · — |
| DropdownMenuGroup | 6.2 kB (0) · 3.4 kB CSS (0) | — | — · — · — |
| DropdownMenuItem | 17.8 kB (0) · 5.5 kB CSS (0) | — | — · — · — |
| DropdownMenuRadioGroup | 6.1 kB (0) · 3.4 kB CSS (0) | — | — · — · — |
| DropdownMenuRadioItem | 18.6 kB (0) · 5.9 kB CSS (0) | — | — · — · — |
| DropdownMenuSubMenu | 56.7 kB (0) · 8.9 kB CSS (0) | — | — · — · — |
| Field | 39.3 kB (0) · 6.3 kB CSS (0) | — | — · — · — |
| FieldLabel | 35.4 kB (0) · 4.8 kB CSS (0) | — | — · — · — |
| FieldStatus · Storybook | 9.6 kB (0) · 3.6 kB CSS (0) | 164 (0) | — · — · — |
| FileInput · Storybook | 53.6 kB (0) · 9.7 kB CSS (0) | 779 (0) | — · — · — |
| Grid | 4.4 kB (0) · 3.8 kB CSS (0) | — | — · — · — |
| GridSpan | 3.5 kB (0) · 3.4 kB CSS (0) | — | — · — · — |
| Heading | 7.9 kB (0) · 4.9 kB CSS (0) | — | — · — · — |
| HoverCard · Storybook | 19.2 kB (0) · 3.9 kB CSS (0) | 730 (0) | — · — · — |
| HStack | 5.9 kB (0) · 1.6 kB CSS (0) | — | — · — · — |
| Markdown · Storybook | 144.7 kB (0) · 18.4 kB CSS (0) | 8,707 (+1) | — · — · — |
Package size: no published package changed; no new dependencies.
API changes: none in this pull request
Committed API report behind this build: cli 8 rows, theme-butter 1 row, theme-chocolate 1 row, theme-gothic 1 row, theme-matcha 1 row, theme-neutral 1 row, theme-stone 1 row, theme-y2k 1 row; run pnpm api:report.
Wide change (12 components): add ci:run-visual, ci:run-a11y or ci:run-rtl to run them.
Generated by PR Enrichment workflow | View full report
meta-cla
Bot
added
the
CLA Signed
![astracat-bot[bot]](https://avatars.githubusercontent.com/in/4796231?s=60&v=4){kind=small}
There was a problem hiding this comment.
This looks ready to approve: keying the evidence folder by repeatEachIndex keeps plain and CI runs on test-results/complex-selector-audit-evidence and stops overlapping --repeat-each copies from clearing each other's receipts. No changes are needed here. Other audit specs that clear their evidence folder in beforeAll share the same race, which could be a separate follow-up.
[Automated review]
e1bf1e6 to
fb8e440
Compare
vjeux
changed the title
|
At fb8e440, rebuilt on current main: the per-copy folder is now one helper in the audit harness, |
There was a problem hiding this comment.
This looks ready to approve: resetAuditEvidence gives each --repeat-each copy of an audit its own evidence folder, plain and CI runs keep writing the folders CI uploads and checks, and all 23 audit specs that cleared a shared folder now use it. No changes are needed.
[Automated review]
fb8e440 to
cf848f7
Compare
There was a problem hiding this comment.
This looks ready to approve: resetAuditEvidence gives each --repeat-each copy of an audit its own evidence folder, and plain and CI runs keep writing the folders CI uploads and checks. No changes are needed.
- Optional: the description says 23 audit specs call the helper; this revision converts 24 (HStack's audit included), as the commit message already says.
[Automated review]
Each component audit's beforeAll deleted its evidence folder, and its last case counted the receipts its cases wrote there. Copies of a file run by --repeat-each share that folder and can run at once in other workers, so a copy that started while another was finishing deleted the other's receipts before its count: "manifest is complete" failed under pressure though each copy passes (seen in ComplexSelector's and DropdownMenuItem's audits). One helper in the audit harness, resetAuditEvidence (exported as @astryxdesign/a11y-spec/evidence), empties and creates this copy's folder: the first copy keeps the folder CI uploads and audit pull requests read, and each further copy gets <folder>-repeat-<n>, keyed by test.info().repeatEachIndex. The 24 audit specs that cleared a shared folder call it. A plain run writes exactly what it wrote before. Tests: the helper's unit cases, and the audits' late-copy profile (--repeat-each=3 at 2 workers), which fails on main.
cf848f7 to
ab1bc66
Compare
There was a problem hiding this comment.
Approved pending CI: CI has not run on this head yet. resetAuditEvidence gives each --repeat-each copy of an audit its own evidence folder, and plain and CI runs keep writing the folders CI uploads and checks; no changes are needed.
Next step: the branch is in a fork and the pull request is a draft, so a maintainer adds the ci:run label and marking it ready for review then starts CI.
- Optional: the description says 23 audit specs call the helper; this revision converts 24, as the commit message says.
[Automated review]
cixzhang
added
the
ci:run
There was a problem hiding this comment.
Approved pending CI: CI has not finished on this head yet. resetAuditEvidence gives each --repeat-each copy of an audit its own evidence folder, and plain and CI runs keep writing the folders CI uploads and checks; no changes are needed.
- Optional: the description says 23 audit specs call the helper; this revision converts 24, as the description's own evidence section counts.
[Automated review]
Maintainer impact
A component audit fails "manifest is complete" when its copies overlap under
--repeat-each, though each copy passes alone. It surfaced in stress runs of the menu and Selector accessibility specs (ComplexSelector's and DropdownMenuItem's audits).Intended invariant
No behavior change. A plain run writes the same evidence it writes today, in each audit's folder, which the accessibility evidence upload and audit pull requests read. Each copy's receipts and manifest belong to that copy alone.
Change
Each audit's
beforeAlldeleted its evidence folder, and its last case counted the receipts its cases wrote there.--repeat-eachcopies share that folder and can run at once in other workers, so a copy that starts while another is finishing deletes the other's receipts before its count.One helper in the audit harness,
resetAuditEvidence(folder, test.info())(exported as@astryxdesign/a11y-spec/evidence), empties and creates this copy's folder. The first copy keeps the folder its readers know; each further copy gets<folder>-repeat-<n>, keyed by itsrepeatEachIndex. The 23 audit specs that cleared a shared folder call it; the three Chat audits keep clearing after their head check. Test-only.Prior art: no spec keyed evidence by repeat index, and the harness had no evidence-folder helper. Of the 527 open pull requests, only #9495 touches these files (DropdownMenuItemAudit, beside the folder line and
beforeAll); whichever lands second rebases.Evidence
--repeat-each=3 --workers=2fails "manifest is complete" in 2 of 2 runs; DropdownMenuItemAudit at--repeat-each=5 --workers=4fails it in 1 of 3 runs (three copies). The race needs a late copy to start while another copy's count is pending, so audits whose copies stay in step show it less often.--repeat-each=10run. The helper's unit cases cover the folder each copy gets and that it clears only its own.Scope
Testing
pnpm exec playwright test --config playwright.config.ts --project=chromium --workers=2 --repeat-each=3 ComplexSelector.a11y.chromiumfails on main and passes here.This head is cf848f7 replayed on main 217fc46. Main now imports Playwright from
@astryxdesign/a11y-spec/playwrightin every spec, so each touched spec keeps that import withresetAuditEvidencebeside it, and the package exports both subpaths; the patch is otherwise unchanged (range-diff). Local gates at ab1bc66: lint on the changed files, the a11y-spec type check and units (792 of 792), a fresh Storybook build, all 24 audits plain (348 passed), and ComplexSelector, DropdownMenuItemAudit and HoverCardAudit at--repeat-each=3on 2 workers (228 passed). Earlier, at 27dfe1c:test:ui18,845 of 18,845;test:node16,059 passed (2 expected failures); DropdownMenuItemAudit--repeat-each=5at 4 workers, 65 passed; HStackAudit--repeat-each=3at 2 workers, 27 passed.