Skip to content

test(perf): overlay-churn-settled lets each entrance end, so its count does not follow the runner's speed - #10496

Open
cixzhang wants to merge 1 commit into
mainfrom
test/perf-driver-timer-wait
Open

cixzhang wants to merge 1 commit into
mainfrom
test/perf-driver-timer-wait

Conversation

@cixzhang

@cixzhang cixzhang commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Refs #10247

Maintainer impact

The nightly perf lane's overlay-churn count followed the runner's speed, not the product. On the same commit, interactionLayouts read 703 on one run and 805 to 988 on another, and the nightly reopened #10247 on such a run. Traces show why. An anchored layer (Popover, DropdownMenu) lays out on every frame of its entrance animation. The driver pressed Escape as soon as a locator wait saw the overlay, so the count included as many entrance frames as the driver's round trip allowed. The locator wait also read the element's style and box on every poll, inside those frames. A slower runner fit more frames and more polls into each entrance.

Intended invariant

spec:AST-030 FR18 measures the product. This PR adds to FR18: "During a measured interaction, a scene's driver adds no rendering work whose amount depends on the runner's speed." It also adds a verification row: medians within 1% between a standard run and a PERF_CPU_THROTTLE=4 run. The product is unchanged.

Change

  • apps/sandbox/perf/scenes.perf.spec.ts:
    • The overlay driver waits for each overlay to open and to close with page.waitForFunction(..., {polling: 50}). The check reads only the open state (:popover-open on the layer that holds the content, dialog[open]), never style or layout.
    • After the open, it waits 250 ms on the Node side, longer than the 230 ms entrance (--duration-fast-max), and then presses Escape. Each entrance plays out, so each open counts the whole entrance however fast the runner is. This measures the full cost of an open, which is what a user's page pays.
    • PERF_CPU_THROTTLE (default 1) slows Chromium's CPU through the DevTools protocol, so anyone can repeat the check: PERF_CPU_THROTTLE=4 pnpm exec playwright test --config playwright.perf.config.ts --grep overlay-churn-settled.
  • The scene is renamed overlay-churn-settled (sceneIds.ts, scenes.tsx, the driver). Its numbers measure something different, so the old overlay-churn series stays readable in the history, and the trend check compares the new scene only with its own runs. perf-trend.mjs needs no change: a scene with fewer than three runs in the window has no baseline yet.
  • playwright.perf.config.ts: the test timeout goes from 6 to 13 minutes, for the longer scene, inside the 15-minute job budget.
  • docs/specs/AST-030/spec.md: the FR18 sentence and the verification row.
  • A comment in LayerCloseLayout.interaction.browser.spec.ts names the renamed scene.

Why not only poll on a timer: without the wait, the count still included however many entrance frames came before Escape. Medians were 702 to 720 on the same commit, 2% apart.

Evidence

interactionLayouts for the settled scene, dispatched perf-nightly.yml runs on the same base, 5 samples each. The 4× rows slow Chromium's CPU 4× (Emulation.setCPUThrottlingRate, as PERF_CPU_THROTTLE=4 does).

Runner Median Samples
Standard 2262 2266, 2232, 2216, 2262, 2274
Standard 2258 2256, 2250, 2266, 2274, 2258
4× CPU 2242 2282, 2266, 2210, 2230, 2242
4× CPU 2272 2346, 2258, 2272, 2284, 2266
  • Failure before the change: the same commit gave 702 to 988 with the old driver, and 702 to 774 with timer polling alone.
  • Success after the change: standard and 4× medians are within 1% (2258 and 2262 against 2242 and 2272).
  • Finding: an anchored Popover or menu entrance costs about 15.6 layouts, about one per frame. 100 anchored opens per cycle account for about 1560 of the 2260. The product question is tracked separately.
  • Product/runtime behavior verified unchanged: no product file changes. The other scenes read the same on the same base (comments-panel 2, table 8, tree-list 5, markdown 1).

Writing rubric results

Over docs/specs/AST-030/spec.md:

R1 narration: none; the FR18 sentence and the verification row state what the lane does, in the present tense.
R2 research: none in the record; the measurements are in this description.
R3 evidence: the new verification row names the scene and the driver option, not a pull request.
R4 rejected alternatives: none in the record.
R5 change references: none added.
R6 open questions: none.

Scope

  • No intended public API, product behavior, visual direction, or policy change. The FR18 sentence states the lane's existing purpose as a rule for drivers.
  • Product changes discovered during the work were removed or split.
  • Public text and artifacts contain no internal Meta context.

Testing

  • The dispatched perf runs above, and ci:run-perf on this PR.
  • check:knowledge passes.

@vercel

vercel Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
astryx Ready Ready Preview Nov 7, 2026 1:09pm UTC

Request Review

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Nov 7, 2026
@astracat-bot

astracat-bot Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

AI review status for this pull request.

Review status Updated
🔄 In progress (for maintainers only) Nov 7, 2026, 1:09 PM UTC

@cixzhang cixzhang added the ci:run-perf Run the perf scenes (perf-nightly.yml) on this pull request label Nov 7, 2026
@github-actions github-actions Bot added the needs:spec-owner-review Current knowledge records await owner approval label Nov 7, 2026
@cixzhang

cixzhang commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

/approve-spec cdc7231

@github-actions github-actions Bot removed the needs:spec-owner-review Current knowledge records await owner approval label Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

PR Analysis Report

🔎 Preview · 85658d4

Storybook · Sandbox · built from this PR's head

Modified components

Component Size LoC Visual · A11y · RTL
LayerProvider 4.2 kB (0) · 20 B CSS (0) — ✓ · ✓ · ✗
layer — — — · ✓ · —
Layer — — ✓ · ✓ · —

Package size: no published package changed; no new dependencies.

API changes: none in this pull request

Committed API report behind this build: cli 19 rows, core 25 rows; run pnpm api:report.

Touched components checked; add ci:run-visual, ci:run-a11y or ci:run-rtl to run the full suites.


Generated by PR Enrichment workflow | View full report

@astracat-bot astracat-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. The new overlay-churn wait looks right, but the FR18 text this pull request adds needs rewording before it can be approved.

  • FR18's new sentence ("adds no main-thread work during a measured interaction beyond the input it sends", docs/specs/AST-030/spec.md:450-452) is untrue for every scene at this head, since waitForOverlay polls in the page, Playwright runs its click checks in animation frames, and the other scenes wait with frames(page, 2) or waitForRest, so narrow it to what the change guarantees, for example "adds no rendering work whose amount depends on the runner's speed".
  • The new FR18 verification row (docs/specs/AST-030/spec.md:662) fails on any difference between the slow and fast counts with no tolerance, which the description's own medians (702 against 704 and 706) already trip, and neither perf-nightly.yml nor the driver can make its 4× CPU run, so add a tolerance and say how the throttled run is produced.

Also worth fixing, non-blocking: the comment at apps/sandbox/perf/scenes.perf.spec.ts:70-73 says a locator's wait polls in a frame callback, but Playwright 1.61.1's locator.waitFor polls on timers and reads the element's style and box on each poll; a suggested rewording is inline.

[Automated review]

Comment thread docs/specs/AST-030/spec.md Resolved
Comment thread docs/specs/AST-030/spec.md Resolved
Comment thread apps/sandbox/perf/scenes.perf.spec.ts Resolved
@cixzhang
cixzhang marked this pull request as draft November 7, 2026 12:00
@cixzhang
cixzhang force-pushed the test/perf-driver-timer-wait branch from cdc7231 to 30aa772 Compare November 7, 2026 12:00
@github-actions github-actions Bot added the needs:spec-owner-review Current knowledge records await owner approval label Nov 7, 2026
@cixzhang

cixzhang commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

/approve-spec 30aa772

@github-actions github-actions Bot removed the needs:spec-owner-review Current knowledge records await owner approval label Nov 7, 2026

@astracat-bot astracat-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. All three points from the earlier review are addressed: FR18's new sentence now states only what the scene drivers guarantee, the new verification row has a 1% tolerance and names how the slow run is made (PERF_CPU_THROTTLE=4, now a driver option), and the waitForOverlay comment names the right cause. A local five-sample run of overlay-churn on this head reads a median of 700 interaction layouts both unthrottled and at PERF_CPU_THROTTLE=4.

Approved pending CI: CI, including the ci:run-perf measurement, has not run on this head yet because the pull request is a draft, and it runs once the pull request is marked ready for review.

[Automated review]

…t does not follow the runner's speed

An anchored layer lays out on every frame of its entrance. The driver
pressed Escape as soon as a locator wait saw the overlay, so the count
held as many entrance frames as its round trip allowed, and the wait
read style and box on every poll. The same commit counted 702 to 988.

The driver now polls the open state on a timer, waits 250 ms for the
entrance to end, and then closes. The scene is renamed
overlay-churn-settled so its new series starts apart from the old one.
PERF_CPU_THROTTLE slows Chromium's CPU to check that the count does not
follow the runner's speed: standard and 4x medians are within 1%.
AST-030 FR18 states the rule.
@cixzhang
cixzhang force-pushed the test/perf-driver-timer-wait branch from 30aa772 to 85658d4 Compare November 7, 2026 13:01
@cixzhang cixzhang changed the title test(perf): the overlay-churn driver waits on a timer, so its count does not follow the runner's speed test(perf): overlay-churn-settled lets each entrance end, so its count does not follow the runner's speed Nov 7, 2026
@github-actions github-actions Bot added the needs:spec-owner-review Current knowledge records await owner approval label Nov 7, 2026
@cixzhang

cixzhang commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

/approve-spec 85658d4

@cixzhang
cixzhang marked this pull request as ready for review November 7, 2026 13:03
@github-actions github-actions Bot removed the needs:spec-owner-review Current knowledge records await owner approval label Nov 7, 2026

This branch was successfully deployed

1 active deployment
Preview — 85658d49 Deployed Nov 7, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:run-perf Run the perf scenes (perf-nightly.yml) on this pull request CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant