Skip to content

test: fix #829 flakes at the source and cut fixed test waits - #855

Open
tt-a1i wants to merge 1 commit into
mainfrom
test-value-round2
Open

tt-a1i wants to merge 1 commit into
mainfrom
test-value-round2

Conversation

@tt-a1i

@tt-a1i tt-a1i commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Second test-value round (methodology: tt-a1i/test-value). Refs #829. Does not touch #832 (setup-config PID reuse); that flake is left to it.

Flaky tests (#829)

Test Root cause Fix Verification
antigravity.test.ts "bounds the full SSE lifetime after the first event" With timeoutMs: 5, the first-event and full-lifetime deadlines are the same instant. A runner that takes >5 ms between attempt start and the first read hits the first-event error instead of the one under test. Freeze Date.now in this case only. The enqueued first event is then always seen before the deadline, and the idle second read expires via its real timer. Injected a 12 ms busy wait into the fetch stub: old test fails with "first SSE event", new test passes. 20/20 local repeats.
session-sidebar.spec.ts:239 trailing-click suppression (Windows) The product suppresses the drag's trailing click for 250 ms of wall time. On Windows the await waitFor(pin) alone can take longer, so the click went through. vi.useFakeTimers({ toFake: ["Date"] }) for this case. The 250 ms window is product behavior, not something a slow runner should consume. Injected a 300 ms delay before the click: old test fails exactly as in CI, new test passes. 10/10 local repeats.
openpi-web.e2e.ts:2510 "Response has been disposed" A snapshot refresh started by the accepted prompt was still in route.fetch()/json() when the context closed. await page.unrouteAll({ behavior: "wait" }) at the end, the same pattern this file already uses after cancel/reconcile. Failed once in the full local baseline E2E run, passed in the full run on this branch. 15/15 isolated repeats on this branch. The isolated base did not reproduce (15/15).
provider-models-settings.spec.ts:474, workbar-session-position.spec.ts:577, document-reading-state.spec.ts:234 (Windows) All three failed inside findByRole/the 5 s test timeout while an App-level jsdom render was still running. The Windows UI job is about 2.5× slower than Linux (environment 148 s vs 56 s), and on it Testing Library's 1 s default give-up is shorter than one such render. configure({ asyncUtilTimeout: 5_000 }) and testTimeout: 15_000 in the UI harness. These are harness give-up limits, not product deadlines: a missing element still fails, just later. No UI spec asserts that something appears within a time bound; product timers are tested with explicit clocks (see the sidebar fix above). Repeated CI runs (see below). This is the one change here that widens a budget instead of removing a timing assumption. It follows the test-value rule that "an unreasonable test timeout can be widened, while a contractual product deadline still needs coverage".
session-history.e2e.ts:340 Superseded by #807; not changed Passed in both full local E2E runs.
Windows setup-config.test.ts:661 Owned by #832; not changed —

Faster feedback

  • Background terminals (product fix). After a child exits, the manager always slept 1 s (SETTLE_GRACE_MS) before checking whether stdio had closed. Session shutdown waited for that sleeper too, even when the terminal had already settled. The grace now races entry.settled, so a normal exit ends it right away; a descendant that holds the pipes open still gets the full 1 s. manager.test.ts: 40.1 s → 16.1 s. On Windows this file runs in the serial isolated-process job. All 102 background-terminal tests pass.
  • Taskkill timeout tests. The ladder cases now run on node:test mock timers instead of waiting out the real 0.4 to 2 s production timeouts. Assertions are unchanged.
  • Web E2E now runs as 3 shards instead of 2: about 8 minutes of single-worker browser time was the PR critical path. Shard job names change to x/3. Required checks are the aggregators (Node 22.19.0, Node 24, Background terminals (Windows)), so branch protection is unaffected.

Audit (tests added since #726: 157 files, +38.6k lines)

Most new tests assert observable behavior through public entry points. Mechanical scans found no copied test titles across files, no source-text assertions, and no snapshot freezes. The cost lives in fixed waits and harness budgets, not in redundant cases, so this round deletes only cases with path-level evidence.

File Decision Evidence
tests/web/web-host.test.ts stop with wall shift 0 / +120000 / -120000 merge → 1 case (-120000) Stop's drain bounds are timers only (no Date.now on the stop path), so all three rows ran the same code. Only a backward shift could expose a regression to a wall-clock deadline; forward or zero could only stop sooner. The surviving case still fails when the drain bound is broken (mutation: SERVER_CLOSE_DRAIN_MS = 10_000 → "host stop did not finish").
tests/web/session-fork.test.ts monotonic budget backward-wall (elapsed 0) delete row It can only fail if the budget always expires, and forward-wall and backward-wall-with-budget-left already reject that. backward-wall-with-expired-budget keeps the wall-clock regression guard.
tests/extensions/background-terminals/manager.test.ts taskkill ladder rewrite (virtual clock) Same assertions, about 9 s less real waiting.
antigravity.test.ts, session-sidebar.spec.ts, openpi-web.e2e.ts cases above rewrite Flake fixes; see the table above.
provider-settings-page.spec.ts C07-C08 × failed/cancelled/unconfirmed keep Each status is a separate member of setupNeedsAttention; removing any one is caught only by its own row.
provider-settings-page.spec.ts plan planning/ready/invalid keep Small gain (about 0.7 s); a mis-enumerated planBlocked would need every value.
subagent-inspection.e2e.ts 1280 mouse / 390 mouse / 390 touch keep Each variant has its own branches (hover-only controls, ≥44 px touch rows, complementary vs main layout).
sidebar-parity.e2e.ts search palette Escape keep Restoring focus after a native <dialog> closes needs a real browser; jsdom specs can't show it.
sidebar-fork-depth.e2e.ts, sidebar-menu-flow.e2e.ts keep Native restart, receipts and worktree removal span the real Host; specs cover only store projections.
session-worktree.test.ts 20,010-file tree (4.4 s) keep It is the only test of the 20,000-entry inspection limit, a real resource bound.
workflows/sandbox.test.ts 6 × non-yielding watchdog (1 s each) keep Each case re-arms the watchdog in a different async context (after await, chained microtasks, a parallel branch). These are separate VM-timeout escape routes. The 1 s is the product watchdog.
openpi-web.e2e.ts mobile focus trap at 320/390 px keep, awaiting evidence Focus logic is width-independent, but there are 360/420 px layout rules in that range.

Timings

Measured locally on the same box with Node 22.19.0 and the same tree before and after. CI job durations are below.

Suite Before (4418140) After
bun run test:node 84.1 s, 2636 tests 74.1 s, 2633 tests
bun run test:ui 43.9 s, 1520 tests 44.7 s, 1520 tests (unchanged within noise)
Web E2E (1 worker, all 157) 8 m 12 s (2 failures: #829 :2510 + local-font PDF) 7 m 40 s (1 failure: local-font PDF*)
background-terminals/manager.test.ts alone 40.1 s 16.1 s

* file-formats.e2e.ts:162 fails identically on base and branch in this sandbox: the box's CJK font maps 文→⽂ (a Kangxi radical). It passes on CI.

CI before (last three green main runs): Web E2E shards 295 to 376 s, Windows Node parallel 243 to 386 s, Windows isolated 138 to 177 s, Windows UI 182 to 311 s.

CI on this PR: 3 consecutive full runs, all green, including every Windows job (run 38830860130, attempts 1–3).

Job Before (main, 3 runs) This PR (3 runs)
Web E2E shards (slowest) 329 / 376 / 328 s (2 shards) 260 / 260 / 276 s (3 shards)
Windows isolated processes 177 / 138 / 149 s 152 / 132 / 149 s
Windows Node parallel (slowest) 299 / 386 / 294 s 362 / 347 / 396 s (setup-config, owned by #832, dominates)
UI windows-latest 301 / 182 / 311 s 231 / 294 / 313 s

- antigravity: freeze Date in the full-SSE-lifetime case so the shared 5 ms
  first-event deadline cannot expire before the first event on a slow runner
- session-sidebar: freeze Date across the 250 ms trailing-click window
- openpi-web E2E: drain the snapshot route handler before teardown
- UI harness: findBy/waitFor give-up budget 1 s -> 5 s (test timeout 15 s);
  a missing element still fails, no product deadline is involved
- background terminals: end the post-exit settle grace when the natural
  close path settles instead of always sleeping 1 s (also on shutdown)
- run the taskkill timeout ladder on node:test's mock clock
- drop wall-clock matrix rows that execute an identical path
- shard Web E2E across three runners

Refs #829
@tt-a1i
tt-a1i requested a review from a team as a code owner October 16, 2026 00:40
@github-actions github-actions Bot added area:background-terminals Background terminal runtime, skill, or tests area:github GitHub workflows, templates, ownership, or tests labels Oct 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:background-terminals Background terminal runtime, skill, or tests area:github GitHub workflows, templates, ownership, or tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant