Skip to content

fix(ci): Windows test legs take 8–15 min since #250 — aqe-store-merge.test.mjs is the critical path #262

Description

@pacphi

Problem

The Windows legs of ci.yml take 8–15 minutes. Ubuntu and macOS legs take 1–3 minutes. Every PR waits on Windows, so feedback is 4–5× slower than it needs to be.

This is not a Node-version effect: Node 22, 24 and 26 are all slow, and which one finishes first changes from run to run. It is a step change that arrived with #250 (the AQE store-integrity PR, merged 2026-09-28). Before #250, every Windows leg took about 3 minutes.

Evidence

When it started. The last 40 ci.yml runs, Windows job minutes as node 22 / 24 / 26:

Run Commit Windows minutes (22 / 24 / 26)
17 runs up to 36358405196 up to 88e26999 (main, 2026-09-27 23:21) 2.3–4.4 on every leg
36379126905 fc7624f6 (#250's branch, 2026-09-28 04:46) 12.7 / 10.7 / 10.6
every run since #250 onward 7.5–26.8, no consistent order by Node version

The only change between the last fast run and the first slow one is #250. The time is all in the "Unit + statusline tests" step. Checkout, setup-node and pnpm setup take under a minute together.

Where it goes. Per-test durations from one run, 36498460373 (node 22), parsed from the TAP duration_ms lines:

Source file Tests Windows Ubuntu Ratio
tests/kit/aqe-store-merge.test.mjs 34 10.9 min 1.7 min 6.3×
tests/kit/provider-cli.test.mjs 26 0.8 min 0.1 min 6.2×
tests/kit/usage-opencode.test.mjs 26 0.3 min <0.1 min 12×
tests/kit/sync-command.test.mjs 53 0.3 min <0.1 min 8.4×
tests/kit/daemons.test.mjs 12 0.2 min <0.1 min 47×
all tests ~5,660 17.1 min 3.7 min 4.6×

Single aqe-store-merge tests take 20–56 s each on Windows against 2–4 s on Ubuntu; the slowest is "AQE starter patterns are counted per stray…" at 55.9 s against 2.9 s. The whole file runs in 7 s on a local macOS machine.

node --test runs test files concurrently, up to CPUs − 1, which is 3 on the 4-vCPU windows-latest. The tests inside one file run one after another. So on Windows this one file is the job's critical path, and the rest of the suite finishes long before it does.

Why this file is slow on Windows (hypotheses, strongest first)

  1. One transaction per statement, which means one disk flush each. Each test's project() fixture builds three stores with buildStore(). That runs the captured AQE 3.14.4 schema (146 statements: 47 tables, 87 indexes, 11 triggers, 1 FTS5 table) one statement at a time, then one INSERT per row, all auto-committed. On Windows every commit is a FlushFileBuffers, which is slow on the runner's disk. The code under test adds a VACUUM INTO backup, copyFileSync rehearsal copies, integrity_check / foreign_key_check passes and recursive cpSync archives.
  2. Microsoft Defender scans every new file. Each test creates dozens of new memory.db, -wal, -shm and journal files under the temp folder, and real-time protection scans each on create and close. This is a well-known cause of slow Node and SQLite test suites on windows-latest.
  3. Coverage runs on every leg. scripts/run-tests.mjs passes --experimental-test-coverage on all 9 matrix legs. V8 coverage writes per-process profiles, and that I/O is slower on Windows. One leg would be enough for the coverage threshold.
  4. The rest is child-process spawning. provider-cli, usage-opencode, sync-command and daemons show a 6–47× Windows penalty. Spawning a process costs far more on Windows. This is secondary: about 2 minutes of test time, and it runs alongside other files.

Proposed fixes, as experiments (measure each on a Windows leg, then keep what pays)

  1. Build the fixture schema once per file. Build one empty 3.14.4 store in a before hook, then copyFileSync it per store. Wrap each buildStore insert batch in BEGIN / COMMIT. Test-only change; no production code changes.
  2. Split aqe-store-merge.test.mjs into 3–4 files by concern: preview/dry-run, holders and refusals, apply and archive, starter patterns. Alternatively, run independent tests concurrently (test(..., { concurrency }) or a describe with concurrency), since each builds its own temp project. Either lets the runner's file-level parallelism cut the critical path.
  3. Exclude the runner's temp and workspace folders from Defender on Windows legs only. For example, a first step: Add-MpPreference -ExclusionPath $env:RUNNER_TEMP, $env:GITHUB_WORKSPACE, guarded by if: runner.os == 'Windows'. This is CI-only and speeds every Windows test.
  4. Collect coverage on one leg (for example ubuntu / node 24) and run the other legs without --experimental-test-coverage. Keep the 70/70/70 threshold on that one leg.
  5. Optional, if the above isn't enough: on Windows, run the full suite on one Node version and a smoke or Windows-focused subset on the other two.

Acceptance criteria

  • The Windows legs' median time is back under 5 minutes (it was about 3 before fix(aqe): one AQE store per project, safe stray-store merge, honest verification #250). Measure over 10 consecutive runs, and post the before/after table on this issue.
  • No Windows-relevant coverage is lost. aqe-store-merge's Windows cases, such as the EBUSY rename, still run on Windows.
  • tests/kit/aqe-store-merge.test.mjs's assertions are unchanged, apart from being moved between files.
  • node scripts/run-tests.mjs unit still passes the real-state tripwire.
  • The 30-minute job limit (ci: stop every job after 30 minutes #261) stays as the backstop and is not raised.

Impact

Every PR waits 8–15 minutes on Windows instead of about 3, and a slow Windows run (26.8 minutes on 2026-09-28) comes close to the 30-minute limit. Faster Windows legs shorten every review cycle, and they also stop the limit being reached by a job that is only slow, not stuck.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    size:MA few modules; one to three daysv4.0.0Routing: v4.0.0 boardws:remediationRemediation leftovers and bugs

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions