Infrastructure: Decide the disposition of the Windows Jest smoke harness

[JasonVMo]

Summary

Make an explicit retire-or-retain decision for the Windows-only Jest smoke
harness in apps/storybook/windows-tests/, then execute the mechanical
consequences. The two Windows validation paths -- Jest plus
@react-native-windows/automation, and WebdriverIO plus the desktop driver --
share WinAppDriver and must not run concurrently; the branch documentation
defers retirement to "once parity is reached" without defining what that means.

This task defines parity, names an explicit retire-or-retain choice with an
expiry date, specifies what to delete or preserve under each choice, resolves
the WinAppDriver version and command conflict, and requires the eleven
focus-crash regression tests added by
4777e54f
to remain executable in CI under either outcome.

Goal

Record a durable, owner-approved disposition for
apps/storybook/windows-tests/storybook-smoke.test.cjs and
apps/storybook/jest.windows.config.cjs, and carry it through to a clean
repository state: one Windows validation path that is actually wired to CI,
zero ambiguity about which command a contributor should run, and no lapse in
coverage of the focus-crash regressions.

Stage

Stage 0 - Decisions. This task gates the
storybook-e2e.md Phase 4 production operations step, which
explicitly defers the harness decision to that phase, and the infrastructure
workstream production exit criterion requiring the harness to be "retired or
documented as intentionally retained, with the port and command conflict
resolved." No CI job may be promoted to required for the Windows Fabric Storybook
endpoint until this disposition is recorded.

Why it matters

Observed. The branch documentation states explicitly: "yarn windows:test and
yarn windows:agent still use the older @react-native-windows/automation Jest
harness. The two paths use different ports and different commands and must not be
run at the same time; the desktop-driver path replaces the smoke harness once it
reaches parity."
(apps/storybook/README.md at 8f971021)

Observed. The harness is never run by any CI workflow.
.github/workflows/pr.yml
has no reference to apps/storybook, and grep -rn "storybook" .github/workflows/ .ado/ returns no matches on main.

Observed. The harness is the only location that runs the eleven focus-crash
regression tests added as regression coverage for
PR #4208.
storybook-e2e.md requires that "the focus-crash regression
coverage added by 4777e54f is either still executed by a CI job or ported to
desktop story tests before the legacy Windows harness is retired."

Observed. The CI Windows PR job installs WinAppDriver 1.1 via msiexec. The
harness requires WinAppDriver 1.2.1, set via WINAPPDRIVERPATH. The desktop-driver
path uses WinAppDriver via Appium's appium-windows-driver. All three share the
same WinAppDriver singleton and its default session port. Running any two
concurrently produces session conflicts.

Inferred. Without an explicit decision, the repository will accumulate two
undocumented Windows validation paths: one that is never run in CI and one that
eventually will be. A contributor cannot know which to run, which is authoritative,
or whether both must agree before a PR is safe.

Observed current state

Jest smoke harness (current main)

Observed.
apps/storybook/jest.windows.config.cjs
runs windows-tests/*.test.cjs under testEnvironment: '@react-native-windows/automation' with maxWorkers: 1, testTimeout: 120000, rootLaunchApp: false, and useRootSession: true. It attaches to a
running Agentic Components Storybook window, identified by
STORYBOOK_WINDOWS_WINDOW_TITLE, using a WinAppDriver binary resolved from
WINAPPDRIVERPATH (default
C:\Program Files (x86)\Windows Application Driver\WinAppDriver.exe).
The @react-native-windows/automation devDependency is pinned to 0.81.32.

Observed.
apps/storybook/windows-tests/storybook-smoke.test.cjs
contains three test groups, run via yarn windows:test:

1. Three smoke visibility tests (parametrized by
   scripts/smoke-stories.json):

   • components-button--default / testId: agentic-storybook-button -- element
     isDisplayed() must be true.
   • primitives-icon--default / testId: agentic-storybook-icon -- element
     isDisplayed() must be true.
   • primitives-callout--default / testId: agentic-storybook-callout-trigger +
     statusTestId: agentic-storybook-callout-status -- element displayed and
     status text equals "Native window: Shown".

2. One focus-movement test: selects components-button--overview, clicks
   agentic-storybook-button-overview-primary, presses Tab, and asserts
   agentic-storybook-button-overview-secondary has HasKeyboardFocus: True.

3. Ten focus-crash regression tests (parametrized):

   Story ID	testId
   components-tag--default	agentic-storybook-tag
   components-accordion--default	accordion-header
   components-tab--selected	agentic-storybook-tab-selected
   components-listboxitem--default	agentic-storybook-listbox-item
   components-checkbox--default	agentic-storybook-checkbox
   components-menuitem--selected	agentic-storybook-menu-item
   components-listitem--selected-focus	agentic-storybook-list-item-selected
   components-radio--default	agentic-storybook-radio
   components-switch--default	agentic-storybook-switch
   components-card--interactive (by XPath)	//Button[@Name="Open report"]

   Each test clicks the element, waits 3 seconds, and asserts HasKeyboardFocus: True. The wait is intentional: the crash described in PR Fix windows crash in components #4208 terminated
   ReactApp.exe with 0xc0000409 in ucrtbase.dll roughly three seconds after
   the click.

Observed. The test writes JSON artifacts to artifacts/windows/automation/,
which is git-ignored. No CI step uploads these artifacts.

Observed. yarn windows:agent (scripts/start-windows-agent-session.ps1 -RunSmokeTest) invokes the same smoke harness as part of its agent-session
orchestration. yarn windows:agent:start and yarn windows:agent:stop manage
the session lifecycle. These scripts are Windows PowerShell only.

Desktop-driver path (test-driver branch)

Observed (branch tip
8f971021).
The wdio.conf.ts created on the branch runs **/*.desktop.spec.ts files
through createDesktopWdioConfig from
@fluentui-react-native/desktop-driver/wdio, using the same
STORYBOOK_WS_PORT / 127.0.0.1:7007 Storybook channel. Scripts added on the
branch include desktop:test:windows, desktop:test:fake, desktop:test:macos,
desktop:generate, and desktop:service:windows. These are separate from
windows:test, windows:agent, and storybook:smoke, and both sets of scripts
remain present on the branch.

Observed. The branch README states "the two paths use different ports and
different commands", meaning the Jest harness connects via
@react-native-windows/automation on WinAppDriver's default TCP session port (4723),
while the desktop-driver path routes through Appium's appium-windows-driver,
which by default also binds on port 4723. Both establish a WinAppDriver session
against the same running application window. Inferred: concurrent execution
produces a session conflict at the WinAppDriver session level.

WinAppDriver version skew

Observed. The Jest harness requires WinAppDriver 1.2.1 and reads it from
WINAPPDRIVERPATH. The CI Windows PR job in
.github/workflows/pr.yml
installs WinAppDriver 1.1 via msiexec. The @react-native-windows/automation
devDependency is pinned to 0.81.32. Inferred: running yarn windows:test
in the CI environment as currently configured requires installing WinAppDriver 1.2.1
separately or setting WINAPPDRIVERPATH to a separately installed binary. The
desktop-driver path faces the same version requirement and must install its own
pinned version explicitly in any CI job it introduces.

macOS equivalent

Observed. There is no macOS equivalent of the Jest smoke harness. The
apps/storybook/scripts/ directory contains only Windows PowerShell session
scripts and the Storybook control CLIs. macOS assertions are entirely absent from
main.

Decision: retire, retain, or conditional retain with expiry

Option A -- Retire

Remove apps/storybook/windows-tests/, apps/storybook/jest.windows.config.cjs,
and the @react-native-windows/automation devDependency from
apps/storybook/package.json. Remove the windows:test and windows:agent
scripts that invoke the harness. Update apps/storybook/README.md to remove
the harness documentation and add a sentence recording the retirement date.

Preconditions before retiring:

1. All three smoke visibility checks (Button, Icon, Callout) are covered by
   parameters.desktopTest story tests that pass the Windows Fabric CI job.
2. The focus-movement test (Button Overview) is covered by a parameters.desktopTest
   story test that asserts HasKeyboardFocus after Tab.
3. All ten focus-crash regression tests are ported to parameters.desktopTest
   story tests -- one per component -- that click the element, wait at least
   3 seconds, and assert HasKeyboardFocus and application survival.
4. These story tests run in the Windows Fabric CI job defined by
   storybook-e2e.md Phase 2 or later and are not quarantined.
5. The CI job reports a terminal failure -- not a passing result -- when the app
   crashes during a test (Priority 1 in
   test-driver-release-readiness.md).

Option B -- Retain permanently

Keep the Jest harness as a second, explicitly manual Windows validation tier.
Document it in apps/storybook/README.md as a separate path for agent sessions
and manual contributor runs, never wired to CI. Add a note that it must not be
run concurrently with any desktop:test:windows or desktop:service:windows
invocation. Assign a named owner responsible for keeping the harness and its
story list current as components evolve.

Preconditions for retaining:

1. The @react-native-windows/automation version and required WinAppDriver version
   are explicitly documented, and the documented version is noted as differing from
   the WinAppDriver 1.1 installed by the existing Windows PR CI job.
2. The coverage overlap with the desktop-driver path is documented: which assertions
   are authoritative, which are supplemental.
3. A named owner reviews the harness when the agentic component list changes and
   when @react-native-windows/automation or react-native-windows releases a new
   minor.
4. The WinAppDriver port conflict with the desktop-driver Appium port is resolved by
   assigning the desktop driver a distinct Appium port or serializing the two paths
   at the CI job level.

Option C -- Conditional retain with explicit expiry

Keep the harness through the period when the desktop-driver path has not yet
reached parity (as defined below). Set an explicit expiry: retire automatically
when all Option A preconditions are satisfied, and no later than the completion of
storybook-e2e.md Phase 4. During the conditional period,
block any new tests from being added to the Jest harness.

Recommended decision

Retire (Option A) with a conditional retain through
storybook-e2e.md Phase 2. The harness is never run by CI
today, so retiring it introduces no CI regression. Its only irreplaceable content
is the eleven focus-crash regression tests; the three smoke visibility tests are a
strict subset of what the desktop-driver path targets. Port the focus-crash tests
as part of storybook-e2e Phase 1 (local real-platform runs), then retire the
harness and remove the @react-native-windows/automation dependency no later than
Phase 2 (first interactive CI job). The harness must not be added to CI in the
interim; it exists only as a local developer fallback during the transition.

The owner of storybook-e2e.md must confirm or override this
recommendation when beginning Phase 1.

Parity requirements

Inferred. "Parity" between the desktop-driver path and the Jest harness means:

1. Story selection parity. All story IDs in smoke-stories.json and in the
   parametrized lists in storybook-smoke.test.cjs have a corresponding
   parameters.desktopTest entry or a linked *.desktop.spec.ts file reachable
   by desktop:generate.
2. Assertion parity. For each covered story:
   • Visibility: an element with the same testID is present and displayed.
   • Status strings: the Callout status assertion ("Native window: Shown") is
     reproduced using the desktop driver's attribute-read API.
   • Focus movement: the Tab-key focus-transfer assertion on Button Overview is
     reproduced using browser.keys or the equivalent desktop driver key action.
   • Focus-crash survival: the 3-second click-and-wait pattern is reproduced, and
     the test fails if the app terminates during the wait, not merely if focus is
     absent.
3. Application crash detection. The desktop-driver run reports a terminal
   failure -- not a passing result -- when ReactApp.exe exits during a test.
   This is Priority 1 in test-driver-release-readiness.md
   and must be satisfied before the harness is retired, because the harness's failure
   mode for a crash is an explicit test failure whereas the desktop-driver path's
   crash detection behavior is not yet confirmed on main.
4. CI execution. The ported story tests run in an actual CI job on
   windows-latest and are not quarantined. Manual-only execution does not satisfy
   parity.

Mechanical consequences

On retire (Option A, after preconditions are met)

Path	Action
apps/storybook/windows-tests/storybook-smoke.test.cjs	Delete
apps/storybook/jest.windows.config.cjs	Delete
apps/storybook/package.json devDependency @react-native-windows/automation	Remove
apps/storybook/package.json script windows:test	Remove
apps/storybook/package.json script windows:agent (-RunSmokeTest variant)	Remove or rewrite to omit smoke invocation
apps/storybook/README.md "Relationship to the Windows Jest smoke harness" section	Remove; add one sentence noting the harness was retired and when
apps/storybook/scripts/smoke-stories.json	Audit first: if storybook-control.cjs smoke does not read it, delete; otherwise retain and document

Observed. yarn storybook:smoke runs node scripts/storybook-control.cjs smoke,
which is the Storybook REST control CLI, not the Jest harness. It is a separate
path and is not removed by retiring the Jest harness. Audit storybook-control.cjs
to determine whether it imports smoke-stories.json before deciding whether to
delete that file.

On retain (Option B)

Item	Action
apps/storybook/README.md	Add a section declaring the harness as manual-only, non-CI, listing the WinAppDriver version requirement, and stating the concurrency constraint
apps/storybook/jest.windows.config.cjs	Add a comment block with the required WinAppDriver version and the concurrency constraint
apps/storybook/package.json	Pin @react-native-windows/automation with a comment noting the minimum WinAppDriver version and its divergence from the CI-installed 1.1
Port conflict	Resolve by assigning the desktop driver a distinct Appium port or serializing the two paths at the CI job level
Owner annotation	Add a named owner in jest.windows.config.cjs and in apps/storybook/README.md

Focus-crash regression coverage preservation

Observed. Commit
4777e54f
("Fix windows crash in components",
PR #4208)
added the eleven tests in storybook-smoke.test.cjs as regression coverage for a
0xc0000409 crash in ucrtbase.dll that terminated ReactApp.exe approximately
three seconds after a click on any agentic component. The fix introduced
packages/agentic/components/src/primitives/focus-visual and migrated eleven
components onto it.

The tests exercise each component's focus acquisition path under Windows Fabric and
are the only CI-eligible evidence that those paths remain crash-free. Inferred:
the three-second wait is load-bearing; removing it or replacing it with an
immediate assertion eliminates the regression guard.

Required under all disposition options:

• The ten focus-crash parametrized tests and the one focus-movement test must remain
  runnable -- either in the retained Jest harness or as ported parameters.desktopTest
  story tests -- and must be included in the review record for any PR that modifies
  packages/agentic/components/src/primitives/focus-visual or any of the eleven
  components it manages.
• Ported story tests must reproduce the 3-second post-click wait. A shorter wait
  does not preserve the regression signal.
• If porting is deferred past storybook-e2e Phase 1, the legacy Jest harness must
  remain locally runnable (not deleted) until the port is confirmed in CI.

Port and version conflict resolution

Observed. jest.windows.config.cjs resolves WINAPPDRIVERPATH for the
@react-native-windows/automation test environment, which initiates a WinAppDriver
session on the default WinAppDriver port (4723). The desktop-driver path's Appium
appium-windows-driver also defaults to port 4723. Inferred: launching both
against the same WinAppDriver binary produces a session conflict; neither will
reliably acquire the application window.

Resolution requirements:

1. Serialized execution rule (required regardless of disposition). Document
   in apps/storybook/README.md and apps/storybook/AGENTS.md that
   yarn windows:test and any desktop:test:windows or
   desktop:service:windows command must not run concurrently. A sequential
   invocation must first stop the previous session and verify that its
   WinAppDriver, Appium, and Storybook service ports are free. Neither script may
   start the other as a subprocess.

2. CI serialization (required if both paths are ever wired to CI). Both paths
   must run in separate jobs with no shared WinAppDriver or Appium process. The job
   running the Jest harness must install WinAppDriver 1.2.1 via WINAPPDRIVERPATH
   independently of the Windows PR WinAppDriver 1.1 installation. The job running
   the desktop driver must pin its own WinAppDriver version and may use a different
   Appium host port to avoid the 4723 default conflict.

3. Storybook channel port is not a conflict. Both paths use 127.0.0.1:7007
   for the Storybook WebSocket channel
   (apps/storybook/storybook-server.cjs).
   This is a shared dependency on a single running Storybook server, not a
   conflict between the two harnesses. Both paths require that server to be started
   before they run.

Acceptance criteria

• A disposition decision (retire, retain, or conditional retain with expiry) is
  recorded in this file, with an owner name and a date.
• If retired: apps/storybook/windows-tests/, jest.windows.config.cjs, and
  the @react-native-windows/automation devDependency are removed from main,
  and apps/storybook/README.md records the retirement.
• If retained: apps/storybook/README.md states whether the harness is a
  required CI owner or a supplementary manual tool, lists its WinAppDriver
  version, and documents serialization. A manual-only harness does not count
  as preserving required coverage; the portable suite must own those cases.
  A CI-owned harness runs in its own serialized job.
• Under either option: all eleven focus-crash regression tests (ten
  parametrized plus the focus-movement test) are covered by an active
  CI-wired path -- either a retained CI-owned harness or ported
  parameters.desktopTest story tests -- and are not quarantined.
• Under either option: the sequential-only constraint on windows:test and
  desktop:test:windows / desktop:service:windows is documented in
  apps/storybook/README.md and apps/storybook/AGENTS.md.
• Under either option: the WinAppDriver version required by the active path is
  explicitly stated in apps/storybook/README.md and in the relevant config
  file, and is noted as differing from the WinAppDriver 1.1 installed by the
  existing Windows PR CI job.
• yarn lage test-links passes for all modified documentation.

Dependencies and ordering

• Blocked by: test-driver.md -- the desktop-driver path
  must be on main before parity can be measured or the focus-crash tests ported.
• Blocks: storybook-e2e.md Phase 4 production operations,
  which requires the harness to be retired or formally retained; and promotion of
  the Windows Fabric Storybook CI job to required, since two competing harnesses
  sharing WinAppDriver create an unresolvable concurrency constraint in a required
  CI context.
• Informs: test-driver-release-readiness.md
  -- parity requirement 3 (application crash must not report success) is already a
  Priority 1 item there and must be closed before the Jest harness is retired.
• Ordering: record the disposition decision and port the focus-crash tests during
  storybook-e2e Phase 1 (local real-platform runs). Do not defer porting to Phase 2
  or later; it is a required precondition for any interactive CI job to claim that
  focus-crash coverage is active.

Risks and open decisions

Item	Evidence	Risk or decision
storybook:smoke vs windows:test scope	Observed: yarn storybook:smoke invokes storybook-control.cjs smoke using the Storybook REST endpoint, not the Jest harness. yarn windows:test invokes the Jest harness. They are separate tools with overlapping story coverage.	Decide whether storybook:smoke is also retired on retire, or kept as a quick local smoke check independent of WinAppDriver.
smoke-stories.json ownership after retire	Observed: smoke-stories.json is imported by storybook-smoke.test.cjs. If storybook-control.cjs smoke also reads it, removing it requires updating that script.	Audit storybook-control.cjs before deleting smoke-stories.json.
3-second wait fidelity in ported tests	Inferred: the 3-second wait is empirically derived from the crash timing in PR #4208. WDIO's browser.waitUntil or an explicit setTimeout can reproduce it, but a synthetic delay is not idiomatic in story tests.	Either retain the explicit delay in the ported spec and document why, or prove that an alternative wait preserves the same regression signal.
windows:agent after harness retirement	Observed: windows:agent launches the smoke harness as part of agent session orchestration via -RunSmokeTest. Retiring the harness removes this from agent sessions.	Decide whether agent smoke validation is replaced by desktop:test:windows, removed, or substituted with a storybook:smoke REST call.
Crash detection in desktop driver before retirement	Observed: Priority 1 reporting in test-driver-release-readiness.md requires classifying application crashes as a distinct failure. This is not yet confirmed on main.	Do not retire the Jest harness until this item is closed; otherwise the ported focus-crash tests produce no CI signal when the app terminates.

Evidence and references

Retrieved 2026-08-21.

• apps/storybook/windows-tests/storybook-smoke.test.cjs,
  apps/storybook/jest.windows.config.cjs,
  apps/storybook/scripts/smoke-stories.json --
  harness source, config, and story list.
• apps/storybook/package.json --
  @react-native-windows/automation@0.81.32 devDependency; windows:test,
  windows:agent, storybook:smoke, desktop:test:windows, and
  desktop:service:windows scripts (the latter two from the test-driver branch).
• apps/storybook/README.md,
  apps/storybook/AGENTS.md --
  WinAppDriver 1.2.1 requirement, WINAPPDRIVERPATH documentation, and Storybook
  channel server on 127.0.0.1:7007.
• apps/storybook/README.md at 8f971021
  -- "Relationship to the Windows Jest smoke harness" section: the two paths use
  different ports and commands, must not run at the same time, and the desktop-driver
  path replaces the harness once it reaches parity.
• apps/storybook/wdio.conf.ts at 8f971021 --
  desktop-driver WDIO config; STORYBOOK_WS_PORT defaults to 7007; platform
  selection via DESKTOP_TEST_PLATFORM; scripts desktop:test:windows,
  desktop:test:fake, desktop:service:windows.
• packages/agentic/test-driver/NEXT-STEPS.md at 8f971021 --
  Priority 1 application crash classification requirement; open decisions on
  WinAppSDK Composition screenshot fidelity and Windows Job Objects.
• 4777e54f
  "Fix windows crash in components"
  (PR #4208) --
  the focus-crash fix, focus-visual primitive introduction, and the eleven
  regression tests added to storybook-smoke.test.cjs.
• .github/workflows/pr.yml --
  confirms no workflow references apps/storybook; WinAppDriver 1.1 installed via
  msiexec in Windows PR and Win32 PR jobs.
• Infrastructure workstream README -- Suggested Addition 1, Stage 0
  decision table, and production exit criterion requiring harness disposition.
• storybook-e2e.md -- Phase 4 defers the disposition to this
  task; acceptance criterion on focus-crash coverage preservation before retirement;
  risks table entry on two competing harnesses and WinAppDriver version skew.
• test-driver.md -- smoke harness preservation requirement
  during landing; this task's landing dependency.
• test-driver-release-readiness.md --
  Priority 1 crash classification (prerequisite for parity requirement 3).