Repository navigation
Replace custom device-lane CI with a standard Expo smoke suite - #212
Merged
Merged
Conversation
The Maestro lanes ran a special build: Metro swapped in readiness and timing markers, a perf command channel, a scroll and pull trace, and an Undo snackbar that never dismissed itself. The store build carried stand-ins instead, so the app under test differed from the app users get. Delete the harness modules, the Metro swap and the Babel plugin that stripped perf hooks, and keep each call site's store behaviour. Undo snackbars now always dismiss on their timer, as in the store app. The fixed test clock (EXPO_PUBLIC_CLOCK_START) stays, because the fake Trakt server's seed counts from it. Delete the tests that only exercised harness code and maestro-ids.test.ts, drop test ids nothing reads any more, and add Jest coverage for two behaviours only the deleted device flows checked: a show marked from the lapsed drawer files back into the queue, and the TV-off empty state leads to Settings.
Pull request CI now runs what ordinary Expo apps run: lint, types, Jest and Vitest in check, plus six Maestro smoke flows on an Android emulator and an iOS simulator. The flows sign in through the device code against the fake Trakt server, mark the next episode from Up Next, visit every tab, open a show and a movie and mark an episode, search, and sign out. Each one asserts the state its taps produce and takes screenshots of what it visits. Each e2e job fingerprints the native project with the @expo/fingerprint CLI and keeps the release build in actions/cache under that hash. On a hit, @expo/repack-app swaps the commit's JavaScript into the cached build; on a miss the job builds and saves the cache straight away, so a failing flow does not throw the build away. The emulator comes from reactivecircus/android-emulator-runner with its documented AVD cache, the simulator from xcrun simctl, and Maestro from its official installer at a pinned version. Nothing retries. Delete what only served the old lanes: the tap proxy and patched Maestro runner, lane health and first-attempt checks, the simulator daemon, spotlight and accessibility tweaks (except disabling apsd and chronod, which slowed the first launch), the JavaScript re-embed scripts, the pixel diff and PR media, the perf probe, the native size measurements, the old flows, and the test/ci tests and config entries for all of them. The footprint job becomes a report-only PR comment of line counts and JavaScript bundle sizes. CI can be dispatched by hand, which is how the suite's determinism is checked.
Sign-out now lives on Profile, so the smoke flow signs out there. Keep the Library sort id the new tests read, drop the harness exclusion the new glyph check used, and leave the old account and library flows deleted.
pnpm audit no longer reports GHSA-vfj7-8cjw-p6xm against the locked braces, so check-stale-audit-ignores fails on any change that touches the lockfile.
Show detail opens with the next episode's season expanded and lists seasons newest first, so Season 1 starts below the fold and Season 3's episodes sit above it.
…rd-expo-ci # Conflicts: # packages/native/__tests__/perf-channel.test.tsx # packages/native/__tests__/runtime-boot.test.tsx # packages/native/src/ui/perf-channel.ts
On Android, Maestro did not find Season 1's episode rows after the season was expanded, although the screen showed them. Expanding a season is covered by season-row.test.tsx, so the smoke flow no longer scrolls to and expands Season 1.
A run failed because Android's launcher (Quickstep) stopped responding over Cue's welcome screen on a freshly booted emulator. Cue itself had drawn its first screen in 2.8 s. The ATD system image is Android's image for automated tests: it removes Launcher3QuickStep, SystemUI and other apps tests do not use (developer.android.com/studio/test/managed-devices#gmd-atd). android-emulator-runner tests itself on aosp_atd at API 34.
The ATD image captured every Android screenshot blank, because it turns off hardware rendering. Instead the emulator gets the runner's four vCPUs (android-emulator-runner's cores input; the default is two), since the launcher stopped responding on a software-rendered emulator. Repeat runs will show whether that is enough.
Two of three runs failed on "Quickstep isn't responding": Android's launcher, not Cue, stopped responding over Cue's screen. Four emulator cores did not change that. Android's Automated Test Device image leaves Launcher3QuickStep out because tests do not need a launcher (developer.android.com/studio/test/managed-devices#gmd-atd), but that image also turns off hardware rendering and captured every screenshot blank. So the default image keeps rendering and the flows disable the launcher package.
In 50 repeat runs, two iOS taps went to SpringBoard's app switcher right after Maestro stopped and relaunched Cue between flows, so later flows now launch with stopApp: false and switch to Up Next. The simulator now runs the iOS runtime matching the selected Xcode SDK; it had picked iOS 26.4. The sign-out flow stops at the confirmation dialog: the confirm tap was cancelled by the simulator in one run, and profile.test.tsx covers confirming sign-out. The Android emulator package is cached with actions/cache (docs.github.com/actions/reference/workflows-and-actions/dependency-caching), since one run failed unpacking a fresh download before the emulator started.
…rd-expo-ci # Conflicts: # packages/native/__tests__/ui/pull-trace.test.tsx # packages/native/src/ui/harness.tsx
In one of 50 repeat runs the tap reached Cue and the dialog appeared, but the simulator's accessibility service stalled for about 33 s, past the 30 s wait.
After the fixable causes were removed, 50 repeat runs passed 50/50 on Android and 48/50 on iOS; the remaining iOS failure is the simulator cancelling a synthesized tap, which has no documented fix (Maestro's iOS tap retry is a no-op, mobile-dev-inc/Maestro#3685). As EAS Maestro jobs do with retries: 1, a failed iOS run is repeated once after resetting the fake Trakt server and reinstalling the app. Both runs' screenshots and logs are uploaded, and a real defect still fails both. Android stays a single run.
The Android smoke flow now swipes a queue row to mark it, so a gesture library change that broke or inverted the swipe fails on a device rather than only in Jest's fake. Gradle's cache is restored only when the native build cache missed, instead of 1.6 GB on every run, and the emulator package cache is keyed on the runner image version. Also drops an unused permission, an upload path nothing reads, a stale comment basis and an unused spelling entry.
Zod before 4.6.6 can exhaust memory validating a large array against an uncapped array schema; Cue's core parses Trakt responses with it. A pnpm override moves every copy to 4.6.6 or later, including Expo CLI's zod 3, which only uses basic schema builders (string, object, enum, array, literal, union) in its dev-tools plugin. The braces audit ignore no longer applies and is removed.
Pull request footprint
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Replace the custom device lanes with six ordered Maestro smoke flows on GitHub Actions. Expo documents running Maestro against a built app and using fingerprint and repack to reuse a compatible native build. This PR uses those tools and removes the repository's tap proxy, patched runner, lane tracking, screenshot diff pipeline and native performance probe.
What CI runs
pnpm check: lint, formatting, spelling, types, architecture, unused code, duplication, quality budget, bundle checks, Vitest and native Jest.reactivecircus/android-emulator-runnerand the fake Trakt server.Pull requests run the checks without draft or path filters; a new push cancels the PR's older run. Pushes on
main,feat/expo-nativeandrelease/*seed native build caches. The workflow also supports scheduled and manual runs. The footprint comment is limited to same-repository, non-Dependabot pull requests.Maestro is the stock CLI, pinned to 2.11.0. Android runs the suite once. If the iOS suite fails, it runs once more after resetting the fake server and reinstalling the app, following the one-retry policy offered by EAS
retries: 1. The flows have no retries, optional steps orretryTapIfNoChange. They wait for and assert on-screen results; these assertions do not independently prove persistence across a restart or server synchronization.The Android launcher is disabled before testing. This follows Google's ATD rationale: remove or disable unrelated apps and services, including Launcher3QuickStep, to reduce resource use during app tests. This job still uses the default API 35 image, not an ATD image. On iOS, apsd and chronod are disabled during simulator setup.
Native builds are saved immediately after compilation so a later smoke failure does not discard them. Gradle's caches and wrapper are restored only when the Android native build cache misses, using the runner OS and lockfile hash. The emulator package cache includes
env.ImageVersion, which runner-images documents as available on GitHub-hosted runners. The AVD snapshot cache remains separate.Both platforms upload screenshots on every non-cancelled run. Android uploads Maestro debug output and the fake-server log on failure. iOS uploads debug output and the fake-server log on every non-cancelled run, including both attempts when a rerun occurs.
Six smoke flows
The flows run in that order. Only sign-in cold-starts the app; the other flows preserve the running app and return to Up Next. Screenshots capture the visited screens in light mode.
Removed
workflow_dispatchonci.yml..maestro/ci,.maestro/flowsand.maestro/perfflows, harness-only Jest tests, unused test IDs, obsolete configuration entries and the unusedxctestspelling entry.test/ci(36 test suites and 11 fixtures), plustest/support/leaked-processes.ts. Remaining CI tests are adapted to the new jobs. The stall guard keeps its 150-second inactivity limit without referring to the deleted PR media test.scripts/mock-traktandpackages/core/test/harnessare unchanged. The unused hidden Atlas file is no longer an upload path, and the footprint artifact download destination matches the bundle directory. The footprint job no longer requestsactions: readfor a same-run download.Where each deleted flow's checks live now
Both new tests were checked against mutations: making
TvShowsOffpush/profilefails the first, and makingclassifyUpNextShowreturnnullor"lapsed"for a pending advance fails the second.Deliberate reductions
Net lines (against the base)
These counts use
git diff --numstatfrom the merge base withorigin/feat/expo-nativethrough the final working tree, including the review edits awaiting commit. They count physical lines, including blanks and comments. The base is8af0c8c4.packages/*/src,packages/native/app, Metro and Babel config)packages/*/test,packages/native/__tests__,test,.maestro).github,scripts)Tracked text-file line totals, before and after (binary assets excluded):
maestro-ids.test.tsValidation
pnpm exec actionlint: passed with actionlint 1.7.12.pnpm exec vitest run test/ci: 28 files and 125 tests passed.heavy-lock.sh run ci-followup -- pnpm check: passed with Node 22.23.4. Vitest: 127 files, 1,192 passes and 15 expected failures. Native Jest: 138 suites and 1,078 tests passed; 2 suites and 6 tests skipped.git diff --check: passed.Dependency advisory
This pull request also carries the fix for GHSA-mrhx-c7xc-m3w5 (zod before 4.6.6 can exhaust memory validating a large array), because it changes the lockfile and the audit gates on it.
packages/coreasks for^4.6.6and a pnpm override moves Expo CLI's zod 3 copy too; it only uses basic schema builders in its dev-tools plugin. The lockfile now holds a single zod, 4.6.7. The braces audit ignore no longer applies and is removed.Merging this PR
Before merging, update the
feat/expo-nativebranch ruleset's required checks:native-android,native-ios,native-e2e,ui-screenshots-ios-dark,ui-screenshots-android-darkandfootprint.ios-e2eandandroid-e2eif missing..github/required-checks.jsonalready lists the new job names for release validation.Apply the same ruleset change to
mainwhenfeat/expo-nativelands there. This PR does not change GitHub's branch rulesets automatically.