Skip to content

Commit 28b4779

Browse files
committed
test(example): iOS re-verification of the findings; README matches
Baselines hold bit-for-bit on Xcode 27.0 once dev-client chrome is off; the realistic break still measures 10,179 px at 0.02 and 0 at 0.1. Corrections from the iOS pass: the iOS dev menu is a one-time onboarding scrim, not a per-relaunch sheet; the floating Tools button also crosses the iOS crop (3,916 px); scroll bottom succeeds on a clean screen; the stale-ref hint quote was wrong; a no-match --scope returns empty nodes with success. README no longer calls any finding an agent-device issue.
1 parent 21dc2ab commit 28b4779

2 files changed

Lines changed: 12 additions & 13 deletions

File tree

‎example/visual/README.md‎

Lines changed: 6 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -98,24 +98,23 @@ The same loop as one script for both platforms, with device pinning, overlay han
9898

9999
## What a real runner has to handle
100100

101-
Each learned while scripting the loop above; documented in `evidence/issues.md`, implemented on the runner branch:
101+
Each learned while scripting the loop above and documented in `evidence/issues.md`; implemented on the runner branch. None of them is an agent-device bug:
102102

103-
- Relaunch before every capture. Fast Refresh silently stopped reaching the Android app; without a fresh bundle a stale screen reads as PASS.
103+
- Reload before every capture with `agent-device metro reload`, then `open --relaunch`. Fast Refresh silently stopped reaching the Android app; without a fresh bundle a stale screen reads as PASS.
104104
- The example app persists navigation state. A relaunch lands on the last screen, so the runner has to press Back to the list root and pick the Surface row, not the header title.
105105
- Dev-client chrome: the dev menu appears seconds after the app is ready, the Android relaunch lands in the dev launcher, and the floating Tools button can appear inside the crop.
106-
- `snapshot` returns an empty `nodes` array when nothing changed unless `--force-full` is passed.
107-
- Set `--threshold` explicitly (0.02 for soft shadows) and verify noise at it; the default hides real regressions.
106+
- Set `--threshold` explicitly (0.02 for soft shadows) and verify noise at it; the default 0.1 is documented as a 44-unit RGB tolerance, far looser than a one-step shadow change.
108107
- Pin the device (UDID, runtime, density) and refuse to compare or re-baseline on anything else; derive expected capture size from the baseline PNG, per story.
109108
- A plain `.mjs` entry guarded by `import.meta.main` does nothing on Node 20/22; guard with an `argv[1]` comparison.
110109

111-
## agent-device issues
110+
## agent-device findings
112111

113-
Nineteen findings with commands and evidence in `evidence/issues.md`, none filed upstream. The four that matter most: the default diff threshold misses soft-shadow regressions (4), sessions are bound to one device per cwd (6), dev-client chrome renders into screenshots and the diff cannot tell (18), `--crop-on` is refused on web (19).
112+
Nineteen observations were recorded during the PoC and every one was re-verified on 2026-09-15 against the 0.21.3 docs and source, the devices, and the upstream tracker (`evidence/issues.md`). None was an unreported agent-device bug. One was a real ergonomic gap, sessions keyed by cwd only [6], and has been fixed upstream. The threshold result [4] stands as our calibration of documented behaviour, with one small ask left: the diff JSON does not record the threshold it was run at. Everything else was documented behaviour, our own build or runner, React Native, or did not survive re-testing; the file says which.
114113

115114
## Evidence
116115

117116
- `evidence/results.csv`: one row per `diff screenshot` run (72 rows): platform, capture, threshold, total and changed pixels, mismatch %, regions, match, and the name of the raw JSON it came from. The raw per-command JSON is on the runner branch.
118-
- `evidence/diff-images/`: one diff image per platform for the realistic break (ring on the Elevation 1 card) and the gross break, both at 0.02, the dev-client Tools-button false FAIL, and the full-page web capture. No 0.1 images exist because `diff screenshot --out` writes nothing on a match (issue 17).
117+
- `evidence/diff-images/`: one diff image per platform for the realistic break (ring on the Elevation 1 card) and the gross break, both at 0.02, the dev-client Tools-button false FAIL, and the full-page web capture. No 0.1 images exist because `diff screenshot --out` writes nothing on a match and deletes any stale file at that path [17].
119118
- `evidence/a11y-excerpt.json`, `evidence/devclient-excerpt.json`, `evidence/web-excerpt.json`: the nodes and responses that matter; full trees on the runner branch.
120119
- `env.json`: the pinned devices, versions and thresholds.
121120

‎example/visual/evidence/issues.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# agent-device findings from the PoC, verified
22

3-
Nineteen observations were recorded while using agent-device 0.21.0 (`npx agent-device@0.21.0`) as a visual regression tool on the example app, on an iPhone 17 Pro simulator (iOS 26.5) and a `Pixel_10_Pro` emulator (API 37). On 2026-09-15 every one was re-checked against the 0.21.3 docs and source, the emulator, and the upstream issue tracker, with one question: is this agent-device's fault, or ours?
3+
Nineteen observations were recorded while using agent-device 0.21.0 (`npx agent-device@0.21.0`) as a visual regression tool on the example app, on an iPhone 17 Pro simulator (iOS 26.5) and a `Pixel_10_Pro` emulator (API 37). On 2026-09-15 and 16 every one was re-checked against the 0.21.3 docs and source, the Android emulator and the iOS simulator (Xcode 27.0), and the upstream issue tracker, with one question: is this agent-device's fault, or ours?
44

55
Result: **none was an unreported agent-device bug.** One was a real ergonomic gap and has since been fixed upstream. Two are documented behaviour worth knowing. The rest were our own build, our runner, React Native, or claims that did not survive re-testing. They are kept below because most of them are things anyone scripting this loop will hit. Original numbers are in brackets so the README references still resolve.
66

@@ -16,26 +16,26 @@ Result: **none was an unreported agent-device bug.** One was a real ergonomic ga
1616

1717
## Our build and our runner, not the tool
1818

19-
**Expo dev-client chrome ends up in screenshots [3, 9, 14, 18].** A dev build cold-starts into the Expo dev launcher on Android and shows the dev menu on iOS a few seconds after the bundle loads. Reproduced with plain `adb shell am force-stop` and `monkey`, no agent-device involved. The dev-client's floating "Tools" button, toggled by the dev menu entry "Tools button", sat inside the Android crop and produced a deterministic 2,822 px false FAIL with `src/` clean (`diff-images/android-devclient-tools-button.png`, `devclient-excerpt.json`). agent-device already flags the loading state: `wait stable` returned "Settled on a nearly-empty tree, the app may still be loading. Wait for specific content." A runner must reconnect to the dev server after any relaunch, wait for real content, dismiss the menu, and check for the floating button. A release build removes the whole class.
19+
**Expo dev-client chrome ends up in screenshots [3, 9, 14, 18].** Three distinct things, all the dev-client's, none agent-device's. On Android a cold start lands in the Expo dev launcher; reproduced with plain `adb shell am force-stop` and `monkey`, no agent-device involved, and there `wait stable` does warn: "Settled on a nearly-empty tree, the app may still be loading." On iOS the dev menu is not shown on every relaunch as this file first claimed; it appears once, as a first-run onboarding sheet that dims the whole app behind a scrim, and `wait stable` reports that as settled with no warning because the tree has real content (39 nodes) under the scrim. Every capture behind it differs 100 %. The floating "Tools" button, toggled by the dev menu entry "Tools button", sits inside the crop on both platforms: 2,822 px on Android (`diff-images/android-devclient-tools-button.png`), 3,916 px on iOS where its bottom arc crosses the crop's top edge; the `flat` section lower on the same screen diffs 0 in both cases. A runner must reconnect to the dev server after a relaunch, dismiss whatever sheet is up, turn the floating button off, and only then capture. A release build removes the whole class.
2020

21-
**Fast Refresh did not reach the Android app after a relaunch [15].** Metro served the new bundle; the app kept the old one until a Reload. Metro and Expo behaviour, and agent-device documents the remedy we did by hand through the dev menu: `agent-device metro reload`, "before `open <app> --relaunch`". A runner should use that, never Fast Refresh.
21+
**Fast Refresh did not reach the Android app after a relaunch [15].** Metro served the new bundle; the app kept the old one until a Reload. Metro and Expo behaviour, and agent-device documents the remedy we did by hand through the dev menu: `agent-device metro reload`, "before `open <app> --relaunch`". Verified on 0.21.0 on iOS: `reloaded: true` and the edited shadow appeared on the next capture. A runner should use that, never Fast Refresh.
2222

2323
**The example app restores its last route on relaunch [13].** `PERSISTENCE_KEY = 'NAVIGATION_STATE'` in `example/src/index.tsx`. A runner cannot assume the list is at the top after `open --relaunch`; it has to press Back to the root and pick the "Surface" row, not the header title with the same label.
2424

2525
**Android exposes the same label on a row and its text child [7].** React Native's accessibility output, visible on every example row and on the Expo launcher itself, so `find 'label="Surface"'` is `AMBIGUOUS_MATCH` there and unique on iOS. agent-device documents the disambiguators: `--first`, `--last`, a `role=` qualifier, and the read-only `list` action that returns the candidates without tapping.
2626

27-
**`scroll bottom` hit its pass limit on the example list [2].** A documented, bounded edge-seek that refuses to swipe forever; the dev menu was on screen absorbing the gestures, which is also why the edge was never reached. Use `scroll down 6 --settle`, or `scroll <dir> --until <selector>` on 0.21.1 and later.
27+
**`scroll bottom` hit its pass limit on the example list [2].** On a clean screen it reaches the bottom in two passes and returns success (re-run on iOS, `passes: 2`). The failure we saw was the dev-client sheet absorbing the gestures; the pass limit is a documented, bounded edge-seek doing its job. Nothing to change in the tool; a runner should dismiss overlays first.
2828

2929
## Withdrawn after re-verification
3030

3131
These were recorded as agent-device problems and are not.
3232

3333
- **`find` taps its match [1].** `find` is documented as `find <locator> <action>` with `click` as the default action and a read-only `list` action that never taps. Upstream #1625 raised the same complaint and was closed in August with `list` as the answer. We ran bare `find` and got the documented default.
3434
- **No timing in `screenshot --json` [5].** Per-command durations are in the session event log (`agent-device events`, `request.finished` with `durationMs`); `open` returns `startup.durationMs` as well.
35-
- **Stale refs rejected with a confusing message [8].** The response carries `hint: "Capture a fresh interactive snapshot (snapshot -i) or use a stable selector, then retry."`, a reason code, and the set of refs the frame does accept. The message says exactly what to do.
35+
- **Stale refs rejected with a confusing message [8].** Every rejection carries a `hint` and a `details.reason` (three seen: `ref_generation_mismatch`, `plain_ref_requires_complete_frame`, an expired frame), for example `"Ref @e12 was minted from snapshot s210062 but the session's ref frame is now s210063 — re-run snapshot -i."` The message says exactly what happened and what to do.
3636
- **`logs` reports success on an inactive log [10].** Bare `logs` is documented as an info query; `success: true` means the query ran, and `active`, `state` and `sizeBytes` are the answer. Logging is documented as off until `logs start`.
3737
- **`--pixel-density` is iOS-only [11].** Documented in the CLI help; Android returns native pixels anyway.
38-
- **`snapshot` returns empty `nodes` when the tree is unchanged [12].** Not reproducible: two consecutive `snapshot --json` returned the full 57 nodes both times, and the source never empties `nodes`, only adds an `unchanged` metadata field. The text-mode "unchanged" acknowledgement is documented. The empty result we saw most likely came from a `--scope` that matched nothing.
38+
- **`snapshot` returns empty `nodes` when the tree is unchanged [12].** Not reproducible: two consecutive `snapshot --json` returned the full tree both times (57 nodes on Android, 22 on iOS), and the source never empties `nodes`; the "unchanged" acknowledgement is text-mode only and documented. The empty result we saw was a `--scope` that matched nothing: `snapshot --json --scope 'id="does-not-exist"'` returns `success: true`, `nodes: []`, `totalNodeCount: 0`, with no diagnostic. That is the one thing worth a sentence for a runner: assert on node count, a no-match scope is not an error.
3939
- **Occasional exit 1 with empty stdout and stderr [16].** Two unlogged occurrences on iOS; 0 of 40 rapid calls on Android reproduced it; nothing upstream. Not supportable.
4040

4141
## Filing history

0 commit comments

Comments
 (0)