You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 28b4779
Browse filesBrowse the repository at this point in the historyBrowse files
test(example): iOS re-verification of the findings; README matches
Baselines hold bit-for-bit on Xcode 27.0 once dev-client chrome is off; the realistic break still measures 10,179 px at 0.02 and 0 at 0.1. Corrections from the iOS pass: the iOS dev menu is a one-time onboarding scrim, not a per-relaunch sheet; the floating Tools button also crosses the iOS crop (3,916 px); scroll bottom succeeds on a clean screen; the stale-ref hint quote was wrong; a no-match --scope returns empty nodes with success. README no longer calls any finding an agent-device issue.
Copy file name to clipboardExpand all lines: example/visual/README.md
+6-7Lines changed: 6 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -98,24 +98,23 @@ The same loop as one script for both platforms, with device pinning, overlay han
98
98
99
99
## What a real runner has to handle
100
100
101
-
Each learned while scripting the loop above; documented in `evidence/issues.md`, implemented on the runner branch:
101
+
Each learned while scripting the loop above and documented in `evidence/issues.md`; implemented on the runner branch. None of them is an agent-device bug:
102
102
103
-
-Relaunch before every capture. Fast Refresh silently stopped reaching the Android app; without a fresh bundle a stale screen reads as PASS.
103
+
-Reload before every capture with `agent-device metro reload`, then `open --relaunch`. Fast Refresh silently stopped reaching the Android app; without a fresh bundle a stale screen reads as PASS.
104
104
- The example app persists navigation state. A relaunch lands on the last screen, so the runner has to press Back to the list root and pick the Surface row, not the header title.
105
105
- Dev-client chrome: the dev menu appears seconds after the app is ready, the Android relaunch lands in the dev launcher, and the floating Tools button can appear inside the crop.
106
-
-`snapshot` returns an empty `nodes` array when nothing changed unless `--force-full` is passed.
107
-
- Set `--threshold` explicitly (0.02 for soft shadows) and verify noise at it; the default hides real regressions.
106
+
- Set `--threshold` explicitly (0.02 for soft shadows) and verify noise at it; the default 0.1 is documented as a 44-unit RGB tolerance, far looser than a one-step shadow change.
108
107
- Pin the device (UDID, runtime, density) and refuse to compare or re-baseline on anything else; derive expected capture size from the baseline PNG, per story.
109
108
- A plain `.mjs` entry guarded by `import.meta.main` does nothing on Node 20/22; guard with an `argv[1]` comparison.
110
109
111
-
## agent-device issues
110
+
## agent-device findings
112
111
113
-
Nineteen findings with commands and evidence in `evidence/issues.md`, none filed upstream. The four that matter most: the default diff threshold misses soft-shadow regressions (4), sessions are bound to one device per cwd (6), dev-client chrome renders into screenshots and the diff cannot tell (18), `--crop-on` is refused on web (19).
112
+
Nineteen observations were recorded during the PoC and every one was re-verified on 2026-09-15 against the 0.21.3 docs and source, the devices, and the upstream tracker (`evidence/issues.md`). None was an unreported agent-device bug. One was a real ergonomic gap, sessions keyed by cwd only [6], and has been fixed upstream. The threshold result [4] stands as our calibration of documented behaviour, with one small ask left: the diff JSON does not record the threshold it was run at. Everything else was documented behaviour, our own build or runner, React Native, or did not survive re-testing; the file says which.
114
113
115
114
## Evidence
116
115
117
116
-`evidence/results.csv`: one row per `diff screenshot` run (72 rows): platform, capture, threshold, total and changed pixels, mismatch %, regions, match, and the name of the raw JSON it came from. The raw per-command JSON is on the runner branch.
118
-
-`evidence/diff-images/`: one diff image per platform for the realistic break (ring on the Elevation 1 card) and the gross break, both at 0.02, the dev-client Tools-button false FAIL, and the full-page web capture. No 0.1 images exist because `diff screenshot --out` writes nothing on a match (issue 17).
117
+
-`evidence/diff-images/`: one diff image per platform for the realistic break (ring on the Elevation 1 card) and the gross break, both at 0.02, the dev-client Tools-button false FAIL, and the full-page web capture. No 0.1 images exist because `diff screenshot --out` writes nothing on a match and deletes any stale file at that path [17].
119
118
-`evidence/a11y-excerpt.json`, `evidence/devclient-excerpt.json`, `evidence/web-excerpt.json`: the nodes and responses that matter; full trees on the runner branch.
120
119
-`env.json`: the pinned devices, versions and thresholds.
Copy file name to clipboardExpand all lines: example/visual/evidence/issues.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# agent-device findings from the PoC, verified
2
2
3
-
Nineteen observations were recorded while using agent-device 0.21.0 (`npx agent-device@0.21.0`) as a visual regression tool on the example app, on an iPhone 17 Pro simulator (iOS 26.5) and a `Pixel_10_Pro` emulator (API 37). On 2026-09-15 every one was re-checked against the 0.21.3 docs and source, the emulator, and the upstream issue tracker, with one question: is this agent-device's fault, or ours?
3
+
Nineteen observations were recorded while using agent-device 0.21.0 (`npx agent-device@0.21.0`) as a visual regression tool on the example app, on an iPhone 17 Pro simulator (iOS 26.5) and a `Pixel_10_Pro` emulator (API 37). On 2026-09-15 and 16 every one was re-checked against the 0.21.3 docs and source, the Android emulator and the iOS simulator (Xcode 27.0), and the upstream issue tracker, with one question: is this agent-device's fault, or ours?
4
4
5
5
Result: **none was an unreported agent-device bug.** One was a real ergonomic gap and has since been fixed upstream. Two are documented behaviour worth knowing. The rest were our own build, our runner, React Native, or claims that did not survive re-testing. They are kept below because most of them are things anyone scripting this loop will hit. Original numbers are in brackets so the README references still resolve.
6
6
@@ -16,26 +16,26 @@ Result: **none was an unreported agent-device bug.** One was a real ergonomic ga
16
16
17
17
## Our build and our runner, not the tool
18
18
19
-
**Expo dev-client chrome ends up in screenshots [3, 9, 14, 18].** A dev build cold-starts into the Expo dev launcher on Android and shows the dev menu on iOS a few seconds after the bundle loads. Reproduced with plain `adb shell am force-stop` and `monkey`, no agent-device involved. The dev-client's floating "Tools" button, toggled by the dev menu entry "Tools button", sat inside the Android crop and produced a deterministic 2,822 px false FAIL with `src/` clean (`diff-images/android-devclient-tools-button.png`, `devclient-excerpt.json`). agent-device already flags the loading state: `wait stable` returned "Settled on a nearly-empty tree, the app may still be loading. Wait for specific content." A runner must reconnect to the dev server after any relaunch, wait for real content, dismiss the menu, and check for the floating button. A release build removes the whole class.
19
+
**Expo dev-client chrome ends up in screenshots [3, 9, 14, 18].** Three distinct things, all the dev-client's, none agent-device's. On Android a cold start lands in the Expo dev launcher; reproduced with plain `adb shell am force-stop` and `monkey`, no agent-device involved, and there `wait stable` does warn: "Settled on a nearly-empty tree, the app may still be loading." On iOS the dev menu is not shown on every relaunch as this file first claimed; it appears once, as a first-run onboarding sheet that dims the whole app behind a scrim, and `wait stable` reports that as settled with no warning because the tree has real content (39 nodes) under the scrim. Every capture behind it differs 100 %. The floating "Tools" button, toggled by the dev menu entry "Tools button", sits inside the crop on both platforms: 2,822 px on Android (`diff-images/android-devclient-tools-button.png`), 3,916 px on iOS where its bottom arc crosses the crop's top edge; the `flat` section lower on the same screen diffs 0 in both cases. A runner must reconnect to the dev server after a relaunch, dismiss whatever sheet is up, turn the floating button off, and only then capture. A release build removes the whole class.
20
20
21
-
**Fast Refresh did not reach the Android app after a relaunch [15].** Metro served the new bundle; the app kept the old one until a Reload. Metro and Expo behaviour, and agent-device documents the remedy we did by hand through the dev menu: `agent-device metro reload`, "before `open <app> --relaunch`". A runner should use that, never Fast Refresh.
21
+
**Fast Refresh did not reach the Android app after a relaunch [15].** Metro served the new bundle; the app kept the old one until a Reload. Metro and Expo behaviour, and agent-device documents the remedy we did by hand through the dev menu: `agent-device metro reload`, "before `open <app> --relaunch`". Verified on 0.21.0 on iOS: `reloaded: true` and the edited shadow appeared on the next capture. A runner should use that, never Fast Refresh.
22
22
23
23
**The example app restores its last route on relaunch [13].**`PERSISTENCE_KEY = 'NAVIGATION_STATE'` in `example/src/index.tsx`. A runner cannot assume the list is at the top after `open --relaunch`; it has to press Back to the root and pick the "Surface" row, not the header title with the same label.
24
24
25
25
**Android exposes the same label on a row and its text child [7].** React Native's accessibility output, visible on every example row and on the Expo launcher itself, so `find 'label="Surface"'` is `AMBIGUOUS_MATCH` there and unique on iOS. agent-device documents the disambiguators: `--first`, `--last`, a `role=` qualifier, and the read-only `list` action that returns the candidates without tapping.
26
26
27
-
**`scroll bottom` hit its pass limit on the example list [2].**A documented, bounded edge-seek that refuses to swipe forever; the dev menu was on screen absorbing the gestures, which is also why the edge was never reached. Use `scroll down 6 --settle`, or `scroll <dir> --until <selector>` on 0.21.1 and later.
27
+
**`scroll bottom` hit its pass limit on the example list [2].**On a clean screen it reaches the bottom in two passes and returns success (re-run on iOS, `passes: 2`). The failure we saw was the dev-client sheet absorbing the gestures; the pass limit is a documented, bounded edge-seek doing its job. Nothing to change in the tool; a runner should dismiss overlays first.
28
28
29
29
## Withdrawn after re-verification
30
30
31
31
These were recorded as agent-device problems and are not.
32
32
33
33
-**`find` taps its match [1].**`find` is documented as `find <locator> <action>` with `click` as the default action and a read-only `list` action that never taps. Upstream #1625 raised the same complaint and was closed in August with `list` as the answer. We ran bare `find` and got the documented default.
34
34
-**No timing in `screenshot --json`[5].** Per-command durations are in the session event log (`agent-device events`, `request.finished` with `durationMs`); `open` returns `startup.durationMs` as well.
35
-
-**Stale refs rejected with a confusing message [8].**The response carries `hint: "Capture a fresh interactive snapshot (snapshot -i) or use a stable selector, then retry."`, a reason code, and the set of refs the frame does accept. The message says exactly what to do.
35
+
-**Stale refs rejected with a confusing message [8].**Every rejection carries a `hint` and a `details.reason` (three seen: `ref_generation_mismatch`, `plain_ref_requires_complete_frame`, an expired frame), for example `"Ref @e12 was minted from snapshot s210062 but the session's ref frame is now s210063 — re-run snapshot -i."` The message says exactly what happened and what to do.
36
36
-**`logs` reports success on an inactive log [10].** Bare `logs` is documented as an info query; `success: true` means the query ran, and `active`, `state` and `sizeBytes` are the answer. Logging is documented as off until `logs start`.
37
37
-**`--pixel-density` is iOS-only [11].** Documented in the CLI help; Android returns native pixels anyway.
38
-
-**`snapshot` returns empty `nodes` when the tree is unchanged [12].** Not reproducible: two consecutive `snapshot --json` returned the full 57 nodes both times, and the source never empties `nodes`, only adds an `unchanged` metadata field. The text-mode "unchanged" acknowledgement is documented. The empty result we saw most likely came from a `--scope` that matched nothing.
38
+
-**`snapshot` returns empty `nodes` when the tree is unchanged [12].** Not reproducible: two consecutive `snapshot --json` returned the full tree both times (57 nodes on Android, 22 on iOS), and the source never empties `nodes`; the "unchanged" acknowledgement is text-mode only and documented. The empty result we saw was a `--scope` that matched nothing: `snapshot --json --scope 'id="does-not-exist"'` returns `success: true`, `nodes: []`, `totalNodeCount: 0`, with no diagnostic. That is the one thing worth a sentence for a runner: assert on node count, a no-match scope is not an error.
39
39
-**Occasional exit 1 with empty stdout and stderr [16].** Two unlogged occurrences on iOS; 0 of 40 rapid calls on Android reproduced it; nothing upstream. Not supportable.
0 commit comments