diff --git a/.claude/skills/video-smoke-test/SKILL.md b/.claude/skills/video-smoke-test/SKILL.md new file mode 100644 index 000000000..8fe7fa41f --- /dev/null +++ b/.claude/skills/video-smoke-test/SKILL.md @@ -0,0 +1,316 @@ +--- +name: video-smoke-test +description: > + Run a basic smoke test of the Stream Video Flutter SDK by driving the dogfooding app through a + two-party call on an Android emulator and an iOS simulator. Covers join, mute/unmute, microphone + input and dominant speaker, speaking-while-muted detection, camera on/off, reactions, video + filter, stats, chat, Picture-in-Picture, reconnect after network loss, deep link joins, and clean + teardown. Does not test ringing. +disable-model-invocation: true +allowed-tools: + - Bash + - Read + - mcp__Claude_Code_iOS_Simulator__control +--- + +# Video SDK smoke test + +Drive the dogfooding app through one two-party call and assert SDK behaviour from logs. +Android is fully scripted. iOS is driven by you through the simulator tool using the +**exact coordinates in the tables below**. + +Budget: ~2 minutes of checks once both apps are built. The iOS build dominates a cold run. + +## Prerequisites + +Three things live outside the scripts, and without them the audio checks fail: + +- **Host microphone access on the AVD**, and **restart the emulator after enabling it**. Emulator + window > Extended controls (`...`) > Microphone > **"Enable Host Microphone Access"**. The + setting only takes effect on a fresh boot — enabling it on a running emulator leaves the + microphone silent, which is exactly what a broken SDK looks like. The emulator console has no + microphone command, so nothing can verify this for you. +- **macOS must allow the emulator to use the microphone**: System Settings > Privacy & Security > + Microphone. Easy to miss, because macOS does not fail an unpermitted recording — it returns + silence. +- **The host must be able to make an audible noise.** The audio checks speak through the Mac's + speakers (`say`) so the host microphone has something to hear. Output volume up, input not muted, + no headphones in the way. + +Even with all three in place, microphone input into the emulator is **intermittent**: the same AVD +instance delivered audio for two runs and then went silent for a third with no configuration +change, and only an emulator restart brought it back. So `dominant-speaker-a2b` and +`swm-detect-android` report `KNOWN-FAIL` — *not verified this run* — when no audio arrives, rather +than failing the run. A `PASS` on either is real news: it means host sound travelled the whole +path. A `KNOWN-FAIL` means check the three things above, and restart the emulator. No other check +is affected either way. + +## Rules — read before doing anything + +1. **Do not investigate failures during the run.** A check fails, you record it and move on. + Root-cause only if the user asks afterwards. +2. **Do not read logs looking for problems.** The scripts assert what matters. Do not grep for + extra errors, do not comment on warnings. +3. **Do not interpret screenshots.** They are evidence for the human. The only screenshot you look + at is the one in step 4. +4. **Do not improvise coordinates.** Every tap is in a table here. If a tap misses, report `FAIL` + for that step and continue — do not hunt for the control. +5. **Do not deviate from the step order.** State depends on it: sheets stay open, keyboards move + layouts. +6. **Never send a second BACK press on Android inside a call.** One closes the keyboard; a second + closes the sheet; a third leaves the call. +7. **Out of scope:** ringing, CallKit, push, screen sharing. Do not test them and do not list them + as gaps. +8. If a setup phase (preflight, launch, android_join) exits non-zero, **stop** and report the single + `FATAL:` line. Do not repair the environment beyond what the scripts already do. + +## Known non-issues — never report these as findings + +- **iOS simulator has no camera.** iOS joins audio-only with the camera control already red. Camera + on/off is an Android-only check. +- **iOS simulator supports neither PiP nor CallKit.** PiP is Android-only here. +- **Emulator camera output is blocky colour bands.** That is the AVD's synthetic camera. +- **Krisp logs `Failed to set wt file`** on the emulator. Emulator-specific. +- **A browsable `https` intent opens Chrome, not the app.** A debug build is not a verified App + Links handler, so `finish.sh` names the activity explicitly. Not a deep link defect. + +## Steps + +### 1. Preflight + +```bash +.claude/skills/video-smoke-test/scripts/preflight.sh +``` + +Pins the emulator serial (so a plugged-in phone cannot break the run), forces Flutter's Swift +Package Manager off (the iOS build cannot link with it on), and pre-grants mic/camera/notification/ +phone permissions on both platforms so **no system dialog can swallow a tap**. On `FATAL:`, relay +that line and stop. + +### 2. Launch both apps + +```bash +.claude/skills/video-smoke-test/scripts/launch.sh +``` + +Blocks until both report a Dart VM Service. Cold iOS run ~10 min (CocoaPods + compile); warm ~1 min. +Wipes Android app state so the login screen is guaranteed. It waits — do not poll it. + +### 3. Android: log in and create the call + +```bash +.claude/skills/video-smoke-test/scripts/android_join.sh +``` + +Prints `CALL_ID=`. Keep it for step 4. A `retry n/4` line is normal, not a failure. + +### 4. iOS: join the same call + +Take one screenshot. Confirm the tool reports a **402x874** coordinate space — if not, stop and ask +for an iPhone 17 simulator, because these coordinates will not work. + +Then pick a branch: + +- Shows **"Enter Username"** → do 4a, then 4b. +- Shows **"Start New Call"** with an **empty** call-id field → do 4b only. +- Shows a **non-empty call-id field** → the app is not freshly launched. The field cannot be cleared + with the tools available, so **re-run `launch.sh`** and start step 4 again. + +**4a. Log in** + +| # | Action | Coordinate | +|---|--------|-----------| +| 1 | tap username field | `201, 532` | +| 2 | type | `smoke_b1` | +| 3 | tap "Sign up with username" | `201, 594` | + +**4b. Join by call ID** + +| # | Action | Coordinate | +|---|--------|-----------| +| 1 | tap "Enter call id" field | `131, 622` | +| 2 | type | the `CALL_ID` from step 3 | +| 3 | tap "Join call" | `325, 621` | +| 4 | tap "Start a test call" | `201, 774` | + +Confirm with **one** command — do not screenshot to check: + +```bash +grep -c sfuJoinResponse /tmp/stream-video-smoke/ios.log +``` + +Non-zero means joined. Zero means `FAIL join-ios`; record it and continue anyway. + +### 5. Android: all in-call checks + +```bash +.claude/skills/video-smoke-test/scripts/android_checks.sh +``` + +Microphone level and dominant speaker, mute/unmute with speaking-while-muted detection, camera +off/on, reaction, blur filter, stats, PiP enter+restore, the full network-loss reconnect cycle, and +chat send plus its delivery to iOS. One `PASS`/`FAIL` line per check, ~3 min. Read only those lines. + +The audio checks talk out loud through the Mac's speakers — that is the script, not a stray +process. It stops on its own. + +**Run it exactly once per call.** The chat step deliberately leaves the chat sheet open, so a second +run's early taps would land on that sheet instead of the control bar. + +### 6. iOS: mute, unmute, chat, leave + +Control bar is at **y = 818**. If iOS is not showing the call, tap the app icon at `247, 223`. + +Step 7 is not optional. The chat sheet opened in step 3 covers the lower screen and dims the +rest; END CALL is visible above it but sits under the sheet's scrim, so tapping it just dismisses +the sheet and the call keeps running. Swipe the sheet away first. + +| # | Action | Coordinate | Why | +|---|--------|-----------|-----| +| 1 | tap mic (mute) | `158, 818` | must take the Apple ADM mute path, not stop the track | +| 2 | wait ~2s, tap mic (unmute) | `158, 818` | | +| 3 | tap chat | `371, 818` | opens the chat sheet | +| 4 | tap composer | `227, 815` | | +| 5 | type | `smoke_ios_1` | exact string — `finish.sh` greps for it | +| 6 | tap send | `361, 824` | | +| 7 | swipe down `201, 325` → `201, 790` | grabber → below | closes the chat sheet; END CALL is behind its scrim and a tap there is eaten by the sheet | +| 8 | tap end call | `371, 90` | | +| 9 | dismiss the feedback modal | `362, 181` | a "We Value Your Feedback!" sheet appears after every leave and blocks the screen | + +Do not verify any of this yourself — `finish.sh` asserts it from the log. + +### 7. Finish + +```bash +.claude/skills/video-smoke-test/scripts/finish.sh +``` + +Asserts the iOS-side behaviour, the iOS chat send and its delivery to Android, Android leave, then +joins the same call again through a deep link and leaves it, and finally that no call foreground +service or ongoing-call notification leaked. Prints the summary; exits non-zero on unexpected failure. +It also stops the run's background processes. + +### If the run is abandoned partway + +A failed phase is normally retried, so the phases between `launch.sh` and `finish.sh` deliberately +leave the apps running — killing them would cost another full build. When a run is given up on +instead of retried, run: + +```bash +.claude/skills/video-smoke-test/scripts/stop.sh +``` + +It stops `adb logcat` and both `flutter run` supervisors (with their build children), and re-enables +the emulator's network in case the run died inside the reconnect step. Logs and screenshots are +kept. `launch.sh` also calls it implicitly at the start of the next run. + +## Reporting + +Terminal only, in this order, nothing more: + +1. One line: `N passed, N failed, N known-fail` plus wall-clock. +2. **The table `finish.sh` prints, relayed verbatim** — one row per check, what it verifies, and + whether it works. Do not rebuild it by hand, do not reorder or reword rows, do not drop the + rows that passed: the point is that the reader sees the whole surface that was covered, not + only the damage. +3. One sentence per failure on what it means for the SDK. No speculation about cause. +4. The screenshot directory path. + +If everything passed, the table plus one sentence is the whole report. No padding, no +recommendations, no suggested follow-ups unless something failed. Publish an artifact **only if +asked**. + +## What each check proves + +| id | proves | +|----|--------| +| `join-android` / `join-ios` | SFU join handshake completes on both platforms | +| `publish-audio-*` / `publish-video-android` | local tracks reach the SFU | +| `peer-seen-*` | both clients see each other in call state | +| `dominant-speaker-a2b` | sound from the emulator's microphone travels capture → publish → SFU → peer, and the peer marks the speaking participant dominant — the state the "who is talking" indicator binds to | +| `mute-android` / `unmute-android` | mute unpublishes and republishes the audio track | +| `swm-detection-started-android` / `swm-detection-stopped-android` | muting arms the speaking-while-muted recognizer and unmuting tears it down | +| `swm-detect-android` | speech while muted actually fires the recognizer (needs the microphone prerequisites) | +| `adm-mute-ios` / `adm-unmute-ios` | iOS mutes at the audio-device-module level and keeps the track alive — the path speaking-while-muted detection depends on | +| `camera-off-android` / `camera-on-android` | video track unpublish/republish | +| `reaction-android` | custom event round-trips through the coordinator | +| `filter-android` | video frame processor initialises on the capture pipeline | +| `stats-android` | the stats route opens and closes without hanging or dropping the call (numbers are for a human, in shot 13) | +| `pip-android` / `pip-restore-android` | PiP entry and restore keep the call alive | +| `reconnect-detect-android` | network loss triggers reconnect / ICE restart scheduling | +| `reconnect-recover-android` | the call recovers and rejoins the SFU (fast-vs-full strategy is reported as info, not asserted — it legitimately varies) | +| `chat-send-android` / `chat-send-ios` | the call's chat channel accepts an outgoing message | +| `chat-a2b` / `chat-b2a` | the peer actually receives it — a `message.new` event lands on the other client | +| `deeplink-route-android` | a `/join/` link resolves the call and lands on its lobby, with no router error and no emptied navigation stack | +| `deeplink-join-android` | the call reached through a link joins the SFU | +| `deeplink-leave-android` | leaving a call a link opened tears down without emptying the stack | +| `leave-*` | call teardown runs | +| `no-service-leak` / `no-notif-leak` | no orphaned call foreground service or ongoing-call notification | + +## Traps already handled (do not "fix" these) + +These are encoded in the scripts for reasons found the hard way: + +- Android username must contain a digit/underscore — the emulator IME autocorrects `smokea` to + `smokes`. +- The soft keyboard covers the sign-up button; it is dismissed before tapping. +- Navigation taps use `tap_until` (retry until the log confirms), never a fixed `sleep`. +- Selecting a **filter leaves the options sheet open**; selecting a **reaction closes it**. The stats + step relies on the sheet still being open — do not add a `MORE` tap between them. +- **A modal sheet eats the first tap outside it.** The chat sheet (iOS and Android) and the options + sheet (Android) are modal: a tap on the dimmed area only closes that sheet, it does not reach the + control underneath. iOS closes the chat sheet with a swipe (step 6, row 7); Android's `finish.sh` + uses `tap_until` on END CALL so the first taps peel off the sheets and the last one lands. +- The stats screen is left via the **top-left** back arrow. The top-right X sits ~16px from END CALL, + and if stats ever fails to open that tap silently ends the call. +- PiP is detected from the task's current `mode=pinned`, never from a `PICTURE_IN_PICTURE_ENTER` + logcat line (stale forever) or from `windowingMode=pinned` (does not match in practice). +- **Chat receipt is asserted from the receiver's `message.new` event, never from the message text.** + No client logs the body of a message it *receives*, so grepping the peer's log for the text can + never pass — that assertion, not the SDK, is what made chat look like a "send-only" defect. Each + direction is checked in a window marked before that send, because a sender gets a `message.new` + for its own message too. +- **The Android send tap is retried across both composer rows** (`android_chat_send`): the soft + keyboard does not always open, which moves the SEND button, and a missed tap leaves the text in + the composer as a draft. Do not assert an Android send by grepping logcat for the bare message + text — logcat echoes the `input text` command itself, so that passes with nothing sent. +- **The service-leak check names `StreamCallService` and polls for 25s.** `grep ServiceRecord` also + matches the app's unrelated short-lived services and a call service still unwinding, so it reports + normal teardown as a leak. +- **The speaking-while-muted checks live inside the mute step and cannot be moved**: the recognizer + is only armed while the user is muted. They read the recognizer's own log lines + (`[startRecognition]`, `isSpeakingWhileMuted: true`, `[stopRecognition]`), because the detection + surfaces in the app only as a snackbar, which no log records. +- **The dominant-speaker check reads the iOS log while Android speaks.** The SFU reports a + participant's level and dominance to the *other* participants, not back to the speaker: Android + never sees its own, so asserting on the Android log looks right and fails with a perfectly good + microphone. It is the server's own measurement, so it cannot pass with a dead microphone the way + a track-publish line can. +- **Do not add an audio-level assertion.** Levels arrive as high-frequency events that update call + state without logging, and `audioLevel:` surfaces only inside the state dumps that track + publish/unpublish print — so a speech window usually contains nothing to match even with a + working microphone. That is what made an earlier `audio-level` check fail while the microphone was + demonstrably fine. Dominant speaker is the low-frequency signal that is actually logged. +- **A successful speaking-while-muted detection puts a snackbar over the control bar.** The app's + floating "You are muted. Unmute to speak." covers the bottom bar, so the next tap hits the + snackbar instead of the mic button. The mute step waits for it to expire — without that, unmute, + camera-on and the filter step all fail for reasons that have nothing to do with them. +- **The deep link is delivered to the activity by name**, not as a browsable intent: a debug build + is not a verified App Links handler, so a plain `am start` opens Chrome and the check would fail + for a reason that has nothing to do with the SDK. +- **The deep link phase runs after the call is over**, so a link cannot disturb the in-call checks, + and before the leak checks, so those still judge a finished call. +- **Android runtime permissions are granted twice, and the second time is the one that counts.** + `flutter run`'s install resets them, so a grant from before it silently does nothing and the + camera dialog appears on the first Start New Call — which reads as "Lobby did not open". + `launch.sh` therefore grants again once both apps report a VM service, and warns if a grant did + not take. (The single `retry 1/4` line on Start New Call is unrelated: the first tap lands while + the home screen is still building. `tap_until` exists for exactly that.) +- `ANDROID_SERIAL` is pinned in preflight; otherwise plugging in a phone mid-run makes every `adb` + call ambiguous and screenshots come back 0 bytes. + +## Files + +`/tmp/stream-video-smoke/`: `results.tsv`, `android.log`, `ios.log`, `shots/*.png`. Reset by +`launch.sh`, so re-running it after a failed attempt starts a clean report. +`service-leak.txt` is written only when `no-service-leak` fails. diff --git a/.claude/skills/video-smoke-test/scripts/android_checks.sh b/.claude/skills/video-smoke-test/scripts/android_checks.sh new file mode 100755 index 000000000..c1515cac4 --- /dev/null +++ b/.claude/skills/video-smoke-test/scripts/android_checks.sh @@ -0,0 +1,252 @@ +#!/usr/bin/env bash +# Phase 4 — Android in-call checks. Run with both clients already in the call. +# Every check is a log assertion scoped to lines produced after its own action, +# so nothing can pass on a stale event. No screenshot interpretation required. +# +# Step order is deliberate. Chat runs LAST because it is the only step that +# opens a modal sheet, and a sheet that fails to dismiss would invalidate every +# check after it. in_call_guard runs before each risky step so a dropped call is +# reported once, clearly, instead of cascading. +cd "$(dirname "$0")" && . ./lib.sh +. "$SMOKE_DIR/env.sh" +android_geometry || exit 1 + +echo "== android: in-call checks ==" + +# The reconnect step cuts the emulator's network. If this script dies between +# the cut and the restore — Ctrl-C, a harness timeout, a kill — the emulator is +# left with no connectivity and nothing on screen says why: every later run just +# fails to log in. The trap restores it on ANY exit path, and is a no-op when +# the network was never cut. +NET_CUT=0 +restore_network() { + [ "${NET_CUT:-0}" = 1 ] || return 0 + NET_CUT=0 + info "restoring network" + adb shell svc wifi enable >/dev/null 2>&1 + adb shell svc data enable >/dev/null 2>&1 +} +trap restore_network EXIT +trap 'restore_network; exit 130' INT TERM + +# This script assumes a freshly joined call with the plain call screen showing — +# which android_join.sh guarantees. Do not run it twice against the same call: +# the chat step deliberately leaves the chat sheet open, and a second run's +# early taps would land on that sheet instead of the control bar. +shot "09-android-precheck-state" + +# --- peer visible ---------------------------------------------------------- +assert_two_participants "peer-seen-android" "Android sees the iOS participant" "$ANDROID_LOG" + +# --- microphone input reaching the peer ------------------------------------ +# Runs before the mute checks, while Android is still unmuted. Needs the +# emulator's host-microphone access and audible sound from the host — see +# MIC_HINT in lib.sh and the prerequisites in SKILL.md. +# +# The assertion reads the *iOS* log while *Android* speaks. That is not a +# mistake: the SFU reports a participant's level and dominance to the other +# participants, not back to the speaker, so Android never sees its own. +# +# Do not try to assert audio levels instead. They arrive as high-frequency +# events that update call state without logging, and `audioLevel:` only shows +# up incidentally inside the state dumps that track publish/unpublish print — +# so during a speech window there is usually nothing to match, working +# microphone or not. Dominant speaker is the low-frequency, logged signal. +if in_call_guard "dominant-speaker-a2b"; then + IOS_M=$(logline "$IOS_LOG") + speak_start 30 + assert_mic "dominant-speaker-a2b" "Android's speech makes iOS mark it the dominant speaker" \ + "$IOS_LOG" "\[sfuDominantSpeakerChanged\]" "$IOS_M" 35 + speak_stop + shot "09b-android-speaking" + anr_check "audio-android" +fi + +# --- mute / unmute + speaking while muted ---------------------------------- +M=$(logline "$ANDROID_LOG"); tap $A_MIC $A_BAR_Y +wait_log "$ANDROID_LOG" "sfuTrackUnpublished.*audio" 20 "$M" +assert_log_since "mute-android" "Mute unpublishes the audio track" "$ANDROID_LOG" "sfuTrackUnpublished.*audio" "$M" +sleep 1; shot "10-android-muted" + +# Still muted: the recognizer must be armed, and must fire on speech. Both can +# only be checked here — detection is only active while the user is muted. +if grep -q "^PASS.mute-android" "$RESULTS"; then + M=$(logline "$ANDROID_LOG") + if wait_log "$ANDROID_LOG" "SV:SpeakingWhileMutedRecognition.*\[startRecognition\]" 15 "$M"; then + pass "swm-detection-started-android" "Muting starts speaking-while-muted detection" + else + fail "swm-detection-started-android" "muting did not start speaking-while-muted detection" + fi + + M=$(logline "$ANDROID_LOG") + speak_start 25 + # The recognizer logs its transitions; the app turns them into a snackbar, + # which is not observable from a log. + assert_mic "swm-detect-android" "Speech while muted is detected" "$ANDROID_LOG" \ + "SV:SpeakingWhileMutedRecognition.*isSpeakingWhileMuted: true" "$M" 30 + speak_stop + shot "10b-android-speaking-while-muted" + # Detection succeeded, which means the app has just shown its floating + # "You are muted. Unmute to speak." snackbar — and that sits over the bottom + # control bar, so the very next tap lands on the snackbar instead of the mic + # button. Wait for it to expire, or unmute, camera-on and the filter step all + # fail for a reason that has nothing to do with them. + sleep 6 +else + skip "swm-detection-started-android" "mute did not take effect" + skip "swm-detect-android" "mute did not take effect" +fi + +M=$(logline "$ANDROID_LOG"); tap $A_MIC $A_BAR_Y +wait_log "$ANDROID_LOG" "sfuTrackPublished.*audio" 20 "$M" +assert_log_since "unmute-android" "Unmute republishes the audio track" "$ANDROID_LOG" "sfuTrackPublished.*audio" "$M" +# Unmuting must also tear the detection down, or it keeps listening for the +# rest of the call. +if wait_log "$ANDROID_LOG" "SV:SpeakingWhileMutedRecognition.*\[stopRecognition\]" 15 "$M"; then + pass "swm-detection-stopped-android" "Unmuting stops speaking-while-muted detection" +else + fail "swm-detection-stopped-android" "detection still running after unmute" +fi +anr_check "mute-android" + +# --- camera off / on ------------------------------------------------------- +M=$(logline "$ANDROID_LOG"); tap $A_CAM $A_BAR_Y +wait_log "$ANDROID_LOG" "sfuTrackUnpublished.*video" 20 "$M" +assert_log_since "camera-off-android" "Camera off unpublishes the video track" "$ANDROID_LOG" "sfuTrackUnpublished.*video" "$M" +sleep 1; shot "11-android-camera-off" + +M=$(logline "$ANDROID_LOG"); tap $A_CAM $A_BAR_Y +wait_log "$ANDROID_LOG" "sfuTrackPublished.*video" 25 "$M" +assert_log_since "camera-on-android" "Camera on republishes the video track" "$ANDROID_LOG" "sfuTrackPublished.*video" "$M" +anr_check "camera-android" + +# --- reaction -------------------------------------------------------------- +M=$(logline "$ANDROID_LOG") +tap $A_MORE $A_BAR_Y; sleep 2 +tap $A_REACT_X $A_REACT_Y; sleep 3 +if tail -n "+$M" "$ANDROID_LOG" | grep -qiE "reaction"; then + pass "reaction-android" "Reaction sent and echoed back" +else + fail "reaction-android" "no reaction event in log"; shot "err-reaction" +fi + +# --- video filter ---------------------------------------------------------- +# Selecting a filter does NOT close the options sheet (selecting a reaction +# does). The sheet is therefore still open when this step ends, and the stats +# step below relies on that — do not add a MORE tap between them. +M=$(logline "$ANDROID_LOG") +tap $A_MORE $A_BAR_Y; sleep 2 +tap $A_BLUR_X $A_BLUR_Y; sleep 4 +assert_log_since "filter-android" "Blur filter initialised its frame processor" "$ANDROID_LOG" \ + "registerBlurEffectProcessors|VideoFrameProcessorWithBitmapFilter" "$M" +shot "12-android-filter" + +# --- stats screen ---------------------------------------------------------- +# Sheet is already open from the filter step; tap Stats directly. +tap $A_STATS_X $A_STATS_Y; sleep 4 +shot "13-android-stats" +tap $A_STATS_BACK_X $A_TOP_Y; sleep 3 # top-LEFT back arrow (see lib.sh) +# Honest scope: this proves the stats route opens and closes without hanging or +# dropping the call. The numbers on it are for a human to read in shot 13. +if anr_check "stats-android" && in_call_guard "stats-android"; then + pass "stats-android" "Stats screen opened and closed with the call intact (values in shot 13)" +fi + +# The sheet is still open after returning from stats. Close it so the bottom +# control bar is reachable for the steps below. +tap $A_MORE $A_BAR_Y; sleep 2 +shot "13b-android-sheet-closed" + +# --- picture in picture ---------------------------------------------------- +if in_call_guard "pip-android"; then + adb shell input keyevent KEYCODE_HOME + if wait_pip 15; then + pass "pip-android" "Entered Picture-in-Picture" + else + fail "pip-android" "no PiP transition observed" + fi + shot "15-android-pip" + + # Restore by re-launching the activity. This resumes the existing task; it must + # NOT restart the process (a restart would drop the call), so verify after. + adb shell am start -n "$ANDROID_PKG/io.getstream.video.flutter.dogfooding.MainActivity" >/dev/null 2>&1 + sleep 6 + if in_call_guard "pip-restore-android"; then + pass "pip-restore-android" "Restored from PiP with the call still connected" + fi +fi + +# --- reconnect ------------------------------------------------------------- +if in_call_guard "reconnect-detect-android"; then + M=$(logline "$ANDROID_LOG") + info "cutting network" + # Flag first: if the disable lands and the script dies before the restore, the + # trap must know the network is down. Setting it after would lose that race. + NET_CUT=1 + adb shell svc wifi disable; adb shell svc data disable + if wait_log "$ANDROID_LOG" "scheduling restartICE|\[reconnect\] isConnecting" 45 "$M"; then + pass "reconnect-detect-android" "Network loss detected; ICE restart / reconnect scheduled" + else + fail "reconnect-detect-android" "no reconnect attempt after network loss" + fi + sleep 2; shot "16-android-reconnecting" + + M=$(logline "$ANDROID_LOG") + # Same restore the trap would do; doing it here clears the flag so the trap + # becomes a no-op on normal exit. + restore_network + # Assert RECOVERY, not the strategy. Whether the SDK picks a fast reconnect or + # a full rejoin legitimately varies with how long the outage lasted, so + # requiring "fast" produces flaky failures. The strategy is reported as info. + if wait_log "$ANDROID_LOG" "sfuJoinResponse" 120 "$M"; then + STRAT=$(tail -n "+$M" "$ANDROID_LOG" | grep -oE "fast reconnecting|migrate|rejoin" | head -1) + pass "reconnect-recover-android" "Recovered and rejoined the SFU (${STRAT:-full rejoin})" + else + fail "reconnect-recover-android" "did not rejoin the SFU after network restore" + fi + sleep 3; shot "17-android-reconnected" + anr_check "reconnect-android" +fi + +# --- chat: send from Android and prove iOS receives it --------------------- +# (LAST — opens a modal sheet.) +# +# Delivery is asserted HERE, not in finish.sh, and that ordering is load +# bearing: the sender also gets a message.new event for its own message, so the +# only window in which a message.new on iOS can *only* be the Android message +# is the one before iOS has sent anything of its own (step 6). +if in_call_guard "chat-send-android"; then + CHAT_MSG="smoke_android_$$" + echo "CHAT_MSG_ANDROID=$CHAT_MSG" >>"$SMOKE_DIR/env.sh" + tap $A_CHAT $A_BAR_Y; sleep 3 + tap $A_CHAT_INPUT_X $A_CHAT_INPUT_Y; sleep 3 # composer, keyboard closed + M=$(logline "$ANDROID_LOG"); IOS_M=$(logline "$IOS_LOG") + if android_chat_send "$CHAT_MSG" "$M"; then + pass "chat-send-android" "Android chat message accepted by the API" + else + fail "chat-send-android" "Android chat message was never sent (send tap did not land)" + fi + shot "14-android-chat" + anr_check "chat-android" + # iOS side of the same message. Only meaningful if the send happened. + if grep -q "^PASS.chat-send-android" "$RESULTS"; then + assert_chat_delivered "chat-a2b" "Android -> iOS chat message delivered" "$IOS_LOG" "$IOS_M" 25 + else + skip "chat-a2b" "no Android message was sent, delivery cannot be judged" + fi + # Mark the Android log AFTER its own message.new echo has landed, so the + # iOS -> Android check in finish.sh cannot be satisfied by Android's own echo. + wait_log "$ANDROID_LOG" "Event received: message\.new" 15 "$M" || true + echo "ANDROID_CHAT_MARK=$(logline "$ANDROID_LOG")" >>"$SMOKE_DIR/env.sh" + + # The keyboard is still open after sending, and it covers the top bar that + # finish.sh needs for END CALL. Exactly ONE back press closes the keyboard and + # leaves the sheet open with the top bar visible. + # + # Do NOT add a second back press: it would close the sheet, and a third would + # leave the call. One is sufficient and safe. + adb shell input keyevent KEYCODE_BACK; sleep 2 + shot "14b-android-chat-after-back" +fi + +echo "OK android checks — now run the iOS check sequence from SKILL.md step 6" diff --git a/.claude/skills/video-smoke-test/scripts/android_join.sh b/.claude/skills/video-smoke-test/scripts/android_join.sh new file mode 100755 index 000000000..4d9adbb68 --- /dev/null +++ b/.claude/skills/video-smoke-test/scripts/android_join.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# Phase 2 — Android: log in, create a call, print the call ID. +# Starts from a wiped app so the login screen is guaranteed to be on screen. +cd "$(dirname "$0")" && . ./lib.sh +. "$SMOKE_DIR/env.sh" # pins ANDROID_SERIAL +android_geometry || exit 1 + +echo "== android: login + create call ==" +# The username MUST contain an underscore or digit. A plain dictionary-ish word +# gets silently rewritten by the emulator IME's autocorrect ("smokea" -> "smokes") +# and the run then asserts against the wrong user id. +USER_A=${1:-smoke_a1} + +tap $A_USER_X $A_USER_Y; sleep 2 +adb shell input text "$USER_A"; sleep 1 +# The soft keyboard covers the sign-up button, so dismiss it before tapping. +adb shell input keyevent KEYCODE_BACK; sleep 2 + +# Coordinator WS health-check traffic only starts once the user is connected. +if tap_until $A_SIGNUP_X $A_SIGNUP_Y "$ANDROID_LOG" "handleHealthCheckEvent|connectUser" 3 25; then + pass "login-android" "Android logged in as $USER_A" +else + fail "login-android" "Android login did not complete"; shot "err-login"; exit 1 +fi +anr_check "login-android" + +# tap_until, not a fixed sleep: the home screen takes a variable moment to build +# after login, and a tap that lands early hits the login screen's Google button. +if tap_until $A_START_NEW_X $A_START_NEW_Y "$ANDROID_LOG" "getOrCreateCall" 4 15; then + pass "lobby-android" "Lobby screen reached" +else + fail "lobby-android" "Lobby did not open"; shot "err-lobby"; exit 1 +fi +sleep 3 +shot "02-android-lobby" + +if tap_until $A_TESTCALL_X $A_TESTCALL_Y "$ANDROID_LOG" "sfuJoinResponse" 3 30; then + pass "join-android" "Android joined the call" +else + fail "join-android" "Android did not join"; shot "err-join"; exit 1 +fi + +CALL_ID=$(grep -oE "callCid: default:[a-z0-9]+" "$ANDROID_LOG" | tail -1 | sed 's/.*://') +if [ -z "$CALL_ID" ]; then fail "callid" "could not extract call id"; exit 1; fi +# Replace any CALL_ID from a previous run rather than appending a second one. +# +# The mv is conditional on purpose. The redirection creates env.tmp whether or +# not grep succeeds, so an unconditional mv turns any grep failure into a +# truncated env.sh — which silently drops ANDROID_SERIAL, and every later phase +# then runs bare `adb` and dies the moment a second device is attached. Keeping +# the old file on failure costs a duplicate CALL_ID line at worst, and the last +# one wins when it is sourced. +if grep -v '^CALL_ID=' "$SMOKE_DIR/env.sh" >"$SMOKE_DIR/env.tmp" 2>/dev/null \ + && [ -s "$SMOKE_DIR/env.tmp" ]; then + mv "$SMOKE_DIR/env.tmp" "$SMOKE_DIR/env.sh" +else + rm -f "$SMOKE_DIR/env.tmp" +fi +echo "CALL_ID=$CALL_ID" >>"$SMOKE_DIR/env.sh" + +sleep 3 +shot "03-android-in-call" +assert_log "publish-audio-android" "Android published its audio track" "$ANDROID_LOG" "\[publishAudioTrack\]" +assert_log "publish-video-android" "Android published its video track" "$ANDROID_LOG" "sfuTrackPublished.*video" + +echo +echo "CALL_ID=$CALL_ID" +echo "OK android joined — now run the iOS join sequence from SKILL.md step 4" diff --git a/.claude/skills/video-smoke-test/scripts/finish.sh b/.claude/skills/video-smoke-test/scripts/finish.sh new file mode 100755 index 000000000..ab8d0c4f9 --- /dev/null +++ b/.claude/skills/video-smoke-test/scripts/finish.sh @@ -0,0 +1,162 @@ +#!/usr/bin/env bash +# Phase 6 — cross-platform assertions, leave, teardown, summary. +# Run after the agent has finished the iOS check sequence and tapped iOS end-call. +cd "$(dirname "$0")" && . ./lib.sh +. "$SMOKE_DIR/env.sh" +android_geometry || exit 1 + +echo "== cross-platform ==" + +# --- iOS side, asserted from its flutter run log --------------------------- +assert_log "join-ios" "iOS joined the same call" "$IOS_LOG" "sfuJoinResponse" +assert_log "publish-audio-ios" "iOS published its audio track" "$IOS_LOG" "\[publishAudioTrack\]" +assert_two_participants "peer-seen-ios" "iOS sees the Android participant" "$IOS_LOG" +# The branch-specific path: iOS must mute at the audio-device-module level and +# keep capturing, not stop the track. Without this, speaking-while-muted +# detection cannot fire on iOS/macOS. +assert_log "adm-mute-ios" "iOS mute goes through the Apple ADM path" "$IOS_LOG" "setAppleAdmMicrophoneMuted\] muted: true" +assert_log "adm-unmute-ios" "iOS unmute releases the ADM mute" "$IOS_LOG" "setAppleAdmMicrophoneMuted\] muted: false" + +# --- chat: iOS send, and delivery back to Android ------------------------- +# The Android -> iOS direction is asserted in android_checks.sh, in the only +# window where a message.new on iOS can only be the Android message. +# +# The iOS send is matched case-insensitively: the iOS keyboard autocapitalises +# the first character of whatever the agent types (Smoke_ios_1). +# Match the outgoing request body, not just the text anywhere in the log, so +# the check cannot pass on a message that only reached the composer. +assert_log "chat-send-ios" "iOS chat message accepted by the API" "$IOS_LOG" \ + "(Message\(.*, [Ss]moke_ios_1, regular|\"text\":\"[Ss]moke_ios_1\")" +# Receipt on Android is the WS event, not the text — a receiving client never +# logs the body of a message it receives. ANDROID_CHAT_MARK was recorded after +# Android's own message.new echo, so a match here is the iOS message. +if grep -q "^PASS.chat-send-ios" "$RESULTS" 2>/dev/null; then + assert_chat_delivered "chat-b2a" "iOS -> Android chat message delivered" \ + "$ANDROID_LOG" "${ANDROID_CHAT_MARK:-1}" 25 +else + skip "chat-b2a" "no iOS message was sent, delivery cannot be judged" +fi + +# --- leave + teardown ------------------------------------------------------ +echo "== teardown ==" +# android_checks.sh ends with the chat sheet open, stacked on top of the +# options sheet. Both are modal: a tap outside a sheet is eaten by its scrim +# and only dismisses that sheet, so a single END CALL tap never reaches the +# button — it just closes the chat sheet and the call keeps running (which is +# what made leave/service/notification all fail together). tap_until re-taps +# until the log confirms the leave: the first taps peel off the sheets, the +# last one lands on END CALL. Do not replace this with a bare tap. +if tap_until $A_END_X $A_TOP_Y "$ANDROID_LOG" "\[leave\] reason|\[clear\] src: disconnect" 4 12; then + pass "leave-android" "Android left the call cleanly" +else + fail "leave-android" "Android leave not observed" + shot "err-leave-android" +fi +sleep 4 +# A feedback modal appears after leaving; dismiss it so the app is left usable. +tap $A_FEEDBACK_X $A_FEEDBACK_Y; sleep 2 + +# --- deep link join -------------------------------------------------------- +# Runs after the call is over so a link cannot disturb the in-call checks, and +# before the leak checks so those still assert on a finished call. +# +# The link is delivered straight to the activity: a debug build is not a +# verified App Links handler, so a browsable VIEW intent opens Chrome instead +# (see `deeplink` in lib.sh). +echo "== deep link ==" +M=$(logline "$ANDROID_LOG") +deeplink "https://pronto.getstream.io/join/$CALL_ID" +if ! wait_log "$ANDROID_LOG" "callCid: default:$CALL_ID" 30 "$M"; then + fail "deeplink-route-android" "call link did not open the lobby for $CALL_ID" + shot "err-deeplink" +elif ! router_ok "$M"; then + fail "deeplink-route-android" "the link resolved the call but the router showed an error" + shot "err-deeplink-router" +else + pass "deeplink-route-android" "Call link opens the lobby for the linked call" +fi +sleep 2; shot "21-android-deeplink-lobby" + +M=$(logline "$ANDROID_LOG") +if tap_until $A_TESTCALL_X $A_TESTCALL_Y "$ANDROID_LOG" "sfuJoinResponse" 3 30; then + pass "deeplink-join-android" "Joined the call reached through the link" +else + fail "deeplink-join-android" "could not join from the link's lobby" +fi + +# Leave again, so the leak checks below judge a torn-down call. This also +# exercises the pop path that a link-built navigation stack produces. +M=$(logline "$ANDROID_LOG") +if ! tap_until $A_END_X $A_TOP_Y "$ANDROID_LOG" "\[leave\] reason|\[clear\] src: disconnect" 4 12; then + fail "deeplink-leave-android" "leaving the linked call failed" +elif ! router_ok "$M"; then + fail "deeplink-leave-android" "leaving the linked call emptied the navigation stack" +else + pass "deeplink-leave-android" "Left the linked call without emptying the navigation stack" +fi +sleep 3 +tap $A_FEEDBACK_X $A_FEEDBACK_Y; sleep 2 +echo "== no-leak checks ==" + +# Only the Stream CALL foreground service counts, and it gets a settle window: +# teardown is asynchronous, and the app also runs unrelated short-lived +# services, so "any ServiceRecord, one second after leaving" is a race that +# reports normal teardown as a leak. On a real leak the dump is saved for a +# human — do not diagnose it from this script. +if wait_call_service_gone 25; then + pass "no-service-leak" "No call foreground service left running after leaving" +else + call_service_dump >"$SMOKE_DIR/service-leak.txt" + adb shell dumpsys activity services "$ANDROID_PKG" 2>/dev/null >>"$SMOKE_DIR/service-leak.txt" + fail "no-service-leak" "call foreground service still alive 25s after leaving (dump: $SMOKE_DIR/service-leak.txt)" +fi + +if [ "$(adb shell dumpsys notification 2>/dev/null | grep -c "pkg=$ANDROID_PKG")" = "0" ]; then + pass "no-notif-leak" "No ongoing-call notification left behind" +else + fail "no-notif-leak" "ongoing-call notification leaked after leaving" +fi + +assert_log "leave-ios" "iOS left the call cleanly" "$IOS_LOG" "\[leave\] reason|\[clear\] src: disconnect" +shot "20-android-after-leave" + +# --- stop background processes -------------------------------------------- +# stop_all_bg, not `kill $(cat *.pid)`: the pid files outlive the run, so after +# a reboot or a long gap those numbers belong to whatever the OS handed them to +# next. Each pid is re-checked against the command it was started with, and the +# flutter run children (gradle, xcodebuild, dart) are stopped too instead of +# being orphaned. +stop_all_bg + +# --- summary -------------------------------------------------------------- +echo +echo "=============== SMOKE TEST SUMMARY ===============" +# `grep -c` prints 0 AND exits 1 when there is no match, so a `|| echo 0` +# fallback here printed the count twice. It already prints a number. +count() { local n; n=$(grep -c "^$1" "$RESULTS" 2>/dev/null); printf '%s' "${n:-0}"; } +printf 'passed: %s\n' "$(count PASS)" +printf 'failed: %s\n' "$(count FAIL)" +printf 'known-fail: %s\n' "$(count KNOWN-FAIL)" +printf 'skipped: %s\n' "$(count SKIP)" +echo + +# The report table. Generated here rather than assembled by hand afterwards, so +# what the user reads is exactly what the checks recorded, in the order they +# ran. Relay it verbatim; see the Reporting section in SKILL.md. +echo "| check | what it verifies | result |" +echo "|---|---|---|" +while IFS=$'\t' read -r res id desc; do + case "$res" in + PASS) mark="works" ;; + FAIL) mark="**FAILED**" ;; + KNOWN-FAIL) mark="not verified" ;; + SKIP) mark="not run" ;; + *) mark="$res" ;; + esac + printf '| `%s` | %s | %s |\n' "$id" "$desc" "$mark" +done <"$RESULTS" +echo +echo "screenshots: $SHOTS" +echo "logs: $ANDROID_LOG $IOS_LOG" +echo "==================================================" +grep -q '^FAIL' "$RESULTS" 2>/dev/null && exit 1 || exit 0 diff --git a/.claude/skills/video-smoke-test/scripts/launch.sh b/.claude/skills/video-smoke-test/scripts/launch.sh new file mode 100755 index 000000000..c0f9afc89 --- /dev/null +++ b/.claude/skills/video-smoke-test/scripts/launch.sh @@ -0,0 +1,70 @@ +#!/usr/bin/env bash +# Phase 1 — build and launch both apps, then block until both are actually up. +# First iOS run pays a CocoaPods + full compile cost (~10 min); later runs are fast. +cd "$(dirname "$0")" && . ./lib.sh +. "$SMOKE_DIR/env.sh" +cd "$(git rev-parse --show-toplevel)/dogfooding" || exit 1 + +echo "== launch ==" + +# Reap the previous run's background processes before starting new ones. An +# earlier run that was interrupted mid-phase never reached finish.sh, so its +# logcat and both flutter run supervisors are still alive — and the old logcat +# is still appending to the very file this run is about to truncate, which +# interleaves two runs' output in one log and makes every assertion suspect. +stop_all_bg + +# Fresh logs so no assertion can match a previous run's output. +adb logcat -c >/dev/null 2>&1 +: >"$ANDROID_LOG"; : >"$IOS_LOG" +# And fresh results. preflight truncates these too, but launch is the real +# start-of-run boundary: it wipes app state, so a relaunch after a failed +# attempt is a new run. Without this the summary table shows the abandoned +# attempt's rows — including its failures — alongside the real ones. +: >"$RESULTS" +nohup adb logcat >"$ANDROID_LOG" 2>&1 & +record_bg logcat $! + +# Wipe Android app state so the login screen is guaranteed to be the first +# screen. Without this, a persisted session lands on Home instead and every +# subsequent tap coordinate is wrong. +adb shell pm clear "$ANDROID_PKG" >/dev/null 2>&1 +# pm clear also wipes granted permissions. Granting here is not enough on its +# own — flutter's install resets them again — so this runs a second time below, +# once both apps are up. +grant_android_permissions + +# Android: install+run via flutter so hot reload and Dart logs are available. +nohup flutter run --flavor dev -d "$ANDROID_SERIAL" \ + >"$SMOKE_DIR/android_run.log" 2>&1 & +record_bg android_run $! + +nohup flutter run --flavor dev -d "$IOS_UDID" >"$IOS_LOG" 2>&1 & +record_bg ios_run $! + +echo " waiting for both apps (up to 15 min; iOS first build is the slow one)" + +if wait_log "$SMOKE_DIR/android_run.log" "Dart VM Service on" 900; then + pass "boot-android" "Android app launched" +else + fail "boot-android" "Android app did not launch — see $SMOKE_DIR/android_run.log" +fi + +if wait_log "$IOS_LOG" "Dart VM Service on" 900; then + pass "boot-ios" "iOS app launched" +else + if grep -qE "requires minimum platform version" "$IOS_LOG"; then + fail "boot-ios" "iOS build failed on SPM platform floor — run: flutter config --no-enable-swift-package-manager" + else + fail "boot-ios" "iOS app did not launch — see $IOS_LOG" + fi +fi + +# The decisive grant: flutter's install has just reset runtime permissions, so +# this is the one that actually holds for the run. Without it the camera dialog +# appears on the first Start New Call and swallows the tap. +grant_android_permissions + +sleep 4 +shot "01-android-launched" +echo "OK launch" diff --git a/.claude/skills/video-smoke-test/scripts/lib.sh b/.claude/skills/video-smoke-test/scripts/lib.sh new file mode 100755 index 000000000..cd238d183 --- /dev/null +++ b/.claude/skills/video-smoke-test/scripts/lib.sh @@ -0,0 +1,493 @@ +#!/usr/bin/env bash +# Shared helpers for the video SDK smoke test. +# Sourced by every phase script. Never run directly. + +set -u + +# ---------------------------------------------------------------- paths +SMOKE_DIR="${SMOKE_DIR:-/tmp/stream-video-smoke}" +mkdir -p "$SMOKE_DIR" + +ANDROID_LOG="$SMOKE_DIR/android.log" # adb logcat capture +IOS_LOG="$SMOKE_DIR/ios.log" # flutter run stdout (iOS) +RESULTS="$SMOKE_DIR/results.tsv" +SHOTS="$SMOKE_DIR/shots" +mkdir -p "$SHOTS" + +ANDROID_PKG="io.getstream.video.flutter.dogfooding.debug" + +# ---------------------------------------------------------------- results +# Every check emits exactly one line. Nothing else writes to stdout with +# these prefixes, so the agent can read results without interpreting prose. +pass() { printf 'PASS\t%s\t%s\n' "$1" "$2"; printf 'PASS\t%s\t%s\n' "$1" "$2" >>"$RESULTS"; } +fail() { printf 'FAIL\t%s\t%s\n' "$1" "$2"; printf 'FAIL\t%s\t%s\n' "$1" "$2" >>"$RESULTS"; } +skip() { printf 'SKIP\t%s\t%s\n' "$1" "$2"; printf 'SKIP\t%s\t%s\n' "$1" "$2" >>"$RESULTS"; } +known() { printf 'KNOWN-FAIL\t%s\t%s\n' "$1" "$2"; printf 'KNOWN-FAIL\t%s\t%s\n' "$1" "$2" >>"$RESULTS"; } +info() { printf '..\t%s\n' "$1"; } + +# assert_log +# Passes if the pattern appears anywhere in the log file. +assert_log() { + local id="$1" desc="$2" logf="$3" pat="$4" + if grep -qE "$pat" "$logf" 2>/dev/null; then pass "$id" "$desc"; else fail "$id" "$desc"; fi +} + +# assert_log_since +# Only looks at lines added after . Use this so an assertion +# cannot be satisfied by an identical event from an earlier step. +assert_log_since() { + local id="$1" desc="$2" logf="$3" pat="$4" from="$5" + if tail -n "+$from" "$logf" 2>/dev/null | grep -qE "$pat"; then pass "$id" "$desc"; else fail "$id" "$desc"; fi +} + +# wait_log [start-line] +# Returns 0 as soon as the pattern appears. Polls at 0.5s. +# Always prefer this over a fixed sleep. +wait_log() { + local logf="$1" pat="$2" timeout="$3" from="${4:-1}" + local deadline=$(( $(date +%s) + timeout )) + while [ "$(date +%s)" -lt "$deadline" ]; do + if tail -n "+$from" "$logf" 2>/dev/null | grep -qE "$pat"; then return 0; fi + sleep 0.5 + done + return 1 +} + +logline() { wc -l <"$1" 2>/dev/null | tr -d ' ' || echo 1; } + +# in_call_guard +# Fails fast if the app has dropped out of the call since it joined. Without +# this, one bad tap early on turns into a pile of confusing downstream failures +# instead of one clear message. +in_call_guard() { + local id="$1" lastjoin lastleave + lastjoin=$(grep -n "sfuJoinResponse" "$ANDROID_LOG" 2>/dev/null | tail -1 | cut -d: -f1) + lastleave=$(grep -nE "\[leave\] reason|\[clear\] src: disconnect" "$ANDROID_LOG" 2>/dev/null | tail -1 | cut -d: -f1) + if [ -n "$lastleave" ] && [ "${lastleave:-0}" -gt "${lastjoin:-0}" ]; then + fail "$id" "app is no longer in the call (an earlier step dropped it) — remaining checks are invalid" + shot "err-$id-not-in-call" + return 1 + fi + return 0 +} + +# assert_two_participants +# Name-independent peer check: at least two distinct user ids appeared in SFU +# participant state. Do not assert on SfuParticipantCount — the join response +# carries the count at join time, so the first client to join always sees 1. +assert_two_participants() { + local id="$1" desc="$2" logf="$3" n + # Exclude the placeholder ids the SDK logs ("0", "null") — counting them would + # let a solo call pass this check. + n=$(grep -oE "userId: [a-zA-Z0-9_]+" "$logf" 2>/dev/null | sort -u \ + | grep -cvE "userId: (0|null)$") + if [ "${n:-0}" -ge 2 ]; then pass "$id" "$desc ($n participants seen)" + else fail "$id" "$desc (only ${n:-0} participant seen)"; fi +} + +# ---------------------------------------------------------------- android input +# Reference geometry: Pixel_10 AVD, 1080x2424. Coordinates below are the +# verified values for that device; SX/SY rescale them if the emulator differs +# in size but not in aspect ratio. +android_geometry() { + local size w h + size=$(adb shell wm size 2>/dev/null | tr -d '\r' | awk -F': ' '{print $2}') + w=${size%x*}; h=${size#*x} + if [ -z "$w" ] || [ -z "$h" ]; then echo "FATAL: cannot read screen size" >&2; return 1; fi + SX=$(awk -v a="$w" 'BEGIN{printf "%.5f", a/1080}') + SY=$(awk -v a="$h" 'BEGIN{printf "%.5f", a/2424}') + local skew + skew=$(awk -v x="$SX" -v y="$SY" 'BEGIN{d=x/y; if(d<1)d=1/d; printf "%.4f", d}') + if awk -v s="$skew" 'BEGIN{exit !(s>1.02)}'; then + echo "FATAL: emulator aspect ratio differs from the 1080x2424 reference (${w}x${h})." >&2 + echo " Taps would land on the wrong controls. Use a Pixel_10-class AVD." >&2 + return 1 + fi + info "android geometry ${w}x${h} (scale $SX/$SY)" +} + +# tap — coordinates in the 1080x2424 reference space +tap() { + local x y + x=$(awk -v a="$1" -v s="$SX" 'BEGIN{printf "%d", a*s}') + y=$(awk -v a="$2" -v s="$SY" 'BEGIN{printf "%d", a*s}') + adb shell input tap "$x" "$y" +} + +shot() { adb exec-out screencap -p >"$SHOTS/$1.png" 2>/dev/null; } + +# tap_until [attempts] [per-attempt-timeout] +# Taps, then waits for the log to confirm the tap had its effect; retries if not. +# +# Use this for every navigation tap. A fixed `sleep` before a tap is a guess +# about how long a screen takes to build, and when the guess is short the tap +# lands on the previous screen — which is how a mistap on the login screen +# turned into "lobby did not open" three steps later. +tap_until() { + local x="$1" y="$2" logf="$3" pat="$4" tries="${5:-3}" t="${6:-12}" i M + for i in $(seq 1 "$tries"); do + M=$(logline "$logf") + tap "$x" "$y" + if wait_log "$logf" "$pat" "$t" "$M"; then return 0; fi + info "retry $i/$tries: tap ($x,$y) produced no '$pat'" + done + return 1 +} + +# wait_pip +# Returns 0 once the app's task is actually in PiP. Polls, so it cannot lose a +# race against the PiP transition. +# +# The probe reads CURRENT state — the task's `mode=pinned` — and is verified to +# go true in PiP and false again after restore. +# +# Two traps to avoid here: +# * Do NOT grep `windowingMode=pinned` out of `dumpsys activity activities`: it +# lives on config lines long enough that the match fails in practice even +# while the app is demonstrably in PiP. +# * Do NOT accept a `PICTURE_IN_PICTURE_ENTER` line from logcat: the buffer +# keeps it forever, so the check passes on a stale event from an earlier run +# even when PiP never happened. +wait_pip() { + local timeout="${1:-12}" deadline + deadline=$(( $(date +%s) + timeout )) + while [ "$(date +%s)" -lt "$deadline" ]; do + if adb shell dumpsys activity activities 2>/dev/null \ + | grep -qE "Task\{.*$ANDROID_PKG.*mode=pinned"; then return 0; fi + sleep 1 + done + return 1 +} + +# Fails the run loudly if Android threw up an ANR dialog. Called between steps +# so a hang is reported as a finding instead of silently eating later taps. +anr_check() { + local id="$1" + if adb shell dumpsys window 2>/dev/null | grep -q "Application Not Responding"; then + fail "$id" "ANR dialog appeared (app main thread blocked)" + shot "anr-$id" + adb shell input keyevent KEYCODE_DPAD_DOWN >/dev/null 2>&1 + return 1 + fi + return 0 +} + +# ---------------------------------------------------------------- audio +# Every microphone-dependent check needs two things that live outside this +# repo: the emulator must be allowed to use the host microphone, and the host +# must actually be making a noise. Neither can be probed — the emulator console +# exposes no microphone command — so the checks play a sound, assert, and name +# the causes when they fail. +MIC_HINT="check, in this order: the emulator's Extended controls > Microphone > 'Enable Host Microphone Access' — restart the emulator after toggling it, the setting only takes effect on a fresh boot; macOS System Settings > Privacy & Security > Microphone, where the emulator must be allowed (macOS hands an unpermitted process silence, not an error); and the Mac's output volume plus an unmuted input" + +# speak_start +# Talks through the host's speakers so the emulator's host-microphone +# passthrough has something to hear. The detector on the Android side is an RMS +# threshold (-45 dBFS over 600ms), so any sustained sound works; speech is used +# because it is what the feature is for. +# +# Sets SPEAK_PID. Always pair with speak_stop, including on the failure path, +# or the run keeps talking through later checks. +speak_start() { + local secs="${1:-10}" deadline + SPEAK_PID="" + if ! command -v say >/dev/null 2>&1; then + info "no 'say' on this host — microphone checks need audible sound from the host" + return 1 + fi + deadline=$(( $(date +%s) + secs )) + ( + while [ "$(date +%s)" -lt "$deadline" ]; do + say -r 170 "one two three four five six seven eight nine ten" >/dev/null 2>&1 + done + ) & + SPEAK_PID=$! + return 0 +} + +speak_stop() { + if [ -n "${SPEAK_PID:-}" ]; then + kill "$SPEAK_PID" 2>/dev/null + # Reap it, or bash prints its own "Terminated: 15" job notice into the + # middle of the check output. + wait "$SPEAK_PID" 2>/dev/null + fi + # `say` is a child of the subshell, so it survives the kill above. + pkill -x say 2>/dev/null + SPEAK_PID="" + return 0 +} + +# assert_mic [timeout] +# A microphone-dependent assertion. A PASS is real news: it means sound from the +# host reached the emulator and travelled the whole path being checked. +# +# Absence is reported as KNOWN-FAIL, not FAIL, because it is almost always the +# environment. Microphone input into the emulator is intermittent: the same AVD +# instance delivered audio for two runs and then went silent for a third with no +# configuration change, and recovering it needs an emulator restart. A hard +# failure here would cry wolf often enough that people stop reading failures, +# which costs more than this check gains. Read a KNOWN-FAIL as "not verified +# this run" and check the prerequisites in SKILL.md. +assert_mic() { + local id="$1" desc="$2" logf="$3" pat="$4" from="$5" t="${6:-25}" + if wait_log "$logf" "$pat" "$t" "$from"; then + pass "$id" "$desc" + else + known "$id" "$desc — not verified: no microphone input observed ($MIC_HINT)" + fi +} + +# ---------------------------------------------------------------- permissions +# grant_android_permissions +# Pre-grants every runtime permission the app asks for, so no system dialog can +# swallow a tap mid-run. +# +# Call this AFTER `flutter run` has installed the app. The install resets +# runtime permissions, so granting only before it silently does nothing: the +# camera dialog then appears on the first Start New Call, eats the tap, and the +# run dies at "Lobby did not open" with a dialog on screen. +grant_android_permissions() { + local p missing="" + for p in RECORD_AUDIO CAMERA POST_NOTIFICATIONS READ_PHONE_STATE BLUETOOTH_CONNECT; do + adb shell pm grant "$ANDROID_PKG" "android.permission.$p" >/dev/null 2>&1 + done + # Verify the two that gate the call. Every later coordinate depends on no + # dialog being on screen, so a silent failure here must not stay silent. + for p in RECORD_AUDIO CAMERA; do + adb shell dumpsys package "$ANDROID_PKG" 2>/dev/null \ + | grep -q "android.permission.$p: granted=true" || missing="$missing $p" + done + if [ -n "$missing" ]; then + info "WARN: permissions not granted:$missing — a system dialog may eat taps" + fi + return 0 +} + +# ---------------------------------------------------------------- deep links +# deeplink +# Delivers a call link to the app. The component is named explicitly because a +# debug build is not a verified App Links handler (its signing fingerprint is +# not in the site's assetlinks.json), and a plain browsable VIEW intent then +# opens Chrome instead of the app — which looks exactly like a broken deep +# link. `adb shell pm get-app-links ` shows the verification state. +deeplink() { + adb shell am start -a android.intent.action.VIEW -d "$1" \ + -n "$ANDROID_PKG/io.getstream.video.flutter.dogfooding.MainActivity" \ + >/dev/null 2>&1 +} + +# router_ok +# False if the router rejected the location it was handed, or if a pop emptied +# the navigation stack. A link that resolves the call but leaves the user on an +# error screen must not pass, and popping the last page is what a link-built +# stack does when nothing was routed beneath it. +router_ok() { + local from="$1" + ! tail -n "+$from" "$ANDROID_LOG" 2>/dev/null \ + | grep -qE "GoException|no routes for location|popped the last page" +} + +# ---------------------------------------------------------------- chat +# android_chat_send +# Types into the already-open Android chat composer and taps SEND +# until the outgoing message request is in the log. Returns 0 only when the +# send is confirmed, so a missed tap is a failure here instead of a mystery +# three checks later. +# +# Two traps this encodes: +# * Whether the soft keyboard opens after `input text` is not under our +# control (Gboard on this AVD sometimes shows only its floating toolbar), +# and that moves the composer between y=2293 (keyboard closed) and y=1434 +# (keyboard open). One fixed tap missed the button, the text sat in the +# composer as a draft, and a stray tap in finish.sh sent it a minute later. +# * Never assert the send by grepping logcat for the bare message text: +# logcat echoes the `input text ` command itself, so that assertion +# passes even when nothing was ever sent. Match the outgoing request body. +# +# Position order is deliberate: keyboard-open first. With the keyboard closed +# that spot is empty message list, so a miss is harmless; the keyboard-closed +# spot lands on a key when the keyboard IS open, which would corrupt the text. +android_chat_send() { + local msg="$1" from="$2" i pat + pat="I flutter.*(\"text\":\"$msg\"|, $msg, regular)" + adb shell input text "$msg"; sleep 2 + for i in 1 2 3 4; do + if [ $((i % 2)) = 1 ]; then tap $A_CHAT_SEND_X $A_CHAT_INPUT2_Y + else tap $A_CHAT_SEND_X $A_CHAT_INPUT_Y; fi + if wait_log "$ANDROID_LOG" "$pat" 8 "$from"; then return 0; fi + info "retry $i/4: chat send tap produced no outgoing message request" + done + return 1 +} + +# assert_chat_delivered [timeout] +# Chat receipt is asserted from the receiver's WS event, NOT from the message +# text. No client logs the body of a message it RECEIVES, so grepping the peer +# log for the text can never pass — that is what made chat look like a +# "send-only" defect for a while: the assertion was wrong, not the SDK. The +# receiving client logs `Event received: message.new` and then POSTs +# channel/read; the event is the portable signal on both platforms. +# +# MUST be taken before the sender sends, and no other message may +# be in flight in that window: the sender also gets a message.new for its own +# message, so an unmarked window cannot tell the two apart. +assert_chat_delivered() { + local id="$1" desc="$2" logf="$3" from="$4" t="${5:-25}" + if wait_log "$logf" "Event received: message\.new" "$t" "$from"; then + pass "$id" "$desc" + else + fail "$id" "$desc" + fi +} + +# ---------------------------------------------------------------- services +# call_service_dump — prints a line per Stream CALL foreground service that is +# alive right now (empty output means none). +# +# Deliberately specific. A bare `dumpsys activity services | grep +# ServiceRecord` reports the app's unrelated short-lived services (push, +# WorkManager) and a call service that is merely still unwinding, so it cannot +# tell a real leak from normal teardown. +call_service_dump() { + adb shell dumpsys activity services "$ANDROID_PKG" 2>/dev/null | tr -d '\r' | awk ' + /ServiceRecord\{/ { rec=$0; keep=(rec ~ /StreamCallService|StreamScreenShareService/); next } + keep && /isForeground=true/ { print rec; keep=0 } + ' +} + +# wait_call_service_gone +# Returns 0 once no Stream call foreground service is left. Polls, because +# teardown is asynchronous — a single-shot check seconds after leaving is a +# race, not a finding. +wait_call_service_gone() { + local timeout="${1:-25}" deadline + deadline=$(( $(date +%s) + timeout )) + while [ "$(date +%s)" -lt "$deadline" ]; do + [ -z "$(call_service_dump)" ] && return 0 + sleep 2 + done + return 1 +} + +# ---------------------------------------------------------------- background procs +# The run leaves three long-lived processes behind it: `adb logcat` and one +# `flutter run` per platform. They are recorded here as "\t" and +# never killed unless the pid is STILL that command. +# +# A bare pid is not safe to kill later. The pid files live in $SMOKE_DIR, which +# survives across runs and reboots, so a stale file plus pid reuse means +# `kill $(cat ...)` can hit an unrelated process of the user's. Recording the +# command line makes the check possible; every kill path below goes through it. + +# record_bg +record_bg() { + local name="$1" pid="$2" cmd + cmd=$(ps -o command= -p "$pid" 2>/dev/null | head -1) + printf '%s\t%s\n' "$pid" "$cmd" >"$SMOKE_DIR/$name.pid" +} + +# proc_is — true only if that pid is alive AND its +# command line still contains what we recorded. An expected string with glob +# characters simply fails to match, which fails in the safe direction: we skip +# the kill rather than kill something unknown. +proc_is() { + local pid="$1" want="$2" cmd + [ -n "$pid" ] && [ -n "$want" ] || return 1 + case "$pid" in *[!0-9]*|'') return 1;; esac + cmd=$(ps -o command= -p "$pid" 2>/dev/null | head -1) + [ -n "$cmd" ] || return 1 + case "$cmd" in *"$want"*) return 0;; *) return 1;; esac +} + +# proc_tree — "\t" for every descendant, deepest last. +# Captured BEFORE the parent is signalled: once it dies its children are +# reparented to launchd and the pgrep -P link is gone. +proc_tree() { + local pid="$1" child cmd + for child in $(pgrep -P "$pid" 2>/dev/null); do + cmd=$(ps -o command= -p "$child" 2>/dev/null | head -1) + [ -n "$cmd" ] && printf '%s\t%s\n' "$child" "$cmd" + proc_tree "$child" + done +} + +# kill_tree — stop a process and anything it spawned. +# `flutter run` is a supervisor: killing only the recorded pid leaves the gradle +# daemon, xcodebuild and the dart VM running. It gets a TERM and five seconds to +# reap its own children first; whatever is still standing after that is killed +# individually, each re-verified against the command captured up front. +kill_tree() { + local pid="$1" kids line cpid ccmd i + kids=$(proc_tree "$pid") + kill "$pid" 2>/dev/null + for i in 1 2 3 4 5 6 7 8 9 10; do + kill -0 "$pid" 2>/dev/null || break + sleep 0.5 + done + kill -0 "$pid" 2>/dev/null && kill -9 "$pid" 2>/dev/null + printf '%s\n' "$kids" | while IFS=$'\t' read -r cpid ccmd; do + [ -n "$cpid" ] || continue + proc_is "$cpid" "$ccmd" && kill "$cpid" 2>/dev/null + done + return 0 +} + +# stop_bg ... — stop recorded background processes and drop their pid +# files. Safe to call when nothing is running, and safe to call twice. +stop_bg() { + local name pid cmd + for name in "$@"; do + [ -f "$SMOKE_DIR/$name.pid" ] || continue + IFS=$'\t' read -r pid cmd <"$SMOKE_DIR/$name.pid" + if [ -z "${cmd:-}" ]; then + # Pre-verification pid file (bare number). Nothing proves this pid is + # still ours, so leave it alone rather than kill a stranger. + info "skipping $name: pid file has no command to verify against" + elif proc_is "$pid" "$cmd"; then + kill_tree "$pid" + info "stopped $name (pid $pid)" + fi + rm -f "$SMOKE_DIR/$name.pid" + done +} + +# stop_all_bg — every background process the run owns. +stop_all_bg() { stop_bg logcat android_run ios_run; } + +# ---------------------------------------------------------------- reference coords +# In-call bottom control bar (y=2300): more / share / mic / camera / people / chat +A_MORE=98; A_SHARE=255; A_MIC=413; A_CAM=570; A_PEOPLE=841; A_CHAT=998 +A_BAR_Y=2300 +# Top bar (y=215) +A_END_X=998; A_TOP_Y=215 +# Login screen (reached deterministically via `pm clear`) +A_USER_X=539; A_USER_Y=1469 +A_SIGNUP_X=539; A_SIGNUP_Y=1632 +# Home screen +A_START_NEW_X=577; A_START_NEW_Y=1871 +A_CALLID_X=350; A_CALLID_Y=1683 +A_JOIN_X=875; A_JOIN_Y=1683 +# Lobby +A_TESTCALL_X=538; A_TESTCALL_Y=2084 +# "Your meeting is live" banner chevron +A_BANNER_X=922; A_BANNER_Y=945 +# Options sheet +A_REACT_X=224; A_REACT_Y=413 # first reaction (party popper) +A_BLUR_X=217; A_BLUR_Y=688 # blur filter +A_STATS_X=202; A_STATS_Y=1277 +# Leave the stats screen with the top-LEFT back arrow, never the top-right X. +# The X sits ~16px from the call screen's END CALL button, so if the stats screen +# ever fails to open, that tap silently ends the call. The top-left position is +# the harmless layout-toggle button on the call screen. +A_STATS_BACK_X=73 +# A "We Value Your Feedback!" modal appears after leaving a call on BOTH +# platforms. It blocks the home screen until dismissed. +A_FEEDBACK_X=973; A_FEEDBACK_Y=541 +# Chat sheet +A_CHAT_INPUT_Y=2293 # composer, keyboard closed (also the SEND row) +A_CHAT_INPUT2_Y=1434 # composer, keyboard open (also the SEND row) +# The keyboard does not always open — android_chat_send taps both rows rather +# than assuming which one is live. +A_CHAT_INPUT_X=605 +A_CHAT_SEND_X=974 diff --git a/.claude/skills/video-smoke-test/scripts/preflight.sh b/.claude/skills/video-smoke-test/scripts/preflight.sh new file mode 100755 index 000000000..c2955f183 --- /dev/null +++ b/.claude/skills/video-smoke-test/scripts/preflight.sh @@ -0,0 +1,95 @@ +#!/usr/bin/env bash +# Phase 0 — environment. Exits non-zero with a single actionable line if the +# run cannot be deterministic. Never guesses. +cd "$(dirname "$0")" && . ./lib.sh +cd "$(git rev-parse --show-toplevel)/dogfooding" || exit 1 + +: >"$RESULTS" +echo "== preflight ==" + +fatal() { echo "FATAL: $1" >&2; exit 1; } + +# --- android emulator ------------------------------------------------------- +# Pin the target explicitly. Every script below runs bare `adb`, which aborts +# with "more than one device/emulator" the moment a phone gets plugged in +# mid-run — screenshots come back 0 bytes and unrelated checks fail. Exporting +# ANDROID_SERIAL makes adb unambiguous for the whole run. +EMUS=$(adb devices | grep -oE "^emulator-[0-9]+" || true) +EMU_COUNT=$(printf '%s\n' "$EMUS" | grep -c . || true) +[ "${EMU_COUNT:-0}" -ge 1 ] \ + || fatal "no Android emulator running. Start one: \$ANDROID_HOME/emulator/emulator -avd &" +if [ "$EMU_COUNT" -gt 1 ]; then + fatal "$EMU_COUNT emulators running; cannot pick one. Leave exactly one booted." +fi +export ANDROID_SERIAL="$EMUS" +if adb devices | grep -vE "^(List|emulator-|$)" | grep -q "device$"; then + echo " note: a physical Android device is attached and will be ignored (pinned to $ANDROID_SERIAL)" +fi +android_geometry || exit 1 + +adb shell pm list packages 2>/dev/null | grep -q "com.google.android.gms" \ + || fatal "emulator has no Google Play services; FCM-dependent paths will misbehave. Use a Play-enabled AVD." + +# --- ios simulator ---------------------------------------------------------- +IOS_UDID=$(xcrun simctl list devices booted 2>/dev/null | grep -oE '[0-9A-F]{8}-[0-9A-F-]{27}' | head -1) +[ -n "$IOS_UDID" ] \ + || fatal "no booted iOS simulator. Boot one: xcrun simctl boot 'iPhone 17' && open -a Simulator" + +IOS_MODEL=$(xcrun simctl list devices booted | grep "$IOS_UDID" | sed 's/ (.*//' | sed 's/^ *//') +case "$IOS_MODEL" in + "iPhone 17"|"iPhone 17 Pro"|"iPhone Air"|"iPhone 16"|"iPhone 16 Pro") + ;; + *) + echo "WARN: booted simulator is '$IOS_MODEL'. The iOS tap coordinates in SKILL.md are" >&2 + echo " for a 402x874pt device (iPhone 17). Verify the panel reports 402x874" >&2 + echo " before tapping, or boot an iPhone 17." >&2 + ;; +esac + +# --- swift package manager must be off (otherwise the iOS build cannot link) - +if grep -q '"enable-swift-package-manager": *true' ~/.config/flutter/settings 2>/dev/null; then + echo " fixing: flutter SPM support is on; the dogfooding iOS build fails with it" + flutter config --no-enable-swift-package-manager >/dev/null 2>&1 +fi + +# --- pre-grant iOS permissions so no system dialog can eat a tap ------------ +# Bundle id differs per flavor; read it off the simulator if the app is already +# installed, else fall back to the dev-flavor default. +IOS_BUNDLE=$(xcrun simctl listapps "$IOS_UDID" 2>/dev/null \ + | grep -oE 'io\.getstream\.video\.flutter\.dogfooding[A-Za-z.]*' | sort -u | head -1) +IOS_BUNDLE=${IOS_BUNDLE:-io.getstream.video.flutter.dogfooding} +for svc in microphone camera notifications; do + xcrun simctl privacy "$IOS_UDID" grant "$svc" "$IOS_BUNDLE" >/dev/null 2>&1 +done +echo " ios permissions pre-granted for $IOS_BUNDLE" + +# --- android permissions, same reason --------------------------------------- +# READ_PHONE_STATE matters as much as the obvious ones: without it the app +# raises a "make and manage phone calls" dialog on the first Start New Call, +# which swallows the tap and derails the run. +grant_android_permissions + +# --- microphone prerequisite ------------------------------------------------- +# The audio checks need real sound to reach the emulator, which depends on two +# things that cannot be read back: host microphone access on the AVD (the +# emulator console has no microphone command) and macOS granting the emulator +# microphone access. So this only reminds; the checks name both causes when they +# fail. Never make it a hard failure — the rest of the run is valid regardless. +if command -v say >/dev/null 2>&1; then + echo " audio checks will speak through the host's speakers" + echo " they need the AVD's 'Enable Host Microphone Access' (Extended controls > Microphone)" +else + echo " WARN: no 'say' on this host — the microphone-dependent checks will fail" +fi + +cat >"$SMOKE_DIR/env.sh" </dev/null | grep -q "device$"; then + adb shell svc wifi enable >/dev/null 2>&1 + adb shell svc data enable >/dev/null 2>&1 + echo " emulator network enabled" +fi + +echo " logs and screenshots kept in $SMOKE_DIR" +echo "OK stop"