Skip to content

fix: rotate landscape screenshots in the pixels, not with an exif tag - #93

Merged
gmegidish merged 1 commit into
mobile-next:mainfrom
mobile-kevin:fix-landscape-screenshot
Oct 1, 2026
Merged

gmegidish merged 1 commit into
mobile-next:mainfrom
mobile-kevin:fix-landscape-screenshot

Conversation

@mobile-kevin

Copy link
Copy Markdown
Contributor

Summary

In landscape, device.screenshot returned an image that disagreed with the UI tree and with tap coordinates.

XCTest hands a rotated screen back as the panel's portrait pixels plus an orientation, and both encoders wrote that orientation as an EXIF tag (8, "rotate 270"). A decoder that ignores the tag (Go's image packages in mobilecli, among others) sees a 1206x2622 portrait image, while device.dump.ui and device.io.tap work in the rotated 874x402 space. Coordinates read off the screenshot land somewhere else.

  • Screenshot.swift: when the captured image carries an orientation, it is redrawn upright before encoding, so the rotation is in the pixels. No EXIF rotation is left for the viewer to apply.
  • Portrait screenshots take the same path as before (pngRepresentation / jpegData on the original image).

Result on iPhone 17 Pro simulator (iOS 26.0), Safari in landscape

Before After
PNG size 1206x2622 2622x1206
JPEG size 1206x2622 2622x1206
EXIF orientation 8 (rotate 270) 1 (normal)
App rect in device.dump.ui 874x402 874x402

Through mobilecli with this runner, mobilecli screenshot in landscape is 2622x1206, and cropping it at the rectangle dump ui reports for a button (x3) lands exactly on that button.

Testing

  • New tests in tests/rpc.test.ts ("device.screenshot in landscape"), for png and jpeg:
    • the screenshot is as wide and as tall as the foreground app's rect times the device scale
    • the screenshot does not ask its viewer to rotate it (EXIF orientation absent or 1)
  • tests/image.ts: a small png/jpeg reader for the size and the orientation tag, so the tests need no image library.
  • All four failed before the fix and pass after.
  • Full suite: 56 passed, 1 failed. The failure is device.io.swipe › swipes over an explicit duration, which passes on its own and fails when it runs after another swipe (about 1390ms against a 1400ms floor); it does not involve the screenshot handler.
  • swiftlint lint --strict reports nothing on the added lines.

Not covered

The MJPEG stream has no orientation handling either; it is not changed or tested here.

In landscape, XCTest returns the panel's portrait pixels plus an orientation,
which the png and jpeg encoders wrote as an EXIF tag. Decoders that ignore the
tag saw a portrait image while the ui tree and taps use the rotated screen, so
coordinates read off a screenshot landed somewhere else. The image is now
redrawn upright before it is encoded.
@coderabbitai

coderabbitai Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 92af5124-29f3-4f48-946a-21ed2deea757

📥 Commits

Reviewing files that changed from the base of the PR and between 5d499ae and 249fba3.

📒 Files selected for processing (3)
  • DeviceKitTests/JSONRPC/Handlers/Screenshot.swift
  • tests/image.ts
  • tests/rpc.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.


Walkthrough

The screenshot handler now rotates image pixels upright before PNG or JPEG encoding, with fallback behavior when rotation is unavailable. New test helpers parse PNG and JPEG dimensions and EXIF orientation. RPC tests capture landscape screenshots and check their dimensions and upright EXIF orientation.

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to 249fb

The screenshot rotation change is mergeable subject to normal checks. The supplied evidence establishes no actionable defect in screenshot encoding or the landscape tests.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 35.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 20 functions across 3 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: rotating landscape screenshots in the pixel data instead of relying on an EXIF orientation tag.
Description check ✅ Passed The description directly explains the screenshot orientation problem, the implementation, test coverage, reported results, and the unchanged MJPEG scope.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@gmegidish
gmegidish merged commit 288af25 into mobile-next:main Oct 1, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants