Skip to content

feat(browser): show an agent cursor where the agent moves and clicks - #2286

Merged
chuks-qua merged 2 commits into
mainfrom
feat/issue-2210
Oct 23, 2026
Merged

chuks-qua merged 2 commits into
mainfrom
feat/issue-2210

Conversation

@chuks-qua

@chuks-qua chuks-qua commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

What

Before this change, the Browser overlay drew a cursor at the click point carried on the controller state. The kernel reported that point only when control changed hands, after the click had already landed, and in page CSS pixels. At any zoom other than 100% the mark missed its target, and repeated clicks often left it parked in the corner.

Now the Electron kernel sends a dedicated pointer event for every point, before it dispatches the input:

click / type target / targeted scroll
  resolvePoint
  admitAgentEffect
  emitPointer(move) → emitPointer(click)      # new, before input
  dispatch input

BrowserAutomationHost.onPointer → schema parse → store.pointers
BrowserActingOverlay
  map x,y through zoomFactor and page placement scale
  ease to the point, one ping per click, dim when idle
  hidden while the user holds control (ADR-0029)
guest preload (only while a tab recording runs)
  draws the same mark inside the page, so recordings include it

BrowserAutomationControllerState.pointer is removed. The new BrowserAutomationPointerEvent contract carries phase, x, y, zoomFactor and a sequence, so the same point clicked twice still counts as a new event. Nothing animates while the agent is idle.

Verified with unit tests at three seams: the kernel emits move then click at the element centre, the overlay maps the point through zoom and placement and hides under human control, and the guest preload draws only while recording. I also drove the real desktop kernel in Electron (see UI Changes).

Closes #2210

Why

People watching an agent use the Browser should see where it is about to act. A cursor that lands next to the button, or arrives after the click, makes that harder to follow than having no cursor at all.

UI Changes

Both clips use the same fixture page at 150% page zoom. A script clicks Alpha, Bravo and Charlie, then Delta below the fold. It calls desktopBridge.preview.automation.execute with a hand-built dispatch, so the real kernel, IPC and overlay run, and only the model is skipped.

Before (main): the cursor appears after each click and misses every button.

After: the cursor travels to each button before the click, pings, and follows the scroll to Delta.

Videos are attached below.

Config Changes

None.

Review Notes

  • Door: two-way. No migration or persisted data. Reverting restores the old controller pointer field. Blast radius: the Browser overlay and the desktop preview kernel.
  • During a tab recording, the host overlay mark and the in-page recording mark can both appear on screen.
  • A zoom change while the mark is showing does not move it until the next point arrives.
  • Out of scope: hover and drag (the kernel has no such operations) and the web runtime preview, which has no kernel.
  • In one live Codex run, the kernel reported a user takeover after the first click. The scripted runs completed four clicks without one, and this diff does not touch the takeover detection path. I could not reproduce it, so I have left it out of this PR.
  • Docs: none needed. The only pages that mention the cursor are point-in-time design and plan notes.
before.mp4
after.mp4

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

The Electron kernel reports each agent point (move, then click) before the
action. The Browser overlay maps it through page zoom and the responsive
scale, pings on click, dims when idle, and hides while the user holds
control. During a tab recording the guest page draws the same mark so the
video includes it.

Closes #2210
Parse pointer events from the desktop bridge with the contract schema before
they reach the overlay store. Report a recording as started only once the
page cursor is on, so the agent's first point lands in the video. Re-admit
control before a targeted scroll shows the mark, and delay the click ping
until the mark finishes its travel.
@chuks-qua
chuks-qua merged commit eed5b72 into main Oct 23, 2026
14 checks passed
@chuks-qua
chuks-qua deleted the feat/issue-2210 branch October 23, 2026 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: show an agent cursor where the agent moves and clicks in the Browser

1 participant