Repository navigation
feat(browser): show an agent cursor where the agent moves and clicks - #2286
Merged
Merged
Conversation
The Electron kernel reports each agent point (move, then click) before the action. The Browser overlay maps it through page zoom and the responsive scale, pings on click, dims when idle, and hides while the user holds control. During a tab recording the guest page draws the same mark so the video includes it. Closes #2210
Parse pointer events from the desktop bridge with the contract schema before they reach the overlay store. Report a recording as started only once the page cursor is on, so the agent's first point lands in the video. Re-admit control before a targeted scroll shows the mark, and delay the click ping until the mark finishes its travel.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Before this change, the Browser overlay drew a cursor at the click point carried on the controller state. The kernel reported that point only when control changed hands, after the click had already landed, and in page CSS pixels. At any zoom other than 100% the mark missed its target, and repeated clicks often left it parked in the corner.
Now the Electron kernel sends a dedicated pointer event for every point, before it dispatches the input:
BrowserAutomationControllerState.pointeris removed. The newBrowserAutomationPointerEventcontract carriesphase,x,y,zoomFactorand asequence, so the same point clicked twice still counts as a new event. Nothing animates while the agent is idle.Verified with unit tests at three seams: the kernel emits move then click at the element centre, the overlay maps the point through zoom and placement and hides under human control, and the guest preload draws only while recording. I also drove the real desktop kernel in Electron (see UI Changes).
Closes #2210
Why
People watching an agent use the Browser should see where it is about to act. A cursor that lands next to the button, or arrives after the click, makes that harder to follow than having no cursor at all.
UI Changes
Both clips use the same fixture page at 150% page zoom. A script clicks Alpha, Bravo and Charlie, then Delta below the fold. It calls
desktopBridge.preview.automation.executewith a hand-built dispatch, so the real kernel, IPC and overlay run, and only the model is skipped.Before (main): the cursor appears after each click and misses every button.
After: the cursor travels to each button before the click, pings, and follows the scroll to Delta.
Videos are attached below.
Config Changes
None.
Review Notes
before.mp4
after.mp4
Need help on this PR? Tag
@codesmith-botwith what you need.