feat(browser): capture mouse buttons, modifiers and wheel for input.json version 3

[krassx]

Summary

The JavaScript half of input.json version 3, the wire contract that lets the viewer render mouse input — buttons, chords, modifiers and wheel deltas — instead of drawing every pointer event as an anonymous touch. The contract is specified once for all four components (iOS, Android, JS, viewer) in bugsee/specs#19; the other three land alongside this.

What the browser SDK now records

Field	Source	Notes
button	PointerEvent.button	The button that changed, remapped from the DOM's order to the contract's: {0:0, 1:2, 2:1, 3:3, 4:4}. The DOM calls the middle button 1 and the secondary 2; the contract is the other way round
buttonMask	PointerEvent.buttons	Bits of everything held, on every stage the platform can read it — begin/end, drag and hover move, and scroll
metaState	the four modifier flags	Android's META_* layout, so one viewer switch reads every platform
scrollX / scrollY	WheelEvent.delta*	Signed, accumulated per frame
scrollUnit	WheelEvent.deltaMode	pixel / line / page

Hover is sampled at a fixed interval rather than per event — a hovering mouse produces a move event per pixel, and the recording only needs the path.

Two decisions worth reviewing rather than skimming

button is never defaulted to 0. Zero means primary on the wire. A button we could not read is therefore omitted, never emitted as 0 — the same rule the other three SDKs follow. This is the single most repeated mistake in this contract; it appeared independently in about six places across the platforms.

Hover sampling is deliberately NOT masked-target-gated, unlike drag moves. A drag has a target to check because it opened on one; hover never opens a gesture, so it has none, and it carries only a bare position — the same as every other pointer coordinate this source already records over a masked field. This was flagged in review as looking like an oversight, and it is not; it is now documented in the file's PII discipline header and pinned by a test. That test was added late: a verification pass found that mutating the hover path to add the mask check left all 141 tests passing, so the decision was stated but not actually held in place.

scroll entries omit id — this SDK's convention of omitting an optional field rather than emitting a placeholder for one that does not apply. Noted because input.md describes id as always present; recorded in docs/design/sdk-design.md §8.4.1.

Testing

Full monorepo: 447 files, 6674 passed, 3 skipped (pre-existing), 0 failures. input-source.ts is at 100% line/branch/function/statement coverage. biome check clean on the touched files; tsc --noEmit clean across the workspace.

Every fix on this branch was then re-checked by reverting its production hunk and confirming a test goes red for the right reason — 15 behaviours, 14 already pinned, the one gap above closed.

Merge order

The viewer PR must land with or before this one. A version 3 recording read by today's viewer mislabels buttons silently — no error, just the wrong button name in the tooltip.

🤖 Generated with Claude Code

[krassx]

A generic question for all the coordinate values in input capture - have we validated that we always get correct X and Y (and/or bounds when required) for input events when page has zoom adjusted (both up and down)?

[krassx]

Good question, and the honest answer is no — zoom is not validated, and checking it turned up a real bug that is not in the mouse code.

What is consistent

Input coordinates are clientX/clientY and viewtree rects are getBoundingClientRect(). Both are CSS pixels relative to the layout viewport, so they stay in step with each other at any zoom — page zoom and pinch zoom both leave that relationship intact (pinch zoom moves the visual viewport; neither API reports it). Target attribution and hit-testing are therefore sound.

What is not

The viewer does not draw coordinates against the viewport. It normalises them against environment.hardware.screen, which for the browser SDK is screen.width/screen.height (packages/browser/src/environment.ts:54). screen.width is itself expressed in CSS pixels, so it moves inversely with zoom while the coordinates do not. Measured, headless Chrome, one page, varying only the CSS-px-to-device-px ratio:

ratio	screen.width	getBoundingClientRect().left
1	800	100
1.5	534	100
2	400	100

The coordinate is stable; the frame it is divided by shrinks. A click at clientX = 600 lands at 600/800 = 75% across the frame at 100% zoom and 600/400 = 150% — off the frame entirely — at 200%.

Two things make it worse:

1. screen.width is the monitor, not the window. Even at 100% zoom, a browser window that is not full-screen is normalised against the whole display. This is wrong in the common case, independent of zoom.
2. It is captured once, at launch — there is no video.aux equivalent for the web SDK, so a zoom change or a window resize mid-recording is never re-stated. (This is the same class of bug as Android's B6, where a recording with no letterboffing emitted no aux event at all and the viewer never learned the screen size.)

Grepped the whole monorepo: innerWidth, innerHeight and visualViewport appear nowhere outside tests. We do not record the viewport at all.

What would fix it

Record the layout viewport (innerWidth/innerHeight) as the frame, re-stated whenever it changes — a resize and a visualViewport listener, emitted as an aux-style entry rather than a launch-time constant — and have the viewer prefer it over screen for app_type web. Pinch zoom additionally needs visualViewport.scale/offsetLeft/offsetTop if we ever want to draw against what the user actually saw rather than the layout.

That is a wire-contract change (a new field, plus a viewer branch), so I would rather agree it than slip it in. Two options:

• Fold it into this PR — the v3 contract is still under review, which is the cheapest moment to add a field.
• Land this and follow up — nothing here regresses: the frame is equally wrong for the touch coordinates this SDK already recorded before v3. The mouse work does not introduce it.

I lean towards folding it in, since the alternative ships a contract we already know is missing a field. Say which and I will do it.

Caveat on the evidence: the table above varies headless Chrome's device-scale-factor, which exercises the same CSS-px ratio that page zoom does. Real ctrl-zoom on a real browser is worth confirming — and screen.width behaviour under zoom differs between Chrome/Firefox and Safari, which is its own argument for not using it as the frame.

🤖 Generated with Claude Code