feat(browser): discover and drive nonstandard controls

[zzalancelot]

Summary

Whitelist-only snapshots could not discover or drive design-system controls (div buttons, date/select triggers, overlay year/month cells). This change adds pointer heuristics with depth, hidden form inventory, full pointer click sequences, overlay leaf discovery, selector/text addressing, browser_focus / browser_upload, and safer Enter handling while a floating panel is open.

Evidence (ByteDance referral apply form):

• Before: ~21 inventory items, no omit note → discovery gap, not budget truncation (maxInteractiveItems default 60)
• After artificial click on 「添加」: new standard inputs appear and browser_type works → gap was collection, not execution
• Date picker: panel can open on deep heuristic targets; year/month cells need overlay discovery or text:"2024" / text:"01"
• Writing controlled hidden inputs does not sync React picker state

Root causes

Layer	Cause
Discovery	Fixed CSS whitelist missed div/pointer controls
Discovery	Outermost-only nesting hid real picker triggers
Discovery	Overlay cells lacked cursor:pointer / roles
Action	HTMLElement.click() alone skips mousedown/focus open paths
Addressing	Index-only targeting could not reach panel cells reliably
Safety	Synthetic Enter→form.submit risked submitting apply forms

Changes

• Heuristic clickables ([heuristic]) with nested depth; filter common field-title labels; heuristic budget cap
• hiddenForms for visually hidden but focusable inputs (readable; typeable)
• browser_click: index | selector | text; full pointerdown→mousedown→focus→mouseup→click sequence
• Overlay short-text leaves as option "…" [overlay] inside panel/picker/dropdown roots
• browser_focus, browser_upload (Host reads absolute path → base64)
• region scopes main text and inventory; missing region errors
• Open Shadow DOM walk; no new permissions
• Do not synthesize form submit on Enter while an overlay is open
• Docs: press/IME limits, controlled hidden inputs, prefer higher depth

Compatibility

• Existing browser_click/browser_type index usage unchanged
• New parameters optional; approval model unchanged (upload still requires approval)
• Extension 0.1.5, bridge @yuxianglin/dsh-bridge-browser@0.0.6

Test plan

• pnpm --filter dsh-browser-extension run test (403)
• pnpm --filter @yuxianglin/dsh-bridge-browser run test (142)
• typecheck + build both packages
• fixture tests/fixtures/nonstandard-controls.html
• Live: click 「添加」; open date panel (high depth); text:"2024" / text:"01"; confirm row shows concrete month
• Regression: links/buttons, existing snapshot/type flows

Known limits

• Closed Shadow DOM still unread
• browser_press is not real Tab focus / IME
• Typing hidden inputs will not drive controlled design-system pickers
• Overlay heuristics depend on class/position/z-index + text click escape hatch

Confidence Score: 5/5

The PR appears safe to merge; no new actionable defects remain, and every previous thread is resolved.

Summary

This PR expands browser automation support for nonstandard design-system controls and addresses all previously reviewed issues.

• Discovers pointer-based controls, nested heuristic targets, overlay options, hidden form controls, and open Shadow DOM elements.
• Adds selector/text addressing, full pointer click sequences, focus and upload tools, region-scoped inventories, and safer Enter handling.
• Masks hidden form values and strengthens sensitive-field detection.
• Accumulates short file reads while preserving the upload-size boundary.
• Updates extension and bridge versions, documentation, fixtures, and tests.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[browser_snapshot] --> B[Visible controls]
  A --> C[Pointer heuristics]
  A --> D[Overlay leaves]
  A --> E[Masked hidden forms]
  B --> F[Stable element registry]
  C --> F
  D --> F
  E --> F
  F --> G[browser_click / type / focus / upload]
  H[Host file path] --> I[Bounded file read]
  I --> J[Base64 payload]
  J --> G

Loading

_{Reviews (3) · Last reviewed commit: "feat(browser): discover and drive nonsta..."}