Skip to content

[nova] tracking: secure Chrome semantic provider #23

Open
12 of 12 issues completed
Open
[nova] tracking: secure Chrome semantic provider#23
12 of 12 issues completed
Labels
enhancementNew feature or request

Description

@bigduu
bigduu
opened on Jul 24, 2026

Delivery tracker (2026-10-03)

This remains a tracking Issue. The merged developer bridge includes explicit selected-tab/site consent, bounded visible text, open-shadow semantics, safe value receipts, authenticated Windows transport, per-user Windows host registration, and opt-in proven same-origin child reads and DOM activation, and bounded top/child activation effect receipts. Explicit ax_read(target="paired_page") and canonical ax_activate now use the consented page; automatic native-window association, mixed browser-chrome/page snapshots and full packaged-browser acceptance remain pending. The original acceptance criteria below are unchanged.

Merged and closed acceptance children:

Each child passed complete independent review and its exact-head CI/acceptance gates. Windows execution uses owned private registry/temp/process fixtures; actual Chrome discovery and packaged GUI are separate. Proven same-origin reads/DOM activation remain partial coverage and do not complete the parent's broader permitted-frame, browser/window identity, canonical routing or trusted-input criteria.

Remaining slices receive focused child Issues before implementation:

  • Broader permitted child-frame coverage and cross-origin actions; child focus/value/scroll remain pending.
  • Native-window/page association for the observed browser-process identity, then automatic canonical ax_read provider routing.
  • Broader action/navigation result confirmation and verified trusted-input fallback.
  • Packaged extension/host smoke on macOS and Windows, plus the full parent regression matrix.

Related #35 owns popup freshness and #36 owns signed distribution under Zenith#185. Do not duplicate that work or equate isolated developer fixtures with production distribution. Source #34/#70 is merged, with manual macOS desktop/cursor acceptance still pending permissions. Zenith#318 / PR#319 delivers the accepted Nova source gitlink at Zenith f45d83bd188e17bae9b2d2f8b894a2b9faeecd0c; both exact-head root PR guards and full independent pointer review passed. The merged-main pointer check passed; the PR-only notes check skips on push. Local root/Nova are synchronized; Bamboo/Bodhi dirty gitlinks and all seven unrelated primary repositories remain unchanged. The already-accepted upstream release config from the fresh pointer base was included by the normal root fast-forward.

There is no monolithic implementation branch for this parent. Every child retains its own bounded scope, estimate, tests and PR. Keep #23 open and all original acceptance boxes unchanged until actual parent-level evidence exists.

Summary

Add a Manifest V3 Chrome/Chromium extension plus a native Nova bridge so browser page content becomes a reliable semantic provider for Nova's ax_read / node-activation contract from #22.

For a Chrome page, Nova should prefer the extension-backed DOM/ARIA snapshot and action channel for page content, while continuing to use operating-system Accessibility for browser chrome such as tabs, toolbar, and address bar. If the extension is absent, disconnected, lacks permission, or cannot instrument the page, Nova must report that reason and follow #22's native-AX → OCR → focused-screenshot fallback rather than pretending the page was empty.

This is a separate Issue from #22 because it introduces a distributable browser extension, Native Messaging registration/lifecycle, tab/frame identity, host permissions, a versioned protocol, and a new security boundary.

Code review: why the current bridge is insufficient

Current master (ff840350) already recognizes that macOS Chrome AX actions are unreliable:

  • src/server.rs::click_cached_mark tries ElementHandle::try_web_click before AX and coordinate fallback.
  • src/platform/mac/elements/webclick.rs runs an AppleScript command against active tab of front window, injecting a fixed document.querySelectorAll(...) / document.elementFromPoint(...).click() snippet.
  • The source explicitly notes that AXPress can return success while doing nothing on web content.

That workaround is useful, but it is not a browser integration contract:

  1. It targets only the front window's active tab, so the tab associated with the AX node is not proven to be the tab that receives the script.
  2. It depends on macOS Automation and Chrome's “Allow JavaScript from Apple Events” setting; failure is collapsed into the generic AX/coordinate fallback.
  3. It returns only a short click description. It cannot supply labels, structured text, form state, links, headings, frame identity, or a page-change signal to ax_read.
  4. The injected script sees only the top document, does not aggregate permitted cross-origin frames, does not traverse open shadow roots, and uses a small hard-coded selector/accessible-name approximation.
  5. HTMLElement.click() does not satisfy every trusted-user-gesture flow. When it fails, Nova lacks a browser-verified fresh element rectangle and an explicit needs_trusted_input result.
  6. There is no extension/native bridge, extension package, installation/status command, permission UX, protocol-version negotiation, or Chrome-specific integration test in the repository.

Relationship to #22

#22 owns the platform-neutral ax:read capability, canonical ax_read tool, snapshot/node model, action routing, and fallback policy.

This Issue owns one provider behind that contract:

Native application:
  ax_read -> OS AX/UIA provider

Chrome/Chromium target:
  browser chrome -> OS AX/UIA provider
  page content   -> Chrome extension provider (preferred)
  bridge unavailable/unsupported -> OS AX/UIA -> OCR -> focused screenshot

The model should normally keep calling ax_read and activating returned nodes. Provider selection belongs inside Nova; the prompt should not require the model to choose AppleScript, extension, or AX manually.

Proposed architecture

Nova MCP server
  <-> owner-only local IPC (Unix socket on macOS, named pipe on Windows)
  <-> `nova browser-host` Native Messaging mode
  <-> Chrome `runtime.connectNative` long-lived port
  <-> MV3 service worker
  <-> per-tab/per-frame content scripts

Native side

  • Add a dedicated nova browser-host mode that implements Chrome Native Messaging framing over stdin/stdout. Protocol stdout is reserved exclusively for framed JSON; diagnostics go to stderr.
  • Register a native-host manifest with an exact allowed_origins entry for the published extension ID. Never use a wildcard extension origin.
  • Add idempotent nova browser-extension install-host, uninstall-host, and status commands for supported Chrome-family browsers/platforms. Report each browser registration path and extension/host version.
  • Route requests from a running Nova MCP server to the Chrome-launched host through per-user local IPC. Do not expose an unauthenticated localhost HTTP/WebSocket port.
  • Authenticate/claim the MCP-to-host session with an owner-only runtime token or equivalent lease, bound to one user and one active Nova session. Define deterministic behavior when multiple Nova servers exist; never let the extension broadcast page data to every process.
  • Use request IDs, deadlines, cancellation, bounded payloads, protocol version negotiation, heartbeats, and reconnect/backoff. Native Messaging messages must remain below Chrome's response-size limit; paginate/truncate snapshots rather than emitting an unbounded DOM.

Extension side

  • Add a self-contained Manifest V3 extension under a reviewed repository path such as extension/chrome/.
  • A service worker owns the Native Messaging port and routes structured commands to the exact tabId, frameId, and documentId.
  • Content scripts are packaged with the extension; do not download or evaluate remote code and do not accept arbitrary JavaScript from Nova/LLM.
  • Support the top document, permitted child frames, and open shadow roots. Every node is frame/document scoped.
  • Keep incognito and file:// access off unless the user explicitly enables them in Chrome.
  • Restricted surfaces (chrome://, Chrome Web Store, extension pages, inaccessible PDF/plugin surfaces, denied origins) return a typed unsupported/permission result.

Permission model

Use least privilege by default:

  • nativeMessaging, scripting, and activeTab for explicit per-tab enablement.
  • Optional host permissions for “allow this site” and an explicit user opt-in for broader browsing. Do not silently ship persistent <all_urls> access as the only mode.
  • Show connection/site-access state in the extension action: disconnected, connected but site denied, enabled for this tab/site, and unsupported page.
  • Revocation must immediately remove content scripts where possible, invalidate snapshots, and stop returning page data.

chrome.debugger is not an MVP dependency. It provides CDP Accessibility/Input domains, but requires the powerful non-optional debugger permission. If DOM activation plus a real Nova foreground click cannot cover required cases, evaluate a separately consented extension variant/follow-up instead of silently adding debugger access.

Versioned provider protocol

At minimum, define typed messages for:

  • hello / capabilities / version negotiation
  • list_targets and exact tab/window selection
  • snapshot
  • activate
  • focus
  • set_value
  • scroll_into_view
  • cancel
  • navigation/document invalidation events

A snapshot response should fit #22's neutral DTO while retaining browser identity:

{
  "provider": "chrome_extension",
  "snapshot_id": "ephemeral",
  "target": {
    "browser": "chrome",
    "window_id": 3,
    "tab_id": 41,
    "frame_id": 0,
    "document_id": "...",
    "url": "https://example.test/",
    "title": "Example"
  },
  "coverage": "complete",
  "nodes": [
    {
      "id": "document-scoped-node",
      "role": "button",
      "name": "Save",
      "value": null,
      "states": { "enabled": true, "focused": false },
      "actions": ["activate"],
      "bounds": { "space": "frame_css_px", "x": 12, "y": 40, "w": 80, "h": 32 }
    }
  ]
}

Requirements:

  • include visible semantic text and controls, not raw innerHTML or the entire DOM;
  • implement/test accessible-name, role, value, description, and state derivation rather than relying only on textContent;
  • preserve deterministic reading/tree order and frame ancestry;
  • include links/headings/static text/form values/selection/checked/expanded/disabled/focused state where safe;
  • redact password fields and sensitive values before filtering, tracing, caching, or IPC;
  • bound node count, depth, text length, and total payload, with explicit partial/truncated coverage;
  • do not leak cookies, storage, network bodies, hidden DOM, or arbitrary page globals.

Snapshot identity and stale safety

Node references are ephemeral and must be bound to (browser instance, tabId, frameId, documentId, snapshot generation).

Before every action, the content script must verify that:

  • the exact tab/frame/document still exists;
  • the node handle is still connected and visible;
  • its role/name/actionability still match the captured fingerprint;
  • the page has not navigated to a different document/origin.

Navigation, content-script restart, permission revocation, or an invalid handle must fail closed with stale_snapshot; Nova then calls ax_read again. Never re-resolve an old node by first matching label text, and never redirect a command to whichever tab is currently frontmost.

Action ladder for Chrome page content

  1. ax_read returns a fresh extension-backed browser node.
  2. activate(snapshot_id, node_id) targets that exact tab/frame/document and revalidates the node.
  3. Prefer a structured DOM action appropriate to the element (click, focus, form-control change, link activation), with an explicit result and post-action document/state observation.
  4. If the site requires trusted input, return needs_trusted_input plus a freshly validated visible rectangle and frame-to-main-viewport mapping.
  5. Nova activates the exact browser window/tab, converts CSS-pixel bounds safely, and performs a real foreground center click; then re-reads through the extension.
  6. Only if the extension cannot identify the target should feat(ax): add first-class ax_read semantic snapshots and an AX-first fallback contract #22 continue to OCR-center and focused visual-coordinate fallback.

Every result reports its route, for example:

  • chrome_dom
  • chrome_dom_then_navigation
  • chrome_trusted_input_fallback
  • native_ax
  • ocr_center
  • visual_coordinate

Do not report success merely because HTMLElement.click() returned. Confirm an observable state/navigation/focus change when the requested action expects one; otherwise return no_observed_effect and let the agent choose the next safe route.

Typed failure/fallback reasons

At minimum:

  • extension_not_installed
  • native_host_not_registered
  • bridge_disconnected
  • protocol_mismatch
  • site_permission_required
  • unsupported_scheme
  • restricted_page
  • tab_not_found
  • frame_not_accessible
  • stale_snapshot
  • ambiguous_target
  • needs_trusted_input
  • no_observed_effect
  • timed_out
  • partial

These feed #22's fallback policy. Missing site permission is an explicit user-consent state, not evidence that the page has no semantics.

Security invariants

  • No arbitrary JavaScript/eval/code payload from MCP, LLM, or a web page.
  • No externally_connectable web origins unless separately justified; ordinary pages must not be able to command Nova.
  • Validate every message at the service-worker, native-host, and MCP boundaries. Treat content-script messages as untrusted.
  • Exact extension origin allowlist in the native-host manifest.
  • Owner-only local IPC and session claim; no network listener exposed to the LAN or other users.
  • Page data is returned only in response to a scoped Nova request; no continuous bulk scraping by default.
  • Redact secrets before they cross the content-script boundary.
  • Actions are limited to the selected tab/site permission and structured verbs.
  • Surface a visible connected/enabled indicator and provide one-step disconnect/revoke controls.

Acceptance criteria

  • A packaged MV3 extension and versioned native-host protocol exist, with macOS and Windows host registration/status/uninstall flows.
  • The extension connects only to Nova's exact registered Native Messaging host; Nova exposes no unauthenticated localhost HTTP/WebSocket endpoint.
  • ax_read automatically uses provider=chrome_extension for permitted Chrome page content and OS AX/UIA for browser chrome, returning one coherent snapshot contract.
  • A fixture page returns buttons, links, headings, labels, static text, field values, and widget states in deterministic order without a screenshot.
  • Permitted child frames and open shadow roots are represented with frame/document-scoped IDs; inaccessible frames are reported as partial coverage.
  • Password/sensitive values never appear in snapshot output, logs, traces, caches, or native messages.
  • Activation targets the exact tab/frame/document even when another Chrome tab/window is frontmost.
  • Navigation and DOM replacement invalidate stale references; old node IDs fail closed.
  • Same-label nodes return distinct candidates and are never resolved by “first substring match.”
  • DOM activation reports no_observed_effect or needs_trusted_input when appropriate; Nova can use the fresh element bounds for a verified real click and then re-read.
  • Extension missing/disconnected, denied site permission, restricted page, and protocol mismatch each produce typed outcomes and follow feat(ax): add first-class ax_read semantic snapshots and an AX-first fallback contract #22's fallback ladder.
  • Default permissions are least-privilege; broader site access is explicit, revocable user consent. Incognito/file access remains opt-in.
  • The release/plugin prompt describes this as an internal semantic provider and still tells the model to use ax_read, not to inject JavaScript or guess coordinates.
  • The existing AppleScript web click remains a bounded compatibility fallback during migration and is not claimed as equivalent to the extension provider.

Deterministic test plan

  • Protocol tests: framing, version mismatch, request IDs, cancellation, timeouts, reconnect, payload limits, malformed/untrusted messages, multiple Nova processes.
  • Extension unit tests: visibility, accessible names/roles/state, same-label nodes, password redaction, open shadow roots, document invalidation, action/no-effect classification.
  • Browser integration fixtures: top frame + cross-origin permitted/denied iframe, dynamic SPA navigation, virtualized list, forms, shadow DOM, user-gesture-required flow, restricted URL.
  • Playwright persistent-context E2E with the unpacked extension and a fake Native Messaging host, followed by live packaged-extension smoke tests.
  • macOS Chrome smoke: extension read, background-tab exact targeting, DOM action, trusted-click fallback, bridge disconnected fallback.
  • Windows Chrome/Edge smoke: same protocol and provider behavior.
  • Security tests: hostile content-script messages, attempted arbitrary-code command, secret fields, revoked permission, wrong extension origin, unauthorized local IPC client.
  • Regression gates: existing read_ui/click_mark, macOS native AX, Windows UIA, OCR, screenshot, Linux headless introspection, plugin packaging/signing.

Non-goals

  • Replacing OS AX/UIA for native applications or Chrome's own toolbar/tabs.
  • A general remote-debugging or arbitrary-JavaScript execution API.
  • Network interception, cookie/storage extraction, or hidden-page scraping.
  • Making restricted Chrome pages scriptable.
  • Requiring chrome.debugger in the first implementation.

Primary references

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions