Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
287 changes: 121 additions & 166 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,175 +1,130 @@
# WebMCP Computer

**A full computer that lives inside a browser tab, with WebMCP as its native control layer.**

Today agents operate computers by imitating humans: screenshots, mouse coordinates, typing
into interfaces built for eyes and hands. WebMCP Computer is a computer built for agents as
first-class citizens. Files, processes, applications, windows, and the terminal are exposed
directly as WebMCP tools. The agent gets a sandboxed machine it can understand and control
natively, and the human watches and intervenes through a familiar desktop GUI on the same
screen.

Entry for the [OpenAI WebMCP Challenge](https://webmcp.devpost.com/).

- **Live:** <https://computer.webmcp.com> (open in ChatGPT's browser or Chrome with
`--enable-features=WebMCP`)
- **License:** MIT (see [LICENSE](./LICENSE))

## What a human and an agent can do together

- Give a coding agent a fresh machine: edit files, run scripts in the in-browser shell,
serve a folder, and preview the result in a sandboxed window the human can see.
- Run agent-generated code without giving the agent your real computer. Preview and
agent-made apps run in opaque sandboxed frames inside the tab.
- Let the agent inspect `dmesg`, list and kill processes, change settings, and repair a
wedged workspace while the human watches every action land.
- Build a small site inside the OS, then let that site register its own WebMCP tools which
the agent calls through the same surface (WebMCP inside WebMCP).
- Drive a shared remote Chrome from the OS's Browser app and call the remote site's own
WebMCP tools from inside the sandbox.
- Opt into a real Linux container (git, node, python) for cloning repos and running test
suites; the human sees the streamed output in Terminal.
- Publish a folder to a public URL with one call, and open it on a phone from the QR toast.

Every control a human can click has a verb an agent can call. Every agent action leaves a
visible trace: an agent cursor jumps to the touched window, an activity toast names the
verb, and agent-written terminal rows are tinted.

## Why WebMCP

WebMCP gives a page one channel to describe itself to an agent: tools with names,
descriptions, JSON schemas, and annotations. WebMCP Computer treats that channel as a
syscall table. Instead of one page exposing a form or two, the whole operating system is
the tool surface, so an agent operating the machine never needs a screenshot or a
coordinate.

The protocol's dynamism is load-bearing here. Per-app tools register when a window opens and
unregister (abort-signal) when it closes. Pages served inside the OS register `site_*` tools
through an injected `document.modelContext` facade that proxies to the host registry.
Chrome's `ontoolchange` is what makes both possible.

## Tool vocabulary

Wire names are `snake_case` verbs. Each declares an invocation class through annotations:
**ask** (read-only), **act** (reversible, visible), **transact** (consequential).

| Domain | Tools |
|---|---|
| Apps and windows | `app_open` `app_close` `app_list` `window_focus` `window_move` `window_resize` |
| Filesystem | `fs_read` `fs_write` `fs_edit` `fs_list` `fs_search` `fs_mkdir` `fs_delete` `fs_move` |
| Terminal and processes | `term_exec` `term_read` `term_state` `term_history` `ps` `kill` |
| System | `sys_status` `os_manual` `os_search` `settings_get` `settings_set` `screensaver_wake` `machine_take_over` |
| Browser (remote Chrome) | `browser_open` `browser_goto` `browser_click` `browser_type` `browser_read` `browser_screenshot` `browser_site_tools` `browser_site_call` |
| Cloud (opt-in) | `cloud_exec` `os_publish` |
| Agent-made apps | `ui_open` |
| Per-app, while the window is open | `editor_open_file` `files_reveal` `notes_append` `notes_preview` `notes_stick` `preview_get_console` `preview_get_url` `preview_reload` |
| Dynamic, from served pages | `site_*` (registered by the page itself; capped at 16 per Preview) |

The manual ships with the machine: `docs/agent-skills/` is seeded into `~/skills/` and served
by `os_manual`, because WebMCP has no resources channel besides tools.

## How it's built

- **Registration:** all tools go through [`@nekuda/webmcp-sdk`](https://www.npmjs.com/package/@nekuda/webmcp-sdk)
(`defineTool` + `registerTools`), which resolves the live `document.modelContext` surface
and pins the spec version. Errors are returned as MCP `{ content, isError: true }` results.
- **Kernel:** Zustand store in pure TypeScript. Process table, window registry, event bus.
Every tool call and human action emits an OS event that AgentPresence, `dmesg`, and the
Tool Monitor all read.
- **Shell:** [just-bash](https://www.npmjs.com/package/just-bash), a bash interpreter in
JavaScript with coreutils, `awk`, and `jq`. The browser shell keeps network commands and
language runtimes off; those live in the opt-in cloud container.
- **Filesystem:** ZenFS on OPFS, in-memory fallback. Session restore across reloads.
- **Preview:** a virtual HTTP server in the tab serves `~` folders into a sandboxed frame and
bridges the frame's `site_*` tool registrations to the host.
- **Browser app:** one shared remote Chrome via Cloudflare Browser Run, controlled over CDP
through a token-holding session Worker.
- **Cloud kernel (opt-in):** a Cloudflare Worker plus container (Debian, git, node 22,
python3) mounts the workspace, executes `cloud_exec` commands, and publishes sites to R2.
- **Stack:** Vite, React 18, TypeScript strict, hand-rolled CSS. No UI kit.

See `docs/BRIEF.md` for the build brief, `docs/webmcp/REFERENCE.md` for the WebMCP surface
as empirically verified, and `docs/features/` for per-feature specs.

## Repository layout

- `web/` — the OS. `cd web && bun install && bun run dev`
- `workers/browser-session/` — token-holding Browser Run session Worker.
- `workers/computer/` — opt-in cloud-workspace and public-site Worker: Durable Object
filesystem, Container execution, R2 publishing.
- `shared/gateway-capability.ts` — provider-neutral signed capability contract shared by
the site and both Workers.
- `shared/session-limits.ts` — the visitor budgets (2 h remote Chrome and 2 h cloud
container per machine per 24-hour accounting window, 5-minute idle stops) that the site, both Workers, and
the client all enforce and display from one place.
- `docs/` — build brief, feature specs, seeded agent manual (`agent-skills/`), testing
charter, WebMCP reference, demo briefs (`demo/`), [`SELF_HOSTING.md`](docs/SELF_HOSTING.md),
and the on-call runbook [`OPERATIONS.md`](docs/OPERATIONS.md).
- [`SECURITY.md`](./SECURITY.md) · [`CONTRIBUTING.md`](./CONTRIBUTING.md) · CI in `.github/workflows/`.

## Running and testing
**One machine. Two users.**

WebMCP Computer is a full computer inside a browser tab, shared by a human and an AI
agent. The human uses a familiar desktop. The agent operates the same files, windows,
applications, processes, and terminal through native WebMCP tools.

No screenshot interpretation. No coordinate guessing. Just structured tools, shared state,
and visible collaboration.

![WebMCP Computer shared by a human and an AI agent](docs/assets/readme/hero.png)

[**Try the live computer**](https://computer.webmcp.com) · [**Watch the demo**](docs/assets/readme/webmcp-computer-demo.mp4) · [**Run locally**](#run-locally)

> Open the live demo in ChatGPT's browser or Chrome 151+ with
> `--enable-features=WebMCP`.

## What this demonstrates

- **A native interface for agents.** The operating system is the WebMCP tool surface:
`fs_write`, `term_exec`, `app_open`, `window_move`, `settings_set`, `ps`, `kill`, and
more. Agents act on named operations and structured data instead of pixels.
- **One shared machine.** Human and agent use the same filesystem, windows, apps, and
terminal. Agent actions are visible as they happen through a live cursor, activity
traces, and attributed terminal output.
- **A dynamic tool surface.** Apps register tools when their windows open and remove them
when they close. The available capabilities always match the machine in front of you.
- **WebMCP inside WebMCP.** A page built and served inside the computer can register its
own `site_*` tools. The agent can create an interface and immediately operate it through
the same WebMCP surface.

## Try it

Open the live computer and ask your agent:

> Open `~/desktop/pizza-demo.md`, build the demo, serve it, add one large pepperoni to
> the cart, and place the demo order.

The agent will edit files, run commands, open Preview, discover the tools exposed by the
new page, and call them. You can watch, interrupt, or take over at any point.

## What it can do

- Edit, search, move, and inspect files in a persistent browser filesystem.
- Run shell commands in an in-browser Bash environment and share one terminal with the
human.
- Build and run sandboxed websites and agent-made HTML applications.
- Inspect processes, change settings, search the machine, and read its built-in manual.
- Drive a shared remote Chrome and call WebMCP tools exposed by the remote page.
- Opt into a Linux container with git, Node.js, and Python for real development work.
- Publish a folder to a temporary public URL and open it from the generated QR code.

## How it works

```mermaid
flowchart LR
Human[Human] -->|Desktop UI| Computer[WebMCP Computer]
Agent[AI agent] -->|WebMCP tools| Computer
Pages[Served pages] -->|Dynamic site_* tools| Computer
Computer --> Machine[Shared kernel<br/>Filesystem / apps / terminal / event log]
```

- [`@nekuda/webmcp-sdk`](https://www.npmjs.com/package/@nekuda/webmcp-sdk) registers the
system and application tools against the browser's live WebMCP surface.
- A TypeScript/Zustand kernel owns the process table, window registry, settings, and event
log.
- [ZenFS](https://zenfs.dev/) stores the local filesystem in OPFS, with an in-memory
fallback. [just-bash](https://www.npmjs.com/package/just-bash) provides the local shell.
- Sandboxed Preview frames bridge their own WebMCP tools into the host registry.
- Optional Cloudflare Workers provide the remote browser, Linux container, and temporary
published sites. The core computer remains local-first.

## Run locally

Requires [Bun](https://bun.sh/) 1.3.x.

```sh
cd web
bun install
bun run dev # http://localhost:5173
bun run build # tsc --noEmit && vite build
bun run dev
```

Then open <http://localhost:5173>. The desktop works in a regular browser; native agent
invocation requires ChatGPT's browser or Chrome 151+ with WebMCP enabled.

```sh
bun test # unit tests
bun run test:e2e # needs Chrome 151+ with native WebMCP; see docs/testing/README.md
bun run build # typecheck and production build
bun run test:e2e # native-Chrome WebMCP suite
```

## Self-hosting

See [`docs/SELF_HOSTING.md`](docs/SELF_HOSTING.md). Templates contain no account ID, API
token, gateway secret, or user data. Wrangler validates required secrets before deployment.

Worker URL resolution in the browser, in order:

1. `?browser_worker=` / `?computer_worker=` query override, or
`localStorage["webmcp_computer.browser_worker"]` /
`localStorage["webmcp_computer.computer_worker"]`. Production accepts these only for
`127.0.0.1` or `localhost`.
2. The signed demo-session configuration returned by the site's `/api/session`.
3. `VITE_BROWSER_WORKER_URL` / `VITE_COMPUTER_WORKER_URL` baked in at build time.

With none configured, cloud features fail with a visible error and the OS stays local.
Site uploads use the browser-held capability route `POST /ws/{wsid}/publish`; no public
`/publish` route exists.

## Network disclosure

When both `VITE_POSTHOG_KEY` and `VITE_POSTHOG_HOST` are set at build time, the web app
sends pseudonymous, content-free typed usage events and privacy-masked session replay to that
PostHog project. Autocapture, page views, person profiles, console/error capture, network
payload capture, surveys, canvas, inputs, app content, and user-site frames are excluded; DNT and
Global Privacy Control are honored. With either value absent, PostHog is not initialized.
The public PostHog project key is browser configuration, not a secret. No project key is
committed here.

WebMCP Computer registers its tools through `@nekuda/webmcp-sdk`, which sends anonymous,
content-free usage beacons (SDK init, tool registration, tool call outcomes) to nekuda's
telemetry endpoint by default. It honors Global Privacy Control and
`globalThis.__WEBMCP_TELEMETRY__ = false`. Every demo visitor receives a random,
cookie-backed machine with short-lived (15-minute, auto-renewed) capabilities; no provider
identity is requested or stored.
Opening Browser contacts the `webmcp-computer-browser-session` Worker, which creates one
remote Chrome per machine on Cloudflare Browser Run and hands back session-scoped URLs; the
client heartbeats only while you are active and the tab is visible, and the Worker deletes
the Chrome after 5 idle minutes. Enabling `cloud_kernel` sends filesystem bytes to a
capability-addressed workspace on the `webmcp-computer-cloud` Worker after reboot; `cloud`
and `cloud_exec` send requested commands and their output streams through that Worker's
container, which is stopped 5 minutes after the last command; `os_publish` sends selected
text files to that Worker's public R2-backed site URL (served `noindex` and sandboxed,
deleted after 30 days; the manifest keeps a pseudonymous publisher record for takedowns).
Each machine may complete 20 publishes, gets 2 hours of remote Chrome, and gets 2 hours of
container time per 24-hour accounting window.
The site backend stores only the gateway signing secret and Worker URLs; the Browser Worker
alone stores the Browser Rendering API token. Local kernel remains the default, and cloud
failure falls back visibly to local.

## Scope notes

One WebMCP Computer machine runs per browser profile (a second tab is blocked unless the
human explicitly selects **Take over**), and its local filesystem persists in OPFS between visits. Git,
node, and real test runners are available only through the opt-in cloud kernel; the
in-browser shell has no git.
The repository has three independent package roots: `web/`, `workers/browser-session/`,
and `workers/computer/`. See [CONTRIBUTING.md](CONTRIBUTING.md) for the complete setup and
CI commands.

## Privacy and network use

The local computer, filesystem, shell, editor, and Preview run in the browser. Networked
features are explicit:

- The hosted demo enables privacy-bounded PostHog analytics only when its public build-time
key and host are configured. Events contain fixed operation categories rather than
commands, paths, file contents, terminal output, URLs, selectors, or typed text. Sensitive
surfaces are excluded from session replay. Self-hosted builds leave PostHog off unless
their operator supplies both values.
- `@nekuda/webmcp-sdk` has a separate anonymous, content-free usage channel enabled by
default. It respects Global Privacy Control and can be disabled page-wide with
`globalThis.__WEBMCP_TELEMETRY__ = false`.
- Opening Browser creates a machine-scoped remote Chrome session through the configured
Browser Worker. Enabling the cloud kernel sends workspace files and requested commands
to the configured Computer Worker. Calling `os_publish` uploads only the selected text
files to a public, sandboxed, `noindex` URL that expires after 30 days.

Hosted visitors use a random cookie-backed machine and short-lived capabilities; no name,
email, or provider identity is requested or stored. See [Self-hosting](docs/SELF_HOSTING.md),
[the analytics contract](docs/features/usage-analytics.md), and [Security](SECURITY.md) for
the complete boundaries and controls.

## Project guide

- [Agent manual](docs/agent-skills/README.md) — the manual shipped inside every machine.
- [WebMCP reference](docs/webmcp/REFERENCE.md) — the browser surface used by the project.
- [Feature specifications](docs/features/) — behavior and design decisions by feature.
- [Self-hosting](docs/SELF_HOSTING.md) — site, Worker, container, and publishing setup.
- [Contributing](CONTRIBUTING.md) · [Security](SECURITY.md) · [MIT license](LICENSE)

WebMCP Computer began as an entry for the
[OpenAI WebMCP Challenge](https://webmcp.devpost.com/) and continues as an open-source
showcase of what agent-native software can feel like.
Binary file added docs/assets/readme/hero.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/readme/webmcp-computer-demo.mp4
Binary file not shown.
Loading