diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0953e1a..b53595d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -23,7 +23,9 @@ bun run check # formatting, lint, types, tests bun run doctor # read-only check against the real engine ``` -`bun test` runs against a fake `codex app-server` (`test/fixtures/fake-codex.ts`) and never touches your apps. +`bun test` runs against a fake `codex app-server` (`test/fixtures/fake-codex.ts`) and a stand-in `ssh` +(`test/fixtures/fake-ssh/ssh`, which runs the remote command on your machine), and never touches your apps. To check a +Mac over SSH, run `bun scripts/smoke.ts --ssh my-mac`. ### Running the plugin from your checkout @@ -37,9 +39,10 @@ project's `plugins`. After editing, run `opencode api post /api/location/reload` | ------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | `src/index.ts` | Plugin entry: options, the `computer_use` tools, the doctor command, turn and session events | | `src/bridge.ts` | One `codex app-server` per plugin instance, one ephemeral Codex thread per OpenCode session, app-access prompts, turn metadata | -| `src/app-server.ts` | JSON-RPC client for `codex app-server --listen stdio://` | +| `src/app-server.ts` | JSON-RPC client for `codex app-server --listen stdio://`, started on the host; its last stderr lines explain an exit | +| `src/host.ts` | Where Codex runs: this machine, or another one over SSH; runs commands there and reports its platform and home folder | | `src/content.ts` | Converts Codex results to OpenCode content: screenshots, OCR, output cap | -| `src/ocr.ts` | On-device OCR with the macOS Vision framework | +| `src/ocr.ts` | On-device OCR with the macOS Vision framework, on the host, with the screenshot on stdin | | `src/surfaces.ts` | Turns native apps or browser tabs off inside the runtime | | `src/doctor.ts` | Setup checks shared by `/computer-use-doctor` and `scripts/smoke.ts` | diff --git a/README.md b/README.md index acc342c..d04cb5d 100644 --- a/README.md +++ b/README.md @@ -59,8 +59,13 @@ Nothing in Codex has to be configured by hand. You need a working Computer Use i The ChatGPT app does not need to be open while you use OpenCode. **Windows and Linux (experimental):** Codex Computer Use also runs on Windows, and its runtime has a Linux target. The -plugin itself is platform-neutral (it only talks to `codex app-server`), but it has only been tested on macOS. On -other platforms, `codex` is found through `PATH` or the `codexPath` option, and OCR is not available yet. +plugin itself is platform-neutral (it only talks to `codex app-server`), but it has only been tested with Computer Use +running on macOS. When Computer Use runs on another platform, `codex` is found through `PATH` or the `codexPath` +option, and OCR is not available there yet. These notes are about the machine that runs Computer Use; OpenCode itself +can run on another machine, as the next paragraph explains. + +**OpenCode on another machine:** OpenCode can also run somewhere else, such as a Linux VM, and operate the apps on a Mac +it reaches over SSH. See [Run Computer Use on another Mac (SSH)](#run-computer-use-on-another-mac-ssh). ## Install @@ -116,7 +121,8 @@ In OpenCode, run the **`/computer-use-doctor`** command. It checks each requirem connected browsers, and says how to fix whatever is missing. The report is added to the session without starting a model turn. -From a checkout, `bun run doctor` runs the same checks in a terminal (`bun scripts/smoke.ts --ocr` also tests OCR). +From a checkout, `bun run doctor` runs the same checks in a terminal (`bun scripts/smoke.ts --ocr` also tests OCR, +and `--ssh my-mac` checks a Mac over SSH). ``` ## Codex Computer Use doctor: ready @@ -161,7 +167,8 @@ Pass options with the object form of the plugin entry. Every option is optional; | `maxOutputKB` | `128` | Maximum text one call returns. Longer output is cut, with a note telling the model to narrow its query. `0` disables the limit. Images are not counted. | | `idleShutdownMinutes` | `0` (never) | Stop the background Codex process after this many minutes without Computer Use calls. The default keeps it running until OpenCode stops, so work can wait indefinitely for you (for example, a login in a tab the agent handed over). | | `callTimeoutSeconds` | `300` | Maximum time for one `computer_use` call. | -| `codexPath` | auto | Path to `codex`. Otherwise `$OPENCODE_CODEX_COMPUTER_USE_CODEX_PATH`, then (on macOS) the ChatGPT app, then the Codex app, then `PATH`. | +| `codexPath` | auto | Path to `codex`. Otherwise `$OPENCODE_CODEX_COMPUTER_USE_CODEX_PATH`, then (on macOS) the ChatGPT app, then the Codex app, then `PATH`. With `ssh`, a path on the Mac. | +| `ssh` | none | Run Computer Use on another Mac, reached over SSH: a destination such as `"my-mac"` (a host from `~/.ssh/config`) or `"me@my-mac"`. See [Run Computer Use on another Mac (SSH)](#run-computer-use-on-another-mac-ssh). | | `debug` | `false` | `true` (or a file path) writes every message exchanged with Codex to `~/.local/share/opencode/log/codex-computer-use.jsonl`, with screenshots replaced by their size. The log contains accessibility trees and page text, so it can hold sensitive data. | ### App access @@ -192,11 +199,69 @@ An `ask` rule is meant to prompt before every call, but OpenCode 2.0.16 does not plugins ([anomalyco/opencode#50652](https://github.com/anomalyco/opencode/issues/50652), fix in [#50657](https://github.com/anomalyco/opencode/pull/50657)); until that fix ships, `ask` behaves like `allow`. +## Run Computer Use on another Mac (SSH) + +Use this when OpenCode runs on one machine, such as a Linux VM, a server or a dev container, and the apps and Chrome +tabs the agent should operate are on a Mac you can reach over SSH. The plugin starts Codex on the Mac through `ssh` and +does everything else there as well: finding `codex`, OCR and the doctor's checks. + +You need: + +- **SSH with a key that works without any prompt.** On the OpenCode machine, `ssh my-mac true` must return without + asking for a password, a passphrase or a host key confirmation. The plugin never answers prompts: a connection that + needs one fails. If your key is in `ssh-agent`, OpenCode has to see the same agent (`SSH_AUTH_SOCK`). +- **A Mac that meets the normal [requirements](#requirements)**, logged in as the user you connect as, and awake. + Computer Use operates that user's desktop. + +Then set the option: + +```jsonc +{ + "plugins": [ + { + "package": "opencode-codex-computer-use", + "options": { "ssh": "my-mac" }, + }, + ], +} +``` + +`ssh` takes a destination the way the `ssh` command does: a host from `~/.ssh/config`, a host name, or `user@host`. +Ports, keys, jump hosts and other settings belong in `~/.ssh/config`. Run `/computer-use-doctor` to check the result. + +What changes: + +- **`codexPath` is a path on the Mac.** Without it, the plugin looks for the ChatGPT and Codex apps in the Mac's + `/Applications`, then for `codex` on the `PATH` an SSH command gets there. `$OPENCODE_CODEX_COMPUTER_USE_CODEX_PATH` + is still read on the OpenCode machine and names a path on the Mac. +- **Codex's working folder is your home folder on the Mac**, because the OpenCode project folder does not exist there. +- **OCR and the doctor's checks run on the Mac.** For OCR, each screenshot is sent back to the Mac over SSH. +- **The plugin adds these options to every `ssh` it runs:** + `-T -o BatchMode=yes -o ConnectTimeout=15 -o ServerAliveInterval=15 -o ServerAliveCountMax=4`. Everything else + comes from your SSH configuration. If the connection drops, the Codex process ends with it; the next call connects + again and tells the model its earlier variables are gone. +- **Optional speed-up:** OCR and the doctor open a new SSH connection each time. Sharing one connection makes them + faster: + + ``` + Host my-mac + ControlMaster auto + ControlPath ~/.ssh/cm-%C + ControlPersist 10m + ``` + +What stays different: + +- **The agent's shell and file tools still run on the OpenCode machine.** Files there are not visible on the Mac until + they are copied (for example with `scp`), and the tool description tells the model so. +- **Handoffs need someone at the Mac.** A login or CAPTCHA the agent hands over waits on the Mac's screen, so you + complete it there (or through screen sharing). + ## How it works ``` OpenCode ── computer_use tool - └─ codex app-server (hidden, started on first use, JSON-RPC over stdio) + └─ codex app-server (hidden, started on first use, JSON-RPC over stdio; through ssh when it runs on another Mac) └─ one ephemeral Codex thread per OpenCode session └─ cua_repl (Codex's Computer Use runtime) ── Codex Computer Use engine ── your apps and tabs ``` @@ -209,6 +274,9 @@ OpenCode ── computer_use tool as Codex does for its own tool calls); the browser surface requires it. When an OpenCode turn finishes or is interrupted, the plugin tells Computer Use the turn ended, which also cleans up agent-created Chrome tabs. Deleting an OpenCode session closes its Codex thread. +- **Another Mac:** with the `ssh` option, the plugin runs `ssh app-server --listen stdio://` + and speaks the same protocol over the connection. Nothing is installed on the Mac beyond what Computer Use already + needs. - **Lifetime:** the Codex process keeps running between calls, so JavaScript variables survive long waits. If it stops anyway (an optional `idleShutdownMinutes`, a crash, or a ChatGPT update), the next call starts it again and tells the model, before anything else, that its earlier variables are gone. @@ -218,20 +286,22 @@ OpenCode ── computer_use tool Computer Use reads the UI and screenshots of the apps and pages it operates, and those are sent to **your OpenCode model provider** as tool results. In Chrome it works in your real profile, with your logged-in sessions. OpenAI's Computer Use component also sends its own usage telemetry to OpenAI, as it does inside ChatGPT; your ChatGPT settings -govern that. With `screenshots: "ocr"` or `"off"`, no screenshot pixels leave your machine. +govern that. With `screenshots: "ocr"` or `"off"`, no screenshot pixels leave your machine; with the `ssh` option, +they only cross your SSH connection between the Mac and the OpenCode machine. ## Troubleshooting Start with `/computer-use-doctor`; it names the missing piece. -| Symptom | Fix | -| ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `Could not find the codex executable` | Install the ChatGPT desktop app in `/Applications`, or set `codexPath`. | -| `Codex has no cua_repl MCP server` | Turn on Computer Use in the ChatGPT/Codex app, then run `/computer-use-doctor` again. | -| Permission errors from the engine | Grant Accessibility and Screen Recording to _Codex Computer Use_, then retry. | -| The agent says an app needs permission | Approve the app permanently in ChatGPT/Codex, or set `approval_policy = "never"`; see [App access](#app-access). | -| No browsers listed / Chrome tab calls fail | Install and connect the ChatGPT for Chrome extension from the ChatGPT/Codex app, keep Chrome running, then run `/computer-use-doctor`. | -| Anything else | Plugin messages are prefixed `[codex-computer-use]` in `~/.local/share/opencode/log/opencode.log`; set `"debug": true` to log the full exchange with Codex. | +| Symptom | Fix | +| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `Could not find the codex executable` | Install the ChatGPT desktop app in `/Applications`, or set `codexPath`. | +| `Could not reach my-mac over SSH` | On the OpenCode machine, `ssh my-mac true` must succeed without a prompt; see [Run Computer Use on another Mac (SSH)](#run-computer-use-on-another-mac-ssh). | +| `Codex has no cua_repl MCP server` | Turn on Computer Use in the ChatGPT/Codex app, then run `/computer-use-doctor` again. | +| Permission errors from the engine | Grant Accessibility and Screen Recording to _Codex Computer Use_, then retry. | +| The agent says an app needs permission | Approve the app permanently in ChatGPT/Codex, or set `approval_policy = "never"`; see [App access](#app-access). | +| No browsers listed / Chrome tab calls fail | Install and connect the ChatGPT for Chrome extension from the ChatGPT/Codex app, keep Chrome running, then run `/computer-use-doctor`. | +| Anything else | Plugin messages are prefixed `[codex-computer-use]` in `~/.local/share/opencode/log/opencode.log`; set `"debug": true` to log the full exchange with Codex. | ## Contributing diff --git a/scripts/smoke.ts b/scripts/smoke.ts index fecc520..58fa416 100644 --- a/scripts/smoke.ts +++ b/scripts/smoke.ts @@ -1,17 +1,26 @@ #!/usr/bin/env bun // Read-only setup check, the same as OpenCode's /computer-use-doctor command. -// Usage: bun scripts/smoke.ts [path/to/codex] [--ocr] +// Usage: bun scripts/smoke.ts [path/to/codex] [--ocr] [--ssh ] import { ComputerUseBridge } from "../src/bridge" import { runDoctor } from "../src/doctor" +import { localHost, sshHost } from "../src/host" import { parseOptions } from "../src/options" const args = process.argv.slice(2) +const sshAt = args.indexOf("--ssh") +if (sshAt >= 0 && !args[sshAt + 1]) { + console.error("Usage: bun scripts/smoke.ts [path/to/codex] [--ocr] [--ssh ]") + process.exit(2) +} const options = parseOptions({ - codexPath: args.find((arg) => !arg.startsWith("--")), + codexPath: args.find((arg, index) => !arg.startsWith("--") && (sshAt < 0 || index !== sshAt + 1)), screenshots: args.includes("--ocr") ? "both" : "image", + ssh: sshAt >= 0 ? args[sshAt + 1] : undefined, }) +const host = options.ssh ? sshHost(options.ssh) : localHost() const bridge = new ComputerUseBridge({ ...options, + host, idleShutdownMs: 0, cwd: process.cwd(), version: "smoke", diff --git a/src/app-server.ts b/src/app-server.ts index 441317e..0eb45c5 100644 --- a/src/app-server.ts +++ b/src/app-server.ts @@ -1,8 +1,15 @@ -import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process" +import type { ChildProcessWithoutNullStreams } from "node:child_process" import { createInterface } from "node:readline" +import { spawnOnHost, type Host } from "./host" + type RequestID = number | string +/** Non-empty stderr lines kept for the error raised when the app-server exits. */ +const STDERR_TAIL_LINES = 3 +/** How long an exit waits for the rest of stderr, which a leftover process could keep open indefinitely. */ +const STDERR_DRAIN_MS = 500 + interface Pending { method: string resolve: (value: unknown) => void @@ -51,6 +58,11 @@ export interface RequestOptions { export class AppServerClient { private nextID = 1 private readonly pending = new Map() + private readonly stderrTail: string[] = [] + /** Stderr received after the last newline. */ + private stderrPartial = "" + /** Settles once an exit has been handled: `onExit` called and pending requests rejected. */ + private readonly exitHandled: Promise private exited = false private constructor( @@ -58,35 +70,48 @@ export class AppServerClient { private readonly handlers: AppServerHandlers, ) { createInterface({ input: child.stdout }).on("line", (line) => this.receive(line)) - createInterface({ input: child.stderr }).on("line", (line) => handlers.onStderr?.(line)) - child.on("exit", (code, signal) => { - this.exited = true - const error = new AppServerClosedError(`codex app-server exited (code ${code}, signal ${signal})`) - for (const pending of this.pending.values()) { - clearTimeout(pending.timer) - pending.reject(error) - } - this.pending.clear() - handlers.onExit?.(code, signal) + // Read stderr by hand rather than with readline: an unfinished last line must still reach the exit error when + // stderr stays open after the exit. + child.stderr.setEncoding("utf8") + child.stderr.on("data", (chunk: string) => { + const lines = (this.stderrPartial + chunk).split(/\r?\n/) + this.stderrPartial = lines.pop() ?? "" + for (const line of lines) this.stderrLine(line) }) + const stderrEnded = new Promise((resolve) => + child.stderr.once("close", () => { + if (this.stderrPartial) this.stderrLine(this.stderrPartial) + this.stderrPartial = "" + resolve() + }), + ) + this.exitHandled = new Promise((resolve) => + child.once("exit", (code, signal) => { + this.exited = true + // The owner hears about the exit at once, before it can start a replacement it might confuse with this one. + handlers.onExit?.(code, signal) + resolve(this.rejectPending(code, signal, stderrEnded)) + }), + ) child.stdin.on("error", () => {}) } static async start(input: { + host: Host codexPath: string clientVersion: string handlers: AppServerHandlers timeoutMs?: number }): Promise { - const child = spawn(input.codexPath, ["app-server", "--listen", "stdio://"], { - stdio: ["pipe", "pipe", "pipe"], - env: process.env, - }) + const child = spawnOnHost(input.host, input.codexPath, ["app-server", "--listen", "stdio://"]) const spawned = await new Promise((resolve) => { child.once("spawn", () => resolve(undefined)) child.once("error", (error) => resolve(error)) }) - if (spawned) throw new Error(`Could not start ${input.codexPath}: ${spawned.message}`) + if (spawned) { + const where = input.host.remote ? ` on ${input.host.label}` : "" + throw new Error(`Could not start ${input.codexPath}${where}: ${spawned.message}`) + } const client = new AppServerClient(child, input.handlers) try { @@ -152,11 +177,9 @@ export class AppServerClient { /** Closes stdin so the app-server shuts down cleanly; kills it if it does not exit in time. */ async close(timeoutMs = 5_000): Promise { - if (this.exited) return - const exited = new Promise((resolve) => this.child.once("exit", () => resolve())) - this.child.stdin.end() + if (!this.exited) this.child.stdin.end() const timer = setTimeout(() => this.kill(), timeoutMs) - await exited + await this.exitHandled clearTimeout(timer) } @@ -164,6 +187,32 @@ export class AppServerClient { if (!this.exited) this.child.kill("SIGTERM") } + private stderrLine(line: string): void { + if (line.trim()) this.stderrTail.push(line.trim()) + if (this.stderrTail.length > STDERR_TAIL_LINES) this.stderrTail.shift() + this.handlers.onStderr?.(line) + } + + private async rejectPending( + code: number | null, + signal: NodeJS.Signals | null, + stderrEnded: Promise, + ): Promise { + // The exit code alone does not say why; the last stderr lines usually do (for example an SSH error). They can + // still be on their way when the process has exited, so wait for the rest of stderr, but only briefly. + let timer: ReturnType | undefined + await Promise.race([stderrEnded, new Promise((resolve) => (timer = setTimeout(resolve, STDERR_DRAIN_MS)))]) + clearTimeout(timer) + const lines = [...this.stderrTail, this.stderrPartial.trim()].filter(Boolean).slice(-STDERR_TAIL_LINES) + const tail = lines.length ? `: ${lines.join("\n")}` : "" + const error = new AppServerClosedError(`codex app-server exited (code ${code}, signal ${signal})${tail}`) + for (const pending of this.pending.values()) { + clearTimeout(pending.timer) + pending.reject(error) + } + this.pending.clear() + } + private write(message: unknown): void { if (this.exited || !this.child.stdin.writable) return this.handlers.onTraffic?.("send", message) diff --git a/src/bridge.ts b/src/bridge.ts index f3e9b38..de26961 100644 --- a/src/bridge.ts +++ b/src/bridge.ts @@ -1,13 +1,18 @@ import { AppServerClient, AppServerError } from "./app-server" import { BUNDLED_CODEX_PATHS, CODEX_PATH_ENV, resolveCodexPath } from "./codex-path" +import type { Host } from "./host" /** Codex's MCP server that hosts the Computer Use JavaScript runtime (`cua`). */ export const CUA_SERVER = "cua_repl" export interface BridgeOptions { + /** The machine that runs Codex. */ + host: Host + /** Path to `codex` on the host. */ codexPath?: string idleShutdownMs: number callTimeoutMs: number + /** The OpenCode location directory; Codex's working folder when Codex runs on this machine. */ cwd: string version: string log: (message: string) => void @@ -55,9 +60,12 @@ interface SessionState { } export class ComputerUseBridge { + readonly host: Host private client?: AppServerClient private starting?: Promise private generation = 0 + /** Numbers every startup attempt, so a process from a failed attempt is never taken for a later one. */ + private startups = 0 private readonly sessions = new Map() private readonly approvalNotes = new Map() private readonly restartedSessions = new Set() @@ -65,7 +73,9 @@ export class ComputerUseBridge { private idleTimer?: ReturnType private disposed = false - constructor(private readonly options: BridgeOptions) {} + constructor(private readonly options: BridgeOptions) { + this.host = options.host + } async run( sessionID: string, @@ -200,20 +210,24 @@ export class ComputerUseBridge { async approvalPolicy(): Promise { return this.busy(async () => { const client = await this.ensureClient() - const response = await client.request("config/read", { cwd: this.options.cwd }, { timeoutMs: 15_000 }) + const cwd = await this.workingDirectory() + const response = await client.request("config/read", { cwd }, { timeoutMs: 15_000 }) const policy = response?.config?.approval_policy if (typeof policy === "string") return policy return policy && typeof policy === "object" ? "granular" : undefined }) } - get codexPath(): string | undefined { - return resolveCodexPath(this.options.codexPath) + /** Where `codex` is on the host, or undefined when it is not installed there. */ + codexPath(): Promise { + return resolveCodexPath(this.host, this.options.codexPath) } async dispose(): Promise { this.disposed = true clearTimeout(this.idleTimer) + // A startup in flight sees `disposed` and closes what it started; wait for it so no app-server outlives the plugin. + await this.starting?.catch(() => {}) await this.stop() } @@ -247,20 +261,24 @@ export class ComputerUseBridge { } private ensureClient(): Promise { - if (this.disposed) return Promise.reject(new Error("opencode-codex-computer-use has been unloaded")) + if (this.disposed) return Promise.reject(unloadedError()) if (this.client && !this.client.closed) return Promise.resolve(this.client) if (this.starting) return this.starting this.starting = (async () => { - const codexPath = resolveCodexPath(this.options.codexPath) + const codexPath = await this.codexPath() + // Finding codex can take an SSH round trip, long enough for the plugin to be unloaded meanwhile. + if (this.disposed) throw unloadedError() + const where = this.host.remote ? ` on ${this.host.label}` : "" if (!codexPath) { throw new SetupError( - `Could not find the \`codex\` executable (looked at the \`codexPath\` option, $${CODEX_PATH_ENV}, ` + - `${[...BUNDLED_CODEX_PATHS, "PATH"].join(", ")}). ${SETUP_HELP}`, + `Could not find the \`codex\` executable${where} (looked at the \`codexPath\` option, ` + + `$${CODEX_PATH_ENV}, ${[...BUNDLED_CODEX_PATHS, "PATH"].join(", ")}). ${SETUP_HELP}`, ) } - const generation = this.generation + 1 + const generation = ++this.startups const client = await AppServerClient.start({ + host: this.host, codexPath, clientVersion: this.options.version, handlers: { @@ -269,9 +287,13 @@ export class ComputerUseBridge { onTraffic: this.options.onTraffic, }, }) + if (this.disposed) { + await client.close() + throw unloadedError() + } this.generation = generation this.client = client - this.options.log(`started codex app-server (pid ${client.pid}) from ${codexPath}`) + this.options.log(`started codex app-server (pid ${client.pid}) from ${codexPath}${where}`) return client })().finally(() => { this.starting = undefined @@ -296,13 +318,14 @@ export class ComputerUseBridge { const existing = this.sessions.get(sessionID) if (existing && existing.generation === this.generation) return existing.threadID - // No approvalPolicy: app access follows the user's Codex approval_policy. - const params: Record = { ephemeral: true, cwd: this.options.cwd } - const threadID = client.request("thread/start", params, { timeoutMs: 60_000 }).then((response: any) => { - const id = response?.thread?.id - if (typeof id !== "string") throw new Error("thread/start returned no thread id") - return id - }) + const threadID = this.workingDirectory() + // No approvalPolicy: app access follows the user's Codex approval_policy. + .then((cwd) => client.request("thread/start", { ephemeral: true, cwd }, { timeoutMs: 60_000 })) + .then((response: any) => { + const id = response?.thread?.id + if (typeof id !== "string") throw new Error("thread/start returned no thread id") + return id + }) const state: SessionState = { generation: this.generation, threadID, dirty: false } this.sessions.set(sessionID, state) void (async () => { @@ -315,6 +338,11 @@ export class ComputerUseBridge { return threadID } + /** Codex's working folder must exist on the machine that runs it; its MCP servers fail to start otherwise. */ + private async workingDirectory(): Promise { + return this.host.remote ? (await this.host.info()).home : this.options.cwd + } + private async answerServerRequest(method: string, params: any): Promise { if (method !== "mcpServer/elicitation/request") { throw new Error(`${method} is not supported by opencode-codex-computer-use`) @@ -371,6 +399,10 @@ export class ComputerUseBridge { } } +function unloadedError(): Error { + return new Error("opencode-codex-computer-use has been unloaded") +} + function appName(params: any): string | undefined { const display = params?._meta?.tool_params_display const entry = Array.isArray(display) ? display.find((item: { name?: string }) => item?.name === "app") : undefined diff --git a/src/codex-path.ts b/src/codex-path.ts index cf4aafc..d6f695f 100644 --- a/src/codex-path.ts +++ b/src/codex-path.ts @@ -1,42 +1,21 @@ -import { accessSync, constants } from "node:fs" -import { delimiter, join } from "node:path" +import type { Host } from "./host" /** macOS app bundles first: their `codex` matches the Computer Use runtime shipped in the same bundle. */ -export const BUNDLED_CODEX_PATHS = - process.platform === "darwin" - ? ["/Applications/ChatGPT.app/Contents/Resources/codex", "/Applications/Codex.app/Contents/Resources/codex"] - : [] +export const BUNDLED_CODEX_PATHS: readonly string[] = [ + "/Applications/ChatGPT.app/Contents/Resources/codex", + "/Applications/Codex.app/Contents/Resources/codex", +] export const CODEX_PATH_ENV = "OPENCODE_CODEX_COMPUTER_USE_CODEX_PATH" -export function resolveCodexPath( +/** Order: explicit option, env[CODEX_PATH_ENV] (this process's env), BUNDLED_CODEX_PATHS when the host runs macOS, + * then `codex` on the host's PATH. Paths are paths on the host. */ +export async function resolveCodexPath( + host: Host, explicit?: string, - input: { env?: NodeJS.ProcessEnv; bundled?: readonly string[]; platform?: NodeJS.Platform } = {}, -): string | undefined { - const env = input.env ?? process.env - const platform = input.platform ?? process.platform - const candidates = [ - explicit, - env[CODEX_PATH_ENV], - ...(input.bundled ?? BUNDLED_CODEX_PATHS), - ...pathCandidates(env, platform), - ] - return candidates.find((candidate): candidate is string => !!candidate && isExecutable(candidate, platform)) -} - -function pathCandidates(env: NodeJS.ProcessEnv, platform: NodeJS.Platform): string[] { - const names = platform === "win32" ? ["codex.exe"] : ["codex"] - return (env.PATH ?? env.Path ?? "") - .split(delimiter) - .filter(Boolean) - .flatMap((directory) => names.map((name) => join(directory, name))) -} - -function isExecutable(path: string, platform: NodeJS.Platform): boolean { - try { - accessSync(path, platform === "win32" ? constants.F_OK : constants.X_OK) - return true - } catch { - return false - } + env: NodeJS.ProcessEnv = process.env, +): Promise { + const { platform } = await host.info() + const bundled = platform === "darwin" ? BUNDLED_CODEX_PATHS : [] + return host.findExecutable([explicit, env[CODEX_PATH_ENV], ...bundled], "codex") } diff --git a/src/content.ts b/src/content.ts index 1bc11f2..0732072 100644 --- a/src/content.ts +++ b/src/content.ts @@ -1,5 +1,5 @@ import type { McpContentBlock } from "./bridge" -import { formatOcr, recognizeText, type OcrResult } from "./ocr" +import { formatOcr, type OcrResult } from "./ocr" import type { ScreenshotMode } from "./options" export type ToolContent = @@ -10,6 +10,7 @@ export interface ContentOptions { screenshots?: ScreenshotMode /** Maximum bytes of text across all blocks; 0 or undefined disables the limit. Notes are never cut. */ maxOutputBytes?: number + /** Text recognition for `ocr` and `both`; it runs where Computer Use runs, so the caller provides it. */ ocr?: (image: Uint8Array) => Promise } @@ -40,7 +41,7 @@ export async function toToolContent( options: ContentOptions = {}, ): Promise { const mode = options.screenshots ?? "image" - const ocr = options.ocr ?? recognizeText + const ocr = options.ocr let images = 0 const pending = blocks.map(async (block): Promise => { if (block.type === "text" && typeof block.text === "string") return [{ type: "text", text: block.text }] @@ -60,6 +61,7 @@ export async function toToolContent( } if (mode === "ocr" || mode === "both") { try { + if (!ocr) throw new Error("no text recognizer is available") items.push({ type: "text", text: formatOcr(await ocr(Buffer.from(block.data, "base64")), index) }) } catch (error) { const reason = error instanceof Error ? error.message : String(error) diff --git a/src/doctor.ts b/src/doctor.ts index f46b3cf..9d0c9f6 100644 --- a/src/doctor.ts +++ b/src/doctor.ts @@ -1,10 +1,8 @@ -import { execFile } from "node:child_process" -import { existsSync } from "node:fs" -import { homedir } from "node:os" -import { join } from "node:path" +import { posix } from "node:path" import { CUA_SERVER, type ComputerUseBridge, type McpContentBlock } from "./bridge" import { imageMime, textOf } from "./content" +import { runOnHost, type Host, type HostInfo } from "./host" import { recognizeText } from "./ocr" import type { Options } from "./options" @@ -24,7 +22,8 @@ export interface DoctorReport { text: string } -/** Read-only checks of everything the plugin needs. Uses its own throwaway Codex thread. */ +/** Read-only checks of everything the plugin needs, on the machine that runs Codex. Uses its own throwaway Codex + * thread. */ export async function runDoctor( bridge: ComputerUseBridge, options: Options, @@ -33,21 +32,36 @@ export async function runDoctor( const checks: Check[] = [] const add = (name: string, status: Status, detail: string) => checks.push({ name, status, detail }) - add("Platform", ...(await platformCheck())) + const host = bridge.host + let info: HostInfo + try { + info = await host.info() + } catch (error) { + add("Platform", "fail", error instanceof Error ? error.message : String(error)) + return report(checks, options) + } + add("Platform", ...platformCheck(info, host)) - const codexPath = bridge.codexPath + let codexPath: string | undefined + try { + codexPath = await bridge.codexPath() + } catch (error) { + add("codex executable", "fail", error instanceof Error ? error.message : String(error)) + return report(checks, options) + } if (!codexPath) { add( "codex executable", "fail", - "Not found. Install the ChatGPT (or Codex) desktop app, or set the `codexPath` option.", + `Not found${host.remote ? ` on ${host.label}` : ""}. Install the ChatGPT (or Codex) desktop app, or set the ` + + "`codexPath` option.", ) return report(checks, options) } - const version = await run(codexPath, ["--version"]).catch(() => "unknown version") + const version = await runOnHost(host, codexPath, ["--version"]).catch(() => "unknown version") add("codex executable", "ok", `${codexPath} (${version.trim()})`) - if (process.platform === "darwin") add("Computer Use app", ...(await computerUseAppCheck())) + if (info.platform === "darwin") add("Computer Use app", ...(await computerUseAppCheck(host, info))) try { const names = await bridge.serverNames(sessionKey) @@ -73,7 +87,7 @@ export async function runDoctor( count ? `Engine reachable, ${count} apps listed` : firstLine(apps.content, "Could not list apps"), ) - if (process.platform === "darwin" && count) { + if (info.platform === "darwin" && count) { const shot = await probe( 'const __doctorFinder = await cua.getApp("com.apple.finder"); await __doctorFinder.getScreenshot({ emit: true }); nodeRepl.write("SHOT");', ).catch((error: unknown) => ({ @@ -89,7 +103,7 @@ export async function runDoctor( : `No screenshot: ${firstLine(shot.content, "unknown error")}. Check Screen Recording permission for Codex Computer Use.`, ) if (options.screenshots === "ocr" || options.screenshots === "both") { - add("OCR", ...(await ocrCheck(image?.data))) + add("OCR", ...(await ocrCheck(host, image?.data))) } } } else { @@ -146,46 +160,48 @@ export function approvalCheck(policy: string | undefined): [Status, string] { ] } -async function platformCheck(): Promise<[Status, string]> { - if (process.platform === "darwin") { - const version = (await run("/usr/bin/sw_vers", ["-productVersion"]).catch(() => "")).trim() +function platformCheck(info: HostInfo, host: Host): [Status, string] { + const on = (system: string) => (host.remote ? `${system} on ${host.label}` : system) + if (info.platform === "darwin") { + const version = info.osVersion ?? "" const [major = 0, minor = 0] = version.split(".").map(Number) const supported = major > 14 || (major === 14 && minor >= 4) - if (process.arch !== "arm64") - return ["fail", `macOS ${version} on ${process.arch}: Computer Use requires Apple Silicon`] + if (info.arch !== "arm64") + return ["fail", `${on(`macOS ${version} on ${info.arch}`)}: Computer Use requires Apple Silicon`] return supported - ? ["ok", `macOS ${version} (arm64)`] - : ["fail", `macOS ${version}: Computer Use requires 14.4 or later`] + ? ["ok", on(`macOS ${version} (arm64)`)] + : ["fail", `${on(`macOS ${version}`)}: Computer Use requires 14.4 or later`] } - if (process.platform === "win32" || process.platform === "linux") { + if (info.platform === "win32" || info.platform === "linux") { return [ "warn", - `${process.platform} (${process.arch}): supported by Codex Computer Use, experimental in this plugin`, + `${on(`${info.platform} (${info.arch})`)}: supported by Codex Computer Use, experimental in this plugin`, ] } - return ["fail", `${process.platform} is not supported by Codex Computer Use`] + return ["fail", `${on(info.platform)} is not supported by Codex Computer Use`] } -async function computerUseAppCheck(): Promise<[Status, string]> { - const codexHome = process.env.CODEX_HOME || join(homedir(), ".codex") - const app = join(codexHome, "computer-use", "Codex Computer Use.app") - if (!existsSync(app)) return ["fail", `Not installed at ${app}. Turn on Computer Use in the ChatGPT/Codex app.`] - const version = await run("/usr/bin/plutil", [ +async function computerUseAppCheck(host: Host, info: HostInfo): Promise<[Status, string]> { + const app = posix.join(info.codexHome, "computer-use", "Codex Computer Use.app") + if (!(await host.exists(app))) { + return ["fail", `Not installed at ${app}. Turn on Computer Use in the ChatGPT/Codex app.`] + } + const version = await runOnHost(host, "/usr/bin/plutil", [ "-extract", "CFBundleShortVersionString", "raw", - join(app, "Contents", "Info.plist"), + posix.join(app, "Contents", "Info.plist"), ]) .then((out) => out.trim()) .catch(() => "unknown version") return ["ok", `${app} (${version})`] } -async function ocrCheck(data: string | undefined): Promise<[Status, string]> { +async function ocrCheck(host: Host, data: string | undefined): Promise<[Status, string]> { if (!data) return ["skip", "No screenshot to test with"] try { const started = performance.now() - const result = await recognizeText(Buffer.from(data, "base64")) + const result = await recognizeText(host, Buffer.from(data, "base64")) return ["ok", `${result.lines.length} text lines recognized in ${Math.round(performance.now() - started)} ms`] } catch (error) { return ["fail", `On-device OCR failed: ${error instanceof Error ? error.message : String(error)}`] @@ -199,9 +215,3 @@ function firstLine(blocks: readonly McpContentBlock[], fallback: string): string .find((line) => line.trim() && !line.startsWith("#")) ?? fallback ).slice(0, 300) } - -function run(command: string, args: string[]): Promise { - return new Promise((resolve, reject) => - execFile(command, args, { timeout: 15_000 }, (error, stdout) => (error ? reject(error) : resolve(stdout))), - ) -} diff --git a/src/host.ts b/src/host.ts new file mode 100644 index 0000000..163c0d6 --- /dev/null +++ b/src/host.ts @@ -0,0 +1,244 @@ +import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process" +import { accessSync, constants, existsSync } from "node:fs" +import { homedir } from "node:os" +import { delimiter, join } from "node:path" + +/** Facts about the machine that runs Codex Computer Use. */ +export interface HostInfo { + /** Node-style platform: "darwin", "linux", "win32"; other systems: lowercased `uname -s`. */ + platform: string + /** Node-style architecture: "arm64" (arm64/aarch64), "x64" (x86_64/amd64); otherwise raw `uname -m`. */ + arch: string + /** Home directory on the host. */ + home: string + /** $CODEX_HOME on the host, or /.codex. */ + codexHome: string + /** `sw_vers -productVersion`, only on darwin (undefined if unavailable). */ + osVersion?: string +} + +export interface RunOptions { + /** Written to stdin, which is then closed. Without it stdin is closed immediately. */ + input?: Uint8Array | string + /** Default 15_000. On timeout the process is killed and the promise rejects. */ + timeoutMs?: number + /** Maximum stdout bytes, default 16 MiB; exceeding it kills the process and rejects. */ + maxBuffer?: number +} + +/** Where Codex Computer Use runs: this machine, or another one reached over SSH. */ +export interface Host { + /** True when commands run on another machine over SSH. */ + readonly remote: boolean + /** For messages: "this machine", or " over SSH". */ + readonly label: string + /** Program and arguments that run `command args` on the host. Local: unchanged. */ + command(command: string, args: readonly string[]): { file: string; args: string[] } + /** Cached after the first success; failures (e.g. SSH unreachable) are not cached. */ + info(): Promise + /** First candidate that is an executable file on the host (undefined/empty entries skipped); otherwise `name` + * looked up on the host's PATH; undefined when nothing is found. */ + findExecutable(candidates: readonly (string | undefined)[], name: string): Promise + /** Whether `path` exists on the host. Any failure reads as false. */ + exists(path: string): Promise +} + +/** + * Added to every SSH connection: no prompts (a password prompt would hang the plugin), a bounded connection time, + * and keep-alives so a dead connection ends the app-server instead of leaving calls waiting. + */ +export const SSH_OPTIONS: readonly string[] = [ + "-T", + "-o", + "BatchMode=yes", + "-o", + "ConnectTimeout=15", + "-o", + "ServerAliveInterval=15", + "-o", + "ServerAliveCountMax=4", +] + +const DEFAULT_MAX_BUFFER = 16 * 1024 * 1024 + +/** This machine. */ +export function localHost(): Host { + const host: Host = { + remote: false, + label: "this machine", + command: (command, args) => ({ file: command, args: [...args] }), + info: cacheSuccess(async () => { + const home = homedir() + return { + platform: process.platform, + arch: process.arch, + home, + codexHome: process.env.CODEX_HOME || join(home, ".codex"), + osVersion: + process.platform === "darwin" + ? await runOnHost(host, "/usr/bin/sw_vers", ["-productVersion"]).then( + (out) => out.trim() || undefined, + () => undefined, + ) + : undefined, + } + }), + findExecutable: async (candidates, name) => { + const onPath = (process.env.PATH ?? process.env.Path ?? "") + .split(delimiter) + .filter(Boolean) + .map((directory) => join(directory, process.platform === "win32" ? `${name}.exe` : name)) + return [...candidates, ...onPath].find((candidate): candidate is string => !!candidate && isExecutable(candidate)) + }, + exists: async (path) => existsSync(path), + } + return host +} + +// The remote side needs a POSIX `sh` (macOS or Linux). Values reach these scripts as positional arguments, never +// inside the script text. +const INFO_SCRIPT = + 'uname -s; uname -m; printf "%s\\n" "$HOME" "${CODEX_HOME:-$HOME/.codex}"; sw_vers -productVersion 2>/dev/null || echo' +const FIND_SCRIPT = + 'name=$1; shift; for p in "$@"; do if [ -f "$p" ] && [ -x "$p" ]; then printf "%s\\n" "$p"; exit 0; fi; done; ' + + 'command -v "$name" || true' + +/** Another machine, reached with the system `ssh` and the user's SSH configuration for `destination`. */ +export function sshHost(destination: string): Host { + // Neither script can fail on its own, so a failure means the machine could not be reached; users see these first, + // from a tool call or the doctor, so they name the machine. + const unreachable = (error: unknown) => + new Error(`Could not reach ${host.label}: ${error instanceof Error ? error.message : String(error)}`) + const host: Host = { + remote: true, + label: `${destination} over SSH`, + // ssh joins the remote command into one string for the remote login shell; quoting each word makes the round + // trip exact. + command: (command, args) => ({ + file: "ssh", + args: [...SSH_OPTIONS, "--", destination, [command, ...args].map(shellQuote).join(" ")], + }), + info: cacheSuccess(async () => { + // Usually the first contact with the machine: wait longer than ssh's ConnectTimeout so ssh's own reason gets + // through. + const output = await runOnHost(host, "sh", ["-c", INFO_SCRIPT], { timeoutMs: 30_000 }).catch((error: unknown) => { + throw unreachable(error) + }) + const [system = "", machine = "", home = "", codexHome = "", version = ""] = output.split("\n") + if (!system.trim() || !home) throw new Error(`unexpected system information: ${JSON.stringify(output)}`) + const platform = system.trim().toLowerCase() + return { + platform, + arch: normalizeArch(machine.trim()), + home, + codexHome, + osVersion: platform === "darwin" ? version.trim() || undefined : undefined, + } + }), + findExecutable: async (candidates, name) => { + const paths = candidates.filter((candidate): candidate is string => !!candidate) + const output = await runOnHost(host, "sh", ["-c", FIND_SCRIPT, "sh", name, ...paths]).catch((error: unknown) => { + throw unreachable(error) + }) + return output.trim() || undefined + }, + exists: (path) => + runOnHost(host, "test", ["-e", path]).then( + () => true, + () => false, + ), + } + return host +} + +/** Starts `command args` on the host with piped stdio (env: process.env). */ +export function spawnOnHost(host: Host, command: string, args: readonly string[]): ChildProcessWithoutNullStreams { + const target = host.command(command, args) + return spawn(target.file, target.args, { stdio: ["pipe", "pipe", "pipe"], env: process.env }) +} + +/** Runs `command args` on the host to completion; resolves with stdout (utf8). Rejects on spawn error, non-zero + * exit (message: trimmed stderr, else " exited with code N"), timeout, or maxBuffer. */ +export function runOnHost( + host: Host, + command: string, + args: readonly string[], + options: RunOptions = {}, +): Promise { + const timeoutMs = options.timeoutMs ?? 15_000 + const maxBuffer = options.maxBuffer ?? DEFAULT_MAX_BUFFER + return new Promise((resolve, reject) => { + const child = spawnOnHost(host, command, args) + const stdout: Buffer[] = [] + const stderr: Buffer[] = [] + let size = 0 + let settled = false + const settle = (error: Error | undefined, output = "") => { + if (settled) return + settled = true + clearTimeout(timer) + if (error) reject(error) + else resolve(output) + } + // Settles at once instead of waiting for "close": descendants of the process (or an ssh ProxyCommand) can keep + // its output open long after it is killed, and our ends of the pipes must not wait for them. + const abandon = (error: Error) => { + settle(error) + child.kill("SIGKILL") + child.stdin.destroy() + child.stdout.destroy() + child.stderr.destroy() + } + const timer = setTimeout(() => abandon(new Error(`${command} timed out after ${timeoutMs} ms`)), timeoutMs) + + child.stdout.on("data", (chunk: Buffer) => { + if (settled) return + size += chunk.length + if (size > maxBuffer) return abandon(new Error(`${command} wrote more than ${maxBuffer} bytes`)) + stdout.push(chunk) + }) + child.stderr.on("data", (chunk: Buffer) => stderr.push(chunk)) + child.once("error", (error) => settle(error)) + child.once("close", (code, signal) => { + if (code === 0) return settle(undefined, Buffer.concat(stdout).toString("utf8")) + const message = Buffer.concat(stderr).toString("utf8").trim() + const status = code === null ? `was stopped by ${signal}` : `exited with code ${code}` + settle(new Error(message || `${command} ${status}`)) + }) + // The process may exit without reading its input. + child.stdin.on("error", () => {}) + child.stdin.end(options.input) + }) +} + +/** POSIX single-quoting of one word: 'it'\''s'. The empty string becomes ''. */ +export function shellQuote(value: string): string { + return `'${value.replaceAll("'", "'\\''")}'` +} + +/** Caches a successful lookup; a failed one (for example while SSH is unreachable) is retried on the next call. */ +function cacheSuccess(load: () => Promise): () => Promise { + let cached: Promise | undefined + return () => { + cached ??= load().catch((error: unknown) => { + cached = undefined + throw error + }) + return cached + } +} + +function normalizeArch(machine: string): string { + if (machine === "arm64" || machine === "aarch64") return "arm64" + if (machine === "x86_64" || machine === "amd64") return "x64" + return machine +} + +function isExecutable(path: string): boolean { + try { + accessSync(path, process.platform === "win32" ? constants.F_OK : constants.X_OK) + return true + } catch { + return false + } +} diff --git a/src/index.ts b/src/index.ts index 3a22adf..9e5c7ab 100644 --- a/src/index.ts +++ b/src/index.ts @@ -4,6 +4,8 @@ import { ComputerUseBridge } from "./bridge" import { textOf, toToolContent } from "./content" import { trafficLogger } from "./debug" import { runDoctor } from "./doctor" +import { localHost, sshHost } from "./host" +import { recognizeText } from "./ocr" import { parseOptions, type Options } from "./options" import { surfaceGuard } from "./surfaces" @@ -23,6 +25,8 @@ export function describeTool(options: Options): string { const targets = [apps && "native desktop apps", browser && "Chrome tabs"].filter(Boolean).join(" and ") const lines = [ `Operate ${targets} through the Codex Computer Use engine installed with the ChatGPT/Codex desktop app.`, + options.ssh && + `The ${targets} are on ${options.ssh} (reached over SSH), a different machine from the one your shell and file tools use, so local files are not visible there until copied (for example with \`scp ${options.ssh}:\`).`, "", "Runs JavaScript in a persistent runtime (per OpenCode session) where a `cua` object is preloaded. Variables persist between calls.", apps && @@ -62,8 +66,10 @@ export default Plugin.define({ async setup(ctx) { const options = parseOptions(ctx.options) for (const warning of options.warnings) log(warning) + const host = options.ssh ? sshHost(options.ssh) : localHost() const bridge = new ComputerUseBridge({ ...options, + host, cwd: ctx.location.directory, version: VERSION, log, @@ -100,7 +106,10 @@ export default Plugin.define({ } const surface = result.meta?.["codex/toolSurface"] as { app?: { appId?: string } } | undefined return { - content: await toToolContent(result.content, result.notes, options), + content: await toToolContent(result.content, result.notes, { + ...options, + ocr: (image) => recognizeText(host, image), + }), metadata: { app: surface?.app?.appId }, } }, diff --git a/src/ocr.ts b/src/ocr.ts index 6c326bc..317494b 100644 --- a/src/ocr.ts +++ b/src/ocr.ts @@ -1,7 +1,4 @@ -import { execFile } from "node:child_process" -import { mkdtemp, rm, writeFile } from "node:fs/promises" -import { tmpdir } from "node:os" -import { join } from "node:path" +import { runOnHost, type Host } from "./host" export interface OcrLine { text: string @@ -16,11 +13,13 @@ export interface OcrResult { lines: OcrLine[] } -// On-device text recognition with macOS's Vision framework, run through JavaScript for Automation. -const SCRIPT = ` -ObjC.import("Vision"); ObjC.import("AppKit"); -function run(argv) { - const image = $.NSImage.alloc.initWithContentsOfURL($.NSURL.fileURLWithPath(argv[0])); +// On-device text recognition with macOS's Vision framework, run through JavaScript for Automation. The image +// arrives on stdin, so the same command works on this machine and over SSH without temporary files; osascript +// prints the JSON the script returns. +const SCRIPT = `ObjC.import("Vision"); ObjC.import("AppKit"); +(() => { + const data = $.NSFileHandle.fileHandleWithStandardInput.readDataToEndOfFile; + const image = $.NSImage.alloc.initWithData(data); const cg = image.CGImageForProposedRectContextHints(null, $(), $()); const w = Number($.CGImageGetWidth(cg)), h = Number($.CGImageGetHeight(cg)); const request = $.VNRecognizeTextRequest.alloc.init; @@ -39,29 +38,19 @@ function run(argv) { }); } return JSON.stringify({ width: w, height: h, lines }); -}` +})()` -export async function recognizeText(image: Uint8Array, timeoutMs = 30_000): Promise { - if (process.platform !== "darwin") { - throw new Error("on-device OCR is currently only available on macOS; use the accessibility tree") - } - const directory = await mkdtemp(join(tmpdir(), "codex-cu-ocr-")) - try { - const imagePath = join(directory, "screenshot") - const scriptPath = join(directory, "ocr.js") - await Promise.all([writeFile(imagePath, image), writeFile(scriptPath, SCRIPT)]) - const stdout = await new Promise((resolve, reject) => - execFile( - "/usr/bin/osascript", - ["-l", "JavaScript", scriptPath, imagePath], - { timeout: timeoutMs, maxBuffer: 16 * 1024 * 1024 }, - (error, out, stderr) => (error ? reject(new Error(stderr.trim() || error.message)) : resolve(out)), - ), - ) - return JSON.parse(stdout) as OcrResult - } finally { - await rm(directory, { recursive: true, force: true }) +/** Recognizes the text in a screenshot on the machine that runs Computer Use. */ +export async function recognizeText(host: Host, image: Uint8Array, timeoutMs = 30_000): Promise { + if ((await host.info()).platform !== "darwin") { + throw new Error("on-device OCR needs macOS on the Computer Use machine; use the accessibility tree") } + const stdout = await runOnHost(host, "/usr/bin/osascript", ["-l", "JavaScript", "-e", SCRIPT], { + input: image, + timeoutMs, + maxBuffer: 16 * 1024 * 1024, + }) + return JSON.parse(stdout) as OcrResult } /** Reading order: top to bottom, then left to right within roughly the same row. */ diff --git a/src/options.ts b/src/options.ts index 6267926..5f6fa3c 100644 --- a/src/options.ts +++ b/src/options.ts @@ -11,6 +11,8 @@ export type Surface = (typeof SURFACES)[number] export interface Options { codexPath?: string + /** SSH destination of the machine that runs Codex Computer Use; undefined runs it on this machine. */ + ssh?: string idleShutdownMs: number callTimeoutMs: number /** Maximum text returned by one call, in bytes. 0 disables the limit. */ @@ -39,6 +41,7 @@ export function parseOptions(raw: PluginOptions): Options { } return { codexPath: typeof raw.codexPath === "string" && raw.codexPath.trim() ? raw.codexPath.trim() : undefined, + ssh: sshDestination(raw.ssh), idleShutdownMs: nonNegative(raw.idleShutdownMinutes, 0) * 60_000, callTimeoutMs: positive(raw.callTimeoutSeconds, 300) * 1_000, maxOutputBytes: Math.round(nonNegative(raw.maxOutputKB, 128) * 1024), @@ -75,6 +78,18 @@ function surfaces(value: unknown): readonly Surface[] { return SURFACES.filter((surface) => list.includes(surface)) } +// SSH settings belong in ~/.ssh/config, so a value that looks like an ssh option or holds several words is a mistake. +function sshDestination(value: unknown): string | undefined { + if (typeof value !== "string" || !value.trim()) return undefined + const destination = value.trim() + if (destination.startsWith("-") || /\s/.test(destination)) { + throw new Error( + `[codex-computer-use] invalid option ssh=${JSON.stringify(value)}; use an SSH destination such as "my-mac" or "me@my-mac"`, + ) + } + return destination +} + function debugLog(value: unknown): string | undefined { if (value === true) return DEFAULT_DEBUG_LOG if (typeof value === "string" && value.trim()) { diff --git a/test/bridge.test.ts b/test/bridge.test.ts index bcf1079..1be7fdf 100644 --- a/test/bridge.test.ts +++ b/test/bridge.test.ts @@ -1,21 +1,52 @@ import { afterEach, describe, expect, test } from "bun:test" import { mkdtempSync, readFileSync, existsSync } from "node:fs" import { tmpdir } from "node:os" -import { join } from "node:path" +import { delimiter, join } from "node:path" +import { AppServerClient } from "../src/app-server" import { ComputerUseBridge, SetupError } from "../src/bridge" +import { localHost, sshHost, type Host } from "../src/host" const FAKE_CODEX = join(import.meta.dir, "fixtures", "fake-codex") +const FAKE_SSH_DIR = join(import.meta.dir, "fixtures", "fake-ssh") +const PATH = process.env.PATH const bridges: ComputerUseBridge[] = [] afterEach(async () => { await Promise.all(bridges.splice(0).map((bridge) => bridge.dispose())) delete process.env.FAKE_CODEX_NO_CUA delete process.env.FAKE_CODEX_POLICY + delete process.env.FAKE_SSH_HOME + delete process.env.FAKE_CODEX_INIT_ERROR_ONCE + process.env.PATH = PATH +}) + +describe("AppServerClient", () => { + test("tells its owner about an exit at once, while pending calls wait for the rest of stderr", async () => { + const at: Record = {} + const client = await AppServerClient.start({ + host: localHost(), + codexPath: FAKE_CODEX, + clientVersion: "test", + handlers: { onServerRequest: async () => ({}), onExit: () => (at.exit = performance.now()) }, + }) + const started = await client.request("thread/start", { ephemeral: true, cwd: "/tmp" }) + await client + .request("mcpServer/tool/call", { + threadId: started.thread.id, + server: "cua_repl", + tool: "js", + arguments: { code: "EXIT_HELD" }, + }) + .catch(() => (at.rejected = performance.now())) + // EXIT_HELD leaves stderr open, so the rejection waits out the drain; the exit must not. + expect(at.rejected! - at.exit!).toBeGreaterThan(300) + }) }) function setup( input: { + host?: Host idleShutdownMs?: number codePrefix?: string onTraffic?: (direction: "send" | "receive", message: unknown) => void @@ -25,6 +56,7 @@ function setup( process.env.FAKE_CODEX_LOG = logFile const logs: string[] = [] const bridge = new ComputerUseBridge({ + host: input.host ?? localHost(), codexPath: FAKE_CODEX, idleShutdownMs: input.idleShutdownMs ?? 0, callTimeoutMs: 10_000, @@ -188,6 +220,49 @@ describe("ComputerUseBridge", () => { expect((await bridge.run("ses_a", "app")).notes).toEqual([]) }) + test("says why the app-server exited, from its last stderr lines", async () => { + const { bridge } = setup() + await expect(bridge.run("ses_a", "EXIT")).rejects.toThrow( + "codex app-server exited (code 1, signal null): fake-codex: stopping on request", + ) + }) + + test("includes stderr output that arrives after the exit, even without a final newline", async () => { + const { bridge } = setup() + await expect(bridge.run("ses_a", "EXIT_LATE")).rejects.toThrow( + "codex app-server exited (code 1, signal null): fake-codex: late reason", + ) + }) + + test("keeps an unfinished stderr line even when stderr stays open", async () => { + const { bridge } = setup() + await expect(bridge.run("ses_a", "EXIT_PARTIAL_HELD")).rejects.toThrow( + "codex app-server exited (code 1, signal null): fake-codex: unfinished reason", + ) + }) + + test("a process from a failed startup never takes the place of the next one", async () => { + process.env.FAKE_CODEX_INIT_ERROR_ONCE = join(mkdtempSync(join(tmpdir(), "cua-init-")), "failed-once") + const { bridge, calls } = setup() + await expect(bridge.run("ses_a", "1 + 1")).rejects.toThrow("initialize failed") + expect(text(await bridge.run("ses_a", "2 + 2"))).toBe("ran on thread-1: 2 + 2") + + // The first process exits a second after it was told to stop, long after the second one took over. + await Bun.sleep(1_500) + const again = await bridge.run("ses_a", "3 + 3") + expect(text(again)).toBe("ran on thread-1: 3 + 3") + expect(again.notes).toEqual([]) + expect(calls("initialize")).toHaveLength(2) + }) + + test("does not wait for a leftover process that keeps stderr open", async () => { + const { bridge } = setup() + await bridge.run("ses_a", "warm up") + const started = performance.now() + await expect(bridge.run("ses_a", "EXIT_HELD")).rejects.toThrow("fake-codex: stopping, stderr stays open") + expect(performance.now() - started).toBeLessThan(1_500) + }) + test("cancelling a call rejects it and interrupts the turn", async () => { const { bridge, calls } = setup() await bridge.run("ses_a", "warm up") @@ -208,6 +283,71 @@ describe("ComputerUseBridge", () => { expect(error.message).toContain("no `cua_repl` MCP server") }) + test("runs Codex on a remote host, in that host's home folder", async () => { + const home = mkdtempSync(join(tmpdir(), "cu remote home ")) + process.env.PATH = `${FAKE_SSH_DIR}${delimiter}${PATH}` + process.env.FAKE_SSH_HOME = home + const { bridge, calls, logs } = setup({ host: sshHost("my-mac") }) + + expect(await bridge.codexPath()).toBe(FAKE_CODEX) + expect(text(await bridge.run("ses_a", "1 + 1"))).toBe("ran on thread-1: 1 + 1") + expect(calls("thread/start")[0].params).toEqual({ ephemeral: true, cwd: home }) + expect(await bridge.approvalPolicy()).toBeUndefined() + expect(calls("config/read")[0].params).toEqual({ cwd: home }) + expect(logs.some((line) => line.endsWith(`from ${FAKE_CODEX} on my-mac over SSH`))).toBe(true) + }) + + test("a call names the remote host it cannot reach", async () => { + process.env.PATH = FAKE_SSH_DIR + const { bridge } = setup({ host: sshHost("my-mac") }) + await expect(bridge.run("ses_a", "1 + 1")).rejects.toThrow(/^Could not reach my-mac over SSH: /) + }) + + test("does not start Codex when the plugin is unloaded while it looks for codex", async () => { + let lookedUp!: () => void + let release!: () => void + const lookingUp = new Promise((resolve) => (lookedUp = resolve)) + const released = new Promise((resolve) => (release = resolve)) + const local = localHost() + const host: Host = { + ...local, + findExecutable: async (candidates, name) => { + lookedUp() + await released + return local.findExecutable(candidates, name) + }, + } + const { bridge, calls } = setup({ host }) + + const call = bridge.run("ses_a", "1 + 1") + await lookingUp + const disposed = bridge.dispose() + release() + await expect(call).rejects.toThrow("opencode-codex-computer-use has been unloaded") + await disposed + expect(calls("initialize")).toHaveLength(0) + }) + + test("closes an app-server that finished starting after the plugin was unloaded", async () => { + let disposed: Promise | undefined + const local = localHost() + const host: Host = { + ...local, + // Unloads the plugin just as the app-server is being started. + command: (command, args) => { + disposed ??= bridge.dispose() + return local.command(command, args) + }, + } + const { bridge, calls, logs } = setup({ host }) + + await expect(bridge.run("ses_a", "1 + 1")).rejects.toThrow("opencode-codex-computer-use has been unloaded") + await disposed + expect(calls("initialize")).toHaveLength(1) + expect(calls("thread/start")).toHaveLength(0) + expect(logs.some((line) => line.startsWith("codex app-server exited"))).toBe(true) + }) + test("stops the app-server after the idle timeout", async () => { const { bridge, calls, logs } = setup({ idleShutdownMs: 200 }) await bridge.run("ses_a", "x") diff --git a/test/codex-path.test.ts b/test/codex-path.test.ts index a9d323b..07c3ef4 100644 --- a/test/codex-path.test.ts +++ b/test/codex-path.test.ts @@ -1,31 +1,65 @@ -import { describe, expect, test } from "bun:test" +import { afterEach, describe, expect, test } from "bun:test" import { join } from "node:path" -import { CODEX_PATH_ENV, resolveCodexPath } from "../src/codex-path" +import { BUNDLED_CODEX_PATHS, CODEX_PATH_ENV, resolveCodexPath } from "../src/codex-path" +import { localHost, type Host } from "../src/host" const FAKE_CODEX = join(import.meta.dir, "fixtures", "fake-codex") const FIXTURES = join(import.meta.dir, "fixtures") +const PATH = process.env.PATH + +afterEach(() => { + process.env.PATH = PATH +}) + +/** This machine, reporting another platform, so the app bundles installed here do not decide the result. */ +function hostOn(platform: string): Host { + const local = localHost() + return { ...local, info: async () => ({ ...(await local.info()), platform }) } +} + +/** What resolveCodexPath asks the host to look for. */ +async function searched(platform: string) { + let request: { candidates: readonly (string | undefined)[]; name: string } | undefined + const host: Host = { + ...hostOn(platform), + findExecutable: async (candidates, name) => { + request = { candidates, name } + return undefined + }, + } + await resolveCodexPath(host, "/x/codex", { [CODEX_PATH_ENV]: "/env/codex" }) + return request +} describe("resolveCodexPath", () => { - test("prefers the explicit option", () => { - expect(resolveCodexPath(FAKE_CODEX, { env: {}, bundled: [] })).toBe(FAKE_CODEX) + test("prefers the explicit option", async () => { + expect(await resolveCodexPath(localHost(), FAKE_CODEX, {})).toBe(FAKE_CODEX) + }) + + test("uses the environment variable next", async () => { + expect(await resolveCodexPath(localHost(), undefined, { [CODEX_PATH_ENV]: FAKE_CODEX })).toBe(FAKE_CODEX) }) - test("uses the environment variable next", () => { - expect(resolveCodexPath(undefined, { env: { [CODEX_PATH_ENV]: FAKE_CODEX }, bundled: [] })).toBe(FAKE_CODEX) + test("skips missing candidates", async () => { + expect(await resolveCodexPath(localHost(), "/nope/codex", { [CODEX_PATH_ENV]: FAKE_CODEX })).toBe(FAKE_CODEX) }) - test("skips missing candidates and prefers app bundles over PATH", () => { - expect(resolveCodexPath("/nope/codex", { env: { PATH: "/nope" }, bundled: ["/missing", FAKE_CODEX] })).toBe( - FAKE_CODEX, - ) + test("considers the app bundles, before PATH, only when the host runs macOS", async () => { + expect(await searched("darwin")).toEqual({ + candidates: ["/x/codex", "/env/codex", ...BUNDLED_CODEX_PATHS], + name: "codex", + }) + expect(await searched("linux")).toEqual({ candidates: ["/x/codex", "/env/codex"], name: "codex" }) }) - test("only finds an executable named codex on PATH", () => { - expect(resolveCodexPath(undefined, { env: { PATH: FIXTURES }, bundled: [] })).toBeUndefined() + test("only finds an executable named codex on PATH", async () => { + process.env.PATH = FIXTURES + expect(await resolveCodexPath(hostOn("linux"), undefined, {})).toBeUndefined() }) - test("returns undefined when nothing is installed", () => { - expect(resolveCodexPath(undefined, { env: { PATH: "/nope" }, bundled: [] })).toBeUndefined() + test("returns undefined when nothing is installed", async () => { + process.env.PATH = "/nope" + expect(await resolveCodexPath(hostOn("linux"), undefined, {})).toBeUndefined() }) }) diff --git a/test/content.test.ts b/test/content.test.ts index e0d9800..bae4825 100644 --- a/test/content.test.ts +++ b/test/content.test.ts @@ -1,7 +1,8 @@ import { describe, expect, test } from "bun:test" import { imageMime, textOf, toToolContent } from "../src/content" -import { formatOcr, type OcrResult } from "../src/ocr" +import { localHost, type Host } from "../src/host" +import { formatOcr, recognizeText, type OcrResult } from "../src/ocr" const IMAGE = { type: "image", data: "/9j/4AAQSkZJRg==", mimeType: "image/png" } const fakeOcr = async (): Promise => ({ @@ -53,6 +54,13 @@ describe("toToolContent", () => { expect((content[0] as { text: string }).text).toContain("text recognition failed (no vision)") }) + test("OCR modes without a text recognizer report the failure per screenshot", async () => { + const content = await toToolContent([IMAGE, IMAGE], [], { screenshots: "both" }) + expect(content.map((item) => item.type)).toEqual(["file", "text", "file", "text"]) + expect((content[1] as { text: string }).text).toStartWith("Screenshot 1: text recognition failed (") + expect((content[3] as { text: string }).text).toStartWith("Screenshot 2: text recognition failed (") + }) + test("caps text output and explains how to narrow it; notes and images are kept", async () => { const big = "x".repeat(3000) const content = await toToolContent( @@ -100,6 +108,16 @@ describe("toToolContent", () => { }) }) +describe("recognizeText", () => { + test("needs macOS on the Computer Use machine", async () => { + const local = localHost() + const linux: Host = { ...local, info: async () => ({ ...(await local.info()), platform: "linux" }) } + await expect(recognizeText(linux, new Uint8Array())).rejects.toThrow( + "on-device OCR needs macOS on the Computer Use machine; use the accessibility tree", + ) + }) +}) + describe("formatOcr", () => { test("reports when nothing was recognized", () => { expect(formatOcr({ width: 10, height: 10, lines: [] }, 2)).toContain("(no text recognized)") diff --git a/test/fixtures/fake-codex.ts b/test/fixtures/fake-codex.ts index 1f12431..d8df815 100644 --- a/test/fixtures/fake-codex.ts +++ b/test/fixtures/fake-codex.ts @@ -1,6 +1,7 @@ #!/usr/bin/env bun // Fake `codex app-server --listen stdio://` for tests. Appends every received message to $FAKE_CODEX_LOG. -import { appendFileSync } from "node:fs" +import { spawn } from "node:child_process" +import { appendFileSync, existsSync, writeFileSync } from "node:fs" import { createInterface } from "node:readline" const log = process.env.FAKE_CODEX_LOG @@ -24,8 +25,16 @@ createInterface({ input: process.stdin }).on("line", async (line) => { } const { id, method, params } = message switch (method) { - case "initialize": + case "initialize": { + // The first process fails to initialize and takes a second to exit when told to stop. + const marker = process.env.FAKE_CODEX_INIT_ERROR_ONCE + if (marker && !existsSync(marker)) { + writeFileSync(marker, "") + process.on("SIGTERM", () => setTimeout(() => process.exit(0), 1_000)) + return send({ id, error: { code: -32000, message: "not ready" } }) + } return send({ id, result: { userAgent: "fake" } }) + } case "initialized": return case "thread/start": @@ -55,7 +64,27 @@ async function toolCall(id: number, params: any) { } if (params.tool !== "js") return send({ id, result: { content: [{ type: "text", text: "{}" }] } }) const code: string = params.arguments.code - if (code.includes("EXIT")) process.exit(1) + if (code.includes("EXIT_LATE")) { + // A leftover process writes the reason after the exit, without a final newline. + spawn("sh", ["-c", "sleep 0.2; printf 'fake-codex: late reason' >&2"], { stdio: ["ignore", "ignore", "inherit"] }) + process.exit(1) + } + if (code.includes("EXIT_HELD")) { + // A leftover process keeps stderr open long after the exit. + spawn("sleep", ["3"], { stdio: ["ignore", "ignore", "inherit"] }) + process.stderr.write("fake-codex: stopping, stderr stays open\n", () => process.exit(1)) + return + } + if (code.includes("EXIT_PARTIAL_HELD")) { + // An unfinished line, then an exit while a leftover process keeps stderr open. + spawn("sleep", ["3"], { stdio: ["ignore", "ignore", "inherit"] }) + process.stderr.write("fake-codex: unfinished reason", () => process.exit(1)) + return + } + if (code.includes("EXIT")) { + process.stderr.write("fake-codex: stopping on request\n", () => process.exit(1)) + return + } if (code.includes("BROWSER")) { const meta = params._meta?.["x-codex-turn-metadata"] if (!meta?.session_id || !meta?.turn_id) { diff --git a/test/fixtures/fake-ssh/ssh b/test/fixtures/fake-ssh/ssh new file mode 100755 index 0000000..1c98307 --- /dev/null +++ b/test/fixtures/fake-ssh/ssh @@ -0,0 +1,10 @@ +#!/bin/sh +# Stand-in for ssh in tests: runs the remote command on this machine through sh, the way sshd hands the joined +# command string to the remote login shell, so the plugin's quoting is exercised for real. +# $FAKE_SSH_HOME plays the remote home directory. +while [ "$#" -gt 0 ] && [ "$1" != "--" ]; do shift; done +shift 2 # "--" and the destination +command="$*" +HOME="${FAKE_SSH_HOME:-$HOME}" +export HOME +exec sh -c "$command" diff --git a/test/host.test.ts b/test/host.test.ts new file mode 100644 index 0000000..614dc27 --- /dev/null +++ b/test/host.test.ts @@ -0,0 +1,220 @@ +import { afterEach, beforeAll, describe, expect, test } from "bun:test" +import { chmodSync, mkdtempSync, writeFileSync } from "node:fs" +import { tmpdir } from "node:os" +import { delimiter, join } from "node:path" + +import { localHost, runOnHost, shellQuote, sshHost } from "../src/host" + +const FIXTURES = join(import.meta.dir, "fixtures") +const FAKE_CODEX = join(FIXTURES, "fake-codex") +const FAKE_SSH_DIR = join(FIXTURES, "fake-ssh") +const PATH = process.env.PATH + +afterEach(() => { + process.env.PATH = PATH + delete process.env.FAKE_SSH_HOME +}) + +/** An executable file whose directory and name need quoting. */ +function awkwardExecutable(): string { + const path = join(mkdtempSync(join(tmpdir(), "cu host ")), "it's codex") + writeFileSync(path, "#!/bin/sh\n") + chmodSync(path, 0o755) + return path +} + +describe("shellQuote", () => { + test("quotes every word so the remote shell reads it back unchanged", () => { + expect(shellQuote("codex")).toBe("'codex'") + expect(shellQuote("a b")).toBe("'a b'") + expect(shellQuote("it's")).toBe("'it'\\''s'") + expect(shellQuote("")).toBe("''") + }) +}) + +describe("command", () => { + test("an SSH host runs the quoted command through ssh with fixed options", () => { + const host = sshHost("me@my-mac") + expect(host.remote).toBe(true) + expect(host.label).toBe("me@my-mac over SSH") + expect( + host.command("/Applications/ChatGPT.app/Contents/Resources/codex", ["app-server", "--listen", "stdio://"]), + ).toEqual({ + file: "ssh", + args: [ + "-T", + "-o", + "BatchMode=yes", + "-o", + "ConnectTimeout=15", + "-o", + "ServerAliveInterval=15", + "-o", + "ServerAliveCountMax=4", + "--", + "me@my-mac", + "'/Applications/ChatGPT.app/Contents/Resources/codex' 'app-server' '--listen' 'stdio://'", + ], + }) + }) + + test("this machine runs the command as is", () => { + const host = localHost() + expect(host.remote).toBe(false) + expect(host.label).toBe("this machine") + expect(host.command("/usr/bin/plutil", ["-extract", "a b"])).toEqual({ + file: "/usr/bin/plutil", + args: ["-extract", "a b"], + }) + }) +}) + +describe("runOnHost", () => { + test("writes the input to stdin and returns stdout", async () => { + expect(await runOnHost(localHost(), "cat", [], { input: "hello" })).toBe("hello") + expect(await runOnHost(localHost(), "cat", [], { input: new Uint8Array([104, 105]) })).toBe("hi") + expect(await runOnHost(localHost(), "cat", [])).toBe("") + }) + + test("rejects with stderr, or the exit code when stderr is empty", async () => { + await expect(runOnHost(localHost(), "sh", ["-c", "echo ' went wrong ' >&2; exit 3"])).rejects.toThrow( + /^went wrong$/, + ) + await expect(runOnHost(localHost(), "sh", ["-c", "exit 4"])).rejects.toThrow("sh exited with code 4") + }) + + test("rejects when the program cannot start", async () => { + await expect(runOnHost(localHost(), join(FIXTURES, "missing"), [])).rejects.toThrow() + }) + + test("kills the process on timeout", async () => { + const started = performance.now() + await expect(runOnHost(localHost(), "sleep", ["5"], { timeoutMs: 100 })).rejects.toThrow( + "sleep timed out after 100 ms", + ) + expect(performance.now() - started).toBeLessThan(2_000) + }) + + test("gives up at once even when leftover processes keep the output open", async () => { + const started = performance.now() + await expect(runOnHost(localHost(), "/bin/sh", ["-c", "sleep 5 & sleep 5"], { timeoutMs: 100 })).rejects.toThrow( + "/bin/sh timed out after 100 ms", + ) + await expect( + runOnHost(localHost(), "/bin/sh", ["-c", "sleep 5 & head -c 100000 /dev/zero"], { + timeoutMs: 5_000, + maxBuffer: 1_000, + }), + ).rejects.toThrow("/bin/sh wrote more than 1000 bytes") + expect(performance.now() - started).toBeLessThan(1_000) + }) + + test("kills the process when stdout exceeds maxBuffer", async () => { + await expect(runOnHost(localHost(), "sh", ["-c", "yes | head -c 100000"], { maxBuffer: 1_000 })).rejects.toThrow( + "sh wrote more than 1000 bytes", + ) + }) +}) + +describe("localHost", () => { + test("describes this machine", async () => { + const info = await localHost().info() + expect(info).toMatchObject({ platform: process.platform, arch: process.arch }) + expect(info.codexHome).toBe(process.env.CODEX_HOME || join(info.home, ".codex")) + expect(info.osVersion === undefined).toBe(process.platform !== "darwin") + }) + + test("finds the first executable candidate, then the name on PATH", async () => { + const host = localHost() + process.env.PATH = FIXTURES + expect(await host.findExecutable([undefined, "", "/nope/codex", FAKE_CODEX], "codex")).toBe(FAKE_CODEX) + expect(await host.findExecutable([join(FIXTURES, "fake-codex.ts")], "fake-codex")).toBe(FAKE_CODEX) + expect(await host.findExecutable([], "codex")).toBeUndefined() + process.env.PATH = "/nope" + expect(await host.findExecutable(["/nope/codex"], "fake-codex")).toBeUndefined() + }) + + test("checks whether a path exists", async () => { + expect(await localHost().exists(FAKE_CODEX)).toBe(true) + expect(await localHost().exists(join(FIXTURES, "missing"))).toBe(false) + }) +}) + +describe("sshHost through a stand-in ssh", () => { + let home: string + + beforeAll(() => { + home = mkdtempSync(join(tmpdir(), "cu remote home ")) + }) + + const useFakeSsh = () => { + process.env.PATH = `${FAKE_SSH_DIR}${delimiter}${PATH}` + process.env.FAKE_SSH_HOME = home + } + + test("passes every argument through the remote shell unchanged", async () => { + useFakeSsh() + const args = ["a b", "it's", "$HOME", "", "two\nlines", "`id` $(id)", '"q"', "back\\slash", "*"] + expect(await runOnHost(sshHost("my-mac"), "printf", ["%s|", ...args])).toBe(args.map((arg) => `${arg}|`).join("")) + }) + + test("sends the input and reports remote failures", async () => { + useFakeSsh() + expect(await runOnHost(sshHost("my-mac"), "cat", [], { input: "over ssh" })).toBe("over ssh") + await expect(runOnHost(sshHost("my-mac"), "sh", ["-c", "echo nope >&2; exit 1"])).rejects.toThrow("nope") + }) + + test("describes the remote machine", async () => { + useFakeSsh() + const local = await localHost().info() + expect(await sshHost("my-mac").info()).toEqual({ + ...local, + home, + codexHome: process.env.CODEX_HOME || join(home, ".codex"), + }) + }) + + test("names the machine it could not reach, with the reason", async () => { + // ssh starts, but the remote side fails: the stand-in cannot find sh on this PATH. + process.env.PATH = FAKE_SSH_DIR + await expect(sshHost("my-mac").info()).rejects.toThrow(/^Could not reach my-mac over SSH: .*not found/) + }) + + test("names the machine when it stops answering after the first contact", async () => { + useFakeSsh() + const host = sshHost("my-mac") + await host.info() + process.env.PATH = FAKE_SSH_DIR + await expect(host.findExecutable([FAKE_CODEX], "codex")).rejects.toThrow( + /^Could not reach my-mac over SSH: .*not found/, + ) + }) + + test("caches the description once it succeeds, and only then", async () => { + const host = sshHost("my-mac") + process.env.PATH = join(FIXTURES, "missing") + await expect(host.info()).rejects.toThrow(/^Could not reach my-mac over SSH: /) + useFakeSsh() + expect((await host.info()).home).toBe(home) + process.env.PATH = join(FIXTURES, "missing") + expect((await host.info()).home).toBe(home) + }) + + test("finds executables on the remote machine", async () => { + useFakeSsh() + const host = sshHost("my-mac") + const awkward = awkwardExecutable() + expect(await host.findExecutable([undefined, "", "/nope/codex", FIXTURES, awkward, FAKE_CODEX], "codex")).toBe( + awkward, + ) + expect(await host.findExecutable([join(FIXTURES, "fake-codex.ts")], "sh")).toMatch(/\/sh$/) + expect(await host.findExecutable(["/nope/codex"], "no-such-command-for-cu-tests")).toBeUndefined() + }) + + test("checks whether a path exists on the remote machine", async () => { + useFakeSsh() + const host = sshHost("my-mac") + expect(await host.exists(awkwardExecutable())).toBe(true) + expect(await host.exists(join(home, "missing"))).toBe(false) + }) +}) diff --git a/test/options.test.ts b/test/options.test.ts index 823c99e..0374665 100644 --- a/test/options.test.ts +++ b/test/options.test.ts @@ -1,9 +1,11 @@ -import { describe, expect, test } from "bun:test" +import { afterEach, describe, expect, test } from "bun:test" import { homedir } from "node:os" -import { join } from "node:path" +import { delimiter, join } from "node:path" +import { ComputerUseBridge } from "../src/bridge" import { stripImages } from "../src/debug" -import { approvalCheck } from "../src/doctor" +import { approvalCheck, runDoctor } from "../src/doctor" +import { sshHost, type Host } from "../src/host" import { describeTool } from "../src/index" import { DEFAULT_DEBUG_LOG, parseOptions } from "../src/options" import { surfaceGuard } from "../src/surfaces" @@ -12,6 +14,7 @@ describe("parseOptions", () => { test("defaults", () => { expect(parseOptions({})).toEqual({ codexPath: undefined, + ssh: undefined, idleShutdownMs: 0, callTimeoutMs: 300_000, maxOutputBytes: 128 * 1024, @@ -26,6 +29,7 @@ describe("parseOptions", () => { expect( parseOptions({ codexPath: " /x/codex ", + ssh: " me@my-mac ", idleShutdownMinutes: 30, callTimeoutSeconds: 60, maxOutputKB: 0, @@ -35,6 +39,7 @@ describe("parseOptions", () => { }), ).toEqual({ codexPath: "/x/codex", + ssh: "me@my-mac", idleShutdownMs: 1_800_000, callTimeoutMs: 60_000, maxOutputBytes: 0, @@ -52,6 +57,15 @@ describe("parseOptions", () => { expect(() => parseOptions({ surfaces: [] })).toThrow("invalid option surfaces") expect(() => parseOptions({ surfaces: ["apps", "files"] })).toThrow("invalid option surfaces") }) + + test("ssh takes one SSH destination", () => { + expect(parseOptions({ ssh: "my-mac" }).ssh).toBe("my-mac") + expect(parseOptions({ ssh: " " }).ssh).toBeUndefined() + expect(() => parseOptions({ ssh: "-oProxyCommand=sh" })).toThrow( + '[codex-computer-use] invalid option ssh="-oProxyCommand=sh"; use an SSH destination such as "my-mac" or "me@my-mac"', + ) + expect(() => parseOptions({ ssh: "me@my-mac -p 2222" })).toThrow('invalid option ssh="me@my-mac -p 2222"') + }) }) test("the removed approvals option is ignored with a warning", () => { @@ -68,6 +82,85 @@ describe("approvalCheck", () => { }) }) +function doctor(host: Host) { + const bridge = new ComputerUseBridge({ + host, + idleShutdownMs: 0, + callTimeoutMs: 1_000, + cwd: "/", + version: "test", + log: () => {}, + }) + return runDoctor(bridge, parseOptions({ ssh: "my-mac" }), "doctor") +} + +describe("runDoctor", () => { + const PATH = process.env.PATH + + afterEach(() => { + process.env.PATH = PATH + }) + + test("stops at the platform check when the Computer Use machine cannot be reached", async () => { + const host: Host = { + ...sshHost("my-mac"), + info: async () => { + throw new Error("Could not reach my-mac over SSH: ssh: Could not resolve hostname my-mac") + }, + } + const report = await doctor(host) + expect(report.ok).toBe(false) + expect(report.checks).toEqual([ + { + name: "Platform", + status: "fail", + detail: "Could not reach my-mac over SSH: ssh: Could not resolve hostname my-mac", + }, + ]) + }) + + test("checks the platform and codex on the Computer Use machine", async () => { + const searched: (string | undefined)[][] = [] + const host: Host = { + ...sshHost("my-mac"), + info: async () => ({ + platform: "darwin", + arch: "arm64", + home: "/Users/me", + codexHome: "/Users/me/.codex", + osVersion: "26.1", + }), + findExecutable: async (candidates) => { + searched.push([...candidates]) + return undefined + }, + } + const report = await doctor(host) + expect(report.checks).toEqual([ + { name: "Platform", status: "ok", detail: "macOS 26.1 (arm64) on my-mac over SSH" }, + { name: "codex executable", status: "fail", detail: expect.stringMatching(/^Not found on my-mac over SSH\. /) }, + ]) + expect(searched[0]).toContain("/Applications/ChatGPT.app/Contents/Resources/codex") + }) + + test("reports a Mac that stops answering after the platform check", async () => { + const fakeSsh = join(import.meta.dir, "fixtures", "fake-ssh") + process.env.PATH = `${fakeSsh}${delimiter}${PATH}` + const host = sshHost("my-mac") + await host.info() + // From now on the stand-in ssh starts but cannot run the remote command. + process.env.PATH = fakeSsh + + const report = await doctor(host) + expect(report.ok).toBe(false) + expect(report.checks.map((check) => check.name)).toEqual(["Platform", "codex executable"]) + expect(report.checks[1]).toMatchObject({ + status: "fail", + detail: expect.stringMatching(/^Could not reach my-mac over SSH: /), + }) + }) +}) + describe("surfaceGuard", () => { test("adds nothing when every surface is enabled", () => { expect(surfaceGuard(["apps", "browser"])).toBeUndefined() @@ -108,6 +201,14 @@ describe("describeTool", () => { expect(appsOnly).toContain("recognized text") expect(describeTool(parseOptions({}))).toContain("native desktop apps and Chrome tabs") }) + + test("says when the apps and tabs are on another machine", () => { + expect(describeTool(parseOptions({}))).not.toContain("SSH") + const remote = describeTool(parseOptions({ ssh: "my-mac" })) + expect(remote).toContain("are on my-mac (reached over SSH), a different machine") + expect(remote).toContain("local files are not visible there until copied (for example with `scp my-mac:`).") + expect(remote.split("\n").filter((line) => line.includes("SSH"))).toHaveLength(1) + }) }) describe("stripImages", () => {