Add Codex desktop app mode

README.md

@@ -12,14 +12,14 @@
 
 Play retro arcade games while your agent works—then jump back the moment it needs you.
 
-VibeSense wraps Claude Code or Codex CLI in a terminal, tracks the agent lifecycle through hooks, and switches one game controller between a browser game and the agent. Your terminal remains the agent's native TUI; VibeSense supplies the game, controller routing, and handoff.
+VibeSense wraps Claude Code or Codex CLI, or drives the Codex desktop app on macOS. It tracks agent lifecycle hooks and switches one game controller between a browser game and the agent.
 
 ## What VibeSense does
 
 - Starts the active game when the wrapped agent begins working, then pauses it when the agent stops or needs approval.
 - Routes controller input to the terminal while the agent needs you and to the game while it is running.
 - Ships five browser games and supports installable web games and external game adapters.
-- Wraps Claude Code by default and supports Codex CLI explicitly.
+- Wraps Claude Code by default and supports Codex CLI plus the Codex desktop app.
 - Shares one controller, game, and local host across multiple wrapped sessions.
 - Provides standalone play and optional continuous-play modes when no agent-driven handoff is wanted.
 
@@ -29,6 +29,7 @@ VibeSense wraps Claude Code or Codex CLI in a terminal, tracks the agent lifecyc
 - macOS is the primary supported platform. Automatic browser opening and keep-awake behavior are macOS-specific.
 - A controller supported by the installed OpenMicro version. Compatibility depends on the device and connection; generic HID support is best-effort.
 - Native build tools for `node-pty` and OpenMicro's `node-hid` dependency, such as Xcode Command Line Tools on macOS.
+- Codex app mode requires the Codex macOS app and Accessibility/Automation permission for the terminal running VibeSense.
 
 Verify the controller before starting:
 
@@ -59,17 +60,19 @@ npx @vibesense/cli
 ```sh
 vibesense                         # wrap Claude Code
 vibesense codex                   # wrap Codex CLI
+vibesense codex-app               # drive the Codex desktop app (macOS)
 vibesense play snake              # play without an agent
 ```
 
-Arguments after `vibesense` are forwarded to Claude Code. Arguments after `vibesense codex` are forwarded to Codex CLI.
+Arguments after `vibesense` are forwarded to Claude Code. Arguments after `vibesense codex` are forwarded to Codex CLI. `vibesense codex-app` is a no-PTY GUI mode and accepts only VibeSense options such as `--no-game` and `--auto-play`.
 
 ## Commands and options
 
 | Command                             | Behavior                                                                     |
 | ----------------------------------- | ---------------------------------------------------------------------------- |
 | `vibesense [claude args...]`        | Wrap Claude Code.                                                            |
 | `vibesense codex [codex args...]`   | Wrap Codex CLI.                                                              |
+| `vibesense codex-app`               | Launch and drive the Codex desktop app on macOS without a PTY.               |
 | `vibesense play [game]`             | Play without an agent; named selection supports installed web games.         |
 | `vibesense games`                   | List installed games; `*` marks the active game.                             |
 | `vibesense install <id-or-package>` | Install an official game ID, npm package, tarball, URL, or local path.       |
@@ -79,6 +82,7 @@ Arguments after `vibesense` are forwarded to Claude Code. Arguments after `vibes
 | `vibesense logout`                  | Remove the marketplace token and cached entitlements.                        |
 | `--no-game`                         | Do not automatically open the browser game tab.                              |
 | `--auto-play`                       | Keep the game running independently of agent state; Menu can still pause it. |
+| `--help`, `-h`                      | Show command help, including the no-PTY Codex app mode.                      |
 | `--version`, `-v`                   | Print the installed VibeSense version.                                       |
 
 `--auto-play` also starts `caffeinate -disu` on macOS for the lifetime of VibeSense, preventing system idle sleep and resetting the OS idle timer. Other platforms still keep the game running but do not receive this keep-awake integration.
@@ -95,16 +99,25 @@ Run Codex with `vibesense codex`. VibeSense installs hooks into `$CODEX_HOME/hoo
 
 Codex must permit hooks through its local or administrator policy. VibeSense does not edit `config.toml`, bypass trust, or override policy. Codex exposes no lifecycle event between approving a tool and that tool starting, so the game can remain paused until the following `PostToolUse` event.
 
+### Codex desktop app
+
+Run `vibesense codex-app` on macOS. VibeSense launches Codex without a PTY and delegates app control to OpenMicro's exported `codex-app` harness. The south button accepts, east rejects or dismisses, north holds push-to-talk, and the D-pad sends arrow keys. Touchpad cycles chats in the current project; L2/LT cycles projects. Each action targets the session that Codex brings frontmost. Continuous right-stick scrolling is intentionally unavailable because the shared app harness has no verified equivalent.
+
+On first use, allow the terminal running VibeSense under **System Settings → Privacy & Security → Accessibility** and **Automation** so it can control System Events and Codex. Codex's `Control+Shift+D` dictation shortcut must remain available for north-button push-to-talk; the other mapped controls use standard Enter, Escape, and arrow keys.
+
+Codex app mode installs both VibeSense's state hook and OpenMicro's shared harness hook in `$CODEX_HOME/hooks.json` (normally `~/.codex/hooks.json`). After the first install or any hook-definition change, open `/hooks` in Codex, inspect the VibeSense and OpenMicro commands, and trust them. Hooks are the state source for the shared Codex host: any headerless non-Claude Codex hook can pause the game when it needs attention, including Codex Desktop sessions in other projects and unwrapped Codex CLI sessions. Controller actions still target whichever Desktop session is frontmost. VibeSense does not copy OpenMicro's GUI or database automation.
+
 ## Controller behavior
 
 OpenMicro owns controller discovery, verification, normalized input, reconnection, and hardware lifecycle. VibeSense consumes those normalized events and decides whether they belong to the agent terminal, game, or game picker.
 
-| Context        | Controls                                                                                                                                                                                                                       |
-| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| Agent terminal | D-pad sends arrow keys; south button accepts; east button cancels; right stick scrolls; north button sends space. In Claude Code, that space invokes its native voice shortcut when configured; Codex receives a normal space. |
-| Browser game   | Left stick and R2/L2 are forwarded to the game. Each game shows its exact controls in the sidebar.                                                                                                                             |
-| Game picker    | View/Share opens the picker; D-pad selects; south confirms; east or View/Share cancels.                                                                                                                                        |
-| Any mode       | Menu/Options manually pauses or resumes the game.                                                                                                                                                                              |
+| Context        | Controls                                                                                                                                                                                                                           |
+| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Agent terminal | D-pad sends arrow keys; south button accepts; east button cancels; right stick scrolls; north button sends space. In Claude Code, that space invokes its native voice shortcut when configured; Codex CLI receives a normal space. |
+| Codex app      | D-pad sends arrow keys; south accepts; east rejects; north holds push-to-talk; Touchpad cycles chats; L2/LT cycles projects. Continuous right-stick scrolling is unmapped.                                                         |
+| Browser game   | Left stick and R2/L2 are forwarded to the game. Each game shows its exact controls in the sidebar.                                                                                                                                 |
+| Game picker    | View/Share opens the picker; D-pad selects; south confirms; east or View/Share cancels.                                                                                                                                            |
+| Any mode       | Menu/Options manually pauses or resumes the game.                                                                                                                                                                                  |
 
 Every terminal/game transition applies a 750 ms guard and ignores buttons held across the mode change, preventing game input from accidentally accepting an agent prompt.
 
@@ -129,13 +142,14 @@ Start with the [game-building guide](docs/building-a-game.md), read the [protoco
 
 ```text
 controller → OpenMicro → VibeSense input router
-                              ├─ agent waiting → keystrokes → node-pty ↔ Claude/Codex
+                              ├─ agent waiting → OpenMicro action → Codex app
+                              ├─ agent waiting → keystrokes → node-pty ↔ Claude/Codex CLI
                               └─ agent working → SSE input/state → browser game
 
 agent lifecycle hooks → http://127.0.0.1:48753 → shared agent-state host
 ```
 
-The first VibeSense process to bind the local port becomes the host and owns the controller and game. Later wrapped processes register as clients. The shared game pauses whenever any tracked session needs attention, while hook events from unrelated Claude Code and Codex CLI sessions are ignored.
+The first VibeSense process to bind the local port becomes the host and owns the controller and game. Later wrapped CLI processes register as clients. Codex app mode must be the host because GUI sessions cannot inherit a wrapper ID. Its shared game pauses whenever any Codex Desktop session needs attention.
 
 Only one host can use the default port `48753`. For a second development instance, choose another port:
 

src/cli.ts

@@ -2,6 +2,7 @@
 // vibesense — wrap an agent CLI in a pty and drive it with a game controller.
 // Usage: vibesense [--no-game] [--auto-play] [claude args...]   (Claude is the default)
 //        vibesense codex [--no-game] [--auto-play] [codex args...]
+//        vibesense codex-app [--no-game] [--auto-play]          (Codex desktop app)
 //        vibesense play [game] [--auto-play]                    (game only, no agent)
 //        vibesense --version | -v                               (print version, exit)
 //
@@ -16,8 +17,9 @@ import { createRequire } from 'node:module'
 import { isVibesenseHost, runAsClient } from './client.js'
 import { runSubcommand, SUBCOMMANDS } from './commands.js'
 import { startController } from './controller.js'
+import { actionForButton, launchGuiHarness } from './gui.js'
 import { harnessFor } from './harness.js'
-import { parseInvocation } from './invocation.js'
+import { parseInvocation, USAGE } from './invocation.js'
 import { KeyRepeater, REPEATING_BUTTONS, TERMINAL_KEYS } from './keymap.js'
 import { logger } from './logger.js'
 import {
@@ -38,6 +40,15 @@ import type { Aggregate } from './state.js'
 
 const args = process.argv.slice(2)
 
+if (
+  args[0] === '--help' ||
+  args[0] === '-h' ||
+  (args[0] === 'codex-app' && (args[1] === '--help' || args[1] === '-h'))
+) {
+  console.log(USAGE)
+  process.exit(0)
+}
+
 // --version/-v must not fall through to claude (which has its own --version).
 if (args[0] === '--version' || args[0] === '-v') {
   // createRequire: package.json sits one level above both src/ (tsx) and dist/ (built).
@@ -61,8 +72,8 @@ const { noGame, autoPlay, agentArgs, agentKind } = invocation
 const harness = agentKind ? harnessFor(agentKind) : null
 const hookHarness = harness ?? harnessFor('claude')
 const hookInstallResult = hookHarness.installHooks()
-if (hookInstallResult === 'changed' && hookHarness.trustNotice) {
-  console.error(hookHarness.trustNotice)
+if (hookInstallResult.changed && hookInstallResult.trustNotice) {
+  console.error(hookInstallResult.trustNotice)
 }
 
 const wrapperId = randomUUID()
@@ -145,29 +156,45 @@ const server = new HostServer(
   },
   process.cwd(),
   wrapperId,
+  harness?.usesPty === false ? harness : undefined,
 )
 const isHost = await server.listen()
 
-if (playMode && !isHost) {
+if ((playMode || harness?.usesPty === false) && !isHost) {
   console.error(`another vibesense is already running — game is at ${HOST_URL}`)
   process.exit(1)
 }
 
 // In play mode there is no agent: the listening server keeps the process
 // alive and stdin is untouched, so Ctrl+C quits by default.
-const agent = playMode
-  ? null
-  : new AgentPty(harness!.command, agentArgs, harness!.childWrapperId(wrapperId), (code) => {
-      shutdown()
-      process.exit(code)
-    })
+const agent =
+  playMode || harness?.usesPty === false
+    ? null
+    : new AgentPty(
+        harness!.command,
+        harness!.buildArgs(agentArgs),
+        harness!.childWrapperId(wrapperId),
+        (code) => {
+          shutdown()
+          process.exit(code)
+        },
+      )
+const gui =
+  harness?.usesPty === false
+    ? launchGuiHarness(harness, agentArgs, undefined, (error) => {
+        logger.error('Codex app failed to launch', error)
+        shutdown()
+        process.exitCode = 1
+      })
+    : null
 
 let stopController: () => void = () => {}
 
 function shutdown(): void {
   stopController()
   stopController = () => {}
   agent?.dispose()
+  gui?.dispose()
   if (isHost) server.close()
 }
 
@@ -204,6 +231,7 @@ if (!isHost) {
     // doesn't change the mode, but the tab still has to show picker mode.
     server.broadcastGameState(shouldPlay ? 'playing' : pickerOpen ? 'picking' : 'paused')
     if (mode === router.currentMode()) return
+    gui?.releasePushToTalk()
     repeater.releaseAll()
     scroller.setValue(0)
     router.setMode(mode)
@@ -264,6 +292,7 @@ if (!isHost) {
     stopController()
     externalGame?.stop()
     scroller.stop()
+    gui?.dispose()
   })
 
   stopController = startController((e) => {
@@ -274,6 +303,7 @@ if (!isHost) {
       }
       if (e.kind === 'disconnected') {
         repeater.releaseAll()
+        gui?.releasePushToTalk()
         scroller.setValue(0)
         return
       }
@@ -309,6 +339,7 @@ if (!isHost) {
             0,
             web.findIndex((g) => g.manifest.id === activeGame?.manifest.id),
           )
+          gui?.releasePushToTalk()
           pickerOpen = true
           repeater.releaseAll() // don't let a held key auto-fire while we're picking
           scroller.setValue(0)
@@ -340,6 +371,17 @@ if (!isHost) {
       // Terminal target — same handling as M1, but writes go to the focused session.
       if (routed.event.kind === 'button') {
         const { button, pressed } = routed.event
+        if (gui) {
+          const action = actionForButton(button, pressed)
+          if (!action) return
+          if (REPEATING_BUTTONS.has(button)) {
+            if (pressed) repeater.press(button, () => gui.perform(action))
+            else repeater.release(button)
+          } else {
+            gui.perform(action)
+          }
+          return
+        }
         const bytes = TERMINAL_KEYS[button]
         if (!bytes) return
         if (REPEATING_BUTTONS.has(button)) {
@@ -351,14 +393,14 @@ if (!isHost) {
         return
       }
 
-      if (routed.event.kind === 'axis' && routed.event.axis === 'right_y') {
+      if (!gui && routed.event.kind === 'axis' && routed.event.axis === 'right_y') {
         scroller.setValue(routed.event.value)
       }
     } catch (err) {
       logger.error('controller event handling failed', err)
     }
   })
-  scroller.start()
+  if (!gui) scroller.start()
   if (playMode || autoPlay) applyMode() // no agent will kick us — start playing now
 
   if (autoPlay) {

src/gui.ts

@@ -0,0 +1,76 @@
+import { spawn } from 'node:child_process'
+import type { ButtonId } from 'openmicro/controller'
+import type { Action, Harness } from 'openmicro/harness'
+import { TERMINAL_KEYS } from './keymap.js'
+
+interface LaunchResult {
+  on(event: 'error', listener: (error: Error) => void): unknown
+  on(event: 'close', listener: (code: number | null) => void): unknown
+}
+
+type AppLauncher = (command: string, args: string[]) => LaunchResult
+
+/** Convert verified Codex app buttons to OpenMicro actions. */
+export function actionForButton(button: ButtonId, pressed: boolean): Action | null {
+  if (button === 'north') return { type: 'push_to_talk', pressed }
+  if (!pressed) return null
+  if (button === 'south') return { type: 'accept' }
+  if (button === 'east') return { type: 'reject' }
+  if (button === 'touchpad') return { type: 'focus_session', index: -1 }
+  if (button === 'l2') return { type: 'herdr_space' }
+  const bytes = TERMINAL_KEYS[button]
+  return bytes ? { type: 'keys', bytes } : null
+}
+
+export interface GuiRuntime {
+  perform(action: Action): boolean
+  releasePushToTalk(): void
+  dispose(): void
+}
+
+/** Launch and drive a no-PTY harness through OpenMicro's lifecycle. */
+export function launchGuiHarness(
+  harness: Harness,
+  userArgs: string[],
+  launch: AppLauncher = (command, args) => spawn(command, args, { stdio: 'ignore' }),
+  onFailure: (error: Error) => void = () => {},
+): GuiRuntime {
+  let disposed = false
+  const dispose = (): void => {
+    if (disposed) return
+    disposed = true
+    harness.dispose?.()
+  }
+  let failed = false
+  const fail = (error: Error): void => {
+    if (failed) return
+    failed = true
+    dispose()
+    onFailure(error)
+  }
+  const child = launch(harness.command, harness.buildArgs(userArgs))
+  child.on('error', fail)
+  child.on('close', (code) => {
+    if (code !== 0) fail(new Error(`${harness.command} exited with code ${code ?? 'unknown'}`))
+  })
+  let pushToTalkHeld = false
+  const perform = (action: Action): boolean => {
+    const resolved = harness.resolveAction(action, { thinkingLevel: 0 })
+    if (!resolved || !harness.execute) return false
+    try {
+      harness.execute(resolved.bytes)
+      if (action.type === 'push_to_talk') pushToTalkHeld = action.pressed !== false
+      return true
+    } catch (error) {
+      dispose()
+      throw error
+    }
+  }
+  return {
+    perform,
+    releasePushToTalk(): void {
+      if (pushToTalkHeld) perform({ type: 'push_to_talk', pressed: false })
+    },
+    dispose,
+  }
+}