Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
243 changes: 148 additions & 95 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,121 +1,174 @@
# GPT-VoiceCoding

Voice-controlling terminal coding agents through a realtime voice call: the system
speaks agent progress to you, and carries your spoken instructions back to the
agents.

> **Status: early build.** Every part is now built and tested — Bridge Core, the
> control plane, the composition root, the Codex and Claude Agent adapters with
> all three Relays, the bridge-owned Live Call, the Companion Channel, the
> menu-bar shell, and the signed app bundle. Starting and closing sessions are
> deliberately not among them: v1.0 is a bridge over the sessions you start
> yourself, and the Session Launcher is parked for a later release. It has not
> been through a release. If you want software with users on it, the
> first-generation implementation lives at
> [GPT-VoiceCoding-legacy](https://github.com/okqixiaobao727-design/GPT-VoiceCoding-legacy).
**for Claude Code & Codex**

![macOS](https://img.shields.io/badge/macOS-only-111111)
![Apple Silicon](https://img.shields.io/badge/Apple%20Silicon-required-111111)
![Licence](https://img.shields.io/badge/licence-MIT-111111)

<!-- PENDING ASSET: the demo video. Drag scratch/demo-recorder/out/take.tvuVMJ/demo.mp4
into any GitHub comment box, then replace this whole comment with the
https://github.com/user-attachments/... URL it returns, on a line of its own with
no Markdown around it, followed by:

*Recorded unattended; the narration and the spoken lines are synthesized speech.*
-->

One voice call covers every Claude Code and Codex session on your Mac: it tells you which one
stopped and what it stopped on, and carries your spoken answer back into it. The call rides
`codex app-server`'s realtime route, so it runs on the ChatGPT account you already signed
`codex` into, and there is no API key to supply.

## How it works

- A session stops and it calls you. The bridge watches the sessions you started yourself; it
does not launch them and does not wrap them.
- You answer by voice. You hear what the session stopped on, say what to do, and your words go
back in as an instruction that session can act on.
- Away from the desk, you answer on Telegram. Every stop arrives there too, and a reply to it
reaches the same session.

## Screenshots

| | |
|---|---|
| ![The Duty Lamp](docs/images/duty-lamp.png)<br>The Duty Lamp sits above everything and counts what is waiting on you. Hovering it offers Hang up, Control and Settings, and shows the session that moved last. | ![The Control Panel](docs/images/control-panel.png)<br>The Control Panel during a call: the three switches, the Call Agent's model and context use, the counts, and every live session. |
| ![Settings](docs/images/settings.png)<br>Settings. Nine voices to pick the Live Call's from, and the realtime model behind it. | ![Telegram](docs/images/telegram.png)<br>Telegram. A stop notice you can reply to, `/sessions` drawn as buttons, and the bot's own command menu. |

## Install

On an Apple Silicon Mac, run this in your terminal:
On an Apple Silicon Mac:

```sh
curl -fsSL https://raw.githubusercontent.com/okqixiaobao727-design/GPT-VoiceCoding/main/scripts/install.sh | sh
```

You need Apple's Command Line Tools (`xcode-select --install`), Python **3.12 or
newer** available as `python3` on your PATH, and HTTPS access to GitHub and PyPI.
The script checks these in order and stops with a fix if one is missing; it
does not install Homebrew or Python. It builds the full ad-hoc-signed app from
source under `~/Library/Application Support/GPT-VoiceCoding/source`, quits a
running copy, places the new bundle in `/Applications`, and opens it. There is
no notarized download.

**Upgrade by running the same command again.** Your existing `config.toml` and
Telegram credential file stay in place. First launch creates a configuration
only if none exists, then walks through Codex login, the installed helpers,
Call Agent, optional Telegram, and a test call. Have Codex installed and log in
with `codex login` when the guide asks. Later launches are silent. Settings that
need an engine restart show one explicit **Restart now** action.

Telegram setup (in the guide or Settings) first checks the bot token. The token
identifies the bot, not the receiving chat. Search for the displayed `@username`
in whichever Telegram client you already use and send `/start` after the token
check. Return to the app, click **I've sent /start — check**, and save after you
see the confirmation message in that chat. The app does not open a Telegram
client for you or check for that message until you click.

The Duty Card starts at the top right of the available screen on a fresh
installation; subsequent launches restore its saved position.

## What it is

You are away from the keyboard. A Claude Code or Codex session stops and needs
you. GPT-VoiceCoding tells you — out loud, in a voice call it holds open — and
carries your spoken answer back into the session. It watches the sessions you
start, relays your words into them, and answers their permission prompts with
your verdict. It does not start them — you do, with your own `claude` or `codex`
command, and the bridge covers whatever is running.

The voice call is owned by this system directly, over `codex app-server`'s realtime
route. There is no API key to supply.

## Shape

A thin Swift menu-bar app spawns a single Python asyncio engine from inside its own
bundle. That engine is **Bridge Core**: it owns every policy and holds the single
source of truth, and reaches everything else — the call, the coding agents, the
Companion Channel — through seams with swappable adapters.
It builds an ad-hoc-signed app from source, puts it in `/Applications` and opens it. There is no
notarized download. Run the same command again to upgrade; your configuration and Telegram
credentials stay where they are.

You need:

- An Apple Silicon Mac, with Apple's Command Line Tools (`xcode-select --install`).
- Python 3.12 or newer, as `python3` on your PATH. The installer checks for it and stops with a
fix rather than installing one.
- Claude Code or Codex, whichever you use. Both work, together or alone.
- Codex signed in with a ChatGPT account (`codex login`). The Live Call needs it.
- Telegram, if you want the second route. It is optional, and first launch walks you through it.

## Architecture

A thin Swift menu-bar app spawns one Python asyncio engine from inside its own bundle. That
engine is **Bridge Core**: it owns every policy and holds the single source of truth, and
reaches everything else through seams with swappable adapters.

The two agents are reached differently because they publish themselves differently. Claude Code
is read from its own roster and two hooks, and written to through the inbox socket every session
already binds. Codex is read and written through the shared `codex app-server` this product
starts and your own terminals join, which is the same process the Live Call's realtime route
rides.

```mermaid
flowchart LR
subgraph terminals["Your own terminals"]
cc["Claude Code Session"]
cx["Codex Session"]
end

appserver["codex app-server<br/>started by this product,<br/>joined by your Sessions"]

subgraph engine["The engine"]
core["Bridge Core<br/>every policy, one source of truth"]
agents["Agent adapters"]
callad["Call adapter"]
chan["Companion Channel adapter"]
plane["Control plane<br/>JSON over a Unix socket"]
core --- agents
core --- callad
core --- chan
core --- plane
end

live["Live Call<br/>Voice speaks, Call Agent acts"]
tg["Telegram"]
shell["Menu-bar app<br/>Duty Lamp, Control Panel"]
ctl["bridgectl"]

cc <-->|"hooks, roster, inbox socket"| agents
cx --- appserver
appserver --- agents
appserver -->|"realtime route"| callad
callad --- live
chan --- tg
plane --- shell
plane --- ctl
```

What happens when a session stops:

```mermaid
sequenceDiagram
participant S as Your Session
participant C as Bridge Core
participant T as Telegram
participant K as Call Keeper
participant V as Voice
participant A as Call Agent
participant U as You

S->>C: stops, waiting on a decision
C->>C: writes the reading to the roster
C->>T: Stop Notice, as an Anchor you can reply to
C->>K: wake
K->>V: dials, briefed from the roster as it stands
V->>U: speaks the Session Brief
U->>V: "use email login, and give me the steps"
V->>A: hands the job over
A->>C: relay
C->>S: Answer Relay
```

Or, without the call:

```mermaid
sequenceDiagram
participant T as Telegram
participant U as You
participant C as Bridge Core
participant S as Your Session

T->>U: Stop Notice
U->>T: replies to it
T->>C: inbound text, naming the Anchor's Session
C->>S: Answer Relay
```

The Voice has no tools. It speaks from what the engine hands it and passes anything that reads
as a job to the Call Agent, which is the only half on the call that can run a control-plane verb.

```
src/gpt_voicecoding/
├── core/ Bridge Core — the hub. All policy, all state.
├── seams/ The interfaces Bridge Core calls through, and the control plane's vocabulary.
├── seams/ The interfaces Bridge Core calls through.
├── adapters/ The implementations behind them. Protocol libraries live only here.
├── control_plane/ The JSON-over-UDS surface: framing, sockets, translation.
├── engine/ The composition root — config in, one running engine out.
├── installation/ What this product puts in files the user owns, and takes back.
└── cli/ bridgectl — a control-plane surface.
shell/ The Swift menu-bar shell (see ADR 0005).
app_bundle/ The build pipeline. Builds the product; is not part of it.
```

One command turns a clean checkout into a signed `.app`:

```bash
scripts/build-app.sh
shell/ The Swift menu-bar shell.
```

See [`docs/app-bundle.md`](docs/app-bundle.md).

The engine runs standalone, without the menu-bar shell:
`python -m gpt_voicecoding.engine --config <file>`. The Live Call's audio path is
an optional extra — `pip install 'gpt-voicecoding[voice]'` — because everything
else runs without a compiled media stack. See
[`docs/control-plane.md`](docs/control-plane.md) for the interface, the command
set and the configuration file.

Start with [`docs/adr/0001`](docs/adr/0001-hub-and-spoke-bridge-core-with-seams.md).

## Platform

macOS only. The microphone grant, the menu-bar shell and the push path are all
macOS-shaped; cross-platform support is not planned.

## Reading order

1. [`CONTEXT.md`](CONTEXT.md) — the vocabulary. Every term in this repo means what
it says there.
2. [`docs/adr/`](docs/adr/README.md) — the decisions, and where each one came from.
3. [`docs/control-plane.md`](docs/control-plane.md) — the interface every surface
speaks, and what the engine is configured with.
4. [`docs/app-bundle.md`](docs/app-bundle.md) — how the `.app` is built and
signed, and every decision that pipeline holds.
The engine runs without the menu-bar app: `python -m gpt_voicecoding.engine --config <file>`.
The Live Call's audio path is an optional extra, `pip install 'gpt-voicecoding[voice]'`, because
everything else runs without a compiled media stack.

## Contributing
Start with [`CONTEXT.md`](CONTEXT.md) for what each term means, then
[`docs/adr/0001`](docs/adr/0001-hub-and-spoke-bridge-core-with-seams.md) for why the shape is
this one. [`docs/control-plane.md`](docs/control-plane.md) has the interface, the command set and
the configuration file; [`docs/app-bundle.md`](docs/app-bundle.md) has the build.

v0 targets developers: build from source, no signed release, no notarization
(ADR 0005). Issues and pull requests are welcome once there is something to build
on — the build issues land next.
macOS only. The microphone grant, the menu-bar app and the push path are all macOS-shaped, and
cross-platform support is not planned.

## Licence

Expand Down
Binary file added docs/images/control-panel.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/duty-lamp.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/settings.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/telegram.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading