CaptionCast

CaptionCast turns one speaker's microphone into live captions for an audience. Listeners open a link or scan a QR code on their phones, choose a language when translation is enabled, and follow along in real time.

The project has two small TypeScript applications:

• web/ — the audience viewer at / and speaker console at /admin.html
• worker/ — the Cloudflare Worker and Durable Object that relay captions, translations, and optional audio

Run locally in five minutes

1. Install the prerequisites

Install Node.js 20 or newer. Node 24 is recommended. For the free Web Speech provider, use Chrome or Edge when opening the speaker console.

Clone the repository, then install both sets of dependencies:

git clone <this-repo>
cd CaptionCast

cd worker && npm install
cd ../web && npm install
cd ..

2. Create your local speaker key

Copy the safe example file:

cp worker/.dev.vars.example worker/.dev.vars

Open worker/.dev.vars and replace change-me-for-local-development with a value you will remember. This is the speaker access key you enter in the console. The file is ignored by Git.

You can leave both provider API keys empty when using Web Speech. Add ELEVENLABS_API_KEY or OPENAI_API_KEY only when you want that provider.

3. Start the relay

In the first terminal:

cd worker
npm run dev

The relay starts at http://localhost:8787. To confirm it is healthy, open that address with /health or run:

curl http://localhost:8787/health

4. Start the web app

In a second terminal:

cd web
npm run dev

Vite prints the local address, normally http://localhost:5173.

5. Start your first broadcast

1. Open http://localhost:5173/admin.html in Chrome or Edge.
2. Choose Free captions — Web Speech.
3. Choose the language you will speak.
4. Enter the same SPEAKER_SECRET value from worker/.dev.vars.
5. Select Start broadcasting and allow microphone access.
6. Give listeners the displayed viewer link or let them scan its QR code.

Listeners open http://localhost:5173/. Captions appear automatically. Stop the broadcast before changing providers or provider settings.

Choose a provider

Speaker option	Provider	What listeners get	What you need
Free captions	Browser Web Speech	Original-language captions	Chrome or Edge; no provider API key
High-accuracy captions	ElevenLabs Scribe	Original-language captions	ELEVENLABS_API_KEY; metered account
Realtime transcription	OpenAI gpt-realtime-whisper	Original-language text captions	OPENAI_API_KEY; metered account
Live translation	OpenAI Realtime	Translated captions and optional translated audio	OPENAI_API_KEY; metered account

Every mode still requires SPEAKER_SECRET. The browser exchanges it over HTTP for a signed 12-hour speaker ticket, clears the master key, and uses only the short-lived ticket for relay and provider-token requests. Provider keys remain on the worker and are never entered into the browser.

ElevenLabs Scribe v2 Realtime sends both partial and committed transcripts. CaptionCast displays partial text immediately while the speaker is talking, then replaces it with the more accurate committed segment after voice-activity detection finds a pause. Its Spoken language setting defaults to automatic detection. Selecting a known language can improve transcription accuracy, and the choice is remembered in that browser.

Everyday use

Speaker

• Choose a provider based on whether you need free captions, higher-accuracy transcription, or translation.
• For ElevenLabs, leave Spoken language on Detect automatically or choose the language being spoken.
• Click Open room first. This admits viewers and lets them choose a translation language without starting microphone capture or paid provider sessions.
• Click Start captions only when the speaker is ready. Stopping captions keeps the room and audience connected while shutting down provider sessions; Close room disconnects the room completely.
• Use Find microphones to grant permission and choose a built-in microphone, external microphone, or headset. ElevenLabs and OpenAI use the selected input directly. Browser Web Speech recognition is limited to the operating system's default input; a selected device still applies to its optional original-audio broadcast.
• Turn on Broadcast original audio only when remote listeners need the source audio. It increases relay traffic.
• You can change microphones while live. OpenAI capture swaps in place; ElevenLabs briefly reconnects its engine; Web Speech captions continue using the operating-system default.
• Open Advanced settings to choose Efficient Opus delivery or PCM compatibility. Nerd stats live below the Live room card and sample only while that card is open.
• The audience card shows how many people are connected in each language and how many enabled audio.
• Stop broadcasting before changing the provider, spoken language, access key, or model. Microphone and audience-audio delivery can change while live.

Listener

• Keep the viewer page open; it reconnects automatically after network interruptions.
• On the first OpenAI translation visit, choose a language from the full-page welcome screen; the choice is remembered and remains changeable in the header. Non-translating providers skip this step.
• Use the EN/日本語 control to localize the viewer interface independently from the caption language.
• Use the speaker button when audio is available.
• Use the three header buttons to make captions smaller, make them larger, or switch directly between Light and Dark.
• Use the download button to export the current local conversation history as a UTF-8 text file. Unfinished partial speech is excluded.
• Committed captions are saved only in that browser's local storage and survive a refresh. CaptionCast does not store a server-side transcript.

Troubleshooting

“Can’t reach the caption relay”

Make sure npm run dev is still running in the worker/ terminal and that the web app is configured for the correct relay URL. Local development defaults to ws://localhost:8787.

The speaker access key was rejected

The value entered in the speaker console must exactly match SPEAKER_SECRET in worker/.dev.vars. Restart the worker after changing that file.

Web Speech is unavailable

Open the speaker console in a current Chrome or Edge browser. Safari and Firefox do not currently expose the compatible recognition API used by this mode.

No captions appear

• Confirm the microphone, provider, and relay diagnostics are green.
• Check that microphone access is allowed for the site.
• Speak for a few seconds; providers may wait for a short pause before committing a line.
• For ElevenLabs or OpenAI, confirm the matching API key is configured and valid.

Listeners cannot hear audio

Browsers require a user gesture before playback. The listener must press the audio button. Original audio also requires Broadcast original audio to be enabled by the speaker.

Test without speaking or using provider keys

With the worker and web app running, start the scripted fake speaker:

cd worker
node test/fake-speaker.mjs

It safely announces Web Speech mode and loops through partial and committed captions without opening a paid provider session.

To exercise the translated-language UI, remove or temporarily blank OPENAI_API_KEY, restart the worker, and then run:

FAKE_TRANSLATION=1 node test/fake-speaker.mjs

The translation flag announces OpenAI mode. If an OpenAI key remains configured, the worker may open real provider sessions and incur usage.

Architecture

A single Cloudflare Durable Object acts as the room and WebSocket fan-out hub. The speaker sends captions or microphone audio to it; viewers subscribe to the original stream or one translated language.

OpenAI transcription and translation are separate speaker modes. Opening the room advertises their capabilities but carries broadcasting: false, so no provider session can run. Start captions changes that state to true. Transcription then connects with intent=transcription, opens a type: "transcription" session, and configures gpt-realtime-whisper under audio.input.transcription.model for text-only output. Translation uses the dedicated gpt-realtime-translate endpoint and keeps one session per subscribed language; hundreds of listeners using the same language share it, and unused sessions close after a grace period. MAX_LANGUAGES—three by default—limits concurrent translated languages and controls cost.

The translation session can additionally configure gpt-realtime-whisper input transcription to show source-language context beside translated output. This does not turn the text-only and translation choices into one model or pipeline.

Opus and PCM audio delivery

CaptionCast keeps provider input as mono PCM16 at 24 kHz, matching the translation service. Listener delivery defaults to Efficient, which negotiates 24 kbit/s mono Opus and automatically falls back to PCM16 if the listener cannot initialize the decoder. Compatibility forces PCM16 for troubleshooting.

The browser captures one 100 ms PCM16 chunk (4,800 bytes) at a time. The Durable Object creates at most one Opus encoder per active audio language, never per viewer, and batches five 20 ms Opus packets into one 100 ms WebSocket frame. It does not decode translated audio, frame PCM, or invoke an Opus encoder unless that language currently has a matching audio subscriber.

Raw PCM payload is about 384 kbit/s per audio listener. The codec smoke fixture currently produces a 331-byte Opus batch per 100 ms—about 26.5 kbit/s of codec payload before WebSocket overhead. Caption-only traffic depends on speech and transcript delta size.

Protocol v2 and coordinated releases

Viewer protocol v2 is intentionally not backward compatible. Viewer WebSockets connect with ?v=2, every subscribe includes audioCodec, and every status contains protocolVersion: 2. Deploy the Worker and web bundle as one coordinated release; old viewers receive WebSocket close code 4406 and must reload.

Run cd worker && npm run test:codec for the local encode/decode and packet-batching smoke check. Run cd worker && npm run test:ticket for valid, wrong-secret, tampered, and expired speaker-ticket checks. Run cd worker && npm run test:hardening for sequence reservation, zero-demand audio suppression, health transitions, exact-origin CORS, limiter responses, OpenAI opening-buffer bounds, fatal failures, and transient retry state.

See docs/PLAN.md for the protocol and architecture, and caption-relay-build-playbook.md for the original milestone guide.

Build and typecheck

cd web && npm run build
cd ../worker && npx tsc --noEmit
npm run test:codec && npm run test:ticket
npm run test:hardening

Deploy

Landing site on GitHub Pages

The bilingual product landing page lives in site/. Its interactive demonstration is deterministic: it animates the speaker, QR join, language choice, and translated captions without contacting the relay or a provider.

Run it locally in a third terminal:

cd site
npm install
npm run dev

Build it with optional public links:

VITE_REPOSITORY_URL=https://github.com/OWNER/CaptionCast \
VITE_LIVE_URL=https://captions.example.com \
npm run build

VITE_REPOSITORY_URL powers the GitHub and self-hosting actions. VITE_LIVE_URL is optional; when it is absent, the landing page hides Try it live. The Vite build uses relative asset paths so it works from a GitHub project URL such as /CaptionCast/.

The included GitHub Pages workflow builds and publishes site/ after landing-site changes reach main, and it can also be run manually. In the repository settings, select GitHub Actions as the Pages source. Add a repository Actions variable named CAPTIONCAST_LIVE_URL only after the product app has a stable public URL.

Landing-page copy is maintained in site/src/main.ts; keep the English and Japanese entries aligned. The five demo steps and their sample captions are defined in the same file so the story remains accurate when the product changes.

1. Deploy the Cloudflare Worker

cd worker
npx wrangler login
npm run deploy
npx wrangler secret put SPEAKER_SECRET
npx wrangler secret put ELEVENLABS_API_KEY
npx wrangler secret put OPENAI_API_KEY

Only set the provider secrets you plan to use. Note the resulting workers.dev host and convert its scheme to wss:// for the web configuration.

Set ALLOWED_ORIGINS in worker/wrangler.toml to the exact production web origins before deploying. The AUTH_RATE_LIMITER binding permits 10 speaker-key exchanges per 60 seconds; TOKEN_RATE_LIMITER permits 20 provider-token requests per 60 seconds per signed-ticket nonce. Requests without Origin remain available to CLI tools and tests.

2. Deploy the web app

The included web/vercel.json builds the Vite app and serves the clean /admin route. In Vercel, use web/ as the project root and add:

VITE_RELAY_URL=wss://caption-relay.<your-subdomain>.workers.dev

Then deploy with the Vercel dashboard or CLI:

cd web
npx vercel --prod

Any static host can serve web/dist; set VITE_RELAY_URL at build time to the deployed relay WebSocket origin.

Privacy and resilience

• Provider credentials remain server-side.
• The long-lived speaker access key is cleared after exchange; only a signed 12-hour ticket is kept for the current browser tab session.
• Viewer transcripts are local to each listener's browser.
• Viewer and speaker relay connections reconnect with backoff.
• Reconnect delays use ±25% jitter. The browser drops new microphone chunks above a 64 KB WebSocket backlog and resumes below 32 KB instead of reconnecting the room.
• Viewer sockets use a server-driven heartbeat; normal tab closes are immediate, while crashed or disconnected viewers time out within about 60 seconds.
• Individual OpenAI language sessions buffer up to two seconds (20 chunks) while opening and reconnect independently with jittered 1/2/4/10-second backoff. Authentication, invalid-model, and invalid-session-configuration failures stay terminal until captions restart.
• The relay limits speaker message rates and slow viewer buffers.

GET /health is public and sanitized. It reports room/broadcast state, aggregate viewer and provider counts, provider readiness, heartbeat/provider-event ages, and backpressure disconnects. While live, ok becomes false only after speaker loss, a terminal provider failure, or a heartbeat age over 30 seconds; it never includes transcripts, credentials, IP addresses, provider error text, or per-language audience counts.

Load validation

The release gate is 500 sustained viewers for 30 minutes plus 1,000 joins spread across 60 seconds, mixing caption-only, Opus, PCM, and three translation languages. Acceptance is no room crash or sequence reuse, under 1% unexpected healthy-client disconnects, p95 join-to-status below one second, and p95 caption fan-out below 250 ms. No deployed environment was available for this repository change, so production measurements remain to be recorded after the coordinated deployment; do not treat local smoke tests as load-test results.

License

MIT — see LICENSE.