diff --git a/README.md b/README.md
index 7047212..6f9b89a 100644
--- a/README.md
+++ b/README.md
@@ -3,59 +3,61 @@
[](https://badge.fury.io/js/react-native-deepgram)
[](https://opensource.org/licenses/MIT)
-**react-native-deepgram** brings Deepgram's AI platform to React Native & Expo.
+The complete client for Deepgram's voice AI platform on **React Native** and **Expo** β real-time transcription, conversational voice agents, text-to-speech, and text intelligence in a single dependency, with hardware echo cancellation and background-audio support wired up for you.
-> β
Supports **Speech-to-Text v1** and the new **Speech-to-Text v2 (Flux)** streaming API alongside Text-to-Speech, Text Intelligence, and the Management API.
+```bash
+yarn add react-native-deepgram
+```
+
+## At a glance
+
+| | |
+| --- | --- |
+| π£οΈ **Voice Agent** | Full-duplex agents over Deepgram's WebSocket. Hardware AEC on iOS (VPIO) and Android (`VOICE_COMMUNICATION` + `MODE_IN_COMMUNICATION`). Function calling, prompt updates, latency telemetry. |
+| ποΈ **Speech-to-Text** | Live PCM streaming over WebSocket on STT v1 *or* Flux v2 (`flux-general-en`). File transcription with summarisation, topics, intents, entities. |
+| π **Text-to-Speech** | One-shot HTTP synthesis or low-latency WebSocket streaming with `Speak` / `Flush` / `Clear` / `Close` controls. |
+| π§ **Text Intelligence** | Summaries, topics, intents, sentiment over text or URLs. |
+| π οΈ **Management API** | Typed wrapper for projects, models, keys, usage, balances, members, invitations, scopes, purchases, and temporary tokens. |
+| βοΈ **Expo plugin** | One-line install β handles permissions, background-audio modes, and Android foreground service automatically. |
+| π§° **Resilient by default** | Audio interruptions, route changes, headphone unplugging, mediaserverd resets, audio-focus loss, and queue overflow are all handled. |
+| π **Modern iOS APIs** | `AVAudioApplication.requestRecordPermission` and `AllowBluetoothHFP` on iOS 17+, with safe fallbacks. |
+
+---
## Table of contents
-1. [Features](#features)
-2. [Installation](#installation)
-3. [Expo config plugin](#expo-config-plugin)
-4. [Configuration](#configuration)
-5. [Usage overview](#usage-overview)
+1. [Quick start (5 minutes)](#quick-start-5-minutes)
+2. [Requirements](#requirements)
+3. [Installation](#installation)
+4. [Core concepts](#core-concepts)
+5. [Audio session, AEC & background](#audio-session-aec--background)
6. [Voice Agent](#voice-agent-usedeepgramvoiceagent)
7. [Speech-to-Text](#speech-to-text-usedeepgramspeechtotext)
8. [Text-to-Speech](#text-to-speech-usedeepgramtexttospeech)
9. [Text Intelligence](#text-intelligence-usedeepgramtextintelligence)
10. [Management API](#management-api-usedeepgrammanagement)
-11. [Example app](#example-app)
-12. [Roadmap](#roadmap)
-13. [Contributing](#contributing)
-14. [License](#license)
+11. [Expo config plugin reference](#expo-config-plugin-reference)
+12. [Common recipes](#common-recipes)
+13. [Troubleshooting](#troubleshooting)
+14. [Example app](#example-app)
+15. [Roadmap](#roadmap)
+16. [Contributing](#contributing)
+17. [License](#license)
---
-## Features
+## Quick start (5 minutes)
-- π **Live Speech-to-Text** β capture PCM audio and stream it over WebSocket (STT v1 or v2/Flux).
-- π **File Transcription** β send audio files/URIs to Deepgram and receive transcripts.
-- π€ **Text-to-Speech** β synthesize speech with HTTP requests or WebSocket streaming controls.
-- π£οΈ **Voice Agent** β orchestrate realtime conversational agents with microphone capture + audio playback.
-- π§ **Text Intelligence** β summarisation, topic detection, intents, sentiment and more.
-- π οΈ **Management API** β list models, keys, usage, projects, balances, etc.
-- βοΈ **Expo config plugin** β automatic native configuration for managed and bare workflows.
-
----
-
-## Installation
+### 1. Install
```bash
yarn add react-native-deepgram
-# or
-npm install react-native-deepgram
```
-### iOS (CocoaPods)
-
-```bash
-cd ios && pod install
-```
-
-### Expo
+### 2. Configure the Expo plugin
```js
-// app.config.js
+// app.config.js (or app.json under "plugins")
module.exports = {
expo: {
plugins: [
@@ -76,19 +78,115 @@ npx expo prebuild
npx expo run:ios # or expo run:android
```
+> Bare React Native? Skip the plugin, run `cd ios && pod install`. The package autolinks via `react-native.config.js`.
+
+### 3. Configure the SDK once at startup
+
+```ts
+// App.tsx
+import { configure } from 'react-native-deepgram';
+
+configure({ apiKey: process.env.EXPO_PUBLIC_DEEPGRAM_API_KEY! });
+```
+
+> **Never ship raw API keys.** Use `EXPO_PUBLIC_*` env vars, secrets storage, or a backend proxy that mints scoped keys.
+
+### 4. Stream microphone audio to Deepgram (live STT)
+
+```tsx
+import { useDeepgramSpeechToText } from 'react-native-deepgram';
+import { Button, Text, View } from 'react-native';
+
+export function LiveCaption() {
+ const {
+ startListening,
+ stopListening,
+ state, // { status: 'idle' | 'listening' | 'error' | ... }
+ transcript, // accumulated final transcript
+ interimTranscript, // current partial
+ } = useDeepgramSpeechToText({
+ trackState: true,
+ trackTranscript: true,
+ live: { punctuate: true, interimResults: true },
+ });
+
+ return (
+
+ {transcript} {interimTranscript}
+
+ );
+}
+```
+
+### 5. Talk to a Voice Agent (full-duplex with AEC)
+
+```tsx
+import { useDeepgramVoiceAgent } from 'react-native-deepgram';
+
+const agent = useDeepgramVoiceAgent({
+ trackState: true,
+ trackConversation: true,
+ defaultSettings: {
+ audio: {
+ input: { encoding: 'linear16', sample_rate: 24_000 },
+ output: { encoding: 'linear16', sample_rate: 24_000, container: 'none' },
+ },
+ agent: {
+ language: 'en',
+ greeting: 'Hi! What can I help you with?',
+ listen: { provider: { type: 'deepgram', model: 'nova-3' } },
+ think: { provider: { type: 'open_ai', model: 'gpt-4o' }, prompt: 'You are a helpful assistant.' },
+ speak: { provider: { type: 'deepgram', model: 'aura-2-asteria-en' } },
+ },
+ },
+});
+
+await agent.connect(); // requests mic, opens WS, starts streaming
+agent.injectUserMessage('Hi'); // optional: prompt the agent with text
+agent.disconnect(); // tears everything down
+```
+
+That's a working, low-latency conversational agent with hardware echo cancellation. Open [the example app](#example-app) to try the full surface area.
+
---
-## Expo config plugin
+## Requirements
-The package ships with an Expo config plugin (exported from `app.plugin.js`) that keeps microphone permissions in sync for both
-platforms:
+| | Minimum |
+| --- | --- |
+| Node | 18 |
+| React | 18.0 |
+| React Native | 0.72 |
+| iOS | 13 (15+ recommended for VPIO AEC) |
+| Android | API 24 (Android 7.0) |
+| Xcode | 15+ if building against the iOS 17 SDK |
-- **Android** β automatically adds `android.permission.RECORD_AUDIO` to your manifest if it is missing.
-- **iOS** β sets `NSMicrophoneUsageDescription` with the message you provide (or a sensible fallback).
+> The library is published with the New Architecture in mind but works on the legacy renderer as well.
-### Options
+---
+
+## Installation
+
+```bash
+yarn add react-native-deepgram
+# or
+npm install react-native-deepgram
+```
+
+### Bare React Native
+
+```bash
+cd ios && pod install
+```
+
+The package is autolinked through `react-native.config.js` β no manual `MainApplication` edits required.
+
+### Expo (managed or bare)
-You can customise the iOS prompt via the `microphonePermission` option:
+Add the plugin to your config:
```js
// app.config.js
@@ -99,7 +197,8 @@ module.exports = {
'react-native-deepgram',
{
microphonePermission:
- 'Allow $(PRODUCT_NAME) to capture audio for real-time transcription.',
+ 'Allow $(PRODUCT_NAME) to access your microphone.',
+ // backgroundAudio: true, // default β see plugin docs below
},
],
],
@@ -107,35 +206,109 @@ module.exports = {
};
```
-> π§ Need the plugin in a bare React Native project? Import it via
-> `require('react-native-deepgram/app.plugin.js')` in your config plugin pipeline.
+Then prebuild and run:
+
+```bash
+npx expo prebuild
+npx expo run:ios # or expo run:android
+```
+
+> Already running an existing development build? Re-run `expo prebuild` (and `pod install` on iOS) so the new permissions, background modes, and Android foreground service are picked up.
---
-## Configuration
+## Core concepts
+
+A few ideas you only have to learn once.
+
+### 1. Configure once, use anywhere
```ts
import { configure } from 'react-native-deepgram';
+configure({ apiKey: process.env.EXPO_PUBLIC_DEEPGRAM_API_KEY! });
+```
+
+`configure` is global β call it at the entry point (e.g. `App.tsx`). Every hook reads from this config. The Management API additionally needs a key with management scopes.
+
+> **Never ship raw API keys.** Use `EXPO_PUBLIC_*` env vars, secrets storage, or a backend proxy that mints scoped keys.
-configure({ apiKey: 'YOUR_DEEPGRAM_API_KEY' });
+### 2. Hooks return reactive state β opt in
+
+Every hook supports `track*` flags that turn on a reactive return value. Without them, the hook stays event-driven (callbacks only) so it never causes unnecessary re-renders.
+
+```ts
+const { transcript, state } = useDeepgramSpeechToText({
+ trackTranscript: true, // exposes `transcript` + `interimTranscript`
+ trackState: true, // exposes `state.status`, `state.error`
+});
```
-> **Headsβup π** The Management API needs a key with management scopes.
-> Do not ship production keys in source controlβprefer environment variables, Expo secrets, or a backend proxy.
+| Hook | Tracking flags |
+| --- | --- |
+| `useDeepgramVoiceAgent` | `trackState`, `trackConversation`, `trackAgentStatus` |
+| `useDeepgramSpeechToText` | `trackState`, `trackTranscript` |
+| `useDeepgramTextToSpeech` | `trackState` |
+| `useDeepgramTextIntelligence` | `trackState` |
+
+### 3. Audio session is automatic
+
+You don't manage `AVAudioSession`, `AudioManager`, or focus. The native module activates the right session category, switches to `VoiceChat` / `MODE_IN_COMMUNICATION` when echo cancellation is enabled, requests/abandons audio focus, starts/stops a foreground service on Android, and rebuilds after `mediaserverd` resets on iOS.
+
+For tuning details, see [Audio session, AEC & background](#audio-session-aec--background).
+
+### 4. Hooks at a glance
+
+| Hook | Purpose | Endpoint |
+| --- | --- | --- |
+| `useDeepgramVoiceAgent` | Conversational agents (full-duplex) | `wss://agent.deepgram.com/v1/agent/converse` |
+| `useDeepgramSpeechToText` | Live STT + file transcription | `wss://api.deepgram.com/v1/listen` (v1) / `/v2/listen` (Flux) |
+| `useDeepgramTextToSpeech` | TTS via REST or streaming | `https://api.deepgram.com/v1/speak` & `wss://api.deepgram.com/v1/speak` |
+| `useDeepgramTextIntelligence` | Summaries, topics, intents, sentiment | `https://api.deepgram.com/v1/read` |
+| `useDeepgramManagement` | Projects / keys / usage REST API | `https://api.deepgram.com/v1` |
---
-## Usage overview
+## Audio session, AEC & background
+
+This section covers what the native module does for you so you can debug or tune behaviour with confidence.
+
+### Hardware echo cancellation (full-duplex)
+
+When you build a Voice Agent (or any duplex flow where Deepgram is speaking *and* the mic is open), set `enableVoiceProcessing: true`:
+
+```ts
+import { Deepgram } from 'react-native-deepgram';
+
+await Deepgram.startRecording({ enableVoiceProcessing: true });
+```
+
+`useDeepgramVoiceAgent` opts into this automatically. What this gives you:
+
+| Platform | Behaviour |
+| --- | --- |
+| iOS | Capture is routed through `AVAudioEngine.inputNode` with `setVoiceProcessingEnabled:YES` on both input and output, and the audio session uses `AVAudioSessionModeVoiceChat`. This engages Apple's Voice-Processing I/O Audio Unit β the only way to get hardware AEC on iOS. |
+| Android | Capture uses `MediaRecorder.AudioSource.VOICE_COMMUNICATION` and the system is set to `MODE_IN_COMMUNICATION` so the platform's hardware AEC engages with the active playback signal as its reference. The previous mode is restored on `stopRecording`. |
+
+For pure speech-to-text usage you generally **don't** want this β leave the option off (the default). STT-only paths use `VOICE_RECOGNITION` on Android and a pure AudioQueue capture on iOS so Deepgram receives the rawest possible signal.
+
+### Background audio
-| Hook | Purpose |
-| ----------------------------- | ---------------------------------------------------- |
-| `useDeepgramVoiceAgent` | Build conversational agents with streaming audio I/O |
-| `useDeepgramSpeechToText` | Live microphone streaming and file transcription |
-| `useDeepgramTextToSpeech` | Text-to-Speech synthesis (HTTP + WebSocket streaming) |
-| `useDeepgramTextIntelligence` | Text analysis (summaries, topics, intents, sentiment) |
-| `useDeepgramManagement` | Typed wrapper around the Management REST API |
+Leave `backgroundAudio: true` on the Expo plugin (it's the default) to keep playback and capture alive when the user leaves the app:
-> π‘ **Pro tip**: All hooks now export a `state` object (and other reactive values) so you can easily track connection status, errors, and transcripts without maintaining your own state.
+- **iOS** β `UIBackgroundModes: ["audio"]` is added to your `Info.plist`.
+- **Android** β a foreground service (`DeepgramAudioService`) is bundled with `foregroundServiceType=microphone|mediaPlayback` and the matching `FOREGROUND_SERVICE`, `FOREGROUND_SERVICE_MICROPHONE`, `FOREGROUND_SERVICE_MEDIA_PLAYBACK` permissions are merged into your manifest. The service is started/stopped automatically whenever recording or playback is active.
+
+### Built-in resilience
+
+The native modules handle the awkward edge cases for you:
+
+- **Audio interruptions** β phone calls, Siri, system alerts pause + resume cleanly on iOS.
+- **Route changes** β unplugging headphones or disconnecting Bluetooth pauses playback (instead of suddenly blasting through the speaker), matching Apple's `AVAudioSessionRouteChangeReasonOldDeviceUnavailable` guidance and Android's `ACTION_AUDIO_BECOMING_NOISY` broadcast.
+- **Audio focus loss (Android)** β transient losses pause playback; permanent losses tear down recording + playback so you don't fight other apps for the mic.
+- **mediaserverd resets (iOS)** β if iOS restarts its audio service mid-session, the audio engine is rebuilt from scratch on the next call.
+- **Bluetooth on iOS 17+** β uses `AllowBluetoothHFP` when available, with a deprecation-safe fallback for older toolchains.
+- **iOS 17+ permission API** β `requestMicPermission` calls `AVAudioApplication.requestRecordPermissionWithCompletionHandler` when available, falling back to `AVAudioSession` on older OS versions.
+- **Bounded playback queue (Android)** β if JS feeds audio faster than the device can play it, the oldest chunks are dropped at ~1.5βMB so you never OOM mid-call.
---
@@ -143,7 +316,7 @@ configure({ apiKey: 'YOUR_DEEPGRAM_API_KEY' });
`useDeepgramVoiceAgent` connects to `wss://agent.deepgram.com/v1/agent/converse`, captures microphone audio, and optionally auto-plays the agent's streamed responses. It wraps the full Voice Agent messaging surface so you can react to conversation updates, function calls, warnings, and raw PCM audio.
-> π **Audio Handling**: This hook uses `AVAudioEngine` on iOS for hardware-accelerated echo cancellation, ensuring the agent doesn't hear itself speak. It also manages the audio session automatically.
+> π **Hardware AEC enabled by default.** The hook calls `startRecording({ enableVoiceProcessing: true })`, so iOS routes capture through Apple's VPIO Audio Unit and Android switches to `VOICE_COMMUNICATION` + `MODE_IN_COMMUNICATION`. The result: the agent never hears itself speak, even on speakerphone. (Note: VPIO is not available in the iOS Simulator β test on a physical device.)
### Quick start
@@ -232,62 +405,132 @@ return (
### API reference (Voice Agent)
+**Signature**
+
+```ts
+const {
+ // Connection
+ connect, disconnect, isConnected,
+ // Messaging
+ sendMessage, sendSettings, sendMedia, sendKeepAlive,
+ injectUserMessage, injectAgentMessage, updatePrompt,
+ // Function calls
+ sendFunctionCallResponse,
+ // Reactive state (opt-in via track* flags)
+ state, conversation, agentStatus, clearConversation,
+} = useDeepgramVoiceAgent(props);
+```
+
#### Hook props
-| Prop | Type | Description |
-| ---- | ---- | ----------- |
-| `endpoint` | `string` | WebSocket endpoint used for the agent conversation (defaults to `wss://agent.deepgram.com/v1/agent/converse`). |
-| `defaultSettings` | `DeepgramVoiceAgentSettings` | Base `Settings` payload sent on connect; merge per-call overrides via `connect(override)`. |
-| `autoStartMicrophone` | `boolean` | Automatically requests mic access and starts streaming PCM when `true` (default). |
-| `autoPlayAudio` | `boolean` | Automatically plays received audio using the native player (default: `true`). |
-| `trackState` | `boolean` | Enable reactive state tracking (connection, errors, warnings) via the `state` return value (default: `false`). |
-| `trackConversation` | `boolean` | Enable conversation history tracking via the `conversation` return value (default: `false`). |
-| `trackAgentStatus` | `boolean` | Enable agent status tracking (thinking, latency) via the `agentStatus` return value (default: `false`). |
-| `downsampleFactor` | `number` | Manually override the downsample ratio applied to captured audio (defaults to a heuristic based on the requested sample rate). |
+**Configuration**
+
+| Prop | Type | Default | Description |
+| ---- | ---- | ------- | ----------- |
+| `endpoint` | `string` | `wss://agent.deepgram.com/v1/agent/converse` | WebSocket endpoint used for the agent conversation. |
+| `defaultSettings` | `DeepgramVoiceAgentSettings` | β | Base `Settings` payload sent on connect; merge per-call overrides via `connect(override)`. |
+| `autoStartMicrophone` | `boolean` | `true` | Automatically requests mic access and starts streaming PCM. |
+| `autoPlayAudio` | `boolean` | `true` | Plays received audio using the native player. |
+| `downsampleFactor` | `number` | heuristic | Manually override the downsample ratio applied to captured audio. |
+
+**Reactive tracking flags** (opt-in β extra renders only when set)
+
+| Prop | Default | Enables |
+| ---- | ------- | ------- |
+| `trackState` | `false` | `state` return β connection status, error, warning. |
+| `trackConversation` | `false` | `conversation` return β running history of `{ role, content }`. |
+| `trackAgentStatus` | `false` | `agentStatus` return β thinking flag and latency metrics. |
#### Callbacks
+
+Lifecycle β connect/disconnect/error
+
| Callback | Signature | Fired when |
| -------- | --------- | ---------- |
-| `onBeforeConnect` | `() => void` | `connect` is calledβbefore requesting mic permissions or opening the socket. |
-| `onConnect` | `() => void` | The socket opens and the initial settings payload is delivered. |
-| `onClose` | `(event?: any) => void` | The socket closes (manual disconnect or remote). |
-| `onError` | `(error: unknown) => void` | Any unexpected error occurs (mic, playback, socket send, etc.). |
-| `onMessage` | `(message: DeepgramVoiceAgentServerMessage) => void` | Every JSON message from the Voice Agent API. |
-| `onWelcome` | `(message: DeepgramVoiceAgentWelcomeMessage) => void` | The agent returns the initial `Welcome` envelope. |
+| `onBeforeConnect` | `() => void` | `connect` is called, before mic prompt or socket open. |
+| `onConnect` | `() => void` | Socket opens and the initial settings payload is delivered. |
+| `onClose` | `(event?: any) => void` | Socket closes (manual disconnect or remote). |
+| `onError` | `(error: unknown) => void` | Unexpected client-side error (mic, playback, socket send). |
+| `onServerError` | `(message: DeepgramVoiceAgentErrorMessage) => void` | API reports a structured error (`description` + `code`). |
+| `onWarning` | `(message: DeepgramVoiceAgentWarningMessage) => void` | Non-fatal warning (e.g. degraded audio quality). |
+
+
+
+
+Conversation β text, audio, agent state
+
+| Callback | Signature | Fired when |
+| -------- | --------- | ---------- |
+| `onWelcome` | `(message: DeepgramVoiceAgentWelcomeMessage) => void` | Agent returns the initial `Welcome` envelope. |
| `onSettingsApplied` | `(message: DeepgramVoiceAgentSettingsAppliedMessage) => void` | Settings are acknowledged by the agent. |
-| `onConversationText` | `(message: DeepgramVoiceAgentConversationTextMessage) => void` | Transcript updates (`role` + `content`) arrive. |
-| `onAgentThinking` | `(message: DeepgramVoiceAgentAgentThinkingMessage) => void` | The agent reports internal reasoning state. |
-| `onAgentStartedSpeaking` | `(message: DeepgramVoiceAgentAgentStartedSpeakingMessage) => void` | A response playback session begins (latency metrics included). |
-| `onAgentAudioDone` | `(message: DeepgramVoiceAgentAgentAudioDoneMessage) => void` | The agent finishes emitting audio for a turn. |
+| `onConversationText` | `(message: DeepgramVoiceAgentConversationTextMessage) => void` | Transcript update (`role` + `content`) arrives. |
| `onUserStartedSpeaking` | `(message: DeepgramVoiceAgentUserStartedSpeakingMessage) => void` | Server-side VAD detects the user speaking. |
-| `onFunctionCallRequest` | `(message: DeepgramVoiceAgentFunctionCallRequestMessage) => void` | The agent asks the client to execute a tool marked `client_side: true`. |
-| `onFunctionCallResponse` | `(message: DeepgramVoiceAgentReceiveFunctionCallResponseMessage) => void` | The server shares the outcome of a non-client-side function call. |
-| `onPromptUpdated` | `(message: DeepgramVoiceAgentPromptUpdatedMessage) => void` | The active prompt is updated (e.g., after `updatePrompt`). |
-| `onSpeakUpdated` | `(message: DeepgramVoiceAgentSpeakUpdatedMessage) => void` | The active speak configuration changes (sent by the server). |
-| `onInjectionRefused` | `(message: DeepgramVoiceAgentInjectionRefusedMessage) => void` | An inject request is rejected (typically while the agent is speaking). |
-| `onWarning` | `(message: DeepgramVoiceAgentWarningMessage) => void` | The API surfaces a non-fatal warning (e.g., degraded audio quality). |
-| `onServerError` | `(message: DeepgramVoiceAgentErrorMessage) => void` | The API reports a structured error payload (`description` + `code`). |
+| `onAgentThinking` | `(message: DeepgramVoiceAgentAgentThinkingMessage) => void` | Agent reports internal reasoning state. |
+| `onAgentStartedSpeaking` | `(message: DeepgramVoiceAgentAgentStartedSpeakingMessage) => void` | Response playback session begins (latency metrics included). |
+| `onAgentAudioDone` | `(message: DeepgramVoiceAgentAgentAudioDoneMessage) => void` | Agent finishes emitting audio for a turn. |
+| `onAudio` | `(audioData: ArrayBuffer) => void` | Raw binary audio chunk received from the socket. |
+| `onPromptUpdated` | `(message: DeepgramVoiceAgentPromptUpdatedMessage) => void` | Active prompt is updated (e.g. after `updatePrompt`). |
+| `onSpeakUpdated` | `(message: DeepgramVoiceAgentSpeakUpdatedMessage) => void` | Active speak configuration changes (server-driven). |
+| `onListenUpdated` | `(message: DeepgramVoiceAgentListenUpdatedMessage) => void` | Listen configuration is updated (e.g. after `updateListen`). |
+| `onThinkUpdated` | `(message: DeepgramVoiceAgentThinkUpdatedMessage) => void` | Think configuration is updated (e.g. after `updateThink`). |
+| `onHistory` | `(message: DeepgramVoiceAgentHistoryMessage) => void` | Conversation history (text turn or function calls) is replayed by the server. |
+| `onInjectionRefused` | `(message: DeepgramVoiceAgentInjectionRefusedMessage) => void` | Inject request rejected (typically while the agent is speaking). |
+| `onMessage` | `(message: DeepgramVoiceAgentServerMessage) => void` | Catch-all for every JSON message from the API. |
+
+
+
+
+Function calls β client-side tools
+
+| Callback | Signature | Fired when |
+| -------- | --------- | ---------- |
+| `onFunctionCallRequest` | `(message: DeepgramVoiceAgentFunctionCallRequestMessage) => void` | Agent asks the client to execute a tool marked `client_side: true`. Respond with `sendFunctionCallResponse`. |
+| `onFunctionCallResponse` | `(message: DeepgramVoiceAgentReceiveFunctionCallResponseMessage) => void` | Server shares the outcome of a non-client-side function call. |
+
+
#### Returned methods
+**Connection**
+
| Method | Signature | Description |
| ------ | --------- | ----------- |
-| `connect` | `(settings?: DeepgramVoiceAgentSettings) => Promise` | Opens the socket, optionally merges additional settings, and begins microphone streaming. |
-| `disconnect` | `() => void` | Tears down the socket, stops recording, and removes listeners. |
-| `sendMessage` | `(message: DeepgramVoiceAgentClientMessage) => boolean` | Sends a pre-built client envelope (handy for custom message types). |
-| `sendSettings` | `(settings: DeepgramVoiceAgentSettings) => boolean` | Sends a `Settings` message mid-session (merged with the `type` field). |
-| `injectUserMessage` | `(content: string) => boolean` | Injects a user-side text message. |
-| `injectAgentMessage` | `(message: string) => boolean` | Injects an assistant-side text message. |
-| `sendFunctionCallResponse` | `(response: Omit) => boolean` | Returns tool results for client-side function calls. |
-| `sendKeepAlive` | `() => boolean` | Emits a `KeepAlive` ping to keep the session warm. |
-| `updatePrompt` | `(prompt: string) => boolean` | Replaces the active system prompt. |
-| `sendMedia` | `(chunk: ArrayBuffer \| Uint8Array \| number[]) => boolean` | Streams additional PCM audio to the agent (e.g., pre-recorded buffers). |
+| `connect` | `(settings?: DeepgramVoiceAgentSettings) => Promise` | Opens the socket, optionally merges settings, starts mic streaming. |
+| `disconnect` | `() => void` | Tears down the socket, stops recording, removes listeners. |
| `isConnected` | `() => boolean` | Returns `true` when the socket is open. |
-| `clearConversation` | `() => void` | Clears the internal conversation history. |
-| `state` | `DeepgramVoiceAgentState` | Reactive state object (requires `trackState: true`). |
-| `conversation` | `DeepgramVoiceAgentConversationMessage[]` | Reactive conversation history (requires `trackConversation: true`). |
-| `agentStatus` | `DeepgramVoiceAgentStatus` | Reactive agent status (requires `trackAgentStatus: true`). |
+
+**Messaging**
+
+| Method | Signature | Description |
+| ------ | --------- | ----------- |
+| `sendMessage` | `(message: DeepgramVoiceAgentClientMessage) => boolean` | Send a pre-built client envelope (custom message types). |
+| `sendSettings` | `(settings: DeepgramVoiceAgentSettings) => boolean` | Update settings mid-session. |
+| `sendMedia` | `(chunk: ArrayBuffer \| Uint8Array \| number[]) => boolean` | Stream additional PCM (e.g. pre-recorded buffer). |
+| `sendKeepAlive` | `() => boolean` | Emit a `KeepAlive` ping. |
+| `injectUserMessage` | `(content: string) => boolean` | Inject a user-side text turn. |
+| `injectAgentMessage` | `(message: string, behavior?: string) => boolean` | Inject an assistant-side text turn (optional `behavior`, e.g. `'default'`). |
+| `updatePrompt` | `(prompt: string) => boolean` | Replace the active system prompt. |
+| `updateListen` | `(listen: DeepgramVoiceAgentListenConfig) => boolean` | Swap the speech-recognition (listen) provider mid-session. |
+| `updateThink` | `(think: DeepgramVoiceAgentThinkConfig) => boolean` | Swap the LLM (think) provider mid-session. |
+| `updateSpeak` | `(speak: DeepgramVoiceAgentSpeakConfig) => boolean` | Swap the text-to-speech (speak) provider mid-session. |
+
+**Function calls**
+
+| Method | Signature | Description |
+| ------ | --------- | ----------- |
+| `sendFunctionCallResponse` | `(response: Omit) => boolean` | Return tool results for a client-side function call. |
+
+#### Reactive state
+
+Each value is `undefined` unless its corresponding `track*` flag is `true`.
+
+| Return | Type | Requires |
+| ------ | ---- | -------- |
+| `state` | `{ connectionState: 'idle' \| 'connecting' \| 'connected' \| 'disconnected'; error: string \| null; warning: string \| null }` | `trackState: true` |
+| `conversation` | `Array<{ role: string; content: string }>` | `trackConversation: true` |
+| `agentStatus` | `{ thinking: string \| null; latency: { total?: number; tts?: number; ttt?: number } \| null }` | `trackAgentStatus: true` |
+| `clearConversation` | `() => void` | `trackConversation: true` |
#### Settings payload (`DeepgramVoiceAgentSettings`)
@@ -302,7 +545,7 @@ return (
| `audio.output` | `DeepgramVoiceAgentAudioConfig` | Choose output encoding/sample rate/bitrate for agent speech. |
| `agent.language` | `string` | Primary language for the conversation. |
| `agent.context.messages` | `DeepgramVoiceAgentContextMessage[]` | Seed the conversation with prior turns or system notes. |
-| `agent.listen.provider` | `DeepgramVoiceAgentListenProvider` | Speech recognition provider/model configuration. |
+| `agent.listen.provider` | `DeepgramVoiceAgentListenProvider` | Speech recognition provider/model configuration (Deepgram listen also accepts Flux fields: `version`, `eot_threshold`, `eager_eot_threshold`, `eot_timeout_ms`, `language_hints`). |
| `agent.think.provider` | `DeepgramVoiceAgentThinkProvider` | LLM selection (`type`, `model`, `temperature`, etc.). |
| `agent.think.functions` | `DeepgramVoiceAgentFunctionConfig[]` | Tooling exposed to the agent (name, parameters, optional endpoint metadata). |
| `agent.think.prompt` | `string` | System prompt presented to the thinking provider. |
@@ -316,7 +559,9 @@ return (
## Speech-to-Text (`useDeepgramSpeechToText`)
-The speech hook streams microphone audio using WebSockets and can also transcribe prerecorded audio sources. It defaults to STT v1 but automatically boots into Flux when `apiVersion: 'v2'` is supplied (defaulting the model to `flux-general-en`).
+The speech hook streams microphone audio over WebSocket and also handles prerecorded transcription. It defaults to STT v1 and auto-boots into Flux v2 when `apiVersion: 'v2'` is supplied (model defaults to `flux-general-en`).
+
+> π **Flux v2 envelope handled correctly.** The hook decodes Deepgram's `TurnInfo` events and treats `event === 'EndOfTurn'` as the final flag, so you don't have to reason about the new envelope yourself. `Connected`, `ConfigureSuccess` / `ConfigureFailure`, and `Error` messages are surfaced via callbacks.
### Live streaming quick start
@@ -341,7 +586,7 @@ const {
Transcript: {transcript}