diff --git a/CHANGELOG.md b/CHANGELOG.md index a47194f..a8761e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,11 +9,64 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **Recording to file.** `useDeepgramSpeechToText` now supports `recordToFile` + option to persist microphone audio to a WAV file while simultaneously streaming + to Deepgram. The resulting `file://` URI is delivered via `onRecordingComplete` + callback and `recordingUri` state property once `stopListening` completes. + Recording format is uncompressed WAV (16 kHz PCM16 mono), mirroring the audio + streamed to Deepgram. +- **Audio-level metering.** Added microphone audio-level (RMS amplitude 0..1) + support via `metering` configuration in `useDeepgramSpeechToText`. When enabled, + the native module emits normalized audio levels at configurable intervals, + surfaced through `audioLevel` state and `onAudioLevel` callback. Useful for + implementing visual feedback like waveforms or voice activity indicators. +- **Auto-reconnect for live streaming.** Introduced `DeepgramReconnectOptions` for + automatic reconnection on unexpected socket close in Speech-to-Text and Voice + Agent. Configure max retries, initial/max backoff delays via `reconnect` prop. + Includes `onReconnecting` and `onReconnected` callbacks for monitoring reconnect + attempts. +- **Pause/resume streaming.** `useDeepgramSpeechToText` now exposes `pause()` and + `resume()` methods to temporarily stop forwarding microphone frames without + tearing down the WebSocket. Sends `Finalize` once (v1) to flush buffered audio + and starts periodic `KeepAlive` to maintain the connection during pause. State + tracked via `isPaused` property when `trackState` is enabled. +- **Mute/unmute for Voice Agent.** `useDeepgramVoiceAgent` now supports + `setMuted(boolean)` to control microphone input while maintaining an active + session. Keep-alive messages are sent while muted to prevent timeout. +- **Text-to-Speech `synthesizeToBytes` method.** Added method to fetch synthesized + audio as raw bytes (`ArrayBuffer`) with MIME type without automatically playing + it, enabling audio caching/persistence. Results are cached in-memory (LRU) keyed + by text + format to avoid duplicate network requests for identical prompts. + Accepts optional per-call `options` to override HTTP format. +- **Token-based authentication.** Introduced global `__DEEPGRAM_GET_TOKEN__` + function alongside existing `__DEEPGRAM_API_KEY__` for dynamic token + provisioning. The SDK automatically caches tokens with TTL-aware refresh and + deduplicates concurrent token requests. New `clearCachedAuthToken()` helper + forces fresh token retrieval. See `src/helpers/auth.ts`. +- **Audio device enumeration and selection.** Added `getAudioDevices()` and + `selectAudioDevice(deviceId)` methods for managing audio output devices, + including Bluetooth headsets and built-in options. Event listeners notify of + audio device changes. Requires Bluetooth permissions on Android 12+. + _(Note: audio route management methods were later removed in a refactor.)_ +- **Structured error handling.** Introduced `DeepgramError` class with typed + `DeepgramErrorCode` for consistent error handling across all hooks. Native + rejections now use stable error codes (`permission_denied`, `init_failed`, + `start_error`, `stop_error`, `audio_start_error`, `audio_stop_error`, + `stop_player_error`, `invalid_data`, `playback_error`) so consumers can branch + programmatically instead of string-matching messages. +- **Configuration overrides.** New `configure()` function accepts `baseUrl`, + `websocketUrl`, and `agentUrl` overrides for custom endpoints or proxying. + Hooks dynamically resolve URLs based on configuration. - **Live `measurements` option.** `DeepgramLiveListenOptions` now supports the `measurements` flag, mirroring the existing prerecorded option. When enabled it is serialized into the live/streaming query (`measurements=true`) for both the v1 and v2 (Flux) realtime paths, converting spoken measurements into abbreviated forms during live transcription. +- **iOS audio session management.** Comprehensive audio session lifecycle + management including activation, deactivation, configuration for recording and + playback, handling for audio route changes, media services resets, and audio + interruptions. Enhanced playback with AVAudioPlayerNode for TTS and Voice Agent + with echo cancellation support when voice processing is enabled. ### Fixed diff --git a/README.md b/README.md index a0a7938..5f78d95 100644 --- a/README.md +++ b/README.md @@ -14,12 +14,12 @@ yarn add react-native-deepgram | | | | --- | --- | | πŸ—£οΈ **Voice Agent** | Full-duplex agents over Deepgram's WebSocket. Hardware AEC on iOS (VPIO) and Android (`VOICE_COMMUNICATION` + `MODE_IN_COMMUNICATION`). Function calling, prompt updates, latency telemetry. | -| πŸŽ™οΈ **Speech-to-Text** | Live PCM streaming over WebSocket on STT v1 *or* Flux v2 (`flux-general-en`). File transcription with summarisation, topics, intents, entities. | +| πŸŽ™οΈ **Speech-to-Text** | Live PCM streaming over WebSocket on STT v1 *or* Flux v2 (`flux-general-en`). File transcription with summarisation, topics, intents, entities. Optionally record the mic to a WAV file while you stream. | | πŸ”Š **Text-to-Speech** | One-shot HTTP synthesis or low-latency WebSocket streaming with `Speak` / `Flush` / `Clear` / `Close` controls. | | 🧠 **Text Intelligence** | Summaries, topics, intents, sentiment over text or URLs. | | πŸ› οΈ **Management API** | Typed wrapper for projects, models, keys, usage, balances, members, invitations, scopes, purchases, and temporary tokens. | | βš™οΈ **Expo plugin** | One-line install β€” handles permissions, background-audio modes, and Android foreground service automatically. | -| 🧰 **Resilient by default** | Audio interruptions, route changes, headphone unplugging, mediaserverd resets, audio-focus loss, and queue overflow are all handled. | +| 🧰 **Resilient by default** | Audio interruptions, route changes, headphone unplugging, mediaserverd resets, audio-focus loss, and queue overflow are all handled β€” plus opt-in WebSocket auto-reconnect with exponential backoff. | | πŸ†• **Modern iOS APIs** | `AVAudioApplication.requestRecordPermission` and `AllowBluetoothHFP` on iOS 17+, with safe fallbacks. | --- @@ -232,6 +232,51 @@ configure({ apiKey: process.env.EXPO_PUBLIC_DEEPGRAM_API_KEY! }); > **Never ship raw API keys.** Use `EXPO_PUBLIC_*` env vars, secrets storage, or a backend proxy that mints scoped keys. +#### Custom endpoints (regional, Dedicated & self-hosted) + +Point the SDK at a non-default Deepgram deployment by passing base URLs to `configure`. Every override is optional and falls back to the public Deepgram endpoints. + +```ts +configure({ + apiKey: process.env.EXPO_PUBLIC_DEEPGRAM_API_KEY!, + baseUrl: 'https://api.beta.deepgram.com/v1', // REST (must include the version segment) + baseWss: 'wss://api.beta.deepgram.com/v1', // live STT + streaming TTS sockets + agentUrl: 'wss://agent.deepgram.com/v1/agent/converse', // Voice Agent socket +}); +``` + +| Option | Applies to | Default | +| ------ | ---------- | ------- | +| `baseUrl` | REST calls (file STT, TTS, Text Intelligence, Management) | `https://api.deepgram.com/v1` | +| `baseWss` | Live STT + streaming TTS sockets | `wss://api.deepgram.com/v1` | +| `agentUrl` | Voice Agent socket | `wss://agent.deepgram.com/v1/agent/converse` | + +`baseUrl`/`baseWss` **must include the version segment** (`/v1`). Flux v2 endpoints are derived automatically by swapping the trailing `/v1` for `/v2`. See Deepgram's [custom endpoints reference](https://developers.deepgram.com/reference/custom-endpoints). + +#### Ephemeral / scoped tokens (keep your API key off-device) + +Instead of shipping a long-lived `apiKey`, you can hand `configure` a `getToken` provider that returns a short-lived Deepgram token minted by your backend (which proxies Deepgram's [`/v1/auth/grant`](https://developers.deepgram.com/reference/token-based-auth-api/grant-token)). The SDK caches the token, refreshes it before it expires, and de-duplicates concurrent refreshes β€” so a burst of requests never triggers multiple grants. + +```ts +configure({ + // No raw apiKey on the device. `getToken` takes precedence when provided. + getToken: async () => { + const res = await fetch('https://your-backend.example.com/deepgram-token'); + const { access_token, expires_in } = await res.json(); + return { token: access_token, expiresInSeconds: expires_in }; + }, +}); +``` + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `token` | `string` | The short-lived token. Sent as `Authorization: Bearer `. | +| `expiresInSeconds` | `number` (optional) | TTL hint used to refresh ahead of expiry. Defaults to `30`. | + +- When both `getToken` and `apiKey` are set, `getToken` wins; the API key is the fallback. +- The token only needs to be valid at connection time β€” live STT / Voice Agent sockets keep running after it expires. +- **Exception:** the Management API (`useDeepgramManagement`) cannot authenticate with temporary tokens, so it always uses the configured `apiKey`. + ### 2. Hooks return reactive state β€” opt in Every hook supports `track*` flags that turn on a reactive return value. Without them, the hook stays event-driven (callbacks only) so it never causes unnecessary re-renders. @@ -266,6 +311,26 @@ For tuning details, see [Audio session, AEC & background](#audio-session-aec--ba | `useDeepgramTextIntelligence` | Summaries, topics, intents, sentiment | `https://api.deepgram.com/v1/read` | | `useDeepgramManagement` | Projects / keys / usage REST API | `https://api.deepgram.com/v1` | +### 5. Typed errors + +Every error the hooks surface β€” mic-permission denials, native start/stop failures, fetch and WebSocket errors β€” is normalized to a `DeepgramError` with a typed `code`, so you can branch on a stable enum instead of string-matching messages. The original failure is preserved on `cause`. + +```ts +import { DeepgramError, type DeepgramErrorCode } from 'react-native-deepgram'; + +useDeepgramSpeechToText({ + onError: (err) => { + if (err instanceof DeepgramError && err.code === 'permission_denied') { + promptForMicAccess(); + } + }, +}); +``` + +The `code` is one of the native reject codes (or `'unknown'` when it can't be mapped): + +`permission_denied` Β· `init_failed` Β· `start_error` Β· `stop_error` Β· `audio_start_error` Β· `audio_stop_error` Β· `stop_player_error` Β· `invalid_data` Β· `playback_error` + --- ## Audio session, AEC & background @@ -411,6 +476,8 @@ return ( const { // Connection connect, disconnect, isConnected, + // Mic control + mute, unmute, isMuted, // Messaging sendMessage, sendSettings, sendMedia, sendKeepAlive, injectUserMessage, injectAgentMessage, updatePrompt, @@ -432,6 +499,7 @@ const { | `autoStartMicrophone` | `boolean` | `true` | Automatically requests mic access and starts streaming PCM. | | `autoPlayAudio` | `boolean` | `true` | Plays received audio using the native player. | | `downsampleFactor` | `number` | heuristic | Manually override the downsample ratio applied to captured audio. | +| `reconnect` | `DeepgramReconnectOptions` | `{ enabled: false }` | Auto-reconnect config for the agent socket. The stored `Settings` payload is re-sent on every successful reconnect. | **Reactive tracking flags** (opt-in β€” extra renders only when set) @@ -452,6 +520,8 @@ const { | `onConnect` | `() => void` | Socket opens and the initial settings payload is delivered. | | `onClose` | `(event?: any) => void` | Socket closes (manual disconnect or remote). | | `onError` | `(error: unknown) => void` | Unexpected client-side error (mic, playback, socket send). | +| `onReconnecting` | `(attempt: number) => void` | A reconnect attempt begins (1-based attempt number). Requires `reconnect.enabled`. | +| `onReconnected` | `() => void` | The socket reconnects and the stored settings are re-sent. | | `onServerError` | `(message: DeepgramVoiceAgentErrorMessage) => void` | API reports a structured error (`description` + `code`). | | `onWarning` | `(message: DeepgramVoiceAgentWarningMessage) => void` | Non-fatal warning (e.g. degraded audio quality). | @@ -499,6 +569,8 @@ const { | `connect` | `(settings?: DeepgramVoiceAgentSettings) => Promise` | Opens the socket, optionally merges settings, starts mic streaming. | | `disconnect` | `() => void` | Tears down the socket, stops recording, removes listeners. | | `isConnected` | `() => boolean` | Returns `true` when the socket is open. | +| `mute` | `() => void` | Stops forwarding mic audio while keeping the socket alive with periodic `KeepAlive` frames. | +| `unmute` | `() => void` | Resumes forwarding microphone audio. | **Messaging** @@ -528,6 +600,7 @@ Each value is `undefined` unless its corresponding `track*` flag is `true`. | Return | Type | Requires | | ------ | ---- | -------- | | `state` | `{ connectionState: 'idle' \| 'connecting' \| 'connected' \| 'disconnected'; error: string \| null; warning: string \| null }` | `trackState: true` | +| `isMuted` | `boolean` | `trackState: true` | | `conversation` | `Array<{ role: string; content: string }>` | `trackConversation: true` | | `agentStatus` | `{ thinking: string \| null; latency: { total?: number; tts?: number; ttt?: number } \| null }` | `trackAgentStatus: true` | | `clearConversation` | `() => void` | `trackConversation: true` | @@ -619,11 +692,11 @@ const pickFile = async () => { ```ts const { // Live streaming - startListening, stopListening, + startListening, stopListening, pause, resume, // File transcription transcribeFile, // Reactive state (opt-in) - state, transcript, interimTranscript, + state, transcript, interimTranscript, isPaused, audioLevel, recordingUri, } = useDeepgramSpeechToText(props); ``` @@ -635,6 +708,9 @@ const { | ---- | ---- | ----------- | | `live` | `DeepgramLiveListenOptions` | Default options merged into every live stream. | | `prerecorded` | `DeepgramPrerecordedOptions` | Default options merged into every file transcription. | +| `reconnect` | `DeepgramReconnectOptions` | Auto-reconnect config for the live socket. Disabled unless `reconnect.enabled` is `true`. | +| `metering` | `{ enabled?: boolean; intervalMs?: number }` | Microphone audio-level (VU meter) events. Disabled by default; set `enabled: true` to emit a normalized RMS level (~10 Hz, tune with `intervalMs`). | +| `recordToFile` | `{ enabled?: boolean; path?: string; format?: 'wav' }` | Persist the captured mic audio to a WAV file while it streams. Disabled by default. Omit `path` to let the native module pick an app-specific location. | **Live streaming callbacks** @@ -644,7 +720,11 @@ const { | `onStart` | `() => void` | The WebSocket opens. | | `onTranscript` | `(transcript: string, event?: DeepgramTranscriptEvent) => void` | Every transcript update (partial and final). | | `onError` | `(error: unknown) => void` | A streaming error occurs. | -| `onEnd` | `() => void` | The socket closes. | +| `onAudioLevel` | `(level: number) => void` | A new mic audio level (`0..1` normalized RMS) is available. Requires `metering.enabled`. | +| `onRecordingComplete` | `(uri: string) => void` | A `recordToFile` session finished; receives the `file://` URI of the saved WAV. Requires `recordToFile.enabled`. | +| `onReconnecting` | `(attempt: number) => void` | A reconnect attempt begins (1-based attempt number). Requires `reconnect.enabled`. | +| `onReconnected` | `() => void` | The live socket successfully reconnects. | +| `onEnd` | `() => void` | The socket closes (manual stop, or after reconnect attempts are exhausted). | **File transcription callbacks** @@ -667,6 +747,8 @@ const { | ------ | --------- | ----------- | | `startListening` | `(options?: DeepgramLiveListenOptions) => Promise` | Requests mic access, starts recording, streams audio to Deepgram. | | `stopListening` | `() => void` | Stops recording and closes the active WebSocket. | +| `pause` | `() => void` | Pauses mic streaming while keeping the socket alive via `KeepAlive`. On v1 a `Finalize` is sent first to flush buffered audio. | +| `resume` | `() => void` | Resumes mic streaming after a `pause`. | | `transcribeFile` | `(file: DeepgramPrerecordedSource, options?: DeepgramPrerecordedOptions) => Promise` | Uploads a file/URI/URL and resolves via the success/error callbacks. | #### Reactive state @@ -674,6 +756,9 @@ const { | Return | Type | Requires | | ------ | ---- | -------- | | `state` | `{ status: 'idle' \| 'loading' \| 'listening' \| 'transcribing' \| 'error'; error: Error \| null }` | `trackState: true` | +| `isPaused` | `boolean` | `trackState: true` | +| `audioLevel` | `number` (0..1 normalized RMS) | `trackState: true` + `metering.enabled` | +| `recordingUri` | `string` (last saved `file://` WAV) | `trackState: true` + `recordToFile.enabled` | | `transcript` | `string` | `trackTranscript: true` | | `interimTranscript` | `string` | `trackTranscript: true` | @@ -926,7 +1011,7 @@ closeStreamGracefully(); ```ts const { // HTTP synthesis - synthesize, + synthesize, synthesizeToBytes, // Streaming startStreaming, sendText, sendMessage, flushStream, clearStream, @@ -975,6 +1060,7 @@ const { | Method | Signature | Description | | ------ | --------- | ----------- | | `synthesize` | `(text: string) => Promise` | Single-shot REST call; resolves with the full audio buffer. | +| `synthesizeToBytes` | `(text: string, options?: DeepgramTextToSpeechHttpOptions) => Promise<{ data: ArrayBuffer; mimeType: string }>` | Like `synthesize` but **does not play** the audio β€” returns the raw bytes + MIME type for saving/caching. Results are cached in-memory (LRU) keyed by text + format, so repeated identical prompts skip the network. Pass `options` to override the format per call. | **Streaming** @@ -1277,6 +1363,54 @@ const stt = useDeepgramSpeechToText({ }); ``` +### Pause and resume a live stream + +```tsx +const { startListening, pause, resume, isPaused } = useDeepgramSpeechToText({ + trackState: true, + live: { punctuate: true, interimResults: true }, +}); + +// Stop forwarding mic audio without dropping the socket. Deepgram is kept warm +// with periodic KeepAlive frames, so resume() continues the same session. +