feat(api): add text-to-speech endpoints backed by Grok TTS - #5791
gooduru-vineeth wants to merge 1 commit into
Conversation
Adds speech synthesis to the proxy so TTS traffic runs through the same credential pool, round-robin, cooldown and usage accounting as chat, images and videos. Two routes, both landing on xAI's POST /v1/tts: - POST /v1/tts (+ GET /v1/tts/voices) — xAI-native passthrough. Primary, because optimize_streaming_latency, text_normalization, replace, with_timestamps, sample_rate and bit_rate have no OpenAI analogue. - POST /v1/audio/speech — OpenAI-compatible façade. Maps input→text, voice→voice_id, response_format→output_format.codec, clamps speed from OpenAI's 0.25-4.0 into xAI's 0.7-1.5, defaults the xAI-required language to "auto", and drops the model field (xAI /v1/tts has none). opus/aac/flac are rejected rather than silently substituted. Real OpenAI platform keys are supported through the existing openai-compatibility provider via a new per-model `speech:` flag. The ChatGPT Codex OAuth backend exposes no synthesis endpoint, so it is not wired up. Follows the openai-image/openai-video precedent: a new "openai-audio" pseudo-format bypasses protocol translation, so internal/translator is untouched. Routing uses a synthetic built-in model id `grok-tts`, which is never forwarded upstream and is rejected on chat routes with a clear error. Three details specific to this being the first binary-body endpoint: - The non-streaming keep-alive writes newlines into the response body, which is invisible ahead of JSON but corrupts audio, so the speech path does not use it. - Upstream headers are dropped unless passthrough-headers is enabled, so Content-Type is derived from the requested codec. - Request logging records a short marker instead of megabytes of audio. Speech also uses the official API base and headers rather than the chat ones: an OAuth credential routed through the chat path would be pinned to the Grok CLI chat proxy, which has no /tts, and the 404 would cool down the whole xAI auth pool. Also fixes buildXAIConfigModels to re-apply WithXAIBuiltins. A configured xai-api-key models[] list previously shadowed the built-in endpoint-only model ids, which already silently removed grok-imagine-image and grok-imagine-video for those operators. Streaming (OpenAI stream_format=sse and the xAI TTS websocket) is out of scope; stream_format=sse returns a clear 400 rather than a body that is not SSE. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019xcgdPAwqg42fDoVsJ6Kxp
|
This pull request targeted The base branch has been automatically changed to |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e6a1fa29b7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| writeSpeechError(c, http.StatusBadRequest, errMessage) | ||
| return | ||
| } | ||
| h.collectSpeech(c, xaiReq, xaiSpeechModel, contentType) |
There was a problem hiding this comment.
When force-model-prefix: true is used with an xAI credential prefix such as the documented xai, the registry contains only xai/grok-tts. This branch accepts that model but replaces it with unprefixed grok-tts before provider resolution, so every such request fails as an unknown model; the native TTS branch performs the same replacement, and the voices endpoint always uses the unprefixed ID. Pass the requested routing model through instead of discarding its prefix.
Useful? React with 👍 / 👎.
| prefix = strings.ToLower(strings.TrimSpace(prefix)) | ||
| return prefix == "" || prefix == "xai" || prefix == "x-ai" || prefix == "grok" |
There was a problem hiding this comment.
xAI credential prefixes are operator-defined, so the registry can advertise an ID such as team/grok-tts; however, this whitelist accepts only xai, x-ai, and grok. Clients using the advertised prefixed ID therefore receive a 400, and with force-model-prefix: true plus any other prefix there is no usable speech model at all. Validate the base model together with its registered xAI provider rather than hard-coding prefix names.
Useful? React with 👍 / 👎.
| if len(text) > xaiSpeechMaxTextLength { | ||
| return nil, "", fmt.Sprintf("Invalid request: input exceeds the %d character limit", xaiSpeechMaxTextLength) |
There was a problem hiding this comment.
Go's len(string) counts UTF-8 bytes, but this check reports and enforces a 15,000-character limit. Consequently, valid multilingual input is rejected early—for example, 6,000 CJK characters are only 6,000 characters but typically exceed 15,000 bytes. The native TTS validator repeats the same check, so both routes should count Unicode characters rather than bytes.
Useful? React with 👍 / 👎.
| if err != nil { | ||
| return resp, err | ||
| } | ||
| applyXAIHeaders(httpReq, auth, token, false, "") |
There was a problem hiding this comment.
When a request-after-auth interceptor adds or replaces headers—for example, to inject a gateway-specific per-request credential—the updated values reside in opts.Headers, but executeSpeech does not receive opts and calls applyXAIHeaders without them. Those headers are therefore silently absent from speech requests even though the xAI image, video, and chat paths forward them, causing affected upstreams to reject the request.
Useful? React with 👍 / 👎.
2 participants
Adds speech synthesis to the proxy so TTS traffic runs through the same credential pool, round-robin, cooldown and usage accounting as chat, images and videos.
Two routes, both landing on xAI's POST /v1/tts:
Real OpenAI platform keys are supported through the existing openai-compatibility provider via a new per-model
speech:flag. The ChatGPT Codex OAuth backend exposes no synthesis endpoint, so it is not wired up.Follows the openai-image/openai-video precedent: a new "openai-audio" pseudo-format bypasses protocol translation, so internal/translator is untouched. Routing uses a synthetic built-in model id
grok-tts, which is never forwarded upstream and is rejected on chat routes with a clear error.Three details specific to this being the first binary-body endpoint:
Speech also uses the official API base and headers rather than the chat ones: an OAuth credential routed through the chat path would be pinned to the Grok CLI chat proxy, which has no /tts, and the 404 would cool down the whole xAI auth pool.
Also fixes buildXAIConfigModels to re-apply WithXAIBuiltins. A configured xai-api-key models[] list previously shadowed the built-in endpoint-only model ids, which already silently removed grok-imagine-image and grok-imagine-video for those operators.
Streaming (OpenAI stream_format=sse and the xAI TTS websocket) is out of scope; stream_format=sse returns a clear 400 rather than a body that is not SSE.
Claude-Session: https://claude.ai/code/session_019xcgdPAwqg42fDoVsJ6Kxp