feat: add Gradium speech-to-speech translation extension - #2292
Open
TiagoAgora wants to merge 6 commits into
Open
feat: add Gradium speech-to-speech translation extension#2292TiagoAgora wants to merge 6 commits into
TiagoAgora wants to merge 6 commits into
Conversation
added 5 commits
August 20, 2026 14:44
…xtension Adds a new mllm extension wiring Gradium's real-time speech-to-speech translation API into TEN, following the same AsyncMLLMBaseExtension contract as openai_mllm_python/glm_mllm_python. Auth, host pattern, and message protocol reuse what's confirmed by the existing gradium_asr_python and gradium_tts_python extensions; the combined s2s endpoint path is not yet confirmed and is documented as such in the extension README. Wires the extension into voice-assistant-realtime as a second predefined graph (gradium_translate_demo, auto_start: false) alongside the existing default graph.
…ed protocol Pratim (Gradium) confirmed the real /api/speech/s2s protocol, fixing a few assumptions from the initial scaffold: - model_name is "s2s-translate", paired with stt_model_name: "stt-translate" and tts_model_name: "default" (not the placeholder "default"/empty values used before) - target_language is not a top-level setup field, it nests inside json_config - vad is not part of this protocol at all (only ready/audio/text/ end_of_stream/error) -- removed the dead handling - text messages carry translated output only, confirming no source-language transcript is available from this endpoint - voice_id must belong to target_language; since there's no voice catalog to validate against, on_init now raises if it's unset instead of defaulting to a voice borrowed from the (possibly different-language) TTS demo
…r handling Adds a tests/ suite mirroring gradium_tts_python's pattern: a real TEN runtime drives the actual extension lifecycle via AsyncExtensionTester, with only GradiumS2SClient mocked. Covers session-ready plus translated text/audio routing, server-side error propagation, connect failures, and missing api_key/voice_id. test_config.py separately unit-tests GradiumMLLMConfig (no TEN runtime needed), guarding the json_config nesting for target_language. Writing the missing-config tests surfaced that on_init raised directly on missing api_key/voice_id, which would crash the extension process instead of reporting a clean error Data -- inconsistent with gradium_tts_python's own on_init pattern. Fixed to catch and report via send_mllm_error, and added a guard in start_connection() so it doesn't attempt a connection with already-known-invalid config.
…ning the tests Running the Tier 2 suite for real (in the TEN dev container, against the real ten_runtime -- not just syntax/logic checks) surfaced a genuine architectural bug: AsyncMLLMBaseExtension.on_start() does `await self.start_connection()` directly. start_connection() looped forever internally (async for message in self.client.messages()), so on_start() never completed, and the framework could never advance the extension to on_stop() -- any clean shutdown attempt hung indefinitely. Fixed by matching gradium_asr_python's proven pattern: start_connection() now connects, sends session-ready, then hands the message loop off to a background asyncio.Task (_receive_loop) and returns immediately. stop_connection() cancels that task before closing the client. Also fixes the mock client in gradium_mocks.py, whose messages() generator blocked on an event that close() never set -- ties close() to the event the generator waits on, mirroring how closing a real websocket ends the async for loop. Verified: all 11 tests pass via `task test-extension EXTENSION=agents/ten_packages/extension/gradium_mllm_python` in the dev container.
…sh voice Pratim confirmed both existing Gradium voice IDs already used in this repo are English: YTpq7expH9539ERJ (Emma) and cLONiZ4hQ8VpQ4Sz (Skyler). Sets Emma as the default voice_id, matching the default target_language: "en", so the extension and demo graph work out of the box for an English demo instead of requiring manual voice_id setup first. Still overridable (env var default in the demo graph) for other target_language values -- voice_id and target_language must be changed together.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gradium_mllm_python, a new extension implementing theAsyncMLLMBaseExtensioncontract (same asopenai_mllm_python/glm_mllm_python/etc.) to integrate Gradium's real-time speech-to-speech Translation APIvoice-assistant-realtimeas a second predefined graph (gradium_translate_demo,auto_start: false) alongside the existing default graph, so it doesn't disturb the current demojson_confignesting fortarget_language) confirmed directly against Gradium and verified against their live APItests/) mirroringgradium_tts_python's pattern: a real TEN runtime drives the extension lifecycle with only the Gradium websocket client mockedAsyncMLLMBaseExtension.on_start()awaitsstart_connection()directly, so astart_connection()that blocks for the life of the connection (as this and sibling MLLM extensions did) prevents the framework from ever advancing toon_stop(). Fixed by connecting, then handing the receive loop off to a background task.Test plan
task test-extension EXTENSION=agents/ten_packages/extension/gradium_mllm_python— 11/11 tests passwss://us.api.gradium.ai/api/speech/s2s) — session handshake succeeds with real credentialsmain_controlbun/QEMU emulation incompatibility unrelated to this extension; not yet tested with a real human speaker)