Skip to content

feat: add speko asr and tts extensions - #2291

Open
Baymurat785 wants to merge 4 commits into
TEN-framework:mainfrom
SpekoAI:feat/speko-extensions
Open

feat: add speko asr and tts extensions#2291
Baymurat785 wants to merge 4 commits into
TEN-framework:mainfrom
SpekoAI:feat/speko-extensions

Conversation

@Baymurat785

Copy link
Copy Markdown

Adds speko_asr_python and speko_tts_python. Speko (https://speko.ai) is a model router for voice AI: one API key, and STT/TTS requests are routed to the best-benchmarked provider for the language, with failover between providers before the first byte. Discussed with Patrick Ferriter at Agora before opening this.

The ASR extension speaks the router's streaming WebSocket (bare s16le PCM in, interim and final transcripts out). The TTS extension extends tts2_http and streams chunked PCM from POST /v1/audio/speech/stream; output is always 24 kHz mono s16le regardless of the serving provider, so failover never changes the audio format. The x-route response header (which provider served the request) is surfaced in TTFB metric metadata.

Both ship standalone pytest suites (mocked, ASR 8/8 and TTS 10/10 in the ten_agent_build container), guarder configs, an example graph (voice_assistant_speko), and an .env.example entry. Modeled on smallest_asr_python and rime_http_tts. Guarder runs against the live API: TTS 15 passed / 2 skipped, ASR 9 passed / 2 skipped / 1 deselected. pylint 10.00/10 on both.

Two registrations in the guarder harnesses, following existing precedent: speko_tts_python in the fixed-sample-rate list (the router normalizes all output to 24 kHz), and speko_asr_python in _EXTENSIONS_USE_SPANISH (the router currently enables en/ar/de/es/fr/hi/nb/ta/te, so the multi-language test uses es instead of zh; property_es.json and property_hi.json are included).

One design note: the router has no flush-in-place frame, so finalize sends {"type":"end"} and the extension reconnects for the next turn, with audio buffered client-side across the gap. A flush primitive is on our roadmap and the extension will pick it up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant