feat: add speko asr and tts extensions - #2291
Open
Baymurat785 wants to merge 4 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
speko_asr_pythonandspeko_tts_python. Speko (https://speko.ai) is a model router for voice AI: one API key, and STT/TTS requests are routed to the best-benchmarked provider for the language, with failover between providers before the first byte. Discussed with Patrick Ferriter at Agora before opening this.The ASR extension speaks the router's streaming WebSocket (bare s16le PCM in, interim and final transcripts out). The TTS extension extends tts2_http and streams chunked PCM from
POST /v1/audio/speech/stream; output is always 24 kHz mono s16le regardless of the serving provider, so failover never changes the audio format. Thex-routeresponse header (which provider served the request) is surfaced in TTFB metric metadata.Both ship standalone pytest suites (mocked, ASR 8/8 and TTS 10/10 in the ten_agent_build container), guarder configs, an example graph (
voice_assistant_speko), and an.env.exampleentry. Modeled onsmallest_asr_pythonandrime_http_tts. Guarder runs against the live API: TTS 15 passed / 2 skipped, ASR 9 passed / 2 skipped / 1 deselected. pylint 10.00/10 on both.Two registrations in the guarder harnesses, following existing precedent:
speko_tts_pythonin the fixed-sample-rate list (the router normalizes all output to 24 kHz), andspeko_asr_pythonin_EXTENSIONS_USE_SPANISH(the router currently enables en/ar/de/es/fr/hi/nb/ta/te, so the multi-language test uses es instead of zh;property_es.jsonandproperty_hi.jsonare included).One design note: the router has no flush-in-place frame, so finalize sends
{"type":"end"}and the extension reconnects for the next turn, with audio buffered client-side across the gap. A flush primitive is on our roadmap and the extension will pick it up.