Repository navigation
Image and speech tools, and /redrob/speech and /redrob/transcribe routes - #85
Merged
Merged
Conversation
Redrob Cowork asks for images through the Redrob gateway, but the key that pays for them belongs to this engine: it lives in the auth store (or REDROB_API_KEY), the desktop app never reads it, and no plugin API hands it out. A Cowork plugin could only reach it by registering a second `redrob` auth hook, which would replace "Connect Redrob" in onboarding, or by fishing it out of an undocumented hook argument. So the tool is a built-in, and the key stays where it is. image_generate posts the prompt to the gateway's /chat/completions with modalities ["image", "text"] (relayed since console #230), writes the returned picture under artifacts/, and returns it as an attachment, which is what puts it inline in the conversation and in front of the model. The file name is chosen before anything is spent and an existing file is never overwritten. Writing is asked as `edit`, like `write`, so plan mode denies it. A gateway refusal (an empty balance, say) and a model that answers in words alone are reported with the gateway's or the model's own message. The registry offers the tool only while a Redrob credential is available, read per request: without one it would be a tool that fails on every call.
The agent could write a script but not voice it. speech_generate sends text to the Redrob gateway's POST /v1/audio/speech on the engine's own credential, for the reason image_generate gives, and saves the mp3 under artifacts/. The default model is multilingual, so a Korean script is spoken as Korean. The audio is saved, not attached. An attachment goes back to the model with the next turn, and audio input would steer auto to an audio-input model and bill the clip as input on every later turn, to tell the model nothing: it wrote the words. Everything that can be refused for free is refused before the request: empty text, more than the gateway's 4096 code points, or no credential. The name is chosen first and an existing file is never overwritten, writing is asked as edit, a gateway refusal is relayed with its own message, and an answer that is not audio is not saved. The registry offers it under the same credential rule as image_generate.
An edit to the expected character count duplicated one line and dropped the model assertion. The block now checks filepath, model, the code-point count and the cost once each.
…he key Redrob Cowork's read-aloud has to play audio in the app, and the Redrob key that pays for it belongs to this engine. The route speaks text through the gateway on the engine's own credential and answers with the mp3, so the key never leaves this process. It shares one path with speech_generate: prepareSpeech validates and picks the model and voice, speechKey finds the credential, and requestSpeech makes the one gateway call and checks the answer is audio. Each fails with a typed SpeechError, which the route maps to its declared errors: invalid text is a 400, no credential a 503 (the engine lacks it, not the caller), a gateway refusal or non-audio answer a 502 carrying the gateway's message, and a timeout a 504. The first two are found before anything is sent. The tool's behaviour is unchanged.
POST /redrob/speech exists for audio a client plays at once: a reply read aloud and the voice conversation's answer, where the wait before the first word is what the user feels. It now defaults to eleven-flash-v2.5 with the same voice, sarah. speech_generate keeps eleven-multilingual-v2 for saved files, and a caller can still name any model.
… Redrob key Redrob Cowork's push-to-talk needs a transcript without holding the key, as read-aloud needed audio. The route takes base64 audio, its format and an optional language and model, refuses anything the gateway would refuse before sending it, and relays /v1/audio/transcriptions.
Both routes refuse with 503 before anything is sent when no Redrob key is connected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The media half of the engine work, re-cut from the closed #69, #71, #73 and #74 onto current
develop, one commit each, plus harness coverage.image_generateandspeech_generatetools, running on the Redrob credential the engine already holds.speech_generatesaves an mp3 underartifacts/(not attached, so the clip is not billed as input on every later turn).POST /redrob/speech(mp3 bytes) andPOST /redrob/transcribe({ text, seconds?, costUsd? }), so an app can play and dictate speech without ever holding the key. They relay to the Console audio API that landed in redrob-console#253.503 service unavailable: redrobbefore anything is sent when there is no key, and400for input the gateway would refuse (empty text, bad base64, unsupported format, over the size cap). Nothing fails silently.Not in this PR
image_generateediting and reference images (#70): nobody asked for it yet. The on-device labellers (#79) and the consolidated #80.Tested
bun turbo typecheck --concurrency=2: 18/18.packages/redrob:image_generate,speech_generate,registry,httpapi-speech,httpapi-transcribetests, 37 pass.httpapi-exercisein all three modes: 219 pass, 0 missing. The two new routes are exercised and refuse with 503 in the key-less exercise home.