Skip to content

Add an image_generate tool on the engine's own Redrob credential - #69

Closed
flxkm522 wants to merge 1 commit into
developfrom
feature/image-generate-tool
Closed

flxkm522 wants to merge 1 commit into
developfrom
feature/image-generate-tool

Conversation

@flxkm522

@flxkm522 flxkm522 commented Oct 9, 2026

Copy link
Copy Markdown

Option 1 from the media-output plan: image generation runs inside the engine, using the Redrob credential the engine already holds.

Why a built-in tool, not a Cowork plugin

The Redrob key belongs to Redrob Code. It lives in auth.json or REDROB_API_KEY, Cowork's server deliberately never reads it, and no plugin API hands it out. A Cowork plugin could only reach the key in two bad ways:

  • Register a second redrob auth hook. ProviderAuth keeps one hook per provider, so this would replace "Connect Redrob" in onboarding.
  • Read provider.key from an undocumented chat.params argument. That's fragile, and it's handling a secret.

A built-in avoids both. The key stays where it is, and the image is billed to the same account as the chat.

What it does

  • image_generate (src/tool/image_generate.ts):
    • Request: POST {CONSOLE_URL}/chat/completions with modalities: ["image","text"]. Default model is google/gemini-2.5-flash-image; model can be overridden. The gateway relays images since mckinley-and-rice/redrob-console#230.
    • Output: writes the picture to artifacts/<slug>.png (or .jpg/.webp/.gif) and returns it as an attachment (data: URL). Cowork already renders tool attachments inline in chat (session-sync.ts → message-list.tsx), and the model sees what it made.
    • File naming: the name is chosen before anything is spent, and an existing file is never overwritten (logo.png → logo-2.png).
    • Permission: asked as edit, like write, so plan mode denies it.
    • Errors: a gateway refusal comes back with the gateway's own message (e.g. 402 out of credit). A model that answers only in text produces a clear error quoting what it said. With no key, the tool fails before anything is sent.
    • Metadata: carries filepath, model, and the billed costUsd from the gateway's redrob block.
    • Accepted images: only inline PNG, JPEG, WebP and GIF (≤ 20 MiB). Remote URLs and SVG are refused.
  • Registry: the tool is offered only while a Redrob credential exists (auth store or env), checked on every request. Without one, it would be a tool that fails on every call.
  • REDROB_CONSOLE_URL is read per call, the same developer seam CONSOLE_URL already honours. That lets the test point at a local stand-in.

Proof

Command Result
bun test test/tool/image_generate.test.ts 7 pass. Uses a real local HTTP stand-in for the gateway. Covers: bearer key from the auth store; request body (model, modalities, prompt); bytes written to artifacts/; attachment shape; costUsd in metadata; edit permission on the relative path; no overwrite; named model and file name; no-image error; 402 relayed; no key means no request sent; slug and data-URL guards
bun test test/tool/registry.test.ts 16 pass, including the new test that hides the tool without a credential and shows it with an auth-store or env key
bun test (all of packages/redrob) 3454 pass / 3 fail. The 3 failures are tui.test.ts, project.test.ts and write.test.ts "OS denies write access"; they also fail on develop in this sandbox (it runs as root)
bun typecheck (turbo, 18 packages) exit 0, also run by the pre-push hook
prettier --check, oxlint on the changed files clean

Not covered: a live call against console.redrob.ai. Reaching users needs a release and a redrobCodeVersion bump in Cowork.

Redrob Cowork asks for images through the Redrob gateway, but the key that
pays for them belongs to this engine: it lives in the auth store (or
REDROB_API_KEY), the desktop app never reads it, and no plugin API hands
it out. A Cowork plugin could only reach it by registering a second
`redrob` auth hook, which would replace "Connect Redrob" in onboarding, or
by fishing it out of an undocumented hook argument. So the tool is a
built-in, and the key stays where it is.

image_generate posts the prompt to the gateway's /chat/completions with
modalities ["image", "text"] (relayed since console #230), writes the
returned picture under artifacts/, and returns it as an attachment, which
is what puts it inline in the conversation and in front of the model. The
file name is chosen before anything is spent and an existing file is never
overwritten. Writing is asked as `edit`, like `write`, so plan mode denies
it. A gateway refusal (an empty balance, say) and a model that answers in
words alone are reported with the gateway's or the model's own message.

The registry offers the tool only while a Redrob credential is available,
read per request: without one it would be a tool that fails on every call.
flxkm522 added a commit that referenced this pull request Oct 12, 2026
…tored

Squash of the redrob-code ENGINE work that was spread across seven open PRs into a single change on develop. The Electron/desktop app and the build-off/Quality-Lab stack move to the alpha/desktop branch separately and are not part of this.

Rolled up:
- #67 insights route + /v1/chat/completions (serve the OpenAI face for real, forward x-redrob-session to the Console).
- #68 regenerate the JS SDK that had fallen behind the routes.
- #69-#74 the media tools and endpoints: image_generate (edit / reference images), speech_generate (mp3), POST /redrob/speech and POST /redrob/transcribe, all on the engine own Redrob credential.
- #79 the shared on-device labellers (route-labeller, work-labeller) as source workspace packages, consumed downstream by Cowork/Office/Design.

Integration fix found by combining the PRs:
- packages/server/src/routes.ts: the chat-completion handler reads LLMClient inside the request, so it was a per-request requirement of the embedded API that Layer.provide(serviceLayer) did not discharge, leaking as a residual requirement and flipping the web.handler signature so the untouched sdk-next caller stopped type-checking. LLMClient is a global service, so it is now provided once via HttpRouter.provideRequest inside makeRoutes.

Review + lean/documentation pass (behaviour-preserving):
- image_generate / speech_generate: size-limit messages derive from their constants instead of hand-kept 10 MB / 50 MB literals (drift-proof).
- speech_generate: characters() uses [...text].length (idiomatic code-point count).
- core/insights.ts: tightened the ErrorEnvelope describe() union guard (identical runtime behaviour).
- plugin/host.ts: removed a misleading comment on skill.list; it matches the uniform mutable() pattern now.
- docs/LOCAL-ENGINE-API.md: documented the shipped /redrob/speech and /redrob/transcribe endpoints (sections 1c, 1d).

Verified: bun turbo typecheck 20/20; bun install --frozen-lockfile clean; engine tool/server tests 42/42; core 6/6; work-labeller 77/77.
@flxkm522

Copy link
Copy Markdown
Author

Superseded by #80, which consolidates all redrob-code engine PRs into a single squashed PR on develop (reviewed + refactored for lean code and docs). Closing this; its content is included there.

@flxkm522 flxkm522 closed this Oct 12, 2026
flxkm522 added a commit that referenced this pull request Oct 12, 2026
…tored

Squash of the redrob-code ENGINE work that was spread across seven open PRs into a single change on develop. The Electron/desktop app and the build-off/Quality-Lab stack move to the alpha/desktop branch separately and are not part of this.

Rolled up:
- #67 insights route + /v1/chat/completions (serve the OpenAI face for real, forward x-redrob-session to the Console).
- #68 regenerate the JS SDK that had fallen behind the routes.
- #69-#74 the media tools and endpoints: image_generate (edit / reference images), speech_generate (mp3), POST /redrob/speech and POST /redrob/transcribe, all on the engine own Redrob credential.
- #79 the shared on-device labellers (route-labeller, work-labeller) as source workspace packages, consumed downstream by Cowork/Office/Design.

Integration fix found by combining the PRs:
- packages/server/src/routes.ts: the chat-completion handler reads LLMClient inside the request, so it was a per-request requirement of the embedded API that Layer.provide(serviceLayer) did not discharge, leaking as a residual requirement and flipping the web.handler signature so the untouched sdk-next caller stopped type-checking. LLMClient is a global service, so it is now provided once via HttpRouter.provideRequest inside makeRoutes.

Review + lean/documentation pass (behaviour-preserving):
- image_generate / speech_generate: size-limit messages derive from their constants instead of hand-kept 10 MB / 50 MB literals (drift-proof).
- speech_generate: characters() uses [...text].length (idiomatic code-point count).
- core/insights.ts: tightened the ErrorEnvelope describe() union guard (identical runtime behaviour).
- plugin/host.ts: removed a misleading comment on skill.list; it matches the uniform mutable() pattern now.
- docs/LOCAL-ENGINE-API.md: documented the shipped /redrob/speech and /redrob/transcribe endpoints (sections 1c, 1d).

Verified: bun turbo typecheck 20/20; bun install --frozen-lockfile clean; engine tool/server tests 42/42; core 6/6; work-labeller 77/77.
flxkm522 added a commit that referenced this pull request Oct 12, 2026
…cs and compat tests

One change on develop that both consolidates the redrob-code ENGINE work and
documents + pins the on-device labelling it introduces. The Electron/desktop app
and the build-off / Quality-Lab stack are the separate alpha/desktop drafts and are
NOT part of this.

== Engine consolidation (was seven PRs) ==
- #67 insights route + /v1/chat/completions (serve the OpenAI face for real; forward
  x-redrob-session to the Console).
- #68 regenerate the JS SDK that had fallen behind the routes.
- #69-#74 the media tools and endpoints: image_generate (edit / reference images),
  speech_generate (mp3), POST /redrob/speech and POST /redrob/transcribe, all on the
  engine own Redrob credential.
- #79 the shared on-device labellers (route-labeller, work-labeller) as source
  workspace packages, consumed downstream by Cowork/Office/Design.

Integration fix: the chat-completion handler reads LLMClient inside the request, so it
was a per-request requirement of the embedded API that Layer.provide(serviceLayer) did
not discharge, leaking as a residual requirement and breaking the untouched sdk-next
caller. LLMClient is a global service, so it is now provided once via
HttpRouter.provideRequest inside makeRoutes (packages/server/src/routes.ts).

Lean/doc pass (behaviour-preserving): size-limit messages derive from their constants;
speech_generate characters() uses [...text].length; the insights describe() union guard
is tightened; a misleading plugin/host comment removed; /redrob/speech + /redrob/transcribe
documented in LOCAL-ENGINE-API.md; and the httpapi route-coverage harness gained scenarios
for the two /redrob/* routes.

== On-device labelling: docs + compatibility (answering the CTO review) ==
- docs/ON-DEVICE-LABELLING.md (new): the single reference for WHY there is encoder code
  (we wrap pinned off-the-shelf models -- distiluse, multilingual-e5-base -- not our own
  encoders, for on-device privacy + one implementation across products), WHERE the tensors
  live (only the small trained layers are ours and in git: work-head.json 159KB,
  prototypes.json 232KB; base weights are hash-pinned downloads), the model-free-CI vs
  opt-in-eval story, and the backwards-compat model for third-party clients. Includes the
  pending checklist to move the e5 release asset from redrob-cowork to redrob-code (SHA
  pins unchanged; the url is not flipped now because a not-yet-uploaded release would 404).
- httpapi-chat-completion.test.ts: two new v1-contract cases -- the newer optional fields
  (max_completion_tokens, reasoning_effort, user) stay accepted, and unknown request fields
  (a Console-style redrob block, a bogus key) are IGNORED not rejected. The second pins the
  exact forward/third-party-compat guarantee and fails CI if the schema rejects excess props.
- test.yml: the labellers job comment now spells out that the unit suites are model-free and
  CI-safe, and the real ~140MB/~278MB encoders are needed ONLY by the opt-in eval/model dev
  commands, which stay off the PR path.

Verified: bun turbo typecheck 20/20; bun install --frozen-lockfile clean; engine tool/server
tests + the 7 chat-completion contract cases pass; core 6/6; work-labeller 77/77.
its-janghoon added a commit that referenced this pull request Oct 12, 2026
…tes (#85)

image_generate and speech_generate on the engine's Redrob credential, registered only when one is connected. POST /redrob/speech and /redrob/transcribe relay to the Console audio API without the app holding the key; 503 before anything is sent when there is no key, 400 for input the gateway would refuse. From the closed #69, #71, #73, #74; image editing (#70) left out.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants