feat(cli): add vlmrun gateway / gw CLI for the OpenAI-compatible model gateway

[spillai]

Adds vlmrun gateway (aliased vlmrun gw), a CLI for the OpenAI-compatible VLM Run gateway — OCR, VQA, embeddings and transcription against third-party models, authenticating with the same VLMRUN_API_KEY as the rest of the SDK.

vlmrun gw health                     Check gateway liveness
vlmrun gw models [<model>]           List models, or detail one model's methods + examples
vlmrun gw chat <files> -m <model>    Run OCR / VLM models over documents and images
vlmrun gw embed <files|-t> -m <model>  Embed text, images or video
vlmrun gw transcribe <audio> -m <model>  Transcribe audio, or a video's audio track

Also adds client.gateway.completions, .embeddings and .transcriptions to the Python SDK.

Verified against the live gateway

Every model and method in the catalog, run end-to-end through the CLI on real files:

Model	Command	Result
paddleocr/pp-ocrv6	gw chat --method ocr <image>.jpg *	✅ 2350 chars — text+score+poly JSON
paddleocr/pp-ocrv6	gw chat --method detect <image>.jpg	✅ 1485 chars — poly only, no text
paddleocr/pp-ocrv6	gw chat --method markdown <image>.jpg	✅ 500 chars — reading-order text
paddleocr/pp-ocrv6	gw chat --method ocr --method-params '{"lang":"en","score_threshold":0.5}' <image>.jpg	✅ threshold sweep 13→12→1→0 regions
zai-org/glm-ocr	gw chat --method ocr <image>.jpg *	✅ 2508 chars
zai-org/glm-ocr	gw chat --method markdown <image>.jpg	✅ 2355 chars
zai-org/glm-ocr	gw chat --method markdown <document>.pdf	✅ 3069 chars, 4 pages, streams per page
rednote-hilab/dots.mocr	gw chat --method parse_layout <image>.jpg *	✅ 1670 chars — bbox + category
rednote-hilab/dots.mocr	gw chat --method parse_layout_only <image>.jpg	✅ 1063 chars — boxes only
rednote-hilab/dots.mocr	gw chat --method ocr <image>.jpg	✅ 694 chars
rednote-hilab/dots.mocr	gw chat --method markdown <image>.jpg	✅ 668 chars
qwen/qwen3.5-0.8b	gw chat -p "what car is this?" <image>.jpg *	✅ VQA answer
qwen/qwen3-vl-embedding-2b	gw embed -t "a blue parrot"	✅ 1 vec, dim 2048
qwen/qwen3-vl-embedding-2b	gw embed <image>.jpg	✅ 1 vec, dim 2048
qwen/qwen3-vl-embedding-2b	gw embed <image>.jpg <image2>.jpg -t "caption"	✅ 3 vecs
qwen/qwen3-vl-embedding-2b	gw embed --join -t "caption" <image>.jpg	✅ 1 joint vec
qwen/qwen3-vl-embedding-2b	gw embed --dimensions 64 -t "hi"	✅ dim 64
qwen/qwen3-vl-embedding-2b	gw embed <video>.mp4	⚠️ returns a vector, but constant across clips — server-side no-op
nvidia/parakeet-tdt-0.6b-v3	gw transcribe -f json/text/verbose_json <audio>.mp3	✅ 493–494 chars
nvidia/parakeet-tdt-0.6b-v3	gw transcribe -f srt <audio>.mp3 / -f vtt	✅ 526 / 532 chars — real timestamps
nvidia/parakeet-tdt-0.6b-v3	gw transcribe <video>.mp4	✅ 380 chars — audio track
nvidia/parakeet-tdt-0.6b-v3	gw transcribe --language en --prompt "..." <audio>.wav	✅ 489 chars

* = model's default method. Every row green except the video-embedding caveat, which is server-side.

Cross-modal retrieval works as expected: cos(parrot image, "a colorful parrot bird") = 0.708 vs cos(pcb image, same text) = 0.146.

Implementation notes

Each of these looks arbitrary in the diff but was forced by observed gateway behaviour. They're the parts most worth reviewing.

Images are sent as image_url, not file_url. file_url is routed through the gateway's document/PDF path: it 400s outright on some images and returns less content when it does work (2350 vs 645 chars on the same file). Confirmed it's the content-part type, not the model — with file_url the same image 400s on all three OCR models; with image_url all three succeed. file_url remains the fallback for anything unidentifiable.

MIME is sniffed from magic bytes, not the extension. Extensions lie — one of our demo files is a WebP named .jpg. Trusting it emitted data:image/jpeg;base64,<webp bytes>; the gateway believes the declared type and misroutes the file. Falls back to the extension when the content is unrecognized.

chat streams only when a document is present. The gateway streams document requests one SSE chunk per page. Image- and text-only requests return a plain chat.completion body still labelled Content-Type: text/event-stream, so an SSE reader finds no events and yields empty content — a silent blank response. Verified directly against the API with no proxy in the path.

Gateway-only fields ride in extra_body. method, method_params, document_dpi, image_resolution are not OpenAI parameters, and create() has an explicit signature that raises TypeError on unknown kwargs. --extra splits keys by introspecting the installed SDK's signature, so it tracks openai upgrades instead of hardcoding a list.

OCR output renders as plain text, not Markdown. Responses are wrapped in <document>/<page> tags; Rich's Markdown renderer treats them as HTML and drops the body entirely, blanking the panel. Payloads starting with markup or JSON render via Text.

gw embed --join is explicit. Models embed at most one image per vector — [[img, img]] returns a 500. Every file and -t is its own vector by default; --join combines them and rejects 2+ files client-side rather than sending a request that 500s.

gw models shows methods, not pricing. The catalog returns no pricing fields at all, so those columns could only ever render -. It now surfaces task and methods (default marked *), and gw models <model> details params and copy-pasteable example commands parsed from the live extra_body_help.

gw chat surfaces usage.cost. The gateway returns a per-request USD cost that was being discarded; it now shows in the panel footer (… · $0.001508 · 1.35s) and under usage.cost in --json. Sub-cent costs print without scientific notation and never collapse a real cost to $0.

Errors dressed as success now fail loudly. An {"error": ...} response body (e.g. an unknown --method, which the gateway returns on a 200) is detected and surfaced as a real error with a non-zero exit, instead of a success-styled panel. Detection is limited to a lone error key, so normal {"text": ...} / <document> output is unaffected.

gw chat --response-format (JSON mode). Accepts text, json_object, or a full JSON object (e.g. json_schema) and sends it as the top-level response_format field. The gateway does not honor it yet — accepted and currently ignored server-side — so it's wired now to test once support lands. Distinct from --json, which controls the CLI's own output.

Notes for the gateway / docs team

Found while testing; all server- or docs-side, none fixable from the client:

• The embeddings docs' multimodal example doesn't work. It shows input: [{"type":"image_url",...}], which 422s. The union is list[union[str, list[EmbeddingContentPart]]] — parts nest one level deeper: input: [[{...}]].
• More than one image in a single joint item 500s ([[img, img]] → Internal server error; [[img, text]] is fine).
• Video embedding is a no-op — two different clips return byte-identical vectors (cosine 1.000000) at 1 token. Supported here, but the help says so plainly rather than letting anyone build a broken index on it.
• A streaming request that would 400 returns 200 with an empty body instead of the error, silently converting failures into blank output.
• An invalid --method returns an {"error": ...} body on a 200, not a 4xx. The CLI now detects and surfaces this (see above), but a proper status code would be cleaner.
• Large PDFs 413. gw chat inlines the file as base64 in the request body, so a PDF over ~20MB hits 413 Request Entity Too Large (a raw HTML error). Batch-OCRing a 26/39/58MB drawing set required splitting each into page-range chunks client-side. A large-file upload path (Files API) or server-side size handling would remove that workaround.

Response format & CLI output

--response-format (the model's output constraint) and --json (the CLI's own stdout format) are deliberately separate flags. usage.cost is surfaced in both the panel and --json.

Tests

93 gateway tests. Full suite 437 passed / 26 skipped, ruff clean. Fakes mirror real API payloads — FakeModel matches an actual /models response (no invented pricing field), and the completions fake asserts on extra_body/response_format rather than accepting any kwarg. Every new test was checked to fail against the pre-change source.

Version bumped 0.6.5 → 0.7.0: minor rather than patch, since this adds a new CLI command group and two SDK resources, following #180 (0.5.12 → 0.6.0 for the execute/executions commands).

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces an OpenAI-compatible model gateway resource, exposing third-party OCR and vision-language models through both the SDK (client.gateway) and the CLI (vlmrun gw). It includes commands for health checks, model listing, chat/OCR completions, embeddings, and audio transcriptions, along with comprehensive test coverage. The feedback highlights opportunities to improve error handling by raising a user-friendly DependencyError when the openai package is missing, validate file types in the embedding endpoint to prevent gateway failures, respect user-configured custom timeouts instead of silently overriding them, and align model IDs in the README with the CLI help strings.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

If a user passes a non-image/non-video file (such as a PDF or text document) to the embed command, it will fall back to treating it as an image_url with an incorrect MIME type, which will fail on the gateway. Consider validating that the file is indeed an image or video and raising a clean error message.

    if mime.startswith("video/"):
        key = "video_url"
    elif mime.startswith("image/"):
        key = "image_url"
    else:
        console.print(
            f"[red]Error:[/] Unsupported file type '{mime}' for {path.name}. "
            "Embedding models only support images and videos."
        )
        raise typer.Exit(1)
    return {"type": key, key: {"url": f"data:{mime};base64,{b64}"}}

[spillai]

Resolved in 1fa3bce. gw embed now rejects any non-image/non-video file client-side with a clear message ("images and video only; use --text for text") instead of sending a mislabelled image_url the gateway 400s on. Verified live: a PDF is rejected before any request. Covered by test_embed_rejects_non_image_file.

[gemini-code-assist]

The README still references the non-existent model IDs paddle-ocrv6 and qwen3.6-0.8b in the gateway examples, even though the CLI help strings were updated to use the correct IDs (pp-ocrv6 and qwen/qwen3.5-0.8b). Let's update the README to be consistent.

[spillai]

Fixed in 1fa3bce. Rewrote the whole README gateway section: corrected the model ids (paddleocr/pp-ocrv6, qwen/qwen3.5-0.8b), and it now uses full <org>/<name> ids throughout.

[devin-ai-integration]

Devin Review found 2 potential issues.

[devin-ai-integration]

🟡 Documentation examples use model names that don't exist, so copy-pasted commands fail

The gateway examples reference the models paddle-ocrv6 and qwen3.6-0.8b (README vlmrun/cli/README.md:246-252), which are the exact identifiers this change states do not exist (the real ones are pp-ocrv6 and qwen/qwen3.5-0.8b), so anyone copy-pasting these commands hits a "model not found" error.
Impact: Users following the documentation run commands that fail immediately against the gateway.

Stale model identifiers in newly added docs

The PR description explicitly lists these as wrong ("paddle-ocrv6 and qwen3.6-0.8b don't exist (pp-ocrv6, qwen/qwen3.5-0.8b)") and the CLI help strings in vlmrun/cli/_cli/gateway.py:52-56 use the corrected ids, but this README section (added in the same change) and the prose at vlmrun/cli/README.md:225 were not updated. AGENTS.md requires docs to be kept in sync with the implementation.

Suggested change

	vlmrun gw chat doc1.pdf doc2.pdf -m paddle-ocrv6
	# OCR an image
	vlmrun gw chat scan.jpg -m paddle-ocrv6
	# Prompt a model that supports text input
	vlmrun gw chat image.jpg -p "describe this image" -m qwen3.6-0.8b
	vlmrun gw chat doc1.pdf doc2.pdf -m pp-ocrv6
	# OCR an image
	vlmrun gw chat scan.jpg -m pp-ocrv6
	# Prompt a model that supports text input
	vlmrun gw chat image.jpg -p "describe this image" -m qwen/qwen3.5-0.8b

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[spillai]

Fixed in 1fa3bce — the stale paddle-ocrv6/qwen3.6-0.8b ids and the prose at line 225 are corrected; the section now uses full <org>/<name> ids.

[devin-ai-integration]

🟡 Documentation claims the model listing shows pricing that the tool no longer displays

The docs describe the models listing as showing pricing per 1M tokens (vlmrun/cli/README.md:238 and vlmrun/cli/README.md:261), but the implementation deliberately removed the pricing columns and now shows only model/task/methods, so the described output does not match what users see.
Impact: Readers are told to expect pricing information from a command that never prints it.

Docs describe removed pricing columns and old file_url routing

The models command in vlmrun/cli/_cli/gateway.py:446-475 builds a table with MODEL/TASK/METHODS columns and no pricing. The PR description confirms pricing columns were removed because "the API returns no pricing fields at all." Additionally vlmrun/cli/README.md:230-231 still says images are sent as file_url parts, whereas the code now routes images through image_url (vlmrun/cli/_cli/gateway.py:123-138). AGENTS.md requires documentation to be kept in sync with the implementation.

Prompt for agents

The README gateway section is out of sync with the implementation in vlmrun/cli/_cli/gateway.py. Two issues: (1) Lines 238 and 261 describe the `vlmrun gw models` command as listing pricing ("pricing ($ per 1M tokens)" and "input/output pricing"), but the models command no longer prints pricing columns — it shows MODEL/TASK/METHODS and supports detailing a single model. Update these to describe methods/task listing and the per-model detail view instead. (2) Lines 230-231 state images are sent as `file_url` content parts, but the code now sends images as `image_url` (documents as `document_url`, `file_url` only as fallback). Update the prose to match.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[spillai]

Fixed in 1fa3bce. Dropped the pricing claim (the API returns no pricing; gw models shows task + methods and details one model), and corrected the file_url prose to image_url for images with file_url as fallback. Also documented embed/transcribe/the gateway alias.