Real-inference model gateway: whisper.cpp, Python Whisper, and OpenAI backends - #2
Real-inference model gateway: whisper.cpp, Python Whisper, and OpenAI backends#2dedmonwalkin wants to merge 1 commit into
Conversation
New model-gateway/ service implementing the HTTP contract both accessibility-lite and accessibility-broadcast already speak (MODEL_HTTP_BASE_URL) — run one gateway, both products get real transcription. Gateway - Backends: whispercpp (local, static binary), python-whisper (pip/brew openai-whisper), openai (API), mock (labelled, confidence 0) - Only transcription runs a model; perception/accessibility endpoints are structural transforms of caller data, scene stays empty — the gateway never fabricates content. Every response tagged _backend. - Confidence semantics per backend: mean token p (whisper.cpp), exp(mean avg_logprob) (python/OpenAI), 0 (mock) - Optional GATEWAY_API_KEY bearer auth (timing-safe), 32MB body cap, /v1/translate proxies to LIBRETRANSLATE_URL or returns 501 - Dockerfile: multi-stage static whisper.cpp build with ggml model baked in, non-root user App integration - MODEL_HTTP_ALLOW_PRIVATE opt-in lets the SSRF guard reach an operator-configured gateway on a private network (docker-compose); scoped to MODEL_HTTP_BASE_URL, never user-supplied URLs - docker-compose: model-gateway service wired in, no host port exposure - npm run start:gateway; README + .env.example updated Verified end to end on real audio: synthesized council-meeting speech → upload → ffmpeg chunking → HTTP provider → gateway (python-whisper) → word-for-word correct WebVTT captions. 9 new gateway tests; suite now 84 passing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 645ce8ef12
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return sendJson(res, 404, { error: 'Not found' }); | ||
| } | ||
| } catch (err) { | ||
| const status = err.statusCode || (err instanceof SyntaxError ? 400 : 500); |
There was a problem hiding this comment.
When a valid request reaches a backend that then throws a SyntaxError while parsing its own JSON output (for example malformed whisper.cpp output, an invalid OpenAI JSON response, or a bad LibreTranslate response), this maps it to HTTP 400 even though the client body was not invalid. That misreports real inference/upstream failures as bad requests, so clients and logs cannot distinguish operator/backend failures from caller mistakes; only JSON body parsing should set an explicit 400 status.
Useful? React with 👍 / 👎.
1 participant
Summary
Stacked on #1. Replaces the mock-only inference story with a real, self-contained model gateway — the first milestone of the local-government demo plan.
Both accessibility-lite and accessibility-broadcast already call
MODEL_HTTP_BASE_URLwith the same contract, so this one service gives both products real transcription.What's here
model-gateway/— zero-dependency Node service with four backends selected byWHISPER_BACKEND:whispercpp— local static binary, audio never leaves the machine (the public-sector privacy story)python-whisper— the referenceopenai-whisperCLI (pip/brew), zero build stepopenai— hosted API, pay per minutemock— clearly labelled placeholder, confidence pinned to0sceneis always empty (no fabricated visual descriptions); every response carries_backendso mock output can never pass as realMODEL_HTTP_ALLOW_PRIVATE=truelets the guard reach the operator-configured gateway on the compose-internal network — scoped toMODEL_HTTP_BASE_URLonly, never user-supplied URLsGATEWAY_API_KEYbearer auth (timing-safe compare), 32MB body cap,/v1/translateproxy to LibreTranslate or honest501Verification
End-to-end on real audio: synthesized council-meeting speech → upload → ffmpeg chunk → HTTP provider → gateway (
python-whisper) → WebVTT:Word-for-word correct, confidence 0.84.
🤖 Generated with Claude Code