URL ingestion: pull municipal video archives by link - #3
Conversation
POST /v1/jobs/from-url accepts a public URL, returns 202 with status "downloading", and the client polls GET /v1/jobs/:id until "pending". Direct media links (.mp4 .webm .mov .mp3 .wav .ogg .m4a) - Streamed fetch with MAX_UPLOAD_MB cap and INGEST_TIMEOUT_MS timeout - Final post-redirect URL re-validated against SSRF rules before the body is read - Magic-byte verification against the extension-derived MIME type (reuses upload.js magicBytesMatch) Platform URLs (YouTube, Granicus, CivicPlus, ...) - Delegated to yt-dlp when installed (--no-playlist, --max-filesize, 720p cap, mp4 remux, arg-injection guarded with --); clear 501 when yt-dlp is absent - Title extracted from --write-info-json - Added yt-dlp to the app Dockerfile (alpine package) Security - User-supplied URLs get full SSRF validation (assertSafeUrl, no private-network exception): cloud-metadata IPs, localhost, RFC1918, and non-http protocols are all refused with 400 UI - "Or paste a video URL" form on the upload page with polling status, aria-live announcements, and redirect to the options page when ready Failed downloads set status "error" and remove the job dir (orphan sweep covers crashes). Verified end to end: W3C Sintel trailer ingested by URL, processed through the python-whisper gateway, dialogue transcribed correctly to WebVTT. 5 new endpoint tests (missing url, bad protocol, metadata IP, localhost, RFC1918); suite now 89 passing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f9844dcd1a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const controller = new AbortController(); | ||
| const timer = setTimeout(() => controller.abort(), config.ingestTimeoutMs); | ||
| try { | ||
| const res = await fetch(url, { signal: controller.signal, redirect: 'follow' }); |
There was a problem hiding this comment.
When a public direct-media URL returns a redirect to a private or metadata address, fetch(..., redirect: 'follow') follows that redirect and sends the request before the later assertSafeUrl(res.url) runs, so the intended SSRF guard is applied only after the internal service has already been contacted. This affects any .mp4/.mp3/etc. URL that can be made to redirect; disable automatic redirects and validate each Location before issuing the next request.
Useful? React with 👍 / 👎.
1 participant
Summary
Stacked on #2 (which is stacked on #1). Second milestone of the local-government demo plan: instead of manual upload only, point the app at a video URL and it pulls the file itself — the intake path for municipal meeting archives.
What's here
POST /v1/jobs/from-url— accepts{ url }, returns202with a job in statusdownloading; pollGET /v1/jobs/:iduntilpending, then process as usual.mp4 .webm .mov .mp3 .wav .ogg .m4a) — streamed native download,MAX_UPLOAD_MBcap enforced mid-stream, magic-byte verification against the extension-derived MIME typeyt-dlpwhen installed (--no-playlist, size cap, 720p sort, mp4 remux,--guard against argument injection); honest501when it isn't. Added to the app Dockerfile.aria-livestatus polling, then redirect into the existing options flowVerification
Real end-to-end run: W3C Sintel trailer (4.4MB mp4) ingested by URL, probed at 52s, processed through the
python-whispergateway from #2:Correct dialogue, correct timing.
Merge order
#1 → #2 → this. GitHub retargets each PR automatically as its base merges.
🤖 Generated with Claude Code