Stream Sentry

Real-Time Live Stream Highlight Clipper, Creator-Controlled Distribution & Content Protection Platform

Stream Sentry solves a critical problem for content creators: during long live broadcasts on Twitch, Kick, YouTube Live, or Rumble, iconic viral moments occur in real time. Manually downloading, editing, and publishing Shorts takes 12–24 hours post-stream. Meanwhile, unauthorized viewers capture raw screen recordings and publish clips across TikTok and X before the broadcaster even finishes streaming, capturing viral attention and infringing intellectual property.

Stream Sentry buffers the live stream, detects viral moments via multimodal telemetry (chat velocity, audio energy, and real-time speech context), automatically crops to vertical 9:16, generates animated karaoke subtitles, and presents clips for explicit creator approval within 60–90 seconds of the event.

Once approved, clips publish instantly to connected creator accounts (YouTube Shorts, TikTok, X), while perceptual video/audio fingerprints (vPDQ, Chromaprint) are registered to detect and flag unauthorized re-uploads online.

---

Table of Contents

• Implemented Capabilities
• System Architecture
• How It Works: 90-Second Latency Pipeline
• Getting Started
• Repository Structure
• Platform Matrix & Access Capabilities
• Security & Compliance Guardrails
• Project Status
• Community
• License

---

Implemented Capabilities

Area	Capability	Notes
File-to-Clip Pipeline	Signed multipart upload → magic-byte/probe validation → deterministic EDL trim → frame-accurate 1080×1920 H.264/AAC render → automated QC → download	Media work always offloaded to the background render worker; corrupt outputs never reach ready state
Shorts Review & Editor	Tenant-scoped review inbox, source claiming, TR/EN word captions (four styles, five layouts), tracked crops, hooks, zoom, verified fonts/logos, opt-in music	Immutable revisions with strong ETag conflict handling; any edit immediately invalidates the rendered final
Live Ingest & Buffer	Authenticated MediaMTX publishing, fenced FFmpeg ingest, disk ring buffer, manual markers, verified private candidate sources	Reuses the same render/QC pipeline as VOD
Viral Moment Analysis	Shared highlight/transcript contracts, receive-only Twitch EventSub adapter, event-time mapping, robust chat features, deterministic candidate fusion and boundary clamping	Python faster-whisper ASR service with word-level timestamps, bridged over HTTP
Approval & Publishing	Explicit approval binding artifact SHA-256, saved revision, account, metadata and privacy for 24 hours; supersession on any change; bounded notifications inbox with authenticated SSE	Provider disabled by default (PUBLISHING_PROVIDER=disabled); resumable uploads reconcile unknown outcomes instead of blind re-upload
Multi-Tenancy & RBAC	PostgreSQL Row-Level Security, tenant-scoped memberships, explicit publish_approver authorization gate, cross-tenant isolation (403/404 barriers)	Ownership alone never grants publishing approval
Credential Vaulting	AES-256-GCM envelope encryption with opaque vault:// references, automatic log redaction, method-bound S3 Signature V4 URLs (TTL ≤ 900 s)	Raw credentials never appear in API payloads or logs
Studio Capacity & Usage Billing	Durable PostgreSQL usage ledger, budget thresholds (80% warning / 100% block), reservation-based concurrency admission, signature-verified fail-closed billing webhook	Usage never derived from worker heartbeats or wall-clock timers
Platform Connectors	Protocol units for Kick, YouTube Live chat, Rumble, TikTok, Meta Reels and X (chunked media, consent flows, bounded polling)	Fixture-tested protocol library; runtime providers stay disabled pending platform review
Content Protection	Suspicious-file intake, versioned fingerprint extraction, binary hash-band retrieval, temporal similarity review, scoped account allowlists, signed evidence packages (ZIP/JSON/Markdown)	Experimental algorithm; a match is never treated as an infringement decision and no automated takedown is sent
Contract Governance	Canonical JSON Schemas with synchronized Zod (TypeScript) and Pydantic (Python) models plus shared test fixtures	Validated across language boundaries on every change

---

System Architecture

Stream Sentry employs an event-driven modular architecture separating lightweight TypeScript orchestration from intensive media decoding and Python AI inference:

flowchart TB
    subgraph Frontend ["Creator Dashboard & Mobile Review (React / TypeScript)"]
        UI["Timeline Review Inbox · Safe-Area Crop Editor · Explicit Approval Gate · Telemetry"]
    end

    subgraph ControlPlane ["Control Plane (Node.js / Fastify)"]
        API["REST & SSE Gateway · Auth & Multi-Tenancy (RLS) · Source Capability Resolver"]
        DB[("PostgreSQL 16 + pgvector<br/>Tenant Schema, Outbox, Ledger")]
        REDIS["Redis 7 + BullMQ<br/>Priority Queues & Stream Manifests"]
        OUTBOX["Transactional Outbox Dispatcher"]
    end

    subgraph IngestLayer ["Live Ingest & Buffer Engine"]
        ING["MediaMTX / FFmpeg Ingest Worker<br/>Authenticated RTMP / SRT Direct Feeds"]
        RING["NVMe 15-Minute Ring Buffer<br/>GOP-Aligned Segment Manifests"]
        CHAT["Official Chat Event Connectors<br/>Twitch EventSub · Kick Webhook · YouTube Live API"]
    end

    subgraph AILayer ["AI & Media Processing Workers (Python)"]
        VAD["Silero VAD & Audio Classifier<br/>(Laughter / Cheer / Energy Jump)"]
        ASR["faster-whisper (CTranslate2)<br/>Word-Level Alignment & Subtitles"]
        CV["MediaPipe Face Tracker<br/>One-Euro Filter Smoothing · 9:16 Reframe"]
        FUSION["Multimodal Fusion & Candidate Engine<br/>Deterministic Sentence Boundary Clamping"]
    end

    subgraph RenderEngine ["Render Engine"]
        FAST["Fast Pipeline: FFmpeg<br/>Sub-30s Render · ASS Karaoke Subtitles · Audio Ducking"]
        RICH["Rich Pipeline: Remotion<br/>Motion Graphics · Zoom Punch · Split-Screen Gaming"]
    end

    subgraph PublisherProtection ["Distribution & Protection"]
        PUB["Resumable Publisher<br/>YouTube Shorts · TikTok Content API · X API v2"]
        FP["Fingerprint Engine<br/>vPDQ Perceptual Hashes · Chromaprint Audio"]
        MATCH["Re-upload Matcher & Evidence Dossier Generator"]
    end

    UI <--> API
    API <--> DB
    API --> OUTBOX --> REDIS
    ING --> RING
    CHAT --> FUSION
    RING --> VAD & ASR & CV
    VAD & ASR & CV --> FUSION
    FUSION --> API
    REDIS --> FAST & RICH
    FAST & RICH --> UI
    UI --> API --> PUB
    PUB --> FP --> MATCH
    MATCH --> DB

Loading

---

How It Works: 90-Second Latency Pipeline

Timeline	Pipeline Stage	Technical Operation
T + 00s	Viral Event Occurs	Streamer delivers punchline, clutch play, or iconic reaction on live broadcast.
T + 04s	Signal Detection	Chat message frequency surges (≥300% above 5-min baseline); VAD classifies peak vocal energy.
T + 15s	Post-Roll Capture	Buffer pins T−40s context to T+15s tail; candidate stream segment is committed.
T + 25s	Boundary Refinement	AI aligns start/end cuts to natural sentence boundaries and silence pauses (20–55s duration).
T + 40s	Smart 9:16 Reframe	Face tracking detects streamer camera; centers speaker or composites split facecam/gameplay layout.
T + 55s	Speech Transcription	faster-whisper aligns word timestamps; generates karaoke-style subtitles within safe areas.
T + 75s	Render & Audio Ducking	FFmpeg bakes video, applies automated ducking (-18dB background music under speech), and runs QC checks.
T + 90s	Review Inbox Alert	Clip is Review-Ready. Push notification dispatched to creator mobile / web review inbox.
Creator	Explicit User Approval	Streamer or authorized editor reviews clip, edits subtitles/crop if needed, and clicks Approve.
T + 120s	Multi-Platform Release	Uploads via resumable APIs to YouTube Shorts, TikTok, and X. Perceptual fingerprints registered.

---

Getting Started

Prerequisites

• Node.js 22.18+ and npm 10+
• Docker & Docker Compose
• Python 3.12 (worker services and pytest suites)
• FFmpeg & ffprobe 7.0+ on system PATH

Setup

# 1. Clone repository
git clone https://github.com/onurozkir/stream-sentry.git
cd stream-sentry

# 2. Install workspace dependencies
npm ci

# 3. Start the local stack (PostgreSQL, Redis, MinIO, API, workers, web)
copy .env.example .env
docker compose --env-file .env.example -f infra/compose/docker-compose.yml up -d --build

# 4. Open the dashboard
# http://localhost:5173

Local login uses the development fixture auth and requires explicitly setting ALLOW_DEV_AUTH=true; production startup fails closed without real production adapters.

Verification

npm run build
npm run typecheck
npm test
python -m pytest tests/ -q
npm run smoke

• npm test builds first, then runs the root test list.
• Real PostgreSQL tests require DATABASE_URL pointing to a dedicated disposable database.
• Real MinIO tests require TEST_STORAGE_ENDPOINT, TEST_STORAGE_ACCESS_KEY and TEST_STORAGE_SECRET_KEY.
• Environment-dependent skips are reported explicitly and are not acceptance evidence.

---

Repository Structure

stream-sentry/
├── apps/                            # TypeScript workspaces
│   ├── api/                         # Fastify control API (auth, RBAC, tenant guards, vault)
│   ├── web/                         # React 18 + Vite + Tailwind creator dashboard
│   ├── render-worker/               # BullMQ media worker (validate / render / QC)
│   ├── ingest-worker/               # Fenced FFmpeg live ingest + ring buffer
│   ├── orchestration-worker/        # Outbox event consumer + job-step ledger
│   └── protection-worker/           # Fingerprint extraction & evidence worker
├── packages/                        # Shared libraries
│   ├── contracts/                   # Canonical JSON Schemas, Zod + Pydantic models
│   ├── media/                       # Media validation, FFmpeg rendering, QC, storage
│   ├── security/                    # AES-256-GCM vault, redaction, scoped S3 URLs
│   ├── observability/               # Trace context, logging, metrics, audit
│   └── connectors/                  # Kick / YouTube / Rumble / TikTok / Meta / X protocol units
├── services/                        # Python AI & protection services
│   ├── ai_worker/                   # faster-whisper ASR service
│   └── protection_worker/           # Fingerprint extraction & matching service
├── infra/
│   ├── compose/                     # docker-compose.yml + optional GPU override
│   ├── db/                          # PostgreSQL schema, FORCE RLS, migrations
│   ├── ingest/                      # MediaMTX ingest configuration
│   └── storage/                     # Pinned MinIO build
├── poc/                             # Feasibility & pilot experiment harnesses
├── scripts/                         # Bootstrap, smoke, demo and evaluation tooling
└── tests/                           # Node/TypeScript + pytest regressions and fixtures

---

Platform Matrix & Access Capabilities

Stream Sentry strictly differentiates Chat/Metadata Access from Raw Video Extraction Rights. Platform capabilities are verified and reported independently:

Platform	Metadata & Status	Live Chat Transport	Raw Media Ingestion Strategy	Publishing Target
Twitch	Helix API / EventSub	Official EventSub WebSockets (channel.chat.message)	Direct OBS RTMP/SRT Relay or Streamlink (Subject to creator auth)	Export / Partner
Kick	Official Developer API	Webhooks (chat.message.sent) & Pusher WS	Direct OBS Feed or Streamlink	Export / Partner
YouTube Live	Data API v3 (Live Broadcast)	liveChatMessages.streamList with continuation tokens	Direct OBS Stream or Authorized Ingest (Policy compliance audited)	YouTube Shorts (videos.insert)
Rumble	Live Stream API v1.0	Creator Live API snapshot (50-msg bounded polling)	Direct RTMP Ingest via Creator Stream Key	Export / Partner
TikTok	Content Posting API	N/A (Source not targeted)	N/A	Direct Post API / Inbox Draft Mode
X (Twitter)	Twitter API v2	N/A (Source not targeted)	N/A	Chunked Media Upload + Post API

---

Security & Compliance Guardrails

• Zero-Trust Input Handling: URLs are strictly validated against allowlists; private IP ranges (RFC 1918) and cloud metadata services (169.254.169.254) are blocked against Server-Side Request Forgery (SSRF).
• Immutable User Approval Binding: Every publication payload requires an approval hash uniquely binding artifact_sha256 + target_account + metadata + privacy_level. Altering any field immediately revokes publishing eligibility.
• Data Minimization & Retention: Raw live stream ring buffers are purged after 15 minutes. Candidate media is retained for 7 days; final clips for 30 days. Raw chat identities are anonymized and retained for at maximum 24 hours.
• Automated PII & Secret Redaction: API tokens, OAuth refresh secrets, and broadcaster stream keys are encrypted via KMS vault adapters and scrubbed from all runtime log streams.
• Strict Human-in-the-Loop Copyright Process: No automated legal notifications or platform DMCA strikes are dispatched autonomously.

---

Project Status

This project is under active development. Current limitations:

• API data, sessions, the credential vault, and audit records run on in-memory development adapters by default; durable behavior is enabled per subsystem through explicit environment configuration.
• The API's /ready endpoint returns 503 until real dependency readiness adapters are connected; /health reports liveness only.
• A worker heartbeat does not prove queue consumption; a healthy process is not evidence of job processing.
• Real publishing to provider platforms stays disabled by default until platform reviews and authorized acceptance evidence are complete.
• No model or media files are committed to the repository.

---

Community

• Code of Conduct: CODE_OF_CONDUCT.md
• Contributing: CONTRIBUTING.md
• Security policy: SECURITY.md — please report vulnerabilities privately, never as a public issue.

---

License

Distributed under the MIT License. Copyright © 2026 Onur Özkır.