A real-time, local-first streaming analytics platform designed to calculate and predict live audience viewer retention using Machine Learning and ultra-fast WebSocket broadcasting!
This repository operates as a full-stack monorepo featuring a machine-learning engine, a high-concurrency Go proxy/backend, and a premium React Dashboard.
Our technology stack was carefully selected to merge data science with extreme real-time throughput:
The core WebSocket engine (backend-go) is built in Go. Live streaming analytics generate an immense volume of micro-events (chat sends, tab hides, volume muting). Go's lightweight goroutines and native channel-based execution model allow us to process tens of thousands of concurrent WebSocket client connections dynamically. Instead of locking up on single threads, Go operates asynchronously mapping our real-time message routing hub with minimal CPU overhead.
The Machine Learning component (ml-engine-python) relies on Scikit-Learn running behind a lightning-fast FastAPI interface. Scikit-learn excels at tabular classification models (like Random Forests) providing incredibly low-latency inference on the fly. Where deep learning models might require GPU cycles to calculate user drop-off risks, Scikit-learn computes tabular statistical probabilities accurately in milliseconds—critical when latency is the enemy of a live streaming dashboard.
The frontend-react dashboard is served up using Vite, offering instantaneous hot-module replacement and bleeding-edge build speeds. We use Recharts for rendering the real-time probability area charts with a lightweight, dark-mode glassmorphism styling foundation built organically from the ground up without heavy external UI libraries.
We've embraced a Docker-first, local-first architecture so that contributors can spin up the entire infrastructure locally entirely for free.
Our docker-compose.yml mounts all three specific services alongside LocalStack—a fully functional local AWS cloud stack. By binding localstack on port 4566, you can develop locally against simulated S3 buckets or ECS APIs without ever needing AWS credentials or incurring bandwidth costs!
Ensure you have Docker running, then deploy the local cloud:
# Spins up Go backend, Python FastAPI, React Nginx frontend, and LocalStack
docker-compose up --buildYour interactive Stream Dashboard is now available at http://localhost:3000!
This repository includes a terraform/ directory containing the theoretical deployment architecture designed for the AWS Cloud. By provisioning through Terraform (main.tf), we eliminate manual console clicking and enforce repeatable immutable infrastructure deployments.
The design implements:
- AWS ECS (Fargate): We deploy all containers into a serverless ECS task definition. Because our containers live inside the same Fargate task, they share an Elastic Network Interface (ENI). The Go backend securely invokes the internal ML Engine over
localhostinstead of traversing a public subnet! - AWS S3: We map up an encrypted, version-controlled S3 Bucket (
aws_s3_bucket.model_logs) tightly scoped using Task IAM roles so the ML Engine can archive inference records securely.
If a broadcast grows to 1,000,000 concurrent streaming connections, our architecture seamlessly scales conceptually:
- Horizontal Go Scaling via Pub/Sub: The Go WebSocket hub currently operates in-memory. However, the exact architectural blueprint is ready to implement Redis Pub/Sub or Kafka. 1 Million viewers will map to thousands of Fargate tasks hidden behind a Network Load Balancer (NLB). The Go nodes will connect to a caching layer to broadcast global connection counts horizontally.
- ML Engine Endpoint Load Balancing: As the volume of inference data skyrockets from 1M WebSockets, the FastAPI/Scikit-learn service will separate independently behind an Application Load Balancer (ALB). We would leverage ECS Auto-Scaling using CloudWatch alarms on CPU saturation to spin up hundreds of Python instances in seconds.
- Data Lake Archival: Local logging to S3 directly per individual viewer would DDOS the network pipeline. A batching layer utilizing AWS Kinesis Firehose would buffer event payloads from Go and stream them into S3 en-masse for data warehousing in isolated parquet files!