LoadForge — Distributed Load Testing Platform

LoadForge is a distributed load-testing platform written in Go. A master node accepts test configurations over a REST API and dispatches jobs to a pool of worker processes over Kafka. Workers generate rate-limited, concurrent HTTP load against a target, track latency percentiles locally, and stream results back to the master, which caches the latest snapshot in Redis and broadcasts it to live clients over WebSockets. A Cobra-based CLI wraps the API for command-line control.

Architecture

 CLI (cobra) ──HTTP──▶ Master API (Chi)
                             │
                    creates test config
                             │
                     POST /tests/{id}/run
                             │
                             ▼
                     Kafka: load-jobs topic
                             │
                             ▼
                    ┌────────────────┐
                    │  Worker pool   │  (N processes)
                    └────────────────┘
                     │              │
         Kafka: heartbeat   Kafka: load-results
                     │              │
                     ▼              ▼
                     Master (registry + Redis cache)
                             │
                             ▼
                     WebSocket Hub ──▶ live clients (run:{id}:latest, 1s tick)

Worker loop

1. Consumes a JobConfig from the load-jobs Kafka topic.
2. Spins up N concurrent goroutines, each throttled by a token-bucket rate limiter (golang.org/x/time/rate), firing HTTP requests at the target URL for the configured duration.
3. Records per-request latency and error state in an in-memory HDR histogram (HdrHistogram-go).
4. Every second, snapshots p50/p95/p99 latency + RPS + error count and publishes it to the load-results topic.
5. Emits a heartbeat every 5 seconds to the heartbeat topic so the master can track worker liveness.

Master loop

1. Consumes heartbeats and updates an in-memory WorkerRegistry (workers are considered alive if seen within the last 15s).
2. Consumes result snapshots and caches the latest one per run in Redis (run:{run_id}:latest).
3. A WebSocket hub polls Redis once a second and pushes the latest snapshot to any client subscribed to that run, powering a live-updating dashboard.

Tech Stack

Layer	Technology
Language	Go
Job distribution / streaming	Apache Kafka (segmentio/kafka-go)
Live results cache & pub	Redis (redis/go-redis/v9)
REST API	Chi (go-chi/chi/v5)
CLI	Cobra (spf13/cobra)
Real-time transport	WebSockets (gorilla/websocket)
Latency tracking	HDR Histogram (HdrHistogram-go)
Persistence (schema defined)	PostgreSQL
Local infra	Docker Compose (Kafka + Zookeeper, Postgres, Redis)

Getting Started

# 1. Start infra dependencies
cd engine/scripts
docker compose up -d

# 2. Run the master
cd ../
go run ./cmd/master

# 3. Run one or more workers (in separate terminals)
go run ./cmd/worker

# 4. Drive it via the CLI
go run ./cmd/cli start --url http://example.com --rps 100 --duration 60s --concurrency 10
go run ./cmd/cli worker      # list alive workers
go run ./cmd/cli status <run_id>
go run ./cmd/cli stop <run_id>

REST API

Method	Path	Description
POST	/tests	Create a test configuration
GET	/tests	List test configurations (stub)
GET	/tests/{id}	Get a test configuration (stub)
POST	/tests/{id}/run	Schedule a run for a test configuration
GET	/runs/{id}	Get run status (stub)
DELETE	/runs/{id}	Abort a run (stub)
GET	/workers	List currently alive workers
GET	/ws?run_id=	WebSocket stream of live metrics for a run