🎯 Status: Feature Complete (Phases 1-12) — Distributed Object Storage Engine with Raft Consensus, Consistent Hashing, and Load Testing
A distributed object storage platform built in Go — inspired by Amazon S3 and MinIO.
AtlasStore separates the Control Plane (API Gateway) from the Data Plane (Storage Nodes) to orchestrate chunked, encrypted, and replicated file storage across a cluster. It was built from scratch as a learning project in distributed systems.
| Phase | Feature | Status |
|---|---|---|
| 1 | Two-server architecture (Gateway + Storage Node) | ✅ Done |
| 1 | JWT Auth (Register / Login) | ✅ Done |
| 1 | PostgreSQL metadata layer | ✅ Done |
| 1 | Web Dashboard (upload, download, delete) | ✅ Done |
| 2 | Chunking engine (5MB chunks, parallel upload/download) | ✅ Done |
| 2 | SHA-256 integrity verification on download | ✅ Done |
| 2 | Multipart upload support | ✅ Done |
| 3 | Storage node auto-registration with Gateway | ✅ Done |
| 3 | Background health checker (gRPC ping) | ✅ Done |
| 4 | Replication (N copies per chunk) | ✅ Done |
| 4 | Fallback download from replica on failure | ✅ Done |
| 5 | gRPC migration (internal HTTP → gRPC) | ✅ Done |
| 6 | Consistent Hashing ring with virtual nodes | ✅ Done |
| 6 | Deterministic chunk placement via Hash Ring | ✅ Done |
| 6 | Background Rebalancing Worker | ✅ Done |
| 7 | Raft Consensus (Leader Election & Cluster State) | ✅ Done |
| 8-10 | Fault Tolerance (Circuit Breakers, Quorum Degradation) | ✅ Done |
| 8-10 | AES-GCM Encryption & Snappy Data Compression | ✅ Done |
| 11 | Prometheus Metrics (Uploads, active nodes, etc) | ✅ Done |
| 12 | Load Testing Suite (2000+ virtual users) | ✅ Done |
[Web Dashboard]
│ (HTTP / JSON)
▼
┌──────────────────────────────────────┐
│ API Gateway │ ← "Brain" / Control Plane
│ - JWT Auth / Circuit Breakers │
│ - Consistent Hash Ring (50 vNodes) │
│ - Background Rebalancer │ ── (Metadata) ──► [ PostgreSQL DB ]
│ - AES-GCM Encrypt & Snappy Compress │
│ - Prometheus Metrics Exporter │
└──────────────────────────────────────┘
│ (internal gRPC)
┌───────┼───────┐
▼ ▼ ▼
┌──────────────────────────────────────┐
│ [Node A] [Node B] [Node C] │ ← "Muscles" / Data Plane
│ │ │ │ │
│ Disk Disk Disk │ ← Encrypted chunk files (SHA-256)
│ │
│ [Raft Consensus Cluster] │ ← Maintains Active Node State
└──────────────────────────────────────┘
- File is split into 5MB chunks (configurable).
- Each chunk is AES-GCM encrypted before leaving the Gateway.
- SHA-256 hash of the plaintext chunk is computed → used as the chunk's filename and lookup key.
- The Consistent Hash Ring maps the chunk hash → N storage node addresses (where N = replication factor).
- Chunks are saved to those nodes in parallel via gRPC.
- If a node is added or removed, the Rebalancer (runs every 60s) detects misplaced chunks and migrates them without downtime.
AtlasStore has been rigorously load-tested to find its breaking point on local hardware. The system architecture (Go Goroutines + gRPC + PostgreSQL) is designed for massive horizontal scalability, but a single local instance provides incredible baseline performance.
Test Setup:
- Concurrency: 2,000 simultaneous virtual users
- Duration: 15 seconds
- Traffic: Continuous stream of Multi-part File Uploads
- Hardware: 12th Gen Intel(R) Core(TM) i5-1235U (12 Cores), 8GB RAM
Results (Single Machine Peak):
| Concurrency (Virtual Users) | Total Requests (15s) | Success Rate | Throughput (Req/Sec) | Notes |
|---|---|---|---|---|
| 100 | ~3,260 | 100.00% | ~210 req/s | Perfect stability. |
| 500 | ~3,464 | 99.83% | ~208 req/s | Slight resource contention. |
| 2,000 | ~4,883 | 92.91% | ~215 req/s | OS file descriptors and DB connections exhausted. |
Note: Each recorded "Request" is a full multi-part file upload consisting of 3 separate HTTP round-trips. The Gateway actually processed over 600 HTTP requests per second at peak.
Bottleneck Analysis:
At 2,000 concurrent users, the Go application itself did not crash. The 7% error rate was caused by hitting physical hardware limits (OS File Descriptors and PostgreSQL max_connections pool exhaustion). To scale beyond this, the architecture supports deploying the Gateway behind a load balancer and horizontally scaling the PostgreSQL database.
| Layer | Technology |
|---|---|
| Backend | Go 1.24 (standard library + minimal dependencies) |
| Database | PostgreSQL (via lib/pq + golang-migrate) |
| Internal Comms | gRPC + Protocol Buffers (pkg/pb) |
| Auth | bcrypt (passwords) + golang-jwt/jwt/v5 (tokens) |
| Encryption | AES-GCM 256-bit (at rest) |
| Hashing | SHA-256 (chunk integrity + consistent ring placement) |
| Frontend | Vanilla HTML/CSS/JavaScript |
docker compose up -dEnsure your .env file exists:
DB_HOST=localhost
DB_PORT=5433
DB_USER=atlasstore
DB_PASSWORD=atlaspassword
DB_NAME=atlasstore
DB_SSLMODE=disable
JWT_SECRET=super_secret_key_change_in_production
CHUNK_SIZE_MB=5
REPLICATION_FACTOR=2
GATEWAY_PORT=8000
STORAGE_NODE_PORT=9001
ENCRYPTION_KEY=0123456789abcdef0123456789abcdefOpen separate terminals for each process:
Terminal 1 — Gateway:
go run ./cmd/gateway/Terminal 2 — Storage Node A:
STORAGE_DATA_DIR=./data/nodeA STORAGE_NODE_PORT=9001 go run ./cmd/storagenode/Terminal 3 — Storage Node B (optional, for replication):
STORAGE_DATA_DIR=./data/nodeB STORAGE_NODE_PORT=9002 go run ./cmd/storagenode/Each storage node automatically registers itself with the Gateway on startup. The Gateway's Hash Ring will update within 10 seconds.
internal.md— Directory maps, request flows, and component breakdown.learning.md— Concept explanations (gRPC, Consistent Hashing, AES-GCM, Connection Pooling, JWTs, etc.).PLAN.md— The full development roadmap.