Optimize LLM Inference on Amazon EKS

End-to-end workshop for deploying, monitoring, benchmarking, and optimizing LLM inference on AWS EKS with GPU acceleration.

Modules

Module	Description	Duration
1. GPU Infrastructure	EKS Auto Mode, ODCR, SOCI, GPU NodePools	8 min
2. Deploy the Model	vLLM + Ministral-3-8B with S3 model loading	20 min
3. Observability	NVIDIA DCGM + Prometheus + Grafana	10 min
4. Agentic App	Tool-calling agent with self-hosted model	15 min
5. Inference Benchmarking	Baseline, saturation tests, optimizations	25 min
6. KV Cache Offloading	LMCache + Valkey tiered caching	20 min

Prerequisites

• AWS account with GPU instance quota (g6e.2xlarge)
• AWS CLI v2 configured
• kubectl, eksctl, helm installed
• Python 3.9+

Architecture

┌─────────────────────────────────────────────────────┐
│                 Amazon EKS (Auto Mode)               │
├─────────────────────────────────────────────────────┤
│                                                      │
│  ┌─────────────┐   ┌──────────────┐                │
│  │ vLLM Server │   │  Prometheus  │                │
│  │ (GPU Node)  │   │  + Grafana   │                │
│  │ g6e.2xlarge │   │  (System)    │                │
│  └──────┬──────┘   └──────┬───────┘                │
│         │                  │                         │
│  ┌──────┴──────┐   ┌──────┴───────┐                │
│  │ S3 CSI      │   │ DCGM Exporter│                │
│  │ (Model)     │   │ (GPU Metrics)│                │
│  └─────────────┘   └──────────────┘                │
│                                                      │
│  ┌─────────────────────────────────┐                │
│  │ LMCache + Valkey (Module 6)     │                │
│  │ GPU → CPU L1 → Valkey L2       │                │
│  └─────────────────────────────────┘                │
└─────────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────┐
│   S3 Bucket     │
│ (Model Weights) │
└─────────────────┘

KV Caching Deep Dive

The SPEC-*.md files provide a comprehensive technical explanation of KV caching in LLMs:

1. SPEC-001: KV Cache Fundamentals
2. SPEC-002: Paged Attention (how vLLM manages memory)
3. SPEC-003: The Eviction Paradox (why naive compression fails)
4. SPEC-004: FlashAttention Incompatibility
5. SPEC-005: TriAttention (NVIDIA's solution)
6. SPEC-006: Summary and Practical Implications

Quick Start

# 1. Clone this repo
git clone https://github.com/Ramu-DE/Optimize--Inference-EKS.git
cd Optimize--Inference-EKS

# 2. Open the notebook
jupyter notebook notebook.ipynb

# 3. Follow modules sequentially

License

MIT