NVIDIA NIM Microservices Demo

This project demonstrates an enterprise-grade deployment of NVIDIA NIM (NVIDIA Inference Microservices) with NeMo Guardrails on AWS EKS. It focuses on providing a scalable, secure, and cost-optimized AI inference environment.

Overview

This project provides a robust solution for deploying NVIDIA NIM containers with NeMo Guardrails on Kubernetes, offering:

Safe AI: Implements content filtering and guardrails for responsible AI interactions.
Scalable: Leverages Kubernetes-native deployment for high availability and elastic scaling.
Enterprise-Grade: Designed for production readiness with secure infrastructure practices.
Cost-Optimized: Ensures efficient GPU resource utilization on AWS EKS.

Video Tutorial

📺 Watch the complete deployment walkthrough

Project Structure

nemo-microservices-demo/
├── infrastructure/             # Infrastructure as Code (IaC)
│   ├── k8s/                    # Kubernetes manifests for deployments and services
│   ├── helm/                   # Helm charts (if used for more complex deployments)
│   └── eks/                    # EKS cluster and nodegroup configurations
├── src/clients/                # Example Python client libraries for interaction
├── examples/                   # Usage examples and end-to-end test scripts
├── scripts/                    # Automation scripts for deployment, port-forwarding, and cleanup
└── notebooks/                  # Jupyter notebooks for advanced testing or exploration

Quick Start

Follow these steps to quickly get the demonstration running.

1. Prerequisites

Ensure you have the following tools installed and configured:

AWS CLI (configured with appropriate credentials)
kubectl (Kubernetes command-line tool)
eksctl (Amazon EKS command-line tool)
helm (Kubernetes package manager)
A valid NVIDIA NGC API key with access to NIM models.

2. Environment Setup

Configure your NVIDIA NGC API key securely:

# Copy the environment template
cp .env.example .env
# Edit the .env file with your NVIDIA NGC API key
# nano .env (or your preferred editor)

3. Deploy Infrastructure

Refer to the complete deployment guide for detailed, step-by-step instructions:

docs/deployment-guide.md

4. Access and Test the Service

Once deployed, set up port forwarding and run the test script:

# Set up port forwarding to the NeMo Guardrails service
./scripts/port_forward.sh

# Run the end-to-end test script
python3 examples/complete_demo.py

Available Scripts

This project includes several utility scripts to streamline operations:

Script	Purpose	Usage
`scripts/deploy.sh`	Deploys applications to an existing cluster	`./scripts/deploy.sh`
`scripts/port_forward.sh`	Sets up local port forwarding for testing	`./scripts/port_forward.sh`
`scripts/health_check.sh`	Verifies system health and component status	`./scripts/health_check.sh`
`scripts/cleanup.sh`	Performs a complete cluster and resource cleanup	`./scripts/cleanup.sh`

Prerequisites and Requirements

A detailed breakdown of the infrastructure and NVIDIA requirements.

AWS Infrastructure

EKS Cluster: Kubernetes version 1.29.15 (or compatible).
GPU Nodes: g5.2xlarge instances recommended (24GB GPU memory, 1x NVIDIA A10G GPU). Crucially, these nodes must use AmazonLinux2023 for GLIBC compatibility.
General Nodes: t3.medium instances for control plane and general workloads.
Network: A properly configured VPC with appropriate subnets and security groups.
IAM: AWS IAM roles with necessary permissions for EKS, EC2, and NGC access.

NVIDIA Requirements

NGC API Key: A valid NVIDIA GPU Cloud account with an active subscription for the desired NIM models.
Container Registry Access: Permissions to pull container images from nvcr.io/nim/*.
CUDA Platform: GPU nodes must support CUDA 12+.
GPU Drivers: NVIDIA device plugin and GPU Operator must be installed and configured.

Kubernetes Components

NVIDIA GPU Operator: Manages GPU drivers and related components.
NVIDIA Device Plugin: Enables Kubernetes to schedule pods on GPU resources.
Resource Limits: Proper memory and GPU allocation defined in deployment manifests.
Node Selectors: Used to target specific GPU instance types for NIM workloads.

Architecture Overview

The system architecture is designed for efficient and secure AI inference:

┌───────────────────┐    ┌───────────────────┐    ┌───────────────────┐
│   NeMo            │    │    NVIDIA         │    │    GPU Node       │
│   Guardrails      │───▶│      NIM          │───▶│  (g5.2xlarge)     │
│   (CPU Node)      │    │   Container       │    │   24GB GPU        │
└───────────────────┘    └───────────────────┘    └───────────────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
                   ┌───────────────────┐
                   │  Llama 3.2 1B     │
                   │      Model        │
                   │  (vLLM Engine)    │
                   └───────────────────┘

Implementation Steps

This section outlines the detailed steps for deploying the NVIDIA NIM Microservices demo.

Step 1: EKS Cluster Setup

Create the EKS cluster with a general-purpose nodegroup.

eksctl create cluster --name nemo-microservices-demo \
  --version 1.29 \
  --region us-east-1 \
  --nodegroup-name general-nodes \
  --node-type t3.medium \
  --nodes 2

Step 2: Add GPU Nodegroup (Critical)

Important: Use AmazonLinux2023 for GLIBC compatibility.

eksctl create nodegroup \
  --cluster nemo-microservices-demo \
  --name gpu-nodes-al2023 \
  --node-type g5.2xlarge \
  --nodes 1 \
  --node-ami-family AmazonLinux2023 \
  --node-volume-size 100 \
  --region us-east-1

Step 3: Install NVIDIA GPU Operator

kubectl create namespace nvidia-operator-system

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install nvidia-operator nvidia/gpu-operator \
  --namespace nvidia-operator-system \
  --set driver.enabled=true \
  --wait

Step 4: Create Application Namespace and Secrets

kubectl create namespace nemo-demo-ns

source .env # Load NGC_API_KEY

kubectl create secret generic ngc-api-key \
  --from-literal=NGC_API_KEY="$NGC_API_KEY" \
  --namespace=nemo-demo-ns

kubectl create secret docker-registry ngc-secret-test \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password="$NGC_API_KEY" \
  --namespace=nemo-demo-ns

Step 5: Verify GPU Setup

Confirm that GPU resources are available and the device plugin is active.

kubectl get nodes --show-labels | grep g5.2xlarge
kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds
kubectl describe nodes | grep nvidia.com/gpu

Expected output should show nvidia.com/gpu: 1 available on your GPU nodes.

Step 6: Deploy Applications Using Organized Structure

Apply the Kubernetes manifests from the infrastructure/k8s/ directory.

kubectl apply -f infrastructure/k8s/nimservice-llama-3-2-1b.yaml
kubectl apply -f infrastructure/k8s/nemo-guardrails-complete.yaml
kubectl apply -f infrastructure/k8s/nemo-guardrails-svc.yaml

Step 7: Wait for Deployment to Complete

NIM model loading can take several minutes. Monitor the status.

kubectl wait --for=condition=available --timeout=600s deployment --all -n nemo-demo-ns
kubectl get pods -n nemo-demo-ns
kubectl logs -f -n nemo-demo-ns -l app=nvidia-nim-llama-1b

Expected status: Both pods should show 1/1 Running.

Step 8: Test the Complete System

Run the provided scripts to confirm functionality.

./scripts/port_forward.sh
python3 examples/complete_demo.py

Expected result: Successful health checks and chat completions through guardrails.

Deployment and Validation Checklist

Your deployment is successful when:

Both nvidia-nim-llama-1b and nemo-fresh-start-guardrails pods show 1/1 Running.
NIM container logs indicate Ready: True.
Port-forwarding connects successfully to localhost:8008.
The health endpoint responds successfully: curl http://localhost:8008/v1/health.
Chat completion API calls function correctly through the guardrails.

Common Issues and Solutions

Issue 1: Image Pull Errors (402 Payment Required)

Cause: The NGC API key may lack access permissions to the specified NIM models. Solution:

Verify your API key in the NGC portal.
Ensure your account has a valid subscription or access to the required NIM models.
Update Kubernetes secrets: kubectl delete secret ngc-api-key -n nemo-demo-ns then recreate it with the correct key.

Issue 2: GLIBC Errors (Container Crashes)

Cause: Incorrect base AMI used for GPU nodes. AmazonLinux2 is incompatible with the NIM container's GLIBC requirements. Solution: Delete the existing GPU nodegroup and recreate it using --node-ami-family AmazonLinux2023.

Issue 3: GPU Resource Not Available

Cause: The NVIDIA device plugin may not be running or properly configured. Solution: Check pods in kube-system: kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds. If not running, wait for the GPU Operator to complete installation or manually apply the device plugin YAML.

Issue 4: Pods Stuck in Pending

Cause: A previous failed pod might have retained GPU allocation, preventing new pods from scheduling. Solution: Temporarily scale down and then scale up the deployment:

kubectl scale deployment nvidia-nim-llama-1b --replicas=0 -n nemo-demo-ns
kubectl scale deployment nvidia-nim-llama-1b --replicas=1 -n nemo-demo-ns

Performance Characteristics

Model: Llama 3.2 1B (1 Billion parameters)

Engine: vLLM with optimized precision (e.g., bf16).
GPU Memory: Approximately 4-8GB during active inference.
Initialization Time: 2-4 minutes for model loading and setup.
Throughput: Expected ~30-60 tokens/second on a g5.2xlarge instance.
Latency: First token generation typically 100-300ms.

Resource Requirements

Minimum GPU: g5.2xlarge (24GB GPU memory) for smaller models like Llama 3.2 1B.
Recommended GPU: g5.2xlarge or larger instances for bigger models or higher throughput.
CPU: 2-4 cores.
System Memory: 8-16GB system memory.

Complete Cleanup Guide

To avoid unexpected AWS charges, follow these steps to deprovision all resources.

# Kill any active port-forward processes
pkill -f "kubectl port-forward"

# Delete the application namespace (removes all NIM and Guardrails deployments/services)
kubectl delete namespace nemo-demo-ns

# Uninstall the NVIDIA GPU Operator
helm uninstall nvidia-operator --namespace nvidia-operator-system
kubectl delete namespace nvidia-operator-system

# Delete the GPU nodegroup
eksctl delete nodegroup --cluster nemo-microservices-demo --name gpu-nodes-al2023

# Delete the general purpose nodegroup
eksctl delete nodegroup --cluster nemo-microservices-demo --name general-nodes

# Delete the EKS cluster (deprovisions control plane, VPC, etc.)
eksctl delete cluster --name nemo-microservices-demo

Expected Deployment Timeline

Step	Description	Estimated Time
1	EKS Cluster creation	8-10 min
2	GPU nodegroup creation	3-5 min
3	GPU Operator installation	5-8 min
4	Secrets creation	<1 min
5	GPU verification	1 min
6	Application deployment	<1 min
7	NIM model loading	3-5 min
8	System testing	2 min
Total Deployment Time	Complete end-to-end setup	20-30 min

Name		Name	Last commit message	Last commit date
examples		examples
infrastructure/k8s		infrastructure/k8s
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation