High-Performance Computing | CUDA Programming | Distributed Systems Architecture
A comprehensive GPU acceleration project implementing Adaptive Histogram Equalization (AHE) with progressively advanced optimization techniques-from foundational CUDA kernels to distributed RDMA-based processing pipelines. Demonstrates performance improvements through parallel algorithm design, memory hierarchy optimization, and zero-copy networking.
This repository contains three progressive projects from Accelerators and Accelerated Systems (046278) – Spring 2024, Technion – Israel Institute of Technology. Each project builds upon the previous one, exploring increasingly sophisticated GPU acceleration and distributed computing techniques.
Core Algorithm: Adaptive Histogram Equalization (AHE) for image enhancement
Progressive Optimization Path:
- HW1: Foundational CUDA kernel implementation with parallel algorithms
- HW2: Advanced asynchronous execution with CUDA Streams and persistent kernels
- HW3: Distributed processing with RDMA-style networking for low-latency GPU-accelerated processing
- ⚡ Parallel Algorithm Implementation – Custom CUDA kernels for histogram calculation, prefix sum, and bilinear interpolation
- 🚀 Memory Optimization – Tile-based processing and shared memory utilization to reduce global memory bandwidth usage
- 🔄 Asynchronous Execution – CUDA Streams and queue-based designs for overlapped computation
- 🌐 RDMA-Oriented Design – Client–server architecture and zero-copy-style transfers for distributed GPU computing
- 📊 Performance Engineering – Register-aware optimization, memory coalescing, and occupancy-oriented tuning
Accelerated-systems/
├── HW1/ # Foundational CUDA Implementation
│ ├── ex1.cu # Parallel histogram, prefix sum, interpolation kernels
│ ├── ex1.pdf # Assignment specification
│ └── homework1.zip # Original homework package
├── HW2/ # Advanced GPU Utilization
│ ├── ex2.cu # CUDA Streams, persistent kernels, queue-based processing
│ ├── ex2.pdf # Design / assignment documentation
│ └── homework2.zip # Original homework package
└── HW3/ # Networked RDMA-Style Architecture
├── Makefile # Build configuration
├── client.cu # Client-side logic, request generation / networking
├── server.cu # Server-side processing with GPU integration
├── common.cu # Shared infrastructure, utilities, protocol logic
├── ex2.cu # Supporting code from previous exercise
├── ex2.h # Headers for HW3 ex2 code
├── ex3.cu # Main GPU-accelerated processing implementation
├── ex3.h # GPU-side interface and definitions
├── ex3-cpu.cu # CPU baseline / reference implementation
├── homework3.pdf # Architecture / assignment specification
└── libutils.a # Precompiled utility library
Goal: Implement a parallel AHE algorithm with optimized memory access patterns.
Parallel Algorithms Implemented:
- Histogram Computation: Thread-level parallelization over image tiles with atomic operations for thread-safe accumulation
- Prefix Sum (Scan): Parallel scan implementation (Hillis–Steele style) with logarithmic depth compared to sequential O(n)
- Bilinear Interpolation: Four-neighbor weighted interpolation with attention to memory coalescing
Processing Strategies:
- Task-Serial Mode: Processes images sequentially with a small memory footprint (approximately one image in memory plus metadata)
- Bulk GPU Mode: Processes multiple images in parallel on the GPU to increase throughput (more images and metadata resident on device)
Observed/Targeted Performance Characteristics:
- Tile-based and shared-memory-aware processing is used to substantially reduce global memory bandwidth usage compared to a naïve implementation (in our experiments, this reduction was on the order of tens of percent; exact values are workload- and GPU-dependent).
- Shared memory is used per thread block to cache partial histograms and intermediate data, reducing global memory traffic.
- Register usage and kernel launch configuration are tuned to keep register pressure reasonable and maintain good multiprocessor occupancy (e.g., on the order of a few dozen registers per thread).
Goal: Increase GPU utilization via asynchronous processing and long-lived GPU kernels.
Key Components:
CUDA Streams:
- Overlapped host–device transfers and computation for latency hiding
- Use of multiple CUDA streams to keep the GPU active while data is being moved
- Pipeline depth chosen to reduce idle GPU time while avoiding excessive buffering
Queue-Based Producer–Consumer System:
- Ring-buffer-like queues coordinating work between host and device
- Use of atomic operations and memory-ordering semantics to ensure safe communication
- Queue sizing and alignment selected to simplify index arithmetic and mitigate contention
Persistent Kernel Design:
- Long-running kernels reduce per-launch overhead by reusing resident thread blocks
- Device-side synchronization (e.g., test-and-set locks or atomics) controls work acquisition
- Thread blocks repeatedly dequeue work, helping maintain steady-state GPU occupancy
Qualitative Optimization Outcomes:
- Reduced kernel launch overhead by switching from per-image launches to persistent kernels
- Improved throughput by allowing continuous work consumption from queues
- Lower end-to-end latency for individual requests through overlapping I/O and compute
GPU Occupancy Considerations (example: NVIDIA Turing sm_75 system used for testing):
- Hardware resources per SM include tens of thousands of registers and tens of KB of shared memory
- Kernel parameters (threads per block, shared memory usage, register pressure) are tuned so that multiple concurrent thread blocks can reside on each SM to improve latency hiding
Goal: Extend GPU acceleration to a networked client–server architecture while keeping CPU overhead low and leveraging zero-copy-style data transfers where possible.
Distributed / RDMA-Style Architecture Features:
Zero-Copy-Oriented Data Transfer:
- Use of pinned host memory (e.g., via
cudaMallocHost) to provide DMA-capable buffers suitable for high-throughput transfers - Design assumes a verbs-style RDMA or InfiniBand/RoCE environment where direct memory access can be used to minimize copies
- Emphasis on avoiding redundant CPU-mediated
memcpywhere network and GPU can transfer data directly between pinned regions
Implementation Modes:
- RPC-Style Mode: Request/response protocol suitable for single or few-client scenarios
- Queue-Based Mode: Persistent queues and long-running server processes to support higher throughput and multi-client readiness
Network Optimization Techniques (Design Targets):
- Direct interaction between client and server over a verbs-like API, aiming for minimal software overhead
- Use of pre-registered memory regions to avoid repeated registration costs on the data path
- Configuration parameters (e.g., MTU, number of in-flight requests or reads) chosen to help saturate available bandwidth on typical lab hardware
Performance Characteristics (Experimental):
- On test systems, server-side CPU utilization can remain low when the GPU and network are the main bottlenecks; the intention is for the CPU to primarily orchestrate rather than perform per-byte processing.
- Throughput is measured in requests per second, comparing different pipeline depths and queue configurations.
- End-to-end latency measurements include both network transfer times and GPU processing durations.
- CUDA 11.0+ – Parallel computing platform and programming model
- C++14/17 – Modern C++ with RAII and atomic operations
- RDMA / Verbs-Style APIs – High-performance networking abstractions (e.g., InfiniBand/RoCE-style environments)
- NVIDIA GPU – Developed and tested on GPUs with compute capability
sm_75(Turing) or newer
GPU Programming:
- CUDA kernel development and optimization
- Memory hierarchy management (global, shared, constant, pinned host memory)
- Synchronization primitives (CUDA atomics, block-level synchronization)
- CUDA Streams for asynchronous execution
Performance Optimization:
- Techniques for reducing effective memory bandwidth consumption
- Occupancy tuning and register pressure management
- Memory access pattern optimization for coalesced global loads/stores
- Latency vs. throughput trade-off analysis
Systems Programming:
- Concurrent data structures with atomic operations
- Memory ordering and synchronization between producer and consumer threads
- Producer–consumer design patterns across CPU and GPU
- High-performance network programming principles in RDMA-style environments
Parallel Algorithms:
- Reduction patterns (histograms)
- Scan patterns (prefix sum)
- Interpolation / stencil-style operations
- Mapping algorithms onto GPU architecture (SMs, warps, threads)
Each implementation includes timing and performance measurement components such as:
- CPU Baseline: Sequential or CPU-only processing for correctness verification and performance comparison
- GPU Variants: Comparisons between task-serial and bulk-processing GPU modes
- Throughput Metrics: Requests-per-second (RPS) style measurements for batches of images or requests
- Latency Tracking: Per-request timing and aggregate statistics (e.g., min/avg/max) to evaluate end-to-end behavior
Memory Hierarchy Optimization:
- Use of shared memory within each block to cache histogram and intermediate data and reduce global memory accesses
- Access patterns chosen to be L1/L2 cache-friendly where possible
- Empirically observed reductions in global memory bandwidth usage when transitioning from naïve to tile- and shared-memory-optimized implementations
Compute Optimization:
- Thread block sizes (e.g., on the order of 256 threads per block) selected for good warp scheduling and occupancy
- Kernel parameters tuned to allow multiple active blocks per SM while respecting register and shared memory limits
Asynchronous Processing:
- Overlapped host-to-device transfers, kernel execution, and device-to-host transfers using CUDA Streams
- Pipeline depth adjusted experimentally to minimize idle time on the GPU
- Queue-based work submission for continuous GPU utilization during heavy workloads
Network / RDMA Optimization:
- Use of pinned memory buffers to enable direct DMA and reduce copy overhead
- Multiple outstanding requests (and, when applicable, reads/writes) to keep the network and GPU pipeline full
- Design tuned for high-throughput, low-latency operation in lab RDMA-capable environments
This project provides hands-on experience with:
Parallel Computing Fundamentals:
- Understanding GPU architecture (SMs, warps, threads, memory hierarchy)
- Designing and implementing parallel algorithms
- Profiling, identifying, and addressing performance bottlenecks
CUDA Development:
- Kernel programming and optimization
- Managing memory across global, shared, constant, and pinned regions
- Using atomics and synchronization for correctness
- Event-driven asynchronous execution with CUDA Streams
High-Performance Systems:
- Lock-free and low-contention concurrent data structures
- Memory consistency models and atomics in host and device code
- Producer–consumer architectures spanning CPU, GPU, and network
- RDMA-style networking for low-latency communication
Performance Engineering:
- Methodical optimization, guided by measurement and profiling
- Occupancy analysis and tuning
- Bandwidth and latency characterization
- Understanding trade-offs between complexity, performance, and portability
Scalability Enhancements:
- Multi-GPU support using libraries such as NVIDIA NCCL for GPU-to-GPU communication
- Dynamic load balancing based on real-time GPU utilization and network conditions
- Adaptive batching strategies for variable-size images and workloads
Advanced Optimizations:
- GPU-accelerated compression to reduce network bandwidth requirements
- GPUDirect RDMA for more direct GPU-to-GPU or GPU-to-network transfers
- Exploring Tensor Core utilization where applicable for specific computational patterns
Course: Accelerators and Accelerated Systems (046278)
Institution: Technion – Israel Institute of Technology
Semester: Spring 2024
Focus Areas: GPU computing, parallel algorithms, high-performance distributed systems
- End-to-end thinking for building distributed, GPU-accelerated systems