cuda-graphs

From-scratch, heavily-annotated CUDA inference runtime for Qwen2.5-Coder-7B on H100 (sm_90). Custom INT4 packer, fused GEMV, paged KV, split-KV attention, CUDA graph decode — every hot path commented for the why. Educational, not a llama.cpp replacement.

GLM-5.2 on 4x DGX Spark with adaptive MTP K2/K4/K5, FULL CUDA graphs, DCP2, 520K context, and a downloadable ARM64 runtime image.

Deterministic sparse-index localization for Apple silicon (MLX/Metal) and CUDA/Triton, with stable ordering, a Python oracle, and reproducible evidence.

A drop-in, hardware-agnostic library for Fused Triangle Multiplicative Updates across AlphaFold3-family models, powered by CUTLASS CuTe kernels.

Measured serving recipes for DeepSeek-V4.1-Flash on 4x NVIDIA DGX Spark (GB10): 1M context on vLLM (CUDA graphs, vision, tools, DSpark) and a switchless-ring SGLang TP4 lane, plus a cross-project reference table. EN + 中文.

fable-5's improvements to the CIFAR-10 speedrun over the hiverge baseline, measured

QSA HiSparse as an SGLang plugin: 256K CPU KV offload and text/image host prefix reuse for Qwen Sparse Attention on unmodified SGLang

Optimized CSM-1B TTS pipeline for RTX 5090 (Blackwell sm_120). CUDA graph replay via patched HF Transformers. ~0.46x RTF. Topics (tags): csm text-to-speech rtx-5090 blackwell cuda-graphs torch-compile sesame streaming pytorch

Hardware-aware Ollama fork with legacy NVIDIA support, controllable runtime tuning, Q4/Q8 KV cache, Flash Attention, CUDA Graphs and optimized multi-GPU inference.

CUDA-graph-safe per-request activation steering plugin for vLLM. pip install hotwire-vllm

A production-ready, high-performance CUDA Runtime and Driver API library for Zig.

Ultra-High-Throughput Transformers Engine: NVIDIA FP8/TMA, Zero-Allocation Slotted Static KV Cache, Sub-5µs CUDA Graphs, Fused Logits Sampling, Tree Speculative Decoding, Split-KV Flash-Decoding & Apple Silicon Metal (MSL) Simdgroup Kernels

Deterministic, allocation-free C++20/CUDA/TensorRT 11 inference runtime with mapped zero-copy I/O, fused GPU pre/postprocessing, fixed execution lanes, and CUDA Graph replay.

Measuring what makes a VLA fast enough to run on the robot: a 5.9x CUDA-graph win, four experiments on why low-bit doesn't, a budget-driven deploy-compiler, and a runtime safety supervisor. Live demo: hf.co/spaces/LaelaZ/embodied-efficiency

From-scratch C++/CUDA LLM inference engine: paged KV cache, continuous batching, CUDA-graph decode. 4,748 tok/s on an RTX 4090 - benchmarked against vLLM and llama.cpp with byte-identical-output gating and fully committed raw data.

Run DeepSeek-V4.1-Flash on 8x RTX 5090 + 503 GiB RAM. 5.6x output throughput vs patched eager mode, with vLLM fixes, CPU offload & reproducible benchmarks.

Bake neural rendering instead of running it every frame: a GPU-resident cache replaces per-frame neural inference, 20x to 34x faster at up to 48.7 dB PSNR. Rust, cuTile, RTX 4070 Ti.

Inference optimization environment for agents

From-scratch LLM inference engine — continuous batching, paged KV cache, prefix caching, hand-written Triton paged attention, CUDA graphs, OpenAI-compatible API. Benchmarked against vLLM on an A100.