AITER (AI Tensor Engine for ROCm) is AMD's high-performance AI operator library, providing optimized GPU kernels for inference and training workloads on ROCm. It serves as a unified collection of production-ready operators that framework developers can integrate directly into their stacks.
- C++ and Python APIs — use operators from either level
- Multiple kernel backends — Triton, Composable Kernel (CK), and hand-tuned ASM
- Inference and training — not just serving kernels, but also training and GEMM+communication fused kernels
- Framework-agnostic — integrate into vLLM, SGLang, or any custom framework
- [2026/07] Kimi-K3 support — FlyDSL SiTUv2 fused-MoE kernels, strided grouped-topk router, and tuned GEMM/fused-MoE configs (BF16, A8W4, FP4) for Kimi-K3
- [2026/04] AITER v0.1.12.post1 Released — patch on v0.1.12 with GEMM and scale masking accuracy fixes; v0.1.12 highlights include blockwise sparse Sage Attention, fused gated RMSNorm+group quantization, etc., plus MI355X tuned configs for Kimi-K2.5 and DeepSeek-V3
- [2026/02] JAX-AITER: Bringing AMD's Optimized AI Kernels to JAX on ROCm
- [2026/02] Beyond Porting: How vLLM Orchestrates High-Performance Inference on AMD ROCm
- [2026/01] Character.ai: 2x Production Inference Performance on AMD Instinct GPUs
- [2026/01] ROCm Becomes a First-Class Platform in the vLLM Ecosystem
- [2025] Accelerated LLM Inference with vLLM 0.9.x and ROCm
- [2025] Accelerate DeepSeek-R1 Inference: Integrate AITER into SGLang
- [2025/08] AITER-Enabled MLA Layer Inference on AMD Instinct MI300X
- [2025/08] Tutorial: MLA Decoding Kernel of the AITER Library to Accelerate LLM Inference
- [2025/03] Accelerating DeepSeek Inference with AMD MI300 — Microsoft
- [2025/03] AITER: AI Tensor Engine For ROCm — Launch Announcement
AITER is the default kernel backend for LLM inference on AMD GPUs, integrated into the major serving frameworks and powering production workloads at scale.
| Framework | Integration | Status | Operators Used |
|---|---|---|---|
| vLLM | Default attention backend on ROCm | Production | MHA, MLA, Paged Attention, Fused MoE, GEMM, RMSNorm, RoPE+KVCache |
| SGLang | Default on ROCm Docker | Production | Attention, Fused MoE, Block-scale GEMM, All-reduce, RMSNorm |
| ATOM | Built natively on AITER | Active development | All AITER operators (attention, MoE, sampling, communication) |
| JAX | XLA FFI bridge, no PyTorch dependency | Experimental | MHA/FMHA, RMSNorm, BF16 GEMM |
| Various customer proprietary inference engines | Kernel-level integration | Production | Attention, MoE, GEMM, quantization |
| Operator | Speedup |
|---|---|
| MLA decode kernel | up to 17x |
| MHA prefill kernel | up to 14x |
| Block-scaled Fused MoE | up to 3x |
| Block-scaled GEMM | up to 2x |
| DeepSeek-R1 e2e (SGLang) | 6,484 → 13,704 tok/s (2.1x) |
| JAX-AITER attention (MI350) | 4.39x median |
For detailed benchmarks, see the ATOM Benchmark Dashboard.
| GPU | Architecture | Status |
|---|---|---|
| AMD Instinct MI300X | gfx942 (CDNA3) | Fully supported |
| AMD Instinct MI325X | gfx942 (CDNA3) | Fully supported |
| AMD Instinct MI350 | gfx950 (CDNA4) | Supported |
| AMD Instinct MI355X | gfx950 (CDNA4) | Supported |
| AMD Pro W7900 | gfx1100 (RDNA3) | Experimental1 |
| AMD AI Max and Max Pro 400/300 Series | gfx1151 (RDNA3.5) | Experimental1 |
| AMD Radeon AI PRO R9700 | gfx1201 (RDNA4) | Experimental1 |
1 On RDNA, Triton and most FlyDSL kernels run, as do most HIP kernels (norm, RoPE, quant, activation, plus some GEMM/attention). Most CK and ASM kernels are CDNA-only.
AITER provides optimized kernels for attention, MoE, GEMM, normalization, quantization, communication, and more. Each operator has unit tests under op_tests/ that you can run directly:
# Example: run a single operator test
python3 op_tests/test_mha.py
python3 op_tests/test_mla.py
python3 op_tests/test_moe.py
python3 op_tests/test_gemm_a8w8.py
python3 op_tests/test_rmsnorm2d.py
# See all available operator tests
ls op_tests/test_*.pyAITER publishes a scheduled release every two weeks. Each scheduled release uses a release branch named after the target version, such as release/v0.1.20, and a matching release tag, such as v0.1.20. The normal version progression moves from one scheduled release tag to the next, for example v0.1.19 to v0.1.20.
For scheduled releases, automation creates any missing release/vX.Y.Z branch and matching vX.Y.Z tag from the configured release source ref, which defaults to main. The tag must point at the release branch HEAD. The GitHub Release page is created with the release branch as the target, generated notes are diffed against the previous scheduled tag, and the notes state the diff base.
If a hotfix is required after a release, the fix is cherry-picked onto the corresponding release branch and published as a post-release tag. Post-releases use the .postN suffix, for example v0.1.20.post1. Post-release tags must already exist on the matching release branch; automation refuses to create a post tag from main.
Release automation validates that the release tag points at the matching release branch HEAD, builds manylinux_2_28 wheels for ROCm 7.0, 7.1, and 7.2 with Python 3.10 and 3.12, validates that exactly six wheels were produced, and uploads the complete wheel set to the matching GitHub Release. It does not upload partial wheel sets.
git clone --recursive https://github.com/ROCm/aiter.git
cd aiter
python3 setup.py developIf you happen to forget the --recursive during clone, you can use the following command after cd aiter
git submodule sync && git submodule update --init --recursiveAITER uses FlyDSL-based kernels across a range of operators (e.g., GEMM and MoE). FlyDSL is a required dependency and is installed automatically when you run python3 setup.py develop.
To install it manually:
pip install -r requirements.txtAITER includes Triton-based operators that require triton from AMD PyPI, with the correct version selected based on your ROCm installation.
If you install with python3 setup.py develop, triton is installed automatically. To skip this and keep your existing triton, set:
AITER_USE_SYSTEM_TRITON=1 python3 setup.py developIf you use pip install -e ., run the install script manually:
./.github/scripts/install_triton.shOpus is a single-header C++ template library (opus.hpp) for writing HIP kernels on AMD GPUs — vectorized load/store, layout abstractions, and MFMA wrappers with a strong focus on build time optimization (up to 61x faster than standard torch extension builds). See the Opus README and op_tests/opus/ for details.
AITER supports GPU-initiated communication using the Iris library. This enables high-performance Triton-based communication primitives like reduce-scatter and all-gather.
pip install -e .
pip install -r requirements-triton-comms.txtFor more details, see docs/triton_comms.md.