🏎️ RACER

Retrieval-Augmented Contextual Rapid Speculative Decoding

---

📰 News

• 2026-09-12: RACER is now implemented in SGLang — see sgl-project/sglang#39211. Feedback is welcome!

---

📋 Table of Contents

• Installation
• Core Code
• Tests
• Supported LLMs
• Inference
  • Evaluation
  • Speed Analysis
  • Free Chat
• Benchmarks
• Acknowledgement

---

🛠️ Installation

conda create -n racer python=3.10
conda activate racer
cd RACER
pip install -r requirements.txt

---

🧩 Core Code

The speculative-decoding automaton is the algorithmic core of RACER:

Path	Description
racer/automaton/src/automaton.cpp	Handwritten C++ core: Logits Tree (Sec. 3.1), Retrieval Tree / LRU / failure links (Sec. 3.2), Tree Attention flatten (Sec. 2)
racer/automaton/automaton.pyx	Cython bindings for the C++ automaton
racer/automaton/__init__.py	Python API: from racer.automaton import Automaton
racer/model/racer_model.py	RACER model wrapper used at inference time

racer/automaton/automaton.cpp is generated by Cython and is not source of truth.

---

🧪 Tests

Rebuild the automaton extension, then run the CPU-only paper-alignment unit tests:

pip install -e ./racer/automaton
python -m unittest tests.test_automaton_paper_alignment -v

These tests cover Logits Tree breadth (Eq. 3 / Algorithm 1), Tree Attention (Eq. 2), default border depth, constructor clamping, and draft-capacity bounds. They do not require a GPU.

---

🤖 Supported LLMs

Modality	Models
📝 Text-only	OpenPangu | Qwen3 (Dense / MoE) | LLaMA 2/3 | Mixtral
🖼️ Multimodal	Qwen2.5/3-VL | LLaVA

---

🚀 Inference

Evaluation

Run benchmarks to measure mean accepted tokens and speedup ratios. Dataset sources, math scoring, and the full --bench-name list are in docs/benchmarks.md.

For a quick sanity check, run GSM8K with thinking turned off on the first 20 questions:

CUDA_VISIBLE_DEVICES=0 python -m evaluation.inference_racer \
  --model-path qwen/qwen3-1.7b \
  --model-id qwen3-1.7b-racer \
  --bench-name gsm8k \
  --enable-thinking False \
  --question-end 20

---

Speed Analysis

Measure decoding throughput and compare against baselines:

# Single-file speed analysis
python evaluation/speed.py <file_path>

# Compare with a baseline (calculates speedup ratio)
python evaluation/speed.py <file_path> --base <base_path>

# Batch analysis across an entire folder
python evaluation/batch_speed.py <folder> --base <base_path>

---

Free Chat

Accepted draft tokens (excluding the sampled next token) are highlighted in green.

Terminal CLI

CUDA_VISIBLE_DEVICES=0 python -m racer.inference.cli --model-path qwen/qwen3-1.7b

# Optional arguments:
#   --temperature   Sampling temperature  (default: 0.7)
#   --top-p         Nucleus sampling p    (default: 0.8)

Web UI

CUDA_VISIBLE_DEVICES=0 python -m racer.inference.webui --model-path qwen/qwen3-1.7b

---

Acknowledgement

We would like to thank openPangu for its support. openPangu is a trademark of Huawei Technologies Co., Ltd. For more information, please refer to the official openPangu repository and the accompanying license files.