Prebuilt binary demos of GPU kernels compiled by Blaze, a DSL and compiler for high-performance ML kernels: LLM inference (Qwen 3.5 9B, Qwen 3.6 27B, SmolLM 135M) and diffusion image generation (FLUX.2-klein, Bonsai 1-bit), for NVIDIA GPUs.
Each demo folder contains only compiler output and its runner: kernel shared objects
(.so), a runner executable, the Blaze runtime libraries, and a run script. There is no
compiler and no source in this repository. Binaries are stored with Git LFS.
Real recorded runs of the binaries in this repository, statistics unedited:
Qwen 3.5 9B Q4_K on RTX 4090 — one warm pass; prefill swings ±6% with GPU boost clocks |
FLUX.2-klein 4-step 1024² generation on RTX 4090 — per-step VAE decodes, image cosine 0.9995 |
Same kernel source on other backends (recordings only — those binaries are not in this repo)
Metal, Apple M4 GPU — Qwen 3.5 9B Q4_K · 155 tok/s prefill · 15.5 tok/s decode |
ARM SME + NEON, Apple M4 CPU only — Qwen 3.5 9B Q4_K · 43 ms/tok prefill · 11.4 tok/s decode |
This repository ships no model weights. You download them separately:
- LLM demos (Qwen, SmolLM): the run script downloads the GGUF from Hugging Face on first run, into the demo folder. You can also place the file there yourself (see the per-demo table for the exact repo and file name).
- Diffusion demos (FLUX.2-klein, Bonsai): run the bundled
export_*.pyscripts once on any machine with Python + PyTorch (klein also needsdiffusers). They download the checkpoint from Hugging Face and write the raw weight files the runners read.
Respect each model's own license when downloading weights.
- NVIDIA GPU matching one of the shipped architectures (see table below)
- NVIDIA driver new enough for your GPU (Blackwell needs r570+)
- Docker with the NVIDIA container toolkit (recommended), or a host with the CUDA 12.6 runtime libraries
git lfs(git lfs installbefore cloning, or rungit lfs pullafter)
Binaries are compiled per GPU architecture from the same Blaze source and schedules. Pick the folder matching your GPU family:
| Folder | Family | Measured on | Also runs on (same family) |
|---|---|---|---|
sm_80 |
Ampere (CC 8.0) | A100 | A30; loads on any CC 8.x board |
sm_89 |
Ada (CC 8.9) | RTX 4090, L4 | RTX 4080 / 4070, L40 / L40S, RTX 6000 Ada |
sm_120 |
Blackwell (CC 12.0) | RTX PRO 6000 Blackwell | RTX 5090 / 5080 (GeForce Blackwell) |
Notes: a cubin runs on its own compute-capability major version at the same or higher
minor version — so sm_80 loads on Ada too, but the sm_89 folder is the one tuned for
Ada. Data-center Blackwell (B200, CC 10.0) is a different family and is not covered.
sm_120 bundles were built with CUDA 12.8 — build the Docker image with the 12.8 base
for those (see below). Performance claims apply only to the boards in the "measured on"
column.
git lfs install
git clone <this-repo> && cd blaze-demos
# Build the measurement environment (CUDA 12.6 runtime)
docker build -t blaze-demos .
# For sm_120 (Blackwell) use the CUDA 12.8 base instead:
# docker build --build-arg CUDA_BASE=nvidia/cuda:12.8.1-runtime-ubuntu22.04 -t blaze-demos .
# Run the Qwen 3.5 9B demo on an sm_89 GPU (downloads the 5.3 GB GGUF on first run)
docker run --rm --gpus all -v "$PWD:/demos" blaze-demos \
bash -c "cd /demos/qwen3.5-9b/sm_89 && ./run_staged.sh"run_staged.sh runs the full measurement protocol — one discarded cold pass plus
warm repetitions — and prints a JSON scorecard with median/min/max prefill and decode
tokens per second.
| Folder | qwen3.5-9b/{sm_89,sm_80,sm_120} |
| Weights | unsloth/Qwen3.5-9B-GGUF → Qwen3.5-9B-Q4_K_M.gguf (5.3 GB, auto-downloaded) |
| Expected (warm) | RTX 4090: ~7,300 prefill / ~120 decode tok/s · RTX PRO 6000: ~7,700 / ~140–160 · A100: ~3,300 / ~117 · L4: ~2,000 / ~42 |
cd qwen3.5-9b/sm_89 && ./run_staged.sh
# or a single process with 5 warm passes (LD_LIBRARY_PATH must point at the demo folder):
LD_LIBRARY_PATH=$PWD:/usr/local/cuda/lib64 \
./qwen35_runner ./qwen35_decode.so ./qwen35_prefill.so ./Qwen3.5-9B-Q4_K_M.gguf \
--tokens 128 --stats-json --ignore-eos \
--prompt-file ./qwen35_prompt.txt --prompt-token-limit 256 --iters 5| Folder | qwen3.6-27b/{sm_89,sm_80,sm_120} |
| Weights | unsloth/Qwen3.6-27B-GGUF → Qwen3.6-27B-Q4_K_M.gguf (16.8 GB, auto-downloaded) |
| Expected (warm) | RTX 4090: ~2,000 prefill / ~32 decode · RTX PRO 6000: ~2,600 / ~38 · A100: ~1,000 / ~31 · L4: ~520 / ~12 |
cd qwen3.6-27b/sm_89 && ./run_staged.shSmall and fast to try — the weights are only ~270 MB.
| Folder | smollm-135m/{sm_89,sm_80,sm_120} |
| Weights | mradermacher/SmolLM-135M-GGUF → SmolLM-135M.f16.gguf (auto-downloaded) |
| Expected (warm) | RTX 4090: ~660–750 decode tok/s · measured on cloud VMs: L4/A100 ~480, RTX PRO 6000 ~670 |
Note on SmolLM numbers: at 135M parameters each decode step is a handful of very small kernels, so throughput is dominated by per-launch overhead rather than compute or bandwidth. That overhead varies a lot between hosts and virtualized cloud GPUs — much more than for the 9B/27B models. Treat the golden-token argmax MATCH plus the ≥350 tok/s floor as the pass criterion; do not read absolute SmolLM tok/s as a GPU comparison.
cd smollm-135m/sm_89 && ./run_staged.sh| Folder | flux2-klein/sm_89 |
| Weights | export once from black-forest-labs/FLUX.2-klein-4B (needs Python + torch + diffusers) |
| Expected | ~1.9 s warm pipeline on RTX 4090; decoded-image cosine ≥ 0.98 vs the bundled reference |
cd flux2-klein/sm_89
# One-time weight export (host side, downloads from Hugging Face):
python3 export_klein_weights.py --output-dir ./flux2_weights
python3 export_klein_vae_weights.py --output-dir ./flux2_vae_weights
# Run: 4 denoise steps -> VAE -> klein_out.ppm, prints image cosine vs reference
./run_pipeline.sh # single pass
./run_pipeline.sh --iters 6 # load weights once, 6 timed passes| Folder | bonsai-1bit/sm_89 |
| Weights | export once from prism-ml/bonsai-image-binary-4B-gemlite-1bit (needs Python + torch) |
| Expected | ~1.6 s warm pipeline on RTX 4090; decoded-image cosine ≥ 0.99 vs the bundled reference |
cd bonsai-1bit/sm_89
python3 export_bonsai_weights.py --outdir ./bonsai_weights
./run_pipeline.sh --iters 6The cond/ folders contain small pinned conditioning tensors (prompt embedding, noise,
RoPE tables) so every run is deterministic and comparable against the bundled
ref_decoded.bin reference image. They are demo inputs, not model weights.
- Numbers above are warm (weights resident, first pass discarded), matching the benchmark method in the infographic. A cold first pass is much slower — that is weight staging and JIT, not kernel speed.
- GPU boost clocks vary run to run by about ±5%. Do not read a single run as a regression or a win.
- Every LLM run self-checks:
"status":"ok"plus internal KV/recurrence health gates. Diffusion runs print the decoded-image cosine against the reference — that is the correctness gate. - Remote GPUs (A100, L4, RTX PRO 6000 Blackwell) were provisioned and driven through
colab-clisessions, running these same staged bundles with the same protocol.
The only numbers we stand behind are ones produced by the shipped protocol:
run_staged.sh / run_pipeline.sh --iters N, or the runner's --iters mode — warm,
weights resident, first pass discarded, best/median of the warm passes, correctness gates
passing. Expect ±5% between runs from GPU boost clocks alone.
We do not support or debate numbers produced other ways: cold single-shot runs quoted as throughput, repeat-loop microbenchmarks over hot caches, mismatched prompt lengths or contexts between systems, or runs whose correctness gates were not checked. If you get a different number, first check: same GPU, same protocol, same prompt, gates green.
The llama.cpp peer numbers in the infographic were measured with llama-completion
(a real end-to-end run, not the llama-bench microbenchmark), built from upstream
llama.cpp at commit 0cea36222fe9bac5ebfc45716c9eef11f37046c4, compiled on the same
machine as each run with its native CUDA architecture (-DLLAMA_BUILD_UI=OFF):
llama-completion -m <model.gguf> -f <prompt-256tok.txt> -n 64 -ngl 99 -c 4096 -t 8 --perfScored the same way as Blaze: 6 runs, first discarded, best of 5 warm, parsing the reported "prompt eval" / "eval" tokens-per-second.
Blaze is a DSL and compiler for high-performance ML kernels. One kernel source and one schedule set produced every binary here — each architecture folder is just a recompile with a different target-architecture flag. The compiler is not yet public; this repository exists so the results can be reproduced and inspected as-is.
See LICENSE: the binaries may be downloaded, run, and benchmarked for evaluation;
redistribution requires permission. Model weights are downloaded separately and carry
their own licenses.