Orin TinyNAS Experiment

Educational TinyNAS-inspired NAS on CIFAR-10 (Linux/ Colab) + TensorRT deployment on Nvidia Orin NX Super (JetPack 6).

Start here: STUDENT_GUIDE.md — full step-by-step reproduction procedure.

Acknowledgments & references

This project is inspired by Professor Song Han's TinyNAS and TinyEngine work from MIT HAN Lab and the MIT 6.5940 Efficient AI Computing course (Lab 3: Neural Architecture Search). We did not fork the MIT codebase; we reimplemented the two-stage NAS idea and adapted the deployment path for Nvidia Jetson Orin NX Super instead of Cortex-M microcontrollers.

Resource	Link
TinyNAS (search space & two-stage NAS)	github.com/mit-han-lab/mcunet/tree/master/mcunet/tinynas
TinyEngine (MCU inference engine)	github.com/mit-han-lab/mcunet/tree/master/mcunet/tinyengine → github.com/mit-han-lab/tinyengine
MIT 6.5940 course (Fall 2026)	hanlab.mit.edu/courses/2026-fall-65940

Results at a glance

End-to-end pipeline: proxy NAS → fine-tune → ONNX export → TensorRT on Orin → benchmark.

Stage	Metric	Result
NAS (proxy, rank #1)	Validation accuracy	85.98%
NAS (proxy, rank #1)	Parameters	111,538 (~112K)
Fine-tune (80 epochs)	Validation accuracy	92.05%
Orin TRT FP16	Top-1 accuracy / latency / FPS	92.03% / 1.00 ms / 1003
Orin TRT INT8	Top-1 accuracy / latency / FPS	91.86% / 0.96 ms / 1041

Measured on real Orin NX Super hardware with local CIFAR-10; see results/benchmark.json.

Discovered architecture (rank #1): 4-stage MBConv subnet, width_mult=0.75 — compact enough for edge deployment while reaching ~92% on CIFAR-10.

Our contributions beyond MIT TinyNAS / TinyEngine

After adopting Professor Han's Lab 3 two-stage NAS concept, we extended it into a full train-on-cloud, deploy-on-edge pipeline on real Orin hardware — not a direct reproduction of the MCU toolchain.

From Lab 3 concept to Orin hardware

MIT TinyNAS searches for MCU-scale architectures; TinyEngine generates hand-written INT8 C for STM32-class devices. We closed the loop on Orin NX Super (JetPack 6): evolutionary proxy NAS on Colab/Linux GPU → fine-tune → ONNX export → on-device TensorRT engine build → benchmark. Measured on real hardware (results/benchmark.json): 92.03% top-1 (FP16), 91.86% (INT8), ~1 ms latency, >1000 FPS batch-1 on 32×32 CIFAR-10 at ~112K parameters.

TinyEngine-inspired engine, TensorRT-backed

Instead of TinyEngine C codegen, we built OrinEngine (deploy/infer_trt.py) — pre-allocated buffers and async TensorRT v3 execution in the spirit of TinyEngine's static-memory design. FP16 and INT8 engines are built via deploy/build_trt_engine.py with an entropy calibrator fed by a training-time calibration cache. INT8 drops only ~0.17% accuracy vs FP16 while achieving slightly lower latency.

Orin-specific reproduction challenges we addressed

The original TinyEngine stack does not cover Jetson GPU deployment. Issues we hit and resolved on Orin:

Challenge	Our solution
TensorRT .plan files are GPU-specific; Colab T4 engines won't run on Orin	Build engines on the Orin after USB transfer (scripts/orin/01_build_engines.sh)
JetPack TensorRT lives in system Python, not conda	PYTHONPATH wiring and verify script (scripts/orin/00_verify_env.sh)
pycuda / ARM compile failures on Jetson	ctypes wrapper over JetPack libcudart.so — no compile step (deploy/cuda_runtime.py)
Long Colab NAS runs disconnect	Resumable search_state.json + Google Drive (notebook §4)
Offline Orin benchmark	artifacts.zip + SHA manifest; no internet required on device (train.py)

Additional adaptations for GPU edge: TRT-friendly search space (MBConv, Conv/BN/ReLU6), efficiency-aware fitness (val_acc - 0.1 × log10(params)), free/pro search tiers, and a MobileNetV2-0.35 baseline for comparison. The Colab notebook, Orin shell scripts, and STUDENT_GUIDE.md form a reproducible lab workflow absent from the MIT MCU toolchain.

Results close to state of the art at ~112K parameters

• MIT target: sub-100KB SRAM models on Cortex-M MCUs via TinyEngine.
• Our target: ~112K params, ~92% CIFAR-10, sub-ms GPU inference on Orin — competitive accuracy and latency for a model small enough for edge deployment.

See Concept mapping below for a side-by-side comparison.

Why this approach

Advantage	What it means
Search + deploy on real hardware	NAS runs on cloud GPU; TensorRT engines are built on the Orin so binaries match the actual edge device.
Tiny but accurate	Evolutionary search under a param budget (~0.1–2.5M) yields a ~112K model at ~92% after fine-tuning — far smaller than typical ResNet-style baselines.
INT8 with minimal loss	Calibration cache from training enables INT8 TRT with only ~0.17% accuracy drop vs FP16, plus slightly lower latency.
Sub-millisecond inference	<1 ms per image and >1000 FPS (batch-1) on Orin for 32×32 CIFAR inputs — suitable for real-time edge vision.
Reproducible two-stage NAS	Short proxy epochs screen hundreds of candidates; full fine-tune only the best subnet — same idea as MIT TinyNAS, adapted for GPU edge.
Jetson-friendly deploy	No pycuda compile: deploy/cuda_runtime.py uses JetPack libcudart.so; system TensorRT + conda orin-tinynas env.
Offline-friendly	CIFAR-10 and artifacts.zip can be transferred manually — no internet required on Orin for benchmark.

Quick overview

Linux / Colab (training)            Transfer              Orin NX (deploy)
────────────────────────             ────────              ─────────────────
train.py / notebook    →           artifacts.zip    →       scripts/orin/*.sh
  Proxy NAS (evolutionary)           SCP / USB              TensorRT FP16/INT8
  Fine-tune rank #1                                         Benchmark
  Export ONNX + INT8 calib

Files

File	Purpose
train.py	Stage 1 NAS + Stage 2 fine-tune + export artifacts.zip
orin_tinynas_colab.ipynb	Self-contained Colab notebook
STUDENT_GUIDE.md	Complete lab procedure for students
environment-orin.yml	Conda env for Orin
scripts/orin/	Verify, build engines, benchmark
deploy/	TensorRT build + OrinEngine inference
results/benchmark.json	FP16/INT8 accuracy, latency, FPS on Orin
search_state.json	NAS result after 20-generation pro-tier search

Orin quick start

cd ~/orin-tinynas
conda env create -f environment-orin.yml
conda activate orin-tinynas
unzip -o artifacts.zip
bash scripts/orin/00_verify_env.sh
bash scripts/orin/01_build_engines.sh
bash scripts/orin/02_benchmark.sh

Important

• TensorRT .plan files must be built on Orin — not on the training GPU (engines are hardware-specific).
• NAS checkpoint resume: search_state.json is saved each generation on disk; Colab users should use Google Drive.
• System python3 on Orin may not have TensorRT — use conda (orin-tinynas env).

Concept mapping

MIT TinyNAS / TinyEngine	This project
Two-stage NAS	Evolutionary proxy search → fine-tune (Linux / Colab)
TinyEngine MCU codegen	TensorRT FP16/INT8 on Orin GPU
INT8 inference	TRT INT8 + calibration cache from training
Sub-100KB MCU models	~112K params, ~92% CIFAR-10, <1 ms on Orin

Team member & contact

Name	Email
Liang Zhang	liangzhang@xidian.edu.cn
Guangming Zhu	gmzhu@xidian.edu.cn
Peiyi Shen	pyshen@xidian.edu.cn
Johann Li	me@qinka.pro
Xunchi Ma	xunchi_ma@stu.xidian.edu.cn
Wenhui Wei	25031212119@stu.xidian.edu.cn
Kexin Zhang	25031111104@stu.xidian.edu.cn

License

Educational use. CIFAR-10 by Toronto; PyTorch/TensorRT per their licenses.