Spectacles LEGO AR

What happens when you add AR glasses to LEGO?

A proof of concept built on Snap Spectacles where Google Gemini sees any LEGO creation you build, generates a unique sound for it, and lets you grab it with your hands.

Shake a plane, hear engines. Wave a dragon, it roars. Every build, every time, completely different.

Watch the full video on LinkedIn

How It Works

1. Pinch to scan — A single pinch captures the scene through the Spectacles camera and depth sensor
2. Gemini 3 Flash analyzes the scene — The camera frame is sent to Gemini 3 Flash as a vision request. Gemini identifies every LEGO creation in view and returns structured JSON with bounding boxes, labels, creative sound prompts, collision sound prompts, dominant colors, a scene description, and a music style — all in a single request
3. Sound generation — Each object's sound prompt is sent to TangoFlux on Replicate, generating unique sounds in just a few seconds. Google Lyria generates background music tailored to the scene description
4. Grab & play — Using Spectacles hand tracking, pick up your LEGO creations, shake them, toss them around — each interaction triggers its sound. Collisions between objects play their own impact sounds

Detection happens only once at the start. After that, everything is driven by real-time hand tracking.

Gemini 3 Flash — The Brain

Gemini 3 Flash (gemini-3-flash-preview) powers the core intelligence of this experience. Here's what it does in a single multimodal request:

• Object detection — Identifies all LEGO creations in the camera frame and returns bounding box coordinates for each one
• Creative labeling — Gives each creation a descriptive name based on what it looks like (car, dragon, spaceship, etc.)
• Sound design — Generates a creative text prompt describing a fun, cartoonish sound for each object (e.g., "cartoon car engine revving with fun vroom sounds") as well as a collision sound prompt (e.g., "plastic brick click clack impact")
• Color extraction — Picks the dominant color of each creation for VFX
• Scene understanding — Writes a short atmospheric description of the overall scene
• Music direction — Suggests a music style that matches the energy of the scene, which is then passed to Google Lyria for background music generation

All of this comes back as structured JSON using Gemini's response_schema feature, so there's no parsing guesswork — the response is guaranteed to match the expected format.

The Bigger Picture

LEGO Smart Bricks already blew our minds. Now imagine combining both: Smart Play's physical sensors inside the bricks + AR + generative AI around them. A LEGO city that reacts from the inside and comes alive through your glasses.

That's not sci-fi. All the pieces exist today. We just played with them.

Tech Stack

• Platform: Lens Studio for Snap Spectacles
• Language: TypeScript
• Vision AI: Google Gemini 3 Flash (scene analysis, object detection, sound/music prompt generation)
• Sound Generation: TangoFlux via Replicate (open-source text-to-audio model)
• Background Music: Google Lyria
• Hand Tracking: Spectacles Interaction Kit (SIK)

Setup

Prerequisites

• Lens Studio 5.15+
• A Google API key (for Gemini + Lyria)
• A Replicate API token (for sound generation)

Configure API Tokens

1. Google token — Open the project in Lens Studio, select the RemoteServiceGatewayCredentials object in the scene, and enter your Google API key in the Inspector panel
2. Replicate token — Open Assets/Scripts/GeminiAPI.ts and replace the placeholder on line 27:

   const REPLICATE_API_TOKEN = "your-replicate-token-here";

Run

1. Open spectacles-lego.esproj in Lens Studio
2. Configure the tokens as described above
3. Build and deploy to Spectacles (or use the Lens Studio preview)

Architecture

User pinch → Capture depth + camera frame → Gemini 3 Flash vision analysis →
Spawn grabbable objects at world positions → Generate unique sounds via TangoFlux →
Generate background music via Lyria → Grab, shake, and play with hand tracking

Script	Role
SceneController.ts	Main orchestrator — coordinates all systems, spawns objects with VFX
GeminiAPI.ts	Gemini 3 Flash vision requests + TangoFlux/Lyria audio generation
DepthCache.ts	Depth frame capture, pixel-to-world coordinate conversion
GrabbableBox.ts	Hand-grabbable AR objects with velocity-triggered sounds and collision detection
VFXPool.ts	Manages spawn and collision VFX instances
WorldLabel.ts	3D world-space labels for detected objects
MusicPlayer.ts	Audio playback for generated sounds and music

Credits

Built by Stijn Spanhove and Pavlo Tkachenko.

License

MIT