Specto

A Snapchat Spectacles lens that captures sports highlights automatically. You tap on the glasses (or in future, say a voice command) and it saves 30 seconds before and 30 seconds after the trigger. So you never miss any exciting moment even if you are late to react.

Built using Lens Studio 5.15.3 with TypeScript.

What it Does

Basically the lens keeps recording camera frames continuously in a rolling buffer. When you tap to trigger capture, it already has 30 seconds of past footage stored. Then it records 30 more seconds after the tap. Total you get around 60 seconds of footage with the highlight moment right in the middle.

After capturing, the frames get uploaded to cloud storage (Supabase) and optionally analyzed using Google Gemini AI to understand whats happening in the sports moment.

How the Flow Works

1. Lens starts and begins buffering camera frames at ~10 FPS
2. Rolling buffer keeps last 30 seconds of frames, old ones get removed automatically
3. You tap on the Spectacles to trigger capture
4. It locks the current 30 second buffer and starts recording 30 more seconds
5. After recording is done, all frames (pre + post) are combined and sorted by timestamp
6. Frames metadata gets uploaded to cloud
7. UI shows "Saved!" and system resets, ready for next capture

The status UI floats in world space in front of you and shows whats happening - "Listening...", "Capturing... 15/30s", "Uploading... 45%", "Saved!" etc.

Project Structure

specto/
├── Assets/
│   ├── Scripts/
│   │   ├── CaptureManager.ts      # Main orchestrator, coordinates everything
│   │   ├── VoiceDetector.ts        # Handles tap trigger (voice ASR planned for future)
│   │   ├── FrameBuffer.ts          # Rolling buffer that stores camera frames
│   │   ├── CloudUploader.ts        # Uploads frame data to Snap Cloud Storage
│   │   ├── StatusUI.ts             # World-space floating status display
│   │   ├── CloudUpload.js          # Supabase + Gemini AI integration
│   │   └── CloudIntegration.js     # Example code for cloud integration patterns
│   ├── Scene.scene                 # Main Lens Studio scene
│   ├── Camera Module.cameraModule  # Camera control
│   ├── Asr Module.asrModule        # Speech recognition (for future use)
│   └── Device Camera Texture       # Live camera feed
├── Packages/
│   ├── SpectaclesInteractionKit.lspkg
│   └── SpectaclesUIKit.lspkg
├── specto.esproj                   # Lens Studio project file
├── tsconfig.json
└── jsconfig.json

Core Components

CaptureManager

The main brain of the whole thing. It connects VoiceDetector, FrameBuffer, CloudUploader and StatusUI together. Manages the capture flow states - Idle, Listening, Recording, Uploading, Complete, Error. Has auto-start option so it begins listening as soon as the lens opens.

VoiceDetector

Currently it uses tap gesture (TouchStartEvent/TapEvent) as the trigger. Voice ASR module input is there but the API is not ready yet in Lens Studio, so tap is the primary method for now. Has a 2 second cooldown so you dont accidentally trigger it twice.

FrameBuffer

This is where the magic happens. It continuously captures frames from the device camera at 10 FPS and keeps them in a rolling buffer. When buffer is 30 seconds full (actually checks for 90% which is 27 seconds), it is considered ready. On trigger, it locks the current buffer and records 30 more seconds. Then combines everything, sorts by timestamp, and sends to the uploader.

CloudUploader

Handles uploading to Snap's CloudStorageModule. Stores session metadata and frame information as key-value pairs. If cloud storage is not available, it simulates the upload (useful for testing). Shows progress in 10 batches from 0 to 100%.

StatusUI

A world-space Text3D display that floats about 80cm in front of the camera. It has billboard effect so it always faces you. Shows different states with different colors - white for listening, red for capturing (with pulsing animation), orange for uploading, green for saved. Auto hides after 3 seconds on success and 5 seconds on error.

CloudUpload.js (Supabase + Gemini)

Separate module for external cloud integration. Uploads moments to Supabase storage bucket and can analyze frames using Google Gemini AI. The AI analysis is meant for sports context - it can identify whats happening in a football game, detect exciting plays, extract game info like teams and scores.

Tech Stack

• Platform: Snapchat Spectacles (Lens Studio 5.15.3)
• Language: TypeScript + JavaScript
• Target: ES2021
• Cloud Storage: Snap CloudStorageModule + Supabase
• AI: Google Gemini 1.5 Flash
• UI: SpectaclesUIKit, SpectaclesInteractionKit
• Min Client: Snapchat 13.60+

Setup

Prerequisites

• Lens Studio 5.15.3 or later
• Snapchat Spectacles device (or use Lens Studio preview)

Opening the Project

1. Open Lens Studio
2. Go to File > Open Project
3. Select specto.esproj

Setting Up Components in Scene

1. Create scene objects for each component (CaptureManager, VoiceDetector, FrameBuffer, CloudUploader, StatusUI)
2. Assign the Device Camera Texture to FrameBuffer's cameraTextureInput
3. Link all components to CaptureManager via the Inspector panel
4. For StatusUI, assign a Text3D component and optionally a Camera reference for billboard effect

Cloud Upload Setup (Optional)

If you want to use Supabase for cloud uploads:

1. Create a free account at supabase.com
2. Create a storage bucket called spectacles-moments
3. Get your project URL and anon key from Supabase Settings
4. Update the values in CloudUpload.js:

   SUPABASE_URL = "https://your-project-id.supabase.co"
   SUPABASE_ANON_KEY = "your-anon-key"
   

Gemini AI Setup (Optional)

For AI analysis of captured moments:

1. Get API key from Google AI Studio
2. Update in CloudUpload.js:

   GEMINI_API_KEY = "your-gemini-api-key"
   

Configuration

Some things you can change through Inspector in Lens Studio:

Parameter	Default	Description
autoStartListening	true	Start listening for triggers on lens open
enableDebugLogs	true	Print debug logs to console
distanceFromCamera	80	Status UI distance from camera (cm)
verticalOffset	-15	Status UI vertical offset (cm below eye level)

In the code you can also change these:

Parameter	Default	Location
BUFFER_DURATION_MS	30000 (30s)	FrameBuffer.ts
RECORD_DURATION_MS	30000 (30s)	FrameBuffer.ts
FRAME_INTERVAL_MS	100 (10 FPS)	FrameBuffer.ts
COOLDOWN_MS	2000 (2s)	VoiceDetector.ts

How to Use

1. Open the lens on your Spectacles
2. Wait for about 27 seconds so the buffer fills up (you will see "Listening..." on screen)
3. When something exciting happens, tap on the Spectacles
4. The lens will record 30 more seconds after your tap
5. After recording, it uploads automatically
6. You will see "Saved!" when its done
7. System resets and you can capture again

Notes

• The buffer needs atleast 27 seconds (90% of 30s) to be ready. If you tap too early, it shows "Wait a moment..."
• Actual frame texture upload to external servers needs a backend with RemoteServiceModule. The CloudUploader currently stores metadata only in Snap's cloud storage
• Voice trigger using ASR is planned but not implemented yet because the Lens Studio API for it is not fully available
• The render target is 900x1600 which is the Spectacles screen aspect ratio
• Both front and back cameras of Spectacles are supported

License

This project is for Snapchat Spectacles development.