Rhythmic Gestures

Rhythmic Gestures is a webcam-controlled rhythm game that challenges players to perform hand gestures while music plays. The application combines real-time computer vision, gesture recognition, audio analysis, and an interactive scoring loop.

Built with Python, MediaPipe, OpenCV, librosa, and Pygame.

How it works

1. OpenCV captures frames from the player’s webcam.
2. MediaPipe classifies the player’s current hand gesture.
3. The game displays a randomly selected target gesture.
4. Matching the target gesture awards 100 points.
5. Pygame plays the selected audio track while the game runs.
6. librosa estimates the track’s tempo for future rhythm synchronization.

The current gesture vocabulary includes:

• Closed Fist
• Open Palm
• Pointing Up
• Thumb Down
• Thumb Up
• Victory
• I Love You

Architecture

Webcam
   |
   v
OpenCV frame capture
   |
   v
MediaPipe gesture recognition
   |
   v
Compare detected gesture with target
   |
   +---- match ----> update score
   |
   +---- no match -> continue game loop

Audio file
   |
   +---- Pygame playback
   |
   +---- librosa tempo analysis

The application separates gesture prompts from the main game controller:

• Gesture selects and displays target gestures.
• Game manages webcam capture, model inference, scoring, audio playback, and the main event loop.
• gesture_recognizer.task provides the MediaPipe gesture-recognition model.

Requirements

• Python 3
• A working webcam
• A graphical desktop environment
• A legally obtained WAV audio file

Python packages:

• mediapipe
• opencv-python
• librosa
• pygame

Installation

Clone the repository:

git clone https://github.com/Colin243/Rhythmic-Gestures.git
cd Rhythmic-Gestures

Create and activate a virtual environment:

python3 -m venv .venv
source .venv/bin/activate

On Windows:

python -m venv .venv
.venv\Scripts\activate

Install the dependencies:

pip install mediapipe opencv-python librosa pygame

Audio setup

The current prototype expects a WAV file whose path is configured near the top of game.py.

Use an audio file that you have permission to use, place it in the project directory, and update both audio references:

y, sr = librosa.load("your-audio-file.wav")
my_sound = pygame.mixer.Sound("your-audio-file.wav")

If the new track has a different duration, update the corresponding track-length value as well.

Running the game

Start the application:

python game.py

A window named Gesture Tracking will display the mirrored webcam feed, target gesture, and score.

Press q to exit.

For the best recognition results:

• Keep your hand clearly visible.
• Use an evenly lit environment.
• Avoid placing your hand too far from the camera.
• Hold each gesture briefly so it can be classified.

Project structure

Rhythmic-Gestures/
├── data/
│   ├── gesture_recognizer.task
│   └── hand_landmarker.task
├── game.py
├── test.py
└── README.md

• game.py contains the playable gesture game.
• data/gesture_recognizer.task contains the MediaPipe model used by the game.
• test.py contains experimental Spotify integration and is not required to run the main application.

Current limitations

This repository is an early prototype.

• Gesture prompts currently change on a timer rather than using the exact beat timestamps returned by librosa.
• Audio configuration and duration are hard-coded.
• The game supports one level and a fixed scoring rule.
• Recognition confidence is not currently displayed or used to adjust scoring.
• There is no start screen, results screen, pause control, or difficulty selection.
• The experimental Spotify code is separate from the playable game.
• Automated tests and a dependency lock file have not yet been added.

Possible future improvements

• Schedule gesture prompts using librosa’s detected beat frames.
• Add configurable songs and automatic duration detection.
• Support difficulty levels and increasingly rapid gesture sequences.
• Introduce streaks, timing accuracy, and confidence-weighted scoring.
• Add start, pause, results, and song-selection screens.
• Separate camera, audio, recognition, and gameplay logic into testable modules.
• Add automated tests for scoring and gesture-transition behavior.
• Package the application with reproducible dependency versions.

Motivation

I built Rhythmic Gestures to explore how real-time computer vision can become an interactive input mechanism. The project gave me experience integrating continuous webcam processing, ML-based gesture classification, audio playback, and stateful game logic in a latency-sensitive loop.

Author

Colin Ho
Computer Science and Mathematics, Tufts University
GitHub