Audora: CNN Audio Classification & Visualizer

Audora is an end-to-end educational and diagnostic application designed to show how a Convolutional Neural Network (CNN) processes, abstracts, and classifies audio signals. The system takes raw audio files, converts them into 2D Mel Spectrograms, feeds them through a custom residual convolutional network, and visualizes both the intermediate feature activation maps and the final predictions.

The codebase is split into two primary components:

1. Core Deep Learning & Inference Serverless Backend (model.py, train.py, main.py) powered by PyTorch, Torchaudio, and Modal.
2. Interactive Visualizer Frontend (audio-cnn-visualization/) built using Next.js (TypeScript, React, Tailwind CSS, shadcn/ui).

---

Machine Learning Pipeline Details

1. Audio Standardization:
   • Multi-channel inputs are average-mixed to mono.
   • Inputs are resampled to 44,100 Hz.
2. Feature Extraction:
   • The waveform is transformed into a Mel Spectrogram using a sample rate of 22,050 Hz, FFT window size of 1024, hop length of 512, and 128 Mel bands.
   • Sound pressure levels are converted to decibels (AmplitudeToDB).
3. Activation Visualization:
   • For an intermediate layer output of shape [batch_size, channels, height, width], the server calculates the mean across the channel dimension: $$A(x, y) = \frac{1}{C}\sum_{c=1}^{C} F(c, x, y)$$ where $A(x, y)$ is the aggregated 2D activation intensity at coordinates $(x, y)$, $C$ is the channel count, and $F(c, x, y)$ is the feature map activation at channel $c$.
   • This down-samples high-dimensional features into a singular 2D grid per block, mapping relative activation intensities directly to CSS color scales.

---

Setup & Deployment Guide

Prerequisites

• Python 3.10+
• Node.js v18+ and npm
• A Modal account (for model training and inference hosting)

1. Backend Setup & Training

Install the local dependencies (preferably in a virtual environment):

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Authenticate with Modal:

modal setup

To run model training on the Modal cloud:

modal run train.py

This downloads the ESC-50 dataset directly into the cloud volume, runs training for 100 epochs using an A10G GPU, logs results via TensorBoard, and saves the best model checkpoint to the esc-model Modal volume.

To deploy the inference server:

modal deploy main.py

This exposes a production-ready HTTP POST endpoint.

2. Frontend Development Setup

Navigate to the frontend application:

cd audio-cnn-visualization

Install packages:

npm install

Start the local development server:

npm run dev

Open http://localhost:3000 to interact with the visualizer dashboard.

Model Performance & Training Specs

• Dataset: ESC-50 (Environmental Sound Classification, 2000 5-second recordings).
• Architecture: Deep Residual CNN (customized ResNet).
• Optimization: AdamW optimizer ($lr=0.0005$, weight decay $= 0.01$), OneCycleLR scheduler ($max_lr = 0.002$), Label Smoothing (0.1) on Cross-Entropy Loss.
• Augmentations: Time & Frequency masking (SpecAugment) + Mixup ($Beta(0.2, 0.2)$).
• Target Metric: Validation accuracy evaluated on Fold 5 (standard out-of-fold benchmark).