Multi-Sequence Speech Recognition using Hidden Markov Models

This project implements Hidden Markov Models (HMMs) for a speech recognition task, extending classic HMM applications to classify sequences of spoken digits ("zero", "one", "two") from noisy acoustic data. The implementation processes multiple observation sequences, performs parameter estimation using the Baum-Welch algorithm, conducts model comparison under varying noise levels, and analyzes robustness.

HMM Specification

The HMM is defined with:

• States: Three hidden states, $S = \lbrace Z, O, T \rbrace$, for "zero," "one," "two."
• Observations: Four acoustic symbols, $V = \lbrace A, B, C, D \rbrace$.
• Transition probabilities (A):

$$ A = \begin{pmatrix} 0.5 & 0.3 & 0.2 \\ 0.3 & 0.4 & 0.3 \\ 0.2 & 0.3 & 0.5 \end{pmatrix} $$

• Emission probabilities (B):

$$ B = \begin{pmatrix} 0.4 & 0.3 & 0.2 & 0.1 \\ 0.2 & 0.3 & 0.3 & 0.2 \\ 0.1 & 0.2 & 0.3 & 0.4 \end{pmatrix} $$

• Initial state probabilities ($\pi$):

$$ \pi = \begin{pmatrix} 0.4 & 0.3 & 0.3 \end{pmatrix} $$

Tasks Implemented

Task 1: Forward Algorithm (Multiple Sequences)

For three observation sequences:

• $O_1 = \lbrace A, B, C, D \rbrace$
• $O_2 = \lbrace B, C, A, D \rbrace$
• $O_3 = \lbrace C, D, A, B \rbrace$

Compute $P(O_i \mid \lambda)$ for each using the forward algorithm.

Task 2: Viterbi Algorithm

For each sequence, find the most likely state sequence using the Viterbi algorithm with backtracking.

Task 3: Baum-Welch Algorithm

Implement Baum-Welch to estimate parameters $(\pi, A, B)$ using all three sequences jointly. Perform two iterations and report updated parameters.

Task 4: Model Comparison

Train two HMMs:

• One with 10% noise (randomly flip observation symbols)
• Another with 20% noise

Compare performance by computing average log-likelihood on a test set of 50 simulated sequences (length 4, no noise). Plot the log-likelihood distributions.

Task 5: Simulation and Robustness

• Simulate 200 sequences (length 4) using the true HMM.
• Split into training (150) and test (50) sets.
• Train HMMs with 0%, 10%, and 20% noise levels (5 iterations each).
• Plot the Mean Absolute Error (MAE) of the transition matrix over iterations for each noise level.

Machine Learning Methodology

Algorithms Implemented

Algorithm	Purpose
Forward	Compute $P(O \mid \lambda)$ with scaling to prevent numerical underflow
Viterbi	Decode the most likely hidden state sequence
Baum-Welch	EM algorithm for parameter estimation from multiple observation sequences

Data Simulation

Sequences are simulated using the true HMM parameters. Noise is introduced by randomly flipping observation symbols with a given probability.

Noise levels tested:

• 0% Noise — Clean data (baseline)
• 10% Noise — Moderate corruption
• 20% Noise — High corruption

Evaluation Metrics

Metric	Description
Log-Likelihood	Measures how well the trained model explains test data
Mean Absolute Error (MAE)	Average absolute difference between estimated and true transition matrices

Project Structure

multi-sequence-speech-recognition/
│
├── code.ipynb                       # Complete HMM implementation
│   ├── forward()                    # Forward algorithm with scaling
│   ├── viterbi()                    # Viterbi decoding with backtracking
│   ├── baum_welch()                 # Parameter estimation (EM)
│   ├── simulate_sequence()          # Generate synthetic data
│   └── add_noise()                  # Inject observation noise
│
├── outputs/
│   ├── log_likelihood_dist.png      # Histogram (Task 4)
│   ├── mae_convergence.png          # MAE over iterations (Task 5)
│   └── console_output.txt           # Forward, Viterbi, Baum-Welch results
│
└── README.md                        # This documentation

Results & Visualizations

Task 1 & 2: Sequence Analysis

Sequence	$P(O \mid \lambda)$ via Forward	Most Likely Hidden Path (Viterbi)
$O_1$: A,B,C,D	$\approx 0.004237$	['Z', 'Z', 'T', 'T']
$O_2$: B,C,A,D	$\approx 0.003521$	['Z', 'Z', 'Z', 'T']
$O_3$: C,D,A,B	$\approx 0.003546$	['T', 'T', 'Z', 'Z']

Task 3: Baum-Welch Parameter Updates

After 2 iterations of Baum-Welch using all three sequences jointly:

Updated Transition Matrix A:

$$ A_{\text{new}} = \begin{pmatrix} 0.4046 & 0.3333 & 0.2621 \\ 0.2542 & 0.3886 & 0.3572 \\ 0.2236 & 0.2763 & 0.5001 \end{pmatrix} $$

Updated Emission Matrix B:

$$ B_{\text{new}} = \begin{pmatrix} 0.3734 & 0.3154 & 0.1985 & 0.1128 \\ 0.2220 & 0.2698 & 0.2792 & 0.2290 \\ 0.1431 & 0.1575 & 0.2767 & 0.4227 \end{pmatrix} $$

Updated Initial Distribution $\pi$:

$$ \pi_{\text{new}} = \begin{pmatrix} 0.4926 & 0.3096 & 0.1977 \end{pmatrix} $$

Task 4: Log-Likelihood Distribution

• 10% Noise Model: Average Log-Likelihood $\approx 5.5676$
• 20% Noise Model: Average Log-Likelihood $\approx 5.5674$

Both models achieve similar test performance, suggesting Baum-Welch is reasonably robust to moderate observation noise.

Task 5: MAE Convergence (Transition Matrix A)

Iteration	0% Noise	10% Noise	20% Noise
0	0.104586	0.103374	0.102812
1	0.101004	0.099931	0.099464
2	0.097769	0.096844	0.097921
3	0.095137	0.094077	0.097058
4	0.094093	0.091587	0.096347

References

As mentioned in the assignment: Hidden Markov Models (HMMs) were pioneered by Leonard E. Baum in the 1960s, with notable applications in IBM's Tangora system (1980s) and Bell Labs' SPHINX system (1980s-1990s) for speech recognition.