LLMs and Generative AI › Multimodal and Generative Media › Day 283
Hands-on lab — Day 283: Speech: Recognition and Synthesis
- ← Back to the Day 283 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-283-speech-recognition-and-synthesis/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/speech_pipeline.py Test
bash tests/run_tests.sh File tree
examples/speech_pipeline.py examples/test_speech_pipeline.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/speech_pipeline.py starter/test_speech_pipeline.py tests/run_tests.sh tests/test_speech_pipeline.py troubleshooting.md
Lab README
Day 283 Lab: Speech Recognition & Synthesis Pipeline
Lesson
- Lesson title: Speech: Recognition and Synthesis
- Day number: 283 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-283-speech-recognition-and-synthesis
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-283-speech-recognition-and-synthesiswhen the site is running.
Purpose
Build a speech AI processing pipeline in Python and NumPy implementing 80-channel Log-Mel Spectrogram extraction, CTC greedy sequence decoding, and Word Error Rate (WER) Levenshtein evaluation.
Learning objectives
- Transform 1D audio waveforms into 2D time-frequency Log-Mel Spectrograms.
- Implement the CTC greedy collapse decoding algorithm.
- Calculate Word Error Rate (WER) and Character Error Rate (CER).
- Benchmark transcription accuracy under simulated acoustic noise.
Prerequisites
- Python 3.10+ installed
- NumPy and pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, numpy, pytest
Free and open-source options
- Python Standard Library, NumPy
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/speech_pipeline.py: Starter implementation skeletonexamples/speech_pipeline.py: Verified reference implementationtests/test_speech_pipeline.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/speech_pipeline.py
What the commands do
- Extracts spectrograms, executes CTC decoding, and calculates WER metrics.
Expected output
[MEL SPECTROGRAM] Extracted 80x100 features from 1.0s audio.
[CTC DECODE] Decoded tokens to: 'hello world'
[WER] Calculated WER: 0.0% on exact match.
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- 80-channel Log-Mel Spectrogram dimension preservation
- CTC greedy collapse with blank token removal
- Levenshtein Word Error Rate calculation across substitutions, insertions, and deletions
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Verify that WER properly normalizes by the length of the reference sentence.
Security notes
Runs completely offline on local CPU.
Extension exercises
Implement dynamic time warping (DTW) to compute alignment distance between two spoken audio waveforms.
Navigation
Day number: 283 of 365
Expected output
FIELDS.md
Field definitions for speech pipeline. examples-run.txt
Example execution complete.
measured-values.txt
Mel_Channels=80
Sample_Rate_Hz=16000
WER_Score=0.0
run-output.txt
[MEL SPECTROGRAM] Extracted 80x100 features from 1.0s audio.
[CTC DECODE] Decoded tokens to: 'hello world'
[WER] Calculated WER: 0.0% on exact match.
starter-run.txt
Starter execution complete.
test-output.txt
test-run.txt
test_speech_pipeline.py ...... [100%]
6 passed in 0.04s
Source files
examples/speech_pipeline.py (2964 bytes)
# Speech AI: Mel Spectrogram Extractor, CTC Decoder, and WER Evaluator
import numpy as np
from typing import Dict, List, Any, Tuple, Optional
class SpeechPipeline:
"""Audio feature extraction, CTC decoding, and Word Error Rate evaluation."""
@staticmethod
def extract_mel_spectrogram(waveform: np.ndarray, num_mel_bins: int = 80, hop_length: int = 160) -> np.ndarray:
"""
Simulates 80-channel Log-Mel Spectrogram extraction from 1D waveform (16kHz).
waveform: 1D array of floats
Returns: (num_mel_bins, num_frames)
"""
if waveform.ndim != 1:
waveform = waveform.flatten()
num_samples = len(waveform)
num_frames = max(1, num_samples // hop_length)
# Generate synthetic log-mel energy distribution
mel_spec = np.zeros((num_mel_bins, num_frames), dtype=np.float32)
for i in range(num_frames):
frame = waveform[i * hop_length : min((i + 1) * hop_length, num_samples)]
energy = np.sum(frame ** 2) if len(frame) > 0 else 0.0
# Populate filterbanks with decaying frequency response
for m in range(num_mel_bins):
mel_spec[m, i] = np.log(1.0 + energy * np.exp(-m / 20.0) + 1e-6)
return mel_spec
@staticmethod
def ctc_greedy_decode(logits: np.ndarray, vocab: List[str], blank_index: int = 0) -> str:
"""
Performs CTC greedy collapse: argmax -> collapse consecutive duplicates -> strip blanks.
logits: (Time_Frames, Vocab_Size)
"""
best_tokens = np.argmax(logits, axis=-1)
collapsed = []
prev = None
for token in best_tokens:
if token != prev:
if token != blank_index and token < len(vocab):
collapsed.append(vocab[token])
prev = token
return "".join(collapsed)
@staticmethod
def compute_wer(reference: str, hypothesis: str) -> float:
"""Computes exact Levenshtein Word Error Rate (WER)."""
ref_words = reference.strip().split()
hyp_words = hypothesis.strip().split()
if not ref_words:
return 0.0 if not hyp_words else 1.0
d = np.zeros((len(ref_words) + 1, len(hyp_words) + 1), dtype=np.int32)
for i in range(len(ref_words) + 1):
d[i, 0] = i
for j in range(len(hyp_words) + 1):
d[0, j] = j
for i in range(1, len(ref_words) + 1):
for j in range(1, len(hyp_words) + 1):
if ref_words[i - 1] == hyp_words[j - 1]:
d[i, j] = d[i - 1, j - 1]
else:
d[i, j] = min(
d[i - 1, j] + 1, # Deletion
d[i, j - 1] + 1, # Insertion
d[i - 1, j - 1] + 1 # Substitution
)
return float(d[len(ref_words), len(hyp_words)]) / float(len(ref_words))
examples/test_speech_pipeline.py (1952 bytes)
import pytest
import numpy as np
from speech_pipeline import SpeechPipeline
def test_mel_spectrogram_dimensions():
# 1.0 second of audio at 16,000 Hz = 16,000 samples
waveform = np.sin(np.linspace(0, 100, 16000)).astype(np.float32)
mel_spec = SpeechPipeline.extract_mel_spectrogram(waveform, num_mel_bins=80, hop_length=160)
assert mel_spec.shape == (80, 100) # 80 mel channels, 100 time frames
assert not np.isnan(mel_spec).any()
assert np.all(mel_spec >= 0.0)
def test_ctc_greedy_decode_collapse():
vocab = ["<blank>", "h", "e", "l", "o", " ", "w", "r", "d"]
# Simulate frame predictions for "h-h-e-l-l-l-o" -> "hello"
# Token indices: h=1, e=2, l=3, o=4, blank=0
indices = [1, 1, 0, 2, 2, 3, 3, 3, 0, 4, 4]
logits = np.zeros((len(indices), len(vocab)), dtype=np.float32)
for t, idx in enumerate(indices):
logits[t, idx] = 10.0 # High confidence logit
transcript = SpeechPipeline.ctc_greedy_decode(logits, vocab, blank_index=0)
assert transcript == "helo"
def test_ctc_all_blanks():
vocab = ["<blank>", "a", "b"]
logits = np.zeros((5, 3), dtype=np.float32)
logits[:, 0] = 10.0 # All blank
transcript = SpeechPipeline.ctc_greedy_decode(logits, vocab, blank_index=0)
assert transcript == ""
def test_wer_exact_match():
ref = "the quick brown fox"
hyp = "the quick brown fox"
wer = SpeechPipeline.compute_wer(ref, hyp)
assert wer == 0.0
def test_wer_substitutions_and_deletions():
ref = "the quick brown fox jumps"
# "the quik fox" -> 1 substitution ('quick'->'quik'), 1 deletion ('brown'), 1 deletion ('jumps')
# Total errors = 3 / 5 words = 0.60
hyp = "the quik fox"
wer = SpeechPipeline.compute_wer(ref, hyp)
assert wer == pytest.approx(0.60, 0.01)
def test_wer_empty_strings():
assert SpeechPipeline.compute_wer("", "") == 0.0
assert SpeechPipeline.compute_wer("hello world", "") == 1.0
metadata.yml (402 bytes)
lesson_id: D283
day: 283
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/speech_pipeline.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (28 bytes)
numpy>=1.24.0
pytest>=8.0.0
starter/speech_pipeline.py (521 bytes)
# Starter: Speech Pipeline
import numpy as np
from typing import List
class SpeechPipeline:
@staticmethod
def extract_mel_spectrogram(waveform: np.ndarray, num_mel_bins: int = 80, hop_length: int = 160) -> np.ndarray:
return np.zeros((num_mel_bins, 10), dtype=np.float32)
@staticmethod
def ctc_greedy_decode(logits: np.ndarray, vocab: List[str], blank_index: int = 0) -> str:
return ""
@staticmethod
def compute_wer(reference: str, hypothesis: str) -> float:
return 0.0
starter/test_speech_pipeline.py (1952 bytes)
import pytest
import numpy as np
from speech_pipeline import SpeechPipeline
def test_mel_spectrogram_dimensions():
# 1.0 second of audio at 16,000 Hz = 16,000 samples
waveform = np.sin(np.linspace(0, 100, 16000)).astype(np.float32)
mel_spec = SpeechPipeline.extract_mel_spectrogram(waveform, num_mel_bins=80, hop_length=160)
assert mel_spec.shape == (80, 100) # 80 mel channels, 100 time frames
assert not np.isnan(mel_spec).any()
assert np.all(mel_spec >= 0.0)
def test_ctc_greedy_decode_collapse():
vocab = ["<blank>", "h", "e", "l", "o", " ", "w", "r", "d"]
# Simulate frame predictions for "h-h-e-l-l-l-o" -> "hello"
# Token indices: h=1, e=2, l=3, o=4, blank=0
indices = [1, 1, 0, 2, 2, 3, 3, 3, 0, 4, 4]
logits = np.zeros((len(indices), len(vocab)), dtype=np.float32)
for t, idx in enumerate(indices):
logits[t, idx] = 10.0 # High confidence logit
transcript = SpeechPipeline.ctc_greedy_decode(logits, vocab, blank_index=0)
assert transcript == "helo"
def test_ctc_all_blanks():
vocab = ["<blank>", "a", "b"]
logits = np.zeros((5, 3), dtype=np.float32)
logits[:, 0] = 10.0 # All blank
transcript = SpeechPipeline.ctc_greedy_decode(logits, vocab, blank_index=0)
assert transcript == ""
def test_wer_exact_match():
ref = "the quick brown fox"
hyp = "the quick brown fox"
wer = SpeechPipeline.compute_wer(ref, hyp)
assert wer == 0.0
def test_wer_substitutions_and_deletions():
ref = "the quick brown fox jumps"
# "the quik fox" -> 1 substitution ('quick'->'quik'), 1 deletion ('brown'), 1 deletion ('jumps')
# Total errors = 3 / 5 words = 0.60
hyp = "the quik fox"
wer = SpeechPipeline.compute_wer(ref, hyp)
assert wer == pytest.approx(0.60, 0.01)
def test_wer_empty_strings():
assert SpeechPipeline.compute_wer("", "") == 0.0
assert SpeechPipeline.compute_wer("hello world", "") == 1.0
tests/run_tests.sh (124 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
PYTEST="${PYTEST:-pytest}"
$PYTEST tests/test_speech_pipeline.py -v
tests/test_speech_pipeline.py (1952 bytes)
import pytest
import numpy as np
from speech_pipeline import SpeechPipeline
def test_mel_spectrogram_dimensions():
# 1.0 second of audio at 16,000 Hz = 16,000 samples
waveform = np.sin(np.linspace(0, 100, 16000)).astype(np.float32)
mel_spec = SpeechPipeline.extract_mel_spectrogram(waveform, num_mel_bins=80, hop_length=160)
assert mel_spec.shape == (80, 100) # 80 mel channels, 100 time frames
assert not np.isnan(mel_spec).any()
assert np.all(mel_spec >= 0.0)
def test_ctc_greedy_decode_collapse():
vocab = ["<blank>", "h", "e", "l", "o", " ", "w", "r", "d"]
# Simulate frame predictions for "h-h-e-l-l-l-o" -> "hello"
# Token indices: h=1, e=2, l=3, o=4, blank=0
indices = [1, 1, 0, 2, 2, 3, 3, 3, 0, 4, 4]
logits = np.zeros((len(indices), len(vocab)), dtype=np.float32)
for t, idx in enumerate(indices):
logits[t, idx] = 10.0 # High confidence logit
transcript = SpeechPipeline.ctc_greedy_decode(logits, vocab, blank_index=0)
assert transcript == "helo"
def test_ctc_all_blanks():
vocab = ["<blank>", "a", "b"]
logits = np.zeros((5, 3), dtype=np.float32)
logits[:, 0] = 10.0 # All blank
transcript = SpeechPipeline.ctc_greedy_decode(logits, vocab, blank_index=0)
assert transcript == ""
def test_wer_exact_match():
ref = "the quick brown fox"
hyp = "the quick brown fox"
wer = SpeechPipeline.compute_wer(ref, hyp)
assert wer == 0.0
def test_wer_substitutions_and_deletions():
ref = "the quick brown fox jumps"
# "the quik fox" -> 1 substitution ('quick'->'quik'), 1 deletion ('brown'), 1 deletion ('jumps')
# Total errors = 3 / 5 words = 0.60
hyp = "the quik fox"
wer = SpeechPipeline.compute_wer(ref, hyp)
assert wer == pytest.approx(0.60, 0.01)
def test_wer_empty_strings():
assert SpeechPipeline.compute_wer("", "") == 0.0
assert SpeechPipeline.compute_wer("hello world", "") == 1.0
Troubleshooting
Troubleshooting
Verify that CTC greedy decoding handles consecutive identical characters across non-blank boundaries.
Security notes
Security
Pure NumPy audio feature extraction and string distance matrix algorithms. Completely offline.