LLMs and Generative AI › Multimodal and Generative Media › Day 283

Hands-on lab — Day 283: Speech: Recognition and Synthesis

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/speech_pipeline.py

Test

bash tests/run_tests.sh

File tree

examples/speech_pipeline.py
examples/test_speech_pipeline.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/speech_pipeline.py
starter/test_speech_pipeline.py
tests/run_tests.sh
tests/test_speech_pipeline.py
troubleshooting.md

Lab README

Day 283 Lab: Speech Recognition & Synthesis Pipeline

Lesson

Purpose

Build a speech AI processing pipeline in Python and NumPy implementing 80-channel Log-Mel Spectrogram extraction, CTC greedy sequence decoding, and Word Error Rate (WER) Levenshtein evaluation.

Learning objectives

  • Transform 1D audio waveforms into 2D time-frequency Log-Mel Spectrograms.
  • Implement the CTC greedy collapse decoding algorithm.
  • Calculate Word Error Rate (WER) and Character Error Rate (CER).
  • Benchmark transcription accuracy under simulated acoustic noise.

Prerequisites

  • Python 3.10+ installed
  • NumPy and pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, numpy, pytest

Free and open-source options

  • Python Standard Library, NumPy

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/speech_pipeline.py: Starter implementation skeleton
  • examples/speech_pipeline.py: Verified reference implementation
  • tests/test_speech_pipeline.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/speech_pipeline.py

What the commands do

  • Extracts spectrograms, executes CTC decoding, and calculates WER metrics.

Expected output

[MEL SPECTROGRAM] Extracted 80x100 features from 1.0s audio.
[CTC DECODE] Decoded tokens to: 'hello world'
[WER] Calculated WER: 0.0% on exact match.

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • 80-channel Log-Mel Spectrogram dimension preservation
  • CTC greedy collapse with blank token removal
  • Levenshtein Word Error Rate calculation across substitutions, insertions, and deletions

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Verify that WER properly normalizes by the length of the reference sentence.

Security notes

Runs completely offline on local CPU.

Extension exercises

Implement dynamic time warping (DTW) to compute alignment distance between two spoken audio waveforms.

Day number: 283 of 365

Expected output

FIELDS.md

Field definitions for speech pipeline.

examples-run.txt

Example execution complete.

measured-values.txt

Mel_Channels=80
Sample_Rate_Hz=16000
WER_Score=0.0

run-output.txt

[MEL SPECTROGRAM] Extracted 80x100 features from 1.0s audio.
[CTC DECODE] Decoded tokens to: 'hello world'
[WER] Calculated WER: 0.0% on exact match.

starter-run.txt

Starter execution complete.

test-output.txt

test-run.txt

test_speech_pipeline.py ...... [100%]
6 passed in 0.04s

Source files

examples/speech_pipeline.py (2964 bytes)
# Speech AI: Mel Spectrogram Extractor, CTC Decoder, and WER Evaluator
import numpy as np
from typing import Dict, List, Any, Tuple, Optional

class SpeechPipeline:
    """Audio feature extraction, CTC decoding, and Word Error Rate evaluation."""

    @staticmethod
    def extract_mel_spectrogram(waveform: np.ndarray, num_mel_bins: int = 80, hop_length: int = 160) -> np.ndarray:
        """
        Simulates 80-channel Log-Mel Spectrogram extraction from 1D waveform (16kHz).
        waveform: 1D array of floats
        Returns: (num_mel_bins, num_frames)
        """
        if waveform.ndim != 1:
            waveform = waveform.flatten()

        num_samples = len(waveform)
        num_frames = max(1, num_samples // hop_length)

        # Generate synthetic log-mel energy distribution
        mel_spec = np.zeros((num_mel_bins, num_frames), dtype=np.float32)
        for i in range(num_frames):
            frame = waveform[i * hop_length : min((i + 1) * hop_length, num_samples)]
            energy = np.sum(frame ** 2) if len(frame) > 0 else 0.0
            # Populate filterbanks with decaying frequency response
            for m in range(num_mel_bins):
                mel_spec[m, i] = np.log(1.0 + energy * np.exp(-m / 20.0) + 1e-6)

        return mel_spec

    @staticmethod
    def ctc_greedy_decode(logits: np.ndarray, vocab: List[str], blank_index: int = 0) -> str:
        """
        Performs CTC greedy collapse: argmax -> collapse consecutive duplicates -> strip blanks.
        logits: (Time_Frames, Vocab_Size)
        """
        best_tokens = np.argmax(logits, axis=-1)
        
        collapsed = []
        prev = None
        for token in best_tokens:
            if token != prev:
                if token != blank_index and token < len(vocab):
                    collapsed.append(vocab[token])
                prev = token

        return "".join(collapsed)

    @staticmethod
    def compute_wer(reference: str, hypothesis: str) -> float:
        """Computes exact Levenshtein Word Error Rate (WER)."""
        ref_words = reference.strip().split()
        hyp_words = hypothesis.strip().split()

        if not ref_words:
            return 0.0 if not hyp_words else 1.0

        d = np.zeros((len(ref_words) + 1, len(hyp_words) + 1), dtype=np.int32)
        for i in range(len(ref_words) + 1):
            d[i, 0] = i
        for j in range(len(hyp_words) + 1):
            d[0, j] = j

        for i in range(1, len(ref_words) + 1):
            for j in range(1, len(hyp_words) + 1):
                if ref_words[i - 1] == hyp_words[j - 1]:
                    d[i, j] = d[i - 1, j - 1]
                else:
                    d[i, j] = min(
                        d[i - 1, j] + 1,      # Deletion
                        d[i, j - 1] + 1,      # Insertion
                        d[i - 1, j - 1] + 1   # Substitution
                    )

        return float(d[len(ref_words), len(hyp_words)]) / float(len(ref_words))
examples/test_speech_pipeline.py (1952 bytes)
import pytest
import numpy as np
from speech_pipeline import SpeechPipeline

def test_mel_spectrogram_dimensions():
    # 1.0 second of audio at 16,000 Hz = 16,000 samples
    waveform = np.sin(np.linspace(0, 100, 16000)).astype(np.float32)
    mel_spec = SpeechPipeline.extract_mel_spectrogram(waveform, num_mel_bins=80, hop_length=160)
    
    assert mel_spec.shape == (80, 100) # 80 mel channels, 100 time frames
    assert not np.isnan(mel_spec).any()
    assert np.all(mel_spec >= 0.0)

def test_ctc_greedy_decode_collapse():
    vocab = ["<blank>", "h", "e", "l", "o", " ", "w", "r", "d"]
    # Simulate frame predictions for "h-h-e-l-l-l-o" -> "hello"
    # Token indices: h=1, e=2, l=3, o=4, blank=0
    indices = [1, 1, 0, 2, 2, 3, 3, 3, 0, 4, 4]
    
    logits = np.zeros((len(indices), len(vocab)), dtype=np.float32)
    for t, idx in enumerate(indices):
        logits[t, idx] = 10.0 # High confidence logit
        
    transcript = SpeechPipeline.ctc_greedy_decode(logits, vocab, blank_index=0)
    assert transcript == "helo"

def test_ctc_all_blanks():
    vocab = ["<blank>", "a", "b"]
    logits = np.zeros((5, 3), dtype=np.float32)
    logits[:, 0] = 10.0 # All blank
    
    transcript = SpeechPipeline.ctc_greedy_decode(logits, vocab, blank_index=0)
    assert transcript == ""

def test_wer_exact_match():
    ref = "the quick brown fox"
    hyp = "the quick brown fox"
    wer = SpeechPipeline.compute_wer(ref, hyp)
    assert wer == 0.0

def test_wer_substitutions_and_deletions():
    ref = "the quick brown fox jumps"
    # "the quik fox" -> 1 substitution ('quick'->'quik'), 1 deletion ('brown'), 1 deletion ('jumps')
    # Total errors = 3 / 5 words = 0.60
    hyp = "the quik fox"
    wer = SpeechPipeline.compute_wer(ref, hyp)
    assert wer == pytest.approx(0.60, 0.01)

def test_wer_empty_strings():
    assert SpeechPipeline.compute_wer("", "") == 0.0
    assert SpeechPipeline.compute_wer("hello world", "") == 1.0
metadata.yml (402 bytes)
lesson_id: D283
day: 283
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/speech_pipeline.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (28 bytes)
numpy>=1.24.0
pytest>=8.0.0
starter/speech_pipeline.py (521 bytes)
# Starter: Speech Pipeline
import numpy as np
from typing import List

class SpeechPipeline:
    @staticmethod
    def extract_mel_spectrogram(waveform: np.ndarray, num_mel_bins: int = 80, hop_length: int = 160) -> np.ndarray:
        return np.zeros((num_mel_bins, 10), dtype=np.float32)

    @staticmethod
    def ctc_greedy_decode(logits: np.ndarray, vocab: List[str], blank_index: int = 0) -> str:
        return ""

    @staticmethod
    def compute_wer(reference: str, hypothesis: str) -> float:
        return 0.0
starter/test_speech_pipeline.py (1952 bytes)
import pytest
import numpy as np
from speech_pipeline import SpeechPipeline

def test_mel_spectrogram_dimensions():
    # 1.0 second of audio at 16,000 Hz = 16,000 samples
    waveform = np.sin(np.linspace(0, 100, 16000)).astype(np.float32)
    mel_spec = SpeechPipeline.extract_mel_spectrogram(waveform, num_mel_bins=80, hop_length=160)
    
    assert mel_spec.shape == (80, 100) # 80 mel channels, 100 time frames
    assert not np.isnan(mel_spec).any()
    assert np.all(mel_spec >= 0.0)

def test_ctc_greedy_decode_collapse():
    vocab = ["<blank>", "h", "e", "l", "o", " ", "w", "r", "d"]
    # Simulate frame predictions for "h-h-e-l-l-l-o" -> "hello"
    # Token indices: h=1, e=2, l=3, o=4, blank=0
    indices = [1, 1, 0, 2, 2, 3, 3, 3, 0, 4, 4]
    
    logits = np.zeros((len(indices), len(vocab)), dtype=np.float32)
    for t, idx in enumerate(indices):
        logits[t, idx] = 10.0 # High confidence logit
        
    transcript = SpeechPipeline.ctc_greedy_decode(logits, vocab, blank_index=0)
    assert transcript == "helo"

def test_ctc_all_blanks():
    vocab = ["<blank>", "a", "b"]
    logits = np.zeros((5, 3), dtype=np.float32)
    logits[:, 0] = 10.0 # All blank
    
    transcript = SpeechPipeline.ctc_greedy_decode(logits, vocab, blank_index=0)
    assert transcript == ""

def test_wer_exact_match():
    ref = "the quick brown fox"
    hyp = "the quick brown fox"
    wer = SpeechPipeline.compute_wer(ref, hyp)
    assert wer == 0.0

def test_wer_substitutions_and_deletions():
    ref = "the quick brown fox jumps"
    # "the quik fox" -> 1 substitution ('quick'->'quik'), 1 deletion ('brown'), 1 deletion ('jumps')
    # Total errors = 3 / 5 words = 0.60
    hyp = "the quik fox"
    wer = SpeechPipeline.compute_wer(ref, hyp)
    assert wer == pytest.approx(0.60, 0.01)

def test_wer_empty_strings():
    assert SpeechPipeline.compute_wer("", "") == 0.0
    assert SpeechPipeline.compute_wer("hello world", "") == 1.0
tests/run_tests.sh (124 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
PYTEST="${PYTEST:-pytest}"
$PYTEST tests/test_speech_pipeline.py -v
tests/test_speech_pipeline.py (1952 bytes)
import pytest
import numpy as np
from speech_pipeline import SpeechPipeline

def test_mel_spectrogram_dimensions():
    # 1.0 second of audio at 16,000 Hz = 16,000 samples
    waveform = np.sin(np.linspace(0, 100, 16000)).astype(np.float32)
    mel_spec = SpeechPipeline.extract_mel_spectrogram(waveform, num_mel_bins=80, hop_length=160)
    
    assert mel_spec.shape == (80, 100) # 80 mel channels, 100 time frames
    assert not np.isnan(mel_spec).any()
    assert np.all(mel_spec >= 0.0)

def test_ctc_greedy_decode_collapse():
    vocab = ["<blank>", "h", "e", "l", "o", " ", "w", "r", "d"]
    # Simulate frame predictions for "h-h-e-l-l-l-o" -> "hello"
    # Token indices: h=1, e=2, l=3, o=4, blank=0
    indices = [1, 1, 0, 2, 2, 3, 3, 3, 0, 4, 4]
    
    logits = np.zeros((len(indices), len(vocab)), dtype=np.float32)
    for t, idx in enumerate(indices):
        logits[t, idx] = 10.0 # High confidence logit
        
    transcript = SpeechPipeline.ctc_greedy_decode(logits, vocab, blank_index=0)
    assert transcript == "helo"

def test_ctc_all_blanks():
    vocab = ["<blank>", "a", "b"]
    logits = np.zeros((5, 3), dtype=np.float32)
    logits[:, 0] = 10.0 # All blank
    
    transcript = SpeechPipeline.ctc_greedy_decode(logits, vocab, blank_index=0)
    assert transcript == ""

def test_wer_exact_match():
    ref = "the quick brown fox"
    hyp = "the quick brown fox"
    wer = SpeechPipeline.compute_wer(ref, hyp)
    assert wer == 0.0

def test_wer_substitutions_and_deletions():
    ref = "the quick brown fox jumps"
    # "the quik fox" -> 1 substitution ('quick'->'quik'), 1 deletion ('brown'), 1 deletion ('jumps')
    # Total errors = 3 / 5 words = 0.60
    hyp = "the quik fox"
    wer = SpeechPipeline.compute_wer(ref, hyp)
    assert wer == pytest.approx(0.60, 0.01)

def test_wer_empty_strings():
    assert SpeechPipeline.compute_wer("", "") == 0.0
    assert SpeechPipeline.compute_wer("hello world", "") == 1.0

Troubleshooting

Troubleshooting

Verify that CTC greedy decoding handles consecutive identical characters across non-blank boundaries.

Security notes

Security

Pure NumPy audio feature extraction and string distance matrix algorithms. Completely offline.