Deep Learning › Sequences and Text › Day 218

Hands-on lab — Day 218: Text Preprocessing and Tokenization

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/text_preprocessing_and_tokenization_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/test_text_preprocessing_and_tokenization_lib.py
examples/text_preprocessing_and_tokenization_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/test_text_preprocessing_and_tokenization_lib.py
starter/text_preprocessing_and_tokenization_lib.py
tests/run_tests.sh
tests/test_text_preprocessing_and_tokenization_lib.py
troubleshooting.md

Lab README

Lab: Day 218 -- Text Preprocessing and Tokenization

Lesson

Day number: 218 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Text Preprocessing and Tokenization Foundations.

Purpose

Build and test a complete text preprocessing and tokenization toolkit in Python. Implement regular expression pre-tokenization, train a Byte-Pair Encoding (BPE) merge registry from scratch on text corpora, manage special tokens and vocabulary indexing, and generate padded batch tensors with attention masks.

Learning objectives

  • Implement regex-based word and punctuation splitting rules.
  • Train Byte-Pair Encoding (BPE) subword merge rules from frequency statistics.
  • Construct vocabulary registries with special tokens (, , , ).
  • Pad variable-length integer token sequences and construct corresponding binary attention masks.

Prerequisites

  • Day 217 (Your Own Image Classifier).
  • Python 3.11+ with standard library modules.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 512 MB RAM.
  • 50 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

All tokenization routines use Python standard library modules (re, collections) and open-source pytest.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/text_preprocessing_and_tokenization_lib.py: Student scaffold file.
  • examples/text_preprocessing_and_tokenization_lib.py: Complete reference implementation.
  • tests/test_text_preprocessing_and_tokenization_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/text_preprocessing_and_tokenization_lib.py

What the commands do

  • Executes regex pre-tokenization across sample sentences.
  • Trains a BPE merge model on text corpus and encodes unseen words into subwords.
  • Formats token sequences into padded rectangular batches with attention masks.

Expected output

Tokenization Demo: Vocab Size = 15, Merges = 5, Padded Len = 3

Validation steps

  1. Verify regex_pre_tokenize separates words, contractions, and punctuation correctly.
  2. Confirm BPETokenizer.train successfully discovers and records subword bigram merges.
  3. Check that pad_and_mask_batch generates identical max-length vectors with matching binary masks.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Infinite Loop in Merge: Ensure bigram counts terminate when no remaining pairs exist in vocabulary.

Security notes

All tokenization and vocabulary generation algorithms execute locally without network access.

Extension exercises

  1. Implement Byte-level BPE supporting raw UTF-8 binary encoding.
  2. Add BPE-dropout stochastic subword sampling.

Expected output

FIELDS.md

# Expected Output Fields: Day 218

- `Vocab Size`: Total number of unique tokens in vocabulary.
- `Merges`: Total number of learned BPE merge operations.
- `Padded Len`: Max sequence length of rectangular batch tensor.

examples-run.txt

Tokenization Demo: Vocab Size = 15, Merges = 5, Padded Len = 3

measured-values.txt

Vocab Size: 15
Merges: 5
Padded Len: 3

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_text_preprocessing_and_tokenization_lib.py::test_regex_pre_tokenize PASSED [ 33%]
tests/test_text_preprocessing_and_tokenization_lib.py::test_bpe_training_and_encoding PASSED [ 66%]
tests/test_text_preprocessing_and_tokenization_lib.py::test_pad_and_mask_batch PASSED [100%]

============================== 3 passed in 0.28s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/test_text_preprocessing_and_tokenization_lib.py (962 bytes)
import pytest
from examples.text_preprocessing_and_tokenization_lib import regex_pre_tokenize, BPETokenizer, pad_and_mask_batch

def test_regex_pre_tokenize():
    text = "Hello, world! It's 2026."
    tokens = regex_pre_tokenize(text)
    assert "Hello" in tokens
    assert "world" in tokens
    assert "," in tokens
    assert "2026" in tokens

def test_bpe_training_and_encoding():
    corpus = ["low lower lowest", "new newer newest", "wide wider widest"]
    tok = BPETokenizer(num_merges=4)
    tok.train(corpus)
    assert len(tok.merges) > 0
    assert len(tok.vocab) >= 4 # Includes special tokens

    ids = tok.encode("lowest")
    assert isinstance(ids, list)
    assert len(ids) > 0

def test_pad_and_mask_batch():
    batch = [[10, 20, 30], [40, 50]]
    padded, masks = pad_and_mask_batch(batch, pad_idx=0)
    assert padded[0] == [10, 20, 30]
    assert padded[1] == [40, 50, 0]
    assert masks[0] == [1, 1, 1]
    assert masks[1] == [1, 1, 0]
examples/text_preprocessing_and_tokenization_lib.py (3981 bytes)
import re
from collections import defaultdict
from typing import List, Dict, Tuple, Any

def regex_pre_tokenize(text: str) -> List[str]:
    pattern = r"'\w+|\w+|[^\w\s]"
    return re.findall(pattern, text)

class BPETokenizer:
    def __init__(self, num_merges: int = 10):
        self.num_merges = num_merges
        self.merges: List[Tuple[str, str]] = []
        self.vocab: Dict[str, int] = {
            "<pad>": 0,
            "<unk>": 1,
            "<bos>": 2,
            "<eos>": 3
        }
        self.inverse_vocab: Dict[int, str] = {v: k for k, v in self.vocab.items()}

    def _get_stats(self, vocab: Dict[Tuple[str, ...], int]) -> Dict[Tuple[str, str], int]:
        pairs = defaultdict(int)
        for word_tuple, freq in vocab.items():
            for i in range(len(word_tuple) - 1):
                pair = (word_tuple[i], word_tuple[i+1])
                pairs[pair] += freq
        return pairs

    def _merge_vocab(self, pair: Tuple[str, str], vocab: Dict[Tuple[str, ...], int]) -> Dict[Tuple[str, ...], int]:
        new_vocab = {}
        bigram = " ".join(pair)
        replacement = "".join(pair)
        for word_tuple, freq in vocab.items():
            w_str = " ".join(word_tuple)
            w_new = w_str.replace(bigram, replacement)
            new_vocab[tuple(w_new.split())] = freq
        return new_vocab

    def train(self, corpus: List[str]):
        word_freqs = defaultdict(int)
        for text in corpus:
            for word in regex_pre_tokenize(text):
                chars = tuple(list(word) + ["</w>"])
                word_freqs[chars] += 1

        current_vocab = word_freqs
        for _ in range(self.num_merges):
            pairs = self._get_stats(current_vocab)
            if not pairs:
                break
            best_pair = max(pairs, key=pairs.get)
            self.merges.append(best_pair)
            current_vocab = self._merge_vocab(best_pair, current_vocab)

        # Build vocabulary
        for word_tuple in current_vocab.keys():
            for subword in word_tuple:
                if subword not in self.vocab:
                    idx = len(self.vocab)
                    self.vocab[subword] = idx
                    self.inverse_vocab[idx] = subword

    def encode_word(self, word: str) -> List[str]:
        tokens = list(word) + ["</w>"]
        for pair in self.merges:
            i = 0
            new_tokens = []
            while i < len(tokens):
                if i < len(tokens) - 1 and (tokens[i], tokens[i+1]) == pair:
                    new_tokens.append(pair[0] + pair[1])
                    i += 2
                else:
                    new_tokens.append(tokens[i])
                    i += 1
            tokens = new_tokens
        return tokens

    def encode(self, text: str) -> List[int]:
        words = regex_pre_tokenize(text)
        token_ids = []
        for word in words:
            subwords = self.encode_word(word)
            for sw in subwords:
                token_ids.append(self.vocab.get(sw, self.vocab["<unk>"]))
        return token_ids

def pad_and_mask_batch(batch_ids: List[List[int]], pad_idx: int = 0) -> Tuple[List[List[int]], List[List[int]]]:
    max_len = max(len(seq) for seq in batch_ids)
    padded = []
    masks = []
    for seq in batch_ids:
        pad_len = max_len - len(seq)
        padded.append(seq + [pad_idx] * pad_len)
        masks.append([1] * len(seq) + [0] * pad_len)
    return padded, masks

def run_tokenization_demo():
    corpus = [
        "low lower lowest",
        "new newer newest",
        "wide wider widest"
    ]
    tok = BPETokenizer(num_merges=5)
    tok.train(corpus)
    encoded = tok.encode("lowest newest")
    padded, masks = pad_and_mask_batch([[1, 2, 3], [4, 5]], pad_idx=0)
    print(f"Tokenization Demo: Vocab Size = {len(tok.vocab)}, Merges = {len(tok.merges)}, Padded Len = {len(padded[0])}")
    return tok, encoded, padded, masks

if __name__ == "__main__":
    run_tokenization_demo()
metadata.yml (448 bytes)
lesson_id: D218
day: 218
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/text_preprocessing_and_tokenization_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 40
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (28 bytes)
numpy>=1.26.0
pytest>=8.0.0
starter/test_text_preprocessing_and_tokenization_lib.py (962 bytes)
import pytest
from examples.text_preprocessing_and_tokenization_lib import regex_pre_tokenize, BPETokenizer, pad_and_mask_batch

def test_regex_pre_tokenize():
    text = "Hello, world! It's 2026."
    tokens = regex_pre_tokenize(text)
    assert "Hello" in tokens
    assert "world" in tokens
    assert "," in tokens
    assert "2026" in tokens

def test_bpe_training_and_encoding():
    corpus = ["low lower lowest", "new newer newest", "wide wider widest"]
    tok = BPETokenizer(num_merges=4)
    tok.train(corpus)
    assert len(tok.merges) > 0
    assert len(tok.vocab) >= 4 # Includes special tokens

    ids = tok.encode("lowest")
    assert isinstance(ids, list)
    assert len(ids) > 0

def test_pad_and_mask_batch():
    batch = [[10, 20, 30], [40, 50]]
    padded, masks = pad_and_mask_batch(batch, pad_idx=0)
    assert padded[0] == [10, 20, 30]
    assert padded[1] == [40, 50, 0]
    assert masks[0] == [1, 1, 1]
    assert masks[1] == [1, 1, 0]
starter/text_preprocessing_and_tokenization_lib.py (676 bytes)
import re
from collections import defaultdict
from typing import List, Dict, Tuple, Any

def regex_pre_tokenize(text: str) -> List[str]:
    # TODO: Split text into words, contractions, numbers, and punctuation using regex
    pass

class BPETokenizer:
    def __init__(self, num_merges: int = 10):
        self.num_merges = num_merges
        self.merges: List[Tuple[str, str]] = []
        self.vocab: Dict[str, int] = {}
        self.inverse_vocab: Dict[int, str] = {}

    def train(self, corpus: List[str]):
        # TODO: Train BPE merge rules on corpus
        pass

    def encode(self, text: str) -> List[int]:
        # TODO: Convert text to token IDs
        pass
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 218 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_text_preprocessing_and_tokenization_lib.py (962 bytes)
import pytest
from examples.text_preprocessing_and_tokenization_lib import regex_pre_tokenize, BPETokenizer, pad_and_mask_batch

def test_regex_pre_tokenize():
    text = "Hello, world! It's 2026."
    tokens = regex_pre_tokenize(text)
    assert "Hello" in tokens
    assert "world" in tokens
    assert "," in tokens
    assert "2026" in tokens

def test_bpe_training_and_encoding():
    corpus = ["low lower lowest", "new newer newest", "wide wider widest"]
    tok = BPETokenizer(num_merges=4)
    tok.train(corpus)
    assert len(tok.merges) > 0
    assert len(tok.vocab) >= 4 # Includes special tokens

    ids = tok.encode("lowest")
    assert isinstance(ids, list)
    assert len(ids) > 0

def test_pad_and_mask_batch():
    batch = [[10, 20, 30], [40, 50]]
    padded, masks = pad_and_mask_batch(batch, pad_idx=0)
    assert padded[0] == [10, 20, 30]
    assert padded[1] == [40, 50, 0]
    assert masks[0] == [1, 1, 1]
    assert masks[1] == [1, 1, 0]

Troubleshooting

Troubleshooting: Day 218 - Text Preprocessing and Tokenization

Common Issues

  1. Unseen character produces key error:
    • Cause: Missing <unk> fallback in vocabulary lookup.
    • Fix: Use vocab.get(token, vocab["<unk>"]).

Security notes

Security & Privacy: Day 218 - Text Preprocessing and Tokenization

Security Guidance

  • All tokenizers operate locally on memory buffers.