Deep Learning › Sequences and Text › Day 218
Hands-on lab — Day 218: Text Preprocessing and Tokenization
- ← Back to the Day 218 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deep-learning/day-218-text-preprocessing-and-tokenization/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/text_preprocessing_and_tokenization_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/test_text_preprocessing_and_tokenization_lib.py examples/text_preprocessing_and_tokenization_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/test_text_preprocessing_and_tokenization_lib.py starter/text_preprocessing_and_tokenization_lib.py tests/run_tests.sh tests/test_text_preprocessing_and_tokenization_lib.py troubleshooting.md
Lab README
Lab: Day 218 -- Text Preprocessing and Tokenization
Lesson
Day number: 218 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Text Preprocessing and Tokenization Foundations.
Purpose
Build and test a complete text preprocessing and tokenization toolkit in Python. Implement regular expression pre-tokenization, train a Byte-Pair Encoding (BPE) merge registry from scratch on text corpora, manage special tokens and vocabulary indexing, and generate padded batch tensors with attention masks.
Learning objectives
- Implement regex-based word and punctuation splitting rules.
- Train Byte-Pair Encoding (BPE) subword merge rules from frequency statistics.
- Construct vocabulary registries with special tokens (, , , ).
- Pad variable-length integer token sequences and construct corresponding binary attention masks.
Prerequisites
- Day 217 (Your Own Image Classifier).
- Python 3.11+ with standard library modules.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 512 MB RAM.
- 50 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
All tokenization routines use Python standard library modules (re, collections) and open-source pytest.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/text_preprocessing_and_tokenization_lib.py: Student scaffold file.examples/text_preprocessing_and_tokenization_lib.py: Complete reference implementation.tests/test_text_preprocessing_and_tokenization_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/text_preprocessing_and_tokenization_lib.py
What the commands do
- Executes regex pre-tokenization across sample sentences.
- Trains a BPE merge model on text corpus and encodes unseen words into subwords.
- Formats token sequences into padded rectangular batches with attention masks.
Expected output
Tokenization Demo: Vocab Size = 15, Merges = 5, Padded Len = 3
Validation steps
- Verify
regex_pre_tokenizeseparates words, contractions, and punctuation correctly. - Confirm
BPETokenizer.trainsuccessfully discovers and records subword bigram merges. - Check that
pad_and_mask_batchgenerates identical max-length vectors with matching binary masks.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Infinite Loop in Merge: Ensure bigram counts terminate when no remaining pairs exist in vocabulary.
Security notes
All tokenization and vocabulary generation algorithms execute locally without network access.
Extension exercises
- Implement Byte-level BPE supporting raw UTF-8 binary encoding.
- Add BPE-dropout stochastic subword sampling.
Navigation
- Lesson title: Text Preprocessing and Tokenization
- Day number: 218 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-218-text-preprocessing-and-tokenization
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-218-text-preprocessing-and-tokenizationwhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 218
- `Vocab Size`: Total number of unique tokens in vocabulary.
- `Merges`: Total number of learned BPE merge operations.
- `Padded Len`: Max sequence length of rectangular batch tensor.
examples-run.txt
Tokenization Demo: Vocab Size = 15, Merges = 5, Padded Len = 3
measured-values.txt
Vocab Size: 15
Merges: 5
Padded Len: 3
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 3 items
tests/test_text_preprocessing_and_tokenization_lib.py::test_regex_pre_tokenize PASSED [ 33%]
tests/test_text_preprocessing_and_tokenization_lib.py::test_bpe_training_and_encoding PASSED [ 66%]
tests/test_text_preprocessing_and_tokenization_lib.py::test_pad_and_mask_batch PASSED [100%]
============================== 3 passed in 0.28s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/test_text_preprocessing_and_tokenization_lib.py (962 bytes)
import pytest
from examples.text_preprocessing_and_tokenization_lib import regex_pre_tokenize, BPETokenizer, pad_and_mask_batch
def test_regex_pre_tokenize():
text = "Hello, world! It's 2026."
tokens = regex_pre_tokenize(text)
assert "Hello" in tokens
assert "world" in tokens
assert "," in tokens
assert "2026" in tokens
def test_bpe_training_and_encoding():
corpus = ["low lower lowest", "new newer newest", "wide wider widest"]
tok = BPETokenizer(num_merges=4)
tok.train(corpus)
assert len(tok.merges) > 0
assert len(tok.vocab) >= 4 # Includes special tokens
ids = tok.encode("lowest")
assert isinstance(ids, list)
assert len(ids) > 0
def test_pad_and_mask_batch():
batch = [[10, 20, 30], [40, 50]]
padded, masks = pad_and_mask_batch(batch, pad_idx=0)
assert padded[0] == [10, 20, 30]
assert padded[1] == [40, 50, 0]
assert masks[0] == [1, 1, 1]
assert masks[1] == [1, 1, 0]
examples/text_preprocessing_and_tokenization_lib.py (3981 bytes)
import re
from collections import defaultdict
from typing import List, Dict, Tuple, Any
def regex_pre_tokenize(text: str) -> List[str]:
pattern = r"'\w+|\w+|[^\w\s]"
return re.findall(pattern, text)
class BPETokenizer:
def __init__(self, num_merges: int = 10):
self.num_merges = num_merges
self.merges: List[Tuple[str, str]] = []
self.vocab: Dict[str, int] = {
"<pad>": 0,
"<unk>": 1,
"<bos>": 2,
"<eos>": 3
}
self.inverse_vocab: Dict[int, str] = {v: k for k, v in self.vocab.items()}
def _get_stats(self, vocab: Dict[Tuple[str, ...], int]) -> Dict[Tuple[str, str], int]:
pairs = defaultdict(int)
for word_tuple, freq in vocab.items():
for i in range(len(word_tuple) - 1):
pair = (word_tuple[i], word_tuple[i+1])
pairs[pair] += freq
return pairs
def _merge_vocab(self, pair: Tuple[str, str], vocab: Dict[Tuple[str, ...], int]) -> Dict[Tuple[str, ...], int]:
new_vocab = {}
bigram = " ".join(pair)
replacement = "".join(pair)
for word_tuple, freq in vocab.items():
w_str = " ".join(word_tuple)
w_new = w_str.replace(bigram, replacement)
new_vocab[tuple(w_new.split())] = freq
return new_vocab
def train(self, corpus: List[str]):
word_freqs = defaultdict(int)
for text in corpus:
for word in regex_pre_tokenize(text):
chars = tuple(list(word) + ["</w>"])
word_freqs[chars] += 1
current_vocab = word_freqs
for _ in range(self.num_merges):
pairs = self._get_stats(current_vocab)
if not pairs:
break
best_pair = max(pairs, key=pairs.get)
self.merges.append(best_pair)
current_vocab = self._merge_vocab(best_pair, current_vocab)
# Build vocabulary
for word_tuple in current_vocab.keys():
for subword in word_tuple:
if subword not in self.vocab:
idx = len(self.vocab)
self.vocab[subword] = idx
self.inverse_vocab[idx] = subword
def encode_word(self, word: str) -> List[str]:
tokens = list(word) + ["</w>"]
for pair in self.merges:
i = 0
new_tokens = []
while i < len(tokens):
if i < len(tokens) - 1 and (tokens[i], tokens[i+1]) == pair:
new_tokens.append(pair[0] + pair[1])
i += 2
else:
new_tokens.append(tokens[i])
i += 1
tokens = new_tokens
return tokens
def encode(self, text: str) -> List[int]:
words = regex_pre_tokenize(text)
token_ids = []
for word in words:
subwords = self.encode_word(word)
for sw in subwords:
token_ids.append(self.vocab.get(sw, self.vocab["<unk>"]))
return token_ids
def pad_and_mask_batch(batch_ids: List[List[int]], pad_idx: int = 0) -> Tuple[List[List[int]], List[List[int]]]:
max_len = max(len(seq) for seq in batch_ids)
padded = []
masks = []
for seq in batch_ids:
pad_len = max_len - len(seq)
padded.append(seq + [pad_idx] * pad_len)
masks.append([1] * len(seq) + [0] * pad_len)
return padded, masks
def run_tokenization_demo():
corpus = [
"low lower lowest",
"new newer newest",
"wide wider widest"
]
tok = BPETokenizer(num_merges=5)
tok.train(corpus)
encoded = tok.encode("lowest newest")
padded, masks = pad_and_mask_batch([[1, 2, 3], [4, 5]], pad_idx=0)
print(f"Tokenization Demo: Vocab Size = {len(tok.vocab)}, Merges = {len(tok.merges)}, Padded Len = {len(padded[0])}")
return tok, encoded, padded, masks
if __name__ == "__main__":
run_tokenization_demo()
metadata.yml (448 bytes)
lesson_id: D218
day: 218
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/text_preprocessing_and_tokenization_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 40
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (28 bytes)
numpy>=1.26.0
pytest>=8.0.0
starter/test_text_preprocessing_and_tokenization_lib.py (962 bytes)
import pytest
from examples.text_preprocessing_and_tokenization_lib import regex_pre_tokenize, BPETokenizer, pad_and_mask_batch
def test_regex_pre_tokenize():
text = "Hello, world! It's 2026."
tokens = regex_pre_tokenize(text)
assert "Hello" in tokens
assert "world" in tokens
assert "," in tokens
assert "2026" in tokens
def test_bpe_training_and_encoding():
corpus = ["low lower lowest", "new newer newest", "wide wider widest"]
tok = BPETokenizer(num_merges=4)
tok.train(corpus)
assert len(tok.merges) > 0
assert len(tok.vocab) >= 4 # Includes special tokens
ids = tok.encode("lowest")
assert isinstance(ids, list)
assert len(ids) > 0
def test_pad_and_mask_batch():
batch = [[10, 20, 30], [40, 50]]
padded, masks = pad_and_mask_batch(batch, pad_idx=0)
assert padded[0] == [10, 20, 30]
assert padded[1] == [40, 50, 0]
assert masks[0] == [1, 1, 1]
assert masks[1] == [1, 1, 0]
starter/text_preprocessing_and_tokenization_lib.py (676 bytes)
import re
from collections import defaultdict
from typing import List, Dict, Tuple, Any
def regex_pre_tokenize(text: str) -> List[str]:
# TODO: Split text into words, contractions, numbers, and punctuation using regex
pass
class BPETokenizer:
def __init__(self, num_merges: int = 10):
self.num_merges = num_merges
self.merges: List[Tuple[str, str]] = []
self.vocab: Dict[str, int] = {}
self.inverse_vocab: Dict[int, str] = {}
def train(self, corpus: List[str]):
# TODO: Train BPE merge rules on corpus
pass
def encode(self, text: str) -> List[int]:
# TODO: Convert text to token IDs
pass
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 218 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_text_preprocessing_and_tokenization_lib.py (962 bytes)
import pytest
from examples.text_preprocessing_and_tokenization_lib import regex_pre_tokenize, BPETokenizer, pad_and_mask_batch
def test_regex_pre_tokenize():
text = "Hello, world! It's 2026."
tokens = regex_pre_tokenize(text)
assert "Hello" in tokens
assert "world" in tokens
assert "," in tokens
assert "2026" in tokens
def test_bpe_training_and_encoding():
corpus = ["low lower lowest", "new newer newest", "wide wider widest"]
tok = BPETokenizer(num_merges=4)
tok.train(corpus)
assert len(tok.merges) > 0
assert len(tok.vocab) >= 4 # Includes special tokens
ids = tok.encode("lowest")
assert isinstance(ids, list)
assert len(ids) > 0
def test_pad_and_mask_batch():
batch = [[10, 20, 30], [40, 50]]
padded, masks = pad_and_mask_batch(batch, pad_idx=0)
assert padded[0] == [10, 20, 30]
assert padded[1] == [40, 50, 0]
assert masks[0] == [1, 1, 1]
assert masks[1] == [1, 1, 0]
Troubleshooting
Troubleshooting: Day 218 - Text Preprocessing and Tokenization
Common Issues
- Unseen character produces key error:
- Cause: Missing
<unk>fallback in vocabulary lookup. - Fix: Use
vocab.get(token, vocab["<unk>"]).
- Cause: Missing
Security notes
Security & Privacy: Day 218 - Text Preprocessing and Tokenization
Security Guidance
- All tokenizers operate locally on memory buffers.