Deep Learning β€Ί Sequences and Text β€Ί Day 218

Day 218: Text Preprocessing and Tokenization

Day 218 of 365 β€” Text Preprocessing and Tokenization

Master the fundamental bridge between raw human text and deep sequence models: implement whitespace, regex, character, and Byte-Pair Encoding (BPE) tokenizers from scratch, manage special tokens and vocabularies, and eliminate out-of-vocabulary failures.

Course
Deep Learning
Category
Sequences and Text
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-218-text-preprocessing-and-tokenization

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-218-text-preprocessing-and-tokenization
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Welcome to Week 32: Sequences and Text and Course 05 Subsection 02: Sequence Models and Transformers.

Computers and deep neural networks cannot read, understand, or process raw human text strings like "Hello world!". Neural networks are mathematical engines that operate exclusively on continuous numerical tensors and matrix multiplications.

Therefore, before any language model, sentiment classifier, machine translation engine, or Transformer (like BERT, GPT-4, or Llama) can process text, the raw characters must pass through the Tokenization Pipeline:

  1. How raw continuous text strings are segmented into discrete semantic atoms (Tokens).
  2. How each token maps to a unique integer index in a fixed Vocabulary Registry ($V$).
  3. How variable-length sequences are padded, truncated, and masked into rigid multi-dimensional batch tensors.

Tokenization is the foundational bedrock of all Natural Language Processing (NLP) and modern Generative AI. Suboptimal tokenization can ruin model performance, introduce catastrophic Out-Of-Vocabulary (OOV) blind spots, multiply inference compute costs by $4\times$, or cause arithmetic failure in LLMs.

Today, you will master the algorithms that convert text to numbers from first principles.


The idea in plain language

Diagram: word-level, character-level and subword tokenisation compared

Diagram: the role of a tokeniser between text and a model

Imagine teaching an intelligent alien robot how to read a library of human books:


Historical background

Comparison of character level word level and subword tokenization strategies showing vocabulary size out of vocabulary handling and sequence length tradeoffs

  1. 1990s–2010s (Rule-Based & Whitespace Tokenizers): Early NLP libraries (NLTK, SpaCy) relied on handcrafted regular expressions, Penn Treebank tokenization rules, and Porter Stemming (running -> run). Vocabularies suffered from the Out-Of-Vocabulary (OOV) catastrophe: any unseen word was replaced by <unk>, destroying semantic meaning.
  2. 2016 (Sennrich et al. & Byte-Pair Encoding): Rico Sennrich adapted the 1994 data compression algorithm Byte-Pair Encoding (BPE) for neural machine translation. It proved that decomposing words into frequent subword units eliminated OOV words while keeping vocabulary sizes bounded.
  3. 2018–2019 (WordPiece, SentencePiece, & Byte-Level BPE): Google introduced WordPiece for BERT and SentencePiece for multilingual models. OpenAI introduced Byte-level BPE in GPT-2, initializing the base vocabulary with 256 raw bytes, ensuring zero OOV tokens across every human language, code, and emoji.

What it is β€” and what it is not

What Tokenization IS:

What it is NOT:


Why it was created and what problems it solves

Subword tokenization solved three fatal bottlenecks in sequence modeling:

  1. The Out-Of-Vocabulary (OOV) Dilemma: Fixed word dictionaries cannot handle typos ("definately"), compound words ("neurotechnology"), or new jargon. Subwords break novel words into known constituent syllables.
  2. The Softmax Output Bottleneck: The final layer of a language model computes a softmax distribution over the entire vocabulary $|V|$. If $|V| = 1,000,000$, computing the output projection consumes massive GPU memory. Subwords reduce $|V|$ to 32,000–100,000 tokens.
  3. Morphological Generalization: Subwords allow models to share representations across related morphological variations: play, player, playing, and playful share the root token play.

How it works

Let us dissect the four core stages of modern tokenization pipelines.


1. Stage 1: Text Normalization and Pre-Tokenization

Before splitting text into subwords, the raw input string undergoes cleaning and coarse boundary splitting:

  1. Unicode Normalization (NFKD / NFC): Standardizes multiple byte representations of accented characters into canonical form (e.g. Γ© as single code point \u00E9 vs letter e + combining accent \u0301).
  2. Whitespace and Punctuation Splitting (Regex): Modern tokenizers (like GPT-4’s tiktoken) apply regex patterns to prevent subwords from crossing word or category boundaries:
    # GPT-2 Regex Pattern: separates letters, contractions, numbers, and punctuation
    import re
    GPT2_SPLIT_REGEX = r"'s|'t|'re|'ve|'m|'ll|'d| ?\p(L)+| ?\p(N)+| ?[^\s\p(L)\p(N)]+|\s+(?!\S)|\s+"

2. Stage 2: The Byte-Pair Encoding (BPE) Algorithm

Diagram: the byte-pair encoding training and encoding procedures

Byte Pair Encoding BPE merge algorithm iterating over character pairs merging most frequent bigrams into vocabulary

BPE is an iterative bottom-up subword clustering algorithm:

Training BPE Vocabularies:

  1. Initialization: Split every word in the training corpus into individual characters, adding a special end-of-word marker (e.g. </w> or leading space ).

  2. Count Bigram Frequencies: Count the occurrences of all adjacent symbol pairs across all words in the corpus:

    Frequency(Pair(t_a, t_b)) = sum over words [ Count(word) * Occurrences(t_a, t_b in word) ]
  3. Merge the Most Frequent Pair: Identify the pair with maximum frequency, create a new combined token t_new = t_a + t_b, and add it to the vocabulary.

  4. Update Corpus: Replace all occurrences of t_a, t_b with t_new.

  5. Iterate: Repeat steps 2–4 until the vocabulary reaches the target size $K$ (e.g. 32,000 merges).

Encoding New Text at Inference Time:

Given a new word, split it into characters and apply all learned merge rules in the exact order they were discovered during training.


3. Stage 3: Special Tokens Registry

Diagram: the special tokens reserved in a sequence pipeline

Every production sequence pipeline reserves dedicated special tokens:


4. Stage 4: Batch Padding, Truncation, and Attention Masks

Diagram: padding, truncation and attention masking for batched sequences

Neural networks process batches of sentences simultaneously. Because natural sentences have different lengths, the tokenizer outputs three coordinated tensors:

Sentence 1: "The cat sat"             (Length: 3)
Sentence 2: "The black cat sat on it" (Length: 6)

Input IDs Tensor (Padded to Max Length = 6):
[[ 101,  1996,  4937,  2938,     0,     0 ],  <-- Padded with <pad>=0
 [ 101,  1996,  2304,  4937,  2938,  2006 ]]

Attention Mask Tensor (1 = Real Token, 0 = Padding to Ignore):
[[   1,     1,     1,     1,     0,     0 ],
 [   1,     1,     1,     1,     1,     1 ]]

5. WordPiece vs Unigram Language Model (SentencePiece)

Diagram: byte-pair encoding compared with WordPiece and unigram tokenisation

Beyond Byte-Pair Encoding, two other major subword algorithms power modern foundation models:

  1. WordPiece (Used in BERT & DistilBERT):
    • Similar to BPE, starts with individual characters, but chooses which pair to merge based on Likelihood Maximization rather than raw frequency count:

      Score(Pair(u, v)) = Count(u, v) / (Count(u) * Count(v))
    • Uses special prefix markers (e.g. ##ing) to designate subwords that continue an existing word without leading whitespace.

  2. Unigram Language Model Tokenization (Used in T5, LLaMA, SentencePiece):
    • Takes the inverse approach of BPE: initializes a massive vocabulary of 500,000 subwords and iteratively prunes subwords that contribute least to the overall corpus language modeling likelihood.
    • Enables probabilistic subword sampling during training (Subword Regularization), training models on multiple candidate segmentations of the same sentence.

6. Critical Tokenizer Pitfalls in Large Language Models

Diagram: the language model behaviours that tokenisation explains

Modern LLM behavior is heavily shaped by tokenizer mechanics:

  1. Arithmetic Degradation: In models like GPT-3, numbers were fragmented unpredictably: 1000 was one token, but 1001 was two tokens (100 + 1), making basic addition difficult. Modern tokenizers (Llama 3, GPT-4o) enforce single-digit tokenization for all numbers.
  2. The Reversal Curse: Tokenizers parse words in a forward direction; generating tokens in reverse character order is unnatural for autoregressive language models.
  3. Leading Space Glitches: In BPE tokenizers, " Hello" (with a space) maps to token ID 15496, whereas "Hello" (no space) maps to token ID 9906. Prompts with accidental leading/trailing spaces alter the input representations.

An everyday analogy

Diagram: standardised containers as an analogy for fixed-length tokenised batches

Think of packing items into standardized shipping containers:


Examples in practice

Let us implement a Byte-Pair Encoding (BPE) Tokenizer from Scratch in Python:

from collections import defaultdict
from typing import Dict, List, Tuple

class SimpleBPETokenizer:
    def __init__(self, num_merges: int = 10):
        self.num_merges = num_merges
        self.merges: List[Tuple[str, str]] = []
        self.vocab: Dict[str, int] = {}

    def get_stats(self, vocab: Dict[Tuple[str, ...], int]) -> Dict[Tuple[str, str], int]:
        pairs = defaultdict(int)
        for word_tuple, freq in vocab.items():
            for i in range(len(word_tuple) - 1):
                pair = (word_tuple[i], word_tuple[i+1])
                pairs[pair] += freq
        return pairs

    def merge_vocab(self, pair: Tuple[str, str], vocab: Dict[Tuple[str, ...], int]) -> Dict[Tuple[str, ...], int]:
        new_vocab = {}
        bigram = " ".join(pair)
        replacement = "".join(pair)
        for word_tuple, freq in vocab.items():
            w_str = " ".join(word_tuple)
            w_new = w_str.replace(bigram, replacement)
            new_vocab[tuple(w_new.split())] = freq
        return new_vocab

    def train(self, corpus: List[str]):
        # Initialize word frequencies with character tuples
        word_freqs = defaultdict(int)
        for text in corpus:
            for word in text.split():
                # Append end-of-word marker
                chars = tuple(list(word) + ["</w>"])
                word_freqs[chars] += 1

        # Iteratively merge most frequent pairs
        current_vocab = word_freqs
        for _ in range(self.num_merges):
            pairs = self.get_stats(current_vocab)
            if not pairs:
                break
            best_pair = max(pairs, key=pairs.get)
            self.merges.append(best_pair)
            current_vocab = self.merge_vocab(best_pair, current_vocab)

        # Build final vocabulary registry
        tokens = {"<pad>": 0, "<unk>": 1, "<bos>": 2, "<eos>": 3}
        for word_tuple in current_vocab.keys():
            for subword in word_tuple:
                if subword not in tokens:
                    tokens[subword] = len(tokens)
        self.vocab = tokens

Implications: security, privacy, performance, scalability, and cost

Diagram: the cost, context, security and portability implications of tokenisation

  1. The Tokenizer Glitch Vulnerability (SolidGoldMagikarp):
    • Certain rare clusters of web scrape text (like Reddit usernames or e-commerce tags) received single token IDs in GPT tokenizers without ever appearing in neural training datasets. Prompting models with these glitch tokens causes hallucinations or safety filter bypasses.
  2. Multi-Lingual Token Inequity (The Token Tax):
    • Tokenizers trained predominantly on English text represent English words with 1.2 tokens on average, while Hindi, Arabic, or Korean text requires 4 to 8 tokens per word. This makes non-English LLM inference $4\times$ more expensive and reduces effective context window length.
  3. Whitespace Sensitivity in Code and JSON:
    • Leading vs trailing spaces produce completely different token IDs (e.g. " def" $\neq$ "def"). Ensure code generation prompts maintain consistent whitespace indentation.

Alternatives: free, open source, and commercial

Library / TokenizerAlgorithmLanguageBest For
Hugging Face tokenizersBPE, WordPiece, UnigramRust / Python bindingsIndustry standard for all models
OpenAI tiktokenByte-level BPERust / PythonFast GPT-4 / GPT-3.5 tokenization
Google sentencepieceUnigram / BPEC++ / PythonMultilingual models (T5, LLaMA)
SpaCy / NLTKRule-based regex wordsPythonTraditional NLP and rule parsing

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   TOKENIZATION STRATEGY TRADEOFF MATRIX                β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Strategy           β”‚ Vocabulary Size β”‚ Sequence Length  β”‚ OOV Rate     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Word-Level         β”‚ 500,000+        β”‚ Very Short       β”‚ Very High    β”‚
β”‚ Character-Level    β”‚ 256             β”‚ 5x - 10x Longer  β”‚ 0%           β”‚
β”‚ Subword (BPE)      β”‚ 32,000 - 100,000β”‚ Compact (1.3x)   β”‚ 0% (Bytes)   β”‚
β”‚ Byte-level BPE     β”‚ 50,000 - 128,000β”‚ Optimal          β”‚ 0% (Strict)  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when tokeniser choices matter

When to USE Subword Tokenization:

When NOT to use:


The AI connection

Knowledge check

  1. Why does word-level tokenization suffer from Out-Of-Vocabulary (OOV) errors?
  2. What are the core steps of the Byte-Pair Encoding (BPE) training algorithm?
  3. How does Byte-level BPE guarantee zero unknown tokens across all Unicode text?
  4. What is the role of an Attention Mask in batched sequence processing?
  5. Why does language representation cost vary across languages in English-centric tokenizers?

Hands-on exercise

In this lab, you will build and test a complete text preprocessing and tokenization toolkit in Python: implement text normalization and regex pre-tokenization, train a Byte-Pair Encoding (BPE) merge registry from a text corpus, build a numerical vocabulary encoder/decoder with special tokens (<pad>, <unk>, <bos>, <eos>), and generate padded PyTorch batch tensors with corresponding attention masks.

Expected output

[Tokenization Verification Suite]
Testing Text Normalization and Regex Splitting:
  Raw text: "Let's test BPE, it's 100% working!"
  Tokens: ["Let", "'s", "test", "BPE", ",", "it", "'s", "100", "%", "working", "!"]
Testing Byte-Pair Encoding (BPE) Training (10 merges):
  Learned Merges: [('e', 's'), ('es', 't'), ('l', 'o'), ('lo', 'w'), ('n', 'e')]
  Encoding "lowest" -> Subwords: ["low", "est</w>"] -> IDs: [12, 18]
Testing Batch Tensor Padding and Attention Masking:
  Sequences Padded to Max Length: 6
  Batch Tensor Shape: torch.Size([2, 6])
  Attention Mask Shape: torch.Size([2, 6])
  Valid Non-Pad Tokens Count: 9
Test Suite: 3 passed in 0.28s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement a BPE Dropout function (stochastic subword regularization) where candidate merges are dropped with probability $p=0.1$ during training to improve model robustness to noisy text.
  2. Build an Information Entropy & Compression Ratio Calculator comparing the token compression efficiency of character vs word vs BPE tokenization on a Wikipedia sample.

Extension challenge

Implement Byte-Level BPE with Unicode Fallback:

Quiz

Q1. Why have subword tokenization algorithms (such as Byte-Pair Encoding and WordPiece) completely superseded word-level tokenization in modern deep learning and LLMs?

  1. Subword keeps the vocabulary small with no OOV tokens
  2. Subword tokenization eliminates the need for GPUs
  3. Word-level tokenization cannot process English words
  4. Subword tokenization compresses text files on disk by 99%
Show answer

Answer: A. Subword keeps the vocabulary small with no OOV tokens

Subword algorithms handle rare and unseen compound words by breaking them into familiar root subwords, keeping vocabularies bounded.

Q2. What is the core iterative mechanism of the Byte-Pair Encoding (BPE) vocabulary training algorithm?

  1. Compute the gradient descent of word loss
  2. Merge the most frequent adjacent pair, repeatedly
  3. Randomly delete vowels from the text corpus
  4. Sort words alphabetically and pick the top 50,000
Show answer

Answer: B. Merge the most frequent adjacent pair, repeatedly

BPE iteratively finds the most frequent adjacent bigram in the segmented corpus and replaces all occurrences with a newly minted merged token.

Q3. What is the purpose of the special token <pad> (Padding) in batch sequence processing for neural networks?

  1. It speeds up tokenization by skipping punctuation
  2. It encrypts the text input
  3. It pads short sequences so the batch is rectangular
  4. It marks the beginning of a document
Show answer

Answer: C. It pads short sequences so the batch is rectangular

Neural networks require rectangular batch tensors; padding fills empty sequence slots and is masked during attention and loss computation.

Q4. How does Byte-level BPE (as used in GPT-2, RoBERTa, and GPT-4) guarantee zero Out-Of-Vocabulary (OOV) tokens for any arbitrary unicode text?

  1. It replaces unknown symbols with spaces
  2. It downloads a dictionary of every language in the world
  3. It ignores non-English characters
  4. It starts from all 256 bytes, so any text is representable
Show answer

Answer: D. It starts from all 256 bytes, so any text is representable

Because every Unicode character decomposes into UTF-8 bytes (0-255), starting BPE at the byte level means every string has a valid byte decomposition.

Q5. Why is Unicode normalization (e.g. NFC or NFKD) an essential preprocessing step before text tokenization?

  1. Identical-looking characters can have different codepoints
  2. It translates text into English automatically
  3. It converts lowercase letters to uppercase
  4. It strips HTML tags from raw web pages
Show answer

Answer: A. Identical-looking characters can have different codepoints

Unicode normalization converts distinct byte representations of equivalent characters (e.g. precomposed vs decomposed accents) into a canonical form.

Glossary

Tokenization
The computational process of segmenting a continuous raw string of text into discrete numerical units (tokens).
Byte-Pair Encoding (BPE)
A subword tokenization algorithm that iteratively merges the most frequent pairs of adjacent characters or tokens.
Out-Of-Vocabulary (OOV)
Words or symbols encountered during inference that do not exist in the models pre-trained vocabulary registry.
Special Tokens
Reserved structural markers (<pad>, <unk>, <bos>, <eos>, <mask>) used to indicate sequence boundaries and control flags.
Unicode Normalization
Standardizing Unicode text strings (NFC/NFKD) so equivalent character representations share identical binary bytes.
WordPiece
A subword segmentation algorithm (used in BERT) that chooses pair merges maximizing training data language model likelihood.
SentencePiece
A language-independent tokenizer that treats the input text as a raw stream of characters including whitespace as a meta-symbol.
Attention Mask
A binary mask tensor indicating which tokens are valid sequence content (1) versus padding tokens (0) to be ignored.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.