Deep Learning β€Ί Sequences and Text β€Ί Day 223

Day 223: Text Classification with Embeddings

Day 223 of 365 β€” Text Classification with Embeddings

Master the core deep learning architectures for text classification: build and benchmark Neural Bag-of-Words (EmbeddingBag), Yoon Kim 1D TextCNN with multi-scale filter banks, and Bidirectional LSTM pooling models, and fine-tune pretrained word embeddings in PyTorch.

Course
Deep Learning
Category
Sequences and Text
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-223-text-classification-with-embeddings

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-223-text-classification-with-embeddings
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

In industrial machine learning, Text Classification is the single most widely deployed Natural Language Processing workload:

While modern 70-billion parameter Large Language Models (LLMs) can perform text classification via prompt engineering, calling a massive LLM API for 10 million incoming user comments per day costs thousands of dollars per month and introduces 500 ms of latency per request.

By contrast, specialized neural text classification architectures:

Today, you will master the three foundational neural document representation paradigms: Neural Bag-of-Words (nn.EmbeddingBag), Yoon Kim 1D TextCNN, and Bidirectional LSTM Pooling.


The idea in plain language

Diagram: classical bag-of-words features compared with averaged embeddings

Diagram: the three neural text classification architectures and what each assumes

Imagine analyzing 10,000 customer reviews for a food delivery app:


Historical background

Comparison of neural text classification architectures including EmbeddingBag average pooling TextCNN 1D convolutions and Bidirectional LSTM pooling

  1. 2014 (Yoon Kim & TextCNN): Yoon Kim published β€œConvolutional Neural Networks for Sentence Classification”, demonstrating that applying standard 1D computer vision convolutions over word embeddings surpassed state-of-the-art SVMs and complex rule-based parsers on 8 standard benchmarks.
  2. 2016 (Armand Joulin & fastText): Facebook AI Research introduced fastText, proving that a simple linear classifier over averaged subword embeddings (EmbeddingBag) achieved accuracy competitive with deep networks while training $1000\times$ faster.
  3. 2017 (Comparative NLP Studies): Wenpeng Yin et al. established the formal trade-off boundary: CNNs excel on local phrase detection and keyphrase sentiment, while BiLSTMs excel on sequential logic, long dependencies, and syntactic agreements.

What it is β€” and what it is not

What Neural Text Classification IS:

What it is NOT:


Why it was created and what problems it solves

Neural text classification solved three major defects of classical TF-IDF / Bag-of-Words models:

  1. Handling Synonyms & Semantic Generalization: Dense embeddings ensure the model generalizes from "unhappy customer" to "disgruntled client" without explicit dictionary engineering.
  2. Local Word Order & Negation Awareness: TextCNN and BiLSTM recognize that "not bad" is positive, whereas Bag-of-Words treats "not" and "bad" as negative tokens.
  3. Sub-Millisecond Edge Serving: Providing lightweight architectures capable of processing thousands of requests per second on edge microservices.

How it works

Let us dissect the three core neural text classification architectures.


1. Model Family 1: Neural Bag-of-Words (nn.EmbeddingBag)

Diagram: separate embedding lookup and pooling compared with a fused kernel

Given a document of $T$ token IDs $(w_1, w_2, \dots, w_T)$:

  1. Look up embeddings $v_1, \dots, v_T \in R^d$.

  2. Compute the global document mean vector:

    v_{\text{doc}} = \frac{1}{T} \sum_{i=1}^T v_i
  3. Pass through a linear projection head with Dropout:

    \hat{y} = W_{\text{head}} v_{\text{doc}} + b

The PyTorch nn.EmbeddingBag Optimization:

Instead of allocating a giant intermediate 3D tensor $(B, T, d)$ in GPU VRAM and then calling torch.mean(), nn.EmbeddingBag executes a fused memory kernel: it loads and sums embedding rows on the fly, reducing memory bandwidth by $95%$ and boosting throughput by $4\times$.


2. Model Family 2: Yoon Kim TextCNN (1D Convolutions)

Diagram: how one-dimensional convolution detects phrases in text

Yoon Kim TextCNN 1D convolutional feature map extraction showing multiple parallel filter window sizes sliding over word embedding matrix

Given a sentence represented as an embedding matrix $X \in R^(T \times d)$:

  1. Parallel 1D Convolutional Filter Banks: Define $K$ filters across multiple window heights (e.g. $h \in [3, 4, 5]$ words):

    • Filter $W_k \in R^(h \times d)$ slides vertically down the sentence across time steps:

      c_{i}^k = \text{ReLU}(W_k * X_{i:i+h-1, :} + b_k)
    • Produces a 1D feature map $c^k \in R^(T - h + 1)$.

  2. 1-Max-Pooling over Time: Extract the single most prominent feature from each map:

    \hat{c}^k = \max_{i} (c_i^k)
  3. Feature Concatenation & Classification: Concatenate all pooled scalar features from all 300 filters into a single vector $z \in R^(300)$, apply Dropout ($p=0.5$), and project to class logits.


3. Model Family 3: Bidirectional LSTM with Temporal Pooling

Given sequential embeddings $(x_1, \dots, x_T)$:

  1. Pass through a 2-layer Bidirectional LSTM:

    [\overrightarrow{h}_t, \overleftarrow{h}_t] = \text{BiLSTM}(x_t, h_{t-1})
  2. Pooling Strategies:

    • Final Hidden State: $h_(doc) = [\overrightarrow(h)_T ,;, \overleftarrow(h)_1] \in R^(2 d_h)$.
    • Global Mean Pooling: $h_(doc) = \frac(1)(T) \sum_(t=1)^T [\overrightarrow(h)_t ,;, \overleftarrow(h)_t]$.
    • Global Max Pooling: $h_(doc) = \max_t [\overrightarrow(h)_t ,;, \overleftarrow(h)_t]$.
  3. Project $h_(doc)$ to class logits.


4. Dilated Convolutions & Temporal Convolutional Networks (TCN)

Standard 1D convolutions have a linear receptive field proportional to kernel size:


5. Hybrid Models: C-LSTM (Convolutional LSTM)

To combine the local phrase extraction of CNNs with the long-range sequential tracking of LSTMs:


6. Imbalanced Text Classification & Focal Loss

Diagram: how focal loss addresses extreme class imbalance

In real-world fraud and toxicity classification, positive classes often comprise $< 1%$ of data:


7. Two-Stage Transfer Learning with Pretrained Embeddings

When fine-tuning on a small labeled dataset with GloVe/FastText embeddings:

  1. Stage 1 (Frozen Embeddings): Freeze embedding.weight.requires_grad = False. Train only the CNN/LSTM parameters for 3 epochs with learning rate $\eta = 10^(-3)$.
  2. Stage 2 (Differential Fine-Tuning): Unfreeze embeddings (requires_grad = True). Set embedding learning rate to $\eta_(embed) = 10^(-4)$ and classifier head to $\eta_(head) = 10^(-3)$.

8. Variable-Length Batch Collation & Dynamic Padding

Diagram: global padding compared with per-batch dynamic padding

In production PyTorch pipelines, padding all batches to the global maximum length (e.g. 512 tokens) wastes huge GPU compute:


9. Regularization Techniques for NLP Classifiers

Diagram: three regularisation techniques for neural text classifiers

Text classification models are prone to overfitting on small vocabulary subsets:


10. Character-Level CNNs (Zhang, Zhao, & LeCun, 2015)

When domain text contains extreme misspellings, slang, or emojis (e.g. social media feeds):


An everyday analogy

Diagram: three ways of summarising a film as an analogy for text classification architectures

Think of assessing a movie:


Examples in practice

Let us build a modular Yoon Kim TextCNN Module in PyTorch:

import torch
import torch.nn as nn
import torch.nn.functional as F

class TextCNN(nn.Module):
    def __init__(self, vocab_size: int, embed_dim: int, num_classes: int,
                 num_filters: int = 100, filter_sizes: list = [3, 4, 5],
                 dropout: float = 0.5):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        
        # Parallel 1D convolutional layers
        self.convs = nn.ModuleList([
            nn.Conv1d(
                in_channels=embed_dim,
                out_channels=num_filters,
                kernel_size=k
            ) for k in filter_sizes
        ])
        
        self.fc = nn.Sequential(
            nn.Dropout(p=dropout),
            nn.Linear(num_filters * len(filter_sizes), num_classes)
        )

    def forward(self, text_indices: torch.Tensor) -> torch.Tensor:
        # text_indices shape: (Batch, Seq_Len)
        embeds = self.embedding(text_indices) # (Batch, Seq_Len, embed_dim)
        
        # Conv1d expects (Batch, Channels=embed_dim, Length=Seq_Len)
        embeds = embeds.permute(0, 2, 1)

        # Apply convolutions and 1-max pooling over time
        pooled_outputs = []
        for conv in self.convs:
            conv_out = F.relu(conv(embeds)) # (Batch, num_filters, Seq_Len - k + 1)
            pooled = F.max_pool1d(conv_out, kernel_size=conv_out.size(2)).squeeze(2)
            pooled_outputs.append(pooled)

        # Concatenate multi-scale features: (Batch, num_filters * 3)
        doc_features = torch.cat(pooled_outputs, dim=1)
        logits = self.fc(doc_features)
        return logits

Implications: security, privacy, performance, scalability, and cost

  1. Adversarial Character Perturbations:
    • Simple character swaps (e.g. "d3licious" or inserting zero-width spaces) can cause out-of-vocabulary fallback, bypassing keyword filters. Combine tokenizers with robust subword or character n-gram defenses.
  2. CPU Inference Latency Profiling:
    • EmbeddingBag: 0.15 ms per batch of 32 on CPU.
    • TextCNN: 1.2 ms per batch of 32 on CPU.
    • BiLSTM: 4.8 ms per batch of 32 on CPU.
    • DistilBERT: 35.0 ms per batch of 32 on CPU.

Alternatives: free, open source, and commercial

Model FamilyTraining TimeCPU LatencyBest For
fastText (EmbeddingBag)30 seconds< 0.2 msReal-time spam/routing at massive scale
TextCNN2 minutes~1 msE-commerce review & intent classification
BiLSTM-Attention5 minutes~4 msComplex clause sentiment and legal tagging
DeBERTa-v3 / BERT30 minutes~40 msMaximum benchmark competition accuracy

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   TEXT CLASSIFICATION TRADE-OFF MATRIX                 β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Architecture       β”‚ Word Order?     β”‚ Negation Catch?  β”‚ GPU Scaling  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ EmbeddingBag       β”‚ No (Bag)        β”‚ Poor             β”‚ Ultra-Fast   β”‚
β”‚ TextCNN            β”‚ Local (n-grams) β”‚ High             β”‚ Fully Para   β”‚
β”‚ BiLSTM             β”‚ Sequential      β”‚ Very High        β”‚ Sequential   β”‚
β”‚ Transformer BERT   β”‚ Full Attn Matrixβ”‚ Perfect          β”‚ Fully Para   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: choosing between lightweight and transformer text classifiers

When to USE EmbeddingBag / TextCNN:

When NOT to use:


The AI connection

Knowledge check

  1. Why is nn.EmbeddingBag more memory-efficient than nn.Embedding + torch.mean?
  2. How do different kernel sizes (k=3, 4, 5) in TextCNN capture multi-scale n-gram features?
  3. What is 1-max-pooling over time and how does it create fixed-size representations?
  4. What are the three primary pooling strategies used for Bidirectional LSTM hidden states?
  5. How does TextCNN inference speed on GPUs compare to Bidirectional LSTMs?

Hands-on exercise

In this lab, you will build and benchmark a complete Neural Text Classification suite in PyTorch: implement an EmbeddingBagClassifier, construct a multi-scale TextCNN architecture with 1D convolutions and global max-over-time pooling, implement a BiLSTMClassifier, handle variable-length sentence batch collation, and benchmark parameter counts and forward latency across all three models.

Expected output

[Neural Text Classification Benchmark Suite]
1. EmbeddingBagClassifier:
   - Parameters: 32,064 | Latency: 0.14 ms | Test Acc: 88.2%
2. Yoon Kim TextCNN (Kernels: [3, 4, 5]):
   - Parameters: 128,602 | Latency: 0.85 ms | Test Acc: 94.1%
3. Bidirectional LSTM (2 Layers, Hidden=64):
   - Parameters: 180,482 | Latency: 3.20 ms | Test Acc: 93.8%
Verification:
  All 3 architectures successfully map variable-length batches to valid logits.
Test Suite: 3 passed in 0.38s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement Multi-Channel TextCNN: create two parallel embedding channels β€” one with static frozen GloVe vectors and one with trainable fine-tuned vectors.
  2. Implement Self-Attention Pooling on top of the BiLSTM feature map and compare accuracy against simple max pooling.

Extension challenge

Implement Depthwise Separable 1D Convolutions in TextCNN:

Quiz

Q1. Why is PyTorch nn.EmbeddingBag significantly faster and more memory-efficient than executing nn.Embedding followed by torch.mean()?

  1. It skips all embedding lookups
  2. It quantizes weights to 1-bit integers
  3. It reduces in one fused kernel, with no giant 3D tensor
  4. It automatically downloads pre-trained weights from HuggingFace
Show answer

Answer: C. It reduces in one fused kernel, with no giant 3D tensor

EmbeddingBag uses a fused kernel that computes the mean/sum reduction directly without instantiating the memory-heavy intermediate 3D embedding tensor.

Q2. What is the purpose of using multiple parallel convolution filter sizes (e.g. k=3, 4, 5) in Yoon Kims TextCNN architecture?

  1. To eliminate the need for an embedding layer
  2. To downsample the spatial resolution of text
  3. To colorize the text characters
  4. To capture several n-gram widths at once
Show answer

Answer: D. To capture several n-gram widths at once

Different kernel heights correspond directly to different n-gram window sizes, capturing diverse phrase lengths in parallel.

Q3. How does "1-Max-Pooling over Time" in TextCNN achieve variable-length sequence invariance?

  1. It takes the highest activation per filter over time
  2. It averages all pixel values to zero
  3. It deletes all sentences shorter than 10 words
  4. It selects the last token in the document
Show answer

Answer: A. It takes the highest activation per filter over time

Global max-over-time pooling picks the most salient feature triggered anywhere in the sequence, producing a fixed-size vector regardless of input length.

Q4. When fine-tuning a text classifier with pretrained GloVe embeddings, what is the recommended strategy for the embedding layer weights?

  1. Randomly re-initialize all embedding weights
  2. Start frozen on pretrained vectors, then unfreeze slowly
  3. Keep embeddings frozen permanently under all circumstances
  4. Delete the vocabulary words not present in the target dataset
Show answer

Answer: B. Start frozen on pretrained vectors, then unfreeze slowly

Two-stage transfer learning (warmup head then unfreeze embeddings with differential learning rate) preserves pretrained semantics while adapting to domain jargon.

Q5. Which document representation architecture is best suited for an ultra-low-latency production microservice requiring < 0.5 ms CPU response times?

  1. A 100-layer CNN
  2. A 24-layer deep Bidirectional LSTM
  3. Neural bag-of-words (EmbeddingBag)
  4. An autoregressive GPT model
Show answer

Answer: C. Neural bag-of-words (EmbeddingBag)

EmbeddingBag requires only a single vector lookup and average, executing in sub-millisecond speeds on standard CPUs.

Glossary

EmbeddingBag
A fused PyTorch module that computes the sum or mean of embedding bags directly without allocating intermediate 3D memory.
TextCNN
A deep learning architecture that applies parallel 1D convolutional filter banks over word embeddings followed by max-over-time pooling.
1-Max-Pooling over Time
A global pooling operation that extracts the maximum activation along the temporal sequence dimension for each feature map channel.
Neural Bag-of-Words (NBoW)
A document classification model that averages dense word vectors across a text and passes the result through a linear classifier.
Collate Function
A custom DataLoader utility that dynamically batches and pads variable-length sequence tensors.
BiLSTM Pooling
Aggregating hidden states from forward and backward LSTM streams via mean pooling, max pooling, or attention pooling.
Static vs Non-Static Embeddings
Static embeddings remain frozen during training; non-static embeddings are fine-tuned via backpropagation gradients.
Temporal Convolutions
Convolutions where kernels slide exclusively across the time/sequence dimension while spanning the entire embedding channel dimension.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.