Deep Learning β€Ί Sequences and Text β€Ί Day 224

Day 224: A Sentiment Analysis Project

Day 224 of 365 β€” A Sentiment Analysis Project

Engineer an end-to-end sentiment classification pipeline in PyTorch: build vocabulary tokenizers with frequency pruning, implement and benchmark 3 model families (Baseline Logistic Regression, fastText EmbeddingBag, and BiLSTM with Attention Pooling), compute precision-recall curves and calibration error, and implement token attribution explainability.

Course
Deep Learning
Category
Sequences and Text
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 55 min
Lesson duration
1h 30m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-224-a-sentiment-analysis-project

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-224-a-sentiment-analysis-project
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Building a production-ready Sentiment Analysis System is the ultimate proving ground for sequential deep learning:

Today, you will design, implement, evaluate, and interpret an end-to-end sentiment classification system in PyTorch.


The idea in plain language

Diagram: dictionary-based sentiment compared with a learned classifier

Diagram: the seven stages of a production sentiment analysis system

Imagine running quality assurance for a national airline receiving 50,000 tweets and customer feedback forms per hour:


Historical background

End to end sentiment analysis system engineering pipeline architecture from text ingestion to explainability and REST API

  1. 2002 (Pang, Lee, & Vaithyanathan): Published the seminal paper β€œThumbs up? Sentiment Classification using Machine Learning Techniques”, establishing benchmark evaluation on movie reviews with Naive Bayes and Support Vector Machines.
  2. 2016 (Yang et al. & Hierarchical Attention): Introduced Hierarchical Attention Networks (HAN), demonstrating that word-level and sentence-level attention weights isolate key sentiment expressions.
  3. 2017 (Sundararajan et al. & Integrated Gradients): Formulated axiomatic gradient-based feature attribution for deep neural networks, providing mathematically sound explainability for text predictions.

What it is β€” and what it is not

What an End-to-End Sentiment Pipeline IS:

What it is NOT:


Why it was created and what problems it solves

Modern sentiment analysis systems solve three critical industrial challenges:

  1. Handling Negations & Clause Shifts: Capturing sentiment reversals across complex grammatical conjunctions (β€œdespite the delay, the flight crew was fantastic”).
  2. Uncertainty Calibration: Ensuring that when the model reports a $90%$ positive prediction, it is genuinely correct $90%$ of the time.
  3. Auditability & Regulatory Explainability: Ensuring automated content filtering and customer routing decisions are explainable to compliance teams.

How it works

Let us examine the core engineering stages of an end-to-end sentiment analysis system.


1. Data Preprocessing and Dynamic Batching

  1. Text Normalization:
    • Strip raw HTML tags, URLs, and special control characters using compiled regular expressions.
    • Standardize contractions ("can't" $\rightarrow$ "cannot", "won't" $\rightarrow$ "will not").
  2. Vocabulary Building:
    • Build a vocabulary dictionary mapping unique tokens to integer IDs with special reserved tokens:
      • 0: <pad> (padding)
      • 1: <unk> (out-of-vocabulary fallback)
      • 2: <bos> (beginning-of-sequence)
      • 3: <eos> (end-of-sequence)
    • Prune tokens appearing fewer than $3$ times.
  3. Dynamic Batch Collate Function:
    • In PyTorch, pad sequences to the maximum length in the batch rather than the global dataset maximum.

2. Deep Model Architecture: BiLSTM with Attention Pooling

Given input tokens $(x_1, \dots, x_T)$:

  1. Embedding Layer: $E = Embedding(x) \in R^(T \times d_e)$.

  2. Bidirectional LSTM:

    H = \text{BiLSTM}(E) \in \mathbb{R}^{T \times 2 d_h}
  3. Learned Attention Pooling: Compute attention alignment weight $\alpha_t$ for each time step:

    u_t = \tanh(W_a * h_t + b_a)
    \alpha_t = \frac{\exp(u_t^\top u_w)}{\sum_{j=1}^T \exp(u_j^\top u_w)}
    v_{\text{doc}} = \sum_{t=1}^T \alpha_t * h_t \in \mathbb{R}^{2 d_h}
  4. Classification Head:

    \hat{y} = \text{Linear}(\text{Dropout}(v_{\text{doc}}, p=0.5))

3. Model Explainability: Gradient-Based Token Attribution

Diagram: how gradient-based attribution identifies influential tokens

Token attribution and saliency explanation heatmap showing positive and negative word feature contributions to predicted sentiment

To discover which words caused a specific prediction:

  1. Compute the gradient of the target class score $y_(target)$ with respect to the input embedding vectors $E$:

    G_i = \nabla_{E_i} y_{\text{target}}
  2. Compute the token attribution score:

    \text{Saliency}(w_i) = E_i \cdot G_i = \sum_{k=1}^d E_{i, k} \frac{\partial y_{\text{target}}}{\partial E_{i, k}}
  3. Positive values indicate words driving the prediction toward the target class; negative values indicate words resisting the prediction.


4. Comprehensive Evaluation Metrics

  1. Macro F1-Score:

    \text{F1}_{\text{macro}} = \frac{1}{K} \sum_{k=1}^K \frac{2 \cdot P_k \cdot R_k}{P_k + R_k}
  2. Expected Calibration Error (ECE): Group predictions into $M = 10$ confidence bins $[B_1, \dots, B_M]$:

    \text{ECE} = \sum_{m=1}^M \frac{|B_m|}{N} |\text{acc}(B_m) - \text{conf}(B_m)|

5. Aspect-Based Sentiment Analysis (ABSA)

Diagram: overall sentiment compared with aspect-based sentiment

In e-commerce and hospitality, customer reviews frequently express mixed sentiment toward different aspects of the experience (e.g. β€œThe food was delicious, but the waiter was extremely rude”):


6. Probability Calibration: Temperature Scaling

Deep neural networks are notorious for overconfidence β€” outputting $99%$ probabilities on incorrect predictions:


7. Axiomatic Feature Attribution: Integrated Gradients

Diagram: the axioms plain gradients violate and how integrated gradients satisfy them

Simple gradient saliency ($\nabla_x y$) violates two fundamental explainability axioms: Completeness and Implementation Invariance:


8. ONNX Export and Sub-Millisecond CPU Inference

To deploy sentiment classifiers to high-throughput production servers:


9. Multi-Lingual Sentiment & Cross-Lingual Zero-Shot Transfer

Global enterprises receive feedback across dozens of languages:


10. Continuous Monitoring: Data Drift & Population Stability Index (PSI)

Over time, language changes β€” new slang emerges, product names launch, and customer sentiment distribution shifts:


11. CheckList Behavioral Testing for NLP Models (Ribeiro et al., 2020)

Diagram: test-set accuracy compared with behavioural capability testing

Standard test-set accuracy fails to uncover hidden vulnerabilities:


12. Active Learning & Human-in-the-Loop Annotation

Instead of labeling thousands of random reviews, modern engineering uses Active Learning:


13. Production Governance: Model Cards & Ethics

Deploying automated sentiment systems requires structured governance:


An everyday analogy

Diagram: manual review compared with automated triage at volume

Think of an executive summary memo presented to a CEO:


Examples in practice

Let us build an End-to-End BiLSTM-Attention Sentiment Analyzer with Explainability in PyTorch:

import torch
import torch.nn as nn
import torch.nn.functional as F

class BiLSTMAttentionSentiment(nn.Module):
    def __init__(self, vocab_size: int, embed_dim: int, hidden_dim: int, num_classes: int = 2):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True, bidirectional=True)
        
        # Attention pooling projection
        self.attn_proj = nn.Linear(hidden_dim * 2, hidden_dim)
        self.attn_vec = nn.Linear(hidden_dim, 1, bias=False)
        
        self.classifier = nn.Sequential(
            nn.Dropout(0.5),
            nn.Linear(hidden_dim * 2, num_classes)
        )

    def forward(self, x: torch.Tensor, mask: torch.Tensor = None):
        embeds = self.embedding(x) # (Batch, Seq_Len, embed_dim)
        h_seq, _ = self.lstm(embeds) # (Batch, Seq_Len, hidden_dim * 2)

        # Compute attention scores
        u = torch.tanh(self.attn_proj(h_seq))
        scores = self.attn_vec(u).squeeze(2) # (Batch, Seq_Len)

        if mask is not None:
            scores = scores.masked_fill(mask == 0, -1e9)

        attn_weights = F.softmax(scores, dim=1) # (Batch, Seq_Len)
        doc_vec = torch.bmm(attn_weights.unsqueeze(1), h_seq).squeeze(1) # (Batch, hidden_dim * 2)

        logits = self.classifier(doc_vec)
        return logits, attn_weights

    def explain_tokens(self, x: torch.Tensor, target_class: int = 1):
        # Computes gradient saliency token attribution scores
        self.eval()
        embeds = self.embedding(x).detach().requires_grad_(True)
        h_seq, _ = self.lstm(embeds)
        u = torch.tanh(self.attn_proj(h_seq))
        scores = self.attn_vec(u).squeeze(2)
        attn_weights = F.softmax(scores, dim=1)
        doc_vec = torch.bmm(attn_weights.unsqueeze(1), h_seq).squeeze(1)
        logits = self.classifier(doc_vec)

        score = logits[0, target_class]
        score.backward()

        saliency = (embeds.grad[0] * embeds[0]).sum(dim=1).detach()
        return saliency, attn_weights[0]

Implications: security, privacy, performance, scalability, and cost

Diagram: the calibration, drift, governance and fairness implications of sentiment systems

  1. Adversarial Sarcasm and Evasion:
    • Malicious users can bypass moderation filters by phrasing toxic reviews as polite sarcasm (β€œThank you so much for deleting all my files, truly stellar work”). Fine-tune on sarcastic corpora.
  2. Selective Classification (Reject Option):
    • In automated customer response pipelines, set an uncertainty threshold: if $\max p(y) < 0.70$, divert the review to human support.

Alternatives: free, open source, and commercial

SolutionDeployment FootprintLatencyExplainability
VADER / TextBlobRule-based lexicon0.05 msLexicon lookups
PyTorch BiLSTM-Attn5 MB model binary1.5 msIntegrated Gradients
RoBERTa / DeBERTa450 MB model binary25 msLayer-wise Attention
Commercial LLM APICloud hosted API400 msPrompt Rationale

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   SENTIMENT ANALYSIS ARCHITECTURES                     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Paradigm           β”‚ Semantic Depth  β”‚ CPU Latency      β”‚ Interpret    β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Lexicon (VADER)    β”‚ Surface Only    β”‚ 0.05 ms          β”‚ Rule list    β”‚
β”‚ EmbeddingBag       β”‚ Bag-of-Words    β”‚ 0.15 ms          β”‚ Word weights β”‚
β”‚ BiLSTM-Attention   β”‚ Deep Context    β”‚ 1.5 ms           β”‚ Attention Mapβ”‚
β”‚ Fine-tuned BERT    β”‚ Maximum SOTA    β”‚ 25.0 ms          β”‚ Multi-Head   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when a custom sentiment model is warranted

When to USE Custom BiLSTM-Attention Sentiment Models:

When NOT to use:


The AI connection

Knowledge check

  1. Why is Accuracy an insufficient evaluation metric on imbalanced sentiment datasets?
  2. How does Attention Pooling capture the core sentiment in contrastive sentences?
  3. How does gradient-based token attribution calculate the importance of each word?
  4. What is Expected Calibration Error (ECE) and why is it important for production serving?
  5. What is Selective Classification (the Reject Option) in automated NLP pipelines?

Hands-on exercise

In this lab, you will build and test an end-to-end Sentiment Analysis system in PyTorch: implement the BiLSTMAttentionSentiment model, train it with cross-entropy loss, compute confusion matrices and Macro F1-scores, extract attention alignment weights, and verify token attribution saliency scores for individual test sentences.

Expected output

[End-to-End Sentiment Analysis Verification Suite]
Testing BiLSTMAttentionSentiment Forward Pass:
  Input Batch: torch.Size([4, 12]) (4 reviews, 12 tokens max)
  Logits Output: torch.Size([4, 2]) [Positive / Negative]
  Attention Weights Sum: 1.0000 across all 4 reviews [VERIFIED]
Testing Token Attribution Explainability:
  Sentence: ["The", "battery", "is", "fantastic", "and", "seamless"]
  Top Attribution Word: "fantastic" (Saliency: +0.88)
  Attention Peak: "fantastic" (Weight: 0.62)
Test Suite: 3 passed in 0.42s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement Temperature Scaling to calibrate model confidence probabilities and reduce Expected Calibration Error.
  2. Build an Interactive HTML Saliency Visualizer that highlights words in green (positive) and red (negative) based on attribution scores.

Extension challenge

Implement Integrated Gradients with Riemann Sum Approximation:

Quiz

Q1. Why is evaluating a sentiment analysis model using only Accuracy misleading on real-world datasets?

  1. Accuracy requires floating-point quantization
  2. Accuracy cannot be calculated on text
  3. PyTorch does not support accuracy calculations
  4. Reviews are imbalanced, so a constant scores 85%
Show answer

Answer: D. Reviews are imbalanced, so a constant scores 85%

Class imbalance makes accuracy misleading; Precision, Recall, and Macro F1-Score are required to measure true performance across all sentiment categories.

Q2. How does Gradient-Based Token Attribution (Saliency / Integrated Gradients) compute the importance of individual words in a predicted review?

  1. The gradient of the output w.r.t. each word embedding
  2. By counting the number of letters in each word
  3. By checking the word against a hardcoded positive dictionary
  4. By deleting random words and checking if the computer restarts
Show answer

Answer: A. The gradient of the output w.r.t. each word embedding

Gradient saliency measures the sensitivity of the output score to infinitesimal shifts in each word embedding vector.

Q3. What is Expected Calibration Error (ECE) in production NLP classification systems?

  1. The clock drift of the GPU server
  2. Whether stated confidence matches empirical accuracy
  3. The difference between train and test loss
  4. The time taken to tokenize 1,000 words
Show answer

Answer: B. Whether stated confidence matches empirical accuracy

ECE quantifies probability calibration, ensuring downstream automated systems can trust the models predicted confidence scores.

Q4. In a customer service sentiment routing system, what is the best strategy for handling ambiguous predictions with low confidence (< 65%)?

  1. Delete the review from the database entirely
  2. Always guess positive
  3. Send ambiguous reviews to a human queue
  4. Double the batch size
Show answer

Answer: C. Send ambiguous reviews to a human queue

Selective classification routes low-confidence predictions to human annotators, maintaining high precision on automated actions.

Q5. Why does a BiLSTM with Attention Pooling outperform simple Global Max Pooling on nuanced sentiment reviews with contrastive clauses ("The food was great, but the service was completely unacceptable")?

  1. Attention pooling only runs on GPUs
  2. BiLSTM has fewer parameters
  3. Max pooling requires sorted words
  4. Attention weights the contrastive clause in context
Show answer

Answer: D. Attention weights the contrastive clause in context

Attention pooling captures the relative syntactic dominance of contrastive clauses, resolving complex sentiment transitions.

Glossary

Sentiment Analysis
The automated classification of subjective text data into emotional polarity categories (e.g. Positive, Negative, Neutral).
Macro F1-Score
The unweighted arithmetic mean of F1-scores calculated independently across each target class, treating all classes equally.
ROC-AUC
Receiver Operating Characteristic Area Under the Curve: measures the models ability to discriminate between positive and negative classes across all classification thresholds.
Integrated Gradients
An axiomatic feature attribution method that calculates word importance by integrating gradients along the straight line from a baseline embedding to the input embedding.
Expected Calibration Error (ECE)
The weighted average difference between predicted confidence bin probabilities and empirical accuracy rates.
Selective Classification
A deployment strategy where predictions below a confidence threshold are rejected and routed to human review.
Token Saliency
The sensitivity of the model output score to individual token embedding vectors calculated via first-order gradients.
Contrastive Clause
A linguistic construction (e.g. "although X, Y") where sentiment shifts across the sentence.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.