Deep Learning β€Ί Transformers β€Ί Day 228

Day 228: Encoder Models: BERT and Friends

Day 228 of 365 β€” Encoder Models: BERT and Friends

Master Encoder-only Transformer foundation models: understand Devlin et al. (2018) BERT, Masked Language Modeling (MLM) and Next Sentence Prediction (NSP), analyze RoBERTa dynamic masking, learn ALBERT parameter reduction via cross-layer weight sharing, understand DeBERTa disentangled attention, and extract contextual representations in PyTorch.

Course
Deep Learning
Category
Transformers
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-228-encoder-models-bert-and-friends

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-228-encoder-models-bert-and-friends
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

In October 2018, Google AI researchers Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova released BERT (Bidirectional Encoder Representations from Transformers).

BERT caused an immediate seismic shift across Natural Language Processing:

Unlike autoregressive language models (like GPT) which are constrained by causal left-to-right masks, BERT utilizes unmasked bidirectional self-attention: every single token simultaneously attends to all words to its left AND all words to its right.

Today, you will master the foundational family of Encoder-Only Foundation Models: BERT, RoBERTa, ALBERT, and DeBERTa-v3.


The idea in plain language

Imagine a literary detective solving a redacted manuscript:


Historical background

Diagram: the progression of encoder model refinements after BERT

BERT pretraining architecture diagram showing input tokens with CLS SEP and MASK tokens bidirectional self attention and dual MLM and NSP heads

  1. 2018 (Devlin et al. & BERT): Introduced Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) on 12-layer (Base, 110M params) and 24-layer (Large, 340M params) architectures.
  2. 2019 (Liu et al. & RoBERTa): Proved BERT was severely undertrained. By removing the NSP task, introducing dynamic masking, using larger mini-batches (8k), and training for 10\times more steps, RoBERTa outperformed BERT across the board.
  3. 2019 (Lan et al. & ALBERT): Introduced factorized embeddings and cross-layer weight sharing, reducing parameter counts by 18\times without losing representational power.
  4. 2021 (He et al. & DeBERTa-v3): Introduced Disentangled Attention (separating content and relative position matrices) and ELECTRA-style Replaced Token Detection, surpassing human baseline scores on the SuperGLUE benchmark.

What it is β€” and what it is not

What Encoder Foundation Models ARE:

What they are NOT:

Diagram: encoder models compared with decoder models


Why it was created and what problems it solves

Encoder models solved the single greatest bottleneck in NLP: Contextual Polysemy.


How it works

Let us dissect the pretraining objectives and architectural refinements of the Encoder family.


1. BERT Pretraining Objectives: MLM and NSP

Diagram: the masked language modelling pretraining objective

  1. Masked Language Modeling (MLM):

    • Randomly sample 15\% of token positions in the corpus.
    • For those 15\% positions, apply the 80/10/10 Corruption Rule:
      • 80\% of the time: Replace with the [MASK] token (β€œThe [MASK] was robbed”).
      • 10\% of the time: Replace with a random word (β€œThe apple was robbed”).
      • 10\% of the time: Keep the original word unchanged (β€œThe bank was robbed”).
    • The model passes contextual vectors through a linear projection tied to vocabulary size and minimizes Cross-Entropy loss over only the corrupted positions.
  2. Next Sentence Prediction (NSP):

    • Feed two sentences A and B separated by [SEP].
    • 50\% of the time B is the actual subsequent sentence; 50\% of the time B is a random sentence from another document.
    • Binary classifier on the [CLS] token predicts IsNext vs NotNext.

2. RoBERTa: Robustly Optimized BERT (Liu et al., 2019)

Evolution roadmap of encoder transformer models from BERT to RoBERTa ALBERT and DeBERTa-v3

RoBERTa introduced four critical empirical refinements:

  1. Dynamic Masking: Generating new random masking patterns every time a sequence is fed to the model (instead of static pre-masked datasets).
  2. Removing the NSP Task: Proved that training on continuous full-sentence blocks without NSP improved downstream performance.
  3. Massive Mini-Batch Scaling: Increasing batch size from 256 to 8,192 sequences stabilized gradient updates and accelerated training.
  4. Byte-Pair Encoding (BPE): Upgrading from a 30k WordPiece vocabulary to a 50k Byte-level BPE vocabulary eliminating out-of-vocabulary tokens.

3. ALBERT: A Lite BERT (Lan et al., 2019)

To shrink BERT’s memory footprint:

  1. Factorized Embedding Parameterization:

    • Vocabulary size V = 30,000 and hidden size H = 768 yields V \times H = 23\text{M} embedding parameters.
    • ALBERT projects tokens to a low dimension E = 128 first, then linearly projects E \rightarrow H:
      O(V \times E + E \times H) = 30,000 \times 128 + 128 \times 768 \approx 3.9\text{M params (83% reduction)}
  2. Cross-Layer Parameter Sharing:

    • Shares all Multi-Head Attention and FFN weights across all 12 layers (12\times parameter reduction).
    • An ALBERT-Base model has only 12M parameters (vs 110M in BERT-Base) while achieving comparable accuracy.

4. DeBERTa: Disentangled Attention (He et al., 2020)

DeBERTa models each word using two independent vectors: a Content Vector (c_i) and a Relative Position Vector (p_{i|j}).

The attention score between token i and token j decomposes into three disentangled cross-terms:

\text{Score}_{i, j} = c_i W_{q, c} (c_j W_{k, c})^\top + c_i W_{q, c} (p_{i|j} W_{k, r})^\top + p_{j|i} W_{q, r} (c_j W_{k, c})^\top
  1. Content-to-Content: Semantic similarity between words.
  2. Content-to-Position: Semantic word querying a relative positional offset.
  3. Position-to-Content: Positional offset querying a semantic word.

Combined with Enhanced Mask Decoding (EMD) and ELECTRA-style replaced token detection, DeBERTa-v3 represents the state of the art in transformer encoders.


5. ELECTRA: Replaced Token Detection (Clark et al., 2020)

Diagram: masked language modelling compared with replaced token detection by loss coverage

A major sample-efficiency bottleneck in BERT MLM is that the loss is only computed on the 15\% corrupted tokens, leaving 85\% of tokens unutilized per pass:


6. Sentence-BERT (SBERT) & Dense Vector Embeddings (Reimers & Gurevych, 2019)

Diagram: how encoder outputs become sentence embeddings

How do we turn an Encoder into an ultra-fast semantic search engine?


7. Modern Embedding Encoders (BGE, E5, GTE, Nomic-Embed)

Modern state-of-the-art embedding models build directly on the DeBERTa/BERT encoder architecture:

  1. InfoNCE Contrastive Pretraining: Optimizes positive pairs against large batches of in-batch negatives:

    \mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(q, p^+) / \tau)}{\exp(\text{sim}(q, p^+) / \tau) + \sum_{j} \exp(\text{sim}(q, n_j^-) / \tau)}
  2. Hard Negative Mining: Automatically mines confusing negative passages using BM25 and cross-encoders to sharpen vector boundary representations for RAG retrieval.

  3. Matryoshka Representation Learning (MRL / Kusupati et al., 2022): Truncates embedding dimensions (e.g. 768 \rightarrow 256 or 128) with < 1\% drop in search retrieval performance, slashing database storage costs by 67\%.


8. Token Classification (NER) vs Sequence Classification Head Mechanics

Diagram: the three output heads attached to an encoder model

Fine-tuning an Encoder model for downstream applications requires attaching specific output heads:


9. Two-Stage RAG: Bi-Encoders vs. Cross-Encoders

Diagram: bi-encoder retrieval compared with cross-encoder reranking

In production Retrieval-Augmented Generation (RAG) pipelines, Encoder models serve two complementary roles:

  1. Stage 1: Bi-Encoder (Dense Retriever):
    • Encodes query and millions of documents into isolated vectors independently.
    • Executes nearest-neighbor search via approximate vector indexers in milliseconds.
  2. Stage 2: Cross-Encoder (Re-Ranker):
    • Concatenates the query and top-100 candidate documents into pairs: [CLS] Query [SEP] Document [SEP].
    • Passes each pair through full bidirectional cross-attention, allowing every query token to directly attend to every document token.
    • Outputs a calibrated relevance score, boosting retrieval precision by 20\% to 35\%.

10. Extractive Question Answering (Span Extraction Head)

In reading comprehension tasks like SQuAD:


An everyday analogy

Diagram: analytical characterisation compared with narrative generation

Think of a specialized medical pathology lab:


Examples in practice

Let us build a BERT Masked Language Model Prediction Pipeline in PyTorch:

import torch
import torch.nn as nn
import torch.nn.functional as F

class BERTMaskedLM(nn.Module):
    def __init__(self, vocab_size: int = 30522, d_model: int = 768,
                 num_layers: int = 6, num_heads: int = 12):
        super().__init__()
        self.token_embeddings = nn.Embedding(vocab_size, d_model, padding_idx=0)
        self.pos_embeddings = nn.Embedding(512, d_model)
        self.norm = nn.LayerNorm(d_model)

        # Stack of Transformer Encoder Layers
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=d_model,
            nhead=num_heads,
            dim_feedforward=4 * d_model,
            activation="gelu",
            batch_first=True,
            norm_first=True # Pre-LN
        )
        self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=num_layers)

        # MLM Prediction Head
        self.mlm_dense = nn.Linear(d_model, d_model)
        self.mlm_norm = nn.LayerNorm(d_model)
        self.mlm_decoder = nn.Linear(d_model, vocab_size, bias=False)

        # Weight tying: tie embedding and decoder weights
        self.mlm_decoder.weight = self.token_embeddings.weight

    def forward(self, input_ids: torch.Tensor,
                attention_mask: torch.Tensor = None) -> torch.Tensor:
        seq_len = input_ids.size(1)
        positions = torch.arange(seq_len, device=input_ids.device).unsqueeze(0)

        # Add token and position embeddings
        x = self.token_embeddings(input_ids) + self.pos_embeddings(positions)
        x = self.norm(x)

        # Pass through bidirectional transformer encoder
        h = self.transformer(x, src_key_padding_mask=attention_mask)

        # Compute MLM vocabulary logits
        h_mlm = F.gelu(self.mlm_dense(h))
        h_mlm = self.mlm_norm(h_mlm)
        logits = self.mlm_decoder(h_mlm)

        return logits

Implications: security, privacy, performance, scalability, and cost

Diagram: the efficiency, deployment, bias and length implications of encoder models

  1. Embedding Extraction & Privacy Leakage:
    • Contextual embedding vectors from BERT can leak private entity attributes if fine-tuned on confidential medical records. Apply differential privacy (DP-SGD) when fine-tuning on sensitive corpora.
  2. Inference Latency Optimization:
    • Running BERT-Base takes ~25 ms per sentence on CPU.
    • DistilBERT (Sanh et al., 2019): Uses knowledge distillation to compress BERT into 6 layers, retaining 97\% of accuracy while running 60\% faster with 40\% fewer parameters.

Alternatives: free, open source, and commercial

ModelParametersGLUE ScoreKey Advantage
BERT-Base110M79.6Standard baseline benchmark
RoBERTa-Base125M86.4Robustly trained, widely supported
ALBERT-Base12M82.3Minimal memory footprint
DeBERTa-v3-Base86M89.9SOTA accuracy for classification/NER
DistilBERT66M77.0High-throughput low-latency serving

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   ENCODER TRANSFORMER COMPARISON                       β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Architecture       β”‚ Masking Style   β”‚ Positional Schemeβ”‚ Parameter Trkβ”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ BERT (2018)        β”‚ Static 15%      β”‚ Learned Absolute β”‚ 110M Base    β”‚
β”‚ RoBERTa (2019)     β”‚ Dynamic 15%     β”‚ Learned Absolute β”‚ 125M Base    β”‚
β”‚ ALBERT (2019)      β”‚ Dynamic 15%     β”‚ Learned Absolute β”‚ 12M (Shared) β”‚
β”‚ DeBERTa-v3 (2021)  β”‚ RTD (ELECTRA)   β”‚ Disentangled Rel β”‚ 86M Base     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when encoder models are the right choice

When to USE Encoder Models (BERT/RoBERTa/DeBERTa):

When NOT to use:


The AI connection

Knowledge check

  1. What is the 80/10/10 rule in BERT Masked Language Modeling and why was it designed?
  2. Why did RoBERTa eliminate the Next Sentence Prediction (NSP) pretraining task?
  3. How does ALBERT reduce parameter count by 18x through cross-layer weight sharing?
  4. What three cross-terms are computed in DeBERTa Disentangled Attention?
  5. Why is bidirectional self-attention superior to causal attention for text classification?

Hands-on exercise

In this lab, you will build and test a modular BERTMaskedLM architecture in PyTorch: implement token and positional embedding lookup, construct the multi-layer bidirectional Transformer encoder stack, implement the masked language model projection head with weight tying, and test mask token prediction recovery.

Expected output

[BERT and Encoder Foundation Models Verification Suite]
Testing BERTMaskedLM Forward Pass:
  Input Batch: torch.Size([2, 10]) (Batch=2, Tokens=10 with [MASK] at index 3)
  Vocabulary Logits Output: torch.Size([2, 10, 1000]) [VOCAB MATCH]
Testing Weight Tying:
  Embedding Weight ID == MLM Decoder Weight ID [WEIGHTS TIED]
Testing Masked Token Prediction:
  Masked Position Top-1 Token ID: 412 (Logit: +14.82)
Test Suite: 3 passed in 0.42s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement Sentence Order Prediction (SOP) from ALBERT and train a miniature binary classification head on the [CLS] token.
  2. Implement Dynamic Masking collation in PyTorch DataLoader.

Extension challenge

Implement Disentangled Attention in Pure PyTorch:

Quiz

Q1. What is the "80/10/10 rule" used in BERT Masked Language Modeling (MLM) for the 15% selected token positions?

  1. 80% [MASK], 10% random, 10% unchanged
  2. 80% of tokens are deleted, 10% are duplicated, and 10% are converted to uppercase
  3. 80% train, 10% validation, 10% test
  4. 80% English, 10% French, 10% German
Show answer

Answer: A. 80% [MASK], 10% random, 10% unchanged

The 80/10/10 rule bridges the pretraining-finetuning discrepancy by ensuring the encoder maintains rich contextual representations even when [MASK] is absent.

Q2. Why did Liu et al. (2019) remove the Next Sentence Prediction (NSP) task in RoBERTa?

  1. NSP was patented
  2. Ablations showed NSP did not help
  3. NSP caused GPU overheating
  4. NSP made the model generate text
Show answer

Answer: B. Ablations showed NSP did not help

RoBERTa showed that BERT was significantly undertrained and that removing NSP while training on contiguous full sequences yielded superior representations.

Q3. How does ALBERT (Lan et al., 2019) achieve an 18x reduction in parameter count compared to BERT-Large while maintaining comparable accuracy?

  1. By discarding the vocabulary table
  2. By quantizing weights to 1-bit
  3. Factorized embeddings, shared layers
  4. By running on a single CPU core
Show answer

Answer: C. Factorized embeddings, shared layers

Cross-layer weight sharing and factorized embedding projections drastically compress parameter counts without sacrificing model depth.

Q4. What is "Disentangled Attention" in DeBERTa (He et al., 2020)?

  1. Running attention on separate GPU threads
  2. Disabling attention for even-numbered tokens
  3. Separating nouns from verbs
  4. Separate content and position vectors
Show answer

Answer: D. Separate content and position vectors

Disentangled attention explicitly models semantic content-to-position and position-to-content cross-terms, achieving state-of-the-art encoder performance.

Q5. Why are Encoder-only models (BERT/RoBERTa/DeBERTa) preferred over Decoder-only models (GPT) for extractive question answering and token classification (NER)?

  1. Encoders are bidirectional, decoders causal
  2. Encoder models are smaller
  3. Decoder models cannot process punctuation
  4. Encoder models do not use PyTorch
Show answer

Answer: A. Encoders are bidirectional, decoders causal

Bidirectional context allows tokens to observe surrounding words on both sides, essential for classification and span extraction.

Glossary

BERT
Bidirectional Encoder Representations from Transformers: an encoder foundation model pretrained with MLM and NSP.
Masked Language Modeling (MLM)
A self-supervised pretraining objective where 15% of tokens are corrupted and the model predicts their original IDs.
Next Sentence Prediction (NSP)
A binary classification pretraining task predicting whether sentence B logically follows sentence A.
RoBERTa
A robustly optimized BERT variant trained with dynamic masking, larger batches, longer steps, and no NSP task.
ALBERT
A Lite BERT architecture using cross-layer weight sharing and factorized embedding projections to slash parameters.
DeBERTa
Decoding-enhanced BERT utilizing disentangled content-position attention and Enhanced Mask Decoding.
Disentangled Attention
Computing separate attention matrices for content-to-content, content-to-position, and position-to-content interactions.
Special Tokens ([CLS], [SEP], [MASK])
Reserved tokens used for sequence classification ([CLS]), sentence separation ([SEP]), and cloze pretraining ([MASK]).

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.