Deep Learning › Transformers › Day 228

Hands-on lab — Day 228: Encoder Models: BERT and Friends

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/encoder_models_bert_and_friends_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/encoder_models_bert_and_friends_lib.py
examples/test_encoder_models_bert_and_friends_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/encoder_models_bert_and_friends_lib.py
starter/test_encoder_models_bert_and_friends_lib.py
tests/run_tests.sh
tests/test_encoder_models_bert_and_friends_lib.py
troubleshooting.md

Lab README

Lab: Day 228 -- Encoder Models: BERT and Friends

Lesson

Day number: 228 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: BERT Encoder Architecture and Masked Language Modeling.

Purpose

Build and test the BERTMaskedLM architecture in PyTorch. Implement token and positional embedding lookup, construct the multi-layer bidirectional Transformer encoder stack, implement the masked language model projection head with weight tying, and test vocabulary logit outputs.

Learning objectives

  • Implement the BERT bidirectional Transformer encoder architecture.
  • Construct the Masked Language Model (MLM) head with GELU activation and LayerNorm.
  • Enforce weight tying between input token embeddings and output decoder matrices.
  • Verify dimensional compatibility and vocabulary logits output shapes.

Prerequisites

  • Day 227 (The Transformer Architecture).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/encoder_models_bert_and_friends_lib.py: Student scaffold file.
  • examples/encoder_models_bert_and_friends_lib.py: Complete reference implementation.
  • tests/test_encoder_models_bert_and_friends_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/encoder_models_bert_and_friends_lib.py

What the commands do

  • Executes forward passes through the BERT Masked LM model.
  • Verifies weight tying and vocabulary projection logits.
  • Runs unit test assertions.

Expected output

BERT Demo: Logits Shape = torch.Size([2, 8, 100])

Validation steps

  1. Verify BERTMaskedLM.forward outputs a tensor of shape (batch_size, seq_len, vocab_size).
  2. Confirm mlm_decoder.weight shares the same underlying memory pointer as token_embeddings.weight.
  3. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Weight tying dimension error: Ensure vocab_size and d_model match exactly between embedding and linear layers.

Security notes

All neural computations execute in local process memory.

Extension exercises

  1. Implement dynamic masking collation in PyTorch.
  2. Build a sequence classification head on the [CLS] token.

Expected output

FIELDS.md

# Expected Output Fields: Day 228

- `Logits Shape`: Tensor shape of the Masked Language Model vocabulary prediction logits.

examples-run.txt

BERT Demo: Logits Shape = torch.Size([2, 8, 100])

measured-values.txt

Logits Shape: torch.Size([2, 8, 100])

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 2 items

tests/test_encoder_models_bert_and_friends_lib.py::test_bert_masked_lm_shapes PASSED [ 50%]
tests/test_encoder_models_bert_and_friends_lib.py::test_weight_tying PASSED          [100%]

============================== 2 passed in 0.42s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/encoder_models_bert_and_friends_lib.py (1892 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Optional

class BERTMaskedLM(nn.Module):
    def __init__(self, vocab_size: int = 1000, d_model: int = 32,
                 num_layers: int = 2, num_heads: int = 4):
        super().__init__()
        self.token_embeddings = nn.Embedding(vocab_size, d_model, padding_idx=0)
        self.pos_embeddings = nn.Embedding(128, d_model)
        self.norm = nn.LayerNorm(d_model)

        encoder_layer = nn.TransformerEncoderLayer(
            d_model=d_model, nhead=num_heads, dim_feedforward=4 * d_model,
            activation="gelu", batch_first=True, norm_first=True
        )
        self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=num_layers)

        self.mlm_dense = nn.Linear(d_model, d_model)
        self.mlm_norm = nn.LayerNorm(d_model)
        self.mlm_decoder = nn.Linear(d_model, vocab_size, bias=False)

        # Weight tying
        self.mlm_decoder.weight = self.token_embeddings.weight

    def forward(self, input_ids: torch.Tensor,
                attention_mask: Optional[torch.Tensor] = None) -> torch.Tensor:
        seq_len = input_ids.size(1)
        positions = torch.arange(seq_len, device=input_ids.device).unsqueeze(0)

        x = self.token_embeddings(input_ids) + self.pos_embeddings(positions)
        x = self.norm(x)

        h = self.transformer(x, src_key_padding_mask=attention_mask)
        h_mlm = F.gelu(self.mlm_dense(h))
        h_mlm = self.mlm_norm(h_mlm)
        logits = self.mlm_decoder(h_mlm)
        return logits

def run_bert_demo():
    torch.manual_seed(42)
    model = BERTMaskedLM(vocab_size=100, d_model=16, num_layers=2, num_heads=2)
    input_ids = torch.randint(1, 100, (2, 8))
    logits = model(input_ids)

    print(f"BERT Demo: Logits Shape = {logits.shape}")
    return logits

if __name__ == "__main__":
    run_bert_demo()
examples/test_encoder_models_bert_and_friends_lib.py (523 bytes)
import pytest
import torch
from examples.encoder_models_bert_and_friends_lib import BERTMaskedLM

def test_bert_masked_lm_shapes():
    torch.manual_seed(42)
    model = BERTMaskedLM(vocab_size=200, d_model=16, num_layers=2, num_heads=2)
    input_ids = torch.randint(0, 200, (3, 6))
    logits = model(input_ids)
    assert logits.shape == (3, 6, 200)

def test_weight_tying():
    model = BERTMaskedLM(vocab_size=100, d_model=16)
    assert model.mlm_decoder.weight.data_ptr() == model.token_embeddings.weight.data_ptr()
metadata.yml (444 bytes)
lesson_id: D228
day: 228
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/encoder_models_bert_and_friends_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/encoder_models_bert_and_friends_lib.py (616 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Optional

class BERTMaskedLM(nn.Module):
    def __init__(self, vocab_size: int = 1000, d_model: int = 32,
                 num_layers: int = 2, num_heads: int = 4):
        super().__init__()
        # TODO: Initialize token embeddings, positional embeddings, transformer, and MLM head
        pass

    def forward(self, input_ids: torch.Tensor,
                attention_mask: Optional[torch.Tensor] = None) -> torch.Tensor:
        # TODO: Add embeddings, pass through transformer encoder, and compute MLM logits
        pass
starter/test_encoder_models_bert_and_friends_lib.py (523 bytes)
import pytest
import torch
from examples.encoder_models_bert_and_friends_lib import BERTMaskedLM

def test_bert_masked_lm_shapes():
    torch.manual_seed(42)
    model = BERTMaskedLM(vocab_size=200, d_model=16, num_layers=2, num_heads=2)
    input_ids = torch.randint(0, 200, (3, 6))
    logits = model(input_ids)
    assert logits.shape == (3, 6, 200)

def test_weight_tying():
    model = BERTMaskedLM(vocab_size=100, d_model=16)
    assert model.mlm_decoder.weight.data_ptr() == model.token_embeddings.weight.data_ptr()
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 228 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_encoder_models_bert_and_friends_lib.py (523 bytes)
import pytest
import torch
from examples.encoder_models_bert_and_friends_lib import BERTMaskedLM

def test_bert_masked_lm_shapes():
    torch.manual_seed(42)
    model = BERTMaskedLM(vocab_size=200, d_model=16, num_layers=2, num_heads=2)
    input_ids = torch.randint(0, 200, (3, 6))
    logits = model(input_ids)
    assert logits.shape == (3, 6, 200)

def test_weight_tying():
    model = BERTMaskedLM(vocab_size=100, d_model=16)
    assert model.mlm_decoder.weight.data_ptr() == model.token_embeddings.weight.data_ptr()

Troubleshooting

Troubleshooting: Day 228 - Encoder Models: BERT and Friends

Common Issues

  1. Dimension mismatch in decoder:
    • Cause: Output projection shape does not match (d_model, vocab_size).
    • Fix: Use nn.Linear(d_model, vocab_size, bias=False).

Security notes

Security & Privacy: Day 228 - Encoder Models: BERT and Friends

Security Guidance

  • All encoder model computations execute locally in memory.