Deep Learning › Transformers › Day 228
Hands-on lab — Day 228: Encoder Models: BERT and Friends
- ← Back to the Day 228 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deep-learning/day-228-encoder-models-bert-and-friends/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/encoder_models_bert_and_friends_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/encoder_models_bert_and_friends_lib.py examples/test_encoder_models_bert_and_friends_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/encoder_models_bert_and_friends_lib.py starter/test_encoder_models_bert_and_friends_lib.py tests/run_tests.sh tests/test_encoder_models_bert_and_friends_lib.py troubleshooting.md
Lab README
Lab: Day 228 -- Encoder Models: BERT and Friends
Lesson
Day number: 228 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: BERT Encoder Architecture and Masked Language Modeling.
Purpose
Build and test the BERTMaskedLM architecture in PyTorch. Implement token and positional embedding lookup, construct the multi-layer bidirectional Transformer encoder stack, implement the masked language model projection head with weight tying, and test vocabulary logit outputs.
Learning objectives
- Implement the BERT bidirectional Transformer encoder architecture.
- Construct the Masked Language Model (MLM) head with GELU activation and LayerNorm.
- Enforce weight tying between input token embeddings and output decoder matrices.
- Verify dimensional compatibility and vocabulary logits output shapes.
Prerequisites
- Day 227 (The Transformer Architecture).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/encoder_models_bert_and_friends_lib.py: Student scaffold file.examples/encoder_models_bert_and_friends_lib.py: Complete reference implementation.tests/test_encoder_models_bert_and_friends_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/encoder_models_bert_and_friends_lib.py
What the commands do
- Executes forward passes through the BERT Masked LM model.
- Verifies weight tying and vocabulary projection logits.
- Runs unit test assertions.
Expected output
BERT Demo: Logits Shape = torch.Size([2, 8, 100])
Validation steps
- Verify
BERTMaskedLM.forwardoutputs a tensor of shape(batch_size, seq_len, vocab_size). - Confirm
mlm_decoder.weightshares the same underlying memory pointer astoken_embeddings.weight. - Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Weight tying dimension error: Ensure
vocab_sizeandd_modelmatch exactly between embedding and linear layers.
Security notes
All neural computations execute in local process memory.
Extension exercises
- Implement dynamic masking collation in PyTorch.
- Build a sequence classification head on the
[CLS]token.
Navigation
- Lesson title: Encoder Models: BERT and Friends
- Day number: 228 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-228-encoder-models-bert-and-friends
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-228-encoder-models-bert-and-friendswhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 228
- `Logits Shape`: Tensor shape of the Masked Language Model vocabulary prediction logits.
examples-run.txt
BERT Demo: Logits Shape = torch.Size([2, 8, 100])
measured-values.txt
Logits Shape: torch.Size([2, 8, 100])
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 2 items
tests/test_encoder_models_bert_and_friends_lib.py::test_bert_masked_lm_shapes PASSED [ 50%]
tests/test_encoder_models_bert_and_friends_lib.py::test_weight_tying PASSED [100%]
============================== 2 passed in 0.42s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/encoder_models_bert_and_friends_lib.py (1892 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Optional
class BERTMaskedLM(nn.Module):
def __init__(self, vocab_size: int = 1000, d_model: int = 32,
num_layers: int = 2, num_heads: int = 4):
super().__init__()
self.token_embeddings = nn.Embedding(vocab_size, d_model, padding_idx=0)
self.pos_embeddings = nn.Embedding(128, d_model)
self.norm = nn.LayerNorm(d_model)
encoder_layer = nn.TransformerEncoderLayer(
d_model=d_model, nhead=num_heads, dim_feedforward=4 * d_model,
activation="gelu", batch_first=True, norm_first=True
)
self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=num_layers)
self.mlm_dense = nn.Linear(d_model, d_model)
self.mlm_norm = nn.LayerNorm(d_model)
self.mlm_decoder = nn.Linear(d_model, vocab_size, bias=False)
# Weight tying
self.mlm_decoder.weight = self.token_embeddings.weight
def forward(self, input_ids: torch.Tensor,
attention_mask: Optional[torch.Tensor] = None) -> torch.Tensor:
seq_len = input_ids.size(1)
positions = torch.arange(seq_len, device=input_ids.device).unsqueeze(0)
x = self.token_embeddings(input_ids) + self.pos_embeddings(positions)
x = self.norm(x)
h = self.transformer(x, src_key_padding_mask=attention_mask)
h_mlm = F.gelu(self.mlm_dense(h))
h_mlm = self.mlm_norm(h_mlm)
logits = self.mlm_decoder(h_mlm)
return logits
def run_bert_demo():
torch.manual_seed(42)
model = BERTMaskedLM(vocab_size=100, d_model=16, num_layers=2, num_heads=2)
input_ids = torch.randint(1, 100, (2, 8))
logits = model(input_ids)
print(f"BERT Demo: Logits Shape = {logits.shape}")
return logits
if __name__ == "__main__":
run_bert_demo()
examples/test_encoder_models_bert_and_friends_lib.py (523 bytes)
import pytest
import torch
from examples.encoder_models_bert_and_friends_lib import BERTMaskedLM
def test_bert_masked_lm_shapes():
torch.manual_seed(42)
model = BERTMaskedLM(vocab_size=200, d_model=16, num_layers=2, num_heads=2)
input_ids = torch.randint(0, 200, (3, 6))
logits = model(input_ids)
assert logits.shape == (3, 6, 200)
def test_weight_tying():
model = BERTMaskedLM(vocab_size=100, d_model=16)
assert model.mlm_decoder.weight.data_ptr() == model.token_embeddings.weight.data_ptr()
metadata.yml (444 bytes)
lesson_id: D228
day: 228
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/encoder_models_bert_and_friends_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/encoder_models_bert_and_friends_lib.py (616 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Optional
class BERTMaskedLM(nn.Module):
def __init__(self, vocab_size: int = 1000, d_model: int = 32,
num_layers: int = 2, num_heads: int = 4):
super().__init__()
# TODO: Initialize token embeddings, positional embeddings, transformer, and MLM head
pass
def forward(self, input_ids: torch.Tensor,
attention_mask: Optional[torch.Tensor] = None) -> torch.Tensor:
# TODO: Add embeddings, pass through transformer encoder, and compute MLM logits
pass
starter/test_encoder_models_bert_and_friends_lib.py (523 bytes)
import pytest
import torch
from examples.encoder_models_bert_and_friends_lib import BERTMaskedLM
def test_bert_masked_lm_shapes():
torch.manual_seed(42)
model = BERTMaskedLM(vocab_size=200, d_model=16, num_layers=2, num_heads=2)
input_ids = torch.randint(0, 200, (3, 6))
logits = model(input_ids)
assert logits.shape == (3, 6, 200)
def test_weight_tying():
model = BERTMaskedLM(vocab_size=100, d_model=16)
assert model.mlm_decoder.weight.data_ptr() == model.token_embeddings.weight.data_ptr()
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 228 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_encoder_models_bert_and_friends_lib.py (523 bytes)
import pytest
import torch
from examples.encoder_models_bert_and_friends_lib import BERTMaskedLM
def test_bert_masked_lm_shapes():
torch.manual_seed(42)
model = BERTMaskedLM(vocab_size=200, d_model=16, num_layers=2, num_heads=2)
input_ids = torch.randint(0, 200, (3, 6))
logits = model(input_ids)
assert logits.shape == (3, 6, 200)
def test_weight_tying():
model = BERTMaskedLM(vocab_size=100, d_model=16)
assert model.mlm_decoder.weight.data_ptr() == model.token_embeddings.weight.data_ptr()
Troubleshooting
Troubleshooting: Day 228 - Encoder Models: BERT and Friends
Common Issues
- Dimension mismatch in decoder:
- Cause: Output projection shape does not match
(d_model, vocab_size). - Fix: Use
nn.Linear(d_model, vocab_size, bias=False).
- Cause: Output projection shape does not match
Security notes
Security & Privacy: Day 228 - Encoder Models: BERT and Friends
Security Guidance
- All encoder model computations execute locally in memory.