Deep Learning › Training at Scale › Day 238
Hands-on lab — Day 238: Section Project: Reproducing a Paper
- ← Back to the Day 238 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deep-learning/day-238-section-project-reproducing-a-paper/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/section_project_reproducing_a_paper_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/section_project_reproducing_a_paper_lib.py examples/test_section_project_reproducing_a_paper_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/section_project_reproducing_a_paper_lib.py starter/test_section_project_reproducing_a_paper_lib.py tests/run_tests.sh tests/test_section_project_reproducing_a_paper_lib.py troubleshooting.md
Lab README
Lab: Day 238 -- Section Project: Reproducing a Paper
Lesson
Day number: 238 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Scientific Paper Reproduction, Single-Batch Overfit, and Multi-Seed Aggregation.
Purpose
Build and test a modular paper reproduction and ablation verification harness in PyTorch. Implement a configurable Transformer architecture supporting Pre-LN and Post-LN variants, execute a single-batch overfit sanity test, and aggregate multi-seed experimental results.
Learning objectives
- Implement the 6-stage scientific paper reproduction protocol.
- Build a single-batch overfit verification harness to debug gradient flow.
- Calculate multi-seed mean and standard deviation aggregates to test robustness.
- Conduct an ablation comparison between Pre-LN and Post-LN Transformer designs.
Prerequisites
- Day 237 (Scaling Laws and What They Bought Us).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/section_project_reproducing_a_paper_lib.py: Student scaffold file.examples/section_project_reproducing_a_paper_lib.py: Complete reference implementation.tests/test_section_project_reproducing_a_paper_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/section_project_reproducing_a_paper_lib.py
What the commands do
- Evaluates Transformer forward passes across Pre-LN and Post-LN modes.
- Executes single-batch overfitting sanity checks.
- Calculates statistical aggregate metrics across multi-seed runs.
- Runs unit test assertions.
Expected output
Reproduction Demo: Single-Batch Overfit = True, Multi-Seed F1 = 0.915 +/- 0.0073
Validation steps
- Verify
ConfigurableTransformerproduces shape(batch, num_classes). - Confirm
run_single_batch_overfitdrives loss below $0.05$. - Confirm
calculate_multi_seed_aggregatecalculates correct mean and sample standard deviation. - Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Overfit failure: Ensure learning rate is sufficiently high ($0.01$) and optimizer step is called.
Security notes
All reproduction benchmarks execute locally in process memory.
Extension exercises
- Implement an automated ablation runner comparing multiple learning rate schedules.
- Build a layer-wise weight similarity comparison tool.
Navigation
- Lesson title: Section Project: Reproducing a Paper
- Day number: 238 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-238-section-project-reproducing-a-paper
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-238-section-project-reproducing-a-paperwhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 238
- `Single-Batch Overfit`: Boolean status confirming loss dropped below 0.05 on fixed batch.
- `Multi-Seed F1`: Mean F1 score and standard deviation across random seeds.
examples-run.txt
Reproduction Demo: Single-Batch Overfit = True, Multi-Seed F1 = 0.915 +/- 0.0073
measured-values.txt
Single-Batch Overfit: True
Multi-Seed F1: 0.915 +/- 0.0073
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 3 items
tests/test_section_project_reproducing_a_paper_lib.py::test_configurable_transformer_forward PASSED [ 33%]
tests/test_section_project_reproducing_a_paper_lib.py::test_single_batch_overfit PASSED [ 66%]
tests/test_section_project_reproducing_a_paper_lib.py::test_multi_seed_aggregate_metrics PASSED [100%]
============================== 3 passed in 0.43s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/section_project_reproducing_a_paper_lib.py (2545 bytes)
import torch
import torch.nn as nn
import numpy as np
from typing import Dict, Any, List
class ConfigurableTransformer(nn.Module):
def __init__(self, vocab_size: int = 100, embed_dim: int = 32, num_classes: int = 2, use_pre_ln: bool = True):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.use_pre_ln = use_pre_ln
self.ln1 = nn.LayerNorm(embed_dim)
self.attn = nn.MultiheadAttention(embed_dim, num_heads=2, batch_first=True)
self.ln2 = nn.LayerNorm(embed_dim)
self.mlp = nn.Sequential(
nn.Linear(embed_dim, embed_dim * 2),
nn.GELU(),
nn.Linear(embed_dim * 2, embed_dim)
)
self.classifier = nn.Linear(embed_dim, num_classes)
def forward(self, x: torch.Tensor) -> torch.Tensor:
h = self.embedding(x)
if self.use_pre_ln:
norm_h = self.ln1(h)
attn_out, _ = self.attn(norm_h, norm_h, norm_h)
h = h + attn_out
h = h + self.mlp(self.ln2(h))
else:
attn_out, _ = self.attn(h, h, h)
h = self.ln1(h + attn_out)
h = self.ln2(h + self.mlp(h))
pooled = h.mean(dim=1)
return self.classifier(pooled)
def run_single_batch_overfit(model: nn.Module, steps: int = 50) -> bool:
optimizer = torch.optim.AdamW(model.parameters(), lr=0.01)
criterion = nn.CrossEntropyLoss()
x = torch.randint(0, 100, (8, 16))
y = torch.randint(0, 2, (8,))
for _ in range(steps):
optimizer.zero_grad()
logits = model(x)
loss = criterion(logits, y)
loss.backward()
optimizer.step()
return bool(loss.item() < 0.05)
def calculate_multi_seed_aggregate(scores: List[float]) -> Dict[str, float]:
arr = np.array(scores)
return {
"mean": round(float(np.mean(arr)), 4),
"std": round(float(np.std(arr)), 4),
"min": round(float(np.min(arr)), 4),
"max": round(float(np.max(arr)), 4)
}
def run_reproduction_demo():
torch.manual_seed(42)
model = ConfigurableTransformer(vocab_size=100, embed_dim=32, num_classes=2, use_pre_ln=True)
overfit_passed = run_single_batch_overfit(model, steps=60)
seed_stats = calculate_multi_seed_aggregate([0.912, 0.925, 0.908])
print(f"Reproduction Demo: Single-Batch Overfit = {overfit_passed}, Multi-Seed F1 = {seed_stats['mean']} +/- {seed_stats['std']}")
return overfit_passed, seed_stats
if __name__ == "__main__":
run_reproduction_demo()
examples/test_section_project_reproducing_a_paper_lib.py (932 bytes)
import pytest
import torch
from examples.section_project_reproducing_a_paper_lib import (
ConfigurableTransformer,
run_single_batch_overfit,
calculate_multi_seed_aggregate
)
def test_configurable_transformer_forward():
model_pre = ConfigurableTransformer(use_pre_ln=True)
model_post = ConfigurableTransformer(use_pre_ln=False)
x = torch.randint(0, 100, (4, 16))
out_pre = model_pre(x)
out_post = model_post(x)
assert out_pre.shape == (4, 2)
assert out_post.shape == (4, 2)
def test_single_batch_overfit():
torch.manual_seed(42)
model = ConfigurableTransformer(embed_dim=32, num_classes=2, use_pre_ln=True)
passed = run_single_batch_overfit(model, steps=60)
assert passed is True
def test_multi_seed_aggregate_metrics():
res = calculate_multi_seed_aggregate([0.90, 0.92, 0.94])
assert res["mean"] == 0.92
assert res["min"] == 0.90
assert res["max"] == 0.94
metadata.yml (448 bytes)
lesson_id: D238
day: 238
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/section_project_reproducing_a_paper_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/section_project_reproducing_a_paper_lib.py (595 bytes)
import torch
import torch.nn as nn
from typing import Dict, Any, List
class ConfigurableTransformer(nn.Module):
def __init__(self, vocab_size: int = 100, embed_dim: int = 32, num_classes: int = 2, use_pre_ln: bool = True):
super().__init__()
# TODO: Initialize configurable transformer architecture
pass
def forward(self, x: torch.Tensor) -> torch.Tensor:
# TODO: Forward pass supporting Pre-LN vs Post-LN
pass
def run_single_batch_overfit(model: nn.Module, steps: int = 50) -> bool:
# TODO: Overfit single batch to near zero loss
pass
starter/test_section_project_reproducing_a_paper_lib.py (932 bytes)
import pytest
import torch
from examples.section_project_reproducing_a_paper_lib import (
ConfigurableTransformer,
run_single_batch_overfit,
calculate_multi_seed_aggregate
)
def test_configurable_transformer_forward():
model_pre = ConfigurableTransformer(use_pre_ln=True)
model_post = ConfigurableTransformer(use_pre_ln=False)
x = torch.randint(0, 100, (4, 16))
out_pre = model_pre(x)
out_post = model_post(x)
assert out_pre.shape == (4, 2)
assert out_post.shape == (4, 2)
def test_single_batch_overfit():
torch.manual_seed(42)
model = ConfigurableTransformer(embed_dim=32, num_classes=2, use_pre_ln=True)
passed = run_single_batch_overfit(model, steps=60)
assert passed is True
def test_multi_seed_aggregate_metrics():
res = calculate_multi_seed_aggregate([0.90, 0.92, 0.94])
assert res["mean"] == 0.92
assert res["min"] == 0.90
assert res["max"] == 0.94
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 238 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_section_project_reproducing_a_paper_lib.py (932 bytes)
import pytest
import torch
from examples.section_project_reproducing_a_paper_lib import (
ConfigurableTransformer,
run_single_batch_overfit,
calculate_multi_seed_aggregate
)
def test_configurable_transformer_forward():
model_pre = ConfigurableTransformer(use_pre_ln=True)
model_post = ConfigurableTransformer(use_pre_ln=False)
x = torch.randint(0, 100, (4, 16))
out_pre = model_pre(x)
out_post = model_post(x)
assert out_pre.shape == (4, 2)
assert out_post.shape == (4, 2)
def test_single_batch_overfit():
torch.manual_seed(42)
model = ConfigurableTransformer(embed_dim=32, num_classes=2, use_pre_ln=True)
passed = run_single_batch_overfit(model, steps=60)
assert passed is True
def test_multi_seed_aggregate_metrics():
res = calculate_multi_seed_aggregate([0.90, 0.92, 0.94])
assert res["mean"] == 0.92
assert res["min"] == 0.90
assert res["max"] == 0.94
Troubleshooting
Troubleshooting: Day 238 - Section Project: Reproducing a Paper
Common Issues
- Single batch overfit divergence:
- Cause: LayerNorm applied incorrectly across sequence dimension instead of embedding dimension.
- Fix: Use
nn.LayerNorm(embed_dim).
Security notes
Security & Privacy: Day 238 - Section Project: Reproducing a Paper
Security Guidance
- All reproduction experimental harnesses execute locally in memory.