Deep Learning › Training at Scale › Day 238

Hands-on lab — Day 238: Section Project: Reproducing a Paper

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/section_project_reproducing_a_paper_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/section_project_reproducing_a_paper_lib.py
examples/test_section_project_reproducing_a_paper_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/section_project_reproducing_a_paper_lib.py
starter/test_section_project_reproducing_a_paper_lib.py
tests/run_tests.sh
tests/test_section_project_reproducing_a_paper_lib.py
troubleshooting.md

Lab README

Lab: Day 238 -- Section Project: Reproducing a Paper

Lesson

Day number: 238 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Scientific Paper Reproduction, Single-Batch Overfit, and Multi-Seed Aggregation.

Purpose

Build and test a modular paper reproduction and ablation verification harness in PyTorch. Implement a configurable Transformer architecture supporting Pre-LN and Post-LN variants, execute a single-batch overfit sanity test, and aggregate multi-seed experimental results.

Learning objectives

  • Implement the 6-stage scientific paper reproduction protocol.
  • Build a single-batch overfit verification harness to debug gradient flow.
  • Calculate multi-seed mean and standard deviation aggregates to test robustness.
  • Conduct an ablation comparison between Pre-LN and Post-LN Transformer designs.

Prerequisites

  • Day 237 (Scaling Laws and What They Bought Us).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/section_project_reproducing_a_paper_lib.py: Student scaffold file.
  • examples/section_project_reproducing_a_paper_lib.py: Complete reference implementation.
  • tests/test_section_project_reproducing_a_paper_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/section_project_reproducing_a_paper_lib.py

What the commands do

  • Evaluates Transformer forward passes across Pre-LN and Post-LN modes.
  • Executes single-batch overfitting sanity checks.
  • Calculates statistical aggregate metrics across multi-seed runs.
  • Runs unit test assertions.

Expected output

Reproduction Demo: Single-Batch Overfit = True, Multi-Seed F1 = 0.915 +/- 0.0073

Validation steps

  1. Verify ConfigurableTransformer produces shape (batch, num_classes).
  2. Confirm run_single_batch_overfit drives loss below $0.05$.
  3. Confirm calculate_multi_seed_aggregate calculates correct mean and sample standard deviation.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Overfit failure: Ensure learning rate is sufficiently high ($0.01$) and optimizer step is called.

Security notes

All reproduction benchmarks execute locally in process memory.

Extension exercises

  1. Implement an automated ablation runner comparing multiple learning rate schedules.
  2. Build a layer-wise weight similarity comparison tool.

Expected output

FIELDS.md

# Expected Output Fields: Day 238

- `Single-Batch Overfit`: Boolean status confirming loss dropped below 0.05 on fixed batch.
- `Multi-Seed F1`: Mean F1 score and standard deviation across random seeds.

examples-run.txt

Reproduction Demo: Single-Batch Overfit = True, Multi-Seed F1 = 0.915 +/- 0.0073

measured-values.txt

Single-Batch Overfit: True
Multi-Seed F1: 0.915 +/- 0.0073

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_section_project_reproducing_a_paper_lib.py::test_configurable_transformer_forward PASSED [ 33%]
tests/test_section_project_reproducing_a_paper_lib.py::test_single_batch_overfit PASSED [ 66%]
tests/test_section_project_reproducing_a_paper_lib.py::test_multi_seed_aggregate_metrics PASSED [100%]

============================== 3 passed in 0.43s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/section_project_reproducing_a_paper_lib.py (2545 bytes)
import torch
import torch.nn as nn
import numpy as np
from typing import Dict, Any, List

class ConfigurableTransformer(nn.Module):
    def __init__(self, vocab_size: int = 100, embed_dim: int = 32, num_classes: int = 2, use_pre_ln: bool = True):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.use_pre_ln = use_pre_ln
        self.ln1 = nn.LayerNorm(embed_dim)
        self.attn = nn.MultiheadAttention(embed_dim, num_heads=2, batch_first=True)
        self.ln2 = nn.LayerNorm(embed_dim)
        self.mlp = nn.Sequential(
            nn.Linear(embed_dim, embed_dim * 2),
            nn.GELU(),
            nn.Linear(embed_dim * 2, embed_dim)
        )
        self.classifier = nn.Linear(embed_dim, num_classes)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        h = self.embedding(x)
        if self.use_pre_ln:
            norm_h = self.ln1(h)
            attn_out, _ = self.attn(norm_h, norm_h, norm_h)
            h = h + attn_out
            h = h + self.mlp(self.ln2(h))
        else:
            attn_out, _ = self.attn(h, h, h)
            h = self.ln1(h + attn_out)
            h = self.ln2(h + self.mlp(h))
            
        pooled = h.mean(dim=1)
        return self.classifier(pooled)

def run_single_batch_overfit(model: nn.Module, steps: int = 50) -> bool:
    optimizer = torch.optim.AdamW(model.parameters(), lr=0.01)
    criterion = nn.CrossEntropyLoss()
    x = torch.randint(0, 100, (8, 16))
    y = torch.randint(0, 2, (8,))

    for _ in range(steps):
        optimizer.zero_grad()
        logits = model(x)
        loss = criterion(logits, y)
        loss.backward()
        optimizer.step()

    return bool(loss.item() < 0.05)

def calculate_multi_seed_aggregate(scores: List[float]) -> Dict[str, float]:
    arr = np.array(scores)
    return {
        "mean": round(float(np.mean(arr)), 4),
        "std": round(float(np.std(arr)), 4),
        "min": round(float(np.min(arr)), 4),
        "max": round(float(np.max(arr)), 4)
    }

def run_reproduction_demo():
    torch.manual_seed(42)
    model = ConfigurableTransformer(vocab_size=100, embed_dim=32, num_classes=2, use_pre_ln=True)
    overfit_passed = run_single_batch_overfit(model, steps=60)
    seed_stats = calculate_multi_seed_aggregate([0.912, 0.925, 0.908])

    print(f"Reproduction Demo: Single-Batch Overfit = {overfit_passed}, Multi-Seed F1 = {seed_stats['mean']} +/- {seed_stats['std']}")
    return overfit_passed, seed_stats

if __name__ == "__main__":
    run_reproduction_demo()
examples/test_section_project_reproducing_a_paper_lib.py (932 bytes)
import pytest
import torch
from examples.section_project_reproducing_a_paper_lib import (
    ConfigurableTransformer,
    run_single_batch_overfit,
    calculate_multi_seed_aggregate
)

def test_configurable_transformer_forward():
    model_pre = ConfigurableTransformer(use_pre_ln=True)
    model_post = ConfigurableTransformer(use_pre_ln=False)
    x = torch.randint(0, 100, (4, 16))

    out_pre = model_pre(x)
    out_post = model_post(x)
    assert out_pre.shape == (4, 2)
    assert out_post.shape == (4, 2)

def test_single_batch_overfit():
    torch.manual_seed(42)
    model = ConfigurableTransformer(embed_dim=32, num_classes=2, use_pre_ln=True)
    passed = run_single_batch_overfit(model, steps=60)
    assert passed is True

def test_multi_seed_aggregate_metrics():
    res = calculate_multi_seed_aggregate([0.90, 0.92, 0.94])
    assert res["mean"] == 0.92
    assert res["min"] == 0.90
    assert res["max"] == 0.94
metadata.yml (448 bytes)
lesson_id: D238
day: 238
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/section_project_reproducing_a_paper_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/section_project_reproducing_a_paper_lib.py (595 bytes)
import torch
import torch.nn as nn
from typing import Dict, Any, List

class ConfigurableTransformer(nn.Module):
    def __init__(self, vocab_size: int = 100, embed_dim: int = 32, num_classes: int = 2, use_pre_ln: bool = True):
        super().__init__()
        # TODO: Initialize configurable transformer architecture
        pass

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # TODO: Forward pass supporting Pre-LN vs Post-LN
        pass

def run_single_batch_overfit(model: nn.Module, steps: int = 50) -> bool:
    # TODO: Overfit single batch to near zero loss
    pass
starter/test_section_project_reproducing_a_paper_lib.py (932 bytes)
import pytest
import torch
from examples.section_project_reproducing_a_paper_lib import (
    ConfigurableTransformer,
    run_single_batch_overfit,
    calculate_multi_seed_aggregate
)

def test_configurable_transformer_forward():
    model_pre = ConfigurableTransformer(use_pre_ln=True)
    model_post = ConfigurableTransformer(use_pre_ln=False)
    x = torch.randint(0, 100, (4, 16))

    out_pre = model_pre(x)
    out_post = model_post(x)
    assert out_pre.shape == (4, 2)
    assert out_post.shape == (4, 2)

def test_single_batch_overfit():
    torch.manual_seed(42)
    model = ConfigurableTransformer(embed_dim=32, num_classes=2, use_pre_ln=True)
    passed = run_single_batch_overfit(model, steps=60)
    assert passed is True

def test_multi_seed_aggregate_metrics():
    res = calculate_multi_seed_aggregate([0.90, 0.92, 0.94])
    assert res["mean"] == 0.92
    assert res["min"] == 0.90
    assert res["max"] == 0.94
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 238 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_section_project_reproducing_a_paper_lib.py (932 bytes)
import pytest
import torch
from examples.section_project_reproducing_a_paper_lib import (
    ConfigurableTransformer,
    run_single_batch_overfit,
    calculate_multi_seed_aggregate
)

def test_configurable_transformer_forward():
    model_pre = ConfigurableTransformer(use_pre_ln=True)
    model_post = ConfigurableTransformer(use_pre_ln=False)
    x = torch.randint(0, 100, (4, 16))

    out_pre = model_pre(x)
    out_post = model_post(x)
    assert out_pre.shape == (4, 2)
    assert out_post.shape == (4, 2)

def test_single_batch_overfit():
    torch.manual_seed(42)
    model = ConfigurableTransformer(embed_dim=32, num_classes=2, use_pre_ln=True)
    passed = run_single_batch_overfit(model, steps=60)
    assert passed is True

def test_multi_seed_aggregate_metrics():
    res = calculate_multi_seed_aggregate([0.90, 0.92, 0.94])
    assert res["mean"] == 0.92
    assert res["min"] == 0.90
    assert res["max"] == 0.94

Troubleshooting

Troubleshooting: Day 238 - Section Project: Reproducing a Paper

Common Issues

  1. Single batch overfit divergence:
    • Cause: LayerNorm applied incorrectly across sequence dimension instead of embedding dimension.
    • Fix: Use nn.LayerNorm(embed_dim).

Security notes

Security & Privacy: Day 238 - Section Project: Reproducing a Paper

Security Guidance

  • All reproduction experimental harnesses execute locally in memory.