Sandeep Bazar 365 Days of AI
Labs

Deep Learning › Training Deep Networks › Day 208

Hands-on lab — Day 208: Dropout, Batch Norm, and Regularization

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/dropout_batch_norm_and_regularization_lib.py

Test

./tests/run_tests.sh

File tree

examples/dropout_batch_norm_and_regularization_lib.py
examples/test_dropout_batch_norm_and_regularization_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/dropout_batch_norm_and_regularization_lib.py
starter/test_dropout_batch_norm_and_regularization_lib.py
tests/run_tests.sh
tests/test_dropout_batch_norm_and_regularization_lib.py
troubleshooting.md

Lab README

Lab: Day 208 -- Dropout, Batch Norm, and Regularization

Lesson

Day number: 208 of 365. Course: Course05-SS01 (Deep Learning - Neural Networks). Topic: Dropout, Batch Normalization, and Regularization in PyTorch.

Purpose

Build and test custom implementations of Inverted Dropout and Batch Normalization from scratch in PyTorch. Verify stochastic masking and inverted scaling during training, deterministic identity passing during evaluation, running statistic accumulation, and complete model mode management.

Learning objectives

  • Implement Inverted Dropout with Bernoulli stochastic masking and 1/(1-p) scaling.
  • Implement CustomBatchNorm1d tracking mini-batch statistics and running statistics.
  • Verify deterministic model execution in model.eval() mode.
  • Integrate normalization and regularization layers into a deep MLP architecture.

Prerequisites

  • Day 207 (Learning Rate Schedules).
  • Python 3.11+ with PyTorch.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is free and open-source under the modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/dropout_batch_norm_and_regularization_lib.py: Student scaffold file.
  • examples/dropout_batch_norm_and_regularization_lib.py: Complete reference implementation.
  • tests/test_dropout_batch_norm_and_regularization_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/dropout_batch_norm_and_regularization_lib.py

What the commands do

  • Evaluates RegularizedMLP forward passes in training versus evaluation modes.
  • Verifies stochastic dropout activation during training.
  • Confirms bitwise deterministic consistency during evaluation.

Expected output

Regularization Demo: Training Stochastic = True, Eval Deterministic = True

Validation steps

  1. Verify that CustomDropout produces zero-masked activations during training.
  2. Confirm that CustomBatchNorm1d updates running_mean and running_var.
  3. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Outputs Differ in Eval Mode: Check that CustomDropout checks self.training before applying the mask.

Security notes

All calculations run in local system memory on CPU hardware.

Extension exercises

  1. Implement RMSNorm and benchmark its memory footprint against LayerNorm.
  2. Implement Monte Carlo Dropout uncertainty estimation over 50 test iterations.

Expected output

FIELDS.md

# Expected Output Fields: Day 208

- `Training Stochastic`: Boolean indicating stochastic forward behavior in train mode.
- `Eval Deterministic`: Boolean indicating deterministic forward behavior in eval mode.

examples-run.txt

Regularization Demo: Training Stochastic = True, Eval Deterministic = True

measured-values.txt

Training Stochastic: True
Eval Deterministic: True

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_dropout_batch_norm_and_regularization_lib.py::test_custom_dropout_training_vs_eval PASSED [ 33%]
tests/test_dropout_batch_norm_and_regularization_lib.py::test_custom_batchnorm_train_eval_statistics PASSED [ 66%]
tests/test_dropout_batch_norm_and_regularization_lib.py::test_regularized_mlp_forward_modes PASSED [100%]

============================== 3 passed in 0.24s ===============================

Source files

examples/dropout_batch_norm_and_regularization_lib.py (2801 bytes) 
import torch
import torch.nn as nn
from typing import Tuple, Dict, Any

class CustomDropout(nn.Module):
    def __init__(self, p: float = 0.5):
        super().__init__()
        self.p = p

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        if not self.training or self.p == 0.0:
            return x
        # Inverted dropout: sample Bernoulli mask and scale by 1 / (1 - p)
        mask = (torch.rand_like(x) > self.p).float()
        return (x * mask) / (1.0 - self.p)

class CustomBatchNorm1d(nn.Module):
    def __init__(self, num_features: int, eps: float = 1e-5, momentum: float = 0.1):
        super().__init__()
        self.num_features = num_features
        self.eps = eps
        self.momentum = momentum

        self.gamma = nn.Parameter(torch.ones(num_features))
        self.beta = nn.Parameter(torch.zeros(num_features))

        self.register_buffer('running_mean', torch.zeros(num_features))
        self.register_buffer('running_var', torch.ones(num_features))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        if self.training:
            mean = x.mean(dim=0)
            var = x.var(dim=0, unbiased=False)

            # Update running stats
            with torch.no_grad():
                self.running_mean = (1.0 - self.momentum) * self.running_mean + self.momentum * mean
                self.running_var = (1.0 - self.momentum) * self.running_var + self.momentum * var

            x_hat = (x - mean) / torch.sqrt(var + self.eps)
        else:
            x_hat = (x - self.running_mean) / torch.sqrt(self.running_var + self.eps)

        return self.gamma * x_hat + self.beta

class RegularizedMLP(nn.Module):
    def __init__(self, in_features: int = 784, hidden_dim: int = 128, num_classes: int = 10, p: float = 0.5):
        super().__init__()
        self.fc1 = nn.Linear(in_features, hidden_dim)
        self.bn1 = CustomBatchNorm1d(hidden_dim)
        self.relu = nn.ReLU()
        self.drop = CustomDropout(p=p)
        self.fc2 = nn.Linear(hidden_dim, num_classes)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        h = self.drop(self.relu(self.bn1(self.fc1(x))))
        return self.fc2(h)

def run_regularization_demo():
    torch.manual_seed(42)
    model = RegularizedMLP(in_features=32, hidden_dim=16, num_classes=2, p=0.5)
    x = torch.randn(10, 32)

    model.train()
    out1 = model(x)
    out2 = model(x)
    is_stochastic = not torch.equal(out1, out2)

    model.eval()
    with torch.no_grad():
        out3 = model(x)
        out4 = model(x)
    is_deterministic = torch.equal(out3, out4)

    print(f"Regularization Demo: Training Stochastic = {is_stochastic}, Eval Deterministic = {is_deterministic}")
    return is_stochastic, is_deterministic

if __name__ == "__main__":
    run_regularization_demo()
examples/test_dropout_batch_norm_and_regularization_lib.py (1598 bytes) 
import pytest
import torch
from examples.dropout_batch_norm_and_regularization_lib import CustomDropout, CustomBatchNorm1d, RegularizedMLP

def test_custom_dropout_training_vs_eval():
    drop = CustomDropout(p=0.5)
    x = torch.ones(100, 100)

    drop.train()
    out_train = drop(x)
    # Check that approx 50% are zero and surviving elements are 2.0
    zero_ratio = (out_train == 0.0).float().mean().item()
    assert 0.40 <= zero_ratio <= 0.60
    assert torch.isclose(out_train.mean(), torch.tensor(1.0), atol=0.1)

    drop.eval()
    out_eval = drop(x)
    assert torch.equal(out_eval, x)

def test_custom_batchnorm_train_eval_statistics():
    bn = CustomBatchNorm1d(num_features=4, momentum=0.5)
    x = torch.tensor([[1.0, 2.0, 3.0, 4.0],
                      [5.0, 6.0, 7.0, 8.0]])

    bn.train()
    out_train = bn(x)
    assert out_train.shape == (2, 4)
    # Output should have approximately zero mean along batch dim
    assert torch.allclose(out_train.mean(dim=0), torch.zeros(4), atol=1e-4)

    # Check that running stats were updated
    assert not torch.equal(bn.running_mean, torch.zeros(4))

    # In eval mode, running stats should be used
    bn.eval()
    out_eval = bn(x)
    assert out_eval.shape == (2, 4)

def test_regularized_mlp_forward_modes():
    model = RegularizedMLP(in_features=20, hidden_dim=10, num_classes=2, p=0.5)
    x = torch.randn(8, 20)

    model.train()
    y1 = model(x)
    y2 = model(x)
    assert not torch.equal(y1, y2)

    model.eval()
    with torch.no_grad():
        y3 = model(x)
        y4 = model(x)
    assert torch.equal(y3, y4)
metadata.yml (450 bytes) 
lesson_id: D208
day: 208
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/dropout_batch_norm_and_regularization_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (27 bytes) 
torch>=2.2.0
pytest>=8.0.0
starter/dropout_batch_norm_and_regularization_lib.py (675 bytes) 
import torch
import torch.nn as nn
from typing import Tuple, Dict, Any

class CustomDropout(nn.Module):
    def __init__(self, p: float = 0.5):
        super().__init__()
        self.p = p

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # TODO: Implement Inverted Dropout
        pass

class CustomBatchNorm1d(nn.Module):
    def __init__(self, num_features: int, eps: float = 1e-5, momentum: float = 0.1):
        super().__init__()
        # TODO: Initialize gamma, beta, running_mean, running_var
        pass

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # TODO: Implement BatchNorm forward pass for train and eval modes
        pass
starter/test_dropout_batch_norm_and_regularization_lib.py (1598 bytes) 
import pytest
import torch
from examples.dropout_batch_norm_and_regularization_lib import CustomDropout, CustomBatchNorm1d, RegularizedMLP

def test_custom_dropout_training_vs_eval():
    drop = CustomDropout(p=0.5)
    x = torch.ones(100, 100)

    drop.train()
    out_train = drop(x)
    # Check that approx 50% are zero and surviving elements are 2.0
    zero_ratio = (out_train == 0.0).float().mean().item()
    assert 0.40 <= zero_ratio <= 0.60
    assert torch.isclose(out_train.mean(), torch.tensor(1.0), atol=0.1)

    drop.eval()
    out_eval = drop(x)
    assert torch.equal(out_eval, x)

def test_custom_batchnorm_train_eval_statistics():
    bn = CustomBatchNorm1d(num_features=4, momentum=0.5)
    x = torch.tensor([[1.0, 2.0, 3.0, 4.0],
                      [5.0, 6.0, 7.0, 8.0]])

    bn.train()
    out_train = bn(x)
    assert out_train.shape == (2, 4)
    # Output should have approximately zero mean along batch dim
    assert torch.allclose(out_train.mean(dim=0), torch.zeros(4), atol=1e-4)

    # Check that running stats were updated
    assert not torch.equal(bn.running_mean, torch.zeros(4))

    # In eval mode, running stats should be used
    bn.eval()
    out_eval = bn(x)
    assert out_eval.shape == (2, 4)

def test_regularized_mlp_forward_modes():
    model = RegularizedMLP(in_features=20, hidden_dim=10, num_classes=2, p=0.5)
    x = torch.randn(8, 20)

    model.train()
    y1 = model(x)
    y2 = model(x)
    assert not torch.equal(y1, y2)

    model.eval()
    with torch.no_grad():
        y3 = model(x)
        y4 = model(x)
    assert torch.equal(y3, y4)
tests/run_tests.sh (227 bytes) 
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 208 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_dropout_batch_norm_and_regularization_lib.py (1598 bytes) 
import pytest
import torch
from examples.dropout_batch_norm_and_regularization_lib import CustomDropout, CustomBatchNorm1d, RegularizedMLP

def test_custom_dropout_training_vs_eval():
    drop = CustomDropout(p=0.5)
    x = torch.ones(100, 100)

    drop.train()
    out_train = drop(x)
    # Check that approx 50% are zero and surviving elements are 2.0
    zero_ratio = (out_train == 0.0).float().mean().item()
    assert 0.40 <= zero_ratio <= 0.60
    assert torch.isclose(out_train.mean(), torch.tensor(1.0), atol=0.1)

    drop.eval()
    out_eval = drop(x)
    assert torch.equal(out_eval, x)

def test_custom_batchnorm_train_eval_statistics():
    bn = CustomBatchNorm1d(num_features=4, momentum=0.5)
    x = torch.tensor([[1.0, 2.0, 3.0, 4.0],
                      [5.0, 6.0, 7.0, 8.0]])

    bn.train()
    out_train = bn(x)
    assert out_train.shape == (2, 4)
    # Output should have approximately zero mean along batch dim
    assert torch.allclose(out_train.mean(dim=0), torch.zeros(4), atol=1e-4)

    # Check that running stats were updated
    assert not torch.equal(bn.running_mean, torch.zeros(4))

    # In eval mode, running stats should be used
    bn.eval()
    out_eval = bn(x)
    assert out_eval.shape == (2, 4)

def test_regularized_mlp_forward_modes():
    model = RegularizedMLP(in_features=20, hidden_dim=10, num_classes=2, p=0.5)
    x = torch.randn(8, 20)

    model.train()
    y1 = model(x)
    y2 = model(x)
    assert not torch.equal(y1, y2)

    model.eval()
    with torch.no_grad():
        y3 = model(x)
        y4 = model(x)
    assert torch.equal(y3, y4)

Troubleshooting

Troubleshooting: Day 208 - Dropout, Batch Norm, and Regularization

Common Issues

  1. Validation accuracy drops drastically:
    • Cause: Forgetting to call model.eval() before validation loop.
    • Fix: Always wrap validation code in model.eval() and with torch.no_grad():.

Security notes

Security & Privacy: Day 208 - Dropout, Batch Norm, and Regularization

Security Guidance

  • All regularization calculations execute in local memory on CPU.