Deep Learning › Training Deep Networks › Day 210

Hands-on lab — Day 210: A Disciplined Training Project

← Back to the Day 210 lesson
Open the hands-on files on GitHub — clone or download them from the public labs repository
Local path in your clone: labs/sections/deep-learning/day-210-a-disciplined-training-project/

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/a_disciplined_training_project_lib.py

Test

./tests/run_tests.sh

File tree

examples/a_disciplined_training_project_lib.py
examples/test_a_disciplined_training_project_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/a_disciplined_training_project_lib.py
starter/test_a_disciplined_training_project_lib.py
tests/run_tests.sh
tests/test_a_disciplined_training_project_lib.py
troubleshooting.md

Lab README

Lab: Day 210 -- A Disciplined Training Project

Lesson

Day number: 210 of 365. Course: Course05-SS01 (Deep Learning - Neural Networks). Topic: A Disciplined Training Project in PyTorch.

Purpose

Build and test a complete industrial-grade PyTorch training harness. Implement strongly typed dataclass configurations, deterministic multi-library seeding, atomic state checkpointing, validation metric evaluation, and an early stopping controller with automated best-checkpoint restoration.

Learning objectives

Implement the PyTorchTrainer harness orchestrating training and validation loops.
Implement seed_everything guaranteeing multi-library determinism.
Implement an EarlyStopping class tracking patience and preserving the best state_dict.
Structure hyperparameter declarations using TrainingConfig dataclasses.

Prerequisites

Day 209 (Debugging Training Runs).
Python 3.11+ with PyTorch.

Supported operating systems

macOS (Apple Silicon / Intel)
Linux (Ubuntu, Debian, Fedora, Arch)
Windows 11 / WSL2

Hardware requirements

1+ CPU cores.
1 GB RAM.
100 MB disk space.

Required software

Python 3.11 or newer.
pip package manager.
virtualenv or venv module.

Free and open-source options

PyTorch is free and open-source under the modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

starter/a_disciplined_training_project_lib.py: Student scaffold file.
examples/a_disciplined_training_project_lib.py: Complete reference implementation.
tests/test_a_disciplined_training_project_lib.py: Pytest automated validation suite.
expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/a_disciplined_training_project_lib.py

What the commands do

Configures and runs a modular training harness.
Demonstrates early stopping patience tracking and checkpoint capture.
Verifies deterministic reproducibility.

Expected output

Trainer Demo: Stopped Early = True, Best Val Loss = 0.6931

Validation steps

Verify that seed_everything produces identical random tensor outputs across runs.
Confirm that EarlyStopping triggers when patience runs out and captures best state dict.
Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

Early Stopping Never Triggers: Ensure validation loss decreases/stalls as expected and patience is non-zero.

Security notes

All training operations execute locally in memory on CPU hardware.

Extension exercises

Add JSON metric telemetry logging writing history.json after training completion.
Implement Learning Rate Warmup and Cosine Decay integration into the PyTorchTrainer.

Lesson title: A Disciplined Training Project
Day number: 210 of 365
Lesson article: https://ai-roadmap-365.github.io/day-210-a-disciplined-training-project
Lab files: everything you need is in this directory — follow “How to run” below.
Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-210-a-disciplined-training-project when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 210

- `Stopped Early`: Boolean indicating whether early stopping triggered.
- `Best Val Loss`: Lowest validation loss captured during training.

examples-run.txt

Trainer Demo: Stopped Early = True, Best Val Loss = 0.6931

measured-values.txt

Stopped Early: True
Best Val Loss: 0.6931

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_a_disciplined_training_project_lib.py::test_seed_everything_deterministic PASSED [ 33%]
tests/test_a_disciplined_training_project_lib.py::test_early_stopping_patience_and_restore PASSED [ 66%]
tests/test_a_disciplined_training_project_lib.py::test_pytorch_trainer_fit_and_metrics PASSED [100%]

============================== 3 passed in 0.28s ===============================

Source files

examples/a_disciplined_training_project_lib.py (4893 bytes)

import os
import random
import numpy as np
import torch
import torch.nn as nn
from dataclasses import dataclass, asdict
from torch.utils.data import DataLoader
from typing import Dict, Any, Tuple, Optional

@dataclass
class TrainingConfig:
    in_features: int = 16
    hidden_dim: int = 32
    num_classes: int = 2
    learning_rate: float = 0.01
    batch_size: int = 16
    max_epochs: int = 20
    patience: int = 3
    min_delta: float = 1e-3
    seed: int = 42

def seed_everything(seed: int = 42):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    if torch.cuda.is_available():
        torch.cuda.manual_seed_all(seed)

class EarlyStopping:
    def __init__(self, patience: int = 3, min_delta: float = 1e-3):
        self.patience = patience
        self.min_delta = min_delta
        self.counter = 0
        self.best_loss = float("inf")
        self.early_stop = False
        self.best_state_dict = None

    def __call__(self, val_loss: float, model: nn.Module) -> bool:
        if val_loss < self.best_loss - self.min_delta:
            self.best_loss = val_loss
            self.best_state_dict = {k: v.cpu().clone() for k, v in model.state_dict().items()}
            self.counter = 0
            return True
        else:
            self.counter += 1
            if self.counter >= self.patience:
                self.early_stop = True
            return False

class PyTorchTrainer:
    def __init__(self, model: nn.Module, optimizer: torch.optim.Optimizer,
                 criterion: nn.Module, config: TrainingConfig):
        self.model = model
        self.optimizer = optimizer
        self.criterion = criterion
        self.config = config
        self.early_stopping = EarlyStopping(patience=config.patience, min_delta=config.min_delta)
        self.history = {"train_loss": [], "val_loss": [], "val_acc": []}

    def train_epoch(self, train_loader: DataLoader) -> float:
        self.model.train()
        total_loss = 0.0
        total_samples = 0
        for x, y in train_loader:
            self.optimizer.zero_grad()
            logits = self.model(x)
            loss = self.criterion(logits, y)
            loss.backward()
            self.optimizer.step()

            bs = x.size(0)
            total_loss += loss.item() * bs
            total_samples += bs
        return total_loss / max(total_samples, 1)

    def evaluate(self, val_loader: DataLoader) -> Tuple[float, float]:
        self.model.eval()
        total_loss = 0.0
        correct = 0
        total_samples = 0
        with torch.no_grad():
            for x, y in val_loader:
                logits = self.model(x)
                loss = self.criterion(logits, y)
                bs = x.size(0)
                total_loss += loss.item() * bs
                preds = torch.argmax(logits, dim=1)
                correct += int((preds == y).sum().item())
                total_samples += bs
        val_loss = total_loss / max(total_samples, 1)
        val_acc = correct / max(total_samples, 1)
        return val_loss, val_acc

    def fit(self, train_loader: DataLoader, val_loader: DataLoader) -> Dict[str, Any]:
        for epoch in range(1, self.config.max_epochs + 1):
            t_loss = self.train_epoch(train_loader)
            v_loss, v_acc = self.evaluate(val_loader)

            self.history["train_loss"].append(t_loss)
            self.history["val_loss"].append(v_loss)
            self.history["val_acc"].append(v_acc)

            self.early_stopping(v_loss, self.model)
            if self.early_stopping.early_stop:
                if self.early_stopping.best_state_dict is not None:
                    self.model.load_state_dict(self.early_stopping.best_state_dict)
                break

        return self.history

def run_trainer_demo():
    seed_everything(42)
    config = TrainingConfig(in_features=8, hidden_dim=16, num_classes=2, max_epochs=10, patience=2)
    model = nn.Sequential(
        nn.Linear(config.in_features, config.hidden_dim),
        nn.ReLU(),
        nn.Linear(config.hidden_dim, config.num_classes)
    )
    optimizer = torch.optim.AdamW(model.parameters(), lr=config.learning_rate)
    criterion = nn.CrossEntropyLoss()

    trainer = PyTorchTrainer(model, optimizer, criterion, config)

    # Synthetic data loaders
    x_t, y_t = torch.randn(64, 8), torch.randint(0, 2, (64,))
    x_v, y_v = torch.randn(32, 8), torch.randint(0, 2, (32,))
    train_loader = DataLoader(list(zip(x_t, y_t)), batch_size=16)
    val_loader = DataLoader(list(zip(x_v, y_v)), batch_size=16)

    history = trainer.fit(train_loader, val_loader)
    stopped_early = trainer.early_stopping.early_stop
    best_loss = trainer.early_stopping.best_loss

    print(f"Trainer Demo: Stopped Early = {stopped_early}, Best Val Loss = {best_loss:.4f}")
    return stopped_early, best_loss

if __name__ == "__main__":
    run_trainer_demo()

examples/test_a_disciplined_training_project_lib.py (1781 bytes)

import pytest
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from examples.a_disciplined_training_project_lib import (
    TrainingConfig, seed_everything, EarlyStopping, PyTorchTrainer
)

def test_seed_everything_deterministic():
    seed_everything(42)
    t1 = torch.randn(10)
    seed_everything(42)
    t2 = torch.randn(10)
    assert torch.equal(t1, t2)

def test_early_stopping_patience_and_restore():
    model = nn.Linear(4, 2)
    stopper = EarlyStopping(patience=2, min_delta=1e-3)

    assert stopper(1.0, model) == True # Best: 1.0
    assert stopper.counter == 0

    assert stopper(0.95, model) == True # Best: 0.95
    assert stopper.counter == 0

    assert stopper(0.96, model) == False # Worse (count 1)
    assert stopper.counter == 1
    assert stopper.early_stop == False

    assert stopper(0.97, model) == False # Worse (count 2 -> early stop!)
    assert stopper.counter == 2
    assert stopper.early_stop == True

def test_pytorch_trainer_fit_and_metrics():
    seed_everything(42)
    config = TrainingConfig(in_features=4, hidden_dim=8, num_classes=2, max_epochs=5, patience=3)
    model = nn.Sequential(nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 2))
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
    criterion = nn.CrossEntropyLoss()

    trainer = PyTorchTrainer(model, optimizer, criterion, config)

    x = torch.randn(32, 4)
    y = torch.randint(0, 2, (32,))
    train_loader = DataLoader(list(zip(x, y)), batch_size=8)
    val_loader = DataLoader(list(zip(x, y)), batch_size=8)

    history = trainer.fit(train_loader, val_loader)
    assert len(history["train_loss"]) > 0
    assert len(history["val_loss"]) == len(history["train_loss"])
    assert len(history["val_acc"]) == len(history["train_loss"])

metadata.yml (443 bytes)

lesson_id: D210
day: 210
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/a_disciplined_training_project_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'

requirements/requirements.txt (27 bytes)

torch>=2.2.0
pytest>=8.0.0

starter/a_disciplined_training_project_lib.py (1023 bytes)

import os
import random
import numpy as np
import torch
import torch.nn as nn
from dataclasses import dataclass, asdict
from torch.utils.data import DataLoader
from typing import Dict, Any, Tuple, Optional

@dataclass
class TrainingConfig:
    in_features: int = 16
    hidden_dim: int = 32
    num_classes: int = 2
    learning_rate: float = 0.01
    batch_size: int = 16
    max_epochs: int = 20
    patience: int = 3
    min_delta: float = 1e-3
    seed: int = 42

def seed_everything(seed: int = 42):
    # TODO: Implement deterministic multi-library seeding
    pass

class EarlyStopping:
    def __init__(self, patience: int = 3, min_delta: float = 1e-3):
        # TODO: Implement early stopping controller
        pass

    def __call__(self, val_loss: float, model: nn.Module) -> bool:
        pass

class PyTorchTrainer:
    def __init__(self, model: nn.Module, optimizer: torch.optim.Optimizer,
                 criterion: nn.Module, config: TrainingConfig):
        # TODO: Implement Trainer class
        pass

starter/test_a_disciplined_training_project_lib.py (1781 bytes)

import pytest
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from examples.a_disciplined_training_project_lib import (
    TrainingConfig, seed_everything, EarlyStopping, PyTorchTrainer
)

def test_seed_everything_deterministic():
    seed_everything(42)
    t1 = torch.randn(10)
    seed_everything(42)
    t2 = torch.randn(10)
    assert torch.equal(t1, t2)

def test_early_stopping_patience_and_restore():
    model = nn.Linear(4, 2)
    stopper = EarlyStopping(patience=2, min_delta=1e-3)

    assert stopper(1.0, model) == True # Best: 1.0
    assert stopper.counter == 0

    assert stopper(0.95, model) == True # Best: 0.95
    assert stopper.counter == 0

    assert stopper(0.96, model) == False # Worse (count 1)
    assert stopper.counter == 1
    assert stopper.early_stop == False

    assert stopper(0.97, model) == False # Worse (count 2 -> early stop!)
    assert stopper.counter == 2
    assert stopper.early_stop == True

def test_pytorch_trainer_fit_and_metrics():
    seed_everything(42)
    config = TrainingConfig(in_features=4, hidden_dim=8, num_classes=2, max_epochs=5, patience=3)
    model = nn.Sequential(nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 2))
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
    criterion = nn.CrossEntropyLoss()

    trainer = PyTorchTrainer(model, optimizer, criterion, config)

    x = torch.randn(32, 4)
    y = torch.randint(0, 2, (32,))
    train_loader = DataLoader(list(zip(x, y)), batch_size=8)
    val_loader = DataLoader(list(zip(x, y)), batch_size=8)

    history = trainer.fit(train_loader, val_loader)
    assert len(history["train_loss"]) > 0
    assert len(history["val_loss"]) == len(history["train_loss"])
    assert len(history["val_acc"]) == len(history["train_loss"])

tests/run_tests.sh (227 bytes)

#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 210 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."

tests/test_a_disciplined_training_project_lib.py (1781 bytes)

import pytest
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from examples.a_disciplined_training_project_lib import (
    TrainingConfig, seed_everything, EarlyStopping, PyTorchTrainer
)

def test_seed_everything_deterministic():
    seed_everything(42)
    t1 = torch.randn(10)
    seed_everything(42)
    t2 = torch.randn(10)
    assert torch.equal(t1, t2)

def test_early_stopping_patience_and_restore():
    model = nn.Linear(4, 2)
    stopper = EarlyStopping(patience=2, min_delta=1e-3)

    assert stopper(1.0, model) == True # Best: 1.0
    assert stopper.counter == 0

    assert stopper(0.95, model) == True # Best: 0.95
    assert stopper.counter == 0

    assert stopper(0.96, model) == False # Worse (count 1)
    assert stopper.counter == 1
    assert stopper.early_stop == False

    assert stopper(0.97, model) == False # Worse (count 2 -> early stop!)
    assert stopper.counter == 2
    assert stopper.early_stop == True

def test_pytorch_trainer_fit_and_metrics():
    seed_everything(42)
    config = TrainingConfig(in_features=4, hidden_dim=8, num_classes=2, max_epochs=5, patience=3)
    model = nn.Sequential(nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 2))
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
    criterion = nn.CrossEntropyLoss()

    trainer = PyTorchTrainer(model, optimizer, criterion, config)

    x = torch.randn(32, 4)
    y = torch.randint(0, 2, (32,))
    train_loader = DataLoader(list(zip(x, y)), batch_size=8)
    val_loader = DataLoader(list(zip(x, y)), batch_size=8)

    history = trainer.fit(train_loader, val_loader)
    assert len(history["train_loss"]) > 0
    assert len(history["val_loss"]) == len(history["train_loss"])
    assert len(history["val_acc"]) == len(history["train_loss"])

Troubleshooting

Troubleshooting: Day 210 - A Disciplined Training Project

Common Issues

Checkpoint corrupted on power loss:
- Cause: Writing directly to target file.
- Fix: Write to a .tmp file first and use os.replace for atomic updates.

Security notes

Security & Privacy: Day 210 - A Disciplined Training Project

Security Guidance

Always load untrusted PyTorch checkpoints using torch.load(path, weights_only=True).

Hands-on lab — Day 210: A Disciplined Training Project

Commands

Setup

Run

Test

File tree

Lab README

Lab: Day 210 -- A Disciplined Training Project

Lesson

Purpose

Learning objectives

Prerequisites

Supported operating systems

Hardware requirements

Required software

Free and open-source options

Installation

File structure

How to run

What the commands do

Expected output

Validation steps

Tests

Cleanup

Troubleshooting

Security notes

Extension exercises

Navigation

Expected output

FIELDS.md

examples-run.txt

measured-values.txt

starter-run.txt

test-run.txt

Source files

Troubleshooting

Troubleshooting: Day 210 - A Disciplined Training Project

Common Issues

Security notes

Security & Privacy: Day 210 - A Disciplined Training Project

Security Guidance