Deep Learning › Training at Scale › Day 235

Hands-on lab — Day 235: Experiment Tracking

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/experiment_tracking_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/experiment_tracking_lib.py
examples/test_experiment_tracking_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/experiment_tracking_lib.py
starter/test_experiment_tracking_lib.py
test_checkpoints/model_epoch_2_loss_0.250.json
test_checkpoints/model_epoch_3_loss_0.200.json
test_runs/demo-run-01.jsonl
tests/run_tests.sh
tests/test_experiment_tracking_lib.py
troubleshooting.md

Lab README

Lab: Day 235 -- Experiment Tracking

Lesson

Day number: 235 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Experiment Tracking, MLOps, and Reproducible Checkpointing.

Purpose

Build and test a modular experiment tracking and checkpoint versioning library in Python. Implement a crash-safe JSONL experiment logger and construct a Best-$K$ checkpoint retention manager.

Learning objectives

  • Build a structured, atomic JSONL experiment logging engine.
  • Implement the 5 pillars of deep learning reproducibility.
  • Construct a Best-$K$ model checkpoint retention and pruning manager.
  • Verify deterministic tracking of hyperparameter trials and evaluation metrics.

Prerequisites

  • Day 234 (Distributed Training Concepts).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

Python and standard JSON/OS libraries are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/experiment_tracking_lib.py: Student scaffold file.
  • examples/experiment_tracking_lib.py: Complete reference implementation.
  • tests/test_experiment_tracking_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/experiment_tracking_lib.py

What the commands do

  • Logs initialization, step, and evaluation events to JSONL.
  • Manages model checkpoint saving and pruning.
  • Runs unit test assertions.

Expected output

Experiment Demo: Logged to ./test_runs/demo-run-01.jsonl, Active Checkpoints = 2

Validation steps

  1. Verify ExperimentLogger creates valid JSON Lines files.
  2. Confirm CheckpointManager preserves exactly the top $K$ lowest validation loss checkpoints.
  3. Confirm older, worse checkpoints are pruned from disk.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

rm -rf test_runs test_checkpoints
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • File lock errors: Ensure file handlers are flushed and closed properly.

Security notes

Do not log API keys or secrets in experiment configs.

Extension exercises

  1. Build an SQLite experiment storage engine.
  2. Integrate Weights & Biases (W&B) into a PyTorch training loop.
  • Lesson title: Experiment Tracking
  • Day number: 235 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-235-experiment-tracking
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-235-experiment-tracking when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 235

- `Logged to`: Path to the generated JSONL experiment telemetry log file.
- `Active Checkpoints`: Number of retained best-performing model checkpoints.

examples-run.txt

Experiment Demo: Logged to ./test_runs/demo-run-01.jsonl, Active Checkpoints = 2

measured-values.txt

Logged to: ./test_runs/demo-run-01.jsonl
Active Checkpoints: 2

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 2 items

tests/test_experiment_tracking_lib.py::test_experiment_logger PASSED     [ 50%]
tests/test_experiment_tracking_lib.py::test_checkpoint_manager_retention PASSED [100%]

============================== 2 passed in 0.42s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/experiment_tracking_lib.py (3033 bytes)
import json
import os
from typing import Dict, Any, List, Optional

class ExperimentLogger:
    def __init__(self, run_id: str, log_dir: str = "./runs", config: Optional[Dict[str, Any]] = None):
        self.run_id = run_id
        self.log_dir = log_dir
        os.makedirs(self.log_dir, exist_ok=True)
        self.log_path = os.path.join(self.log_dir, f"{run_id}.jsonl")
        
        init_event = {
            "event": "init",
            "run_id": run_id,
            "config": config or {}
        }
        self._append(init_event)

    def _append(self, record: Dict[str, Any]):
        with open(self.log_path, "a", encoding="utf-8") as f:
            f.write(json.dumps(record) + "\n")

    def log_step(self, step: int, epoch: int, metrics: Dict[str, float]):
        record = {
            "event": "step",
            "step": step,
            "epoch": epoch,
            **metrics
        }
        self._append(record)

    def log_eval(self, epoch: int, metrics: Dict[str, float]):
        record = {
            "event": "eval",
            "epoch": epoch,
            **metrics
        }
        self._append(record)

class CheckpointManager:
    def __init__(self, save_dir: str = "./checkpoints", max_to_keep: int = 2):
        self.save_dir = save_dir
        self.max_to_keep = max_to_keep
        self.checkpoints: List[Dict[str, Any]] = []
        os.makedirs(self.save_dir, exist_ok=True)

    def save_checkpoint(self, epoch: int, val_loss: float, model_state: Dict[str, Any]) -> List[str]:
        ckpt_filename = f"model_epoch_{epoch}_loss_{val_loss:.3f}.json"
        ckpt_path = os.path.join(self.save_dir, ckpt_filename)
        
        with open(ckpt_path, "w", encoding="utf-8") as f:
            json.dump({"epoch": epoch, "val_loss": val_loss, "state": model_state}, f)

        self.checkpoints.append({"epoch": epoch, "val_loss": val_loss, "path": ckpt_path})
        # Sort by validation loss ascending (lower is better)
        self.checkpoints.sort(key=lambda x: x["val_loss"])

        # Prune if exceeding max_to_keep
        while len(self.checkpoints) > self.max_to_keep:
            to_remove = self.checkpoints.pop(-1)
            if os.path.exists(to_remove["path"]):
                os.remove(to_remove["path"])

        return [c["path"] for c in self.checkpoints]

def run_experiment_tracking_demo():
    logger = ExperimentLogger("demo-run-01", log_dir="./test_runs", config={"lr": 3e-4, "batch_size": 32})
    logger.log_step(1, 1, {"loss": 0.45})
    logger.log_eval(1, {"val_loss": 0.38, "val_acc": 0.88})

    manager = CheckpointManager(save_dir="./test_checkpoints", max_to_keep=2)
    manager.save_checkpoint(1, 0.38, {"w": [1.0, 2.0]})
    manager.save_checkpoint(2, 0.25, {"w": [1.1, 2.1]})
    active_ckpts = manager.save_checkpoint(3, 0.20, {"w": [1.2, 2.2]})

    print(f"Experiment Demo: Logged to {logger.log_path}, Active Checkpoints = {len(active_ckpts)}")
    return logger.log_path, active_ckpts

if __name__ == "__main__":
    run_experiment_tracking_demo()
examples/test_experiment_tracking_lib.py (1136 bytes)
import pytest
import os
import json
from examples.experiment_tracking_lib import ExperimentLogger, CheckpointManager

def test_experiment_logger(tmp_path):
    log_dir = str(tmp_path / "runs")
    logger = ExperimentLogger("test_run", log_dir=log_dir, config={"lr": 1e-4})
    logger.log_step(1, 1, {"loss": 0.5})
    logger.log_eval(1, {"val_acc": 0.9})

    assert os.path.exists(logger.log_path)
    with open(logger.log_path, "r", encoding="utf-8") as f:
        lines = [json.loads(line) for line in f]
    
    assert len(lines) == 3
    assert lines[0]["event"] == "init"
    assert lines[1]["loss"] == 0.5
    assert lines[2]["val_acc"] == 0.9

def test_checkpoint_manager_retention(tmp_path):
    save_dir = str(tmp_path / "ckpts")
    mgr = CheckpointManager(save_dir=save_dir, max_to_keep=2)
    
    mgr.save_checkpoint(1, 0.50, {"step": 1})
    mgr.save_checkpoint(2, 0.30, {"step": 2})
    active = mgr.save_checkpoint(3, 0.20, {"step": 3})

    assert len(active) == 2
    # Best 2: loss 0.20 (epoch 3) and loss 0.30 (epoch 2)
    assert any("epoch_3" in p for p in active)
    assert any("epoch_2" in p for p in active)
metadata.yml (472 bytes)
lesson_id: D235
day: 235
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/experiment_tracking_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'rm -rf test_runs test_checkpoints'
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/experiment_tracking_lib.py (778 bytes)
import json
import os
from typing import Dict, Any, List, Optional

class ExperimentLogger:
    def __init__(self, run_id: str, log_dir: str = "./runs", config: Optional[Dict[str, Any]] = None):
        # TODO: Initialize experiment logger and log init event to JSONL
        pass

    def log_step(self, step: int, epoch: int, metrics: Dict[str, float]):
        # TODO: Append step metrics to JSONL file
        pass

class CheckpointManager:
    def __init__(self, save_dir: str = "./checkpoints", max_to_keep: int = 2):
        # TODO: Initialize checkpoint manager
        pass

    def save_checkpoint(self, epoch: int, val_loss: float, model_state: Dict[str, Any]) -> List[str]:
        # TODO: Save checkpoint and prune older checkpoints beyond max_to_keep
        pass
starter/test_experiment_tracking_lib.py (1136 bytes)
import pytest
import os
import json
from examples.experiment_tracking_lib import ExperimentLogger, CheckpointManager

def test_experiment_logger(tmp_path):
    log_dir = str(tmp_path / "runs")
    logger = ExperimentLogger("test_run", log_dir=log_dir, config={"lr": 1e-4})
    logger.log_step(1, 1, {"loss": 0.5})
    logger.log_eval(1, {"val_acc": 0.9})

    assert os.path.exists(logger.log_path)
    with open(logger.log_path, "r", encoding="utf-8") as f:
        lines = [json.loads(line) for line in f]
    
    assert len(lines) == 3
    assert lines[0]["event"] == "init"
    assert lines[1]["loss"] == 0.5
    assert lines[2]["val_acc"] == 0.9

def test_checkpoint_manager_retention(tmp_path):
    save_dir = str(tmp_path / "ckpts")
    mgr = CheckpointManager(save_dir=save_dir, max_to_keep=2)
    
    mgr.save_checkpoint(1, 0.50, {"step": 1})
    mgr.save_checkpoint(2, 0.30, {"step": 2})
    active = mgr.save_checkpoint(3, 0.20, {"step": 3})

    assert len(active) == 2
    # Best 2: loss 0.20 (epoch 3) and loss 0.30 (epoch 2)
    assert any("epoch_3" in p for p in active)
    assert any("epoch_2" in p for p in active)
test_checkpoints/model_epoch_2_loss_0.250.json (58 bytes)
{"epoch": 2, "val_loss": 0.25, "state": {"w": [1.1, 2.1]}}
test_checkpoints/model_epoch_3_loss_0.200.json (57 bytes)
{"epoch": 3, "val_loss": 0.2, "state": {"w": [1.2, 2.2]}}
test_runs/demo-run-01.jsonl (207 bytes)
{"event": "init", "run_id": "demo-run-01", "config": {"lr": 0.0003, "batch_size": 32}}
{"event": "step", "step": 1, "epoch": 1, "loss": 0.45}
{"event": "eval", "epoch": 1, "val_loss": 0.38, "val_acc": 0.88}
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 235 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_experiment_tracking_lib.py (1136 bytes)
import pytest
import os
import json
from examples.experiment_tracking_lib import ExperimentLogger, CheckpointManager

def test_experiment_logger(tmp_path):
    log_dir = str(tmp_path / "runs")
    logger = ExperimentLogger("test_run", log_dir=log_dir, config={"lr": 1e-4})
    logger.log_step(1, 1, {"loss": 0.5})
    logger.log_eval(1, {"val_acc": 0.9})

    assert os.path.exists(logger.log_path)
    with open(logger.log_path, "r", encoding="utf-8") as f:
        lines = [json.loads(line) for line in f]
    
    assert len(lines) == 3
    assert lines[0]["event"] == "init"
    assert lines[1]["loss"] == 0.5
    assert lines[2]["val_acc"] == 0.9

def test_checkpoint_manager_retention(tmp_path):
    save_dir = str(tmp_path / "ckpts")
    mgr = CheckpointManager(save_dir=save_dir, max_to_keep=2)
    
    mgr.save_checkpoint(1, 0.50, {"step": 1})
    mgr.save_checkpoint(2, 0.30, {"step": 2})
    active = mgr.save_checkpoint(3, 0.20, {"step": 3})

    assert len(active) == 2
    # Best 2: loss 0.20 (epoch 3) and loss 0.30 (epoch 2)
    assert any("epoch_3" in p for p in active)
    assert any("epoch_2" in p for p in active)

Troubleshooting

Troubleshooting: Day 235 - Experiment Tracking

Common Issues

  1. Corrupt JSONL records:
    • Cause: Writing un-serialized objects to file.
    • Fix: Use json.dumps() on primitive Python dictionaries.

Security notes

Security & Privacy: Day 235 - Experiment Tracking

Security Guidance

  • Mask all sensitive credentials before logging config dictionaries.