Deep Learning › Training at Scale › Day 235
Hands-on lab — Day 235: Experiment Tracking
- ← Back to the Day 235 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deep-learning/day-235-experiment-tracking/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/experiment_tracking_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/experiment_tracking_lib.py examples/test_experiment_tracking_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/experiment_tracking_lib.py starter/test_experiment_tracking_lib.py test_checkpoints/model_epoch_2_loss_0.250.json test_checkpoints/model_epoch_3_loss_0.200.json test_runs/demo-run-01.jsonl tests/run_tests.sh tests/test_experiment_tracking_lib.py troubleshooting.md
Lab README
Lab: Day 235 -- Experiment Tracking
Lesson
Day number: 235 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Experiment Tracking, MLOps, and Reproducible Checkpointing.
Purpose
Build and test a modular experiment tracking and checkpoint versioning library in Python. Implement a crash-safe JSONL experiment logger and construct a Best-$K$ checkpoint retention manager.
Learning objectives
- Build a structured, atomic JSONL experiment logging engine.
- Implement the 5 pillars of deep learning reproducibility.
- Construct a Best-$K$ model checkpoint retention and pruning manager.
- Verify deterministic tracking of hyperparameter trials and evaluation metrics.
Prerequisites
- Day 234 (Distributed Training Concepts).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
Python and standard JSON/OS libraries are open source.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/experiment_tracking_lib.py: Student scaffold file.examples/experiment_tracking_lib.py: Complete reference implementation.tests/test_experiment_tracking_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/experiment_tracking_lib.py
What the commands do
- Logs initialization, step, and evaluation events to JSONL.
- Manages model checkpoint saving and pruning.
- Runs unit test assertions.
Expected output
Experiment Demo: Logged to ./test_runs/demo-run-01.jsonl, Active Checkpoints = 2
Validation steps
- Verify
ExperimentLoggercreates valid JSON Lines files. - Confirm
CheckpointManagerpreserves exactly the top $K$ lowest validation loss checkpoints. - Confirm older, worse checkpoints are pruned from disk.
- Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
rm -rf test_runs test_checkpoints
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- File lock errors: Ensure file handlers are flushed and closed properly.
Security notes
Do not log API keys or secrets in experiment configs.
Extension exercises
- Build an SQLite experiment storage engine.
- Integrate Weights & Biases (W&B) into a PyTorch training loop.
Navigation
- Lesson title: Experiment Tracking
- Day number: 235 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-235-experiment-tracking
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-235-experiment-trackingwhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 235
- `Logged to`: Path to the generated JSONL experiment telemetry log file.
- `Active Checkpoints`: Number of retained best-performing model checkpoints.
examples-run.txt
Experiment Demo: Logged to ./test_runs/demo-run-01.jsonl, Active Checkpoints = 2
measured-values.txt
Logged to: ./test_runs/demo-run-01.jsonl
Active Checkpoints: 2
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 2 items
tests/test_experiment_tracking_lib.py::test_experiment_logger PASSED [ 50%]
tests/test_experiment_tracking_lib.py::test_checkpoint_manager_retention PASSED [100%]
============================== 2 passed in 0.42s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/experiment_tracking_lib.py (3033 bytes)
import json
import os
from typing import Dict, Any, List, Optional
class ExperimentLogger:
def __init__(self, run_id: str, log_dir: str = "./runs", config: Optional[Dict[str, Any]] = None):
self.run_id = run_id
self.log_dir = log_dir
os.makedirs(self.log_dir, exist_ok=True)
self.log_path = os.path.join(self.log_dir, f"{run_id}.jsonl")
init_event = {
"event": "init",
"run_id": run_id,
"config": config or {}
}
self._append(init_event)
def _append(self, record: Dict[str, Any]):
with open(self.log_path, "a", encoding="utf-8") as f:
f.write(json.dumps(record) + "\n")
def log_step(self, step: int, epoch: int, metrics: Dict[str, float]):
record = {
"event": "step",
"step": step,
"epoch": epoch,
**metrics
}
self._append(record)
def log_eval(self, epoch: int, metrics: Dict[str, float]):
record = {
"event": "eval",
"epoch": epoch,
**metrics
}
self._append(record)
class CheckpointManager:
def __init__(self, save_dir: str = "./checkpoints", max_to_keep: int = 2):
self.save_dir = save_dir
self.max_to_keep = max_to_keep
self.checkpoints: List[Dict[str, Any]] = []
os.makedirs(self.save_dir, exist_ok=True)
def save_checkpoint(self, epoch: int, val_loss: float, model_state: Dict[str, Any]) -> List[str]:
ckpt_filename = f"model_epoch_{epoch}_loss_{val_loss:.3f}.json"
ckpt_path = os.path.join(self.save_dir, ckpt_filename)
with open(ckpt_path, "w", encoding="utf-8") as f:
json.dump({"epoch": epoch, "val_loss": val_loss, "state": model_state}, f)
self.checkpoints.append({"epoch": epoch, "val_loss": val_loss, "path": ckpt_path})
# Sort by validation loss ascending (lower is better)
self.checkpoints.sort(key=lambda x: x["val_loss"])
# Prune if exceeding max_to_keep
while len(self.checkpoints) > self.max_to_keep:
to_remove = self.checkpoints.pop(-1)
if os.path.exists(to_remove["path"]):
os.remove(to_remove["path"])
return [c["path"] for c in self.checkpoints]
def run_experiment_tracking_demo():
logger = ExperimentLogger("demo-run-01", log_dir="./test_runs", config={"lr": 3e-4, "batch_size": 32})
logger.log_step(1, 1, {"loss": 0.45})
logger.log_eval(1, {"val_loss": 0.38, "val_acc": 0.88})
manager = CheckpointManager(save_dir="./test_checkpoints", max_to_keep=2)
manager.save_checkpoint(1, 0.38, {"w": [1.0, 2.0]})
manager.save_checkpoint(2, 0.25, {"w": [1.1, 2.1]})
active_ckpts = manager.save_checkpoint(3, 0.20, {"w": [1.2, 2.2]})
print(f"Experiment Demo: Logged to {logger.log_path}, Active Checkpoints = {len(active_ckpts)}")
return logger.log_path, active_ckpts
if __name__ == "__main__":
run_experiment_tracking_demo()
examples/test_experiment_tracking_lib.py (1136 bytes)
import pytest
import os
import json
from examples.experiment_tracking_lib import ExperimentLogger, CheckpointManager
def test_experiment_logger(tmp_path):
log_dir = str(tmp_path / "runs")
logger = ExperimentLogger("test_run", log_dir=log_dir, config={"lr": 1e-4})
logger.log_step(1, 1, {"loss": 0.5})
logger.log_eval(1, {"val_acc": 0.9})
assert os.path.exists(logger.log_path)
with open(logger.log_path, "r", encoding="utf-8") as f:
lines = [json.loads(line) for line in f]
assert len(lines) == 3
assert lines[0]["event"] == "init"
assert lines[1]["loss"] == 0.5
assert lines[2]["val_acc"] == 0.9
def test_checkpoint_manager_retention(tmp_path):
save_dir = str(tmp_path / "ckpts")
mgr = CheckpointManager(save_dir=save_dir, max_to_keep=2)
mgr.save_checkpoint(1, 0.50, {"step": 1})
mgr.save_checkpoint(2, 0.30, {"step": 2})
active = mgr.save_checkpoint(3, 0.20, {"step": 3})
assert len(active) == 2
# Best 2: loss 0.20 (epoch 3) and loss 0.30 (epoch 2)
assert any("epoch_3" in p for p in active)
assert any("epoch_2" in p for p in active)
metadata.yml (472 bytes)
lesson_id: D235
day: 235
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/experiment_tracking_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'rm -rf test_runs test_checkpoints'
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/experiment_tracking_lib.py (778 bytes)
import json
import os
from typing import Dict, Any, List, Optional
class ExperimentLogger:
def __init__(self, run_id: str, log_dir: str = "./runs", config: Optional[Dict[str, Any]] = None):
# TODO: Initialize experiment logger and log init event to JSONL
pass
def log_step(self, step: int, epoch: int, metrics: Dict[str, float]):
# TODO: Append step metrics to JSONL file
pass
class CheckpointManager:
def __init__(self, save_dir: str = "./checkpoints", max_to_keep: int = 2):
# TODO: Initialize checkpoint manager
pass
def save_checkpoint(self, epoch: int, val_loss: float, model_state: Dict[str, Any]) -> List[str]:
# TODO: Save checkpoint and prune older checkpoints beyond max_to_keep
pass
starter/test_experiment_tracking_lib.py (1136 bytes)
import pytest
import os
import json
from examples.experiment_tracking_lib import ExperimentLogger, CheckpointManager
def test_experiment_logger(tmp_path):
log_dir = str(tmp_path / "runs")
logger = ExperimentLogger("test_run", log_dir=log_dir, config={"lr": 1e-4})
logger.log_step(1, 1, {"loss": 0.5})
logger.log_eval(1, {"val_acc": 0.9})
assert os.path.exists(logger.log_path)
with open(logger.log_path, "r", encoding="utf-8") as f:
lines = [json.loads(line) for line in f]
assert len(lines) == 3
assert lines[0]["event"] == "init"
assert lines[1]["loss"] == 0.5
assert lines[2]["val_acc"] == 0.9
def test_checkpoint_manager_retention(tmp_path):
save_dir = str(tmp_path / "ckpts")
mgr = CheckpointManager(save_dir=save_dir, max_to_keep=2)
mgr.save_checkpoint(1, 0.50, {"step": 1})
mgr.save_checkpoint(2, 0.30, {"step": 2})
active = mgr.save_checkpoint(3, 0.20, {"step": 3})
assert len(active) == 2
# Best 2: loss 0.20 (epoch 3) and loss 0.30 (epoch 2)
assert any("epoch_3" in p for p in active)
assert any("epoch_2" in p for p in active)
test_checkpoints/model_epoch_2_loss_0.250.json (58 bytes)
{"epoch": 2, "val_loss": 0.25, "state": {"w": [1.1, 2.1]}} test_checkpoints/model_epoch_3_loss_0.200.json (57 bytes)
{"epoch": 3, "val_loss": 0.2, "state": {"w": [1.2, 2.2]}} test_runs/demo-run-01.jsonl (207 bytes)
{"event": "init", "run_id": "demo-run-01", "config": {"lr": 0.0003, "batch_size": 32}}
{"event": "step", "step": 1, "epoch": 1, "loss": 0.45}
{"event": "eval", "epoch": 1, "val_loss": 0.38, "val_acc": 0.88}
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 235 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_experiment_tracking_lib.py (1136 bytes)
import pytest
import os
import json
from examples.experiment_tracking_lib import ExperimentLogger, CheckpointManager
def test_experiment_logger(tmp_path):
log_dir = str(tmp_path / "runs")
logger = ExperimentLogger("test_run", log_dir=log_dir, config={"lr": 1e-4})
logger.log_step(1, 1, {"loss": 0.5})
logger.log_eval(1, {"val_acc": 0.9})
assert os.path.exists(logger.log_path)
with open(logger.log_path, "r", encoding="utf-8") as f:
lines = [json.loads(line) for line in f]
assert len(lines) == 3
assert lines[0]["event"] == "init"
assert lines[1]["loss"] == 0.5
assert lines[2]["val_acc"] == 0.9
def test_checkpoint_manager_retention(tmp_path):
save_dir = str(tmp_path / "ckpts")
mgr = CheckpointManager(save_dir=save_dir, max_to_keep=2)
mgr.save_checkpoint(1, 0.50, {"step": 1})
mgr.save_checkpoint(2, 0.30, {"step": 2})
active = mgr.save_checkpoint(3, 0.20, {"step": 3})
assert len(active) == 2
# Best 2: loss 0.20 (epoch 3) and loss 0.30 (epoch 2)
assert any("epoch_3" in p for p in active)
assert any("epoch_2" in p for p in active)
Troubleshooting
Troubleshooting: Day 235 - Experiment Tracking
Common Issues
- Corrupt JSONL records:
- Cause: Writing un-serialized objects to file.
- Fix: Use
json.dumps()on primitive Python dictionaries.
Security notes
Security & Privacy: Day 235 - Experiment Tracking
Security Guidance
- Mask all sensitive credentials before logging config dictionaries.