Deep Learning βΊ Training at Scale βΊ Day 235
Day 235: Experiment Tracking
Master systematic experiment tracking and scientific reproducibility: understand why unorganized notebook scripts lead to irreproducible AI models, compare tracking platforms (Weights & Biases, MLflow, TensorBoard), design structured experiment schemas (hyperparameters, metrics, hardware telemetry, model artifacts, git commit hashes), build a lightweight JSONL/SQLite experiment logger from scratch, implement automated checkpoint versioning, and configure Bayesian hyperparameter sweeps.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-235-experiment-tracking
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β section / subsection / week / day:
cd labs/sections/deep-learning/day-235-experiment-tracking - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Identify the 5 pillars of AI experiment reproducibility: Code Version (Git SHA), Data Version, Environment (Dependencies), Hyperparameters, and Random Seeds.
- Compare enterprise experiment tracking platforms: Weights & Biases (W&B), MLflow, TensorBoard, and Neptune.
- Build a self-contained, crash-safe JSONL experiment logger and telemetry emitter in Python.
- Implement automated model checkpoint versioning with metadata sidecars and best-model retention.
- Configure systematic hyperparameter search strategies: Grid Search, Random Search, and Bayesian Optimization with Hyperband pruning.
Prerequisites
Why this matters
In software engineering, code that is not under version control is a liability.
In deep learning, an un-tracked training run is worse: it is a catastrophic waste of electricity, money, and time:
- You spend 3 days training a model on 8 GPUs. You get an amazing
94.2\%accuracy score. - Two weeks later, your team lead asks: βWhich learning rate schedule did you use? What was the random seed? Which version of the dataset was that?β
- You open your Jupyter Notebook history: the cells were overwritten, the print outputs are gone, and you cannot reproduce the result.
Science without reproducibility is not scienceβit is alchemy.
Today, you will learn how to build and operate enterprise-grade Experiment Tracking Systems: capturing complete code lineage, hardware telemetry, step-by-step metrics, versioned model artifacts, and automated Bayesian hyperparameter sweeps.
The idea in plain language
Think of clinical drug trials in medical science:
- The Amateur Chemist: Mixes random chemicals in a beaker without measuring grams, leaves no notes, and discovers a cure for a disease. But because they took no notes (No Experiment Tracking), they can never manufacture the cure again.
- The Modern Pharmaceutical Laboratory: Every batch has a barcoded lot number (Model Version), exact chemical proportions recorded down to the milligram (Hyperparameters), temperature sensors recording every minute (Hardware Telemetry), and patient blood assays recorded daily (Step Metrics).
Every trial is 100\% reproducible by any laboratory in the world.
Historical background
- 2015 (TensorBoard & Google TensorFlow): Google released TensorBoard, introducing visual scalar loss curves, computation graph rendering, and weight histogram visualization.
- 2018 (MLflow & Databricks): Matei Zaharia and Databricks open-sourced MLflow, establishing the four pillars of ML lifecycle management: Tracking, Projects, Models, and Registry.
- 2019 (Weights & Biases / W&B): Lukas Biewald and Chris Van Pelt founded W&B, creating the gold standard cloud-native collaborative experiment dashboard for deep learning teams.
- 2021βPresent (Full Lineage & DVC): Integration of data version control (DVC), hardware MFU telemetry, and automated artifact registries into modern CI/CD pipelines.
What it is β and what it is not
What Experiment Tracking IS:
- Complete Provenance & Lineage: Capturing Git commit SHAs, uncommitted diffs, environment requirements, random seeds, hyperparameters, training loss curves, and evaluation metrics.
- Atomic & Crash-Safe Logging: Streaming metrics to persistent storage (JSONL, SQLite, remote server) asynchronously without impacting training speed.
What it is NOT:
- Not Casual
print()Statements: Piping stdout to a text fileoutput.txtis fragile, unsearchable, and impossible to compare across 100 parallel hyperparameter trials.
Why it was created and what problems it solves
Experiment tracking eliminates the primary pathologies of machine learning development:
- The βIt Worked Yesterdayβ Syndrome: Inability to reproduce a previous peak validation metric.
- Silent Degradation: Catching gradient vanishing, exploding loss spikes, or GPU thermal throttling early in multi-day training runs.
- Hyperparameter Blindness: Visualizing multi-dimensional parameter spaces to find optimal learning rate and weight decay regimes.
How it works
Let us dissect the 5 reproducibility pillars, structured logging formats, and hyperparameter search algorithms.
1. The 5 Pillars of Deep Learning Reproducibility
To guarantee that any training run can be reproduced bit-for-bit, your tracking system must record:
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β THE 5 PILLARS OF REPRODUCIBILITY β
ββββββββββββββββββββββ¬ββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββ€
β Pillar β What is Captured β Production Mechanism β
ββββββββββββββββββββββΌββββββββββββββββββββββββΌββββββββββββββββββββββββββββ€
β 1. Code Lineage β Git SHA, Branch, Diff β git rev-parse HEAD β
β 2. Data Provenance β Dataset Hash / Versionβ SHA-256 Manifest / DVC β
β 3. Environment β Package versions, CUDAβ pip freeze, Docker Image β
β 4. Configuration β All Hyperparameters β YAML / Dict Config Schema β
β 5. Randomness β PRNG Seeds β torch.manual_seed(42) β
ββββββββββββββββββββββ΄ββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββ
2. Structured Metrics Logging with JSON Lines (JSONL)
A production training loop should never append to raw CSV files (which require rewriting headers and corrupt on unbuffered crashes).
JSON Lines (JSONL) provides crash-safe atomic logging:
{"event": "init", "run_id": "exp-2026-08-29-01", "git_sha": "a1b2c3d", "lr": 0.0003, "batch_size": 64}
{"event": "step", "epoch": 1, "step": 100, "train_loss": 0.542, "lr": 0.00015, "gpu_util_pct": 94.2}
{"event": "step", "epoch": 1, "step": 200, "train_loss": 0.381, "lr": 0.00030, "gpu_util_pct": 95.1}
{"event": "eval", "epoch": 1, "val_loss": 0.320, "val_acc": 0.912, "val_f1": 0.908}
{"event": "artifact", "type": "checkpoint", "path": "checkpoints/best_model.pt", "val_loss": 0.320}
3. Hyperparameter Sweeps & The Pareto Frontier
How do you discover optimal hyperparameters across a large search space?
- Grid Search: Evaluates every combination on a fixed grid. Thorough but scales exponentially (
O(K^D)), wasting90\%of compute on unpromising combinations. - Random Search (Bergstra & Bengio, 2012): Randomly samples configurations. Statistically proven to outperform grid search for deep learning because different parameters have unequal importance.
- Bayesian Optimization (GPyOpt / Optuna): Fits a probabilistic model (e.g. Gaussian Process or Tree-structured Parzen Estimator) to prior trial results, balancing Exploration (high uncertainty regions) and Exploitation (promising minima).
- Hyperband & Successive Halving (Li et al., 2016):
- Starts 64 trials for 1 epoch.
- Halts the bottom
50\%worst performers. - Trains the top 32 for 2 epochs
\rightarrowhalts bottom50\%\rightarrowtrains top 16 for 4 epochs\rightarrowtrains the final champion for the full budget!
4. Model Checkpoint Versioning & Metadata Sidecars
Saving model_final.pt is dangerous. Always save checkpoints with structured metadata sidecars:
checkpoints/
βββ run_exp01_epoch03_val0.215.pt # PyTorch State Dict
βββ run_exp01_epoch03_val0.215.json # Metadata Sidecar
The metadata JSON contains:
{
"checkpoint_epoch": 3,
"val_loss": 0.215,
"val_accuracy": 0.938,
"model_architecture": "DistilBertForSequenceClassification",
"num_parameters": 66955010,
"commit_sha": "a1b2c3d",
"timestamp": "2026-08-29T12:00:00Z"
}
5. Data Version Control (DVC) & Data Lineage
Code versioning with Git is straightforward because text files are small. But machine learning datasets are gigabytes or terabytes in size:
- DVC (Data Version Control): Solves data provenance by replacing large dataset directories with small, text-based metadata pointer files (
dataset.dvc) that are committed directly into Git. - Content-Addressable Storage: Datasets are hashed via SHA-256 and stored in centralized cloud blob storage (AWS S3, Google Cloud Storage, or Azure Blob).
- Lineage Guarantee: When you check out a Git commit from 6 months ago, running
dvc checkoutrestores the exact matching dataset slice used to train that specific model checkpoint!
6. Gradient & Weight Histogram Telemetry
Scalar loss curves only tell you that a model is failing; Tensor Histograms tell you why it is failing:
- Vanishing Gradients: If gradient norm histograms across bottom layers collapse to zero (
< 10^{-7}), backpropagation is dying. - Exploding Gradients: If top-layer gradient norms spike past
10^{2}, weights will oscillate wildly. - Dead Neurons (ReLU/GELU Saturation): If activation histograms show
90\%+of values pinned at zero, whole layer subspaces have died. - Weight Norm Drift: Tracking
\ell_2norm of weight matrices reveals whether weight decay is appropriately regularizing parameter growth.
7. Distributed Telemetry & Hardware Health Telemetry
In multi-node GPU cluster training, tracking software metrics alone is insufficient:
- GPU Thermal Throttling: If GPU 14 in Node 3 reaches
85^\circ\text{C}, it automatically throttles its clock speed from1,800\text{ MHz} \rightarrow 1,100\text{ MHz}, slowing down the entire synchronous All-Reduce ring. - VRAM Fragmentation Tracking: Monitoring peak allocated vs reserved PyTorch memory (
torch.cuda.memory_allocated()vstorch.cuda.memory_reserved()) catches out-of-memory memory leaks before crashes occur. - Continuous MFU Profiling: Real-time computation of Model FLOPs Utilization (MFU) to detect communication bottlenecks across InfiniBand switches.
8. Enterprise Model Registry & Deployment Governance
A trained checkpoint is not ready for customer traffic until it passes through an Immutable Model Registry (such as MLflow Models or Hugging Face Hub):
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β ENTERPRISE MODEL LIFECYCLE STAGING β
ββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Stage β Verification Gate & Criteria β
ββββββββββββββββββββββΌββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β 1. Development β Raw training checkpoint + JSON metadata sidecar β
β 2. Candidate β Automated eval suite: Accuracy, F1, Latency tests β
β 3. Staging β Shadow deployment: Comparing outputs vs champion β
β 4. Production β Canary deployment (5% -> 100% traffic routing) β
β 5. Archived β Deprecated version retained for regulatory audit β
ββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββββββ
9. Multi-Objective Pareto Optimization (Accuracy vs. Latency vs. Cost)
In production AI engineering, single-metric optimization is rarely the goal:
- You do not just want the highest accuracy; you want the fastest model that meets an accuracy threshold to minimize cloud inference serving costs.
- The Pareto Frontier: The set of non-dominated solutions where you cannot improve accuracy without increasing latency or memory consumption.
- Tracking systems plot multi-dimensional Pareto charts allowing engineers to choose the optimal trade-off point for mobile edge deployment vs datacenter serving.
10. PyTorch Distributed Checkpointing (DCP / PyTorch 2.x)
In massive distributed clusters (e.g. 512 GPUs), standard PyTorch checkpointing (torch.save(model.state_dict())) fails catastrophically:
- Gathering a 70B parameter model to GPU 0 for serial disk writing causes CPU memory crashes and stalls the cluster for 30 minutes.
- PyTorch Distributed Checkpoint (DCP): Every GPU rank writes its own local sharded tensor slice directly to parallel NVMe storage asynchronously in parallel.
- Re-sharding Agility: DCP checkpoints can be re-loaded onto a different cluster topology (e.g. saved on 64 GPUs, reloaded on 128 GPUs) without manual tensor restructuring!
- Continuous Metric Assertion & Alerting: Modern MLOps pipelines run automated assertions against logged metrics (e.g. asserting validation loss decreases by at least
5\%per epoch). If an anomaly or NaN occurs, webhooks instantly notify on-call engineers via Slack or PagerDuty. This prevents wasted cloud expenditure on silent divergence.
An everyday analogy
Think of an elite flight data recorder (the βBlack Boxβ) on a modern commercial airliner:
- During Normal Flight: It continuously records airspeed (Training Loss), altitude (Validation Accuracy), engine temperature (GPU VRAM & Thermals), and pilot control inputs (Hyperparameters & Learning Rate).
- If an Anomaly Occurs: Flight investigators do not have to guess why a failure happenedβthey review the synchronized telemetry timeline down to the millisecond to identify the exact root cause.
Examples in practice
Let us build a Lightweight, Crash-Safe Experiment Logger in Python:
import json
import os
import time
from typing import Dict, Any, Optional
class ExperimentLogger:
def __init__(self, run_id: str, log_dir: str = "./runs", config: Optional[Dict[str, Any]] = None):
self.run_id = run_id
self.log_dir = log_dir
os.makedirs(self.log_dir, exist_ok=True)
self.log_path = os.path.join(self.log_dir, f"{run_id}.jsonl")
# Log initialization metadata
init_event = {
"event": "init",
"run_id": run_id,
"timestamp": time.time(),
"config": config or {}
}
self._append(init_event)
def _append(self, record: Dict[str, Any]):
with open(self.log_path, "a", encoding="utf-8") as f:
f.write(json.dumps(record) + "\n")
def log_step(self, step: int, epoch: int, metrics: Dict[str, float]):
record = {
"event": "step",
"step": step,
"epoch": epoch,
"timestamp": time.time(),
**metrics
}
self._append(record)
def log_eval(self, epoch: int, metrics: Dict[str, float]):
record = {
"event": "eval",
"epoch": epoch,
"timestamp": time.time(),
**metrics
}
self._append(record)
def log_artifact(self, name: str, artifact_type: str, file_path: str, metadata: Optional[Dict[str, Any]] = None):
record = {
"event": "artifact",
"name": name,
"type": artifact_type,
"path": file_path,
"metadata": metadata or {},
"timestamp": time.time()
}
self._append(record)
Implications: security, privacy, performance, scalability, and cost
- Secret Leakage in Configs:
- Developers frequently accidentally log API tokens (OpenAI, Hugging Face, AWS) or database connection strings in experiment configuration dictionaries. Always sanitize configs with regex masking prior to logging.
- Local Disk Space Exhaustion:
- Saving uncompressed 10 GB checkpoints every 500 steps can fill a 2 TB SSD in a single afternoon. Always configure Best-
KCheckpoint Retention (e.g. keep only top 3 checkpoints and delete older ones).
- Saving uncompressed 10 GB checkpoints every 500 steps can fill a 2 TB SSD in a single afternoon. Always configure Best-
Alternatives: free, open source, and commercial
| Tool | Deployment Model | Artifact Storage | Best Feature | License |
|---|---|---|---|---|
| Weights & Biases (W&B) | Cloud SaaS / Enterprise | S3 / GCS / Cloud | World-class dashboards | Commercial / Free Tier |
| MLflow | Self-Hosted / Databricks | Local / Cloud Store | Central Model Registry | Open Source (Apache 2.0) |
| TensorBoard | Local / Open Source | Local Event Files | Native PyTorch/TF integ | Open Source (Apache 2.0) |
| Aim / Neptune | Hybrid / Cloud | Flexible | High-volume scalar logging | Open / Commercial |
Comparison with related concepts
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β EXPERIMENT TRACKING COMPARISON β
ββββββββββββββββββββββ¬ββββββββββββββββββ¬βββββββββββββββββββ¬βββββββββββββββ€
β Feature β TensorBoard β MLflow β W&B β
ββββββββββββββββββββββΌββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββ€
β Hosting β Local process β Local / Server β Cloud SaaS β
β Model Registry β None β Built-in β Built-in β
β Hyperparam Sweeps β Basic (HParams) β External β Native Bayes β
β Collaborative UI β Single User β Multi-User Team β Enterprise β
ββββββββββββββββββββββ΄ββββββββββββββββββ΄βββββββββββββββββββ΄βββββββββββββββ
When to use it β and when not to
When to Use Experiment Tracking:
- Every deep learning project, hyperparameter tuning sweep, distributed training run, research paper reproduction, and production model deployment.
When NOT to use:
- Trivial one-line Python calculator scripts or exploratory scratch REPL sessions.
The AI connection
- Machine learning is empirical, so the experiment log is the primary artefact β without it, a good result is an anecdote rather than a finding.
- Reproducibility is a systems problem, not a discipline problem: code, data, weights, environment and randomness all have to be pinned together.
- Data versioning is what most teams skip and later regret, because a model is a function of its training set as much as its architecture.
- The registry is where research meets operations, turning a checkpoint into something auditable, approvable and rollback-able.
- Trading accuracy against latency and cost is the real decision, and only a tracked sweep makes that frontier visible.
Knowledge check
- What are the 5 pillars of deep learning reproducibility?
- Why is JSONL preferable to CSV for high-frequency step logging?
- How does Bayesian Hyperparameter Optimization differ from Random Search?
- How does Hyperband use Successive Halving to save GPU compute hours?
- What metadata should always accompany a saved model checkpoint file?
Hands-on exercise
In this lab, you will build and test a modular experiment tracking and checkpoint versioning library in Python: implement a crash-safe JSONL experiment logger, build a Best-K checkpoint retention manager with metadata sidecars, simulate a hyperparameter search trial evaluator, and verify reproducibility.
Expected output
[Experiment Tracking Verification Suite]
Testing JSONL Experiment Logger:
Initialized Run: exp-test-01 with 3 Hyperparameters
Logged 5 Step Metrics and 2 Eval Events [VERIFIED]
Testing Best-K Checkpoint Manager (K=2):
Saved Epoch 1 (Val Loss: 0.450) -> Retained [Epoch 1]
Saved Epoch 2 (Val Loss: 0.320) -> Retained [Epoch 2, Epoch 1]
Saved Epoch 3 (Val Loss: 0.210) -> Retained [Epoch 3, Epoch 2] (Epoch 1 Pruned) [PRUNING VERIFIED]
Testing Hyperparameter Sweep Evaluator:
Swept 3 Configurations: Best LR = 0.0003 with Min Val Loss = 0.210 [OPTIMAL FOUND]
Test Suite: 3 passed in 0.42s
Validate your work
Run the automated test runner:
./tests/run_tests.sh
Troubleshooting
- When writing to JSONL files, always flush or close file handles to ensure data is committed to disk.
Common mistakes
- Overwriting checkpoints without metadata: Makes it impossible to know what hyperparameters or validation scores correspond to a weight file.
Practice assignment
- Implement a Weights & Biases (W&B) integration script that logs training loss, validation accuracy, and sample generated text outputs.
- Build an Optuna Bayesian Hyperparameter Sweep optimizing learning rate and weight decay on a small classifier.
Extension challenge
Implement a Lightweight SQLite Experiment Registry:
- Build a Python registry class that creates a local SQLite database table indexing runs, parameters, best metrics, and artifact file paths with SQL query support.
Quiz
Q1. What are the 5 foundational metadata elements required to guarantee 100% scientific reproducibility of a deep learning training run?
- The computer username and wallpaper
- Only the model weights file
- Commit, data, env, config, seeds
- The time of day the training started
Show answer
Answer: C. Commit, data, env, config, seeds
Reproducibility requires capturing the exact code commit, dataset hash, environment lockfile, hyperparameter configuration, and random seeds.
Q2. Why is logging experiment metrics to structured JSON Lines (JSONL) files superior to printing raw strings to standard output or CSV files?
- CSV files cannot store numbers
- JSONL compresses files into audio
- JSONL disables backpropagation
- It appends safely on a crash
Show answer
Answer: D. It appends safely on a crash
JSONL provides crash-safe line-by-line atomic appending of flexible schema dictionaries.
Q3. What is the primary advantage of Bayesian Hyperparameter Optimization over brute-force Grid Search?
- It builds a surrogate model to pick promising points
- Bayesian optimization tries every single number from 1 to 1,000,000
- Bayesian optimization runs without a CPU
- Grid search is illegal in deep learning
Show answer
Answer: A. It builds a surrogate model to pick promising points
Bayesian optimization uses prior trial history to guide intelligent exploration toward the global minimum.
Q4. How does the Hyperband algorithm accelerate hyperparameter sweeps?
- It doubles the learning rate on every epoch
- It halts the worst runs early
- It compresses the dataset text
- It deletes failed runs from git
Show answer
Answer: B. It halts the worst runs early
Hyperband uses Successive Halving to terminate unpromising runs early and focus compute on top configurations.
Q5. What is a Model Artifact Registry and why is it necessary in production MLOps?
- A folder on a personal laptop desktop
- A list of GPU serial numbers
- A versioned store of weights and metrics
- A software license agreement
Show answer
Answer: C. A versioned store of weights and metrics
A model registry provides immutable versioning, lineage tracking, and deployment governance for trained models.
Glossary
- Experiment Tracking
- The systematic recording of code versions, datasets, hyperparameters, training metrics, hardware telemetry, and model artifacts across training runs.
- Lineage
- The complete provenance graph linking a trained model artifact back to the exact code commit, training script, hyperparameters, and dataset version that created it.
- JSONL (JSON Lines)
- A structured text file format where each line is a self-contained, valid JSON object, enabling crash-safe atomic logging.
- Weights & Biases (W&B)
- An enterprise MLOps platform for experiment tracking, dashboard visualization, hyperparameter sweeps, and artifact management.
- MLflow
- An open-source machine learning lifecycle platform featuring tracking, project packaging, model serving, and a central model registry.
- Hyperparameter Sweep
- An automated search process evaluating multiple hyperparameter combinations across a predefined search space to optimize a target validation metric.
- Hyperband
- An adaptive hyperparameter optimization algorithm combining random exploration with principled early stopping via successive halving.
- Model Registry
- A centralized catalog for storing, staging, versioning, and deploying trained neural network model weights with metadata.
Sources and further reading
- Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization β JMLR 2017 (accessed 2026-08-29)
- MLflow: A Platform for Managing the Machine Learning Lifecycle β Linux Foundation (accessed 2026-08-29)
- Reproducible Machine Learning: A Guide to Best Practices β ACM Computing Surveys (accessed 2026-08-29)
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.