Sandeep Bazar 365 Days of AI
Labs

Machine Learning › Evaluation and Interpretation › Day 177

Hands-on lab — Day 177: Learning Curves and Diagnostics

Commands

Setup

python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt

Run

.venv/bin/python examples/learning_curves_and_diagnostics_lib.py

Test

./tests/run_tests.sh

File tree

examples/metric_diagnostics_lib.py
examples/test_metric_diagnostics_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/metric_diagnostics_lib.py
starter/test_metric_diagnostics_lib.py
tests/run_tests.sh
troubleshooting.md

Lab README

Day 177 Lab: Learning Curves and Diagnostics

Day number: 177 of 365.

Lesson

Covering day-177-learning-curves-and-diagnostics.

Purpose

Master high bias vs high variance diagnosis, learning curves over training sample size, validation curves, and error decomposition. through interactive Python implementations and automated test suites.

Learning objectives

  • Implement core mathematical algorithms for learning curves and diagnostics.
  • Benchmark models against rigorous baselines.
  • Execute automated unit and integration tests.
  • Analyze failure modes and edge cases.

Prerequisites

  • Python 3.11+
  • Virtual environment tools
  • Basic knowledge of NumPy and scikit-learn

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu 22.04+, Debian, Fedora, Arch)
  • Windows (WSL2 recommended)

Hardware requirements

  • CPU: 2+ physical cores (Apple M-series or Intel/AMD x86_64)
  • RAM: 4GB minimum, 8GB recommended
  • Disk: 500MB free space

Required software

  • Python 3.11 or higher
  • Git
  • Bash shell

Free and open-source options

  • Python: python.org (PSFL)
  • scikit-learn: BSD 3-Clause
  • pytest: MIT License

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/: Scaffolded implementation files for student completion.
  • examples/: Fully functional reference library implementation.
  • tests/: Pytest suite and shell validation runners.
  • expected-output/: Captured reference terminal logs.
  • requirements/: Python package dependency specifications.
  • troubleshooting.md: Common runtime failure solutions.
  • security.md: Local execution safety guidance.

How to run

python3 examples/learning_curves_and_diagnostics_lib.py

What the commands do

  • Executes reference implementation demonstration and benchmarks.

Expected output

Reference logs are captured in expected-output/run-output.txt and expected-output/test-output.txt.

Validation steps

  1. Run ./tests/run_tests.sh.
  2. Ensure exit code is 0.

Tests

pytest tests/ -v

Cleanup

rm -rf .venv __pycache__ .pytest_cache

Troubleshooting

Refer to troubleshooting.md for common import or version issues.

Security notes

Refer to security.md for isolation and data safety guidance.

Extension exercises

  • Test on imbalanced real-world datasets.
  • Profile runtime latency and memory utilization.

Expected output

FIELDS.md

# Output Fields
- train_sizes, train_scores_mean, val_scores_mean, gap
- regime, diagnosis, actionable_remedies

examples-run.txt

=== Learning Curve Diagnostics ===
Sample Size  40 | Train R2: 0.9965 | Val R2: 0.9946 | Gap: 0.0019
Sample Size 100 | Train R2: 0.9975 | Val R2: 0.9972 | Gap: 0.0003
Sample Size 160 | Train R2: 0.9978 | Val R2: 0.9975 | Gap: 0.0003
Sample Size 200 | Train R2: 0.9980 | Val R2: 0.9976 | Gap: 0.0004

Model Regime: OPTIMAL
Diagnosis: Optimal Regime: Balanced bias-variance tradeoff with high validation performance and low generalization gap.

measured-values.txt

=== Learning Curve Diagnostics ===
Sample Size  40 | Train R2: 0.9965 | Val R2: 0.9946 | Gap: 0.0019
Sample Size 100 | Train R2: 0.9975 | Val R2: 0.9972 | Gap: 0.0003
Sample Size 160 | Train R2: 0.9978 | Val R2: 0.9975 | Gap: 0.0003
Sample Size 200 | Train R2: 0.9980 | Val R2: 0.9976 | Gap: 0.0004

Model Regime: OPTIMAL
Diagnosis: Optimal Regime: Balanced bias-variance tradeoff with high validation performance and low generalization gap.

starter-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>
collecting ... collected 3 items

labs/sections/machine-learning/day-177-learning-curves-and-diagnostics/starter/test_metric_diagnostics_lib.py::test_compute_learning_curves FAILED [ 33%]
labs/sections/machine-learning/day-177-learning-curves-and-diagnostics/starter/test_metric_diagnostics_lib.py::test_compute_validation_curves FAILED [ 66%]
labs/sections/machine-learning/day-177-learning-curves-and-diagnostics/starter/test_metric_diagnostics_lib.py::test_diagnose_model_regime FAILED [100%]

=================================== FAILURES ===================================
_________________________ test_compute_learning_curves _________________________

regression_data = (array([[-2.87226221,  0.32316771,  0.51360011, ..., -0.4660365 ,
        -1.16991684, -1.76843937],
       [-0.276813...0739,  204.63132389,  -59.88072323, -144.55824299,
         93.88603709,   90.09488867,   53.26181392,  145.36974683]))

    def test_compute_learning_curves(regression_data):
        X, y = regression_data
        model = Ridge(alpha=1.0)
        res = compute_learning_curves(model, X, y, train_sizes=[0.2, 0.5, 1.0], cv=3)
    
>       assert len(res["train_sizes"]) == 3
                   ^^^^^^^^^^^^^^^^^^
E       TypeError: 'NoneType' object is not subscriptable

labs/sections/machine-learning/day-177-learning-curves-and-diagnostics/starter/test_metric_diagnostics_lib.py:22: TypeError
________________________ test_compute_validation_curves ________________________

regression_data = (array([[-2.87226221,  0.32316771,  0.51360011, ..., -0.4660365 ,
        -1.16991684, -1.76843937],
       [-0.276813...0739,  204.63132389,  -59.88072323, -144.55824299,
         93.88603709,   90.09488867,   53.26181392,  145.36974683]))

    def test_compute_validation_curves(regression_data):
        X, y = regression_data
        model = DecisionTreeRegressor(random_state=42)
        depths = [1, 3, 6, 10]
        res = compute_validation_curves(model, X, y, param_name="max_depth", param_range=depths, cv=3)
    
>       assert res["param_name"] == "max_depth"
               ^^^^^^^^^^^^^^^^^
E       TypeError: 'NoneType' object is not subscriptable

labs/sections/machine-learning/day-177-learning-curves-and-diagnostics/starter/test_metric_diagnostics_lib.py:36: TypeError
__________________________ test_diagnose_model_regime __________________________

    def test_diagnose_model_regime():
        # Case 1: High Bias (both train and val scores low)
        d_bias = diagnose_model_regime(train_score=0.45, val_score=0.42, benchmark_score=0.85)
>       assert d_bias["regime"] == "HIGH_BIAS"
               ^^^^^^^^^^^^^^^^
E       TypeError: 'NoneType' object is not subscriptable

labs/sections/machine-learning/day-177-learning-curves-and-diagnostics/starter/test_metric_diagnostics_lib.py:42: TypeError
=========================== short test summary info ============================
FAILED labs/sections/machine-learning/day-177-learning-curves-and-diagnostics/starter/test_metric_diagnostics_lib.py::test_compute_learning_curves
FAILED labs/sections/machine-learning/day-177-learning-curves-and-diagnostics/starter/test_metric_diagnostics_lib.py::test_compute_validation_curves
FAILED labs/sections/machine-learning/day-177-learning-curves-and-diagnostics/starter/test_metric_diagnostics_lib.py::test_diagnose_model_regime
============================== 3 failed in 0.80s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>
collecting ... collected 3 items

labs/sections/machine-learning/day-177-learning-curves-and-diagnostics/examples/test_metric_diagnostics_lib.py::test_compute_learning_curves PASSED [ 33%]
labs/sections/machine-learning/day-177-learning-curves-and-diagnostics/examples/test_metric_diagnostics_lib.py::test_compute_validation_curves PASSED [ 66%]
labs/sections/machine-learning/day-177-learning-curves-and-diagnostics/examples/test_metric_diagnostics_lib.py::test_diagnose_model_regime PASSED [100%]

============================== 3 passed in 4.88s ===============================

Source files

examples/metric_diagnostics_lib.py (3525 bytes) 
import numpy as np
from sklearn.model_selection import learning_curve, validation_curve
from sklearn.base import clone

def compute_learning_curves(estimator, X, y, train_sizes=None, cv=5, scoring="r2"):
    """
    Compute training and validation scores across increasing training sample sizes.
    """
    if train_sizes is None:
        train_sizes = np.linspace(0.1, 1.0, 5)
        
    sizes, train_scores, val_scores = learning_curve(
        estimator, X, y, train_sizes=train_sizes, cv=cv, scoring=scoring, n_jobs=1, random_state=42
    )
    
    train_mean = np.mean(train_scores, axis=1)
    train_std = np.std(train_scores, axis=1)
    val_mean = np.mean(val_scores, axis=1)
    val_std = np.std(val_scores, axis=1)
    
    return {
        "train_sizes": sizes.tolist(),
        "train_scores_mean": train_mean.tolist(),
        "train_scores_std": train_std.tolist(),
        "val_scores_mean": val_mean.tolist(),
        "val_scores_std": val_std.tolist(),
    }

def compute_validation_curves(estimator, X, y, param_name, param_range, cv=5, scoring="r2"):
    """
    Compute training and validation scores across a single hyperparameter range.
    """
    train_scores, val_scores = validation_curve(
        estimator, X, y, param_name=param_name, param_range=param_range, cv=cv, scoring=scoring, n_jobs=1
    )
    
    return {
        "param_name": param_name,
        "param_range": list(param_range),
        "train_scores_mean": np.mean(train_scores, axis=1).tolist(),
        "val_scores_mean": np.mean(val_scores, axis=1).tolist(),
    }

def diagnose_model_regime(train_score, val_score, benchmark_score=0.90, generalization_gap_threshold=0.15):
    """
    Diagnose whether a model is suffering from High Bias (Underfitting), High Variance (Overfitting),
    or operating in the Optimal Tradeoff Regime.
    """
    gap = train_score - val_score
    
    if train_score < (benchmark_score - 0.15) and val_score < (benchmark_score - 0.15):
        return {
            "regime": "HIGH_BIAS",
            "diagnosis": "Underfitting: Model has insufficient representational capacity. Both training and validation errors are unacceptably high.",
            "actionable_remedies": [
                "Add polynomial features or interaction terms",
                "Decrease regularization strength (e.g. reduce Ridge alpha or L1 penalty)",
                "Increase model complexity (e.g. deeper decision trees, more neural units)",
                "Note: Adding more training data will NOT resolve high bias"
            ]
        }
    elif gap > generalization_gap_threshold:
        return {
            "regime": "HIGH_VARIANCE",
            "diagnosis": "Overfitting: Model memorized training noise. Large generalization gap between training and validation scores.",
            "actionable_remedies": [
                "Collect more training data (learning curve shows validation error steadily decreasing)",
                "Increase regularization strength (increase L2 weight decay / Ridge alpha)",
                "Apply feature selection to prune uninformative noise dimensions",
                "Apply ensemble bagging / Random Forests with subspace sampling"
            ]
        }
    else:
        return {
            "regime": "OPTIMAL",
            "diagnosis": "Optimal Regime: Balanced bias-variance tradeoff with high validation performance and low generalization gap.",
            "actionable_remedies": ["Model is ready for holdout testing and canary deployment"]
        }
examples/test_metric_diagnostics_lib.py (2134 bytes) 
import pytest
import numpy as np
from sklearn.linear_model import Ridge
from sklearn.tree import DecisionTreeRegressor
from sklearn.datasets import make_regression
from metric_diagnostics_lib import (
    compute_learning_curves,
    compute_validation_curves,
    diagnose_model_regime
)

@pytest.fixture
def regression_data():
    X, y = make_regression(n_samples=200, n_features=10, noise=5.0, random_state=42)
    return X, y

def test_compute_learning_curves(regression_data):
    X, y = regression_data
    model = Ridge(alpha=1.0)
    res = compute_learning_curves(model, X, y, train_sizes=[0.2, 0.5, 1.0], cv=3)
    
    assert len(res["train_sizes"]) == 3
    assert len(res["train_scores_mean"]) == 3
    assert len(res["val_scores_mean"]) == 3
    # Training score typically starts high and stabilizes
    assert res["train_scores_mean"][0] >= res["train_scores_mean"][-1] - 0.20
    # Validation score typically increases with sample size
    assert res["val_scores_mean"][-1] > res["val_scores_mean"][0]

def test_compute_validation_curves(regression_data):
    X, y = regression_data
    model = DecisionTreeRegressor(random_state=42)
    depths = [1, 3, 6, 10]
    res = compute_validation_curves(model, X, y, param_name="max_depth", param_range=depths, cv=3)
    
    assert res["param_name"] == "max_depth"
    assert len(res["val_scores_mean"]) == 4

def test_diagnose_model_regime():
    # Case 1: High Bias (both train and val scores low)
    d_bias = diagnose_model_regime(train_score=0.45, val_score=0.42, benchmark_score=0.85)
    assert d_bias["regime"] == "HIGH_BIAS"
    assert "Add polynomial features" in d_bias["actionable_remedies"][0]
    
    # Case 2: High Variance (train score high, val score low)
    d_var = diagnose_model_regime(train_score=0.98, val_score=0.65, generalization_gap_threshold=0.15)
    assert d_var["regime"] == "HIGH_VARIANCE"
    assert "Collect more training data" in d_var["actionable_remedies"][0]
    
    # Case 3: Optimal Regime
    d_opt = diagnose_model_regime(train_score=0.92, val_score=0.89, generalization_gap_threshold=0.10)
    assert d_opt["regime"] == "OPTIMAL"
metadata.yml (660 bytes) 
lesson_id: D177
day: 177
kind: applied-ml-diagnostics
languages:
  - python
setup_commands:
  - python3 -m venv .venv
  - .venv/bin/pip install -r requirements/requirements.txt
run_commands:
  - .venv/bin/python examples/learning_curves_and_diagnostics_lib.py
test_commands:
  - ./tests/run_tests.sh
cleanup_commands:
  - rm -rf .venv __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: >-
  macOS (Apple Silicon, arm64, CPU only), Python 3.14.0, scikit-learn 1.9.0, pytest 9.1.1 -- bash tests/run_tests.sh -> 4 checks, 0 failure(s), exit 0. Verified Day 177 implementation.
requirements/requirements.txt (62 bytes) 
numpy>=1.24.0
scipy>=1.10.0
scikit-learn>=1.3.0
pytest>=7.4.0
starter/metric_diagnostics_lib.py (548 bytes) 
import numpy as np

def compute_learning_curves(estimator, X, y, train_sizes=None, cv=5, scoring="r2"):
    # TODO: Implement learning curve calculation across sample sizes
    pass

def compute_validation_curves(estimator, X, y, param_name, param_range, cv=5, scoring="r2"):
    # TODO: Implement validation curve calculation across hyperparameter range
    pass

def diagnose_model_regime(train_score, val_score, benchmark_score=0.90, generalization_gap_threshold=0.15):
    # TODO: Implement automated Bias vs Variance rule diagnostics
    pass
starter/test_metric_diagnostics_lib.py (2134 bytes) 
import pytest
import numpy as np
from sklearn.linear_model import Ridge
from sklearn.tree import DecisionTreeRegressor
from sklearn.datasets import make_regression
from metric_diagnostics_lib import (
    compute_learning_curves,
    compute_validation_curves,
    diagnose_model_regime
)

@pytest.fixture
def regression_data():
    X, y = make_regression(n_samples=200, n_features=10, noise=5.0, random_state=42)
    return X, y

def test_compute_learning_curves(regression_data):
    X, y = regression_data
    model = Ridge(alpha=1.0)
    res = compute_learning_curves(model, X, y, train_sizes=[0.2, 0.5, 1.0], cv=3)
    
    assert len(res["train_sizes"]) == 3
    assert len(res["train_scores_mean"]) == 3
    assert len(res["val_scores_mean"]) == 3
    # Training score typically starts high and stabilizes
    assert res["train_scores_mean"][0] >= res["train_scores_mean"][-1] - 0.20
    # Validation score typically increases with sample size
    assert res["val_scores_mean"][-1] > res["val_scores_mean"][0]

def test_compute_validation_curves(regression_data):
    X, y = regression_data
    model = DecisionTreeRegressor(random_state=42)
    depths = [1, 3, 6, 10]
    res = compute_validation_curves(model, X, y, param_name="max_depth", param_range=depths, cv=3)
    
    assert res["param_name"] == "max_depth"
    assert len(res["val_scores_mean"]) == 4

def test_diagnose_model_regime():
    # Case 1: High Bias (both train and val scores low)
    d_bias = diagnose_model_regime(train_score=0.45, val_score=0.42, benchmark_score=0.85)
    assert d_bias["regime"] == "HIGH_BIAS"
    assert "Add polynomial features" in d_bias["actionable_remedies"][0]
    
    # Case 2: High Variance (train score high, val score low)
    d_var = diagnose_model_regime(train_score=0.98, val_score=0.65, generalization_gap_threshold=0.15)
    assert d_var["regime"] == "HIGH_VARIANCE"
    assert "Collect more training data" in d_var["actionable_remedies"][0]
    
    # Case 3: Optimal Regime
    d_opt = diagnose_model_regime(train_score=0.92, val_score=0.89, generalization_gap_threshold=0.10)
    assert d_opt["regime"] == "OPTIMAL"
tests/run_tests.sh (36 bytes) 
#!/bin/bash
set -e
pytest tests/ -v

Troubleshooting

Troubleshooting Learning Curves

1. Flat Learning Curves

If validation score does not increase as training size increases, the model has hit a High Bias ceiling. Increasing dataset size will not improve performance; add features or increase model capacity.

2. Inverted Learning Curves (Train Score Below Validation)

Can occur when heavy regularization (e.g. Dropout, large L2 weight decay) is applied during training but disabled during validation evaluation.

Security notes

Security Considerations for Model Diagnostics

1. Data Integrity and Synthetic Probing

When evaluating diagnostics with synthetic feature probes, ensure no confidential training rows are exposed in diagnostic log files.

2. Leakage in Learning Curve Cross-Validation

Always ensure preprocessing transformers are cloned inside cross-validation splits during learning curve computation to avoid optimistic validation curve artifacts.