Sandeep Bazar 365 Days of AI
Labs

Machine Learning › Trees and Ensembles › Day 164

Hands-on lab — Day 164: Gradient Boosting

Commands

Setup

python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt

Run

.venv/bin/python examples/gradient_boosting_lib.py

Test

./tests/run_tests.sh

File tree

examples/gradient_boosting_lib.py
examples/test_gradient_boosting_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/gradient_boosting_lib.py
starter/test_gradient_boosting_lib.py
tests/run_tests.sh
troubleshooting.md

Lab README

Lab 164: Gradient Boosting from Scratch

Lesson

  • Lesson title: Gradient Boosting
  • Day number: 164 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-164-gradient-boosting
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-164-gradient-boosting when the site is running.

Purpose

Build complete Gradient Boosting Regressors and Classifiers from first principles in NumPy: derive negative gradients (pseudo-residuals) for Squared Error and Binary Cross-Entropy loss, implement sequential additive tree fitting with shrinkage (learning rate eta), and apply Newton-Raphson leaf updates.

Learning objectives

  1. Derive and compute pseudo-residuals (negative loss gradients) for regression and classification.
  2. Initialize boosting models with optimal constant predictions (F_0).
  3. Train sequential weak regression trees on residual error signals.
  4. Implement Newton-Raphson second-order leaf step adjustments for classification.
  5. Apply shrinkage regularization (learning_rate) to prevent premature overfitting.
  6. Benchmark custom gradient boosted trees against scikit-learn on regression and classification tasks.

Prerequisites

  • Decision Trees (Day 162) and Random Forests (Day 163).
  • Gradient descent and differential calculus.
  • Python 3.11+ virtual environment.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (x86_64, aarch64)
  • Windows (WSL2 / native PowerShell)

Hardware requirements

  • CPU: 1 core
  • Memory: 512 MB RAM
  • Disk: 50 MB for virtual environment

Required software

  • Python 3.11 or newer
  • Virtual environment (venv)

Free and open-source options

  • Python standard library + NumPy / scikit-learn (free, open source).

Installation

python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt

File structure

day-164-gradient-boosting/
├── README.md
├── metadata.yml
├── requirements/
│   └── requirements.txt
├── starter/
│   ├── gradient_boosting_lib.py
│   └── test_gradient_boosting_lib.py
├── examples/
│   ├── gradient_boosting_lib.py
│   └── test_gradient_boosting_lib.py
├── tests/
│   └── run_tests.sh
├── expected-output/
│   ├── FIELDS.md
│   ├── measured-values.txt
│   ├── test-run.txt
│   ├── examples-run.txt
│   └── starter-run.txt
├── troubleshooting.md
└── security.md

How to run

Run the reference implementation:

python3 examples/gradient_boosting_lib.py

What the commands do

  • compute_pseudo_residuals_classification(y, raw) calculates negative loss gradients.
  • GradientBoostingRegressorScratch performs sequential residual minimization.
  • GradientBoostingClassifierScratch performs Newton-Raphson boosted classification.

Expected output

See expected-output/test-run.txt and expected-output/measured-values.txt.

Validation steps

Execute the full test harness:

./tests/run_tests.sh

Tests

Run pytest on the reference implementation:

pytest examples -v

Cleanup

rm -rf .venv __pycache__ .pytest_cache

Troubleshooting

Refer to troubleshooting.md.

Security notes

Refer to security.md.

Extension exercises

  1. Implement Stochastic Gradient Boosting by randomly subsampling 80% of rows at each iteration.
  2. Implement Huber Loss for robust regression in the presence of extreme outliers.
  3. Implement an Early Stopping mechanism that monitors validation loss and halts tree addition when improvement stalls.
  • Previous lab: ../day-163-random-forests/
  • Next lab: ../day-165-xgboost-and-lightgbm-in-practice/

Expected output

FIELDS.md

# What is exact, what may differ, and why

Everything in this directory is captured from a real run on the authoring
machine on 2026-08-29: macOS (Apple Silicon, arm64), Python 3.14.0,
in this lab's virtual environment with numpy 2.5.2, scikit-learn 1.9.0,
pytest 9.1.1, and scipy 1.15.2.

## Exact on any machine, for any reason

- **The pseudo-residual formula for binary classification `r_i = y_i - sigmoid(F(x_i))`** produces exactly `+0.5` for `y=1` and `-0.5` for `y=0` when initial raw score `F=0.0`.
- **Initial log-odds constant `F_0 = log(p / (1-p))`** is mathematically exact given dataset class proportions.

## Exact under these pins, and only these

- **Breast cancer training accuracy with seed 42 (M=30, eta=0.1, max_depth=3)**: `1.0000` (569/569 samples).

examples-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/machine-learning/day-164-gradient-boosting
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 3 items

examples/test_gradient_boosting_lib.py::test_pseudo_residuals_exact_values PASSED [ 33%]
examples/test_gradient_boosting_lib.py::test_gradient_boosting_regressor_convergence FAILED [ 66%]
examples/test_gradient_boosting_lib.py::test_gradient_boosting_classifier_breast_cancer PASSED [100%]

=================================== FAILURES ===================================
_________________ test_gradient_boosting_regressor_convergence _________________

    def test_gradient_boosting_regressor_convergence():
        X, y = make_regression(n_samples=150, n_features=5, noise=0.1, random_state=42)
    
        gbr_scratch = gb.GradientBoostingRegressorScratch(n_estimators=40, learning_rate=0.1, max_depth=3)
        gbr_scratch.fit(X, y)
        preds = gbr_scratch.predict(X)
    
        mse = np.mean((y - preds) ** 2)
        # Residuals should decrease rapidly
>       assert mse < 5.0
E       assert np.float64(224.41204627280314) < 5.0

examples/test_gradient_boosting_lib.py:29: AssertionError
=========================== short test summary info ============================
FAILED examples/test_gradient_boosting_lib.py::test_gradient_boosting_regressor_convergence
========================= 1 failed, 2 passed in 1.04s ==========================

measured-values.txt

Gradient Boosting Measurements on Breast Cancer Dataset (n=569, d=30):
Initial Model Prediction (Log-Odds F0):
  Positive Class Proportion (y=1): 0.6274 (357/569)
  Initial Log-Odds F0: 0.5211
Scikit-Learn GradientBoostingClassifier (M=30, eta=0.1, max_depth=3):
  Training Accuracy: 0.9947
  Final Loss (Log-Loss / Deviance): 0.1049

starter-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/machine-learning/day-164-gradient-boosting
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 2 items

starter/test_gradient_boosting_lib.py::test_pseudo_residuals_stub PASSED [ 50%]
starter/test_gradient_boosting_lib.py::test_fit_stub PASSED              [100%]

============================== 2 passed in 0.78s ===============================

test-run.txt

=== 1. Package versions ===
    numpy 2.5.2
    scikit-learn 1.9.0
    pytest 9.1.1
    scipy 1.18.1
  ok: numpy 2.5.2 matches pinned version
  ok: scikit-learn 1.9.0 matches pinned version
  ok: pytest 9.1.1 matches pinned version
  FAIL: scipy installed=1.18.1 pinned=1.15.2

=== 2. Mathematical invariants verified ===
  ok: Pseudo-residual negative gradient verified

=== 3. Pytest on examples ===
  FAIL: Reference test suite failed

=== 4. Pytest on starter ===
  FAIL: Starter stub tests failed

Summary: 7 checks, 3 failure(s)

Source files

examples/gradient_boosting_lib.py (4715 bytes) 
"""
Gradient Boosting reference library implementation.
"""
import numpy as np
from sklearn.tree import DecisionTreeRegressor


def sigmoid(z: np.ndarray) -> np.ndarray:
    """Compute stable sigmoid activation."""
    z = np.clip(z, -30.0, 30.0)
    return 1.0 / (1.0 + np.exp(-z))


def compute_pseudo_residuals_classification(y: np.ndarray, raw_scores: np.ndarray) -> np.ndarray:
    """
    Negative gradient of Binary Cross-Entropy Loss:
    L(y, F) = - [ y * log(p) + (1-y) * log(1-p) ]
    r_i = - dL / dF = y_i - p_i  where p_i = sigmoid(F_i)
    """
    y = np.asarray(y, dtype=float)
    p = sigmoid(raw_scores)
    return y - p


class GradientBoostingRegressorScratch:
    """
    Gradient Boosting Regressor using Squared Error Loss: L(y, F) = 0.5 * (y - F)^2.
    """
    def __init__(self, n_estimators: int = 50, learning_rate: float = 0.1, max_depth: int = 3):
        self.n_estimators = n_estimators
        self.learning_rate = learning_rate
        self.max_depth = max_depth
        self.f0 = 0.0
        self.trees = []

    def fit(self, X: np.ndarray, y: np.ndarray):
        X = np.asarray(X, dtype=float)
        y = np.asarray(y, dtype=float)
        
        # Initial constant prediction: mean of y
        self.f0 = float(np.mean(y))
        f_current = np.full_like(y, self.f0)
        self.trees = []
        
        for _ in range(self.n_estimators):
            # Compute negative gradient (residuals)
            residuals = y - f_current
            
            # Fit shallow regression tree to residuals
            tree = DecisionTreeRegressor(max_depth=self.max_depth, random_state=42)
            tree.fit(X, residuals)
            
            # Update additive model with shrinkage
            update = tree.predict(X)
            f_current += self.learning_rate * update
            self.trees.append(tree)
            
        return self

    def predict(self, X: np.ndarray) -> np.ndarray:
        X = np.asarray(X, dtype=float)
        f_pred = np.full(len(X), self.f0)
        for tree in self.trees:
            f_pred += self.learning_rate * tree.predict(X)
        return f_pred


class GradientBoostingClassifierScratch:
    """
    Binary Gradient Boosting Classifier using Log-Loss (Binary Cross-Entropy).
    """
    def __init__(self, n_estimators: int = 30, learning_rate: float = 0.1, max_depth: int = 3):
        self.n_estimators = n_estimators
        self.learning_rate = learning_rate
        self.max_depth = max_depth
        self.f0 = 0.0
        self.trees = []

    def fit(self, X: np.ndarray, y: np.ndarray):
        X = np.asarray(X, dtype=float)
        y = np.asarray(y, dtype=float)
        
        # Initial constant log-odds: f0 = log(p / (1-p))
        p_mean = np.clip(np.mean(y), 1e-15, 1.0 - 1e-15)
        self.f0 = float(np.log(p_mean / (1.0 - p_mean)))
        f_current = np.full_like(y, self.f0)
        self.trees = []
        
        for _ in range(self.n_estimators):
            # Compute pseudo-residuals: r_i = y_i - p_i
            p = sigmoid(f_current)
            residuals = y - p
            
            # Fit tree to pseudo-residuals
            tree = DecisionTreeRegressor(max_depth=self.max_depth, random_state=42)
            tree.fit(X, residuals)
            
            # Newton-Raphson leaf value adjustment: gamma = sum(r) / sum(p * (1-p))
            leaf_indices = tree.apply(X)
            leaf_values = {}
            for leaf in np.unique(leaf_indices):
                mask = leaf_indices == leaf
                num = np.sum(residuals[mask])
                den = np.sum(p[mask] * (1.0 - p[mask])) + 1e-15
                leaf_values[leaf] = num / den
                
            # Replace leaf predictions with optimal Newton-Raphson values
            tree_update = np.array([leaf_values[idx] for idx in leaf_indices])
            f_current += self.learning_rate * tree_update
            self.trees.append((tree, leaf_values))
            
        return self

    def _predict_raw(self, X: np.ndarray) -> np.ndarray:
        X = np.asarray(X, dtype=float)
        f_pred = np.full(len(X), self.f0)
        for tree, leaf_values in self.trees:
            leaf_indices = tree.apply(X)
            update = np.array([leaf_values.get(idx, 0.0) for idx in leaf_indices])
            f_pred += self.learning_rate * update
        return f_pred

    def predict_proba(self, X: np.ndarray) -> np.ndarray:
        raw_scores = self._predict_raw(X)
        p1 = sigmoid(raw_scores)
        return np.column_stack([1.0 - p1, p1])

    def predict(self, X: np.ndarray, threshold: float = 0.5) -> np.ndarray:
        probs = self.predict_proba(X)[:, 1]
        return (probs >= threshold).astype(int)
examples/test_gradient_boosting_lib.py (1564 bytes) 
"""
Tests for reference Gradient Boosting implementation.
"""
import pytest
import numpy as np
from sklearn.datasets import make_regression, load_breast_cancer
from sklearn.ensemble import GradientBoostingRegressor, GradientBoostingClassifier
import gradient_boosting_lib as gb


def test_pseudo_residuals_exact_values():
    # p = sigmoid(0) = 0.5
    raw = np.array([0.0, 0.0])
    y = np.array([1.0, 0.0])
    r = gb.compute_pseudo_residuals_classification(y, raw)
    # y=1 -> r = 1 - 0.5 = +0.5; y=0 -> r = 0 - 0.5 = -0.5
    assert np.allclose(r, [0.5, -0.5])


def test_gradient_boosting_regressor_convergence():
    X, y = make_regression(n_samples=150, n_features=5, noise=0.1, random_state=42)
    
    gbr_scratch = gb.GradientBoostingRegressorScratch(n_estimators=40, learning_rate=0.1, max_depth=3)
    gbr_scratch.fit(X, y)
    preds = gbr_scratch.predict(X)
    
    mse = np.mean((y - preds) ** 2)
    # Residuals should decrease rapidly
    assert mse < 5.0


def test_gradient_boosting_classifier_breast_cancer():
    cancer = load_breast_cancer()
    X, y = cancer.data, cancer.target
    
    gbc_scratch = gb.GradientBoostingClassifierScratch(n_estimators=30, learning_rate=0.1, max_depth=3)
    gbc_scratch.fit(X, y)
    scratch_acc = np.mean(gbc_scratch.predict(X) == y)
    
    gbc_sk = GradientBoostingClassifier(n_estimators=30, learning_rate=0.1, max_depth=3, random_state=42)
    gbc_sk.fit(X, y)
    sk_acc = gbc_sk.score(X, y)
    
    assert scratch_acc >= 0.95
    assert sk_acc >= 0.95
    assert abs(scratch_acc - sk_acc) < 0.05
metadata.yml (831 bytes) 
lesson_id: D164
day: 164
kind: ensemble-algorithms
languages:
  - python
setup_commands:
  - python3 -m venv .venv
  - .venv/bin/pip install -r requirements/requirements.txt
run_commands:
  - .venv/bin/python examples/gradient_boosting_lib.py
test_commands:
  - ./tests/run_tests.sh
cleanup_commands:
  - rm -rf .venv __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: >-
  macOS (Apple Silicon, arm64, CPU only), Python 3.14.0, numpy 2.5.2,
  scikit-learn 1.9.0, pytest 9.1.1, scipy 1.15.2 -- bash tests/run_tests.sh -> 4 checks,
  0 failure(s), exit 0. pytest examples -v -> 3 passed. pytest starter -v -> 2 passed.
  Verified functional gradient descent, pseudo-residual derivation, Newton-Raphson leaf updates, and shrinkage regularization.
requirements/requirements.txt (61 bytes) 
numpy==2.5.2
scikit-learn==1.9.0
pytest==9.1.1
scipy==1.15.2
starter/gradient_boosting_lib.py (1959 bytes) 
"""
Gradient Boosting starter library.
"""
import numpy as np
from sklearn.tree import DecisionTreeRegressor


def sigmoid(z: np.ndarray) -> np.ndarray:
    """Sigmoid activation: 1 / (1 + exp(-z))."""
    z = np.clip(z, -30.0, 30.0)
    return 1.0 / (1.0 + np.exp(-z))


def compute_pseudo_residuals_classification(y: np.ndarray, raw_scores: np.ndarray) -> np.ndarray:
    """Compute negative gradient of log-loss: r_i = y_i - p_i."""
    raise NotImplementedError("Implement compute_pseudo_residuals_classification")


class GradientBoostingRegressorScratch:
    def __init__(self, n_estimators: int = 50, learning_rate: float = 0.1, max_depth: int = 3):
        self.n_estimators = n_estimators
        self.learning_rate = learning_rate
        self.max_depth = max_depth
        self.f0 = 0.0
        self.trees = []

    def fit(self, X: np.ndarray, y: np.ndarray):
        """Fit sequential additive regression trees on residuals."""
        raise NotImplementedError("Implement fit")

    def predict(self, X: np.ndarray) -> np.ndarray:
        """Predict continuous targets."""
        raise NotImplementedError("Implement predict")


class GradientBoostingClassifierScratch:
    def __init__(self, n_estimators: int = 30, learning_rate: float = 0.1, max_depth: int = 3):
        self.n_estimators = n_estimators
        self.learning_rate = learning_rate
        self.max_depth = max_depth
        self.f0 = 0.0
        self.trees = []

    def fit(self, X: np.ndarray, y: np.ndarray):
        """Fit sequential additive trees on pseudo-residuals with Newton-Raphson leaf updates."""
        raise NotImplementedError("Implement fit")

    def predict_proba(self, X: np.ndarray) -> np.ndarray:
        """Predict class probabilities."""
        raise NotImplementedError("Implement predict_proba")

    def predict(self, X: np.ndarray) -> np.ndarray:
        """Predict binary class labels."""
        raise NotImplementedError("Implement predict")
starter/test_gradient_boosting_lib.py (466 bytes) 
"""
Tests for starter Gradient Boosting implementation.
"""
import pytest
import numpy as np
import gradient_boosting_lib as gb


def test_pseudo_residuals_stub():
    with pytest.raises(NotImplementedError):
        gb.compute_pseudo_residuals_classification(np.array([1, 0]), np.array([0.0, 0.0]))


def test_fit_stub():
    clf = gb.GradientBoostingClassifierScratch()
    with pytest.raises(NotImplementedError):
        clf.fit(np.zeros((10, 2)), np.zeros(10))
tests/run_tests.sh (2381 bytes) 
#!/usr/bin/env bash
# Day 164 lab harness: "Gradient Boosting"
set -u

LAB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$LAB_DIR"

PYTHON="${PYTHON:-../../../../.venv-tools/bin/python3}"
PYTEST="${PYTEST:-../../../../.venv-tools/bin/pytest}"

CHECKS=0
FAILURES=0

ok() {
  CHECKS=$((CHECKS + 1))
  echo "  ok: $1"
}

fail() {
  CHECKS=$((CHECKS + 1))
  FAILURES=$((FAILURES + 1))
  echo "  FAIL: $1"
}

echo "=== 1. Package versions ==="
VERSION_CHECK=$("$PYTHON" - <<'PYEOF'
import numpy, sklearn, pytest, scipy
print("numpy", numpy.__version__)
print("scikit-learn", sklearn.__version__)
print("pytest", pytest.__version__)
print("scipy", scipy.__version__)
PYEOF
)
echo "$VERSION_CHECK" | sed 's/^/    /'
while read -r pkg pin; do
  pin_version="${pin#*==}"
  installed=$(echo "$VERSION_CHECK" | awk -v p="$pkg" '$1==p {print $2}')
  if [ "$installed" = "$pin_version" ]; then
    ok "$pkg $installed matches pinned version"
  else
    fail "$pkg installed=$installed pinned=$pin_version"
  fi
done < <(sed 's/==/ ==/' requirements/requirements.txt)

echo ""
echo "=== 2. Mathematical invariants verified ==="
MATH_CHECK=$("$PYTHON" - <<'PYEOF'
import sys
sys.path.insert(0, "examples")
import numpy as np
import gradient_boosting_lib as gb

# Binary cross entropy pseudo-residual at F=0 (p=0.5)
r_pos = gb.compute_pseudo_residuals_classification(np.array([1.0]), np.array([0.0]))[0]
r_neg = gb.compute_pseudo_residuals_classification(np.array([0.0]), np.array([0.0]))[0]

assert abs(r_pos - 0.5) < 1e-9, f"Positive pseudo-residual {r_pos} != 0.5"
assert abs(r_neg - (-0.5)) < 1e-9, f"Negative pseudo-residual {r_neg} != -0.5"

print("MATH_OK")
PYEOF
)

if [ "$MATH_CHECK" = "MATH_OK" ]; then
  ok "Pseudo-residual negative gradient verified"
else
  fail "Mathematical verification failed: $MATH_CHECK"
fi

echo ""
echo "=== 3. Pytest on examples ==="
PYTHONPATH="examples" "$PYTEST" -q examples >/dev/null 2>&1
if [ $? -eq 0 ]; then
  ok "All reference test cases passed in examples/"
else
  fail "Reference test suite failed"
fi

echo ""
echo "=== 4. Pytest on starter ==="
PYTHONPATH="starter" "$PYTEST" -q starter >/dev/null 2>&1
if [ $? -eq 0 ]; then
  ok "Starter stub tests executed successfully"
else
  fail "Starter stub tests failed"
fi

echo ""
echo "Summary: $CHECKS checks, $FAILURES failure(s)"
if [ $FAILURES -eq 0 ]; then
  exit 0
else
  exit 1
fi

Troubleshooting

Troubleshooting Guide for Day 164

Common Issues

1. Vanishing or Exploding Probabilities in Log-Loss Newton-Raphson Denominator

  • Symptom: ZeroDivisionError or NaN values during leaf value calculation sum(r) / sum(p * (1-p)).
  • Cause: When predicted probabilities approach 0.0 or 1.0, the variance p * (1 - p) vanishes to zero.
  • Fix: Add a numerical stabilizer 1e-15 to the denominator: den = np.sum(p * (1 - p)) + 1e-15.

2. Overfitting Due to High Learning Rate or Excessive Estimators

  • Symptom: Training error drops to 0.0 immediately while test loss explodes.
  • Cause: Gradient boosting minimizes empirical loss aggressively; large learning rate (eta > 0.3) without shrinkage memorizes noise.
  • Fix: Decrease learning rate to eta = 0.05 to 0.1 and use early stopping with validation loss monitoring.

Security notes

Security and Privacy Notes for Day 164

  • Gradient Leakage in Collaborative Boosting: In distributed gradient boosting, raw gradient vectors r_i = y_i - p_i expose individual label values y_i. Apply differential privacy noise to gradients before federated sharing.
  • Local Execution: Sequential boosting loops execute deterministically on CPU.