Deep Learning › Neural Network Foundations › Day 200

Hands-on lab — Day 200: Backpropagation

← Back to the Day 200 lesson
Open the hands-on files on GitHub — clone or download them from the public labs repository
Local path in your clone: labs/sections/deep-learning/day-200-backpropagation/

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/backpropagation_lib.py

Test

./tests/run_tests.sh

File tree

examples/backpropagation_lib.py
examples/test_backpropagation_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/backpropagation_lib.py
starter/test_backpropagation_lib.py
tests/run_tests.sh
tests/test_backpropagation_lib.py
troubleshooting.md

Lab README

Lab: Day 200 -- Backpropagation

Lesson

Day number: 200 of 365. Course: Course05-SS01 (Deep Learning - Neural Networks). Topic: Backpropagation and Gradient Checking.

Purpose

Build a fully vectorized, modular BackpropEngine in pure NumPy. You will derive and implement the four fundamental equations of backpropagation, propagate error residuals through activation layers, evaluate exact parameter gradients, and verify gradient accuracy using numerical gradient checking.

Learning objectives

Formulate and code output error residuals dZ^[L] = A^[L] - Y.
Implement upstream gradient propagation dA^[l-1] = (W^[l])^T dZ^[l].
Calculate parameter gradients dW^[l] and db^[l] across mini-batches.
Validate analytical gradients against numerical finite differences to 1e-7 tolerance.

Prerequisites

Multivariable calculus (partial derivatives, chain rule).
Python 3.11+ with NumPy.

Supported operating systems

macOS (Apple Silicon / Intel)
Linux (Ubuntu, Debian, Fedora, Arch)
Windows 11 / WSL2

Hardware requirements

1+ CPU cores.
512 MB RAM.
50 MB disk space.

Required software

Python 3.11 or newer.
pip package manager.
virtualenv or venv module.

Free and open-source options

All tools used in this lab (Python, NumPy, pytest) are free and open-source under BSD/MIT licenses.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

starter/backpropagation_lib.py: Student scaffold file.
examples/backpropagation_lib.py: Complete reference implementation.
tests/test_backpropagation_lib.py: Pytest automated validation suite.
expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/backpropagation_lib.py

What the commands do

Executes a 2-layer network forward pass.
Sweeps backward computing exact gradients dW and db.
Validates parameter shapes and gradient checks.

Expected output

Backprop Demo: dW2 Shape = (3, 16), dW1 Shape = (16, 8)

Validation steps

Verify that dW and db shapes match parameter matrices for all layers.
Confirm that relative error between analytical and numerical gradients is less than 1e-6.
Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

Gradient Shape Inversion: Ensure np.dot(dZ, A_prev.T) is used for weight gradients.

Security notes

All derivative calculus executes locally without network access.

Extension exercises

Implement L2 Weight Regularization gradients.
Code backward propagation for Leaky ReLU and Tanh.

Lesson title: Backpropagation
Day number: 200 of 365
Lesson article: https://ai-roadmap-365.github.io/day-200-backpropagation
Lab files: everything you need is in this directory — follow “How to run” below.
Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-200-backpropagation when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 200

- `dW2 Shape`: Dimensions of layer 2 weight gradient.
- `dW1 Shape`: Dimensions of layer 1 weight gradient.

examples-run.txt

Backprop Demo: dW2 Shape = (3, 16), dW1 Shape = (16, 8)

measured-values.txt

dW2 Shape: (3, 16)
dW1 Shape: (16, 8)

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_backpropagation_lib.py::test_linear_and_relu_backward PASSED          [ 33%]
tests/test_backpropagation_lib.py::test_full_backpropagation_gradient_shapes PASSED [ 66%]
tests/test_backpropagation_lib.py::test_numerical_gradient_check PASSED          [100%]

============================== 3 passed in 0.12s ===============================

Source files

examples/backpropagation_lib.py (3359 bytes)

import numpy as np
from typing import List, Dict, Tuple, Any

class BackpropEngine:
    @staticmethod
    def relu_backward(dA: np.ndarray, Z: np.ndarray) -> np.ndarray:
        dZ = np.array(dA, copy=True)
        dZ[Z <= 0.0] = 0.0
        return dZ

    @staticmethod
    def tanh_backward(dA: np.ndarray, Z: np.ndarray) -> np.ndarray:
        return dA * (1.0 - np.tanh(Z) ** 2)

    @staticmethod
    def linear_backward(dZ: np.ndarray, cache: Dict[str, np.ndarray]) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
        A_prev = cache["A_prev"]
        W = cache["W"]
        b = cache["b"]
        m = A_prev.shape[1]

        dW = (1.0 / m) * np.dot(dZ, A_prev.T)
        db = (1.0 / m) * np.sum(dZ, axis=1, keepdims=True)
        dA_prev = np.dot(W.T, dZ)

        return dA_prev, dW, db

    @staticmethod
    def full_backward_pass(A_last: np.ndarray, Y: np.ndarray, caches: List[Dict[str, np.ndarray]], activations: List[str]) -> Dict[str, np.ndarray]:
        grads = {}
        L = len(caches)

        dZ_last = A_last - Y
        dA_prev, dW_last, db_last = BackpropEngine.linear_backward(dZ_last, caches[L-1])
        grads[f"dW{L}"] = dW_last
        grads[f"db{L}"] = db_last
        dA = dA_prev

        for l in reversed(range(L - 1)):
            cache = caches[l]
            act = activations[l]
            Z = cache["Z"]

            if act == "relu":
                dZ = BackpropEngine.relu_backward(dA, Z)
            elif act == "tanh":
                dZ = BackpropEngine.tanh_backward(dA, Z)
            else:
                dZ = dA

            dA_prev, dW, db = BackpropEngine.linear_backward(dZ, cache)
            grads[f"dW{l+1}"] = dW
            grads[f"db{l+1}"] = db
            dA = dA_prev

        return grads

def gradient_check_layer(W: np.ndarray, dW_analytical: np.ndarray, forward_fn, eps: float = 1e-5) -> float:
    # Compute numerical gradient for a weight matrix
    dW_num = np.zeros_like(W)
    for i in range(W.shape[0]):
        for j in range(W.shape[1]):
            orig = W[i, j]
            W[i, j] = orig + eps
            loss_plus = forward_fn()
            W[i, j] = orig - eps
            loss_minus = forward_fn()
            W[i, j] = orig
            dW_num[i, j] = (loss_plus - loss_minus) / (2.0 * eps)

    num = np.linalg.norm(dW_analytical - dW_num)
    den = np.linalg.norm(dW_analytical) + np.linalg.norm(dW_num) + 1e-8
    return float(num / den)

def run_backprop_demo():
    np.random.seed(42)
    m = 16
    X = np.random.randn(8, m)
    Y = np.zeros((3, m))
    for i in range(m):
        Y[np.random.randint(0, 3), i] = 1.0

    # Layer 1: 8 -> 16
    W1 = np.random.randn(16, 8) * 0.1
    b1 = np.zeros((16, 1))
    Z1 = np.dot(W1, X) + b1
    A1 = np.maximum(0.0, Z1)

    # Layer 2: 16 -> 3
    W2 = np.random.randn(3, 16) * 0.1
    b2 = np.zeros((3, 1))
    Z2 = np.dot(W2, A1) + b2
    exp_Z2 = np.exp(Z2 - np.max(Z2, axis=0, keepdims=True))
    A2 = exp_Z2 / np.sum(exp_Z2, axis=0, keepdims=True)

    caches = [
        {"A_prev": X, "Z": Z1, "W": W1, "b": b1},
        {"A_prev": A1, "Z": Z2, "W": W2, "b": b2}
    ]
    grads = BackpropEngine.full_backward_pass(A2, Y, caches, ["relu", "softmax"])

    print(f"Backprop Demo: dW2 Shape = {grads['dW2'].shape}, dW1 Shape = {grads['dW1'].shape}")
    return grads

if __name__ == "__main__":
    run_backprop_demo()

examples/test_backpropagation_lib.py (2352 bytes)

import pytest
import numpy as np
from examples.backpropagation_lib import BackpropEngine, gradient_check_layer

def test_linear_and_relu_backward():
    m = 4
    A_prev = np.random.randn(5, m)
    W = np.random.randn(3, 5) * 0.1
    b = np.zeros((3, 1))
    Z = np.dot(W, A_prev) + b
    dA = np.random.randn(3, m)

    dZ = BackpropEngine.relu_backward(dA, Z)
    assert dZ.shape == (3, m)
    # Check that where Z <= 0, dZ is strictly 0.0
    assert np.all(dZ[Z <= 0.0] == 0.0)

    cache = {"A_prev": A_prev, "W": W, "b": b, "Z": Z}
    dA_prev, dW, db = BackpropEngine.linear_backward(dZ, cache)
    assert dW.shape == (3, 5)
    assert db.shape == (3, 1)
    assert dA_prev.shape == (5, m)

def test_full_backpropagation_gradient_shapes():
    m = 8
    X = np.random.randn(6, m)
    Y = np.zeros((2, m))
    Y[0, :] = 1.0

    W1 = np.random.randn(4, 6) * 0.1
    b1 = np.zeros((4, 1))
    Z1 = np.dot(W1, X) + b1
    A1 = np.maximum(0.0, Z1)

    W2 = np.random.randn(2, 4) * 0.1
    b2 = np.zeros((2, 1))
    Z2 = np.dot(W2, A1) + b2
    exp_Z2 = np.exp(Z2 - np.max(Z2, axis=0, keepdims=True))
    A2 = exp_Z2 / np.sum(exp_Z2, axis=0, keepdims=True)

    caches = [
        {"A_prev": X, "Z": Z1, "W": W1, "b": b1},
        {"A_prev": A1, "Z": Z2, "W": W2, "b": b2}
    ]
    grads = BackpropEngine.full_backward_pass(A2, Y, caches, ["relu", "softmax"])

    assert grads["dW1"].shape == (4, 6)
    assert grads["db1"].shape == (4, 1)
    assert grads["dW2"].shape == (2, 4)
    assert grads["db2"].shape == (2, 1)

def test_numerical_gradient_check():
    np.random.seed(123)
    m = 10
    X = np.random.randn(4, m)
    Y = np.zeros((2, m))
    Y[0, :] = 1.0

    W = np.random.randn(2, 4) * 0.1
    b = np.zeros((2, 1))
    Z = np.dot(W, X) + b
    exp_Z = np.exp(Z - np.max(Z, axis=0, keepdims=True))
    A = exp_Z / np.sum(exp_Z, axis=0, keepdims=True)

    cache = {"A_prev": X, "Z": Z, "W": W, "b": b}
    grads = BackpropEngine.full_backward_pass(A, Y, [cache], ["softmax"])

    def forward_loss():
        Z_cur = np.dot(W, X) + b
        exp_cur = np.exp(Z_cur - np.max(Z_cur, axis=0, keepdims=True))
        A_cur = exp_cur / np.sum(exp_cur, axis=0, keepdims=True)
        return - (1.0 / m) * np.sum(Y * np.log(A_cur + 1e-15))

    rel_err = gradient_check_layer(W, grads["dW1"], forward_loss, eps=1e-5)
    assert rel_err < 1e-6

metadata.yml (428 bytes)

lesson_id: D200
day: 200
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/backpropagation_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'

requirements/requirements.txt (28 bytes)

numpy>=1.26.0
pytest>=8.0.0

starter/backpropagation_lib.py (715 bytes)

import numpy as np
from typing import List, Dict, Tuple, Any

class BackpropEngine:
    @staticmethod
    def linear_backward(dZ: np.ndarray, cache: Dict[str, np.ndarray]) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
        # TODO: Implement dW = (1/m) dZ A_prev.T, db = (1/m) sum(dZ), dA_prev = W.T dZ
        pass

    @staticmethod
    def relu_backward(dA: np.ndarray, Z: np.ndarray) -> np.ndarray:
        # TODO: Implement ReLU backward: dA where Z > 0 else 0
        pass

    @staticmethod
    def full_backward_pass(A_last: np.ndarray, Y: np.ndarray, caches: List[Dict[str, np.ndarray]], activations: List[str]) -> Dict[str, np.ndarray]:
        # TODO: Execute full L-layer backpropagation
        pass

starter/test_backpropagation_lib.py (2352 bytes)

import pytest
import numpy as np
from examples.backpropagation_lib import BackpropEngine, gradient_check_layer

def test_linear_and_relu_backward():
    m = 4
    A_prev = np.random.randn(5, m)
    W = np.random.randn(3, 5) * 0.1
    b = np.zeros((3, 1))
    Z = np.dot(W, A_prev) + b
    dA = np.random.randn(3, m)

    dZ = BackpropEngine.relu_backward(dA, Z)
    assert dZ.shape == (3, m)
    # Check that where Z <= 0, dZ is strictly 0.0
    assert np.all(dZ[Z <= 0.0] == 0.0)

    cache = {"A_prev": A_prev, "W": W, "b": b, "Z": Z}
    dA_prev, dW, db = BackpropEngine.linear_backward(dZ, cache)
    assert dW.shape == (3, 5)
    assert db.shape == (3, 1)
    assert dA_prev.shape == (5, m)

def test_full_backpropagation_gradient_shapes():
    m = 8
    X = np.random.randn(6, m)
    Y = np.zeros((2, m))
    Y[0, :] = 1.0

    W1 = np.random.randn(4, 6) * 0.1
    b1 = np.zeros((4, 1))
    Z1 = np.dot(W1, X) + b1
    A1 = np.maximum(0.0, Z1)

    W2 = np.random.randn(2, 4) * 0.1
    b2 = np.zeros((2, 1))
    Z2 = np.dot(W2, A1) + b2
    exp_Z2 = np.exp(Z2 - np.max(Z2, axis=0, keepdims=True))
    A2 = exp_Z2 / np.sum(exp_Z2, axis=0, keepdims=True)

    caches = [
        {"A_prev": X, "Z": Z1, "W": W1, "b": b1},
        {"A_prev": A1, "Z": Z2, "W": W2, "b": b2}
    ]
    grads = BackpropEngine.full_backward_pass(A2, Y, caches, ["relu", "softmax"])

    assert grads["dW1"].shape == (4, 6)
    assert grads["db1"].shape == (4, 1)
    assert grads["dW2"].shape == (2, 4)
    assert grads["db2"].shape == (2, 1)

def test_numerical_gradient_check():
    np.random.seed(123)
    m = 10
    X = np.random.randn(4, m)
    Y = np.zeros((2, m))
    Y[0, :] = 1.0

    W = np.random.randn(2, 4) * 0.1
    b = np.zeros((2, 1))
    Z = np.dot(W, X) + b
    exp_Z = np.exp(Z - np.max(Z, axis=0, keepdims=True))
    A = exp_Z / np.sum(exp_Z, axis=0, keepdims=True)

    cache = {"A_prev": X, "Z": Z, "W": W, "b": b}
    grads = BackpropEngine.full_backward_pass(A, Y, [cache], ["softmax"])

    def forward_loss():
        Z_cur = np.dot(W, X) + b
        exp_cur = np.exp(Z_cur - np.max(Z_cur, axis=0, keepdims=True))
        A_cur = exp_cur / np.sum(exp_cur, axis=0, keepdims=True)
        return - (1.0 / m) * np.sum(Y * np.log(A_cur + 1e-15))

    rel_err = gradient_check_layer(W, grads["dW1"], forward_loss, eps=1e-5)
    assert rel_err < 1e-6

tests/run_tests.sh (227 bytes)

#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 200 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."

tests/test_backpropagation_lib.py (2352 bytes)

import pytest
import numpy as np
from examples.backpropagation_lib import BackpropEngine, gradient_check_layer

def test_linear_and_relu_backward():
    m = 4
    A_prev = np.random.randn(5, m)
    W = np.random.randn(3, 5) * 0.1
    b = np.zeros((3, 1))
    Z = np.dot(W, A_prev) + b
    dA = np.random.randn(3, m)

    dZ = BackpropEngine.relu_backward(dA, Z)
    assert dZ.shape == (3, m)
    # Check that where Z <= 0, dZ is strictly 0.0
    assert np.all(dZ[Z <= 0.0] == 0.0)

    cache = {"A_prev": A_prev, "W": W, "b": b, "Z": Z}
    dA_prev, dW, db = BackpropEngine.linear_backward(dZ, cache)
    assert dW.shape == (3, 5)
    assert db.shape == (3, 1)
    assert dA_prev.shape == (5, m)

def test_full_backpropagation_gradient_shapes():
    m = 8
    X = np.random.randn(6, m)
    Y = np.zeros((2, m))
    Y[0, :] = 1.0

    W1 = np.random.randn(4, 6) * 0.1
    b1 = np.zeros((4, 1))
    Z1 = np.dot(W1, X) + b1
    A1 = np.maximum(0.0, Z1)

    W2 = np.random.randn(2, 4) * 0.1
    b2 = np.zeros((2, 1))
    Z2 = np.dot(W2, A1) + b2
    exp_Z2 = np.exp(Z2 - np.max(Z2, axis=0, keepdims=True))
    A2 = exp_Z2 / np.sum(exp_Z2, axis=0, keepdims=True)

    caches = [
        {"A_prev": X, "Z": Z1, "W": W1, "b": b1},
        {"A_prev": A1, "Z": Z2, "W": W2, "b": b2}
    ]
    grads = BackpropEngine.full_backward_pass(A2, Y, caches, ["relu", "softmax"])

    assert grads["dW1"].shape == (4, 6)
    assert grads["db1"].shape == (4, 1)
    assert grads["dW2"].shape == (2, 4)
    assert grads["db2"].shape == (2, 1)

def test_numerical_gradient_check():
    np.random.seed(123)
    m = 10
    X = np.random.randn(4, m)
    Y = np.zeros((2, m))
    Y[0, :] = 1.0

    W = np.random.randn(2, 4) * 0.1
    b = np.zeros((2, 1))
    Z = np.dot(W, X) + b
    exp_Z = np.exp(Z - np.max(Z, axis=0, keepdims=True))
    A = exp_Z / np.sum(exp_Z, axis=0, keepdims=True)

    cache = {"A_prev": X, "Z": Z, "W": W, "b": b}
    grads = BackpropEngine.full_backward_pass(A, Y, [cache], ["softmax"])

    def forward_loss():
        Z_cur = np.dot(W, X) + b
        exp_cur = np.exp(Z_cur - np.max(Z_cur, axis=0, keepdims=True))
        A_cur = exp_cur / np.sum(exp_cur, axis=0, keepdims=True)
        return - (1.0 / m) * np.sum(Y * np.log(A_cur + 1e-15))

    rel_err = gradient_check_layer(W, grads["dW1"], forward_loss, eps=1e-5)
    assert rel_err < 1e-6

Troubleshooting

Troubleshooting: Day 200 - Backpropagation

Common Issues

Gradient Dimension Mismatch:
- Cause: Inverted matrix multiplication during backward step.
- Fix: Ensure dW is (dZ @ A_prev.T) and dA_prev is (W.T @ dZ).

Security notes

Security & Privacy: Day 200 - Backpropagation

Security Guidance

All gradient operations run in memory without external data leaks.

Hands-on lab — Day 200: Backpropagation

Commands

Setup

Run

Test

File tree

Lab README

Lab: Day 200 -- Backpropagation

Lesson

Purpose

Learning objectives

Prerequisites

Supported operating systems

Hardware requirements

Required software

Free and open-source options

Installation

File structure

How to run

What the commands do

Expected output

Validation steps

Tests

Cleanup

Troubleshooting

Security notes

Extension exercises

Navigation

Expected output

FIELDS.md

examples-run.txt

measured-values.txt

starter-run.txt

test-run.txt

Source files

Troubleshooting

Troubleshooting: Day 200 - Backpropagation

Common Issues

Security notes

Security & Privacy: Day 200 - Backpropagation

Security Guidance