Sandeep Bazar 365 Days of AI
Labs

Deep Learning › Neural Network Foundations › Day 199

Hands-on lab — Day 199: Forward Propagation

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/forward_propagation_lib.py

Test

./tests/run_tests.sh

File tree

examples/forward_propagation_lib.py
examples/test_forward_propagation_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/forward_propagation_lib.py
starter/test_forward_propagation_lib.py
tests/run_tests.sh
tests/test_forward_propagation_lib.py
troubleshooting.md

Lab README

Lab: Day 199 -- Forward Propagation

Lesson

Day number: 199 of 365. Course: Course05-SS01 (Deep Learning - Neural Networks). Topic: Multi-Layer Forward Propagation and Activation Caching.

Purpose

Build a generalized, modular L-Layer Forward Propagation engine in pure NumPy. You will formulate linear affine transformations, manage activation functions across layers, enforce strict matrix dimension contracts, construct forward activation caches, and calculate mini-batch Categorical Cross-Entropy loss.

Learning objectives

  • Implement vectorized dense layer affine transformations Z = W A_prev + b.
  • Structure modular L-layer forward passes with activation caching.
  • Enforce strict matrix dimension contracts across mini-batches.
  • Calculate numerically stable Categorical Cross-Entropy (CCE) loss.

Prerequisites

  • Linear algebra (matrix multiplication, broadcasting).
  • Python 3.11+ with NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 512 MB RAM.
  • 50 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

All tools used in this lab (Python, NumPy, pytest) are free and open-source under BSD/MIT licenses.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/forward_propagation_lib.py: Student scaffold file.
  • examples/forward_propagation_lib.py: Complete reference implementation.
  • tests/test_forward_propagation_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/forward_propagation_lib.py

What the commands do

  • Constructs a 3-layer neural network [784, 128, 64, 10].
  • Executes forward propagation across a mini-batch of 32 samples.
  • Computes initial Categorical Cross-Entropy loss.

Expected output

Forward Demo: Output Shape = (10, 32), Initial CCE Loss = 2.3026

Validation steps

  1. Verify that output probabilities sum to 1.0 along the class axis for every sample.
  2. Confirm that forward caches store A_prev, Z, W, and b with correct dimensions.
  3. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Dimension Mismatch: Ensure input matrix has shape (features, batch_size).

Security notes

All tensor calculations execute locally in system RAM.

Extension exercises

  1. Implement Binary Cross-Entropy (BCE) loss evaluation.
  2. Build an inverted Dropout forward pass.
  • Lesson title: Forward Propagation
  • Day number: 199 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-199-forward-propagation
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-199-forward-propagation when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 199

- `Output Shape`: Shape of final output probability tensor.
- `Initial CCE Loss`: Categorical cross-entropy loss evaluated on random weights.

examples-run.txt

Forward Demo: Output Shape = (10, 32), Initial CCE Loss = 2.3026

measured-values.txt

Output Shape: (10, 32)
Initial CCE Loss: 2.3026

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_forward_propagation_lib.py::test_dense_layer_shapes_and_caching PASSED [ 33%]
tests/test_forward_propagation_lib.py::test_multilayer_network_forward_propagation PASSED [ 66%]
tests/test_forward_propagation_lib.py::test_categorical_crossentropy_loss PASSED [100%]

============================== 3 passed in 0.09s ===============================

Source files

examples/forward_propagation_lib.py (2569 bytes) 
import numpy as np
from typing import List, Tuple, Dict, Any

class DenseLayer:
    def __init__(self, in_features: int, out_features: int, activation: str = "relu"):
        self.in_features = in_features
        self.out_features = out_features
        self.activation = activation
        limit = np.sqrt(2.0 / in_features) if activation == "relu" else np.sqrt(1.0 / in_features)
        self.W = np.random.randn(out_features, in_features) * limit
        self.b = np.zeros((out_features, 1))

    def forward(self, A_prev: np.ndarray) -> Tuple[np.ndarray, Dict[str, np.ndarray]]:
        Z = np.dot(self.W, A_prev) + self.b

        if self.activation == "relu":
            A = np.maximum(0.0, Z)
        elif self.activation == "sigmoid":
            A = np.where(Z >= 0, 1.0 / (1.0 + np.exp(-Z)), np.exp(Z) / (1.0 + np.exp(Z)))
        elif self.activation == "softmax":
            Z_shift = Z - np.max(Z, axis=0, keepdims=True)
            exp_Z = np.exp(Z_shift)
            A = exp_Z / np.sum(exp_Z, axis=0, keepdims=True)
        else:
            A = Z

        cache = {"A_prev": A_prev, "Z": Z, "W": self.W, "b": self.b}
        return A, cache

class MultiLayerNetwork:
    def __init__(self, layer_dims: List[int], activations: List[str]):
        self.layer_dims = layer_dims
        self.activations = activations
        self.layers = []
        for i in range(len(layer_dims) - 1):
            self.layers.append(DenseLayer(layer_dims[i], layer_dims[i+1], activations[i]))

    def forward(self, X: np.ndarray) -> Tuple[np.ndarray, List[Dict[str, np.ndarray]]]:
        A = X
        caches = []
        for layer in self.layers:
            A, cache = layer.forward(A)
            caches.append(cache)
        return A, caches

    @staticmethod
    def compute_categorical_crossentropy(A_last: np.ndarray, Y_onehot: np.ndarray) -> float:
        m = Y_onehot.shape[1]
        eps = 1e-15
        loss = - (1.0 / m) * np.sum(Y_onehot * np.log(A_last + eps))
        return float(loss)

def run_forward_demo():
    np.random.seed(42)
    # Architecture: 784 -> 128 -> 64 -> 10
    net = MultiLayerNetwork([784, 128, 64, 10], ["relu", "relu", "softmax"])
    m = 32
    X = np.random.randn(784, m)
    Y = np.zeros((10, m))
    for i in range(m):
        Y[np.random.randint(0, 10), i] = 1.0

    A_out, caches = net.forward(X)
    loss = net.compute_categorical_crossentropy(A_out, Y)

    print(f"Forward Demo: Output Shape = {A_out.shape}, Initial CCE Loss = {loss:.4f}")
    return A_out, loss

if __name__ == "__main__":
    run_forward_demo()
examples/test_forward_propagation_lib.py (1306 bytes) 
import pytest
import numpy as np
from examples.forward_propagation_lib import DenseLayer, MultiLayerNetwork

def test_dense_layer_shapes_and_caching():
    layer = DenseLayer(in_features=20, out_features=10, activation="relu")
    m = 8
    A_prev = np.random.randn(20, m)
    A, cache = layer.forward(A_prev)

    assert A.shape == (10, m)
    assert cache["A_prev"].shape == (20, m)
    assert cache["Z"].shape == (10, m)
    assert cache["W"].shape == (10, 20)
    assert cache["b"].shape == (10, 1)

def test_multilayer_network_forward_propagation():
    net = MultiLayerNetwork([10, 16, 8, 4], ["relu", "relu", "softmax"])
    m = 12
    X = np.random.randn(10, m)
    A_out, caches = net.forward(X)

    assert A_out.shape == (4, m)
    assert len(caches) == 3
    # Check that softmax probabilities sum to 1.0 along class dimension (axis 0)
    assert np.allclose(np.sum(A_out, axis=0), np.ones(m))

def test_categorical_crossentropy_loss():
    m = 4
    # Perfect predictions
    Y_onehot = np.array([[1.0, 0.0, 0.0, 0.0],
                         [0.0, 1.0, 0.0, 0.0],
                         [0.0, 0.0, 1.0, 0.0],
                         [0.0, 0.0, 0.0, 1.0]])
    A_pred = Y_onehot.copy()
    loss = MultiLayerNetwork.compute_categorical_crossentropy(A_pred, Y_onehot)
    assert loss < 1e-5
metadata.yml (432 bytes) 
lesson_id: D199
day: 199
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/forward_propagation_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (28 bytes) 
numpy>=1.26.0
pytest>=8.0.0
starter/forward_propagation_lib.py (979 bytes) 
import numpy as np
from typing import List, Tuple, Dict, Any

class DenseLayer:
    def __init__(self, in_features: int, out_features: int, activation: str = "relu"):
        self.in_features = in_features
        self.out_features = out_features
        self.activation = activation
        self.W = None
        self.b = None

    def forward(self, A_prev: np.ndarray) -> Tuple[np.ndarray, Dict[str, np.ndarray]]:
        # TODO: Compute Z = W.A_prev + b, apply activation, and return (A, cache)
        pass

class MultiLayerNetwork:
    def __init__(self, layer_dims: List[int], activations: List[str]):
        # TODO: Initialize layers list
        pass

    def forward(self, X: np.ndarray) -> Tuple[np.ndarray, List[Dict[str, np.ndarray]]]:
        # TODO: Execute layer-by-layer forward propagation
        pass

    @staticmethod
    def compute_categorical_crossentropy(A_last: np.ndarray, Y_onehot: np.ndarray) -> float:
        # TODO: Compute CCE loss
        pass
starter/test_forward_propagation_lib.py (1306 bytes) 
import pytest
import numpy as np
from examples.forward_propagation_lib import DenseLayer, MultiLayerNetwork

def test_dense_layer_shapes_and_caching():
    layer = DenseLayer(in_features=20, out_features=10, activation="relu")
    m = 8
    A_prev = np.random.randn(20, m)
    A, cache = layer.forward(A_prev)

    assert A.shape == (10, m)
    assert cache["A_prev"].shape == (20, m)
    assert cache["Z"].shape == (10, m)
    assert cache["W"].shape == (10, 20)
    assert cache["b"].shape == (10, 1)

def test_multilayer_network_forward_propagation():
    net = MultiLayerNetwork([10, 16, 8, 4], ["relu", "relu", "softmax"])
    m = 12
    X = np.random.randn(10, m)
    A_out, caches = net.forward(X)

    assert A_out.shape == (4, m)
    assert len(caches) == 3
    # Check that softmax probabilities sum to 1.0 along class dimension (axis 0)
    assert np.allclose(np.sum(A_out, axis=0), np.ones(m))

def test_categorical_crossentropy_loss():
    m = 4
    # Perfect predictions
    Y_onehot = np.array([[1.0, 0.0, 0.0, 0.0],
                         [0.0, 1.0, 0.0, 0.0],
                         [0.0, 0.0, 1.0, 0.0],
                         [0.0, 0.0, 0.0, 1.0]])
    A_pred = Y_onehot.copy()
    loss = MultiLayerNetwork.compute_categorical_crossentropy(A_pred, Y_onehot)
    assert loss < 1e-5
tests/run_tests.sh (227 bytes) 
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 199 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_forward_propagation_lib.py (1306 bytes) 
import pytest
import numpy as np
from examples.forward_propagation_lib import DenseLayer, MultiLayerNetwork

def test_dense_layer_shapes_and_caching():
    layer = DenseLayer(in_features=20, out_features=10, activation="relu")
    m = 8
    A_prev = np.random.randn(20, m)
    A, cache = layer.forward(A_prev)

    assert A.shape == (10, m)
    assert cache["A_prev"].shape == (20, m)
    assert cache["Z"].shape == (10, m)
    assert cache["W"].shape == (10, 20)
    assert cache["b"].shape == (10, 1)

def test_multilayer_network_forward_propagation():
    net = MultiLayerNetwork([10, 16, 8, 4], ["relu", "relu", "softmax"])
    m = 12
    X = np.random.randn(10, m)
    A_out, caches = net.forward(X)

    assert A_out.shape == (4, m)
    assert len(caches) == 3
    # Check that softmax probabilities sum to 1.0 along class dimension (axis 0)
    assert np.allclose(np.sum(A_out, axis=0), np.ones(m))

def test_categorical_crossentropy_loss():
    m = 4
    # Perfect predictions
    Y_onehot = np.array([[1.0, 0.0, 0.0, 0.0],
                         [0.0, 1.0, 0.0, 0.0],
                         [0.0, 0.0, 1.0, 0.0],
                         [0.0, 0.0, 0.0, 1.0]])
    A_pred = Y_onehot.copy()
    loss = MultiLayerNetwork.compute_categorical_crossentropy(A_pred, Y_onehot)
    assert loss < 1e-5

Troubleshooting

Troubleshooting: Day 199 - Forward Propagation

Common Issues

  1. Matrix Incompatible Shapes:
    • Cause: Layer weights dimension does not match input activation dimension.
    • Fix: Ensure W has shape (out_features, in_features).

Security notes

Security & Privacy: Day 199 - Forward Propagation

Security Guidance

  • All forward activations execute strictly in local memory.