Sandeep Bazar 365 Days of AI
Labs

Deep Learning › Neural Network Foundations › Day 201

Hands-on lab — Day 201: A Neural Network in Pure NumPy

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/a_neural_network_in_pure_numpy_lib.py

Test

./tests/run_tests.sh

File tree

examples/a_neural_network_in_pure_numpy_lib.py
examples/test_a_neural_network_in_pure_numpy_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/a_neural_network_in_pure_numpy_lib.py
starter/test_a_neural_network_in_pure_numpy_lib.py
tests/run_tests.sh
tests/test_a_neural_network_in_pure_numpy_lib.py
troubleshooting.md

Lab README

Lab: Day 201 -- A Neural Network in Pure NumPy

Lesson

Day number: 201 of 365. Course: Course05-SS01 (Deep Learning - Neural Networks). Topic: Complete Neural Network Engine in Pure NumPy.

Purpose

Build a complete, standalone, object-oriented NeuralNetwork in pure NumPy. You will integrate He parameter initialization, forward caching, analytical backpropagation, mini-batch shuffling, and SGD with Momentum, training the network to solve complex non-linear classification manifolds (Two Moons).

Learning objectives

  • Implement He and Xavier parameter initialization breaking symmetry.
  • Integrate modular forward passes, activation functions, and backpropagation.
  • Code Mini-Batch SGD with Momentum parameter update dynamics.
  • Train the network on non-linear datasets and achieve > 90% classification accuracy.

Prerequisites

  • Days 197-200 (Perceptron, Activations, Forward Prop, Backprop).
  • Python 3.11+ with NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 512 MB RAM.
  • 50 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

All tools used in this lab (Python, NumPy, pytest) are free and open-source under BSD/MIT licenses.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/a_neural_network_in_pure_numpy_lib.py: Student scaffold file.
  • examples/a_neural_network_in_pure_numpy_lib.py: Complete reference implementation.
  • tests/test_a_neural_network_in_pure_numpy_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/a_neural_network_in_pure_numpy_lib.py

What the commands do

  • Generates synthetic Two Moons dataset.
  • Trains a 3-layer neural network [2, 16, 8, 2].
  • Evaluates classification accuracy and loss convergence.

Expected output

Pure NumPy Demo: Final Loss = 0.0245, Accuracy = 99.2%

Validation steps

  1. Verify that training loss decreases across epochs.
  2. Confirm that final classification accuracy exceeds 90% on the Two Moons dataset.
  3. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Accuracy Stuck at 50%: Ensure learning rate is sufficiently large (lr >= 0.05) and activations are non-linear.

Security notes

All training runs locally on CPU memory without external telemetric transmissions.

Extension exercises

  1. Implement Learning Rate Decay.
  2. Code the Adam optimization algorithm in pure NumPy.

Expected output

FIELDS.md

# Expected Output Fields: Day 201

- `Final Loss`: Average cross-entropy loss on final training epoch.
- `Accuracy`: Percentage of correctly classified test samples.

examples-run.txt

Pure NumPy Demo: Final Loss = 0.0245, Accuracy = 99.2%

measured-values.txt

Final Loss: 0.0245
Accuracy: 0.9920

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_a_neural_network_in_pure_numpy_lib.py::test_neural_network_he_initialization PASSED [ 33%]
tests/test_a_neural_network_in_pure_numpy_lib.py::test_neural_network_trains_and_reduces_loss PASSED [ 66%]
tests/test_a_neural_network_in_pure_numpy_lib.py::test_neural_network_achieves_high_accuracy_on_moons PASSED [100%]

============================== 3 passed in 0.15s ===============================

Source files

examples/a_neural_network_in_pure_numpy_lib.py (5573 bytes) 
import numpy as np
from typing import List, Tuple, Dict, Any

class NeuralNetwork:
    def __init__(self, layer_dims: List[int], activations: List[str], learning_rate: float = 0.05, momentum: float = 0.9):
        self.layer_dims = layer_dims
        self.activations = activations
        self.lr = learning_rate
        self.beta = momentum
        self.L = len(layer_dims) - 1

        self.params = {}
        self.velocities = {}
        self._initialize_parameters()

    def _initialize_parameters(self):
        np.random.seed(42)
        for l in range(1, self.L + 1):
            n_in = self.layer_dims[l-1]
            n_out = self.layer_dims[l]
            std = np.sqrt(2.0 / n_in) if self.activations[l-1] == "relu" else np.sqrt(1.0 / n_in)
            self.params[f"W{l}"] = np.random.randn(n_out, n_in) * std
            self.params[f"b{l}"] = np.zeros((n_out, 1))

            self.velocities[f"V_dW{l}"] = np.zeros_like(self.params[f"W{l}"])
            self.velocities[f"V_db{l}"] = np.zeros_like(self.params[f"b{l}"])

    def forward(self, X: np.ndarray) -> Tuple[np.ndarray, List[Dict[str, np.ndarray]]]:
        A = X
        caches = []
        for l in range(1, self.L + 1):
            W = self.params[f"W{l}"]
            b = self.params[f"b{l}"]
            act = self.activations[l-1]

            Z = np.dot(W, A) + b
            if act == "relu":
                A_next = np.maximum(0.0, Z)
            elif act == "softmax":
                Z_shift = Z - np.max(Z, axis=0, keepdims=True)
                exp_Z = np.exp(Z_shift)
                A_next = exp_Z / np.sum(exp_Z, axis=0, keepdims=True)
            elif act == "sigmoid":
                A_next = np.where(Z >= 0, 1.0 / (1.0 + np.exp(-Z)), np.exp(Z) / (1.0 + np.exp(Z)))
            else:
                A_next = Z

            caches.append({"A_prev": A, "Z": Z, "W": W, "b": b})
            A = A_next

        return A, caches

    def backward(self, A_last: np.ndarray, Y: np.ndarray, caches: List[Dict[str, np.ndarray]]) -> Dict[str, np.ndarray]:
        grads = {}
        m = A_last.shape[1]

        dZ = A_last - Y
        for l in reversed(range(1, self.L + 1)):
            cache = caches[l-1]
            A_prev = cache["A_prev"]
            W = cache["W"]
            Z = cache["Z"]

            if l < self.L:
                act = self.activations[l-1]
                if act == "relu":
                    dZ = dA * np.where(Z > 0.0, 1.0, 0.0)
                elif act == "tanh":
                    dZ = dA * (1.0 - np.tanh(Z) ** 2)

            grads[f"dW{l}"] = (1.0 / m) * np.dot(dZ, A_prev.T)
            grads[f"db{l}"] = (1.0 / m) * np.sum(dZ, axis=1, keepdims=True)
            dA = np.dot(W.T, dZ)

        return grads

    def update_parameters(self, grads: Dict[str, np.ndarray]):
        for l in range(1, self.L + 1):
            self.velocities[f"V_dW{l}"] = self.beta * self.velocities[f"V_dW{l}"] + (1.0 - self.beta) * grads[f"dW{l}"]
            self.velocities[f"V_db{l}"] = self.beta * self.velocities[f"V_db{l}"] + (1.0 - self.beta) * grads[f"db{l}"]

            self.params[f"W{l}"] -= self.lr * self.velocities[f"V_dW{l}"]
            self.params[f"b{l}"] -= self.lr * self.velocities[f"V_db{l}"]

    def fit(self, X: np.ndarray, Y: np.ndarray, epochs: int = 100, batch_size: int = 32) -> List[float]:
        m = X.shape[1]
        loss_history = []

        for epoch in range(epochs):
            permutation = np.random.permutation(m)
            X_shuffled = X[:, permutation]
            Y_shuffled = Y[:, permutation]

            num_batches = int(np.ceil(m / batch_size))
            epoch_loss = 0.0

            for b in range(num_batches):
                start = b * batch_size
                end = min(start + batch_size, m)
                X_batch = X_shuffled[:, start:end]
                Y_batch = Y_shuffled[:, start:end]

                A_out, caches = self.forward(X_batch)
                batch_loss = - (1.0 / (end - start)) * np.sum(Y_batch * np.log(A_out + 1e-15))
                epoch_loss += batch_loss * (end - start)

                grads = self.backward(A_out, Y_batch, caches)
                self.update_parameters(grads)

            loss_history.append(float(epoch_loss / m))

        return loss_history

    def predict(self, X: np.ndarray) -> np.ndarray:
        A_out, _ = self.forward(X)
        return np.argmax(A_out, axis=0)

def generate_two_moons(n_samples: int = 400) -> Tuple[np.ndarray, np.ndarray]:
    np.random.seed(42)
    n = n_samples // 2
    theta = np.linspace(0, np.pi, n)
    moon1_x = np.cos(theta) + np.random.randn(n) * 0.08
    moon1_y = np.sin(theta) + np.random.randn(n) * 0.08

    moon2_x = 1 - np.cos(theta) + np.random.randn(n) * 0.08
    moon2_y = 1 - np.sin(theta) - 0.5 + np.random.randn(n) * 0.08

    X = np.vstack([np.hstack([moon1_x, moon2_x]), np.hstack([moon1_y, moon2_y])])
    y = np.hstack([np.zeros(n, dtype=int), np.ones(n, dtype=int)])

    Y_onehot = np.zeros((2, n_samples))
    for i in range(n_samples):
        Y_onehot[y[i], i] = 1.0

    return X, Y_onehot, y

def run_pure_numpy_demo():
    X, Y_onehot, y_true = generate_two_moons(n_samples=300)
    net = NeuralNetwork([2, 16, 8, 2], ["relu", "relu", "softmax"], learning_rate=0.1, momentum=0.9)
    losses = net.fit(X, Y_onehot, epochs=60, batch_size=32)

    preds = net.predict(X)
    acc = float(np.mean(preds == y_true))

    print(f"Pure NumPy Demo: Final Loss = {losses[-1]:.4f}, Accuracy = {acc * 100:.1f}%")
    return net, acc

if __name__ == "__main__":
    run_pure_numpy_demo()
examples/test_a_neural_network_in_pure_numpy_lib.py (1234 bytes) 
import pytest
import numpy as np
from examples.a_neural_network_in_pure_numpy_lib import NeuralNetwork, generate_two_moons

def test_neural_network_he_initialization():
    net = NeuralNetwork([2, 20, 10, 2], ["relu", "relu", "softmax"])
    assert net.params["W1"].shape == (20, 2)
    assert net.params["b1"].shape == (20, 1)
    assert net.params["W2"].shape == (10, 20)
    assert net.params["W3"].shape == (2, 10)
    # Check that weights are not all zeros
    assert np.count_nonzero(net.params["W1"]) > 0

def test_neural_network_trains_and_reduces_loss():
    X, Y_onehot, _ = generate_two_moons(n_samples=200)
    net = NeuralNetwork([2, 16, 8, 2], ["relu", "relu", "softmax"], learning_rate=0.1)
    losses = net.fit(X, Y_onehot, epochs=50, batch_size=32)

    assert len(losses) == 50
    # Check that loss decreased significantly
    assert losses[-1] < losses[0]

def test_neural_network_achieves_high_accuracy_on_moons():
    X, Y_onehot, y_true = generate_two_moons(n_samples=300)
    net = NeuralNetwork([2, 32, 16, 2], ["relu", "relu", "softmax"], learning_rate=0.1, momentum=0.9)
    net.fit(X, Y_onehot, epochs=80, batch_size=32)

    preds = net.predict(X)
    acc = np.mean(preds == y_true)
    assert acc > 0.90
metadata.yml (443 bytes) 
lesson_id: D201
day: 201
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/a_neural_network_in_pure_numpy_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (28 bytes) 
numpy>=1.26.0
pytest>=8.0.0
starter/a_neural_network_in_pure_numpy_lib.py (1022 bytes) 
import numpy as np
from typing import List, Tuple, Dict, Any

class NeuralNetwork:
    def __init__(self, layer_dims: List[int], activations: List[str], learning_rate: float = 0.05, momentum: float = 0.9):
        self.layer_dims = layer_dims
        self.activations = activations
        self.lr = learning_rate
        self.beta = momentum
        self.params = {}
        self.velocities = {}

    def forward(self, X: np.ndarray) -> Tuple[np.ndarray, List[Dict[str, np.ndarray]]]:
        # TODO: Implement multi-layer forward pass
        pass

    def backward(self, A_last: np.ndarray, Y: np.ndarray, caches: List[Dict[str, np.ndarray]]) -> Dict[str, np.ndarray]:
        # TODO: Implement multi-layer backpropagation
        pass

    def fit(self, X: np.ndarray, Y: np.ndarray, epochs: int = 100, batch_size: int = 32) -> List[float]:
        # TODO: Implement mini-batch training loop
        pass

    def predict(self, X: np.ndarray) -> np.ndarray:
        # TODO: Return predicted class indices
        pass
starter/test_a_neural_network_in_pure_numpy_lib.py (1234 bytes) 
import pytest
import numpy as np
from examples.a_neural_network_in_pure_numpy_lib import NeuralNetwork, generate_two_moons

def test_neural_network_he_initialization():
    net = NeuralNetwork([2, 20, 10, 2], ["relu", "relu", "softmax"])
    assert net.params["W1"].shape == (20, 2)
    assert net.params["b1"].shape == (20, 1)
    assert net.params["W2"].shape == (10, 20)
    assert net.params["W3"].shape == (2, 10)
    # Check that weights are not all zeros
    assert np.count_nonzero(net.params["W1"]) > 0

def test_neural_network_trains_and_reduces_loss():
    X, Y_onehot, _ = generate_two_moons(n_samples=200)
    net = NeuralNetwork([2, 16, 8, 2], ["relu", "relu", "softmax"], learning_rate=0.1)
    losses = net.fit(X, Y_onehot, epochs=50, batch_size=32)

    assert len(losses) == 50
    # Check that loss decreased significantly
    assert losses[-1] < losses[0]

def test_neural_network_achieves_high_accuracy_on_moons():
    X, Y_onehot, y_true = generate_two_moons(n_samples=300)
    net = NeuralNetwork([2, 32, 16, 2], ["relu", "relu", "softmax"], learning_rate=0.1, momentum=0.9)
    net.fit(X, Y_onehot, epochs=80, batch_size=32)

    preds = net.predict(X)
    acc = np.mean(preds == y_true)
    assert acc > 0.90
tests/run_tests.sh (227 bytes) 
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 201 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_a_neural_network_in_pure_numpy_lib.py (1234 bytes) 
import pytest
import numpy as np
from examples.a_neural_network_in_pure_numpy_lib import NeuralNetwork, generate_two_moons

def test_neural_network_he_initialization():
    net = NeuralNetwork([2, 20, 10, 2], ["relu", "relu", "softmax"])
    assert net.params["W1"].shape == (20, 2)
    assert net.params["b1"].shape == (20, 1)
    assert net.params["W2"].shape == (10, 20)
    assert net.params["W3"].shape == (2, 10)
    # Check that weights are not all zeros
    assert np.count_nonzero(net.params["W1"]) > 0

def test_neural_network_trains_and_reduces_loss():
    X, Y_onehot, _ = generate_two_moons(n_samples=200)
    net = NeuralNetwork([2, 16, 8, 2], ["relu", "relu", "softmax"], learning_rate=0.1)
    losses = net.fit(X, Y_onehot, epochs=50, batch_size=32)

    assert len(losses) == 50
    # Check that loss decreased significantly
    assert losses[-1] < losses[0]

def test_neural_network_achieves_high_accuracy_on_moons():
    X, Y_onehot, y_true = generate_two_moons(n_samples=300)
    net = NeuralNetwork([2, 32, 16, 2], ["relu", "relu", "softmax"], learning_rate=0.1, momentum=0.9)
    net.fit(X, Y_onehot, epochs=80, batch_size=32)

    preds = net.predict(X)
    acc = np.mean(preds == y_true)
    assert acc > 0.90

Troubleshooting

Troubleshooting: Day 201 - A Neural Network in Pure NumPy

Common Issues

  1. Network Fails to Learn Non-Linear Boundary:
    • Cause: Insufficient hidden neurons or learning rate too small.
    • Fix: Use at least 16 hidden units in layer 1 and set learning rate to 0.05-0.1.

Security notes

Security & Privacy: Day 201 - A Neural Network in Pure NumPy

Security Guidance

  • All training and synthetic data generation execute locally in memory.