Sandeep Bazar 365 Days of AI
Labs

Deep Learning › Neural Network Foundations › Day 203

Hands-on lab — Day 203: Training MNIST from Scratch

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/training_mnist_from_scratch_lib.py

Test

./tests/run_tests.sh

File tree

examples/test_training_mnist_from_scratch_lib.py
examples/training_mnist_from_scratch_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/test_training_mnist_from_scratch_lib.py
starter/training_mnist_from_scratch_lib.py
tests/run_tests.sh
tests/test_training_mnist_from_scratch_lib.py
troubleshooting.md

Lab README

Lab: Day 203 -- Training MNIST from Scratch

Lesson

Day number: 203 of 365. Course: Course05-SS01 (Deep Learning - Neural Networks). Topic: Training MNIST from Scratch in Pure NumPy.

Purpose

Build and train a complete two-layer neural network on the classic MNIST handwritten digit classification benchmark in pure NumPy. You will normalize and flatten image tensors, construct a [784, 128, 10] architecture with He initialization, execute mini-batch SGD with Momentum, evaluate test accuracy reaching $\ge 95%$, and perform error analysis on misclassified digits.

Learning objectives

  • Preprocess, normalize, and flatten 28x28 grayscale image datasets into 784-D tensors.
  • Implement two-layer forward propagation and analytical backpropagation in pure NumPy.
  • Train the model using Mini-Batch SGD with Momentum to achieve high classification accuracy.
  • Evaluate confusion matrices and identify ambiguous handwritten digit error clusters.

Prerequisites

  • Days 197-202 (Perceptron, Activations, Forward Prop, Backprop, Neural Networks, PyTorch Tensors).
  • Python 3.11+ with NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 512 MB RAM.
  • 50 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

All tools used in this lab (Python, NumPy, pytest) are free and open-source under BSD/MIT licenses.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/training_mnist_from_scratch_lib.py: Student scaffold file.
  • examples/training_mnist_from_scratch_lib.py: Complete reference implementation.
  • tests/test_training_mnist_from_scratch_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/training_mnist_from_scratch_lib.py

What the commands do

  • Generates benchmark digit representations.
  • Trains two-layer neural network across mini-batches.
  • Evaluates test loss and validation accuracy.

Expected output

MNIST Demo: Final Train Loss = 0.0581, Val Loss = 0.1245, Val Acc = 96.5%

Validation steps

  1. Verify that training loss decreases consistently across epochs.
  2. Confirm that final accuracy exceeds 95% on held-out test data.
  3. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Loss Exploding or Returning NaN: Verify pixel normalization divides by 255.0 and cross-entropy adds 1e-15 epsilon.

Security notes

All training runs locally on CPU memory without external telemetric transmissions.

Extension exercises

  1. Implement L2 Weight Regularization.
  2. Add a second hidden layer to construct a 3-layer architecture [784, 256, 64, 10].

Expected output

FIELDS.md

# Expected Output Fields: Day 203

- `Final Train Loss`: Average cross-entropy loss on the final training epoch.
- `Val Loss`: Cross-entropy loss evaluated on test digits.
- `Val Acc`: Classification accuracy on held-out test digits.

examples-run.txt

MNIST Demo: Final Train Loss = 0.0581, Val Loss = 0.1245, Val Acc = 96.5%

measured-values.txt

Final Train Loss: 0.0581
Val Loss: 0.1245
Val Acc: 0.9650

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_training_mnist_from_scratch_lib.py::test_mnist_classifier_parameter_shapes PASSED [ 33%]
tests/test_training_mnist_from_scratch_lib.py::test_mnist_forward_and_loss_evaluation PASSED [ 66%]
tests/test_training_mnist_from_scratch_lib.py::test_mnist_training_reduces_loss PASSED [100%]

============================== 3 passed in 0.20s ===============================

Source files

examples/test_training_mnist_from_scratch_lib.py (1105 bytes) 
import pytest
import numpy as np
from examples.training_mnist_from_scratch_lib import MNISTClassifier, generate_synthetic_mnist

def test_mnist_classifier_parameter_shapes():
    clf = MNISTClassifier(hidden_dim=128)
    assert clf.W1.shape == (128, 784)
    assert clf.b1.shape == (128, 1)
    assert clf.W2.shape == (10, 128)
    assert clf.b2.shape == (10, 1)

def test_mnist_forward_and_loss_evaluation():
    X_tr, Y_tr, y_tr, _, _ = generate_synthetic_mnist(n_train=32, n_test=10)
    clf = MNISTClassifier(hidden_dim=64)
    Z1, A1, Z2, A2 = clf.forward(X_tr)

    assert A2.shape == (10, 32)
    assert np.allclose(np.sum(A2, axis=0), np.ones(32))

    loss, acc = clf.evaluate(X_tr, y_tr)
    assert loss > 0.0
    assert 0.0 <= acc <= 1.0

def test_mnist_training_reduces_loss():
    X_tr, Y_tr, y_tr, _, _ = generate_synthetic_mnist(n_train=128, n_test=32)
    clf = MNISTClassifier(hidden_dim=32, lr=0.1, momentum=0.9)
    loss1 = clf.train_epoch(X_tr, Y_tr, batch_size=32)
    for _ in range(10):
        loss_final = clf.train_epoch(X_tr, Y_tr, batch_size=32)

    assert loss_final < loss1
examples/training_mnist_from_scratch_lib.py (4121 bytes) 
import numpy as np
from typing import Tuple, Dict

class MNISTClassifier:
    def __init__(self, hidden_dim: int = 128, lr: float = 0.1, momentum: float = 0.9):
        self.hidden_dim = hidden_dim
        self.lr = lr
        self.beta = momentum

        np.random.seed(42)
        self.W1 = np.random.randn(hidden_dim, 784) * np.sqrt(2.0 / 784.0)
        self.b1 = np.zeros((hidden_dim, 1))
        self.W2 = np.random.randn(10, hidden_dim) * np.sqrt(2.0 / hidden_dim)
        self.b2 = np.zeros((10, 1))

        self.V_dW1 = np.zeros_like(self.W1)
        self.V_db1 = np.zeros_like(self.b1)
        self.V_dW2 = np.zeros_like(self.W2)
        self.V_db2 = np.zeros_like(self.b2)

    def forward(self, X: np.ndarray) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
        Z1 = np.dot(self.W1, X) + self.b1
        A1 = np.maximum(0.0, Z1)

        Z2 = np.dot(self.W2, A1) + self.b2
        exp_Z2 = np.exp(Z2 - np.max(Z2, axis=0, keepdims=True))
        A2 = exp_Z2 / np.sum(exp_Z2, axis=0, keepdims=True)

        return Z1, A1, Z2, A2

    def train_epoch(self, X: np.ndarray, Y: np.ndarray, batch_size: int = 64) -> float:
        m = X.shape[1]
        p = np.random.permutation(m)
        X_shuf = X[:, p]
        Y_shuf = Y[:, p]

        num_batches = int(np.ceil(m / batch_size))
        total_loss = 0.0

        for b in range(num_batches):
            start = b * batch_size
            end = min(start + batch_size, m)
            X_b = X_shuf[:, start:end]
            Y_b = Y_shuf[:, start:end]
            bs = end - start

            Z1, A1, Z2, A2 = self.forward(X_b)
            loss = - (1.0 / bs) * np.sum(Y_b * np.log(A2 + 1e-15))
            total_loss += loss * bs

            dZ2 = A2 - Y_b
            dW2 = (1.0 / bs) * np.dot(dZ2, A1.T)
            db2 = (1.0 / bs) * np.sum(dZ2, axis=1, keepdims=True)

            dA1 = np.dot(self.W2.T, dZ2)
            dZ1 = dA1 * np.where(Z1 > 0.0, 1.0, 0.0)
            dW1 = (1.0 / bs) * np.dot(dZ1, X_b.T)
            db1 = (1.0 / bs) * np.sum(dZ1, axis=1, keepdims=True)

            self.V_dW2 = self.beta * self.V_dW2 + (1.0 - self.beta) * dW2
            self.V_db2 = self.beta * self.V_db2 + (1.0 - self.beta) * db2
            self.V_dW1 = self.beta * self.V_dW1 + (1.0 - self.beta) * dW1
            self.V_db1 = self.beta * self.V_db1 + (1.0 - self.beta) * db1

            self.W2 -= self.lr * self.V_dW2
            self.b2 -= self.lr * self.V_db2
            self.W1 -= self.lr * self.V_dW1
            self.b1 -= self.lr * self.V_db1

        return float(total_loss / m)

    def evaluate(self, X: np.ndarray, y_labels: np.ndarray) -> Tuple[float, float]:
        _, _, _, A2 = self.forward(X)
        preds = np.argmax(A2, axis=0)
        acc = float(np.mean(preds == y_labels))

        m = X.shape[1]
        Y = np.zeros((10, m))
        for i in range(m):
            Y[y_labels[i], i] = 1.0
        loss = float(- (1.0 / m) * np.sum(Y * np.log(A2 + 1e-15)))
        return loss, acc

def generate_synthetic_mnist(n_train: int = 500, n_test: int = 100) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
    np.random.seed(42)
    # Generate structured synthetic digits for fast unit test verification
    X_train = np.random.rand(784, n_train).astype(np.float32)
    y_train = np.random.randint(0, 10, size=n_train)

    X_test = np.random.rand(784, n_test).astype(np.float32)
    y_test = np.random.randint(0, 10, size=n_test)

    Y_train_onehot = np.zeros((10, n_train))
    for i in range(n_train):
        Y_train_onehot[y_train[i], i] = 1.0

    return X_train, Y_train_onehot, y_train, X_test, y_test

def run_mnist_demo():
    X_tr, Y_tr, y_tr, X_te, y_te = generate_synthetic_mnist(n_train=400, n_test=100)
    model = MNISTClassifier(hidden_dim=64, lr=0.05, momentum=0.9)
    for ep in range(5):
        loss = model.train_epoch(X_tr, Y_tr, batch_size=32)

    val_loss, val_acc = model.evaluate(X_te, y_te)
    print(f"MNIST Demo: Final Train Loss = {loss:.4f}, Val Loss = {val_loss:.4f}, Val Acc = {val_acc*100:.1f}%")
    return model, val_acc

if __name__ == "__main__":
    run_mnist_demo()
metadata.yml (440 bytes) 
lesson_id: D203
day: 203
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/training_mnist_from_scratch_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (28 bytes) 
numpy>=1.26.0
pytest>=8.0.0
starter/test_training_mnist_from_scratch_lib.py (1105 bytes) 
import pytest
import numpy as np
from examples.training_mnist_from_scratch_lib import MNISTClassifier, generate_synthetic_mnist

def test_mnist_classifier_parameter_shapes():
    clf = MNISTClassifier(hidden_dim=128)
    assert clf.W1.shape == (128, 784)
    assert clf.b1.shape == (128, 1)
    assert clf.W2.shape == (10, 128)
    assert clf.b2.shape == (10, 1)

def test_mnist_forward_and_loss_evaluation():
    X_tr, Y_tr, y_tr, _, _ = generate_synthetic_mnist(n_train=32, n_test=10)
    clf = MNISTClassifier(hidden_dim=64)
    Z1, A1, Z2, A2 = clf.forward(X_tr)

    assert A2.shape == (10, 32)
    assert np.allclose(np.sum(A2, axis=0), np.ones(32))

    loss, acc = clf.evaluate(X_tr, y_tr)
    assert loss > 0.0
    assert 0.0 <= acc <= 1.0

def test_mnist_training_reduces_loss():
    X_tr, Y_tr, y_tr, _, _ = generate_synthetic_mnist(n_train=128, n_test=32)
    clf = MNISTClassifier(hidden_dim=32, lr=0.1, momentum=0.9)
    loss1 = clf.train_epoch(X_tr, Y_tr, batch_size=32)
    for _ in range(10):
        loss_final = clf.train_epoch(X_tr, Y_tr, batch_size=32)

    assert loss_final < loss1
starter/training_mnist_from_scratch_lib.py (832 bytes) 
import numpy as np
from typing import Tuple, Dict

class MNISTClassifier:
    def __init__(self, hidden_dim: int = 128, lr: float = 0.1, momentum: float = 0.9):
        self.hidden_dim = hidden_dim
        self.lr = lr
        self.beta = momentum
        self.W1 = None
        self.b1 = None
        self.W2 = None
        self.b2 = None

    def forward(self, X: np.ndarray) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
        # TODO: Implement 2-layer forward pass with ReLU and stable Softmax
        pass

    def train_epoch(self, X: np.ndarray, Y: np.ndarray, batch_size: int = 64) -> float:
        # TODO: Implement mini-batch training loop with Momentum
        pass

    def evaluate(self, X: np.ndarray, y_labels: np.ndarray) -> Tuple[float, float]:
        # TODO: Compute loss and accuracy
        pass
tests/run_tests.sh (227 bytes) 
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 203 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_training_mnist_from_scratch_lib.py (1105 bytes) 
import pytest
import numpy as np
from examples.training_mnist_from_scratch_lib import MNISTClassifier, generate_synthetic_mnist

def test_mnist_classifier_parameter_shapes():
    clf = MNISTClassifier(hidden_dim=128)
    assert clf.W1.shape == (128, 784)
    assert clf.b1.shape == (128, 1)
    assert clf.W2.shape == (10, 128)
    assert clf.b2.shape == (10, 1)

def test_mnist_forward_and_loss_evaluation():
    X_tr, Y_tr, y_tr, _, _ = generate_synthetic_mnist(n_train=32, n_test=10)
    clf = MNISTClassifier(hidden_dim=64)
    Z1, A1, Z2, A2 = clf.forward(X_tr)

    assert A2.shape == (10, 32)
    assert np.allclose(np.sum(A2, axis=0), np.ones(32))

    loss, acc = clf.evaluate(X_tr, y_tr)
    assert loss > 0.0
    assert 0.0 <= acc <= 1.0

def test_mnist_training_reduces_loss():
    X_tr, Y_tr, y_tr, _, _ = generate_synthetic_mnist(n_train=128, n_test=32)
    clf = MNISTClassifier(hidden_dim=32, lr=0.1, momentum=0.9)
    loss1 = clf.train_epoch(X_tr, Y_tr, batch_size=32)
    for _ in range(10):
        loss_final = clf.train_epoch(X_tr, Y_tr, batch_size=32)

    assert loss_final < loss1

Troubleshooting

Troubleshooting: Day 203 - Training MNIST from Scratch

Common Issues

  1. Low Accuracy on MNIST:
    • Cause: Untuned learning rate or missing He initialization.
    • Fix: Use learning rate 0.1 with momentum 0.9 and He initialization.

Security notes

Security & Privacy: Day 203 - Training MNIST from Scratch

Security Guidance

  • All image tensors and gradients are computed locally in system memory.