Sandeep Bazar 365 Days of AI
Labs

Deep Learning › Neural Network Foundations › Day 198

Hands-on lab — Day 198: Activation Functions

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/activation_functions_lib.py

Test

./tests/run_tests.sh

File tree

examples/activation_functions_lib.py
examples/test_activation_functions_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/activation_functions_lib.py
starter/test_activation_functions_lib.py
tests/run_tests.sh
tests/test_activation_functions_lib.py
troubleshooting.md

Lab README

Lab: Day 198 -- Activation Functions

Lesson

Day number: 198 of 365. Course: Course05-SS01 (Deep Learning - Neural Networks). Topic: Activation Functions and Numerical Gradient Checking.

Purpose

Build a complete, vectorized, high-performance ActivationEngine in pure NumPy. You will implement Sigmoid, Tanh, ReLU, Leaky ReLU, GeLU, and numerically stable Softmax, derive analytical gradients, perform central finite-difference gradient checks, and analyze vanishing gradient behavior across deep layers.

Learning objectives

  • Implement Sigmoid, Tanh, ReLU, Leaky ReLU, GeLU, and Softmax forward functions.
  • Compute analytical derivatives for each activation function.
  • Execute numerical gradient checking to verify derivative precision.
  • Prevent floating-point overflow in Softmax with maximum subtraction.

Prerequisites

  • Calculus (derivatives, chain rule).
  • Python 3.11+ with NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 512 MB RAM.
  • 50 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

All tools used in this lab (Python, NumPy, pytest) are free and open-source under BSD/MIT licenses.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/activation_functions_lib.py: Student scaffold file.
  • examples/activation_functions_lib.py: Complete reference implementation.
  • tests/test_activation_functions_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/activation_functions_lib.py

What the commands do

  • Executes forward activations and analytical derivatives.
  • Verifies numerical gradient checks against finite differences.
  • Evaluates numerically stable Softmax on extreme logits.

Expected output

Activation Demo: Sigmoid Relative Error = 1.2415e-11, Stable Softmax Sum = 1.0000

Validation steps

  1. Verify that numerical gradient check error is less than 1e-5 for all activations.
  2. Confirm that Softmax does not return NaN or inf on logits of 5000.0.
  3. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Numerical Gradient Discontinuity: For ReLU, evaluate away from the non-differentiable cusp z = 0.0.

Security notes

All mathematical computations execute locally in process memory.

Extension exercises

  1. Implement the SELU (Scaled Exponential Linear Unit) activation.
  2. Code the Mish activation function and compute its analytical gradient.
  • Lesson title: Activation Functions
  • Day number: 198 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-198-activation-functions
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-198-activation-functions when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 198

- `Sigmoid Relative Error`: Relative discrepancy between analytical and numerical derivatives.
- `Stable Softmax Sum`: Sum of output probabilities from extreme logits.

examples-run.txt

Activation Demo: Sigmoid Relative Error = 1.2415e-11, Stable Softmax Sum = 1.0000

measured-values.txt

Sigmoid Relative Error: 1.2415e-11
Stable Softmax Sum: 1.0000

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 4 items

tests/test_activation_functions_lib.py::test_sigmoid_and_tanh_derivatives PASSED [ 25%]
tests/test_activation_functions_lib.py::test_relu_and_leaky_relu_behavior PASSED [ 50%]
tests/test_activation_functions_lib.py::test_stable_softmax_prevents_overflow PASSED [ 75%]
tests/test_activation_functions_lib.py::test_gelu_gradient_accuracy PASSED       [100%]

============================== 4 passed in 0.10s ===============================

Source files

examples/activation_functions_lib.py (2307 bytes) 
import numpy as np

class ActivationEngine:
    @staticmethod
    def sigmoid(z: np.ndarray) -> np.ndarray:
        return np.where(z >= 0, 1.0 / (1.0 + np.exp(-z)), np.exp(z) / (1.0 + np.exp(z)))

    @staticmethod
    def sigmoid_grad(z: np.ndarray) -> np.ndarray:
        s = ActivationEngine.sigmoid(z)
        return s * (1.0 - s)

    @staticmethod
    def tanh(z: np.ndarray) -> np.ndarray:
        return np.tanh(z)

    @staticmethod
    def tanh_grad(z: np.ndarray) -> np.ndarray:
        t = np.tanh(z)
        return 1.0 - t ** 2

    @staticmethod
    def relu(z: np.ndarray) -> np.ndarray:
        return np.maximum(0.0, z)

    @staticmethod
    def relu_grad(z: np.ndarray) -> np.ndarray:
        return np.where(z > 0.0, 1.0, 0.0)

    @staticmethod
    def leaky_relu(z: np.ndarray, alpha: float = 0.01) -> np.ndarray:
        return np.where(z > 0.0, z, alpha * z)

    @staticmethod
    def leaky_relu_grad(z: np.ndarray, alpha: float = 0.01) -> np.ndarray:
        return np.where(z > 0.0, 1.0, alpha)

    @staticmethod
    def gelu(z: np.ndarray) -> np.ndarray:
        return 0.5 * z * (1.0 + np.tanh(np.sqrt(2.0 / np.pi) * (z + 0.044715 * (z ** 3))))

    @staticmethod
    def stable_softmax(z: np.ndarray, axis: int = -1) -> np.ndarray:
        z_shifted = z - np.max(z, axis=axis, keepdims=True)
        exp_z = np.exp(z_shifted)
        return exp_z / np.sum(exp_z, axis=axis, keepdims=True)

def numerical_gradient_check(func, z: np.ndarray, analytical_grad: np.ndarray, eps: float = 1e-5) -> float:
    grad_approx = (func(z + eps) - func(z - eps)) / (2.0 * eps)
    numerator = np.linalg.norm(analytical_grad - grad_approx)
    denominator = np.linalg.norm(analytical_grad) + np.linalg.norm(grad_approx) + 1e-8
    return float(numerator / denominator)

def run_activation_demo():
    z = np.array([-2.0, -0.5, 0.0, 0.5, 2.0])
    s = ActivationEngine.sigmoid(z)
    sg = ActivationEngine.sigmoid_grad(z)
    err = numerical_gradient_check(ActivationEngine.sigmoid, z, sg)

    extreme_logits = np.array([[1000.0, 2000.0, 3000.0]])
    probs = ActivationEngine.stable_softmax(extreme_logits)

    print(f"Activation Demo: Sigmoid Relative Error = {err:.4e}, Stable Softmax Sum = {np.sum(probs):.4f}")
    return s, probs

if __name__ == "__main__":
    run_activation_demo()
examples/test_activation_functions_lib.py (1565 bytes) 
import pytest
import numpy as np
from examples.activation_functions_lib import ActivationEngine, numerical_gradient_check

def test_sigmoid_and_tanh_derivatives():
    z = np.array([-2.5, -1.0, 0.2, 1.5, 3.0])
    sig_grad = ActivationEngine.sigmoid_grad(z)
    err_sig = numerical_gradient_check(ActivationEngine.sigmoid, z, sig_grad)
    assert err_sig < 1e-5

    tanh_grad = ActivationEngine.tanh_grad(z)
    err_tanh = numerical_gradient_check(ActivationEngine.tanh, z, tanh_grad)
    assert err_tanh < 1e-5

def test_relu_and_leaky_relu_behavior():
    z = np.array([-3.0, -1.0, 1.0, 3.0])
    r = ActivationEngine.relu(z)
    assert np.array_equal(r, np.array([0.0, 0.0, 1.0, 3.0]))
    assert np.array_equal(ActivationEngine.relu_grad(z), np.array([0.0, 0.0, 1.0, 1.0]))

    lr = ActivationEngine.leaky_relu(z, alpha=0.1)
    assert np.allclose(lr, np.array([-0.3, -0.1, 1.0, 3.0]))

def test_stable_softmax_prevents_overflow():
    extreme_logits = np.array([[5000.0, 5001.0, 5002.0], [-1000.0, -1000.0, -1000.0]])
    probs = ActivationEngine.stable_softmax(extreme_logits, axis=-1)
    assert not np.isnan(probs).any()
    assert not np.isinf(probs).any()
    assert np.allclose(np.sum(probs, axis=-1), np.array([1.0, 1.0]))

def test_gelu_gradient_accuracy():
    z = np.array([-1.5, -0.5, 0.5, 1.5])
    # Approximate gelu gradient numerically
    eps = 1e-5
    grad_approx = (ActivationEngine.gelu(z + eps) - ActivationEngine.gelu(z - eps)) / (2.0 * eps)
    # Check that gelu output is continuous and smooth
    assert grad_approx.shape == z.shape
metadata.yml (433 bytes) 
lesson_id: D198
day: 198
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/activation_functions_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (28 bytes) 
numpy>=1.26.0
pytest>=8.0.0
starter/activation_functions_lib.py (898 bytes) 
import numpy as np

class ActivationEngine:
    @staticmethod
    def sigmoid(z: np.ndarray) -> np.ndarray:
        # TODO: Implement numerically stable sigmoid
        pass

    @staticmethod
    def sigmoid_grad(z: np.ndarray) -> np.ndarray:
        # TODO: Implement sigmoid analytical derivative
        pass

    @staticmethod
    def relu(z: np.ndarray) -> np.ndarray:
        # TODO: Implement ReLU
        pass

    @staticmethod
    def relu_grad(z: np.ndarray) -> np.ndarray:
        # TODO: Implement ReLU derivative
        pass

    @staticmethod
    def stable_softmax(z: np.ndarray, axis: int = -1) -> np.ndarray:
        # TODO: Implement numerically stable softmax with max subtraction
        pass

def numerical_gradient_check(func, z: np.ndarray, analytical_grad: np.ndarray, eps: float = 1e-5) -> float:
    # TODO: Implement central finite differences gradient check
    pass
starter/test_activation_functions_lib.py (1565 bytes) 
import pytest
import numpy as np
from examples.activation_functions_lib import ActivationEngine, numerical_gradient_check

def test_sigmoid_and_tanh_derivatives():
    z = np.array([-2.5, -1.0, 0.2, 1.5, 3.0])
    sig_grad = ActivationEngine.sigmoid_grad(z)
    err_sig = numerical_gradient_check(ActivationEngine.sigmoid, z, sig_grad)
    assert err_sig < 1e-5

    tanh_grad = ActivationEngine.tanh_grad(z)
    err_tanh = numerical_gradient_check(ActivationEngine.tanh, z, tanh_grad)
    assert err_tanh < 1e-5

def test_relu_and_leaky_relu_behavior():
    z = np.array([-3.0, -1.0, 1.0, 3.0])
    r = ActivationEngine.relu(z)
    assert np.array_equal(r, np.array([0.0, 0.0, 1.0, 3.0]))
    assert np.array_equal(ActivationEngine.relu_grad(z), np.array([0.0, 0.0, 1.0, 1.0]))

    lr = ActivationEngine.leaky_relu(z, alpha=0.1)
    assert np.allclose(lr, np.array([-0.3, -0.1, 1.0, 3.0]))

def test_stable_softmax_prevents_overflow():
    extreme_logits = np.array([[5000.0, 5001.0, 5002.0], [-1000.0, -1000.0, -1000.0]])
    probs = ActivationEngine.stable_softmax(extreme_logits, axis=-1)
    assert not np.isnan(probs).any()
    assert not np.isinf(probs).any()
    assert np.allclose(np.sum(probs, axis=-1), np.array([1.0, 1.0]))

def test_gelu_gradient_accuracy():
    z = np.array([-1.5, -0.5, 0.5, 1.5])
    # Approximate gelu gradient numerically
    eps = 1e-5
    grad_approx = (ActivationEngine.gelu(z + eps) - ActivationEngine.gelu(z - eps)) / (2.0 * eps)
    # Check that gelu output is continuous and smooth
    assert grad_approx.shape == z.shape
tests/run_tests.sh (227 bytes) 
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 198 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_activation_functions_lib.py (1565 bytes) 
import pytest
import numpy as np
from examples.activation_functions_lib import ActivationEngine, numerical_gradient_check

def test_sigmoid_and_tanh_derivatives():
    z = np.array([-2.5, -1.0, 0.2, 1.5, 3.0])
    sig_grad = ActivationEngine.sigmoid_grad(z)
    err_sig = numerical_gradient_check(ActivationEngine.sigmoid, z, sig_grad)
    assert err_sig < 1e-5

    tanh_grad = ActivationEngine.tanh_grad(z)
    err_tanh = numerical_gradient_check(ActivationEngine.tanh, z, tanh_grad)
    assert err_tanh < 1e-5

def test_relu_and_leaky_relu_behavior():
    z = np.array([-3.0, -1.0, 1.0, 3.0])
    r = ActivationEngine.relu(z)
    assert np.array_equal(r, np.array([0.0, 0.0, 1.0, 3.0]))
    assert np.array_equal(ActivationEngine.relu_grad(z), np.array([0.0, 0.0, 1.0, 1.0]))

    lr = ActivationEngine.leaky_relu(z, alpha=0.1)
    assert np.allclose(lr, np.array([-0.3, -0.1, 1.0, 3.0]))

def test_stable_softmax_prevents_overflow():
    extreme_logits = np.array([[5000.0, 5001.0, 5002.0], [-1000.0, -1000.0, -1000.0]])
    probs = ActivationEngine.stable_softmax(extreme_logits, axis=-1)
    assert not np.isnan(probs).any()
    assert not np.isinf(probs).any()
    assert np.allclose(np.sum(probs, axis=-1), np.array([1.0, 1.0]))

def test_gelu_gradient_accuracy():
    z = np.array([-1.5, -0.5, 0.5, 1.5])
    # Approximate gelu gradient numerically
    eps = 1e-5
    grad_approx = (ActivationEngine.gelu(z + eps) - ActivationEngine.gelu(z - eps)) / (2.0 * eps)
    # Check that gelu output is continuous and smooth
    assert grad_approx.shape == z.shape

Troubleshooting

Troubleshooting: Day 198 - Activation Functions

Common Issues

  1. Floating Point Overflow in Exp:
    • Cause: Exponentiating large positive numbers (> 709 in float64).
    • Fix: Use numerically stable formulations (e.g. subtract max(z) in Softmax).

Security notes

Security & Privacy: Day 198 - Activation Functions

Security Guidance

  • Numerical tensor operations run locally without external data exfiltration.