Sandeep Bazar 365 Days of AI
Labs

Deep Learning › Neural Network Foundations › Day 202

Hands-on lab — Day 202: PyTorch Tensors

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/pytorch_tensors_lib.py

Test

./tests/run_tests.sh

File tree

examples/pytorch_tensors_lib.py
examples/test_pytorch_tensors_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/pytorch_tensors_lib.py
starter/test_pytorch_tensors_lib.py
tests/run_tests.sh
tests/test_pytorch_tensors_lib.py
troubleshooting.md

Lab README

Lab: Day 202 -- PyTorch Tensors

Lesson

Day number: 202 of 365. Course: Course05-SS01 (Deep Learning - Neural Networks). Topic: PyTorch Tensors, Strides, and Autograd Mechanics.

Purpose

Master multidimensional tensor manipulation, memory contiguity contracts, hardware device abstraction, and dynamic automatic differentiation. You will evaluate tensor memory layouts, test stride modifications across views and permutations, and verify analytical gradients against Autograd computational graph passes.

Learning objectives

  • Manipulate multidimensional tensors, shapes, strides, and memory buffers.
  • Enforce the contiguity contract when reshaping permuted tensors.
  • Implement device-agnostic code running on CPU, Apple Silicon MPS, and NVIDIA CUDA.
  • Verify Autograd automatic differentiation against exact analytical derivatives.

Prerequisites

  • Days 199-201 (Forward Prop, Backprop, Neural Networks).
  • Python 3.11+ with NumPy and PyTorch (CPU-only wheel supported).

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 512 MB RAM.
  • 50 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

All tools used in this lab (Python, NumPy, PyTorch CPU, pytest) are free and open-source under BSD/MIT/Apache licenses.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

Note on PyTorch Installation: To install PyTorch CPU-only wheels on constrained environments: pip install torch --extra-index-url https://download.pytorch.org/whl/cpu

File structure

  • starter/pytorch_tensors_lib.py: Student scaffold file.
  • examples/pytorch_tensors_lib.py: Complete reference implementation.
  • tests/test_pytorch_tensors_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/pytorch_tensors_lib.py

What the commands do

  • Executes tensor memory stride evaluations.
  • Computes forward and backward autograd gradient simulations.
  • Verifies numerical gradient precision.

Expected output

PyTorch Tensors Demo: Output Z shape = (2, 2), dW shape = (2, 2)

Validation steps

  1. Verify that non-contiguous transposed tensors are safely flattened via contiguous memory buffers.
  2. Confirm that analytical gradients match numerical finite differences to 1e-6 precision.
  3. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Non-Contiguous Error: Call .contiguous() before .view() when modifying tensor dimensions.

Security notes

All tensor calculations execute locally without external network transmission.

Extension exercises

  1. Implement Higher-Order Gradients (Hessian-Vector Products).
  2. Benchmark tensor performance across CPU, MPS, and CUDA devices.
  • Lesson title: PyTorch Tensors
  • Day number: 202 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-202-pytorch-tensors
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-202-pytorch-tensors when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 202

- `Output Z Shape`: Dimensions of linear layer output tensor.
- `dW Shape`: Dimensions of evaluated weight gradient.

examples-run.txt

PyTorch Tensors Demo: Output Z shape = (2, 2), dW shape = (2, 2)

measured-values.txt

Output Z Shape: (2, 2)
dW Shape: (2, 2)

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_pytorch_tensors_lib.py::test_tensor_strides_and_contiguity PASSED [ 33%]
tests/test_pytorch_tensors_lib.py::test_linear_autograd_gradients_accuracy PASSED [ 66%]
tests/test_pytorch_tensors_lib.py::test_numerical_gradient_verification PASSED [100%]

============================== 3 passed in 0.12s ===============================

Source files

examples/pytorch_tensors_lib.py (1590 bytes) 
import numpy as np
from typing import Tuple, Dict, Any

class PyTorchTensorToolkit:
    @staticmethod
    def tensor_strides_demo(arr: np.ndarray) -> Dict[str, Any]:
        shape = arr.shape
        strides = arr.strides
        is_c_contiguous = arr.flags["C_CONTIGUOUS"]
        # Transpose
        arr_t = arr.T
        is_t_contiguous = arr_t.flags["C_CONTIGUOUS"]
        # Make contiguous and flatten
        flattened = np.ascontiguousarray(arr_t).reshape(-1)

        return {
            "orig_shape": shape,
            "orig_strides": strides,
            "orig_contiguous": is_c_contiguous,
            "transposed_contiguous": is_t_contiguous,
            "flattened_shape": flattened.shape
        }

    @staticmethod
    def linear_autograd_simulation(X: np.ndarray, W: np.ndarray, b: np.ndarray) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
        # Forward pass: Z = W X + b, Loss = sum(Z^2)
        Z = np.dot(W, X) + b
        loss = float(np.sum(Z ** 2))

        # Backward pass: dL/dZ = 2 * Z
        dZ = 2.0 * Z
        dW = np.dot(dZ, X.T)
        db = np.sum(dZ, axis=1, keepdims=True)

        return Z, dW, db

def run_pytorch_tensors_demo():
    X = np.array([[1.0, 2.0], [3.0, 4.0]], dtype=np.float32)
    W = np.array([[0.5, -0.5], [1.0, 2.0]], dtype=np.float32)
    b = np.array([[0.1], [-0.1]], dtype=np.float32)

    Z, dW, db = PyTorchTensorToolkit.linear_autograd_simulation(X, W, b)
    print(f"PyTorch Tensors Demo: Output Z shape = {Z.shape}, dW shape = {dW.shape}")
    return Z, dW, db

if __name__ == "__main__":
    run_pytorch_tensors_demo()
examples/test_pytorch_tensors_lib.py (1665 bytes) 
import pytest
import numpy as np
from examples.pytorch_tensors_lib import PyTorchTensorToolkit

def test_tensor_strides_and_contiguity():
    arr = np.arange(24).reshape(2, 3, 4)
    info = PyTorchTensorToolkit.tensor_strides_demo(arr)

    assert info["orig_shape"] == (2, 3, 4)
    assert info["orig_contiguous"] is True
    assert info["transposed_contiguous"] is False
    assert info["flattened_shape"] == (24,)

def test_linear_autograd_gradients_accuracy():
    np.random.seed(42)
    m = 5
    in_dim = 3
    out_dim = 2

    X = np.random.randn(in_dim, m)
    W = np.random.randn(out_dim, in_dim)
    b = np.random.randn(out_dim, 1)

    Z, dW, db = PyTorchTensorToolkit.linear_autograd_simulation(X, W, b)

    assert Z.shape == (out_dim, m)
    assert dW.shape == (out_dim, in_dim)
    assert db.shape == (out_dim, 1)

def test_numerical_gradient_verification():
    X = np.array([[1.0, 2.0]], dtype=float).T # (2, 1)
    W = np.array([[2.0, 3.0]], dtype=float)    # (1, 2)
    b = np.array([[1.0]], dtype=float)         # (1, 1)

    Z, dW_ana, db_ana = PyTorchTensorToolkit.linear_autograd_simulation(X, W, b)

    # Numerical gradient check on W
    eps = 1e-5
    dW_num = np.zeros_like(W)
    for i in range(W.shape[0]):
        for j in range(W.shape[1]):
            orig = W[i, j]
            W[i, j] = orig + eps
            loss_p = np.sum((np.dot(W, X) + b) ** 2)
            W[i, j] = orig - eps
            loss_m = np.sum((np.dot(W, X) + b) ** 2)
            W[i, j] = orig
            dW_num[i, j] = (loss_p - loss_m) / (2.0 * eps)

    rel_err = np.linalg.norm(dW_ana - dW_num) / (np.linalg.norm(dW_ana) + 1e-8)
    assert rel_err < 1e-6
metadata.yml (428 bytes) 
lesson_id: D202
day: 202
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/pytorch_tensors_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (28 bytes) 
numpy>=1.26.0
pytest>=8.0.0
starter/pytorch_tensors_lib.py (547 bytes) 
import numpy as np
from typing import Tuple, Dict

# Note: Student implementation can bridge NumPy and PyTorch or test mathematical tensors
class PyTorchTensorToolkit:
    @staticmethod
    def tensor_strides_demo(arr: np.ndarray) -> Dict[str, Any]:
        # TODO: Compute shape, strides, and contiguous flattening
        pass

    @staticmethod
    def linear_autograd_simulation(X: np.ndarray, W: np.ndarray, b: np.ndarray) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
        # TODO: Simulate forward and backward autograd pass
        pass
starter/test_pytorch_tensors_lib.py (1665 bytes) 
import pytest
import numpy as np
from examples.pytorch_tensors_lib import PyTorchTensorToolkit

def test_tensor_strides_and_contiguity():
    arr = np.arange(24).reshape(2, 3, 4)
    info = PyTorchTensorToolkit.tensor_strides_demo(arr)

    assert info["orig_shape"] == (2, 3, 4)
    assert info["orig_contiguous"] is True
    assert info["transposed_contiguous"] is False
    assert info["flattened_shape"] == (24,)

def test_linear_autograd_gradients_accuracy():
    np.random.seed(42)
    m = 5
    in_dim = 3
    out_dim = 2

    X = np.random.randn(in_dim, m)
    W = np.random.randn(out_dim, in_dim)
    b = np.random.randn(out_dim, 1)

    Z, dW, db = PyTorchTensorToolkit.linear_autograd_simulation(X, W, b)

    assert Z.shape == (out_dim, m)
    assert dW.shape == (out_dim, in_dim)
    assert db.shape == (out_dim, 1)

def test_numerical_gradient_verification():
    X = np.array([[1.0, 2.0]], dtype=float).T # (2, 1)
    W = np.array([[2.0, 3.0]], dtype=float)    # (1, 2)
    b = np.array([[1.0]], dtype=float)         # (1, 1)

    Z, dW_ana, db_ana = PyTorchTensorToolkit.linear_autograd_simulation(X, W, b)

    # Numerical gradient check on W
    eps = 1e-5
    dW_num = np.zeros_like(W)
    for i in range(W.shape[0]):
        for j in range(W.shape[1]):
            orig = W[i, j]
            W[i, j] = orig + eps
            loss_p = np.sum((np.dot(W, X) + b) ** 2)
            W[i, j] = orig - eps
            loss_m = np.sum((np.dot(W, X) + b) ** 2)
            W[i, j] = orig
            dW_num[i, j] = (loss_p - loss_m) / (2.0 * eps)

    rel_err = np.linalg.norm(dW_ana - dW_num) / (np.linalg.norm(dW_ana) + 1e-8)
    assert rel_err < 1e-6
tests/run_tests.sh (227 bytes) 
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 202 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_pytorch_tensors_lib.py (1665 bytes) 
import pytest
import numpy as np
from examples.pytorch_tensors_lib import PyTorchTensorToolkit

def test_tensor_strides_and_contiguity():
    arr = np.arange(24).reshape(2, 3, 4)
    info = PyTorchTensorToolkit.tensor_strides_demo(arr)

    assert info["orig_shape"] == (2, 3, 4)
    assert info["orig_contiguous"] is True
    assert info["transposed_contiguous"] is False
    assert info["flattened_shape"] == (24,)

def test_linear_autograd_gradients_accuracy():
    np.random.seed(42)
    m = 5
    in_dim = 3
    out_dim = 2

    X = np.random.randn(in_dim, m)
    W = np.random.randn(out_dim, in_dim)
    b = np.random.randn(out_dim, 1)

    Z, dW, db = PyTorchTensorToolkit.linear_autograd_simulation(X, W, b)

    assert Z.shape == (out_dim, m)
    assert dW.shape == (out_dim, in_dim)
    assert db.shape == (out_dim, 1)

def test_numerical_gradient_verification():
    X = np.array([[1.0, 2.0]], dtype=float).T # (2, 1)
    W = np.array([[2.0, 3.0]], dtype=float)    # (1, 2)
    b = np.array([[1.0]], dtype=float)         # (1, 1)

    Z, dW_ana, db_ana = PyTorchTensorToolkit.linear_autograd_simulation(X, W, b)

    # Numerical gradient check on W
    eps = 1e-5
    dW_num = np.zeros_like(W)
    for i in range(W.shape[0]):
        for j in range(W.shape[1]):
            orig = W[i, j]
            W[i, j] = orig + eps
            loss_p = np.sum((np.dot(W, X) + b) ** 2)
            W[i, j] = orig - eps
            loss_m = np.sum((np.dot(W, X) + b) ** 2)
            W[i, j] = orig
            dW_num[i, j] = (loss_p - loss_m) / (2.0 * eps)

    rel_err = np.linalg.norm(dW_ana - dW_num) / (np.linalg.norm(dW_ana) + 1e-8)
    assert rel_err < 1e-6

Troubleshooting

Troubleshooting: Day 202 - PyTorch Tensors

Common Issues

  1. Contiguity Runtime Error:
    • Cause: Calling .view() on a transposed or permuted tensor without re-allocating memory.
    • Fix: Use tensor.contiguous().view() or tensor.reshape().

Security notes

Security & Privacy: Day 202 - PyTorch Tensors

Security Guidance

  • All tensor operations run in local memory without outbound network access.