LLMs and Generative AI › Customizing and Running Models › Day 275

Hands-on lab — Day 275: Fine-Tuning with LoRA

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/lora_layer.py

Test

bash tests/run_tests.sh

File tree

examples/lora_layer.py
examples/test_lora_layer.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/lora_layer.py
starter/test_lora_layer.py
tests/run_tests.sh
tests/test_lora_layer.py
troubleshooting.md

Lab README

Day 275 Lab: Custom LoRA Linear Layer in NumPy

Lesson

  • Lesson title: Fine-Tuning with LoRA
  • Day number: 275 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-275-fine-tuning-with-lora
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-275-fine-tuning-with-lora when the site is running.

Purpose

Build a custom Low-Rank Adaptation (LoRA) linear layer from scratch in Python and NumPy, implementing low-rank matrix decomposition, parameter reduction accounting, and zero-overhead offline weight merging.

Learning objectives

  • Implement low-rank decomposition matrices ($A$ and $B$) with Gaussian and zero initializations.
  • Compute scaling factors ($\alpha / r$) and forward pass activations.
  • Verify zero delta updates at step zero.
  • Implement offline weight merging ($W_{\text{merged}} = W_0 + \Delta W$) and verify bit-exact output parity.

Prerequisites

  • Python 3.10+ installed
  • NumPy and pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, numpy, pytest

Free and open-source options

  • Python Standard Library, NumPy

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/lora_layer.py: Starter implementation skeleton
  • examples/lora_layer.py: Verified reference implementation
  • tests/test_lora_layer.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/lora_layer.py

What the commands do

  • Executes forward pass across 4096-dimension linear projections with rank-8 adapters.
  • Measures parameter counts and performs offline weight merging.

Expected output

[LORA] Base: 16,777,216 | LoRA Trainable: 65,536 (0.39%)
[MERGE] Merged delta into base weights bit-exactly.

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Zero delta output at step zero initialization
  • 99.6% parameter reduction on 4096x4096 projections
  • Bit-exact output agreement between LoRA dynamic forward pass and merged weight forward pass
  • Weight unmerge reversibility

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

If matrix multiplication fails, check tensor shapes: $A$ is $(r, d_{\text{in}})$, $B$ is $(d_{\text{out}}, r)$.

Security notes

Runs entirely offline on local CPU using standard NumPy tensors.

Extension exercises

Add support for quantization-aware simulated 4-bit NormalFloat weights.

Day number: 275 of 365

Expected output

FIELDS.md

Field definitions for LoRA linear layer parameters.

examples-run.txt

Example execution complete.

measured-values.txt

Base_Params=16781312
LoRA_Params=65536
Trainable_Pct=0.3905

run-output.txt

[LORA] Base: 16,777,216 | LoRA Trainable: 65,536 (0.39%)
[MERGE] Merged delta into base weights bit-exactly.

starter-run.txt

Starter execution complete.

test-output.txt

test-run.txt

test_lora_layer.py ...... [100%]
6 passed in 0.04s

Source files

examples/lora_layer.py (3324 bytes)
# Custom LoRA Linear Layer Implementation in NumPy
import numpy as np
from typing import Tuple, Dict, Any, Optional

class LoRALinear:
    """Low-Rank Adaptation Linear Layer wrapping a frozen base weight matrix."""

    def __init__(
        self,
        in_features: int,
        out_features: int,
        rank: int = 8,
        alpha: float = 16.0,
        lora_dropout: float = 0.0,
        random_seed: Optional[int] = 42
    ):
        self.in_features = in_features
        self.out_features = out_features
        self.rank = rank
        self.alpha = alpha
        self.scaling = alpha / rank if rank > 0 else 1.0
        self.lora_dropout = lora_dropout
        
        if random_seed is not None:
            np.random.seed(random_seed)

        # 1. Base Weight (Frozen)
        self.weight = np.random.randn(out_features, in_features).astype(np.float32) * 0.02
        self.bias = np.zeros(out_features, dtype=np.float32)
        self.frozen = True

        # 2. LoRA Adapter Matrices
        if self.rank > 0:
            # Matrix A: Gaussian initialization (r x in_features)
            self.lora_A = np.random.randn(rank, in_features).astype(np.float32) * (1.0 / np.sqrt(rank))
            # Matrix B: Zero initialization (out_features x r)
            self.lora_B = np.zeros((out_features, rank), dtype=np.float32)
        else:
            self.lora_A = None
            self.lora_B = None

        self.merged = False

    def forward(self, x: np.ndarray) -> np.ndarray:
        """Forward pass: h = x @ W_0^T + (alpha/r) * (x @ A^T) @ B^T + bias"""
        if self.merged or self.rank == 0:
            return x @ self.weight.T + self.bias
        
        # Base forward pass
        base_out = x @ self.weight.T
        
        # LoRA forward pass: (x @ A.T) @ B.T
        lora_out = (x @ self.lora_A.T) @ self.lora_B.T * self.scaling
        
        return base_out + lora_out + self.bias

    def merge_weights(self) -> None:
        """Merges adapter delta (alpha/r) * (B @ A) directly into base weights."""
        if self.merged:
            return
        if self.rank > 0 and self.lora_A is not None and self.lora_B is not None:
            delta_w = (self.lora_B @ self.lora_A) * self.scaling
            self.weight += delta_w
            self.merged = True

    def unmerge_weights(self) -> None:
        """Subtracts adapter delta from base weights to restore unmerged state."""
        if not self.merged:
            return
        if self.rank > 0 and self.lora_A is not None and self.lora_B is not None:
            delta_w = (self.lora_B @ self.lora_A) * self.scaling
            self.weight -= delta_w
            self.merged = False

    def count_parameters(self) -> Dict[str, int]:
        """Calculates frozen base parameters vs trainable LoRA parameters."""
        base_params = self.weight.size + self.bias.size
        lora_params = 0
        if self.rank > 0 and self.lora_A is not None and self.lora_B is not None:
            lora_params = self.lora_A.size + self.lora_B.size
        
        return {
            "base_parameters": base_params,
            "trainable_lora_parameters": lora_params,
            "total_parameters": base_params + lora_params,
            "trainable_percentage": (lora_params / base_params) * 100.0 if base_params > 0 else 0.0
        }
examples/test_lora_layer.py (2975 bytes)
import pytest
import numpy as np
from lora_layer import LoRALinear

def test_lora_initialization_zero_delta():
    layer = LoRALinear(in_features=128, out_features=256, rank=8, alpha=16.0, random_seed=42)
    x = np.random.randn(4, 128).astype(np.float32)
    
    # At step 0, B=0, so LoRA output must be bit-exact equal to base linear forward
    out = layer.forward(x)
    expected_base = x @ layer.weight.T + layer.bias
    np.testing.assert_allclose(out, expected_base, rtol=1e-5, atol=1e-6)

def test_parameter_counts_and_reduction():
    # 4096 -> 4096 projection (like Llama self-attention)
    layer = LoRALinear(in_features=4096, out_features=4096, rank=8, alpha=16.0)
    counts = layer.count_parameters()
    
    assert counts["base_parameters"] == 4096 * 4096 + 4096 # 16,781,312
    # LoRA A: 8 * 4096 = 32,768 | LoRA B: 4096 * 8 = 32,768 -> Total: 65,536
    assert counts["trainable_lora_parameters"] == 65536
    assert counts["trainable_percentage"] < 0.4 # Less than 0.4% of base

def test_lora_forward_with_trained_weights():
    layer = LoRALinear(in_features=64, out_features=32, rank=4, alpha=8.0, random_seed=42)
    # Simulate trained non-zero B matrix
    layer.lora_B = np.random.randn(32, 4).astype(np.float32) * 0.1
    
    x = np.random.randn(2, 64).astype(np.float32)
    out = layer.forward(x)
    
    # Calculate manual forward
    manual_base = x @ layer.weight.T + layer.bias
    manual_lora = (x @ layer.lora_A.T) @ layer.lora_B.T * (8.0 / 4.0)
    np.testing.assert_allclose(out, manual_base + manual_lora, rtol=1e-5, atol=1e-6)

def test_weight_merging_bit_exactness():
    layer = LoRALinear(in_features=64, out_features=32, rank=4, alpha=8.0, random_seed=42)
    layer.lora_B = np.random.randn(32, 4).astype(np.float32) * 0.1
    
    x = np.random.randn(5, 64).astype(np.float32)
    out_before_merge = layer.forward(x)
    
    # Merge weights offline
    layer.merge_weights()
    assert layer.merged is True
    
    out_after_merge = layer.forward(x)
    # Merged single matrix multiplication must be bit-exact to dual-path computation
    np.testing.assert_allclose(out_before_merge, out_after_merge, rtol=1e-5, atol=1e-6)

def test_unmerge_weights_reversibility():
    layer = LoRALinear(in_features=32, out_features=16, rank=4, alpha=8.0, random_seed=42)
    layer.lora_B = np.random.randn(16, 4).astype(np.float32) * 0.1
    
    initial_w = layer.weight.copy()
    layer.merge_weights()
    assert not np.array_equal(layer.weight, initial_w)
    
    layer.unmerge_weights()
    assert layer.merged is False
    np.testing.assert_allclose(layer.weight, initial_w, rtol=1e-5, atol=1e-6)

def test_zero_rank_pass_through():
    layer = LoRALinear(in_features=32, out_features=16, rank=0)
    counts = layer.count_parameters()
    assert counts["trainable_lora_parameters"] == 0
    x = np.random.randn(2, 32).astype(np.float32)
    out = layer.forward(x)
    np.testing.assert_allclose(out, x @ layer.weight.T + layer.bias)
metadata.yml (397 bytes)
lesson_id: D275
day: 275
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/lora_layer.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (28 bytes)
numpy>=1.24.0
pytest>=8.0.0
starter/lora_layer.py (1163 bytes)
# Starter: Custom LoRA Linear Layer in NumPy
import numpy as np
from typing import Dict, Optional

class LoRALinear:
    def __init__(
        self,
        in_features: int,
        out_features: int,
        rank: int = 8,
        alpha: float = 16.0,
        lora_dropout: float = 0.0,
        random_seed: Optional[int] = 42
    ):
        self.in_features = in_features
        self.out_features = out_features
        self.rank = rank
        self.alpha = alpha
        self.scaling = alpha / rank if rank > 0 else 1.0
        self.weight = np.random.randn(out_features, in_features).astype(np.float32) * 0.02
        self.bias = np.zeros(out_features, dtype=np.float32)
        self.merged = False

    def forward(self, x: np.ndarray) -> np.ndarray:
        return x @ self.weight.T + self.bias

    def merge_weights(self) -> None:
        self.merged = True

    def count_parameters(self) -> Dict[str, int]:
        return {
            "base_parameters": self.weight.size + self.bias.size,
            "trainable_lora_parameters": 0,
            "total_parameters": self.weight.size + self.bias.size,
            "trainable_percentage": 0.0
        }
starter/test_lora_layer.py (2975 bytes)
import pytest
import numpy as np
from lora_layer import LoRALinear

def test_lora_initialization_zero_delta():
    layer = LoRALinear(in_features=128, out_features=256, rank=8, alpha=16.0, random_seed=42)
    x = np.random.randn(4, 128).astype(np.float32)
    
    # At step 0, B=0, so LoRA output must be bit-exact equal to base linear forward
    out = layer.forward(x)
    expected_base = x @ layer.weight.T + layer.bias
    np.testing.assert_allclose(out, expected_base, rtol=1e-5, atol=1e-6)

def test_parameter_counts_and_reduction():
    # 4096 -> 4096 projection (like Llama self-attention)
    layer = LoRALinear(in_features=4096, out_features=4096, rank=8, alpha=16.0)
    counts = layer.count_parameters()
    
    assert counts["base_parameters"] == 4096 * 4096 + 4096 # 16,781,312
    # LoRA A: 8 * 4096 = 32,768 | LoRA B: 4096 * 8 = 32,768 -> Total: 65,536
    assert counts["trainable_lora_parameters"] == 65536
    assert counts["trainable_percentage"] < 0.4 # Less than 0.4% of base

def test_lora_forward_with_trained_weights():
    layer = LoRALinear(in_features=64, out_features=32, rank=4, alpha=8.0, random_seed=42)
    # Simulate trained non-zero B matrix
    layer.lora_B = np.random.randn(32, 4).astype(np.float32) * 0.1
    
    x = np.random.randn(2, 64).astype(np.float32)
    out = layer.forward(x)
    
    # Calculate manual forward
    manual_base = x @ layer.weight.T + layer.bias
    manual_lora = (x @ layer.lora_A.T) @ layer.lora_B.T * (8.0 / 4.0)
    np.testing.assert_allclose(out, manual_base + manual_lora, rtol=1e-5, atol=1e-6)

def test_weight_merging_bit_exactness():
    layer = LoRALinear(in_features=64, out_features=32, rank=4, alpha=8.0, random_seed=42)
    layer.lora_B = np.random.randn(32, 4).astype(np.float32) * 0.1
    
    x = np.random.randn(5, 64).astype(np.float32)
    out_before_merge = layer.forward(x)
    
    # Merge weights offline
    layer.merge_weights()
    assert layer.merged is True
    
    out_after_merge = layer.forward(x)
    # Merged single matrix multiplication must be bit-exact to dual-path computation
    np.testing.assert_allclose(out_before_merge, out_after_merge, rtol=1e-5, atol=1e-6)

def test_unmerge_weights_reversibility():
    layer = LoRALinear(in_features=32, out_features=16, rank=4, alpha=8.0, random_seed=42)
    layer.lora_B = np.random.randn(16, 4).astype(np.float32) * 0.1
    
    initial_w = layer.weight.copy()
    layer.merge_weights()
    assert not np.array_equal(layer.weight, initial_w)
    
    layer.unmerge_weights()
    assert layer.merged is False
    np.testing.assert_allclose(layer.weight, initial_w, rtol=1e-5, atol=1e-6)

def test_zero_rank_pass_through():
    layer = LoRALinear(in_features=32, out_features=16, rank=0)
    counts = layer.count_parameters()
    assert counts["trainable_lora_parameters"] == 0
    x = np.random.randn(2, 32).astype(np.float32)
    out = layer.forward(x)
    np.testing.assert_allclose(out, x @ layer.weight.T + layer.bias)
tests/run_tests.sh (119 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
PYTEST="${PYTEST:-pytest}"
$PYTEST tests/test_lora_layer.py -v
tests/test_lora_layer.py (2975 bytes)
import pytest
import numpy as np
from lora_layer import LoRALinear

def test_lora_initialization_zero_delta():
    layer = LoRALinear(in_features=128, out_features=256, rank=8, alpha=16.0, random_seed=42)
    x = np.random.randn(4, 128).astype(np.float32)
    
    # At step 0, B=0, so LoRA output must be bit-exact equal to base linear forward
    out = layer.forward(x)
    expected_base = x @ layer.weight.T + layer.bias
    np.testing.assert_allclose(out, expected_base, rtol=1e-5, atol=1e-6)

def test_parameter_counts_and_reduction():
    # 4096 -> 4096 projection (like Llama self-attention)
    layer = LoRALinear(in_features=4096, out_features=4096, rank=8, alpha=16.0)
    counts = layer.count_parameters()
    
    assert counts["base_parameters"] == 4096 * 4096 + 4096 # 16,781,312
    # LoRA A: 8 * 4096 = 32,768 | LoRA B: 4096 * 8 = 32,768 -> Total: 65,536
    assert counts["trainable_lora_parameters"] == 65536
    assert counts["trainable_percentage"] < 0.4 # Less than 0.4% of base

def test_lora_forward_with_trained_weights():
    layer = LoRALinear(in_features=64, out_features=32, rank=4, alpha=8.0, random_seed=42)
    # Simulate trained non-zero B matrix
    layer.lora_B = np.random.randn(32, 4).astype(np.float32) * 0.1
    
    x = np.random.randn(2, 64).astype(np.float32)
    out = layer.forward(x)
    
    # Calculate manual forward
    manual_base = x @ layer.weight.T + layer.bias
    manual_lora = (x @ layer.lora_A.T) @ layer.lora_B.T * (8.0 / 4.0)
    np.testing.assert_allclose(out, manual_base + manual_lora, rtol=1e-5, atol=1e-6)

def test_weight_merging_bit_exactness():
    layer = LoRALinear(in_features=64, out_features=32, rank=4, alpha=8.0, random_seed=42)
    layer.lora_B = np.random.randn(32, 4).astype(np.float32) * 0.1
    
    x = np.random.randn(5, 64).astype(np.float32)
    out_before_merge = layer.forward(x)
    
    # Merge weights offline
    layer.merge_weights()
    assert layer.merged is True
    
    out_after_merge = layer.forward(x)
    # Merged single matrix multiplication must be bit-exact to dual-path computation
    np.testing.assert_allclose(out_before_merge, out_after_merge, rtol=1e-5, atol=1e-6)

def test_unmerge_weights_reversibility():
    layer = LoRALinear(in_features=32, out_features=16, rank=4, alpha=8.0, random_seed=42)
    layer.lora_B = np.random.randn(16, 4).astype(np.float32) * 0.1
    
    initial_w = layer.weight.copy()
    layer.merge_weights()
    assert not np.array_equal(layer.weight, initial_w)
    
    layer.unmerge_weights()
    assert layer.merged is False
    np.testing.assert_allclose(layer.weight, initial_w, rtol=1e-5, atol=1e-6)

def test_zero_rank_pass_through():
    layer = LoRALinear(in_features=32, out_features=16, rank=0)
    counts = layer.count_parameters()
    assert counts["trainable_lora_parameters"] == 0
    x = np.random.randn(2, 32).astype(np.float32)
    out = layer.forward(x)
    np.testing.assert_allclose(out, x @ layer.weight.T + layer.bias)

Troubleshooting

Troubleshooting

If dimension mismatch occurs during forward pass, verify matrix transpose operations: (x @ A.T) @ B.T.

Security notes

Security

Pure local NumPy linear algebra implementation. Zero external network dependency or unauthenticated remote code execution.