LLMs and Generative AI › Customizing and Running Models › Day 275
Hands-on lab — Day 275: Fine-Tuning with LoRA
- ← Back to the Day 275 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-275-fine-tuning-with-lora/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/lora_layer.py Test
bash tests/run_tests.sh File tree
examples/lora_layer.py examples/test_lora_layer.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/lora_layer.py starter/test_lora_layer.py tests/run_tests.sh tests/test_lora_layer.py troubleshooting.md
Lab README
Day 275 Lab: Custom LoRA Linear Layer in NumPy
Lesson
- Lesson title: Fine-Tuning with LoRA
- Day number: 275 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-275-fine-tuning-with-lora
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-275-fine-tuning-with-lorawhen the site is running.
Purpose
Build a custom Low-Rank Adaptation (LoRA) linear layer from scratch in Python and NumPy, implementing low-rank matrix decomposition, parameter reduction accounting, and zero-overhead offline weight merging.
Learning objectives
- Implement low-rank decomposition matrices ($A$ and $B$) with Gaussian and zero initializations.
- Compute scaling factors ($\alpha / r$) and forward pass activations.
- Verify zero delta updates at step zero.
- Implement offline weight merging ($W_{\text{merged}} = W_0 + \Delta W$) and verify bit-exact output parity.
Prerequisites
- Python 3.10+ installed
- NumPy and pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, numpy, pytest
Free and open-source options
- Python Standard Library, NumPy
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/lora_layer.py: Starter implementation skeletonexamples/lora_layer.py: Verified reference implementationtests/test_lora_layer.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/lora_layer.py
What the commands do
- Executes forward pass across 4096-dimension linear projections with rank-8 adapters.
- Measures parameter counts and performs offline weight merging.
Expected output
[LORA] Base: 16,777,216 | LoRA Trainable: 65,536 (0.39%)
[MERGE] Merged delta into base weights bit-exactly.
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Zero delta output at step zero initialization
- 99.6% parameter reduction on 4096x4096 projections
- Bit-exact output agreement between LoRA dynamic forward pass and merged weight forward pass
- Weight unmerge reversibility
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
If matrix multiplication fails, check tensor shapes: $A$ is $(r, d_{\text{in}})$, $B$ is $(d_{\text{out}}, r)$.
Security notes
Runs entirely offline on local CPU using standard NumPy tensors.
Extension exercises
Add support for quantization-aware simulated 4-bit NormalFloat weights.
Navigation
Day number: 275 of 365
Expected output
FIELDS.md
Field definitions for LoRA linear layer parameters. examples-run.txt
Example execution complete.
measured-values.txt
Base_Params=16781312
LoRA_Params=65536
Trainable_Pct=0.3905
run-output.txt
[LORA] Base: 16,777,216 | LoRA Trainable: 65,536 (0.39%)
[MERGE] Merged delta into base weights bit-exactly.
starter-run.txt
Starter execution complete.
test-output.txt
test-run.txt
test_lora_layer.py ...... [100%]
6 passed in 0.04s
Source files
examples/lora_layer.py (3324 bytes)
# Custom LoRA Linear Layer Implementation in NumPy
import numpy as np
from typing import Tuple, Dict, Any, Optional
class LoRALinear:
"""Low-Rank Adaptation Linear Layer wrapping a frozen base weight matrix."""
def __init__(
self,
in_features: int,
out_features: int,
rank: int = 8,
alpha: float = 16.0,
lora_dropout: float = 0.0,
random_seed: Optional[int] = 42
):
self.in_features = in_features
self.out_features = out_features
self.rank = rank
self.alpha = alpha
self.scaling = alpha / rank if rank > 0 else 1.0
self.lora_dropout = lora_dropout
if random_seed is not None:
np.random.seed(random_seed)
# 1. Base Weight (Frozen)
self.weight = np.random.randn(out_features, in_features).astype(np.float32) * 0.02
self.bias = np.zeros(out_features, dtype=np.float32)
self.frozen = True
# 2. LoRA Adapter Matrices
if self.rank > 0:
# Matrix A: Gaussian initialization (r x in_features)
self.lora_A = np.random.randn(rank, in_features).astype(np.float32) * (1.0 / np.sqrt(rank))
# Matrix B: Zero initialization (out_features x r)
self.lora_B = np.zeros((out_features, rank), dtype=np.float32)
else:
self.lora_A = None
self.lora_B = None
self.merged = False
def forward(self, x: np.ndarray) -> np.ndarray:
"""Forward pass: h = x @ W_0^T + (alpha/r) * (x @ A^T) @ B^T + bias"""
if self.merged or self.rank == 0:
return x @ self.weight.T + self.bias
# Base forward pass
base_out = x @ self.weight.T
# LoRA forward pass: (x @ A.T) @ B.T
lora_out = (x @ self.lora_A.T) @ self.lora_B.T * self.scaling
return base_out + lora_out + self.bias
def merge_weights(self) -> None:
"""Merges adapter delta (alpha/r) * (B @ A) directly into base weights."""
if self.merged:
return
if self.rank > 0 and self.lora_A is not None and self.lora_B is not None:
delta_w = (self.lora_B @ self.lora_A) * self.scaling
self.weight += delta_w
self.merged = True
def unmerge_weights(self) -> None:
"""Subtracts adapter delta from base weights to restore unmerged state."""
if not self.merged:
return
if self.rank > 0 and self.lora_A is not None and self.lora_B is not None:
delta_w = (self.lora_B @ self.lora_A) * self.scaling
self.weight -= delta_w
self.merged = False
def count_parameters(self) -> Dict[str, int]:
"""Calculates frozen base parameters vs trainable LoRA parameters."""
base_params = self.weight.size + self.bias.size
lora_params = 0
if self.rank > 0 and self.lora_A is not None and self.lora_B is not None:
lora_params = self.lora_A.size + self.lora_B.size
return {
"base_parameters": base_params,
"trainable_lora_parameters": lora_params,
"total_parameters": base_params + lora_params,
"trainable_percentage": (lora_params / base_params) * 100.0 if base_params > 0 else 0.0
}
examples/test_lora_layer.py (2975 bytes)
import pytest
import numpy as np
from lora_layer import LoRALinear
def test_lora_initialization_zero_delta():
layer = LoRALinear(in_features=128, out_features=256, rank=8, alpha=16.0, random_seed=42)
x = np.random.randn(4, 128).astype(np.float32)
# At step 0, B=0, so LoRA output must be bit-exact equal to base linear forward
out = layer.forward(x)
expected_base = x @ layer.weight.T + layer.bias
np.testing.assert_allclose(out, expected_base, rtol=1e-5, atol=1e-6)
def test_parameter_counts_and_reduction():
# 4096 -> 4096 projection (like Llama self-attention)
layer = LoRALinear(in_features=4096, out_features=4096, rank=8, alpha=16.0)
counts = layer.count_parameters()
assert counts["base_parameters"] == 4096 * 4096 + 4096 # 16,781,312
# LoRA A: 8 * 4096 = 32,768 | LoRA B: 4096 * 8 = 32,768 -> Total: 65,536
assert counts["trainable_lora_parameters"] == 65536
assert counts["trainable_percentage"] < 0.4 # Less than 0.4% of base
def test_lora_forward_with_trained_weights():
layer = LoRALinear(in_features=64, out_features=32, rank=4, alpha=8.0, random_seed=42)
# Simulate trained non-zero B matrix
layer.lora_B = np.random.randn(32, 4).astype(np.float32) * 0.1
x = np.random.randn(2, 64).astype(np.float32)
out = layer.forward(x)
# Calculate manual forward
manual_base = x @ layer.weight.T + layer.bias
manual_lora = (x @ layer.lora_A.T) @ layer.lora_B.T * (8.0 / 4.0)
np.testing.assert_allclose(out, manual_base + manual_lora, rtol=1e-5, atol=1e-6)
def test_weight_merging_bit_exactness():
layer = LoRALinear(in_features=64, out_features=32, rank=4, alpha=8.0, random_seed=42)
layer.lora_B = np.random.randn(32, 4).astype(np.float32) * 0.1
x = np.random.randn(5, 64).astype(np.float32)
out_before_merge = layer.forward(x)
# Merge weights offline
layer.merge_weights()
assert layer.merged is True
out_after_merge = layer.forward(x)
# Merged single matrix multiplication must be bit-exact to dual-path computation
np.testing.assert_allclose(out_before_merge, out_after_merge, rtol=1e-5, atol=1e-6)
def test_unmerge_weights_reversibility():
layer = LoRALinear(in_features=32, out_features=16, rank=4, alpha=8.0, random_seed=42)
layer.lora_B = np.random.randn(16, 4).astype(np.float32) * 0.1
initial_w = layer.weight.copy()
layer.merge_weights()
assert not np.array_equal(layer.weight, initial_w)
layer.unmerge_weights()
assert layer.merged is False
np.testing.assert_allclose(layer.weight, initial_w, rtol=1e-5, atol=1e-6)
def test_zero_rank_pass_through():
layer = LoRALinear(in_features=32, out_features=16, rank=0)
counts = layer.count_parameters()
assert counts["trainable_lora_parameters"] == 0
x = np.random.randn(2, 32).astype(np.float32)
out = layer.forward(x)
np.testing.assert_allclose(out, x @ layer.weight.T + layer.bias)
metadata.yml (397 bytes)
lesson_id: D275
day: 275
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/lora_layer.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (28 bytes)
numpy>=1.24.0
pytest>=8.0.0
starter/lora_layer.py (1163 bytes)
# Starter: Custom LoRA Linear Layer in NumPy
import numpy as np
from typing import Dict, Optional
class LoRALinear:
def __init__(
self,
in_features: int,
out_features: int,
rank: int = 8,
alpha: float = 16.0,
lora_dropout: float = 0.0,
random_seed: Optional[int] = 42
):
self.in_features = in_features
self.out_features = out_features
self.rank = rank
self.alpha = alpha
self.scaling = alpha / rank if rank > 0 else 1.0
self.weight = np.random.randn(out_features, in_features).astype(np.float32) * 0.02
self.bias = np.zeros(out_features, dtype=np.float32)
self.merged = False
def forward(self, x: np.ndarray) -> np.ndarray:
return x @ self.weight.T + self.bias
def merge_weights(self) -> None:
self.merged = True
def count_parameters(self) -> Dict[str, int]:
return {
"base_parameters": self.weight.size + self.bias.size,
"trainable_lora_parameters": 0,
"total_parameters": self.weight.size + self.bias.size,
"trainable_percentage": 0.0
}
starter/test_lora_layer.py (2975 bytes)
import pytest
import numpy as np
from lora_layer import LoRALinear
def test_lora_initialization_zero_delta():
layer = LoRALinear(in_features=128, out_features=256, rank=8, alpha=16.0, random_seed=42)
x = np.random.randn(4, 128).astype(np.float32)
# At step 0, B=0, so LoRA output must be bit-exact equal to base linear forward
out = layer.forward(x)
expected_base = x @ layer.weight.T + layer.bias
np.testing.assert_allclose(out, expected_base, rtol=1e-5, atol=1e-6)
def test_parameter_counts_and_reduction():
# 4096 -> 4096 projection (like Llama self-attention)
layer = LoRALinear(in_features=4096, out_features=4096, rank=8, alpha=16.0)
counts = layer.count_parameters()
assert counts["base_parameters"] == 4096 * 4096 + 4096 # 16,781,312
# LoRA A: 8 * 4096 = 32,768 | LoRA B: 4096 * 8 = 32,768 -> Total: 65,536
assert counts["trainable_lora_parameters"] == 65536
assert counts["trainable_percentage"] < 0.4 # Less than 0.4% of base
def test_lora_forward_with_trained_weights():
layer = LoRALinear(in_features=64, out_features=32, rank=4, alpha=8.0, random_seed=42)
# Simulate trained non-zero B matrix
layer.lora_B = np.random.randn(32, 4).astype(np.float32) * 0.1
x = np.random.randn(2, 64).astype(np.float32)
out = layer.forward(x)
# Calculate manual forward
manual_base = x @ layer.weight.T + layer.bias
manual_lora = (x @ layer.lora_A.T) @ layer.lora_B.T * (8.0 / 4.0)
np.testing.assert_allclose(out, manual_base + manual_lora, rtol=1e-5, atol=1e-6)
def test_weight_merging_bit_exactness():
layer = LoRALinear(in_features=64, out_features=32, rank=4, alpha=8.0, random_seed=42)
layer.lora_B = np.random.randn(32, 4).astype(np.float32) * 0.1
x = np.random.randn(5, 64).astype(np.float32)
out_before_merge = layer.forward(x)
# Merge weights offline
layer.merge_weights()
assert layer.merged is True
out_after_merge = layer.forward(x)
# Merged single matrix multiplication must be bit-exact to dual-path computation
np.testing.assert_allclose(out_before_merge, out_after_merge, rtol=1e-5, atol=1e-6)
def test_unmerge_weights_reversibility():
layer = LoRALinear(in_features=32, out_features=16, rank=4, alpha=8.0, random_seed=42)
layer.lora_B = np.random.randn(16, 4).astype(np.float32) * 0.1
initial_w = layer.weight.copy()
layer.merge_weights()
assert not np.array_equal(layer.weight, initial_w)
layer.unmerge_weights()
assert layer.merged is False
np.testing.assert_allclose(layer.weight, initial_w, rtol=1e-5, atol=1e-6)
def test_zero_rank_pass_through():
layer = LoRALinear(in_features=32, out_features=16, rank=0)
counts = layer.count_parameters()
assert counts["trainable_lora_parameters"] == 0
x = np.random.randn(2, 32).astype(np.float32)
out = layer.forward(x)
np.testing.assert_allclose(out, x @ layer.weight.T + layer.bias)
tests/run_tests.sh (119 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
PYTEST="${PYTEST:-pytest}"
$PYTEST tests/test_lora_layer.py -v
tests/test_lora_layer.py (2975 bytes)
import pytest
import numpy as np
from lora_layer import LoRALinear
def test_lora_initialization_zero_delta():
layer = LoRALinear(in_features=128, out_features=256, rank=8, alpha=16.0, random_seed=42)
x = np.random.randn(4, 128).astype(np.float32)
# At step 0, B=0, so LoRA output must be bit-exact equal to base linear forward
out = layer.forward(x)
expected_base = x @ layer.weight.T + layer.bias
np.testing.assert_allclose(out, expected_base, rtol=1e-5, atol=1e-6)
def test_parameter_counts_and_reduction():
# 4096 -> 4096 projection (like Llama self-attention)
layer = LoRALinear(in_features=4096, out_features=4096, rank=8, alpha=16.0)
counts = layer.count_parameters()
assert counts["base_parameters"] == 4096 * 4096 + 4096 # 16,781,312
# LoRA A: 8 * 4096 = 32,768 | LoRA B: 4096 * 8 = 32,768 -> Total: 65,536
assert counts["trainable_lora_parameters"] == 65536
assert counts["trainable_percentage"] < 0.4 # Less than 0.4% of base
def test_lora_forward_with_trained_weights():
layer = LoRALinear(in_features=64, out_features=32, rank=4, alpha=8.0, random_seed=42)
# Simulate trained non-zero B matrix
layer.lora_B = np.random.randn(32, 4).astype(np.float32) * 0.1
x = np.random.randn(2, 64).astype(np.float32)
out = layer.forward(x)
# Calculate manual forward
manual_base = x @ layer.weight.T + layer.bias
manual_lora = (x @ layer.lora_A.T) @ layer.lora_B.T * (8.0 / 4.0)
np.testing.assert_allclose(out, manual_base + manual_lora, rtol=1e-5, atol=1e-6)
def test_weight_merging_bit_exactness():
layer = LoRALinear(in_features=64, out_features=32, rank=4, alpha=8.0, random_seed=42)
layer.lora_B = np.random.randn(32, 4).astype(np.float32) * 0.1
x = np.random.randn(5, 64).astype(np.float32)
out_before_merge = layer.forward(x)
# Merge weights offline
layer.merge_weights()
assert layer.merged is True
out_after_merge = layer.forward(x)
# Merged single matrix multiplication must be bit-exact to dual-path computation
np.testing.assert_allclose(out_before_merge, out_after_merge, rtol=1e-5, atol=1e-6)
def test_unmerge_weights_reversibility():
layer = LoRALinear(in_features=32, out_features=16, rank=4, alpha=8.0, random_seed=42)
layer.lora_B = np.random.randn(16, 4).astype(np.float32) * 0.1
initial_w = layer.weight.copy()
layer.merge_weights()
assert not np.array_equal(layer.weight, initial_w)
layer.unmerge_weights()
assert layer.merged is False
np.testing.assert_allclose(layer.weight, initial_w, rtol=1e-5, atol=1e-6)
def test_zero_rank_pass_through():
layer = LoRALinear(in_features=32, out_features=16, rank=0)
counts = layer.count_parameters()
assert counts["trainable_lora_parameters"] == 0
x = np.random.randn(2, 32).astype(np.float32)
out = layer.forward(x)
np.testing.assert_allclose(out, x @ layer.weight.T + layer.bias)
Troubleshooting
Troubleshooting
If dimension mismatch occurs during forward pass, verify matrix transpose operations: (x @ A.T) @ B.T.
Security notes
Security
Pure local NumPy linear algebra implementation. Zero external network dependency or unauthenticated remote code execution.