LLMs and Generative AI › Customizing and Running Models › Day 280
Hands-on lab — Day 280: Fine-Tune and Serve Your Own Model
- ← Back to the Day 280 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-280-fine-tune-and-serve-your-own/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/finetune_pipeline.py Test
bash tests/run_tests.sh File tree
examples/finetune_pipeline.py examples/test_finetune_pipeline.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/finetune_pipeline.py starter/test_finetune_pipeline.py tests/run_tests.sh tests/test_finetune_pipeline.py troubleshooting.md
Lab README
Day 280 Lab: End-to-End Fine-Tuning Pipeline
Lesson
- Lesson title: Fine-Tune and Serve Your Own Model
- Day number: 280 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-280-fine-tune-and-serve-your-own
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-280-fine-tune-and-serve-your-ownwhen the site is running.
Purpose
Build an end-to-end Python pipeline that merges LoRA adapter weights into base tensors, exposes a local mock serving endpoint, and quantitatively evaluates accuracy and schema compliance gains.
Learning objectives
- Implement exact LoRA weight merging: $W_{\text{merged}} = W_{\text{base}} + (B \cdot A) \cdot (\alpha / r)$.
- Simulate local REST model serving endpoints.
- Build an automated evaluation benchmark computing JSON compliance and Exact Match (EM).
- Quantify performance improvements over baseline models.
Prerequisites
- Python 3.10+ installed
- NumPy and pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, numpy, pytest
Free and open-source options
- Python Standard Library, NumPy
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/finetune_pipeline.py: Starter implementation skeletonexamples/finetune_pipeline.py: Verified reference implementationtests/test_finetune_pipeline.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/finetune_pipeline.py
What the commands do
- Executes LoRA adapter weight merging, simulates local inference, and runs benchmark evaluation.
Expected output
[PIPELINE] Merged LoRA Adapter into base model.
[BENCHMARK] Fine-Tuned Model achieved 100% JSON Schema Compliance vs 0% Base Model.
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Matrix multiplication and scaling for weight merging
- JSON syntax validation logic
- Exact match string parsing
- Comparative benchmark metrics
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Ensure matrix dimensions for LoRA $B$ and $A$ align with $(d_{\text{out}}, r) \times (r, d_{\text{in}})$.
Security notes
Runs completely offline on local CPU without third-party network access.
Extension exercises
Add AST-based SQL query parsing evaluation using sqlparse.
Navigation
Day number: 280 of 365
Expected output
FIELDS.md
Field definitions for fine-tuning pipeline. examples-run.txt
Example execution complete.
measured-values.txt
Base_JSON_Compliance=0.0
FineTuned_JSON_Compliance=100.0
Accuracy_Boost=100.0
run-output.txt
[PIPELINE] Merged LoRA Adapter into base model.
[BENCHMARK] Fine-Tuned Model achieved 100% JSON Schema Compliance vs 0% Base Model.
starter-run.txt
Starter execution complete.
test-output.txt
test-run.txt
test_finetune_pipeline.py ..... [100%]
5 passed in 0.04s
Source files
examples/finetune_pipeline.py (4070 bytes)
# End-to-End Fine-Tuning, Merging, Serving, and Evaluation Pipeline
import numpy as np
import json
from typing import Dict, List, Any, Tuple, Optional
class LoRAWeightMerger:
"""Performs exact mathematical merging of LoRA adapters into base model weights."""
@staticmethod
def merge_weights(base_w: np.ndarray, lora_a: np.ndarray, lora_b: np.ndarray, r: int, alpha: float) -> np.ndarray:
"""
Merges W_merged = W_base + (B @ A) * (alpha / r)
base_w: (d_out, d_in)
lora_a: (r, d_in)
lora_b: (d_out, r)
"""
if r <= 0:
raise ValueError("LoRA rank r must be > 0")
scaling = float(alpha) / float(r)
delta_w = np.matmul(lora_b, lora_a) * scaling
if delta_w.shape != base_w.shape:
raise ValueError(f"Shape mismatch: delta_w {delta_w.shape} vs base_w {base_w.shape}")
return (base_w + delta_w).astype(np.float32)
class MockModelServer:
"""Simulates local OpenAI-compatible inference server."""
def __init__(self, is_finetuned: bool = True):
self.is_finetuned = is_finetuned
def generate_completion(self, prompt: str) -> str:
"""Simulates generation based on model type."""
if "sql" in prompt.lower() or "query" in prompt.lower():
if self.is_finetuned:
return '```sql\nSELECT id, username, email FROM users WHERE active = 1;\n```'
else:
return 'Sure! Here is some information about users: You can write a query like SELECT * FROM users probably.'
elif "json" in prompt.lower() or "schema" in prompt.lower():
if self.is_finetuned:
return '{"status": "success", "user_id": 1042, "role": "admin"}'
else:
return 'Here is your JSON output: status: success, user_id: 1042 (Note: this is not valid JSON)'
else:
return "Standard response."
class ModelEvaluationBenchmark:
"""Automated benchmark evaluating schema compliance and exact match accuracy."""
@staticmethod
def evaluate_json_compliance(predictions: List[str]) -> float:
"""Measures percentage of predictions that are valid JSON."""
if not predictions:
return 0.0
valid_count = 0
for pred in predictions:
try:
json.loads(pred.strip())
valid_count += 1
except Exception:
pass
return (valid_count / len(predictions)) * 100.0
@staticmethod
def evaluate_exact_match(predictions: List[str], targets: List[str]) -> float:
"""Measures percentage of exact match against golden targets."""
if not predictions or len(predictions) != len(targets):
return 0.0
matches = sum(1 for p, t in zip(predictions, targets) if p.strip() == t.strip())
return (matches / len(predictions)) * 100.0
@staticmethod
def run_benchmark_comparison(base_server: MockModelServer, ft_server: MockModelServer, test_cases: List[Dict[str, str]]) -> Dict[str, Any]:
"""Compares base model vs fine-tuned model on test cases."""
prompts = [tc["prompt"] for tc in test_cases]
targets = [tc["target"] for tc in test_cases]
base_preds = [base_server.generate_completion(p) for p in prompts]
ft_preds = [ft_server.generate_completion(p) for p in prompts]
base_json_rate = ModelEvaluationBenchmark.evaluate_json_compliance(base_preds)
ft_json_rate = ModelEvaluationBenchmark.evaluate_json_compliance(ft_preds)
base_em = ModelEvaluationBenchmark.evaluate_exact_match(base_preds, targets)
ft_em = ModelEvaluationBenchmark.evaluate_exact_match(ft_preds, targets)
return {
"total_cases": len(test_cases),
"base_json_compliance": base_json_rate,
"finetuned_json_compliance": ft_json_rate,
"base_exact_match": base_em,
"finetuned_exact_match": ft_em,
"accuracy_improvement": ft_em - base_em
}
examples/test_finetune_pipeline.py (2316 bytes)
import pytest
import numpy as np
from finetune_pipeline import LoRAWeightMerger, MockModelServer, ModelEvaluationBenchmark
def test_lora_merging_math():
np.random.seed(42)
d_out, d_in, r = 32, 32, 4
alpha = 8.0
base_w = np.random.randn(d_out, d_in).astype(np.float32)
lora_a = np.random.randn(r, d_in).astype(np.float32)
lora_b = np.random.randn(d_out, r).astype(np.float32)
merged = LoRAWeightMerger.merge_weights(base_w, lora_a, lora_b, r=r, alpha=alpha)
expected_delta = np.matmul(lora_b, lora_a) * (8.0 / 4.0)
expected_merged = base_w + expected_delta
assert np.allclose(merged, expected_merged, atol=1e-5)
def test_lora_invalid_rank():
base_w = np.zeros((10, 10))
lora_a = np.zeros((0, 10))
lora_b = np.zeros((10, 0))
with pytest.raises(ValueError, match="LoRA rank r must be > 0"):
LoRAWeightMerger.merge_weights(base_w, lora_a, lora_b, r=0, alpha=16.0)
def test_json_compliance_evaluation():
valid_cases = ['{"name": "test"}', '{"status": 200, "items": [1, 2]}']
invalid_cases = ['Not JSON', '{"broken": json}']
score = ModelEvaluationBenchmark.evaluate_json_compliance(valid_cases + invalid_cases)
assert score == 50.0 # 2 out of 4 valid
def test_exact_match_evaluation():
preds = ["SELECT * FROM users;", " SELECT id FROM items; ", "INVALID"]
targets = ["SELECT * FROM users;", "SELECT id FROM items;", "SELECT * FROM orders;"]
em = ModelEvaluationBenchmark.evaluate_exact_match(preds, targets)
assert em == pytest.approx(66.66, 0.1) # 2 out of 3 match
def test_end_to_end_benchmark_comparison():
base_srv = MockModelServer(is_finetuned=False)
ft_srv = MockModelServer(is_finetuned=True)
test_cases = [
{"prompt": "Generate JSON schema for user 1042", "target": '{"status": "success", "user_id": 1042, "role": "admin"}'},
{"prompt": "Generate JSON schema for admin", "target": '{"status": "success", "user_id": 1042, "role": "admin"}'}
]
results = ModelEvaluationBenchmark.run_benchmark_comparison(base_srv, ft_srv, test_cases)
assert results["finetuned_json_compliance"] == 100.0
assert results["base_json_compliance"] == 0.0
assert results["finetuned_exact_match"] == 100.0
assert results["accuracy_improvement"] == 100.0
metadata.yml (404 bytes)
lesson_id: D280
day: 280
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/finetune_pipeline.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (28 bytes)
numpy>=1.24.0
pytest>=8.0.0
starter/finetune_pipeline.py (725 bytes)
# Starter: End-to-End Pipeline
import numpy as np
from typing import Dict, List, Any
class LoRAWeightMerger:
@staticmethod
def merge_weights(base_w: np.ndarray, lora_a: np.ndarray, lora_b: np.ndarray, r: int, alpha: float) -> np.ndarray:
return base_w
class MockModelServer:
def __init__(self, is_finetuned: bool = True):
self.is_finetuned = is_finetuned
def generate_completion(self, prompt: str) -> str:
return "response"
class ModelEvaluationBenchmark:
@staticmethod
def evaluate_json_compliance(predictions: List[str]) -> float:
return 0.0
@staticmethod
def evaluate_exact_match(predictions: List[str], targets: List[str]) -> float:
return 0.0
starter/test_finetune_pipeline.py (2316 bytes)
import pytest
import numpy as np
from finetune_pipeline import LoRAWeightMerger, MockModelServer, ModelEvaluationBenchmark
def test_lora_merging_math():
np.random.seed(42)
d_out, d_in, r = 32, 32, 4
alpha = 8.0
base_w = np.random.randn(d_out, d_in).astype(np.float32)
lora_a = np.random.randn(r, d_in).astype(np.float32)
lora_b = np.random.randn(d_out, r).astype(np.float32)
merged = LoRAWeightMerger.merge_weights(base_w, lora_a, lora_b, r=r, alpha=alpha)
expected_delta = np.matmul(lora_b, lora_a) * (8.0 / 4.0)
expected_merged = base_w + expected_delta
assert np.allclose(merged, expected_merged, atol=1e-5)
def test_lora_invalid_rank():
base_w = np.zeros((10, 10))
lora_a = np.zeros((0, 10))
lora_b = np.zeros((10, 0))
with pytest.raises(ValueError, match="LoRA rank r must be > 0"):
LoRAWeightMerger.merge_weights(base_w, lora_a, lora_b, r=0, alpha=16.0)
def test_json_compliance_evaluation():
valid_cases = ['{"name": "test"}', '{"status": 200, "items": [1, 2]}']
invalid_cases = ['Not JSON', '{"broken": json}']
score = ModelEvaluationBenchmark.evaluate_json_compliance(valid_cases + invalid_cases)
assert score == 50.0 # 2 out of 4 valid
def test_exact_match_evaluation():
preds = ["SELECT * FROM users;", " SELECT id FROM items; ", "INVALID"]
targets = ["SELECT * FROM users;", "SELECT id FROM items;", "SELECT * FROM orders;"]
em = ModelEvaluationBenchmark.evaluate_exact_match(preds, targets)
assert em == pytest.approx(66.66, 0.1) # 2 out of 3 match
def test_end_to_end_benchmark_comparison():
base_srv = MockModelServer(is_finetuned=False)
ft_srv = MockModelServer(is_finetuned=True)
test_cases = [
{"prompt": "Generate JSON schema for user 1042", "target": '{"status": "success", "user_id": 1042, "role": "admin"}'},
{"prompt": "Generate JSON schema for admin", "target": '{"status": "success", "user_id": 1042, "role": "admin"}'}
]
results = ModelEvaluationBenchmark.run_benchmark_comparison(base_srv, ft_srv, test_cases)
assert results["finetuned_json_compliance"] == 100.0
assert results["base_json_compliance"] == 0.0
assert results["finetuned_exact_match"] == 100.0
assert results["accuracy_improvement"] == 100.0
tests/run_tests.sh (126 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
PYTEST="${PYTEST:-pytest}"
$PYTEST tests/test_finetune_pipeline.py -v
tests/test_finetune_pipeline.py (2316 bytes)
import pytest
import numpy as np
from finetune_pipeline import LoRAWeightMerger, MockModelServer, ModelEvaluationBenchmark
def test_lora_merging_math():
np.random.seed(42)
d_out, d_in, r = 32, 32, 4
alpha = 8.0
base_w = np.random.randn(d_out, d_in).astype(np.float32)
lora_a = np.random.randn(r, d_in).astype(np.float32)
lora_b = np.random.randn(d_out, r).astype(np.float32)
merged = LoRAWeightMerger.merge_weights(base_w, lora_a, lora_b, r=r, alpha=alpha)
expected_delta = np.matmul(lora_b, lora_a) * (8.0 / 4.0)
expected_merged = base_w + expected_delta
assert np.allclose(merged, expected_merged, atol=1e-5)
def test_lora_invalid_rank():
base_w = np.zeros((10, 10))
lora_a = np.zeros((0, 10))
lora_b = np.zeros((10, 0))
with pytest.raises(ValueError, match="LoRA rank r must be > 0"):
LoRAWeightMerger.merge_weights(base_w, lora_a, lora_b, r=0, alpha=16.0)
def test_json_compliance_evaluation():
valid_cases = ['{"name": "test"}', '{"status": 200, "items": [1, 2]}']
invalid_cases = ['Not JSON', '{"broken": json}']
score = ModelEvaluationBenchmark.evaluate_json_compliance(valid_cases + invalid_cases)
assert score == 50.0 # 2 out of 4 valid
def test_exact_match_evaluation():
preds = ["SELECT * FROM users;", " SELECT id FROM items; ", "INVALID"]
targets = ["SELECT * FROM users;", "SELECT id FROM items;", "SELECT * FROM orders;"]
em = ModelEvaluationBenchmark.evaluate_exact_match(preds, targets)
assert em == pytest.approx(66.66, 0.1) # 2 out of 3 match
def test_end_to_end_benchmark_comparison():
base_srv = MockModelServer(is_finetuned=False)
ft_srv = MockModelServer(is_finetuned=True)
test_cases = [
{"prompt": "Generate JSON schema for user 1042", "target": '{"status": "success", "user_id": 1042, "role": "admin"}'},
{"prompt": "Generate JSON schema for admin", "target": '{"status": "success", "user_id": 1042, "role": "admin"}'}
]
results = ModelEvaluationBenchmark.run_benchmark_comparison(base_srv, ft_srv, test_cases)
assert results["finetuned_json_compliance"] == 100.0
assert results["base_json_compliance"] == 0.0
assert results["finetuned_exact_match"] == 100.0
assert results["accuracy_improvement"] == 100.0
Troubleshooting
Troubleshooting
Verify scaling calculation uses float division alpha / r.
Security notes
Security
Pure NumPy matrix math and local mock serving logic. No external network dependencies.