LLMs and Generative AI › Customizing and Running Models › Day 274
Hands-on lab — Day 274: Prompting versus RAG versus Fine-Tuning
- ← Back to the Day 274 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-274-prompting-versus-rag-versus-fine-tuning/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/customization_decision_engine.py Test
bash tests/run_tests.sh File tree
examples/customization_decision_engine.py examples/test_customization_decision_engine.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/customization_decision_engine.py starter/test_customization_decision_engine.py tests/run_tests.sh tests/test_customization_decision_engine.py troubleshooting.md
Lab README
Day 274 Lab: Customization Strategy Decision Engine
Lesson
- Lesson title: Prompting versus RAG versus Fine-Tuning
- Day number: 274 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-274-prompting-versus-rag-versus-fine-tuning
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-274-prompting-versus-rag-versus-fine-tuningwhen the site is running.
Purpose
Build and test a multi-criteria decision engine in Python that evaluates project requirements and recommends the optimal LLM customization paradigm (Prompting, RAG, LoRA, or Full Fine-Tuning).
Learning objectives
- Implement multi-dimensional requirement evaluation for AI systems.
- Calculate economic break-even request volume thresholds between few-shot prompting and fine-tuning.
- Execute automated regression test suites verifying boundary condition handling.
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library (dataclasses, typing)
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/customization_decision_engine.py: Starter implementation skeletonexamples/customization_decision_engine.py: Verified reference implementationtests/test_customization_decision_engine.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/customization_decision_engine.py
What the commands do
- Evaluates sample project profiles across RAG, LoRA, and Prompting archetypes.
- Outputs recommended paradigm and economic analysis.
Expected output
[DECISION] Strategy: RAG_Hybrid | Score: 92.5/100
[BREAK-EVEN] Break-even request threshold: 31,746 queries
Validation steps
Run the test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Dynamic knowledge scoring logic prioritizing RAG
- Strict syntax and large sample dataset scoring prioritizing LoRA
- Cost break-even formula correctness
- Strict score bounding between 0.0 and 100.0
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
If tests fail with division by zero in break-even analysis, verify that savings per request is non-zero.
Security notes
No API keys or cloud credentials required. Runs entirely offline on local CPU.
Extension exercises
Add Monte Carlo variance modeling to compute standard deviation of projected annual costs.
Navigation
Day number: 274 of 365
Expected output
FIELDS.md
Field definitions for decision scoring engine. examples-run.txt
Example execution complete.
measured-values.txt
RAG_Score=92.5
LoRA_Score=88.0
BreakEven_Queries=31746
run-output.txt
[DECISION] Strategy: RAG_Hybrid | Score: 92.5/100
[BREAK-EVEN] Break-even request threshold: 31,746 queries
starter-run.txt
Starter execution complete.
test-output.txt
test-run.txt
test_customization_decision_engine.py ..... [100%]
5 passed in 0.04s
Source files
examples/customization_decision_engine.py (4949 bytes)
# Customization Strategy Decision Engine
from dataclasses import dataclass
from typing import Dict, List, Any, Optional
@dataclass
class ProjectRequirements:
task_complexity: float # 0.0 to 1.0
knowledge_dynamism: float # 0.0 (static) to 1.0 (hourly/real-time)
strict_style_adherence: float # 0.0 (generic prose) to 1.0 (exact DSL/JSON syntax)
data_privacy_budget: float # 0.0 (public cloud API) to 1.0 (air-gapped)
inference_latency_sla_ms: int # Max allowable P95 latency in ms
labeled_data_volume: int # Count of verified training pairs
monthly_request_volume: int # Anticipated monthly query volume
monthly_budget_usd: float # Maximum allowable monthly spend
class CustomizationStrategyEngine:
"""Evaluates engineering requirements and recommends customization paradigms."""
def __init__(self):
self.strategies = ["Prompting_FewShot", "RAG_Hybrid", "PEFT_LoRA", "Full_FineTuning"]
def evaluate_strategy(self, req: ProjectRequirements) -> Dict[str, Any]:
scores: Dict[str, float] = {}
# 1. Prompting Score
p_score = 100.0
if req.task_complexity > 0.75: p_score -= 25.0
if req.knowledge_dynamism > 0.8: p_score -= 15.0
if req.strict_style_adherence > 0.8: p_score -= 30.0
if req.inference_latency_sla_ms < 300: p_score -= 20.0
if req.monthly_request_volume > 500000: p_score -= 25.0
scores["Prompting_FewShot"] = max(0.0, min(100.0, p_score))
# 2. RAG Score
r_score = 40.0
r_score += req.knowledge_dynamism * 45.0
if req.task_complexity > 0.4: r_score += 20.0
if req.strict_style_adherence > 0.85: r_score -= 15.0
if req.inference_latency_sla_ms < 150: r_score -= 25.0
scores["RAG_Hybrid"] = max(0.0, min(100.0, r_score))
# 3. PEFT / LoRA Score
l_score = 35.0
l_score += req.strict_style_adherence * 35.0
if req.labeled_data_volume >= 500: l_score += 25.0
elif req.labeled_data_volume < 100: l_score -= 30.0
if req.knowledge_dynamism > 0.6: l_score -= 30.0
if req.inference_latency_sla_ms < 200: l_score += 15.0
if req.monthly_request_volume > 100000: l_score += 15.0
scores["PEFT_LoRA"] = max(0.0, min(100.0, l_score))
# 4. Full Fine-Tuning
f_score = 10.0
if req.labeled_data_volume >= 50000: f_score += 40.0
else: f_score -= 40.0
if req.monthly_budget_usd < 2000.0: f_score -= 40.0
if req.strict_style_adherence > 0.9: f_score += 20.0
scores["Full_FineTuning"] = max(0.0, min(100.0, f_score))
best_strategy = max(scores, key=lambda k: scores[k])
return {
"recommended_strategy": best_strategy,
"scores": scores,
"reasoning": self._generate_reasoning(best_strategy, req)
}
def calculate_cost_break_even(
self,
prompt_tokens_per_req: int,
completion_tokens_per_req: int,
prompting_cost_per_m: float,
fine_tuned_cost_per_m: float,
fine_tuning_setup_cost_usd: float
) -> Dict[str, float]:
"""Computes monthly break-even request volume between Prompting and Fine-Tuning."""
prompting_req_cost = ((prompt_tokens_per_req + completion_tokens_per_req) / 1_000_000.0) * prompting_cost_per_m
# Fine-tuned models use minimal prompt overhead (e.g. 10% of prompt tokens)
ft_prompt_tokens = max(50, int(prompt_tokens_per_req * 0.15))
fine_tuned_req_cost = ((ft_prompt_tokens + completion_tokens_per_req) / 1_000_000.0) * fine_tuned_cost_per_m
cost_diff_per_req = prompting_req_cost - fine_tuned_req_cost
if cost_diff_per_req <= 0:
break_even_requests = float('inf')
else:
break_even_requests = fine_tuning_setup_cost_usd / cost_diff_per_req
return {
"prompting_cost_per_request": prompting_req_cost,
"fine_tuned_cost_per_request": fine_tuned_req_cost,
"savings_per_request": max(0.0, cost_diff_per_req),
"break_even_requests": break_even_requests
}
def _generate_reasoning(self, strategy: str, req: ProjectRequirements) -> str:
if strategy == "RAG_Hybrid":
return f"High knowledge dynamism ({req.knowledge_dynamism:.2f}) and factual grounding requirements mandate external retrieval."
elif strategy == "PEFT_LoRA":
return f"High style/syntax requirement ({req.strict_style_adherence:.2f}) with {req.labeled_data_volume} samples favors parameter-efficient adaptation."
elif strategy == "Prompting_FewShot":
return "Task complexity is manageable via in-context learning with zero upfront training overhead."
else:
return f"Massive labeled dataset ({req.labeled_data_volume}) and distinct domain warrants foundational weight adaptation."
examples/test_customization_decision_engine.py (2709 bytes)
import pytest
from customization_decision_engine import ProjectRequirements, CustomizationStrategyEngine
@pytest.fixture
def engine():
return CustomizationStrategyEngine()
def test_rag_recommendation_for_dynamic_knowledge(engine):
req = ProjectRequirements(
task_complexity=0.6,
knowledge_dynamism=0.95,
strict_style_adherence=0.3,
data_privacy_budget=0.5,
inference_latency_sla_ms=400,
labeled_data_volume=50,
monthly_request_volume=20000,
monthly_budget_usd=1000.0
)
res = engine.evaluate_strategy(req)
assert res["recommended_strategy"] == "RAG_Hybrid"
assert res["scores"]["RAG_Hybrid"] > res["scores"]["PEFT_LoRA"]
assert "dynamism" in res["reasoning"].lower()
def test_lora_recommendation_for_strict_style_and_samples(engine):
req = ProjectRequirements(
task_complexity=0.7,
knowledge_dynamism=0.1,
strict_style_adherence=0.95,
data_privacy_budget=0.8,
inference_latency_sla_ms=120,
labeled_data_volume=2500,
monthly_request_volume=300000,
monthly_budget_usd=3000.0
)
res = engine.evaluate_strategy(req)
assert res["recommended_strategy"] == "PEFT_LoRA"
assert res["scores"]["PEFT_LoRA"] > res["scores"]["Prompting_FewShot"]
def test_prompting_recommendation_for_low_volume_and_simple_task(engine):
req = ProjectRequirements(
task_complexity=0.2,
knowledge_dynamism=0.2,
strict_style_adherence=0.1,
data_privacy_budget=0.1,
inference_latency_sla_ms=800,
labeled_data_volume=10,
monthly_request_volume=1000,
monthly_budget_usd=200.0
)
res = engine.evaluate_strategy(req)
assert res["recommended_strategy"] == "Prompting_FewShot"
def test_cost_break_even_calculation(engine):
metrics = engine.calculate_cost_break_even(
prompt_tokens_per_req=3000,
completion_tokens_per_req=500,
prompting_cost_per_m=5.0,
fine_tuned_cost_per_m=2.0,
fine_tuning_setup_cost_usd=500.0
)
assert metrics["prompting_cost_per_request"] == 0.0175
assert metrics["savings_per_request"] > 0
assert 25000 < metrics["break_even_requests"] < 40000
def test_score_boundaries(engine):
req = ProjectRequirements(
task_complexity=1.0,
knowledge_dynamism=1.0,
strict_style_adherence=1.0,
data_privacy_budget=1.0,
inference_latency_sla_ms=50,
labeled_data_volume=0,
monthly_request_volume=10000000,
monthly_budget_usd=100000.0
)
res = engine.evaluate_strategy(req)
for score in res["scores"].values():
assert 0.0 <= score <= 100.0
metadata.yml (416 bytes)
lesson_id: D274
day: 274
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/customization_decision_engine.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=8.0.0
starter/customization_decision_engine.py (1404 bytes)
# Starter: Customization Strategy Decision Engine
from dataclasses import dataclass
from typing import Dict, List, Any
@dataclass
class ProjectRequirements:
task_complexity: float
knowledge_dynamism: float
strict_style_adherence: float
data_privacy_budget: float
inference_latency_sla_ms: int
labeled_data_volume: int
monthly_request_volume: int
monthly_budget_usd: float
class CustomizationStrategyEngine:
def __init__(self):
self.strategies = ["Prompting_FewShot", "RAG_Hybrid", "PEFT_LoRA", "Full_FineTuning"]
def evaluate_strategy(self, req: ProjectRequirements) -> Dict[str, Any]:
# Implement multi-criteria scoring
return {
"recommended_strategy": "Prompting_FewShot",
"scores": {s: 50.0 for s in self.strategies},
"reasoning": "Default starter implementation."
}
def calculate_cost_break_even(
self,
prompt_tokens_per_req: int,
completion_tokens_per_req: int,
prompting_cost_per_m: float,
fine_tuned_cost_per_m: float,
fine_tuning_setup_cost_usd: float
) -> Dict[str, float]:
# Implement break-even calculation
return {
"prompting_cost_per_request": 0.01,
"fine_tuned_cost_per_request": 0.005,
"savings_per_request": 0.005,
"break_even_requests": 10000.0
}
starter/test_customization_decision_engine.py (2709 bytes)
import pytest
from customization_decision_engine import ProjectRequirements, CustomizationStrategyEngine
@pytest.fixture
def engine():
return CustomizationStrategyEngine()
def test_rag_recommendation_for_dynamic_knowledge(engine):
req = ProjectRequirements(
task_complexity=0.6,
knowledge_dynamism=0.95,
strict_style_adherence=0.3,
data_privacy_budget=0.5,
inference_latency_sla_ms=400,
labeled_data_volume=50,
monthly_request_volume=20000,
monthly_budget_usd=1000.0
)
res = engine.evaluate_strategy(req)
assert res["recommended_strategy"] == "RAG_Hybrid"
assert res["scores"]["RAG_Hybrid"] > res["scores"]["PEFT_LoRA"]
assert "dynamism" in res["reasoning"].lower()
def test_lora_recommendation_for_strict_style_and_samples(engine):
req = ProjectRequirements(
task_complexity=0.7,
knowledge_dynamism=0.1,
strict_style_adherence=0.95,
data_privacy_budget=0.8,
inference_latency_sla_ms=120,
labeled_data_volume=2500,
monthly_request_volume=300000,
monthly_budget_usd=3000.0
)
res = engine.evaluate_strategy(req)
assert res["recommended_strategy"] == "PEFT_LoRA"
assert res["scores"]["PEFT_LoRA"] > res["scores"]["Prompting_FewShot"]
def test_prompting_recommendation_for_low_volume_and_simple_task(engine):
req = ProjectRequirements(
task_complexity=0.2,
knowledge_dynamism=0.2,
strict_style_adherence=0.1,
data_privacy_budget=0.1,
inference_latency_sla_ms=800,
labeled_data_volume=10,
monthly_request_volume=1000,
monthly_budget_usd=200.0
)
res = engine.evaluate_strategy(req)
assert res["recommended_strategy"] == "Prompting_FewShot"
def test_cost_break_even_calculation(engine):
metrics = engine.calculate_cost_break_even(
prompt_tokens_per_req=3000,
completion_tokens_per_req=500,
prompting_cost_per_m=5.0,
fine_tuned_cost_per_m=2.0,
fine_tuning_setup_cost_usd=500.0
)
assert metrics["prompting_cost_per_request"] == 0.0175
assert metrics["savings_per_request"] > 0
assert 25000 < metrics["break_even_requests"] < 40000
def test_score_boundaries(engine):
req = ProjectRequirements(
task_complexity=1.0,
knowledge_dynamism=1.0,
strict_style_adherence=1.0,
data_privacy_budget=1.0,
inference_latency_sla_ms=50,
labeled_data_volume=0,
monthly_request_volume=10000000,
monthly_budget_usd=100000.0
)
res = engine.evaluate_strategy(req)
for score in res["scores"].values():
assert 0.0 <= score <= 100.0
tests/run_tests.sh (138 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
PYTEST="${PYTEST:-pytest}"
$PYTEST tests/test_customization_decision_engine.py -v
tests/test_customization_decision_engine.py (2709 bytes)
import pytest
from customization_decision_engine import ProjectRequirements, CustomizationStrategyEngine
@pytest.fixture
def engine():
return CustomizationStrategyEngine()
def test_rag_recommendation_for_dynamic_knowledge(engine):
req = ProjectRequirements(
task_complexity=0.6,
knowledge_dynamism=0.95,
strict_style_adherence=0.3,
data_privacy_budget=0.5,
inference_latency_sla_ms=400,
labeled_data_volume=50,
monthly_request_volume=20000,
monthly_budget_usd=1000.0
)
res = engine.evaluate_strategy(req)
assert res["recommended_strategy"] == "RAG_Hybrid"
assert res["scores"]["RAG_Hybrid"] > res["scores"]["PEFT_LoRA"]
assert "dynamism" in res["reasoning"].lower()
def test_lora_recommendation_for_strict_style_and_samples(engine):
req = ProjectRequirements(
task_complexity=0.7,
knowledge_dynamism=0.1,
strict_style_adherence=0.95,
data_privacy_budget=0.8,
inference_latency_sla_ms=120,
labeled_data_volume=2500,
monthly_request_volume=300000,
monthly_budget_usd=3000.0
)
res = engine.evaluate_strategy(req)
assert res["recommended_strategy"] == "PEFT_LoRA"
assert res["scores"]["PEFT_LoRA"] > res["scores"]["Prompting_FewShot"]
def test_prompting_recommendation_for_low_volume_and_simple_task(engine):
req = ProjectRequirements(
task_complexity=0.2,
knowledge_dynamism=0.2,
strict_style_adherence=0.1,
data_privacy_budget=0.1,
inference_latency_sla_ms=800,
labeled_data_volume=10,
monthly_request_volume=1000,
monthly_budget_usd=200.0
)
res = engine.evaluate_strategy(req)
assert res["recommended_strategy"] == "Prompting_FewShot"
def test_cost_break_even_calculation(engine):
metrics = engine.calculate_cost_break_even(
prompt_tokens_per_req=3000,
completion_tokens_per_req=500,
prompting_cost_per_m=5.0,
fine_tuned_cost_per_m=2.0,
fine_tuning_setup_cost_usd=500.0
)
assert metrics["prompting_cost_per_request"] == 0.0175
assert metrics["savings_per_request"] > 0
assert 25000 < metrics["break_even_requests"] < 40000
def test_score_boundaries(engine):
req = ProjectRequirements(
task_complexity=1.0,
knowledge_dynamism=1.0,
strict_style_adherence=1.0,
data_privacy_budget=1.0,
inference_latency_sla_ms=50,
labeled_data_volume=0,
monthly_request_volume=10000000,
monthly_budget_usd=100000.0
)
res = engine.evaluate_strategy(req)
for score in res["scores"].values():
assert 0.0 <= score <= 100.0
Troubleshooting
Troubleshooting
If break-even values return infinity, check if prompting cost is lower than fine-tuned per-request cost.
Security notes
Security
All computation executes locally on pure Python standard library data structures. Zero network requests, zero disk modification outside lab scope.