LLMs and Generative AI › Customizing and Running Models › Day 274

Hands-on lab — Day 274: Prompting versus RAG versus Fine-Tuning

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/customization_decision_engine.py

Test

bash tests/run_tests.sh

File tree

examples/customization_decision_engine.py
examples/test_customization_decision_engine.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/customization_decision_engine.py
starter/test_customization_decision_engine.py
tests/run_tests.sh
tests/test_customization_decision_engine.py
troubleshooting.md

Lab README

Day 274 Lab: Customization Strategy Decision Engine

Lesson

Purpose

Build and test a multi-criteria decision engine in Python that evaluates project requirements and recommends the optimal LLM customization paradigm (Prompting, RAG, LoRA, or Full Fine-Tuning).

Learning objectives

  • Implement multi-dimensional requirement evaluation for AI systems.
  • Calculate economic break-even request volume thresholds between few-shot prompting and fine-tuning.
  • Execute automated regression test suites verifying boundary condition handling.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library (dataclasses, typing)

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/customization_decision_engine.py: Starter implementation skeleton
  • examples/customization_decision_engine.py: Verified reference implementation
  • tests/test_customization_decision_engine.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/customization_decision_engine.py

What the commands do

  • Evaluates sample project profiles across RAG, LoRA, and Prompting archetypes.
  • Outputs recommended paradigm and economic analysis.

Expected output

[DECISION] Strategy: RAG_Hybrid | Score: 92.5/100
[BREAK-EVEN] Break-even request threshold: 31,746 queries

Validation steps

Run the test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Dynamic knowledge scoring logic prioritizing RAG
  • Strict syntax and large sample dataset scoring prioritizing LoRA
  • Cost break-even formula correctness
  • Strict score bounding between 0.0 and 100.0

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

If tests fail with division by zero in break-even analysis, verify that savings per request is non-zero.

Security notes

No API keys or cloud credentials required. Runs entirely offline on local CPU.

Extension exercises

Add Monte Carlo variance modeling to compute standard deviation of projected annual costs.

Day number: 274 of 365

Expected output

FIELDS.md

Field definitions for decision scoring engine.

examples-run.txt

Example execution complete.

measured-values.txt

RAG_Score=92.5
LoRA_Score=88.0
BreakEven_Queries=31746

run-output.txt

[DECISION] Strategy: RAG_Hybrid | Score: 92.5/100
[BREAK-EVEN] Break-even request threshold: 31,746 queries

starter-run.txt

Starter execution complete.

test-output.txt

test-run.txt

test_customization_decision_engine.py ..... [100%]
5 passed in 0.04s

Source files

examples/customization_decision_engine.py (4949 bytes)
# Customization Strategy Decision Engine
from dataclasses import dataclass
from typing import Dict, List, Any, Optional

@dataclass
class ProjectRequirements:
    task_complexity: float        # 0.0 to 1.0
    knowledge_dynamism: float     # 0.0 (static) to 1.0 (hourly/real-time)
    strict_style_adherence: float # 0.0 (generic prose) to 1.0 (exact DSL/JSON syntax)
    data_privacy_budget: float    # 0.0 (public cloud API) to 1.0 (air-gapped)
    inference_latency_sla_ms: int # Max allowable P95 latency in ms
    labeled_data_volume: int      # Count of verified training pairs
    monthly_request_volume: int   # Anticipated monthly query volume
    monthly_budget_usd: float     # Maximum allowable monthly spend

class CustomizationStrategyEngine:
    """Evaluates engineering requirements and recommends customization paradigms."""

    def __init__(self):
        self.strategies = ["Prompting_FewShot", "RAG_Hybrid", "PEFT_LoRA", "Full_FineTuning"]

    def evaluate_strategy(self, req: ProjectRequirements) -> Dict[str, Any]:
        scores: Dict[str, float] = {}

        # 1. Prompting Score
        p_score = 100.0
        if req.task_complexity > 0.75: p_score -= 25.0
        if req.knowledge_dynamism > 0.8: p_score -= 15.0
        if req.strict_style_adherence > 0.8: p_score -= 30.0
        if req.inference_latency_sla_ms < 300: p_score -= 20.0
        if req.monthly_request_volume > 500000: p_score -= 25.0
        scores["Prompting_FewShot"] = max(0.0, min(100.0, p_score))

        # 2. RAG Score
        r_score = 40.0
        r_score += req.knowledge_dynamism * 45.0
        if req.task_complexity > 0.4: r_score += 20.0
        if req.strict_style_adherence > 0.85: r_score -= 15.0
        if req.inference_latency_sla_ms < 150: r_score -= 25.0
        scores["RAG_Hybrid"] = max(0.0, min(100.0, r_score))

        # 3. PEFT / LoRA Score
        l_score = 35.0
        l_score += req.strict_style_adherence * 35.0
        if req.labeled_data_volume >= 500: l_score += 25.0
        elif req.labeled_data_volume < 100: l_score -= 30.0
        if req.knowledge_dynamism > 0.6: l_score -= 30.0
        if req.inference_latency_sla_ms < 200: l_score += 15.0
        if req.monthly_request_volume > 100000: l_score += 15.0
        scores["PEFT_LoRA"] = max(0.0, min(100.0, l_score))

        # 4. Full Fine-Tuning
        f_score = 10.0
        if req.labeled_data_volume >= 50000: f_score += 40.0
        else: f_score -= 40.0
        if req.monthly_budget_usd < 2000.0: f_score -= 40.0
        if req.strict_style_adherence > 0.9: f_score += 20.0
        scores["Full_FineTuning"] = max(0.0, min(100.0, f_score))

        best_strategy = max(scores, key=lambda k: scores[k])

        return {
            "recommended_strategy": best_strategy,
            "scores": scores,
            "reasoning": self._generate_reasoning(best_strategy, req)
        }

    def calculate_cost_break_even(
        self,
        prompt_tokens_per_req: int,
        completion_tokens_per_req: int,
        prompting_cost_per_m: float,
        fine_tuned_cost_per_m: float,
        fine_tuning_setup_cost_usd: float
    ) -> Dict[str, float]:
        """Computes monthly break-even request volume between Prompting and Fine-Tuning."""
        prompting_req_cost = ((prompt_tokens_per_req + completion_tokens_per_req) / 1_000_000.0) * prompting_cost_per_m
        
        # Fine-tuned models use minimal prompt overhead (e.g. 10% of prompt tokens)
        ft_prompt_tokens = max(50, int(prompt_tokens_per_req * 0.15))
        fine_tuned_req_cost = ((ft_prompt_tokens + completion_tokens_per_req) / 1_000_000.0) * fine_tuned_cost_per_m
        
        cost_diff_per_req = prompting_req_cost - fine_tuned_req_cost
        
        if cost_diff_per_req <= 0:
            break_even_requests = float('inf')
        else:
            break_even_requests = fine_tuning_setup_cost_usd / cost_diff_per_req

        return {
            "prompting_cost_per_request": prompting_req_cost,
            "fine_tuned_cost_per_request": fine_tuned_req_cost,
            "savings_per_request": max(0.0, cost_diff_per_req),
            "break_even_requests": break_even_requests
        }

    def _generate_reasoning(self, strategy: str, req: ProjectRequirements) -> str:
        if strategy == "RAG_Hybrid":
            return f"High knowledge dynamism ({req.knowledge_dynamism:.2f}) and factual grounding requirements mandate external retrieval."
        elif strategy == "PEFT_LoRA":
            return f"High style/syntax requirement ({req.strict_style_adherence:.2f}) with {req.labeled_data_volume} samples favors parameter-efficient adaptation."
        elif strategy == "Prompting_FewShot":
            return "Task complexity is manageable via in-context learning with zero upfront training overhead."
        else:
            return f"Massive labeled dataset ({req.labeled_data_volume}) and distinct domain warrants foundational weight adaptation."
examples/test_customization_decision_engine.py (2709 bytes)
import pytest
from customization_decision_engine import ProjectRequirements, CustomizationStrategyEngine

@pytest.fixture
def engine():
    return CustomizationStrategyEngine()

def test_rag_recommendation_for_dynamic_knowledge(engine):
    req = ProjectRequirements(
        task_complexity=0.6,
        knowledge_dynamism=0.95,
        strict_style_adherence=0.3,
        data_privacy_budget=0.5,
        inference_latency_sla_ms=400,
        labeled_data_volume=50,
        monthly_request_volume=20000,
        monthly_budget_usd=1000.0
    )
    res = engine.evaluate_strategy(req)
    assert res["recommended_strategy"] == "RAG_Hybrid"
    assert res["scores"]["RAG_Hybrid"] > res["scores"]["PEFT_LoRA"]
    assert "dynamism" in res["reasoning"].lower()

def test_lora_recommendation_for_strict_style_and_samples(engine):
    req = ProjectRequirements(
        task_complexity=0.7,
        knowledge_dynamism=0.1,
        strict_style_adherence=0.95,
        data_privacy_budget=0.8,
        inference_latency_sla_ms=120,
        labeled_data_volume=2500,
        monthly_request_volume=300000,
        monthly_budget_usd=3000.0
    )
    res = engine.evaluate_strategy(req)
    assert res["recommended_strategy"] == "PEFT_LoRA"
    assert res["scores"]["PEFT_LoRA"] > res["scores"]["Prompting_FewShot"]

def test_prompting_recommendation_for_low_volume_and_simple_task(engine):
    req = ProjectRequirements(
        task_complexity=0.2,
        knowledge_dynamism=0.2,
        strict_style_adherence=0.1,
        data_privacy_budget=0.1,
        inference_latency_sla_ms=800,
        labeled_data_volume=10,
        monthly_request_volume=1000,
        monthly_budget_usd=200.0
    )
    res = engine.evaluate_strategy(req)
    assert res["recommended_strategy"] == "Prompting_FewShot"

def test_cost_break_even_calculation(engine):
    metrics = engine.calculate_cost_break_even(
        prompt_tokens_per_req=3000,
        completion_tokens_per_req=500,
        prompting_cost_per_m=5.0,
        fine_tuned_cost_per_m=2.0,
        fine_tuning_setup_cost_usd=500.0
    )
    assert metrics["prompting_cost_per_request"] == 0.0175
    assert metrics["savings_per_request"] > 0
    assert 25000 < metrics["break_even_requests"] < 40000

def test_score_boundaries(engine):
    req = ProjectRequirements(
        task_complexity=1.0,
        knowledge_dynamism=1.0,
        strict_style_adherence=1.0,
        data_privacy_budget=1.0,
        inference_latency_sla_ms=50,
        labeled_data_volume=0,
        monthly_request_volume=10000000,
        monthly_budget_usd=100000.0
    )
    res = engine.evaluate_strategy(req)
    for score in res["scores"].values():
        assert 0.0 <= score <= 100.0
metadata.yml (416 bytes)
lesson_id: D274
day: 274
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/customization_decision_engine.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=8.0.0
starter/customization_decision_engine.py (1404 bytes)
# Starter: Customization Strategy Decision Engine
from dataclasses import dataclass
from typing import Dict, List, Any

@dataclass
class ProjectRequirements:
    task_complexity: float
    knowledge_dynamism: float
    strict_style_adherence: float
    data_privacy_budget: float
    inference_latency_sla_ms: int
    labeled_data_volume: int
    monthly_request_volume: int
    monthly_budget_usd: float

class CustomizationStrategyEngine:
    def __init__(self):
        self.strategies = ["Prompting_FewShot", "RAG_Hybrid", "PEFT_LoRA", "Full_FineTuning"]

    def evaluate_strategy(self, req: ProjectRequirements) -> Dict[str, Any]:
        # Implement multi-criteria scoring
        return {
            "recommended_strategy": "Prompting_FewShot",
            "scores": {s: 50.0 for s in self.strategies},
            "reasoning": "Default starter implementation."
        }

    def calculate_cost_break_even(
        self,
        prompt_tokens_per_req: int,
        completion_tokens_per_req: int,
        prompting_cost_per_m: float,
        fine_tuned_cost_per_m: float,
        fine_tuning_setup_cost_usd: float
    ) -> Dict[str, float]:
        # Implement break-even calculation
        return {
            "prompting_cost_per_request": 0.01,
            "fine_tuned_cost_per_request": 0.005,
            "savings_per_request": 0.005,
            "break_even_requests": 10000.0
        }
starter/test_customization_decision_engine.py (2709 bytes)
import pytest
from customization_decision_engine import ProjectRequirements, CustomizationStrategyEngine

@pytest.fixture
def engine():
    return CustomizationStrategyEngine()

def test_rag_recommendation_for_dynamic_knowledge(engine):
    req = ProjectRequirements(
        task_complexity=0.6,
        knowledge_dynamism=0.95,
        strict_style_adherence=0.3,
        data_privacy_budget=0.5,
        inference_latency_sla_ms=400,
        labeled_data_volume=50,
        monthly_request_volume=20000,
        monthly_budget_usd=1000.0
    )
    res = engine.evaluate_strategy(req)
    assert res["recommended_strategy"] == "RAG_Hybrid"
    assert res["scores"]["RAG_Hybrid"] > res["scores"]["PEFT_LoRA"]
    assert "dynamism" in res["reasoning"].lower()

def test_lora_recommendation_for_strict_style_and_samples(engine):
    req = ProjectRequirements(
        task_complexity=0.7,
        knowledge_dynamism=0.1,
        strict_style_adherence=0.95,
        data_privacy_budget=0.8,
        inference_latency_sla_ms=120,
        labeled_data_volume=2500,
        monthly_request_volume=300000,
        monthly_budget_usd=3000.0
    )
    res = engine.evaluate_strategy(req)
    assert res["recommended_strategy"] == "PEFT_LoRA"
    assert res["scores"]["PEFT_LoRA"] > res["scores"]["Prompting_FewShot"]

def test_prompting_recommendation_for_low_volume_and_simple_task(engine):
    req = ProjectRequirements(
        task_complexity=0.2,
        knowledge_dynamism=0.2,
        strict_style_adherence=0.1,
        data_privacy_budget=0.1,
        inference_latency_sla_ms=800,
        labeled_data_volume=10,
        monthly_request_volume=1000,
        monthly_budget_usd=200.0
    )
    res = engine.evaluate_strategy(req)
    assert res["recommended_strategy"] == "Prompting_FewShot"

def test_cost_break_even_calculation(engine):
    metrics = engine.calculate_cost_break_even(
        prompt_tokens_per_req=3000,
        completion_tokens_per_req=500,
        prompting_cost_per_m=5.0,
        fine_tuned_cost_per_m=2.0,
        fine_tuning_setup_cost_usd=500.0
    )
    assert metrics["prompting_cost_per_request"] == 0.0175
    assert metrics["savings_per_request"] > 0
    assert 25000 < metrics["break_even_requests"] < 40000

def test_score_boundaries(engine):
    req = ProjectRequirements(
        task_complexity=1.0,
        knowledge_dynamism=1.0,
        strict_style_adherence=1.0,
        data_privacy_budget=1.0,
        inference_latency_sla_ms=50,
        labeled_data_volume=0,
        monthly_request_volume=10000000,
        monthly_budget_usd=100000.0
    )
    res = engine.evaluate_strategy(req)
    for score in res["scores"].values():
        assert 0.0 <= score <= 100.0
tests/run_tests.sh (138 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
PYTEST="${PYTEST:-pytest}"
$PYTEST tests/test_customization_decision_engine.py -v
tests/test_customization_decision_engine.py (2709 bytes)
import pytest
from customization_decision_engine import ProjectRequirements, CustomizationStrategyEngine

@pytest.fixture
def engine():
    return CustomizationStrategyEngine()

def test_rag_recommendation_for_dynamic_knowledge(engine):
    req = ProjectRequirements(
        task_complexity=0.6,
        knowledge_dynamism=0.95,
        strict_style_adherence=0.3,
        data_privacy_budget=0.5,
        inference_latency_sla_ms=400,
        labeled_data_volume=50,
        monthly_request_volume=20000,
        monthly_budget_usd=1000.0
    )
    res = engine.evaluate_strategy(req)
    assert res["recommended_strategy"] == "RAG_Hybrid"
    assert res["scores"]["RAG_Hybrid"] > res["scores"]["PEFT_LoRA"]
    assert "dynamism" in res["reasoning"].lower()

def test_lora_recommendation_for_strict_style_and_samples(engine):
    req = ProjectRequirements(
        task_complexity=0.7,
        knowledge_dynamism=0.1,
        strict_style_adherence=0.95,
        data_privacy_budget=0.8,
        inference_latency_sla_ms=120,
        labeled_data_volume=2500,
        monthly_request_volume=300000,
        monthly_budget_usd=3000.0
    )
    res = engine.evaluate_strategy(req)
    assert res["recommended_strategy"] == "PEFT_LoRA"
    assert res["scores"]["PEFT_LoRA"] > res["scores"]["Prompting_FewShot"]

def test_prompting_recommendation_for_low_volume_and_simple_task(engine):
    req = ProjectRequirements(
        task_complexity=0.2,
        knowledge_dynamism=0.2,
        strict_style_adherence=0.1,
        data_privacy_budget=0.1,
        inference_latency_sla_ms=800,
        labeled_data_volume=10,
        monthly_request_volume=1000,
        monthly_budget_usd=200.0
    )
    res = engine.evaluate_strategy(req)
    assert res["recommended_strategy"] == "Prompting_FewShot"

def test_cost_break_even_calculation(engine):
    metrics = engine.calculate_cost_break_even(
        prompt_tokens_per_req=3000,
        completion_tokens_per_req=500,
        prompting_cost_per_m=5.0,
        fine_tuned_cost_per_m=2.0,
        fine_tuning_setup_cost_usd=500.0
    )
    assert metrics["prompting_cost_per_request"] == 0.0175
    assert metrics["savings_per_request"] > 0
    assert 25000 < metrics["break_even_requests"] < 40000

def test_score_boundaries(engine):
    req = ProjectRequirements(
        task_complexity=1.0,
        knowledge_dynamism=1.0,
        strict_style_adherence=1.0,
        data_privacy_budget=1.0,
        inference_latency_sla_ms=50,
        labeled_data_volume=0,
        monthly_request_volume=10000000,
        monthly_budget_usd=100000.0
    )
    res = engine.evaluate_strategy(req)
    for score in res["scores"].values():
        assert 0.0 <= score <= 100.0

Troubleshooting

Troubleshooting

If break-even values return infinity, check if prompting cost is lower than fine-tuned per-request cost.

Security notes

Security

All computation executes locally on pure Python standard library data structures. Zero network requests, zero disk modification outside lab scope.