LLMs and Generative AI › The LLM Landscape › Day 241
Hands-on lab — Day 241: The Model Landscape: Claude, GPT, Gemini, Llama
- ← Back to the Day 241 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-241-the-model-landscape-claude-gpt-gemini/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/the_model_landscape_claude_gpt_gemini_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/test_the_model_landscape_claude_gpt_gemini_lib.py examples/the_model_landscape_claude_gpt_gemini_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/test_the_model_landscape_claude_gpt_gemini_lib.py starter/the_model_landscape_claude_gpt_gemini_lib.py tests/run_tests.sh tests/test_the_model_landscape_claude_gpt_gemini_lib.py troubleshooting.md
Lab README
Lab: Day 241 -- The Model Landscape: Claude, GPT, Gemini, Llama
Lesson
Day number: 241 of 365. Course: Course06-SS01 (LLMs and Generative AI - Working with LLMs). Topic: Frontier LLM Landscape, Model Routing, and Cost Optimization.
Purpose
Build and test a modular model routing and cost calculation engine in Python. Implement heuristic and keyword query classification, dispatch prompts to appropriate cost/latency tiers, calculate multi-tier API billing costs, and simulate blended enterprise cost savings.
Learning objectives
- Analyze the operational trade-offs between Claude 3.5 Sonnet, GPT-4o, Gemini 2.0 Flash, and open-weights models.
- Implement rule-based and keyword-driven semantic query dispatching.
- Calculate exact multi-tier API costs incorporating input/output token pricing and prompt caching discounts.
- Simulate blended fleet cost savings comparing monolithic routing vs multi-tier routing.
Prerequisites
- Day 240 (Pretraining, Fine-Tuning, and RLHF).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
Python and standard math libraries are open source.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/the_model_landscape_claude_gpt_gemini_lib.py: Student scaffold file.examples/the_model_landscape_claude_gpt_gemini_lib.py: Complete reference implementation.tests/test_the_model_landscape_claude_gpt_gemini_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/the_model_landscape_claude_gpt_gemini_lib.py
What the commands do
- Evaluates query routing decisions across code, classification, and long-context prompts.
- Computes API query costs with prompt caching discounts.
- Runs unit test assertions.
Expected output
Router Demo: Code -> claude-3.5-sonnet, Quick -> gpt-4o-mini, Long -> gemini-2.0-flash, Sonnet Cost = $0.1650, Cached = $0.0525
Validation steps
- Verify code queries route to
claude-3.5-sonnet. - Confirm short classification queries route to
gpt-4o-mini. - Confirm long context queries ($> 150k$ tokens) route to
gemini-2.0-flash. - Confirm
is_cached=Trueapplies a $75%$ input token discount. - Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Cost scaling error: Ensure token counts are divided by $1,000,000$ when multiplying by rate card prices.
Security notes
All routing calculations execute locally in process memory.
Extension exercises
- Implement a latency-aware router incorporating provider status checks.
- Build an automated fallback cascade handling simulated 429 rate limit exceptions.
Navigation
- Lesson title: The Model Landscape: Claude, GPT, Gemini, Llama
- Day number: 241 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-241-the-model-landscape-claude-gpt-gemini
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-241-the-model-landscape-claude-gpt-geminiwhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 241
- `Code Route`: Selected model tier for complex code reasoning.
- `Quick Route`: Selected model tier for lightweight fast tasks.
- `Long Route`: Selected model tier for massive context queries.
- `Sonnet Cost`: Total standard query cost in USD.
- `Cached Cost`: Total discounted query cost with prompt caching.
examples-run.txt
Router Demo: Code -> claude-3.5-sonnet, Quick -> gpt-4o-mini, Long -> gemini-2.0-flash, Sonnet Cost = $0.1650, Cached = $0.0525
measured-values.txt
Code Route: claude-3.5-sonnet
Quick Route: gpt-4o-mini
Long Route: gemini-2.0-flash
Sonnet Cost: $0.1650
Cached Cost: $0.0525
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 3 items
tests/test_the_model_landscape_claude_gpt_gemini_lib.py::test_router_dispatch_decisions PASSED [ 33%]
tests/test_the_model_landscape_claude_gpt_gemini_lib.py::test_cost_calculation PASSED [ 66%]
tests/test_the_model_landscape_claude_gpt_gemini_lib.py::test_prompt_caching_discount PASSED [100%]
============================== 3 passed in 0.41s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/test_the_model_landscape_claude_gpt_gemini_lib.py (1280 bytes)
import pytest
from examples.the_model_landscape_claude_gpt_gemini_lib import ModelRouter
def test_router_dispatch_decisions():
router = ModelRouter()
# Code prompt should go to Claude Sonnet
code_prompt = "def merge_sort(arr):\n if len(arr) <= 1: return arr"
assert router.route_query(code_prompt) == "claude-3.5-sonnet"
# Fast classification prompt should go to GPT-4o-mini
simple_prompt = "Classify sentiment: Great movie!"
assert router.route_query(simple_prompt, token_count=20) == "gpt-4o-mini"
# Long document should go to Gemini Flash
assert router.route_query("Document text", token_count=200000) == "gemini-2.0-flash"
def test_cost_calculation():
router = ModelRouter()
# 1M input + 1M output on Sonnet = $3 + $15 = $18
cost = router.calculate_cost("claude-3.5-sonnet", 1000000, 1000000)
assert cost == pytest.approx(18.00, rel=1e-3)
def test_prompt_caching_discount():
router = ModelRouter()
cost_regular = router.calculate_cost("claude-3.5-sonnet", 1000000, 0, is_cached=False)
cost_cached = router.calculate_cost("claude-3.5-sonnet", 1000000, 0, is_cached=True)
assert cost_regular == pytest.approx(3.00, rel=1e-3)
assert cost_cached == pytest.approx(0.75, rel=1e-3) # 75% discount
examples/the_model_landscape_claude_gpt_gemini_lib.py (2017 bytes)
from typing import Dict, Any, Tuple
class ModelRouter:
def __init__(self):
# Pricing per 1M tokens: (input_price, output_price)
self.pricing = {
"claude-3.5-sonnet": (3.00, 15.00),
"gpt-4o": (2.50, 10.00),
"gemini-2.0-flash": (0.10, 0.40),
"gpt-4o-mini": (0.15, 0.60)
}
def route_query(self, prompt: str, token_count: int = 100) -> str:
prompt_lower = prompt.lower()
if token_count > 150000:
return "gemini-2.0-flash"
if "def " in prompt or "class " in prompt or "theorem" in prompt_lower or "refactor" in prompt_lower:
return "claude-3.5-sonnet"
if token_count < 300 and ("classify" in prompt_lower or "extract" in prompt_lower or "sentiment" in prompt_lower):
return "gpt-4o-mini"
return "gemini-2.0-flash"
def calculate_cost(self, model: str, input_tokens: int, output_tokens: int, is_cached: bool = False) -> float:
in_rate, out_rate = self.pricing.get(model, (1.0, 3.0))
if is_cached:
in_rate = in_rate * 0.25 # 75% prompt cache discount
cost = (input_tokens / 1e6) * in_rate + (output_tokens / 1e6) * out_rate
return round(cost, 6)
def run_routing_demo():
router = ModelRouter()
m1 = router.route_query("def fibonacci(n): return n if n < 2 else fibonacci(n-1) + fibonacci(n-2)")
m2 = router.route_query("Classify the sentiment of this review", token_count=50)
m3 = router.route_query("Analyze complete archive", token_count=500000)
cost_sonnet = router.calculate_cost("claude-3.5-sonnet", input_tokens=50000, output_tokens=1000)
cost_cached = router.calculate_cost("claude-3.5-sonnet", input_tokens=50000, output_tokens=1000, is_cached=True)
print(f"Router Demo: Code -> {m1}, Quick -> {m2}, Long -> {m3}, Sonnet Cost = ${cost_sonnet:.4f}, Cached = ${cost_cached:.4f}")
return m1, m2, m3, cost_sonnet, cost_cached
if __name__ == "__main__":
run_routing_demo()
metadata.yml (450 bytes)
lesson_id: D241
day: 241
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/the_model_landscape_claude_gpt_gemini_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/test_the_model_landscape_claude_gpt_gemini_lib.py (1280 bytes)
import pytest
from examples.the_model_landscape_claude_gpt_gemini_lib import ModelRouter
def test_router_dispatch_decisions():
router = ModelRouter()
# Code prompt should go to Claude Sonnet
code_prompt = "def merge_sort(arr):\n if len(arr) <= 1: return arr"
assert router.route_query(code_prompt) == "claude-3.5-sonnet"
# Fast classification prompt should go to GPT-4o-mini
simple_prompt = "Classify sentiment: Great movie!"
assert router.route_query(simple_prompt, token_count=20) == "gpt-4o-mini"
# Long document should go to Gemini Flash
assert router.route_query("Document text", token_count=200000) == "gemini-2.0-flash"
def test_cost_calculation():
router = ModelRouter()
# 1M input + 1M output on Sonnet = $3 + $15 = $18
cost = router.calculate_cost("claude-3.5-sonnet", 1000000, 1000000)
assert cost == pytest.approx(18.00, rel=1e-3)
def test_prompt_caching_discount():
router = ModelRouter()
cost_regular = router.calculate_cost("claude-3.5-sonnet", 1000000, 0, is_cached=False)
cost_cached = router.calculate_cost("claude-3.5-sonnet", 1000000, 0, is_cached=True)
assert cost_regular == pytest.approx(3.00, rel=1e-3)
assert cost_cached == pytest.approx(0.75, rel=1e-3) # 75% discount
starter/the_model_landscape_claude_gpt_gemini_lib.py (687 bytes)
from typing import Dict, Any, Tuple
class ModelRouter:
def __init__(self):
# Pricing per 1M tokens: (input_price, output_price)
self.pricing = {
"claude-3.5-sonnet": (3.00, 15.00),
"gpt-4o": (2.50, 10.00),
"gemini-2.0-flash": (0.10, 0.40),
"gpt-4o-mini": (0.15, 0.60)
}
def route_query(self, prompt: str, token_count: int = 100) -> str:
# TODO: Route prompt to optimal model tier
pass
def calculate_cost(self, model: str, input_tokens: int, output_tokens: int, is_cached: bool = False) -> float:
# TODO: Calculate query cost with optional prompt caching discount
pass
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 241 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_the_model_landscape_claude_gpt_gemini_lib.py (1280 bytes)
import pytest
from examples.the_model_landscape_claude_gpt_gemini_lib import ModelRouter
def test_router_dispatch_decisions():
router = ModelRouter()
# Code prompt should go to Claude Sonnet
code_prompt = "def merge_sort(arr):\n if len(arr) <= 1: return arr"
assert router.route_query(code_prompt) == "claude-3.5-sonnet"
# Fast classification prompt should go to GPT-4o-mini
simple_prompt = "Classify sentiment: Great movie!"
assert router.route_query(simple_prompt, token_count=20) == "gpt-4o-mini"
# Long document should go to Gemini Flash
assert router.route_query("Document text", token_count=200000) == "gemini-2.0-flash"
def test_cost_calculation():
router = ModelRouter()
# 1M input + 1M output on Sonnet = $3 + $15 = $18
cost = router.calculate_cost("claude-3.5-sonnet", 1000000, 1000000)
assert cost == pytest.approx(18.00, rel=1e-3)
def test_prompt_caching_discount():
router = ModelRouter()
cost_regular = router.calculate_cost("claude-3.5-sonnet", 1000000, 0, is_cached=False)
cost_cached = router.calculate_cost("claude-3.5-sonnet", 1000000, 0, is_cached=True)
assert cost_regular == pytest.approx(3.00, rel=1e-3)
assert cost_cached == pytest.approx(0.75, rel=1e-3) # 75% discount
Troubleshooting
Troubleshooting: Day 241 - The Model Landscape: Claude, GPT, Gemini, Llama
Common Issues
- Pricing calculation mismatch:
- Cause: Input tokens multiplied by output rates.
- Fix: Use separate pricing variables
in_rateandout_rate.
Security notes
Security & Privacy: Day 241 - The Model Landscape: Claude, GPT, Gemini, Llama
Security Guidance
- All routing optimization engines execute locally in memory.