LLMs and Generative AI › The LLM Landscape › Day 241

Hands-on lab — Day 241: The Model Landscape: Claude, GPT, Gemini, Llama

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/the_model_landscape_claude_gpt_gemini_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/test_the_model_landscape_claude_gpt_gemini_lib.py
examples/the_model_landscape_claude_gpt_gemini_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/test_the_model_landscape_claude_gpt_gemini_lib.py
starter/the_model_landscape_claude_gpt_gemini_lib.py
tests/run_tests.sh
tests/test_the_model_landscape_claude_gpt_gemini_lib.py
troubleshooting.md

Lab README

Lab: Day 241 -- The Model Landscape: Claude, GPT, Gemini, Llama

Lesson

Day number: 241 of 365. Course: Course06-SS01 (LLMs and Generative AI - Working with LLMs). Topic: Frontier LLM Landscape, Model Routing, and Cost Optimization.

Purpose

Build and test a modular model routing and cost calculation engine in Python. Implement heuristic and keyword query classification, dispatch prompts to appropriate cost/latency tiers, calculate multi-tier API billing costs, and simulate blended enterprise cost savings.

Learning objectives

  • Analyze the operational trade-offs between Claude 3.5 Sonnet, GPT-4o, Gemini 2.0 Flash, and open-weights models.
  • Implement rule-based and keyword-driven semantic query dispatching.
  • Calculate exact multi-tier API costs incorporating input/output token pricing and prompt caching discounts.
  • Simulate blended fleet cost savings comparing monolithic routing vs multi-tier routing.

Prerequisites

  • Day 240 (Pretraining, Fine-Tuning, and RLHF).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

Python and standard math libraries are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/the_model_landscape_claude_gpt_gemini_lib.py: Student scaffold file.
  • examples/the_model_landscape_claude_gpt_gemini_lib.py: Complete reference implementation.
  • tests/test_the_model_landscape_claude_gpt_gemini_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/the_model_landscape_claude_gpt_gemini_lib.py

What the commands do

  • Evaluates query routing decisions across code, classification, and long-context prompts.
  • Computes API query costs with prompt caching discounts.
  • Runs unit test assertions.

Expected output

Router Demo: Code -> claude-3.5-sonnet, Quick -> gpt-4o-mini, Long -> gemini-2.0-flash, Sonnet Cost = $0.1650, Cached = $0.0525

Validation steps

  1. Verify code queries route to claude-3.5-sonnet.
  2. Confirm short classification queries route to gpt-4o-mini.
  3. Confirm long context queries ($> 150k$ tokens) route to gemini-2.0-flash.
  4. Confirm is_cached=True applies a $75%$ input token discount.
  5. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Cost scaling error: Ensure token counts are divided by $1,000,000$ when multiplying by rate card prices.

Security notes

All routing calculations execute locally in process memory.

Extension exercises

  1. Implement a latency-aware router incorporating provider status checks.
  2. Build an automated fallback cascade handling simulated 429 rate limit exceptions.

Expected output

FIELDS.md

# Expected Output Fields: Day 241

- `Code Route`: Selected model tier for complex code reasoning.
- `Quick Route`: Selected model tier for lightweight fast tasks.
- `Long Route`: Selected model tier for massive context queries.
- `Sonnet Cost`: Total standard query cost in USD.
- `Cached Cost`: Total discounted query cost with prompt caching.

examples-run.txt

Router Demo: Code -> claude-3.5-sonnet, Quick -> gpt-4o-mini, Long -> gemini-2.0-flash, Sonnet Cost = $0.1650, Cached = $0.0525

measured-values.txt

Code Route: claude-3.5-sonnet
Quick Route: gpt-4o-mini
Long Route: gemini-2.0-flash
Sonnet Cost: $0.1650
Cached Cost: $0.0525

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_the_model_landscape_claude_gpt_gemini_lib.py::test_router_dispatch_decisions PASSED [ 33%]
tests/test_the_model_landscape_claude_gpt_gemini_lib.py::test_cost_calculation PASSED [ 66%]
tests/test_the_model_landscape_claude_gpt_gemini_lib.py::test_prompt_caching_discount PASSED [100%]

============================== 3 passed in 0.41s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/test_the_model_landscape_claude_gpt_gemini_lib.py (1280 bytes)
import pytest
from examples.the_model_landscape_claude_gpt_gemini_lib import ModelRouter

def test_router_dispatch_decisions():
    router = ModelRouter()
    
    # Code prompt should go to Claude Sonnet
    code_prompt = "def merge_sort(arr):\n    if len(arr) <= 1: return arr"
    assert router.route_query(code_prompt) == "claude-3.5-sonnet"

    # Fast classification prompt should go to GPT-4o-mini
    simple_prompt = "Classify sentiment: Great movie!"
    assert router.route_query(simple_prompt, token_count=20) == "gpt-4o-mini"

    # Long document should go to Gemini Flash
    assert router.route_query("Document text", token_count=200000) == "gemini-2.0-flash"

def test_cost_calculation():
    router = ModelRouter()
    # 1M input + 1M output on Sonnet = $3 + $15 = $18
    cost = router.calculate_cost("claude-3.5-sonnet", 1000000, 1000000)
    assert cost == pytest.approx(18.00, rel=1e-3)

def test_prompt_caching_discount():
    router = ModelRouter()
    cost_regular = router.calculate_cost("claude-3.5-sonnet", 1000000, 0, is_cached=False)
    cost_cached = router.calculate_cost("claude-3.5-sonnet", 1000000, 0, is_cached=True)

    assert cost_regular == pytest.approx(3.00, rel=1e-3)
    assert cost_cached == pytest.approx(0.75, rel=1e-3) # 75% discount
examples/the_model_landscape_claude_gpt_gemini_lib.py (2017 bytes)
from typing import Dict, Any, Tuple

class ModelRouter:
    def __init__(self):
        # Pricing per 1M tokens: (input_price, output_price)
        self.pricing = {
            "claude-3.5-sonnet": (3.00, 15.00),
            "gpt-4o": (2.50, 10.00),
            "gemini-2.0-flash": (0.10, 0.40),
            "gpt-4o-mini": (0.15, 0.60)
        }

    def route_query(self, prompt: str, token_count: int = 100) -> str:
        prompt_lower = prompt.lower()
        if token_count > 150000:
            return "gemini-2.0-flash"

        if "def " in prompt or "class " in prompt or "theorem" in prompt_lower or "refactor" in prompt_lower:
            return "claude-3.5-sonnet"

        if token_count < 300 and ("classify" in prompt_lower or "extract" in prompt_lower or "sentiment" in prompt_lower):
            return "gpt-4o-mini"

        return "gemini-2.0-flash"

    def calculate_cost(self, model: str, input_tokens: int, output_tokens: int, is_cached: bool = False) -> float:
        in_rate, out_rate = self.pricing.get(model, (1.0, 3.0))
        if is_cached:
            in_rate = in_rate * 0.25 # 75% prompt cache discount

        cost = (input_tokens / 1e6) * in_rate + (output_tokens / 1e6) * out_rate
        return round(cost, 6)

def run_routing_demo():
    router = ModelRouter()
    m1 = router.route_query("def fibonacci(n): return n if n < 2 else fibonacci(n-1) + fibonacci(n-2)")
    m2 = router.route_query("Classify the sentiment of this review", token_count=50)
    m3 = router.route_query("Analyze complete archive", token_count=500000)

    cost_sonnet = router.calculate_cost("claude-3.5-sonnet", input_tokens=50000, output_tokens=1000)
    cost_cached = router.calculate_cost("claude-3.5-sonnet", input_tokens=50000, output_tokens=1000, is_cached=True)

    print(f"Router Demo: Code -> {m1}, Quick -> {m2}, Long -> {m3}, Sonnet Cost = ${cost_sonnet:.4f}, Cached = ${cost_cached:.4f}")
    return m1, m2, m3, cost_sonnet, cost_cached

if __name__ == "__main__":
    run_routing_demo()
metadata.yml (450 bytes)
lesson_id: D241
day: 241
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/the_model_landscape_claude_gpt_gemini_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/test_the_model_landscape_claude_gpt_gemini_lib.py (1280 bytes)
import pytest
from examples.the_model_landscape_claude_gpt_gemini_lib import ModelRouter

def test_router_dispatch_decisions():
    router = ModelRouter()
    
    # Code prompt should go to Claude Sonnet
    code_prompt = "def merge_sort(arr):\n    if len(arr) <= 1: return arr"
    assert router.route_query(code_prompt) == "claude-3.5-sonnet"

    # Fast classification prompt should go to GPT-4o-mini
    simple_prompt = "Classify sentiment: Great movie!"
    assert router.route_query(simple_prompt, token_count=20) == "gpt-4o-mini"

    # Long document should go to Gemini Flash
    assert router.route_query("Document text", token_count=200000) == "gemini-2.0-flash"

def test_cost_calculation():
    router = ModelRouter()
    # 1M input + 1M output on Sonnet = $3 + $15 = $18
    cost = router.calculate_cost("claude-3.5-sonnet", 1000000, 1000000)
    assert cost == pytest.approx(18.00, rel=1e-3)

def test_prompt_caching_discount():
    router = ModelRouter()
    cost_regular = router.calculate_cost("claude-3.5-sonnet", 1000000, 0, is_cached=False)
    cost_cached = router.calculate_cost("claude-3.5-sonnet", 1000000, 0, is_cached=True)

    assert cost_regular == pytest.approx(3.00, rel=1e-3)
    assert cost_cached == pytest.approx(0.75, rel=1e-3) # 75% discount
starter/the_model_landscape_claude_gpt_gemini_lib.py (687 bytes)
from typing import Dict, Any, Tuple

class ModelRouter:
    def __init__(self):
        # Pricing per 1M tokens: (input_price, output_price)
        self.pricing = {
            "claude-3.5-sonnet": (3.00, 15.00),
            "gpt-4o": (2.50, 10.00),
            "gemini-2.0-flash": (0.10, 0.40),
            "gpt-4o-mini": (0.15, 0.60)
        }

    def route_query(self, prompt: str, token_count: int = 100) -> str:
        # TODO: Route prompt to optimal model tier
        pass

    def calculate_cost(self, model: str, input_tokens: int, output_tokens: int, is_cached: bool = False) -> float:
        # TODO: Calculate query cost with optional prompt caching discount
        pass
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 241 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_the_model_landscape_claude_gpt_gemini_lib.py (1280 bytes)
import pytest
from examples.the_model_landscape_claude_gpt_gemini_lib import ModelRouter

def test_router_dispatch_decisions():
    router = ModelRouter()
    
    # Code prompt should go to Claude Sonnet
    code_prompt = "def merge_sort(arr):\n    if len(arr) <= 1: return arr"
    assert router.route_query(code_prompt) == "claude-3.5-sonnet"

    # Fast classification prompt should go to GPT-4o-mini
    simple_prompt = "Classify sentiment: Great movie!"
    assert router.route_query(simple_prompt, token_count=20) == "gpt-4o-mini"

    # Long document should go to Gemini Flash
    assert router.route_query("Document text", token_count=200000) == "gemini-2.0-flash"

def test_cost_calculation():
    router = ModelRouter()
    # 1M input + 1M output on Sonnet = $3 + $15 = $18
    cost = router.calculate_cost("claude-3.5-sonnet", 1000000, 1000000)
    assert cost == pytest.approx(18.00, rel=1e-3)

def test_prompt_caching_discount():
    router = ModelRouter()
    cost_regular = router.calculate_cost("claude-3.5-sonnet", 1000000, 0, is_cached=False)
    cost_cached = router.calculate_cost("claude-3.5-sonnet", 1000000, 0, is_cached=True)

    assert cost_regular == pytest.approx(3.00, rel=1e-3)
    assert cost_cached == pytest.approx(0.75, rel=1e-3) # 75% discount

Troubleshooting

Troubleshooting: Day 241 - The Model Landscape: Claude, GPT, Gemini, Llama

Common Issues

  1. Pricing calculation mismatch:
    • Cause: Input tokens multiplied by output rates.
    • Fix: Use separate pricing variables in_rate and out_rate.

Security notes

Security & Privacy: Day 241 - The Model Landscape: Claude, GPT, Gemini, Llama

Security Guidance

  • All routing optimization engines execute locally in memory.