LLMs and Generative AI › LLM APIs › Day 258

Hands-on lab — Day 258: Cost, Caching, and Rate Limits

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/cost_caching_and_rate_limits_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/cost_caching_and_rate_limits_lib.py
examples/test_cost_caching_and_rate_limits_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/cost_caching_and_rate_limits_lib.py
starter/test_cost_caching_and_rate_limits_lib.py
tests/run_tests.sh
tests/test_cost_caching_and_rate_limits_lib.py
troubleshooting.md

Lab README

Lab: Day 258 -- Cost, Caching, and Rate Limits

Lesson

Day number: 258 of 365. Course: Course06-SS02 (LLMs and Generative AI - LLM APIs). Topic: Cost Optimization, Prompt Caching, and Rate Limits.

Purpose

Build and test a Rate-Limited Cost Optimization Ledger in Python. Calculate multi-tier prompt token costs with cache discounts, implement a Token Bucket rate limiter, and compute full jitter backoff retry intervals.

Learning objectives

  • Calculate exact token costs across standard, cached, and completion tiers.
  • Implement a Token Bucket traffic shaping rate limiter.
  • Compute exponential backoff with full jitter for HTTP 429 resilience.
  • Maintain a cumulative multi-model token expenditure ledger.

Prerequisites

  • Day 257 (Working with Images and Documents).
  • Python 3.11+ with Pytest.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

Python and standard time/random modules are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/cost_caching_and_rate_limits_lib.py: Student scaffold file.
  • examples/cost_caching_and_rate_limits_lib.py: Complete reference implementation.
  • tests/test_cost_caching_and_rate_limits_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/cost_caching_and_rate_limits_lib.py

What the commands do

  • Records cached prompt usage.
  • Consumes tokens from TokenBucket.
  • Runs unit test assertions.

Expected output

Cost Demo Executed. Cost: $0.0075, Token Consumed: True

Validation steps

  1. Verify cached token cost reflects 90% discount rate.
  2. Confirm TokenBucket rejects requests exceeding capacity.
  3. Validate exponential jitter stays within mathematical bounds.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Cost scaling error: Ensure pricing rates per million tokens are divided by 1,000,000.

Security notes

All cost accounting runs locally in memory.

Extension exercises

  1. Implement a distributed Redis Token Bucket using atomic scripts.
  2. Build an automated prompt cache prefix linter.

Expected output

FIELDS.md

# Expected Output Fields: Day 258

- `Cost Demo`: Status of cost and token bucket execution.

examples-run.txt

Cost Demo Executed. Cost: $0.0075, Token Consumed: True

measured-values.txt

Cost Demo: Success
Calculated Cost: $0.0075
Token Bucket Consumed: True

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_cost_caching_and_rate_limits_lib.py::test_cost_calculation PASSED     [ 33%]
tests/test_cost_caching_and_rate_limits_lib.py::test_token_bucket PASSED         [ 66%]
tests/test_cost_caching_and_rate_limits_lib.py::test_full_jitter_range PASSED    [100%]

============================== 3 passed in 0.38s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/cost_caching_and_rate_limits_lib.py (2482 bytes)
import time
import random
from typing import Dict, Any

class TokenBucket:
    def __init__(self, capacity: float, refill_rate: float):
        self.capacity = capacity
        self.refill_rate = refill_rate
        self.tokens = capacity
        self.last_update = time.time()

    def consume(self, amount: float = 1.0) -> bool:
        now = time.time()
        elapsed = now - self.last_update
        self.last_update = now
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)

        if self.tokens >= amount:
            self.tokens -= amount
            return True
        return False

class LLMCostLedger:
    PRICING = {
        "claude-3-5-sonnet": {"input": 3.00, "cache_read": 0.30, "output": 15.00},
        "gpt-4o": {"input": 2.50, "cache_read": 1.25, "output": 10.00}
    }

    def __init__(self):
        self.total_cost_usd = 0.0
        self.total_input_tokens = 0
        self.total_cached_tokens = 0
        self.total_output_tokens = 0

    def record_usage(self, model: str, input_tokens: int, cached_tokens: int, output_tokens: int) -> float:
        rates = self.PRICING.get(model, {"input": 3.0, "cache_read": 0.3, "output": 15.0})
        cost = (
            (input_tokens * rates["input"] / 1_000_000) +
            (cached_tokens * rates["cache_read"] / 1_000_000) +
            (output_tokens * rates["output"] / 1_000_000)
        )
        self.total_cost_usd += cost
        self.total_input_tokens += input_tokens
        self.total_cached_tokens += cached_tokens
        self.total_output_tokens += output_tokens

        # Return the same value that was accumulated. Rounding here while
        # adding the unrounded figure to the total meant the returned costs
        # did not sum to total_cost_usd. Round at the point of DISPLAY, not
        # inside the ledger.
        return cost

def calculate_full_jitter_backoff(attempt: int, base_delay: float = 1.0, max_delay: float = 32.0) -> float:
    ceiling = min(max_delay, base_delay * (2 ** attempt))
    return random.uniform(0, ceiling)

def run_cost_demo():
    ledger = LLMCostLedger()
    cost = ledger.record_usage("claude-3-5-sonnet", 500, 10000, 200)
    bucket = TokenBucket(10, 2)
    consumed = bucket.consume(5)
    # Round HERE, at the point of display -- the ledger keeps full precision.
    print(f"Cost Demo Executed. Cost: ${cost:.4f}, Token Consumed: {consumed}")
    return cost, consumed

if __name__ == "__main__":
    run_cost_demo()
examples/test_cost_caching_and_rate_limits_lib.py (1085 bytes)
import pytest
from examples.cost_caching_and_rate_limits_lib import (
    TokenBucket,
    LLMCostLedger,
    calculate_full_jitter_backoff
)

def test_cost_calculation():
    ledger = LLMCostLedger()
    # 500 input @ $3 = 0.0015, 10000 cached @ $0.3 = 0.003, 200 out @ $15 = 0.003 -> Total = 0.0075
    cost = ledger.record_usage("claude-3-5-sonnet", 500, 10000, 200)

    # Binary floating point cannot represent 0.0075 exactly, so a computed
    # cost is never == the decimal literal. Compare with a tolerance; the
    # ledger keeps full precision and rounding happens at display time.
    assert cost == pytest.approx(0.0075)
    assert ledger.total_cost_usd == pytest.approx(0.0075)
    assert ledger.total_cached_tokens == 10000

def test_token_bucket():
    bucket = TokenBucket(10, 1) # 10 capacity, 1 per sec
    assert bucket.consume(8) is True
    assert bucket.consume(5) is False # Only 2 left

def test_full_jitter_range():
    for _ in range(10):
        backoff = calculate_full_jitter_backoff(3, base_delay=1.0, max_delay=32.0)
        assert 0.0 <= backoff <= 8.0
metadata.yml (441 bytes)
lesson_id: D258
day: 258
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/cost_caching_and_rate_limits_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/cost_caching_and_rate_limits_lib.py (597 bytes)
import time
import random
from typing import Dict, Any

class TokenBucket:
    def __init__(self, capacity: float, refill_rate: float):
        # TODO: Initialize bucket capacity and refill rate
        pass

    def consume(self, amount: float = 1.0) -> bool:
        # TODO: Refill and consume tokens
        pass

class LLMCostLedger:
    def __init__(self):
        # TODO: Initialize cost tracking attributes
        pass

    def record_usage(self, model: str, input_tokens: int, cached_tokens: int, output_tokens: int) -> float:
        # TODO: Calculate cost and record usage
        pass
starter/test_cost_caching_and_rate_limits_lib.py (1085 bytes)
import pytest
from examples.cost_caching_and_rate_limits_lib import (
    TokenBucket,
    LLMCostLedger,
    calculate_full_jitter_backoff
)

def test_cost_calculation():
    ledger = LLMCostLedger()
    # 500 input @ $3 = 0.0015, 10000 cached @ $0.3 = 0.003, 200 out @ $15 = 0.003 -> Total = 0.0075
    cost = ledger.record_usage("claude-3-5-sonnet", 500, 10000, 200)

    # Binary floating point cannot represent 0.0075 exactly, so a computed
    # cost is never == the decimal literal. Compare with a tolerance; the
    # ledger keeps full precision and rounding happens at display time.
    assert cost == pytest.approx(0.0075)
    assert ledger.total_cost_usd == pytest.approx(0.0075)
    assert ledger.total_cached_tokens == 10000

def test_token_bucket():
    bucket = TokenBucket(10, 1) # 10 capacity, 1 per sec
    assert bucket.consume(8) is True
    assert bucket.consume(5) is False # Only 2 left

def test_full_jitter_range():
    for _ in range(10):
        backoff = calculate_full_jitter_backoff(3, base_delay=1.0, max_delay=32.0)
        assert 0.0 <= backoff <= 8.0
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 258 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_cost_caching_and_rate_limits_lib.py (1085 bytes)
import pytest
from examples.cost_caching_and_rate_limits_lib import (
    TokenBucket,
    LLMCostLedger,
    calculate_full_jitter_backoff
)

def test_cost_calculation():
    ledger = LLMCostLedger()
    # 500 input @ $3 = 0.0015, 10000 cached @ $0.3 = 0.003, 200 out @ $15 = 0.003 -> Total = 0.0075
    cost = ledger.record_usage("claude-3-5-sonnet", 500, 10000, 200)

    # Binary floating point cannot represent 0.0075 exactly, so a computed
    # cost is never == the decimal literal. Compare with a tolerance; the
    # ledger keeps full precision and rounding happens at display time.
    assert cost == pytest.approx(0.0075)
    assert ledger.total_cost_usd == pytest.approx(0.0075)
    assert ledger.total_cached_tokens == 10000

def test_token_bucket():
    bucket = TokenBucket(10, 1) # 10 capacity, 1 per sec
    assert bucket.consume(8) is True
    assert bucket.consume(5) is False # Only 2 left

def test_full_jitter_range():
    for _ in range(10):
        backoff = calculate_full_jitter_backoff(3, base_delay=1.0, max_delay=32.0)
        assert 0.0 <= backoff <= 8.0

Troubleshooting

Troubleshooting: Day 258 - Cost, Caching, and Rate Limits

Common Issues

  1. Token rate limit overage:
    • Cause: Sending large unthrottled burst requests.
    • Fix: Wrap request dispatchers in TokenBucket rate limit checks.

Security notes

Security & Privacy: Day 258 - Cost, Caching, and Rate Limits

Security Guidance

  • Set hard spending quotas and circuit breakers to prevent financial denial-of-wallet attacks.