LLMs and Generative AI › LLM APIs › Day 258
Hands-on lab — Day 258: Cost, Caching, and Rate Limits
- ← Back to the Day 258 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-258-cost-caching-and-rate-limits/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/cost_caching_and_rate_limits_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/cost_caching_and_rate_limits_lib.py examples/test_cost_caching_and_rate_limits_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/cost_caching_and_rate_limits_lib.py starter/test_cost_caching_and_rate_limits_lib.py tests/run_tests.sh tests/test_cost_caching_and_rate_limits_lib.py troubleshooting.md
Lab README
Lab: Day 258 -- Cost, Caching, and Rate Limits
Lesson
Day number: 258 of 365. Course: Course06-SS02 (LLMs and Generative AI - LLM APIs). Topic: Cost Optimization, Prompt Caching, and Rate Limits.
Purpose
Build and test a Rate-Limited Cost Optimization Ledger in Python. Calculate multi-tier prompt token costs with cache discounts, implement a Token Bucket rate limiter, and compute full jitter backoff retry intervals.
Learning objectives
- Calculate exact token costs across standard, cached, and completion tiers.
- Implement a Token Bucket traffic shaping rate limiter.
- Compute exponential backoff with full jitter for HTTP 429 resilience.
- Maintain a cumulative multi-model token expenditure ledger.
Prerequisites
- Day 257 (Working with Images and Documents).
- Python 3.11+ with Pytest.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
Python and standard time/random modules are open source.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/cost_caching_and_rate_limits_lib.py: Student scaffold file.examples/cost_caching_and_rate_limits_lib.py: Complete reference implementation.tests/test_cost_caching_and_rate_limits_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/cost_caching_and_rate_limits_lib.py
What the commands do
- Records cached prompt usage.
- Consumes tokens from TokenBucket.
- Runs unit test assertions.
Expected output
Cost Demo Executed. Cost: $0.0075, Token Consumed: True
Validation steps
- Verify cached token cost reflects 90% discount rate.
- Confirm TokenBucket rejects requests exceeding capacity.
- Validate exponential jitter stays within mathematical bounds.
- Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Cost scaling error: Ensure pricing rates per million tokens are divided by 1,000,000.
Security notes
All cost accounting runs locally in memory.
Extension exercises
- Implement a distributed Redis Token Bucket using atomic scripts.
- Build an automated prompt cache prefix linter.
Navigation
- Lesson title: Cost, Caching, and Rate Limits
- Day number: 258 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-258-cost-caching-and-rate-limits
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-258-cost-caching-and-rate-limitswhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 258
- `Cost Demo`: Status of cost and token bucket execution.
examples-run.txt
Cost Demo Executed. Cost: $0.0075, Token Consumed: True
measured-values.txt
Cost Demo: Success
Calculated Cost: $0.0075
Token Bucket Consumed: True
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 3 items
tests/test_cost_caching_and_rate_limits_lib.py::test_cost_calculation PASSED [ 33%]
tests/test_cost_caching_and_rate_limits_lib.py::test_token_bucket PASSED [ 66%]
tests/test_cost_caching_and_rate_limits_lib.py::test_full_jitter_range PASSED [100%]
============================== 3 passed in 0.38s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/cost_caching_and_rate_limits_lib.py (2482 bytes)
import time
import random
from typing import Dict, Any
class TokenBucket:
def __init__(self, capacity: float, refill_rate: float):
self.capacity = capacity
self.refill_rate = refill_rate
self.tokens = capacity
self.last_update = time.time()
def consume(self, amount: float = 1.0) -> bool:
now = time.time()
elapsed = now - self.last_update
self.last_update = now
self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
if self.tokens >= amount:
self.tokens -= amount
return True
return False
class LLMCostLedger:
PRICING = {
"claude-3-5-sonnet": {"input": 3.00, "cache_read": 0.30, "output": 15.00},
"gpt-4o": {"input": 2.50, "cache_read": 1.25, "output": 10.00}
}
def __init__(self):
self.total_cost_usd = 0.0
self.total_input_tokens = 0
self.total_cached_tokens = 0
self.total_output_tokens = 0
def record_usage(self, model: str, input_tokens: int, cached_tokens: int, output_tokens: int) -> float:
rates = self.PRICING.get(model, {"input": 3.0, "cache_read": 0.3, "output": 15.0})
cost = (
(input_tokens * rates["input"] / 1_000_000) +
(cached_tokens * rates["cache_read"] / 1_000_000) +
(output_tokens * rates["output"] / 1_000_000)
)
self.total_cost_usd += cost
self.total_input_tokens += input_tokens
self.total_cached_tokens += cached_tokens
self.total_output_tokens += output_tokens
# Return the same value that was accumulated. Rounding here while
# adding the unrounded figure to the total meant the returned costs
# did not sum to total_cost_usd. Round at the point of DISPLAY, not
# inside the ledger.
return cost
def calculate_full_jitter_backoff(attempt: int, base_delay: float = 1.0, max_delay: float = 32.0) -> float:
ceiling = min(max_delay, base_delay * (2 ** attempt))
return random.uniform(0, ceiling)
def run_cost_demo():
ledger = LLMCostLedger()
cost = ledger.record_usage("claude-3-5-sonnet", 500, 10000, 200)
bucket = TokenBucket(10, 2)
consumed = bucket.consume(5)
# Round HERE, at the point of display -- the ledger keeps full precision.
print(f"Cost Demo Executed. Cost: ${cost:.4f}, Token Consumed: {consumed}")
return cost, consumed
if __name__ == "__main__":
run_cost_demo()
examples/test_cost_caching_and_rate_limits_lib.py (1085 bytes)
import pytest
from examples.cost_caching_and_rate_limits_lib import (
TokenBucket,
LLMCostLedger,
calculate_full_jitter_backoff
)
def test_cost_calculation():
ledger = LLMCostLedger()
# 500 input @ $3 = 0.0015, 10000 cached @ $0.3 = 0.003, 200 out @ $15 = 0.003 -> Total = 0.0075
cost = ledger.record_usage("claude-3-5-sonnet", 500, 10000, 200)
# Binary floating point cannot represent 0.0075 exactly, so a computed
# cost is never == the decimal literal. Compare with a tolerance; the
# ledger keeps full precision and rounding happens at display time.
assert cost == pytest.approx(0.0075)
assert ledger.total_cost_usd == pytest.approx(0.0075)
assert ledger.total_cached_tokens == 10000
def test_token_bucket():
bucket = TokenBucket(10, 1) # 10 capacity, 1 per sec
assert bucket.consume(8) is True
assert bucket.consume(5) is False # Only 2 left
def test_full_jitter_range():
for _ in range(10):
backoff = calculate_full_jitter_backoff(3, base_delay=1.0, max_delay=32.0)
assert 0.0 <= backoff <= 8.0
metadata.yml (441 bytes)
lesson_id: D258
day: 258
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/cost_caching_and_rate_limits_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/cost_caching_and_rate_limits_lib.py (597 bytes)
import time
import random
from typing import Dict, Any
class TokenBucket:
def __init__(self, capacity: float, refill_rate: float):
# TODO: Initialize bucket capacity and refill rate
pass
def consume(self, amount: float = 1.0) -> bool:
# TODO: Refill and consume tokens
pass
class LLMCostLedger:
def __init__(self):
# TODO: Initialize cost tracking attributes
pass
def record_usage(self, model: str, input_tokens: int, cached_tokens: int, output_tokens: int) -> float:
# TODO: Calculate cost and record usage
pass
starter/test_cost_caching_and_rate_limits_lib.py (1085 bytes)
import pytest
from examples.cost_caching_and_rate_limits_lib import (
TokenBucket,
LLMCostLedger,
calculate_full_jitter_backoff
)
def test_cost_calculation():
ledger = LLMCostLedger()
# 500 input @ $3 = 0.0015, 10000 cached @ $0.3 = 0.003, 200 out @ $15 = 0.003 -> Total = 0.0075
cost = ledger.record_usage("claude-3-5-sonnet", 500, 10000, 200)
# Binary floating point cannot represent 0.0075 exactly, so a computed
# cost is never == the decimal literal. Compare with a tolerance; the
# ledger keeps full precision and rounding happens at display time.
assert cost == pytest.approx(0.0075)
assert ledger.total_cost_usd == pytest.approx(0.0075)
assert ledger.total_cached_tokens == 10000
def test_token_bucket():
bucket = TokenBucket(10, 1) # 10 capacity, 1 per sec
assert bucket.consume(8) is True
assert bucket.consume(5) is False # Only 2 left
def test_full_jitter_range():
for _ in range(10):
backoff = calculate_full_jitter_backoff(3, base_delay=1.0, max_delay=32.0)
assert 0.0 <= backoff <= 8.0
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 258 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_cost_caching_and_rate_limits_lib.py (1085 bytes)
import pytest
from examples.cost_caching_and_rate_limits_lib import (
TokenBucket,
LLMCostLedger,
calculate_full_jitter_backoff
)
def test_cost_calculation():
ledger = LLMCostLedger()
# 500 input @ $3 = 0.0015, 10000 cached @ $0.3 = 0.003, 200 out @ $15 = 0.003 -> Total = 0.0075
cost = ledger.record_usage("claude-3-5-sonnet", 500, 10000, 200)
# Binary floating point cannot represent 0.0075 exactly, so a computed
# cost is never == the decimal literal. Compare with a tolerance; the
# ledger keeps full precision and rounding happens at display time.
assert cost == pytest.approx(0.0075)
assert ledger.total_cost_usd == pytest.approx(0.0075)
assert ledger.total_cached_tokens == 10000
def test_token_bucket():
bucket = TokenBucket(10, 1) # 10 capacity, 1 per sec
assert bucket.consume(8) is True
assert bucket.consume(5) is False # Only 2 left
def test_full_jitter_range():
for _ in range(10):
backoff = calculate_full_jitter_backoff(3, base_delay=1.0, max_delay=32.0)
assert 0.0 <= backoff <= 8.0
Troubleshooting
Troubleshooting: Day 258 - Cost, Caching, and Rate Limits
Common Issues
- Token rate limit overage:
- Cause: Sending large unthrottled burst requests.
- Fix: Wrap request dispatchers in TokenBucket rate limit checks.
Security notes
Security & Privacy: Day 258 - Cost, Caching, and Rate Limits
Security Guidance
- Set hard spending quotas and circuit breakers to prevent financial denial-of-wallet attacks.