LLMs and Generative AI β€Ί LLM APIs β€Ί Day 258

Day 258: Cost, Caching, and Rate Limits

Day 258 of 365 β€” Cost, Caching, and Rate Limits

Master production token economics and traffic resilience: optimize costs using Prompt Caching (Anthropic Prompt Caching & OpenAI KV Caching with 90% cost discounts), implement Token Bucket and Leaky Bucket rate limit algorithms in Python, handle HTTP 429 Too Many Requests with exponential backoff and jitter, track multi-model token expenditure ledgers with exact pricing matrices, and build a rate-limited cost optimization proxy in Python.

Course
LLMs and Generative AI
Category
LLM APIs
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-258-cost-caching-and-rate-limits

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/llms-and-generative-ai/day-258-cost-caching-and-rate-limits
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Deploying language model applications to thousands of active users without cost and traffic controls is a recipe for financial and operational disaster:

Production engineering requires strict token economics and traffic resilience. By mastering:

Today, you will master prompt cache breakpoint architectures, calculate multi-model token pricing ledgers, implement a client-side Token Bucket rate limiter in Python, and build an automated cost-optimizing API proxy.


The idea in plain language

Diagram: how a cached prompt prefix reduces cost and latency

Think of a research library with an expert consultant:


Historical background

Diagram: the historical progression of language model pricing and cost control

Prompt caching and KV cache breakpoint architecture diagram

  1. 2023 (Token Billing Explosions): Early RAG systems suffered severe cost inflation as developers passed massive document contexts on every conversational turn.
  2. 2024 (Anthropic Prompt Caching): Anthropic pioneered explicit cache_control breakpoints for Claude, providing a 90% discount on cached tokens with a 5-minute rolling TTL.
  3. 2024 (OpenAI Automatic Prefix Caching): OpenAI introduced automatic prompt caching for prompts exceeding 1,024 tokens in GPT-4o, matching the 90% discount tier.
  4. 2024 (vLLM Chunked Prefill & RadixAttention): Open-source inference engines implemented automated tree-based KV cache sharing, bringing enterprise prompt caching to self-hosted GPU clusters.

What it is β€” and what it is not

What Prompt Caching IS:

What it is NOT:


Why it was created and what problems it solves

Prompt caching and rate limiting solve 4 critical enterprise infrastructure challenges:

  1. Multi-Turn Cost Collapse: Reduces the cost of multi-turn chat agents with large system prompts by up to 90%.
  2. Sub-Second TTFT on Large Contexts: Slashes Time-To-First-Token on 50,000-token prompts from 5 seconds to under 400 milliseconds.
  3. Thundering Herd Prevention: Full jitter retries prevent thousands of microservices from crashing upstream APIs simultaneously after an outage.
  4. Budget Guardrails: Hard spending limits prevent runaway billing spikes caused by recursive agent loops.

How it works

Let us analyze prompt cache mechanics, rate-limit mathematics, and retry algorithms.


1. Anthropic Explicit Prompt Cache Breakpoints

In Claude, place cache_control: {"type": "ephemeral"} on static message blocks (minimum 1,024 tokens for Claude 3.5 Sonnet):

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "You are an expert legal assistant with deep knowledge of this contract.",
        },
        {
            "type": "text",
            "text": "FULL_50_PAGE_LEGAL_CONTRACT_TEXT_HERE...",
            "cache_control": {"type": "ephemeral"} # Cache Breakpoint!
        }
    ],
    messages=[{"role": "user", "content": "What is the termination clause?"}]
)

# Telemetry Inspection
print("Cache Creation Tokens:", response.usage.cache_creation_input_tokens)
print("Cache Read Tokens:", response.usage.cache_read_input_tokens)
print("Uncached Input Tokens:", response.usage.input_tokens)
print("Total Cost Savings Verified via Prompt Caching Breakpoint.")

2. Token Pricing Matrix & Cache Cost Mathematics

Diagram: the four separately priced token categories in a model request

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   PROMPT CACHING TOKEN ECONOMICS (PER MTOK)            β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Operation          β”‚ Standard Price  β”‚ Cached Price     β”‚ Discount     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Claude 3.5 Sonnet  β”‚ $3.00 / Mtok    β”‚ $0.30 / Mtok     β”‚ 90% OFF      β”‚
β”‚ GPT-4o             β”‚ $2.50 / Mtok    β”‚ $1.25 / Mtok     β”‚ 50% OFF      β”‚
β”‚ DeepSeek V3        β”‚ $0.14 / Mtok    β”‚ $0.014 / Mtok    β”‚ 90% OFF      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Cost Calculation Formula: Total Cost = (N_{uncached} * P_{standard}) + (N_{read} * P_{cached}) + (N_{output} * P_{out})


3. The Token Bucket Rate Limiting Algorithm

Diagram: how a token bucket rate limiter admits and rejects requests

Token bucket rate limiting and exponential backoff retry diagram

The Token Bucket algorithm regulates request rates using two parameters:

  1. Capacity (C): Maximum burst tokens the bucket can hold.
  2. Refill Rate (r): Tokens added to the bucket per second.
import time

class TokenBucket:
    def __init__(self, capacity: float, refill_rate: float):
        self.capacity = capacity
        self.refill_rate = refill_rate
        self.tokens = capacity
        self.last_update = time.time()

    def consume(self, amount: float = 1.0) -> bool:
        now = time.time()
        elapsed = now - self.last_update
        self.last_update = now
        # Refill tokens
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)

        if self.tokens >= amount:
            self.tokens -= amount
            return True
        return False

4. Exponential Backoff with Full Jitter

When upstream APIs return HTTP 429 Too Many Requests:


5. Multi-Model Token Cost Tracking Ledger

Diagram: the four dimensions along which model spending is tracked

Track expenses across OpenAI, Anthropic, and local models inside a centralized Python accounting class:


6. Prefix Invariance & Radix Tree Cache Eviction Policies

To maximize prompt cache hit rates across high-throughput production services:


7. Two-Dimensional Rate Limiting: Requests-Per-Minute (RPM) vs. Tokens-Per-Minute (TPM)

Diagram: request-rate limits compared with token-rate limits

Enterprise foundation model providers enforce two concurrent rate limit dimensions:


8. Inspecting Upstream Retry-After Headers & Decorrelated Jitter

When upstream cloud APIs reject traffic with HTTP 429 Too Many Requests:


9. Hard Budget Guardrails & Automated Circuit Breakers

Diagram: the four controls that bound spending on a metered model API

Autonomous agents and recursive tool calling loops can easily enter runaway cycles if unmonitored:


10. FinOps for AI: Real-Time Unit Economics & Pareto Optimization

Enterprise AI platform engineering requires continuous visibility into unit economics:


An everyday analogy

Diagram: fixed-cost infrastructure compared with metered per-request cost

Think of a fast-food drive-through:


Examples in practice

Let us build a Rate-Limited Cost Optimization Ledger in Python:

import time
import random
from typing import Dict, Any

class LLMCostLedger:
    PRICING = {
        "claude-3-5-sonnet": {"input": 3.00, "cache_read": 0.30, "output": 15.00},
        "gpt-4o": {"input": 2.50, "cache_read": 1.25, "output": 10.00}
    }

    def __init__(self):
        self.total_cost_usd = 0.0
        self.total_input_tokens = 0
        self.total_cached_tokens = 0
        self.total_output_tokens = 0

    def record_usage(self, model: str, input_tokens: int, cached_tokens: int, output_tokens: int) -> float:
        rates = self.PRICING.get(model, {"input": 3.0, "cache_read": 0.3, "output": 15.0})
        cost = (
            (input_tokens * rates["input"] / 1_000_000) +
            (cached_tokens * rates["cache_read"] / 1_000_000) +
            (output_tokens * rates["output"] / 1_000_000)
        )
        self.total_cost_usd += cost
        self.total_input_tokens += input_tokens
        self.total_cached_tokens += cached_tokens
        self.total_output_tokens += output_tokens
        return round(cost, 6)

def calculate_full_jitter_backoff(attempt: int, base_delay: float = 1.0, max_delay: float = 32.0) -> float:
    ceiling = min(max_delay, base_delay * (2 ** attempt))
    return random.uniform(0, ceiling)

Implications: security, privacy, performance, scalability, and cost

Diagram: the design, monitoring, abuse and architecture implications of per-token pricing

  1. Deterministic Cache Placement:
    • Placing static system prompts at the beginning guarantees 90% cost savings across all multi-turn sessions.
  2. Preventing Financial Denial-of-Wallet:
    • Enforcing strict token bucket quotas and circuit breakers protects applications from malicious API cost depletion attacks.

Alternatives: free, open source, and commercial

SolutionTypeCaching MechanismBest For
Anthropic Prompt CacheCloud APIExplicit BreakpointsLarge system prompts & documents
OpenAI Prefix CacheCloud APIAutomatic (1024+ Tok)Standard chat completions
vLLM RadixAttentionOSS EngineTree-based KV CacheHigh-throughput GPU serving
Redis Semantic CacheCache LayerEmbedding SimilarityExact/Near-exact query deduplication

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   CACHING MECHANISM COMPARISON                         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Feature            β”‚ KV Prompt Cache β”‚ Semantic Cache   β”‚ Response Cacheβ”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Level              β”‚ Transformer KV  β”‚ Vector Database  β”‚ HTTP Key-Valueβ”‚
β”‚ Output Diversity   β”‚ Fresh & Dynamic β”‚ Stored Response  β”‚ Stored Output β”‚
β”‚ Invalidation Risk  β”‚ Prefix Change   β”‚ Low Threshold    β”‚ Cache TTL     β”‚
β”‚ Cost Reduction     β”‚ 50% - 90%       β”‚ 100%             β”‚ 100%          β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when prompt caching is worthwhile

When to Use Prompt Caching:

When NOT to use:


The AI connection

Knowledge check

  1. What is the standard cost discount for prompt cache reads in Claude 3.5 Sonnet?
  2. How does the Token Bucket algorithm prevent sudden request burst overflows?
  3. Why does Full Jitter backoff prevent β€œthundering herd” retry collisions?
  4. Why must static context be placed at the very beginning of the prompt array for caching?
  5. What HTTP status code signals that rate limit quotas have been exceeded?

Hands-on exercise

In this lab, you will build and test a Rate-Limited Cost Optimization Ledger in Python: track token costs across standard, cached, and output token tiers, implement a Token Bucket rate limiter with refill mechanics, calculate full jitter exponential backoff intervals, and verify output assertions against automated test suites.

Expected output

[Cost, Caching, and Rate Limits Verification Suite]
Testing Token Cost Calculation Matrix:
  Claude 3.5 Sonnet (10k cached, 500 input, 200 output): $0.0075 [PASS]
  Savings vs Uncached: 76.5% cost reduction [PASS]
Testing Token Bucket Rate Limiter:
  Burst Consumption: 5 tokens consumed successfully [PASS]
  Depleted Bucket Throttling: Rejected when empty [PASS]
Testing Exponential Backoff with Jitter:
  Attempt 3 Jitter Interval: Within range [0, 8.0s] [VERIFIED]
Test Suite: 3 passed in 0.38s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement a Distributed Token Bucket Rate Limiter using Redis atomic scripts (EVAL) that synchronizes rate limits across 10 worker containers.
  2. Build an automated Prompt Cache Breakpoint Linter that scans message arrays and warns if dynamic content precedes cache breakpoints.

Extension challenge

Implement an Adaptive Model Cost-Routing Circuit Breaker:

Quiz

Q1. What is the primary financial benefit of Prompt Caching (KV Caching) in modern LLM APIs?

  1. Free unlimited output generation, forever and always
  2. Up to 90% off cached prefix tokens, and faster TTFT
  3. Free GPU hardware upgrades
  4. Zero network bandwidth usage
Show answer

Answer: B. Up to 90% off cached prefix tokens, and faster TTFT

Cached prompt tokens bypass recomputation, resulting in a 90% price discount and drastic latency improvements.

Q2. How does the Token Bucket algorithm control API request volume?

  1. It resets user passwords when limit is exceeded
  2. It randomly deletes 50% of user requests
  3. Tokens refill at a fixed rate and each request spends some
  4. It increases prompt costs dynamically
Show answer

Answer: C. Tokens refill at a fixed rate and each request spends some

Token bucket smoothly shapes traffic bursts while enforcing strict average rate constraints.

Q3. What is Full Jitter in exponential backoff retry algorithms?

  1. Disabling SSL certificates
  2. Shaking the computer monitor
  3. Retrying 100 times per second
  4. A random sleep added to each backoff interval
Show answer

Answer: D. A random sleep added to each backoff interval

Full jitter decorrelates retry intervals across distributed workers, eliminating thundering herd collision waves.

Q4. In Anthropic Claude Messages API, how do you explicitly declare a prompt cache breakpoint?

  1. Add cache_control ephemeral to the content block
  2. Passing ?cache=true in the URL
  3. Setting temperature=0.0
  4. Anthropic does not support prompt caching at all
Show answer

Answer: A. Add cache_control ephemeral to the content block

Setting cache_control: {"type": "ephemeral"} explicitly marks a prompt caching breakpoint in Anthropic.

Q5. What HTTP status code is returned when an API client exceeds request-per-minute (RPM) or token-per-minute (TPM) quotas?

  1. HTTP 200 OK
  2. HTTP 429 Too Many Requests
  3. HTTP 404 Not Found
  4. HTTP 500 Internal Server Error
Show answer

Answer: B. HTTP 429 Too Many Requests

HTTP 429 indicates rate limiting, often accompanied by a Retry-After response header.

Glossary

Prompt Caching
An optimization technique storing computed Key-Value (KV) attention states for static prompt prefixes across API calls.
Cache Breakpoint
An explicit metadata marker placed on a message block instructing the server to freeze and persist the preceding KV cache state.
Token Bucket Algorithm
A rate-limiting algorithm that maintains a bucket of tokens refilled at a constant rate, consuming tokens per request or byte.
Requests Per Minute (RPM)
An API rate-limit quota constraining the total number of discrete HTTP requests permitted within a 60-second window.
Tokens Per Minute (TPM)
An API rate-limit quota constraining the aggregate number of input plus output tokens processed within a 60-second window.
Exponential Backoff
An error retry strategy that doubles the waiting duration after each consecutive failure to reduce load on the failing server.
Full Jitter
Introducing uniform random variation to retry backoff intervals to prevent synchronized collision waves across worker pools.
HTTP 429 Too Many Requests
The standard HTTP status code signaling that the client has exceeded its allotted rate or concurrency limits.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.