AI Engineering: Agents and Applications › Production Retrieval and Pipelines › Day 327
Hands-on lab — Day 327: Cost Engineering for AI Systems
- ← Back to the Day 327 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/ai-engineering/day-327-cost-engineering-for-ai-systems/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/costs_demo.py Test
bash tests/run_tests.sh File tree
examples/costs_demo.py examples/costs.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/costs.py tests/run_tests.sh tests/test_costs.py troubleshooting.md
Lab README
Lab — Day 327: Cost Engineering for AI Systems
Lesson
- Lesson title: Cost Engineering for AI Systems
- Day number: 327 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-327-cost-engineering-for-ai-systems
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-327-cost-engineering-for-ai-systemswhen the site is running.
Purpose
Make the cost of an AI pipeline visible, then reduce it deliberately. You build a ledger that attributes spend per model, a router that sends easy work to a cheaper model, a context trimmer, and a spend cap that degrades before it refuses — then measure what each lever actually saves on the same workload.
Everything runs offline. Prices are a fixed table and tokens are estimated crudely; the accounting is real.
Learning objectives
- Attribute spend per model rather than reporting a single total.
- Demonstrate that cache hit rate and cost saving are different measurements.
- Route requests to the cheapest adequate model and trim context to a token budget.
- Enforce a cap checked before the call, with degradation before refusal.
- Project cost per request to expected volume.
Prerequisites
- Day 326, "Scaling Retrieval".
- Comfortable with Python dataclasses and dictionaries.
- No API key and no account — the model is simulated.
Supported operating systems
macOS, Linux and Windows (via WSL or PowerShell). Nothing is platform-specific.
Hardware requirements
Any machine running Python 3.10 or newer. No GPU. The suite completes in hundredths of a second.
Required software
- Python 3.10 or newer.
pytest7 or newer.
Free and open-source options
Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no API key, no spend — the price table is a constant in the source. The lesson discusses LiteLLM, Langfuse and Helicone as the real open-source options for budgets and cost observability.
Installation
python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt
Removing .venv fully reverses the installation.
File structure
starter/costs.py your work: cost_of, route, trim_context,
unit_economics, Pipeline.answer
examples/costs.py reference implementation
examples/costs_demo.py runs one workload four ways
tests/test_costs.py grouped by lever
tests/run_tests.sh suite entry point
expected-output/ real captured output and measured values
requirements/ pinned dependency
How to run
python3 examples/costs_demo.py # compare the levers
bash tests/run_tests.sh # run the suite
To work on the exercise, edit starter/costs.py, then copy it over the reference:
cp starter/costs.py examples/costs.py
bash tests/run_tests.sh
Keep a copy of the reference first if you want to compare.
What the commands do
python3 examples/costs_demo.py runs the same six requests four ways — no cache, cached, cached with trimmed context, and under a budget too small to complete — printing total spend, spend by model, and cost per request for each, then projecting to a million requests.
bash tests/run_tests.sh runs pytest over twenty tests grouped by lever: accounting, cache, router, trim, budget and unit economics.
Expected output
workload: 6 requests, 3 of them the same question
no cache, full ctx calls=6 cached=0 total=$0.0085 per_request=$0.00141
by model: large=$0.0073, small=$0.0012
cache, full ctx calls=6 cached=2 total=$0.0079 per_request=$0.00131
by model: cache=$0.0000, large=$0.0073, small=$0.0006
cache, trimmed ctx calls=6 cached=2 total=$0.0066 per_request=$0.00109
by model: cache=$0.0000, large=$0.0061, small=$0.0005
tight budget STOPPED: request would cost $0.0036, only $0.0009 left
tight budget calls=2 cached=0 total=$0.0006 per_request=$0.00030
by model: small=$0.0006
projection at 1M requests: $1,092 (from $0.00109 per request)
The second row is the one to study: a 33 percent cache hit rate produced a 7 percent saving, because the repeated question routed to the cheap model. expected-output/FIELDS.md works through the attribution.
Validation steps
bash tests/run_tests.shreports20 passed.- Run the demo and confirm the
tight budgetrow stops rather than completing, and that its total is at or below the budget — never above. - Check the
by modelline includescache=$0.0000. If cache hits are absent from the ledger rather than present at zero cost, your hit rate is invisible.
Tests
Twenty tests in six groups:
- accounting — output costs more than input, cost is linear in tokens, the large model is an order of magnitude dearer, an unknown model raises rather than costing zero, and a token estimate is never zero.
- cache — a repeat is free, hits appear in the ledger, and different context is a different key.
- router — reasoning and long prompts go large, simple ones go small.
- trim — keeps what fits in priority order and stops rather than skipping ahead.
- budget — checked before spending, degrades before refusing, and
remainingnever goes negative. - unit economics — scales linearly, handles zero requests, and attributes by model.
Cleanup
rm -rf __pycache__ .pytest_cache
rm -rf .venv
Both are safe to remove and fully reversible.
Troubleshooting
See troubleshooting.md for each failure this lab is designed to produce.
Security notes
See security.md. In short: no network, no credentials, no spend. The lesson treats unbounded spend as a security concern — a denial-of-wallet attack — rather than only a budgeting one.
Extension exercises
- Per-tenant budgets. Enforce a per-tenant cap alongside the global one, and prove both that one tenant cannot exhaust another's budget and that per-tenant limits do not sum to a safe global limit.
- Semantic caching. Cache on similarity rather than exact match, and measure the wrong answers served at each threshold alongside the savings.
- Self-hosted crossover. Add a fixed hourly cost for a self-hosted model and compute the request volume at which it beats per-token pricing.
Navigation
- Lesson
- Previous: Day 326 — Scaling Retrieval
- Next: Day 328 — Privacy in AI Systems
Expected output
FIELDS.md
# Reading the cost output
## The per-configuration lines
cache, trimmed ctx calls=6 cached=2 total=$0.0066 per_request=$0.00109
by model: cache=$0.0000, large=$0.0061, small=$0.0005
| Field | Meaning |
| --- | --- |
| `calls` | Entries in the ledger, including cache hits. |
| `cached` | How many of those were hits. Recorded as zero-cost calls rather than skipped, so the hit rate stays visible. |
| `total` | Dollar spend for the workload. |
| `per_request` | Total divided by calls — the number that gets multiplied by volume. |
| `by model` | Attribution. This line is the whole point: a total says what you spent, attribution says where to act. |
## Why `by model` matters more than `total`
Compare the baseline and the cached run:
no cache large=$0.0073, small=$0.0012 total $0.0085
+ cache large=$0.0073, small=$0.0006 total $0.0079
The cache changed only the small-model column. The large model — 86 percent of spend from two of six requests — was untouched, because the repeated question was a cheap one.
A dashboard reporting "33% cache hit rate" would call this a success. The attribution shows it saved 7 percent.
## The budget line
tight budget STOPPED: request would cost $0.0036, only $0.0009 left
The cap is checked **before** the call. Spend stops at or below the budget rather than overshooting by the size of the last request. Note also that the pipeline degrades — downgrading large-model requests to the small model — before it refuses outright.
## The projection
projection at 1M requests: $1,092 (from $0.00109 per request)
Cost per request multiplied by expected volume. This belongs in a design document before the feature ships, not in a retrospective after the invoice.
examples-run.txt
workload: 6 requests, 3 of them the same question
no cache, full ctx calls=6 cached=0 total=$0.0085 per_request=$0.00141
by model: large=$0.0073, small=$0.0012
cache, full ctx calls=6 cached=2 total=$0.0079 per_request=$0.00131
by model: cache=$0.0000, large=$0.0073, small=$0.0006
cache, trimmed ctx calls=6 cached=2 total=$0.0066 per_request=$0.00109
by model: cache=$0.0000, large=$0.0061, small=$0.0005
tight budget STOPPED: request would cost $0.0036, only $0.0009 left
tight budget calls=2 cached=0 total=$0.0006 per_request=$0.00030
by model: small=$0.0006
projection at 1M requests: $1,092 (from $0.00109 per request)
measured-values.txt
Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.
python3 examples/costs_demo.py -> 9 lines, exit 0
bash tests/run_tests.sh -> 20 passed in 0.01s
The four configurations, same six-request workload:
configuration total per request note
no cache, full ctx $0.0085 $0.00141 baseline
cache, full ctx $0.0079 $0.00131 2 hits, saved 7%
cache, trimmed ctx $0.0066 $0.00109 saved 22% against baseline
tight budget stopped -- refused before overspending
The number worth staring at is the second row. The cache hit 2 of 6 requests --
a 33% hit rate -- and saved 7% of the cost, because the repeated question
routed to the SMALL model. Spend by model on the baseline:
large $0.0073 (86% of spend, from 2 of 6 requests)
small $0.0012
Hit rate is not savings. Trimming context saved more precisely because it acted
on the large-model calls.
Projection: $1,092 per million requests, from $0.00109 per request.
run-output.txt
workload: 6 requests, 3 of them the same question
no cache, full ctx calls=6 cached=0 total=$0.0085 per_request=$0.00141
by model: large=$0.0073, small=$0.0012
cache, full ctx calls=6 cached=2 total=$0.0079 per_request=$0.00131
by model: cache=$0.0000, large=$0.0073, small=$0.0006
cache, trimmed ctx calls=6 cached=2 total=$0.0066 per_request=$0.00109
by model: cache=$0.0000, large=$0.0061, small=$0.0005
tight budget STOPPED: request would cost $0.0036, only $0.0009 left
tight budget calls=2 cached=0 total=$0.0006 per_request=$0.00030
by model: small=$0.0006
projection at 1M requests: $1,092 (from $0.00109 per request)
starter-run.txt
An untouched starter fails most of the suite:
17 failed, 3 passed in 0.05s
The three that pass check the PRICES table and estimate_tokens, neither of
which is stubbed:
test_output_tokens_cost_more_than_input_tokens
test_large_model_costs_an_order_of_magnitude_more (partially)
test_token_estimate_is_never_zero
Everything else depends on cost_of, route, trim_context, unit_economics or
Pipeline.answer, all of which raise NotImplementedError.
test-output.txt
.................... [100%]
20 passed in 0.01s
test-run.txt
.................... [100%]
20 passed in 0.01s
Source files
examples/costs_demo.py (2141 bytes)
#!/usr/bin/env python3
"""Run a small workload four ways and compare what each lever saves."""
from __future__ import annotations
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from costs import BudgetExceeded, Pipeline, unit_economics
CONTEXT = [f"retrieved passage {i} " * 20 for i in range(6)]
PROMPTS = [
"What is the refund window?",
"What is the refund window?", # exact repeat -> cache
"Analyse why refunds spiked in March", # reasoning -> large model
"List the supported currencies",
"What is the refund window?", # repeat again
"Compare the standard and premium tiers",
]
def run(label: str, *, budget: float, context_budget: int, use_cache: bool) -> None:
pipe = Pipeline(budget=budget)
for prompt in PROMPTS:
if not use_cache:
pipe.cache.clear()
try:
pipe.answer(prompt, CONTEXT, context_budget=context_budget)
except BudgetExceeded as exc:
print(f"{label:<22} STOPPED: {exc}")
break
econ = unit_economics(pipe.ledger, len(pipe.ledger.calls))
models = ", ".join(f"{m}=${c:.4f}" for m, c in sorted(pipe.ledger.by_model().items()))
print(f"{label:<22} {pipe.ledger.summary()} per_request=${econ['per_request']:.5f}")
print(f"{'':<22} by model: {models}")
def main() -> int:
print("workload: 6 requests, 3 of them the same question")
run("no cache, full ctx", budget=1.0, context_budget=10_000, use_cache=False)
run("cache, full ctx", budget=1.0, context_budget=10_000, use_cache=True)
run("cache, trimmed ctx", budget=1.0, context_budget=400, use_cache=True)
run("tight budget", budget=0.0015, context_budget=10_000, use_cache=False)
pipe = Pipeline(budget=1.0)
for prompt in PROMPTS:
pipe.answer(prompt, CONTEXT, context_budget=400)
econ = unit_economics(pipe.ledger, len(PROMPTS))
print(
f"projection at 1M requests: ${econ['per_million_requests']:,.0f} "
f"(from ${econ['per_request']:.5f} per request)"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
examples/costs.py (6762 bytes)
"""Cost accounting, caching, tiering and budget enforcement for an AI pipeline.
Offline and standard-library only. Prices are a fixed table rather than a live
API, and "tokens" are counted with a crude word-based estimator -- both are
simulations. What is real is the accounting: where cost accumulates, which
levers move it, and what happens when a budget runs out.
The four levers, in the order they usually pay off:
cache an exact repeat costs nothing
tier route easy work to a cheaper model
trim send less context
cap refuse, or degrade, rather than overspend
The last one is the one teams skip, and it is the only one that bounds the
worst case. Without a cap, a retry loop against a paid API is unbounded spend.
"""
from __future__ import annotations
from dataclasses import dataclass, field
# Illustrative prices in dollars per million tokens. The shape is what
# matters: output costs several times more than input, and the gap between
# tiers is an order of magnitude rather than a few percent.
PRICES: dict[str, tuple[float, float]] = {
"small": (0.25, 1.25),
"large": (3.00, 15.00),
}
class BudgetExceeded(RuntimeError):
"""Raised when a request would push spend past its cap."""
def estimate_tokens(text: str) -> int:
"""Crude token estimate: roughly four characters per token.
Real tokenisers are model-specific and this is not one. It is honest about
being an estimate, which is the right posture for a pre-flight cost check --
you cap on the estimate, then reconcile against the provider's reported
usage afterwards.
"""
return max(1, len(text) // 4)
def cost_of(model: str, input_tokens: int, output_tokens: int) -> float:
"""Dollar cost of one call, from the price table."""
if model not in PRICES:
raise KeyError(f"no price for model '{model}'")
in_price, out_price = PRICES[model]
return (input_tokens * in_price + output_tokens * out_price) / 1_000_000
@dataclass
class Call:
model: str
input_tokens: int
output_tokens: int
cost: float
cached: bool = False
@dataclass
class Ledger:
"""Every call, so cost can be attributed rather than merely totalled."""
calls: list[Call] = field(default_factory=list)
@property
def total(self) -> float:
return sum(c.cost for c in self.calls)
@property
def cache_hits(self) -> int:
return sum(1 for c in self.calls if c.cached)
def by_model(self) -> dict[str, float]:
out: dict[str, float] = {}
for call in self.calls:
out[call.model] = out.get(call.model, 0.0) + call.cost
return out
def summary(self) -> str:
return (
f"calls={len(self.calls)} cached={self.cache_hits} "
f"total=${self.total:.4f}"
)
def route(prompt: str, *, hard_words: tuple[str, ...] = ("analyse", "compare", "explain why")) -> str:
"""Pick the cheapest model that can plausibly do the job.
A real router uses a classifier or a length heuristic; this one looks for
words that signal reasoning. The mechanism matters less than the principle:
most traffic does not need the expensive model, and sending all of it there
is the single most common source of avoidable spend.
"""
lowered = prompt.lower()
if any(word in lowered for word in hard_words) or estimate_tokens(prompt) > 200:
return "large"
return "small"
def trim_context(chunks: list[str], budget_tokens: int) -> list[str]:
"""Keep the highest-priority chunks that fit a token budget.
Chunks arrive in priority order. Stopping at the first one that does not
fit -- rather than skipping it and trying the next -- keeps the context
contiguous in relevance, which matters more than packing the budget full.
"""
kept: list[str] = []
used = 0
for chunk in chunks:
size = estimate_tokens(chunk)
if used + size > budget_tokens:
break
kept.append(chunk)
used += size
return kept
@dataclass
class Pipeline:
"""A request pipeline with a cache, a router and a spend cap."""
budget: float
ledger: Ledger = field(default_factory=Ledger)
cache: dict[str, str] = field(default_factory=dict)
degrade_at: float = 0.8 # fraction of budget after which we downgrade
def spent(self) -> float:
return self.ledger.total
def remaining(self) -> float:
return max(0.0, self.budget - self.spent())
def answer(self, prompt: str, context: list[str] | None = None, *, context_budget: int = 400) -> str:
"""Handle one request, respecting the cache, the router and the cap."""
context = context or []
kept = trim_context(context, context_budget)
key = prompt + "||" + "|".join(kept)
if key in self.cache:
# A cache hit is recorded as a zero-cost call rather than skipped
# entirely, so the hit rate is visible in the ledger.
self.ledger.calls.append(
Call(model="cache", input_tokens=0, output_tokens=0, cost=0.0, cached=True)
)
return self.cache[key]
model = route(prompt)
# Degrade before refusing: past the threshold, downgrade rather than
# fail. A cheaper answer beats no answer for most workloads.
if model == "large" and self.spent() >= self.budget * self.degrade_at:
model = "small"
input_tokens = estimate_tokens(prompt) + sum(estimate_tokens(c) for c in kept)
output_tokens = 120
cost = cost_of(model, input_tokens, output_tokens)
# Check BEFORE spending. Estimating after the call is an audit, not a
# control.
if self.spent() + cost > self.budget:
raise BudgetExceeded(
f"request would cost ${cost:.4f}, only ${self.remaining():.4f} left"
)
answer = f"[{model}] answer to: {prompt[:40]}"
self.cache[key] = answer
self.ledger.calls.append(
Call(model=model, input_tokens=input_tokens, output_tokens=output_tokens, cost=cost)
)
return answer
def unit_economics(ledger: Ledger, requests: int) -> dict[str, float]:
"""Cost per request and projected cost at scale.
Totals are not decisions. Cost per request multiplied by expected volume is
what tells you whether a feature is viable, and it is the number that
should appear in a design document.
"""
if requests <= 0:
return {"per_request": 0.0, "per_1k_requests": 0.0, "per_million_requests": 0.0}
per = ledger.total / requests
return {
"per_request": per,
"per_1k_requests": per * 1_000,
"per_million_requests": per * 1_000_000,
}
metadata.yml (873 bytes)
lesson_id: D327
day: 327
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/costs_demo.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-30'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3 examples/costs_demo.py
reproduced the documented four-configuration comparison exactly; bash tests/run_tests.sh -> 20 passed
in 0.01s. The untouched starter fails 17 of 20; the 3 that pass check the price table and the token
estimator, neither of which is stubbed. requires_network is false and requires_api_key is false: the
model is simulated and prices are a constant table, so the lab costs nothing to run.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/costs.py (6088 bytes)
"""Cost accounting, caching, tiering and budget enforcement for an AI pipeline.
Offline and standard-library only. Prices are a fixed table rather than a live
API, and "tokens" are counted with a crude word-based estimator -- both are
simulations. What is real is the accounting: where cost accumulates, which
levers move it, and what happens when a budget runs out.
The four levers, in the order they usually pay off:
cache an exact repeat costs nothing
tier route easy work to a cheaper model
trim send less context
cap refuse, or degrade, rather than overspend
The last one is the one teams skip, and it is the only one that bounds the
worst case. Without a cap, a retry loop against a paid API is unbounded spend.
"""
from __future__ import annotations
from dataclasses import dataclass, field
# Illustrative prices in dollars per million tokens. The shape is what
# matters: output costs several times more than input, and the gap between
# tiers is an order of magnitude rather than a few percent.
PRICES: dict[str, tuple[float, float]] = {
"small": (0.25, 1.25),
"large": (3.00, 15.00),
}
class BudgetExceeded(RuntimeError):
"""Raised when a request would push spend past its cap."""
def estimate_tokens(text: str) -> int:
"""Crude token estimate: roughly four characters per token.
Real tokenisers are model-specific and this is not one. It is honest about
being an estimate, which is the right posture for a pre-flight cost check --
you cap on the estimate, then reconcile against the provider's reported
usage afterwards.
"""
return max(1, len(text) // 4)
def cost_of(model: str, input_tokens: int, output_tokens: int) -> float:
"""Dollar cost of one call, from the price table."""
# TASK 1: dollar cost of one call from PRICES.
# Prices are per MILLION tokens. Raise KeyError for an unknown model --
# pricing it at zero is how a new model silently escapes accounting.
raise NotImplementedError("implement cost_of")
@dataclass
class Call:
model: str
input_tokens: int
output_tokens: int
cost: float
cached: bool = False
@dataclass
class Ledger:
"""Every call, so cost can be attributed rather than merely totalled."""
calls: list[Call] = field(default_factory=list)
@property
def total(self) -> float:
return sum(c.cost for c in self.calls)
@property
def cache_hits(self) -> int:
return sum(1 for c in self.calls if c.cached)
def by_model(self) -> dict[str, float]:
out: dict[str, float] = {}
for call in self.calls:
out[call.model] = out.get(call.model, 0.0) + call.cost
return out
def summary(self) -> str:
return (
f"calls={len(self.calls)} cached={self.cache_hits} "
f"total=${self.total:.4f}"
)
def route(prompt: str, *, hard_words: tuple[str, ...] = ("analyse", "compare", "explain why")) -> str:
"""Pick the cheapest model that can plausibly do the job.
A real router uses a classifier or a length heuristic; this one looks for
words that signal reasoning. The mechanism matters less than the principle:
most traffic does not need the expensive model, and sending all of it there
is the single most common source of avoidable spend.
"""
# TASK 2: return "small" or "large".
# "large" when the lowered prompt contains any of hard_words, or when
# estimate_tokens(prompt) > 200. Otherwise "small".
raise NotImplementedError("implement route")
def trim_context(chunks: list[str], budget_tokens: int) -> list[str]:
"""Keep the highest-priority chunks that fit a token budget.
Chunks arrive in priority order. Stopping at the first one that does not
fit -- rather than skipping it and trying the next -- keeps the context
contiguous in relevance, which matters more than packing the budget full.
"""
# TASK 3: keep chunks in order while they fit budget_tokens.
# STOP at the first chunk that does not fit rather than skipping ahead to
# a smaller one -- contiguity in relevance beats filling the budget.
raise NotImplementedError("implement trim_context")
@dataclass
class Pipeline:
"""A request pipeline with a cache, a router and a spend cap."""
budget: float
ledger: Ledger = field(default_factory=Ledger)
cache: dict[str, str] = field(default_factory=dict)
degrade_at: float = 0.8 # fraction of budget after which we downgrade
def spent(self) -> float:
return self.ledger.total
def remaining(self) -> float:
return max(0.0, self.budget - self.spent())
def answer(self, prompt: str, context: list[str] | None = None, *, context_budget: int = 400) -> str:
# TASK 5: handle one request.
# - trim the context, then build a cache key from prompt AND the
# kept chunks; a hit appends a zero-cost Call(cached=True) so
# the hit rate stays visible in the ledger, and returns
# - route; if the route is large and spent() has passed
# budget * degrade_at, downgrade to small rather than failing
# - estimate input tokens (prompt + kept chunks), assume 120
# output tokens, and price the call
# - raise BudgetExceeded BEFORE recording anything if
# spent() + cost would exceed the budget
# - otherwise cache the answer, append the Call, and return it
raise NotImplementedError("implement Pipeline.answer")
def unit_economics(ledger: Ledger, requests: int) -> dict[str, float]:
"""Cost per request and projected cost at scale.
Totals are not decisions. Cost per request multiplied by expected volume is
what tells you whether a feature is viable, and it is the number that
should appear in a design document.
"""
# TASK 4: return per_request, per_1k_requests and per_million_requests.
# Zero requests is a normal state on a fresh ledger: return zeros rather
# than dividing.
raise NotImplementedError("implement unit_economics")
tests/run_tests.sh (206 bytes)
#!/usr/bin/env bash
# Run the day 327 cost engineering test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_costs.py (5360 bytes)
"""Grouped by lever, so a failure names which control broke.
Run with: bash tests/run_tests.sh
"""
from __future__ import annotations
import os
import sys
import pytest
HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))
from costs import ( # noqa: E402
PRICES,
BudgetExceeded,
Ledger,
Pipeline,
cost_of,
estimate_tokens,
route,
trim_context,
unit_economics,
)
# ---------------------------------------------------------------- accounting
def test_output_tokens_cost_more_than_input_tokens():
# True of every major provider, and the reason "just send more context" is
# cheaper than "just generate more".
for model, (inp, out) in PRICES.items():
assert out > inp, f"{model}: output should cost more than input"
def test_cost_scales_linearly_with_tokens():
single = cost_of("small", 1000, 0)
double = cost_of("small", 2000, 0)
assert abs(double - 2 * single) < 1e-12
def test_large_model_costs_an_order_of_magnitude_more():
small = cost_of("small", 1000, 1000)
large = cost_of("large", 1000, 1000)
assert large > small * 5
def test_unknown_model_raises_rather_than_costing_zero():
# Silently pricing an unknown model at zero is how spend goes unnoticed.
with pytest.raises(KeyError):
cost_of("gpt-imaginary", 10, 10)
def test_token_estimate_is_never_zero():
assert estimate_tokens("") >= 1
assert estimate_tokens("a") >= 1
# --------------------------------------------------------------------- cache
def test_repeat_request_is_free_and_recorded():
pipe = Pipeline(budget=1.0)
pipe.answer("What is the refund window?")
first = pipe.spent()
pipe.answer("What is the refund window?")
assert pipe.spent() == first, "a cache hit must not add cost"
assert pipe.ledger.cache_hits == 1
def test_cache_hits_appear_in_the_ledger_not_hidden():
# A hit is a zero-cost call rather than a skipped one, so the hit rate is
# visible when attributing spend.
pipe = Pipeline(budget=1.0)
pipe.answer("same")
pipe.answer("same")
assert len(pipe.ledger.calls) == 2
assert pipe.ledger.by_model()["cache"] == 0.0
def test_different_context_is_a_different_cache_key():
pipe = Pipeline(budget=1.0)
pipe.answer("question", ["context A"])
before = pipe.spent()
pipe.answer("question", ["context B"])
assert pipe.spent() > before
# -------------------------------------------------------------------- router
def test_reasoning_prompts_route_to_the_large_model():
assert route("Analyse why sales fell") == "large"
assert route("Compare the two plans") == "large"
def test_simple_prompts_route_to_the_small_model():
assert route("What is the refund window?") == "small"
assert route("List supported currencies") == "small"
def test_very_long_prompts_route_to_the_large_model():
assert route("word " * 500) == "large"
# ---------------------------------------------------------------------- trim
def test_trim_keeps_what_fits_in_priority_order():
chunks = ["a" * 400, "b" * 400, "c" * 400] # ~100 tokens each
kept = trim_context(chunks, 250)
assert kept == chunks[:2]
def test_trim_stops_rather_than_skipping_to_a_smaller_chunk():
# Contiguity in relevance beats packing the budget full.
chunks = ["a" * 4000, "b" * 40]
assert trim_context(chunks, 100) == []
def test_trim_with_a_generous_budget_keeps_everything():
chunks = ["a" * 40, "b" * 40]
assert trim_context(chunks, 10_000) == chunks
# -------------------------------------------------------------------- budget
def test_budget_is_checked_before_spending_not_after():
pipe = Pipeline(budget=0.0005)
with pytest.raises(BudgetExceeded):
for i in range(50):
pipe.answer(f"Analyse question number {i}")
assert pipe.spent() <= pipe.budget, "spend must never exceed the cap"
def test_degrades_to_the_cheap_model_before_refusing():
# Past the threshold a large-model request is downgraded rather than
# failed: a cheaper answer beats no answer for most workloads.
pipe = Pipeline(budget=0.01, degrade_at=0.0)
pipe.answer("Analyse the quarterly numbers")
assert pipe.ledger.calls[-1].model == "small"
def test_remaining_never_goes_negative():
pipe = Pipeline(budget=0.0)
assert pipe.remaining() == 0.0
# ----------------------------------------------------------- unit economics
def test_unit_economics_scales_from_a_single_request():
ledger = Ledger()
pipe = Pipeline(budget=1.0, ledger=ledger)
pipe.answer("What is the refund window?")
econ = unit_economics(ledger, 1)
assert econ["per_1k_requests"] == pytest.approx(econ["per_request"] * 1000)
assert econ["per_million_requests"] == pytest.approx(econ["per_request"] * 1_000_000)
def test_unit_economics_of_zero_requests_is_zero_not_an_error():
assert unit_economics(Ledger(), 0)["per_request"] == 0.0
def test_cost_attribution_splits_by_model():
pipe = Pipeline(budget=1.0)
pipe.answer("What is the refund window?") # small
pipe.answer("Analyse the refund spike") # large
by_model = pipe.ledger.by_model()
assert set(by_model) == {"small", "large"}
assert by_model["large"] > by_model["small"]
Troubleshooting
Troubleshooting — Day 327
Spend exceeds the budget
You are recording the call before checking the cap, or checking after the call returns. The check must be spent() + cost > budget before anything is appended to the ledger. Checking afterwards means the cap can be exceeded by the size of the last request — or without limit, if the caller retries.
The cache never hits
Your key does not include the trimmed context, or it includes something that varies between calls. Two identical prompts with identical context must produce the same key. Build it from the prompt plus the kept chunks, after trimming.
Cache hits do not appear in by model
You are returning early on a hit without appending a Call. Record the hit as a zero-cost call with cached=True instead — otherwise the hit rate is invisible in exactly the place you need it when attributing spend.
Everything routes to the large model
Your heuristic is matching too eagerly. Lower the prompt once and test each hard word as a substring; do not compare the whole prompt for equality, and check the length threshold is > 200 estimated tokens rather than characters.
trim_context returns more than the budget allows
You are adding the chunk and then checking, rather than checking and then adding. Compute used + size > budget_tokens first and break.
trim_context skips a large chunk and keeps a later small one
That is the packing behaviour, and this lab deliberately does not want it. Chunks arrive in priority order, so stopping at the first one that does not fit keeps the context contiguous in relevance. test_trim_stops_rather_than_skipping_to_a_smaller_chunk asserts this.
ZeroDivisionError in unit_economics
A fresh ledger with zero requests is a normal state, not an edge case. Return zeros.
cost_of returns 0.0 for a model you have not priced
You are using PRICES.get(model, (0, 0)). Raise KeyError instead — silently pricing an unknown model at zero is how a newly added model escapes accounting entirely, and the bill is the first anyone hears of it.
The demo numbers do not match
The workload and prices are fixed, so a correct implementation reproduces the table exactly. Differences usually come from the assumed output token count (120), the token estimator (four characters per token), or from trimming before rather than after building the cache key.
NotImplementedError on most tests
Expected. The starter stubs five functions — see expected-output/starter-run.txt, which also lists the three tests that pass without them.
Security notes
Security notes — Day 327
What this lab does and does not touch
- No network. The model is simulated and prices are a constant table. No test opens a socket.
- No credentials and no spend.
requires_api_keyisfalse. Running this lab costs nothing. - No writes outside this directory. Only
__pycache__/,.pytest_cache/and, if you create it,.venv/. - No
sudo, no global installs.
Unbounded spend is a security problem, not just a budgeting one
The spend cap in this lab looks like housekeeping. In production it is a control, and its absence is an exploitable weakness — OWASP tracks it as unbounded consumption in the LLM Top 10.
Three ways spend goes unbounded:
- A retry loop. A transient failure, an automatic retry, a failure again. Nothing in the loop knows what it has already cost.
- A recursive agent. A tool-using agent that can invoke itself, or two agents that call each other, has no natural termination.
- A hostile client. Anything a legitimate user can trigger, an attacker can trigger deliberately and repeatedly. This is a denial-of-wallet attack: the target is the budget rather than the CPU, and it is often cheaper to mount than a traditional denial of service.
Prompt injection makes the third worse: text inside a retrieved document can induce expensive behaviour, so the attacker does not even need to reach your endpoint directly.
The mitigations are per-request caps, per-user and per-tenant budgets, a maximum tool-call depth, and alerting on spend rate rather than only on the total. A cap that is checked before the call is the only one that bounds the worst case; checking afterwards is an audit.
Caching is user data
A response cache stores prompts and completions, which are user content. Two consequences:
- Key on tenant as well as content. A shared cache keyed only on prompt text serves one tenant's response to another when prompts collide, which is a data leak with a very ordinary-looking cause.
- Retention rules apply. Cached prompts and responses inherit whatever deletion and retention obligations the original data has. A right-to-erasure request that clears the database and leaves the cache has not been honoured.
Semantic caching, in the extension exercise, sharpens both problems: near-matches hit across users more readily than exact matches do.
Reversing everything this lab did
rm -rf __pycache__ .pytest_cache .venv
Nothing else was created, and nothing outside this directory was modified.