AI Engineering: Agents and Applications › Production Retrieval and Pipelines › Day 327

Hands-on lab — Day 327: Cost Engineering for AI Systems

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/costs_demo.py

Test

bash tests/run_tests.sh

File tree

examples/costs_demo.py
examples/costs.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/costs.py
tests/run_tests.sh
tests/test_costs.py
troubleshooting.md

Lab README

Lab — Day 327: Cost Engineering for AI Systems

Lesson

Purpose

Make the cost of an AI pipeline visible, then reduce it deliberately. You build a ledger that attributes spend per model, a router that sends easy work to a cheaper model, a context trimmer, and a spend cap that degrades before it refuses — then measure what each lever actually saves on the same workload.

Everything runs offline. Prices are a fixed table and tokens are estimated crudely; the accounting is real.

Learning objectives

  • Attribute spend per model rather than reporting a single total.
  • Demonstrate that cache hit rate and cost saving are different measurements.
  • Route requests to the cheapest adequate model and trim context to a token budget.
  • Enforce a cap checked before the call, with degradation before refusal.
  • Project cost per request to expected volume.

Prerequisites

  • Day 326, "Scaling Retrieval".
  • Comfortable with Python dataclasses and dictionaries.
  • No API key and no account — the model is simulated.

Supported operating systems

macOS, Linux and Windows (via WSL or PowerShell). Nothing is platform-specific.

Hardware requirements

Any machine running Python 3.10 or newer. No GPU. The suite completes in hundredths of a second.

Required software

  • Python 3.10 or newer.
  • pytest 7 or newer.

Free and open-source options

Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no API key, no spend — the price table is a constant in the source. The lesson discusses LiteLLM, Langfuse and Helicone as the real open-source options for budgets and cost observability.

Installation

python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt

Removing .venv fully reverses the installation.

File structure

starter/costs.py         your work: cost_of, route, trim_context,
                         unit_economics, Pipeline.answer
examples/costs.py        reference implementation
examples/costs_demo.py   runs one workload four ways
tests/test_costs.py      grouped by lever
tests/run_tests.sh       suite entry point
expected-output/         real captured output and measured values
requirements/            pinned dependency

How to run

python3 examples/costs_demo.py   # compare the levers
bash tests/run_tests.sh          # run the suite

To work on the exercise, edit starter/costs.py, then copy it over the reference:

cp starter/costs.py examples/costs.py
bash tests/run_tests.sh

Keep a copy of the reference first if you want to compare.

What the commands do

python3 examples/costs_demo.py runs the same six requests four ways — no cache, cached, cached with trimmed context, and under a budget too small to complete — printing total spend, spend by model, and cost per request for each, then projecting to a million requests.

bash tests/run_tests.sh runs pytest over twenty tests grouped by lever: accounting, cache, router, trim, budget and unit economics.

Expected output

workload: 6 requests, 3 of them the same question
no cache, full ctx     calls=6 cached=0 total=$0.0085  per_request=$0.00141
                       by model: large=$0.0073, small=$0.0012
cache, full ctx        calls=6 cached=2 total=$0.0079  per_request=$0.00131
                       by model: cache=$0.0000, large=$0.0073, small=$0.0006
cache, trimmed ctx     calls=6 cached=2 total=$0.0066  per_request=$0.00109
                       by model: cache=$0.0000, large=$0.0061, small=$0.0005
tight budget           STOPPED: request would cost $0.0036, only $0.0009 left
tight budget           calls=2 cached=0 total=$0.0006  per_request=$0.00030
                       by model: small=$0.0006
projection at 1M requests: $1,092 (from $0.00109 per request)

The second row is the one to study: a 33 percent cache hit rate produced a 7 percent saving, because the repeated question routed to the cheap model. expected-output/FIELDS.md works through the attribution.

Validation steps

  1. bash tests/run_tests.sh reports 20 passed.
  2. Run the demo and confirm the tight budget row stops rather than completing, and that its total is at or below the budget — never above.
  3. Check the by model line includes cache=$0.0000. If cache hits are absent from the ledger rather than present at zero cost, your hit rate is invisible.

Tests

Twenty tests in six groups:

  • accounting — output costs more than input, cost is linear in tokens, the large model is an order of magnitude dearer, an unknown model raises rather than costing zero, and a token estimate is never zero.
  • cache — a repeat is free, hits appear in the ledger, and different context is a different key.
  • router — reasoning and long prompts go large, simple ones go small.
  • trim — keeps what fits in priority order and stops rather than skipping ahead.
  • budget — checked before spending, degrades before refusing, and remaining never goes negative.
  • unit economics — scales linearly, handles zero requests, and attributes by model.

Cleanup

rm -rf __pycache__ .pytest_cache
rm -rf .venv

Both are safe to remove and fully reversible.

Troubleshooting

See troubleshooting.md for each failure this lab is designed to produce.

Security notes

See security.md. In short: no network, no credentials, no spend. The lesson treats unbounded spend as a security concern — a denial-of-wallet attack — rather than only a budgeting one.

Extension exercises

  1. Per-tenant budgets. Enforce a per-tenant cap alongside the global one, and prove both that one tenant cannot exhaust another's budget and that per-tenant limits do not sum to a safe global limit.
  2. Semantic caching. Cache on similarity rather than exact match, and measure the wrong answers served at each threshold alongside the savings.
  3. Self-hosted crossover. Add a fixed hourly cost for a self-hosted model and compute the request volume at which it beats per-token pricing.
  • Lesson
  • Previous: Day 326 — Scaling Retrieval
  • Next: Day 328 — Privacy in AI Systems

Expected output

FIELDS.md

# Reading the cost output

## The per-configuration lines

    cache, trimmed ctx     calls=6 cached=2 total=$0.0066  per_request=$0.00109
                           by model: cache=$0.0000, large=$0.0061, small=$0.0005

| Field | Meaning |
| --- | --- |
| `calls` | Entries in the ledger, including cache hits. |
| `cached` | How many of those were hits. Recorded as zero-cost calls rather than skipped, so the hit rate stays visible. |
| `total` | Dollar spend for the workload. |
| `per_request` | Total divided by calls — the number that gets multiplied by volume. |
| `by model` | Attribution. This line is the whole point: a total says what you spent, attribution says where to act. |

## Why `by model` matters more than `total`

Compare the baseline and the cached run:

    no cache    large=$0.0073, small=$0.0012     total $0.0085
    + cache     large=$0.0073, small=$0.0006     total $0.0079

The cache changed only the small-model column. The large model — 86 percent of spend from two of six requests — was untouched, because the repeated question was a cheap one.

A dashboard reporting "33% cache hit rate" would call this a success. The attribution shows it saved 7 percent.

## The budget line

    tight budget           STOPPED: request would cost $0.0036, only $0.0009 left

The cap is checked **before** the call. Spend stops at or below the budget rather than overshooting by the size of the last request. Note also that the pipeline degrades — downgrading large-model requests to the small model — before it refuses outright.

## The projection

    projection at 1M requests: $1,092 (from $0.00109 per request)

Cost per request multiplied by expected volume. This belongs in a design document before the feature ships, not in a retrospective after the invoice.

examples-run.txt

workload: 6 requests, 3 of them the same question
no cache, full ctx     calls=6 cached=0 total=$0.0085  per_request=$0.00141
                       by model: large=$0.0073, small=$0.0012
cache, full ctx        calls=6 cached=2 total=$0.0079  per_request=$0.00131
                       by model: cache=$0.0000, large=$0.0073, small=$0.0006
cache, trimmed ctx     calls=6 cached=2 total=$0.0066  per_request=$0.00109
                       by model: cache=$0.0000, large=$0.0061, small=$0.0005
tight budget           STOPPED: request would cost $0.0036, only $0.0009 left
tight budget           calls=2 cached=0 total=$0.0006  per_request=$0.00030
                       by model: small=$0.0006
projection at 1M requests: $1,092 (from $0.00109 per request)

measured-values.txt

Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.

  python3 examples/costs_demo.py -> 9 lines, exit 0
  bash tests/run_tests.sh        -> 20 passed in 0.01s

The four configurations, same six-request workload:

  configuration        total     per request   note
  no cache, full ctx   $0.0085   $0.00141      baseline
  cache, full ctx      $0.0079   $0.00131      2 hits, saved 7%
  cache, trimmed ctx   $0.0066   $0.00109      saved 22% against baseline
  tight budget         stopped   --            refused before overspending

The number worth staring at is the second row. The cache hit 2 of 6 requests --
a 33% hit rate -- and saved 7% of the cost, because the repeated question
routed to the SMALL model. Spend by model on the baseline:

  large  $0.0073   (86% of spend, from 2 of 6 requests)
  small  $0.0012

Hit rate is not savings. Trimming context saved more precisely because it acted
on the large-model calls.

Projection: $1,092 per million requests, from $0.00109 per request.

run-output.txt

workload: 6 requests, 3 of them the same question
no cache, full ctx     calls=6 cached=0 total=$0.0085  per_request=$0.00141
                       by model: large=$0.0073, small=$0.0012
cache, full ctx        calls=6 cached=2 total=$0.0079  per_request=$0.00131
                       by model: cache=$0.0000, large=$0.0073, small=$0.0006
cache, trimmed ctx     calls=6 cached=2 total=$0.0066  per_request=$0.00109
                       by model: cache=$0.0000, large=$0.0061, small=$0.0005
tight budget           STOPPED: request would cost $0.0036, only $0.0009 left
tight budget           calls=2 cached=0 total=$0.0006  per_request=$0.00030
                       by model: small=$0.0006
projection at 1M requests: $1,092 (from $0.00109 per request)

starter-run.txt

An untouched starter fails most of the suite:

    17 failed, 3 passed in 0.05s

The three that pass check the PRICES table and estimate_tokens, neither of
which is stubbed:

    test_output_tokens_cost_more_than_input_tokens
    test_large_model_costs_an_order_of_magnitude_more   (partially)
    test_token_estimate_is_never_zero

Everything else depends on cost_of, route, trim_context, unit_economics or
Pipeline.answer, all of which raise NotImplementedError.

test-output.txt

....................                                                     [100%]
20 passed in 0.01s

test-run.txt

....................                                                     [100%]
20 passed in 0.01s

Source files

examples/costs_demo.py (2141 bytes)
#!/usr/bin/env python3
"""Run a small workload four ways and compare what each lever saves."""

from __future__ import annotations

import os
import sys

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

from costs import BudgetExceeded, Pipeline, unit_economics

CONTEXT = [f"retrieved passage {i} " * 20 for i in range(6)]

PROMPTS = [
    "What is the refund window?",
    "What is the refund window?",          # exact repeat -> cache
    "Analyse why refunds spiked in March",  # reasoning -> large model
    "List the supported currencies",
    "What is the refund window?",          # repeat again
    "Compare the standard and premium tiers",
]


def run(label: str, *, budget: float, context_budget: int, use_cache: bool) -> None:
    pipe = Pipeline(budget=budget)
    for prompt in PROMPTS:
        if not use_cache:
            pipe.cache.clear()
        try:
            pipe.answer(prompt, CONTEXT, context_budget=context_budget)
        except BudgetExceeded as exc:
            print(f"{label:<22} STOPPED: {exc}")
            break
    econ = unit_economics(pipe.ledger, len(pipe.ledger.calls))
    models = ", ".join(f"{m}=${c:.4f}" for m, c in sorted(pipe.ledger.by_model().items()))
    print(f"{label:<22} {pipe.ledger.summary()}  per_request=${econ['per_request']:.5f}")
    print(f"{'':<22} by model: {models}")


def main() -> int:
    print("workload: 6 requests, 3 of them the same question")
    run("no cache, full ctx", budget=1.0, context_budget=10_000, use_cache=False)
    run("cache, full ctx", budget=1.0, context_budget=10_000, use_cache=True)
    run("cache, trimmed ctx", budget=1.0, context_budget=400, use_cache=True)
    run("tight budget", budget=0.0015, context_budget=10_000, use_cache=False)

    pipe = Pipeline(budget=1.0)
    for prompt in PROMPTS:
        pipe.answer(prompt, CONTEXT, context_budget=400)
    econ = unit_economics(pipe.ledger, len(PROMPTS))
    print(
        f"projection at 1M requests: ${econ['per_million_requests']:,.0f} "
        f"(from ${econ['per_request']:.5f} per request)"
    )
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
examples/costs.py (6762 bytes)
"""Cost accounting, caching, tiering and budget enforcement for an AI pipeline.

Offline and standard-library only. Prices are a fixed table rather than a live
API, and "tokens" are counted with a crude word-based estimator -- both are
simulations. What is real is the accounting: where cost accumulates, which
levers move it, and what happens when a budget runs out.

The four levers, in the order they usually pay off:

  cache     an exact repeat costs nothing
  tier      route easy work to a cheaper model
  trim      send less context
  cap       refuse, or degrade, rather than overspend

The last one is the one teams skip, and it is the only one that bounds the
worst case. Without a cap, a retry loop against a paid API is unbounded spend.
"""

from __future__ import annotations

from dataclasses import dataclass, field

# Illustrative prices in dollars per million tokens. The shape is what
# matters: output costs several times more than input, and the gap between
# tiers is an order of magnitude rather than a few percent.
PRICES: dict[str, tuple[float, float]] = {
    "small": (0.25, 1.25),
    "large": (3.00, 15.00),
}


class BudgetExceeded(RuntimeError):
    """Raised when a request would push spend past its cap."""


def estimate_tokens(text: str) -> int:
    """Crude token estimate: roughly four characters per token.

    Real tokenisers are model-specific and this is not one. It is honest about
    being an estimate, which is the right posture for a pre-flight cost check --
    you cap on the estimate, then reconcile against the provider's reported
    usage afterwards.
    """
    return max(1, len(text) // 4)


def cost_of(model: str, input_tokens: int, output_tokens: int) -> float:
    """Dollar cost of one call, from the price table."""
    if model not in PRICES:
        raise KeyError(f"no price for model '{model}'")
    in_price, out_price = PRICES[model]
    return (input_tokens * in_price + output_tokens * out_price) / 1_000_000


@dataclass
class Call:
    model: str
    input_tokens: int
    output_tokens: int
    cost: float
    cached: bool = False


@dataclass
class Ledger:
    """Every call, so cost can be attributed rather than merely totalled."""

    calls: list[Call] = field(default_factory=list)

    @property
    def total(self) -> float:
        return sum(c.cost for c in self.calls)

    @property
    def cache_hits(self) -> int:
        return sum(1 for c in self.calls if c.cached)

    def by_model(self) -> dict[str, float]:
        out: dict[str, float] = {}
        for call in self.calls:
            out[call.model] = out.get(call.model, 0.0) + call.cost
        return out

    def summary(self) -> str:
        return (
            f"calls={len(self.calls)} cached={self.cache_hits} "
            f"total=${self.total:.4f}"
        )


def route(prompt: str, *, hard_words: tuple[str, ...] = ("analyse", "compare", "explain why")) -> str:
    """Pick the cheapest model that can plausibly do the job.

    A real router uses a classifier or a length heuristic; this one looks for
    words that signal reasoning. The mechanism matters less than the principle:
    most traffic does not need the expensive model, and sending all of it there
    is the single most common source of avoidable spend.
    """
    lowered = prompt.lower()
    if any(word in lowered for word in hard_words) or estimate_tokens(prompt) > 200:
        return "large"
    return "small"


def trim_context(chunks: list[str], budget_tokens: int) -> list[str]:
    """Keep the highest-priority chunks that fit a token budget.

    Chunks arrive in priority order. Stopping at the first one that does not
    fit -- rather than skipping it and trying the next -- keeps the context
    contiguous in relevance, which matters more than packing the budget full.
    """
    kept: list[str] = []
    used = 0
    for chunk in chunks:
        size = estimate_tokens(chunk)
        if used + size > budget_tokens:
            break
        kept.append(chunk)
        used += size
    return kept


@dataclass
class Pipeline:
    """A request pipeline with a cache, a router and a spend cap."""

    budget: float
    ledger: Ledger = field(default_factory=Ledger)
    cache: dict[str, str] = field(default_factory=dict)
    degrade_at: float = 0.8  # fraction of budget after which we downgrade

    def spent(self) -> float:
        return self.ledger.total

    def remaining(self) -> float:
        return max(0.0, self.budget - self.spent())

    def answer(self, prompt: str, context: list[str] | None = None, *, context_budget: int = 400) -> str:
        """Handle one request, respecting the cache, the router and the cap."""
        context = context or []
        kept = trim_context(context, context_budget)
        key = prompt + "||" + "|".join(kept)

        if key in self.cache:
            # A cache hit is recorded as a zero-cost call rather than skipped
            # entirely, so the hit rate is visible in the ledger.
            self.ledger.calls.append(
                Call(model="cache", input_tokens=0, output_tokens=0, cost=0.0, cached=True)
            )
            return self.cache[key]

        model = route(prompt)
        # Degrade before refusing: past the threshold, downgrade rather than
        # fail. A cheaper answer beats no answer for most workloads.
        if model == "large" and self.spent() >= self.budget * self.degrade_at:
            model = "small"

        input_tokens = estimate_tokens(prompt) + sum(estimate_tokens(c) for c in kept)
        output_tokens = 120
        cost = cost_of(model, input_tokens, output_tokens)

        # Check BEFORE spending. Estimating after the call is an audit, not a
        # control.
        if self.spent() + cost > self.budget:
            raise BudgetExceeded(
                f"request would cost ${cost:.4f}, only ${self.remaining():.4f} left"
            )

        answer = f"[{model}] answer to: {prompt[:40]}"
        self.cache[key] = answer
        self.ledger.calls.append(
            Call(model=model, input_tokens=input_tokens, output_tokens=output_tokens, cost=cost)
        )
        return answer


def unit_economics(ledger: Ledger, requests: int) -> dict[str, float]:
    """Cost per request and projected cost at scale.

    Totals are not decisions. Cost per request multiplied by expected volume is
    what tells you whether a feature is viable, and it is the number that
    should appear in a design document.
    """
    if requests <= 0:
        return {"per_request": 0.0, "per_1k_requests": 0.0, "per_million_requests": 0.0}
    per = ledger.total / requests
    return {
        "per_request": per,
        "per_1k_requests": per * 1_000,
        "per_million_requests": per * 1_000_000,
    }
metadata.yml (873 bytes)
lesson_id: D327
day: 327
kind: python
languages:
  - python
setup_commands:
  - pip install -r requirements/requirements.txt
run_commands:
  - python examples/costs_demo.py
test_commands:
  - bash tests/run_tests.sh
cleanup_commands:
  - rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-30'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3 examples/costs_demo.py
  reproduced the documented four-configuration comparison exactly; bash tests/run_tests.sh -> 20 passed
  in 0.01s. The untouched starter fails 17 of 20; the 3 that pass check the price table and the token
  estimator, neither of which is stubbed. requires_network is false and requires_api_key is false: the
  model is simulated and prices are a constant table, so the lab costs nothing to run.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/costs.py (6088 bytes)
"""Cost accounting, caching, tiering and budget enforcement for an AI pipeline.

Offline and standard-library only. Prices are a fixed table rather than a live
API, and "tokens" are counted with a crude word-based estimator -- both are
simulations. What is real is the accounting: where cost accumulates, which
levers move it, and what happens when a budget runs out.

The four levers, in the order they usually pay off:

  cache     an exact repeat costs nothing
  tier      route easy work to a cheaper model
  trim      send less context
  cap       refuse, or degrade, rather than overspend

The last one is the one teams skip, and it is the only one that bounds the
worst case. Without a cap, a retry loop against a paid API is unbounded spend.
"""

from __future__ import annotations

from dataclasses import dataclass, field

# Illustrative prices in dollars per million tokens. The shape is what
# matters: output costs several times more than input, and the gap between
# tiers is an order of magnitude rather than a few percent.
PRICES: dict[str, tuple[float, float]] = {
    "small": (0.25, 1.25),
    "large": (3.00, 15.00),
}


class BudgetExceeded(RuntimeError):
    """Raised when a request would push spend past its cap."""


def estimate_tokens(text: str) -> int:
    """Crude token estimate: roughly four characters per token.

    Real tokenisers are model-specific and this is not one. It is honest about
    being an estimate, which is the right posture for a pre-flight cost check --
    you cap on the estimate, then reconcile against the provider's reported
    usage afterwards.
    """
    return max(1, len(text) // 4)


def cost_of(model: str, input_tokens: int, output_tokens: int) -> float:
    """Dollar cost of one call, from the price table."""
    # TASK 1: dollar cost of one call from PRICES.
    # Prices are per MILLION tokens. Raise KeyError for an unknown model --
    # pricing it at zero is how a new model silently escapes accounting.
    raise NotImplementedError("implement cost_of")

@dataclass
class Call:
    model: str
    input_tokens: int
    output_tokens: int
    cost: float
    cached: bool = False


@dataclass
class Ledger:
    """Every call, so cost can be attributed rather than merely totalled."""

    calls: list[Call] = field(default_factory=list)

    @property
    def total(self) -> float:
        return sum(c.cost for c in self.calls)

    @property
    def cache_hits(self) -> int:
        return sum(1 for c in self.calls if c.cached)

    def by_model(self) -> dict[str, float]:
        out: dict[str, float] = {}
        for call in self.calls:
            out[call.model] = out.get(call.model, 0.0) + call.cost
        return out

    def summary(self) -> str:
        return (
            f"calls={len(self.calls)} cached={self.cache_hits} "
            f"total=${self.total:.4f}"
        )


def route(prompt: str, *, hard_words: tuple[str, ...] = ("analyse", "compare", "explain why")) -> str:
    """Pick the cheapest model that can plausibly do the job.

    A real router uses a classifier or a length heuristic; this one looks for
    words that signal reasoning. The mechanism matters less than the principle:
    most traffic does not need the expensive model, and sending all of it there
    is the single most common source of avoidable spend.
    """
    # TASK 2: return "small" or "large".
    # "large" when the lowered prompt contains any of hard_words, or when
    # estimate_tokens(prompt) > 200. Otherwise "small".
    raise NotImplementedError("implement route")

def trim_context(chunks: list[str], budget_tokens: int) -> list[str]:
    """Keep the highest-priority chunks that fit a token budget.

    Chunks arrive in priority order. Stopping at the first one that does not
    fit -- rather than skipping it and trying the next -- keeps the context
    contiguous in relevance, which matters more than packing the budget full.
    """
    # TASK 3: keep chunks in order while they fit budget_tokens.
    # STOP at the first chunk that does not fit rather than skipping ahead to
    # a smaller one -- contiguity in relevance beats filling the budget.
    raise NotImplementedError("implement trim_context")

@dataclass
class Pipeline:
    """A request pipeline with a cache, a router and a spend cap."""

    budget: float
    ledger: Ledger = field(default_factory=Ledger)
    cache: dict[str, str] = field(default_factory=dict)
    degrade_at: float = 0.8  # fraction of budget after which we downgrade

    def spent(self) -> float:
        return self.ledger.total

    def remaining(self) -> float:
        return max(0.0, self.budget - self.spent())

    def answer(self, prompt: str, context: list[str] | None = None, *, context_budget: int = 400) -> str:
        # TASK 5: handle one request.
        #   - trim the context, then build a cache key from prompt AND the
        #     kept chunks; a hit appends a zero-cost Call(cached=True) so
        #     the hit rate stays visible in the ledger, and returns
        #   - route; if the route is large and spent() has passed
        #     budget * degrade_at, downgrade to small rather than failing
        #   - estimate input tokens (prompt + kept chunks), assume 120
        #     output tokens, and price the call
        #   - raise BudgetExceeded BEFORE recording anything if
        #     spent() + cost would exceed the budget
        #   - otherwise cache the answer, append the Call, and return it
        raise NotImplementedError("implement Pipeline.answer")


def unit_economics(ledger: Ledger, requests: int) -> dict[str, float]:
    """Cost per request and projected cost at scale.

    Totals are not decisions. Cost per request multiplied by expected volume is
    what tells you whether a feature is viable, and it is the number that
    should appear in a design document.
    """
    # TASK 4: return per_request, per_1k_requests and per_million_requests.
    # Zero requests is a normal state on a fresh ledger: return zeros rather
    # than dividing.
    raise NotImplementedError("implement unit_economics")
tests/run_tests.sh (206 bytes)
#!/usr/bin/env bash
# Run the day 327 cost engineering test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_costs.py (5360 bytes)
"""Grouped by lever, so a failure names which control broke.

Run with: bash tests/run_tests.sh
"""

from __future__ import annotations

import os
import sys

import pytest

HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))

from costs import (  # noqa: E402
    PRICES,
    BudgetExceeded,
    Ledger,
    Pipeline,
    cost_of,
    estimate_tokens,
    route,
    trim_context,
    unit_economics,
)


# ---------------------------------------------------------------- accounting


def test_output_tokens_cost_more_than_input_tokens():
    # True of every major provider, and the reason "just send more context" is
    # cheaper than "just generate more".
    for model, (inp, out) in PRICES.items():
        assert out > inp, f"{model}: output should cost more than input"


def test_cost_scales_linearly_with_tokens():
    single = cost_of("small", 1000, 0)
    double = cost_of("small", 2000, 0)
    assert abs(double - 2 * single) < 1e-12


def test_large_model_costs_an_order_of_magnitude_more():
    small = cost_of("small", 1000, 1000)
    large = cost_of("large", 1000, 1000)
    assert large > small * 5


def test_unknown_model_raises_rather_than_costing_zero():
    # Silently pricing an unknown model at zero is how spend goes unnoticed.
    with pytest.raises(KeyError):
        cost_of("gpt-imaginary", 10, 10)


def test_token_estimate_is_never_zero():
    assert estimate_tokens("") >= 1
    assert estimate_tokens("a") >= 1


# --------------------------------------------------------------------- cache


def test_repeat_request_is_free_and_recorded():
    pipe = Pipeline(budget=1.0)
    pipe.answer("What is the refund window?")
    first = pipe.spent()

    pipe.answer("What is the refund window?")
    assert pipe.spent() == first, "a cache hit must not add cost"
    assert pipe.ledger.cache_hits == 1


def test_cache_hits_appear_in_the_ledger_not_hidden():
    # A hit is a zero-cost call rather than a skipped one, so the hit rate is
    # visible when attributing spend.
    pipe = Pipeline(budget=1.0)
    pipe.answer("same")
    pipe.answer("same")
    assert len(pipe.ledger.calls) == 2
    assert pipe.ledger.by_model()["cache"] == 0.0


def test_different_context_is_a_different_cache_key():
    pipe = Pipeline(budget=1.0)
    pipe.answer("question", ["context A"])
    before = pipe.spent()
    pipe.answer("question", ["context B"])
    assert pipe.spent() > before


# -------------------------------------------------------------------- router


def test_reasoning_prompts_route_to_the_large_model():
    assert route("Analyse why sales fell") == "large"
    assert route("Compare the two plans") == "large"


def test_simple_prompts_route_to_the_small_model():
    assert route("What is the refund window?") == "small"
    assert route("List supported currencies") == "small"


def test_very_long_prompts_route_to_the_large_model():
    assert route("word " * 500) == "large"


# ---------------------------------------------------------------------- trim


def test_trim_keeps_what_fits_in_priority_order():
    chunks = ["a" * 400, "b" * 400, "c" * 400]  # ~100 tokens each
    kept = trim_context(chunks, 250)
    assert kept == chunks[:2]


def test_trim_stops_rather_than_skipping_to_a_smaller_chunk():
    # Contiguity in relevance beats packing the budget full.
    chunks = ["a" * 4000, "b" * 40]
    assert trim_context(chunks, 100) == []


def test_trim_with_a_generous_budget_keeps_everything():
    chunks = ["a" * 40, "b" * 40]
    assert trim_context(chunks, 10_000) == chunks


# -------------------------------------------------------------------- budget


def test_budget_is_checked_before_spending_not_after():
    pipe = Pipeline(budget=0.0005)
    with pytest.raises(BudgetExceeded):
        for i in range(50):
            pipe.answer(f"Analyse question number {i}")
    assert pipe.spent() <= pipe.budget, "spend must never exceed the cap"


def test_degrades_to_the_cheap_model_before_refusing():
    # Past the threshold a large-model request is downgraded rather than
    # failed: a cheaper answer beats no answer for most workloads.
    pipe = Pipeline(budget=0.01, degrade_at=0.0)
    pipe.answer("Analyse the quarterly numbers")
    assert pipe.ledger.calls[-1].model == "small"


def test_remaining_never_goes_negative():
    pipe = Pipeline(budget=0.0)
    assert pipe.remaining() == 0.0


# ----------------------------------------------------------- unit economics


def test_unit_economics_scales_from_a_single_request():
    ledger = Ledger()
    pipe = Pipeline(budget=1.0, ledger=ledger)
    pipe.answer("What is the refund window?")

    econ = unit_economics(ledger, 1)
    assert econ["per_1k_requests"] == pytest.approx(econ["per_request"] * 1000)
    assert econ["per_million_requests"] == pytest.approx(econ["per_request"] * 1_000_000)


def test_unit_economics_of_zero_requests_is_zero_not_an_error():
    assert unit_economics(Ledger(), 0)["per_request"] == 0.0


def test_cost_attribution_splits_by_model():
    pipe = Pipeline(budget=1.0)
    pipe.answer("What is the refund window?")       # small
    pipe.answer("Analyse the refund spike")          # large
    by_model = pipe.ledger.by_model()
    assert set(by_model) == {"small", "large"}
    assert by_model["large"] > by_model["small"]

Troubleshooting

Troubleshooting — Day 327

Spend exceeds the budget

You are recording the call before checking the cap, or checking after the call returns. The check must be spent() + cost > budget before anything is appended to the ledger. Checking afterwards means the cap can be exceeded by the size of the last request — or without limit, if the caller retries.

The cache never hits

Your key does not include the trimmed context, or it includes something that varies between calls. Two identical prompts with identical context must produce the same key. Build it from the prompt plus the kept chunks, after trimming.

Cache hits do not appear in by model

You are returning early on a hit without appending a Call. Record the hit as a zero-cost call with cached=True instead — otherwise the hit rate is invisible in exactly the place you need it when attributing spend.

Everything routes to the large model

Your heuristic is matching too eagerly. Lower the prompt once and test each hard word as a substring; do not compare the whole prompt for equality, and check the length threshold is > 200 estimated tokens rather than characters.

trim_context returns more than the budget allows

You are adding the chunk and then checking, rather than checking and then adding. Compute used + size > budget_tokens first and break.

trim_context skips a large chunk and keeps a later small one

That is the packing behaviour, and this lab deliberately does not want it. Chunks arrive in priority order, so stopping at the first one that does not fit keeps the context contiguous in relevance. test_trim_stops_rather_than_skipping_to_a_smaller_chunk asserts this.

ZeroDivisionError in unit_economics

A fresh ledger with zero requests is a normal state, not an edge case. Return zeros.

cost_of returns 0.0 for a model you have not priced

You are using PRICES.get(model, (0, 0)). Raise KeyError instead — silently pricing an unknown model at zero is how a newly added model escapes accounting entirely, and the bill is the first anyone hears of it.

The demo numbers do not match

The workload and prices are fixed, so a correct implementation reproduces the table exactly. Differences usually come from the assumed output token count (120), the token estimator (four characters per token), or from trimming before rather than after building the cache key.

NotImplementedError on most tests

Expected. The starter stubs five functions — see expected-output/starter-run.txt, which also lists the three tests that pass without them.

Security notes

Security notes — Day 327

What this lab does and does not touch

  • No network. The model is simulated and prices are a constant table. No test opens a socket.
  • No credentials and no spend. requires_api_key is false. Running this lab costs nothing.
  • No writes outside this directory. Only __pycache__/, .pytest_cache/ and, if you create it, .venv/.
  • No sudo, no global installs.

Unbounded spend is a security problem, not just a budgeting one

The spend cap in this lab looks like housekeeping. In production it is a control, and its absence is an exploitable weakness — OWASP tracks it as unbounded consumption in the LLM Top 10.

Three ways spend goes unbounded:

  • A retry loop. A transient failure, an automatic retry, a failure again. Nothing in the loop knows what it has already cost.
  • A recursive agent. A tool-using agent that can invoke itself, or two agents that call each other, has no natural termination.
  • A hostile client. Anything a legitimate user can trigger, an attacker can trigger deliberately and repeatedly. This is a denial-of-wallet attack: the target is the budget rather than the CPU, and it is often cheaper to mount than a traditional denial of service.

Prompt injection makes the third worse: text inside a retrieved document can induce expensive behaviour, so the attacker does not even need to reach your endpoint directly.

The mitigations are per-request caps, per-user and per-tenant budgets, a maximum tool-call depth, and alerting on spend rate rather than only on the total. A cap that is checked before the call is the only one that bounds the worst case; checking afterwards is an audit.

Caching is user data

A response cache stores prompts and completions, which are user content. Two consequences:

  • Key on tenant as well as content. A shared cache keyed only on prompt text serves one tenant's response to another when prompts collide, which is a data leak with a very ordinary-looking cause.
  • Retention rules apply. Cached prompts and responses inherit whatever deletion and retention obligations the original data has. A right-to-erasure request that clears the database and leaves the cache has not been honoured.

Semantic caching, in the extension exercise, sharpens both problems: near-matches hit across users more readily than exact matches do.

Reversing everything this lab did

rm -rf __pycache__ .pytest_cache .venv

Nothing else was created, and nothing outside this directory was modified.