AI Engineering: Agents and Applications › Full-Stack AI Applications › Day 316

Hands-on lab — Day 316: Architecture of an AI Product

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/product_dispatcher.py

Test

bash tests/run_tests.sh

File tree

examples/product_dispatcher.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/product_dispatcher.py
tests/run_tests.sh
tests/test_product_dispatcher.py
troubleshooting.md

Lab README

Day 316 Lab: Architecture of an AI Product

Lesson

Purpose

Build a Full-Stack AI Product Architecture Dispatcher in Python that integrates authentication, token-bucket rate limiting, credit quota gating, semantic caching, and streaming token delivery.

Learning objectives

  • Implement multi-tenant authentication and quota verification.
  • Enforce token-bucket rate limits per organization.
  • Manage response caching for identical queries.
  • Execute simulated token streaming with accurate credit deductions.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library, Pytest

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/product_dispatcher.py: Starter implementation skeleton
  • examples/product_dispatcher.py: Verified reference implementation
  • tests/test_product_dispatcher.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/product_dispatcher.py

What the commands do

  • Executes request dispatch across authentication, rate limiting, caching, and generation.

Expected output

Org Alpha: {'status': 'SUCCESS', 'tokens_streamed': ['This', ' is', ' a', ' streamed', ' AI', ' response.'], 'response': 'This is a streamed AI response.', 'cost_usd': 0.002, 'cached': False, 'remaining_credits': 49.998}

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Valid tenant authentication and request dispatch
  • Rejection on invalid tenant ID
  • Rejection on rate limit exhaustion
  • Rejection on zero credit balance
  • Zero-cost delivery on cached query hits

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Ensure credit balance deductions occur only on successful model executions.

Security notes

Runs completely offline on local CPU.

Extension exercises

Add multi-provider fallback routing when primary mock fails.

Day number: 316 of 365

Expected output

FIELDS.md

# Measured Values
- `dispatched_requests`: 10
- `cached_requests`: 1

examples-run.txt

Org Alpha: {'status': 'SUCCESS', 'tokens_streamed': ['This', ' is', ' a', ' streamed', ' AI', ' response.'], 'response': 'This is a streamed AI response.', 'cost_usd': 0.002, 'cached': False, 'remaining_credits': 49.998}

measured-values.txt

dispatched_requests=10
cached_requests=1

run-output.txt

Org Alpha: {'status': 'SUCCESS', 'tokens_streamed': ['This', ' is', ' a', ' streamed', ' AI', ' response.'], 'response': 'This is a streamed AI response.', 'cost_usd': 0.002, 'cached': False, 'remaining_credits': 49.998}

starter-run.txt

An untouched starter fails the suite. Captured from a real run:


self = <examples.product_dispatcher.AIProductDispatcher object at 0x1069ca520>
tenant_id = 'org_gamma', prompt = 'Q1'

    def dispatch_chat(self, tenant_id: str, prompt: str) -> Dict[str, Any]:
>       raise NotImplementedError('TASK 2: implement dispatch_chat.')
E       NotImplementedError: TASK 2: implement dispatch_chat.

examples/product_dispatcher.py:22: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_product_dispatcher.py::test_valid_dispatch_and_credit_deduction
FAILED tests/test_product_dispatcher.py::test_cached_query_zero_cost - NotImp...
FAILED tests/test_product_dispatcher.py::test_unauthorized_tenant - NotImplem...
FAILED tests/test_product_dispatcher.py::test_zero_credit_rejection - NotImpl...
FAILED tests/test_product_dispatcher.py::test_rate_limit_rejection - NotImple...
============================== 5 failed in 0.03s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-316-architecture-of-an-ai-product
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_product_dispatcher.py::test_valid_dispatch_and_credit_deduction PASSED [ 20%]
tests/test_product_dispatcher.py::test_cached_query_zero_cost PASSED     [ 40%]
tests/test_product_dispatcher.py::test_unauthorized_tenant PASSED        [ 60%]
tests/test_product_dispatcher.py::test_zero_credit_rejection PASSED      [ 80%]
tests/test_product_dispatcher.py::test_rate_limit_rejection PASSED       [100%]

============================== 5 passed in 0.01s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-316-architecture-of-an-ai-product
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_product_dispatcher.py::test_valid_dispatch_and_credit_deduction PASSED [ 20%]
tests/test_product_dispatcher.py::test_cached_query_zero_cost PASSED     [ 40%]
tests/test_product_dispatcher.py::test_unauthorized_tenant PASSED        [ 60%]
tests/test_product_dispatcher.py::test_zero_credit_rejection PASSED      [ 80%]
tests/test_product_dispatcher.py::test_rate_limit_rejection PASSED       [100%]

============================== 5 passed in 0.01s ===============================

Source files

examples/product_dispatcher.py (2630 bytes)
import time
import json
from typing import Dict, Any, Tuple, List, Optional

class AIProductDispatcher:
    def __init__(self):
        self.tenants: Dict[str, Dict[str, Any]] = {
            "org_alpha": {"credits": 50.0, "rate_limit_per_min": 60, "requests_this_min": 0, "last_reset": time.time()},
            "org_beta": {"credits": 0.0, "rate_limit_per_min": 10, "requests_this_min": 0, "last_reset": time.time()},
            "org_gamma": {"credits": 10.0, "rate_limit_per_min": 2, "requests_this_min": 0, "last_reset": time.time()}
        }
        self.cache: Dict[str, str] = {}
        
    def authenticate_and_gate(self, tenant_id: str) -> Tuple[bool, str]:
        if tenant_id not in self.tenants:
            return False, "Unauthorized: Invalid Tenant ID"
            
        tenant = self.tenants[tenant_id]
        now = time.time()
        
        if now - tenant["last_reset"] > 60:
            tenant["requests_this_min"] = 0
            tenant["last_reset"] = now
            
        if tenant["requests_this_min"] >= tenant["rate_limit_per_min"]:
            return False, "Too Many Requests: Rate limit exceeded"
            
        if tenant["credits"] <= 0.0:
            return False, "Payment Required: Zero credit balance"
            
        tenant["requests_this_min"] += 1
        return True, "OK"

    def dispatch_chat(self, tenant_id: str, prompt: str) -> Dict[str, Any]:
        auth_ok, msg = self.authenticate_and_gate(tenant_id)
        if not auth_ok:
            return {"status": "ERROR", "error_code": msg, "tokens_streamed": []}
            
        cache_key = f"{tenant_id}:{prompt.strip().lower()}"
        if cache_key in self.cache:
            return {
                "status": "CACHED",
                "response": self.cache[cache_key],
                "cost_usd": 0.0,
                "cached": True,
                "remaining_credits": round(self.tenants[tenant_id]["credits"], 4)
            }
            
        mock_tokens = ["This", " is", " a", " streamed", " AI", " response."]
        full_response = "".join(mock_tokens)
        
        self.tenants[tenant_id]["credits"] -= 0.002
        self.cache[cache_key] = full_response
        
        return {
            "status": "SUCCESS",
            "tokens_streamed": mock_tokens,
            "response": full_response,
            "cost_usd": 0.002,
            "cached": False,
            "remaining_credits": round(self.tenants[tenant_id]["credits"], 4)
        }

if __name__ == "__main__":
    dispatcher = AIProductDispatcher()
    print("Org Alpha:", dispatcher.dispatch_chat("org_alpha", "Hello"))
metadata.yml (405 bytes)
lesson_id: D316
day: 316
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/product_dispatcher.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/product_dispatcher.py (1258 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import time
import json
from typing import Dict, Any, Tuple, List, Optional

class AIProductDispatcher:

    def __init__(self):
        self.tenants: Dict[str, Dict[str, Any]] = {'org_alpha': {'credits': 50.0, 'rate_limit_per_min': 60, 'requests_this_min': 0, 'last_reset': time.time()}, 'org_beta': {'credits': 0.0, 'rate_limit_per_min': 10, 'requests_this_min': 0, 'last_reset': time.time()}, 'org_gamma': {'credits': 10.0, 'rate_limit_per_min': 2, 'requests_this_min': 0, 'last_reset': time.time()}}
        self.cache: Dict[str, str] = {}

    def authenticate_and_gate(self, tenant_id: str) -> Tuple[bool, str]:
        raise NotImplementedError('TASK 1: implement authenticate_and_gate.')

    def dispatch_chat(self, tenant_id: str, prompt: str) -> Dict[str, Any]:
        raise NotImplementedError('TASK 2: implement dispatch_chat.')
if __name__ == '__main__':
    dispatcher = AIProductDispatcher()
    print('Org Alpha:', dispatcher.dispatch_chat('org_alpha', 'Hello'))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_product_dispatcher.py (1651 bytes)
import pytest
from examples.product_dispatcher import AIProductDispatcher

def test_valid_dispatch_and_credit_deduction():
    dispatcher = AIProductDispatcher()
    res = dispatcher.dispatch_chat("org_alpha", "Tell me a joke")
    assert res["status"] == "SUCCESS"
    assert len(res["tokens_streamed"]) > 0
    assert res["cached"] is False
    assert res["remaining_credits"] == 49.998

def test_cached_query_zero_cost():
    dispatcher = AIProductDispatcher()
    res1 = dispatcher.dispatch_chat("org_alpha", "Summarize article")
    res2 = dispatcher.dispatch_chat("org_alpha", "Summarize article")
    assert res2["status"] == "CACHED"
    assert res2["cached"] is True
    assert res2["cost_usd"] == 0.0
    assert res2["remaining_credits"] == 49.998

def test_unauthorized_tenant():
    dispatcher = AIProductDispatcher()
    res = dispatcher.dispatch_chat("org_unknown", "Hello")
    assert res["status"] == "ERROR"
    assert "Unauthorized" in res["error_code"]

def test_zero_credit_rejection():
    dispatcher = AIProductDispatcher()
    res = dispatcher.dispatch_chat("org_beta", "Generate report")
    assert res["status"] == "ERROR"
    assert "Payment Required" in res["error_code"]

def test_rate_limit_rejection():
    dispatcher = AIProductDispatcher()
    # org_gamma has limit of 2 requests/min
    res1 = dispatcher.dispatch_chat("org_gamma", "Q1")
    res2 = dispatcher.dispatch_chat("org_gamma", "Q2")
    res3 = dispatcher.dispatch_chat("org_gamma", "Q3")
    assert res1["status"] == "SUCCESS"
    assert res2["status"] == "SUCCESS"
    assert res3["status"] == "ERROR"
    assert "Rate limit exceeded" in res3["error_code"]

Troubleshooting

Troubleshooting - Day 316\nEnsure tenant dictionaries initialize with valid credit and rate limit parameters.\n

Security notes

Security - Day 316\nFrontend clients never hold raw provider credentials; all requests pass through the edge gateway.\n