AI Engineering: Agents and Applications › Full-Stack AI Applications › Day 317

Hands-on lab — Day 317: Backend Patterns for LLM Apps

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/backend_patterns.py

Test

bash tests/run_tests.sh

File tree

examples/backend_patterns.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/backend_patterns.py
tests/run_tests.sh
tests/test_backend_patterns.py
troubleshooting.md

Lab README

Day 317 Lab: Backend Patterns for LLM Apps

Lesson

Purpose

Build a Resilient LLM Backend Engine in Python supporting Idempotency Key deduplication and a 3-State Circuit Breaker with automated fallback routing.

Learning objectives

  • Implement idempotent request deduplication to prevent duplicate inference billing.
  • Build a 3-state Circuit Breaker (CLOSED, OPEN, HALF-OPEN).
  • Route traffic automatically to fallback replicas when upstream providers degrade.
  • Verify self-healing recovery when probe requests succeed in HALF-OPEN state.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library, Pytest

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/backend_patterns.py: Starter implementation skeleton
  • examples/backend_patterns.py: Verified reference implementation
  • tests/test_backend_patterns.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/backend_patterns.py

What the commands do

  • Executes requests across idempotency checks and circuit breaker state transitions.

Expected output

Normal: {'status': 'SUCCESS', 'provider': 'primary_model', 'response': '[PRIMARY_CLAUDE] Processed: Hello', 'cached': False}

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Normal primary model execution in CLOSED state
  • Idempotent response replay on duplicate keys
  • Circuit breaker tripping to OPEN after threshold failures
  • Automatic fallback routing when circuit is OPEN
  • Self-healing state recovery to CLOSED upon successful probe

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Ensure reset timeout expires before expecting HALF-OPEN state transition.

Security notes

Runs completely offline on local CPU.

Extension exercises

Add sliding window error rate calculation.

Day number: 317 of 365

Expected output

FIELDS.md

# Measured Values
- `requests_handled`: 15
- `circuit_trips`: 2

examples-run.txt

Normal: {'status': 'SUCCESS', 'provider': 'primary_model', 'response': '[PRIMARY_CLAUDE] Processed: Hello', 'cached': False}

measured-values.txt

requests_handled=15
circuit_trips=2

run-output.txt

Normal: {'status': 'SUCCESS', 'provider': 'primary_model', 'response': '[PRIMARY_CLAUDE] Processed: Hello', 'cached': False}

starter-run.txt

An untouched starter fails the suite. Captured from a real run:


self = <examples.backend_patterns.ResilientLLMBackend object at 0x106d62520>
idempotency_key = 'k_err', prompt = 'P1', simulate_upstream_fail = True

    def process_request(self, idempotency_key: str, prompt: str, simulate_upstream_fail: bool=False) -> Dict[str, Any]:
>       raise NotImplementedError('TASK 4: implement process_request.')
E       NotImplementedError: TASK 4: implement process_request.

examples/backend_patterns.py:32: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_backend_patterns.py::test_successful_request - NotImplement...
FAILED tests/test_backend_patterns.py::test_idempotent_replay - NotImplemente...
FAILED tests/test_backend_patterns.py::test_circuit_trips_to_open - NotImplem...
FAILED tests/test_backend_patterns.py::test_fallback_routing_when_open - NotI...
FAILED tests/test_backend_patterns.py::test_self_healing_recovery - NotImplem...
============================== 5 failed in 0.03s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-317-backend-patterns-for-llm-apps
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_backend_patterns.py::test_successful_request PASSED           [ 20%]
tests/test_backend_patterns.py::test_idempotent_replay PASSED            [ 40%]
tests/test_backend_patterns.py::test_circuit_trips_to_open PASSED        [ 60%]
tests/test_backend_patterns.py::test_fallback_routing_when_open PASSED   [ 80%]
tests/test_backend_patterns.py::test_self_healing_recovery PASSED        [100%]

============================== 5 passed in 0.08s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-317-backend-patterns-for-llm-apps
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_backend_patterns.py::test_successful_request PASSED           [ 20%]
tests/test_backend_patterns.py::test_idempotent_replay PASSED            [ 40%]
tests/test_backend_patterns.py::test_circuit_trips_to_open PASSED        [ 60%]
tests/test_backend_patterns.py::test_fallback_routing_when_open PASSED   [ 80%]
tests/test_backend_patterns.py::test_self_healing_recovery PASSED        [100%]

============================== 5 passed in 0.08s ===============================

Source files

examples/backend_patterns.py (2593 bytes)
import time
import json
from typing import Dict, Any, List, Optional, Tuple

class ResilientLLMBackend:
    def __init__(self, failure_threshold: int = 3, reset_timeout_seconds: float = 1.0):
        self.failure_threshold = int(failure_threshold)
        self.reset_timeout = float(reset_timeout_seconds)
        self.state = "CLOSED"  # CLOSED, OPEN, HALF-OPEN
        self.failure_count = 0
        self.last_state_change = time.time()
        self.idempotency_store: Dict[str, Dict[str, Any]] = {}
        
    def record_failure(self):
        self.failure_count += 1
        if self.failure_count >= self.failure_threshold:
            self.state = "OPEN"
            self.last_state_change = time.time()
            
    def record_success(self):
        self.failure_count = 0
        self.state = "CLOSED"
        
    def check_circuit_state(self) -> str:
        if self.state == "OPEN":
            if time.time() - self.last_state_change >= self.reset_timeout:
                self.state = "HALF-OPEN"
        return self.state

    def process_request(self, idempotency_key: str, prompt: str, simulate_upstream_fail: bool = False) -> Dict[str, Any]:
        if idempotency_key in self.idempotency_store:
            return {
                "status": "REPLAYED",
                "idempotency_key": idempotency_key,
                "cached": True,
                "response": self.idempotency_store[idempotency_key]["response"]
            }
            
        current_state = self.check_circuit_state()
        if current_state == "OPEN":
            fallback_response = f"[FALLBACK_BACKUP_MODEL] Processed: {prompt}"
            self.idempotency_store[idempotency_key] = {"response": fallback_response}
            return {
                "status": "CIRCUIT_OPEN_FALLBACK",
                "provider": "backup_replica",
                "response": fallback_response,
                "cached": False
            }
            
        if simulate_upstream_fail:
            self.record_failure()
            return {"status": "UPSTREAM_ERROR", "error": "503 Service Unavailable"}
            
        self.record_success()
        primary_response = f"[PRIMARY_CLAUDE] Processed: {prompt}"
        self.idempotency_store[idempotency_key] = {"response": primary_response}
        
        return {
            "status": "SUCCESS",
            "provider": "primary_model",
            "response": primary_response,
            "cached": False
        }

if __name__ == "__main__":
    backend = ResilientLLMBackend()
    print("Normal:", backend.process_request("k1", "Hello"))
metadata.yml (403 bytes)
lesson_id: D317
day: 317
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/backend_patterns.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/backend_patterns.py (1389 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import time
import json
from typing import Dict, Any, List, Optional, Tuple

class ResilientLLMBackend:

    def __init__(self, failure_threshold: int=3, reset_timeout_seconds: float=1.0):
        self.failure_threshold = int(failure_threshold)
        self.reset_timeout = float(reset_timeout_seconds)
        self.state = 'CLOSED'
        self.failure_count = 0
        self.last_state_change = time.time()
        self.idempotency_store: Dict[str, Dict[str, Any]] = {}

    def record_failure(self):
        raise NotImplementedError('TASK 1: implement record_failure.')

    def record_success(self):
        raise NotImplementedError('TASK 2: implement record_success.')

    def check_circuit_state(self) -> str:
        raise NotImplementedError('TASK 3: implement check_circuit_state.')

    def process_request(self, idempotency_key: str, prompt: str, simulate_upstream_fail: bool=False) -> Dict[str, Any]:
        raise NotImplementedError('TASK 4: implement process_request.')
if __name__ == '__main__':
    backend = ResilientLLMBackend()
    print('Normal:', backend.process_request('k1', 'Hello'))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_backend_patterns.py (1818 bytes)
import pytest
import time
from examples.backend_patterns import ResilientLLMBackend

def test_successful_request():
    backend = ResilientLLMBackend()
    res = backend.process_request("key_1", "What is Python?")
    assert res["status"] == "SUCCESS"
    assert res["provider"] == "primary_model"
    assert res["cached"] is False

def test_idempotent_replay():
    backend = ResilientLLMBackend()
    res1 = backend.process_request("key_2", "Prompt A")
    res2 = backend.process_request("key_2", "Prompt A")
    assert res2["status"] == "REPLAYED"
    assert res2["cached"] is True
    assert res2["response"] == res1["response"]

def test_circuit_trips_to_open():
    backend = ResilientLLMBackend(failure_threshold=2)
    # 2 failures
    backend.process_request("k_err1", "P1", simulate_upstream_fail=True)
    backend.process_request("k_err2", "P2", simulate_upstream_fail=True)
    assert backend.state == "OPEN"

def test_fallback_routing_when_open():
    backend = ResilientLLMBackend(failure_threshold=1)
    backend.process_request("k_err", "P1", simulate_upstream_fail=True)
    assert backend.state == "OPEN"
    
    res = backend.process_request("k_fall", "P2", simulate_upstream_fail=False)
    assert res["status"] == "CIRCUIT_OPEN_FALLBACK"
    assert res["provider"] == "backup_replica"

def test_self_healing_recovery():
    backend = ResilientLLMBackend(failure_threshold=1, reset_timeout_seconds=0.05)
    backend.process_request("k_err", "P1", simulate_upstream_fail=True)
    assert backend.state == "OPEN"
    
    time.sleep(0.06)
    assert backend.check_circuit_state() == "HALF-OPEN"
    
    res = backend.process_request("k_probe", "P_probe", simulate_upstream_fail=False)
    assert res["status"] == "SUCCESS"
    assert backend.state == "CLOSED"
    assert backend.failure_count == 0

Troubleshooting

Troubleshooting - Day 317\nConfirm that failure count resets to zero on successful probe requests.\n

Security notes

Security - Day 317\nIdempotency keys must be tenant-scoped to prevent key collision attacks.\n