AI Engineering: Agents and Applications › Full-Stack AI Applications › Day 317
Hands-on lab — Day 317: Backend Patterns for LLM Apps
- ← Back to the Day 317 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/ai-engineering/day-317-backend-patterns-for-llm-apps/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/backend_patterns.py Test
bash tests/run_tests.sh File tree
examples/backend_patterns.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/backend_patterns.py tests/run_tests.sh tests/test_backend_patterns.py troubleshooting.md
Lab README
Day 317 Lab: Backend Patterns for LLM Apps
Lesson
- Lesson title: Backend Patterns for LLM Apps
- Day number: 317 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-317-backend-patterns-for-llm-apps
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-317-backend-patterns-for-llm-appswhen the site is running.
Purpose
Build a Resilient LLM Backend Engine in Python supporting Idempotency Key deduplication and a 3-State Circuit Breaker with automated fallback routing.
Learning objectives
- Implement idempotent request deduplication to prevent duplicate inference billing.
- Build a 3-state Circuit Breaker (CLOSED, OPEN, HALF-OPEN).
- Route traffic automatically to fallback replicas when upstream providers degrade.
- Verify self-healing recovery when probe requests succeed in HALF-OPEN state.
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library, Pytest
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/backend_patterns.py: Starter implementation skeletonexamples/backend_patterns.py: Verified reference implementationtests/test_backend_patterns.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/backend_patterns.py
What the commands do
- Executes requests across idempotency checks and circuit breaker state transitions.
Expected output
Normal: {'status': 'SUCCESS', 'provider': 'primary_model', 'response': '[PRIMARY_CLAUDE] Processed: Hello', 'cached': False}
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Normal primary model execution in CLOSED state
- Idempotent response replay on duplicate keys
- Circuit breaker tripping to OPEN after threshold failures
- Automatic fallback routing when circuit is OPEN
- Self-healing state recovery to CLOSED upon successful probe
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Ensure reset timeout expires before expecting HALF-OPEN state transition.
Security notes
Runs completely offline on local CPU.
Extension exercises
Add sliding window error rate calculation.
Navigation
Day number: 317 of 365
Expected output
FIELDS.md
# Measured Values
- `requests_handled`: 15
- `circuit_trips`: 2
examples-run.txt
Normal: {'status': 'SUCCESS', 'provider': 'primary_model', 'response': '[PRIMARY_CLAUDE] Processed: Hello', 'cached': False}
measured-values.txt
requests_handled=15
circuit_trips=2
run-output.txt
Normal: {'status': 'SUCCESS', 'provider': 'primary_model', 'response': '[PRIMARY_CLAUDE] Processed: Hello', 'cached': False}
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
self = <examples.backend_patterns.ResilientLLMBackend object at 0x106d62520>
idempotency_key = 'k_err', prompt = 'P1', simulate_upstream_fail = True
def process_request(self, idempotency_key: str, prompt: str, simulate_upstream_fail: bool=False) -> Dict[str, Any]:
> raise NotImplementedError('TASK 4: implement process_request.')
E NotImplementedError: TASK 4: implement process_request.
examples/backend_patterns.py:32: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_backend_patterns.py::test_successful_request - NotImplement...
FAILED tests/test_backend_patterns.py::test_idempotent_replay - NotImplemente...
FAILED tests/test_backend_patterns.py::test_circuit_trips_to_open - NotImplem...
FAILED tests/test_backend_patterns.py::test_fallback_routing_when_open - NotI...
FAILED tests/test_backend_patterns.py::test_self_healing_recovery - NotImplem...
============================== 5 failed in 0.03s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-317-backend-patterns-for-llm-apps
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_backend_patterns.py::test_successful_request PASSED [ 20%]
tests/test_backend_patterns.py::test_idempotent_replay PASSED [ 40%]
tests/test_backend_patterns.py::test_circuit_trips_to_open PASSED [ 60%]
tests/test_backend_patterns.py::test_fallback_routing_when_open PASSED [ 80%]
tests/test_backend_patterns.py::test_self_healing_recovery PASSED [100%]
============================== 5 passed in 0.08s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-317-backend-patterns-for-llm-apps
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_backend_patterns.py::test_successful_request PASSED [ 20%]
tests/test_backend_patterns.py::test_idempotent_replay PASSED [ 40%]
tests/test_backend_patterns.py::test_circuit_trips_to_open PASSED [ 60%]
tests/test_backend_patterns.py::test_fallback_routing_when_open PASSED [ 80%]
tests/test_backend_patterns.py::test_self_healing_recovery PASSED [100%]
============================== 5 passed in 0.08s ===============================
Source files
examples/backend_patterns.py (2593 bytes)
import time
import json
from typing import Dict, Any, List, Optional, Tuple
class ResilientLLMBackend:
def __init__(self, failure_threshold: int = 3, reset_timeout_seconds: float = 1.0):
self.failure_threshold = int(failure_threshold)
self.reset_timeout = float(reset_timeout_seconds)
self.state = "CLOSED" # CLOSED, OPEN, HALF-OPEN
self.failure_count = 0
self.last_state_change = time.time()
self.idempotency_store: Dict[str, Dict[str, Any]] = {}
def record_failure(self):
self.failure_count += 1
if self.failure_count >= self.failure_threshold:
self.state = "OPEN"
self.last_state_change = time.time()
def record_success(self):
self.failure_count = 0
self.state = "CLOSED"
def check_circuit_state(self) -> str:
if self.state == "OPEN":
if time.time() - self.last_state_change >= self.reset_timeout:
self.state = "HALF-OPEN"
return self.state
def process_request(self, idempotency_key: str, prompt: str, simulate_upstream_fail: bool = False) -> Dict[str, Any]:
if idempotency_key in self.idempotency_store:
return {
"status": "REPLAYED",
"idempotency_key": idempotency_key,
"cached": True,
"response": self.idempotency_store[idempotency_key]["response"]
}
current_state = self.check_circuit_state()
if current_state == "OPEN":
fallback_response = f"[FALLBACK_BACKUP_MODEL] Processed: {prompt}"
self.idempotency_store[idempotency_key] = {"response": fallback_response}
return {
"status": "CIRCUIT_OPEN_FALLBACK",
"provider": "backup_replica",
"response": fallback_response,
"cached": False
}
if simulate_upstream_fail:
self.record_failure()
return {"status": "UPSTREAM_ERROR", "error": "503 Service Unavailable"}
self.record_success()
primary_response = f"[PRIMARY_CLAUDE] Processed: {prompt}"
self.idempotency_store[idempotency_key] = {"response": primary_response}
return {
"status": "SUCCESS",
"provider": "primary_model",
"response": primary_response,
"cached": False
}
if __name__ == "__main__":
backend = ResilientLLMBackend()
print("Normal:", backend.process_request("k1", "Hello"))
metadata.yml (403 bytes)
lesson_id: D317
day: 317
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/backend_patterns.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/backend_patterns.py (1389 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import time
import json
from typing import Dict, Any, List, Optional, Tuple
class ResilientLLMBackend:
def __init__(self, failure_threshold: int=3, reset_timeout_seconds: float=1.0):
self.failure_threshold = int(failure_threshold)
self.reset_timeout = float(reset_timeout_seconds)
self.state = 'CLOSED'
self.failure_count = 0
self.last_state_change = time.time()
self.idempotency_store: Dict[str, Dict[str, Any]] = {}
def record_failure(self):
raise NotImplementedError('TASK 1: implement record_failure.')
def record_success(self):
raise NotImplementedError('TASK 2: implement record_success.')
def check_circuit_state(self) -> str:
raise NotImplementedError('TASK 3: implement check_circuit_state.')
def process_request(self, idempotency_key: str, prompt: str, simulate_upstream_fail: bool=False) -> Dict[str, Any]:
raise NotImplementedError('TASK 4: implement process_request.')
if __name__ == '__main__':
backend = ResilientLLMBackend()
print('Normal:', backend.process_request('k1', 'Hello'))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_backend_patterns.py (1818 bytes)
import pytest
import time
from examples.backend_patterns import ResilientLLMBackend
def test_successful_request():
backend = ResilientLLMBackend()
res = backend.process_request("key_1", "What is Python?")
assert res["status"] == "SUCCESS"
assert res["provider"] == "primary_model"
assert res["cached"] is False
def test_idempotent_replay():
backend = ResilientLLMBackend()
res1 = backend.process_request("key_2", "Prompt A")
res2 = backend.process_request("key_2", "Prompt A")
assert res2["status"] == "REPLAYED"
assert res2["cached"] is True
assert res2["response"] == res1["response"]
def test_circuit_trips_to_open():
backend = ResilientLLMBackend(failure_threshold=2)
# 2 failures
backend.process_request("k_err1", "P1", simulate_upstream_fail=True)
backend.process_request("k_err2", "P2", simulate_upstream_fail=True)
assert backend.state == "OPEN"
def test_fallback_routing_when_open():
backend = ResilientLLMBackend(failure_threshold=1)
backend.process_request("k_err", "P1", simulate_upstream_fail=True)
assert backend.state == "OPEN"
res = backend.process_request("k_fall", "P2", simulate_upstream_fail=False)
assert res["status"] == "CIRCUIT_OPEN_FALLBACK"
assert res["provider"] == "backup_replica"
def test_self_healing_recovery():
backend = ResilientLLMBackend(failure_threshold=1, reset_timeout_seconds=0.05)
backend.process_request("k_err", "P1", simulate_upstream_fail=True)
assert backend.state == "OPEN"
time.sleep(0.06)
assert backend.check_circuit_state() == "HALF-OPEN"
res = backend.process_request("k_probe", "P_probe", simulate_upstream_fail=False)
assert res["status"] == "SUCCESS"
assert backend.state == "CLOSED"
assert backend.failure_count == 0