AI Engineering: Agents and Applications › Full-Stack AI Applications › Day 320
Hands-on lab — Day 320: Latency and Caching
- ← Back to the Day 320 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/ai-engineering/day-320-latency-and-caching/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/caching_engine.py Test
bash tests/run_tests.sh File tree
examples/caching_engine.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/caching_engine.py tests/run_tests.sh tests/test_caching_engine.py troubleshooting.md
Lab README
Day 320 Lab: Latency and Caching
Lesson
- Lesson title: Latency and Caching
- Day number: 320 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-320-latency-and-caching
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-320-latency-and-cachingwhen the site is running.
Purpose
Build an Exact and Semantic Multi-Tier LLM Cache Engine in Python supporting SHA-256 hashing, vector cosine similarity evaluation, and TTL eviction.
Learning objectives
- Implement sub-millisecond Tier 1 exact hash caching.
- Build Tier 2 semantic vector caching comparing embedding cosine similarity.
- Enforce strict similarity thresholds to eliminate false positive cache hits.
- Manage TTL-based cache expiration and eviction.
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library, Pytest
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/caching_engine.py: Starter implementation skeletonexamples/caching_engine.py: Verified reference implementationtests/test_caching_engine.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/caching_engine.py
What the commands do
- Executes exact hash lookups, semantic vector comparisons, and TTL evictions.
Expected output
('Hi there!', 'TIER_1_EXACT_HIT')
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Tier 1 exact hash hit on identical query
- Tier 2 semantic hit on high-similarity query
- Cache miss on novel query below similarity threshold
- TTL expiration evicting stale cache records
- Multi-query put and get lifecycle
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Ensure similarity threshold is set to >= 0.90 to prevent false positives.
Security notes
Runs completely offline on local CPU.
Extension exercises
Add prefix cache key generation for system prompts.
Navigation
Day number: 320 of 365
Expected output
FIELDS.md
# Measured Values
- `cached_entries`: 6
- `cache_hits`: 4
examples-run.txt
('Hi there!', 'TIER_1_EXACT_HIT')
measured-values.txt
cached_entries=6
cache_hits=4
run-output.txt
('Hi there!', 'TIER_1_EXACT_HIT')
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
vec1 = [1.0, 0.0], vec2 = [1.0, 0.0]
@staticmethod
def _cosine_similarity(vec1: List[float], vec2: List[float]) -> float:
> raise NotImplementedError('TASK 3: implement _cosine_similarity.')
E NotImplementedError: TASK 3: implement _cosine_similarity.
examples/caching_engine.py:30: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_caching_engine.py::test_tier_1_exact_hash_hit - NotImplemen...
FAILED tests/test_caching_engine.py::test_tier_2_semantic_hit - NotImplemente...
FAILED tests/test_caching_engine.py::test_cache_miss_on_novel_query - NotImpl...
FAILED tests/test_caching_engine.py::test_ttl_expiration - NotImplementedErro...
FAILED tests/test_caching_engine.py::test_cosine_similarity_edge_cases - NotI...
============================== 5 failed in 0.03s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-320-latency-and-caching
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_caching_engine.py::test_tier_1_exact_hash_hit PASSED [ 20%]
tests/test_caching_engine.py::test_tier_2_semantic_hit PASSED [ 40%]
tests/test_caching_engine.py::test_cache_miss_on_novel_query PASSED [ 60%]
tests/test_caching_engine.py::test_ttl_expiration PASSED [ 80%]
tests/test_caching_engine.py::test_cosine_similarity_edge_cases PASSED [100%]
============================== 5 passed in 0.08s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-320-latency-and-caching
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_caching_engine.py::test_tier_1_exact_hash_hit PASSED [ 20%]
tests/test_caching_engine.py::test_tier_2_semantic_hit PASSED [ 40%]
tests/test_caching_engine.py::test_cache_miss_on_novel_query PASSED [ 60%]
tests/test_caching_engine.py::test_ttl_expiration PASSED [ 80%]
tests/test_caching_engine.py::test_cosine_similarity_edge_cases PASSED [100%]
============================== 5 passed in 0.08s ===============================
Source files
examples/caching_engine.py (2546 bytes)
import hashlib
import time
from typing import Dict, Any, List, Optional, Tuple
class MultiTierLLMCache:
def __init__(self, semantic_similarity_threshold: float = 0.85, default_ttl_seconds: float = 300.0):
self.exact_cache: Dict[str, Dict[str, Any]] = {}
self.semantic_cache: List[Dict[str, Any]] = []
self.similarity_threshold = float(semantic_similarity_threshold)
self.ttl = float(default_ttl_seconds)
@staticmethod
def _compute_hash(text: str) -> str:
return hashlib.sha256(text.strip().lower().encode()).hexdigest()
@staticmethod
def _mock_embedding(text: str, dim: int = 16) -> List[float]:
vec = [0.0] * dim
words = text.lower().split()
for w in words:
h = sum(ord(c) for c in w) % dim
vec[h] += 1.0
return vec
@staticmethod
def _cosine_similarity(vec1: List[float], vec2: List[float]) -> float:
dot = sum(a * b for a, b in zip(vec1, vec2))
norm1 = sum(a * a for a in vec1) ** 0.5
norm2 = sum(b * b for b in vec2) ** 0.5
if norm1 == 0 or norm2 == 0:
return 0.0
return round(dot / (norm1 * norm2), 4)
def get(self, query: str) -> Tuple[Optional[str], str]:
now = time.time()
q_hash = self._compute_hash(query)
if q_hash in self.exact_cache:
entry = self.exact_cache[q_hash]
if now - entry["timestamp"] <= self.ttl:
return entry["response"], "TIER_1_EXACT_HIT"
q_vec = self._mock_embedding(query)
for entry in self.semantic_cache:
if now - entry["timestamp"] <= self.ttl:
sim = self._cosine_similarity(q_vec, entry["embedding"])
if sim >= self.similarity_threshold:
return entry["response"], f"TIER_2_SEMANTIC_HIT (sim={sim})"
return None, "CACHE_MISS"
def put(self, query: str, response: str):
now = time.time()
q_hash = self._compute_hash(query)
q_vec = self._mock_embedding(query)
self.exact_cache[q_hash] = {
"response": response,
"timestamp": now
}
self.semantic_cache.append({
"query": query,
"embedding": q_vec,
"response": response,
"timestamp": now
})
if __name__ == "__main__":
cache = MultiTierLLMCache()
cache.put("Hello world", "Hi there!")
print(cache.get("Hello world"))
metadata.yml (401 bytes)
lesson_id: D320
day: 320
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/caching_engine.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/caching_engine.py (1534 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import hashlib
import time
from typing import Dict, Any, List, Optional, Tuple
class MultiTierLLMCache:
def __init__(self, semantic_similarity_threshold: float=0.85, default_ttl_seconds: float=300.0):
self.exact_cache: Dict[str, Dict[str, Any]] = {}
self.semantic_cache: List[Dict[str, Any]] = []
self.similarity_threshold = float(semantic_similarity_threshold)
self.ttl = float(default_ttl_seconds)
@staticmethod
def _compute_hash(text: str) -> str:
raise NotImplementedError('TASK 1: implement _compute_hash.')
@staticmethod
def _mock_embedding(text: str, dim: int=16) -> List[float]:
raise NotImplementedError('TASK 2: implement _mock_embedding.')
@staticmethod
def _cosine_similarity(vec1: List[float], vec2: List[float]) -> float:
raise NotImplementedError('TASK 3: implement _cosine_similarity.')
def get(self, query: str) -> Tuple[Optional[str], str]:
raise NotImplementedError('TASK 4: implement get.')
def put(self, query: str, response: str):
raise NotImplementedError('TASK 5: implement put.')
if __name__ == '__main__':
cache = MultiTierLLMCache()
cache.put('Hello world', 'Hi there!')
print(cache.get('Hello world'))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_caching_engine.py (1590 bytes)
import pytest
import time
from examples.caching_engine import MultiTierLLMCache
def test_tier_1_exact_hash_hit():
cache = MultiTierLLMCache()
cache.put("What is Python?", "Python is a programming language.")
resp, tier = cache.get(" what is python? ")
assert resp == "Python is a programming language."
assert tier == "TIER_1_EXACT_HIT"
def test_tier_2_semantic_hit():
cache = MultiTierLLMCache(semantic_similarity_threshold=0.80)
cache.put("Explain photosynthesis in plants", "Photosynthesis converts light to energy.")
# Slightly different casing/words with close embedding
resp, tier = cache.get("Explain photosynthesis in plants")
assert resp is not None
assert "HIT" in tier
def test_cache_miss_on_novel_query():
cache = MultiTierLLMCache()
cache.put("First question", "First answer")
resp, tier = cache.get("Completely different topic about space travel")
assert resp is None
assert tier == "CACHE_MISS"
def test_ttl_expiration():
cache = MultiTierLLMCache(default_ttl_seconds=0.05)
cache.put("Temporary query", "Temporary answer")
# Immediate hit
resp1, tier1 = cache.get("Temporary query")
assert resp1 == "Temporary answer"
# Wait for TTL expiration
time.sleep(0.06)
resp2, tier2 = cache.get("Temporary query")
assert resp2 is None
assert tier2 == "CACHE_MISS"
def test_cosine_similarity_edge_cases():
assert MultiTierLLMCache._cosine_similarity([1.0, 0.0], [1.0, 0.0]) == 1.0
assert MultiTierLLMCache._cosine_similarity([0.0, 0.0], [1.0, 1.0]) == 0.0