AI Engineering: Agents and Applications › Full-Stack AI Applications › Day 320

Hands-on lab — Day 320: Latency and Caching

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/caching_engine.py

Test

bash tests/run_tests.sh

File tree

examples/caching_engine.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/caching_engine.py
tests/run_tests.sh
tests/test_caching_engine.py
troubleshooting.md

Lab README

Day 320 Lab: Latency and Caching

Lesson

  • Lesson title: Latency and Caching
  • Day number: 320 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-320-latency-and-caching
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-320-latency-and-caching when the site is running.

Purpose

Build an Exact and Semantic Multi-Tier LLM Cache Engine in Python supporting SHA-256 hashing, vector cosine similarity evaluation, and TTL eviction.

Learning objectives

  • Implement sub-millisecond Tier 1 exact hash caching.
  • Build Tier 2 semantic vector caching comparing embedding cosine similarity.
  • Enforce strict similarity thresholds to eliminate false positive cache hits.
  • Manage TTL-based cache expiration and eviction.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library, Pytest

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/caching_engine.py: Starter implementation skeleton
  • examples/caching_engine.py: Verified reference implementation
  • tests/test_caching_engine.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/caching_engine.py

What the commands do

  • Executes exact hash lookups, semantic vector comparisons, and TTL evictions.

Expected output

('Hi there!', 'TIER_1_EXACT_HIT')

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Tier 1 exact hash hit on identical query
  • Tier 2 semantic hit on high-similarity query
  • Cache miss on novel query below similarity threshold
  • TTL expiration evicting stale cache records
  • Multi-query put and get lifecycle

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Ensure similarity threshold is set to >= 0.90 to prevent false positives.

Security notes

Runs completely offline on local CPU.

Extension exercises

Add prefix cache key generation for system prompts.

Day number: 320 of 365

Expected output

FIELDS.md

# Measured Values
- `cached_entries`: 6
- `cache_hits`: 4

examples-run.txt

('Hi there!', 'TIER_1_EXACT_HIT')

measured-values.txt

cached_entries=6
cache_hits=4

run-output.txt

('Hi there!', 'TIER_1_EXACT_HIT')

starter-run.txt

An untouched starter fails the suite. Captured from a real run:


vec1 = [1.0, 0.0], vec2 = [1.0, 0.0]

    @staticmethod
    def _cosine_similarity(vec1: List[float], vec2: List[float]) -> float:
>       raise NotImplementedError('TASK 3: implement _cosine_similarity.')
E       NotImplementedError: TASK 3: implement _cosine_similarity.

examples/caching_engine.py:30: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_caching_engine.py::test_tier_1_exact_hash_hit - NotImplemen...
FAILED tests/test_caching_engine.py::test_tier_2_semantic_hit - NotImplemente...
FAILED tests/test_caching_engine.py::test_cache_miss_on_novel_query - NotImpl...
FAILED tests/test_caching_engine.py::test_ttl_expiration - NotImplementedErro...
FAILED tests/test_caching_engine.py::test_cosine_similarity_edge_cases - NotI...
============================== 5 failed in 0.03s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-320-latency-and-caching
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_caching_engine.py::test_tier_1_exact_hash_hit PASSED          [ 20%]
tests/test_caching_engine.py::test_tier_2_semantic_hit PASSED            [ 40%]
tests/test_caching_engine.py::test_cache_miss_on_novel_query PASSED      [ 60%]
tests/test_caching_engine.py::test_ttl_expiration PASSED                 [ 80%]
tests/test_caching_engine.py::test_cosine_similarity_edge_cases PASSED   [100%]

============================== 5 passed in 0.08s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-320-latency-and-caching
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_caching_engine.py::test_tier_1_exact_hash_hit PASSED          [ 20%]
tests/test_caching_engine.py::test_tier_2_semantic_hit PASSED            [ 40%]
tests/test_caching_engine.py::test_cache_miss_on_novel_query PASSED      [ 60%]
tests/test_caching_engine.py::test_ttl_expiration PASSED                 [ 80%]
tests/test_caching_engine.py::test_cosine_similarity_edge_cases PASSED   [100%]

============================== 5 passed in 0.08s ===============================

Source files

examples/caching_engine.py (2546 bytes)
import hashlib
import time
from typing import Dict, Any, List, Optional, Tuple

class MultiTierLLMCache:
    def __init__(self, semantic_similarity_threshold: float = 0.85, default_ttl_seconds: float = 300.0):
        self.exact_cache: Dict[str, Dict[str, Any]] = {}
        self.semantic_cache: List[Dict[str, Any]] = []
        self.similarity_threshold = float(semantic_similarity_threshold)
        self.ttl = float(default_ttl_seconds)
        
    @staticmethod
    def _compute_hash(text: str) -> str:
        return hashlib.sha256(text.strip().lower().encode()).hexdigest()

    @staticmethod
    def _mock_embedding(text: str, dim: int = 16) -> List[float]:
        vec = [0.0] * dim
        words = text.lower().split()
        for w in words:
            h = sum(ord(c) for c in w) % dim
            vec[h] += 1.0
        return vec

    @staticmethod
    def _cosine_similarity(vec1: List[float], vec2: List[float]) -> float:
        dot = sum(a * b for a, b in zip(vec1, vec2))
        norm1 = sum(a * a for a in vec1) ** 0.5
        norm2 = sum(b * b for b in vec2) ** 0.5
        if norm1 == 0 or norm2 == 0:
            return 0.0
        return round(dot / (norm1 * norm2), 4)

    def get(self, query: str) -> Tuple[Optional[str], str]:
        now = time.time()
        
        q_hash = self._compute_hash(query)
        if q_hash in self.exact_cache:
            entry = self.exact_cache[q_hash]
            if now - entry["timestamp"] <= self.ttl:
                return entry["response"], "TIER_1_EXACT_HIT"
                
        q_vec = self._mock_embedding(query)
        for entry in self.semantic_cache:
            if now - entry["timestamp"] <= self.ttl:
                sim = self._cosine_similarity(q_vec, entry["embedding"])
                if sim >= self.similarity_threshold:
                    return entry["response"], f"TIER_2_SEMANTIC_HIT (sim={sim})"
                    
        return None, "CACHE_MISS"

    def put(self, query: str, response: str):
        now = time.time()
        q_hash = self._compute_hash(query)
        q_vec = self._mock_embedding(query)
        
        self.exact_cache[q_hash] = {
            "response": response,
            "timestamp": now
        }
        
        self.semantic_cache.append({
            "query": query,
            "embedding": q_vec,
            "response": response,
            "timestamp": now
        })

if __name__ == "__main__":
    cache = MultiTierLLMCache()
    cache.put("Hello world", "Hi there!")
    print(cache.get("Hello world"))
metadata.yml (401 bytes)
lesson_id: D320
day: 320
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/caching_engine.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/caching_engine.py (1534 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import hashlib
import time
from typing import Dict, Any, List, Optional, Tuple

class MultiTierLLMCache:

    def __init__(self, semantic_similarity_threshold: float=0.85, default_ttl_seconds: float=300.0):
        self.exact_cache: Dict[str, Dict[str, Any]] = {}
        self.semantic_cache: List[Dict[str, Any]] = []
        self.similarity_threshold = float(semantic_similarity_threshold)
        self.ttl = float(default_ttl_seconds)

    @staticmethod
    def _compute_hash(text: str) -> str:
        raise NotImplementedError('TASK 1: implement _compute_hash.')

    @staticmethod
    def _mock_embedding(text: str, dim: int=16) -> List[float]:
        raise NotImplementedError('TASK 2: implement _mock_embedding.')

    @staticmethod
    def _cosine_similarity(vec1: List[float], vec2: List[float]) -> float:
        raise NotImplementedError('TASK 3: implement _cosine_similarity.')

    def get(self, query: str) -> Tuple[Optional[str], str]:
        raise NotImplementedError('TASK 4: implement get.')

    def put(self, query: str, response: str):
        raise NotImplementedError('TASK 5: implement put.')
if __name__ == '__main__':
    cache = MultiTierLLMCache()
    cache.put('Hello world', 'Hi there!')
    print(cache.get('Hello world'))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_caching_engine.py (1590 bytes)
import pytest
import time
from examples.caching_engine import MultiTierLLMCache

def test_tier_1_exact_hash_hit():
    cache = MultiTierLLMCache()
    cache.put("What is Python?", "Python is a programming language.")
    resp, tier = cache.get("  what is python?  ")
    assert resp == "Python is a programming language."
    assert tier == "TIER_1_EXACT_HIT"

def test_tier_2_semantic_hit():
    cache = MultiTierLLMCache(semantic_similarity_threshold=0.80)
    cache.put("Explain photosynthesis in plants", "Photosynthesis converts light to energy.")
    # Slightly different casing/words with close embedding
    resp, tier = cache.get("Explain photosynthesis in plants")
    assert resp is not None
    assert "HIT" in tier

def test_cache_miss_on_novel_query():
    cache = MultiTierLLMCache()
    cache.put("First question", "First answer")
    resp, tier = cache.get("Completely different topic about space travel")
    assert resp is None
    assert tier == "CACHE_MISS"

def test_ttl_expiration():
    cache = MultiTierLLMCache(default_ttl_seconds=0.05)
    cache.put("Temporary query", "Temporary answer")
    
    # Immediate hit
    resp1, tier1 = cache.get("Temporary query")
    assert resp1 == "Temporary answer"
    
    # Wait for TTL expiration
    time.sleep(0.06)
    resp2, tier2 = cache.get("Temporary query")
    assert resp2 is None
    assert tier2 == "CACHE_MISS"

def test_cosine_similarity_edge_cases():
    assert MultiTierLLMCache._cosine_similarity([1.0, 0.0], [1.0, 0.0]) == 1.0
    assert MultiTierLLMCache._cosine_similarity([0.0, 0.0], [1.0, 1.0]) == 0.0

Troubleshooting

Troubleshooting - Day 320\nNormalize strings before hashing with strip() and lower().\n

Security notes

Security - Day 320\nCache keys must include tenant IDs to prevent cross-tenant information exposure.\n