AI Engineering: Agents and Applications › Production Retrieval and Pipelines › Day 329
Hands-on lab — Day 329: Section Project: A Production Assistant
- ← Back to the Day 329 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/ai-engineering/day-329-section-project-a-production-assistant/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/seam_audit_demo.py Test
bash tests/run_tests.sh File tree
examples/seam_audit_demo.py examples/seam_audit.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/seam_audit.py tests/run_tests.sh tests/test_seam_audit.py troubleshooting.md
Lab README
Lab — Day 329: Section Project: A Production Assistant
Lesson
- Lesson title: Section Project: A Production Assistant
- Day number: 329 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-329-section-project-a-production-assistant
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-329-section-project-a-production-assistantwhen the site is running.
Purpose
Build a conformance auditor for the three seams of a production assistant.
The week 47 project builds an assistant. This lab checks one — a different skill, and the one you need when reviewing someone else's system, or your own six months later. Each check targets an interaction no single stage's tests can reach.
The auditor is run against two assistants: one that conforms, and one deliberately broken in three specific ways. The second matters more, because an auditor that has never caught anything is not evidence of anything.
Learning objectives
- Express a system property as an executable check rather than a review comment.
- Verify that redaction precedes indexing, by inspecting what actually reached the chunks.
- Verify that retrieval and generation share one ledger, so no paid path escapes the cap.
- Verify an erasure by reading back from every store rather than trusting the delete calls.
- Demonstrate that a checker catches real defects, and exactly those defects.
Prerequisites
- Days 323 to 328 — each check corresponds to one of them.
- Comfortable with Python dataclasses, and able to read a
Protocol.
Supported operating systems
macOS, Linux and Windows (via WSL or PowerShell). Nothing is platform-specific.
Hardware requirements
Any machine running Python 3.10 or newer. No GPU. The suite completes in hundredths of a second.
Required software
- Python 3.10 or newer.
pytest7 or newer.
Free and open-source options
Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no API key, no network.
Installation
python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt
Removing .venv fully reverses the installation.
File structure
starter/seam_audit.py your work: the five check functions
examples/seam_audit.py reference auditor plus two assistants to audit
examples/seam_audit_demo.py audits both and prints the verdicts
tests/test_seam_audit.py the auditor passes good, catches broken
tests/run_tests.sh suite entry point
expected-output/ real captured output and measured values
requirements/ pinned dependency
How to run
python3 examples/seam_audit_demo.py # audit both assistants
bash tests/run_tests.sh # run the suite
To work on the exercise, edit starter/seam_audit.py, then copy it over the reference:
cp starter/seam_audit.py examples/seam_audit.py
bash tests/run_tests.sh
Keep a copy of the reference first if you want to compare.
What the commands do
python3 examples/seam_audit_demo.py runs all five checks against ConformantAssistant and then against BrokenAssistant, printing a line per check and a verdict for each.
bash tests/run_tests.sh runs pytest over twelve tests in three groups: the auditor passes a conformant system, the auditor catches a broken one, and the reference assistant itself behaves correctly.
Expected output
--- conformant assistant ---
PASS redaction_before_indexing
PASS shared_budget
PASS erasure_is_complete
PASS cursor_advances_on_failure
PASS no_orphans_on_shrink
CONFORMANT (5/5 checks passed)
--- broken assistant ---
FAIL redaction_before_indexing -- 2 chunk(s) contain an address: ['policy::0', 'policy::1']
FAIL shared_budget -- retrieval is not charged to the request ledger
FAIL erasure_is_complete -- still present in: cache
PASS cursor_advances_on_failure
PASS no_orphans_on_shrink
NON-CONFORMANT (2/5 checks passed)
expected-output/FIELDS.md explains each field and why two assistants are audited.
Validation steps
bash tests/run_tests.shreports12 passed.- The conformant assistant must pass all five checks; the broken one must fail exactly three — the three planted defects and no others. An auditor that fails everything is as useless as one that fails nothing.
- Every
FAILmust carry a detail naming what was found. A failure without a detail is a bug report nobody can act on.
Tests
Twelve tests in three groups:
- the auditor passes good — a conformant assistant is
CONFORMANT, and all five checks actually run. - the auditor catches broken — each of the three planted defects is detected with an informative detail, and the failure list is exactly those three.
- the reference assistant — idempotent ingest, cursor advances past a dead letter, erasure clears all three stores, a shrinking document leaves no orphans, and a cached answer is free and recorded.
Cleanup
rm -rf __pycache__ .pytest_cache
rm -rf .venv
Both are safe to remove and fully reversible.
Troubleshooting
See troubleshooting.md.
Security notes
See security.md. In short: no network, no credentials, and the only identifier in the fixtures uses the reserved example.com domain.
Extension exercises
- A fourth defect. Add a
BrokenAssistantvariant that leaves orphans on shrink, and confirm the audit catches it without any change to the checks. - Audit the week 47 project. Point the auditor at the assistant you built for the project. It should be conformant — and if it is not, you have found something worth fixing.
- Severity. Give each check a severity and make the verdict distinguish a warning from a blocker, so an audit can gate a deployment rather than only report.
Navigation
- Lesson
- Previous: Day 328 — Privacy in AI Systems
- Next: Day 330 — Docker Fundamentals
Expected output
FIELDS.md
# Reading the audit output
## A check line
FAIL erasure_is_complete -- still present in: cache
| Part | Meaning |
| --- | --- |
| `PASS` / `FAIL` | Whether this seam holds in the assistant under audit. |
| name | Which interaction was checked. |
| detail | Present only on failure, and names *what* was found — the offending chunk ids, the missing ledger stage, or the stores that still hold the subject. A failure without a detail is a bug report nobody can act on. |
## The summary line
NON-CONFORMANT (2/5 checks passed)
The verdict plus the count. `CONFORMANT` requires every check to pass — these are properties, not a score to average.
## The five checks
| Check | The failure it catches |
| --- | --- |
| `redaction_before_indexing` | Identifiers reached the chunks, so they are inside the vectors and removing them means re-embedding. |
| `shared_budget` | Retrieval has its own accounting, so it is a path the spend cap does not bound. |
| `erasure_is_complete` | A deletion reached some stores and not others — usually the cache, which then keeps answering from deleted content. |
| `cursor_advances_on_failure` | A dead-lettered document blocks the head of the queue and is retried on every run. |
| `no_orphans_on_shrink` | A shorter document left stale chunks behind, still retrievable. |
## Why two assistants
The demo audits a conformant assistant and a deliberately broken one. The second is the important half: **an auditor that has never caught anything is not evidence of anything.** The broken assistant is wrong in exactly three ways, and the audit is expected to name exactly those three — no fewer, and no more.
examples-run.txt
--- conformant assistant ---
PASS redaction_before_indexing
PASS shared_budget
PASS erasure_is_complete
PASS cursor_advances_on_failure
PASS no_orphans_on_shrink
CONFORMANT (5/5 checks passed)
--- broken assistant ---
FAIL redaction_before_indexing -- 2 chunk(s) contain an address: ['policy::0', 'policy::1']
FAIL shared_budget -- retrieval is not charged to the request ledger
FAIL erasure_is_complete -- still present in: cache
PASS cursor_advances_on_failure
PASS no_orphans_on_shrink
NON-CONFORMANT (2/5 checks passed)
measured-values.txt
Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.
python3 examples/seam_audit_demo.py -> 14 lines, exit 0
bash tests/run_tests.sh -> 12 passed in 0.01s
The conformant assistant passes all five checks. The broken one fails exactly
three -- the three defects planted in it, and no others:
FAIL redaction_before_indexing -- 2 chunk(s) contain an address
FAIL shared_budget -- retrieval is not charged to the ledger
FAIL erasure_is_complete -- still present in: cache
PASS cursor_advances_on_failure
PASS no_orphans_on_shrink
Precision matters as much as recall here. An auditor that fails everything is
as useless as one that fails nothing, which is why
test_it_names_exactly_the_three_planted_defects asserts the failure list
exactly rather than merely asserting non-conformance.
run-output.txt
--- conformant assistant ---
PASS redaction_before_indexing
PASS shared_budget
PASS erasure_is_complete
PASS cursor_advances_on_failure
PASS no_orphans_on_shrink
CONFORMANT (5/5 checks passed)
--- broken assistant ---
FAIL redaction_before_indexing -- 2 chunk(s) contain an address: ['policy::0', 'policy::1']
FAIL shared_budget -- retrieval is not charged to the request ledger
FAIL erasure_is_complete -- still present in: cache
PASS cursor_advances_on_failure
PASS no_orphans_on_shrink
NON-CONFORMANT (2/5 checks passed)
starter-run.txt
An untouched starter fails most of the suite:
7 failed, 5 passed in 0.04s
The five that pass exercise ConformantAssistant directly -- idempotent ingest,
cursor advance, erasure, orphan cleanup and the free cached answer -- none of
which depends on the five stubbed check functions.
Everything that runs an audit fails until the checks are implemented.
test-output.txt
............ [100%]
12 passed in 0.01s
test-run.txt
............ [100%]
12 passed in 0.01s
Source files
examples/seam_audit_demo.py (674 bytes)
#!/usr/bin/env python3
"""Audit two assistants: one that conforms, one that does not."""
from __future__ import annotations
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from seam_audit import BrokenAssistant, ConformantAssistant, audit
def main() -> int:
for label, factory in (
("conformant assistant", ConformantAssistant),
("broken assistant", BrokenAssistant),
):
report = audit(factory)
print(f"--- {label} ---")
for check in report.checks:
print(check.line())
print(report.summary())
return 0
if __name__ == "__main__":
raise SystemExit(main())
examples/seam_audit.py (9835 bytes)
"""A conformance auditor for the three seams of a production assistant.
The week 47 project builds an assistant. This lab checks one -- which is a
different skill, and the one you need when reviewing someone else's system or
your own six months later.
Each check targets an interaction that no single stage's tests can reach:
redaction_before_indexing identifiers must not survive into any chunk
shared_budget retrieval must charge the same ledger as
generation, or one path escapes the cap
erasure_is_complete a deletion must reach the index, the content
hashes AND the response cache
cursor_advances_on_failure a dead-lettered document must not block the head
of the queue forever
no_orphans_on_shrink a shorter document must not leave stale chunks
The audit works against any object exposing the small protocol below, so it can
be pointed at a real implementation. Two are supplied: one that conforms, and
one that is deliberately broken in three specific ways -- because an auditor
that has never caught anything is not evidence of anything.
"""
from __future__ import annotations
import hashlib
import re
from dataclasses import dataclass, field
from typing import Protocol
EMAIL = re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b")
@dataclass(frozen=True)
class Doc:
doc_id: str
seq: int
body: str | None
class AssistantLike(Protocol):
"""The surface an assistant must expose to be auditable."""
chunks: dict[str, str] # chunk_id -> text
hashes: dict[str, str] # doc_id -> content hash
cache: dict[str, str] # question -> answer
cursor: int
ledger: list[tuple[str, float]] # (stage, cost)
def ingest(self, docs: list[Doc]) -> None: ...
def answer(self, question: str) -> str: ...
def erase(self, doc_id: str) -> None: ...
@dataclass
class Check:
name: str
passed: bool
detail: str = ""
def line(self) -> str:
mark = "PASS" if self.passed else "FAIL"
return f" {mark} {self.name}" + (f" -- {self.detail}" if self.detail else "")
@dataclass
class AuditReport:
checks: list[Check] = field(default_factory=list)
@property
def conformant(self) -> bool:
return all(c.passed for c in self.checks)
def failures(self) -> list[str]:
return [c.name for c in self.checks if not c.passed]
def summary(self) -> str:
passed = sum(1 for c in self.checks if c.passed)
verdict = "CONFORMANT" if self.conformant else "NON-CONFORMANT"
return f"{verdict} ({passed}/{len(self.checks)} checks passed)"
def sample_docs() -> list[Doc]:
return [
Doc("policy", 1, "The refund window is thirty days. Write to ada@example.com. " * 3),
Doc("sla", 2, "Uptime guarantee is 99.99 percent per calendar month. " * 3),
Doc("broken", 3, None),
]
# ------------------------------------------------------------------ checks
def check_redaction_before_indexing(make: type) -> Check:
bot = make()
bot.ingest(sample_docs())
leaked = [cid for cid, text in bot.chunks.items() if EMAIL.search(text)]
return Check(
"redaction_before_indexing",
not leaked,
f"{len(leaked)} chunk(s) contain an address: {leaked[:2]}" if leaked else "",
)
def check_shared_budget(make: type) -> Check:
bot = make()
bot.ingest(sample_docs())
bot.answer("What is the refund window?")
stages = {stage for stage, _ in bot.ledger}
return Check(
"shared_budget",
"retrieval" in stages,
"retrieval is not charged to the request ledger" if "retrieval" not in stages else "",
)
def check_erasure_is_complete(make: type) -> Check:
bot = make()
bot.ingest(sample_docs())
bot.answer("What is the refund window?")
bot.erase("policy")
remaining = []
if any(cid.startswith("policy::") for cid in bot.chunks):
remaining.append("index")
if "policy" in bot.hashes:
remaining.append("hashes")
if any("policy" in answer for answer in bot.cache.values()):
remaining.append("cache")
return Check(
"erasure_is_complete",
not remaining,
f"still present in: {', '.join(remaining)}" if remaining else "",
)
def check_cursor_advances_on_failure(make: type) -> Check:
bot = make()
bot.ingest(sample_docs())
before = bot.cursor
bot.ingest(sample_docs())
return Check(
"cursor_advances_on_failure",
before >= 3,
f"cursor stuck at {before}; the dead-lettered document blocks the queue"
if before < 3
else "",
)
def check_no_orphans_on_shrink(make: type) -> Check:
bot = make()
bot.ingest([Doc("long", 1, "alpha beta gamma delta " * 30)])
many = sum(1 for cid in bot.chunks if cid.startswith("long::"))
bot.ingest([Doc("long", 2, "alpha beta")])
ids = {cid for cid in bot.chunks if cid.startswith("long::")}
expected = {f"long::{i}" for i in range(len(ids))}
return Check(
"no_orphans_on_shrink",
many > 1 and ids == expected,
f"orphans left behind: {sorted(ids - expected)}" if ids != expected else "",
)
CHECKS = (
check_redaction_before_indexing,
check_shared_budget,
check_erasure_is_complete,
check_cursor_advances_on_failure,
check_no_orphans_on_shrink,
)
def audit(make: type) -> AuditReport:
"""Run every seam check against a factory producing a fresh assistant."""
return AuditReport(checks=[check(make) for check in CHECKS])
# ------------------------------------------------- two assistants to audit
def _digest(text: str) -> str:
return hashlib.sha256(" ".join(text.split()).encode()).hexdigest()
@dataclass
class ConformantAssistant:
"""Does all three seams correctly."""
chunks: dict[str, str] = field(default_factory=dict)
hashes: dict[str, str] = field(default_factory=dict)
cache: dict[str, str] = field(default_factory=dict)
cursor: int = 0
ledger: list[tuple[str, float]] = field(default_factory=list)
def ingest(self, docs: list[Doc]) -> None:
for doc in sorted((d for d in docs if d.seq > self.cursor), key=lambda d: d.seq):
if doc.body is None:
self.cursor = doc.seq # advance anyway, or the queue head blocks
continue
clean = EMAIL.sub("[email]", doc.body) # redact BEFORE anything else
digest = _digest(clean)
if self.hashes.get(doc.doc_id) == digest:
self.cursor = doc.seq
continue
words = clean.split()
fresh = set()
for position, start in enumerate(range(0, len(words), 20)):
cid = f"{doc.doc_id}::{position}"
self.chunks[cid] = " ".join(words[start : start + 20])
fresh.add(cid)
for stale in [
c for c in list(self.chunks) if c.startswith(f"{doc.doc_id}::") and c not in fresh
]:
del self.chunks[stale]
self.hashes[doc.doc_id] = digest
self.cursor = doc.seq
def answer(self, question: str) -> str:
if question in self.cache:
self.ledger.append(("cache", 0.0))
return self.cache[question]
self.ledger.append(("retrieval", 0.000002)) # same ledger as generation
self.ledger.append(("small", 0.00002))
hit = next((c for c in self.chunks if c.startswith("policy")), "none")
reply = f"answer citing {hit}"
self.cache[question] = reply
return reply
def erase(self, doc_id: str) -> None:
for cid in [c for c in list(self.chunks) if c.startswith(f"{doc_id}::")]:
del self.chunks[cid]
self.hashes.pop(doc_id, None)
self.cache = {q: a for q, a in self.cache.items() if doc_id not in a}
@dataclass
class BrokenAssistant(ConformantAssistant):
"""Broken in three specific ways, so the audit has something to catch.
1. redacts after chunking, so identifiers reach the index
2. charges retrieval to a separate counter, so it escapes the cap
3. erases the index but forgets the cache
"""
side_ledger: list[tuple[str, float]] = field(default_factory=list)
def ingest(self, docs: list[Doc]) -> None:
for doc in sorted((d for d in docs if d.seq > self.cursor), key=lambda d: d.seq):
if doc.body is None:
self.cursor = doc.seq
continue
words = doc.body.split() # NOT redacted
fresh = set()
for position, start in enumerate(range(0, len(words), 20)):
cid = f"{doc.doc_id}::{position}"
self.chunks[cid] = " ".join(words[start : start + 20])
fresh.add(cid)
for stale in [
c for c in list(self.chunks) if c.startswith(f"{doc.doc_id}::") and c not in fresh
]:
del self.chunks[stale]
self.hashes[doc.doc_id] = _digest(doc.body)
self.cursor = doc.seq
def answer(self, question: str) -> str:
if question in self.cache:
self.ledger.append(("cache", 0.0))
return self.cache[question]
self.side_ledger.append(("retrieval", 0.000002)) # escapes the cap
self.ledger.append(("small", 0.00002))
hit = next((c for c in self.chunks if c.startswith("policy")), "none")
reply = f"answer citing {hit}"
self.cache[question] = reply
return reply
def erase(self, doc_id: str) -> None:
for cid in [c for c in list(self.chunks) if c.startswith(f"{doc_id}::")]:
del self.chunks[cid]
self.hashes.pop(doc_id, None)
# cache deliberately untouched
metadata.yml (853 bytes)
lesson_id: D329
day: 329
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/seam_audit_demo.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 60
last_executed: '2026-08-31'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3
examples/seam_audit_demo.py reproduced the documented verdicts exactly, with the broken assistant
failing exactly the three planted defects; bash tests/run_tests.sh -> 12 passed in 0.01s. The untouched
starter fails 7 of 12; the 5 that pass exercise the reference assistant rather than the stubbed checks.
requires_network is false: everything is in-process and no test opens a socket.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/seam_audit.py (9205 bytes)
"""A conformance auditor for the three seams of a production assistant.
The week 47 project builds an assistant. This lab checks one -- which is a
different skill, and the one you need when reviewing someone else's system or
your own six months later.
Each check targets an interaction that no single stage's tests can reach:
redaction_before_indexing identifiers must not survive into any chunk
shared_budget retrieval must charge the same ledger as
generation, or one path escapes the cap
erasure_is_complete a deletion must reach the index, the content
hashes AND the response cache
cursor_advances_on_failure a dead-lettered document must not block the head
of the queue forever
no_orphans_on_shrink a shorter document must not leave stale chunks
The audit works against any object exposing the small protocol below, so it can
be pointed at a real implementation. Two are supplied: one that conforms, and
one that is deliberately broken in three specific ways -- because an auditor
that has never caught anything is not evidence of anything.
"""
from __future__ import annotations
import hashlib
import re
from dataclasses import dataclass, field
from typing import Protocol
EMAIL = re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b")
@dataclass(frozen=True)
class Doc:
doc_id: str
seq: int
body: str | None
class AssistantLike(Protocol):
"""The surface an assistant must expose to be auditable."""
chunks: dict[str, str] # chunk_id -> text
hashes: dict[str, str] # doc_id -> content hash
cache: dict[str, str] # question -> answer
cursor: int
ledger: list[tuple[str, float]] # (stage, cost)
def ingest(self, docs: list[Doc]) -> None: ...
def answer(self, question: str) -> str: ...
def erase(self, doc_id: str) -> None: ...
@dataclass
class Check:
name: str
passed: bool
detail: str = ""
def line(self) -> str:
mark = "PASS" if self.passed else "FAIL"
return f" {mark} {self.name}" + (f" -- {self.detail}" if self.detail else "")
@dataclass
class AuditReport:
checks: list[Check] = field(default_factory=list)
@property
def conformant(self) -> bool:
return all(c.passed for c in self.checks)
def failures(self) -> list[str]:
return [c.name for c in self.checks if not c.passed]
def summary(self) -> str:
passed = sum(1 for c in self.checks if c.passed)
verdict = "CONFORMANT" if self.conformant else "NON-CONFORMANT"
return f"{verdict} ({passed}/{len(self.checks)} checks passed)"
def sample_docs() -> list[Doc]:
return [
Doc("policy", 1, "The refund window is thirty days. Write to ada@example.com. " * 3),
Doc("sla", 2, "Uptime guarantee is 99.99 percent per calendar month. " * 3),
Doc("broken", 3, None),
]
# ------------------------------------------------------------------ checks
def check_redaction_before_indexing(make: type) -> Check:
# TASK 1: ingest sample_docs() into a fresh assistant and assert that no
# chunk text still matches EMAIL. Report the offending chunk ids in detail.
raise NotImplementedError("implement check_redaction_before_indexing")
def check_shared_budget(make: type) -> Check:
# TASK 2: ingest, ask one question, then assert "retrieval" appears among
# the ledger stages. A retrieval stage with its own counter is a path the
# spend cap does not bound.
raise NotImplementedError("implement check_shared_budget")
def check_erasure_is_complete(make: type) -> Check:
# TASK 3: ingest, ask a question so the cache holds an answer, erase
# "policy", then check ALL THREE stores by reading back: chunks, hashes and
# cache. Name whichever still hold it in detail.
raise NotImplementedError("implement check_erasure_is_complete")
def check_cursor_advances_on_failure(make: type) -> Check:
# TASK 4: ingest twice and assert the cursor passed the dead-lettered
# document (seq 3). A cursor that stops there retries the poison record on
# every run and the queue head never moves.
raise NotImplementedError("implement check_cursor_advances_on_failure")
def check_no_orphans_on_shrink(make: type) -> Check:
# TASK 5: ingest a long document, then a much shorter one with the same
# doc_id, and assert the surviving chunk ids are exactly long::0 .. long::n-1
# with no gaps -- otherwise stale chunks are still retrievable.
raise NotImplementedError("implement check_no_orphans_on_shrink")
CHECKS = (
check_redaction_before_indexing,
check_shared_budget,
check_erasure_is_complete,
check_cursor_advances_on_failure,
check_no_orphans_on_shrink,
)
def audit(make: type) -> AuditReport:
"""Run every seam check against a factory producing a fresh assistant."""
return AuditReport(checks=[check(make) for check in CHECKS])
# ------------------------------------------------- two assistants to audit
def _digest(text: str) -> str:
return hashlib.sha256(" ".join(text.split()).encode()).hexdigest()
@dataclass
class ConformantAssistant:
"""Does all three seams correctly."""
chunks: dict[str, str] = field(default_factory=dict)
hashes: dict[str, str] = field(default_factory=dict)
cache: dict[str, str] = field(default_factory=dict)
cursor: int = 0
ledger: list[tuple[str, float]] = field(default_factory=list)
def ingest(self, docs: list[Doc]) -> None:
for doc in sorted((d for d in docs if d.seq > self.cursor), key=lambda d: d.seq):
if doc.body is None:
self.cursor = doc.seq # advance anyway, or the queue head blocks
continue
clean = EMAIL.sub("[email]", doc.body) # redact BEFORE anything else
digest = _digest(clean)
if self.hashes.get(doc.doc_id) == digest:
self.cursor = doc.seq
continue
words = clean.split()
fresh = set()
for position, start in enumerate(range(0, len(words), 20)):
cid = f"{doc.doc_id}::{position}"
self.chunks[cid] = " ".join(words[start : start + 20])
fresh.add(cid)
for stale in [
c for c in list(self.chunks) if c.startswith(f"{doc.doc_id}::") and c not in fresh
]:
del self.chunks[stale]
self.hashes[doc.doc_id] = digest
self.cursor = doc.seq
def answer(self, question: str) -> str:
if question in self.cache:
self.ledger.append(("cache", 0.0))
return self.cache[question]
self.ledger.append(("retrieval", 0.000002)) # same ledger as generation
self.ledger.append(("small", 0.00002))
hit = next((c for c in self.chunks if c.startswith("policy")), "none")
reply = f"answer citing {hit}"
self.cache[question] = reply
return reply
def erase(self, doc_id: str) -> None:
for cid in [c for c in list(self.chunks) if c.startswith(f"{doc_id}::")]:
del self.chunks[cid]
self.hashes.pop(doc_id, None)
self.cache = {q: a for q, a in self.cache.items() if doc_id not in a}
@dataclass
class BrokenAssistant(ConformantAssistant):
"""Broken in three specific ways, so the audit has something to catch.
1. redacts after chunking, so identifiers reach the index
2. charges retrieval to a separate counter, so it escapes the cap
3. erases the index but forgets the cache
"""
side_ledger: list[tuple[str, float]] = field(default_factory=list)
def ingest(self, docs: list[Doc]) -> None:
for doc in sorted((d for d in docs if d.seq > self.cursor), key=lambda d: d.seq):
if doc.body is None:
self.cursor = doc.seq
continue
words = doc.body.split() # NOT redacted
fresh = set()
for position, start in enumerate(range(0, len(words), 20)):
cid = f"{doc.doc_id}::{position}"
self.chunks[cid] = " ".join(words[start : start + 20])
fresh.add(cid)
for stale in [
c for c in list(self.chunks) if c.startswith(f"{doc.doc_id}::") and c not in fresh
]:
del self.chunks[stale]
self.hashes[doc.doc_id] = _digest(doc.body)
self.cursor = doc.seq
def answer(self, question: str) -> str:
if question in self.cache:
self.ledger.append(("cache", 0.0))
return self.cache[question]
self.side_ledger.append(("retrieval", 0.000002)) # escapes the cap
self.ledger.append(("small", 0.00002))
hit = next((c for c in self.chunks if c.startswith("policy")), "none")
reply = f"answer citing {hit}"
self.cache[question] = reply
return reply
def erase(self, doc_id: str) -> None:
for cid in [c for c in list(self.chunks) if c.startswith(f"{doc_id}::")]:
del self.chunks[cid]
self.hashes.pop(doc_id, None)
# cache deliberately untouched
tests/run_tests.sh (200 bytes)
#!/usr/bin/env bash
# Run the day 329 seam audit test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_seam_audit.py (3946 bytes)
"""Tests for the seam auditor.
An auditor needs two kinds of test: that it passes a conformant system, and --
more importantly -- that it FAILS a broken one. A checker that has never caught
anything is not evidence of anything.
Run with: bash tests/run_tests.sh
"""
from __future__ import annotations
import os
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))
from seam_audit import ( # noqa: E402
CHECKS,
BrokenAssistant,
ConformantAssistant,
Doc,
audit,
check_erasure_is_complete,
check_redaction_before_indexing,
check_shared_budget,
sample_docs,
)
# ------------------------------------------------- the auditor passes good
def test_conformant_assistant_passes_every_check():
report = audit(ConformantAssistant)
assert report.conformant, f"unexpected failures: {report.failures()}"
assert report.summary() == "CONFORMANT (5/5 checks passed)"
def test_every_check_is_actually_run():
report = audit(ConformantAssistant)
assert len(report.checks) == len(CHECKS) == 5
names = {c.name for c in report.checks}
assert names == {
"redaction_before_indexing",
"shared_budget",
"erasure_is_complete",
"cursor_advances_on_failure",
"no_orphans_on_shrink",
}
# ------------------------------------------- the auditor catches the broken
def test_broken_assistant_is_reported_non_conformant():
report = audit(BrokenAssistant)
assert not report.conformant
def test_it_catches_redaction_running_after_indexing():
check = check_redaction_before_indexing(BrokenAssistant)
assert not check.passed
assert "address" in check.detail
def test_it_catches_retrieval_escaping_the_shared_budget():
check = check_shared_budget(BrokenAssistant)
assert not check.passed
assert "retrieval" in check.detail
def test_it_catches_an_erasure_that_forgets_the_cache():
check = check_erasure_is_complete(BrokenAssistant)
assert not check.passed
assert "cache" in check.detail
def test_it_names_exactly_the_three_planted_defects():
# Precision matters as much as recall: an auditor that fails everything is
# as useless as one that fails nothing.
report = audit(BrokenAssistant)
assert sorted(report.failures()) == [
"erasure_is_complete",
"redaction_before_indexing",
"shared_budget",
]
# ------------------------------------------------ the reference assistant
def test_conformant_ingest_is_idempotent():
bot = ConformantAssistant()
bot.ingest(sample_docs())
first = dict(bot.chunks)
bot.ingest(sample_docs())
assert bot.chunks == first
def test_dead_letter_advances_the_cursor():
bot = ConformantAssistant()
bot.ingest(sample_docs())
# The third document has no body; the cursor must still pass it.
assert bot.cursor == 3
def test_conformant_erase_clears_all_three_stores():
bot = ConformantAssistant()
bot.ingest(sample_docs())
bot.answer("What is the refund window?")
bot.erase("policy")
assert not any(c.startswith("policy::") for c in bot.chunks)
assert "policy" not in bot.hashes
assert not any("policy" in a for a in bot.cache.values())
def test_shrinking_document_leaves_no_orphans():
bot = ConformantAssistant()
bot.ingest([Doc("long", 1, "alpha beta gamma delta " * 30)])
many = sum(1 for c in bot.chunks if c.startswith("long::"))
bot.ingest([Doc("long", 2, "alpha beta")])
few = sum(1 for c in bot.chunks if c.startswith("long::"))
assert 1 == few < many
def test_cached_answer_is_free_and_recorded():
bot = ConformantAssistant()
bot.ingest(sample_docs())
bot.answer("q")
spend = sum(c for _, c in bot.ledger)
bot.answer("q")
assert sum(c for _, c in bot.ledger) == spend
assert any(stage == "cache" for stage, _ in bot.ledger)
Troubleshooting
Troubleshooting — Day 329
The broken assistant passes a check it should fail
Your check is looking at the wrong place. Three specific traps:
redaction_before_indexingmust inspectbot.chunks, not the source documents. The broken assistant redacts nothing, so the identifier is in the chunk text.shared_budgetmust inspectbot.ledger. The broken assistant records retrieval intoside_ledger, which is exactly the defect — a paid stage the cap does not see.erasure_is_completemust read all three stores back after erasing. The broken assistant clears the chunks and hashes correctly and leaves the cache, so a check that stops at the index passes.
The audit fails checks on the conformant assistant
Usually a fixture problem rather than a check problem. check_erasure_is_complete must ask a question before erasing, or the cache is empty and the check proves nothing. check_no_orphans_on_shrink needs the first document to produce more than one chunk, or there is nothing to orphan.
test_it_names_exactly_the_three_planted_defects fails
Your audit is failing more than the three planted defects. Precision matters as much as recall: an auditor that fails everything is as useless as one that fails nothing. Look at which extra check fired and why — most often it is a check that reports failure on an empty result rather than on a genuine violation.
A failing check has no detail
Every FAIL should name what was found — the offending chunk ids, the missing ledger stage, the stores still holding the subject. A bare FAIL is a bug report nobody can act on. Build the detail string from the actual finding, not a fixed message.
AttributeError on side_ledger
Only BrokenAssistant has it. Your check is reaching for it directly instead of inspecting bot.ledger, which is the surface the AssistantLike protocol defines. Audit the protocol, not the implementation.
The cursor check passes on both assistants
That is correct. BrokenAssistant inherits the cursor behaviour and is not broken in that respect — only three of the five properties are violated. If your version fails it on the conformant assistant, check that you are comparing against seq 3, the dead-lettered document.
NotImplementedError on most tests
Expected. The starter stubs all five check functions — see expected-output/starter-run.txt, which also names the five tests that pass without them.
Security notes
Security notes — Day 329
What this lab does and does not touch
- No network. Everything is in-process. No test opens a socket.
- No credentials. No API key, token or account.
- No writes outside this directory. Only
__pycache__/,.pytest_cache/and, if you create it,.venv/. - No
sudo, no global installs. - No real personal data. The single identifier in the fixtures uses
example.com, a domain reserved by RFC 2606.
An auditor is a security control, and it needs its own evidence
The checks here encode three properties that are security or privacy properties rather than functional ones:
redaction_before_indexing— identifiers inside embeddings can only be removed by re-embedding, so this ordering determines whether a future incident is a config change or a corpus rebuild.shared_budget— a paid stage with separate accounting is a path the spend cap does not bound, which is how unbounded consumption gets in.erasure_is_complete— a deletion that reaches some stores and not others leaves a system able to answer from content someone had a right to have removed.
The important design decision is the BrokenAssistant. A checker that has never failed anything provides no assurance: it may be correct, or it may be checking nothing. Planting known defects and asserting that the audit finds exactly those — no fewer, and no more — is what makes the auditor's own passing result meaningful.
That is worth generalising. Any test that guards a security property should have a companion proving it fails when the property is violated, or it is decoration.
What this lab does not do
It does not sandbox anything, and it audits an in-process object rather than a running service. A real conformance harness would run against a deployed system, which brings authentication, rate limits and test-data handling with it — all of which are out of scope here and none of which change the checks themselves.
Reversing everything this lab did
rm -rf __pycache__ .pytest_cache .venv
Nothing else was created, and nothing outside this directory was modified.