AI Engineering: Agents and Applications › Production Retrieval and Pipelines › Day 329

Hands-on lab — Day 329: Section Project: A Production Assistant

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/seam_audit_demo.py

Test

bash tests/run_tests.sh

File tree

examples/seam_audit_demo.py
examples/seam_audit.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/seam_audit.py
tests/run_tests.sh
tests/test_seam_audit.py
troubleshooting.md

Lab README

Lab — Day 329: Section Project: A Production Assistant

Lesson

Purpose

Build a conformance auditor for the three seams of a production assistant.

The week 47 project builds an assistant. This lab checks one — a different skill, and the one you need when reviewing someone else's system, or your own six months later. Each check targets an interaction no single stage's tests can reach.

The auditor is run against two assistants: one that conforms, and one deliberately broken in three specific ways. The second matters more, because an auditor that has never caught anything is not evidence of anything.

Learning objectives

  • Express a system property as an executable check rather than a review comment.
  • Verify that redaction precedes indexing, by inspecting what actually reached the chunks.
  • Verify that retrieval and generation share one ledger, so no paid path escapes the cap.
  • Verify an erasure by reading back from every store rather than trusting the delete calls.
  • Demonstrate that a checker catches real defects, and exactly those defects.

Prerequisites

  • Days 323 to 328 — each check corresponds to one of them.
  • Comfortable with Python dataclasses, and able to read a Protocol.

Supported operating systems

macOS, Linux and Windows (via WSL or PowerShell). Nothing is platform-specific.

Hardware requirements

Any machine running Python 3.10 or newer. No GPU. The suite completes in hundredths of a second.

Required software

  • Python 3.10 or newer.
  • pytest 7 or newer.

Free and open-source options

Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no API key, no network.

Installation

python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt

Removing .venv fully reverses the installation.

File structure

starter/seam_audit.py         your work: the five check functions
examples/seam_audit.py        reference auditor plus two assistants to audit
examples/seam_audit_demo.py   audits both and prints the verdicts
tests/test_seam_audit.py      the auditor passes good, catches broken
tests/run_tests.sh            suite entry point
expected-output/              real captured output and measured values
requirements/                 pinned dependency

How to run

python3 examples/seam_audit_demo.py   # audit both assistants
bash tests/run_tests.sh               # run the suite

To work on the exercise, edit starter/seam_audit.py, then copy it over the reference:

cp starter/seam_audit.py examples/seam_audit.py
bash tests/run_tests.sh

Keep a copy of the reference first if you want to compare.

What the commands do

python3 examples/seam_audit_demo.py runs all five checks against ConformantAssistant and then against BrokenAssistant, printing a line per check and a verdict for each.

bash tests/run_tests.sh runs pytest over twelve tests in three groups: the auditor passes a conformant system, the auditor catches a broken one, and the reference assistant itself behaves correctly.

Expected output

--- conformant assistant ---
  PASS  redaction_before_indexing
  PASS  shared_budget
  PASS  erasure_is_complete
  PASS  cursor_advances_on_failure
  PASS  no_orphans_on_shrink
CONFORMANT (5/5 checks passed)
--- broken assistant ---
  FAIL  redaction_before_indexing -- 2 chunk(s) contain an address: ['policy::0', 'policy::1']
  FAIL  shared_budget -- retrieval is not charged to the request ledger
  FAIL  erasure_is_complete -- still present in: cache
  PASS  cursor_advances_on_failure
  PASS  no_orphans_on_shrink
NON-CONFORMANT (2/5 checks passed)

expected-output/FIELDS.md explains each field and why two assistants are audited.

Validation steps

  1. bash tests/run_tests.sh reports 12 passed.
  2. The conformant assistant must pass all five checks; the broken one must fail exactly three — the three planted defects and no others. An auditor that fails everything is as useless as one that fails nothing.
  3. Every FAIL must carry a detail naming what was found. A failure without a detail is a bug report nobody can act on.

Tests

Twelve tests in three groups:

  • the auditor passes good — a conformant assistant is CONFORMANT, and all five checks actually run.
  • the auditor catches broken — each of the three planted defects is detected with an informative detail, and the failure list is exactly those three.
  • the reference assistant — idempotent ingest, cursor advances past a dead letter, erasure clears all three stores, a shrinking document leaves no orphans, and a cached answer is free and recorded.

Cleanup

rm -rf __pycache__ .pytest_cache
rm -rf .venv

Both are safe to remove and fully reversible.

Troubleshooting

See troubleshooting.md.

Security notes

See security.md. In short: no network, no credentials, and the only identifier in the fixtures uses the reserved example.com domain.

Extension exercises

  1. A fourth defect. Add a BrokenAssistant variant that leaves orphans on shrink, and confirm the audit catches it without any change to the checks.
  2. Audit the week 47 project. Point the auditor at the assistant you built for the project. It should be conformant — and if it is not, you have found something worth fixing.
  3. Severity. Give each check a severity and make the verdict distinguish a warning from a blocker, so an audit can gate a deployment rather than only report.
  • Lesson
  • Previous: Day 328 — Privacy in AI Systems
  • Next: Day 330 — Docker Fundamentals

Expected output

FIELDS.md

# Reading the audit output

## A check line

      FAIL  erasure_is_complete -- still present in: cache

| Part | Meaning |
| --- | --- |
| `PASS` / `FAIL` | Whether this seam holds in the assistant under audit. |
| name | Which interaction was checked. |
| detail | Present only on failure, and names *what* was found — the offending chunk ids, the missing ledger stage, or the stores that still hold the subject. A failure without a detail is a bug report nobody can act on. |

## The summary line

    NON-CONFORMANT (2/5 checks passed)

The verdict plus the count. `CONFORMANT` requires every check to pass — these are properties, not a score to average.

## The five checks

| Check | The failure it catches |
| --- | --- |
| `redaction_before_indexing` | Identifiers reached the chunks, so they are inside the vectors and removing them means re-embedding. |
| `shared_budget` | Retrieval has its own accounting, so it is a path the spend cap does not bound. |
| `erasure_is_complete` | A deletion reached some stores and not others — usually the cache, which then keeps answering from deleted content. |
| `cursor_advances_on_failure` | A dead-lettered document blocks the head of the queue and is retried on every run. |
| `no_orphans_on_shrink` | A shorter document left stale chunks behind, still retrievable. |

## Why two assistants

The demo audits a conformant assistant and a deliberately broken one. The second is the important half: **an auditor that has never caught anything is not evidence of anything.** The broken assistant is wrong in exactly three ways, and the audit is expected to name exactly those three — no fewer, and no more.

examples-run.txt

--- conformant assistant ---
  PASS  redaction_before_indexing
  PASS  shared_budget
  PASS  erasure_is_complete
  PASS  cursor_advances_on_failure
  PASS  no_orphans_on_shrink
CONFORMANT (5/5 checks passed)
--- broken assistant ---
  FAIL  redaction_before_indexing -- 2 chunk(s) contain an address: ['policy::0', 'policy::1']
  FAIL  shared_budget -- retrieval is not charged to the request ledger
  FAIL  erasure_is_complete -- still present in: cache
  PASS  cursor_advances_on_failure
  PASS  no_orphans_on_shrink
NON-CONFORMANT (2/5 checks passed)

measured-values.txt

Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.

  python3 examples/seam_audit_demo.py -> 14 lines, exit 0
  bash tests/run_tests.sh             -> 12 passed in 0.01s

The conformant assistant passes all five checks. The broken one fails exactly
three -- the three defects planted in it, and no others:

  FAIL  redaction_before_indexing -- 2 chunk(s) contain an address
  FAIL  shared_budget             -- retrieval is not charged to the ledger
  FAIL  erasure_is_complete       -- still present in: cache
  PASS  cursor_advances_on_failure
  PASS  no_orphans_on_shrink

Precision matters as much as recall here. An auditor that fails everything is
as useless as one that fails nothing, which is why
test_it_names_exactly_the_three_planted_defects asserts the failure list
exactly rather than merely asserting non-conformance.

run-output.txt

--- conformant assistant ---
  PASS  redaction_before_indexing
  PASS  shared_budget
  PASS  erasure_is_complete
  PASS  cursor_advances_on_failure
  PASS  no_orphans_on_shrink
CONFORMANT (5/5 checks passed)
--- broken assistant ---
  FAIL  redaction_before_indexing -- 2 chunk(s) contain an address: ['policy::0', 'policy::1']
  FAIL  shared_budget -- retrieval is not charged to the request ledger
  FAIL  erasure_is_complete -- still present in: cache
  PASS  cursor_advances_on_failure
  PASS  no_orphans_on_shrink
NON-CONFORMANT (2/5 checks passed)

starter-run.txt

An untouched starter fails most of the suite:

    7 failed, 5 passed in 0.04s

The five that pass exercise ConformantAssistant directly -- idempotent ingest,
cursor advance, erasure, orphan cleanup and the free cached answer -- none of
which depends on the five stubbed check functions.

Everything that runs an audit fails until the checks are implemented.

test-output.txt

............                                                             [100%]
12 passed in 0.01s

test-run.txt

............                                                             [100%]
12 passed in 0.01s

Source files

examples/seam_audit_demo.py (674 bytes)
#!/usr/bin/env python3
"""Audit two assistants: one that conforms, one that does not."""

from __future__ import annotations

import os
import sys

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

from seam_audit import BrokenAssistant, ConformantAssistant, audit


def main() -> int:
    for label, factory in (
        ("conformant assistant", ConformantAssistant),
        ("broken assistant", BrokenAssistant),
    ):
        report = audit(factory)
        print(f"--- {label} ---")
        for check in report.checks:
            print(check.line())
        print(report.summary())
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
examples/seam_audit.py (9835 bytes)
"""A conformance auditor for the three seams of a production assistant.

The week 47 project builds an assistant. This lab checks one -- which is a
different skill, and the one you need when reviewing someone else's system or
your own six months later.

Each check targets an interaction that no single stage's tests can reach:

  redaction_before_indexing   identifiers must not survive into any chunk
  shared_budget               retrieval must charge the same ledger as
                              generation, or one path escapes the cap
  erasure_is_complete         a deletion must reach the index, the content
                              hashes AND the response cache
  cursor_advances_on_failure  a dead-lettered document must not block the head
                              of the queue forever
  no_orphans_on_shrink        a shorter document must not leave stale chunks

The audit works against any object exposing the small protocol below, so it can
be pointed at a real implementation. Two are supplied: one that conforms, and
one that is deliberately broken in three specific ways -- because an auditor
that has never caught anything is not evidence of anything.
"""

from __future__ import annotations

import hashlib
import re
from dataclasses import dataclass, field
from typing import Protocol

EMAIL = re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b")


@dataclass(frozen=True)
class Doc:
    doc_id: str
    seq: int
    body: str | None


class AssistantLike(Protocol):
    """The surface an assistant must expose to be auditable."""

    chunks: dict[str, str]  # chunk_id -> text
    hashes: dict[str, str]  # doc_id -> content hash
    cache: dict[str, str]  # question -> answer
    cursor: int
    ledger: list[tuple[str, float]]  # (stage, cost)

    def ingest(self, docs: list[Doc]) -> None: ...
    def answer(self, question: str) -> str: ...
    def erase(self, doc_id: str) -> None: ...


@dataclass
class Check:
    name: str
    passed: bool
    detail: str = ""

    def line(self) -> str:
        mark = "PASS" if self.passed else "FAIL"
        return f"  {mark}  {self.name}" + (f" -- {self.detail}" if self.detail else "")


@dataclass
class AuditReport:
    checks: list[Check] = field(default_factory=list)

    @property
    def conformant(self) -> bool:
        return all(c.passed for c in self.checks)

    def failures(self) -> list[str]:
        return [c.name for c in self.checks if not c.passed]

    def summary(self) -> str:
        passed = sum(1 for c in self.checks if c.passed)
        verdict = "CONFORMANT" if self.conformant else "NON-CONFORMANT"
        return f"{verdict} ({passed}/{len(self.checks)} checks passed)"


def sample_docs() -> list[Doc]:
    return [
        Doc("policy", 1, "The refund window is thirty days. Write to ada@example.com. " * 3),
        Doc("sla", 2, "Uptime guarantee is 99.99 percent per calendar month. " * 3),
        Doc("broken", 3, None),
    ]


# ------------------------------------------------------------------ checks


def check_redaction_before_indexing(make: type) -> Check:
    bot = make()
    bot.ingest(sample_docs())
    leaked = [cid for cid, text in bot.chunks.items() if EMAIL.search(text)]
    return Check(
        "redaction_before_indexing",
        not leaked,
        f"{len(leaked)} chunk(s) contain an address: {leaked[:2]}" if leaked else "",
    )


def check_shared_budget(make: type) -> Check:
    bot = make()
    bot.ingest(sample_docs())
    bot.answer("What is the refund window?")
    stages = {stage for stage, _ in bot.ledger}
    return Check(
        "shared_budget",
        "retrieval" in stages,
        "retrieval is not charged to the request ledger" if "retrieval" not in stages else "",
    )


def check_erasure_is_complete(make: type) -> Check:
    bot = make()
    bot.ingest(sample_docs())
    bot.answer("What is the refund window?")
    bot.erase("policy")

    remaining = []
    if any(cid.startswith("policy::") for cid in bot.chunks):
        remaining.append("index")
    if "policy" in bot.hashes:
        remaining.append("hashes")
    if any("policy" in answer for answer in bot.cache.values()):
        remaining.append("cache")
    return Check(
        "erasure_is_complete",
        not remaining,
        f"still present in: {', '.join(remaining)}" if remaining else "",
    )


def check_cursor_advances_on_failure(make: type) -> Check:
    bot = make()
    bot.ingest(sample_docs())
    before = bot.cursor
    bot.ingest(sample_docs())
    return Check(
        "cursor_advances_on_failure",
        before >= 3,
        f"cursor stuck at {before}; the dead-lettered document blocks the queue"
        if before < 3
        else "",
    )


def check_no_orphans_on_shrink(make: type) -> Check:
    bot = make()
    bot.ingest([Doc("long", 1, "alpha beta gamma delta " * 30)])
    many = sum(1 for cid in bot.chunks if cid.startswith("long::"))
    bot.ingest([Doc("long", 2, "alpha beta")])
    ids = {cid for cid in bot.chunks if cid.startswith("long::")}
    expected = {f"long::{i}" for i in range(len(ids))}
    return Check(
        "no_orphans_on_shrink",
        many > 1 and ids == expected,
        f"orphans left behind: {sorted(ids - expected)}" if ids != expected else "",
    )


CHECKS = (
    check_redaction_before_indexing,
    check_shared_budget,
    check_erasure_is_complete,
    check_cursor_advances_on_failure,
    check_no_orphans_on_shrink,
)


def audit(make: type) -> AuditReport:
    """Run every seam check against a factory producing a fresh assistant."""
    return AuditReport(checks=[check(make) for check in CHECKS])


# ------------------------------------------------- two assistants to audit


def _digest(text: str) -> str:
    return hashlib.sha256(" ".join(text.split()).encode()).hexdigest()


@dataclass
class ConformantAssistant:
    """Does all three seams correctly."""

    chunks: dict[str, str] = field(default_factory=dict)
    hashes: dict[str, str] = field(default_factory=dict)
    cache: dict[str, str] = field(default_factory=dict)
    cursor: int = 0
    ledger: list[tuple[str, float]] = field(default_factory=list)

    def ingest(self, docs: list[Doc]) -> None:
        for doc in sorted((d for d in docs if d.seq > self.cursor), key=lambda d: d.seq):
            if doc.body is None:
                self.cursor = doc.seq  # advance anyway, or the queue head blocks
                continue
            clean = EMAIL.sub("[email]", doc.body)  # redact BEFORE anything else
            digest = _digest(clean)
            if self.hashes.get(doc.doc_id) == digest:
                self.cursor = doc.seq
                continue
            words = clean.split()
            fresh = set()
            for position, start in enumerate(range(0, len(words), 20)):
                cid = f"{doc.doc_id}::{position}"
                self.chunks[cid] = " ".join(words[start : start + 20])
                fresh.add(cid)
            for stale in [
                c for c in list(self.chunks) if c.startswith(f"{doc.doc_id}::") and c not in fresh
            ]:
                del self.chunks[stale]
            self.hashes[doc.doc_id] = digest
            self.cursor = doc.seq

    def answer(self, question: str) -> str:
        if question in self.cache:
            self.ledger.append(("cache", 0.0))
            return self.cache[question]
        self.ledger.append(("retrieval", 0.000002))  # same ledger as generation
        self.ledger.append(("small", 0.00002))
        hit = next((c for c in self.chunks if c.startswith("policy")), "none")
        reply = f"answer citing {hit}"
        self.cache[question] = reply
        return reply

    def erase(self, doc_id: str) -> None:
        for cid in [c for c in list(self.chunks) if c.startswith(f"{doc_id}::")]:
            del self.chunks[cid]
        self.hashes.pop(doc_id, None)
        self.cache = {q: a for q, a in self.cache.items() if doc_id not in a}


@dataclass
class BrokenAssistant(ConformantAssistant):
    """Broken in three specific ways, so the audit has something to catch.

    1. redacts after chunking, so identifiers reach the index
    2. charges retrieval to a separate counter, so it escapes the cap
    3. erases the index but forgets the cache
    """

    side_ledger: list[tuple[str, float]] = field(default_factory=list)

    def ingest(self, docs: list[Doc]) -> None:
        for doc in sorted((d for d in docs if d.seq > self.cursor), key=lambda d: d.seq):
            if doc.body is None:
                self.cursor = doc.seq
                continue
            words = doc.body.split()  # NOT redacted
            fresh = set()
            for position, start in enumerate(range(0, len(words), 20)):
                cid = f"{doc.doc_id}::{position}"
                self.chunks[cid] = " ".join(words[start : start + 20])
                fresh.add(cid)
            for stale in [
                c for c in list(self.chunks) if c.startswith(f"{doc.doc_id}::") and c not in fresh
            ]:
                del self.chunks[stale]
            self.hashes[doc.doc_id] = _digest(doc.body)
            self.cursor = doc.seq

    def answer(self, question: str) -> str:
        if question in self.cache:
            self.ledger.append(("cache", 0.0))
            return self.cache[question]
        self.side_ledger.append(("retrieval", 0.000002))  # escapes the cap
        self.ledger.append(("small", 0.00002))
        hit = next((c for c in self.chunks if c.startswith("policy")), "none")
        reply = f"answer citing {hit}"
        self.cache[question] = reply
        return reply

    def erase(self, doc_id: str) -> None:
        for cid in [c for c in list(self.chunks) if c.startswith(f"{doc_id}::")]:
            del self.chunks[cid]
        self.hashes.pop(doc_id, None)
        # cache deliberately untouched
metadata.yml (853 bytes)
lesson_id: D329
day: 329
kind: python
languages:
  - python
setup_commands:
  - pip install -r requirements/requirements.txt
run_commands:
  - python examples/seam_audit_demo.py
test_commands:
  - bash tests/run_tests.sh
cleanup_commands:
  - rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 60
last_executed: '2026-08-31'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3
  examples/seam_audit_demo.py reproduced the documented verdicts exactly, with the broken assistant
  failing exactly the three planted defects; bash tests/run_tests.sh -> 12 passed in 0.01s. The untouched
  starter fails 7 of 12; the 5 that pass exercise the reference assistant rather than the stubbed checks.
  requires_network is false: everything is in-process and no test opens a socket.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/seam_audit.py (9205 bytes)
"""A conformance auditor for the three seams of a production assistant.

The week 47 project builds an assistant. This lab checks one -- which is a
different skill, and the one you need when reviewing someone else's system or
your own six months later.

Each check targets an interaction that no single stage's tests can reach:

  redaction_before_indexing   identifiers must not survive into any chunk
  shared_budget               retrieval must charge the same ledger as
                              generation, or one path escapes the cap
  erasure_is_complete         a deletion must reach the index, the content
                              hashes AND the response cache
  cursor_advances_on_failure  a dead-lettered document must not block the head
                              of the queue forever
  no_orphans_on_shrink        a shorter document must not leave stale chunks

The audit works against any object exposing the small protocol below, so it can
be pointed at a real implementation. Two are supplied: one that conforms, and
one that is deliberately broken in three specific ways -- because an auditor
that has never caught anything is not evidence of anything.
"""

from __future__ import annotations

import hashlib
import re
from dataclasses import dataclass, field
from typing import Protocol

EMAIL = re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b")


@dataclass(frozen=True)
class Doc:
    doc_id: str
    seq: int
    body: str | None


class AssistantLike(Protocol):
    """The surface an assistant must expose to be auditable."""

    chunks: dict[str, str]  # chunk_id -> text
    hashes: dict[str, str]  # doc_id -> content hash
    cache: dict[str, str]  # question -> answer
    cursor: int
    ledger: list[tuple[str, float]]  # (stage, cost)

    def ingest(self, docs: list[Doc]) -> None: ...
    def answer(self, question: str) -> str: ...
    def erase(self, doc_id: str) -> None: ...


@dataclass
class Check:
    name: str
    passed: bool
    detail: str = ""

    def line(self) -> str:
        mark = "PASS" if self.passed else "FAIL"
        return f"  {mark}  {self.name}" + (f" -- {self.detail}" if self.detail else "")


@dataclass
class AuditReport:
    checks: list[Check] = field(default_factory=list)

    @property
    def conformant(self) -> bool:
        return all(c.passed for c in self.checks)

    def failures(self) -> list[str]:
        return [c.name for c in self.checks if not c.passed]

    def summary(self) -> str:
        passed = sum(1 for c in self.checks if c.passed)
        verdict = "CONFORMANT" if self.conformant else "NON-CONFORMANT"
        return f"{verdict} ({passed}/{len(self.checks)} checks passed)"


def sample_docs() -> list[Doc]:
    return [
        Doc("policy", 1, "The refund window is thirty days. Write to ada@example.com. " * 3),
        Doc("sla", 2, "Uptime guarantee is 99.99 percent per calendar month. " * 3),
        Doc("broken", 3, None),
    ]


# ------------------------------------------------------------------ checks


def check_redaction_before_indexing(make: type) -> Check:
    # TASK 1: ingest sample_docs() into a fresh assistant and assert that no
    # chunk text still matches EMAIL. Report the offending chunk ids in detail.
    raise NotImplementedError("implement check_redaction_before_indexing")

def check_shared_budget(make: type) -> Check:
    # TASK 2: ingest, ask one question, then assert "retrieval" appears among
    # the ledger stages. A retrieval stage with its own counter is a path the
    # spend cap does not bound.
    raise NotImplementedError("implement check_shared_budget")

def check_erasure_is_complete(make: type) -> Check:
    # TASK 3: ingest, ask a question so the cache holds an answer, erase
    # "policy", then check ALL THREE stores by reading back: chunks, hashes and
    # cache. Name whichever still hold it in detail.
    raise NotImplementedError("implement check_erasure_is_complete")

def check_cursor_advances_on_failure(make: type) -> Check:
    # TASK 4: ingest twice and assert the cursor passed the dead-lettered
    # document (seq 3). A cursor that stops there retries the poison record on
    # every run and the queue head never moves.
    raise NotImplementedError("implement check_cursor_advances_on_failure")

def check_no_orphans_on_shrink(make: type) -> Check:
    # TASK 5: ingest a long document, then a much shorter one with the same
    # doc_id, and assert the surviving chunk ids are exactly long::0 .. long::n-1
    # with no gaps -- otherwise stale chunks are still retrievable.
    raise NotImplementedError("implement check_no_orphans_on_shrink")

CHECKS = (
    check_redaction_before_indexing,
    check_shared_budget,
    check_erasure_is_complete,
    check_cursor_advances_on_failure,
    check_no_orphans_on_shrink,
)


def audit(make: type) -> AuditReport:
    """Run every seam check against a factory producing a fresh assistant."""
    return AuditReport(checks=[check(make) for check in CHECKS])


# ------------------------------------------------- two assistants to audit


def _digest(text: str) -> str:
    return hashlib.sha256(" ".join(text.split()).encode()).hexdigest()


@dataclass
class ConformantAssistant:
    """Does all three seams correctly."""

    chunks: dict[str, str] = field(default_factory=dict)
    hashes: dict[str, str] = field(default_factory=dict)
    cache: dict[str, str] = field(default_factory=dict)
    cursor: int = 0
    ledger: list[tuple[str, float]] = field(default_factory=list)

    def ingest(self, docs: list[Doc]) -> None:
        for doc in sorted((d for d in docs if d.seq > self.cursor), key=lambda d: d.seq):
            if doc.body is None:
                self.cursor = doc.seq  # advance anyway, or the queue head blocks
                continue
            clean = EMAIL.sub("[email]", doc.body)  # redact BEFORE anything else
            digest = _digest(clean)
            if self.hashes.get(doc.doc_id) == digest:
                self.cursor = doc.seq
                continue
            words = clean.split()
            fresh = set()
            for position, start in enumerate(range(0, len(words), 20)):
                cid = f"{doc.doc_id}::{position}"
                self.chunks[cid] = " ".join(words[start : start + 20])
                fresh.add(cid)
            for stale in [
                c for c in list(self.chunks) if c.startswith(f"{doc.doc_id}::") and c not in fresh
            ]:
                del self.chunks[stale]
            self.hashes[doc.doc_id] = digest
            self.cursor = doc.seq

    def answer(self, question: str) -> str:
        if question in self.cache:
            self.ledger.append(("cache", 0.0))
            return self.cache[question]
        self.ledger.append(("retrieval", 0.000002))  # same ledger as generation
        self.ledger.append(("small", 0.00002))
        hit = next((c for c in self.chunks if c.startswith("policy")), "none")
        reply = f"answer citing {hit}"
        self.cache[question] = reply
        return reply

    def erase(self, doc_id: str) -> None:
        for cid in [c for c in list(self.chunks) if c.startswith(f"{doc_id}::")]:
            del self.chunks[cid]
        self.hashes.pop(doc_id, None)
        self.cache = {q: a for q, a in self.cache.items() if doc_id not in a}


@dataclass
class BrokenAssistant(ConformantAssistant):
    """Broken in three specific ways, so the audit has something to catch.

    1. redacts after chunking, so identifiers reach the index
    2. charges retrieval to a separate counter, so it escapes the cap
    3. erases the index but forgets the cache
    """

    side_ledger: list[tuple[str, float]] = field(default_factory=list)

    def ingest(self, docs: list[Doc]) -> None:
        for doc in sorted((d for d in docs if d.seq > self.cursor), key=lambda d: d.seq):
            if doc.body is None:
                self.cursor = doc.seq
                continue
            words = doc.body.split()  # NOT redacted
            fresh = set()
            for position, start in enumerate(range(0, len(words), 20)):
                cid = f"{doc.doc_id}::{position}"
                self.chunks[cid] = " ".join(words[start : start + 20])
                fresh.add(cid)
            for stale in [
                c for c in list(self.chunks) if c.startswith(f"{doc.doc_id}::") and c not in fresh
            ]:
                del self.chunks[stale]
            self.hashes[doc.doc_id] = _digest(doc.body)
            self.cursor = doc.seq

    def answer(self, question: str) -> str:
        if question in self.cache:
            self.ledger.append(("cache", 0.0))
            return self.cache[question]
        self.side_ledger.append(("retrieval", 0.000002))  # escapes the cap
        self.ledger.append(("small", 0.00002))
        hit = next((c for c in self.chunks if c.startswith("policy")), "none")
        reply = f"answer citing {hit}"
        self.cache[question] = reply
        return reply

    def erase(self, doc_id: str) -> None:
        for cid in [c for c in list(self.chunks) if c.startswith(f"{doc_id}::")]:
            del self.chunks[cid]
        self.hashes.pop(doc_id, None)
        # cache deliberately untouched
tests/run_tests.sh (200 bytes)
#!/usr/bin/env bash
# Run the day 329 seam audit test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_seam_audit.py (3946 bytes)
"""Tests for the seam auditor.

An auditor needs two kinds of test: that it passes a conformant system, and --
more importantly -- that it FAILS a broken one. A checker that has never caught
anything is not evidence of anything.

Run with: bash tests/run_tests.sh
"""

from __future__ import annotations

import os
import sys

HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))

from seam_audit import (  # noqa: E402
    CHECKS,
    BrokenAssistant,
    ConformantAssistant,
    Doc,
    audit,
    check_erasure_is_complete,
    check_redaction_before_indexing,
    check_shared_budget,
    sample_docs,
)


# ------------------------------------------------- the auditor passes good


def test_conformant_assistant_passes_every_check():
    report = audit(ConformantAssistant)
    assert report.conformant, f"unexpected failures: {report.failures()}"
    assert report.summary() == "CONFORMANT (5/5 checks passed)"


def test_every_check_is_actually_run():
    report = audit(ConformantAssistant)
    assert len(report.checks) == len(CHECKS) == 5
    names = {c.name for c in report.checks}
    assert names == {
        "redaction_before_indexing",
        "shared_budget",
        "erasure_is_complete",
        "cursor_advances_on_failure",
        "no_orphans_on_shrink",
    }


# ------------------------------------------- the auditor catches the broken


def test_broken_assistant_is_reported_non_conformant():
    report = audit(BrokenAssistant)
    assert not report.conformant


def test_it_catches_redaction_running_after_indexing():
    check = check_redaction_before_indexing(BrokenAssistant)
    assert not check.passed
    assert "address" in check.detail


def test_it_catches_retrieval_escaping_the_shared_budget():
    check = check_shared_budget(BrokenAssistant)
    assert not check.passed
    assert "retrieval" in check.detail


def test_it_catches_an_erasure_that_forgets_the_cache():
    check = check_erasure_is_complete(BrokenAssistant)
    assert not check.passed
    assert "cache" in check.detail


def test_it_names_exactly_the_three_planted_defects():
    # Precision matters as much as recall: an auditor that fails everything is
    # as useless as one that fails nothing.
    report = audit(BrokenAssistant)
    assert sorted(report.failures()) == [
        "erasure_is_complete",
        "redaction_before_indexing",
        "shared_budget",
    ]


# ------------------------------------------------ the reference assistant


def test_conformant_ingest_is_idempotent():
    bot = ConformantAssistant()
    bot.ingest(sample_docs())
    first = dict(bot.chunks)
    bot.ingest(sample_docs())
    assert bot.chunks == first


def test_dead_letter_advances_the_cursor():
    bot = ConformantAssistant()
    bot.ingest(sample_docs())
    # The third document has no body; the cursor must still pass it.
    assert bot.cursor == 3


def test_conformant_erase_clears_all_three_stores():
    bot = ConformantAssistant()
    bot.ingest(sample_docs())
    bot.answer("What is the refund window?")
    bot.erase("policy")
    assert not any(c.startswith("policy::") for c in bot.chunks)
    assert "policy" not in bot.hashes
    assert not any("policy" in a for a in bot.cache.values())


def test_shrinking_document_leaves_no_orphans():
    bot = ConformantAssistant()
    bot.ingest([Doc("long", 1, "alpha beta gamma delta " * 30)])
    many = sum(1 for c in bot.chunks if c.startswith("long::"))
    bot.ingest([Doc("long", 2, "alpha beta")])
    few = sum(1 for c in bot.chunks if c.startswith("long::"))
    assert 1 == few < many


def test_cached_answer_is_free_and_recorded():
    bot = ConformantAssistant()
    bot.ingest(sample_docs())
    bot.answer("q")
    spend = sum(c for _, c in bot.ledger)
    bot.answer("q")
    assert sum(c for _, c in bot.ledger) == spend
    assert any(stage == "cache" for stage, _ in bot.ledger)

Troubleshooting

Troubleshooting — Day 329

The broken assistant passes a check it should fail

Your check is looking at the wrong place. Three specific traps:

  • redaction_before_indexing must inspect bot.chunks, not the source documents. The broken assistant redacts nothing, so the identifier is in the chunk text.
  • shared_budget must inspect bot.ledger. The broken assistant records retrieval into side_ledger, which is exactly the defect — a paid stage the cap does not see.
  • erasure_is_complete must read all three stores back after erasing. The broken assistant clears the chunks and hashes correctly and leaves the cache, so a check that stops at the index passes.

The audit fails checks on the conformant assistant

Usually a fixture problem rather than a check problem. check_erasure_is_complete must ask a question before erasing, or the cache is empty and the check proves nothing. check_no_orphans_on_shrink needs the first document to produce more than one chunk, or there is nothing to orphan.

test_it_names_exactly_the_three_planted_defects fails

Your audit is failing more than the three planted defects. Precision matters as much as recall: an auditor that fails everything is as useless as one that fails nothing. Look at which extra check fired and why — most often it is a check that reports failure on an empty result rather than on a genuine violation.

A failing check has no detail

Every FAIL should name what was found — the offending chunk ids, the missing ledger stage, the stores still holding the subject. A bare FAIL is a bug report nobody can act on. Build the detail string from the actual finding, not a fixed message.

AttributeError on side_ledger

Only BrokenAssistant has it. Your check is reaching for it directly instead of inspecting bot.ledger, which is the surface the AssistantLike protocol defines. Audit the protocol, not the implementation.

The cursor check passes on both assistants

That is correct. BrokenAssistant inherits the cursor behaviour and is not broken in that respect — only three of the five properties are violated. If your version fails it on the conformant assistant, check that you are comparing against seq 3, the dead-lettered document.

NotImplementedError on most tests

Expected. The starter stubs all five check functions — see expected-output/starter-run.txt, which also names the five tests that pass without them.

Security notes

Security notes — Day 329

What this lab does and does not touch

  • No network. Everything is in-process. No test opens a socket.
  • No credentials. No API key, token or account.
  • No writes outside this directory. Only __pycache__/, .pytest_cache/ and, if you create it, .venv/.
  • No sudo, no global installs.
  • No real personal data. The single identifier in the fixtures uses example.com, a domain reserved by RFC 2606.

An auditor is a security control, and it needs its own evidence

The checks here encode three properties that are security or privacy properties rather than functional ones:

  • redaction_before_indexing — identifiers inside embeddings can only be removed by re-embedding, so this ordering determines whether a future incident is a config change or a corpus rebuild.
  • shared_budget — a paid stage with separate accounting is a path the spend cap does not bound, which is how unbounded consumption gets in.
  • erasure_is_complete — a deletion that reaches some stores and not others leaves a system able to answer from content someone had a right to have removed.

The important design decision is the BrokenAssistant. A checker that has never failed anything provides no assurance: it may be correct, or it may be checking nothing. Planting known defects and asserting that the audit finds exactly those — no fewer, and no more — is what makes the auditor's own passing result meaningful.

That is worth generalising. Any test that guards a security property should have a companion proving it fails when the property is violated, or it is decoration.

What this lab does not do

It does not sandbox anything, and it audits an in-process object rather than a running service. A real conformance harness would run against a deployed system, which brings authentication, rate limits and test-data handling with it — all of which are out of scope here and none of which change the checks themselves.

Reversing everything this lab did

rm -rf __pycache__ .pytest_cache .venv

Nothing else was created, and nothing outside this directory was modified.