AI Engineering: Agents and Applications › Production Retrieval and Pipelines › Day 323

Hands-on lab — Day 323: Data Ingestion Pipelines

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/ingest_demo.py

Test

bash tests/run_tests.sh

File tree

examples/ingest_demo.py
examples/ingest.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/ingest.py
tests/run_tests.sh
tests/test_ingest.py
troubleshooting.md

Lab README

Lab — Day 323: Data Ingestion Pipelines

Lesson

  • Lesson title: Data Ingestion Pipelines
  • Day number: 323 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-323-data-ingestion-pipelines
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-323-data-ingestion-pipelines when the site is running.

Purpose

Build an incremental ingestion pipeline that is safe to schedule and forget. You implement content-hash change detection, stable chunk identifiers, a durable cursor committed in the correct order, and a dead-letter queue — the four mechanisms that make a re-run harmless instead of destructive.

Everything runs offline against an in-memory source and index. The endpoints are simulated; the mechanisms are real, which is what makes their behaviour testable.

Learning objectives

  • Derive a chunk identifier that is reproducible across runs, so a second write upserts rather than duplicates.
  • Prove a document is unchanged with a content hash rather than trusting modification metadata.
  • Order the cursor commit after the batch write, so a crash reprocesses instead of losing records.
  • Capture a failing record in a dead-letter queue and let the batch continue.
  • Delete orphaned chunks when a document shrinks.

Prerequisites

  • Day 322, "A Full-Stack AI Application".
  • Comfortable with Python dataclasses, dictionaries and exceptions.
  • Familiar with pytest basics.

Supported operating systems

macOS, Linux and Windows (via WSL or PowerShell). Nothing in this lab is platform-specific.

Hardware requirements

Any machine that runs Python 3.10 or newer. No GPU. The full suite finishes in well under a second and uses a few megabytes of memory.

Required software

  • Python 3.10 or newer (uses X | None type syntax).
  • pytest 7 or newer.

Free and open-source options

Every dependency is free and open source. Python is PSF-licensed; pytest is MIT-licensed. There is no paid tier, no account and no hosted service anywhere in this lab.

Installation

Build a virtual environment inside this lab directory so nothing touches your system Python:

python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt

Removing .venv fully reverses the installation.

File structure

starter/ingest.py          your work: content_hash, chunk_document, run_once
examples/ingest.py         reference implementation
examples/ingest_demo.py    three consecutive runs over a changing source
tests/test_ingest.py       one test per property
tests/run_tests.sh         suite entry point
expected-output/           real captured output, including the starter's failures
requirements/              pinned dependency

How to run

python3 examples/ingest_demo.py     # see the reference pipeline behave
bash tests/run_tests.sh             # run the suite

To work on the exercise, edit starter/ingest.py, then copy it over the reference to test your version:

cp starter/ingest.py examples/ingest.py
bash tests/run_tests.sh

Keep a copy of the reference first if you want to compare.

What the commands do

python3 examples/ingest_demo.py runs the pipeline three times over a source where one document is edited between runs and one document is permanently malformed. It prints a summary line per run plus the final index state.

bash tests/run_tests.sh changes into the lab root and runs python3 -m pytest tests -q. The suite is grouped by property, so a failure names which of idempotent, incremental, resumable or isolated is not yet satisfied.

Expected output

run 1: scanned=5 changed=4 embedded=7 indexed=7 dead_lettered=1 cursor=5
run 2: scanned=0 changed=0 embedded=0 indexed=0 dead_lettered=0 cursor=5
run 3: scanned=1 changed=1 embedded=2 indexed=2 dead_lettered=0 cursor=6
index: 7 chunks across 4 documents
dead letters: 1 (doc-3: ExtractionError)

Run 2 scans nothing because the cursor has not moved. expected-output/FIELDS.md explains every field; expected-output/starter-run.txt records what an untouched starter produces.

Validation steps

  1. bash tests/run_tests.sh reports 11 passed.
  2. Run python3 examples/ingest_demo.py twice in succession. The index size must be identical both times — if it grows, your chunk ids are not stable.
  3. Confirm run 2 reports embedded=0. If it does not, you are comparing timestamps rather than content hashes.

Tests

Eleven tests across five groups:

  • idempotent — a second run leaves the index identical; ids are stable across independent runs; reindexing from a clean checkpoint does not duplicate.
  • incremental — unchanged documents are not re-embedded even when re-presented; edited documents are; the hash ignores whitespace reformatting.
  • resumable — a simulated crash resumes from the last committed cursor; the cursor never advances past uncommitted work.
  • isolated — the failing record is dead-lettered with its error and the other four documents still index.
  • reconciliation — a shrinking document leaves no orphaned chunks.

Cleanup

rm -rf __pycache__ .pytest_cache
rm -rf .venv

Both are safe to remove and fully reversible: the virtual environment rebuilds from requirements/requirements.txt.

Troubleshooting

See troubleshooting.md for the failure modes this lab is designed to produce, including a doubling index, a run that re-embeds everything, and a crash that loses records.

Security notes

See security.md. In short: no network, no credentials, no API key, and every write stays inside this directory.

Extension exercises

  1. Deletion reconciliation. Add deleted_ids to the source and remove those documents' chunks. Then add a full reconciliation pass comparing all source ids against all indexed ids.
  2. Bounded concurrency. Extract with a small ThreadPoolExecutor while preserving the cursor guarantee — the cursor may only pass a record once it and everything before it has committed. Measure both versions and record the real numbers, including a null result.
  3. Retry policy. Give dead letters a retry count and an abandon threshold, so a transient failure recovers but a permanently poisoned record does not retry forever.
  • Lesson
  • Previous: Day 322 — A Full-Stack AI Application
  • Next: Day 324 — Document Processing at Scale

Expected output

FIELDS.md

# What each field in the run summary means

Each run prints one line in this shape:

    run N: scanned=A changed=B embedded=C indexed=D dead_lettered=E cursor=F

| Field | Meaning |
| --- | --- |
| `scanned` | Records with `seq` greater than the committed cursor that this run looked at. Zero means the cursor has not moved since the last run. |
| `changed` | Records whose content hash differed from the stored hash, so their body genuinely changed. A record that fails extraction is never hashed and so is never counted here. |
| `embedded` | Chunks passed to the embedding stage. In production this is the line item you pay for, which is why `changed` gates it. |
| `indexed` | Chunks written to the index. Equal to `embedded` in this lab because every embedded chunk is upserted. |
| `dead_lettered` | Records that failed extraction and were captured with their error so the run could continue. |
| `cursor` | The highest `seq` fully processed. Committed only after the batch is written, which is what makes a crash reprocess rather than lose work. |

The final two lines report the index size and the dead-letter contents:

    index: 7 chunks across 4 documents
    dead letters: 1 (doc-3: ExtractionError)

Four documents rather than five because `doc-3` has no payload and fails
extraction on every attempt.

examples-run.txt

run 1: scanned=5 changed=4 embedded=7 indexed=7 dead_lettered=1 cursor=5
run 2: scanned=0 changed=0 embedded=0 indexed=0 dead_lettered=0 cursor=5
run 3: scanned=1 changed=1 embedded=2 indexed=2 dead_lettered=0 cursor=6
index: 7 chunks across 4 documents
dead letters: 1 (doc-3: ExtractionError)

run-output.txt

run 1: scanned=5 changed=4 embedded=7 indexed=7 dead_lettered=1 cursor=5
run 2: scanned=0 changed=0 embedded=0 indexed=0 dead_lettered=0 cursor=5
run 3: scanned=1 changed=1 embedded=2 indexed=2 dead_lettered=0 cursor=6
index: 7 chunks across 4 documents
dead letters: 1 (doc-3: ExtractionError)

starter-run.txt

Before you implement the TODOs, the starter is expected to fail. Running

    bash tests/run_tests.sh

against an untouched starter/ingest.py produces:

    10 failed, 1 passed

The single pass is test_extract_raises_on_unreadable_payload, which exercises
extract() and does not depend on any of the stubbed functions. Every other test
fails with NotImplementedError until you complete content_hash, chunk_document
and run_once.

test-output.txt

...........                                                              [100%]
11 passed in 0.01s

test-run.txt

...........                                                              [100%]
11 passed in 0.01s

Source files

examples/ingest_demo.py (1753 bytes)
#!/usr/bin/env python3
"""Three consecutive runs over a source that changes between them.

Run 2 is the line worth reading: it scans nothing and embeds nothing, because
the cursor has not moved. Run 3 shows an edited document re-embedded and
upserted without changing the document count.
"""

from __future__ import annotations

import os
import sys

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

from ingest import Checkpoint, DeadLetter, Index, SourceRecord, run_once


def main() -> int:
    # doc-3 has no payload: it fails extraction on every attempt and must not
    # stop the other four documents from being indexed.
    source = [
        SourceRecord("doc-1", 1, "alpha " * 45),
        SourceRecord("doc-2", 2, "beta " * 80),
        SourceRecord("doc-3", 3, None),
        SourceRecord("doc-4", 4, "delta " * 30),
        SourceRecord("doc-5", 5, "epsilon " * 55),
    ]

    index = Index()
    checkpoint = Checkpoint()
    dead_letters: list[DeadLetter] = []

    stats1 = run_once(source, index, checkpoint, dead_letters)
    print(stats1.line("run 1"))

    # Nothing changed and the cursor has not moved.
    stats2 = run_once(source, index, checkpoint, dead_letters)
    print(stats2.line("run 2"))

    # An edited document arrives with a new sequence number.
    source.append(SourceRecord("doc-1", 6, "alpha revised " * 40))
    stats3 = run_once(source, index, checkpoint, dead_letters)
    print(stats3.line("run 3"))

    print(f"index: {len(index.chunks)} chunks across {len(index.doc_ids())} documents")
    detail = ", ".join(f"{dl.doc_id}: {dl.error}" for dl in dead_letters)
    print(f"dead letters: {len(dead_letters)} ({detail})")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
examples/ingest.py (7299 bytes)
"""A small but complete incremental ingestion pipeline.

Everything here runs offline with the standard library only. The source and the
index are in-memory fakes; the mechanisms around them -- content hashing, stable
chunk ids, a durable cursor and a dead-letter queue -- are the real thing, which
is what makes their behaviour testable.

The four properties the tests check:

  idempotent  running twice over unchanged sources leaves the index identical
  incremental a second run embeds only documents whose content hash changed
  resumable   a crash resumes from the last committed cursor
  isolated    a record that fails extraction is dead-lettered; the run continues
"""

from __future__ import annotations

import hashlib
from dataclasses import dataclass, field
from typing import Callable, Iterable


class ExtractionError(RuntimeError):
    """Raised when a source record cannot be turned into text."""


@dataclass(frozen=True)
class SourceRecord:
    """A record as the source system presents it."""

    doc_id: str
    seq: int  # monotonic; the cursor is expressed in these terms
    payload: str | None  # None simulates a document that cannot be extracted


@dataclass
class Chunk:
    chunk_id: str
    doc_id: str
    text: str


@dataclass
class DeadLetter:
    doc_id: str
    error: str


@dataclass
class RunStats:
    scanned: int = 0
    changed: int = 0
    embedded: int = 0
    indexed: int = 0
    dead_lettered: int = 0
    cursor: int = 0

    def line(self, label: str) -> str:
        return (
            f"{label}: scanned={self.scanned} changed={self.changed} "
            f"embedded={self.embedded} indexed={self.indexed} "
            f"dead_lettered={self.dead_lettered} cursor={self.cursor}"
        )


@dataclass
class Index:
    """A fake vector index with upsert semantics keyed on chunk_id."""

    chunks: dict[str, Chunk] = field(default_factory=dict)

    def upsert(self, chunks: Iterable[Chunk]) -> int:
        written = 0
        for chunk in chunks:
            self.chunks[chunk.chunk_id] = chunk
            written += 1
        return written

    def delete_orphans(self, doc_id: str, keep: set[str]) -> int:
        """Remove chunks of `doc_id` that are not in `keep`.

        A document that shrinks from twelve chunks to eight leaves four
        orphans behind. They stay retrievable and surface as stale answers, so
        reconciliation is part of a correct write, not an optimisation.
        """
        stale = [
            cid
            for cid, chunk in self.chunks.items()
            if chunk.doc_id == doc_id and cid not in keep
        ]
        for cid in stale:
            del self.chunks[cid]
        return len(stale)

    def doc_ids(self) -> set[str]:
        return {chunk.doc_id for chunk in self.chunks.values()}


@dataclass
class Checkpoint:
    """Durable-ish state: the cursor plus one content hash per document.

    Deliberately small. Storing extracted text here would make the checkpoint a
    second copy of the corpus.
    """

    cursor: int = 0
    hashes: dict[str, str] = field(default_factory=dict)


def content_hash(text: str) -> str:
    """Digest of the normalised body.

    Normalising before hashing matters: a source that rewrites whitespace on
    every read would otherwise look changed on every run.
    """
    normalised = " ".join(text.split())
    return hashlib.sha256(normalised.encode("utf-8")).hexdigest()


def extract(record: SourceRecord) -> str:
    """Turn a source record into text, or fail loudly.

    This is where a pipeline meets the real world's file formats, so it is the
    stage most likely to raise. Callers must treat a raise as a dead letter
    rather than letting it end the run.
    """
    if record.payload is None:
        raise ExtractionError(f"unreadable payload for {record.doc_id}")
    return record.payload


def chunk_document(doc_id: str, text: str, size: int = 40) -> list[Chunk]:
    """Split text and give each piece a STABLE id.

    The id is derived from the document id and the chunk's position within that
    document. It never depends on run order or on how many other documents were
    processed, which is what makes a second write an update rather than a
    duplicate.
    """
    words = text.split()
    if not words:
        return []
    chunks: list[Chunk] = []
    for position, start in enumerate(range(0, len(words), size)):
        body = " ".join(words[start : start + size])
        chunks.append(
            Chunk(chunk_id=f"{doc_id}::{position}", doc_id=doc_id, text=body)
        )
    return chunks


def embed(chunks: list[Chunk]) -> list[Chunk]:
    """Stand-in for the expensive stage.

    Real embedding costs money per chunk, which is the whole reason the content
    hash exists: an unchanged document must never reach this function.
    """
    return chunks


def run_once(
    source: list[SourceRecord],
    index: Index,
    checkpoint: Checkpoint,
    dead_letters: list[DeadLetter],
    *,
    fail_after: int | None = None,
    on_commit: Callable[[int], None] | None = None,
) -> RunStats:
    """Process everything above the cursor, in order.

    `fail_after` simulates a crash after N documents have been committed, so a
    test can prove the run resumes rather than restarts.
    """
    stats = RunStats(cursor=checkpoint.cursor)
    pending = sorted(
        (r for r in source if r.seq > checkpoint.cursor), key=lambda r: r.seq
    )

    committed = 0
    for record in pending:
        stats.scanned += 1

        try:
            text = extract(record)
        except ExtractionError as exc:
            # Isolated: the record is captured with its error and the run
            # continues. The cursor still advances, because retrying this
            # record on every future run would stall the pipeline forever --
            # draining the dead-letter queue is separate work.
            dead_letters.append(DeadLetter(doc_id=record.doc_id, error=type(exc).__name__))
            stats.dead_lettered += 1
            checkpoint.cursor = record.seq
            stats.cursor = record.seq
            continue

        digest = content_hash(text)
        if checkpoint.hashes.get(record.doc_id) == digest:
            # Incremental: provably unchanged, so skip the expensive stage.
            checkpoint.cursor = record.seq
            stats.cursor = record.seq
            continue

        stats.changed += 1
        chunks = embed(chunk_document(record.doc_id, text))
        stats.embedded += len(chunks)

        # Idempotent: stable ids mean this replaces rather than appends.
        stats.indexed += index.upsert(chunks)
        index.delete_orphans(record.doc_id, {c.chunk_id for c in chunks})

        # Resumable: the cursor advances only after the batch is written. A
        # crash between the write and this line reprocesses the record, which
        # is harmless precisely because the write was idempotent.
        checkpoint.hashes[record.doc_id] = digest
        checkpoint.cursor = record.seq
        stats.cursor = record.seq

        committed += 1
        if on_commit is not None:
            on_commit(committed)
        if fail_after is not None and committed >= fail_after:
            raise RuntimeError("simulated crash after commit")

    return stats
metadata.yml (753 bytes)
lesson_id: D323
day: 323
kind: python
languages:
  - python
setup_commands:
  - pip install -r requirements/requirements.txt
run_commands:
  - python examples/ingest_demo.py
test_commands:
  - bash tests/run_tests.sh
cleanup_commands:
  - rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-30'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3 examples/ingest_demo.py
  reproduced the documented run summary exactly; bash tests/run_tests.sh -> 11 passed in 0.01s. The
  untouched starter was verified to fail 10 of 11 tests, so the exercise is real. requires_network is
  false: the source and index are in-memory and no test opens a socket.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/ingest.py (6788 bytes)
"""A small but complete incremental ingestion pipeline.

Everything here runs offline with the standard library only. The source and the
index are in-memory fakes; the mechanisms around them -- content hashing, stable
chunk ids, a durable cursor and a dead-letter queue -- are the real thing, which
is what makes their behaviour testable.

The four properties the tests check:

  idempotent  running twice over unchanged sources leaves the index identical
  incremental a second run embeds only documents whose content hash changed
  resumable   a crash resumes from the last committed cursor
  isolated    a record that fails extraction is dead-lettered; the run continues
"""

from __future__ import annotations

import hashlib
from dataclasses import dataclass, field
from typing import Callable, Iterable


class ExtractionError(RuntimeError):
    """Raised when a source record cannot be turned into text."""


@dataclass(frozen=True)
class SourceRecord:
    """A record as the source system presents it."""

    doc_id: str
    seq: int  # monotonic; the cursor is expressed in these terms
    payload: str | None  # None simulates a document that cannot be extracted


@dataclass
class Chunk:
    chunk_id: str
    doc_id: str
    text: str


@dataclass
class DeadLetter:
    doc_id: str
    error: str


@dataclass
class RunStats:
    scanned: int = 0
    changed: int = 0
    embedded: int = 0
    indexed: int = 0
    dead_lettered: int = 0
    cursor: int = 0

    def line(self, label: str) -> str:
        return (
            f"{label}: scanned={self.scanned} changed={self.changed} "
            f"embedded={self.embedded} indexed={self.indexed} "
            f"dead_lettered={self.dead_lettered} cursor={self.cursor}"
        )


@dataclass
class Index:
    """A fake vector index with upsert semantics keyed on chunk_id."""

    chunks: dict[str, Chunk] = field(default_factory=dict)

    def upsert(self, chunks: Iterable[Chunk]) -> int:
        written = 0
        for chunk in chunks:
            self.chunks[chunk.chunk_id] = chunk
            written += 1
        return written

    def delete_orphans(self, doc_id: str, keep: set[str]) -> int:
        """Remove chunks of `doc_id` that are not in `keep`.

        A document that shrinks from twelve chunks to eight leaves four
        orphans behind. They stay retrievable and surface as stale answers, so
        reconciliation is part of a correct write, not an optimisation.
        """
        stale = [
            cid
            for cid, chunk in self.chunks.items()
            if chunk.doc_id == doc_id and cid not in keep
        ]
        for cid in stale:
            del self.chunks[cid]
        return len(stale)

    def doc_ids(self) -> set[str]:
        return {chunk.doc_id for chunk in self.chunks.values()}


@dataclass
class Checkpoint:
    """Durable-ish state: the cursor plus one content hash per document.

    Deliberately small. Storing extracted text here would make the checkpoint a
    second copy of the corpus.
    """

    cursor: int = 0
    hashes: dict[str, str] = field(default_factory=dict)


def content_hash(text: str) -> str:
    """Digest of the normalised body.

    Normalising before hashing matters: a source that rewrites whitespace on
    every read would otherwise look changed on every run.
    """
    # TODO(1): return a stable digest of the NORMALISED body.
    # Normalise whitespace first, or a source that reformats on every read will
    # look changed on every run. hashlib.sha256 over utf-8 bytes is fine.
    raise NotImplementedError("implement content_hash")

def extract(record: SourceRecord) -> str:
    """Turn a source record into text, or fail loudly.

    This is where a pipeline meets the real world's file formats, so it is the
    stage most likely to raise. Callers must treat a raise as a dead letter
    rather than letting it end the run.
    """
    if record.payload is None:
        raise ExtractionError(f"unreadable payload for {record.doc_id}")
    return record.payload


def chunk_document(doc_id: str, text: str, size: int = 40) -> list[Chunk]:
    """Split text and give each piece a STABLE id.

    The id is derived from the document id and the chunk's position within that
    document. It never depends on run order or on how many other documents were
    processed, which is what makes a second write an update rather than a
    duplicate.
    """
    # TODO(2): split `text` into chunks of `size` words and give each a STABLE
    # id of the form f"{doc_id}::{position}". The id must depend only on the
    # document and the position within it -- never on run order or a global
    # counter. Return [] for empty text.
    raise NotImplementedError("implement chunk_document")

def embed(chunks: list[Chunk]) -> list[Chunk]:
    """Stand-in for the expensive stage.

    Real embedding costs money per chunk, which is the whole reason the content
    hash exists: an unchanged document must never reach this function.
    """
    return chunks


def run_once(
    source: list[SourceRecord],
    index: Index,
    checkpoint: Checkpoint,
    dead_letters: list[DeadLetter],
    *,
    fail_after: int | None = None,
    on_commit: Callable[[int], None] | None = None,
) -> RunStats:
    """Process everything above the cursor, in order.

    TODO(3): implement the run loop so that all four properties hold.

      idempotent  stable chunk ids mean a second write REPLACES; after writing,
                  call index.delete_orphans(doc_id, keep) so a document that
                  shrinks leaves nothing behind.

      incremental compare content_hash(text) against checkpoint.hashes[doc_id].
                  If they match, the body is provably unchanged -- advance the
                  cursor and skip the expensive embed step entirely.

      resumable   advance checkpoint.cursor ONLY AFTER the batch is written.
                  A crash between write and commit then reprocesses a record,
                  which is harmless because the write was idempotent. Commit
                  first and the same crash loses the record silently.

      isolated    wrap extract() in try/except ExtractionError. On failure,
                  append a DeadLetter(doc_id, error=type(exc).__name__), count
                  it, advance the cursor, and CONTINUE the loop.

    `fail_after` simulates a crash: after that many documents have been
    committed, raise RuntimeError("simulated crash after commit"). Call
    on_commit(n) with the running commit count when it is not None. Both exist
    so the tests can prove resumability.

    Process records with seq > checkpoint.cursor, in ascending seq order, and
    return a RunStats describing what happened.
    """
    raise NotImplementedError("implement run_once")
tests/run_tests.sh (208 bytes)
#!/usr/bin/env bash
# Run the day 323 ingestion pipeline test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_ingest.py (5989 bytes)
"""One test per property, so a failure names what is broken.

Run with: bash tests/run_tests.sh
"""

from __future__ import annotations

import os
import sys

import pytest

HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))

from ingest import (  # noqa: E402
    Checkpoint,
    DeadLetter,
    ExtractionError,
    Index,
    SourceRecord,
    chunk_document,
    content_hash,
    extract,
    run_once,
)


def make_source() -> list[SourceRecord]:
    return [
        SourceRecord("doc-1", 1, "alpha " * 45),
        SourceRecord("doc-2", 2, "beta " * 80),
        SourceRecord("doc-3", 3, None),
        SourceRecord("doc-4", 4, "delta " * 30),
        SourceRecord("doc-5", 5, "epsilon " * 55),
    ]


def fresh():
    return Index(), Checkpoint(), []


# ---------------------------------------------------------------- idempotent


def test_second_run_over_unchanged_source_leaves_index_identical():
    source = make_source()
    index, checkpoint, dlq = fresh()

    run_once(source, index, checkpoint, dlq)
    after_first = dict(index.chunks)

    run_once(source, index, checkpoint, dlq)

    assert index.chunks.keys() == after_first.keys()
    assert len(index.chunks) == len(after_first)


def test_chunk_ids_are_stable_across_independent_runs():
    text = "gamma " * 100
    first = [c.chunk_id for c in chunk_document("doc-7", text)]
    second = [c.chunk_id for c in chunk_document("doc-7", text)]
    assert first == second
    # Scoped to the document, not to global processing order.
    assert all(cid.startswith("doc-7::") for cid in first)


def test_reindexing_from_a_clean_checkpoint_does_not_duplicate():
    source = make_source()
    index, checkpoint, dlq = fresh()
    run_once(source, index, checkpoint, dlq)
    size_once = len(index.chunks)

    # Same index, brand-new checkpoint: every document is reprocessed, but the
    # stable ids mean the writes replace rather than append.
    run_once(source, index, Checkpoint(), [])
    assert len(index.chunks) == size_once


# --------------------------------------------------------------- incremental


def test_unchanged_documents_are_not_embedded_again():
    source = make_source()
    index, checkpoint, dlq = fresh()
    run_once(source, index, checkpoint, dlq)

    # Cursor has not moved, so nothing is even scanned.
    second = run_once(source, index, checkpoint, dlq)
    assert second.scanned == 0
    assert second.embedded == 0

    # A document re-presented at a higher sequence with identical content is
    # scanned but must NOT be embedded -- that is the content hash working.
    source.append(SourceRecord("doc-1", 6, "alpha " * 45))
    third = run_once(source, index, checkpoint, dlq)
    assert third.scanned == 1
    assert third.changed == 0
    assert third.embedded == 0


def test_edited_document_is_embedded_again():
    source = make_source()
    index, checkpoint, dlq = fresh()
    run_once(source, index, checkpoint, dlq)

    source.append(SourceRecord("doc-1", 6, "alpha revised " * 40))
    stats = run_once(source, index, checkpoint, dlq)
    assert stats.changed == 1
    assert stats.embedded > 0


def test_content_hash_ignores_whitespace_reformatting():
    assert content_hash("a  b\n\nc") == content_hash("a b c")


# ------------------------------------------------------------------ resumable


def test_crash_resumes_from_last_committed_cursor():
    source = make_source()
    index, checkpoint, dlq = fresh()

    with pytest.raises(RuntimeError, match="simulated crash"):
        run_once(source, index, checkpoint, dlq, fail_after=2)

    # Two documents committed; the cursor reflects exactly that.
    crashed_at = checkpoint.cursor
    assert crashed_at > 0
    indexed_before = len(index.chunks)

    stats = run_once(source, index, checkpoint, dlq)

    # It resumed rather than restarted.
    assert stats.cursor == 5
    assert len(index.chunks) > indexed_before
    # And nothing below the cursor was scanned a second time.
    assert stats.scanned == len([r for r in source if r.seq > crashed_at])


def test_cursor_never_advances_past_uncommitted_work():
    source = make_source()
    index, checkpoint, dlq = fresh()
    seen: list[int] = []

    with pytest.raises(RuntimeError):
        run_once(source, index, checkpoint, dlq, fail_after=1, on_commit=seen.append)

    # Exactly one commit happened, and the cursor is at a real record boundary.
    assert seen == [1]
    assert checkpoint.cursor in {r.seq for r in source}


# ------------------------------------------------------------------- isolated


def test_failing_record_is_dead_lettered_and_run_continues():
    source = make_source()
    index, checkpoint, dlq = fresh()

    stats = run_once(source, index, checkpoint, dlq)

    assert stats.dead_lettered == 1
    assert dlq[0].doc_id == "doc-3"
    assert dlq[0].error == "ExtractionError"
    # The four healthy documents still made it in.
    assert index.doc_ids() == {"doc-1", "doc-2", "doc-4", "doc-5"}


def test_extract_raises_on_unreadable_payload():
    with pytest.raises(ExtractionError):
        extract(SourceRecord("doc-x", 99, None))


# ------------------------------------------------------- orphan reconciliation


def test_shrinking_document_leaves_no_orphaned_chunks():
    index, checkpoint, dlq = fresh()
    long_doc = [SourceRecord("doc-1", 1, "word " * 200)]
    run_once(long_doc, index, checkpoint, dlq)
    many = len([c for c in index.chunks.values() if c.doc_id == "doc-1"])
    assert many > 2

    short_doc = long_doc + [SourceRecord("doc-1", 2, "word " * 20)]
    run_once(short_doc, index, checkpoint, dlq)
    few = len([c for c in index.chunks.values() if c.doc_id == "doc-1"])

    assert few < many, "chunk count should shrink with the document"
    ids = {c.chunk_id for c in index.chunks.values() if c.doc_id == "doc-1"}
    assert ids == {f"doc-1::{i}" for i in range(few)}, "orphans were left behind"

Troubleshooting

Troubleshooting — Day 323

The index doubles on the second run

Your chunk id depends on something that varies between runs — a global counter, a timestamp, uuid4(), or id(). A stable id must be derived from the document id plus the chunk's position within that document, so the same input always produces the same id and the write becomes an update.

Check with:

python3 -c "
import sys; sys.path.insert(0,'examples')
from ingest import chunk_document
a=[c.chunk_id for c in chunk_document('d','x '*100)]
b=[c.chunk_id for c in chunk_document('d','x '*100)]
print('stable' if a==b else 'NOT STABLE')"

Run 2 re-embeds everything

Either you are comparing modification times instead of content hashes, or you are hashing something that varies. Hash the normalised body only — " ".join(text.split()) before digesting — so a source that reformats whitespace on read does not look changed.

test_crash_resumes_from_last_committed_cursor fails

You are committing the cursor before the batch is written. Move checkpoint.cursor = record.seq to after the index.upsert(...) call. Committing first means a crash between the two silently skips the record forever.

One bad document ends the run

extract() raises ExtractionError for doc-3, which has no payload. Wrap the call in try/except ExtractionError, append a DeadLetter(doc_id, error=type(exc).__name__), count it, advance the cursor and continue. If you re-raise, the run dies and the four healthy documents never index.

test_shrinking_document_leaves_no_orphaned_chunks fails

Upserting the new chunks is not enough. After writing, call index.delete_orphans(doc_id, keep) with the set of chunk ids you just wrote, so chunks from the longer previous version are removed. Orphans stay retrievable and produce stale answers.

NotImplementedError everywhere

Expected. The starter stubs content_hash, chunk_document and run_once. See expected-output/starter-run.txt — an untouched starter fails 10 of 11 tests, and the one that passes does not depend on the stubs.

ModuleNotFoundError: No module named 'ingest'

Run the tests through bash tests/run_tests.sh, which changes into the lab root first. The test file inserts examples/ onto sys.path relative to its own location, so running pytest from elsewhere can miss it.

TypeError on str | None

You are on Python 3.9 or older. This lab needs 3.10 or newer for that type syntax. Check with python3 --version.

Security notes

Security notes — Day 323

What this lab does and does not touch

  • No network. The source and the index are in-memory Python objects. No test opens a socket, and requires_network is false in metadata.yml.
  • No credentials. There is no API key, token or account anywhere in this lab.
  • No writes outside this directory. The pipeline holds its state in memory. The only files created are __pycache__/, .pytest_cache/ and, if you create it, .venv/ — all inside this lab directory and all removable with the documented cleanup commands.
  • No sudo, no global installs. Dependencies go into a local virtual environment.

What the lesson says about real pipelines, which this lab deliberately simulates

A production ingestion pipeline is a genuinely sensitive component, and it is worth understanding why even though this lab exercises none of it:

  • It holds credentials for every source it reads. That makes it a high-value target. Grant the narrowest read-only scope each source supports, and keep those credentials out of the repository.
  • Extraction parses untrusted input. A PDF, an archive or an office document is an attack surface — decompression bombs and malformed structures are real. In production, run extractors with memory and time limits, and preferably in a sandbox. This lab's extract() raises a plain exception instead, which is the behaviour you must handle either way.
  • Ingestion is where personal data enters the system, so it is the right place to redact it, and the right place to honour deletion. A source record deleted for legal reasons must be removed from the index and any cache; a deletion that does not propagate is a compliance failure rather than a stale cache.

Reversing everything this lab did

rm -rf __pycache__ .pytest_cache .venv

Nothing else was created, and nothing outside this directory was modified.