AI Engineering: Agents and Applications › Production Retrieval and Pipelines › Day 323
Hands-on lab — Day 323: Data Ingestion Pipelines
- ← Back to the Day 323 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/ai-engineering/day-323-data-ingestion-pipelines/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/ingest_demo.py Test
bash tests/run_tests.sh File tree
examples/ingest_demo.py examples/ingest.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/ingest.py tests/run_tests.sh tests/test_ingest.py troubleshooting.md
Lab README
Lab — Day 323: Data Ingestion Pipelines
Lesson
- Lesson title: Data Ingestion Pipelines
- Day number: 323 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-323-data-ingestion-pipelines
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-323-data-ingestion-pipelineswhen the site is running.
Purpose
Build an incremental ingestion pipeline that is safe to schedule and forget. You implement content-hash change detection, stable chunk identifiers, a durable cursor committed in the correct order, and a dead-letter queue — the four mechanisms that make a re-run harmless instead of destructive.
Everything runs offline against an in-memory source and index. The endpoints are simulated; the mechanisms are real, which is what makes their behaviour testable.
Learning objectives
- Derive a chunk identifier that is reproducible across runs, so a second write upserts rather than duplicates.
- Prove a document is unchanged with a content hash rather than trusting modification metadata.
- Order the cursor commit after the batch write, so a crash reprocesses instead of losing records.
- Capture a failing record in a dead-letter queue and let the batch continue.
- Delete orphaned chunks when a document shrinks.
Prerequisites
- Day 322, "A Full-Stack AI Application".
- Comfortable with Python dataclasses, dictionaries and exceptions.
- Familiar with
pytestbasics.
Supported operating systems
macOS, Linux and Windows (via WSL or PowerShell). Nothing in this lab is platform-specific.
Hardware requirements
Any machine that runs Python 3.10 or newer. No GPU. The full suite finishes in well under a second and uses a few megabytes of memory.
Required software
- Python 3.10 or newer (uses
X | Nonetype syntax). pytest7 or newer.
Free and open-source options
Every dependency is free and open source. Python is PSF-licensed; pytest is MIT-licensed. There is no paid tier, no account and no hosted service anywhere in this lab.
Installation
Build a virtual environment inside this lab directory so nothing touches your system Python:
python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt
Removing .venv fully reverses the installation.
File structure
starter/ingest.py your work: content_hash, chunk_document, run_once
examples/ingest.py reference implementation
examples/ingest_demo.py three consecutive runs over a changing source
tests/test_ingest.py one test per property
tests/run_tests.sh suite entry point
expected-output/ real captured output, including the starter's failures
requirements/ pinned dependency
How to run
python3 examples/ingest_demo.py # see the reference pipeline behave
bash tests/run_tests.sh # run the suite
To work on the exercise, edit starter/ingest.py, then copy it over the reference to test your version:
cp starter/ingest.py examples/ingest.py
bash tests/run_tests.sh
Keep a copy of the reference first if you want to compare.
What the commands do
python3 examples/ingest_demo.py runs the pipeline three times over a source where one document is edited between runs and one document is permanently malformed. It prints a summary line per run plus the final index state.
bash tests/run_tests.sh changes into the lab root and runs python3 -m pytest tests -q. The suite is grouped by property, so a failure names which of idempotent, incremental, resumable or isolated is not yet satisfied.
Expected output
run 1: scanned=5 changed=4 embedded=7 indexed=7 dead_lettered=1 cursor=5
run 2: scanned=0 changed=0 embedded=0 indexed=0 dead_lettered=0 cursor=5
run 3: scanned=1 changed=1 embedded=2 indexed=2 dead_lettered=0 cursor=6
index: 7 chunks across 4 documents
dead letters: 1 (doc-3: ExtractionError)
Run 2 scans nothing because the cursor has not moved. expected-output/FIELDS.md explains every field; expected-output/starter-run.txt records what an untouched starter produces.
Validation steps
bash tests/run_tests.shreports11 passed.- Run
python3 examples/ingest_demo.pytwice in succession. The index size must be identical both times — if it grows, your chunk ids are not stable. - Confirm run 2 reports
embedded=0. If it does not, you are comparing timestamps rather than content hashes.
Tests
Eleven tests across five groups:
- idempotent — a second run leaves the index identical; ids are stable across independent runs; reindexing from a clean checkpoint does not duplicate.
- incremental — unchanged documents are not re-embedded even when re-presented; edited documents are; the hash ignores whitespace reformatting.
- resumable — a simulated crash resumes from the last committed cursor; the cursor never advances past uncommitted work.
- isolated — the failing record is dead-lettered with its error and the other four documents still index.
- reconciliation — a shrinking document leaves no orphaned chunks.
Cleanup
rm -rf __pycache__ .pytest_cache
rm -rf .venv
Both are safe to remove and fully reversible: the virtual environment rebuilds from requirements/requirements.txt.
Troubleshooting
See troubleshooting.md for the failure modes this lab is designed to produce, including a doubling index, a run that re-embeds everything, and a crash that loses records.
Security notes
See security.md. In short: no network, no credentials, no API key, and every write stays inside this directory.
Extension exercises
- Deletion reconciliation. Add
deleted_idsto the source and remove those documents' chunks. Then add a full reconciliation pass comparing all source ids against all indexed ids. - Bounded concurrency. Extract with a small
ThreadPoolExecutorwhile preserving the cursor guarantee — the cursor may only pass a record once it and everything before it has committed. Measure both versions and record the real numbers, including a null result. - Retry policy. Give dead letters a retry count and an abandon threshold, so a transient failure recovers but a permanently poisoned record does not retry forever.
Navigation
- Lesson
- Previous: Day 322 — A Full-Stack AI Application
- Next: Day 324 — Document Processing at Scale
Expected output
FIELDS.md
# What each field in the run summary means
Each run prints one line in this shape:
run N: scanned=A changed=B embedded=C indexed=D dead_lettered=E cursor=F
| Field | Meaning |
| --- | --- |
| `scanned` | Records with `seq` greater than the committed cursor that this run looked at. Zero means the cursor has not moved since the last run. |
| `changed` | Records whose content hash differed from the stored hash, so their body genuinely changed. A record that fails extraction is never hashed and so is never counted here. |
| `embedded` | Chunks passed to the embedding stage. In production this is the line item you pay for, which is why `changed` gates it. |
| `indexed` | Chunks written to the index. Equal to `embedded` in this lab because every embedded chunk is upserted. |
| `dead_lettered` | Records that failed extraction and were captured with their error so the run could continue. |
| `cursor` | The highest `seq` fully processed. Committed only after the batch is written, which is what makes a crash reprocess rather than lose work. |
The final two lines report the index size and the dead-letter contents:
index: 7 chunks across 4 documents
dead letters: 1 (doc-3: ExtractionError)
Four documents rather than five because `doc-3` has no payload and fails
extraction on every attempt.
examples-run.txt
run 1: scanned=5 changed=4 embedded=7 indexed=7 dead_lettered=1 cursor=5
run 2: scanned=0 changed=0 embedded=0 indexed=0 dead_lettered=0 cursor=5
run 3: scanned=1 changed=1 embedded=2 indexed=2 dead_lettered=0 cursor=6
index: 7 chunks across 4 documents
dead letters: 1 (doc-3: ExtractionError)
run-output.txt
run 1: scanned=5 changed=4 embedded=7 indexed=7 dead_lettered=1 cursor=5
run 2: scanned=0 changed=0 embedded=0 indexed=0 dead_lettered=0 cursor=5
run 3: scanned=1 changed=1 embedded=2 indexed=2 dead_lettered=0 cursor=6
index: 7 chunks across 4 documents
dead letters: 1 (doc-3: ExtractionError)
starter-run.txt
Before you implement the TODOs, the starter is expected to fail. Running
bash tests/run_tests.sh
against an untouched starter/ingest.py produces:
10 failed, 1 passed
The single pass is test_extract_raises_on_unreadable_payload, which exercises
extract() and does not depend on any of the stubbed functions. Every other test
fails with NotImplementedError until you complete content_hash, chunk_document
and run_once.
test-output.txt
........... [100%]
11 passed in 0.01s
test-run.txt
........... [100%]
11 passed in 0.01s
Source files
examples/ingest_demo.py (1753 bytes)
#!/usr/bin/env python3
"""Three consecutive runs over a source that changes between them.
Run 2 is the line worth reading: it scans nothing and embeds nothing, because
the cursor has not moved. Run 3 shows an edited document re-embedded and
upserted without changing the document count.
"""
from __future__ import annotations
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from ingest import Checkpoint, DeadLetter, Index, SourceRecord, run_once
def main() -> int:
# doc-3 has no payload: it fails extraction on every attempt and must not
# stop the other four documents from being indexed.
source = [
SourceRecord("doc-1", 1, "alpha " * 45),
SourceRecord("doc-2", 2, "beta " * 80),
SourceRecord("doc-3", 3, None),
SourceRecord("doc-4", 4, "delta " * 30),
SourceRecord("doc-5", 5, "epsilon " * 55),
]
index = Index()
checkpoint = Checkpoint()
dead_letters: list[DeadLetter] = []
stats1 = run_once(source, index, checkpoint, dead_letters)
print(stats1.line("run 1"))
# Nothing changed and the cursor has not moved.
stats2 = run_once(source, index, checkpoint, dead_letters)
print(stats2.line("run 2"))
# An edited document arrives with a new sequence number.
source.append(SourceRecord("doc-1", 6, "alpha revised " * 40))
stats3 = run_once(source, index, checkpoint, dead_letters)
print(stats3.line("run 3"))
print(f"index: {len(index.chunks)} chunks across {len(index.doc_ids())} documents")
detail = ", ".join(f"{dl.doc_id}: {dl.error}" for dl in dead_letters)
print(f"dead letters: {len(dead_letters)} ({detail})")
return 0
if __name__ == "__main__":
raise SystemExit(main())
examples/ingest.py (7299 bytes)
"""A small but complete incremental ingestion pipeline.
Everything here runs offline with the standard library only. The source and the
index are in-memory fakes; the mechanisms around them -- content hashing, stable
chunk ids, a durable cursor and a dead-letter queue -- are the real thing, which
is what makes their behaviour testable.
The four properties the tests check:
idempotent running twice over unchanged sources leaves the index identical
incremental a second run embeds only documents whose content hash changed
resumable a crash resumes from the last committed cursor
isolated a record that fails extraction is dead-lettered; the run continues
"""
from __future__ import annotations
import hashlib
from dataclasses import dataclass, field
from typing import Callable, Iterable
class ExtractionError(RuntimeError):
"""Raised when a source record cannot be turned into text."""
@dataclass(frozen=True)
class SourceRecord:
"""A record as the source system presents it."""
doc_id: str
seq: int # monotonic; the cursor is expressed in these terms
payload: str | None # None simulates a document that cannot be extracted
@dataclass
class Chunk:
chunk_id: str
doc_id: str
text: str
@dataclass
class DeadLetter:
doc_id: str
error: str
@dataclass
class RunStats:
scanned: int = 0
changed: int = 0
embedded: int = 0
indexed: int = 0
dead_lettered: int = 0
cursor: int = 0
def line(self, label: str) -> str:
return (
f"{label}: scanned={self.scanned} changed={self.changed} "
f"embedded={self.embedded} indexed={self.indexed} "
f"dead_lettered={self.dead_lettered} cursor={self.cursor}"
)
@dataclass
class Index:
"""A fake vector index with upsert semantics keyed on chunk_id."""
chunks: dict[str, Chunk] = field(default_factory=dict)
def upsert(self, chunks: Iterable[Chunk]) -> int:
written = 0
for chunk in chunks:
self.chunks[chunk.chunk_id] = chunk
written += 1
return written
def delete_orphans(self, doc_id: str, keep: set[str]) -> int:
"""Remove chunks of `doc_id` that are not in `keep`.
A document that shrinks from twelve chunks to eight leaves four
orphans behind. They stay retrievable and surface as stale answers, so
reconciliation is part of a correct write, not an optimisation.
"""
stale = [
cid
for cid, chunk in self.chunks.items()
if chunk.doc_id == doc_id and cid not in keep
]
for cid in stale:
del self.chunks[cid]
return len(stale)
def doc_ids(self) -> set[str]:
return {chunk.doc_id for chunk in self.chunks.values()}
@dataclass
class Checkpoint:
"""Durable-ish state: the cursor plus one content hash per document.
Deliberately small. Storing extracted text here would make the checkpoint a
second copy of the corpus.
"""
cursor: int = 0
hashes: dict[str, str] = field(default_factory=dict)
def content_hash(text: str) -> str:
"""Digest of the normalised body.
Normalising before hashing matters: a source that rewrites whitespace on
every read would otherwise look changed on every run.
"""
normalised = " ".join(text.split())
return hashlib.sha256(normalised.encode("utf-8")).hexdigest()
def extract(record: SourceRecord) -> str:
"""Turn a source record into text, or fail loudly.
This is where a pipeline meets the real world's file formats, so it is the
stage most likely to raise. Callers must treat a raise as a dead letter
rather than letting it end the run.
"""
if record.payload is None:
raise ExtractionError(f"unreadable payload for {record.doc_id}")
return record.payload
def chunk_document(doc_id: str, text: str, size: int = 40) -> list[Chunk]:
"""Split text and give each piece a STABLE id.
The id is derived from the document id and the chunk's position within that
document. It never depends on run order or on how many other documents were
processed, which is what makes a second write an update rather than a
duplicate.
"""
words = text.split()
if not words:
return []
chunks: list[Chunk] = []
for position, start in enumerate(range(0, len(words), size)):
body = " ".join(words[start : start + size])
chunks.append(
Chunk(chunk_id=f"{doc_id}::{position}", doc_id=doc_id, text=body)
)
return chunks
def embed(chunks: list[Chunk]) -> list[Chunk]:
"""Stand-in for the expensive stage.
Real embedding costs money per chunk, which is the whole reason the content
hash exists: an unchanged document must never reach this function.
"""
return chunks
def run_once(
source: list[SourceRecord],
index: Index,
checkpoint: Checkpoint,
dead_letters: list[DeadLetter],
*,
fail_after: int | None = None,
on_commit: Callable[[int], None] | None = None,
) -> RunStats:
"""Process everything above the cursor, in order.
`fail_after` simulates a crash after N documents have been committed, so a
test can prove the run resumes rather than restarts.
"""
stats = RunStats(cursor=checkpoint.cursor)
pending = sorted(
(r for r in source if r.seq > checkpoint.cursor), key=lambda r: r.seq
)
committed = 0
for record in pending:
stats.scanned += 1
try:
text = extract(record)
except ExtractionError as exc:
# Isolated: the record is captured with its error and the run
# continues. The cursor still advances, because retrying this
# record on every future run would stall the pipeline forever --
# draining the dead-letter queue is separate work.
dead_letters.append(DeadLetter(doc_id=record.doc_id, error=type(exc).__name__))
stats.dead_lettered += 1
checkpoint.cursor = record.seq
stats.cursor = record.seq
continue
digest = content_hash(text)
if checkpoint.hashes.get(record.doc_id) == digest:
# Incremental: provably unchanged, so skip the expensive stage.
checkpoint.cursor = record.seq
stats.cursor = record.seq
continue
stats.changed += 1
chunks = embed(chunk_document(record.doc_id, text))
stats.embedded += len(chunks)
# Idempotent: stable ids mean this replaces rather than appends.
stats.indexed += index.upsert(chunks)
index.delete_orphans(record.doc_id, {c.chunk_id for c in chunks})
# Resumable: the cursor advances only after the batch is written. A
# crash between the write and this line reprocesses the record, which
# is harmless precisely because the write was idempotent.
checkpoint.hashes[record.doc_id] = digest
checkpoint.cursor = record.seq
stats.cursor = record.seq
committed += 1
if on_commit is not None:
on_commit(committed)
if fail_after is not None and committed >= fail_after:
raise RuntimeError("simulated crash after commit")
return stats
metadata.yml (753 bytes)
lesson_id: D323
day: 323
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/ingest_demo.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-30'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3 examples/ingest_demo.py
reproduced the documented run summary exactly; bash tests/run_tests.sh -> 11 passed in 0.01s. The
untouched starter was verified to fail 10 of 11 tests, so the exercise is real. requires_network is
false: the source and index are in-memory and no test opens a socket.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/ingest.py (6788 bytes)
"""A small but complete incremental ingestion pipeline.
Everything here runs offline with the standard library only. The source and the
index are in-memory fakes; the mechanisms around them -- content hashing, stable
chunk ids, a durable cursor and a dead-letter queue -- are the real thing, which
is what makes their behaviour testable.
The four properties the tests check:
idempotent running twice over unchanged sources leaves the index identical
incremental a second run embeds only documents whose content hash changed
resumable a crash resumes from the last committed cursor
isolated a record that fails extraction is dead-lettered; the run continues
"""
from __future__ import annotations
import hashlib
from dataclasses import dataclass, field
from typing import Callable, Iterable
class ExtractionError(RuntimeError):
"""Raised when a source record cannot be turned into text."""
@dataclass(frozen=True)
class SourceRecord:
"""A record as the source system presents it."""
doc_id: str
seq: int # monotonic; the cursor is expressed in these terms
payload: str | None # None simulates a document that cannot be extracted
@dataclass
class Chunk:
chunk_id: str
doc_id: str
text: str
@dataclass
class DeadLetter:
doc_id: str
error: str
@dataclass
class RunStats:
scanned: int = 0
changed: int = 0
embedded: int = 0
indexed: int = 0
dead_lettered: int = 0
cursor: int = 0
def line(self, label: str) -> str:
return (
f"{label}: scanned={self.scanned} changed={self.changed} "
f"embedded={self.embedded} indexed={self.indexed} "
f"dead_lettered={self.dead_lettered} cursor={self.cursor}"
)
@dataclass
class Index:
"""A fake vector index with upsert semantics keyed on chunk_id."""
chunks: dict[str, Chunk] = field(default_factory=dict)
def upsert(self, chunks: Iterable[Chunk]) -> int:
written = 0
for chunk in chunks:
self.chunks[chunk.chunk_id] = chunk
written += 1
return written
def delete_orphans(self, doc_id: str, keep: set[str]) -> int:
"""Remove chunks of `doc_id` that are not in `keep`.
A document that shrinks from twelve chunks to eight leaves four
orphans behind. They stay retrievable and surface as stale answers, so
reconciliation is part of a correct write, not an optimisation.
"""
stale = [
cid
for cid, chunk in self.chunks.items()
if chunk.doc_id == doc_id and cid not in keep
]
for cid in stale:
del self.chunks[cid]
return len(stale)
def doc_ids(self) -> set[str]:
return {chunk.doc_id for chunk in self.chunks.values()}
@dataclass
class Checkpoint:
"""Durable-ish state: the cursor plus one content hash per document.
Deliberately small. Storing extracted text here would make the checkpoint a
second copy of the corpus.
"""
cursor: int = 0
hashes: dict[str, str] = field(default_factory=dict)
def content_hash(text: str) -> str:
"""Digest of the normalised body.
Normalising before hashing matters: a source that rewrites whitespace on
every read would otherwise look changed on every run.
"""
# TODO(1): return a stable digest of the NORMALISED body.
# Normalise whitespace first, or a source that reformats on every read will
# look changed on every run. hashlib.sha256 over utf-8 bytes is fine.
raise NotImplementedError("implement content_hash")
def extract(record: SourceRecord) -> str:
"""Turn a source record into text, or fail loudly.
This is where a pipeline meets the real world's file formats, so it is the
stage most likely to raise. Callers must treat a raise as a dead letter
rather than letting it end the run.
"""
if record.payload is None:
raise ExtractionError(f"unreadable payload for {record.doc_id}")
return record.payload
def chunk_document(doc_id: str, text: str, size: int = 40) -> list[Chunk]:
"""Split text and give each piece a STABLE id.
The id is derived from the document id and the chunk's position within that
document. It never depends on run order or on how many other documents were
processed, which is what makes a second write an update rather than a
duplicate.
"""
# TODO(2): split `text` into chunks of `size` words and give each a STABLE
# id of the form f"{doc_id}::{position}". The id must depend only on the
# document and the position within it -- never on run order or a global
# counter. Return [] for empty text.
raise NotImplementedError("implement chunk_document")
def embed(chunks: list[Chunk]) -> list[Chunk]:
"""Stand-in for the expensive stage.
Real embedding costs money per chunk, which is the whole reason the content
hash exists: an unchanged document must never reach this function.
"""
return chunks
def run_once(
source: list[SourceRecord],
index: Index,
checkpoint: Checkpoint,
dead_letters: list[DeadLetter],
*,
fail_after: int | None = None,
on_commit: Callable[[int], None] | None = None,
) -> RunStats:
"""Process everything above the cursor, in order.
TODO(3): implement the run loop so that all four properties hold.
idempotent stable chunk ids mean a second write REPLACES; after writing,
call index.delete_orphans(doc_id, keep) so a document that
shrinks leaves nothing behind.
incremental compare content_hash(text) against checkpoint.hashes[doc_id].
If they match, the body is provably unchanged -- advance the
cursor and skip the expensive embed step entirely.
resumable advance checkpoint.cursor ONLY AFTER the batch is written.
A crash between write and commit then reprocesses a record,
which is harmless because the write was idempotent. Commit
first and the same crash loses the record silently.
isolated wrap extract() in try/except ExtractionError. On failure,
append a DeadLetter(doc_id, error=type(exc).__name__), count
it, advance the cursor, and CONTINUE the loop.
`fail_after` simulates a crash: after that many documents have been
committed, raise RuntimeError("simulated crash after commit"). Call
on_commit(n) with the running commit count when it is not None. Both exist
so the tests can prove resumability.
Process records with seq > checkpoint.cursor, in ascending seq order, and
return a RunStats describing what happened.
"""
raise NotImplementedError("implement run_once")
tests/run_tests.sh (208 bytes)
#!/usr/bin/env bash
# Run the day 323 ingestion pipeline test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_ingest.py (5989 bytes)
"""One test per property, so a failure names what is broken.
Run with: bash tests/run_tests.sh
"""
from __future__ import annotations
import os
import sys
import pytest
HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))
from ingest import ( # noqa: E402
Checkpoint,
DeadLetter,
ExtractionError,
Index,
SourceRecord,
chunk_document,
content_hash,
extract,
run_once,
)
def make_source() -> list[SourceRecord]:
return [
SourceRecord("doc-1", 1, "alpha " * 45),
SourceRecord("doc-2", 2, "beta " * 80),
SourceRecord("doc-3", 3, None),
SourceRecord("doc-4", 4, "delta " * 30),
SourceRecord("doc-5", 5, "epsilon " * 55),
]
def fresh():
return Index(), Checkpoint(), []
# ---------------------------------------------------------------- idempotent
def test_second_run_over_unchanged_source_leaves_index_identical():
source = make_source()
index, checkpoint, dlq = fresh()
run_once(source, index, checkpoint, dlq)
after_first = dict(index.chunks)
run_once(source, index, checkpoint, dlq)
assert index.chunks.keys() == after_first.keys()
assert len(index.chunks) == len(after_first)
def test_chunk_ids_are_stable_across_independent_runs():
text = "gamma " * 100
first = [c.chunk_id for c in chunk_document("doc-7", text)]
second = [c.chunk_id for c in chunk_document("doc-7", text)]
assert first == second
# Scoped to the document, not to global processing order.
assert all(cid.startswith("doc-7::") for cid in first)
def test_reindexing_from_a_clean_checkpoint_does_not_duplicate():
source = make_source()
index, checkpoint, dlq = fresh()
run_once(source, index, checkpoint, dlq)
size_once = len(index.chunks)
# Same index, brand-new checkpoint: every document is reprocessed, but the
# stable ids mean the writes replace rather than append.
run_once(source, index, Checkpoint(), [])
assert len(index.chunks) == size_once
# --------------------------------------------------------------- incremental
def test_unchanged_documents_are_not_embedded_again():
source = make_source()
index, checkpoint, dlq = fresh()
run_once(source, index, checkpoint, dlq)
# Cursor has not moved, so nothing is even scanned.
second = run_once(source, index, checkpoint, dlq)
assert second.scanned == 0
assert second.embedded == 0
# A document re-presented at a higher sequence with identical content is
# scanned but must NOT be embedded -- that is the content hash working.
source.append(SourceRecord("doc-1", 6, "alpha " * 45))
third = run_once(source, index, checkpoint, dlq)
assert third.scanned == 1
assert third.changed == 0
assert third.embedded == 0
def test_edited_document_is_embedded_again():
source = make_source()
index, checkpoint, dlq = fresh()
run_once(source, index, checkpoint, dlq)
source.append(SourceRecord("doc-1", 6, "alpha revised " * 40))
stats = run_once(source, index, checkpoint, dlq)
assert stats.changed == 1
assert stats.embedded > 0
def test_content_hash_ignores_whitespace_reformatting():
assert content_hash("a b\n\nc") == content_hash("a b c")
# ------------------------------------------------------------------ resumable
def test_crash_resumes_from_last_committed_cursor():
source = make_source()
index, checkpoint, dlq = fresh()
with pytest.raises(RuntimeError, match="simulated crash"):
run_once(source, index, checkpoint, dlq, fail_after=2)
# Two documents committed; the cursor reflects exactly that.
crashed_at = checkpoint.cursor
assert crashed_at > 0
indexed_before = len(index.chunks)
stats = run_once(source, index, checkpoint, dlq)
# It resumed rather than restarted.
assert stats.cursor == 5
assert len(index.chunks) > indexed_before
# And nothing below the cursor was scanned a second time.
assert stats.scanned == len([r for r in source if r.seq > crashed_at])
def test_cursor_never_advances_past_uncommitted_work():
source = make_source()
index, checkpoint, dlq = fresh()
seen: list[int] = []
with pytest.raises(RuntimeError):
run_once(source, index, checkpoint, dlq, fail_after=1, on_commit=seen.append)
# Exactly one commit happened, and the cursor is at a real record boundary.
assert seen == [1]
assert checkpoint.cursor in {r.seq for r in source}
# ------------------------------------------------------------------- isolated
def test_failing_record_is_dead_lettered_and_run_continues():
source = make_source()
index, checkpoint, dlq = fresh()
stats = run_once(source, index, checkpoint, dlq)
assert stats.dead_lettered == 1
assert dlq[0].doc_id == "doc-3"
assert dlq[0].error == "ExtractionError"
# The four healthy documents still made it in.
assert index.doc_ids() == {"doc-1", "doc-2", "doc-4", "doc-5"}
def test_extract_raises_on_unreadable_payload():
with pytest.raises(ExtractionError):
extract(SourceRecord("doc-x", 99, None))
# ------------------------------------------------------- orphan reconciliation
def test_shrinking_document_leaves_no_orphaned_chunks():
index, checkpoint, dlq = fresh()
long_doc = [SourceRecord("doc-1", 1, "word " * 200)]
run_once(long_doc, index, checkpoint, dlq)
many = len([c for c in index.chunks.values() if c.doc_id == "doc-1"])
assert many > 2
short_doc = long_doc + [SourceRecord("doc-1", 2, "word " * 20)]
run_once(short_doc, index, checkpoint, dlq)
few = len([c for c in index.chunks.values() if c.doc_id == "doc-1"])
assert few < many, "chunk count should shrink with the document"
ids = {c.chunk_id for c in index.chunks.values() if c.doc_id == "doc-1"}
assert ids == {f"doc-1::{i}" for i in range(few)}, "orphans were left behind"
Troubleshooting
Troubleshooting — Day 323
The index doubles on the second run
Your chunk id depends on something that varies between runs — a global counter, a timestamp, uuid4(), or id(). A stable id must be derived from the document id plus the chunk's position within that document, so the same input always produces the same id and the write becomes an update.
Check with:
python3 -c "
import sys; sys.path.insert(0,'examples')
from ingest import chunk_document
a=[c.chunk_id for c in chunk_document('d','x '*100)]
b=[c.chunk_id for c in chunk_document('d','x '*100)]
print('stable' if a==b else 'NOT STABLE')"
Run 2 re-embeds everything
Either you are comparing modification times instead of content hashes, or you are hashing something that varies. Hash the normalised body only — " ".join(text.split()) before digesting — so a source that reformats whitespace on read does not look changed.
test_crash_resumes_from_last_committed_cursor fails
You are committing the cursor before the batch is written. Move checkpoint.cursor = record.seq to after the index.upsert(...) call. Committing first means a crash between the two silently skips the record forever.
One bad document ends the run
extract() raises ExtractionError for doc-3, which has no payload. Wrap the call in try/except ExtractionError, append a DeadLetter(doc_id, error=type(exc).__name__), count it, advance the cursor and continue. If you re-raise, the run dies and the four healthy documents never index.
test_shrinking_document_leaves_no_orphaned_chunks fails
Upserting the new chunks is not enough. After writing, call index.delete_orphans(doc_id, keep) with the set of chunk ids you just wrote, so chunks from the longer previous version are removed. Orphans stay retrievable and produce stale answers.
NotImplementedError everywhere
Expected. The starter stubs content_hash, chunk_document and run_once. See expected-output/starter-run.txt — an untouched starter fails 10 of 11 tests, and the one that passes does not depend on the stubs.
ModuleNotFoundError: No module named 'ingest'
Run the tests through bash tests/run_tests.sh, which changes into the lab root first. The test file inserts examples/ onto sys.path relative to its own location, so running pytest from elsewhere can miss it.
TypeError on str | None
You are on Python 3.9 or older. This lab needs 3.10 or newer for that type syntax. Check with python3 --version.
Security notes
Security notes — Day 323
What this lab does and does not touch
- No network. The source and the index are in-memory Python objects. No test opens a socket, and
requires_networkisfalseinmetadata.yml. - No credentials. There is no API key, token or account anywhere in this lab.
- No writes outside this directory. The pipeline holds its state in memory. The only files created are
__pycache__/,.pytest_cache/and, if you create it,.venv/— all inside this lab directory and all removable with the documented cleanup commands. - No
sudo, no global installs. Dependencies go into a local virtual environment.
What the lesson says about real pipelines, which this lab deliberately simulates
A production ingestion pipeline is a genuinely sensitive component, and it is worth understanding why even though this lab exercises none of it:
- It holds credentials for every source it reads. That makes it a high-value target. Grant the narrowest read-only scope each source supports, and keep those credentials out of the repository.
- Extraction parses untrusted input. A PDF, an archive or an office document is an attack surface — decompression bombs and malformed structures are real. In production, run extractors with memory and time limits, and preferably in a sandbox. This lab's
extract()raises a plain exception instead, which is the behaviour you must handle either way. - Ingestion is where personal data enters the system, so it is the right place to redact it, and the right place to honour deletion. A source record deleted for legal reasons must be removed from the index and any cache; a deletion that does not propagate is a compliance failure rather than a stale cache.
Reversing everything this lab did
rm -rf __pycache__ .pytest_cache .venv
Nothing else was created, and nothing outside this directory was modified.