AI Engineering: Agents and Applications › Production Retrieval and Pipelines › Day 324
Hands-on lab — Day 324: Document Processing at Scale
- ← Back to the Day 324 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/ai-engineering/day-324-document-processing-at-scale/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/process_demo.py Test
bash tests/run_tests.sh File tree
examples/corpus.py examples/process_demo.py examples/process.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/process.py tests/run_tests.sh tests/test_process.py troubleshooting.md
Lab README
Lab — Day 324: Document Processing at Scale
Lesson
- Lesson title: Document Processing at Scale
- Day number: 324 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-324-document-processing-at-scale
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-324-document-processing-at-scalewhen the site is running.
Purpose
Build a document processing pipeline that turns arbitrary files into clean text without silently corrupting anything. You implement byte-level format detection, a per-format extractor registry, a hard time budget enforced by a process you can actually kill, and a quality gate with three outcomes.
Everything runs offline against eight synthetic documents. Each one reproduces a real failure: a scan with no text layer, mojibake, a file that never returns, and an unknown format.
Learning objectives
- Detect a document's real format from its leading bytes rather than its name.
- Dispatch to per-format extractors through a registry instead of a growing conditional.
- Enforce a per-document budget from the harness, and understand why a thread cannot provide one.
- Score extracted text on yield, alphabetic ratio and word length.
- Justify three outcomes — accepted, flagged, dead — over a binary decision.
Prerequisites
- Day 323, "Data Ingestion Pipelines".
- Comfortable with Python dataclasses and exceptions.
- Aware of what
multiprocessingdoes, though the lab supplies the harness.
Supported operating systems
macOS, Linux and Windows (via WSL). The budget uses multiprocessing, which behaves consistently across all three; on Windows and macOS the child is spawned rather than forked, which the code accounts for.
Hardware requirements
Any machine running Python 3.10 or newer. No GPU. Peak memory is a few megabytes.
Required software
- Python 3.10 or newer (uses
X | Nonetype syntax). pytest7 or newer.
Free and open-source options
Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no API key, no hosted service. The lesson discusses Tesseract, pdfplumber and Docling as the real-world open-source options; none is required here.
Installation
python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt
Removing .venv fully reverses the installation.
File structure
starter/process.py your work: detect, score, process_one
examples/process.py reference implementation
examples/corpus.py eight synthetic documents
examples/process_demo.py runs the corpus and prints the table
tests/test_process.py grouped by concern
tests/run_tests.sh suite entry point
expected-output/ real captured output and measured values
requirements/ pinned dependency
How to run
python3 examples/process_demo.py # process the corpus
bash tests/run_tests.sh # run the suite
To work on the exercise, edit starter/process.py, then copy it over the reference:
cp starter/process.py examples/process.py
bash tests/run_tests.sh
Keep a copy of the reference first if you want to compare.
What the commands do
python3 examples/process_demo.py processes eight documents and prints one line each plus a summary. It takes about 2.4 seconds, almost all of which is doc-06.slow exhausting its 2-second budget before being terminated.
bash tests/run_tests.sh runs pytest over fourteen tests grouped by concern — detection, dispatch, budget, scoring and gating — so a failure names what is broken.
Expected output
doc-01.txt accepted yield=1.00 alpha=0.98 words=4.8
doc-02.pdf accepted yield=0.98 alpha=0.98 words=4.8
doc-03.scan flagged yield=0.02 alpha=0.81 words=3.2 low text yield
doc-04.bin flagged yield=1.00 alpha=0.00 words=600.0 low alphabetic ratio
doc-05.html accepted yield=0.61 alpha=0.98 words=5.3
doc-06.slow dead timeout after 2.00s
doc-07.xyz dead no extractor for format 'unknown'
doc-08.pdf accepted yield=0.96 alpha=0.98 words=4.8
summary: accepted=4 flagged=2 dead=2
The two flagged documents are caught by different signals: the scan by yield alone (its few characters are all letters), the mojibake by alphabetic ratio alone (every byte decoded, so its yield is perfect). expected-output/FIELDS.md explains every column.
Validation steps
bash tests/run_tests.shreports14 passed.- Time the demo: it must finish in roughly 2 to 3 seconds. If it takes 30, your budget is not being enforced — the pathological extractor sleeps for 30 seconds.
- Confirm
doc-04.binisflaggedand notaccepted. Extraction "succeeded" for it, which is exactly why the gate exists.
Tests
Fourteen tests in five groups:
- detect — content is read rather than the extension; a scan is distinguished from a born-digital PDF.
- dispatch — an unknown format is dead-lettered rather than raising.
- budget — the pathological document times out in well under five seconds, and the rest of the run still completes.
- score — empty text does not divide by zero; yield is a ratio; the alphabetic ratio counts only letters.
- gate — each flagged document is caught by the expected signal, flagged documents keep their text, and the summary counts all three outcomes.
Cleanup
rm -rf __pycache__ .pytest_cache
rm -rf .venv
Both are safe to remove and fully reversible.
Troubleshooting
See troubleshooting.md for each failure this lab is designed to produce.
Security notes
See security.md. In short: no network, no credentials, no API key, and extraction runs in a child process — which is also how you would contain a real parser handling untrusted input.
Extension exercises
- A fifth signal. Add a repetition ratio to catch a page header repeated once per page, which passes all four existing signals. Prove the gap first, then close it.
- Escalation. Add a slow but capable second extractor and route only flagged documents to it. Measure how many documents were recovered and what it cost, and report the real numbers.
- Memory budget. Add a memory ceiling alongside the time budget using
resource.setrlimitin the child, and a document that would exceed it.
Navigation
- Lesson
- Previous: Day 323 — Data Ingestion Pipelines
- Next: Day 325 — Keeping Indexes Fresh
Expected output
FIELDS.md
# What each column in the run output means
doc-01.txt accepted yield=1.00 alpha=0.98 words=4.8
| Column | Meaning |
| --- | --- |
| name | The document. Two are deliberately named for formats that do not exist (`.scan`, `.slow`) so a pipeline that dispatches on the extension gets them wrong. |
| outcome | `accepted`, `flagged` or `dead`. Three outcomes, not two — see below. |
| `yield` | Extracted characters per source byte, capped at 1.0. A ratio, so a genuinely short document is not confused with a failed extraction of a long one. |
| `alpha` | Proportion of non-space characters that are letters. Mojibake and OCR noise score near zero. |
| `words` | Mean word length. Natural prose sits near 4-6; a page whose spaces were lost scores in the hundreds. |
| detail | For a flagged document, the first failing signal. For a dead one, why it died. |
## The three outcomes
- **accepted** — scored cleanly, goes to the index.
- **flagged** — produced text that scored badly. Still indexed, because partial text usually beats none, but recorded so it can be re-routed to a costlier extractor.
- **dead** — failed outright: a timeout, a crash, or a format with no registered extractor.
A binary accept-or-reject cannot express the middle case, and the middle case is where scans and encoding failures live.
examples-run.txt
doc-01.txt accepted yield=1.00 alpha=0.98 words=4.8
doc-02.pdf accepted yield=0.98 alpha=0.98 words=4.8
doc-03.scan flagged yield=0.02 alpha=0.81 words=3.2 low text yield
doc-04.bin flagged yield=1.00 alpha=0.00 words=600.0 low alphabetic ratio
doc-05.html accepted yield=0.61 alpha=0.98 words=5.3
doc-06.slow dead timeout after 2.00s
doc-07.xyz dead no extractor for format 'unknown'
doc-08.pdf accepted yield=0.96 alpha=0.98 words=4.8
summary: accepted=4 flagged=2 dead=2
measured-values.txt
Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.
python3 examples/process_demo.py -> 9 lines, exit 0, ~2.4s wall clock
bash tests/run_tests.sh -> 14 passed in 5.15s
The demo takes about 2.4 seconds because doc-06.slow deliberately exhausts its
full 2.00s budget before being terminated. The test suite passes a smaller
budget (0.3s) wherever the exact timeout value does not matter, which is why 14
tests finish in about 5 seconds rather than 17.
Scores worth noting, because each flagged document is caught by a DIFFERENT
signal:
doc-03.scan yield=0.02 alpha=0.81 -> flagged on yield alone
doc-04.bin yield=1.00 alpha=0.00 -> flagged on alphabetic ratio alone
Neither is caught by both. That is the argument for several cheap signals.
run-output.txt
doc-01.txt accepted yield=1.00 alpha=0.98 words=4.8
doc-02.pdf accepted yield=0.98 alpha=0.98 words=4.8
doc-03.scan flagged yield=0.02 alpha=0.81 words=3.2 low text yield
doc-04.bin flagged yield=1.00 alpha=0.00 words=600.0 low alphabetic ratio
doc-05.html accepted yield=0.61 alpha=0.98 words=5.3
doc-06.slow dead timeout after 2.00s
doc-07.xyz dead no extractor for format 'unknown'
doc-08.pdf accepted yield=0.96 alpha=0.98 words=4.8
summary: accepted=4 flagged=2 dead=2
starter-run.txt
An untouched starter fails every test:
14 failed in 0.06s
All three stubbed functions -- detect, score and process_one -- raise
NotImplementedError, and every test depends on at least one of them. Unlike
Day 323, there is no test that passes by accident here.
test-output.txt
.............. [100%]
14 passed in 5.12s
test-run.txt
.............. [100%]
14 passed in 5.12s
Source files
examples/corpus.py (2050 bytes)
"""Eight synthetic documents, each reproducing a real failure mode.
Two are deliberately named to look like formats that do not exist, so a
pipeline that dispatches on the extension gets them wrong.
"""
from __future__ import annotations
from process import Document
PROSE = (
"Retrieval quality depends on the text that reaches the index. "
"A document that extracts badly cannot be rescued by a better model. "
"Clean text is the input every later stage assumes it already has. "
)
def build_corpus() -> list[Document]:
return [
# Plain prose. The easy majority; should stay cheap.
Document("doc-01.txt", (PROSE * 3).encode("utf-8")),
# Born-digital PDF: has a font resource and a real text layer.
Document("doc-02.pdf", b"%PDF-1.7 /Font\n" + (PROSE * 3).encode("utf-8")),
# A scan: PDF header, no font resource, no text layer. Extraction
# returns a few stray characters rather than raising.
Document("doc-03.scan", b"%PDF-1.4\n" + b"\x00" * 900 + b"scanned page 1 of 40"),
# Mojibake: decodes, but almost nothing is a letter.
Document("doc-04.bin", ("=?#" * 200).encode("utf-8")),
# HTML where most of the bytes are navigation and boilerplate.
Document(
"doc-05.html",
(
"<html><head><title>t</title></head><body>"
"<nav><ul><li>Home</li><li>Docs</li><li>Pricing</li></ul></nav>"
f"<main><p>{PROSE}</p></main>"
"<footer><p>Cookie notice</p></footer></body></html>"
).encode("utf-8"),
),
# Pathological: the extractor never returns within any sane budget.
Document("doc-06.slow", b"SLOW:" + b"x" * 100),
# Unknown format: undecodable bytes with no recognised signature.
Document("doc-07.xyz", bytes([0xFF, 0xFE, 0x00, 0x01] * 50)),
# A second normal PDF, so the summary is not dominated by failures.
Document("doc-08.pdf", b"%PDF-1.7 /Font\n" + (PROSE * 2).encode("utf-8")),
]
examples/process_demo.py (521 bytes)
#!/usr/bin/env python3
"""Process the synthetic corpus and print one line per document."""
from __future__ import annotations
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from corpus import build_corpus
from process import format_line, process_all
def main() -> int:
report = process_all(build_corpus())
for result in report.results:
print(format_line(result))
print(report.summary())
return 0
if __name__ == "__main__":
raise SystemExit(main())
examples/process.py (8457 bytes)
"""Document processing with format dispatch, budgets and a quality gate.
Offline and standard-library only. The documents are synthetic, but each one
reproduces a real failure mode: a scan with no text layer, mojibake, a
pathological file that never returns, and an unknown format.
The pipeline has three outcomes rather than two. `accepted` goes downstream,
`dead` failed outright, and `flagged` produced text that scored badly -- still
indexed, because partial text usually beats none, but recorded for review. That
third outcome is what keeps the gate honest: a binary decision either discards
recoverable documents or silently admits garbage.
"""
from __future__ import annotations
import time
import multiprocessing as mp
from dataclasses import dataclass, field
from typing import Callable
# Cheap thresholds. None is conclusive alone; together they separate "fine"
# from "catastrophic", and catastrophic is what must not reach the index.
MIN_YIELD = 0.10 # extracted chars per source byte
MIN_ALPHA = 0.55 # proportion of characters that are letters
MAX_WORD_LEN = 12.0 # mean word length; runaway means spaces were lost
class UnknownFormat(RuntimeError):
"""Detection could not identify the document."""
@dataclass(frozen=True)
class Document:
name: str
data: bytes
@dataclass
class Scores:
text_yield: float
alpha_ratio: float
mean_word_len: float
def reasons(self) -> list[str]:
"""Why this text looks wrong. Empty means it looks fine."""
out: list[str] = []
if self.text_yield < MIN_YIELD:
out.append("low text yield")
if self.alpha_ratio < MIN_ALPHA:
out.append("low alphabetic ratio")
if self.mean_word_len > MAX_WORD_LEN:
out.append("implausible word length")
return out
@dataclass
class Result:
name: str
outcome: str # accepted | flagged | dead
text: str = ""
scores: Scores | None = None
detail: str = ""
@dataclass
class Report:
results: list[Result] = field(default_factory=list)
def count(self, outcome: str) -> int:
return sum(1 for r in self.results if r.outcome == outcome)
def summary(self) -> str:
return (
f"summary: accepted={self.count('accepted')} "
f"flagged={self.count('flagged')} dead={self.count('dead')}"
)
# ------------------------------------------------------------------- detect
def detect(doc: Document) -> str:
"""Identify the format from leading bytes, never from the extension.
The extension lies: a `.pdf` may be a scan, a `.txt` may be UTF-16. Two of
the demo documents are named to look like formats that do not exist,
precisely so a pipeline that trusts the name gets them wrong.
"""
data = doc.data
if data.startswith(b"%PDF-"):
# A PDF with no font resources and a large image per page is a scan,
# and needs a different route than a born-digital PDF.
return "pdf-scan" if b"/Font" not in data else "pdf"
if data.startswith(b"PK\x03\x04"):
return "office"
if data.lstrip()[:5].lower().startswith(b"<html") or b"<body" in data[:200].lower():
return "html"
if data.startswith(b"SLOW:"):
return "pathological"
try:
data.decode("utf-8")
except UnicodeDecodeError:
return "unknown"
return "text"
# ------------------------------------------------------------------ extract
def extract_text(doc: Document) -> str:
return doc.data.decode("utf-8", errors="replace")
def extract_pdf(doc: Document) -> str:
"""Stand-in for a born-digital PDF with a real text layer."""
body = doc.data.split(b"\n", 1)[1] if b"\n" in doc.data else b""
return body.decode("utf-8", errors="replace")
def extract_pdf_scan(doc: Document) -> str:
"""A scan has no text layer, so a direct extractor recovers almost nothing.
It does NOT raise. That is the whole problem: it returns a short string that
would enter the index silently without a quality gate.
"""
return "".join(chr(c) for c in doc.data[-24:] if 32 <= c < 127)
def extract_html(doc: Document) -> str:
"""Crudely strip tags. Real extraction would isolate the main content."""
text = doc.data.decode("utf-8", errors="replace")
out, depth = [], 0
for ch in text:
if ch == "<":
depth += 1
elif ch == ">":
depth = max(0, depth - 1)
elif depth == 0:
out.append(ch)
return " ".join("".join(out).split())
def extract_pathological(doc: Document) -> str:
"""Never returns within any sane budget. The harness must stop it."""
time.sleep(30)
return "unreachable"
EXTRACTORS: dict[str, Callable[[Document], str]] = {
"text": extract_text,
"pdf": extract_pdf,
"pdf-scan": extract_pdf_scan,
"html": extract_html,
"pathological": extract_pathological,
}
# -------------------------------------------------------------------- score
def score(doc: Document, text: str) -> Scores:
"""Four cheap signals over the extracted text.
Empty output is a common case, not an edge case, so every ratio guards its
denominator.
"""
source_len = max(1, len(doc.data))
text_yield = min(1.0, len(text) / source_len)
chars = [c for c in text if not c.isspace()]
alpha_ratio = (sum(1 for c in chars if c.isalpha()) / len(chars)) if chars else 0.0
words = text.split()
mean_word_len = (sum(len(w) for w in words) / len(words)) if words else 0.0
return Scores(text_yield=text_yield, alpha_ratio=alpha_ratio, mean_word_len=mean_word_len)
# ------------------------------------------------------------------ process
def _extract_worker(fmt: str, doc: Document, out: "mp.Queue") -> None:
"""Run one extractor in a child process and post the result back."""
try:
out.put(("ok", EXTRACTORS[fmt](doc)))
except Exception as exc: # noqa: BLE001 - any parser failure is a dead letter
out.put(("err", type(exc).__name__))
def extract_with_budget(fmt: str, doc: Document, budget_s: float) -> tuple[str, str]:
"""Run an extractor under a hard deadline. Returns (status, payload).
The budget is enforced by the HARNESS, not by the extractor. A parser stuck
in a tight loop cannot be interrupted cooperatively, and a thread cannot be
killed at all -- an earlier version of this lab used a thread pool and the
30-second sleep blocked the whole run anyway, because the pool's shutdown
waits for its workers. A child process can genuinely be terminated, which
is why production extraction belongs in one.
"""
queue: "mp.Queue" = mp.Queue()
child = mp.Process(target=_extract_worker, args=(fmt, doc, queue), daemon=True)
child.start()
child.join(timeout=budget_s)
if child.is_alive():
child.terminate()
child.join(timeout=1.0)
return ("timeout", f"timeout after {budget_s:.2f}s")
try:
status, payload = queue.get_nowait()
except Exception: # noqa: BLE001 - child died without posting
return ("error", "ExtractorCrashed")
return (status, payload)
def process_one(doc: Document, *, budget_s: float = 2.0) -> Result:
"""Detect, dispatch under a budget, score, then gate."""
fmt = detect(doc)
if fmt not in EXTRACTORS:
# An unrecognised format is a normal outcome, not an exception.
return Result(doc.name, "dead", detail=f"no extractor for format '{fmt}'")
status, payload = extract_with_budget(fmt, doc, budget_s)
if status != "ok":
return Result(doc.name, "dead", detail=payload)
text = payload
scores = score(doc, text)
reasons = scores.reasons()
if reasons:
# Flagged, not rejected: partial text usually beats none.
return Result(doc.name, "flagged", text=text, scores=scores, detail=reasons[0])
return Result(doc.name, "accepted", text=text, scores=scores)
def process_all(docs: list[Document], *, budget_s: float = 2.0) -> Report:
return Report(results=[process_one(d, budget_s=budget_s) for d in docs])
def format_line(r: Result) -> str:
if r.scores is None:
return f"{r.name:<15} {r.outcome:<10} {r.detail}"
s = r.scores
line = (
f"{r.name:<15} {r.outcome:<10} "
f"yield={s.text_yield:.2f} alpha={s.alpha_ratio:.2f} words={s.mean_word_len:.1f}"
)
return f"{line} {r.detail}" if r.detail else line
metadata.yml (828 bytes)
lesson_id: D324
day: 324
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/process_demo.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-30'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3 examples/process_demo.py
reproduced the documented table exactly in ~2.4s; bash tests/run_tests.sh -> 14 passed in 5.15s. The
untouched starter was verified to fail all 14. requires_network is false: the corpus is synthetic and
in-memory, and no test opens a socket. Extraction runs in a child process that is terminated on
timeout, so no worker outlives the run.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/process.py (8228 bytes)
"""Document processing with format dispatch, budgets and a quality gate.
Offline and standard-library only. The documents are synthetic, but each one
reproduces a real failure mode: a scan with no text layer, mojibake, a
pathological file that never returns, and an unknown format.
The pipeline has three outcomes rather than two. `accepted` goes downstream,
`dead` failed outright, and `flagged` produced text that scored badly -- still
indexed, because partial text usually beats none, but recorded for review. That
third outcome is what keeps the gate honest: a binary decision either discards
recoverable documents or silently admits garbage.
"""
from __future__ import annotations
import time
import multiprocessing as mp
from dataclasses import dataclass, field
from typing import Callable
# Cheap thresholds. None is conclusive alone; together they separate "fine"
# from "catastrophic", and catastrophic is what must not reach the index.
MIN_YIELD = 0.10 # extracted chars per source byte
MIN_ALPHA = 0.55 # proportion of characters that are letters
MAX_WORD_LEN = 12.0 # mean word length; runaway means spaces were lost
class UnknownFormat(RuntimeError):
"""Detection could not identify the document."""
@dataclass(frozen=True)
class Document:
name: str
data: bytes
@dataclass
class Scores:
text_yield: float
alpha_ratio: float
mean_word_len: float
def reasons(self) -> list[str]:
"""Why this text looks wrong. Empty means it looks fine."""
out: list[str] = []
if self.text_yield < MIN_YIELD:
out.append("low text yield")
if self.alpha_ratio < MIN_ALPHA:
out.append("low alphabetic ratio")
if self.mean_word_len > MAX_WORD_LEN:
out.append("implausible word length")
return out
@dataclass
class Result:
name: str
outcome: str # accepted | flagged | dead
text: str = ""
scores: Scores | None = None
detail: str = ""
@dataclass
class Report:
results: list[Result] = field(default_factory=list)
def count(self, outcome: str) -> int:
return sum(1 for r in self.results if r.outcome == outcome)
def summary(self) -> str:
return (
f"summary: accepted={self.count('accepted')} "
f"flagged={self.count('flagged')} dead={self.count('dead')}"
)
# ------------------------------------------------------------------- detect
def detect(doc: Document) -> str:
"""Identify the format from leading bytes, never from the extension.
The extension lies: a `.pdf` may be a scan, a `.txt` may be UTF-16. Two of
the demo documents are named to look like formats that do not exist,
precisely so a pipeline that trusts the name gets them wrong.
"""
# TASK 1: identify the format from doc.data's leading bytes.
# Return one of: "pdf", "pdf-scan", "office", "html", "pathological",
# "text", "unknown". A %PDF- header with no b"/Font" anywhere is a scan.
# b"SLOW:" marks the pathological document. Bytes that will not decode as
# UTF-8 are "unknown". Never look at doc.name.
raise NotImplementedError("implement detect")
# ------------------------------------------------------------------ extract
def extract_text(doc: Document) -> str:
return doc.data.decode("utf-8", errors="replace")
def extract_pdf(doc: Document) -> str:
"""Stand-in for a born-digital PDF with a real text layer."""
body = doc.data.split(b"\n", 1)[1] if b"\n" in doc.data else b""
return body.decode("utf-8", errors="replace")
def extract_pdf_scan(doc: Document) -> str:
"""A scan has no text layer, so a direct extractor recovers almost nothing.
It does NOT raise. That is the whole problem: it returns a short string that
would enter the index silently without a quality gate.
"""
return "".join(chr(c) for c in doc.data[-24:] if 32 <= c < 127)
def extract_html(doc: Document) -> str:
"""Crudely strip tags. Real extraction would isolate the main content."""
text = doc.data.decode("utf-8", errors="replace")
out, depth = [], 0
for ch in text:
if ch == "<":
depth += 1
elif ch == ">":
depth = max(0, depth - 1)
elif depth == 0:
out.append(ch)
return " ".join("".join(out).split())
def extract_pathological(doc: Document) -> str:
"""Never returns within any sane budget. The harness must stop it."""
time.sleep(30)
return "unreachable"
EXTRACTORS: dict[str, Callable[[Document], str]] = {
"text": extract_text,
"pdf": extract_pdf,
"pdf-scan": extract_pdf_scan,
"html": extract_html,
"pathological": extract_pathological,
}
# -------------------------------------------------------------------- score
def score(doc: Document, text: str) -> Scores:
"""Four cheap signals over the extracted text.
Empty output is a common case, not an edge case, so every ratio guards its
denominator.
"""
# TASK 2: compute the three signals.
# text_yield len(text) / len(doc.data), capped at 1.0 -- a RATIO, so a
# short document is not confused with a failed extraction
# alpha_ratio letters / non-space characters (letters only, not every
# non-space character)
# mean_word_len mean length of text.split()
# Empty output is a common case: guard every denominator.
raise NotImplementedError("implement score")
# ------------------------------------------------------------------ process
def _extract_worker(fmt: str, doc: Document, out: "mp.Queue") -> None:
"""Run one extractor in a child process and post the result back."""
try:
out.put(("ok", EXTRACTORS[fmt](doc)))
except Exception as exc: # noqa: BLE001 - any parser failure is a dead letter
out.put(("err", type(exc).__name__))
def extract_with_budget(fmt: str, doc: Document, budget_s: float) -> tuple[str, str]:
"""Run an extractor under a hard deadline. Returns (status, payload).
The budget is enforced by the HARNESS, not by the extractor. A parser stuck
in a tight loop cannot be interrupted cooperatively, and a thread cannot be
killed at all -- an earlier version of this lab used a thread pool and the
30-second sleep blocked the whole run anyway, because the pool's shutdown
waits for its workers. A child process can genuinely be terminated, which
is why production extraction belongs in one.
"""
queue: "mp.Queue" = mp.Queue()
child = mp.Process(target=_extract_worker, args=(fmt, doc, queue), daemon=True)
child.start()
child.join(timeout=budget_s)
if child.is_alive():
child.terminate()
child.join(timeout=1.0)
return ("timeout", f"timeout after {budget_s:.2f}s")
try:
status, payload = queue.get_nowait()
except Exception: # noqa: BLE001 - child died without posting
return ("error", "ExtractorCrashed")
return (status, payload)
def process_one(doc: Document, *, budget_s: float = 2.0) -> Result:
"""Detect, dispatch under a budget, score, then gate."""
# TASK 3: detect, dispatch under a budget, score, then gate.
# - a format with no registered extractor is a normal "dead" outcome
# with detail f"no extractor for format '{fmt}'", not an exception
# - run the extractor via extract_with_budget(fmt, doc, budget_s); a
# non-"ok" status is a "dead" Result carrying the returned payload
# - score the text; if scores.reasons() is non-empty the outcome is
# "flagged" (still carrying the text, detail = the first reason)
# - otherwise "accepted"
raise NotImplementedError("implement process_one")
def process_all(docs: list[Document], *, budget_s: float = 2.0) -> Report:
return Report(results=[process_one(d, budget_s=budget_s) for d in docs])
def format_line(r: Result) -> str:
if r.scores is None:
return f"{r.name:<15} {r.outcome:<10} {r.detail}"
s = r.scores
line = (
f"{r.name:<15} {r.outcome:<10} "
f"yield={s.text_yield:.2f} alpha={s.alpha_ratio:.2f} words={s.mean_word_len:.1f}"
)
return f"{line} {r.detail}" if r.detail else line
tests/run_tests.sh (209 bytes)
#!/usr/bin/env bash
# Run the day 324 document processing test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_process.py (5059 bytes)
"""Grouped by concern, so a failure names what is broken.
Run with: bash tests/run_tests.sh
"""
from __future__ import annotations
import os
import sys
import time
HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))
from corpus import build_corpus # noqa: E402
from process import ( # noqa: E402
Document,
detect,
format_line,
process_all,
process_one,
score,
)
def by_name(report, name):
return next(r for r in report.results if r.name == name)
# ------------------------------------------------------------------ detect
def test_detection_reads_content_not_the_extension():
# Named .scan and .slow -- formats that do not exist. Detection must not
# care about the name.
corpus = {d.name: d for d in build_corpus()}
assert detect(corpus["doc-03.scan"]) == "pdf-scan"
assert detect(corpus["doc-06.slow"]) == "pathological"
# A .txt name over undecodable bytes is still unknown.
assert detect(Document("innocent.txt", bytes([0xFF, 0xFE, 0x00]))) == "unknown"
def test_scan_and_born_digital_pdf_are_distinguished():
born = Document("a.pdf", b"%PDF-1.7 /Font\nhello world")
scan = Document("b.pdf", b"%PDF-1.4\n" + b"\x00" * 100)
assert detect(born) == "pdf"
assert detect(scan) == "pdf-scan"
# ---------------------------------------------------------------- dispatch
def test_unknown_format_is_dead_lettered_not_raised():
result = process_one(Document("x.xyz", bytes([0xFF, 0xFE, 0x00, 0x01] * 10)))
assert result.outcome == "dead"
assert "no extractor" in result.detail
# ------------------------------------------------------------------ budget
def test_pathological_document_times_out_and_does_not_hang():
doc = Document("slow.bin", b"SLOW:" + b"x" * 10)
started = time.monotonic()
result = process_one(doc, budget_s=0.5)
elapsed = time.monotonic() - started
assert result.outcome == "dead"
assert "timeout" in result.detail
# The extractor sleeps for 30s; the harness must give up long before.
assert elapsed < 5.0, f"budget not enforced, took {elapsed:.1f}s"
def test_budget_does_not_stop_the_rest_of_the_run():
report = process_all(build_corpus(), budget_s=0.3)
assert by_name(report, "doc-06.slow").outcome == "dead"
# Everything after the pathological document still processed.
assert by_name(report, "doc-08.pdf").outcome == "accepted"
# ------------------------------------------------------------------- score
def test_scores_handle_empty_text_without_dividing_by_zero():
s = score(Document("e.txt", b"abc"), "")
assert s.text_yield == 0.0
assert s.alpha_ratio == 0.0
assert s.mean_word_len == 0.0
def test_yield_is_a_ratio_not_a_length():
# A genuinely short document is fine; a failed extraction of a big one is
# not. Only the ratio tells them apart.
short = score(Document("s.txt", b"hi"), "hi")
failed = score(Document("b.txt", b"x" * 1000), "hi")
assert short.text_yield > failed.text_yield
assert short.text_yield >= 0.9
assert failed.text_yield < 0.1
def test_alphabetic_ratio_counts_only_letters():
s = score(Document("m.txt", b"=?#" * 10), "=?#" * 10)
assert s.alpha_ratio == 0.0
# -------------------------------------------------------------------- gate
def test_scan_is_flagged_by_low_yield_not_by_alphabetic_ratio():
report = process_all(build_corpus(), budget_s=0.3)
scan = by_name(report, "doc-03.scan")
assert scan.outcome == "flagged"
assert scan.detail == "low text yield"
# Its letters ARE letters -- only the yield reveals the failure.
assert scan.scores.alpha_ratio > 0.5
def test_mojibake_is_flagged_by_alphabetic_ratio_not_by_yield():
report = process_all(build_corpus(), budget_s=0.3)
junk = by_name(report, "doc-04.bin")
assert junk.outcome == "flagged"
assert junk.detail == "low alphabetic ratio"
# Every byte decoded, so yield is perfect. The mirror image of the scan.
assert junk.scores.text_yield >= 0.99
def test_flagged_documents_still_carry_their_text():
# Flagged is not rejected: partial text usually beats none.
report = process_all(build_corpus(), budget_s=0.3)
for name in ("doc-03.scan", "doc-04.bin"):
assert by_name(report, name).text != ""
def test_summary_counts_all_three_outcomes():
report = process_all(build_corpus(), budget_s=0.3)
assert report.summary() == "summary: accepted=4 flagged=2 dead=2"
assert len(report.results) == 8
def test_healthy_documents_are_accepted():
report = process_all(build_corpus(), budget_s=0.3)
for name in ("doc-01.txt", "doc-02.pdf", "doc-05.html", "doc-08.pdf"):
assert by_name(report, name).outcome == "accepted"
def test_format_line_is_stable_for_dead_and_scored_results():
report = process_all(build_corpus(), budget_s=0.3)
assert "timeout" in format_line(by_name(report, "doc-06.slow"))
assert "yield=" in format_line(by_name(report, "doc-01.txt"))
Troubleshooting
Troubleshooting — Day 324
The demo takes 30 seconds instead of about 2.4
Your budget is not being enforced. extract_pathological sleeps for 30 seconds, and it will not stop on its own.
The common cause is trying to enforce the deadline with a thread. A thread cannot be killed, and ThreadPoolExecutor used as a context manager waits for its workers on exit — so future.result(timeout=...) returns but the with block then blocks for the full 30 seconds anyway. This lab's first version had exactly that bug. Use extract_with_budget, which runs the extractor in a child process and terminates it.
Everything is accepted, including doc-03.scan
Your yield signal is a length rather than a ratio. A genuinely short document and a failed extraction of a long one both produce little text; only len(text) / len(doc.data) distinguishes them.
Check with:
python3 -c "
import sys; sys.path.insert(0,'examples')
from process import score, Document
print(score(Document('s',b'hi'),'hi').text_yield) # near 1.0
print(score(Document('b',b'x'*1000),'hi').text_yield) # near 0.0"
doc-04.bin is accepted
Your alphabetic ratio counts every non-space character rather than only letters. =?# are not letters. Use str.isalpha().
doc-07.xyz raises instead of being dead-lettered
An unrecognised format is a normal outcome. Check fmt not in EXTRACTORS and return a dead Result with f"no extractor for format '{fmt}'" rather than raising or looking up a missing key.
test_detection_reads_content_not_the_extension fails
You are branching on doc.name. Detection must look only at doc.data. doc-03.scan and doc-06.slow are named for formats that do not exist precisely to catch this.
ZeroDivisionError in score
Empty extracted text is a common case, not an edge case — a scan can yield nothing at all. Guard every denominator, and return 0.0 rather than raising.
The suite takes 17 seconds
Tests that do not care about the exact timeout should pass a smaller budget: process_all(build_corpus(), budget_s=0.3). Only the test that asserts on timeout behaviour needs a realistic one.
RuntimeError about the start method, or the demo hangs on Windows
multiprocessing needs the entry point guarded. examples/process_demo.py already has if __name__ == "__main__":; keep it if you write your own harness.
NotImplementedError everywhere
Expected. The starter stubs detect, score and process_one, and every test depends on at least one of them — see expected-output/starter-run.txt.
Security notes
Security notes — Day 324
What this lab does and does not touch
- No network. The corpus is eight synthetic in-memory documents. No test opens a socket, and
requires_networkisfalse. - No credentials. No API key, token or account anywhere.
- No writes outside this directory. Only
__pycache__/,.pytest_cache/and, if you create it,.venv/— all inside this lab and all removable with the documented cleanup. - No
sudo, no global installs. - Child processes are cleaned up. Extraction runs in a
multiprocessingchild markeddaemon=Trueand is explicitly terminated on timeout, then joined. No worker outlives the run.
Why extraction runs in a child process, and why that matters beyond this lab
The budget in this lab exists to stop a synthetic sleep. In production the same mechanism is a genuine security control:
- Document parsers are a real attack surface. They are typically complex C libraries consuming fully untrusted input, and they have a long history of memory-safety vulnerabilities. Decompression bombs, deeply nested structures and malformed cross-reference tables are all used in practice.
- A process boundary is what makes a limit enforceable. A thread cannot be killed, so a cooperative timeout only works on code that cooperates — which excludes exactly the inputs you are defending against. An earlier version of this lab used a thread pool and the 30-second sleep blocked the whole run.
- In production, add a memory ceiling too (
resource.setrlimitin the child), and prefer a sandbox where one is available. Extension exercise 3 covers the memory half. - Never let a parser fetch remote resources referenced by the document. That turns document processing into a server-side request forgery primitive.
Privacy
Extraction is the point where document contents become plain, greppable text — so it is the right stage for redaction, and the stage where sending content to a third-party OCR or document-intelligence API becomes a data-residency decision rather than a technical one. This lab sends nothing anywhere.
Reversing everything this lab did
rm -rf __pycache__ .pytest_cache .venv
Nothing else was created, and nothing outside this directory was modified.