AI Engineering: Agents and Applications › Production Retrieval and Pipelines › Day 324

Hands-on lab — Day 324: Document Processing at Scale

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/process_demo.py

Test

bash tests/run_tests.sh

File tree

examples/corpus.py
examples/process_demo.py
examples/process.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/process.py
tests/run_tests.sh
tests/test_process.py
troubleshooting.md

Lab README

Lab — Day 324: Document Processing at Scale

Lesson

Purpose

Build a document processing pipeline that turns arbitrary files into clean text without silently corrupting anything. You implement byte-level format detection, a per-format extractor registry, a hard time budget enforced by a process you can actually kill, and a quality gate with three outcomes.

Everything runs offline against eight synthetic documents. Each one reproduces a real failure: a scan with no text layer, mojibake, a file that never returns, and an unknown format.

Learning objectives

  • Detect a document's real format from its leading bytes rather than its name.
  • Dispatch to per-format extractors through a registry instead of a growing conditional.
  • Enforce a per-document budget from the harness, and understand why a thread cannot provide one.
  • Score extracted text on yield, alphabetic ratio and word length.
  • Justify three outcomes — accepted, flagged, dead — over a binary decision.

Prerequisites

  • Day 323, "Data Ingestion Pipelines".
  • Comfortable with Python dataclasses and exceptions.
  • Aware of what multiprocessing does, though the lab supplies the harness.

Supported operating systems

macOS, Linux and Windows (via WSL). The budget uses multiprocessing, which behaves consistently across all three; on Windows and macOS the child is spawned rather than forked, which the code accounts for.

Hardware requirements

Any machine running Python 3.10 or newer. No GPU. Peak memory is a few megabytes.

Required software

  • Python 3.10 or newer (uses X | None type syntax).
  • pytest 7 or newer.

Free and open-source options

Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no API key, no hosted service. The lesson discusses Tesseract, pdfplumber and Docling as the real-world open-source options; none is required here.

Installation

python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt

Removing .venv fully reverses the installation.

File structure

starter/process.py         your work: detect, score, process_one
examples/process.py        reference implementation
examples/corpus.py         eight synthetic documents
examples/process_demo.py   runs the corpus and prints the table
tests/test_process.py      grouped by concern
tests/run_tests.sh         suite entry point
expected-output/           real captured output and measured values
requirements/              pinned dependency

How to run

python3 examples/process_demo.py    # process the corpus
bash tests/run_tests.sh             # run the suite

To work on the exercise, edit starter/process.py, then copy it over the reference:

cp starter/process.py examples/process.py
bash tests/run_tests.sh

Keep a copy of the reference first if you want to compare.

What the commands do

python3 examples/process_demo.py processes eight documents and prints one line each plus a summary. It takes about 2.4 seconds, almost all of which is doc-06.slow exhausting its 2-second budget before being terminated.

bash tests/run_tests.sh runs pytest over fourteen tests grouped by concern — detection, dispatch, budget, scoring and gating — so a failure names what is broken.

Expected output

doc-01.txt      accepted   yield=1.00 alpha=0.98 words=4.8
doc-02.pdf      accepted   yield=0.98 alpha=0.98 words=4.8
doc-03.scan     flagged    yield=0.02 alpha=0.81 words=3.2   low text yield
doc-04.bin      flagged    yield=1.00 alpha=0.00 words=600.0   low alphabetic ratio
doc-05.html     accepted   yield=0.61 alpha=0.98 words=5.3
doc-06.slow     dead       timeout after 2.00s
doc-07.xyz      dead       no extractor for format 'unknown'
doc-08.pdf      accepted   yield=0.96 alpha=0.98 words=4.8
summary: accepted=4 flagged=2 dead=2

The two flagged documents are caught by different signals: the scan by yield alone (its few characters are all letters), the mojibake by alphabetic ratio alone (every byte decoded, so its yield is perfect). expected-output/FIELDS.md explains every column.

Validation steps

  1. bash tests/run_tests.sh reports 14 passed.
  2. Time the demo: it must finish in roughly 2 to 3 seconds. If it takes 30, your budget is not being enforced — the pathological extractor sleeps for 30 seconds.
  3. Confirm doc-04.bin is flagged and not accepted. Extraction "succeeded" for it, which is exactly why the gate exists.

Tests

Fourteen tests in five groups:

  • detect — content is read rather than the extension; a scan is distinguished from a born-digital PDF.
  • dispatch — an unknown format is dead-lettered rather than raising.
  • budget — the pathological document times out in well under five seconds, and the rest of the run still completes.
  • score — empty text does not divide by zero; yield is a ratio; the alphabetic ratio counts only letters.
  • gate — each flagged document is caught by the expected signal, flagged documents keep their text, and the summary counts all three outcomes.

Cleanup

rm -rf __pycache__ .pytest_cache
rm -rf .venv

Both are safe to remove and fully reversible.

Troubleshooting

See troubleshooting.md for each failure this lab is designed to produce.

Security notes

See security.md. In short: no network, no credentials, no API key, and extraction runs in a child process — which is also how you would contain a real parser handling untrusted input.

Extension exercises

  1. A fifth signal. Add a repetition ratio to catch a page header repeated once per page, which passes all four existing signals. Prove the gap first, then close it.
  2. Escalation. Add a slow but capable second extractor and route only flagged documents to it. Measure how many documents were recovered and what it cost, and report the real numbers.
  3. Memory budget. Add a memory ceiling alongside the time budget using resource.setrlimit in the child, and a document that would exceed it.
  • Lesson
  • Previous: Day 323 — Data Ingestion Pipelines
  • Next: Day 325 — Keeping Indexes Fresh

Expected output

FIELDS.md

# What each column in the run output means

    doc-01.txt      accepted   yield=1.00 alpha=0.98 words=4.8

| Column | Meaning |
| --- | --- |
| name | The document. Two are deliberately named for formats that do not exist (`.scan`, `.slow`) so a pipeline that dispatches on the extension gets them wrong. |
| outcome | `accepted`, `flagged` or `dead`. Three outcomes, not two — see below. |
| `yield` | Extracted characters per source byte, capped at 1.0. A ratio, so a genuinely short document is not confused with a failed extraction of a long one. |
| `alpha` | Proportion of non-space characters that are letters. Mojibake and OCR noise score near zero. |
| `words` | Mean word length. Natural prose sits near 4-6; a page whose spaces were lost scores in the hundreds. |
| detail | For a flagged document, the first failing signal. For a dead one, why it died. |

## The three outcomes

- **accepted** — scored cleanly, goes to the index.
- **flagged** — produced text that scored badly. Still indexed, because partial text usually beats none, but recorded so it can be re-routed to a costlier extractor.
- **dead** — failed outright: a timeout, a crash, or a format with no registered extractor.

A binary accept-or-reject cannot express the middle case, and the middle case is where scans and encoding failures live.

examples-run.txt

doc-01.txt      accepted   yield=1.00 alpha=0.98 words=4.8
doc-02.pdf      accepted   yield=0.98 alpha=0.98 words=4.8
doc-03.scan     flagged    yield=0.02 alpha=0.81 words=3.2   low text yield
doc-04.bin      flagged    yield=1.00 alpha=0.00 words=600.0   low alphabetic ratio
doc-05.html     accepted   yield=0.61 alpha=0.98 words=5.3
doc-06.slow     dead       timeout after 2.00s
doc-07.xyz      dead       no extractor for format 'unknown'
doc-08.pdf      accepted   yield=0.96 alpha=0.98 words=4.8
summary: accepted=4 flagged=2 dead=2

measured-values.txt

Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.

  python3 examples/process_demo.py   -> 9 lines, exit 0, ~2.4s wall clock
  bash tests/run_tests.sh            -> 14 passed in 5.15s

The demo takes about 2.4 seconds because doc-06.slow deliberately exhausts its
full 2.00s budget before being terminated. The test suite passes a smaller
budget (0.3s) wherever the exact timeout value does not matter, which is why 14
tests finish in about 5 seconds rather than 17.

Scores worth noting, because each flagged document is caught by a DIFFERENT
signal:

  doc-03.scan   yield=0.02  alpha=0.81  -> flagged on yield alone
  doc-04.bin    yield=1.00  alpha=0.00  -> flagged on alphabetic ratio alone

Neither is caught by both. That is the argument for several cheap signals.

run-output.txt

doc-01.txt      accepted   yield=1.00 alpha=0.98 words=4.8
doc-02.pdf      accepted   yield=0.98 alpha=0.98 words=4.8
doc-03.scan     flagged    yield=0.02 alpha=0.81 words=3.2   low text yield
doc-04.bin      flagged    yield=1.00 alpha=0.00 words=600.0   low alphabetic ratio
doc-05.html     accepted   yield=0.61 alpha=0.98 words=5.3
doc-06.slow     dead       timeout after 2.00s
doc-07.xyz      dead       no extractor for format 'unknown'
doc-08.pdf      accepted   yield=0.96 alpha=0.98 words=4.8
summary: accepted=4 flagged=2 dead=2

starter-run.txt

An untouched starter fails every test:

    14 failed in 0.06s

All three stubbed functions -- detect, score and process_one -- raise
NotImplementedError, and every test depends on at least one of them. Unlike
Day 323, there is no test that passes by accident here.

test-output.txt

..............                                                           [100%]
14 passed in 5.12s

test-run.txt

..............                                                           [100%]
14 passed in 5.12s

Source files

examples/corpus.py (2050 bytes)
"""Eight synthetic documents, each reproducing a real failure mode.

Two are deliberately named to look like formats that do not exist, so a
pipeline that dispatches on the extension gets them wrong.
"""

from __future__ import annotations

from process import Document

PROSE = (
    "Retrieval quality depends on the text that reaches the index. "
    "A document that extracts badly cannot be rescued by a better model. "
    "Clean text is the input every later stage assumes it already has. "
)


def build_corpus() -> list[Document]:
    return [
        # Plain prose. The easy majority; should stay cheap.
        Document("doc-01.txt", (PROSE * 3).encode("utf-8")),
        # Born-digital PDF: has a font resource and a real text layer.
        Document("doc-02.pdf", b"%PDF-1.7 /Font\n" + (PROSE * 3).encode("utf-8")),
        # A scan: PDF header, no font resource, no text layer. Extraction
        # returns a few stray characters rather than raising.
        Document("doc-03.scan", b"%PDF-1.4\n" + b"\x00" * 900 + b"scanned page 1 of 40"),
        # Mojibake: decodes, but almost nothing is a letter.
        Document("doc-04.bin", ("=?#" * 200).encode("utf-8")),
        # HTML where most of the bytes are navigation and boilerplate.
        Document(
            "doc-05.html",
            (
                "<html><head><title>t</title></head><body>"
                "<nav><ul><li>Home</li><li>Docs</li><li>Pricing</li></ul></nav>"
                f"<main><p>{PROSE}</p></main>"
                "<footer><p>Cookie notice</p></footer></body></html>"
            ).encode("utf-8"),
        ),
        # Pathological: the extractor never returns within any sane budget.
        Document("doc-06.slow", b"SLOW:" + b"x" * 100),
        # Unknown format: undecodable bytes with no recognised signature.
        Document("doc-07.xyz", bytes([0xFF, 0xFE, 0x00, 0x01] * 50)),
        # A second normal PDF, so the summary is not dominated by failures.
        Document("doc-08.pdf", b"%PDF-1.7 /Font\n" + (PROSE * 2).encode("utf-8")),
    ]
examples/process_demo.py (521 bytes)
#!/usr/bin/env python3
"""Process the synthetic corpus and print one line per document."""

from __future__ import annotations

import os
import sys

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

from corpus import build_corpus
from process import format_line, process_all


def main() -> int:
    report = process_all(build_corpus())
    for result in report.results:
        print(format_line(result))
    print(report.summary())
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
examples/process.py (8457 bytes)
"""Document processing with format dispatch, budgets and a quality gate.

Offline and standard-library only. The documents are synthetic, but each one
reproduces a real failure mode: a scan with no text layer, mojibake, a
pathological file that never returns, and an unknown format.

The pipeline has three outcomes rather than two. `accepted` goes downstream,
`dead` failed outright, and `flagged` produced text that scored badly -- still
indexed, because partial text usually beats none, but recorded for review. That
third outcome is what keeps the gate honest: a binary decision either discards
recoverable documents or silently admits garbage.
"""

from __future__ import annotations

import time
import multiprocessing as mp
from dataclasses import dataclass, field
from typing import Callable

# Cheap thresholds. None is conclusive alone; together they separate "fine"
# from "catastrophic", and catastrophic is what must not reach the index.
MIN_YIELD = 0.10  # extracted chars per source byte
MIN_ALPHA = 0.55  # proportion of characters that are letters
MAX_WORD_LEN = 12.0  # mean word length; runaway means spaces were lost


class UnknownFormat(RuntimeError):
    """Detection could not identify the document."""


@dataclass(frozen=True)
class Document:
    name: str
    data: bytes


@dataclass
class Scores:
    text_yield: float
    alpha_ratio: float
    mean_word_len: float

    def reasons(self) -> list[str]:
        """Why this text looks wrong. Empty means it looks fine."""
        out: list[str] = []
        if self.text_yield < MIN_YIELD:
            out.append("low text yield")
        if self.alpha_ratio < MIN_ALPHA:
            out.append("low alphabetic ratio")
        if self.mean_word_len > MAX_WORD_LEN:
            out.append("implausible word length")
        return out


@dataclass
class Result:
    name: str
    outcome: str  # accepted | flagged | dead
    text: str = ""
    scores: Scores | None = None
    detail: str = ""


@dataclass
class Report:
    results: list[Result] = field(default_factory=list)

    def count(self, outcome: str) -> int:
        return sum(1 for r in self.results if r.outcome == outcome)

    def summary(self) -> str:
        return (
            f"summary: accepted={self.count('accepted')} "
            f"flagged={self.count('flagged')} dead={self.count('dead')}"
        )


# ------------------------------------------------------------------- detect


def detect(doc: Document) -> str:
    """Identify the format from leading bytes, never from the extension.

    The extension lies: a `.pdf` may be a scan, a `.txt` may be UTF-16. Two of
    the demo documents are named to look like formats that do not exist,
    precisely so a pipeline that trusts the name gets them wrong.
    """
    data = doc.data
    if data.startswith(b"%PDF-"):
        # A PDF with no font resources and a large image per page is a scan,
        # and needs a different route than a born-digital PDF.
        return "pdf-scan" if b"/Font" not in data else "pdf"
    if data.startswith(b"PK\x03\x04"):
        return "office"
    if data.lstrip()[:5].lower().startswith(b"<html") or b"<body" in data[:200].lower():
        return "html"
    if data.startswith(b"SLOW:"):
        return "pathological"
    try:
        data.decode("utf-8")
    except UnicodeDecodeError:
        return "unknown"
    return "text"


# ------------------------------------------------------------------ extract


def extract_text(doc: Document) -> str:
    return doc.data.decode("utf-8", errors="replace")


def extract_pdf(doc: Document) -> str:
    """Stand-in for a born-digital PDF with a real text layer."""
    body = doc.data.split(b"\n", 1)[1] if b"\n" in doc.data else b""
    return body.decode("utf-8", errors="replace")


def extract_pdf_scan(doc: Document) -> str:
    """A scan has no text layer, so a direct extractor recovers almost nothing.

    It does NOT raise. That is the whole problem: it returns a short string that
    would enter the index silently without a quality gate.
    """
    return "".join(chr(c) for c in doc.data[-24:] if 32 <= c < 127)


def extract_html(doc: Document) -> str:
    """Crudely strip tags. Real extraction would isolate the main content."""
    text = doc.data.decode("utf-8", errors="replace")
    out, depth = [], 0
    for ch in text:
        if ch == "<":
            depth += 1
        elif ch == ">":
            depth = max(0, depth - 1)
        elif depth == 0:
            out.append(ch)
    return " ".join("".join(out).split())


def extract_pathological(doc: Document) -> str:
    """Never returns within any sane budget. The harness must stop it."""
    time.sleep(30)
    return "unreachable"


EXTRACTORS: dict[str, Callable[[Document], str]] = {
    "text": extract_text,
    "pdf": extract_pdf,
    "pdf-scan": extract_pdf_scan,
    "html": extract_html,
    "pathological": extract_pathological,
}


# -------------------------------------------------------------------- score


def score(doc: Document, text: str) -> Scores:
    """Four cheap signals over the extracted text.

    Empty output is a common case, not an edge case, so every ratio guards its
    denominator.
    """
    source_len = max(1, len(doc.data))
    text_yield = min(1.0, len(text) / source_len)

    chars = [c for c in text if not c.isspace()]
    alpha_ratio = (sum(1 for c in chars if c.isalpha()) / len(chars)) if chars else 0.0

    words = text.split()
    mean_word_len = (sum(len(w) for w in words) / len(words)) if words else 0.0

    return Scores(text_yield=text_yield, alpha_ratio=alpha_ratio, mean_word_len=mean_word_len)


# ------------------------------------------------------------------ process


def _extract_worker(fmt: str, doc: Document, out: "mp.Queue") -> None:
    """Run one extractor in a child process and post the result back."""
    try:
        out.put(("ok", EXTRACTORS[fmt](doc)))
    except Exception as exc:  # noqa: BLE001 - any parser failure is a dead letter
        out.put(("err", type(exc).__name__))


def extract_with_budget(fmt: str, doc: Document, budget_s: float) -> tuple[str, str]:
    """Run an extractor under a hard deadline. Returns (status, payload).

    The budget is enforced by the HARNESS, not by the extractor. A parser stuck
    in a tight loop cannot be interrupted cooperatively, and a thread cannot be
    killed at all -- an earlier version of this lab used a thread pool and the
    30-second sleep blocked the whole run anyway, because the pool's shutdown
    waits for its workers. A child process can genuinely be terminated, which
    is why production extraction belongs in one.
    """
    queue: "mp.Queue" = mp.Queue()
    child = mp.Process(target=_extract_worker, args=(fmt, doc, queue), daemon=True)
    child.start()
    child.join(timeout=budget_s)

    if child.is_alive():
        child.terminate()
        child.join(timeout=1.0)
        return ("timeout", f"timeout after {budget_s:.2f}s")

    try:
        status, payload = queue.get_nowait()
    except Exception:  # noqa: BLE001 - child died without posting
        return ("error", "ExtractorCrashed")
    return (status, payload)


def process_one(doc: Document, *, budget_s: float = 2.0) -> Result:
    """Detect, dispatch under a budget, score, then gate."""
    fmt = detect(doc)
    if fmt not in EXTRACTORS:
        # An unrecognised format is a normal outcome, not an exception.
        return Result(doc.name, "dead", detail=f"no extractor for format '{fmt}'")

    status, payload = extract_with_budget(fmt, doc, budget_s)
    if status != "ok":
        return Result(doc.name, "dead", detail=payload)
    text = payload

    scores = score(doc, text)
    reasons = scores.reasons()
    if reasons:
        # Flagged, not rejected: partial text usually beats none.
        return Result(doc.name, "flagged", text=text, scores=scores, detail=reasons[0])
    return Result(doc.name, "accepted", text=text, scores=scores)


def process_all(docs: list[Document], *, budget_s: float = 2.0) -> Report:
    return Report(results=[process_one(d, budget_s=budget_s) for d in docs])


def format_line(r: Result) -> str:
    if r.scores is None:
        return f"{r.name:<15} {r.outcome:<10} {r.detail}"
    s = r.scores
    line = (
        f"{r.name:<15} {r.outcome:<10} "
        f"yield={s.text_yield:.2f} alpha={s.alpha_ratio:.2f} words={s.mean_word_len:.1f}"
    )
    return f"{line}   {r.detail}" if r.detail else line
metadata.yml (828 bytes)
lesson_id: D324
day: 324
kind: python
languages:
  - python
setup_commands:
  - pip install -r requirements/requirements.txt
run_commands:
  - python examples/process_demo.py
test_commands:
  - bash tests/run_tests.sh
cleanup_commands:
  - rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-30'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3 examples/process_demo.py
  reproduced the documented table exactly in ~2.4s; bash tests/run_tests.sh -> 14 passed in 5.15s. The
  untouched starter was verified to fail all 14. requires_network is false: the corpus is synthetic and
  in-memory, and no test opens a socket. Extraction runs in a child process that is terminated on
  timeout, so no worker outlives the run.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/process.py (8228 bytes)
"""Document processing with format dispatch, budgets and a quality gate.

Offline and standard-library only. The documents are synthetic, but each one
reproduces a real failure mode: a scan with no text layer, mojibake, a
pathological file that never returns, and an unknown format.

The pipeline has three outcomes rather than two. `accepted` goes downstream,
`dead` failed outright, and `flagged` produced text that scored badly -- still
indexed, because partial text usually beats none, but recorded for review. That
third outcome is what keeps the gate honest: a binary decision either discards
recoverable documents or silently admits garbage.
"""

from __future__ import annotations

import time
import multiprocessing as mp
from dataclasses import dataclass, field
from typing import Callable

# Cheap thresholds. None is conclusive alone; together they separate "fine"
# from "catastrophic", and catastrophic is what must not reach the index.
MIN_YIELD = 0.10  # extracted chars per source byte
MIN_ALPHA = 0.55  # proportion of characters that are letters
MAX_WORD_LEN = 12.0  # mean word length; runaway means spaces were lost


class UnknownFormat(RuntimeError):
    """Detection could not identify the document."""


@dataclass(frozen=True)
class Document:
    name: str
    data: bytes


@dataclass
class Scores:
    text_yield: float
    alpha_ratio: float
    mean_word_len: float

    def reasons(self) -> list[str]:
        """Why this text looks wrong. Empty means it looks fine."""
        out: list[str] = []
        if self.text_yield < MIN_YIELD:
            out.append("low text yield")
        if self.alpha_ratio < MIN_ALPHA:
            out.append("low alphabetic ratio")
        if self.mean_word_len > MAX_WORD_LEN:
            out.append("implausible word length")
        return out


@dataclass
class Result:
    name: str
    outcome: str  # accepted | flagged | dead
    text: str = ""
    scores: Scores | None = None
    detail: str = ""


@dataclass
class Report:
    results: list[Result] = field(default_factory=list)

    def count(self, outcome: str) -> int:
        return sum(1 for r in self.results if r.outcome == outcome)

    def summary(self) -> str:
        return (
            f"summary: accepted={self.count('accepted')} "
            f"flagged={self.count('flagged')} dead={self.count('dead')}"
        )


# ------------------------------------------------------------------- detect


def detect(doc: Document) -> str:
    """Identify the format from leading bytes, never from the extension.

    The extension lies: a `.pdf` may be a scan, a `.txt` may be UTF-16. Two of
    the demo documents are named to look like formats that do not exist,
    precisely so a pipeline that trusts the name gets them wrong.
    """
    # TASK 1: identify the format from doc.data's leading bytes.
    # Return one of: "pdf", "pdf-scan", "office", "html", "pathological",
    # "text", "unknown". A %PDF- header with no b"/Font" anywhere is a scan.
    # b"SLOW:" marks the pathological document. Bytes that will not decode as
    # UTF-8 are "unknown". Never look at doc.name.
    raise NotImplementedError("implement detect")

# ------------------------------------------------------------------ extract


def extract_text(doc: Document) -> str:
    return doc.data.decode("utf-8", errors="replace")


def extract_pdf(doc: Document) -> str:
    """Stand-in for a born-digital PDF with a real text layer."""
    body = doc.data.split(b"\n", 1)[1] if b"\n" in doc.data else b""
    return body.decode("utf-8", errors="replace")


def extract_pdf_scan(doc: Document) -> str:
    """A scan has no text layer, so a direct extractor recovers almost nothing.

    It does NOT raise. That is the whole problem: it returns a short string that
    would enter the index silently without a quality gate.
    """
    return "".join(chr(c) for c in doc.data[-24:] if 32 <= c < 127)


def extract_html(doc: Document) -> str:
    """Crudely strip tags. Real extraction would isolate the main content."""
    text = doc.data.decode("utf-8", errors="replace")
    out, depth = [], 0
    for ch in text:
        if ch == "<":
            depth += 1
        elif ch == ">":
            depth = max(0, depth - 1)
        elif depth == 0:
            out.append(ch)
    return " ".join("".join(out).split())


def extract_pathological(doc: Document) -> str:
    """Never returns within any sane budget. The harness must stop it."""
    time.sleep(30)
    return "unreachable"


EXTRACTORS: dict[str, Callable[[Document], str]] = {
    "text": extract_text,
    "pdf": extract_pdf,
    "pdf-scan": extract_pdf_scan,
    "html": extract_html,
    "pathological": extract_pathological,
}


# -------------------------------------------------------------------- score


def score(doc: Document, text: str) -> Scores:
    """Four cheap signals over the extracted text.

    Empty output is a common case, not an edge case, so every ratio guards its
    denominator.
    """
    # TASK 2: compute the three signals.
    #   text_yield    len(text) / len(doc.data), capped at 1.0 -- a RATIO, so a
    #                 short document is not confused with a failed extraction
    #   alpha_ratio   letters / non-space characters (letters only, not every
    #                 non-space character)
    #   mean_word_len mean length of text.split()
    # Empty output is a common case: guard every denominator.
    raise NotImplementedError("implement score")

# ------------------------------------------------------------------ process


def _extract_worker(fmt: str, doc: Document, out: "mp.Queue") -> None:
    """Run one extractor in a child process and post the result back."""
    try:
        out.put(("ok", EXTRACTORS[fmt](doc)))
    except Exception as exc:  # noqa: BLE001 - any parser failure is a dead letter
        out.put(("err", type(exc).__name__))


def extract_with_budget(fmt: str, doc: Document, budget_s: float) -> tuple[str, str]:
    """Run an extractor under a hard deadline. Returns (status, payload).

    The budget is enforced by the HARNESS, not by the extractor. A parser stuck
    in a tight loop cannot be interrupted cooperatively, and a thread cannot be
    killed at all -- an earlier version of this lab used a thread pool and the
    30-second sleep blocked the whole run anyway, because the pool's shutdown
    waits for its workers. A child process can genuinely be terminated, which
    is why production extraction belongs in one.
    """
    queue: "mp.Queue" = mp.Queue()
    child = mp.Process(target=_extract_worker, args=(fmt, doc, queue), daemon=True)
    child.start()
    child.join(timeout=budget_s)

    if child.is_alive():
        child.terminate()
        child.join(timeout=1.0)
        return ("timeout", f"timeout after {budget_s:.2f}s")

    try:
        status, payload = queue.get_nowait()
    except Exception:  # noqa: BLE001 - child died without posting
        return ("error", "ExtractorCrashed")
    return (status, payload)


def process_one(doc: Document, *, budget_s: float = 2.0) -> Result:
    """Detect, dispatch under a budget, score, then gate."""
    # TASK 3: detect, dispatch under a budget, score, then gate.
    #   - a format with no registered extractor is a normal "dead" outcome
    #     with detail f"no extractor for format '{fmt}'", not an exception
    #   - run the extractor via extract_with_budget(fmt, doc, budget_s); a
    #     non-"ok" status is a "dead" Result carrying the returned payload
    #   - score the text; if scores.reasons() is non-empty the outcome is
    #     "flagged" (still carrying the text, detail = the first reason)
    #   - otherwise "accepted"
    raise NotImplementedError("implement process_one")

def process_all(docs: list[Document], *, budget_s: float = 2.0) -> Report:
    return Report(results=[process_one(d, budget_s=budget_s) for d in docs])


def format_line(r: Result) -> str:
    if r.scores is None:
        return f"{r.name:<15} {r.outcome:<10} {r.detail}"
    s = r.scores
    line = (
        f"{r.name:<15} {r.outcome:<10} "
        f"yield={s.text_yield:.2f} alpha={s.alpha_ratio:.2f} words={s.mean_word_len:.1f}"
    )
    return f"{line}   {r.detail}" if r.detail else line
tests/run_tests.sh (209 bytes)
#!/usr/bin/env bash
# Run the day 324 document processing test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_process.py (5059 bytes)
"""Grouped by concern, so a failure names what is broken.

Run with: bash tests/run_tests.sh
"""

from __future__ import annotations

import os
import sys
import time

HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))

from corpus import build_corpus  # noqa: E402
from process import (  # noqa: E402
    Document,
    detect,
    format_line,
    process_all,
    process_one,
    score,
)


def by_name(report, name):
    return next(r for r in report.results if r.name == name)


# ------------------------------------------------------------------ detect


def test_detection_reads_content_not_the_extension():
    # Named .scan and .slow -- formats that do not exist. Detection must not
    # care about the name.
    corpus = {d.name: d for d in build_corpus()}
    assert detect(corpus["doc-03.scan"]) == "pdf-scan"
    assert detect(corpus["doc-06.slow"]) == "pathological"
    # A .txt name over undecodable bytes is still unknown.
    assert detect(Document("innocent.txt", bytes([0xFF, 0xFE, 0x00]))) == "unknown"


def test_scan_and_born_digital_pdf_are_distinguished():
    born = Document("a.pdf", b"%PDF-1.7 /Font\nhello world")
    scan = Document("b.pdf", b"%PDF-1.4\n" + b"\x00" * 100)
    assert detect(born) == "pdf"
    assert detect(scan) == "pdf-scan"


# ---------------------------------------------------------------- dispatch


def test_unknown_format_is_dead_lettered_not_raised():
    result = process_one(Document("x.xyz", bytes([0xFF, 0xFE, 0x00, 0x01] * 10)))
    assert result.outcome == "dead"
    assert "no extractor" in result.detail


# ------------------------------------------------------------------ budget


def test_pathological_document_times_out_and_does_not_hang():
    doc = Document("slow.bin", b"SLOW:" + b"x" * 10)
    started = time.monotonic()
    result = process_one(doc, budget_s=0.5)
    elapsed = time.monotonic() - started

    assert result.outcome == "dead"
    assert "timeout" in result.detail
    # The extractor sleeps for 30s; the harness must give up long before.
    assert elapsed < 5.0, f"budget not enforced, took {elapsed:.1f}s"


def test_budget_does_not_stop_the_rest_of_the_run():
    report = process_all(build_corpus(), budget_s=0.3)
    assert by_name(report, "doc-06.slow").outcome == "dead"
    # Everything after the pathological document still processed.
    assert by_name(report, "doc-08.pdf").outcome == "accepted"


# ------------------------------------------------------------------- score


def test_scores_handle_empty_text_without_dividing_by_zero():
    s = score(Document("e.txt", b"abc"), "")
    assert s.text_yield == 0.0
    assert s.alpha_ratio == 0.0
    assert s.mean_word_len == 0.0


def test_yield_is_a_ratio_not_a_length():
    # A genuinely short document is fine; a failed extraction of a big one is
    # not. Only the ratio tells them apart.
    short = score(Document("s.txt", b"hi"), "hi")
    failed = score(Document("b.txt", b"x" * 1000), "hi")
    assert short.text_yield > failed.text_yield
    assert short.text_yield >= 0.9
    assert failed.text_yield < 0.1


def test_alphabetic_ratio_counts_only_letters():
    s = score(Document("m.txt", b"=?#" * 10), "=?#" * 10)
    assert s.alpha_ratio == 0.0


# -------------------------------------------------------------------- gate


def test_scan_is_flagged_by_low_yield_not_by_alphabetic_ratio():
    report = process_all(build_corpus(), budget_s=0.3)
    scan = by_name(report, "doc-03.scan")
    assert scan.outcome == "flagged"
    assert scan.detail == "low text yield"
    # Its letters ARE letters -- only the yield reveals the failure.
    assert scan.scores.alpha_ratio > 0.5


def test_mojibake_is_flagged_by_alphabetic_ratio_not_by_yield():
    report = process_all(build_corpus(), budget_s=0.3)
    junk = by_name(report, "doc-04.bin")
    assert junk.outcome == "flagged"
    assert junk.detail == "low alphabetic ratio"
    # Every byte decoded, so yield is perfect. The mirror image of the scan.
    assert junk.scores.text_yield >= 0.99


def test_flagged_documents_still_carry_their_text():
    # Flagged is not rejected: partial text usually beats none.
    report = process_all(build_corpus(), budget_s=0.3)
    for name in ("doc-03.scan", "doc-04.bin"):
        assert by_name(report, name).text != ""


def test_summary_counts_all_three_outcomes():
    report = process_all(build_corpus(), budget_s=0.3)
    assert report.summary() == "summary: accepted=4 flagged=2 dead=2"
    assert len(report.results) == 8


def test_healthy_documents_are_accepted():
    report = process_all(build_corpus(), budget_s=0.3)
    for name in ("doc-01.txt", "doc-02.pdf", "doc-05.html", "doc-08.pdf"):
        assert by_name(report, name).outcome == "accepted"


def test_format_line_is_stable_for_dead_and_scored_results():
    report = process_all(build_corpus(), budget_s=0.3)
    assert "timeout" in format_line(by_name(report, "doc-06.slow"))
    assert "yield=" in format_line(by_name(report, "doc-01.txt"))

Troubleshooting

Troubleshooting — Day 324

The demo takes 30 seconds instead of about 2.4

Your budget is not being enforced. extract_pathological sleeps for 30 seconds, and it will not stop on its own.

The common cause is trying to enforce the deadline with a thread. A thread cannot be killed, and ThreadPoolExecutor used as a context manager waits for its workers on exit — so future.result(timeout=...) returns but the with block then blocks for the full 30 seconds anyway. This lab's first version had exactly that bug. Use extract_with_budget, which runs the extractor in a child process and terminates it.

Everything is accepted, including doc-03.scan

Your yield signal is a length rather than a ratio. A genuinely short document and a failed extraction of a long one both produce little text; only len(text) / len(doc.data) distinguishes them.

Check with:

python3 -c "
import sys; sys.path.insert(0,'examples')
from process import score, Document
print(score(Document('s',b'hi'),'hi').text_yield)          # near 1.0
print(score(Document('b',b'x'*1000),'hi').text_yield)      # near 0.0"

doc-04.bin is accepted

Your alphabetic ratio counts every non-space character rather than only letters. =?# are not letters. Use str.isalpha().

doc-07.xyz raises instead of being dead-lettered

An unrecognised format is a normal outcome. Check fmt not in EXTRACTORS and return a dead Result with f"no extractor for format '{fmt}'" rather than raising or looking up a missing key.

test_detection_reads_content_not_the_extension fails

You are branching on doc.name. Detection must look only at doc.data. doc-03.scan and doc-06.slow are named for formats that do not exist precisely to catch this.

ZeroDivisionError in score

Empty extracted text is a common case, not an edge case — a scan can yield nothing at all. Guard every denominator, and return 0.0 rather than raising.

The suite takes 17 seconds

Tests that do not care about the exact timeout should pass a smaller budget: process_all(build_corpus(), budget_s=0.3). Only the test that asserts on timeout behaviour needs a realistic one.

RuntimeError about the start method, or the demo hangs on Windows

multiprocessing needs the entry point guarded. examples/process_demo.py already has if __name__ == "__main__":; keep it if you write your own harness.

NotImplementedError everywhere

Expected. The starter stubs detect, score and process_one, and every test depends on at least one of them — see expected-output/starter-run.txt.

Security notes

Security notes — Day 324

What this lab does and does not touch

  • No network. The corpus is eight synthetic in-memory documents. No test opens a socket, and requires_network is false.
  • No credentials. No API key, token or account anywhere.
  • No writes outside this directory. Only __pycache__/, .pytest_cache/ and, if you create it, .venv/ — all inside this lab and all removable with the documented cleanup.
  • No sudo, no global installs.
  • Child processes are cleaned up. Extraction runs in a multiprocessing child marked daemon=True and is explicitly terminated on timeout, then joined. No worker outlives the run.

Why extraction runs in a child process, and why that matters beyond this lab

The budget in this lab exists to stop a synthetic sleep. In production the same mechanism is a genuine security control:

  • Document parsers are a real attack surface. They are typically complex C libraries consuming fully untrusted input, and they have a long history of memory-safety vulnerabilities. Decompression bombs, deeply nested structures and malformed cross-reference tables are all used in practice.
  • A process boundary is what makes a limit enforceable. A thread cannot be killed, so a cooperative timeout only works on code that cooperates — which excludes exactly the inputs you are defending against. An earlier version of this lab used a thread pool and the 30-second sleep blocked the whole run.
  • In production, add a memory ceiling too (resource.setrlimit in the child), and prefer a sandbox where one is available. Extension exercise 3 covers the memory half.
  • Never let a parser fetch remote resources referenced by the document. That turns document processing into a server-side request forgery primitive.

Privacy

Extraction is the point where document contents become plain, greppable text — so it is the right stage for redaction, and the stage where sending content to a third-party OCR or document-intelligence API becomes a data-residency decision rather than a technical one. This lab sends nothing anywhere.

Reversing everything this lab did

rm -rf __pycache__ .pytest_cache .venv

Nothing else was created, and nothing outside this directory was modified.