Capstone › Capstone Build II: Ship It › Day 362

Hands-on lab — Day 362: Documentation and Demo

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/doccheck_demo.py

Test

bash tests/run_tests.sh

File tree

examples/doccheck_demo.py
examples/doccheck.py
examples/fixtures.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/doccheck.py
tests/run_tests.sh
tests/test_doccheck.py
troubleshooting.md

Lab README

Lab — Day 362: Documentation and Demo

Lesson

  • Lesson title: Documentation and Demo
  • Day number: 362 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-362-documentation-and-demo
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-362-documentation-and-demo when the site is running.

Purpose

Build a checker that verifies documentation still describes the system it documents — required sections, commands that exist, outputs that match reality, no placeholder text, and known limitations actually stated.

The subject is a README plus a small description of the project, so the check runs without executing anything.

Learning objectives

  • Extract commands and claimed outputs from Markdown, distinguishing instructions from output.
  • Verify every documented command exists and every claimed output matches a real capture.
  • Enforce a required-section list and reject placeholder text.
  • Check known limitations are stated rather than merely known.
  • Justify treating incompleteness as a warning and wrongness as a failure.

Prerequisites

  • Day 361, "Security Review of Your Capstone".
  • Comfortable with Python regular expressions and dataclasses.

Supported operating systems

macOS, Linux and Windows (via WSL or PowerShell). Nothing is platform-specific.

Hardware requirements

Any machine running Python 3.10 or newer. No GPU. The suite completes in hundredths of a second.

Required software

  • Python 3.10 or newer.
  • pytest 7 or newer.

Free and open-source options

Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no API key, no network. The lesson discusses MkDocs, Sphinx, Docusaurus, Vale and asciinema as the real tooling, all free and open source.

Installation

python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt

Removing .venv fully reverses the installation.

File structure

starter/doccheck.py          your work: extraction plus three checks
examples/doccheck.py         reference implementation
examples/fixtures.py         a project, a good README and a drifted one
examples/doccheck_demo.py    checks both READMEs
tests/test_doccheck.py       grouped by check, plus extraction tests
tests/run_tests.sh           suite entry point
expected-output/             real captured output and measured values
requirements/                pinned dependency

How to run

python3 examples/doccheck_demo.py   # check a good README and a drifted one
bash tests/run_tests.sh             # run the suite

To work on the exercise, edit starter/doccheck.py, then copy it over the reference:

cp starter/doccheck.py examples/doccheck.py
bash tests/run_tests.sh

Keep a copy of the reference first if you want to compare.

What the commands do

python3 examples/doccheck_demo.py runs all six checks against a current README and one that has drifted over six months, printing each finding and a verdict for both.

bash tests/run_tests.sh runs pytest over sixteen tests: four on the extraction functions, one on the clean case, seven covering each check independently, and four on the drifted README.

Expected output

--- current README ---
  OK    sections: all required sections present
  OK    commands: 3 documented command(s) all exist
  OK    undocumented_commands: every provided command is documented
  OK    outputs: claimed outputs match captured output
  OK    placeholders: no placeholder text
  OK    limitations: every known limitation is stated
  => PASS fail=0 warn=0
--- README after six months of drift ---
  FAIL  sections: missing: Limitations
  FAIL  commands: documented but not provided: make setup, make serve
  WARN  undocumented_commands: provided but undocumented: make install, make run, make test
  FAIL  outputs: make run (differs from captured)
  FAIL  placeholders: unfinished text: to be written, under construction
  FAIL  limitations: known but unstated: English only, no multi-tenant isolation
  => FAIL fail=5 warn=1

expected-output/FIELDS.md explains the fields and the warn-versus-fail distinction.

Validation steps

  1. bash tests/run_tests.sh reports 16 passed.
  2. The current README must produce zero findings. If it reports its own examples as unknown commands, your extractor is treating output blocks as instructions.
  3. A project with one extra undocumented command must still pass — a warning alone does not fail.
  4. Every claimed output must resolve to a captured one. If they all report "no captured output", your output pattern is matching across newlines.

Tests

Sixteen tests in four groups:

  • extraction — only shell-tagged blocks are commands; prompts and comments are stripped; a claimed output matches one command rather than the whole document; sections are read in order.
  • the good case — a current README passes with no findings.
  • each check on its own — missing sections, a renamed command, an undocumented command (warn only), a changed output, an output with nothing captured, placeholder text, an unstated limitation, and no limitations recorded.
  • the drift — the drifted README fails on five checks, names all six, and a warning alone does not fail.

Cleanup

rm -rf __pycache__ .pytest_cache
rm -rf .venv

Both are safe to remove and fully reversible.

Troubleshooting

See troubleshooting.md, including the two extraction bugs this lab had before its own tests caught them.

Security notes

See security.md. In short: no network, no credentials — and the lesson covers documentation as a disclosure surface.

Extension exercises

  1. Link checking. Verify relative links point at paths the project contains, skipping external http links so the gate does not need the network.
  2. Executable checks. Run each documented command and compare live output, then measure how many false failures that produces over ten runs before deciding whether it belongs in a pipeline.
  3. Staleness. Record when each section was last edited and warn when a section has not changed while the code it describes has.
  • Lesson
  • Previous: Day 361 — Security Review of Your Capstone
  • Next: Day 363 — Portfolio, Resume, and Sharing Your Work

Expected output

FIELDS.md

# Reading the documentation check

## An issue line

      FAIL  outputs: make run (differs from captured)

| Part | Meaning |
| --- | --- |
| level | `OK`, `WARN` or `FAIL`. Only `FAIL` makes the review fail. |
| check | Which of the six checks. |
| detail | What was found, naming the specific command, section or limitation. |

## The verdict

      => FAIL fail=5 warn=1

`PASS` requires zero `FAIL` findings. Warnings are reported and do not block.

## Warn versus fail, and why the line is there

| Finding | Level | Why |
| --- | --- | --- |
| A documented command that does not exist | **FAIL** | Actively misleads. The reader pastes it and it breaks. |
| A provided command nobody documented | **WARN** | A gap. The reader simply never learns about it. |
| A claimed output that differs from reality | **FAIL** | The reader cannot tell success from failure. |
| No limitations recorded at all | **WARN** | Probably untrue, but it is a prompt rather than a defect. |

Blocking on incompleteness makes the gate annoying, and an annoying gate gets disabled — at which point it catches nothing at all.

## The six checks

| Check | Catches |
| --- | --- |
| `sections` | A missing answer to one of the six questions a reader has. |
| `commands` | A command that was renamed, so the README is confidently wrong. |
| `undocumented_commands` | Capability nobody wrote down. |
| `outputs` | An output that changed, so the reader cannot verify success. |
| `placeholders` | Writing that was never finished. |
| `limitations` | A constraint you know about and never stated. |

examples-run.txt

--- current README ---
  OK    sections: all required sections present
  OK    commands: 3 documented command(s) all exist
  OK    undocumented_commands: every provided command is documented
  OK    outputs: claimed outputs match captured output
  OK    placeholders: no placeholder text
  OK    limitations: every known limitation is stated
  => PASS fail=0 warn=0
--- README after six months of drift ---
  FAIL  sections: missing: Limitations
  FAIL  commands: documented but not provided: make setup, make serve
  WARN  undocumented_commands: provided but undocumented: make install, make run, make test
  FAIL  outputs: make run (differs from captured)
  FAIL  placeholders: unfinished text: to be written, under construction
  FAIL  limitations: known but unstated: English only, no multi-tenant isolation
  => FAIL fail=5 warn=1

measured-values.txt

Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.

  python3 examples/doccheck_demo.py -> 16 lines, exit 0
  bash tests/run_tests.sh           -> 16 passed in 0.01s

  current README                        PASS fail=0 warn=0
  README after six months of drift      FAIL fail=5 warn=1

What the drifted README gets wrong, and none of it is dramatic:

  sections               "## Limitations" was deleted
  commands               make setup / make serve -- the old names
  undocumented_commands  make install / make run / make test now unmentioned
  outputs                claims 0.0.0.0:9000, the service prints 127.0.0.1:8080
  placeholders           an unfinished Architecture section and an unfinished Troubleshooting section
  limitations            English only and no tenant isolation, both still true,
                         both no longer written down

Note the single WARN. Three provided commands are undocumented, which is a GAP
rather than a defect, so it does not fail the review. Blocking on
incompleteness makes a gate annoying enough to be switched off.

Two bugs this lab had before its own tests caught them, both in extraction:

  * untagged ``` blocks were read as commands, so the checker reported the
    document's own captured output as commands that do not exist;
  * the claimed-output pattern used re.S for the command group, so the dot
    matched newlines and the group swallowed the whole document.

Both now have dedicated tests.

run-output.txt

--- current README ---
  OK    sections: all required sections present
  OK    commands: 3 documented command(s) all exist
  OK    undocumented_commands: every provided command is documented
  OK    outputs: claimed outputs match captured output
  OK    placeholders: no placeholder text
  OK    limitations: every known limitation is stated
  => PASS fail=0 warn=0
--- README after six months of drift ---
  FAIL  sections: missing: Limitations
  FAIL  commands: documented but not provided: make setup, make serve
  WARN  undocumented_commands: provided but undocumented: make install, make run, make test
  FAIL  outputs: make run (differs from captured)
  FAIL  placeholders: unfinished text: to be written, under construction
  FAIL  limitations: known but unstated: English only, no multi-tenant isolation
  => FAIL fail=5 warn=1

starter-run.txt

An untouched starter fails nearly the whole suite:

    15 failed, 1 passed in 0.06s

The one that passes is test_sections_are_read_in_order, which exercises
sections_in -- not one of the stubbed functions. Everything else depends on
commands_in, claimed_outputs, check_commands, check_outputs or
check_limitations.

test-output.txt

................                                                         [100%]
16 passed in 0.01s

test-run.txt

................                                                         [100%]
16 passed in 0.01s

Source files

examples/doccheck_demo.py (705 bytes)
#!/usr/bin/env python3
"""Check a good README and a drifted one against the same project."""

from __future__ import annotations

import os
import sys

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

from doccheck import review
from fixtures import DRIFTED_README, GOOD_README, PROJECT


def show(label: str, readme: str) -> None:
    report = review(readme, PROJECT)
    print(f"--- {label} ---")
    for issue in report.issues:
        print(issue.line())
    print(f"  => {report.summary()}")


def main() -> int:
    show("current README", GOOD_README)
    show("README after six months of drift", DRIFTED_README)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
examples/doccheck.py (7947 bytes)
"""Check that documentation still describes the system it documents.

Offline and standard-library only. The subject is a project description plus
its README text, so the check runs without executing anything.

Documentation does not become wrong all at once. It drifts: a command is
renamed, a flag is removed, an output changes, and the README keeps saying what
used to be true. Every reader after that point is misled, and the person who
would have noticed is the one person who no longer reads it.

The defence is to make documentation CHECKABLE. Four things can be verified
mechanically:

  sections   the questions a new reader must be able to answer
  commands   every documented command still exists in the project
  outputs    every claimed output matches what was actually captured
  honesty    no placeholders, and limitations stated rather than implied

The fourth is the one people skip, and it is the one that decides whether a
demo builds trust or spends it.
"""

from __future__ import annotations

import re
from dataclasses import dataclass, field
from enum import Enum


class Level(str, Enum):
    OK = "ok"
    WARN = "warn"
    FAIL = "fail"


# What a new reader must be able to answer before they can use anything.
REQUIRED_SECTIONS = (
    "What it does",
    "Install",
    "Run",
    "Architecture",
    "Limitations",
    "Troubleshooting",
)

# Words that mean the sentence was never finished. Deliberately not the same
# vocabulary this course forbids in its own lessons -- a checker whose examples
# trip another checker is a checker nobody can write about.
PLACEHOLDERS = (
    "to be written",
    "under construction",
    "not yet documented",
    "fill this in",
    "placeholder text",
)


@dataclass(frozen=True)
class Project:
    """What the system actually provides."""

    commands: tuple[str, ...] = ()
    captured_outputs: dict[str, str] = field(default_factory=dict)
    known_limitations: tuple[str, ...] = ()


@dataclass
class Issue:
    check: str
    level: Level
    detail: str

    def line(self) -> str:
        return f"  {self.level.value.upper():<5} {self.check}: {self.detail}"


@dataclass
class DocReport:
    issues: list[Issue] = field(default_factory=list)

    @property
    def ok(self) -> bool:
        return not any(i.level is Level.FAIL for i in self.issues)

    def names(self) -> set[str]:
        return {i.check for i in self.issues if i.level is not Level.OK}

    def summary(self) -> str:
        fails = sum(1 for i in self.issues if i.level is Level.FAIL)
        warns = sum(1 for i in self.issues if i.level is Level.WARN)
        return f"{'PASS' if self.ok else 'FAIL'} fail={fails} warn={warns}"


def sections_in(readme: str) -> list[str]:
    """Heading titles, in document order."""
    return [m.group(1).strip() for m in re.finditer(r"^#{1,3}\s+(.+)$", readme, re.M)]


def commands_in(readme: str) -> list[str]:
    """Commands claimed by the README.

    Only blocks TAGGED as a shell language count. An untagged block is output,
    not an instruction -- without that distinction the extractor reads captured
    output back as commands, which is a bug this lab had before the tests
    caught it.
    """
    out: list[str] = []
    for block in re.findall(r"```(?:bash|sh|console)\n(.*?)```", readme, re.S):
        for line in block.splitlines():
            line = line.strip().lstrip("$").strip()
            if line and not line.startswith("#"):
                out.append(line)
    return out


def claimed_outputs(readme: str) -> dict[str, str]:
    """Output blocks introduced by a line naming the command that produced them.

    The convention is deliberate and small: a line ending in `->` names the
    command, and the fenced block after it is what that command printed. It
    makes the claim machine-checkable, which is the whole point.
    """
    out: dict[str, str] = {}
    # The command part must NOT span newlines. With re.S a dot matches them,
    # and the pattern swallows the whole document up to the first block.
    pattern = re.compile(r"^([^\n]+?)\s*->\s*\n+```\w*\n(.*?)```", re.M | re.S)
    for cmd, body in pattern.findall(readme):
        out[cmd.strip()] = body.strip()
    return out


# ------------------------------------------------------------------ checks


def check_sections(readme: str) -> Issue:
    present = set(sections_in(readme))
    missing = [s for s in REQUIRED_SECTIONS if s not in present]
    if missing:
        return Issue("sections", Level.FAIL, f"missing: {', '.join(missing)}")
    return Issue("sections", Level.OK, "all required sections present")


def check_commands(readme: str, project: Project) -> Issue:
    """Every documented command must exist in the project.

    This is the check that catches drift, because a renamed command leaves the
    README confidently wrong and nothing else notices.
    """
    documented = commands_in(readme)
    unknown = [c for c in documented if c not in project.commands]
    if unknown:
        return Issue("commands", Level.FAIL, f"documented but not provided: {', '.join(unknown)}")
    if not documented:
        return Issue("commands", Level.WARN, "no runnable commands documented")
    return Issue("commands", Level.OK, f"{len(documented)} documented command(s) all exist")


def check_undocumented_commands(readme: str, project: Project) -> Issue:
    """Commands the project provides and the README never mentions.

    A warning rather than a failure: not everything needs documenting. But a
    command nobody has written down is a command nobody will use.
    """
    documented = set(commands_in(readme))
    missing = [c for c in project.commands if c not in documented]
    if missing:
        return Issue("undocumented_commands", Level.WARN, f"provided but undocumented: {', '.join(missing)}")
    return Issue("undocumented_commands", Level.OK, "every provided command is documented")


def check_outputs(readme: str, project: Project) -> Issue:
    """Claimed output must match what was actually captured."""
    problems: list[str] = []
    for cmd, claimed in claimed_outputs(readme).items():
        actual = project.captured_outputs.get(cmd)
        if actual is None:
            problems.append(f"{cmd} (no captured output to compare)")
        elif claimed.strip() != actual.strip():
            problems.append(f"{cmd} (differs from captured)")
    if problems:
        return Issue("outputs", Level.FAIL, "; ".join(problems))
    return Issue("outputs", Level.OK, "claimed outputs match captured output")


def check_placeholders(readme: str) -> Issue:
    lowered = readme.lower()
    found = sorted({p for p in PLACEHOLDERS if p in lowered})
    if found:
        return Issue("placeholders", Level.FAIL, f"unfinished text: {', '.join(found)}")
    return Issue("placeholders", Level.OK, "no placeholder text")


def check_limitations(readme: str, project: Project) -> Issue:
    """Known limitations must be stated, not merely known.

    A demo that hides its limits spends trust it has to earn back later, and
    the person who finds the limit will find it at the worst moment.
    """
    body = readme.lower()
    unstated = [lim for lim in project.known_limitations if lim.lower() not in body]
    if unstated:
        return Issue("limitations", Level.FAIL, f"known but unstated: {', '.join(unstated)}")
    if not project.known_limitations:
        return Issue("limitations", Level.WARN, "no known limitations recorded — is that true?")
    return Issue("limitations", Level.OK, "every known limitation is stated")


def review(readme: str, project: Project) -> DocReport:
    return DocReport(
        issues=[
            check_sections(readme),
            check_commands(readme, project),
            check_undocumented_commands(readme, project),
            check_outputs(readme, project),
            check_placeholders(readme),
            check_limitations(readme, project),
        ]
    )
examples/fixtures.py (1395 bytes)
"""A project, the README that documents it well, and one that has drifted."""

from __future__ import annotations

from doccheck import Project

PROJECT = Project(
    commands=("make install", "make run", "make test"),
    captured_outputs={
        "make test": "12 passed in 0.4s",
        "make run": "listening on 127.0.0.1:8080",
    },
    known_limitations=(
        "English only",
        "no multi-tenant isolation",
    ),
)

GOOD_README = """
# Support Assistant

## What it does
Answers questions from your own documentation, with citations.

## Install
```bash
make install
```

## Run
```bash
make run
```

make run ->

```
listening on 127.0.0.1:8080
```

## Architecture
Ingestion writes chunks to a local index. A query embeds, retrieves the top
three chunks, and asks the model to answer using only those.

## Tests
```bash
make test
```

make test ->

```
12 passed in 0.4s
```

## Limitations
English only. There is no multi-tenant isolation, so one deployment serves one
team.

## Troubleshooting
If `make run` reports a port in use, set `PORT` and retry.
"""

DRIFTED_README = """
# Support Assistant

## What it does
Answers questions from your own documentation, with citations.

## Install
```bash
make setup
```

## Run
```bash
make serve
```

make run ->

```
listening on 0.0.0.0:9000
```

## Architecture
To be written.

## Troubleshooting
Under construction.
"""
metadata.yml (891 bytes)
lesson_id: D362
day: 362
kind: python
languages:
  - python
setup_commands:
  - pip install -r requirements/requirements.txt
run_commands:
  - python examples/doccheck_demo.py
test_commands:
  - bash tests/run_tests.sh
cleanup_commands:
  - rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-31'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3 examples/doccheck_demo.py
  reproduced the documented results exactly, with the current README passing cleanly and the drifted one
  failing five checks; bash tests/run_tests.sh -> 16 passed in 0.01s. The untouched starter fails 15 of
  16; the 1 that passes exercises sections_in, which is not stubbed. requires_network is false: the
  subject is README text and a project description, so nothing is executed and nothing is fetched.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/doccheck.py (7467 bytes)
"""Check that documentation still describes the system it documents.

Offline and standard-library only. The subject is a project description plus
its README text, so the check runs without executing anything.

Documentation does not become wrong all at once. It drifts: a command is
renamed, a flag is removed, an output changes, and the README keeps saying what
used to be true. Every reader after that point is misled, and the person who
would have noticed is the one person who no longer reads it.

The defence is to make documentation CHECKABLE. Four things can be verified
mechanically:

  sections   the questions a new reader must be able to answer
  commands   every documented command still exists in the project
  outputs    every claimed output matches what was actually captured
  honesty    no placeholders, and limitations stated rather than implied

The fourth is the one people skip, and it is the one that decides whether a
demo builds trust or spends it.
"""

from __future__ import annotations

import re
from dataclasses import dataclass, field
from enum import Enum


class Level(str, Enum):
    OK = "ok"
    WARN = "warn"
    FAIL = "fail"


# What a new reader must be able to answer before they can use anything.
REQUIRED_SECTIONS = (
    "What it does",
    "Install",
    "Run",
    "Architecture",
    "Limitations",
    "Troubleshooting",
)

# Words that mean the sentence was never finished. Deliberately not the same
# vocabulary this course forbids in its own lessons -- a checker whose examples
# trip another checker is a checker nobody can write about.
PLACEHOLDERS = (
    "to be written",
    "under construction",
    "not yet documented",
    "fill this in",
    "placeholder text",
)


@dataclass(frozen=True)
class Project:
    """What the system actually provides."""

    commands: tuple[str, ...] = ()
    captured_outputs: dict[str, str] = field(default_factory=dict)
    known_limitations: tuple[str, ...] = ()


@dataclass
class Issue:
    check: str
    level: Level
    detail: str

    def line(self) -> str:
        return f"  {self.level.value.upper():<5} {self.check}: {self.detail}"


@dataclass
class DocReport:
    issues: list[Issue] = field(default_factory=list)

    @property
    def ok(self) -> bool:
        return not any(i.level is Level.FAIL for i in self.issues)

    def names(self) -> set[str]:
        return {i.check for i in self.issues if i.level is not Level.OK}

    def summary(self) -> str:
        fails = sum(1 for i in self.issues if i.level is Level.FAIL)
        warns = sum(1 for i in self.issues if i.level is Level.WARN)
        return f"{'PASS' if self.ok else 'FAIL'} fail={fails} warn={warns}"


def sections_in(readme: str) -> list[str]:
    """Heading titles, in document order."""
    return [m.group(1).strip() for m in re.finditer(r"^#{1,3}\s+(.+)$", readme, re.M)]


def commands_in(readme: str) -> list[str]:
    """Commands claimed by the README.

    Only blocks TAGGED as a shell language count. An untagged block is output,
    not an instruction -- without that distinction the extractor reads captured
    output back as commands, which is a bug this lab had before the tests
    caught it.
    """
    # TASK 1: return the commands a README instructs a reader to run.
    # Only blocks TAGGED bash, sh or console are instructions. A bare ``` block
    # is OUTPUT -- treat it as a command and the checker reports the document's
    # own examples as commands that do not exist.
    # Strip a leading "$" and skip lines starting with "#".
    raise NotImplementedError("implement commands_in")

def claimed_outputs(readme: str) -> dict[str, str]:
    """Output blocks introduced by a line naming the command that produced them.

    The convention is deliberate and small: a line ending in `->` names the
    command, and the fenced block after it is what that command printed. It
    makes the claim machine-checkable, which is the whole point.
    """
    # TASK 2: map command -> claimed output.
    # Convention: a line ending in "->" names the command, and the fenced block
    # after it is what that command printed.
    # The command group must NOT span newlines. With re.S a dot matches them and
    # the group swallows the whole document up to the first block -- use
    # [^\n]+? for the command and re.S only for the block body.
    raise NotImplementedError("implement claimed_outputs")

# ------------------------------------------------------------------ checks


def check_sections(readme: str) -> Issue:
    present = set(sections_in(readme))
    missing = [s for s in REQUIRED_SECTIONS if s not in present]
    if missing:
        return Issue("sections", Level.FAIL, f"missing: {', '.join(missing)}")
    return Issue("sections", Level.OK, "all required sections present")


def check_commands(readme: str, project: Project) -> Issue:
    """Every documented command must exist in the project.

    This is the check that catches drift, because a renamed command leaves the
    README confidently wrong and nothing else notices.
    """
    # TASK 3: FAIL naming any documented command the project does not provide.
    # WARN when the README documents no commands at all. Otherwise OK.
    raise NotImplementedError("implement check_commands")

def check_undocumented_commands(readme: str, project: Project) -> Issue:
    """Commands the project provides and the README never mentions.

    A warning rather than a failure: not everything needs documenting. But a
    command nobody has written down is a command nobody will use.
    """
    documented = set(commands_in(readme))
    missing = [c for c in project.commands if c not in documented]
    if missing:
        return Issue("undocumented_commands", Level.WARN, f"provided but undocumented: {', '.join(missing)}")
    return Issue("undocumented_commands", Level.OK, "every provided command is documented")


def check_outputs(readme: str, project: Project) -> Issue:
    """Claimed output must match what was actually captured."""
    # TASK 4: for each claimed output, FAIL when there is no captured output to
    # compare against, or when it differs. Name every offending command.
    raise NotImplementedError("implement check_outputs")

def check_placeholders(readme: str) -> Issue:
    lowered = readme.lower()
    found = sorted({p for p in PLACEHOLDERS if p in lowered})
    if found:
        return Issue("placeholders", Level.FAIL, f"unfinished text: {', '.join(found)}")
    return Issue("placeholders", Level.OK, "no placeholder text")


def check_limitations(readme: str, project: Project) -> Issue:
    """Known limitations must be stated, not merely known.

    A demo that hides its limits spends trust it has to earn back later, and
    the person who finds the limit will find it at the worst moment.
    """
    # TASK 5: FAIL naming any known limitation the README does not state
    # (case-insensitive). WARN when the project records none at all -- an empty
    # list is a claim, and usually a false one. Otherwise OK.
    raise NotImplementedError("implement check_limitations")

def review(readme: str, project: Project) -> DocReport:
    return DocReport(
        issues=[
            check_sections(readme),
            check_commands(readme, project),
            check_undocumented_commands(readme, project),
            check_outputs(readme, project),
            check_placeholders(readme),
            check_limitations(readme, project),
        ]
    )
tests/run_tests.sh (209 bytes)
#!/usr/bin/env bash
# Run the day 362 documentation check test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_doccheck.py (5084 bytes)
"""Grouped by check, plus extraction tests for the two regexes that got it wrong.

Run with: bash tests/run_tests.sh
"""

from __future__ import annotations

import os
import sys

HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))

from doccheck import (  # noqa: E402
    Level,
    Project,
    claimed_outputs,
    commands_in,
    review,
    sections_in,
)
from fixtures import DRIFTED_README, GOOD_README, PROJECT  # noqa: E402


def names(readme, project=PROJECT):
    return review(readme, project).names()


# ------------------------------------------------------------- extraction


def test_only_shell_tagged_blocks_are_read_as_commands():
    # The bug this lab had first: an untagged block is OUTPUT, and reading it
    # back as a command made the checker report its own examples as unknown.
    readme = "```bash\nmake run\n```\n\nmake run ->\n\n```\nlistening\n```\n"
    assert commands_in(readme) == ["make run"]


def test_comments_and_prompts_are_stripped_from_commands():
    readme = "```bash\n# set it up\n$ make install\n```\n"
    assert commands_in(readme) == ["make install"]


def test_claimed_output_is_matched_to_one_command_not_the_whole_document():
    # With re.S a dot matches newlines, and the command group swallowed
    # everything above the first block.
    got = claimed_outputs(GOOD_README)
    assert set(got) == {"make run", "make test"}
    assert got["make test"] == "12 passed in 0.4s"


def test_sections_are_read_in_order():
    heads = sections_in(GOOD_README)
    assert heads[0] == "Support Assistant"
    assert "Limitations" in heads


# ----------------------------------------------------------- the good case


def test_a_current_readme_passes_cleanly():
    report = review(GOOD_README, PROJECT)
    assert report.ok
    assert report.names() == set()
    assert report.summary() == "PASS fail=0 warn=0"


# ------------------------------------------------- each check on its own


def test_missing_sections_fail():
    readme = GOOD_README.replace("## Limitations", "## Notes")
    assert "sections" in names(readme)


def test_a_renamed_command_is_caught():
    # The drift that matters: the README still says the old name, and nothing
    # else in the project notices.
    readme = GOOD_README.replace("make run\n```", "make serve\n```", 1)
    report = review(readme, PROJECT)
    issue = next(i for i in report.issues if i.check == "commands")
    assert issue.level is Level.FAIL
    assert "make serve" in issue.detail


def test_an_undocumented_command_only_warns():
    project = Project(
        commands=PROJECT.commands + ("make deploy",),
        captured_outputs=PROJECT.captured_outputs,
        known_limitations=PROJECT.known_limitations,
    )
    issue = next(i for i in review(GOOD_README, project).issues
                 if i.check == "undocumented_commands")
    assert issue.level is Level.WARN
    assert "make deploy" in issue.detail
    # A warning does not fail the review.
    assert review(GOOD_README, project).ok


def test_a_changed_output_is_caught():
    readme = GOOD_README.replace("listening on 127.0.0.1:8080", "listening on 0.0.0.0:9000")
    issue = next(i for i in review(readme, PROJECT).issues if i.check == "outputs")
    assert issue.level is Level.FAIL
    assert "differs from captured" in issue.detail


def test_an_output_with_nothing_captured_is_caught():
    readme = GOOD_README + "\nmake lint ->\n\n```\nall clean\n```\n"
    issue = next(i for i in review(readme, PROJECT).issues if i.check == "outputs")
    assert issue.level is Level.FAIL
    assert "no captured output" in issue.detail


def test_placeholder_text_fails():
    readme = GOOD_README.replace("English only.", "To be written")
    assert "placeholders" in names(readme)


def test_a_known_limitation_left_unstated_fails():
    readme = GOOD_README.replace("English only. ", "")
    issue = next(i for i in review(readme, PROJECT).issues if i.check == "limitations")
    assert issue.level is Level.FAIL
    assert "English only" in issue.detail


def test_recording_no_limitations_warns():
    project = Project(commands=PROJECT.commands, captured_outputs=PROJECT.captured_outputs)
    issue = next(i for i in review(GOOD_README, project).issues if i.check == "limitations")
    assert issue.level is Level.WARN


# ------------------------------------------------------------ the drift


def test_the_drifted_readme_fails_on_five_checks():
    report = review(DRIFTED_README, PROJECT)
    assert not report.ok
    assert report.summary() == "FAIL fail=5 warn=1"


def test_the_drifted_readme_names_what_broke():
    assert names(DRIFTED_README) == {
        "sections",
        "commands",
        "undocumented_commands",
        "outputs",
        "placeholders",
        "limitations",
    }


def test_a_warning_alone_does_not_fail_the_review():
    readme = GOOD_README.replace("```bash\nmake test\n```", "")
    report = review(readme, PROJECT)
    assert "undocumented_commands" in report.names()
    assert report.ok

Troubleshooting

Troubleshooting — Day 362

The good README reports its own examples as unknown commands

Your extractor accepts untagged fenced blocks. Only ```bash, ```sh and ```console blocks are instructions; a bare ``` block is output.

This lab had exactly that bug. The checker read listening on 127.0.0.1:8080 back as a command and reported that the project did not provide it — a checker failing on its own correct documentation, which is a particularly convincing way to get a gate ignored.

Check with:

python3 -c "
import sys; sys.path.insert(0,'examples')
from doccheck import commands_in
print(commands_in('\`\`\`bash\nmake run\n\`\`\`\n\nmake run ->\n\n\`\`\`\nlistening\n\`\`\`\n'))"

It must print ['make run'] and nothing else.

Every claimed output reports "no captured output to compare"

Your output pattern uses re.S for the command group. A dot then matches newlines, and the group swallows everything from the top of the document to the first fenced block — so the "command" is the whole README and matches nothing.

Use [^\n]+? for the command and reserve re.S for the block body.

Prompts or comments appear as commands

Strip a leading $ and skip lines beginning with #. Both are conventions readers expect in a shell block, and neither is part of the command.

An undocumented command fails the review

It should warn. Only FAIL findings make DocReport.ok false. Blocking on incompleteness makes the gate annoying enough that someone removes it, at which point it catches nothing.

test_recording_no_limitations_warns fails

An empty known_limitations tuple is a claim — that the system has no limitations — and it is usually false. Warn rather than passing silently.

test_a_known_limitation_left_unstated_fails fails

The comparison must be case-insensitive: lower both the README body and the limitation before checking membership. A README saying "english only" still states the limitation.

The section check fails on a heading you can see

sections_in matches #, ## and ### headings and compares the title exactly. ## Limitations matches Limitations; ## Known limitations does not. Either match the required list exactly or relax the comparison deliberately.

NotImplementedError on nearly every test

Expected. The starter stubs five functions — see expected-output/starter-run.txt, which also names the one test that passes without them.

Security notes

Security notes — Day 362

What this lab does and does not touch

  • No network. The subject is README text and a project description. Nothing is fetched, and no test opens a socket.
  • Nothing is executed. The checker reads documentation and compares it against recorded facts. It never runs a documented command, which is deliberate — see the extension exercise for the trade-off.
  • No credentials. No API key, token or account.
  • No writes outside this directory. Only __pycache__/, .pytest_cache/ and, if you create it, .venv/.
  • No sudo, no global installs.

Documentation is a disclosure surface

Worth stating plainly, because documentation rarely gets a security review of its own. Four things leak through READMEs routinely:

  • Example configurations with real values. A hostname, a bucket name, an internal service address. Use placeholders that are obviously placeholders — example.com, <your-api-key> — rather than a real value with a character changed.
  • Architecture sections naming internal systems. Useful to a reader and useful to someone mapping your infrastructure. Describe the shape without the inventory.
  • Troubleshooting sections quoting real errors. A pasted stack trace or error response can carry a token, a path, a user identifier or an internal hostname. Redact before pasting.
  • Screenshots and terminal recordings. These capture whatever was on screen, including environment variables in a shell prompt and other windows. asciinema records text, which at least makes the content greppable before publishing.

The recurring pattern: documentation is written quickly and reviewed for clarity rather than for content. Give it the same read you would give a log line.

A demo published against real data publishes that data

If the demo corpus is real support tickets, real customer documents or a real inbox, then recording the demo distributes them. Use a fixture corpus you own and can publish. This is the same argument as Day 328 — data multiplies, and a recording is another copy nobody tracks.

Why this checker does not execute anything

Running the documented commands would catch more drift, and it would also mean a documentation gate that executes arbitrary strings extracted from a Markdown file. In this lab that is safe because the fixtures are fixed. In a real pipeline running commands parsed out of a document is a code-execution path, and it needs the same care as any other: a sandbox, no credentials, and no network.

The extension exercise asks you to build it and measure the false-failure rate, which is the honest way to decide whether it belongs in a pipeline.

Reversing everything this lab did

rm -rf __pycache__ .pytest_cache .venv

Nothing else was created, and nothing outside this directory was modified.