Deployment, MLOps, and Security › Containers and Cloud › Day 330

Hands-on lab — Day 330: Docker Fundamentals

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/cache_demo.py

Test

bash tests/run_tests.sh

File tree

examples/cache_demo.py
examples/layer_cache.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/layer_cache.py
tests/run_tests.sh
tests/test_layer_cache.py
troubleshooting.md

Lab README

Day 330 Lab: Docker Fundamentals

Lesson

  • Lesson title: Docker Fundamentals
  • Day number: 330 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-330-docker-fundamentals
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-330-docker-fundamentals when the site is running.

Purpose

Model Docker's build cache so you can see what an edit actually costs — without installing Docker, and without waiting three minutes to discover you were wrong.

The cache follows one rule, and almost everyone gets it wrong at least once:

a layer is reused only if every layer before it was reused, and its own inputs are unchanged

You will build a parser, a cache model and a linter, then measure the difference the rule makes on two Dockerfiles that install the same packages from the same base image.

Learning objectives

  • Parse a Dockerfile into layers, joining continuations and keeping real line numbers.
  • Decide which sources a COPY reads, and which project files those cover.
  • Apply the cache rule, including the part where a layer misses because something above it did.
  • Explain why instruction order changes warm-build time but not cold-build time.
  • Detect the ordering, pinning, cleanup and privilege mistakes that cost time or ship a risk.

Prerequisites

  • Day 329, "Section Project: A Production Assistant".
  • Comfortable with Python dataclasses and basic regular expressions.

Supported operating systems

macOS, Linux and Windows (via WSL or PowerShell). Nothing is platform-specific.

Hardware requirements

Any machine running Python 3.10 or newer. No GPU, no container runtime. The suite completes in hundredths of a second.

Required software

  • Python 3.10 or newer.
  • pytest 7 or newer.

Docker is not required. This lab is about the rule the cache follows, which you can reason about on paper — and reasoning about it is what stops you burning ten minutes per rebuild for a year. If you do have Docker installed, the extension exercises suggest checking the model against a real build.

Free and open-source options

Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no registry login, no network. Docker Engine itself is Apache-2.0; Podman and Buildah are drop-in, daemonless alternatives that read the same Dockerfiles.

Installation

python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt

Removing .venv fully reverses the installation.

File structure

starter/layer_cache.py    your work: twelve stubbed tasks
examples/layer_cache.py   reference implementation
examples/cache_demo.py    two Dockerfiles, one edited file
tests/test_layer_cache.py grouped by what the cache model decides
tests/run_tests.sh        suite entry point
expected-output/          real captured output and measured values
requirements/             pinned dependency

How to run

python3 examples/cache_demo.py   # compare two Dockerfiles after one edit
bash tests/run_tests.sh          # run the suite

To work on the exercise, edit starter/layer_cache.py, then copy it over the reference:

cp starter/layer_cache.py examples/layer_cache.py
bash tests/run_tests.sh

Keep a copy of the reference first if you want to compare.

What the commands do

python3 examples/cache_demo.py parses two Dockerfiles that build the same application, plans a cold build and a warm build after one source file changes, prints which layer missed first and why, and lints both.

bash tests/run_tests.sh runs pytest over twenty-eight tests grouped by what the model decides: parsing, cost, copied paths, the cache rule, linting and the digest.

Expected output

--- naive: COPY . . then install ---
  cold build   0/6 layers cached  rebuild 76.0s
  after edit   2/6 layers cached  rebuild 76.0s
  first miss   line 3: COPY . .
               reads changed file(s): src/app.py
--- ordered: manifest, install, then source ---
  cold build   0/8 layers cached  rebuild 77.0s
  after edit   4/8 layers cached  rebuild 1.0s
--- the cost of one edit ---
  naive 76.0s vs ordered 1.0s
  ordering the same instructions is 76x faster to rebuild

Same base image, same packages, same application. Only the order differs — and the cold builds are within a second of each other, so the ordering costs nothing to adopt.

Validation steps

  1. bash tests/run_tests.sh reports 28 passed.
  2. The two cold builds must be within a few seconds of each other. If reordering appears to speed up a first build, plan_build is skipping layers in cold mode.
  3. Editing README.md must cost 0.0s in the ordered Dockerfile and the full 76.0s in the naive one — COPY . . reads the README too.
  4. Changing requirements.txt must re-run the install in both files. That is correct: the install genuinely depends on it.
  5. The linter must report nothing on the ordered Dockerfile and three findings on the naive one.

Tests

Twenty-eight tests in six groups:

  • parsing — comments and blanks produce no layer; line numbers are the real ones; a backslash continuation is one instruction; the verb is uppercased and the arguments are not.
  • cost — an install dwarfs a copy; metadata instructions are free; a RUN that installs nothing is cheap.
  • copied paths — the last token is the destination; flags are not sources; a RUN has no sources.
  • the cache rule — a cold build reuses nothing; an untouched project reuses everything; COPY . . puts the install downstream of every file; copying the manifest first protects it; everything after the first miss also misses; an unread file costs nothing; the cold builds match.
  • lint — each rule fires when it should and stays quiet when it should not; findings carry the line to look at.
  • digest — stable across comments and spacing, different when an instruction changes.

Cleanup

rm -rf __pycache__ .pytest_cache
rm -rf .venv

Both are safe to remove and fully reversible.

Troubleshooting

See troubleshooting.md.

Security notes

See security.md. In short: no Docker, no network, no credentials — and the reason a secret you COPY and later rm is still fully readable in the image.

Extension exercises

  1. Add .dockerignore. Give plan_build a set of ignored patterns and exclude matching files from what a broad COPY reads. Then measure how much of the naive Dockerfile's penalty it removes — the answer is instructive, because it is not all of it.
  2. Model multi-stage builds. COPY --from=builder reads from an earlier stage rather than the project. Track stages by their AS name and let a stage's cache state depend on the stage it copies from.
  3. Check the model against reality. If you have Docker or Podman, build both Dockerfiles, touch one source file, rebuild, and compare the real timings with the predicted ratio. Record where the model is wrong — it will be, and knowing how is the point.
  • Lesson
  • Previous: Day 329 — Section Project: A Production Assistant
  • Next: Day 331 — Dockerizing an AI Application

Expected output

FIELDS.md

# Reading the build plan

## The plan summary

    cold build   0/6 layers cached  rebuild 76.0s
    after edit   2/6 layers cached  rebuild 76.0s

| Part | Meaning |
| --- | --- |
| `n/m layers cached` | How many layers were reused out of the total. |
| `rebuild Ns` | Wall-clock cost of the layers that must run. Cached layers cost nothing. |

The two lines are the whole argument. A cold build is the same either way; the
warm build is where ordering pays.

## The first miss

    first miss   line 3: COPY . .
                 reads changed file(s): src/app.py

The first layer that could not be reused, and why. Everything below it rebuilds
regardless of its own inputs, so this one line explains the entire cost.

## The reasons a layer is not reused

| Reason | Meaning |
| --- | --- |
| `no cache (cold build)` | Nothing to reuse; every layer runs. |
| `reads changed file(s): …` | This layer copies something that changed. It is the first miss. |
| `a previous layer changed` | This layer's own inputs are fine. It rebuilds anyway. |

The third is the rule people forget. A layer cannot be reused just because
nothing it touches has changed.

## The cost model

| Instruction | Seconds |
| --- | --- |
| `RUN` that installs dependencies | 75 |
| any other `RUN` | 8 |
| `COPY` / `ADD` | 1 |
| `FROM`, `WORKDIR`, `ENV`, `EXPOSE`, `USER`, `CMD`, … | 0 |

The absolute numbers are nominal. What matters is the **ratio**: an install is
about seventy-five times a copy, which is why an ordering mistake that
invalidates the install costs so much more than it looks like it should.

## The lint rules

| Rule | What it costs you |
| --- | --- |
| `copy-before-install` | Every source edit re-runs the install. Time. |
| `unpinned-base` | A rebuild months later gets a different image. Reproducibility. |
| `apt-cache-kept` | The package index ships inside the layer. Size, and see `security.md`. |
| `pip-cache-kept` | The wheel cache ships inside the layer. Size. |
| `runs-as-root` | The process runs as uid 0 in the container. Blast radius. |

`copy-before-install` is reported at the line of the **COPY**, not the install,
because the copy is the line you move.

examples-run.txt

--- naive: COPY . . then install ---
  cold build   0/6 layers cached  rebuild 76.0s
  after edit   2/6 layers cached  rebuild 76.0s
  first miss   line 3: COPY . .
               reads changed file(s): src/app.py
--- ordered: manifest, install, then source ---
  cold build   0/8 layers cached  rebuild 77.0s
  after edit   4/8 layers cached  rebuild 1.0s
  first miss   line 5: COPY src/ ./src/
               reads changed file(s): src/app.py
--- the cost of one edit ---
  naive 76.0s vs ordered 1.0s
  ordering the same instructions is 76x faster to rebuild
--- lint: naive ---
  line  3 [copy-before-install] COPY . . before the install layer: every source edit re-runs the i
  line  4 [pip-cache-kept] pip install without --no-cache-dir bakes the wheel cache into the 
  line  0 [runs-as-root] no USER instruction: the container runs as root
--- lint: ordered ---
  no findings

measured-values.txt

Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.

  python3 examples/cache_demo.py -> 19 lines, exit 0
  bash tests/run_tests.sh        -> 28 passed in 0.02s
  the untouched starter          -> 28 failed in 0.08s

Two Dockerfiles for the same application. One line of source code changed.

  naive    COPY . . then install     cold 76.0s   after the edit  76.0s
  ordered  manifest, install, source cold 77.0s   after the edit   1.0s

That is the number this lab exists for: 76x, and the cold builds are within one
second of each other. Ordering the instructions costs nothing on a first build
and returns the entire install on every build after it.

The mechanism, from the plan output:

  naive    first miss line 3  COPY . .          reads changed file(s): src/app.py
  ordered  first miss line 5  COPY src/ ./src/  reads changed file(s): src/app.py

Both miss on the same file. The difference is WHERE. In the naive file the miss
happens before the install, so the install is downstream of it and re-runs. In
the ordered file the install sits above the miss and is reused.

Note what is NOT different: the same instructions, the same base image, the same
packages. Only the order.

One thing worth checking yourself. Editing README.md costs 0.0s in the ordered
Dockerfile and the full 76.0s in the naive one, because `COPY . .` reads the
README too. A file that is not part of the build still invalidates the build
when a broad copy pulls it in -- which is what .dockerignore is actually for.

The linter finds three problems in the naive file and none in the ordered one:

  line 3  copy-before-install   every source edit re-runs the install
  line 4  pip-cache-kept        no --no-cache-dir, so the wheel cache ships
  line 0  runs-as-root          no USER instruction

Line 0 means the finding is about the file as a whole rather than one line --
the absence of an instruction has no line number to point at.

run-output.txt

--- naive: COPY . . then install ---
  cold build   0/6 layers cached  rebuild 76.0s
  after edit   2/6 layers cached  rebuild 76.0s
  first miss   line 3: COPY . .
               reads changed file(s): src/app.py
--- ordered: manifest, install, then source ---
  cold build   0/8 layers cached  rebuild 77.0s
  after edit   4/8 layers cached  rebuild 1.0s
  first miss   line 5: COPY src/ ./src/
               reads changed file(s): src/app.py
--- the cost of one edit ---
  naive 76.0s vs ordered 1.0s
  ordering the same instructions is 76x faster to rebuild
--- lint: naive ---
  line  3 [copy-before-install] COPY . . before the install layer: every source edit re-runs the i
  line  4 [pip-cache-kept] pip install without --no-cache-dir bakes the wheel cache into the 
  line  0 [runs-as-root] no USER instruction: the container runs as root
--- lint: ordered ---
  no findings

starter-run.txt

An untouched starter fails the suite. Captured from a real run:

FAILED tests/test_layer_cache.py::test_changing_the_manifest_does_re_run_the_install
FAILED tests/test_layer_cache.py::test_a_change_to_a_file_nobody_copies_costs_nothing
FAILED tests/test_layer_cache.py::test_the_cold_builds_cost_about_the_same - ...
FAILED tests/test_layer_cache.py::test_copy_before_install_is_reported - NotI...
FAILED tests/test_layer_cache.py::test_a_correctly_ordered_dockerfile_is_clean
FAILED tests/test_layer_cache.py::test_an_unpinned_base_image_is_reported - N...
FAILED tests/test_layer_cache.py::test_a_pinned_base_image_is_not_reported - ...
FAILED tests/test_layer_cache.py::test_apt_without_cleanup_in_the_same_layer_is_reported
FAILED tests/test_layer_cache.py::test_apt_with_cleanup_in_the_same_layer_is_clean
FAILED tests/test_layer_cache.py::test_a_missing_user_instruction_is_reported
FAILED tests/test_layer_cache.py::test_findings_carry_the_line_to_look_at - N...
FAILED tests/test_layer_cache.py::test_the_digest_ignores_comments_and_spacing
FAILED tests/test_layer_cache.py::test_the_digest_changes_when_an_instruction_changes
28 failed in 0.08s

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

............................                                             [100%]
28 passed in 0.01s

test-run.txt

............................                                             [100%]
28 passed in 0.01s

Source files

examples/cache_demo.py (2023 bytes)
#!/usr/bin/env python3
"""The same application, two Dockerfiles, one edited source file."""

from __future__ import annotations

import os
import sys

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

from layer_cache import lint, parse, plan_build

NAIVE = """\
FROM python:3.12-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
EXPOSE 8000
CMD ["python", "-m", "app"]
"""

ORDERED = """\
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ ./src/
USER 10001
EXPOSE 8000
CMD ["python", "-m", "app"]
"""

# One line of application code changed. Nothing else.
EDIT = {"src/app.py"}


def show(name: str, dockerfile: str) -> None:
    ins = parse(dockerfile)
    cold = plan_build(ins, cold=True)
    warm = plan_build(ins, EDIT)
    print(f"--- {name} ---")
    print(f"  cold build   {cold.summary()}")
    print(f"  after edit   {warm.summary()}")
    miss = warm.first_miss
    if miss:
        print(f"  first miss   line {miss.instruction.line_no}: {miss.instruction.text[:44]}")
        print(f"               {miss.reason}")


def main() -> int:
    show("naive: COPY . . then install", NAIVE)
    show("ordered: manifest, install, then source", ORDERED)

    naive_warm = plan_build(parse(NAIVE), EDIT).seconds
    ordered_warm = plan_build(parse(ORDERED), EDIT).seconds
    print("--- the cost of one edit ---")
    print(f"  naive {naive_warm:.1f}s vs ordered {ordered_warm:.1f}s")
    if ordered_warm:
        print(f"  ordering the same instructions is {naive_warm / ordered_warm:.0f}x faster to rebuild")

    print("--- lint: naive ---")
    for f in lint(parse(NAIVE)):
        print(f"  line {f.line_no:>2} [{f.rule}] {f.message[:66]}")
    print("--- lint: ordered ---")
    found = lint(parse(ORDERED))
    for f in found:
        print(f"  line {f.line_no:>2} [{f.rule}] {f.message[:66]}")
    if not found:
        print("  no findings")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
examples/layer_cache.py (8574 bytes)
"""Model Docker's build cache so you can see what an edit actually costs.

Offline and standard-library only. Nothing is built and Docker does not need
to be installed -- the point is the RULE the cache follows, which you can
reason about without waiting three minutes to find out you were wrong.

The rule is short and almost everyone gets it wrong at least once:

  a layer is reused only if every layer before it was reused, AND
  its own inputs are unchanged

The consequence is the whole lesson. `COPY . .` early in a Dockerfile makes
every later layer depend on every file in your project, so editing one line of
Python re-runs `pip install`. Copying the dependency manifest first, installing,
and only then copying source costs nothing extra on a cold build and saves the
install on every warm one.

Nothing here judges whether an image is good. It computes which layers survive
an edit, which is the part that decides how long you wait.
"""

from __future__ import annotations

import hashlib
import re
from dataclasses import dataclass, field

# Rough wall-clock seconds each kind of instruction costs when it has to run.
# The absolute numbers do not matter; the RATIO between installing dependencies
# and copying a file is what makes cache ordering worth caring about.
COST_SECONDS = {
    "FROM": 0.0,
    "WORKDIR": 0.0,
    "ENV": 0.0,
    "ARG": 0.0,
    "EXPOSE": 0.0,
    "USER": 0.0,
    "LABEL": 0.0,
    "ENTRYPOINT": 0.0,
    "CMD": 0.0,
    "HEALTHCHECK": 0.0,
    "COPY": 1.0,
    "ADD": 1.0,
    "RUN": 8.0,
}

# A RUN that installs dependencies is the expensive one, and it is the layer
# people accidentally invalidate.
INSTALL_RE = re.compile(
    r"\b(pip\s+install|apt-get\s+install|apk\s+add|npm\s+(ci|install)|poetry\s+install|uv\s+pip\s+install)\b",
    re.I,
)
INSTALL_SECONDS = 75.0


@dataclass(frozen=True)
class Instruction:
    line_no: int
    verb: str
    args: str

    @property
    def text(self) -> str:
        return f"{self.verb} {self.args}".strip()

    @property
    def cost_seconds(self) -> float:
        """What this layer costs when it actually has to run."""
        if self.verb == "RUN" and INSTALL_RE.search(self.args):
            return INSTALL_SECONDS
        return COST_SECONDS.get(self.verb, 1.0)

    @property
    def copied_paths(self) -> list[str]:
        """Sources a COPY or ADD reads. The last token is the destination."""
        if self.verb not in ("COPY", "ADD"):
            return []
        parts = [p for p in self.args.split() if not p.startswith("--")]
        return parts[:-1] if len(parts) >= 2 else []


def parse(dockerfile: str) -> list[Instruction]:
    """Parse a Dockerfile, joining backslash continuations.

    Comments and blank lines produce no layer, so they are dropped rather than
    numbered -- but the line numbers kept are the real ones, because that is
    what you need when the linter tells you where to look.
    """
    out: list[Instruction] = []
    pending = ""
    start = 0
    for i, raw in enumerate(dockerfile.split("\n"), start=1):
        line = raw.strip()
        if not pending and (not line or line.startswith("#")):
            continue
        if not pending:
            start = i
        if line.endswith("\\"):
            pending += line[:-1].strip() + " "
            continue
        full = (pending + line).strip()
        pending = ""
        if not full:
            continue
        verb, _, args = full.partition(" ")
        out.append(Instruction(line_no=start, verb=verb.upper(), args=args.strip()))
    return out


def _matches(pattern: str, path: str) -> bool:
    """Whether a COPY source covers a project file.

    Deliberately simple: `.` and `./` cover everything, a directory prefix
    covers what is under it, and anything else must match exactly. Docker's
    real matching is richer; this is the part that decides cache behaviour.
    """
    if pattern in (".", "./"):
        return True
    p = pattern.rstrip("/")
    return path == p or path.startswith(p + "/")


@dataclass
class Layer:
    instruction: Instruction
    reused: bool
    reason: str
    cost_seconds: float


@dataclass
class BuildPlan:
    layers: list[Layer] = field(default_factory=list)

    @property
    def rebuilt(self) -> list[Layer]:
        return [x for x in self.layers if not x.reused]

    @property
    def seconds(self) -> float:
        return round(sum(x.cost_seconds for x in self.rebuilt), 1)

    @property
    def first_miss(self) -> Layer | None:
        return self.rebuilt[0] if self.rebuilt else None

    def summary(self) -> str:
        hit = len(self.layers) - len(self.rebuilt)
        return f"{hit}/{len(self.layers)} layers cached  rebuild {self.seconds:.1f}s"


def plan_build(
    instructions: list[Instruction],
    changed_files: set[str] | None = None,
    *,
    cold: bool = False,
) -> BuildPlan:
    """Which layers survive, given the files that changed since the last build.

    Once a layer misses, everything after it misses too -- that is the whole
    rule, and it is why the ORDER of instructions matters more than their
    content.
    """
    changed = changed_files or set()
    plan = BuildPlan()
    invalidated = False
    for ins in instructions:
        if cold:
            reason = "no cache (cold build)"
            reused = False
        elif invalidated:
            reason = "a previous layer changed"
            reused = False
        else:
            touched = [f for f in sorted(changed) if any(_matches(p, f) for p in ins.copied_paths)]
            if touched:
                reason = f"reads changed file(s): {', '.join(touched[:3])}"
                reused = False
                invalidated = True
            else:
                reason = "unchanged"
                reused = True
        plan.layers.append(
            Layer(
                instruction=ins,
                reused=reused,
                reason=reason,
                cost_seconds=0.0 if reused else ins.cost_seconds,
            )
        )
    return plan


def digest(dockerfile: str) -> str:
    """A stable identifier for a Dockerfile's instruction sequence."""
    text = "\n".join(i.text for i in parse(dockerfile))
    return hashlib.sha256(text.encode()).hexdigest()[:12]


@dataclass(frozen=True)
class Finding:
    line_no: int
    rule: str
    message: str


def lint(instructions: list[Instruction]) -> list[Finding]:
    """The mistakes that cost real time or ship a real risk."""
    out: list[Finding] = []
    install_idx = next(
        (i for i, x in enumerate(instructions) if x.verb == "RUN" and INSTALL_RE.search(x.args)),
        None,
    )
    broad_copy_idx = next(
        (i for i, x in enumerate(instructions) if x.copied_paths and any(p in (".", "./") for p in x.copied_paths)),
        None,
    )
    if install_idx is not None and broad_copy_idx is not None and broad_copy_idx < install_idx:
        out.append(
            Finding(
                instructions[broad_copy_idx].line_no,
                "copy-before-install",
                "COPY . . before the install layer: every source edit re-runs the install. "
                "Copy the dependency manifest first, install, then copy the source.",
            )
        )

    for ins in instructions:
        if ins.verb == "FROM":
            ref = ins.args.split(" AS ")[0].strip()
            if ":" not in ref.rsplit("/", 1)[-1] or ref.endswith(":latest"):
                out.append(
                    Finding(ins.line_no, "unpinned-base",
                            f"base image '{ref}' is unpinned; a rebuild months later gets a different image")
                )
        if ins.verb == "RUN" and re.search(r"\bapt-get\s+install\b", ins.args, re.I):
            if "rm -rf /var/lib/apt/lists" not in ins.args:
                out.append(
                    Finding(ins.line_no, "apt-cache-kept",
                            "apt-get install without removing /var/lib/apt/lists in the SAME layer "
                            "leaves the package index in the image")
                )
        if ins.verb == "RUN" and re.search(r"\bpip\s+install\b", ins.args, re.I):
            if "--no-cache-dir" not in ins.args:
                out.append(
                    Finding(ins.line_no, "pip-cache-kept",
                            "pip install without --no-cache-dir bakes the wheel cache into the layer")
                )

    if not any(x.verb == "USER" for x in instructions):
        out.append(Finding(0, "runs-as-root", "no USER instruction: the container runs as root"))
    return out
metadata.yml (885 bytes)
lesson_id: D330
day: 330
kind: python
languages:
  - python
setup_commands:
  - pip install -r requirements/requirements.txt
run_commands:
  - python3 examples/cache_demo.py
test_commands:
  - bash tests/run_tests.sh
cleanup_commands:
  - rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-31'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3 examples/cache_demo.py
  reproduced the documented figures exactly (naive 76.0s vs ordered 1.0s after one source edit, 76x,
  with cold builds within 1s of each other); bash tests/run_tests.sh -> 28 passed in 0.02s. The
  untouched starter fails all 28. requires_network is false and Docker is NOT required: the lab models
  the cache rule rather than invoking a builder, which is the point — the rule is what you reason about.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/layer_cache.py (4976 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.

Model Docker's build cache so you can see what an edit actually costs.

Offline and standard-library only. Nothing is built and Docker does not need
to be installed -- the point is the RULE the cache follows, which you can
reason about without waiting three minutes to find out you were wrong.

The rule is short and almost everyone gets it wrong at least once:

  a layer is reused only if every layer before it was reused, AND
  its own inputs are unchanged

The consequence is the whole lesson. `COPY . .` early in a Dockerfile makes
every later layer depend on every file in your project, so editing one line of
Python re-runs `pip install`. Copying the dependency manifest first, installing,
and only then copying source costs nothing extra on a cold build and saves the
install on every warm one.

Nothing here judges whether an image is good. It computes which layers survive
an edit, which is the part that decides how long you wait.
"""
from __future__ import annotations
import hashlib
import re
from dataclasses import dataclass, field
COST_SECONDS = {'FROM': 0.0, 'WORKDIR': 0.0, 'ENV': 0.0, 'ARG': 0.0, 'EXPOSE': 0.0, 'USER': 0.0, 'LABEL': 0.0, 'ENTRYPOINT': 0.0, 'CMD': 0.0, 'HEALTHCHECK': 0.0, 'COPY': 1.0, 'ADD': 1.0, 'RUN': 8.0}
INSTALL_RE = re.compile('\\b(pip\\s+install|apt-get\\s+install|apk\\s+add|npm\\s+(ci|install)|poetry\\s+install|uv\\s+pip\\s+install)\\b', re.I)
INSTALL_SECONDS = 75.0

@dataclass(frozen=True)
class Instruction:
    line_no: int
    verb: str
    args: str

    @property
    def text(self) -> str:
        raise NotImplementedError('TASK 1: implement text.')

    @property
    def cost_seconds(self) -> float:
        """What this layer costs when it actually has to run."""
        raise NotImplementedError('TASK 2: implement cost_seconds. What this layer costs when it actually has to run.')

    @property
    def copied_paths(self) -> list[str]:
        """Sources a COPY or ADD reads. The last token is the destination."""
        raise NotImplementedError('TASK 3: implement copied_paths. Sources a COPY or ADD reads. The last token is the destination.')

def parse(dockerfile: str) -> list[Instruction]:
    """Parse a Dockerfile, joining backslash continuations.

    Comments and blank lines produce no layer, so they are dropped rather than
    numbered -- but the line numbers kept are the real ones, because that is
    what you need when the linter tells you where to look.
    """
    raise NotImplementedError('TASK 4: implement parse. Parse a Dockerfile, joining backslash continuations.')

def _matches(pattern: str, path: str) -> bool:
    """Whether a COPY source covers a project file.

    Deliberately simple: `.` and `./` cover everything, a directory prefix
    covers what is under it, and anything else must match exactly. Docker's
    real matching is richer; this is the part that decides cache behaviour.
    """
    raise NotImplementedError('TASK 5: implement _matches. Whether a COPY source covers a project file.')

@dataclass
class Layer:
    instruction: Instruction
    reused: bool
    reason: str
    cost_seconds: float

@dataclass
class BuildPlan:
    layers: list[Layer] = field(default_factory=list)

    @property
    def rebuilt(self) -> list[Layer]:
        raise NotImplementedError('TASK 6: implement rebuilt.')

    @property
    def seconds(self) -> float:
        raise NotImplementedError('TASK 7: implement seconds.')

    @property
    def first_miss(self) -> Layer | None:
        raise NotImplementedError('TASK 8: implement first_miss.')

    def summary(self) -> str:
        raise NotImplementedError('TASK 9: implement summary.')

def plan_build(instructions: list[Instruction], changed_files: set[str] | None=None, *, cold: bool=False) -> BuildPlan:
    """Which layers survive, given the files that changed since the last build.

    Once a layer misses, everything after it misses too -- that is the whole
    rule, and it is why the ORDER of instructions matters more than their
    content.
    """
    raise NotImplementedError('TASK 10: implement plan_build. Which layers survive, given the files that changed since the last build.')

def digest(dockerfile: str) -> str:
    """A stable identifier for a Dockerfile's instruction sequence."""
    raise NotImplementedError("TASK 11: implement digest. A stable identifier for a Dockerfile's instruction sequence.")

@dataclass(frozen=True)
class Finding:
    line_no: int
    rule: str
    message: str

def lint(instructions: list[Instruction]) -> list[Finding]:
    """The mistakes that cost real time or ship a real risk."""
    raise NotImplementedError('TASK 12: implement lint. The mistakes that cost real time or ship a real risk.')
tests/run_tests.sh (201 bytes)
#!/usr/bin/env bash
# Run the day 330 layer-cache test suite.
# Offline, standard library plus pytest. Docker is not required.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_layer_cache.py (6379 bytes)
"""Grouped by what the cache model decides.

Run with: bash tests/run_tests.sh
"""

from __future__ import annotations

import os
import sys

HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.join(os.path.dirname(HERE), "examples"))

from layer_cache import (  # noqa: E402
    INSTALL_SECONDS,
    Instruction,
    digest,
    lint,
    parse,
    plan_build,
)

NAIVE = """\
FROM python:3.12-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD ["python", "-m", "app"]
"""

ORDERED = """\
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ ./src/
USER 10001
CMD ["python", "-m", "app"]
"""


# ------------------------------------------------------------------- parsing


def test_comments_and_blank_lines_produce_no_layer():
    ins = parse("# a comment\n\nFROM python:3.12-slim\n\n# another\nWORKDIR /app\n")
    assert [i.verb for i in ins] == ["FROM", "WORKDIR"]


def test_line_numbers_are_the_real_ones():
    # The linter reports these, so an off-by-one sends people to the wrong line.
    ins = parse("# comment\n\nFROM python:3.12-slim\nWORKDIR /app\n")
    assert [i.line_no for i in ins] == [3, 4]


def test_a_backslash_continuation_is_one_instruction():
    ins = parse("RUN apt-get update \\\n && apt-get install -y curl \\\n && rm -rf /var/lib/apt/lists/*\n")
    assert len(ins) == 1
    assert "apt-get install" in ins[0].args and "rm -rf" in ins[0].args


def test_the_verb_is_uppercased_but_the_arguments_are_not():
    ins = parse("from python:3.12-slim AS Builder\n")
    assert ins[0].verb == "FROM"
    assert "Builder" in ins[0].args


# ---------------------------------------------------------------------- cost


def test_an_install_layer_costs_far_more_than_a_copy():
    run = Instruction(1, "RUN", "pip install -r requirements.txt")
    copy = Instruction(2, "COPY", "src/ ./src/")
    assert run.cost_seconds == INSTALL_SECONDS
    assert run.cost_seconds > copy.cost_seconds * 50


def test_metadata_instructions_are_free():
    for verb in ("FROM", "WORKDIR", "ENV", "EXPOSE", "USER", "CMD"):
        assert Instruction(1, verb, "x").cost_seconds == 0.0


def test_a_run_that_installs_nothing_is_cheap():
    assert Instruction(1, "RUN", "mkdir -p /app/data").cost_seconds < INSTALL_SECONDS


# ------------------------------------------------------------- copied paths


def test_the_last_token_is_the_destination_not_a_source():
    assert Instruction(1, "COPY", "requirements.txt .").copied_paths == ["requirements.txt"]
    assert Instruction(1, "COPY", "a.txt b.txt /dest/").copied_paths == ["a.txt", "b.txt"]


def test_flags_are_not_sources():
    assert Instruction(1, "COPY", "--from=builder /wheels /wheels").copied_paths == ["/wheels"]


def test_an_instruction_that_copies_nothing_has_no_sources():
    assert Instruction(1, "RUN", "pip install x").copied_paths == []


# --------------------------------------------------------------- cache rule


def test_a_cold_build_reuses_nothing():
    plan = plan_build(parse(ORDERED), cold=True)
    assert plan.rebuilt == plan.layers


def test_an_untouched_project_reuses_everything():
    plan = plan_build(parse(ORDERED), set())
    assert plan.rebuilt == []
    assert plan.seconds == 0.0


def test_copy_dot_makes_every_later_layer_depend_on_every_file():
    plan = plan_build(parse(NAIVE), {"src/app.py"})
    assert plan.first_miss.instruction.verb == "COPY"
    assert plan.seconds >= INSTALL_SECONDS  # the install re-runs


def test_copying_the_manifest_first_protects_the_install():
    plan = plan_build(parse(ORDERED), {"src/app.py"})
    assert plan.seconds < INSTALL_SECONDS
    assert not any(x.instruction.verb == "RUN" and not x.reused for x in plan.layers)


def test_everything_after_the_first_miss_also_misses():
    # THE rule. A later layer cannot be reused just because its own inputs
    # are unchanged.
    plan = plan_build(parse(ORDERED), {"requirements.txt"})
    idx = plan.layers.index(plan.first_miss)
    assert all(not x.reused for x in plan.layers[idx:])


def test_changing_the_manifest_does_re_run_the_install():
    plan = plan_build(parse(ORDERED), {"requirements.txt"})
    assert plan.seconds >= INSTALL_SECONDS


def test_a_change_to_a_file_nobody_copies_costs_nothing():
    assert plan_build(parse(ORDERED), {"README.md"}).seconds == 0.0


def test_the_cold_builds_cost_about_the_same():
    # The ordering trick is free: it buys warm-build speed, not cold.
    naive = plan_build(parse(NAIVE), cold=True).seconds
    ordered = plan_build(parse(ORDERED), cold=True).seconds
    assert abs(naive - ordered) < 5.0


# --------------------------------------------------------------------- lint


def test_copy_before_install_is_reported():
    rules = {f.rule for f in lint(parse(NAIVE))}
    assert "copy-before-install" in rules


def test_a_correctly_ordered_dockerfile_is_clean():
    assert lint(parse(ORDERED)) == []


def test_an_unpinned_base_image_is_reported():
    for ref in ("FROM python\n", "FROM python:latest\n"):
        assert "unpinned-base" in {f.rule for f in lint(parse(ref + "USER 1\n"))}


def test_a_pinned_base_image_is_not_reported():
    assert "unpinned-base" not in {f.rule for f in lint(parse("FROM python:3.12-slim\nUSER 1\n"))}


def test_apt_without_cleanup_in_the_same_layer_is_reported():
    df = "FROM debian:12\nRUN apt-get update && apt-get install -y curl\nUSER 1\n"
    assert "apt-cache-kept" in {f.rule for f in lint(parse(df))}


def test_apt_with_cleanup_in_the_same_layer_is_clean():
    df = ("FROM debian:12\n"
          "RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*\n"
          "USER 1\n")
    assert "apt-cache-kept" not in {f.rule for f in lint(parse(df))}


def test_a_missing_user_instruction_is_reported():
    assert "runs-as-root" in {f.rule for f in lint(parse("FROM python:3.12-slim\n"))}


def test_findings_carry_the_line_to_look_at():
    f = next(x for x in lint(parse(NAIVE)) if x.rule == "copy-before-install")
    assert f.line_no == 3


# ------------------------------------------------------------------- digest


def test_the_digest_ignores_comments_and_spacing():
    assert digest(ORDERED) == digest("# a note\n\n" + ORDERED)


def test_the_digest_changes_when_an_instruction_changes():
    assert digest(NAIVE) != digest(ORDERED)

Troubleshooting

Troubleshooting — Day 330

NotImplementedError on every test

Expected. The starter stubs twelve functions — see expected-output/starter-run.txt.

Fix them in order. parse comes first; nothing can be checked until instructions exist.

test_line_numbers_are_the_real_ones fails

You are numbering the instructions you kept rather than the lines they came from. Comments and blank lines produce no layer but they do occupy lines, and the linter reports these numbers — an off-by-one sends the learner to the wrong line.

Track the line index while scanning, and record it when an instruction starts.

A backslash continuation becomes several instructions

RUN a \ / && b is one instruction. Accumulate while a line ends with \, and only emit when it does not. Remember to record the line number of the first line of the group.

test_the_last_token_is_the_destination_not_a_source fails

COPY a.txt b.txt /dest/ has two sources and one destination. Drop the last token, and drop --from=... style flags before you do — otherwise COPY --from=builder /wheels /wheels reports the flag as a source.

Everything after the first miss is still marked cached

The rule this lab exists for: a layer is reused only if every layer before it was reused as well. Once you record a miss, set a flag and mark the rest as misses regardless of their own inputs.

The naive and ordered Dockerfiles rebuild in the same time

Check _matches. COPY . . must match every project path — if . only matches a file literally named ., the naive Dockerfile looks as good as the ordered one and the whole demonstration collapses.

test_a_change_to_a_file_nobody_copies_costs_nothing fails

Editing README.md should cost nothing when no COPY reads it. If everything rebuilds, your matcher is treating an unrelated path as a match — most often by using in rather than a prefix test, so src matches not-src/x.py.

test_the_cold_builds_cost_about_the_same fails

This is the honest part of the argument: reordering does not make a first build faster. If your cold builds differ by more than a few seconds, plan_build is skipping layers in cold mode instead of running all of them.

The linter reports apt-cache-kept on a Dockerfile that does clean up

The cleanup must be in the same RUN. Check that your parser joined the backslash continuation into one instruction — if && rm -rf /var/lib/apt/lists/* ended up as a separate entry, the check cannot see it.

test_a_correctly_ordered_dockerfile_is_clean fails

Something is firing on the good Dockerfile. Print the findings and read them: the usual cause is copy-before-install triggering on COPY requirements.txt ., because the check is looking for any COPY rather than a broad one (. or ./).

Security notes

Security notes — Day 330

What this lab does and does not touch

  • No Docker required and none is invoked. The lab parses Dockerfile text and models the cache rule. Nothing is built, pulled, or run in a container.
  • No network. The Dockerfiles are strings in the demo.
  • No credentials, no API key, no registry login.
  • No writes outside this directory. Only __pycache__/, .pytest_cache/ and, if you create it, .venv/.
  • No sudo, no global installs. Worth stating plainly, because installing Docker Desktop is neither, and this lab deliberately does not ask you to.

Why the linter's three security rules are the ones it has

An unpinned base image. FROM python or FROM python:latest resolves to whatever that tag points at on the day you build. A rebuild six months later silently gets a different operating system, a different OpenSSL, and a different set of vulnerabilities. Pin to a specific tag, and pin to a digest when the build must be reproducible.

A container running as root. Without a USER instruction the process runs as uid 0 inside the container. If it is ever able to write to a mounted host path, or if a kernel escape exists, root inside is a much worse starting position than an unprivileged uid. Adding USER 10001 costs one line.

A package index left in the image. apt-get install without rm -rf /var/lib/apt/lists/* in the same layer ships the index. Removing it in a later layer does not shrink the image — the bytes are still in the earlier layer, and anyone who pulls the image can read them. That is the same mechanism by which a secret COPYd and later deleted remains fully recoverable.

The layer rule is a security property, not only a speed one

Every layer is kept. COPY .env . followed by RUN rm .env produces an image where the file is gone from the final filesystem and completely present in the layer beneath it, readable with docker save and a tar extractor.

This is the single most common way credentials escape into a registry. The fix is not to delete the file later; it is never to copy it in — use a .dockerignore, a build secret, or a runtime environment variable.

Reversing everything this lab did

rm -rf __pycache__ .pytest_cache .venv

Nothing else was created, and nothing outside this directory was modified.