Capstone › Capstone Build II: Ship It › Day 361

Hands-on lab — Day 361: Security Review of Your Capstone

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/review_demo.py

Test

bash tests/run_tests.sh

File tree

examples/review_demo.py
examples/review.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/review.py
tests/run_tests.sh
tests/test_review.py
troubleshooting.md

Lab README

Lab — Day 361: Security Review of Your Capstone

Lesson

Purpose

Build a defensive security review for an AI application: seven checks over a described configuration, each producing a finding with a severity and a remediation.

Everything here describes a system and reports weaknesses so they can be fixed. There is no exploit code, and nothing attacks anything. The subject is a Posture — a flat description — so the review can run against a design document before any code exists, which is when changes are cheapest.

Learning objectives

  • Express a system's security decisions as a reviewable posture.
  • Implement seven checks, each owning exactly one failure class.
  • Weight severity by blast radius rather than sophistication.
  • Attach a remediation to every finding.
  • Prove the review can fail, by running it against a deliberately weak configuration.

Prerequisites

  • Day 360, "Monitoring and Cost Controls".
  • Day 328, "Privacy in AI Systems", for the retention and redaction concepts.
  • Comfortable with Python dataclasses and enums.

Supported operating systems

macOS, Linux and Windows (via WSL or PowerShell). Nothing is platform-specific.

Hardware requirements

Any machine running Python 3.10 or newer. No GPU. The suite completes in hundredths of a second.

Required software

  • Python 3.10 or newer.
  • pytest 7 or newer.

Free and open-source options

Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no API key, no network. The lesson points at the OWASP Top 10 for LLM Applications as the reference list, and at gitleaks, Trivy, Semgrep and Bandit as the tools worth adding to a real project.

Installation

python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt

Removing .venv fully reverses the installation.

File structure

starter/review.py         your work: the seven check functions
examples/review.py        reference review, plus a sound and a weak posture
examples/review_demo.py   reviews both, then changes one setting at a time
tests/test_review.py      grouped by check, plus "the review can fail"
tests/run_tests.sh        suite entry point
expected-output/          real captured output and measured values
requirements/             pinned dependency

How to run

python3 examples/review_demo.py   # review both postures
bash tests/run_tests.sh           # run the suite

To work on the exercise, edit starter/review.py, then copy it over the reference:

cp starter/review.py examples/review.py
bash tests/run_tests.sh

Keep a copy of the reference first if you want to compare.

What the commands do

python3 examples/review_demo.py runs all seven checks against a sound posture and a deliberately weak one, printing each finding with its remediation, then makes three single-setting changes to a sound posture to show each check firing in isolation.

bash tests/run_tests.sh runs pytest over eighteen tests in three groups: the sound baseline, one test per check catching its own flaw, and a group proving the review can fail.

Expected output

--- sound posture ---
  OK      untrusted_content_boundary: retrieved content is marked untrusted
  OK      output_handling: output is escaped and never executed
  ...
  => PASS high=0 medium=0 low=0
--- weak posture ---
  HIGH    tool_permissions: state-changing scopes granted without confirmation:
          write:tickets, delete:records
          fix: grant the narrowest scope the task needs, and require explicit
          confirmation for anything that writes, deletes or spends
  ...
  => FAIL high=6 medium=1 low=0
--- one change at a time ---
  drop the spend caps                    FAIL high=1   ['spend_bounds']
  grant delete without confirmation      FAIL high=1   ['tool_permissions']
  keep traces forever                    FAIL high=0 medium=1

expected-output/FIELDS.md explains each field and the severity scale.

Validation steps

  1. bash tests/run_tests.sh reports 18 passed.
  2. The sound posture must report zero findings and the weak one must trip all seven — precision as well as recall.
  3. Each isolation run must produce exactly one finding. If changing one setting lights up several checks, they are reading overlapping fields and cannot be prioritised.
  4. Every failing finding must carry a remediation.

Tests

Eighteen tests in three groups:

  • the sound baseline — a clean posture passes with no findings, and all seven checks run.
  • each check catches its own flaw — one test per check, asserting the finding set is exactly that check.
  • severity and the ability to fail — severity tracks blast radius, one missing cap is less severe than two, a confirmed destructive scope is only low, the weak posture fails loudly and trips every check, and failing findings carry remediations.

Cleanup

rm -rf __pycache__ .pytest_cache
rm -rf .venv

Both are safe to remove and fully reversible.

Troubleshooting

See troubleshooting.md.

Security notes

See security.md. In short: this lab is entirely defensive, contains no exploit code, and its own subject is a fictional configuration.

Extension exercises

  1. A rate-limiting check. Add per-user and per-IP limits, and prove a posture with spend caps but no rate limits still fails — a cap bounds cost, not concurrency.
  2. A prioritised remediation plan. Order findings by consequence per unit of effort rather than by severity alone.
  3. Review your own capstone. Write its posture honestly, run the review, and record any finding you decide not to fix along with the reason.
  • Lesson
  • Previous: Day 360 — Monitoring and Cost Controls
  • Next: Day 362 — Documentation and Demo

Expected output

FIELDS.md

# Reading the review output

## A finding line

      HIGH    spend_bounds: no per-request and no per-user daily cap
              fix: cap before the call, not after; anything a user can trigger
              an attacker can trigger repeatedly

| Part | Meaning |
| --- | --- |
| severity | `OK`, `LOW`, `MEDIUM` or `HIGH`, weighted by **blast radius** rather than sophistication. |
| check | Which of the seven areas. Each owns one failure. |
| detail | What was actually found, with the specifics — which scopes, which caps, which controls. |
| fix | The remediation. A finding without one is a complaint rather than a review. |

## The verdict line

      => FAIL high=6 medium=1 low=0

`PASS` requires the worst finding to be `OK` or `LOW`. Any medium or high fails — a low finding is worth recording without blocking.

## The severity scale

| Severity | Meaning |
| --- | --- |
| `HIGH` | An attacker can act, data leaves, or the service can be taken down. |
| `MEDIUM` | Real exposure by a narrower path, or one of a pair of controls missing. |
| `LOW` | Worth noting. A destructive scope that exists but is gated behind confirmation. |
| `OK` | The control is present. |

## The isolation runs

    drop the spend caps                  FAIL high=1   ['spend_bounds']

One configuration change should produce one finding. This is how you know the checks are independent, and it is what makes the output prioritisable: each finding points at exactly one decision you could reverse.

## Why two postures are reviewed

The demo reviews a sound configuration and a deliberately weak one. The second is the important half — a review that has never found anything is not evidence of safety, because a clean report and a review that checks nothing look identical from outside.

examples-run.txt

--- sound posture ---
  OK      untrusted_content_boundary: retrieved content is marked untrusted
  OK      output_handling: output is escaped and never executed
  OK      tool_permissions: read-only scopes only
  OK      spend_bounds: per-request and per-user caps present
  OK      secret_handling: credentials come from the environment
  OK      data_retention: traces are bounded, deletable and redacted
  OK      dependency_pinning: image and dependencies are pinned
  => PASS high=0 medium=0 low=0
--- weak posture ---
  HIGH    untrusted_content_boundary: retrieved and user content is concatenated into the prompt without a boundary
          fix: keep instructions and untrusted content in separate, clearly delimited sections, and treat retrieved text as data the model may summarise but must not obey
  HIGH    output_handling: model output is rendered without escaping
          fix: escape on every append; a tag split across two streamed tokens is not a tag in either half, so a whole-document sanitiser will miss it
  HIGH    tool_permissions: state-changing scopes granted without confirmation: write:tickets, delete:records
          fix: grant the narrowest scope the task needs, and require explicit confirmation for anything that writes, deletes or spends
  HIGH    spend_bounds: no per-request and no per-user daily cap
          fix: cap before the call, not after; anything a user can trigger an attacker can trigger repeatedly, and unbounded spend is an availability risk
  HIGH    secret_handling: credentials are baked into the container image
          fix: an image is a distributable artefact and a later RUN rm does not remove a layer; supply secrets through the environment at run time
  HIGH    data_retention: traces are retained indefinitely; traces are not covered by deletion; content is stored before redaction
          fix: traces hold full prompts and completions -- give them a retention period, include them in the erasure path, and redact before writing
  MEDIUM  dependency_pinning: image referenced by mutable tag; dependencies unlocked
          fix: pin the image by digest and commit a lockfile, so what you deploy is provably what you tested
  => FAIL high=6 medium=1 low=0
--- one change at a time ---
  drop the spend caps                    FAIL high=1 medium=0 low=0       ['spend_bounds']
  grant delete without confirmation      FAIL high=1 medium=0 low=0       ['tool_permissions']
  keep traces forever                    FAIL high=0 medium=1 low=0       []

measured-values.txt

Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.

  python3 examples/review_demo.py -> 26 lines, exit 0
  bash tests/run_tests.sh         -> 18 passed in 0.02s

  sound posture   PASS high=0 medium=0 low=0
  weak posture    FAIL high=6 medium=1 low=0

The weak posture trips all seven checks and the sound one trips none. Both
halves matter: recall without precision would mean a review that fails
everything, which is as useless as one that fails nothing.

Isolation runs, one configuration change each:

  drop the spend caps                  FAIL high=1   ['spend_bounds']
  grant delete without confirmation    FAIL high=1   ['tool_permissions']
  keep traces forever                  FAIL high=0 medium=1

One change, one finding. A review where changing one setting lights up four
checks cannot prioritise, because it cannot say which decision caused what.

Severity accumulation, same store:

  traces kept forever                                        MEDIUM
  kept forever + no deletion path + no redaction             HIGH

Each control missing alone leaves a partial mitigation. All three missing
removes every mitigation from the most sensitive store in the system.

run-output.txt

--- sound posture ---
  OK      untrusted_content_boundary: retrieved content is marked untrusted
  OK      output_handling: output is escaped and never executed
  OK      tool_permissions: read-only scopes only
  OK      spend_bounds: per-request and per-user caps present
  OK      secret_handling: credentials come from the environment
  OK      data_retention: traces are bounded, deletable and redacted
  OK      dependency_pinning: image and dependencies are pinned
  => PASS high=0 medium=0 low=0
--- weak posture ---
  HIGH    untrusted_content_boundary: retrieved and user content is concatenated into the prompt without a boundary
          fix: keep instructions and untrusted content in separate, clearly delimited sections, and treat retrieved text as data the model may summarise but must not obey
  HIGH    output_handling: model output is rendered without escaping
          fix: escape on every append; a tag split across two streamed tokens is not a tag in either half, so a whole-document sanitiser will miss it
  HIGH    tool_permissions: state-changing scopes granted without confirmation: write:tickets, delete:records
          fix: grant the narrowest scope the task needs, and require explicit confirmation for anything that writes, deletes or spends
  HIGH    spend_bounds: no per-request and no per-user daily cap
          fix: cap before the call, not after; anything a user can trigger an attacker can trigger repeatedly, and unbounded spend is an availability risk
  HIGH    secret_handling: credentials are baked into the container image
          fix: an image is a distributable artefact and a later RUN rm does not remove a layer; supply secrets through the environment at run time
  HIGH    data_retention: traces are retained indefinitely; traces are not covered by deletion; content is stored before redaction
          fix: traces hold full prompts and completions -- give them a retention period, include them in the erasure path, and redact before writing
  MEDIUM  dependency_pinning: image referenced by mutable tag; dependencies unlocked
          fix: pin the image by digest and commit a lockfile, so what you deploy is provably what you tested
  => FAIL high=6 medium=1 low=0
--- one change at a time ---
  drop the spend caps                    FAIL high=1 medium=0 low=0       ['spend_bounds']
  grant delete without confirmation      FAIL high=1 medium=0 low=0       ['tool_permissions']
  keep traces forever                    FAIL high=0 medium=1 low=0       []

starter-run.txt

An untouched starter fails every test:

    18 failed in 0.07s

All seven checks raise NotImplementedError, and every test runs at least one of
them -- including the tests that assert the SOUND posture passes, since those
still have to run the checks to find out.

test-output.txt

..................                                                       [100%]
18 passed in 0.01s

test-run.txt

..................                                                       [100%]
18 passed in 0.01s

Source files

examples/review_demo.py (1130 bytes)
#!/usr/bin/env python3
"""Review a sound posture and a deliberately weak one."""

from __future__ import annotations

import os
import sys

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

from review import SOUND, WEAK, Posture, Severity, review


def show(label: str, posture: Posture) -> None:
    report = review(posture)
    print(f"--- {label} ---")
    for finding in report.findings:
        print(finding.line())
    print(f"  => {report.summary()}")


def main() -> int:
    show("sound posture", SOUND)
    show("weak posture", WEAK)

    print("--- one change at a time ---")
    for label, posture in (
        ("drop the spend caps", Posture(per_request_cap=None, per_user_daily_cap=None)),
        ("grant delete without confirmation", Posture(tool_scopes=("read:docs", "delete:records"))),
        ("keep traces forever", Posture(trace_retention_days=None)),
    ):
        report = review(posture)
        highs = [f.check for f in report.at_least(Severity.HIGH)]
        print(f"  {label:<38} {report.summary():<32} {highs}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
examples/review.py (10263 bytes)
"""A security posture review for an AI application.

Offline and standard-library only. Everything here is DEFENSIVE: the checks
describe a system's configuration and report weaknesses so they can be fixed.
There is no exploit code, and nothing here attacks anything.

The checks follow the shape of the OWASP Top 10 for LLM Applications, reduced
to the seven that a capstone can actually get wrong:

  untrusted_content_boundary   retrieved text treated as instructions
  output_handling              model output rendered or executed unescaped
  tool_permissions             an agent that can do more than the task needs
  spend_bounds                 no cap, so cost is an availability risk
  secret_handling              credentials in the image or the repository
  data_retention               traces kept forever, undeleted
  dependency_pinning           mutable references in the supply chain

Two ideas run through it.

  * Severity is about BLAST RADIUS, not about how clever the flaw is. A missing
    spend cap is dull and can take the service down.
  * A review that finds nothing is not evidence of safety unless you know it
    can find something -- which is why a deliberately weak configuration ships
    alongside a sound one.
"""

from __future__ import annotations

from dataclasses import dataclass, field
from enum import Enum


class Severity(str, Enum):
    OK = "ok"
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"


ORDER = {Severity.OK: 0, Severity.LOW: 1, Severity.MEDIUM: 2, Severity.HIGH: 3}


@dataclass(frozen=True)
class Posture:
    """How an application is configured. The subject of the review.

    Deliberately a flat description rather than live introspection: a review
    should be runnable against a design document before any code exists.
    """

    # Is retrieved/user content kept separate from instructions?
    marks_untrusted_content: bool = True
    # Is model output escaped before rendering, and never executed?
    escapes_output: bool = True
    executes_model_output: bool = False
    # What may the agent's tools do?
    tool_scopes: tuple[str, ...] = ("read:docs",)
    tools_require_confirmation: tuple[str, ...] = ()
    # Cost bounds
    per_request_cap: float | None = 0.05
    per_user_daily_cap: float | None = 1.0
    # Secrets
    secrets_in_environment: bool = True
    secrets_in_image: bool = False
    secrets_in_repository: bool = False
    # Data handling
    trace_retention_days: int | None = 30
    traces_in_erasure_path: bool = True
    redacts_before_storage: bool = True
    # Supply chain
    image_pinned_by_digest: bool = True
    dependencies_locked: bool = True


# Tool scopes that can change or destroy state, and so should not be granted
# without an explicit confirmation step.
DESTRUCTIVE_SCOPES = ("write:", "delete:", "admin:", "exec:", "payments:")


@dataclass
class Finding:
    check: str
    severity: Severity
    detail: str
    remediation: str = ""

    def line(self) -> str:
        base = f"  {self.severity.value.upper():<7} {self.check}: {self.detail}"
        return f"{base}\n          fix: {self.remediation}" if self.remediation else base


@dataclass
class Report:
    findings: list[Finding] = field(default_factory=list)

    @property
    def worst(self) -> Severity:
        return max((f.severity for f in self.findings), key=lambda s: ORDER[s], default=Severity.OK)

    def at_least(self, severity: Severity) -> list[Finding]:
        return [f for f in self.findings if ORDER[f.severity] >= ORDER[severity]]

    def names(self) -> set[str]:
        return {f.check for f in self.findings if f.severity is not Severity.OK}

    def summary(self) -> str:
        counts = {s: 0 for s in Severity}
        for f in self.findings:
            counts[f.severity] += 1
        verdict = "PASS" if self.worst in (Severity.OK, Severity.LOW) else "FAIL"
        return (
            f"{verdict} high={counts[Severity.HIGH]} medium={counts[Severity.MEDIUM]} "
            f"low={counts[Severity.LOW]}"
        )


# ------------------------------------------------------------------ checks


def check_untrusted_content_boundary(p: Posture) -> Finding:
    if p.marks_untrusted_content:
        return Finding("untrusted_content_boundary", Severity.OK, "retrieved content is marked untrusted")
    return Finding(
        "untrusted_content_boundary",
        Severity.HIGH,
        "retrieved and user content is concatenated into the prompt without a boundary",
        "keep instructions and untrusted content in separate, clearly delimited sections, "
        "and treat retrieved text as data the model may summarise but must not obey",
    )


def check_output_handling(p: Posture) -> Finding:
    if p.executes_model_output:
        return Finding(
            "output_handling",
            Severity.HIGH,
            "model output is executed",
            "never execute generated code or queries directly; run them in a sandbox with no "
            "credentials, or require a human to approve",
        )
    if not p.escapes_output:
        return Finding(
            "output_handling",
            Severity.HIGH,
            "model output is rendered without escaping",
            "escape on every append; a tag split across two streamed tokens is not a tag in "
            "either half, so a whole-document sanitiser will miss it",
        )
    return Finding("output_handling", Severity.OK, "output is escaped and never executed")


def check_tool_permissions(p: Posture) -> Finding:
    risky = [s for s in p.tool_scopes if s.startswith(DESTRUCTIVE_SCOPES)]
    unguarded = [s for s in risky if s not in p.tools_require_confirmation]
    if unguarded:
        return Finding(
            "tool_permissions",
            Severity.HIGH,
            f"state-changing scopes granted without confirmation: {', '.join(unguarded)}",
            "grant the narrowest scope the task needs, and require explicit confirmation for "
            "anything that writes, deletes or spends",
        )
    if risky:
        return Finding(
            "tool_permissions",
            Severity.LOW,
            f"state-changing scopes present but confirmed: {', '.join(risky)}",
        )
    return Finding("tool_permissions", Severity.OK, "read-only scopes only")


def check_spend_bounds(p: Posture) -> Finding:
    missing = [
        name
        for name, value in (("per-request", p.per_request_cap), ("per-user daily", p.per_user_daily_cap))
        if value is None
    ]
    if missing:
        return Finding(
            "spend_bounds",
            Severity.HIGH if len(missing) == 2 else Severity.MEDIUM,
            f"no {' and no '.join(missing)} cap",
            "cap before the call, not after; anything a user can trigger an attacker can "
            "trigger repeatedly, and unbounded spend is an availability risk",
        )
    return Finding("spend_bounds", Severity.OK, "per-request and per-user caps present")


def check_secret_handling(p: Posture) -> Finding:
    if p.secrets_in_repository:
        return Finding(
            "secret_handling",
            Severity.HIGH,
            "credentials are committed to the repository",
            "rotate them first -- history is not erased by deleting the file -- then move them "
            "to the environment",
        )
    if p.secrets_in_image:
        return Finding(
            "secret_handling",
            Severity.HIGH,
            "credentials are baked into the container image",
            "an image is a distributable artefact and a later RUN rm does not remove a layer; "
            "supply secrets through the environment at run time",
        )
    if not p.secrets_in_environment:
        return Finding("secret_handling", Severity.MEDIUM, "no declared source of credentials")
    return Finding("secret_handling", Severity.OK, "credentials come from the environment")


def check_data_retention(p: Posture) -> Finding:
    problems: list[str] = []
    if p.trace_retention_days is None:
        problems.append("traces are retained indefinitely")
    if not p.traces_in_erasure_path:
        problems.append("traces are not covered by deletion")
    if not p.redacts_before_storage:
        problems.append("content is stored before redaction")
    if problems:
        return Finding(
            "data_retention",
            Severity.HIGH if len(problems) > 1 else Severity.MEDIUM,
            "; ".join(problems),
            "traces hold full prompts and completions -- give them a retention period, include "
            "them in the erasure path, and redact before writing",
        )
    return Finding("data_retention", Severity.OK, "traces are bounded, deletable and redacted")


def check_dependency_pinning(p: Posture) -> Finding:
    problems: list[str] = []
    if not p.image_pinned_by_digest:
        problems.append("image referenced by mutable tag")
    if not p.dependencies_locked:
        problems.append("dependencies unlocked")
    if problems:
        return Finding(
            "dependency_pinning",
            Severity.MEDIUM,
            "; ".join(problems),
            "pin the image by digest and commit a lockfile, so what you deploy is provably what "
            "you tested",
        )
    return Finding("dependency_pinning", Severity.OK, "image and dependencies are pinned")


CHECKS = (
    check_untrusted_content_boundary,
    check_output_handling,
    check_tool_permissions,
    check_spend_bounds,
    check_secret_handling,
    check_data_retention,
    check_dependency_pinning,
)


def review(posture: Posture) -> Report:
    """Run every check against one posture."""
    return Report(findings=[check(posture) for check in CHECKS])


# ------------------------------------------------- two postures to review


SOUND = Posture()

WEAK = Posture(
    marks_untrusted_content=False,
    escapes_output=False,
    tool_scopes=("read:docs", "write:tickets", "delete:records"),
    tools_require_confirmation=(),
    per_request_cap=None,
    per_user_daily_cap=None,
    secrets_in_environment=False,
    secrets_in_image=True,
    trace_retention_days=None,
    traces_in_erasure_path=False,
    redacts_before_storage=False,
    image_pinned_by_digest=False,
    dependencies_locked=False,
)
metadata.yml (902 bytes)
lesson_id: D361
day: 361
kind: python
languages:
  - python
setup_commands:
  - pip install -r requirements/requirements.txt
run_commands:
  - python examples/review_demo.py
test_commands:
  - bash tests/run_tests.sh
cleanup_commands:
  - rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-31'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3 examples/review_demo.py
  reproduced the documented reviews exactly, with the weak posture tripping all seven checks and the
  sound one tripping none; bash tests/run_tests.sh -> 18 passed in 0.02s. The untouched starter fails all
  18. requires_network is false: the subject is a described configuration, not a running system, so
  nothing is probed and nothing leaves the machine. The lab is defensive throughout and contains no
  exploit code.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/review.py (7182 bytes)
"""A security posture review for an AI application.

Offline and standard-library only. Everything here is DEFENSIVE: the checks
describe a system's configuration and report weaknesses so they can be fixed.
There is no exploit code, and nothing here attacks anything.

The checks follow the shape of the OWASP Top 10 for LLM Applications, reduced
to the seven that a capstone can actually get wrong:

  untrusted_content_boundary   retrieved text treated as instructions
  output_handling              model output rendered or executed unescaped
  tool_permissions             an agent that can do more than the task needs
  spend_bounds                 no cap, so cost is an availability risk
  secret_handling              credentials in the image or the repository
  data_retention               traces kept forever, undeleted
  dependency_pinning           mutable references in the supply chain

Two ideas run through it.

  * Severity is about BLAST RADIUS, not about how clever the flaw is. A missing
    spend cap is dull and can take the service down.
  * A review that finds nothing is not evidence of safety unless you know it
    can find something -- which is why a deliberately weak configuration ships
    alongside a sound one.
"""

from __future__ import annotations

from dataclasses import dataclass, field
from enum import Enum


class Severity(str, Enum):
    OK = "ok"
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"


ORDER = {Severity.OK: 0, Severity.LOW: 1, Severity.MEDIUM: 2, Severity.HIGH: 3}


@dataclass(frozen=True)
class Posture:
    """How an application is configured. The subject of the review.

    Deliberately a flat description rather than live introspection: a review
    should be runnable against a design document before any code exists.
    """

    # Is retrieved/user content kept separate from instructions?
    marks_untrusted_content: bool = True
    # Is model output escaped before rendering, and never executed?
    escapes_output: bool = True
    executes_model_output: bool = False
    # What may the agent's tools do?
    tool_scopes: tuple[str, ...] = ("read:docs",)
    tools_require_confirmation: tuple[str, ...] = ()
    # Cost bounds
    per_request_cap: float | None = 0.05
    per_user_daily_cap: float | None = 1.0
    # Secrets
    secrets_in_environment: bool = True
    secrets_in_image: bool = False
    secrets_in_repository: bool = False
    # Data handling
    trace_retention_days: int | None = 30
    traces_in_erasure_path: bool = True
    redacts_before_storage: bool = True
    # Supply chain
    image_pinned_by_digest: bool = True
    dependencies_locked: bool = True


# Tool scopes that can change or destroy state, and so should not be granted
# without an explicit confirmation step.
DESTRUCTIVE_SCOPES = ("write:", "delete:", "admin:", "exec:", "payments:")


@dataclass
class Finding:
    check: str
    severity: Severity
    detail: str
    remediation: str = ""

    def line(self) -> str:
        base = f"  {self.severity.value.upper():<7} {self.check}: {self.detail}"
        return f"{base}\n          fix: {self.remediation}" if self.remediation else base


@dataclass
class Report:
    findings: list[Finding] = field(default_factory=list)

    @property
    def worst(self) -> Severity:
        return max((f.severity for f in self.findings), key=lambda s: ORDER[s], default=Severity.OK)

    def at_least(self, severity: Severity) -> list[Finding]:
        return [f for f in self.findings if ORDER[f.severity] >= ORDER[severity]]

    def names(self) -> set[str]:
        return {f.check for f in self.findings if f.severity is not Severity.OK}

    def summary(self) -> str:
        counts = {s: 0 for s in Severity}
        for f in self.findings:
            counts[f.severity] += 1
        verdict = "PASS" if self.worst in (Severity.OK, Severity.LOW) else "FAIL"
        return (
            f"{verdict} high={counts[Severity.HIGH]} medium={counts[Severity.MEDIUM]} "
            f"low={counts[Severity.LOW]}"
        )


# ------------------------------------------------------------------ checks


def check_untrusted_content_boundary(p: Posture) -> Finding:
    # TASK 1: HIGH when marks_untrusted_content is False, with a remediation
    # saying to delimit untrusted content and treat it as data to summarise
    # rather than instructions to obey. Otherwise OK.
    raise NotImplementedError("implement check_untrusted_content_boundary")

def check_output_handling(p: Posture) -> Finding:
    # TASK 2: HIGH if executes_model_output, else HIGH if not escapes_output,
    # else OK. Executing is the worse of the two, so check it first.
    raise NotImplementedError("implement check_output_handling")

def check_tool_permissions(p: Posture) -> Finding:
    # TASK 3: find scopes starting with any DESTRUCTIVE_SCOPES prefix.
    #   any not in tools_require_confirmation -> HIGH, naming them
    #   all confirmed                          -> LOW
    #   none present                           -> OK
    # Use a PREFIX test, not a substring one, or "read:docs" matches.
    raise NotImplementedError("implement check_tool_permissions")

def check_spend_bounds(p: Posture) -> Finding:
    # TASK 4: both caps missing -> HIGH, one missing -> MEDIUM, none -> OK.
    # Severity accumulates because the consequence does.
    raise NotImplementedError("implement check_spend_bounds")

def check_secret_handling(p: Posture) -> Finding:
    # TASK 5: in the repository -> HIGH, and the remediation must say ROTATE
    # first: deleting the file does not remove it from history. In the image ->
    # HIGH. No declared source -> MEDIUM. Otherwise OK.
    raise NotImplementedError("implement check_secret_handling")

def check_data_retention(p: Posture) -> Finding:
    # TASK 6: collect the problems -- no retention period, not in the erasure
    # path, not redacted before storage. More than one -> HIGH, exactly one ->
    # MEDIUM, none -> OK.
    raise NotImplementedError("implement check_data_retention")

def check_dependency_pinning(p: Posture) -> Finding:
    # TASK 7: an unpinned image or unlocked dependencies -> MEDIUM, naming
    # both when both apply. Otherwise OK.
    raise NotImplementedError("implement check_dependency_pinning")

CHECKS = (
    check_untrusted_content_boundary,
    check_output_handling,
    check_tool_permissions,
    check_spend_bounds,
    check_secret_handling,
    check_data_retention,
    check_dependency_pinning,
)


def review(posture: Posture) -> Report:
    """Run every check against one posture."""
    return Report(findings=[check(posture) for check in CHECKS])


# ------------------------------------------------- two postures to review


SOUND = Posture()

WEAK = Posture(
    marks_untrusted_content=False,
    escapes_output=False,
    tool_scopes=("read:docs", "write:tickets", "delete:records"),
    tools_require_confirmation=(),
    per_request_cap=None,
    per_user_daily_cap=None,
    secrets_in_environment=False,
    secrets_in_image=True,
    trace_retention_days=None,
    traces_in_erasure_path=False,
    redacts_before_storage=False,
    image_pinned_by_digest=False,
    dependencies_locked=False,
)
tests/run_tests.sh (205 bytes)
#!/usr/bin/env bash
# Run the day 361 security review test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_review.py (5254 bytes)
"""Grouped by check, plus a group proving the review CAN fail.

A review that has never found anything is not evidence of safety. Half of this
suite exists to show each check catches the weakness it owns, and only that one.

Run with: bash tests/run_tests.sh
"""

from __future__ import annotations

import os
import sys

HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))

from review import (  # noqa: E402
    CHECKS,
    SOUND,
    WEAK,
    Posture,
    Severity,
    review,
)


def failing(posture) -> set[str]:
    return review(posture).names()


# ------------------------------------------------------- the sound baseline


def test_a_sound_posture_passes_cleanly():
    report = review(SOUND)
    assert report.worst is Severity.OK
    assert report.names() == set()
    assert report.summary() == "PASS high=0 medium=0 low=0"


def test_every_check_runs():
    assert len(review(SOUND).findings) == len(CHECKS) == 7


# ------------------------------------------- each check catches its own flaw


def test_unmarked_untrusted_content_is_high():
    assert failing(Posture(marks_untrusted_content=False)) == {"untrusted_content_boundary"}


def test_unescaped_output_is_high():
    assert failing(Posture(escapes_output=False)) == {"output_handling"}


def test_executing_model_output_is_high():
    report = review(Posture(executes_model_output=True))
    finding = next(f for f in report.findings if f.check == "output_handling")
    assert finding.severity is Severity.HIGH
    assert "executed" in finding.detail


def test_destructive_scope_without_confirmation_is_high():
    assert failing(Posture(tool_scopes=("read:docs", "delete:records"))) == {"tool_permissions"}


def test_missing_spend_caps_is_high():
    assert failing(Posture(per_request_cap=None, per_user_daily_cap=None)) == {"spend_bounds"}


def test_secrets_in_the_image_is_high():
    assert failing(Posture(secrets_in_image=True)) == {"secret_handling"}


def test_secrets_in_the_repository_is_high_and_mentions_rotation():
    report = review(Posture(secrets_in_repository=True))
    finding = next(f for f in report.findings if f.check == "secret_handling")
    assert finding.severity is Severity.HIGH
    # Deleting the file does not remove it from history, so rotation comes first.
    assert "rotate" in finding.remediation


def test_unpinned_dependencies_are_medium():
    report = review(Posture(image_pinned_by_digest=False, dependencies_locked=False))
    finding = next(f for f in report.findings if f.check == "dependency_pinning")
    assert finding.severity is Severity.MEDIUM


# ---------------------------------------------------------------- severity


def test_severity_reflects_blast_radius_not_novelty():
    # A missing spend cap is dull and can take the service down; an unlocked
    # dependency is more interesting and less immediately dangerous.
    spend = next(f for f in review(Posture(per_request_cap=None, per_user_daily_cap=None)).findings
                 if f.check == "spend_bounds")
    deps = next(f for f in review(Posture(dependencies_locked=False)).findings
                if f.check == "dependency_pinning")
    assert spend.severity is Severity.HIGH
    assert deps.severity is Severity.MEDIUM


def test_one_missing_cap_is_less_severe_than_both():
    one = next(f for f in review(Posture(per_user_daily_cap=None)).findings
               if f.check == "spend_bounds")
    both = next(f for f in review(Posture(per_request_cap=None, per_user_daily_cap=None)).findings
                if f.check == "spend_bounds")
    assert one.severity is Severity.MEDIUM
    assert both.severity is Severity.HIGH


def test_a_confirmed_destructive_scope_is_only_low():
    report = review(
        Posture(tool_scopes=("read:docs", "write:tickets"),
                tools_require_confirmation=("write:tickets",))
    )
    finding = next(f for f in report.findings if f.check == "tool_permissions")
    assert finding.severity is Severity.LOW
    # Low findings do not fail the review.
    assert report.summary().startswith("PASS")


def test_a_high_finding_fails_the_review():
    assert review(Posture(escapes_output=False)).summary().startswith("FAIL")


# ------------------------------------------- the review can actually fail


def test_the_weak_posture_fails_loudly():
    report = review(WEAK)
    assert report.worst is Severity.HIGH
    assert report.summary() == "FAIL high=6 medium=1 low=0"


def test_the_weak_posture_trips_every_check():
    # Precision as well as recall: all seven checks report a problem, and the
    # sound posture reports none. A checker that cannot fail proves nothing.
    assert review(WEAK).names() == {c.__name__.replace("check_", "") for c in CHECKS}


def test_findings_that_fail_carry_remediation():
    for finding in review(WEAK).findings:
        if finding.severity is not Severity.OK:
            assert finding.remediation or finding.check == "tool_permissions", (
                f"{finding.check} has no remediation"
            )


def test_at_least_filters_by_severity():
    report = review(WEAK)
    assert len(report.at_least(Severity.HIGH)) == 6
    assert len(report.at_least(Severity.MEDIUM)) == 7

Troubleshooting

Troubleshooting — Day 361

The sound posture reports findings

Most often check_tool_permissions matching read:docs. DESTRUCTIVE_SCOPES must be tested as a prefix — s.startswith(DESTRUCTIVE_SCOPES) — not as a substring search, or any scope containing those characters matches.

A single configuration change trips several checks

Your checks are reading overlapping fields. Each should own one area and consult only its own settings. This matters beyond tidiness: if one change lights up four findings, the report cannot tell you which decision to reverse.

A confirmed destructive scope still reports HIGH

Compare the risky scopes against tools_require_confirmation before choosing the severity. Present-and-confirmed is LOW; present-and-unconfirmed is HIGH.

test_one_missing_cap_is_less_severe_than_both fails

Severity accumulates. Both caps missing is HIGH; exactly one missing is MEDIUM. Count the missing caps rather than returning a fixed severity whenever any is absent.

The verdict passes with a MEDIUM finding

Only OK and LOW pass. Check the comparison in summary — worst in (Severity.OK, Severity.LOW).

test_findings_that_fail_carry_remediation fails

Every non-OK finding needs a remediation, with one deliberate exception: the LOW tool-permissions finding, which records a confirmed destructive scope and has nothing to fix. If yours fails on a different check, that check is producing a complaint rather than a review.

test_secrets_in_the_repository_is_high_and_mentions_rotation fails

The remediation must contain the word "rotate". This is not a formatting requirement — deleting a committed secret does not remove it from history, so rotation is the actual first step and the finding should say so.

test_the_weak_posture_trips_every_check fails

Precision as well as recall. All seven checks must report a problem on WEAK and none on SOUND. If a check reports OK on the weak posture, it is reading a field the weak posture does not change.

NotImplementedError on every test

Expected. The starter stubs all seven checks, and every test runs at least one of them — including the tests asserting the sound posture passes, since those still have to run the checks to find out.

Security notes

Security notes — Day 361

This lab is defensive, and deliberately so

Every check here describes a configuration and reports a weakness so it can be fixed. There is no exploit code, nothing probes a running system, and the subject of the review is a fictional Posture dataclass rather than any real service.

That framing is the point rather than a disclaimer. A security review is an inspection, not an attack, and the skill being taught is finding your own weaknesses before someone else does.

What this lab does and does not touch

  • No network. The subject is a description, not a running system. No test opens a socket.
  • No credentials. secrets_in_environment and friends are booleans on a dataclass. There is no secret anywhere in this lab.
  • No writes outside this directory. Only __pycache__/, .pytest_cache/ and, if you create it, .venv/.
  • No sudo, no global installs.

Prompt injection: what the checks can and cannot do

check_untrusted_content_boundary is the most important check here and also the most limited, and it is worth being precise about why.

A language model receives instructions and data through one channel. There is no mechanism in the architecture that distinguishes "this is your instruction" from "this is a document I retrieved". Marking untrusted content with delimiters helps, and it is not a guarantee — a sufficiently persuasive document can still influence behaviour.

Which is why the remediation names the permission model rather than the prompt. The realistic posture is:

  • Assume the model may be persuaded. Design so that a persuaded model achieves little.
  • Grant the narrowest authority. An agent that cannot delete cannot be talked into deleting.
  • Require confirmation for consequential actions. A human in the loop for anything that writes, deletes or spends.
  • Never execute model output. Sandbox it, or have a person approve it.

Anyone claiming a filter that solves prompt injection is describing a mitigation, not a fix.

If a secret has been committed

The remediation on check_secret_handling says rotate first, and the ordering matters. Deleting the file removes it from the working tree and not from history — anyone with the repository, including anyone who cloned or forked it, still has the credential. Rewriting history helps only if you can guarantee every copy is rewritten, which for a public repository you cannot.

So: rotate the credential, then clean up. gitleaks and trufflehog scan history rather than the current tree, which is the right thing to check.

What a passing review does and does not mean

Seven checks catch seven classes of failure. A clean report means those seven are addressed. It does not mean the system is secure, and treating it that way is how a review becomes theatre.

The WEAK posture exists to make the passing result meaningful. Without evidence that the review can fail, a clean report and a review that checks nothing are indistinguishable from outside.

Reversing everything this lab did

rm -rf __pycache__ .pytest_cache .venv

Nothing else was created, and nothing outside this directory was modified.