Capstone › Capstone Build II: Ship It › Day 361
Hands-on lab — Day 361: Security Review of Your Capstone
- ← Back to the Day 361 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/capstone/day-361-security-review-of-your-capstone/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/review_demo.py Test
bash tests/run_tests.sh File tree
examples/review_demo.py examples/review.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/review.py tests/run_tests.sh tests/test_review.py troubleshooting.md
Lab README
Lab — Day 361: Security Review of Your Capstone
Lesson
- Lesson title: Security Review of Your Capstone
- Day number: 361 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-361-security-review-of-your-capstone
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-361-security-review-of-your-capstonewhen the site is running.
Purpose
Build a defensive security review for an AI application: seven checks over a described configuration, each producing a finding with a severity and a remediation.
Everything here describes a system and reports weaknesses so they can be fixed. There is no exploit code, and nothing attacks anything. The subject is a Posture — a flat description — so the review can run against a design document before any code exists, which is when changes are cheapest.
Learning objectives
- Express a system's security decisions as a reviewable posture.
- Implement seven checks, each owning exactly one failure class.
- Weight severity by blast radius rather than sophistication.
- Attach a remediation to every finding.
- Prove the review can fail, by running it against a deliberately weak configuration.
Prerequisites
- Day 360, "Monitoring and Cost Controls".
- Day 328, "Privacy in AI Systems", for the retention and redaction concepts.
- Comfortable with Python dataclasses and enums.
Supported operating systems
macOS, Linux and Windows (via WSL or PowerShell). Nothing is platform-specific.
Hardware requirements
Any machine running Python 3.10 or newer. No GPU. The suite completes in hundredths of a second.
Required software
- Python 3.10 or newer.
pytest7 or newer.
Free and open-source options
Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no API key, no network. The lesson points at the OWASP Top 10 for LLM Applications as the reference list, and at gitleaks, Trivy, Semgrep and Bandit as the tools worth adding to a real project.
Installation
python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt
Removing .venv fully reverses the installation.
File structure
starter/review.py your work: the seven check functions
examples/review.py reference review, plus a sound and a weak posture
examples/review_demo.py reviews both, then changes one setting at a time
tests/test_review.py grouped by check, plus "the review can fail"
tests/run_tests.sh suite entry point
expected-output/ real captured output and measured values
requirements/ pinned dependency
How to run
python3 examples/review_demo.py # review both postures
bash tests/run_tests.sh # run the suite
To work on the exercise, edit starter/review.py, then copy it over the reference:
cp starter/review.py examples/review.py
bash tests/run_tests.sh
Keep a copy of the reference first if you want to compare.
What the commands do
python3 examples/review_demo.py runs all seven checks against a sound posture and a deliberately weak one, printing each finding with its remediation, then makes three single-setting changes to a sound posture to show each check firing in isolation.
bash tests/run_tests.sh runs pytest over eighteen tests in three groups: the sound baseline, one test per check catching its own flaw, and a group proving the review can fail.
Expected output
--- sound posture ---
OK untrusted_content_boundary: retrieved content is marked untrusted
OK output_handling: output is escaped and never executed
...
=> PASS high=0 medium=0 low=0
--- weak posture ---
HIGH tool_permissions: state-changing scopes granted without confirmation:
write:tickets, delete:records
fix: grant the narrowest scope the task needs, and require explicit
confirmation for anything that writes, deletes or spends
...
=> FAIL high=6 medium=1 low=0
--- one change at a time ---
drop the spend caps FAIL high=1 ['spend_bounds']
grant delete without confirmation FAIL high=1 ['tool_permissions']
keep traces forever FAIL high=0 medium=1
expected-output/FIELDS.md explains each field and the severity scale.
Validation steps
bash tests/run_tests.shreports18 passed.- The sound posture must report zero findings and the weak one must trip all seven — precision as well as recall.
- Each isolation run must produce exactly one finding. If changing one setting lights up several checks, they are reading overlapping fields and cannot be prioritised.
- Every failing finding must carry a remediation.
Tests
Eighteen tests in three groups:
- the sound baseline — a clean posture passes with no findings, and all seven checks run.
- each check catches its own flaw — one test per check, asserting the finding set is exactly that check.
- severity and the ability to fail — severity tracks blast radius, one missing cap is less severe than two, a confirmed destructive scope is only low, the weak posture fails loudly and trips every check, and failing findings carry remediations.
Cleanup
rm -rf __pycache__ .pytest_cache
rm -rf .venv
Both are safe to remove and fully reversible.
Troubleshooting
See troubleshooting.md.
Security notes
See security.md. In short: this lab is entirely defensive, contains no exploit code, and its own subject is a fictional configuration.
Extension exercises
- A rate-limiting check. Add per-user and per-IP limits, and prove a posture with spend caps but no rate limits still fails — a cap bounds cost, not concurrency.
- A prioritised remediation plan. Order findings by consequence per unit of effort rather than by severity alone.
- Review your own capstone. Write its posture honestly, run the review, and record any finding you decide not to fix along with the reason.
Navigation
- Lesson
- Previous: Day 360 — Monitoring and Cost Controls
- Next: Day 362 — Documentation and Demo
Expected output
FIELDS.md
# Reading the review output
## A finding line
HIGH spend_bounds: no per-request and no per-user daily cap
fix: cap before the call, not after; anything a user can trigger
an attacker can trigger repeatedly
| Part | Meaning |
| --- | --- |
| severity | `OK`, `LOW`, `MEDIUM` or `HIGH`, weighted by **blast radius** rather than sophistication. |
| check | Which of the seven areas. Each owns one failure. |
| detail | What was actually found, with the specifics — which scopes, which caps, which controls. |
| fix | The remediation. A finding without one is a complaint rather than a review. |
## The verdict line
=> FAIL high=6 medium=1 low=0
`PASS` requires the worst finding to be `OK` or `LOW`. Any medium or high fails — a low finding is worth recording without blocking.
## The severity scale
| Severity | Meaning |
| --- | --- |
| `HIGH` | An attacker can act, data leaves, or the service can be taken down. |
| `MEDIUM` | Real exposure by a narrower path, or one of a pair of controls missing. |
| `LOW` | Worth noting. A destructive scope that exists but is gated behind confirmation. |
| `OK` | The control is present. |
## The isolation runs
drop the spend caps FAIL high=1 ['spend_bounds']
One configuration change should produce one finding. This is how you know the checks are independent, and it is what makes the output prioritisable: each finding points at exactly one decision you could reverse.
## Why two postures are reviewed
The demo reviews a sound configuration and a deliberately weak one. The second is the important half — a review that has never found anything is not evidence of safety, because a clean report and a review that checks nothing look identical from outside.
examples-run.txt
--- sound posture ---
OK untrusted_content_boundary: retrieved content is marked untrusted
OK output_handling: output is escaped and never executed
OK tool_permissions: read-only scopes only
OK spend_bounds: per-request and per-user caps present
OK secret_handling: credentials come from the environment
OK data_retention: traces are bounded, deletable and redacted
OK dependency_pinning: image and dependencies are pinned
=> PASS high=0 medium=0 low=0
--- weak posture ---
HIGH untrusted_content_boundary: retrieved and user content is concatenated into the prompt without a boundary
fix: keep instructions and untrusted content in separate, clearly delimited sections, and treat retrieved text as data the model may summarise but must not obey
HIGH output_handling: model output is rendered without escaping
fix: escape on every append; a tag split across two streamed tokens is not a tag in either half, so a whole-document sanitiser will miss it
HIGH tool_permissions: state-changing scopes granted without confirmation: write:tickets, delete:records
fix: grant the narrowest scope the task needs, and require explicit confirmation for anything that writes, deletes or spends
HIGH spend_bounds: no per-request and no per-user daily cap
fix: cap before the call, not after; anything a user can trigger an attacker can trigger repeatedly, and unbounded spend is an availability risk
HIGH secret_handling: credentials are baked into the container image
fix: an image is a distributable artefact and a later RUN rm does not remove a layer; supply secrets through the environment at run time
HIGH data_retention: traces are retained indefinitely; traces are not covered by deletion; content is stored before redaction
fix: traces hold full prompts and completions -- give them a retention period, include them in the erasure path, and redact before writing
MEDIUM dependency_pinning: image referenced by mutable tag; dependencies unlocked
fix: pin the image by digest and commit a lockfile, so what you deploy is provably what you tested
=> FAIL high=6 medium=1 low=0
--- one change at a time ---
drop the spend caps FAIL high=1 medium=0 low=0 ['spend_bounds']
grant delete without confirmation FAIL high=1 medium=0 low=0 ['tool_permissions']
keep traces forever FAIL high=0 medium=1 low=0 []
measured-values.txt
Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.
python3 examples/review_demo.py -> 26 lines, exit 0
bash tests/run_tests.sh -> 18 passed in 0.02s
sound posture PASS high=0 medium=0 low=0
weak posture FAIL high=6 medium=1 low=0
The weak posture trips all seven checks and the sound one trips none. Both
halves matter: recall without precision would mean a review that fails
everything, which is as useless as one that fails nothing.
Isolation runs, one configuration change each:
drop the spend caps FAIL high=1 ['spend_bounds']
grant delete without confirmation FAIL high=1 ['tool_permissions']
keep traces forever FAIL high=0 medium=1
One change, one finding. A review where changing one setting lights up four
checks cannot prioritise, because it cannot say which decision caused what.
Severity accumulation, same store:
traces kept forever MEDIUM
kept forever + no deletion path + no redaction HIGH
Each control missing alone leaves a partial mitigation. All three missing
removes every mitigation from the most sensitive store in the system.
run-output.txt
--- sound posture ---
OK untrusted_content_boundary: retrieved content is marked untrusted
OK output_handling: output is escaped and never executed
OK tool_permissions: read-only scopes only
OK spend_bounds: per-request and per-user caps present
OK secret_handling: credentials come from the environment
OK data_retention: traces are bounded, deletable and redacted
OK dependency_pinning: image and dependencies are pinned
=> PASS high=0 medium=0 low=0
--- weak posture ---
HIGH untrusted_content_boundary: retrieved and user content is concatenated into the prompt without a boundary
fix: keep instructions and untrusted content in separate, clearly delimited sections, and treat retrieved text as data the model may summarise but must not obey
HIGH output_handling: model output is rendered without escaping
fix: escape on every append; a tag split across two streamed tokens is not a tag in either half, so a whole-document sanitiser will miss it
HIGH tool_permissions: state-changing scopes granted without confirmation: write:tickets, delete:records
fix: grant the narrowest scope the task needs, and require explicit confirmation for anything that writes, deletes or spends
HIGH spend_bounds: no per-request and no per-user daily cap
fix: cap before the call, not after; anything a user can trigger an attacker can trigger repeatedly, and unbounded spend is an availability risk
HIGH secret_handling: credentials are baked into the container image
fix: an image is a distributable artefact and a later RUN rm does not remove a layer; supply secrets through the environment at run time
HIGH data_retention: traces are retained indefinitely; traces are not covered by deletion; content is stored before redaction
fix: traces hold full prompts and completions -- give them a retention period, include them in the erasure path, and redact before writing
MEDIUM dependency_pinning: image referenced by mutable tag; dependencies unlocked
fix: pin the image by digest and commit a lockfile, so what you deploy is provably what you tested
=> FAIL high=6 medium=1 low=0
--- one change at a time ---
drop the spend caps FAIL high=1 medium=0 low=0 ['spend_bounds']
grant delete without confirmation FAIL high=1 medium=0 low=0 ['tool_permissions']
keep traces forever FAIL high=0 medium=1 low=0 []
starter-run.txt
An untouched starter fails every test:
18 failed in 0.07s
All seven checks raise NotImplementedError, and every test runs at least one of
them -- including the tests that assert the SOUND posture passes, since those
still have to run the checks to find out.
test-output.txt
.................. [100%]
18 passed in 0.01s
test-run.txt
.................. [100%]
18 passed in 0.01s
Source files
examples/review_demo.py (1130 bytes)
#!/usr/bin/env python3
"""Review a sound posture and a deliberately weak one."""
from __future__ import annotations
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from review import SOUND, WEAK, Posture, Severity, review
def show(label: str, posture: Posture) -> None:
report = review(posture)
print(f"--- {label} ---")
for finding in report.findings:
print(finding.line())
print(f" => {report.summary()}")
def main() -> int:
show("sound posture", SOUND)
show("weak posture", WEAK)
print("--- one change at a time ---")
for label, posture in (
("drop the spend caps", Posture(per_request_cap=None, per_user_daily_cap=None)),
("grant delete without confirmation", Posture(tool_scopes=("read:docs", "delete:records"))),
("keep traces forever", Posture(trace_retention_days=None)),
):
report = review(posture)
highs = [f.check for f in report.at_least(Severity.HIGH)]
print(f" {label:<38} {report.summary():<32} {highs}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
examples/review.py (10263 bytes)
"""A security posture review for an AI application.
Offline and standard-library only. Everything here is DEFENSIVE: the checks
describe a system's configuration and report weaknesses so they can be fixed.
There is no exploit code, and nothing here attacks anything.
The checks follow the shape of the OWASP Top 10 for LLM Applications, reduced
to the seven that a capstone can actually get wrong:
untrusted_content_boundary retrieved text treated as instructions
output_handling model output rendered or executed unescaped
tool_permissions an agent that can do more than the task needs
spend_bounds no cap, so cost is an availability risk
secret_handling credentials in the image or the repository
data_retention traces kept forever, undeleted
dependency_pinning mutable references in the supply chain
Two ideas run through it.
* Severity is about BLAST RADIUS, not about how clever the flaw is. A missing
spend cap is dull and can take the service down.
* A review that finds nothing is not evidence of safety unless you know it
can find something -- which is why a deliberately weak configuration ships
alongside a sound one.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
class Severity(str, Enum):
OK = "ok"
LOW = "low"
MEDIUM = "medium"
HIGH = "high"
ORDER = {Severity.OK: 0, Severity.LOW: 1, Severity.MEDIUM: 2, Severity.HIGH: 3}
@dataclass(frozen=True)
class Posture:
"""How an application is configured. The subject of the review.
Deliberately a flat description rather than live introspection: a review
should be runnable against a design document before any code exists.
"""
# Is retrieved/user content kept separate from instructions?
marks_untrusted_content: bool = True
# Is model output escaped before rendering, and never executed?
escapes_output: bool = True
executes_model_output: bool = False
# What may the agent's tools do?
tool_scopes: tuple[str, ...] = ("read:docs",)
tools_require_confirmation: tuple[str, ...] = ()
# Cost bounds
per_request_cap: float | None = 0.05
per_user_daily_cap: float | None = 1.0
# Secrets
secrets_in_environment: bool = True
secrets_in_image: bool = False
secrets_in_repository: bool = False
# Data handling
trace_retention_days: int | None = 30
traces_in_erasure_path: bool = True
redacts_before_storage: bool = True
# Supply chain
image_pinned_by_digest: bool = True
dependencies_locked: bool = True
# Tool scopes that can change or destroy state, and so should not be granted
# without an explicit confirmation step.
DESTRUCTIVE_SCOPES = ("write:", "delete:", "admin:", "exec:", "payments:")
@dataclass
class Finding:
check: str
severity: Severity
detail: str
remediation: str = ""
def line(self) -> str:
base = f" {self.severity.value.upper():<7} {self.check}: {self.detail}"
return f"{base}\n fix: {self.remediation}" if self.remediation else base
@dataclass
class Report:
findings: list[Finding] = field(default_factory=list)
@property
def worst(self) -> Severity:
return max((f.severity for f in self.findings), key=lambda s: ORDER[s], default=Severity.OK)
def at_least(self, severity: Severity) -> list[Finding]:
return [f for f in self.findings if ORDER[f.severity] >= ORDER[severity]]
def names(self) -> set[str]:
return {f.check for f in self.findings if f.severity is not Severity.OK}
def summary(self) -> str:
counts = {s: 0 for s in Severity}
for f in self.findings:
counts[f.severity] += 1
verdict = "PASS" if self.worst in (Severity.OK, Severity.LOW) else "FAIL"
return (
f"{verdict} high={counts[Severity.HIGH]} medium={counts[Severity.MEDIUM]} "
f"low={counts[Severity.LOW]}"
)
# ------------------------------------------------------------------ checks
def check_untrusted_content_boundary(p: Posture) -> Finding:
if p.marks_untrusted_content:
return Finding("untrusted_content_boundary", Severity.OK, "retrieved content is marked untrusted")
return Finding(
"untrusted_content_boundary",
Severity.HIGH,
"retrieved and user content is concatenated into the prompt without a boundary",
"keep instructions and untrusted content in separate, clearly delimited sections, "
"and treat retrieved text as data the model may summarise but must not obey",
)
def check_output_handling(p: Posture) -> Finding:
if p.executes_model_output:
return Finding(
"output_handling",
Severity.HIGH,
"model output is executed",
"never execute generated code or queries directly; run them in a sandbox with no "
"credentials, or require a human to approve",
)
if not p.escapes_output:
return Finding(
"output_handling",
Severity.HIGH,
"model output is rendered without escaping",
"escape on every append; a tag split across two streamed tokens is not a tag in "
"either half, so a whole-document sanitiser will miss it",
)
return Finding("output_handling", Severity.OK, "output is escaped and never executed")
def check_tool_permissions(p: Posture) -> Finding:
risky = [s for s in p.tool_scopes if s.startswith(DESTRUCTIVE_SCOPES)]
unguarded = [s for s in risky if s not in p.tools_require_confirmation]
if unguarded:
return Finding(
"tool_permissions",
Severity.HIGH,
f"state-changing scopes granted without confirmation: {', '.join(unguarded)}",
"grant the narrowest scope the task needs, and require explicit confirmation for "
"anything that writes, deletes or spends",
)
if risky:
return Finding(
"tool_permissions",
Severity.LOW,
f"state-changing scopes present but confirmed: {', '.join(risky)}",
)
return Finding("tool_permissions", Severity.OK, "read-only scopes only")
def check_spend_bounds(p: Posture) -> Finding:
missing = [
name
for name, value in (("per-request", p.per_request_cap), ("per-user daily", p.per_user_daily_cap))
if value is None
]
if missing:
return Finding(
"spend_bounds",
Severity.HIGH if len(missing) == 2 else Severity.MEDIUM,
f"no {' and no '.join(missing)} cap",
"cap before the call, not after; anything a user can trigger an attacker can "
"trigger repeatedly, and unbounded spend is an availability risk",
)
return Finding("spend_bounds", Severity.OK, "per-request and per-user caps present")
def check_secret_handling(p: Posture) -> Finding:
if p.secrets_in_repository:
return Finding(
"secret_handling",
Severity.HIGH,
"credentials are committed to the repository",
"rotate them first -- history is not erased by deleting the file -- then move them "
"to the environment",
)
if p.secrets_in_image:
return Finding(
"secret_handling",
Severity.HIGH,
"credentials are baked into the container image",
"an image is a distributable artefact and a later RUN rm does not remove a layer; "
"supply secrets through the environment at run time",
)
if not p.secrets_in_environment:
return Finding("secret_handling", Severity.MEDIUM, "no declared source of credentials")
return Finding("secret_handling", Severity.OK, "credentials come from the environment")
def check_data_retention(p: Posture) -> Finding:
problems: list[str] = []
if p.trace_retention_days is None:
problems.append("traces are retained indefinitely")
if not p.traces_in_erasure_path:
problems.append("traces are not covered by deletion")
if not p.redacts_before_storage:
problems.append("content is stored before redaction")
if problems:
return Finding(
"data_retention",
Severity.HIGH if len(problems) > 1 else Severity.MEDIUM,
"; ".join(problems),
"traces hold full prompts and completions -- give them a retention period, include "
"them in the erasure path, and redact before writing",
)
return Finding("data_retention", Severity.OK, "traces are bounded, deletable and redacted")
def check_dependency_pinning(p: Posture) -> Finding:
problems: list[str] = []
if not p.image_pinned_by_digest:
problems.append("image referenced by mutable tag")
if not p.dependencies_locked:
problems.append("dependencies unlocked")
if problems:
return Finding(
"dependency_pinning",
Severity.MEDIUM,
"; ".join(problems),
"pin the image by digest and commit a lockfile, so what you deploy is provably what "
"you tested",
)
return Finding("dependency_pinning", Severity.OK, "image and dependencies are pinned")
CHECKS = (
check_untrusted_content_boundary,
check_output_handling,
check_tool_permissions,
check_spend_bounds,
check_secret_handling,
check_data_retention,
check_dependency_pinning,
)
def review(posture: Posture) -> Report:
"""Run every check against one posture."""
return Report(findings=[check(posture) for check in CHECKS])
# ------------------------------------------------- two postures to review
SOUND = Posture()
WEAK = Posture(
marks_untrusted_content=False,
escapes_output=False,
tool_scopes=("read:docs", "write:tickets", "delete:records"),
tools_require_confirmation=(),
per_request_cap=None,
per_user_daily_cap=None,
secrets_in_environment=False,
secrets_in_image=True,
trace_retention_days=None,
traces_in_erasure_path=False,
redacts_before_storage=False,
image_pinned_by_digest=False,
dependencies_locked=False,
)
metadata.yml (902 bytes)
lesson_id: D361
day: 361
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/review_demo.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-31'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3 examples/review_demo.py
reproduced the documented reviews exactly, with the weak posture tripping all seven checks and the
sound one tripping none; bash tests/run_tests.sh -> 18 passed in 0.02s. The untouched starter fails all
18. requires_network is false: the subject is a described configuration, not a running system, so
nothing is probed and nothing leaves the machine. The lab is defensive throughout and contains no
exploit code.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/review.py (7182 bytes)
"""A security posture review for an AI application.
Offline and standard-library only. Everything here is DEFENSIVE: the checks
describe a system's configuration and report weaknesses so they can be fixed.
There is no exploit code, and nothing here attacks anything.
The checks follow the shape of the OWASP Top 10 for LLM Applications, reduced
to the seven that a capstone can actually get wrong:
untrusted_content_boundary retrieved text treated as instructions
output_handling model output rendered or executed unescaped
tool_permissions an agent that can do more than the task needs
spend_bounds no cap, so cost is an availability risk
secret_handling credentials in the image or the repository
data_retention traces kept forever, undeleted
dependency_pinning mutable references in the supply chain
Two ideas run through it.
* Severity is about BLAST RADIUS, not about how clever the flaw is. A missing
spend cap is dull and can take the service down.
* A review that finds nothing is not evidence of safety unless you know it
can find something -- which is why a deliberately weak configuration ships
alongside a sound one.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
class Severity(str, Enum):
OK = "ok"
LOW = "low"
MEDIUM = "medium"
HIGH = "high"
ORDER = {Severity.OK: 0, Severity.LOW: 1, Severity.MEDIUM: 2, Severity.HIGH: 3}
@dataclass(frozen=True)
class Posture:
"""How an application is configured. The subject of the review.
Deliberately a flat description rather than live introspection: a review
should be runnable against a design document before any code exists.
"""
# Is retrieved/user content kept separate from instructions?
marks_untrusted_content: bool = True
# Is model output escaped before rendering, and never executed?
escapes_output: bool = True
executes_model_output: bool = False
# What may the agent's tools do?
tool_scopes: tuple[str, ...] = ("read:docs",)
tools_require_confirmation: tuple[str, ...] = ()
# Cost bounds
per_request_cap: float | None = 0.05
per_user_daily_cap: float | None = 1.0
# Secrets
secrets_in_environment: bool = True
secrets_in_image: bool = False
secrets_in_repository: bool = False
# Data handling
trace_retention_days: int | None = 30
traces_in_erasure_path: bool = True
redacts_before_storage: bool = True
# Supply chain
image_pinned_by_digest: bool = True
dependencies_locked: bool = True
# Tool scopes that can change or destroy state, and so should not be granted
# without an explicit confirmation step.
DESTRUCTIVE_SCOPES = ("write:", "delete:", "admin:", "exec:", "payments:")
@dataclass
class Finding:
check: str
severity: Severity
detail: str
remediation: str = ""
def line(self) -> str:
base = f" {self.severity.value.upper():<7} {self.check}: {self.detail}"
return f"{base}\n fix: {self.remediation}" if self.remediation else base
@dataclass
class Report:
findings: list[Finding] = field(default_factory=list)
@property
def worst(self) -> Severity:
return max((f.severity for f in self.findings), key=lambda s: ORDER[s], default=Severity.OK)
def at_least(self, severity: Severity) -> list[Finding]:
return [f for f in self.findings if ORDER[f.severity] >= ORDER[severity]]
def names(self) -> set[str]:
return {f.check for f in self.findings if f.severity is not Severity.OK}
def summary(self) -> str:
counts = {s: 0 for s in Severity}
for f in self.findings:
counts[f.severity] += 1
verdict = "PASS" if self.worst in (Severity.OK, Severity.LOW) else "FAIL"
return (
f"{verdict} high={counts[Severity.HIGH]} medium={counts[Severity.MEDIUM]} "
f"low={counts[Severity.LOW]}"
)
# ------------------------------------------------------------------ checks
def check_untrusted_content_boundary(p: Posture) -> Finding:
# TASK 1: HIGH when marks_untrusted_content is False, with a remediation
# saying to delimit untrusted content and treat it as data to summarise
# rather than instructions to obey. Otherwise OK.
raise NotImplementedError("implement check_untrusted_content_boundary")
def check_output_handling(p: Posture) -> Finding:
# TASK 2: HIGH if executes_model_output, else HIGH if not escapes_output,
# else OK. Executing is the worse of the two, so check it first.
raise NotImplementedError("implement check_output_handling")
def check_tool_permissions(p: Posture) -> Finding:
# TASK 3: find scopes starting with any DESTRUCTIVE_SCOPES prefix.
# any not in tools_require_confirmation -> HIGH, naming them
# all confirmed -> LOW
# none present -> OK
# Use a PREFIX test, not a substring one, or "read:docs" matches.
raise NotImplementedError("implement check_tool_permissions")
def check_spend_bounds(p: Posture) -> Finding:
# TASK 4: both caps missing -> HIGH, one missing -> MEDIUM, none -> OK.
# Severity accumulates because the consequence does.
raise NotImplementedError("implement check_spend_bounds")
def check_secret_handling(p: Posture) -> Finding:
# TASK 5: in the repository -> HIGH, and the remediation must say ROTATE
# first: deleting the file does not remove it from history. In the image ->
# HIGH. No declared source -> MEDIUM. Otherwise OK.
raise NotImplementedError("implement check_secret_handling")
def check_data_retention(p: Posture) -> Finding:
# TASK 6: collect the problems -- no retention period, not in the erasure
# path, not redacted before storage. More than one -> HIGH, exactly one ->
# MEDIUM, none -> OK.
raise NotImplementedError("implement check_data_retention")
def check_dependency_pinning(p: Posture) -> Finding:
# TASK 7: an unpinned image or unlocked dependencies -> MEDIUM, naming
# both when both apply. Otherwise OK.
raise NotImplementedError("implement check_dependency_pinning")
CHECKS = (
check_untrusted_content_boundary,
check_output_handling,
check_tool_permissions,
check_spend_bounds,
check_secret_handling,
check_data_retention,
check_dependency_pinning,
)
def review(posture: Posture) -> Report:
"""Run every check against one posture."""
return Report(findings=[check(posture) for check in CHECKS])
# ------------------------------------------------- two postures to review
SOUND = Posture()
WEAK = Posture(
marks_untrusted_content=False,
escapes_output=False,
tool_scopes=("read:docs", "write:tickets", "delete:records"),
tools_require_confirmation=(),
per_request_cap=None,
per_user_daily_cap=None,
secrets_in_environment=False,
secrets_in_image=True,
trace_retention_days=None,
traces_in_erasure_path=False,
redacts_before_storage=False,
image_pinned_by_digest=False,
dependencies_locked=False,
)
tests/run_tests.sh (205 bytes)
#!/usr/bin/env bash
# Run the day 361 security review test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_review.py (5254 bytes)
"""Grouped by check, plus a group proving the review CAN fail.
A review that has never found anything is not evidence of safety. Half of this
suite exists to show each check catches the weakness it owns, and only that one.
Run with: bash tests/run_tests.sh
"""
from __future__ import annotations
import os
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))
from review import ( # noqa: E402
CHECKS,
SOUND,
WEAK,
Posture,
Severity,
review,
)
def failing(posture) -> set[str]:
return review(posture).names()
# ------------------------------------------------------- the sound baseline
def test_a_sound_posture_passes_cleanly():
report = review(SOUND)
assert report.worst is Severity.OK
assert report.names() == set()
assert report.summary() == "PASS high=0 medium=0 low=0"
def test_every_check_runs():
assert len(review(SOUND).findings) == len(CHECKS) == 7
# ------------------------------------------- each check catches its own flaw
def test_unmarked_untrusted_content_is_high():
assert failing(Posture(marks_untrusted_content=False)) == {"untrusted_content_boundary"}
def test_unescaped_output_is_high():
assert failing(Posture(escapes_output=False)) == {"output_handling"}
def test_executing_model_output_is_high():
report = review(Posture(executes_model_output=True))
finding = next(f for f in report.findings if f.check == "output_handling")
assert finding.severity is Severity.HIGH
assert "executed" in finding.detail
def test_destructive_scope_without_confirmation_is_high():
assert failing(Posture(tool_scopes=("read:docs", "delete:records"))) == {"tool_permissions"}
def test_missing_spend_caps_is_high():
assert failing(Posture(per_request_cap=None, per_user_daily_cap=None)) == {"spend_bounds"}
def test_secrets_in_the_image_is_high():
assert failing(Posture(secrets_in_image=True)) == {"secret_handling"}
def test_secrets_in_the_repository_is_high_and_mentions_rotation():
report = review(Posture(secrets_in_repository=True))
finding = next(f for f in report.findings if f.check == "secret_handling")
assert finding.severity is Severity.HIGH
# Deleting the file does not remove it from history, so rotation comes first.
assert "rotate" in finding.remediation
def test_unpinned_dependencies_are_medium():
report = review(Posture(image_pinned_by_digest=False, dependencies_locked=False))
finding = next(f for f in report.findings if f.check == "dependency_pinning")
assert finding.severity is Severity.MEDIUM
# ---------------------------------------------------------------- severity
def test_severity_reflects_blast_radius_not_novelty():
# A missing spend cap is dull and can take the service down; an unlocked
# dependency is more interesting and less immediately dangerous.
spend = next(f for f in review(Posture(per_request_cap=None, per_user_daily_cap=None)).findings
if f.check == "spend_bounds")
deps = next(f for f in review(Posture(dependencies_locked=False)).findings
if f.check == "dependency_pinning")
assert spend.severity is Severity.HIGH
assert deps.severity is Severity.MEDIUM
def test_one_missing_cap_is_less_severe_than_both():
one = next(f for f in review(Posture(per_user_daily_cap=None)).findings
if f.check == "spend_bounds")
both = next(f for f in review(Posture(per_request_cap=None, per_user_daily_cap=None)).findings
if f.check == "spend_bounds")
assert one.severity is Severity.MEDIUM
assert both.severity is Severity.HIGH
def test_a_confirmed_destructive_scope_is_only_low():
report = review(
Posture(tool_scopes=("read:docs", "write:tickets"),
tools_require_confirmation=("write:tickets",))
)
finding = next(f for f in report.findings if f.check == "tool_permissions")
assert finding.severity is Severity.LOW
# Low findings do not fail the review.
assert report.summary().startswith("PASS")
def test_a_high_finding_fails_the_review():
assert review(Posture(escapes_output=False)).summary().startswith("FAIL")
# ------------------------------------------- the review can actually fail
def test_the_weak_posture_fails_loudly():
report = review(WEAK)
assert report.worst is Severity.HIGH
assert report.summary() == "FAIL high=6 medium=1 low=0"
def test_the_weak_posture_trips_every_check():
# Precision as well as recall: all seven checks report a problem, and the
# sound posture reports none. A checker that cannot fail proves nothing.
assert review(WEAK).names() == {c.__name__.replace("check_", "") for c in CHECKS}
def test_findings_that_fail_carry_remediation():
for finding in review(WEAK).findings:
if finding.severity is not Severity.OK:
assert finding.remediation or finding.check == "tool_permissions", (
f"{finding.check} has no remediation"
)
def test_at_least_filters_by_severity():
report = review(WEAK)
assert len(report.at_least(Severity.HIGH)) == 6
assert len(report.at_least(Severity.MEDIUM)) == 7
Troubleshooting
Troubleshooting — Day 361
The sound posture reports findings
Most often check_tool_permissions matching read:docs. DESTRUCTIVE_SCOPES must be tested as a prefix — s.startswith(DESTRUCTIVE_SCOPES) — not as a substring search, or any scope containing those characters matches.
A single configuration change trips several checks
Your checks are reading overlapping fields. Each should own one area and consult only its own settings. This matters beyond tidiness: if one change lights up four findings, the report cannot tell you which decision to reverse.
A confirmed destructive scope still reports HIGH
Compare the risky scopes against tools_require_confirmation before choosing the severity. Present-and-confirmed is LOW; present-and-unconfirmed is HIGH.
test_one_missing_cap_is_less_severe_than_both fails
Severity accumulates. Both caps missing is HIGH; exactly one missing is MEDIUM. Count the missing caps rather than returning a fixed severity whenever any is absent.
The verdict passes with a MEDIUM finding
Only OK and LOW pass. Check the comparison in summary — worst in (Severity.OK, Severity.LOW).
test_findings_that_fail_carry_remediation fails
Every non-OK finding needs a remediation, with one deliberate exception: the LOW tool-permissions finding, which records a confirmed destructive scope and has nothing to fix. If yours fails on a different check, that check is producing a complaint rather than a review.
test_secrets_in_the_repository_is_high_and_mentions_rotation fails
The remediation must contain the word "rotate". This is not a formatting requirement — deleting a committed secret does not remove it from history, so rotation is the actual first step and the finding should say so.
test_the_weak_posture_trips_every_check fails
Precision as well as recall. All seven checks must report a problem on WEAK and none on SOUND. If a check reports OK on the weak posture, it is reading a field the weak posture does not change.
NotImplementedError on every test
Expected. The starter stubs all seven checks, and every test runs at least one of them — including the tests asserting the sound posture passes, since those still have to run the checks to find out.
Security notes
Security notes — Day 361
This lab is defensive, and deliberately so
Every check here describes a configuration and reports a weakness so it can be fixed. There is no exploit code, nothing probes a running system, and the subject of the review is a fictional Posture dataclass rather than any real service.
That framing is the point rather than a disclaimer. A security review is an inspection, not an attack, and the skill being taught is finding your own weaknesses before someone else does.
What this lab does and does not touch
- No network. The subject is a description, not a running system. No test opens a socket.
- No credentials.
secrets_in_environmentand friends are booleans on a dataclass. There is no secret anywhere in this lab. - No writes outside this directory. Only
__pycache__/,.pytest_cache/and, if you create it,.venv/. - No
sudo, no global installs.
Prompt injection: what the checks can and cannot do
check_untrusted_content_boundary is the most important check here and also the most limited, and it is worth being precise about why.
A language model receives instructions and data through one channel. There is no mechanism in the architecture that distinguishes "this is your instruction" from "this is a document I retrieved". Marking untrusted content with delimiters helps, and it is not a guarantee — a sufficiently persuasive document can still influence behaviour.
Which is why the remediation names the permission model rather than the prompt. The realistic posture is:
- Assume the model may be persuaded. Design so that a persuaded model achieves little.
- Grant the narrowest authority. An agent that cannot delete cannot be talked into deleting.
- Require confirmation for consequential actions. A human in the loop for anything that writes, deletes or spends.
- Never execute model output. Sandbox it, or have a person approve it.
Anyone claiming a filter that solves prompt injection is describing a mitigation, not a fix.
If a secret has been committed
The remediation on check_secret_handling says rotate first, and the ordering matters. Deleting the file removes it from the working tree and not from history — anyone with the repository, including anyone who cloned or forked it, still has the credential. Rewriting history helps only if you can guarantee every copy is rewritten, which for a public repository you cannot.
So: rotate the credential, then clean up. gitleaks and trufflehog scan history rather than the current tree, which is the right thing to check.
What a passing review does and does not mean
Seven checks catch seven classes of failure. A clean report means those seven are addressed. It does not mean the system is secure, and treating it that way is how a review becomes theatre.
The WEAK posture exists to make the passing result meaningful. Without evidence that the review can fail, a clean report and a review that checks nothing are indistinguishable from outside.
Reversing everything this lab did
rm -rf __pycache__ .pytest_cache .venv
Nothing else was created, and nothing outside this directory was modified.