Deployment, MLOps, and Security › AI Security and Privacy › Day 344

Day 344: Threat Modeling AI Systems

Day 344 of 365 — Threat Modeling AI Systems

Master systematic AI threat modeling: apply STRIDE for AI, map attack surfaces across OWASP Top 10 for LLMs, construct formal attack trees, and quantify threat risks using DREAD scoring in Python.

Course
Deployment, MLOps, and Security
Category
AI Security and Privacy
Reading time
≈ 35 min
Practical time
≈ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deployment-mlops-and-security/day-344-threat-modeling-ai-systems

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
    cd labs/sections/deployment-mlops-and-security/day-344-threat-modeling-ai-systems
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Deploying Generative AI models and autonomous agent workflows without rigorous threat modeling exposes organizations to devastating security breaches:

Applying STRIDE for AI, mapping the OWASP Top 10 for LLMs, constructing hierarchical Attack Trees, and calculating quantitative DREAD risk scores builds a defensible, multi-layered security posture for enterprise AI applications.

STRIDE AI Threat Matrix and Trust Boundaries showing Untrusted Ingress Domain, AI Gateway Trust Boundary, Agent Tool Execution Sandbox, and Backend Enterprise Data Stores with STRIDE Threat Classification

The idea in plain language

Diagram: a system prompt compared with an architectural trust boundary

Diagram: the four steps of threat modelling an AI system

Think of threat modeling an AI system like security-proofing a high-security automated bank vault:

Historical background

Diagram: the progression from STRIDE to the OWASP Top 10 for LLM applications

Threat modeling methodologies evolved across three major paradigms:

1. The Classic Microsoft STRIDE Era (2000–2015)

Microsoft introduced STRIDE to analyze deterministic C++ and web architectures: Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, and Elevation of Privilege.

2. Cloud and Microservice Threat Modeling (2016–2022)

Threat modeling shifted toward container isolation, IAM role boundary analysis, and API gateway security perimeters.

3. Generative AI and Agent Threat Modeling (2023–Present)

The OWASP Foundation released the OWASP Top 10 for LLMs, establishing global standards for non-deterministic AI threats, including indirect prompt injection, vector embedding denial-of-service, and autonomous agent tool abuse.

What it is — and what it is not

Let us define the core architecture of production AI threat modeling:

What AI Threat Modeling Is:

What AI Threat Modeling Is Not:

Diagram: what AI threat modelling is compared with two ways of getting it wrong

+-------------------------------------------------------------------------------+
|                    OWASP TOP 10 FOR LLM APPLICATIONS (2025)                   |
+-------------------+-----------------------+-----------------------------------+
| Vulnerability ID  | Threat Name           | Primary Exploit Vector            |
+-------------------+-----------------------+-----------------------------------+
| LLM01             | Prompt Injection      | Direct / Indirect input hijacking |
| LLM02             | Insecure Output       | Raw LLM output passed to SQL/Shell|
| LLM03             | Training Data Poison  | Manipulated pre-training / RAG doc|
| LLM04             | Model Denial of Serv  | Token exhaustion & sponge queries |
| LLM05             | Supply Chain Vuln     | Malicious pickle model checkpoints|
| LLM06             | Sensitive Info Leak   | System prompt & PII extraction    |
| LLM07             | Insecure Tool Calling | Excessive autonomous permissions  |
| LLM08             | Excessive Agency      | Autonomous actions without human  |
| LLM09             | Overreliance          | Hallucinated code / advice used   |
| LLM10             | Model Theft           | Weight extraction via distillation|
+-------------------+-----------------------+-----------------------------------+

Why it was created and what problems it solves

Formal AI threat modeling resolves the three primary vulnerabilities of modern intelligent applications:

1. Eliminating the “Prompt as a Security Boundary” Fallacy

LLMs are probabilistic token predictors that cannot fundamentally distinguish between control instructions and untrusted user data. Threat modeling enforces deterministic architectural barriers around the model.

2. Quantifying and Prioritizing Security Risks via DREAD

Rather than guessing which vulnerability to fix first, DREAD scoring provides a mathematical framework prioritizing critical high-damage, easily-reproducible vulnerabilities.

3. Hardening Autonomous Tool Calling Perimeters

Mapping agent trust boundaries prevents compromised LLM outputs from executing unauthorized destructive actions against enterprise databases or cloud infrastructure.

How it works

Diagram: the five DREAD factors and how they produce a priority

Let us inspect the complete AI Threat Modeling and DREAD Assessment Workflow:


Construct Hierarchical Attack Tree] - Adversarial Goal: "Exfiltrate Customer PII via Agent" ├── Path A: Exploit RAG via Indirect Injection └── Path B: Exploit SQL Tool via Insecure Output Handling │ ▼ [4. Calculate DREAD Risk Score] - Score D(9) + R(8) + E(7) + A(10) + D(8) = 8.4 / 10 (CRITICAL) │ ▼ [5. Generate Architectural Mitigation Action Plan] - Deploy In-Memory PII Scrubber & Read-Only DB Gateway

Hierarchical AI Attack Tree and DREAD Scoring decomposing an Adversarial Goal into Indirect Injection, Tool Poisoning, and Exfiltration paths with quantitative DREAD Risk Heatmap

An everyday analogy

Diagram: relying on instructions compared with an architectural barrier, in a cockpit

Think of threat modeling an AI agent like auditing the security of a commercial airline cockpit:

Examples in practice

Let us build a complete Python AI Threat Modeling & DREAD Risk Scorer that evaluates system threats, computes composite DREAD scores, and generates prioritized remediation plans:

from typing import Dict, Any, List, Optional

class AIThreatModelScorer:
    def __init__(self):
        self.identified_threats: List[Dict[str, Any]] = []

    def add_threat(self, threat_id: str, title: str, category_stride: str, 
                   owasp_id: str, damage: int, reproducibility: int, 
                   exploitability: int, affected_users: int, discoverability: int,
                   description: str) -> Dict[str, Any]:
        # Validate scores in [1, 10]
        scores = [damage, reproducibility, exploitability, affected_users, discoverability]
        for s in scores:
            if not (1 <= s <= 10):
                raise ValueError("All DREAD criteria must be integers between 1 and 10.")

        composite_score = round(sum(scores) / 5.0, 2)
        
        if composite_score >= 7.5:
            severity = "CRITICAL"
        elif composite_score >= 5.0:
            severity = "HIGH"
        elif composite_score >= 3.0:
            severity = "MEDIUM"
        else:
            severity = "LOW"

        record = {
            "threat_id": threat_id,
            "title": title,
            "stride_category": category_stride,
            "owasp_id": owasp_id,
            "dread_scores": {
                "damage": damage,
                "reproducibility": reproducibility,
                "exploitability": exploitability,
                "affected_users": affected_users,
                "discoverability": discoverability
            },
            "composite_score": composite_score,
            "severity": severity,
            "description": description
        }
        self.identified_threats.append(record)
        return record

    def get_prioritized_remediation_plan(self) -> List[Dict[str, Any]]:
        # Sort threats descending by composite DREAD score
        return sorted(self.identified_threats, key=lambda x: x["composite_score"], reverse=True)

if __name__ == "__main__":
    scorer = AIThreatModelScorer()
    
    # 1. Add Indirect Prompt Injection Threat
    scorer.add_threat(
        threat_id="THR-001",
        title="Indirect Prompt Injection via Ingested PDF Invoices",
        category_stride="Tampering / Elevation of Privilege",
        owasp_id="LLM01",
        damage=9, reproducibility=8, exploitability=8, affected_users=9, discoverability=8,
        description="Malicious invoice overrides agent prompt to trigger unauthorized bank transfers."
    )

    # 2. Add System Prompt Leakage Threat
    scorer.add_threat(
        threat_id="THR-002",
        title="System Prompt Extraction via Formatting Jailbreak",
        category_stride="Information Disclosure",
        owasp_id="LLM06",
        damage=5, reproducibility=7, exploitability=6, affected_users=4, discoverability=7,
        description="User tricks LLM into dumping secret system instructions."
    )

    plan = scorer.get_prioritized_remediation_plan()
    print(f"Evaluated {len(plan)} threats:")
    for t in plan:
        print(f"[{t['severity']}] {t['threat_id']}: {t['title']} (DREAD: {t['composite_score']}/10)")

Production Architecture Case Study: Enterprise AI Threat Model Document

Consider an enterprise threat model document for an autonomous AI customer service agent connected to internal SQL databases.

# AI Application Threat Model: Customer Support Copilot

**Application Tier:** Tier 1 (Production Customer-Facing)  
**LLM Engine:** Anthropic Claude 3.5 Sonnet / Llama-3.1-70B  
**Lead Security Architect:** Elena Rostova (Principal AI Security Engineer)  

## 1. System Data Flow Diagram & Trust Boundaries
- **Trust Boundary 0 (Public Internet):** User browsers submitting support tickets.
- **Trust Boundary 1 (API Gateway):** WAF, rate limiters, and input length filters.
- **Trust Boundary 2 (Agent Core):** LLM reasoner, prompt templates, and tool schemas.
- **Trust Boundary 3 (Enterprise Data):** Customer PostgreSQL database and Zendesk API.

## 2. STRIDE & OWASP Risk Assessment
| Threat ID | STRIDE Category | OWASP ID | DREAD Score | Severity | Mitigation Strategy |
| :--- | :--- | :--- | :--- | :--- | :--- |
| TM-01 | Elevation of Privilege | LLM01 | 8.6 / 10 | CRITICAL | Implement Dual-LLM Sandboxing & Tool Permission Guardrails |
| TM-02 | Information Disclosure | LLM06 | 7.8 / 10 | CRITICAL | Enforce In-Memory PII Regex Sanitizer & Output Canaries |
| TM-03 | Denial of Service | LLM04 | 6.4 / 10 | HIGH | Implement 4k Max Token Ingress Limit & Rate Limiters |
| TM-04 | Tampering | LLM03 | 5.8 / 10 | HIGH | Compute SHA256 hashes on RAG documentation chunks |

## 3. Mandatory Security Architecture Controls
1. **Parameterized Tool Execution:** The LLM never writes raw SQL queries; tools only accept structured scalar parameters (`user_id: int`, `order_id: str`).
2. **Read-Only Database Credentials:** The database connection pool assigned to the AI agent service account has strictly read-only SELECT permissions.

Detailed Failure Mode Taxonomy: AI Threat Modeling Hazards

Diagram: how an instruction in a third-party document reaches an agent's tools

+-------------------------------------------------------------------------------+
|                       AI THREAT MODELING FAILURE TAXONOMY                     |
+-------------------+-----------------------+-----------------------------------+
| Failure Mode      | Root Cause Analysis   | Architectural Mitigation          |
+-------------------+-----------------------+-----------------------------------+
| Relying on System | Assuming prompt words | Treat LLM output as 100% untrusted|
| Prompt as Defense | provide security      | and enforce code-level validation |
| Missing Indirect  | Only modeling direct  | Sanitize all third-party files,   |
| Injection Vectors | user prompt input     | web searches, and database inputs |
| Excessive Tool    | Agent granted global  | Enforce least-privilege IAM roles |
| Privileges        | database write access | and human-in-the-loop approvals   |
| Unbounded Token   | No max context bounds | Set strict token ingress caps     |
| Exhaustion        | on user input files   | and rate-limit customer tenants   |
+-------------------+-----------------------+-----------------------------------+

Deep Architectural Breakdown: Dual-LLM Sandboxing for Indirect Injection Defense

Diagram: how a quarantined model extracts structured data for a trusted controller

When processing untrusted external documents:

1. The Untrusted Quarantine Worker (LLM 1)

2. The Trusted Controller Agent (LLM 2)

+-------------------------------------------------------------------------------+
|                    DUAL-LLM QUARANTINE ARCHITECTURE                           |
+-------------------+-------------------+-------------------+-------------------+
| Pipeline Layer    | Execution Domain  | Tool Permissions  | Ingress Data      |
+-------------------+-------------------+-------------------+-------------------+
| Raw Document Ingest| Quarantine LLM 1 | ZERO Tools / Net  | Untrusted Raw PDF |
| Sanitized JSON Map| Gateway Parser    | Schema Validation | Extracted Fields  |
| Agent Controller  | Trusted LLM 2     | Authorized Tools  | Validated JSON    |
+-------------------+-------------------+-------------------+-------------------+

Implications: security, privacy, performance, scalability, and cost

Diagram: the security, operational, cost and process implications of AI threat modelling

1. Continuous Threat Model Auditing

Threat models are living documents. Whenever a new tool is added to an agent or a new external data source is connected to a RAG pipeline, the threat model must be updated and re-evaluated.

2. Latency and Cost Overhead of Defensive Sandboxing

Running dual-LLM quarantine pipelines increases total latency by 80ms and doubles token processing costs for document ingestion, but neutralizes 99% of indirect prompt injection attacks.

Diagram: the latency and token cost of dual-LLM quarantine against the risk it removes

Alternatives: free, open source, and commercial

Tool / FrameworkTypeBest Suited ForKey Capabilities
OWASP LLM Top 10Open FrameworkThreat TaxonomyGlobal vulnerability benchmark
Microsoft Threat Modeling ToolOpen SourceArchitectural ModelingData flow diagrams, STRIDE analysis
IriusRiskCommercial SaaSAutomated Threat ModelingContinuous compliance, architectural rules
Garak / PyRITOpen SourceAutomated AI Red TeamingLLM vulnerability scanner, jailbreak fuzzing

Diagram: architectural threat modelling compared with automated red teaming

+-----------------------+-----------------------+-----------------------+
| Dimension             | Ad-Hoc Penetration    | Systematic Threat Model|
+-----------------------+-----------------------+-----------------------+
| Timing                | Post-Deployment       | Pre-Architecture Design|
| Scope                 | Point-in-Time Bugs    | Full Architectural Map|
| Prioritization        | Subjective            | Quantitative (DREAD)  |
| Cost to Remediate     | 10x (Late stage)      | 1x (Designed in)      |
+-----------------------+-----------------------+-----------------------+

When to use it — and when not to

Diagram: when formal AI threat modelling is warranted and what triggers a re-run

Perform Formal AI Threat Modeling for:

Do NOT require formal DREAD audits for:

Production Checklist: AI Threat Modeling Hardening

The AI connection

Knowledge check

  1. What are the six STRIDE categories adapted for AI? Spoofing (Agent impersonation), Tampering (RAG poisoning), Repudiation (Unlogged actions), Information Disclosure (Prompt leaks), Denial of Service (Token depletion), and Elevation of Privilege (Tool hijacking).
  2. What is an architectural trust boundary? A perimeter separating untrusted external data inputs from trusted internal execution and credential domains.
  3. How does DREAD scoring prioritize vulnerabilities? By averaging scores across Damage, Reproducibility, Exploitability, Affected Users, and Discoverability.

Hands-on exercise

In this lab, you will build an AI Threat Modeling & DREAD Risk Scorer in Python. Your implementation will:

  1. Register threat vectors with STRIDE categories and OWASP LLM mappings.
  2. Validate DREAD criteria scores (1–10) and compute composite risk averages.
  3. Assign risk severity levels (CRITICAL, HIGH, MEDIUM, LOW).
  4. Generate prioritized remediation plans sorted by composite risk.

Expected output

============================= test session starts ==============================
collected 5 items

tests/test_threat_modeling.py .....                                      [100%]

============================== 5 passed in 0.02s ===============================
[THREAT SCORER] Initialized AI Threat Model Scorer.
[CRITICAL RISK] Evaluated THR-001: Indirect Prompt Injection (DREAD: 8.4/10).
[HIGH RISK] Evaluated THR-002: System Prompt Leakage (DREAD: 5.8/10).
[REMEDIATION PLAN] Generated prioritized action plan sorted by composite risk.

Validate your work

Run the pytest test suite:

pytest tests/run_tests.sh

Troubleshooting

  1. Score validation: Ensure all 5 DREAD inputs are integers between 1 and 10.
  2. Sorting logic: Use sorted(threats, key=lambda x: x["composite_score"], reverse=True).

Common mistakes

Practice assignment

Add a Mitigation Cost Estimator calculating the return on investment (Risk Reduction / Engineering Hours) for each remediation task.

Extension challenge

Implement a DOT Graphviz Exporter visualizing hierarchical attack trees as graphical diagrams.

Quiz

Q1. How is the traditional STRIDE security framework (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) adapted for Generative AI applications?

  1. It is completely discarded and replaced with random guessing
  2. STRIDE only applies to desktop video games
  3. It maps each STRIDE category onto AI-specific attacks
  4. STRIDE is only used for hardware CPU design
Show answer

Answer: C. It maps each STRIDE category onto AI-specific attacks

STRIDE for AI systematically categorizes vulnerabilities across agent identities, RAG data integrity, prompt extraction, and tool execution privileges.

Q2. What is the primary risk identified in OWASP LLM01 (Prompt Injection)?

  1. The model running out of electricity mid-request
  2. Typing in lowercase letters
  3. The prompt being too short
  4. Untrusted input bending the model's instructions
Show answer

Answer: D. Untrusted input bending the model's instructions

OWASP LLM01 occurs when untrusted input hijacks the LLM's operational control flow.

Q3. In DREAD risk scoring (Damage, Reproducibility, Exploitability, Affected Users, Discoverability), what does a high score indicate?

  1. An urgent risk to fix before deployment
  2. The system is 100% secure
  3. The model is simply running at high speed
  4. The server needs more RAM
Show answer

Answer: A. An urgent risk to fix before deployment

DREAD calculates composite risk severity; higher scores demand immediate remediation priority.

Q4. What is a 'Trust Boundary' in an AI agent architecture?

  1. A fence around the datacenter
  2. The line between untrusted data and trusted execution
  3. The boundary between CPU and RAM
  4. A legal nondisclosure agreement
Show answer

Answer: B. The line between untrusted data and trusted execution

Trust boundaries isolate unverified inputs from sensitive credentials and internal execution backends.

Q5. What is an 'AI Attack Tree'?

  1. A tree planted near the server room
  2. A binary search algorithm
  3. A diagram decomposing a goal into exploit paths
  4. A Python decision tree classifier
Show answer

Answer: C. A diagram decomposing a goal into exploit paths

Attack trees break down adversarial objectives into actionable, structured technical attack paths.

Glossary

STRIDE for AI
A structured methodology classifying threats: Spoofing, Tampering, Repudiation, Info Disclosure, DoS, Elevation.
OWASP Top 10 for LLMs
The industry-standard consensus list of the most critical security vulnerabilities in LLM applications.
DREAD Scoring
A quantitative risk assessment model evaluating Damage, Reproducibility, Exploitability, Affected Users, Discoverability.
Attack Tree
A tree diagram representing hierarchical paths and conditions an adversary uses to exploit a system.
Trust Boundary
A perimeter separating system components of differing trust levels, requiring validation upon crossing.
Indirect Prompt Injection
Exploiting an LLM via untrusted external third-party data (web pages, PDFs, emails) ingested at runtime.
Token Depletion Attack
An adversarial exploit flooding an AI service with complex computations to exhaust API budgets or compute.
Insecure Output Handling
Passing raw LLM generations directly into downstream parsers, SQL interpreters, or shells without sanitization.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.