# How Should Teams Threat Model Agentic AI Systems in 2026?

Savannah Jenkins · September 23, 2026

> What Is the Best Approach to Threat Modeling Agentic AI? As of 24 September 2026, the strongest approach combines system architecture review, data-flow...

## What Is the Best Approach to Threat Modeling Agentic AI?

As of 24 September 2026, the strongest approach combines system architecture review, data-flow diagrams, attack trees, adversarial testing, and runtime control validation. The unit of analysis is not the language model by itself; it is the complete agentic system, including prompts, memory, identities, tools, data sources, external services, and people who approve its actions. Conventional threat modeling still provides the foundation, but it must be extended to cover non-deterministic decisions, delegated authority, indirect prompt injection, memory poisoning, and interactions that span multiple systems. The objective is not to prove that every malicious instruction will be blocked, which is not a realistic assurance claim, but to establish what the agent can access and bound the damage it can cause when instructions, models, or data are compromised.

**Also worth reading:** [How Does an Agentic AI Microsegmentation Zero Trust Framework Protect Enterprise Systems in 2026?](https://agustin-otegui.com/knowledge/how_does_an_agentic_ai_microsegmentation_zero_trust_framework_protect_enterprise_systems_in_2026.php) · [What Are the Definitive Architectural Best Practices for Governing Autonomous Agentic AI Systems in 2026?](https://agustin-otegui.com/knowledge/what_are_the_definitive_architectural_best_practices_for_governing_autonomous_agentic_ai_systems_in_2026.php) · [What are deterministic AI verification methods and why do they matter for agentic systems in 2026?](https://agustin-otegui.com/knowledge/what_are_deterministic_ai_verification_methods_and_why_do_they_matter_for_agentic_systems_in_2026.php)

A practical program treats threat modeling as a continuous engineering activity tied to architecture changes, model releases, new tools, and revised permissions. That matters because an agent that can only search a private knowledge base presents a different risk from one that can search that knowledge base, modify customer records, and send external email. Teams should document several concrete failure paths before deployment, assign an owner to each high-risk path, and test whether the proposed controls work under realistic conditions. In this sense, agentic AI threat modeling is both a design method and a recurring verification process rather than a one-time document created immediately before launch.

## Why Traditional Threat Models Are Not Enough for Autonomous Agents

Traditional application threat models usually assume a relatively stable mapping between user input, application logic, and backend permissions. Agentic systems weaken that assumption because the model interprets goals, decomposes tasks, selects tools, generates arguments, and can revise its next step after observing an external response. An error that would produce a bad sentence in a chatbot can instead become a database update, a payment instruction, a credential disclosure, or a lateral movement step. The model is therefore a policy interpreter and action broker, not merely an output generator.

Non-determinism also changes how attacks are observed and reproduced. A malicious goal may be rejected on one run, partially executed on another, and redirected on a third because sampling, tool responses, memory, and environmental state differ. Unit 42's published account titled How AI Agents Breached a Network in 10 Hours illustrates how quickly an autonomous testing agent progressed through a network engagement, although the ten-hour figure describes a particular exercise rather than a universal capability or incident rate. The lesson is that response time and tool autonomy must be included in the model, not treated as external implementation details.

This creates a new insider-threat problem as well. A well-behaved employee may grant an agent broad permissions because manual approval appears too slow, while the employee or operator may not realize how easily an injected instruction can activate those permissions. Dark Reading's discussion of agentic insider risks reflects this concern, but it should not be read as evidence that every human operator is malicious. The more accurate point is that delegated authority concentrates both helpful capability and potential misuse in an identity that can act faster than ordinary review processes. Controls must therefore reduce reliance on continuous human vigilance and place enforceable limits around the agent itself.

## How Should Teams Model the Agent, Its Tools, and Its Data?

Start with a diagram that can be read by engineering, security, data, and business owners. Show the model boundary, orchestration layer, memory stores, retrieval systems, tool endpoints, credentials, approval interfaces, external channels, and every trust boundary crossed by data or authority. Include agents that are not visible in the primary product, such as background workers, evaluator models, scheduled jobs, and supervisor agents. A useful scope question is whether the system can change state outside its own conversation; if it can, the security model must follow that state change all the way to the affected business process.

Threat categories such as STRIDE and privacy-oriented methods such as LINDDUN remain useful, but they need agent-specific scenarios. OWASP's generative AI security work includes prompt injection, sensitive information disclosure, supply-chain weaknesses, improper output handling, excessive agency, and unbounded consumption, all of which can affect an agentic application. MITRE ATLAS supplies an adversarial vocabulary for discussing attacks against AI systems, while NIST's AI Risk Management Framework, published in January 2023, organizes work into Govern, Map, Measure, and Manage. These references help teams communicate, but they do not replace a system-specific model of who can invoke which tool under which conditions.

Prioritization should be explicit rather than based on alarm. A five-by-five likelihood and impact matrix is adequate for many teams, provided that impact includes confidentiality, integrity, availability, financial loss, safety, and regulatory exposure. A score of 15 or higher can trigger executive review and a named remediation owner, but the threshold is an internal convention rather than an official standard. Illustrate the rating with a sequence: an attacker embeds instructions in a supplier document, the agent reads it, retrieves a customer record, composes an email, and sends it without confirmation. That sequence exposes the exact data, tool, credential, and missing control that must be addressed.

## Which Agentic AI Threat Modeling Techniques Work Best?

No single technique sees the entire system. Manual workshops are good at exposing business and organizational assumptions, code-derived tools are good at tracing implemented routes, adversarial exercises test runtime behavior, and policy validation checks whether governance matches actual enforcement. The most defensible result comes from combining at least three of these methods and recording disagreements between their findings. Automation is particularly useful for keeping diagrams and attack paths current, but generating more findings does not guarantee that the most dangerous failures are understood.

| Feature | Manual Architecture Workshop | Code-Derived Analysis | Adversarial Simulation | Runtime Control Validation |
| --- | --- | --- | --- | --- |
| Best use | Business logic, abuse cases, ownership, and organizational failure | Credentials, routes, dependencies, and data flows | Prompt injection, tool misuse, memory attacks, and unsafe sequences | Enforcement of approvals, egress rules, timeouts, budgets, and kill switches |
| Main strength | Surfaces assumptions that code and diagrams do not show | Repeats structural analysis as the codebase changes | Tests behavior rather than intended design | Confirms that controls work when the agent actually executes tasks |
| Main blind spot | Becomes stale and depends heavily on participant knowledge | Can miss external channels, prompts, human decisions, and indirect data injection | Expensive, probabilistic, and difficult to reproduce | Usually tests known attack paths rather than discovering novel ones |
| Useful cadence | At design and at least quarterly | On every relevant code or infrastructure change | Before release and after meaningful model, tool, or policy changes | Daily in production, with deeper reviews after alerts or incidents |
| Typical cost profile | Professional facilitation and staff time | Free or low-cost tools, plus engineering review | Model usage, test data, engineering time, and containment preparation | Monitoring, policy enforcement, identity, and incident response staffing |

TITO, presented on Hacker News as an open-source project for automated threat modeling from code, illustrates the code-derived category well. Such a tool can identify routes, sinks, and architectural relationships more consistently than a workshop held months earlier, but a clean code scan does not prove that the production system is safe. An LLM can also assist with abuse-case generation or diagram interpretation, yet its output needs source review because it may invent endpoints, misread authorization boundaries, or treat a plausible attack as technically possible. Organizations should prefer evidence-linked findings over eloquent reports. Each proposed threat should cite the relevant architecture, code, configuration, prompt, or observed execution trace, and reviewers should reject anything that cannot be reproduced or connected to a business impact.

## What Does a Practical 30-Day Threat Modeling Cycle Look Like?

During days 1 through 5, inventory every model, agent, tool, memory store, credential, external data source, and administrative path. During days 6 through 10, create the system diagram and identify at least 3 trust zones, even if the architecture is a single cloud account. During days 11 through 15, write abuse cases for direct prompt injection, indirect injection, malicious retrieved content, memory tampering, tool argument manipulation, secret exposure, and excessive autonomy. The team should then select scenarios whose failure could affect money, customers, regulated data, or production availability rather than attempting to enumerate every theoretical input.

Days 16 through 20 should be used to run those scenarios in a test environment with realistic but synthetic data. Test whether the agent changes its plan after a failure, whether an attacker can influence a second agent, and whether an approval request reveals enough detail for a human to make an informed decision. Include resource attacks such as recursive tool calls, long-running loops, and attacker-selected URLs. Do not treat a final response containing a warning as a successful control if the model has already accessed a secret or invoked a tool before producing that warning.

Days 21 through 25 should translate the findings into enforceable controls, while days 26 through 30 should validate them through tabletop exercises and a second agent run. A useful action model has 3 levels: observe, propose, and execute. Read-only retrieval may operate in observe mode; drafting a customer response may require proposal and approval; changing a production record should require narrowly scoped authorization and a reversible transaction. Teams can set explicit budgets, such as 10 tool calls per task, 1,000 retrieved records, a 60-minute execution window, and a fixed spend ceiling, although the correct values depend on the business process. Controls should fail closed for destructive actions, unknown tools, malformed outputs, and authorization failures rather than silently falling back to a broader permission set.

## Which Guardrails and Detection Rules Should Be Tested?

Agent guardrails operate at several layers, and no single prompt instruction is dependable enough to control every layer by itself. Input inspection can flag suspicious instructions, but benign documents can contain those patterns, so filtering must include context and allowlists. The stronger controls usually sit in infrastructure: short-lived credentials, per-tool authorization, read-only defaults, output validation, destination allowlists, restricted network access, memory write policies, and confirmation before irreversible external actions. A human approval button is useful only when the agent presents the intended action, target, data, and consequence in a form the reviewer can verify.

Runtime detection should reconstruct what happened across the model, orchestrator, tools, and external systems. Record agent identity, session, model version, prompt-template version, selected tool, redacted arguments, approval status, result, latency, token use, memory writes, and final business effect. Logs must avoid copying unnecessary secrets, because observability systems can become high-value data stores when they capture prompts and tool results verbatim. Access to those logs should itself follow least privilege, retention rules, and tamper-resistant storage.

A pilot can use measurable acceptance gates without pretending that they are universal. For example, run at least 100 seeded attacks across high-risk scenarios and require 100% blocking of unauthorized external writes and production privilege changes, plus at least 95% blocking of the remaining critical abuse cases. A safer interpretation is that a single successful unauthorized write fails the gate, regardless of the overall percentage. Measure false positives, human override time, mean time to detect, mean time to contain, cost per completed task, and the proportion of actions that reached execution without an independent control. As of 24 September 2026, no consensus methodology supports claims that an agent is safe because a vendor scored it highly on a general benchmark.

## What Are the Most Common Threat Modeling Mistakes?

The most frequent error is treating the language model as the entire system and ignoring tools, identities, memory, retrieval, and external communications. A second error is performing one penetration test before launch and assuming that later prompt or model changes cannot alter the attack surface. A third is equating a high task-completion rate with security, particularly when the agent becomes more capable by using more permissions. A fourth is assuming that prompt injection can be completely solved by asking the model to ignore other instructions; that expectation ignores direct and indirect channels, content interpretation errors, and non-deterministic behavior.

Organizations also make the mistake of giving an agent a permanent credential because short-lived identity integration is inconvenient. They may set no tool-call limit because cost monitoring seems unnecessary during a successful pilot, and they may enable unrestricted network access to support research tasks without considering command execution or data-transfer risks. Another recurring mistake is testing only obvious attacks while neglecting compromised documents, stale memory, tool metadata, and instructions embedded in structured output. The final error is declaring a kill switch during an exercise without confirming that operators can use it, restore clean state, revoke active tokens, and resume safely.

These mistakes often reflect under-scoped governance rather than ignorance among individual engineers. A model owner may not control the identity platform, while a security team may not know that a research notebook has acquired production credentials. Dark Reading's framing of agents as insider risks is useful because it highlights governance, delegation, and human incentives, but the remedy is not to treat every operator as hostile. Remedy the design by making normal behavior safe, making high-impact actions attributable, and ensuring that suspicious or unauthorized behavior produces evidence an independent team can inspect.

## When Should a Team Act, and What Will Threat Modeling Cost?

Threat modeling should begin before an agent receives production data, external write access, or a budget. Prioritize it when one workflow crosses 3 trust zones, interacts with untrusted customer or public content, retains memory across sessions, delegates work to another agent, or can trigger financial or administrative actions. A read-only assistant over a small, curated corpus may justify a lighter review, but the decisive factors are consequence and change rate rather than the label agentic. A slow agent that modifies payment instructions can still create more exposure than a fast search tool that returns public documents. Regulated deployments also need documented risk ownership, although legal interpretation should come from qualified counsel rather than a threat-modeling report.

The European Union's AI Act entered into force on 1 August 2024 and applies in stages, so teams should not reduce compliance to a single deadline or assume every AI feature has identical obligations. Technical evidence, control ownership, testing records, and incident procedures support governance, but they do not create a legal safe harbor. As of 24 September 2026, there is no standard market price for an agentic AI threat model. Open-source and manual methods can begin at zero direct licence cost, while hosted analysis and monitoring tools may range from free tiers to several hundred US dollars per month for a small team; an independent enterprise review often falls within a broad planning band of $25,000 to $150,000, depending on integration depth and assurance expectations.

These figures are planning estimates, not quotations, and model usage, data volume, cloud infrastructure, and response requirements can move them substantially. A useful pilot might consume $500 to $5,000 in engineering and testing effort before any production rollout, while a large multi-agent platform can require dedicated identity, policy, telemetry, and incident-response work. Teams should fund the highest-consequence paths first rather than buying an elaborate platform before defining the system. If an external consultant is used, the deliverable should expose assumptions, diagrams, reproducible scenarios, test evidence, and unresolved residual risk, not a branded score with no engineering value.

## What Does a Mature Agentic Security Practice Look Like?

Maturity means the model, security team, data owner, platform engineer, and business process owner can explain the same system in compatible terms. Security should participate when tools and identity boundaries are designed, while engineering should own reproducible control tests and runtime evidence. NIST's Govern, Map, Measure, and Manage functions provide a useful management frame, and AWS guidance on agentic security similarly emphasizes constrained autonomy, appropriate identity, guardrails, and observability. Public frameworks such as AEGIS also reinforce the need for governance across the lifecycle, but framework alignment should support decisions rather than become a compliance exercise performed after design.

Run the full review at least quarterly for stable systems and whenever a model, tool, permission, memory policy, data source, or agent topology changes materially. Maintain a small set of operational measures, including unauthorized-action attempts, approval bypasses, policy denials, rollback time, containment time, cost anomalies, and the share of agent activity that is attributable to a named identity. Review false negatives as carefully as false positives, because repeated approvals or ignored alerts can quietly train people to accept ineffective controls. A good report distinguishes prevented attacks, detected attacks, attacks that caused no effect, and attacks that exposed a residual risk, rather than compressing everything into a green or red status.

The best answer is therefore a living model of intent, authority, data, and execution, tested continuously and enforced outside the model whenever possible. Agentic AI threat modeling does not eliminate uncertainty, but it makes that uncertainty visible, limits the blast radius, and gives decision-makers defensible evidence. That is a higher standard than claiming that a model is secure because it passed a prompt evaluation, and a more useful one than treating every AI deployment as equally dangerous. As of 24 September 2026, architecture discipline, least-privilege execution, adversarial evidence, and operational accountability remain more reliable than confidence in a single scanner or vendor promise.

## Quick answers

### Is agentic AI threat modeling different from standard LLM security testing?

Yes. Conventional LLM testing often focuses on harmful outputs, data leakage, or prompt injection, while agentic threat modeling also follows tool selection, credentials, memory, delegation, and external actions. The system can be technically accurate yet unsafe if a single erroneous action can alter a customer record or send sensitive data.

### What is the most important control for a tool-using AI agent?

Narrow, task-specific authorization is usually the most important control. An agent should receive only the data and tool permissions required for the current task, with short-lived credentials and independent enforcement outside the prompt. A human approval step adds value when the consequence is high, but it does not replace limits on what the agent can do.

### Can automated code scanners fully replace threat modeling workshops?

No. Code-derived tools can identify routes, dependencies, and sinks efficiently, but they may miss malicious content, human decisions, external channels, and organizational incentives. Manual workshops and adversarial tests are still needed to connect technical findings to actual business harm.

### How often should an agentic AI system be threat modeled?

Review it at design, before production release, and whenever models, tools, permissions, data sources, or memory behavior change. A full quarterly review is reasonable for stable systems, while production controls and alerts should operate continuously. Risk-based review frequency is more useful than a fixed schedule for an agent that never changes.

### How can teams test for indirect prompt injection?

Place realistic but safe instructions in documents, web pages, emails, records, and other content the agent is likely to retrieve. Observe whether it follows those instructions, attempts unauthorized tool calls, or exposes protected data, and verify that infrastructure controls stop harm even if the model fails. Repeat tests across multiple runs because agent behavior is non-deterministic.

Canonical: https://agustin-otegui.com/knowledge/how_should_teams_threat_model_agentic_ai_systems_in_2026.php
Markdown: https://agustin-otegui.com/knowledge/how_should_teams_threat_model_agentic_ai_systems_in_2026.php/index.md