Practical guide · Ravindra Annam

How to Threat Model an AI Agent

A workflow review for teams building agents that read information, use tools, remember context, and delegate work.

By Ravindra AnnamUpdated October 11, 2026All resources →
Start with a real task

Follow the action and its authority.

An agent threat model begins with the task a system is allowed to complete. Describe the result in business terms, then identify the actions that can change data, move information, spend money, or grant access. Review the whole workflow with the people who own those systems.

1. Define the permitted outcome

Write one sentence that names the actor, allowed operation, target, and limit. For example: an invoice assistant may prepare a payment proposal for an approved vendor; releasing the payment requires a separate approval. Decide how the system handles uncertainty or an unavailable approver.

2. Draw the trust boundaries

Inventory user requests, retrieved documents, tool descriptions, tool results, durable memory, and other agents. Mark where each item came from and who can change it. Content arriving through these paths should not acquire permission merely because the model includes it in a plan.

3. Trace authority through the workflow

For every consequential call, record the acting identity, credential, permitted resource, policy decision, and delegation chain. Ask which component enforces the limit. A model-generated explanation of permission is insufficient evidence that the receiving service checked it.

Example: a changed recipient

A retrieved invoice contains an instruction to send payment to a different account. The review should test whether that text can alter the proposal, whether the service constrains the recipient, and whether changing the final parameters invalidates an earlier approval.

4. Turn abuse paths into test cases

  • Place conflicting instructions in a document or tool response.
  • Try an action outside the credential's resource scope.
  • Revoke a grant before a queued or delegated action executes.
  • Attempt to store an instruction that changes a later workflow.
  • Change an approved target before execution.

For each case, record the expected result, the enforcement point, and the evidence that demonstrates the result. Keep a named owner for gaps and unresolved risk.

5. Review changes and recovery

Revisit the model when tools, permissions, memory behavior, or integrations change. Include a containment drill: stopping the model alone may leave scheduled work or downstream credentials active. Define what must be checked before execution resumes.

Use the worksheet to make the discussion reproducible. It is a starting structure for a team review; the scope and test depth depend on the system.