Most organizations have invested in AI security controls. Fewer have tested whether those controls would actually stop a failure.
Threat modeling is the discipline of asking that question before an incident forces the answer. It does not require a technical team or a specialized tool. It requires leaders who are willing to challenge their assumptions about how AI could be misused, how monitoring could miss it, and whether the organization could reconstruct what happened afterward.
This brief introduces three questions that structure that challenge. Each question targets a gap that conventional security reviews often leave open.
Could an Adversary Use Our AI Against Us?
AI systems introduce attack surfaces that do not appear in traditional vulnerability assessments. An adversary who cannot breach a network perimeter may still be able to manipulate an AI system through the inputs it accepts, the data it retrieves, or the instructions embedded in content it processes.
Prompt injection is the most widely documented example. An attacker embeds instructions in a document, email, or web page that the AI system retrieves and acts on as if those instructions came from an authorized user.2 The attack does not require a software vulnerability. It exploits the model's core function: following instructions.
MITRE ATLAS documents a broader taxonomy of AI-specific adversarial techniques, including model evasion, data poisoning, and the exploitation of retrieval pipelines.1 These techniques are not theoretical. They have been demonstrated against production systems in healthcare, financial services, and critical infrastructure.
The leadership question is not whether these techniques exist. It is whether the organization has tested its own systems against them. A threat model that does not include AI-specific attack paths will not identify AI-specific control gaps.
Useful questions for a leadership review include: Which AI systems in our environment accept external input? What happens if that input contains adversarial instructions? Which downstream actions could an AI system take before a human reviews the output? Have we tested those paths?
Would Our Monitoring See It?
Security monitoring is designed to detect known threat patterns. AI-enabled attacks often do not match those patterns.
Consider a scenario in which an AI agent with access to a clinical records system retrieves a patient file, summarizes it, and sends the summary to an external address. The agent uses valid credentials. The retrieval matches normal access patterns. The outbound message uses an approved communication channel. No individual action triggers an alert.
A conventional vulnerability scan might identify no relevant software flaw. A conventional access review might confirm that the agent's permissions are appropriate for its stated purpose. The failure is not in the controls. It is in the absence of a control that evaluates the sequence of actions and their combined effect.
This is the monitoring gap that AI introduces. Traditional security tools monitor for anomalous access. AI-enabled threats can operate entirely within normal access parameters while producing outcomes that no authorized user would have approved.
The NIST AI Risk Management Framework identifies the need for ongoing monitoring of AI system behavior, not just the underlying infrastructure.3 That distinction matters. Monitoring the infrastructure confirms that the system is running. Monitoring the behavior confirms that the system is doing what it is supposed to do.
Useful questions for a leadership review include: Does our monitoring evaluate what AI systems do, or only whether they are running? Can we detect a sequence of individually permitted actions that produces an unauthorized outcome? Who reviews AI system behavior, and how often?
Could We Reconstruct What Happened?
When an AI-related incident occurs, the investigation will require evidence that most organizations have not yet decided to collect.
A complete reconstruction of an AI-enabled event requires records of the initiating source, the interaction with the model, the tool actions the model requested, the identity under which those actions were executed, and the destination of any output or data transfer. Each element corresponds to a log that must be created, retained, and protected before the incident occurs.
NIST SP 800-154 addresses data-centric threat modeling and the importance of identifying evidence requirements as part of the design process, not after a failure.4 That principle applies directly to AI systems. The time to identify what evidence an investigation would need is before the system is deployed, not after an incident is reported.
Prepare the evidence before the investigation.
Forensic readiness for AI systems also requires attention to the integrity of the evidence itself. An AI agent that can write to a log can also be manipulated to alter one. Audit records must be protected from modification by the system they are monitoring.
ENISA's threat landscape for AI identifies the manipulation of training data and operational logs as a recognized attack category.5 Organizations that have not addressed log integrity for AI systems may find that the evidence they need has been altered or is unavailable.
Useful questions for a leadership review include: What logs does each AI system generate? Are those logs protected from modification by the system itself? Have we identified what a complete reconstruction of an AI-related incident would require? Do we retain that evidence long enough to support an investigation?
At Your Next Leadership Meeting
These three questions can structure a focused leadership discussion without requiring technical expertise from every participant. The goal is not to produce a complete threat model in a single session. It is to identify which questions the organization cannot yet answer and to assign responsibility for answering them.
Consider asking:
- Which AI systems in our environment accept external input, and have we tested what happens when that input is adversarial?
- Does our monitoring evaluate what AI systems do, or only whether they are running? Can we detect a sequence of individually permitted actions that produces an unauthorized outcome?
- If an AI-related incident occurred today, what evidence would an investigation require? Do we collect and protect that evidence now?
The Takeaway
Threat modeling is not a technical exercise reserved for security teams. It is a leadership discipline that asks whether the organization has genuinely challenged how its AI systems could be used against it.
The three questions in this brief — Can an adversary use our AI against us? Would our monitoring see it? Could we reconstruct what happened? — are not exhaustive. They are a starting point for a conversation that most leadership teams have not yet had.
Organizations that ask these questions before an incident are in a position to close the gaps. Organizations that wait for an incident to ask them will find that the gaps are larger than they expected.
Continue the Exploration
The questions in this brief are drawn from the threat modeling and AI governance frameworks developed across the vCSO Series™. Three volumes address these topics in depth for security and technology leaders.
Threat Modeling Healthcare AI (Volume 6) applies structured threat modeling to AI systems in clinical and operational environments. AI Security Governance for the Enterprise (Volume 8) provides a governance framework for managing AI risk at the organizational level. The Future of Threat Modeling (Volume 7) examines how threat modeling must evolve to address AI-enabled attack surfaces across sectors.
Sources
Ken Williams, CISSP, CFE, CPP
The vCSO Series™ | Leadership Brief
Issue 003 — October 2026