How Do You Build a RAG Threat Model That Catches Real AI Attacks in 2026?

What RAG Threat Modeling Actually Means

RAG threat modeling is the structured process of identifying assets, trust boundaries, attacker paths, and failure conditions in a retrieval-augmented generation system. It treats the model, document store, embedding service, orchestration layer, user interface, connected tools, and human operators as one socio-technical system rather than securing the language model in isolation. A useful model answers four questions: what can an attacker manipulate, what can they steal or disrupt, which routes are plausible, and what evidence would reveal misuse? For a B2B customer-signal inbox, those questions might concern customer conversations, account records, internal support procedures, retrieved policies, generated replies, and actions taken by an AI agent. The objective is not to predict every possible prompt injection. It is to prioritize scenarios that could cause material confidentiality, integrity, availability, regulatory, or business harm, then connect each scenario to measurable controls and tests.

Also worth reading: How Do Product and Support Teams Build an Effective B2B Feedback Scoring Model? · How Should a B2B Company Build a Customer Health Score Model in 2026? · How Can Customer Signal Management Turn B2B Inbox Noise Into Product and Support Action?

RAG introduces risk because generated answers depend on several components whose behavior is not determined by the base model alone. Retrieving the wrong document can disclose information or produce a false answer, while malicious text in a retrieved document may influence the model even when the user entered a harmless question. The same defense cannot simply block selected words because attackers can paraphrase requests, conceal instructions in images, distribute payloads across documents, or exploit ambiguous tool permissions. Threat modeling therefore combines conventional application security, data security, AI-specific abuse cases, and business-impact analysis. It should be repeated when the model, data sources, agent tools, or deployment architecture change, not treated as a one-time design document.

The Main Threats and Attack Paths

Prompt injection is usually the first concern, but it is only one member of a broader attack family. Direct injection attempts to make the model ignore its system instructions, reveal prompts, or perform unauthorized actions. Indirect injection places adversarial instructions in material that the system later retrieves, such as a customer email, uploaded PDF, public knowledge-base article, or compromised web page. Poisoning modifies the knowledge base so that selected chunks consistently distort answers, while evasion can manipulate embeddings or retrieval rankings so attackers reach their content more often. Data exfiltration may occur through generated text, logs, traces, embedded images, side channels, or an agent that is tricked into sending retrieved data elsewhere.

The most damaging path often crosses trust boundaries. For example, an external customer can submit a message containing instructions, a connector retrieves another customer’s account context, the model treats both records as equally trusted evidence, and an agent drafts or sends a response. Traditional access control may permit this workflow because each service used a valid API key, even though the end-to-end action was unauthorized. Attackers can also exploit insecure agent tool use, excessive permissions, weak tenant isolation, over-broad searches, exposed vector stores, vulnerable parsers, or model and gateway logs containing sensitive context. Denial-of-service risks include crafted inputs that trigger expensive retrieval, long agent loops, repeated tool calls, and pipelines that process oversized documents without bounds.

## How to Build the Threat Model Step by Step

Begin with an architecture and data-flow inventory. Record where documents originate, who can modify them, how they are parsed and chunked, which embedding model is used, what metadata accompanies each chunk, which filters run before retrieval, and which services receive model output. Draw every trust boundary, including browsers, APIs, identity providers, vector databases, model endpoints, observability platforms, ticketing systems, and outbound email or messaging tools. Assign an owner and sensitivity classification to each asset; RAG evaluations often degrade when teams discuss “the data” without distinguishing public content, internal procedures, regulated records, authentication data, and customer-generated text. A current diagram should also show whether authorization happens before retrieval, after retrieval, before tool execution, or only in the final application.

Next, define realistic abuse cases and attacker capabilities. Model insiders, ordinary authenticated users, malicious document authors, compromised integrations, and external attackers separately because their access and motivation differ. Describe at least confidentiality, integrity, availability, misuse, and compliance scenarios, then include agent-specific paths such as unauthorized emails, record changes, code execution, or purchases. A practical scoring method can assign likelihood and impact from 1 to 5, producing scores from 1 to 25; risks scoring 15 or higher can enter an accelerated review, while 10–14 require scheduled treatment and scores below 10 can be monitored. These thresholds are planning conventions, not universal industry standards, and should be adjusted to legal obligations, expected loss, and customer commitments.

Convert each high-risk scenario into an attack tree. For an exfiltration path, consider direct requests, retrieved instructions, encoded content, metadata leakage, tool misuse, and denial of service as distinct branches. Then define preventive controls, detective controls, and recovery controls rather than writing vague goals such as “make the model safe.” For example, prevent broad retrieval with tenant filters and per-document authorization; detect suspicious instruction patterns and anomalous tool calls; and recover through token rotation, vector-store quarantine, log preservation, and notification. Test whether controls work together, because a sound prompt filter offers little protection if an agent retains unrestricted write access after generation.

Comparing RAG Security Approaches

There is no single product category that replaces a complete threat model. Gateways and AI firewalls can add useful inspection and policy enforcement, but they operate near the model boundary and may not understand every business authorization rule. Retrieval platforms can improve filtering and provenance, yet they do not determine whether generated text is truthful or whether a tool action is safe. Agent platforms can provide identity, approval, and tracing, although a platform feature is ineffective when teams configure permissions too broadly. Conventional application security remains necessary for injection flaws, insecure APIs, weak credentials, vulnerable parsers, and tenant-isolation defects, while data-governance tools help classify, redact, and govern information independently of model behavior.

Fine-tuning and carefully written prompts can improve task behavior, but they do not establish reliable prompt-injection resistance. Model training may reduce certain failures while leaving attackers free to use new phrasings or malicious retrieved context. A defense-in-depth approach is generally stronger than relying on one detector, especially because no commercial filter claims perfect prevention across models, languages, and attack techniques. The right comparison is based on coverage, integration effort, false-positive rate, latency, data residency, audit evidence, and failure behavior rather than a generic security score.

## Controls That Survive Contact with Attackers

The strongest architecture treats all retrieved content as untrusted input, even when it comes from an internal database. Separate system instructions from source material, tell the model to ignore instructions embedded in documents, and still enforce permissions outside the model. Apply document-level authorization during retrieval rather than asking the model not to reveal restricted data. Scope service accounts narrowly, use short-lived credentials where possible, and require explicit authorization immediately before any tool that writes data, sends communications, changes permissions, or executes code. For consequential actions, use a human approval gate that displays the evidence, intended action, destination, and affected account.

Monitoring should cover behavior rather than only known prohibited phrases. Useful signals include unusual document-modification rates, cross-tenant lookup attempts, repeated retrieval failures, sudden changes in token consumption, long agent loops, novel tool destinations, and instructions that conflict with approved workflows. A practical initial threshold can be more than 10 failed authorization events from one identity in 15 minutes or more than 3 high-impact tool calls from one conversation without approval; organizations should tune these to normal operations. Trace retention should be sufficient for investigation but bounded by legal and contractual requirements, with secrets removed before logs reach general analytics systems. As a privacy baseline, review whether raw prompts and retrieved chunks need to remain for 30 days rather than defaulting to indefinite storage, while preserving longer evidence only for a justified incident window.

Evaluation must include adversarial retrieval, not merely direct user prompts. Maintain a versioned test set of benign questions, authorized cross-role questions, direct and indirect injection attempts, poisoned documents, multilingual paraphrases, data-exfiltration requests, and unsafe tool plans. Measure unauthorized-answer rate, sensitive-data exposure, policy violations, tool-action rejection, task success, latency, and false positives. A useful release gate might require zero known cross-tenant disclosures, at least 99% compliance with approved high-risk tool policies, and no statistically meaningful regression in core answer quality, although exact targets should reflect risk tolerance. Track the model, prompt, retriever, embedding model, document corpus, and tool permissions because results from one version do not automatically transfer to another.

Common Mistakes and Why Threat Models Fail

The most common error is treating prompt injection as a string-filtering problem. Attackers do not need to repeat a banned phrase when they can issue a legitimate-seeming task, distribute instructions across several chunks, hide them in structured content, or exploit the model’s role. Another mistake is equating retrieved knowledge with trusted instructions, which creates a data-plane and control-plane confusion. Teams also often test the model while omitting the full application, so they miss insecure document ingestion, missing tenant predicates, exposed object storage, excessive tool scopes, or secrets sent to third-party endpoints.

Security controls can also fail through unmeasured automation. A gateway that flags many harmless customer phrases becomes business noise, while one tuned too narrowly creates false confidence. Excessive human approval can train users to click through warnings, and a model tasked with evaluating its own output is not an independent control. Teams frequently omit attackers who already hold valid accounts, because threat modeling incorrectly starts with anonymous internet traffic. Finally, RAG projects change quickly as sources, models, and connectors evolve, but risk registers remain frozen after an initial review.

To avoid these failures, assign control owners and evidence requirements. Security may own monitoring and red-team governance, data owners may approve classification and retention, and application teams must maintain authorization and tool policies. Record assumptions explicitly, including whether a connector is trusted and whether metadata is cryptographically trustworthy. Use incident exercises that simulate compromised support articles, malicious customer messages, and unintended tool execution, then measure time to detection, containment, credential rotation, and customer notification. A mature model is revised after incidents, material architecture changes, new agent capabilities, and relevant regulatory or contractual shifts; an annual review is a floor rather than a sufficient cadence for fast-changing systems.

When to Act and What It May Cost

Organizations should act before a RAG system handles production customer data or can take external actions. Early action allows teams to correct document permissions, tenant filters, data flows, and tool scopes before stored prompts, integrations, and customer expectations make redesign expensive. It is also important to act when adding new retrieval sources, allowing customers to upload content, connecting write-capable tools, changing the embedding model, or expanding from internal drafting to automated replies. Even a read-only assistant can create serious risks, but write-capable agents increase both impact and urgency because one manipulated generation can cause many downstream changes.

Cost varies more by architecture and governance maturity than by document count. A small internal pilot may require several weeks of engineering, security, legal, and product review, while a regulated enterprise deployment may need months of identity redesign, procurement, red-team evaluation, monitoring, and evidence collection. Infrastructure costs can include vector storage, embedding calls, model inference, retrieval, observability, and security gateways; high-volume systems should calculate cost per conversation and impose token, chunk, tool-call, and time budgets. For example, limits of 20 retrieved chunks, 8,000 context tokens, 10 tool calls, and 60 seconds per workflow can stop runaway behavior, but they must be tested against legitimate complex tasks rather than applied blindly.

By September 2026, teams should expect deployment patterns to remain heterogeneous: managed models, private endpoints, hybrid gateways, multiple retrieval stores, and agent frameworks are all common. The durable investment is therefore in data classification, identity, authorization, test sets, observability, and rollback mechanisms. Cloud security certification can provide a useful control framework, but it does not certify the correctness or safety of a particular RAG workflow. For B2B customer-signal products, the most relevant validation includes tenant separation, evidence provenance, prompt and retrieved-content privacy, outbound-message approval, and measurable recovery from a manipulated source.

A Minimum Decision Framework

Start by deciding whether the system only retrieves and drafts, or whether it can send, modify, purchase, delete, or execute. Then identify whether documents are fully internal, customer-controlled, or publicly writable, because each source demands different ingestion controls. Confirm that authorization is enforced below the model and that every tool uses least privilege. Ensure the organization can detect abnormal retrieval and tool activity, quarantine suspect documents, revoke credentials, restore trusted content, and explain which records and outputs were affected. Finally, require a documented release gate based on adversarial testing and business-risk thresholds.

This framework is deliberately more demanding than asking whether a vendor offers an “AI firewall.” It recognizes that RAG risk emerges from interactions among data, software, models, and people. No method provides absolute safety, and no percentage of prompt detection guarantees that an entire system is secure. The defensible claim is narrower: a documented threat model, layered controls, continuous evaluation, and rehearsed response reduce the likelihood and impact of foreseeable attacks, while making residual risk visible to decision-makers.