Guide

What Is RAG?

A plain-English guide to retrieval-augmented generation: what RAG is, how a RAG pipeline works step by step, RAG vs fine-tuning vs long context, the main types of RAG including GraphRAG and agentic RAG, what the evidence says about hallucinations, the security risks, how to evaluate a RAG system, and the managed RAG services from OpenAI, Google, Amazon and Microsoft.

October 11, 2026 · The AI Rankings

Quick answer: RAG, or retrieval-augmented generation, is a technique in which an AI system first searches a collection of documents for passages relevant to a question and then passes those passages to a large language model, so the model answers from the retrieved text rather than from memory alone. RAG is the standard way to make a general model such as Claude, ChatGPT or Gemini answer from private, recent or specialist data without retraining it, and it lets the answer cite its sources. The term comes from a 2020 paper by researchers at Facebook AI Research, University College London and New York University, and Menlo Ventures’ enterprise survey found RAG was the most-used enterprise AI design pattern, at 51% adoption. The one caveat: RAG reduces hallucination but does not remove it, and Stanford researchers found commercial legal research tools built on retrieval still hallucinated on 17% to 33% of queries.

This page is the architecture explainer: what RAG is, how each stage works and when to use it. It does not rank products. For the components, see best vector databases for where the retrievable text is stored and best embedding models for the models that turn text into searchable vectors. For systems that retrieve as one step in a longer plan, see what is agentic AI. For one-line definitions of the surrounding vocabulary, see the AI glossary.

What is RAG?

Retrieval-augmented generation is an AI architecture that combines a search step with a text-generation step: a retriever finds relevant passages, and a large language model writes the answer using those passages as its evidence.

The name breaks down into its three parts:

• Retrieval is the search step. The system looks through a knowledge base, such as company documents, a product manual, a legal database or the web, and pulls out the passages most likely to answer the question.
• Augmented means the retrieved passages are added to the prompt. The model receives the user’s question plus the evidence.
• Generation is the language model writing the answer, ideally quoting or citing the passages it used.

The simplest way to picture it is an open-book exam. A model without RAG answers from what it memorised during training. A model with RAG is handed the relevant pages of the textbook before it answers.

What RAG is not

RAG does not change the model. The model’s weights, the learned numbers that encode what it knows, stay exactly the same. RAG changes only what the model is shown at the moment it answers. That is the core difference between RAG and fine-tuning, which retrains the model itself (see RAG vs fine-tuning vs long context below).

RAG is also not a product. It is a pattern that many products use. ChatGPT Enterprise file uploads, Claude Projects, Google’s NotebookLM, Microsoft 365 Copilot and AI search engines such as Perplexity all retrieve before they generate.

Who named it

Patrick Lewis was the lead author of the 2020 paper that coined the term, and he has said the team never loved the name. “We definitely would have put more thought into the name had we known our work would become so widespread,” Lewis told NVIDIA’s blog.

How does RAG work?

A RAG system works in two phases: an indexing phase that prepares the documents once, and a query phase that runs every time someone asks a question.

1. Ingest and parse

The first step is getting documents into plain text. PDFs, slide decks, spreadsheets, web pages and scanned images all need converting, and tables and images are where most parsing quality is lost. Managed services handle this step for you: OpenAI’s file search accepts PDF, Word, PowerPoint, Markdown, HTML, JSON and common code files.

2. Chunk

Chunking splits each document into passages small enough to retrieve precisely. A chunk is usually a few hundred tokens, roughly a few paragraphs. Chunks often overlap slightly so a sentence cut at a boundary still appears whole in one of them.

Published defaults differ by vendor:

• OpenAI file search uses 800-token chunks with a 400-token overlap by default, according to its official Java SDK documentation, and allows chunk sizes from 100 to 4,096 tokens.
• Microsoft recommends starting with 512-token chunks and a 25% (128-token) overlap in Azure AI Search.

Chunk size is a trade-off. Small chunks retrieve precisely but lose surrounding context. Large chunks keep context but dilute the match and use more of the model’s input budget.

3. Embed

An embedding model converts each chunk into a vector, a fixed-length list of numbers positioned so that passages with similar meaning sit close together. This is what lets a search for “how do I cancel” find a document that only says “terminating your subscription”. The choice of embedding model sets the ceiling on retrieval quality, and changing it usually means re-embedding the whole corpus. Our best embedding models page compares the current options on price, licence and measured retrieval quality.

4. Store

The vectors and their original text are stored in an index that can find the nearest vectors quickly. That index can be a dedicated vector database or a vector extension to a database you already run. Our best vector databases page covers the trade-off, and its main finding is that most teams that already run PostgreSQL can start with the free pgvector extension rather than buying a separate system.

5. Retrieve

At question time, the system embeds the user’s question with the same embedding model and fetches the chunks whose vectors are closest. This is called dense or semantic retrieval.

Most production systems now combine it with keyword search, usually the BM25 algorithm, in what is called hybrid search. Keyword search catches exact terms such as product codes, names and error messages that semantic search can miss. An independent study of text-and-table document retrieval across 23,088 queries found that a two-stage pipeline combining hybrid retrieval with a reranker performed best.

6. Rerank

A reranker is a second model that re-scores the retrieved candidates against the question and keeps only the most relevant. Retrieval casts a wide net; reranking picks the best few for the model to read.

• In Anthropic’s contextual retrieval tests, retrieving 150 chunks, reranking them and passing the top 20 to the model cut the retrieval failure rate from 5.7% to 1.9%, a 67% reduction, when combined with contextual embeddings and BM25.
• In the same retrieval study, hybrid retrieval plus reranking was the best configuration tested.
• Reranking is not free quality: an independent evaluation across about 25,000 question-answer pairs found “only marginal performance improvements accompanied by higher response latency”. Our embedding models page explains when rerankers help and when they hurt.

7. Generate

The final step puts the question and the retrieved chunks into the prompt and asks the model to answer from them. A well-built system instructs the model to use only the supplied passages, to cite which passage supports each claim, and to say so when the passages do not contain the answer.

Model providers now support this directly. Anthropic’s Citations feature returns the exact text each claim is based on, and its search result content blocks let developers pass their own retrieval results to Claude and receive citations back. OpenAI’s file search tool returns file citations with its answers.

Why RAG exists: the problems it solves

RAG exists because a language model on its own has four limits that matter in real use.

RAG is also the cheapest of the three ways to add knowledge to a model. Updating a RAG system means adding or deleting documents in an index. Updating a fine-tuned model means training it again.

RAG vs fine-tuning vs long context

There are three ways to get a model to use knowledge it was not trained on: retrieve the relevant parts (RAG), retrain the model (fine-tuning), or paste everything into the prompt (long context).

The practical rule, which the AI glossary also states: fine-tune to change how a model behaves, and use RAG to supply facts. Menlo Ventures’ survey of 600 US enterprise IT decision-makers found 51% used RAG, up from 31% the year before, while only 9% of production models were fine-tuned.

When you do not need RAG

If the whole knowledge base fits comfortably in the model’s context window, you may not need retrieval at all. Anthropic advises that a knowledge base under about 200,000 tokens, roughly 500 pages, can simply be placed in the prompt, with prompt caching to cut the repeat cost.

The ceiling for that approach has risen. The leading models now accept about 1 million tokens:

RAG vs long context: what the studies show

A bigger context window is a capacity, not a guarantee that the model uses everything in it. Four studies set out the trade-off:

• Position matters. The Lost in the Middle study (Liu et al.) found models use information at the start and end of a long input far better than information in the middle. With 20 documents, GPT-3.5-Turbo scored 75.8% when the answer was in the first document and 53.8% when it was in the middle, below the 56.1% it scored with no documents at all.
• Longer inputs degrade accuracy. Chroma’s Context Rot report tested 18 models and found that “model performance consistently degrades with increasing input length”, even on simple tasks.
• More retrieved context helps, up to a point. A Databricks study of more than 2,000 experiments across 13 models found retrieval recall rose with longer context, but answer quality fell for some models past a threshold: Llama 3.1 405B declined after 32,000 tokens and GPT-4-0125-preview after 64,000.
• Routing between the two saves money. A Google DeepMind and University of Michigan study found long context outperformed RAG on average when cost was no object, but the two gave identical answers on 63% of queries. Its Self-Route method, which uses RAG first and falls back to long context only when needed, cut cost by 65% for Gemini 1.5 Pro and 39% for GPT-4o.

What the difference costs

The cost gap is easy to calculate from published prices. The worked example below is our arithmetic, using Claude Sonnet 5.5’s list prices and an 800,000-token knowledge base, roughly 2,000 pages by Anthropic’s 500-pages-per-200,000-tokens rule of thumb.

Output costs are the same in all three cases. The RAG row excludes retrieval costs, which are small by comparison: OpenAI charges $2.50 per 1,000 file-search calls. Caching closes most of the gap, but a cache expires and must be rewritten at a premium, and a knowledge base larger than the context window cannot be cached whole. At the scale of a company wiki or a legal database, retrieval is the only option.

Types of RAG

“RAG” now covers a family of designs. The table runs from the simplest to the most complex.

Contextual retrieval

Contextual retrieval fixes a basic chunking problem: a chunk that says “revenue grew 3%” means little without knowing which company and which quarter. Anthropic’s method uses a language model to write a short explanation of where each chunk sits in its document and prepends it before embedding. Anthropic measured retrieval failures falling 35% with contextual embeddings alone and 49% with contextual BM25 added, at a one-time cost of $1.02 per million document tokens using prompt caching. These are Anthropic’s own measurements.

GraphRAG

GraphRAG uses a language model to extract entities and relationships from every document, builds a knowledge graph, and summarises clusters of related entities in advance. Microsoft Research published the method and released it as open source. Its strength is “global” questions that no single passage answers, such as “what are the main themes in these 5,000 support tickets?”

GraphRAG’s weakness is cost and consistency. Building the graph requires running a language model over the entire corpus. Microsoft’s follow-up, LazyGraphRAG, claims indexing costs “identical to vector RAG and 0.1% of the costs of full GraphRAG”; that is Microsoft’s claim. Independent results are mixed:

• A systematic evaluation by Han et al. found plain RAG did better on single-hop, detail-oriented questions, GraphRAG did better on multi-hop reasoning, and combining them gave consistent gains.
• The GraphRAG-Bench study found that “GraphRAG frequently underperforms vanilla RAG on many real-world tasks”.

The independent evidence supports GraphRAG for cross-document and multi-hop questions, not as a default replacement for standard RAG.

Agentic RAG

Agentic RAG puts an AI agent in charge of retrieval. Instead of one search followed by one answer, the agent decides whether to search at all, rewrites the query, searches several sources, reads the results and searches again if the evidence is thin. Our what is agentic AI guide explains the agent loop this builds on, and best AI agent frameworks covers the libraries used to build it.

Research systems introduced the core ideas:

• Self-RAG trains a model to decide when to retrieve and to critique the retrieved passages and its own answer.
• Corrective RAG scores retrieved documents and falls back to web search when retrieval looks poor.
• HyDE has the model write a hypothetical answer first and uses that answer to search for real documents.

The managed platforms have since productised the pattern. Amazon Bedrock’s Managed Knowledge Base includes agentic retrieval, and Microsoft’s Azure AI Search agentic retrieval uses a language model to plan several subqueries, run them in parallel and merge the results.

Is RAG dead?

No. The single vector lookup is becoming one tool among several, not the whole system. The “RAG is dead” argument comes mainly from AI coding agents:

• Boris Cherny, who created Anthropic’s Claude Code, said the team replaced its vector database with agentic search, in which the agent runs file-search commands itself. “It outperformed everything else by a lot,” he said, while describing the evidence as internal “vibes” (OfficeChai).
• Cursor reported the opposite result for its own coding agent: adding semantic search improved answer accuracy by 12.5% on average, ranging from 6.5% to 23.5% depending on the model. That is Cursor’s own measurement.

The positions converge on the same architecture: give the agent a semantic search tool alongside keyword and file search, and let it choose. Our embedding models page sets out the full evidence on both sides. For documents rather than code, the retrieval study above found hybrid retrieval with reranking performed best.

Does RAG stop hallucinations?

No. RAG reduces hallucinations but does not stop them. A model can still miss the right passage, misread a correct one, blend two sources, or answer beyond what the passages say.

The pattern across these studies is consistent. RAG lifts accuracy substantially on knowledge-heavy questions, and a well-built RAG system still gets a meaningful share of answers wrong. No high-quality study yet gives a single pooled figure for how much RAG cuts hallucination.

The legal result is the most cited warning. The vendors of those tools had marketed them as reducing or avoiding hallucination, and the Stanford team’s paper found they still produced wrong or unsupported answers on between one in six and one in three queries.

Where RAG fails

Most RAG failures are retrieval failures, not model failures. If the right passage never reaches the model, no model can answer correctly.

RAG security risks

RAG creates security risks that a model alone does not have, because it feeds outside content straight into the model’s prompt. The OWASP Top 10 for LLM Applications 2025 added a dedicated entry, LLM08:2025 Vector and Embedding Weaknesses, in response to requests for guidance on securing RAG.

Indirect prompt injection

Indirect prompt injection is the biggest RAG risk. An attacker plants instructions inside a document, email or message that the system later retrieves, and the model follows them as if they came from the user. OWASP’s LLM01:2025 Prompt Injection entry lists a poisoned document in a RAG application’s repository as a named attack scenario.

Two incidents show the pattern in production:

• Slack AI. Security firm PromptArmor showed that an attacker could post instructions in a public Slack channel that caused Slack AI to leak a victim’s private data through a link. Slack described the underlying search behaviour as intended.
• Microsoft 365 Copilot “EchoLeak”. Aim Security disclosed a zero-click flaw, CVE-2025-32711, in which a crafted email was pulled in by Copilot’s retrieval engine and caused it to leak data. Microsoft patched it, and The Hacker News reported a CVSS severity score of 9.3 and no evidence of exploitation.

Knowledge-base poisoning

An attacker who can add documents to the knowledge base can steer answers. The PoisonedRAG study achieved a 90% attack success rate by injecting five malicious texts per target question into a knowledge base of millions of texts.

Embeddings leak text

Embeddings are not anonymised data. The vec2text study recovered 92% of 32-token text inputs exactly from their embeddings. A vector database should be secured like the documents it was built from.

Permissions

A RAG system must only retrieve what the person asking is allowed to see. OWASP recommends “fine-grained access controls and permission-aware vector and embedding stores”. Microsoft states that Microsoft 365 Copilot “only surfaces organizational data to which individual users have at least view permissions”. A system that indexes everything into one shared store and filters afterwards is the common way this goes wrong.

How to evaluate a RAG system

A RAG system should be tested at both stages: did retrieval find the right passages, and did the model answer faithfully from them? Three open frameworks define the standard metrics:

The practical method is to build a test set of real questions with known correct answers and source passages, then measure retrieval recall and answer faithfulness on every change to chunking, embedding model, retrieval settings or prompt. Tools for tracing and scoring RAG pipelines in production are compared on our best LLM observability tools page.

Managed RAG services

Every major AI platform now sells RAG as a managed service that handles parsing, chunking, embedding, storage and retrieval. This table is a reference, not a ranking.

RAG in apps you already use

Many consumer and workplace AI apps run RAG behind the scenes when you upload files:

• ChatGPT Enterprise places up to about 110,000 tokens of an uploaded file directly in context and sends the rest to a private search index using hybrid keyword and semantic search, according to OpenAI’s help centre.
• Claude Projects switch to a RAG mode automatically when project knowledge nears the context limit, allowing up to 10 times more content, on Pro, Max, Team and Enterprise plans, according to Anthropic’s help centre.
• NotebookLM from Google answers only from the sources a user uploads, with each source up to 500,000 words or 200 MB, according to Google’s help centre.
• Microsoft 365 Copilot grounds answers in a user’s emails, files and chats through its Semantic Index, limited to content that user can already view, according to Microsoft’s documentation.
• AI search engines such as Perplexity apply the same pattern to the open web: search first, then answer with citations. Our best AI search engines page compares them.

How to build a RAG system

The evidence above points to a sensible default for a first build. This is a starting order, not a fixed recipe.

1. Check whether you need retrieval at all. If the knowledge base is under about 200,000 tokens, put it in the prompt with caching and measure that first.
2. Decide managed or self-built. A managed service from the table above is the fastest route; a self-built pipeline gives control over every stage. Frameworks for self-built pipelines are covered in best AI agent frameworks.
3. Parse carefully. Check how your tables, PDFs and scans come out as text before tuning anything else.
4. Chunk at a few hundred tokens with overlap. Start from a vendor default such as Microsoft’s 512 tokens with 25% overlap, then test.
5. Pick an embedding model on your own data. Use best embedding models to shortlist, then test two or three on real questions.
6. Store vectors where your data already lives if you can. See best vector databases for when pgvector is enough and when a dedicated system pays off.
7. Use hybrid retrieval from the start. Combine vector and keyword search.
8. Add a reranker and measure it. Retrieve a deep candidate pool, rerank and pass the best 5 to 20 chunks.
9. Require citations and an “I don’t know”. Instruct the model to answer only from the passages and to say when they do not contain the answer. Any model on best LLM APIs can do this; for a private deployment, see best open-source LLMs and best local LLMs.
10. Enforce permissions at retrieval time and evaluate continuously. Filter by the user’s access before results reach the model, and track retrieval recall and faithfulness on a fixed test set.

A short history of RAG

The original 2020 system paired a retriever from Facebook’s Dense Passage Retrieval (DPR) project with BART-large, a generator of about 400 million parameters, over a Wikipedia dump split into 21 million 100-word passages (paper). The architecture has changed little in outline since then; what changed is that the generator became a frontier LLM and the retriever became hybrid, reranked and, increasingly, agent-driven.

Related pages

• Best vector databases: where RAG systems store and search their vectors
• Best embedding models: the models that make semantic retrieval work, plus rerankers
• What is agentic AI: the agent loop behind agentic RAG
• Best AI agent frameworks: libraries for building RAG and agent pipelines
• Best LLM observability tools: tracing and evaluating RAG in production
• Best AI search engines: RAG applied to the open web
• AI glossary: one-line definitions of embedding, vector database, grounding and more

Frequently asked questions

What is RAG in simple terms?

RAG, or retrieval-augmented generation, is a way of making an AI model answer from a set of documents rather than from memory. The system first searches the documents for passages relevant to the question, then gives those passages to the language model along with the question, and the model writes its answer from that evidence, often with citations.

What does RAG stand for in AI?

RAG stands for retrieval-augmented generation. The term was coined in a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University.

How does RAG work?

RAG works in two phases. In the indexing phase, documents are split into chunks, each chunk is converted into a vector by an embedding model, and the vectors are stored in a searchable index such as a vector database. In the query phase, the question is converted into a vector, the closest chunks are retrieved and often reranked, and a language model answers using those chunks as its evidence.

What is the difference between RAG and fine-tuning?

RAG changes what a model is shown when it answers, while fine-tuning changes the model itself by retraining its weights. RAG is better for supplying facts that change or that must be cited, because updating it means updating documents. Fine-tuning is better for changing a model’s behaviour, tone or format. Menlo Ventures’ enterprise survey found 51% of enterprises used RAG, while only 9% of production models were fine-tuned.

Is RAG still needed with 1 million token context windows?

RAG is still needed for most large knowledge bases, but not for small ones. Anthropic advises that a knowledge base under about 200,000 tokens, roughly 500 pages, can simply go in the prompt. Above that, RAG is far cheaper per question, works on corpora larger than any context window, and avoids the accuracy loss that studies such as Lost in the Middle and Chroma’s Context Rot report measured on very long inputs.

Does RAG eliminate hallucinations?

No. RAG reduces hallucinations but does not eliminate them. Meta’s CRAG benchmark found the best industry RAG systems answered only 63% of questions without hallucination, and Stanford researchers found commercial legal research tools built on retrieval hallucinated on 17% to 33% of queries. A model can miss the right passage, misread it, or answer beyond it.

What is a vector database in RAG?

A vector database stores the numerical vectors that represent each chunk of text and finds the vectors closest to a question’s vector quickly. It is the storage and search layer of a RAG system. Many teams use a vector extension such as pgvector inside PostgreSQL instead of a separate database; our best vector databases page compares the options.

What is an embedding in RAG?

An embedding is a list of numbers that represents the meaning of a piece of text, produced by an embedding model. Texts with similar meanings get similar embeddings, which is what lets RAG find relevant passages even when they use different words from the question. Our best embedding models page compares the current models on price and measured retrieval quality.

What is GraphRAG?

GraphRAG is a variant of RAG, published by Microsoft Research, that uses a language model to build a knowledge graph of the entities and relationships in a corpus and retrieves over that graph. It is strongest on questions about themes across many documents and on multi-hop questions, but it costs more to build, and an independent benchmark found it frequently underperforms standard RAG on many real-world tasks.

What is agentic RAG?

Agentic RAG is RAG in which an AI agent controls retrieval: it decides whether to search, rewrites queries, searches several sources and searches again if the evidence is weak, rather than running one fixed search. Amazon Bedrock and Azure AI Search both now offer agentic retrieval as a managed feature.

Is RAG dead?

No. Some AI coding tools, including Anthropic’s Claude Code, replaced vector search with agentic file search, but Cursor reported that adding semantic search improved its coding agent’s accuracy by 12.5% on average. The common position is to give an AI agent semantic search as one tool alongside keyword and file search, which is an evolution of RAG rather than its end.

What are the security risks of RAG?

The main RAG security risks are indirect prompt injection, where instructions hidden in a retrieved document hijack the model; knowledge-base poisoning, where planted documents steer answers; leakage of data a user should not see; and leakage of text from stored embeddings. OWASP’s Top 10 for LLM Applications lists vector and embedding weaknesses as entry LLM08.

Does ChatGPT use RAG?

Yes, for large files. OpenAI’s help centre states that ChatGPT Enterprise sends the part of a large uploaded file beyond about 110,000 tokens to a private search index using hybrid keyword and semantic search. Developers can build RAG on OpenAI’s models with the file search tool in the Responses API, priced at $2.50 per 1,000 calls.

How much does RAG cost?

RAG costs depend mostly on the language model’s input price and how many chunks are passed per question. At Claude Sonnet 5.5’s $2 per million input tokens, passing 20 chunks of 800 tokens costs about $0.03 per question in input, against $1.60 for pasting an 800,000-token knowledge base into the prompt uncached. Managed retrieval adds a small charge, such as OpenAI’s $2.50 per 1,000 file-search calls or Amazon Bedrock’s $1.00 per 1,000 retrieve calls.

← All guides