Retrieval-augmented generation (RAG)
Last verified: June 2026· definition
Retrieval-augmented generation (RAG) connects a language model to an external knowledge source. At query time, the system embeds the question, retrieves the most relevant chunks from a vector store or search index, and includes them in the prompt so the model grounds its answer in real data.
RAG is the default way to give a model knowledge it wasn’t trained on — internal docs, current facts, customer-specific data — without retraining. Update the index and the answers update; no model training required.
Good RAG lives and dies on retrieval quality and citation. Poor chunking or a weak retriever produces confident, wrong answers; strong retrieval with citations dramatically reduces hallucination.
What the term does and does not cover
RAG names a pattern, not a product or a specific architecture: retrieve relevant material at query time, put it in the prompt, generate an answer from it. Anything following that shape qualifies, whether retrieval is vector search, keyword search, a database query, or an API call. This is worth stating because the term is often used as though it implied embeddings and a vector database specifically, which has led teams to adopt infrastructure they did not need for a retrieval problem a plain query would have solved.
Why it displaced fine-tuning for knowledge
Before RAG became standard practice, giving a model information it lacked meant training it in. That is expensive, slow to update, and produces knowledge the model recalls unreliably and cannot cite. Retrieval changed the default because the knowledge stays outside the model: update the index and answers update immediately, remove a document and it stops being used, and every claim can point at a source. Fine-tuning remains the right tool for shaping behaviour and format — the split is knowledge versus behaviour.
The vocabulary around it
Discussion of RAG uses a fairly stable set of terms worth knowing: chunking is splitting documents into retrievable units; embedding turns those units into vectors; the retriever finds candidates; a reranker reorders them by relevance to the query; grounding means the answer is derived from retrieved material; and citation is the link back to the passage supporting a claim. Agentic RAG describes systems where the model decides what to retrieve and may search repeatedly, rather than retrieving once before generating.
Where the term gets stretched
Usage has broadened to the point where it is worth reading carefully. Some systems described as RAG retrieve nothing at query time and simply have a large static prompt. Others perform retrieval but ignore the results in generation, which produces the appearance of grounding without the substance. The useful test is whether a claim in an answer can be traced to a passage that was retrieved for that specific query — if not, the label is doing more work than the system is.
Common misconceptions
MythRAG stops the model hallucinating.
RealityIt reduces it by grounding answers in real passages, but the model can still misread retrieved text or fill gaps when retrieval returns nothing useful. Citations are what let you verify rather than trust.
MythRAG is an alternative to fine-tuning.
RealityThey address different problems. RAG supplies knowledge the model lacks; fine-tuning shapes behaviour, format, or style. Needing current or private facts is a RAG problem, and fine-tuning will not solve it.
MythBigger context windows make RAG unnecessary.
RealityLoading everything is more expensive, slower, and often less accurate than retrieving the relevant part. Large windows relax a constraint; they do not remove the value of selecting well.
Frequently asked questions
Why does our RAG system give confident wrong answers?
Almost always because retrieval returned chunks that are topically related but do not contain the answer, and the model generated from them anyway. Start by logging what was retrieved for failing queries — the cause is usually obvious on inspection. The common fixes are chunking on structure so facts stay with their context, adding keyword search alongside vector search, adding a reranker, and giving the model explicit permission to say the answer is not in the source material.
How many chunks should I retrieve?
Enough to contain the answer and few enough that it is not buried. More retrieved context costs more, adds latency, and dilutes attention, so retrieving a large number to be safe frequently makes answers worse rather than better. Measure it on your own queries rather than adopting a default, since the right number varies with chunk size and how densely your corpus states facts.
When should I use RAG instead of putting everything in the prompt?
Whenever the corpus is larger than a small, stable set of documents, or changes at all. RAG lets you update the index without touching prompts or retraining anything, keeps per-request cost proportional to what is actually needed, and scales to corpora no window could hold. Loading everything only makes sense when the material is genuinely small, static, and relevant to every query.