What is RAG (retrieval-augmented generation)?
Last verified: July 2026· LLM evals
Retrieval-augmented generation is the most reliable way to make a language model answer from your data rather than its training set. Instead of hoping the right facts are buried in the model's weights, you retrieve the relevant documents at query time and hand them to the model as context, along with an instruction to answer only from what it was given and to cite its sources. The result is grounded, current, and auditable — and the engineering work is mostly in the retrieval half, not the generation. This guide walks through what RAG is, how the pipeline fits together, how it compares to fine-tuning, when to use it, and how to evaluate it so you know it actually works.
What RAG is, and why it beats parametric memory
A language model stores what it learned during training as parametric memory — knowledge baked into its weights. That memory is fixed at the training cutoff, blind to your private data, and impossible to cite. Ask it about last week's incident or a policy in an internal wiki and it will either say it doesn't know or, worse, confidently invent an answer. Retrieval-augmented generation fixes this by separating knowledge from reasoning: the model keeps its language and reasoning ability, but the facts are supplied fresh at query time from a source you control.
Concretely, RAG retrieves the documents or passages relevant to a question — usually via embeddings and a vector store, often combined with lexical search — and places them in the prompt. The model is instructed to answer from that context and to cite which passage each claim came from. This buys you four things that matter in production: grounding (answers trace back to real source text), freshness (update the corpus, not the model), citations (a reader can verify each claim), and less hallucination (the model is steered toward retrieved evidence rather than filling gaps from memory).
RAG does not make hallucination impossible — a model can still misread or over-extend the context, and if retrieval returns nothing useful the model may fall back on its parametric guess. But it changes the failure mode from silent fabrication to something you can measure and constrain, because every answer is supposed to be backed by a passage you can point to.
How the RAG pipeline works: chunk, embed, store, retrieve, rerank, generate
RAG has two phases. The first is indexing, done offline whenever your data changes. You split source documents into chunks — passages small enough to be specific but large enough to carry meaning — because retrieving a whole 40-page PDF is useless as context. Chunking strategy matters: split on natural boundaries (sections, paragraphs) rather than arbitrary character counts, keep some overlap so a sentence isn't cut mid-thought, and attach metadata (source, title, date, section) to every chunk so you can filter and cite later. Each chunk is then passed through an embedding model that turns it into a vector — a numeric representation where semantically similar text lands close together — and those vectors are written to a vector database alongside the original text and metadata.
The second phase is query time. The incoming question is embedded with the same model, and the vector store returns the nearest chunks by similarity — this is the retrieval step. A strong pipeline doesn't stop there: it often runs hybrid retrieval (combining semantic vector search with lexical keyword search so exact terms, codes, and names aren't lost), then applies a reranker — a model that scores each candidate chunk against the query directly and reorders them, so the few passages actually placed in the prompt are the most relevant, not just the most vector-similar.
Finally comes grounded generation. The top-ranked chunks are assembled into a prompt with an instruction along the lines of: answer using only the provided context, cite the source of each claim, and say you don't know if the context doesn't contain the answer. The model produces the answer with citations back to specific chunks. Every stage is a lever: better chunking and embeddings improve what's retrievable, hybrid search and reranking improve what reaches the prompt, and prompt design plus a capable model improve how faithfully the answer sticks to the evidence.
Semantic search vs hybrid retrieval — and do you need a vector database?
Semantic search is the retrieval engine inside most RAG systems, but the two are not the same thing. Semantic search is a component — embed a query, find the nearest vectors — that returns relevant passages. RAG is the whole system that uses that retrieval to condition a generative model and produce a written, cited answer. Semantic search returns a ranked list of documents; RAG returns a synthesised answer grounded in them. You can have semantic search without RAG (a search box), but you can't have good RAG without good retrieval.
Pure semantic search has a well-known weakness: embeddings capture meaning but blur exact tokens, so a query for a specific error code, product SKU, surname, or legal clause number can rank below passages that are merely topically related. Hybrid retrieval addresses this by running lexical search (classic keyword/BM25-style matching) alongside vector search and fusing the results, so you get semantic recall and exact-match precision. For most real corpora — full of identifiers, jargon, and proper nouns — hybrid meaningfully outperforms either method alone, which is why it's the default we reach for.
Do you need a dedicated vector database? Not always. A vector store's job is fast approximate nearest-neighbour search over embeddings at scale. If your corpus is small — thousands of chunks — an in-memory index or a vector extension on the database you already run (for example a Postgres extension) is usually enough, and it keeps your stack simple. A purpose-built vector database earns its place when you have millions of vectors, need low-latency retrieval under load, want metadata filtering and hybrid search built in, or need to update the index continuously. The honest default: start with what you have, add a specialised store when scale or latency requirements force the issue — not before.
RAG vs fine-tuning: different tools, often complementary
The most common confusion in this space is treating RAG and fine-tuning as competing answers to the same question. They solve different problems. RAG changes what the model knows: it injects knowledge and keeps it fresh, and it produces citations because the facts came from retrievable sources. Fine-tuning changes how the model behaves: it adjusts style, tone, format, and task-specific patterns by updating the weights on examples. If your problem is 'the model doesn't know our data' or 'our data changes weekly,' that's a retrieval problem — reach for RAG. If your problem is 'the model won't follow our house format' or 'it needs to reliably output this exact structure,' that's a behaviour problem — reach for fine-tuning.
The practical differences follow from that. RAG's knowledge updates by re-indexing documents — cheap, fast, auditable, and instantly revertible. Fine-tuning bakes patterns into weights, which is powerful for behaviour but a poor way to store facts: retraining to add a document is slow and expensive, and the model still can't cite where a fact came from. RAG also degrades more gracefully — if a fact isn't retrieved, the system can say so — whereas a fine-tuned model has no way to signal that a memorised fact is stale.
In production the two are frequently complementary, not either/or. A mature system might fine-tune a model to reliably adopt a domain's voice and output structure, and use RAG to feed it the current, cited facts it should reason over. Use fine-tuning to shape the behaviour; use RAG to supply the knowledge. Start with RAG — it addresses the more common need, ships faster, and is easier to evaluate — and add fine-tuning only when you've proven that a behaviour or format gap remains that prompting and retrieval can't close.
When to use RAG (and when not to)
RAG is the right tool when answers must come from a specific, changing body of knowledge that the base model doesn't reliably contain: internal documentation, product manuals, support tickets, policies, contracts, research libraries, or any corpus that updates faster than you could retrain. It's especially valuable when you need traceability — regulated, legal, medical, or financial settings where an answer without a verifiable source is worthless, and citations are a requirement rather than a nicety.
It's a weaker fit in a few cases. If the knowledge is small, stable, and general enough that the model already handles it well, RAG just adds moving parts. If the task is pure reasoning, transformation, or generation over text the user already provides in full (summarising a pasted document, rewriting supplied copy), there's nothing to retrieve. And if your real need is a change in behaviour or format rather than knowledge, that's a fine-tuning or prompting problem, as above. RAG also cannot rescue a corpus that doesn't contain the answer — retrieval can only surface what exists, so garbage or gaps in the source data become garbage or gaps in the output.
A useful test before committing: could a knowledgeable colleague answer this question correctly given the right document open in front of them? If yes, RAG's job is to reliably put that document in front of the model — and the engineering effort should go where the difficulty actually lives, which is retrieval quality.
Evaluating a RAG system: retrieval and generation, separately
The single most important discipline in RAG is evaluating the two halves independently, because a bad answer has two very different causes and they need different fixes. Either retrieval failed to surface the right passage (a retrieval problem), or the right passage was retrieved and the model still answered wrong (a generation problem). Grade the end-to-end answer only and you'll waste time tuning prompts when the real fault was an empty retrieval — or vice versa.
Evaluate retrieval with a labelled set of questions paired with the passages that should answer them, and measure whether the relevant chunks appear in the results and how highly they rank — recall (did we retrieve the needed passage at all?) and rank-quality metrics that reward putting the right chunk near the top. This is where you diagnose chunking, embedding choice, hybrid search, and reranking: if the correct passage never gets retrieved, no amount of prompt tuning downstream will save the answer.
Evaluate generation given the retrieved context, focusing on two properties. Faithfulness (or groundedness): is every claim in the answer actually supported by the retrieved passages, or did the model add unsupported detail? Answer relevance: does the response actually address the question? Check that citations point to the passages that genuinely support each claim, and that the system correctly abstains — says it doesn't know — when the context lacks the answer, rather than guessing. Run these evals continuously so a change to chunking, an embedding-model swap, or a prompt tweak is caught by numbers rather than by a user filing a complaint. RAG quality is an evals problem as much as a retrieval one, which is exactly why the two are worth building together.
Frequently asked questions
RAG vs fine-tuning — which should I use?
They solve different problems, so the question is often 'which first,' not 'which instead.' RAG adds knowledge, freshness, and citations by retrieving your data at query time; fine-tuning changes the model's behaviour, style, and output format by updating its weights. If the model 'doesn't know your data' or your data changes often, use RAG. If it 'won't follow your format or voice,' use fine-tuning. They're frequently complementary — fine-tune for behaviour, use RAG for the facts — and most teams should start with RAG because it ships faster, updates cheaply by re-indexing, and is easier to evaluate.
Do I need a vector database?
Not always. A vector database provides fast nearest-neighbour search over embeddings, plus metadata filtering and often hybrid search. For a small corpus — thousands of chunks — an in-memory index or a vector extension on a database you already run is usually enough and keeps the stack simple. A dedicated vector database earns its place at scale: millions of vectors, low-latency retrieval under load, continuous index updates, or built-in hybrid and metadata filtering. Start with what you have and add a specialised store when scale or latency actually forces it.
What's the difference between semantic search and RAG?
Semantic search is a component; RAG is the system built on top of it. Semantic search embeds a query and returns a ranked list of relevant passages. RAG uses that retrieval to condition a language model and produce a synthesised, cited answer. You can run semantic search on its own as a search box, but you can't build good RAG without good retrieval underneath it — and in practice most RAG systems combine semantic search with lexical keyword search (hybrid retrieval) so exact terms and identifiers aren't lost.
Does RAG stop hallucination?
It reduces it but doesn't eliminate it. By grounding answers in retrieved passages and instructing the model to answer only from that context and cite sources, RAG steers the model toward real evidence instead of filling gaps from memory. But the model can still misread context or over-extend a claim, and if retrieval returns nothing useful it may fall back on a parametric guess. The value is that RAG turns silent fabrication into something measurable: you can evaluate faithfulness, check citations, and require the system to abstain when the context lacks the answer.
How should I chunk documents for RAG?
Chunk on natural boundaries — sections, paragraphs, logical units — rather than fixed character counts, so a passage carries coherent meaning. Keep chunks small enough to be specific but large enough to stand alone, add a little overlap so sentences aren't cut mid-thought, and attach metadata (source, title, date, section) to every chunk for filtering and citation. There's no universal size; the right setting is whatever your retrieval evaluation shows reliably surfaces the passage that answers a question. Treat chunking as a tunable parameter you measure, not a constant you guess once.