Skip to main content
AI

RAG vs fine-tuning: an open-book exam or a semester of study?

RAG gives a model the textbook during the exam. Fine-tuning makes it study beforehand. The right choice depends on what you need it to know, how often that changes, and whether people need to check the answer.

By Anvexis Stack Engineering8 min read

Side by side

AspectRAGFine-tuning
TeachesFacts and knowledgeStyle, format, and behaviour
UpdatingRe-index documentsRetrain the model
CitationsNaturalNot available
PermissionsApplied at retrievalHard to enforce
Best forPolicies, manuals, changing dataConsistent tone, specialized formats

What each approach actually changes

Retrieval-augmented generation leaves the model untouched. When a question arrives, the system searches your documents, picks the most relevant passages, and hands them to the model with the question. The model answers from that material and can cite exactly where each statement came from. The idea was formalized in a 2020 paper by Lewis and colleagues, which described models that combine the knowledge stored in their weights with an external, searchable memory, partly to tackle the open problems of provenance and keeping knowledge up to dateSource [1].

Fine-tuning changes the model itself. You train it further on examples of inputs and desired outputs, and it adjusts its weights to produce that kind of output more reliably. Microsoft's documentation lists the situations where that helps: improving accuracy on a specific task, reducing prompt overhead, aligning style and structure, improving tool use, using retrieved context better, and specializing a smaller modelSource [3].

Try these in order

  1. Better prompt

    Clear instructions and a few good examples solve more than people expect. Always start here.

  2. Put it in the prompt

    If the knowledge fits in the model's context, include it directly and cache itSource [2].

  3. RAG

    If the knowledge is large, changing, or permissioned, add retrieval with citations.

  4. Fine-tune

    If behaviour or format is still inconsistent at volume, fine-tune, and measure the difference.

Do you need retrieval at all?

Not always. Anthropic's write-up on contextual retrieval makes a point many teams miss: if your knowledge base is smaller than about 200,000 tokens, you can simply include all of it in the prompt, with prompt caching making that faster and cheaperSource [2]. That's a few hundred pages of policies or product documentation, which covers a surprising number of internal assistants.

Skipping retrieval removes a whole class of failures, such as the right passage never being found. Add a retrieval pipeline when the material outgrows the context window, changes frequently, or must respect per-user permissions.

When RAG is the right answer

Choose retrieval when the model needs to know things specific to your organization: policies, product details, contracts, procedures, or past tickets. These change regularly, and retrieval picks up changes as soon as documents are re-indexed. Citations let users check answers, which matters for anything with legal, financial, or safety consequences.

Retrieval also respects permissions naturally. Each passage carries the access rights of its source document, so two users asking the same question can get different answers based on what they're allowed to see. That's very hard to achieve with knowledge baked into a model's weights. Our RAG knowledge systems are built around exactly this.

Retrieval quality is where RAG succeeds or fails

A RAG system is only as good as the passages it finds. Splitting documents into chunks strips context: a chunk that says “revenue grew 3% over the previous quarter” is useless if it no longer says which company or quarter. Anthropic's experiments with adding that context back are instructive. Prepending a short, generated summary of each chunk's context before embedding reduced the rate of failing to retrieve a relevant chunk in the top 20 by 35%; combining that with keyword search cut failures by 49%; adding a reranking step brought the reduction to 67%Source [2].

Those numbers come from Anthropic's own benchmarks, so your mileage will vary, but the lesson travels: invest in chunking, hybrid search, and reranking before you blame the model. Clean, deduplicated, current source documents matter just as much. Retrieval over a messy shared drive returns confident answers from outdated files.

How to evaluate a RAG system

Evaluate retrieval and answering separately, because they fail differently. For retrieval, collect real questions and mark which documents or passages should be found, then measure how often they appear in the top results. For answering, check whether the response is correct, whether every claim is supported by the retrieved passages, and whether the model says “I don't know” when the passages don't contain the answer.

Keep the evaluation set alive. Add every question a user flags as wrong, every new document type, and the awkward phrasings people actually use. When you change the chunk size, the embedding model, the reranker, or the prompt, rerun the whole set and compare. Most RAG improvements are small, measurable gains like these rather than one big breakthrough.

Permissions and injection in retrieval

Apply access control at retrieval time, not after the answer is written. Filter the search to documents the user is allowed to see before any passage reaches the model; otherwise a well-phrased question can pull restricted content into an answer. Test it with two users of different roles asking the same question.

Retrieved passages are also a channel for prompt injection: a document can contain text that looks like instructions. Mark retrieved content clearly as data, keep the model's tools and permissions minimal, and never let a passage change what the system is allowed to do.

Keeping the index fresh

A RAG system is a data pipeline with a model at the end. Documents change, get deleted, and gain new versions, and the index has to follow. Track each source document's last-modified date, re-index changed documents automatically, remove deleted ones, and record which version of each document an answer was based on.

Plan for re-embedding, too. When you switch embedding models, every document needs new vectors, so keep the original text and chunking logic, not just the vectors. Teams that store only embeddings discover this the expensive way.

When fine-tuning earns its cost

Fine-tuning makes sense when the problem is behaviour rather than knowledge: producing a very specific output format reliably, adopting a consistent tone across thousands of responses, classifying text into your own categories, calling tools more accurately, or making a smaller, cheaper model perform a narrow task as well as a larger oneSource [3].

It isn't free. Microsoft notes that fine-tuning adds training and hosting costs, isn't guaranteed to help, and needs held-out evaluation to prove it beat the baselineSource [3]. You also need a plan for retraining when base models improve or are retired. For many teams, careful prompting with a few examples gets most of the benefit at a fraction of the effort, so try that properly first.

Common mistakes

The most common mistake is fine-tuning to teach facts. The model may learn them partially, mix them up, and keep repeating outdated versions after the source documents change. Microsoft is explicit that fine-tuning doesn't replace retrieval for current informationSource [3].

Another is building retrieval over messy, duplicated, or outdated documents, then blaming the model for confident wrong answers that came from bad sources. A third is skipping evaluation. Whichever approach you choose, build a set of real questions with known good answers and measure every change against it. Without it, decisions about chunk sizes, models, and prompts are guesses.

Using both together

Mature systems sometimes combine the two: a model fine-tuned to follow a specific answer format or tone, or to make better use of retrieved context, with retrieval supplying the factsSource [3]. This works well when you have high volume, strict output requirements, and the engineering capacity to maintain both pipelines.

Most teams should start with prompting, add retrieval when the model needs their knowledge, and consider fine-tuning only when evaluation shows a behaviour problem that prompting can't fix. Whatever you build, treat retrieved documents as data, never as instructions, which we cover in building production AI agents.

Where to start this month

Pick one narrow question set with real users, such as internal policy questions or product support for one product line. Gather fifty real questions with good answers, try the documents in the prompt first if they fit, and measure. Only build retrieval if the material doesn't fit or changes too often, and only consider fine-tuning once evaluation shows a behaviour problem prompting can't fix. Small, measured steps beat a grand architecture that nobody can evaluate.

Costs and maintenance compared

CostRAGFine-tuning
BuildIngestion pipeline, index, search tuningTraining data, training runs, evaluation
Per requestExtra tokens for retrieved passagesOften fewer prompt tokens
Keeping currentRe-index changed documentsRetrain when behaviour needs change
When models changeUsually swap the model and re-testRetrain on the new base model

Questions that decide it

  • Does the knowledge fit in the prompt? Try that before building retrieval.
  • Does the model need facts that change? Use retrieval.
  • Do users need to see sources? Use retrieval.
  • Do answers depend on who is asking? Use retrieval with permissions.
  • Is the problem format, tone, tool use, or a narrow task? Consider fine-tuning.
  • Is there an evaluation set to measure the difference? Build one first.

Frequently asked questions

Can fine-tuning teach a model our company's knowledge?
Partly, but it's the wrong tool. Facts learned in training go stale, can't be cited, and can't be restricted per user. Use retrieval for knowledge and fine-tuning for behaviourSource [3].
Do we always need a vector database for RAG?
No. If your material is small enough, put it straight in the promptSource [2]. When you do need search, a vector index combined with keyword search and reranking usually beats either aloneSource [2]. PostgreSQL with pgvector is often enough.
Is RAG more expensive than fine-tuning?
They cost money in different places. RAG costs an ingestion pipeline and extra tokens per request; fine-tuning costs training runs, hosting, and retraining over timeSource [3]. Compare them on your own volume and evaluation results.
How do we stop a RAG assistant from making things up?
Retrieve good passages, instruct the model to answer only from them and say when it can't, show citations, and measure grounded accuracy on an evaluation set. Our LLM agents follow this pattern.

Sources

  1. [1]Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (opens in a new tab)Lewis et al., arXiv, 22 May 2020
  2. [2]Introducing Contextual Retrieval (opens in a new tab)Anthropic, 19 September 2024
  3. [3]Fine-tuning in Microsoft Foundry (opens in a new tab)Microsoft Learn, 6 October 2026

Written by Anvexis Stack

We build Odoo, AI, web, and cloud systems for growing companies, and we write about the patterns we use. This is engineering guidance, not a product guarantee: check vendor details for your version before you decide. Last reviewed .