Retrieval or Fine-Tuning: How to Actually Decide
Mostafa Dhouib··6 min readThe decision is not about quality and it is not close. Retrieval changes what the model knows; fine-tuning changes how it behaves. Almost every team asking this question has a knowledge problem and is being sold a behaviour solution, and the tell is whether the right answer changes when your documents change.
Retrieval or Fine-Tuning: How to Actually Decide
The short answer. They solve different problems. Retrieval changes what the model can see; fine-tuning changes how the model behaves. If the correct answer to a question changes when your documents change, you have a knowledge problem and retrieval is the answer. If the correct answer is stable but the model expresses it in the wrong form, tone, or structure, you have a behaviour problem and fine-tuning may help. Most teams asking this question have the first and are being offered the second.
The one question that decides it
Take a question your system gets wrong. Now update a document so the correct answer changes.
Does the desired output change? Yes means knowledge. Retrieval.
Does the desired output stay the same, but you want it said differently? That is behaviour. Fine-tuning is on the table.
This works because it isolates the variable. Everything about a knowledge problem is downstream of what is in the corpus at the moment of the query, and no weight update tracks a document edited this afternoon.
The decision table
| Retrieval | Fine-tuning | |
|---|---|---|
| Changes | What the model can see | How the model behaves |
| Right when | Facts live in documents, change over time, or are private to a tenant | Output form, tone, structure, or a domain idiom is wrong |
| Updating a fact costs | Re-index a document | A retraining run |
| Attribution | Natural: you know which passage was used | None: the fact is diffused into weights |
| Per-tenant isolation | Enforceable at the index or data layer | Requires a model per tenant |
| Handles a fact added today | Yes, immediately | No, until the next training run |
| Fails by | Retrieving the wrong passage, or none | Behaving consistently and being confidently wrong |
| First cost | Index, embedding, evaluation set | Data curation, training, evaluation set |
| Ongoing cost | Re-indexing, drift monitoring | Retraining as behaviour requirements move |
| Regulated setting | Evidence is a citation to a source | Evidence is a training-data provenance argument |
The row that decides most real cases is attribution. If someone will ask "where did that come from," retrieval can answer and a fine-tuned model cannot, because the fact no longer exists as a locatable thing.
Why fine-tuning gets chosen anyway
Worth naming, because the pull is real and it is not stupidity.
It feels more sophisticated. Retrieval is plumbing: chunking, indexing, ranking, an evaluation set. Fine-tuning is machine learning. Given a choice between the two, teams gravitate to the one that reads as harder.
It produces a deliverable. A fine-tuned model is an artifact with a version. A working retrieval system is a set of unglamorous components that nobody photographs.
The first demo is better. A model fine-tuned on your domain sounds right immediately, in a way a partially built retrieval system does not. That impression fades exactly when the first fact goes stale, which is after the decision was made.
The knowledge problem is disguised as a tone problem. "It does not sound like it understands our business" is usually a report that it lacked the specific facts, filtered through how a non-technical reviewer experiences the gap.
What people miss about fine-tuning
Three properties that change the calculus once they are on the table.
It does not reliably add facts. Training on documents teaches style and distribution far more readily than it installs retrievable knowledge, and what it does install cannot be updated, audited, or attributed. Facts injected this way surface unpredictably and are indistinguishable from confabulation at the point of use.
It makes wrongness more confident. Fine-tuning aligns the model to your domain's register, so a wrong answer now arrives in fluent, house-style, senior-sounding prose. That is worse than an obviously out-of-place wrong answer, because the surface cue that prompts a reader to check has been removed.
Behaviour requirements move too. The assumption that fine-tuning is a one-off and retrieval is ongoing maintenance is usually backwards after the first year.
What people miss about retrieval
To be fair to the other side.
Recall is the ceiling and almost nobody measures it. If the passage containing the answer is not retrieved, nothing downstream recovers it: not a better prompt, not a bigger model, not a reranker. Teams spend quarters on generation while the retrieval half is the constraint.
A corpus that contradicts itself cannot be fixed by retrieval. If three documents disagree and none is marked authoritative, the system will faithfully surface whichever ranked highest. That is a content governance problem wearing an engineering costume.
It adds latency and a failure mode. Every query now depends on an index being available, current, and correctly scoped.
Where both is correct
- Build retrieval and measure recallthe ceiling on everything downstream
- Split the remaining failures by halfpassage retrieved or not, on fifty known-wrong questions
- If the errors are about content, keep working on retrieval
- If they are about form, consider a light tunestructured output a prompt cannot produce, a domain register, a consistent refusal
- Facts still come from retrieval at query timethe tune governs only how they are presented
The genuine both case is narrower than it is sold as: retrieval carries the facts, and a light tune fixes the form.
Structured output that a prompt cannot reliably produce. A domain register that matters to your users. A consistent refusal behaviour. In each, the facts still come from retrieval at query time and the tune governs only how they are presented.
The order matters and is frequently reversed. Build retrieval first and measure it. Then, if the remaining errors are about form rather than content, consider a tune. Tuning first hides the retrieval gap behind fluency, and you lose the ability to tell which half is failing.
The sequence to follow
Split your failures by half. Take fifty questions the system gets wrong. For each, check whether the passage containing the answer was retrieved. Two buckets, one afternoon, and it determines everything.
If retrieval failures dominate, the answer is retrieval work and fine-tuning is a distraction with a cost attached.
If the passage was retrieved and the answer was still wrong, look at what kind of wrong. Contradicted the source, ignored it, or cited it while saying something else is a grounding problem. Correct but in the wrong form is where a tune belongs.
Only then price them, and price the ongoing side rather than the first build.
FAQ
Should I use RAG or fine-tuning? Ask whether the desired output changes when your documents change. If yes, it is a knowledge problem and retrieval is the answer. If the desired answer stays the same but you want it expressed differently, it is a behaviour problem and fine-tuning may help.
Can fine-tuning teach a model new facts? Not reliably. Training on documents teaches style and distribution far more readily than it installs retrievable knowledge, and what it does install cannot be updated, audited, or attributed. Those facts surface unpredictably and are indistinguishable from confabulation at the point of use.
When do you need both? When retrieval carries the facts and a light tune fixes the form: structured output a prompt cannot reliably produce, a domain register, or a consistent refusal behaviour. Build and measure retrieval first, because tuning first hides the retrieval gap behind fluency.
Why does fine-tuning make wrong answers worse? Because it aligns the model to your domain's register, so a wrong answer arrives in fluent, house-style prose. The surface cue that would have prompted a reader to check has been removed, which is why a confidently wrong tuned model is more dangerous than an obviously out-of-place one.
What should I measure before deciding? Split fifty known failures into two buckets: the passage containing the answer was retrieved, or it was not. That takes an afternoon and settles which half your budget belongs in.
Split your failures into retrieved and not-retrieved in an afternoon, measure recall across k, score by slice, and check citation support mechanically. Ends in a seven-question hiring sheet with a scoring column.
Tell us the system, the stakes, and the date that matters. You get a straight technical reply from the person who would lead the work, within 24 hours.
Bring us the program