Retrieval or Generation? Why You Can't Tell Which Half of Your RAG System Broke

Mostafa Dhouib·September 3, 2026·10 min read

Retrieval or Generation? Why You Can't Tell Which Half of Your RAG System Broke

The short answer. A wrong RAG answer is not one failure, it is two different diseases with one symptom. Either the search pulled the wrong pages, or the search worked and the model ignored them and invented an answer. From the output they are identical, same confident tone, same citation, but the fixes are opposite. And a single blended quality score averages the two into one green number that hides which half is actually broken.

An internal RAG tool answered a refund-window question. Fluently, specifically, citing a real contract section.

It was completely wrong. The cited section had nothing to do with the thirty-day claim, and the number was invented. The dashboard showed a quality score of 94, green, and stayed green while this happened.

The answer was wrong wearing the exact uniform of every correct answer the system had ever given. To the person reading it, it was indistinguishable from the truth, and they acted on it, because not rereading the contract is the entire reason the tool exists.

Every number on the dashboard is a messenger

You cannot personally read all forty thousand answers, so you appoint a messenger, one number, to go check the truth and report back. The 94 was that messenger.

You do not care about the number. You care whether the answers are true. And the failure is always the same shape: the messenger comes back and says fine, everyone stops looking, and nobody asks whether it actually checked the thing it was sent to check.

This is not a log entry that sits quietly. In these systems the output is the start of a chain. A wrong answer becomes a wrong action. Closing a refund window becomes a commitment the company has to honor. The next decision builds on it, and the next, and no stage downstream ever rereads the original document.

That is the difference between a system that is wrong, which costs you one answer, and a system that is load-bearing and wrong, which costs you everything standing on it. The green number does not tell you which one you have.

One symptom, two diseases

Here is the center of it. A RAG system has two jobs that are nothing alike. Find the right documents, which is search. Write an answer from what it found, which is the model. Two separate machines, two ways to fail.

Disease one, the search fails. You ask about the refund window. The contract says "the buyer's right to rescind within thirty days." To you and me those are the same thing. To the search they are not, because it is matching on a mathematical score, not on meaning as you understand it. So it pulls three vaguely related, useless pages, and the one page with the answer never gets picked.

It is like trying to find a song stuck in your head by humming it into a text box. You get nothing, because the index speaks words and you are holding a melody. The question and the document meant the same thing in two different languages, and the model has not even started writing yet.

Disease two, the generation fails. This time the search nails it and places the exact right page in front of the model, and the model writes something else anyway. It drifts off the page and invents from its training.

The model is blending two sources: the document you handed it, and a vast fuzzy memory of how these words usually go. It writes what sounds right based on the thousand other contracts it has seen. It is like a senior expert so experienced they answer before reading the file you gave them, right nine times out of ten, which is exactly why the tenth time, when your file said something unusual, is so dangerous.

From the output, both are identical. Same confident tone, same clean format, same citation. There is no way to tell which machine is broken by looking at the answer.

It is a fever that could be a harmless flu or a bacterial infection that kills you. Same symptom, opposite treatments. The retrieval fix, how you chunk and index and rank, lives somewhere completely different from the generation fix, how you constrain the model to the page. Spend a month improving the model when the disease was search, and it is all wasted, and now you also believe you fixed it.

The average hides the room that is on fire

So why did the green number betray them?

The quality score did the most natural and most destructive thing available: it averaged the search signal and the writing signal into one number.

Picture a thermostat wired to two rooms, one freezing and one on fire. It reads the average and reports a pleasant 72, which is technically correct and completely useless, because no single room is 72, and it hides the room that is burning in plain sight.

The 94 was the average of "search struggling" and "writing fine," blended into one number that describes no real part of the system. An average always throws information away, and this one threw away the single thing they needed: which half was broken.

The three numbers you actually need

Accuracy is close to a useless word for a retrieval system. It is not merely vague, it is measuring three different machines at once. Retrieval systems break in three specific places, and each has its own name and its own fix.

Recall. Did it even find the right pages? This is the silent killer. If the answer lives in a document the system never pulled up, nothing downstream can rescue it. The model cannot quote a page that was never opened. Recall failures are invisible from the output, because what you see is a confident answer built from whatever it did find.

Precision. Were the pages it found actually useful, and in a sensible order? Or was the right page technically retrieved but buried under a pile of junk, far enough down that it may as well not have been there? A system can have adequate recall and still fail, because the signal was drowned in the noise it retrieved alongside it.

Faithfulness. Did the answer stick to what was actually on those pages, or did the model make things up?

Three numbers, three machines, three different fixes. One blended score tells you about none of them.

The pairing trap

Here is the part that trips people up, and it is worth stating as a rule: recall and faithfulness have to be read together, never separately.

Consider a system that has become very cautious and mostly refuses to answer. Measure faithfulness alone and it looks immaculate. It never invents anything, because it barely says anything. Perfect score, and the dashboard is delighted.

Meanwhile recall has collapsed and the system is useless. Nobody is getting answers. The faithfulness number is not lying about faithfulness, it is just answering a question nobody needed answered.

The reverse pairing fails too. A system that always answers something will have excellent coverage and terrible faithfulness, and reading coverage alone will tell you it is working.

Any one of these numbers read on its own will lie to you. Read them as a set or do not bother reading them.

And check the test itself

One more check before you trust any of the three. If your test questions leaked into what the system was trained or tuned on, the system is not understanding anything, it is remembering the answers.

This is the student who studied by memorizing last year's answer key. The score goes up dramatically. Nothing was learned. That kind of leak can inflate a score substantially while the underlying capability is unchanged, which means every decision you make from that number is made on a fiction.

So the question worth asking, of any evaluation: how do you know it learned this rather than memorized it?

How to actually run the split

The diagnosis is not difficult once you decide to do it. It takes a set of real questions with known answers and a couple of days.

For each question in your set, record which documents were retrieved, in what order, before the model sees anything. Then ask a separate question of each stage.

Did the document containing the answer appear in the retrieved set at all? If no, this is a retrieval failure, and nothing about the model is implicated. Stop. Go work on chunking, indexing, embeddings, query expansion, or hybrid search.

If yes, did it appear high enough to matter given how many documents you pass to the model? If it was retrieved at rank forty and you pass the top five, that is still a retrieval failure, specifically a ranking one.

If the right document was retrieved and passed to the model, and the answer is still wrong, now you have a generation failure. Only now. And the fixes are different: constrain the model to the provided context, require span-level citation, penalize claims not supported by the passed text, or reduce the noise in the context window.

Run that over a few dozen real failures and the split is usually obvious within a day. Most teams have never run it, and have been arguing about the model for months.

The problem was never quality

The real lesson underneath all of this is that the system was not untrustworthy because it was bad. The model was fine, the engineering was fine, the team was good.

It was untrustworthy because when it failed, there was no way to find where.

A thing you cannot diagnose is a thing you cannot trust, no matter how good it is. And that is genuinely good news, because quality is hard to manufacture and visibility you can always build. Instrument the search and the writing separately. Build a real test set from actual questions your users ask. Never report the average.

What you buy with that work is not a better answer. It is the ability to know which half is broken before you lay a finger on it. That, more than anything, is what separates people who have seen this pattern across many systems from the ones who reach straight for a bigger model, which is treating the patient before running the scan.

FAQ

Why does my RAG system give confident, cited answers that are wrong? Because a wrong answer has two possible causes that look identical: the search pulled the wrong pages, or the search worked and the model ignored them and invented an answer from its training. Both come back with the same confident tone and a citation, so the output alone cannot tell you which.

How do I tell whether retrieval or generation is the problem? Record what was retrieved, in what order, before the model sees it. If the document containing the answer was never retrieved, or was ranked below what you pass to the model, it is a retrieval failure. If it was retrieved and passed and the answer is still wrong, it is a generation failure. Run that over a few dozen real failures.

What metrics should I use for a RAG system instead of accuracy? Recall, whether the right pages were found at all. Precision, whether what was found was useful and well ranked. Faithfulness, whether the answer stuck to those pages. Three machines, three numbers, three different fixes.

Why can't I read faithfulness on its own? Because a system that mostly refuses to answer scores nearly perfectly on faithfulness while its recall has collapsed and it is useless. Faithfulness and recall have to be read as a pair, or the number will confirm that a broken system is working.

Why is a single accuracy number misleading for RAG? Because it averages independent things, retrieval quality and generation faithfulness, into one figure that describes neither, the way a thermostat wired to a freezing room and a burning room reads a comfortable average. The average throws away exactly the information you need: which half is on fire.