How to Fix a Broken RAG System (Without Rebuilding the Model)

Mostafa Dhouib·September 3, 2026·15 min read

How to Fix a Broken RAG System (Without Rebuilding the Model)

The short answer. When a RAG system gives wrong answers, everyone points at the model, but the model is one of six places it can break, and the score usually only checks one of them. The fix is four steps, cheapest first: slice the score to find the one broken corner, measure recall, precision, and faithfulness separately instead of one blended number, build a check that is not the same thing it is checking, and re-check on a schedule. Days of work, not a rebuild.

A company had a RAG system whose whole job was to answer questions over their own documents, ask a question, it finds the right pages, writes an answer, attaches a citation. It scored 95% in testing. A week later, when the internal team started using it, it dropped below 44%. Two contractors had already looked at it, both said rebuild the model, one quoted four months, the other six. The actual fix took five days, and it was not the model. Here is what the five-day fix really was, and how to use it on any system, not just RAG.

The model is one of six places it can break

When someone asks your system a question, six things happen: the user asks, the system grabs the documents it thinks are relevant, it orders them and guesses which are best, the model reads them and writes an answer, it attaches a citation, and a person reads the answer and acts on it. Those are also the six places it can go wrong.

Here is what trips people up. That 95% accuracy score is almost always measured on one of those steps, usually the fourth, the model writing an answer, checked against a fixed list of test questions. If the problem is in any of the other five steps, the score has no idea, because it was never looking there.

That is why "rebuild the model" is usually wrong. You end up rebuilding the one box that was probably fine.

The order matters, and so does the cost

The four steps below are ordered deliberately, cheapest and most informative first, and the ordering is most of the value.

Steps one and two are effectively free. They are analysis of data you already have, they take days at most, and they tell you where to spend. Step three costs real money, because it is the only one that builds something durable. Step four sits in between: a small ongoing cost forever, rather than a large one once.

Compare that against the alternative that was actually proposed. A full rebuild is the most expensive option available, and it was aimed at the one component that was working. The two contractors were not amateurs and were not trying to deceive anyone. They were aiming at the wrong thing, which is a much more common failure than dishonesty and considerably harder to detect.

The general rule underneath: do the free diagnosis before you authorize the expensive repair. It sounds obvious written down, and it is skipped constantly, because the expensive repair is the thing that feels like action.

Step 1: Find the broken corner

Wrong answers are not spread out evenly. They clump. It is almost never wrong 10% across the board, it is badly wrong in one specific corner and fine everywhere else. Your job, even as a non-technical decision maker, is to find which corner. Find it, and you are 90% of the way to the fix. Skip it, and you rebuild the whole thing to patch one small part.

Here is the trick. Break the score down by what kind of documents the answer came from. In this system, contracts scored 94%, policies 91%, specs 89%, all healthy, average 88%, a number that looks great on a dashboard. But scanned PDFs scored 41%. Anyone asking about a scanned document got a wrong answer more than half the time.

Nobody noticed because scanned PDFs were a small slice of the total, so they barely dented the average of 88%. Old contracts that were photographed years ago look like normal documents to you and me, but to the system they are pictures it cannot read as cleanly as text, so it fumbles those answers quietly and never tells you, and the average hides it completely. The fix was not a better number, it was breaking one number into many. The second you break one score into many, the broken parts stop hiding.

Cut along the lines where your reality actually changes: by question type (a simple lookup is a different task than "combine these three documents"), by document type (clean text versus scanned), by age (recent versus old documents), by how deep it had to dig (answered from the first result versus buried down the list), and by team. Two practical notes on doing this, because it is easy to do badly.

Slice on attributes you already have. Document type, source system, age, requesting team, question length, and retrieval depth are all recorded or trivially derivable, and none of them requires labeling anything. Teams often stall here because they assume slicing requires a new annotation project. It does not, at least not for the first pass.

And watch the small slices, not the big ones. The whole reason the failure hid is that it lived in a slice too small to move the average. That means the slices most likely to contain your problem are exactly the ones a summary statistic is designed to ignore, so look at the worst slice rather than the biggest, and look at slices with enough volume to be real but small enough to be invisible.

This is also your hire test. Ask whoever is fixing it to show you accuracy broken down by what varies in your business, not just the overall number. A good hire either already did it or can in an afternoon. If they only ever show you one big number, that is your tell, they have not actually looked.

Step 2: Measure the right thing

You can find the broken corner, fix it, and still fail, because the number you were watching was measuring the wrong thing all along. Every number stands in for something it is not. That 95% stood in for "people get correct answers at work," but those are not the same thing, they are connected by assumptions, and the gap is where things break. There is a name for it, Goodhart's law: the moment a number becomes the goal, people find a way to make the number go up in a way that has nothing to do with what you wanted, like a kid studying by memorizing last year's answer key. The number climbs while the actual quality quietly turns and heads down.

The cheap fix: make whoever is fixing this write down, in one sentence, what each number is actually standing for. Half the time the problem is obvious the second it is in plain words, because nobody had ever bothered to write it down.

For RAG specifically, "accuracy" is close to a garbage word, because retrieval breaks in three distinct places and you need to measure all three:

• Recall. Did it even find the right pages? This is the silent killer. If the answer is in a document the system never pulled up, it cannot answer correctly and cannot cite a page it never opened.
• Precision. Were the pages it found useful and in a good order, or was the right page buried under junk?
• Faithfulness. Did the answer stick to what was on those pages, or did it make things up?

Read recall and faithfulness together. A system that just refuses to answer looks perfect on faithfulness, it never makes anything up, while recall has totally collapsed. One number read on its own lies to you. And check the test itself: if the test questions leaked into what the system was trained on, it is remembering the answers, not understanding, which can pump the score more than 20% while it learned nothing.

Step 3: Build a check, and never let it grade itself

This is the actual repair, and the one that costs real money. It turns on a distinction even smart technical people mix up. There are two completely different questions you can ask about your system: is it up, responding, fast, no errors, and is the answer right? These feel like the same question and they are not close.

Everything your ops monitoring watches all day answers only the first one. It is a smoke alarm that only checks its own battery, "battery good, all green," while it is not actually smelling any smoke. A system can respond in a third of a second with a clean green light and no errors anywhere, and hand every user a confident wrong answer. Building the second dial, is it right, is the repair.

Three layers build it:

• A golden set. A list of questions you already know the correct answer to, a trusted answer key. Every time anyone changes anything, run those and check you still get them right. This alone would have caught the problem on day one.
• Shadow and canary. Test a change quietly on real traffic without showing anyone the results, then release it to 1% of users with an automatic undo if it goes bad. Do not flip it on for everyone and pray.
• Human review. A person reads a small sample, one or two answers out of a hundred, against the actual sources. This is the most expensive layer, the one teams cut first, and the one that matters most, because it is the only place real truth actually enters the system.

A golden set is worth being concrete about, because it is the cheapest of the three and the one most often built wrong.

It is not a list of questions someone invented at a desk. It is drawn from questions people actually asked, weighted toward the corners you found in step one, with answers verified by someone who knows the domain and a note recording which document the answer came from and when it was verified. Fifty to a couple of hundred items is usually enough to be useful, which is a week of someone's attention rather than a project.

It has to be owned and versioned like code, because a golden set that is never updated will eventually certify that your system still gives the answer that was correct two years ago. When a policy changes, the answer key changes with it, and the process that changes the policy is the one that has to touch it.

And the single most important rule in all of this, the one question that tells you the most about whoever you hire: the check cannot be the same thing it is checking. You cannot let the AI grade its own homework. It goes easy on itself, and that is measurable, it prefers longer answers and its own writing style and scores those higher, like a student grading their own exam. The right way is the model writes the answer and something else grades it, a different model plus a human on a sample. So the sharpest hire question, in plain English: how do you check that the answers are right, and what is doing the checking? If the answer is any version of "the AI checks itself," you have found your problem standing right in front of you.

The honest caveat: sometimes you cannot grade in real time

Anyone good will admit this. Sometimes you cannot know if an answer was right until much later, or ever. You do not know if approving a loan was the right call until years down the road. You do not find out a transaction was fraud until the chargeback shows up months later. When the truth arrives late, you cannot grade in real time, so people fall back to estimating quality from indirect clues, and there are clever tools for this. But they have a blind spot: they can tell you when your incoming questions start looking different, but not when the correct answer changed underneath while the questions look exactly the same, which is the most dangerous kind of failure there is. The tell: if someone promises fully automatic certainty, no humans, everything handled, they are overselling. The honest version always has a human in the room and admits where its blind spots are.

Step 4: Keep it fixed

Everything above describes your system today, but a RAG system answers from a pile of documents, and that pile is always changing. The fix you ship today degrades. That is entropy, and it applies to every system, not just this one. So the last step is to find out before you blow up.

There are two kinds of drift. The easy one: the questions start looking different, common, because inputs change. The nasty one: the questions look exactly the same but the right answer changed. You updated your refund policy in January, but the old policy document is still sitting in the system, so it keeps answering with the old rule. The question looks normal, the answer sounds confident, and it is wrong, and nothing tripped because nothing looked wrong.

The fix is not clever, it is a clear calendar. Check every so many days against your baseline. There is a measure for it, but the threshold for when to worry is a judgment call, pure experience, there is no magic number, and if someone quotes you an exact one as if it were a law, they are bullshitting. And the most important rule for this step, the one that separates the careful from the reckless: when the drift alarm goes off, wake up a human, do not automatically rebuild. At least half the time the alarm fires because something broke in your data plumbing, someone upstream changed a file format, not because the answer changed, and auto-retraining the second it fires means retraining on garbage. So ask whoever handles this: what happens when a drift alarm goes off? If they say everything is automatic, they are bullshitting. A human should look at why before anything is rebuilt.

The decision-maker's checklist

You are not going to rebuild this yourself, but you are the one spending the money, so you need to be able to tell from someone's answers whether they actually get your problem. Ask:

• How did you rule out the other five steps, not just the model?
• Can you show me accuracy broken down by source, question type, and age?
• Are you measuring recall, precision, and faithfulness, or one blended number?
• How do you know it learned this instead of memorizing pre-made answers?
• How do you check that the answers are right, and what is doing the checking? Is there an answer key? Does a human read a sample, and how often?
• And the big one that beats any fancy pitch: what does your approach actually miss? A person who really does this has a quick, specific answer, because they live with the limit every day. Someone overselling tells you it does not miss anything, which is always false, because nothing is perfect. That answer, more than any resume or portfolio, tells you whether they can actually fix your system.

The takeaway

You do not need to rebuild the whole system. You need to build the thing that would have caught the failure in the first place: slice the number, measure the right thing, check a guess against reality with a human in the loop, and keep checking on a schedule. That was the five-day fix, and it was days of work against a four-to-six-month rebuild of a part that was never broken.

FAQ

Why does a RAG system pass testing but fail in production? Because the test score usually measures only one of the six steps in a RAG pipeline, typically the model writing an answer against fixed questions. If the failure is in retrieval, ordering, citations, or the changing document set, the score never sees it, so it stays green while production breaks.

Should I rebuild the model to fix RAG accuracy? Almost never as the first move. Wrong answers clump in one corner of the pipeline, and slicing the score by document type, question type, and age usually points to it. The fix is typically days of retrieval and verification work, not a multi-month model rebuild.

What metrics should I use to measure a RAG system? Not one blended accuracy number. Measure recall (did it find the right pages), precision (were they useful and well ordered), and faithfulness (did the answer stick to the pages) separately, and read recall and faithfulness together, since a system that rarely answers can look faithful while recall has collapsed.

Can I use an LLM to evaluate its own answers? No. A model grading its own output goes easy on itself in measurable ways, preferring longer answers and its own style. Use an independent grader, a different model plus a human reviewing a sample, and never let the check be the same thing it is checking.