Index Freshness and Drift: Why a Working RAG System Quietly Goes Wrong

Mostafa Dhouib·September 3, 2026·9 min read

Index Freshness and Drift: Why a Working RAG System Quietly Goes Wrong

The short answer. A RAG system answers from a pile of documents, and that pile never stops changing, so the fix you shipped today decays. There are two kinds of drift: the easy one where questions start looking different, and the nasty one where questions look identical but the correct answer changed underneath. The fix is not clever, it is a calendar: re-check against a baseline on a schedule, and when the alarm fires, wake a human, because half the time it is a broken data feed, not a real change.

You fixed your retrieval system. You found the broken corner, you started measuring the right thing, you built a real check against reality. It works.

It will quietly stop working.

Not because you did anything wrong, and not because of a bug you will eventually find. A RAG system answers from a pile of documents, and that pile is always changing. Documents get added. Documents get superseded. Someone reorganizes a folder. A policy changes and the old version stays where it is. The system you validated was validated against a snapshot, and the snapshot is already out of date.

This is ordinary entropy, and it applies to every system, not just this one. Nothing stays fixed unless it is maintained. So the last piece of a reliable retrieval system is not a better retriever. It is finding out you are drifting before it becomes a disaster.

Two kinds of drift, and only one of them announces itself

There are two, and treating them as one problem is why most monitoring misses the one that matters.

The easy kind: the questions change. Your users start asking about things they did not ask about before. A new product launches, a regulation changes, a season turns, a different team starts using the tool. The distribution of incoming queries visibly shifts.

This one is tractable. The inputs are right there, you can compare this month against your baseline, and there are established statistical measures and off-the-shelf tools that will tell you when the incoming distribution has moved. If someone tells you they are monitoring drift, this is almost always the kind they mean.

The nasty kind: the questions stay the same and the right answer changes underneath.

Say you updated your refund policy in January. The new policy is correct, approved, and published. But the old policy document is still sitting in the index, because nobody pulled it out, because removing documents is nobody's job and no ticket was filed.

The system keeps answering with the old rule.

Look at what every signal reports. The question looks completely normal, because it is the same question people have always asked. The retrieval succeeds and returns a real document. The answer is faithful to that document, so a faithfulness check passes cleanly. The answer sounds confident. There are no errors, no latency spike, no exception in any log.

And it is wrong, and it will keep being wrong, and someone on your team will act on it.

This is the most dangerous failure available in a retrieval system, because it is invisible in every dimension you are likely to be watching. The input distribution has not moved. The system has not degraded technically. The failure is entirely in the relationship between the index and a world that changed, and nothing inside the system has visibility into that relationship.

It is worth being explicit about the limit of the tooling here, because vendors are not always. The good drift-detection tools can tell you when your incoming questions start looking different. They cannot tell you when the correct answer changed while the questions looked exactly the same. That is not a shortcoming of any particular product, it is a consequence of what the tool can observe: it sees your inputs, and this failure is not in your inputs.

Catching the nasty kind

Since the surface signals are all clean, the only thing that catches it is comparing against something that knows the truth.

Your golden set is the primary defense, and it only works if it is maintained as a living thing. A set of question and answer pairs written eighteen months ago and never revisited will happily confirm that your system still gives the answer that was correct eighteen months ago. When a policy changes, the golden set has to change with it, which means someone owns it and the process that changes the policy also touches the answer key.

Beyond that, three checks are worth the small amount of work they take.

Index provenance and recency. For each answer, know which document it came from, when that document last changed, and whether anything supersedes it. A large fraction of this failure class is a superseded document that was never removed, and simply surfacing the age of the source document next to the answer catches a surprising amount of it by eye.

Reconciliation against the source of truth. If the authoritative policy lives in a system of record somewhere, periodically check that what the index contains matches it. This is boring, mechanical, and catches the exact case where a document was updated in one place and not the other.

A human sample. One or two answers in a hundred, read by a person who knows the domain and checks them against current reality rather than against the retrieved document. This is the only place actual truth enters the system, and it is the first thing cut when a schedule tightens.

The fix is a calendar, not a clever tool

Here is the part people do not want to hear, because it is unglamorous.

The fix for drift is not a technique. It is a schedule. Re-run your golden set against your baseline every so many days, on the calendar, whether or not anything seems wrong. Track the result over time so a slow slide is visible as a slope rather than as a single reading you can dismiss.

That is genuinely most of it.

The difference between a system that bleeds down for a year until something visible breaks, and one that dips and recovers and dips and recovers, is not sophistication. It is whether anybody bothered to measure on a schedule.

There is a statistical measure for how far your input distribution has moved, and it is useful. But the threshold for when to worry is a judgment call, specific to your system and your tolerance, and it comes from experience with your own baseline. There is no magic number. If someone quotes you an exact universal threshold as though it were a law, they are overselling.

When the alarm fires, wake a human

This is the rule that separates the careful from the reckless, and it is the one most often gotten wrong by teams who are otherwise doing everything right.

When the drift alarm goes off, a person looks at why. Do not automatically rebuild. Do not automatically retrain.

The reason is specific. At least half the time, the alarm is firing because something broke in your data plumbing, not because the world changed. Somebody upstream changed a file format. An export job started truncating. A parser silently began failing on a document type and returning empty text. An encoding changed. A permissions change quietly removed a whole folder from the crawl.

Every one of those produces a drift signal that looks exactly like a real distribution shift.

If your system is wired to retrain the moment the alarm fires, then on those occasions you retrain on garbage. You take a system that had one broken input feed and you bake the breakage into the model, which converts a plumbing problem you could have fixed in an afternoon into a model problem you will chase for a month.

Automation that reacts to a signal without diagnosing it is not intelligence. It is an amplifier pointed at your own failures.

So the question to ask whoever owns this part of your system: what happens when a drift alarm goes off? If the answer is that everything is automatic and it handles itself, that is your tell. The correct answer is that a human looks at why before anything is rebuilt or retrained, and that the first thing they check is whether the data pipeline is intact.

What this looks like in practice

Put it together and the operating discipline is short enough to fit on one page.

A golden set that is owned, versioned, and updated when the world changes. A scheduled run of it, on the calendar, with results tracked over time. Provenance and recency attached to every answer, so a stale source is visible. Periodic reconciliation against the systems of record that hold the truth. A small human sample, continuously, not as a launch activity. Input drift monitoring for the easy kind, understood as a partial signal rather than a complete one. And an alarm that pages a person, never a retrain job.

None of that is expensive. All of it is boring. It is the difference between a retrieval system that is trustworthy in year two and one that everybody quietly stopped believing.

FAQ

Why does a RAG system get worse over time even though nothing changed in the code? Because the documents it answers from keep changing, and the fix you shipped assumed the pile as it was. As documents are added, removed, or superseded, retrieval quality drifts, and without scheduled checks against a baseline, the decline is invisible until it is severe.

What is the difference between the two kinds of retrieval drift? The easy kind is when incoming questions start looking different, which is visible and detectable with standard tooling. The nasty kind is when the questions look identical but the correct answer changed underneath, for example an old policy document that was never removed. No input-side signal catches it, because the inputs did not change.

Why didn't my drift monitoring catch a stale document? Because input drift tools observe your incoming questions, and in this failure the questions did not change. The change was in the relationship between your index and a world that moved. Catching it requires a maintained golden set, source recency and provenance, reconciliation against the system of record, and a human sample.

Should a RAG system automatically retrain when it detects drift? No. At least half the time a drift alarm is a broken data feed, a changed file format, a failing parser, or a permissions change, not a real shift. Auto-retraining on that bakes the breakage in and converts a one-afternoon plumbing fix into a month-long model investigation. A human should look at why the alarm fired first.

How often should we re-check a retrieval system? On a fixed calendar rather than on suspicion, with the interval set by how fast your document base actually changes. The specific number matters much less than the fact that it is scheduled, tracked over time so a slow slide shows as a slope, and owned by someone.