Questions to Ask Before Hiring Anyone to Fix Your RAG System

Mostafa Dhouib·September 6, 2026·7 min read

Questions to Ask Before Hiring Anyone to Fix Your RAG System

The short answer. Seven questions, ordered so the cheap ones sort first. Ask which half broke before anything else, because a candidate who cannot separate retrieval failure from generation failure will spend your budget on the wrong half. Then ask about recall, about the evaluation set, about drift, about what happens on a citation that does not support the claim, and finally what their approach misses. The last one is close to unfakeable.

A retrieval system that gives wrong answers has at least six places it can break and most teams have instrumented one. So the hiring problem is not finding someone who knows the components. It is finding someone who will locate the failure before proposing a fix.

These seven questions do that sorting, and none of them requires you to be technical.

One: how will you tell whether retrieval or generation broke?

Ask first, always. Everything else depends on the answer.

A RAG answer fails in one of two fundamentally different ways. Either the right passage was never retrieved, in which case the model was asked to answer from material that did not contain the answer and did the best it could. Or the right passage was retrieved and the model ignored it, contradicted it, or cited it while saying something else.

Those have different fixes and there is almost no overlap between them. Chunking, embeddings, and ranking are the first. Prompting, grounding constraints, and model choice are the second.

A good answer describes a procedure: take a set of questions with known correct sources, check whether the correct passage appears in what was retrieved, and only then look at what the model did with it. That splits the problem in an afternoon.

A bad answer goes straight to a fix. "We would move to a better embedding model" is an answer to a question nobody asked yet.

Two: what is our retrieval recall, and how would you measure it?

Recall is the share of questions where the passage containing the answer made it into the retrieved set at all. It is the ceiling on everything downstream: if the passage is not there, no prompt, no model, and no reranker recovers it.

It is also the number almost nobody measures, because measuring it requires knowing the correct source for a set of questions, and building that set is unglamorous work.

A good answer treats recall as the first measurement to establish and describes how they would build the question-to-source set, including who in your organisation would need to be involved.

A bad answer proposes improvements to ranking. Ranking rearranges what was retrieved. If the passage is not in the set, ranking has nothing to work with, and this is the single most common wasted quarter in retrieval work.

Three: what does your evaluation set look like, and who writes it?

A good answer has two properties. The questions come from real users rather than from the team that built the system, and the correct answers were established by someone with the domain knowledge to judge them, which is usually not the engineer.

A bad answer is that the model will grade the outputs. A model checking its own retrieval and generation shares the priors that produced the error, so where the system is confidently wrong the grader is confidently agreeable. That is a verifier that verifies nothing wearing an evaluation label.

Follow up with: how many questions, and how are they sliced? One aggregate number over a mixed set hides the slice where it fails, and the slice where it fails is usually the slice that matters most.

Four: how will we know when the index has gone stale?

Retrieval systems degrade without anyone changing anything. Documents get superseded, policies get revised, the language people use drifts away from the language in the corpus, and the index quietly starts returning things that were correct last year.

A good answer names a cadence and an owner, and describes what gets re-measured on that cadence: recall on the maintained question set, the age distribution of retrieved documents, and how often a retrieved passage comes from a superseded version.

A bad answer is that they will re-index regularly. Re-indexing keeps the corpus current. It says nothing about whether retrieval is still finding the right thing, and those are different failures.

Five: what happens when a citation does not support the claim?

This is the failure that damages trust fastest, because it is the one your users find rather than your metrics.

An answer with a citation attached that does not actually support it is worse than an answer with no citation, since the citation is what persuaded the reader to stop checking.

A good answer describes a check that runs on the output: does the cited passage contain the claim, verified mechanically rather than by asking the model whether it is confident. This is a rung above an opinion because it compares against something the system did not generate.

A bad answer is that the prompt instructs the model to cite accurately. A prompt is a request, and the component receiving it is nondeterministic.

Six: what would you do first, and what would it cost to find out you were wrong?

The point of this question is not the plan. It is whether the first step is cheap.

A good answer starts with a measurement that takes days and could invalidate their own hypothesis. Establish recall. Split the failures by half. Slice the evaluation by document type.

A bad answer starts with a migration. Changing the vector database, the embedding model, or the chunking strategy before knowing which half broke is an expensive way to gather no information, and it is the most common opening move in this field.

Seven: what does your approach miss?

The highest-signal question available, and close to unfakeable.

Someone who has run retrieval systems in production has watched their own method fail and can describe the conditions. "This works well on a stable document corpus and badly where the source of truth is a database that changes hourly." "We are strong on the retrieval side and weaker where the real problem is that the underlying documents contradict each other, which no retrieval system fixes."

Three answers should worry you:

Nothing. They have not run their method into its limits, or they will not say.

Only things outside their scope. "We do not do infrastructure" is a boundary, not a limitation.

A repackaged strength. "We are perhaps too rigorous about evaluation" is a sales answer, and it tells you which conversation you are in.

The order matters

Ask these in sequence rather than as a checklist, because the sequence itself is the test.

Questions one and two establish whether the candidate diagnoses before prescribing. That single trait predicts the outcome of the engagement better than any credential, because the expensive failure in retrieval work is almost never incompetence. It is a competent team improving the half that was already working.

FAQ

How do I hire someone to fix a RAG system? Ask how they will determine whether retrieval or generation broke before you ask anything about their solution. A candidate who proposes a fix before splitting the problem will improve the half that was already working, which is the most common way retrieval budgets are wasted.

What is retrieval recall and why does it matter most? It is the share of questions where the passage containing the answer made it into the retrieved set at all. It is the ceiling on everything downstream, because no prompt, model, or reranker can use a passage that was never retrieved. Most teams have never measured it.

Can we use a model to grade our RAG evaluation set? Not as the primary judge. A model grading retrieval and generation shares the priors that produced the error, so it agrees confidently exactly where the system is confidently wrong. Correct answers should be established by someone with the domain knowledge to judge them.

What is the wrong first move on a failing RAG system? Migrating the vector database, swapping the embedding model, or changing the chunking strategy before knowing which half broke. Those are expensive changes that gather no diagnostic information, and they are the most common opening move in this work.

What single question sorts candidates fastest? "What does your approach miss?" Someone who has run retrieval in production can name the conditions under which their method fails. Nothing, a list of out-of-scope items, or a repackaged strength are each informative answers in themselves.