Tagged rag
6 pieces
- · 7 min read
Shadow and Canary: Shipping an AI Change Without Betting the Business
Two unglamorous techniques between a change and everyone. Shadow runs the new version against real traffic without showing anyone the results, so you find out how it behaves on production inputs before a single user sees it. Canary releases it to a small fraction with an automatic undo. The alternative, which is what most teams do, is to flip it on for everyone and hope.
- · 8 min read
Recall Is the Silent Killer of Your RAG System
If the answer lives in a document your system never pulled up, nothing downstream can rescue it. The model cannot quote a page that was never opened, so it answers confidently from whatever it did find, and the output looks exactly like a good answer. Recall is the one failure that leaves no trace in the output, which is why it is the first thing to measure and the last thing anyone does.
- · 7 min read
Why You Can't Let an AI Grade Its Own Homework
Use the same model to write an answer and to judge it, and the loop closes on itself: it keeps the same blind spots twice, and it goes easy on itself in ways that are measurable rather than figurative. It prefers longer answers and its own writing style, and scores those higher. The single sharpest question to ask anyone building your evaluation is what is doing the checking.
- · 10 min read
Retrieval or Generation? Why You Can't Tell Which Half of Your RAG System Broke
A wrong RAG answer is not one failure, it is two: the search pulled the wrong pages, or the search worked and the model ignored them. From the output they look identical, same confident tone, same citation, but they need opposite fixes. And one blended quality score averages the two into a green number that hides which half is on fire.
- · 9 min read
Index Freshness and Drift: Why a Working RAG System Quietly Goes Wrong
A RAG system that works today degrades quietly as its documents change. There are two kinds of drift: the easy one, where the questions start looking different, and the nasty one, where the questions look identical but the right answer changed underneath. The fix is re-checking on a schedule against a baseline, and never auto-rebuilding when the alarm fires, because half the time it is a data-plumbing break, not a real change.
- · 15 min read
How to Fix a Broken RAG System (Without Rebuilding the Model)
A RAG system that scored 95% in testing and collapses in production almost never needs a model rebuild. The failure is usually in one corner of the pipeline, and the fix is four cheap steps: slice the score to find the broken corner, measure recall precision and faithfulness separately, build a check that is not the thing being checked, and re-check on a schedule.