Tagged rag
12 pieces
- · 7 min read
Your Hybrid Search Is Silently Dense-Only, and Reciprocal Rank Fusion Will Not Tell You
The widely copied hybrid search recipe pairs a vector index with Postgres full-text search and fuses the two with reciprocal rank fusion. Run a Chinese query against it and the full-text side returns zero rows, because there is no Chinese-capable text search configuration. Fusion then produces the dense-only ordering, with no error and no warning, and every test that checks for results still passes.
- · 8 min read
You Asked for 40 Rows and Got 4: What a Filter Does to an HNSW Vector Search
Add a WHERE clause to a vector search and the index does not search harder to compensate. It walks its usual neighbourhood, most of what it finds fails your filter, and you get a fraction of the rows you asked for. At a 10 percent selective filter on default ef_search, a reproduction on pgvector returned 3.93 rows on average against a request for 40. No error, no warning, just a short list that looks like a thin corpus.
- · 8 min read
Schema-Valid and Ten Thousand Times Wrong: Units Are Semantics, Not Syntax
A strict JSON schema on an extraction pipeline validates type, range and required fields. It cannot see that the number 12 in a table headed in ten-thousands means 120,000. The row passes every check, lands in the database as 12, and is wrong by four orders of magnitude with no validation error anywhere. The fix is an arithmetic cross-check against a total the document already states.
- · 6 min read
Retrieval or Fine-Tuning: How to Actually Decide
The decision is not about quality and it is not close. Retrieval changes what the model knows; fine-tuning changes how it behaves. Almost every team asking this question has a knowledge problem and is being sold a behaviour solution, and the tell is whether the right answer changes when your documents change.
- · 7 min read
Multi-Tenant Retrieval: Leakage Is a Design Decision
Cross-tenant leakage in a retrieval system is not an accident that happens to careless teams. It is the default outcome of a design that treats tenancy as a filter applied at query time, because a filter is something that can be forgotten, bypassed, or applied one layer too late.
- · 7 min read
Questions to Ask Before Hiring Anyone to Fix Your RAG System
Seven questions, in the order that sorts candidates fastest. The first three separate people who have run a retrieval system in production from people who have read about one, and the last one beats any pitch, because someone who has done this work can tell you exactly where their approach falls down.
- · 7 min read
Shadow and Canary: Shipping an AI Change Without Betting the Business
Two unglamorous techniques between a change and everyone. Shadow runs the new version against real traffic without showing anyone the results, so you find out how it behaves on production inputs before a single user sees it. Canary releases it to a small fraction with an automatic undo. The alternative, which is what most teams do, is to flip it on for everyone and hope.
- · 7 min read
Why You Can't Let an AI Grade Its Own Homework
Use the same model to write an answer and to judge it, and the loop closes on itself: it keeps the same blind spots twice, and it goes easy on itself in ways that are measurable rather than figurative. It prefers longer answers and its own writing style, and scores those higher. The single sharpest question to ask anyone building your evaluation is what is doing the checking.
- · 8 min read
Recall Is the Silent Killer of Your RAG System
If the answer lives in a document your system never pulled up, nothing downstream can rescue it. The model cannot quote a page that was never opened, so it answers confidently from whatever it did find, and the output looks exactly like a good answer. Recall is the one failure that leaves no trace in the output, which is why it is the first thing to measure and the last thing anyone does.
- · 10 min read
Retrieval or Generation? Why You Can't Tell Which Half of Your RAG System Broke
A wrong RAG answer is not one failure, it is two: the search pulled the wrong pages, or the search worked and the model ignored them. From the output they look identical, same confident tone, same citation, but they need opposite fixes. And one blended quality score averages the two into a green number that hides which half is on fire.
- · 9 min read
Index Freshness and Drift: Why a Working RAG System Quietly Goes Wrong
A RAG system that works today degrades quietly as its documents change. There are two kinds of drift: the easy one, where the questions start looking different, and the nasty one, where the questions look identical but the right answer changed underneath. The fix is re-checking on a schedule against a baseline, and never auto-rebuilding when the alarm fires, because half the time it is a data-plumbing break, not a real change.
- · 15 min read
How to Fix a Broken RAG System (Without Rebuilding the Model)
A RAG system that scored 95% in testing and collapses in production almost never needs a model rebuild. The failure is usually in one corner of the pipeline, and the fix is four cheap steps: slice the score to find the broken corner, measure recall precision and faithfulness separately, build a check that is not the thing being checked, and re-check on a schedule.