Tagged retrieval
12 pieces
- · 7 min read
Your Hybrid Search Is Silently Dense-Only, and Reciprocal Rank Fusion Will Not Tell You
The widely copied hybrid search recipe pairs a vector index with Postgres full-text search and fuses the two with reciprocal rank fusion. Run a Chinese query against it and the full-text side returns zero rows, because there is no Chinese-capable text search configuration. Fusion then produces the dense-only ordering, with no error and no warning, and every test that checks for results still passes.
- · 10 min read
Plant the Faults So the Harness Proves It Can See Them
A harness that has never detected a known fault has not been shown to detect anything. Injecting deliberate, known faults is the step that converts a green suite into a suite that can fail for the right reasons. This covers the fault classes worth planting in an extraction or retrieval pipeline, how each one should present, and why a planted fault is the positive half of a pair whose other half is the negative control.
- · 8 min read
You Asked for 40 Rows and Got 4: What a Filter Does to an HNSW Vector Search
Add a WHERE clause to a vector search and the index does not search harder to compensate. It walks its usual neighbourhood, most of what it finds fails your filter, and you get a fraction of the rows you asked for. At a 10 percent selective filter on default ef_search, a reproduction on pgvector returned 3.93 rows on average against a request for 40. No error, no warning, just a short list that looks like a thin corpus.
- · 10 min read
Score Each Stage or You Are Tuning Blind
A multi-stage pipeline judged only on its combined output cannot tell you which stage contributed nothing. An extraction error, a dead retrieval branch, a fusion step absorbing an empty list and generation failing on good context all produce the same thin answer. Per-stage instrumentation means candidate counts per branch, per-stage scoring against a source-derived reference, and attribution before any improvement work starts.
- · 6 min read
Retrieval or Fine-Tuning: How to Actually Decide
The decision is not about quality and it is not close. Retrieval changes what the model knows; fine-tuning changes how it behaves. Almost every team asking this question has a knowledge problem and is being sold a behaviour solution, and the tell is whether the right answer changes when your documents change.
- · 7 min read
Questions to Ask Before Hiring Anyone to Fix Your RAG System
Seven questions, in the order that sorts candidates fastest. The first three separate people who have run a retrieval system in production from people who have read about one, and the last one beats any pitch, because someone who has done this work can tell you exactly where their approach falls down.
- · 7 min read
Shadow and Canary: Shipping an AI Change Without Betting the Business
Two unglamorous techniques between a change and everyone. Shadow runs the new version against real traffic without showing anyone the results, so you find out how it behaves on production inputs before a single user sees it. Canary releases it to a small fraction with an automatic undo. The alternative, which is what most teams do, is to flip it on for everyone and hope.
- · 7 min read
Why You Can't Let an AI Grade Its Own Homework
Use the same model to write an answer and to judge it, and the loop closes on itself: it keeps the same blind spots twice, and it goes easy on itself in ways that are measurable rather than figurative. It prefers longer answers and its own writing style, and scores those higher. The single sharpest question to ask anyone building your evaluation is what is doing the checking.
- · 8 min read
Recall Is the Silent Killer of Your RAG System
If the answer lives in a document your system never pulled up, nothing downstream can rescue it. The model cannot quote a page that was never opened, so it answers confidently from whatever it did find, and the output looks exactly like a good answer. Recall is the one failure that leaves no trace in the output, which is why it is the first thing to measure and the last thing anyone does.
- · 10 min read
Retrieval or Generation? Why You Can't Tell Which Half of Your RAG System Broke
A wrong RAG answer is not one failure, it is two: the search pulled the wrong pages, or the search worked and the model ignored them. From the output they look identical, same confident tone, same citation, but they need opposite fixes. And one blended quality score averages the two into a green number that hides which half is on fire.
- · 9 min read
Index Freshness and Drift: Why a Working RAG System Quietly Goes Wrong
A RAG system that works today degrades quietly as its documents change. There are two kinds of drift: the easy one, where the questions start looking different, and the nasty one, where the questions look identical but the right answer changed underneath. The fix is re-checking on a schedule against a baseline, and never auto-rebuilding when the alarm fires, because half the time it is a data-plumbing break, not a real change.
- · 15 min read
How to Fix a Broken RAG System (Without Rebuilding the Model)
A RAG system that scored 95% in testing and collapses in production almost never needs a model rebuild. The failure is usually in one corner of the pipeline, and the fix is four cheap steps: slice the score to find the broken corner, measure recall precision and faithfulness separately, build a check that is not the thing being checked, and re-check on a schedule.