This is Failure Modes #2, a little ahead of its first-Tuesday slot. One production failure pattern a month: how it looks, why it happens, and the fix. #1 covered silent degradation, and the Substack gets each installment the day it ships.
A RAG system gives a wrong answer in production. The post-mortem follows a script I could recite in my sleep: someone reads the prompt, someone suggests a better model, someone adds a line that says answer only from the provided context, and the ticket closes. Two weeks later the same class of wrong answer is back, because in most RAG failures the model did its job. It answered faithfully from the context it was given. The context was wrong, and the component that chose the context was never on trial.
Retrieval failures are the quietest failures in the stack. The vector database returned 200. The pipeline ran end to end. The answer came out fluent and formatted. Nothing in the infrastructure graph distinguishes a run that retrieved the right paragraph from a run that retrieved plausible garbage, which makes this a direct sequel to the silent degradation problem: no deploy, no alert, no stack trace. Here are the six ways I see it happen, roughly in order of how often I find them.
1. Top-k Always Returns k
Vector search is nearest-neighbor search, and nearest does not mean near. Ask for the five closest chunks and you get five chunks, whether the best of them answers the question or merely shares some vocabulary with it. Ask about something that is not in the corpus at all and you still get five chunks, now actively misleading, handed to a model that has been instructed to trust them.
This is the failure mode behind most hallucination complaints in RAG systems. The model gets blamed for inventing an answer when the honest description is that the retriever invented the evidence. The fixes are old and boring: a similarity floor below which chunks do not ship, an explicit no-answer path that fires when nothing clears the floor, and logged scores on every query. If you do not log the scores, you cannot see the difference between a confident retrieval and a shrug, and neither can anyone debugging the system after you.
2. Chunk Boundaries Cut the Answer in Half
When the answer to a question straddles a chunk boundary, neither half embeds like the whole, and the right chunk loses a similarity contest it should have won. This is invisible in any individual trace, because the retrieved chunks still look topical. It only shows up in aggregate, and the aggregate is expensive to ignore: on 800 real questions I benchmarked, naive fixed-size chunks found the right chunk at rank one 61.1 percent of the time, and structure-aware recursive splitting of the same corpus hit 70.8 percent at identical embedding cost. Nearly ten points of recall, free, and the naive splitter is still what half the tutorials ship.
The fix costs an afternoon. Split on structure, then look at your actual boundaries instead of trusting the splitter's defaults, and if you want numbers instead of eyeballs, run your own corpus through the benchmark lab and see where your boundaries fall before the embedding model does.
3. The Query and the Index Speak Different Embeddings
Vectors are only comparable inside the model that produced them. Embed the corpus with one model and the query with another, and cosine similarity quietly becomes a random number generator with good manners. Nothing throws. The shapes match, the math runs, and retrieval quality falls off a cliff that no error message marks.
Teams do not do this on purpose. It arrives as an embedding model upgrade applied to new documents only, a client library bump that changed the default model string, a second service embedding queries with its own config, or a partial backfill that ran out of budget halfway. The symptom is distinctive if you know to look: quality drops across the board right after an unrelated-looking deploy, with no single query that reproduces cleanly. The protections are mechanical. Stamp every vector with the model name and version that produced it, reject queries whose embedding config does not match the index, and treat an embedding model change like a schema migration: full reindex, all or nothing, never a blend.
4. The Index Is a Snapshot Nobody Rebuilds
Retrieval serves the corpus as it existed at index time, and index time recedes into the past one day per day. The pricing page changed, the policy was rewritten, the feature was deprecated, and the system keeps citing the old version with full confidence, because for the retriever the old version is simply what exists. Deletes are the worst case: the document someone removed for being wrong is still in the index, still retrievable, and now being served with a straight face by your support bot.
Users only need to catch this once to stop trusting the system, and they catch it on exactly the documents that change most, which are usually the ones that matter most. The fix is to stop treating the index as setup and start treating it as a build artifact: an owned refresh cadence matched to corpus churn, deletes propagated synchronously rather than on the nightly job, and a freshness timestamp carried on every chunk so staleness is at least visible when it reaches the model and the logs.
5. Cranking k Instead of Fixing Recall
The reflexive fix for retrieval misses is to raise top-k. Five chunks missed, so ship twenty. Recall at twenty is indeed higher than recall at five, so the eval that measures retrieval improves, while the answers mostly do not: the right chunk is now buried in the middle of a long context where models attend worst, wrapped in fifteen distractors the model has been told are relevant. And the cost moves in exactly the wrong direction, because input is the dominant term in what a task costs, and you just quadrupled the retrieved share of it as a workaround.
Cranking k treats a precision problem with volume. If the right chunk is not in the top five, the honest diagnosis is upstream: boundaries, embeddings, or staleness, the modes above. Fix those first. Where the candidate set genuinely is large and ambiguous, a reranker earns its latency: retrieve wide, rerank hard, and ship the model a short context that is actually relevant rather than a long one that is statistically relevant.
6. Filters That Quietly Empty the Set
Production RAG rarely searches one big happy corpus. There are tenant filters, permission filters, date filters, product-line filters, and every one of them interacts with k. Apply a strict filter after retrieving the top twenty and some queries now return three chunks, or none, at which point this collapses into failure mode one with extra steps: the system shrugs fluently and the logs show a successful request. Loosen the filter to make the misses stop and you get the opposite failure, which is a permissions leak wearing a relevance badge, serving one customer's documents to another.
Both directions are quiet. The fixes: filter inside the search where your engine supports it rather than trimming afterward, alert on the empty-after-filter rate because it is the cheapest canary in the whole pipeline, and keep a permission test suite of per-role golden queries, because access control that is only tested by production traffic is access control that is tested by incidents.
What Catches All Six
Every mode on this list shares one property: the request succeeds. HTTP 200, latency normal, answer fluent. No infrastructure monitor will ever page you for any of them, which is why the catching has to be built on purpose.
Measure retrieval separately from generation. One end-to-end quality number blends two failure sources into something unexplainable. Keep a golden set of queries with known correct chunks and track retrieval recall on its own, so that when quality moves you already know which half of the pipeline moved.
Re-run it on a schedule. Staleness, drift, and filter rot are functions of time, not deploys. A golden set that runs weekly catches what a pre-launch eval structurally cannot. Size it honestly: small suites cannot see small drops, and the sample size calculator will tell you what your suite can actually detect before you trust a green week.
Log the scores. Similarity scores, chunks served, filter counts, index version. The distribution of retrieval scores over time is the single cheapest early-warning signal in RAG, and almost nobody keeps it.
The retriever is the half of RAG that does not demo. It has no personality, it writes nothing quotable, and it decides more of your answer quality than the model does. Put it on trial first.
FAQ
Why does my RAG system give wrong answers with high confidence?
Usually retrieval handed the model wrong context and the model answered faithfully from it. Vector search returns k chunks whether or not anything relevant exists. A similarity floor and an explicit no-answer path fix the worst of it.
What is the most common RAG failure mode?
Quiet retrieval misses: the right information exists but is not surfaced, because of chunk boundaries, a stale index, or mismatched embedding versions. All of them return HTTP 200, which is why they persist.
How do you debug a RAG pipeline?
Separate retrieval from generation. Golden queries with known correct chunks, retrieval recall tracked per stage on a schedule, and logged similarity scores. Final-answer quality alone blends two failure sources into one number.
How often should you reindex?
Match refresh cadence to corpus churn, propagate deletes synchronously, and treat any embedding model change as a full reindex. The index is a build artifact, not a setup step.