RAG in Production: What Breaks After the Demo

Every RAG demo works. Embed some PDFs, wire up a similarity search, pipe the chunks into a prompt — the answers look great on the five questions you tried. Then real users arrive with real data that changes hourly, and a different system is required. We run RAG over nine live business sources in production; this is the list of what broke and what we did about it.

Failure 1: Stale Data Destroys Trust Fastest

The first serious complaint we got wasn't a hallucination — it was the agent citing a task as "in progress" that had been closed that morning. One confidently wrong answer about data the user could verify cost more trust than any "I don't know."

Fix: sync workers on per-source cadences (tasks every 10 minutes, email every 15, documents hourly, chat every 30), and — critically — every answer shows the sync timestamp of its sources. "As of 12 minutes ago" changes how users read the answer.

Failure 2: Retrieval Finds Text, Not Answers

"What's the status of the Meridian contract?" retrieves eight chunks that mention Meridian and contracts — from four months of history, contradicting each other. The model synthesizes a plausible blend. Fixes that moved the needle:

• Recency weighting in the ranking function for status-type queries
• Metadata pre-filtering by entity and source before similarity
• Timestamps injected into every chunk so the model can reason about which fact is current

Failure 3: Permissions as an Afterthought

A shared vector index happily retrieves the salary spreadsheet for anyone who asks the right question. Permission scope must be a metadata field on every vector, enforced as a hard filter inside the retrieval query — never post-filtered in application code, and never left for the prompt to handle. We treat any retrieval path without a permission filter as a security bug.

Failure 4: No One Defined "Good"

For the first month, quality debates were vibes. Then we built a 120-query eval set from real logged questions with known correct answers, scored on every retrieval change. Arguments about chunk size, top-k, and rerankers ended the same week. Nothing about your pipeline is real until it moves a number on an eval set drawn from actual user queries.

What the Production Stack Actually Looks Like

Failure 5: The "No Answer" Path Doesn't Exist

Demos never test the question with no answer in the corpus. Production is full of them, and default LLM behavior is to improvise from whatever weakly-related chunks came back. We added an explicit gate: if the top result scores below a similarity threshold we tuned on the eval set, the agent says what it searched and found nothing, and suggests where the answer might live. Users rate these responses higher than mediocre real answers. Saying "I don't know" well is a feature you have to build.

Failure 6: One Source Goes Down, Everything Looks Broken

Third-party APIs fail weekly — rate limits, expired OAuth tokens, schema changes. Early on, one dead connector made the whole agent seem broken because answers silently lost a source. Now each connector has independent health status, sync failures alert us before users notice, and answers note when a relevant source is degraded.

The demo is the retrieval algorithm. The product is everything wrapped around it: freshness, permissions, provenance, and honest failure.

Budget accordingly. In our experience the pipeline that impressed everyone in week one was about 15% of the code that eventually shipped — and the other 85% is where the trust comes from.