RAG systems that survive production

A retrieval demo is easy. You take fifty clean documents, embed them, ask three questions you already know the answers to, and everyone applauds. Production is a different animal: ten thousand documents of wildly uneven quality, users who ask half-questions, and a business that will only forgive so many confident lies.

Retrieval is a ranking problem, not a search problem

Pure vector similarity finds text that sounds like the query. It does not find text that answers the query. The fix is hybrid: lexical search for exact identifiers, vector search for meaning, then a reranking pass that actually reads candidates before deciding what reaches the model.

• Chunk per document class, a contract and a changelog do not chunk the same way
• Store structural metadata (section, version, owner) and filter before you rank
• Rerank the top 50 down to the top 5; the extra latency buys accuracy you cannot get elsewhere
• Always return citations, and make the UI show them without the user asking

If you cannot measure it, you cannot ship it

Every production system I have shipped carries an eval harness: a golden set of real questions with graded answers, run on every change to prompts, chunking or models. Without it, a prompt tweak that feels better is indistinguishable from a regression.

The honest measure of an AI feature is not how good it is on its best day, but how bad it is on its worst.

Design for the failure case first

Grounded systems should refuse. A clean 'I do not have a source for that' preserves trust far longer than a plausible paragraph of invention. Build refusal, escalation and human handoff on day one, not as a patch after the first incident.