Every RAG demo looks the same: upload a handful of clean PDFs, ask a friendly question, watch the model answer correctly. Then it goes to production, meets your actual document set, and the wheels come off.
The chunking decision nobody revisits
Most teams pick a chunk size once, during the demo, and never touch it again. But a 500-token chunk that works for a policy handbook falls apart on a table-heavy financial filing. The fix isn’t a bigger model, it’s treating chunking strategy as an engineering decision with its own test suite, not a default left over from a tutorial.
Freshness is a pipeline, not a checkbox
Retrieval systems quietly go stale the moment nobody owns re-indexing. We’ve seen production systems answering confidently from a knowledge base that was six months out of date, because “someone will re-run the embedding job eventually” is not a pipeline.
- Version your embeddings alongside your source documents
- Automate re-indexing on a real schedule, not a manual one
- Build an evaluation harness before you need one, not after an incident
The fix is architectural, not another prompt tweak
When a RAG system gives a wrong answer, the instinct is to patch the prompt. Usually the actual fault is upstream: bad chunking, stale indices, or no source-of-truth pipeline at all. Fix the retrieval layer first. The prompt was never the problem.