Retrieval-augmented generation (RAG) is deceptively simple to demo and genuinely hard to get right at enterprise scale. A few lessons from shipping RAG systems in regulated, high-stakes environments:
1. Chunking strategy matters more than model choice. Naive fixed-size chunking breaks context at the worst possible points. Structure-aware chunking — respecting headings, tables and document boundaries — consistently outperformed model upgrades in our evaluations.
2. Hybrid search beats pure vector search. Combining keyword (BM25) and vector retrieval catches both semantic matches and exact-term queries — critical when users search for a specific policy number or SKU.
3. Citations are not optional. Every production RAG system we’ve shipped surfaces the source passage. It’s the single biggest driver of user trust, and it turns “the AI was wrong” into a two-second fact-check instead of a crisis.
4. Permissions have to be enforced at retrieval, not just at the UI. If a document is restricted, the retriever must never fetch it for an unauthorized user — regardless of what the front end shows.
5. Stale knowledge is worse than no knowledge. Build the sync pipeline before you build the chat interface. An assistant confidently citing a policy that changed last month is a liability.
6. Evaluation has to be continuous, not a one-time benchmark. We run scheduled evaluation suites against production RAG systems, because both the underlying documents and user query patterns drift over time.
7. Budget for the un-glamorous 20%. Access control, monitoring, sync pipelines and evaluation tooling are most of the engineering effort in a serious RAG deployment — the retrieval-and-generate loop itself is the easy part.