Retrieval-augmented generation demos well and degrades quietly. The prototype answers the ten questions it was built against; six months later the corpus has changed, the questions have moved on, and nobody can say whether quality dropped because the retrieval stopped finding the right documents or because the model changed. The engineering work that prevents this is mostly unglamorous data plumbing and measurement.
Retrieval quality is a separate problem from generation quality
Treat them as two systems with two measurements. If the correct passage was never retrieved, no amount of prompt work will fix the answer, and tuning the prompt will make the underlying problem harder to see. Measure recall of the relevant passage in the top-k results independently of whether the final answer was judged good.
Chunking follows document structure, not a token count
- Split on structural boundaries — headings, sections, table rows, function definitions — before falling back to size limits.
- Carry the parent context (document title, section path, effective date) into each chunk's text, not only its metadata.
- Keep tables and code intact; a table sliced across chunks retrieves badly and reads worse.
- Store a stable chunk identifier so evaluation sets survive a re-index.
Hybrid retrieval and reranking beat embedding-only search
Dense vectors handle paraphrase well and exact identifiers badly. Real corpora are full of part numbers, error codes, policy names and version strings that users type verbatim. Combining lexical search with vector search and then reranking the merged candidate set is a consistently better default than either method alone, at the cost of one extra hop of latency.
Freshness and permissions are correctness requirements
Two failure modes damage trust faster than a mediocre answer: citing a superseded document as current, and surfacing content the user is not entitled to see. Both are index-side problems.
- 1.Re-index incrementally on source change events rather than on a nightly full rebuild; track and alert on index lag.
- 2.Delete and tombstone removed source documents — orphaned chunks outlive their sources and keep being cited.
- 3.Store the source access control list on each chunk and filter at query time by the caller's identity, before the model sees anything.
- 4.Show the source, its date and a link on every answer, so a stale result is visible rather than confidently wrong.
Permission filtering applied after generation is not permission filtering. The retrieved text must be scoped to the caller before it reaches the model context.
Build the evaluation set before the launch, not after the complaint
A few hundred question-and-expected-source pairs, drawn from real user questions and reviewed by someone who knows the domain, is enough to make changes measurable. Run it in CI against every change to chunking, embedding model, retrieval parameters or prompt, and record retrieval recall and answer judgments separately.
- Version the evaluation set and treat regressions as build failures, not discussion topics.
- Log queries that returned low-confidence or no results; they are the highest-value additions to the set.
- Re-run the full set before any model or embedding upgrade — embeddings are not comparable across versions and require a full re-index.
- Sample production traffic for periodic human review; automated judgments drift in the same direction as the system being judged.
Retrieval systems stay accurate because someone measures them continuously, not because the initial architecture was clever. The instrumentation is the deliverable as much as the pipeline is.