Building A Production RAG System: The Complete Playbook For Chunking, Retrieval, Reranking And Evaluation

Retrieval - The quality of a RAG answer is capped by the quality of what you retrieve - fix retrieval first  ·  Hybrid - Dense embeddings plus keyword search beats either alone on most real corpora  ·  Rerank - A cross-encoder reranker over the top candidates is the highest-ROI upgrade in most pipelines  ·  Eval - Without an evaluation harness you are tuning blind - build it before you optimise

Retrieval-augmented generation is one of the easiest things in AI to prototype and one of the hardest to make reliable. The prototype is a weekend project: chunk some documents, embed them, drop them in a vector store, stuff the top matches into a prompt. It demos well. Then it meets production - a corpus of millions of documents, users asking questions the naive pipeline retrieves the wrong context for, hallucinated answers with no citations, and no way to tell whether a change made things better or worse. This playbook walks the full production pipeline stage by stage, and ends where every serious RAG project should start: evaluation.

Stage 1: Ingestion And Chunking

RAG quality is decided before a single query runs, at ingestion. The goal of chunking is to produce passages that are self-contained enough to answer a question yet focused enough to embed cleanly. Fixed-size chunks with overlap are a fine default, but structure-aware chunking - splitting on document structure (headings, sections, semantic boundaries) rather than arbitrary character counts - almost always retrieves better, because it keeps coherent ideas together. Attach metadata to every chunk (source, section, date, permissions) because you will need it for filtering, citations and access control. And preserve enough context in each chunk - a section title, a parent summary - that a retrieved passage makes sense on its own.

• Chunk on structure where you can (headings, sections), falling back to fixed-size-with-overlap for unstructured text.
• Right-size chunks to your embedding model and content - too large dilutes relevance, too small loses context; test, do not guess.
• Attach rich metadata (source, section, timestamp, access level) to every chunk for filtering, citation and permission enforcement.
• Store the original document reference so you can cite precisely and let users click through to the source.

Stage 2: Retrieval - Go Hybrid

Dense vector search (semantic similarity via embeddings) is excellent at matching meaning but can miss exact terms - product codes, names, acronyms, rare keywords. Sparse keyword search (BM25) is the mirror image: great at exact matches, blind to paraphrase. On most real corpora, hybrid search - running both and fusing the results - beats either alone, because user questions mix conceptual and literal intent. Add metadata filtering (date ranges, source, permissions) at the retrieval layer so you never even consider chunks the user should not see or that are out of scope. Retrieval is the ceiling on answer quality: if the right passage is not in the retrieved set, no amount of prompt engineering downstream will save the answer.

Stage 3: Reranking - The Highest-ROI Upgrade

Initial retrieval optimises for recall: cast a wide net and pull, say, the top 20-50 candidates. But you only want to hand the LLM the few passages that genuinely matter. A reranking step - a cross-encoder that scores each candidate against the query with far more precision than the initial vector similarity - reorders those candidates so the best few rise to the top. In practice, adding a reranker over a decent retriever is the single highest-ROI change in most RAG pipelines: retrieve broadly, rerank precisely, then pass only the top few reranked passages into generation. It cuts irrelevant context, reduces hallucination, and lowers token cost all at once.

Stage 4: Grounded Generation With Citations

Generation is where discipline pays off. Instruct the model to answer only from the provided context, to cite which passage each claim comes from, and - critically - to say it does not know when the context does not contain the answer. A RAG system that confidently fabricates when retrieval fails is worse than one that says 'I could not find that in the sources', because the failure is invisible. Return citations to the user so answers are verifiable and trust is earned. This 'ground, cite, or refuse' contract is what separates a RAG system people can rely on from a plausible-sounding guess machine.

Stage 5: Evaluation - Build This First, Not Last

Here is the part almost everyone skips and every serious team regrets skipping: evaluation. Without a harness you are tuning blind - you change the chunk size, the demo looks better, and you have no idea whether real accuracy went up or down. Build an evaluation set of representative questions with known-good answers and known source passages, then measure the pipeline on the dimensions that matter: retrieval quality (did we fetch the right passages?), faithfulness (is the answer grounded in the retrieved context, or invented?), answer relevance (does it actually address the question?), and citation accuracy. Run it on every change. This is what turns RAG from guesswork into engineering.

• Retrieval metrics - did the correct passage appear in the retrieved set, and how highly was it ranked?
• Faithfulness / groundedness - is every claim in the answer supported by the retrieved context, or hallucinated?
• Answer relevance - does the response actually address the user's question?
• Citation accuracy - do the cited sources genuinely support the statements attributed to them?

Operating RAG Over Time

A production RAG system is not a one-time build; it is a living pipeline. Corpora grow and drift, user questions change, and embedding models improve. Keep ingestion incremental so new and updated documents flow in without a full rebuild, monitor retrieval and faithfulness metrics continuously, and re-run your evaluation set whenever you change any component. Get the pipeline right - structure-aware chunking, hybrid retrieval, precise reranking, grounded-and-cited generation, continuous evaluation - and RAG becomes exactly what it promises: a way to give a frontier model reliable, current, private knowledge without retraining it.

References & Further Reading

• Lewis et al. - Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (the original RAG paper): https://arxiv.org/abs/2005.11401
• Anthropic - Contextual Retrieval (improving retrieval accuracy): https://www.anthropic.com/news/contextual-retrieval
• Pinecone - Learn: hybrid search and RAG fundamentals: https://www.pinecone.io/learn/
• Ragas - open-source RAG evaluation framework: https://docs.ragas.io/
• Robertson & Zaragoza - The Probabilistic Relevance Framework: BM25 and Beyond: https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf