When a RAG system gives a wrong answer, the instinct is to blame the model or rewrite the prompt. In practice the answer was usually doomed before generation: the retriever never surfaced the passage that contained the fact. If the right chunk is not in the context window, no amount of prompt engineering will conjure it.
The useful mental model is a two-stage pipeline where retrieval quality caps everything downstream. Fix retrieval first, and generation problems shrink to the ones actually worth solving.
Chunking is a retrieval decision, not a formatting one
Fixed 500-token windows split tables mid-row and separate a claim from the sentence that qualifies it. Chunk on structure — headings, list items, table rows — so each unit is independently answerable. Store enough surrounding metadata that a matched chunk can be traced back to its document and section.
- Chunk on semantic boundaries, not a fixed character count.
- Attach source, section and page metadata to every chunk for citation and filtering.
- Overlap adjacent chunks slightly so a fact split across a boundary is still recoverable.
- Keep a rough token budget per chunk so a handful fit the context window with room to spare.
Hybrid search beats pure vectors more often than people admit
Dense embeddings are strong at paraphrase and weak at exact tokens — part numbers, error codes, proper nouns. A user searching for 'ERR_4021' wants a keyword match, not the semantically nearest neighbour. Combining BM25 with vector search and fusing the ranks recovers the cases each method misses on its own.
Add a reranker on the top 20 to 50 candidates. A cross-encoder that scores query and passage jointly is slower per pair but dramatically sharpens the final ordering, and you only run it on a short list.
Measure retrieval separately from generation
Build a labelled set of questions with the passages that should answer them, then track recall at k for the retriever alone. This isolates where a regression came from: a drop in retrieval recall and a drop in answer quality are different bugs with different fixes.
- Track recall at k and mean reciprocal rank on a fixed evaluation set.
- Log which chunks were retrieved for every production answer so failures are reproducible.
- Re-embed and re-index when you change the embedding model — mixing model versions silently corrupts similarity.
Say 'I don't know' on purpose
A retriever that returns nothing relevant should trigger an abstention, not a confident hallucination. Set a similarity floor below which the system declines to answer and offers to escalate. Users forgive 'I couldn't find that' far more readily than an invented answer delivered with certainty.