RAG that holds up in production

The model never saw the answer

When a RAG assistant gets something wrong, the first instinct is to blame the model. Teams swap in a bigger one, rewrite the prompt, add "do not make things up" in capital letters. In our experience that is often the wrong fix. The model answered badly because the paragraph that held the answer never reached it.

Retrieval-augmented generation, RAG for short, is a simple idea¹. Before the model answers, you search your own documents, paste the best matches into the prompt and ask the model to answer from them. The first version is easy to build: split the documents into chunks, turn each chunk into an embedding, store the embeddings in a vector database and fetch the five closest ones for every question. The research literature calls this "naive RAG"², and on a demo with ten hand-picked questions it feels like magic.

Then real users arrive. They ask about the exception, not the rule. They paste an error code. They use the name the company dropped two years ago. Researchers at Deakin University who built RAG systems for research, education and biomedical question answering wrote down where those systems broke, and found seven distinct failure points³. Three of the seven happen before the model is even called.

The list is worth reading slowly. In order: the answer isn't in the documents, but the system answers anyway. The answer exists but ranks too low, say sixth when only five chunks are kept. It is retrieved, then dropped when the prompt is assembled. It reaches the model, which fails to pull it out of the noise. The last three are about the reply itself: the wrong format, the wrong level of detail, or only half the answer. The same paper makes a point we agree with from our own projects: you can only really validate a RAG system once real questions hit it, and its robustness grows over time rather than being designed in on day one³.

One check comes before everything else. If the whole knowledge base is under about 200,000 tokens, roughly 500 pages, you may not need retrieval at all: put all of it in the prompt and let prompt caching keep it fast and cheap⁴. When retrieval is needed, the rest of this post follows a question through the system in the order we fix things: how documents are cut, how they are searched, how results are re-ordered, what goes into the prompt, how the answer shows its sources, and how we know any of it helped.

Chunking decides what can be found

Every RAG system starts by cutting documents into pieces small enough to search and to fit in a prompt. It sounds like plumbing. In practice it decides what the model can ever be shown.

The common default is a fixed size: every 500 or 800 tokens, sometimes with an overlap. It is easy, and it ignores the document completely. A rule that spans two sentences gets cut between them. A table loses its header row. A clause that says "except in the cases above" lands in a chunk without the cases. When Chroma's researchers compared chunking strategies on the same retrieval task, some beat others by up to 9% in recall. The default OpenAI documented for its file search at the time, 800 tokens with a 400-token overlap, came out slightly below average on recall and last on every other measure⁵.

What works better is to cut along the structure the author already gave the document: headings, sections, list items, table rows. A section about carrying over leave should be one chunk, not two halves. Where a section is long, we split it on paragraph boundaries and repeat the heading on every piece.

Structure alone leaves one problem. A chunk that says "up to 10 unused days carry into the next year" is correct, but it doesn't say which company, office or year it belongs to. If the knowledge base holds leave policies for two offices, their carry-over sections look almost identical to a search engine. Anthropic's fix is contextual retrieval: before indexing, a model writes a short note for each chunk, usually 50 to 100 tokens, explaining where it sits in the document, and the note is indexed together with the chunk⁴. Try the three approaches on one short policy below.

In Anthropic's tests, adding that context cut the share of failed retrievals by 35%⁴. It costs money once, at indexing time: they estimated $1.02 per million document tokens using prompt caching⁴. For most business knowledge bases that is a small, one-off bill, and an easy one to justify against a single wrong answer given to a customer.

Keywords and meaning, searched together

Embeddings search by meaning. Ask "how do I get my money back?" and an embedding search can find the refund policy even though the words "money back" appear nowhere in it. That is why vector search became the default for RAG. On open-domain question answering it beat keyword search by a wide margin: the Dense Passage Retrieval paper reported top-20 accuracy 9 to 19 points higher than a strong BM25 baseline⁶.

But meaning is not all that users type. They type error codes, invoice numbers, SKUs, section numbers and names. To an embedding model, "E-4012" is a fuzzy point somewhere near other error codes. A keyword index knows exactly which documents contain that string, which is why Anthropic singles out unique identifiers and technical terms as the queries where BM25 helps most⁴. BM25, the classic keyword ranking function, is also hard to beat on text it has never seen. The BEIR benchmark tested retrievers on 18 datasets they weren't trained on and found BM25 a robust baseline, while dense retrievers often did worse⁷.

So we run both. Keyword search and vector search each return a ranked list, and we merge the two. The simplest merge that works well is reciprocal rank fusion, or RRF: every list gives a document 1 / (60 + its rank), and the scores add up⁸. It ignores the raw scores, which is a feature, because BM25 scores and cosine similarities sit on different scales and don't compare directly. A document that both methods like rises to the top.

The 60 isn't magic. The authors found it near-optimal in their tests and said the exact choice was not critical⁸, and Elasticsearch ships the same default⁹. In Anthropic's tests, adding a contextual BM25 index next to contextual embeddings took the reduction in failed retrievals from 35% to 49%⁴.

The embedding model is a choice too. The MTEB benchmark found that no single embedding method wins on every task¹⁰, so a leaderboard gives us a shortlist, not an answer. We test two or three candidates on a sample of the client's own documents and questions, written the way their users actually write.

The question itself can be improved before it is searched. Users write short, vague queries; documents are written in long, formal sentences. A model can turn a follow-up like "and for contractors?" into a full question using the chat history, or split a question that asks two things into two searches. HyDE goes further: it asks a model to write a hypothetical answer, embeds that, and searches with it, because a made-up answer looks more like the real document than the question does¹¹. The made-up details don't matter, since the search only uses the text to find real documents nearby. Every rewrite adds a model call before retrieval, so we add one only where the evaluation shows it helps.

Rerank, then send less

First-stage search is built for speed. The embedding model encodes the question and each chunk separately, so chunk vectors can be computed once and searched in milliseconds. The price is that the question and the chunk never meet.

A cross-encoder reranker reads the question and one candidate chunk together and scores how well that chunk answers that question. Because it attends across both texts at once, it is more accurate, and far too slow to run over a whole collection¹². So the two work as a funnel: retrieve 50 to 150 candidates cheaply, rerank them, keep the best handful. The idea has a strong track record. In early 2019 a BERT reranker reached the top of the MS MARCO passage ranking leaderboard, beating the previous best by 27% on its main metric¹³, and on BEIR, rerankers had the best zero-shot results, at a higher compute cost⁷.

Anthropic's numbers show how the pieces stack. Retrieving 150 candidates, reranking them and keeping the top 20 cut the failure rate to a third of where it started⁴.

Reranking is also what makes the next step safe: sending less. It is tempting to fill a large context window with everything that might be relevant, but long contexts don't work that way. In "Lost in the middle", Liu and colleagues placed the one document holding the answer at different positions among 20 retrieved documents¹⁴. Accuracy was highest when it came first or last and sagged in between. In the middle, GPT-3.5-Turbo scored 53.8%, lower than the 56.1% it scored with no documents at all¹⁴.

Those are 2023 models, so we don't treat the exact numbers as current. The design lesson still holds, and it is cheap to follow. The same paper found that giving the model 50 documents instead of 20 improved accuracy by only about 1.5% while making every request longer¹⁴. Anthropic, on the other hand, found 20 chunks worked better than 10 or 5 in their setup⁴. The right number depends on the data, which is the point: we treat context as a budget we measure. A fixed number of reranked chunks, the strongest first, a cap per source so one long document can't crowd out the rest, and nothing added just because there was room.

Answers that show their sources

Even with the right chunks in the prompt, a model can blend them with what it remembers from training, or stretch a passage to cover a question it doesn't quite answer. Users can't tell the difference by reading. That is why we treat citations as a product feature, not decoration.

Generative search engines show what happens otherwise. When Liu, Zhang and Liang audited four popular ones, only 51.5% of generated sentences were fully supported by their citations, and only 74.5% of citations supported the sentence they were attached to¹⁵. The answers looked sourced. Often they weren't.

A grounded answer needs three things from the system, not just from the prompt. Number every chunk in the prompt and ask for the number after each claim. Check the citations in code, because a citation that points to a chunk that was never sent is a bug you can catch before anyone sees it. And give the model a clean way out: when the chunks don't contain the answer, the right reply is "I couldn't find this", plus where to ask. That covers the first failure point on the list, missing content, which no amount of retrieval tuning can fix.

import re


def build_prompt(question, chunks):
    sources = "\n\n".join(
        f"[{i}] {c.title}\n{c.text}"
        for i, c in enumerate(chunks, start=1)
    )
    return f"""Answer using only the numbered sources.
Cite the source after each claim, like [2].
If the sources don't answer it, say you couldn't find it.

{sources}

Question: {question}"""


def unknown_citations(answer, chunks):
    cited = {int(n) for n in re.findall(r"\[(\d+)\]", answer)}
    return cited - set(range(1, len(chunks) + 1))

In the interface, the citation should sit right next to the claim and open the exact passage, so checking an answer takes one tap. The refusal needs as much care as the answer.

Tune the refusal in both directions. Make the instruction too loose and the model fills gaps from memory; make it too strict and it refuses questions the documents do answer. You find the balance with real questions, which brings us to the part that holds everything else together.

Measure every change

Every technique in this post can make things worse on your data. A smaller chunk helps one kind of question and hurts another. A reranker fixes the order and adds latency. Query rewriting rescues vague questions and occasionally rewrites a precise one into something else. Without measurement, RAG tuning turns into opinion: someone tries five questions, likes the answers and ships.

So we build the evaluation set before we tune anything: real questions from the people who will use the system, each with a reference answer and the document it comes from, plus questions the documents can't answer. Then we score retrieval and generation separately, because a single "accuracy" number can't tell you what to fix. RAGAS, an open-source evaluation framework, includes metrics that split the problem cleanly^16,17.

These metrics use a model as the judge, so they cost money and they aren't perfect. In the RAGAS paper's tests, the metrics matched human annotators' side-by-side judgements 95% of the time for faithfulness, 78% for answer relevance and 70% for context relevance¹⁶. That is good enough to catch regressions and compare two versions. It isn't good enough to stop reading answers yourself, so we still review a sample by hand.

The workflow is boring on purpose. Change one thing. Run the set. Compare the four numbers with the last run. Keep the change only if it helps the questions that matter without hurting the rest. Then keep feeding the set with real questions from production, especially the ones users marked as unhelpful. That is the robustness Barnett and his colleagues described: it grows over time, run by run³.

Users only ever see the answer. Almost everything that makes it right happens before the model writes a word.