Retrieval-Augmented Generation

Six steps, usually drawn as one box labeled RAG

Most diagrams show retrieval-augmented generation as a single arrow: question in, grounded answer out. Every piece of that arrow is a separate step, built from the last two articles, and each one is a place quality can be won or lost independently of the others.

1def answer_question(query):
2    candidates = vector_index.query(embed(query), top_k=50)   # retrieve
3    reranked = rerank(query, candidates)[:5]                   # rerank
4    context = build_context(reranked)                          # assemble
5    return client.chat.completions.create(
6        model="gpt-4o",
7        messages=[
8            {"role": "system", "content": "Answer using only the provided context."},
9            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"},
10        ],
11    )                                                            # generate

Ingestion and chunking happened earlier, offline, to build the index this function queries. Embedding, retrieval, and reranking were the subject of the last article. What's new here is the last step: handing the retrieved context to the model with an explicit instruction to answer from it, and what happens when that context isn't actually enough to answer the question.

RAG doesn't remove hallucination, it moves what gets hallucinated about

"Grounded in retrieved documents" sounds like it should eliminate made-up answers, and it removes exactly one cause of them, the model inventing a fact it never actually knew. It does nothing about a second cause: the retrieved context being incomplete, mismatched, or subtly wrong, which the model has no way to detect on its own.

Feed the model five chunks that are topically related but don't actually answer the question, the exact failure mode from the last article's discussion of similarity search, and it doesn't respond "the context doesn't cover this." It answers anyway, blending whatever the context does say with its own general knowledge, confidently, in a single fluent paragraph that reads exactly like a well-grounded answer.

This is a more dangerous failure than a model with no retrieval at all making something up, because the presence of citations and retrieved text creates an appearance of rigor the answer hasn't actually earned. A prompt that explicitly permits "say you don't know if the context doesn't contain the answer" helps, but only if the retrieval and reranking steps upstream are actually good enough that most failures are rare edge cases, not the everyday result of weak retrieval papered over by a confident model.

Retrieve wide and cheap, then rerank narrow and precise

A cross-encoder, a model that scores a query and a candidate document together rather than comparing two precomputed vectors, is more accurate at judging real relevance than embedding similarity. It's also too slow to run against an entire corpus, scoring every document takes a full model pass per document, not a vector comparison.

The practical pattern uses both, at different scales. Vector search retrieves a wide net of candidates cheaply, fifty rather than five, then a cross-encoder rescoring pass runs only against that fifty and picks the true top five to actually send to the model.

1candidates = vector_index.query(embed(query), top_k=50)  # cheap, wide, approximate
2reranked = cross_encoder.score(query, candidates)          # precise, narrow, slower
3top_five = sorted(reranked, key=lambda r: r.score, reverse=True)[:5]

Fifty cross-encoder calls per query is a real cost, but it's a fixed, bounded one, unlike running the same model against a corpus of millions. Whether it's worth adding depends on whether the last article's similarity-search gap, related but not useful, is actually showing up often enough in your own retrieval results to justify it, which is exactly the kind of thing the evaluation techniques covered later in this series exist to measure rather than guess at.

RAG is for facts that live in a specific, changing corpus

RAG earns its complexity for one specific job: answering from information that lives in documents you control and that changes over time, your product docs, your internal wiki, this quarter's policy update. That's a genuinely different problem from what a model already knows, and a genuinely different problem from what fine-tuning solves, covered in full later in this series.

It's the wrong tool in a few specific shapes that are worth recognizing before building the pipeline. A task that's really about reasoning, working through a multi-step problem, doesn't improve because you handed it more retrieved paragraphs. A task that needs broad general knowledge the model already has doesn't need retrieval at all, and adding it just adds latency and a place for irrelevant context to creep in. And a task that genuinely needs to synthesize across hundreds of documents at once runs into the same context-window ceiling from earlier in this series, retrieval narrows the field, it doesn't remove the limit.

The honest test before reaching for RAG: is the answer actually written down somewhere in a document that changes over time, and would a person with access to that document but no other context be able to answer the question. If not, retrieval isn't the missing piece.