RAG Is a Search Problem Before It Is a Generation Problem

A language model cannot cite a document it never received. It cannot repair a paragraph that was split away from its definition, recover a table discarded during parsing, or distinguish the authoritative policy from five near-duplicates if the retrieval layer gives every version the same score.

Yet teams often debug retrieval-augmented generation from the rightmost edge of the architecture. They tune the prompt, change the model, add a longer context window, and ask the generator to “be more accurate.” Those changes can improve presentation. They cannot manufacture missing evidence.

Retrieval sets the ceiling

It helps to write the pipeline without the product language around it:

query
  → candidate documents
  → parsed content
  → chunks
  → retrieved candidates
  → ranked context
  → generated answer

Every arrow can lose information. A good answer requires the relevant source to survive all of them. The original Retrieval-Augmented Generation paper describes generation that combines parametric memory in the model with non-parametric memory in a retrievable index. In a production system, that non-parametric memory is not an abstract database. It is a chain of parsing, segmentation, indexing, filtering, and ranking decisions.

The first question in a bad-answer investigation should therefore be: Was sufficient evidence present in the context? If the answer is no, generation is downstream noise.

Chunking is a boundary problem

“Use 500-token chunks with 50-token overlap” sounds like a configuration choice. It is really a claim about the shape of knowledge in your corpus.

A chunk should be large enough to preserve the unit needed to answer a question and small enough to rank for that question. Those goals conflict. A tiny chunk may match the query precisely while omitting the exception in the next paragraph. A large chunk may preserve the exception while diluting the terms that make the passage retrievable.

Useful boundaries often come from the document itself:

• headings and their child paragraphs,
• table titles plus headers and rows,
• function signatures plus their documentation,
• policy clauses plus definitions and exceptions,
• conversation turns rather than arbitrary token windows.

Keep document identity and hierarchy as metadata. A chunk should know its source, section, version, timestamp, and access scope. That metadata is how you filter stale content, expand to a parent section, and explain where an answer came from.

Build a ranking stack, not a single search box

Dense retrieval is good at semantic similarity. Lexical retrieval is good at exact identifiers, rare terms, error codes, and names. Neither deserves to be the only door into the corpus.

A practical stack usually has several jobs:

1. Normalize the query. Resolve obvious abbreviations, preserve exact entities, and attach known filters.
2. Generate candidates. Use lexical and semantic retrieval where each is strong.
3. Fuse the lists. Combine rankings without pretending their raw scores are directly comparable.
4. Rerank a small set. Spend the expensive relevance model on dozens of candidates, not the entire corpus.
5. Assemble context. Deduplicate, enforce diversity, expand parent context when needed, and stay inside the token budget.

Query rewriting can help, but it should create additional retrieval attempts rather than erase the user's original language. The original query contains exact clues that a rewritten query may smooth away.

Evaluate the stages separately

End-to-end answer scores tell you that the system failed. They do not tell you where. Build an evaluation set with questions, acceptable sources, and the facts an answer must contain. Then evaluate at least two layers.

Retrieval evaluation

• Recall@k: did any acceptable evidence appear in the first k results?
• Rank quality: did the best evidence appear early enough to survive context assembly?
• Context coverage: did the final prompt contain every fact required for the answer?

Generation evaluation

• Does each material claim follow from the supplied context?
• Does the answer cover the required facts?
• Are citations attached to the claims they support?
• Does the model abstain when evidence is insufficient?

This separation makes experiments legible. If recall improves and answer quality does not, inspect ranking, context packing, or the generator. If answer quality improves while recall is flat, the change probably affected reasoning or presentation rather than retrieval.

def evaluate_case(case, system):
    candidates = system.retrieve(case.question)
    context = system.assemble(candidates)
    answer = system.generate(case.question, context)

    return {
        "evidence_recall": recall_at_k(candidates, case.sources, k=10),
        "context_coverage": coverage(context, case.required_facts),
        "groundedness": grounded_claims(answer, context),
        "completeness": covered_facts(answer, case.required_facts),
    }

A useful failure taxonomy

Logging the query, retrieved identifiers, ranking scores, assembled context, answer, and citations turns this taxonomy into an operational tool. Without that trace, every error becomes “the AI was weird.”

The operating rule

When a RAG answer fails, walk left through the pipeline. Verify evidence in the final context, then the ranked set, then the candidates, then the parsed corpus. Stop at the first place the evidence disappears.

Prompt work still matters. Model choice still matters. But both become productive only after the system can reliably place the right facts in front of the model. Retrieval is not the prelude to the product. For knowledge-intensive AI, retrieval is the product.