Evaluating RAG Systems

Retrieval-augmented generation (RAG) means fetching documents first and then answering from them. It's the most common way LLMs are used in production, so it's the system you're most likely to be asked to evaluate. Half the work is classic information retrieval, which search engineers already know well.

The idea that makes RAG manageable is that it has two places to fail, and you measure them separately before you measure them together. When an answer is wrong, first ask whether retrieval failed or generation failed. The fixes are different, and an end-to-end score can't tell them apart.

Stage 1: retrieval

Did the right documents reach the model's context? This is standard search evaluation: recall@k, precision@k, MRR and nDCG. If retrieval misses, no prompt will rescue the answer, because the facts were never in front of the model.

Stage 2: generation

Given the context, was the answer good? The usual checks are faithfulness (is every claim supported by the context?), answer relevance (does it address the question?), context precision (was the retrieved material useful, or mostly padding?) and citation accuracy (do the cited sources back up what they're cited for?).

Both cases look like "a wrong answer" to the user, and they need opposite fixes.

Lab code · runs free (mock judge by default)

USE_REAL_API = False        # free mock by default; True needs a key

def recall_at_k(retrieved, gold, k):
    top = retrieved[:k]
    return len(set(top) & set(gold)) / len(gold) if gold else 0.0

if USE_REAL_API:
    import anthropic
    from pydantic import BaseModel
    client = anthropic.Anthropic()
    class Grounded(BaseModel):
        reason: str
        grounded: bool
    def faithful(context, answer):
        r = client.messages.parse(model="claude-opus-4-8", max_tokens=300,
            messages=[{"role": "user", "content":
                f"Is every claim in the ANSWER supported by the CONTEXT? "
                f"Reason, then grounded: true/false.\n\nCTX:\n{context}\n\nANS:\n{answer}"}],
            output_format=Grounded)
        return r.parsed_output.grounded
else:
    def faithful(context, answer):
        return "[INVENTED]" not in answer    # mock: ungrounded if it invents

CASES = [   # (query, retrieved_ids, gold_ids, context, answer)
    ("refund window?", ["d2","d9","d4"], ["d1"], "",
     "You have 30 days. [INVENTED]"),
    ("reset password?", ["d5","d1","d7"], ["d5"], "Click 'Forgot password'.",
     "Click 'Forgot password' and check your email."),
    ("export data?", ["d8","d3"], ["d8"], "Export is on the Settings page.",
     "Data export is under Billing. [INVENTED]"),
]

for q, retrieved, gold, ctx, ans in CASES:
    r = recall_at_k(retrieved, gold, k=3)
    if r < 1.0:
        verdict = "RETRIEVAL -> fix chunking / embeddings / reranking"
    elif not faithful(ctx, ans):
        verdict = "GENERATION -> fix prompt / grounding"
    else:
        verdict = "OK"
    print(f"{q:16} recall@3={r:.2f}  {verdict}")

refund window?   recall@3=0.00  RETRIEVAL -> fix chunking / embeddings / reranking
reset password?  recall@3=1.00  OK
export data?     recall@3=1.00  GENERATION -> fix prompt / grounding