Evaluating RAG Systems

Retrieval-Augmented Generation — fetch documents, then answer using them — is the most common production LLM architecture, so it's the one you're most likely to be asked to evaluate. The trick that makes it manageable: it has two separate failure surfaces.

The idea that makes RAG eval tractable: a RAG system has two failure surfaces, measured separately before together. When an answer is wrong, the first question is — did retrieval fail, or did generation fail? Different fixes; a single end-to-end number can't tell them apart.

Surface 1 — retrieval

Did the right documents reach the context? Classic IR: recall@k, precision@k, MRR, nDCG. If retrieval fails, no prompt cleverness saves the answer — the facts aren't in front of the model.

Surface 2 — generation (given the context)

• Faithfulness / groundedness — every claim supported by the context? (Anti-hallucination.)
• Answer relevance — does it address the question?
• Context precision — was retrieved context useful or padded?
• Citation accuracy — do cited sources support their claims?

Two identical-looking "wrong answers," two opposite fixes. Separating the surfaces is what turns RAG debugging from guesswork into a decision.

Lab code — runs free (mock judge by default)

USE_REAL_API = False        # free mock by default; True needs a key

def recall_at_k(retrieved, gold, k):
    top = retrieved[:k]
    return len(set(top) & set(gold)) / len(gold) if gold else 0.0

if USE_REAL_API:
    import anthropic
    from pydantic import BaseModel
    client = anthropic.Anthropic()
    class Grounded(BaseModel):
        reason: str
        grounded: bool
    def faithful(context, answer):
        r = client.messages.parse(model="claude-opus-4-8", max_tokens=300,
            messages=[{"role": "user", "content":
                f"Is every claim in the ANSWER supported by the CONTEXT? "
                f"Reason, then grounded: true/false.\n\nCTX:\n{context}\n\nANS:\n{answer}"}],
            output_format=Grounded)
        return r.parsed_output.grounded
else:
    def faithful(context, answer):
        return "[INVENTED]" not in answer    # mock: ungrounded if it invents

CASES = [   # (query, retrieved_ids, gold_ids, context, answer)
    ("refund window?", ["d2","d9","d4"], ["d1"], "",
     "You have 30 days. [INVENTED]"),
    ("reset password?", ["d5","d1","d7"], ["d5"], "Click 'Forgot password'.",
     "Click 'Forgot password' and check your email."),
    ("export data?", ["d8","d3"], ["d8"], "Export is on the Settings page.",
     "Data export is under Billing. [INVENTED]"),
]

for q, retrieved, gold, ctx, ans in CASES:
    r = recall_at_k(retrieved, gold, k=3)
    if r < 1.0:
        verdict = "RETRIEVAL -> fix chunking / embeddings / reranking"
    elif not faithful(ctx, ans):
        verdict = "GENERATION -> fix prompt / grounding"
    else:
        verdict = "OK"
    print(f"{q:16} recall@3={r:.2f}  {verdict}")

refund window?   recall@3=0.00  RETRIEVAL -> fix chunking / embeddings / reranking
reset password?  recall@3=1.00  OK
export data?     recall@3=1.00  GENERATION -> fix prompt / grounding