Chapter 11

Advanced Retrieval

The standard pipeline handles most questions. The ones it can't are filtered searches, multi-step questions and requests that shouldn't trigger retrieval at all. This chapter pairs each of those with the pattern that fixes it.

Case study

ResearchMate retrieves small and reads big

ResearchMate answers questions over scientific papers and hit the chunking trade-off from both sides. Small chunks retrieved precisely, but on their own they lacked the context needed for a full answer, and completeness scores sat at 3.4 out of 5. Large chunks carried context but blurred the embeddings.

They switched to small-to-big retrieval: match on paragraph-sized chunks, then give the model the whole section each match came from. Completeness rose to 4.3 with no loss in retrieval precision.

Metadata filtering

An enterprise customer asks for the refund policy, and enterprise terms differ from consumer terms. Semantic search happily returns the consumer chunk, which is very similar and has no idea about the customer's tier. Metadata filtering attaches fields such as tier, product, date and language to each chunk and restricts the search to the eligible ones. It's often the biggest precision gain available for documents with natural partitions, and it doubles as access control: never retrieve a chunk the user isn't allowed to see.

Multi-step and agentic retrieval

"Which of our SOC 2 certified vendors also support SSO?" needs two retrievals: first the certified vendors, then SSO support among them. No single chunk holds the answer. In agentic RAG, the model decides what to search for, reads the results and searches again based on what it learned. It's more capable and slower, since every step is a model call plus a retrieval, so keep it for questions that need it.

Knowing when not to retrieve

"Rewrite this paragraph to be shorter" doesn't need your documents. Retrieving anyway adds noise and cost and can make the answer worse. Mature systems put a cheap router in front that decides whether a query needs retrieval and, if so, which index.

Match the pattern to the failure

Partitioned documents call for metadata filters. Multi-step questions call for agentic retrieval. Mixed traffic calls for a router. Questions about relationships across many documents call for GraphRAG, which retrieves over a graph of entities. Each pattern adds latency and complexity, so add the one your questions need and prove it with the Chapter 8 harness.

Try it · ~25 minFree · no API key

Use a metadata filter to fix a confident wrong answer

Retrieve over documents where the best topical match is for the wrong customer tier. Show that filtering by tier before ranking turns a wrong but relevant answer into the right one, and measure accuracy before and after.

Starter code

CHUNKS = [
  {"id":"c1","tier":"consumer",  "topical":0.95, "text":"Consumer refunds within 14 days."},
  {"id":"c2","tier":"enterprise","topical":0.80, "text":"Enterprise refunds are governed by contract terms."},
  {"id":"c3","tier":"consumer",  "topical":0.90, "text":"Consumer plan cancellation policy."},
  {"id":"c4","tier":"enterprise","topical":0.78, "text":"Enterprise cancellation requires 30-day notice."},
]
# (query, asker's tier, correct chunk id)
QUERIES = [
  ("refund policy",       "enterprise", "c2"),
  ("cancellation policy", "enterprise", "c4"),
]

def best(cands):  # highest topical score
    return max(cands, key=lambda c: c["topical"])

# TODO unfiltered: best(all chunks) — does it match the correct id?
# TODO filtered:   best(chunks whose tier == asker tier)
# Print accuracy (fraction correct) for each.

Worked solution

def accuracy(use_filter):
    correct = 0
    for q, tier, gold in QUERIES:
        cands = [c for c in CHUNKS if (c["tier"] == tier)] if use_filter else CHUNKS
        correct += best(cands)["id"] == gold
    return correct / len(QUERIES)

print(f"unfiltered accuracy: {accuracy(False):.2f}")
print(f"filtered   accuracy: {accuracy(True):.2f}")

Accuracy goes from 0.00 without the filter to 1.00 with it. Without it, the consumer chunks win every time because they're cleaner textual matches, and an enterprise customer gets the wrong policy with full confidence. The filter restricts the candidates to the asker's tier before ranking. No embedding model could have fixed this, because the wrong chunk really was relevant; it just wasn't eligible.

For a harder version, combine filtering with hybrid search and reranking on a mix of queries where some have no tier constraint. You'll need a router to avoid over-filtering those, which is what production systems look like.

Running case · Acme Vault

Acme retrieves on paragraph-sized chunks and hands the model the full section. An answer about vault.export now arrives with its authentication prerequisites attached, instead of leaving out the setup step.

If you're starting out

You don't need to have built GraphRAG or an agentic retriever to discuss them well. What interviewers look for is knowing which problem each pattern solves and what it costs. Pairing each pattern with the failure it fixes, as this chapter does, is a good way to organize your answer.

If you run the team

The most important item in this chapter for you is access control. If the assistant searches one index covering HR files, finance and every team's documents, anyone can phrase a question that surfaces something they were never allowed to read. Make per-user permission checks in retrieval a launch requirement.

Treat documents the assistant ingests as untrusted, too. A document or web page can contain instructions that try to hijack the assistant, so ask the team how they test for that.

Back Continue