AI Search / RAG Engineer · Chapter 3

Ingest & Chunking

Chunking is the least glamorous stage and the one that quietly caps your ceiling. Get it wrong and no retriever, reranker, or model can recover — the answer was never in a retrievable unit to begin with. This is where recall is won or lost before search even starts.

A chunk is the atomic unit your retriever returns. That single fact has sharp consequences: if the complete answer doesn't live inside one chunk, retrieval literally cannot return it. Split a rate-limit table so the header ("/export") is in chunk A and the value ("20 req/min") is in chunk B, and every query for that limit now depends on getting both chunks — doubling your failure surface for one fact. Chunking decides what "an answer" can even be.

The two forces you're trading off

Every chunking decision is one tension: too big vs. too small, and both fail, in opposite ways.

Chunk size	What breaks
Too large (e.g. whole page, 1500 tokens)	The embedding averages several topics into a muddy vector that matches everything weakly and nothing strongly; and you burn context tokens on irrelevant text, worsening lost-in-the-middle at synthesis.
Too small (e.g. one sentence, 40 tokens)	Answers get split across chunks; each chunk lacks the context to be interpretable ("It rotates every 90 days" — what rotates?); pronouns and headers dangle.

The industry lands, for prose, around 256–512 tokens with 10–20% overlap — not because it's magic, but because it's the empirical sweet spot where a chunk holds one coherent idea plus enough surrounding context to stand alone. Overlap (repeating the last ~50 tokens of one chunk at the start of the next) is cheap insurance against splitting an answer exactly on a boundary.

Structure beats fixed-size

Blind fixed-size splitting (every N characters) is the naive baseline and it cuts through the middle of sentences, tables, and code. Structure-aware chunking respects the document's own boundaries — split on headings, paragraphs, list items, Markdown sections — so each chunk is a natural semantic unit. The rule of thumb:

Chunk on meaning, not on character count

Split at the largest structural boundary that keeps chunks under your size cap: section → paragraph → sentence, in that order. Never split inside a table row, a code block, or a numbered step. A chunk should be something a human could read in isolation and understand.

One more high-leverage move: context enrichment. Prepend each chunk with its document title and section heading before embedding — "Acme Vault API › Rate Limits › The /export endpoint is rate limited to 20 req/min." Now a chunk that said only "20 requests per minute" carries its subject, so both its embedding and the model's reading of it are unambiguous. This one trick routinely lifts recall several points for near-zero cost.

Try it · ~30 min

Measure recall as a function of chunk size

You'll chunk the same document three ways — tiny, medium, huge — build a lexical retriever over each, and measure which chunking recovers the most gold answers. You'll watch the U-shape: small loses to splitting, huge loses to muddiness, medium wins. No API key, no cost.

Setup: pip install rank-bm25 (pure-Python). Nothing else.

Step 1. Chunk the doc at sizes 1, 3, and 8 sentences per chunk.

Step 2. For each chunking, BM25-retrieve top-2 for each gold query.

Step 3. Score recall = fraction of queries whose answer substring is present in a retrieved chunk. Print all three.

Your goal: three recall numbers showing medium ≥ small and medium ≥ huge — and the ability to name which query each extreme lost.

Starter code

from rank_bm25 import BM25Okapi
import re

DOC = (
 "The Acme Vault API secures secrets for teams. "
 "The /export endpoint is rate limited to 20 requests per minute per API key. "
 "Exceeding the limit returns HTTP 429 with a Retry-After header. "
 "Secrets are encrypted with AES-256. Encryption keys rotate every 90 days automatically. "
 "The free tier allows 3 projects. The Team tier allows 50 projects. "
 "Webhooks retry failed deliveries up to 5 times with exponential backoff. "
 "The legacy /dump endpoint was removed in v4; use /export instead."
)
SENTS = [s.strip() for s in DOC.split(". ") if s.strip()]

# (query, answer substring that must appear in a retrieved chunk)
GOLD = [
    ("export endpoint rate limit",        "20 requests per minute"),
    ("what happens when you exceed limit", "429"),
    ("how often do encryption keys rotate","90 days"),
    ("free tier project count",            "3 projects"),
    ("webhook retry attempts",             "5 times"),
]

def toks(s): return re.findall(r"[a-z0-9]+", s.lower())

def chunk(sents_per):
    return [". ".join(SENTS[i:i+sents_per]) for i in range(0, len(SENTS), sents_per)]

def recall(chunks, k=2):
    bm25 = BM25Okapi([toks(c) for c in chunks])
    hits = 0
    for q, ans in GOLD:
        ranked = sorted(range(len(chunks)), key=lambda i: bm25.get_scores(toks(q))[i], reverse=True)
        top = " ".join(chunks[i] for i in ranked[:k])
        hits += ans.lower() in top.lower()
    return hits / len(GOLD)

# TODO: print recall for chunk(1), chunk(3), chunk(8)

Worked solution

for n in (1, 3, 8):
    print(f"{n} sent/chunk  ->  recall@2 = {recall(chunk(n)):.2f}")

What you should see (approximately): 1 → 0.60, 3 → 1.00, 8 → 0.80. The medium chunking wins. Read why each extreme lost:

Too small (1 sentence). "Exceeding the limit returns HTTP 429..." and "...rate limited to 20 requests per minute" are separate chunks. A query about the limit retrieves one, and with k=2 a related-but-wrong sentence can crowd out the other — answers that needed two adjacent sentences get split.
Too big (8 sentences). One giant chunk holds five topics; BM25 scores it high for everything, so a query pulls the mega-chunk plus one more and can still miss a specific fact buried among unrelated text — and at synthesis you'd pay for all five topics' tokens.

The lesson: chunking has a recall optimum, not a "bigger/smaller is better" monotone. You find it by measuring on your own gold set — exactly this harness — not by copying a number from a blog. The right size is corpus-specific: dense API docs chunk smaller than narrative prose.

Going further: add overlap — repeat the last sentence of each chunk at the start of the next — and re-run the size-1 case. Watch split-answer recall recover, because the boundary that used to sever an answer now appears in both neighbors. That's overlap earning its keep.

Case study

Statute: when a rule doesn't fit in one chunk

Statute runs search over regulatory text. Their pipeline split documents into fixed 512-token windows — clean and simple, and quietly broken. Many rules ran across the boundary: the condition landed in one chunk and its exception in the next, so no single retrieved chunk contained the whole answer. Recall@10 sat stubbornly at 0.68 and no amount of retriever tuning moved it, because a chunk is the atomic unit — if the full answer doesn't fit in one, no retriever can return it.

Two changes fixed it. They switched to section-aware chunking (split on the document's own headings, so a rule and its exceptions stay together), and they prepended each chunk's heading path — "§7.2 Encryption keys · Rotation" — so a chunk carries its subject even in isolation. Recall@10 rose to 0.86. Same embeddings, same retriever; the win was entirely in how the text was cut.

Running case · Acme Vault

Acme's changelog was being sliced mid-entry, so "what changed in v4.2?" pulled half an entry. They chunk on entry boundaries and prepend the version and endpoint to each chunk — and those version queries start landing.