Ingest & Chunking
Chunking gets little attention, and it sets a ceiling on everything after it. If an answer never ends up whole inside one retrievable piece, no retriever, reranker or model can recover it.
Statute's rules didn't fit in one chunk
Statute runs search over regulatory text. It split documents into fixed 512-token windows, which was simple and had a hidden flaw: many rules crossed a boundary, with the condition in one chunk and its exception in the next. Recall@10 stayed at 0.68 no matter how they tuned the retriever.
Two changes fixed it. They split on the documents' own section headings, so each rule stayed with its exceptions, and they added the heading path, such as "§7.2 Encryption keys · Rotation," to the start of each chunk. Recall@10 rose to 0.86 with the same embeddings and the same retriever.
A chunk is the smallest thing your retriever can return. If the full answer isn't inside one chunk, retrieval can't hand it over in one piece. Put the header "/export" in chunk A and the value "20 requests per minute" in chunk B, and every question about that limit now depends on retrieving both. Chunking decides what an answer can even look like to the rest of the system.
Too big or too small
| Chunk size | What goes wrong |
|---|---|
| Too large (a whole page, 1,500 tokens) | The embedding averages several topics into a blurry vector that matches everything a little and nothing well. You also spend context on irrelevant text, which makes lost-in-the-middle worse. |
| Too small (one sentence, 40 tokens) | Answers get split. Chunks lose the context that makes them readable: "It rotates every 90 days," but what does? |
For prose, many teams settle on 256 to 512 tokens with 10 to 20% overlap. The range comes from experience: it's usually enough for one coherent idea plus the context to stand alone. Overlap, repeating the last 50 or so tokens of one chunk at the start of the next, is cheap protection against an answer falling right on a boundary.
Split on structure
Splitting every N characters cuts through sentences, tables and code. Structure-aware chunking follows the document's own boundaries (headings, paragraphs, list items) so each chunk is a natural unit. Split at the largest boundary that keeps chunks under your size limit, working down from section to paragraph to sentence, and never split inside a table row, a code block or a numbered step. A person should be able to read any chunk on its own and understand it.
One more cheap improvement is context enrichment: add the document title and section heading to the start of each chunk before indexing, as in "Acme Vault API › Rate Limits › The /export endpoint is limited to 20 requests per minute." A chunk that only said "20 requests per minute" now carries its subject, which helps both the embedding and the model. It often lifts recall by several points.
Measure recall at different chunk sizes
Chunk one document three ways (1, 3 and 8 sentences per chunk), build a BM25 retriever over each, and measure which version finds the most gold answers. You'll need pip install rank-bm25. Look for the U-shape, and be ready to name the question each extreme lost.
Starter code
from rank_bm25 import BM25Okapi
import re
DOC = (
"The Acme Vault API secures secrets for teams. "
"The /export endpoint is rate limited to 20 requests per minute per API key. "
"Exceeding the limit returns HTTP 429 with a Retry-After header. "
"Secrets are encrypted with AES-256. Encryption keys rotate every 90 days automatically. "
"The free tier allows 3 projects. The Team tier allows 50 projects. "
"Webhooks retry failed deliveries up to 5 times with exponential backoff. "
"The legacy /dump endpoint was removed in v4; use /export instead."
)
SENTS = [s.strip() for s in DOC.split(". ") if s.strip()]
# (query, answer substring that must appear in a retrieved chunk)
GOLD = [
("export endpoint rate limit", "20 requests per minute"),
("what happens when you exceed limit", "429"),
("how often do encryption keys rotate","90 days"),
("free tier project count", "3 projects"),
("webhook retry attempts", "5 times"),
]
def toks(s): return re.findall(r"[a-z0-9]+", s.lower())
def chunk(sents_per):
return [". ".join(SENTS[i:i+sents_per]) for i in range(0, len(SENTS), sents_per)]
def recall(chunks, k=2):
bm25 = BM25Okapi([toks(c) for c in chunks])
hits = 0
for q, ans in GOLD:
ranked = sorted(range(len(chunks)), key=lambda i: bm25.get_scores(toks(q))[i], reverse=True)
top = " ".join(chunks[i] for i in ranked[:k])
hits += ans.lower() in top.lower()
return hits / len(GOLD)
# TODO: print recall for chunk(1), chunk(3), chunk(8)for n in (1, 3, 8):
print(f"{n} sent/chunk -> recall@2 = {recall(chunk(n)):.2f}")You should see roughly 0.60 for one sentence, 1.00 for three and 0.80 for eight. With one-sentence chunks, the sentence about the HTTP 429 error and the sentence with the limit end up apart, and answers that needed both get split. With eight-sentence chunks, one giant chunk covers five topics, scores well for everything, and can still crowd out the specific fact you wanted.
The best size depends on your documents: dense API docs usually want smaller chunks than narrative prose. Find it by measuring on your own questions, as you just did. For more practice, add one sentence of overlap to the one-sentence version and watch the split answers come back.
Acme's changelog was being cut in the middle of entries, so "what changed in v4.2?" retrieved half an entry. Acme now splits on entry boundaries and adds the version and endpoint to the start of each chunk, and version questions start working.
Chunking questions are common in interviews because there's no single right answer, which lets the interviewer see how you reason. Talk about the trade-off between too big and too small, mention structure-aware splitting and overlap, and say you'd pick the size by measuring recall on real questions.
A small project that stands out: take a public documentation site, chunk it three ways, and publish the recall numbers. It's quick and shows you measure instead of copying defaults.
The documents you point the system at matter more than which model you pick. Chunking is one half of that; the other is the documents themselves. If the corpus has outdated drafts, duplicates or two versions of the same policy, a perfect retriever will faithfully hand the model the wrong one, with a citation.
When the team asks for time to clean up the documents before launch, it's usually worth giving. Removing duplicates and superseded versions, dating documents and naming an owner are unglamorous tasks that prevent the "the assistant quoted last year's policy" incidents.
Quiz · Chapter 3
- Why does chunking set a ceiling on the whole pipeline?
- A chunk says only "It rotates every 90 days," and it ranks low for "how often do encryption keys rotate." Best fix?
- Why do very large chunks tend to hurt retrieval?
- Why is recall against chunk size usually U-shaped?
- Why does structure-aware chunking beat fixed-size splitting?