What It Costs — and How Not to Overpay
RAG can be remarkably cheap or shockingly expensive for the same feature, and the difference is an architecture choice your team makes early. You should understand the one trade-off that drives the bill.
Every question you send to a language model costs money in proportion to how much text you send with it. The tempting shortcut — "context windows are huge now, just send the whole document set with every question" — is where budgets explode: you pay to process your entire corpus on every single query. Retrieval is the cost discipline: find the handful of relevant passages and send only those, and the same question can cost a fraction of a cent instead of a dollar.
The gap compounds. At scale, "send everything" versus "retrieve the few relevant passages" can be a hundredfold difference in the monthly bill — and, as we saw earlier, sending everything often makes answers worse, not better, because the model attends poorly across a giant context. So the cheap architecture is usually also the accurate one. Your job as a leader is to ask what drives the per-question cost and whether there's a runaway path hiding in it.
The surprise invoice
Usage doubled but your AI bill went up sixfold, and finance wants an explanation. What's the most likely architectural cause, and what do you ask your team?
The classic cause is sending too much text per question — often "stuff the whole document set (or huge chunks of it) into every prompt" — so cost scales with corpus size, not just query volume. Ask: "how much text do we send to the model per question, and does it grow as our document set grows?" If the answer is "we include everything," that's your sixfold. The fix is real retrieval: send only the few relevant passages. Bonus — it usually improves accuracy too, so it's not a cost-versus-quality trade, it's both.
The startup's $120k surprise
A startup shipped a document assistant that pasted its whole knowledge base — around 240,000 words — into every prompt, because "it was simpler and the context window was big enough." It worked in testing with light usage. At 100,000 questions a month, the architecture meant paying to process the entire corpus 100,000 times: the projected bill hit roughly $120,000 a month, versus about $300 for the same feature built on retrieval that sent only the relevant passages.
Same product, same model, same questions — a ~400× difference per query driven entirely by how much text they sent. When they switched to retrieving the few relevant passages, the bill collapsed and accuracy improved, because the model was no longer hunting for the answer in a wall of irrelevant text. The founder's lesson: ask about per-question cost before scale makes it a crisis, not after.
Before scaling company-wide, Brightline models cost per question and spots the runaway path early: an engineer had prototyped by sending large document dumps. They lock in retrieve-only-what's-relevant, set a per-question budget, and the projected bill drops from alarming to boring.
Quiz · Chapter 9
- The cost of a question scales mainly with:
- "Just put everything in the context window" tends to:
- Retrieval is a cost discipline because:
- A bill growing faster than usage suggests: