Build vs. Buy & Who to Hire
You can buy a "chat with your docs" product off the shelf or build one on your own stack, and both are right for different companies. The mistake is deciding by hype instead of by what's actually your differentiator.
Off-the-shelf RAG products get you to a working assistant fast and are often the right call when the feature is a convenience, not your core product. Building gives you control over quality, cost, security, and the corpus — worth it when answers-from-your-documents is the product, when your data is sensitive, or when scale makes per-question cost a strategic number. A common middle path: buy the infrastructure (the search index, the model) and build the parts that are yours — retrieval quality, evaluation, and permissions.
Whichever you choose, someone has to own quality. That's the hire: not necessarily a machine-learning PhD, but an engineer who thinks in retrieval quality, slices, faithfulness, and cost — who treats "how do we know it's good?" as their core job. The appendix gives you the questions to find that person.
Build or buy?
Two of your peers made opposite calls: one bought an off-the-shelf doc assistant, one built in-house. Both are happy. What single question decides which is right for you?
Ask: "Is answering from our documents a core differentiator — and is our data sensitive or our scale large enough that cost and control matter?" If yes to those, build (or buy-the-infra-build-the-logic), because quality, security, and per-question cost are strategic and you can't outsource them. If it's a convenience feature on non-sensitive data at modest scale, buy and move on. Either way, one rule holds: even if you buy, measure the vendor's answers on your gold set — never accept a black box you can't grade.
Two startups, two right answers
Two startups faced the same choice. DeskZero, a small B2B tool, needed a help widget over public docs — not a differentiator, not sensitive. They bought an off-the-shelf assistant, wired it up in a week, and spent their engineering time on their actual product. Right call. MedLedger, a healthcare platform, needed answers over sensitive patient-facing documents where accuracy and access control were existential. They bought the search infrastructure but built retrieval, evaluation, and permission-aware access in-house. Also the right call.
The difference wasn't budget or sophistication; it was whether RAG was core and whether the data was sensitive. Both, notably, measured quality on their own gold set — DeskZero to keep the vendor honest, MedLedger to govern their own build. The leadership lesson: decide build-vs-buy on differentiation and risk, and measure the output either way.
Brightline lands in the middle: it buys the vector infrastructure and the model, but builds its own retrieval logic, evaluation harness, and permission-aware access — the parts tied to its quality and its data. To own it, they hire one engineer whose whole job is "how do we know the answers are good?" The next section is how they found her.
Quiz · Chapter 12
- You should lean toward building when:
- A common sensible middle path is:
- Even if you buy, you should:
- The key hire is someone who: