Build vs. Buy & Who to Hire
You've seen what good evaluation looks like. The last decisions are yours: what to build versus buy, and who to put in the seat.
Build vs. buy. The tooling landscape has eval frameworks, observability/tracing platforms (logging every prompt, response, cost, and latency), and annotation tools. Underneath, they all do the same four things — dataset, task, scorer, report — so don't over-buy. A sensible default: buy observability/tracing (it's plumbing you don't want to maintain) and build your eval sets and scorers (they encode your definition of quality and are your real moat). Who to hire. An AI Evaluation Engineer is part data scientist (statistics, datasets), part software engineer (pipelines, CI), part product thinker (turning "good" into measurable criteria). The best ones often come from ML, data science, or search/ranking-quality backgrounds — measurement rigor transfers directly; the LLM-specific surface is learnable in weeks.
Two candidates
Candidate 1 lists five eval tools and the latest models. Candidate 2 asks what your product does, what "wrong" looks like for your users, and how you'd know quality dropped — then sketches a sliced dataset and a validated judge. Who do you hire?
Candidate 2, decisively. Tools change every quarter and are learnable; the scarce, durable skill is turning a fuzzy "is it good?" into a measurable plan — datasets, slices, validated judges, sample sizes, gates. Candidate 1's tool fluency is a thin veneer over no measurement instinct; Candidate 2 is already doing the job in the interview. The strongest eval hires frequently come from search-quality or ML backgrounds precisely because that rigor is exactly what the role needs — don't screen them out for lacking "years of LLM experience." Ask any finalist to show you an eval harness they built.
Two companies, two right answers
Two companies faced the build-vs-buy call on evaluation. DeskZero, a small team, needed basic quality checks on a low-stakes feature; they bought an off-the-shelf eval tool, wired it up in days, and spent their engineering time on the product. Right call. MedLedger, in healthcare, had high-stakes, sensitive outputs where quality and safety were existential; they bought the tooling but built their own datasets, judges, and gates in-house and hired an evaluation specialist to own it. Also the right call.
The difference wasn't budget or sophistication — it was how core and consequential quality was to each business. Both, tellingly, measured on their own representative datasets rather than trusting a vendor's benchmark. The leadership lesson: decide build-vs-buy on how much evaluation is a differentiator and a risk for you, and either way, own a dataset that reflects your reality and someone accountable for the number.
The whole arc: Meridian bought its eval tooling but built what was theirs — Remi's sliced dataset, the validated judge, the refund gate, the red-team suite, the CI gate, and live monitoring — and hired one evaluation engineer to own "how do we know Remi is good, and safe?" That role is why Remi shipped without the incident Chapter 1 was afraid of. The appendix is how they hired for it.
Quiz · Chapter 12
- A sensible build-vs-buy default is:
- The scarce, durable skill in an eval hire is:
- A strong eval-engineer candidate, handed a vague problem, will first:
- Candidates from search-quality or ML backgrounds are: