Learn evaluation on real data
Each lesson starts with a decision an evaluation engineer faces at work. You make predictions, work through real public data by hand, run the full analysis yourself, and then face a new case with no hints. They're free, they take about an hour each, and they build on each other, so doing them in order helps.
Is the New Model Better, or Is It Noise?
Decide whether a bigger model's higher benchmark score is worth paying for, and find out why a leaderboard of close scores is mostly noise.
Data: MT-Bench grades for 34 chatbots · You'll use: paired differences, 95% ranges, resampling
Does Your Test Set Look Like Your Users?
Compare a polished benchmark with 60,000 real chatbot conversations, label real messages yourself, and build a test set that covers what users send.
Data: WildChat real conversations · You'll use: traffic profiles, slices, sampling with weights
Can an Automatic Score Tell a Good Summary From a Bad One?
Compute ROUGE by hand, test it against expert ratings of 1,600 summaries, and find the damaging errors it rewards.
Data: SummEval expert ratings · You'll use: word-overlap scores, correlation, system vs. single-output evaluation
Can You Trust an AI Judge?
Judge real answer pairs yourself, then measure how often GPT-4 agrees with human raters, and how often it changes its mind when the answers swap places.
Data: MT-Bench human and GPT-4 judgments · You'll use: agreement rates, human ceilings, position bias
Do Your Human Raters Agree?
Rate real responses yourself, compute chance-corrected agreement by hand, avoid a common counting mistake, and plan a labeling budget.
Data: HelpSteer2 individual annotations · You'll use: kappa, weighted agreement, raters per item
Did the Model Make That Up?
Check real answers against their sources, measure how often six models invent things even with documents in hand, and test cheap automatic checks.
Data: RAGTruth human-marked answers · You'll use: made-up rates by model and task, recall, precision and base rates
Did Search Find the Right Passage?
Play the search engine, score search by hand, then discover why a famous benchmark flatters keyword search and test it with your own wording.
Data: SQuAD questions over Wikipedia · You'll use: BM25, top-k rates, mean reciprocal rank, slicing by word overlap
Can You Rely on the Agent?
Read two real runs of a customer-service agent on the same request, measure how consistently agents succeed across repeated runs, and decide where business rules belong.
Data: τ-bench agent runs · You'll use: success per run vs. every run vs. any run, failure types, rules in tools
How Safe Is the Model?
Predict how one model fares under three jailbreak attacks, measure how much the choice of judge moves a safety score, and decide which safety report to trust.
Data: JailbreakBench attack results and human labels · You'll use: attack success rate, worst case across attacks, judge vs. people, over-refusal
Is Something Wrong in Production?
Read a year of real chatbot traffic, find out what moved the dashboard, trace a drop nobody announced, and decide which alerts deserve to wake someone up.
Data: WildChat traffic, April 2023 to April 2024 · You'll use: per-sender breakdowns, weekly noise, alert rules, change logs
Ship or Hold?
The final lesson. Decide whether to switch a support agent to a newer model: test whether the gain is real, read the regressions, weigh the failures that cost money, and write the release memo.
Data: τ-bench agent runs, two models · You'll use: paired ranges, per-task regressions, failure costs, a staged launch plan