Tools of the Trade & Interview Prep

You don't need to memorize a tool to interview well — you need the categories and vocabulary, so you can map any company's stack onto concepts you understand. Then you assemble the whole thing once, yourself.

The tooling landscape, by category

• Eval frameworks & harnesses — define datasets, run tasks, score (open-source and vendor SDKs). They standardize the dataset→task→scorer→report loop.
• RAG-specific eval — faithfulness/relevance/context metrics out of the box (Chapter 7).
• Observability & tracing — log every prompt, response, tool call, latency, cost; sample and score production traffic. Where online evals live.
• Experiment tracking — version prompts, datasets, eval runs so results are comparable.
• Annotation & red-team tooling — human labeling interfaces and adversarial suites.

The throughline: every tool implements the four-part eval (dataset, task, scorer, report) plus a way to track results over time. Learn the concept and the tools become interchangeable.

Questions you should be ready for

• "How would you evaluate a support chatbot / a summarizer / a RAG Q&A system?" → name dimensions, propose dataset + scorer per dimension, separate components (Ch 2, 4, 7).
• "You changed a prompt and it seems better — how do you know?" → baseline, eval set, pairwise judge, slices, CI guard (Ch 5, 11).
• "How do you trust an LLM judge?" → validate vs. human labels on the hard slice, mitigate biases, low-cardinality rubric, re-validate on change (Ch 5).
• "How would you catch a regression after the provider updates the model?" → production monitoring, drift detection, CI regression suite (Ch 10).
• "How do you evaluate an agent?" → outcome vs. trajectory, success rate, tool-use correctness, cost/latency, sandboxes (Ch 8).
• "How do you evaluate the quality and reliability of AI systems and their output — what tools and metrics?" → the broad opener; answer in three pillars — offline, online, operational (below).

The broad opener, answered in three pillars

The most common screening question is also the vaguest: "How do you evaluate the quality and reliability of AI technologies and their output? What tools or metrics do you use?" It's an invitation to show you have a system, not a grab-bag of metrics. Structure the answer in three pillars and you cover the whole field in about ninety seconds:

• Pillar 1 — Offline (pre-release). Reproducible benchmarks on curated golden datasets — standard queries, edge cases, adversarial prompts. Deterministic scorers where there's a key; a validated LLM-as-judge (G-Eval-style rubric) where there isn't (Ch 4–5). For RAG, the RAG Triad — context relevance, groundedness, answer relevance — plus retrieval recall@k (Ch 7). Gate releases on it in CI (Ch 11).
• Pillar 2 — Online (production). Static sets go stale the moment traffic shifts, so you sample real traffic, run an online judge, and watch implicit signals — acceptance rate, edit distance, regenerate/abandon — alongside explicit thumbs. Real-time guardrail classifiers (Llama Guard) catch toxicity, PII, and jailbreaks in flight. Observability via LangSmith / Arize Phoenix / Langfuse; drift detection makes silent regressions loud (Ch 10).
• Pillar 3 — Operational (cost & latency). Quality isn't free: TTFT, tokens/second, cost and latency per request, utilization — and the routing trade-off of a small fine-tuned model for the easy majority versus a frontier model for the hard tail, proven with the eval harness (Ch 10).

Capstone code — runs free (mock by default)

"""Minimal eval framework: dataset -> task -> scorer -> report.
Every tool you'll meet is a version of these four parts."""
from statistics import mean

USE_REAL_API = False        # free mock by default; True needs a key

DATASET = [
    {"q": "refund window?",  "ctx": "Returns accepted within 30 days.", "slice": "policy"},
    {"q": "reset password?", "ctx": "Use the 'Forgot password' link.",   "slice": "howto"},
    {"q": "data export?",    "ctx": "Export lives on the Settings page.", "slice": "howto"},
]

if USE_REAL_API:
    import anthropic
    from pydantic import BaseModel
    client = anthropic.Anthropic()
    class G(BaseModel):
        reason: str
        grounded: bool
    def task(c):                                   # 2. TASK (system under test)
        r = client.messages.create(model="claude-opus-4-8", max_tokens=200,
            messages=[{"role": "user", "content": f"{c['ctx']}\n\nQ: {c['q']}"}])
        return next(b.text for b in r.content if b.type == "text")
    def judge(ans, c):                             # 3. SCORER (LLM-as-judge)
        r = client.messages.parse(model="claude-opus-4-8", max_tokens=200,
            messages=[{"role": "user", "content":
                f"Grounded in context? true/false.\nCTX:{c['ctx']}\nANS:{ans}"}],
            output_format=G)
        return float(r.parsed_output.grounded)
else:
    def task(c):       return c["ctx"]             # stub: answer = the context
    def judge(ans, c): return float(c["ctx"] in ans)   # grounded if it quotes ctx

def report(dataset):                               # 4. REPORT (overall + slices)
    rows = [{"slice": c["slice"], "score": judge(task(c), c)} for c in dataset]   # 1. DATASET
    by_slice = {s: mean(r["score"] for r in rows if r["slice"] == s)
                for s in {r["slice"] for r in rows}}
    return {"overall": mean(r["score"] for r in rows), "by_slice": by_slice, "n": len(rows)}

print(report(DATASET))

{'overall': 1.0, 'by_slice': {'policy': 1.0, 'howto': 1.0}, 'n': 3}