Tools, Interviews & Hiring

You don't need to memorize tools. You need to know the categories well enough to map any company's stack onto ideas you already understand. Then you build the whole thing once yourself, and you'll have something to show.

Tools, by category

• Eval frameworks and harnesses define datasets, run tasks and score them. Open-source and vendor versions all standardize the dataset, task, scorer and report loop.
• RAG evaluation tools ship faithfulness, relevance and context metrics (Chapter 7).
• Observability and tracing platforms log every prompt, response, tool call, latency and cost, and let you sample and score production traffic. Online evals usually live here.
• Experiment tracking versions prompts, datasets and eval runs so results can be compared.
• Annotation and red-team tools provide labeling interfaces and attack suites.

Every one of these implements the four parts from Chapter 2 plus a way to track results over time. Once you understand those, switching tools is mostly a matter of reading the docs.

The broad opener, answered in three parts

The most common screening question is also the vaguest: "How do you evaluate the quality and reliability of AI systems? What tools or metrics do you use?" The interviewer wants to hear a system. Three parts cover it in about ninety seconds:

• Offline, before release. Golden datasets with normal, edge and adversarial cases. Deterministic scorers where there's an answer key, and a validated model judge where there isn't (Chapters 4 and 5). For RAG, the Triad plus retrieval recall@k (Chapter 7). A gate in CI (Chapter 11).
• Online, in production. Sampled live traffic with an online judge, implicit signals such as acceptance, edits and retries, explicit ratings, guardrail classifiers such as Llama Guard, observability through tools like LangSmith, Phoenix or Langfuse, and drift detection (Chapters 9 and 10).
• Operational. TTFT, tokens per second, cost and latency per request, utilization, and the routing trade-off between a small model and a frontier model, proven with the eval harness (Chapter 10).

Capstone code · runs free (mock by default)

"""Minimal eval framework: dataset -> task -> scorer -> report.
Every tool you'll meet is a version of these four parts."""
from statistics import mean

USE_REAL_API = False        # free mock by default; True needs a key

DATASET = [
    {"q": "refund window?",  "ctx": "Returns accepted within 30 days.", "slice": "policy"},
    {"q": "reset password?", "ctx": "Use the 'Forgot password' link.",   "slice": "howto"},
    {"q": "data export?",    "ctx": "Export lives on the Settings page.", "slice": "howto"},
]

if USE_REAL_API:
    import anthropic
    from pydantic import BaseModel
    client = anthropic.Anthropic()
    class G(BaseModel):
        reason: str
        grounded: bool
    def task(c):                                   # 2. TASK (system under test)
        r = client.messages.create(model="claude-opus-4-8", max_tokens=200,
            messages=[{"role": "user", "content": f"{c['ctx']}\n\nQ: {c['q']}"}])
        return next(b.text for b in r.content if b.type == "text")
    def judge(ans, c):                             # 3. SCORER (LLM-as-judge)
        r = client.messages.parse(model="claude-opus-4-8", max_tokens=200,
            messages=[{"role": "user", "content":
                f"Grounded in context? true/false.\nCTX:{c['ctx']}\nANS:{ans}"}],
            output_format=G)
        return float(r.parsed_output.grounded)
else:
    def task(c):       return c["ctx"]             # stub: answer = the context
    def judge(ans, c): return float(c["ctx"] in ans)   # grounded if it quotes ctx

def report(dataset):                               # 4. REPORT (overall + slices)
    rows = [{"slice": c["slice"], "score": judge(task(c), c)} for c in dataset]   # 1. DATASET
    by_slice = {s: mean(r["score"] for r in rows if r["slice"] == s)
                for s in {r["slice"] for r in rows}}
    return {"overall": mean(r["score"] for r in rows), "by_slice": by_slice, "n": len(rows)}

print(report(DATASET))

{'overall': 1.0, 'by_slice': {'policy': 1.0, 'howto': 1.0}, 'n': 3}