Production Evaluation

Offline evals tell you about the cases you thought of. Production shows you everything else. Online evaluation measures behavior on real traffic and sends what it finds back into your offline suite.

Signals, guardrails and silent regressions

On live traffic you can collect user behavior signals, run a judge on a sample of responses, and enforce guardrails that block or repair a response before it's sent. Guardrails act in the moment; evals measure over time. You want both.

The failure to fear is the silent regression. Nothing crashes and nothing logs an error, but quality drops, because users started asking different things or because the model provider updated the model. You find out when customers leave. Drift detection is how you notice sooner.

Explicit and implicit feedback

Explicit feedback is what users choose to give: thumbs up or down, star ratings, a report button. It's clear but rare. On most products only a percent or two of responses get rated, and the people who rate tend to be either delighted or annoyed. Implicit feedback is what their behavior shows, and there's far more of it:

• Acceptance rate. For inline suggestions such as code completions or reply drafts, did the user keep it? For a completion product this is often the main quality number.
• Edit distance. How much did they change an accepted suggestion before using it? A suggestion that was accepted and then rewritten wasn't much of a win.
• Copy, retry and abandon. Copying a response suggests it was useful. Regenerating or rephrasing the question suggests it wasn't. Leaving in the middle of a conversation is often the clearest signal of all.

Each implicit signal is noisy, but in aggregate they're strong, and they cost nothing to label. Use them for the main dashboard, a sampled judge to understand why things change, and explicit ratings as spot checks.

Operational metrics

Quality always trades off against speed and cost, and at scale managing that trade-off is much of the engineering. Track these next to quality:

• Time to first token (TTFT). How long until the user sees anything. In a streaming interface this is what feels fast or slow: under a second feels instant, several seconds feels broken.
• Tokens per second. How fast output streams once it starts, and how many requests a deployment can handle at once before it slows down.
• Cost and latency per request. Dollars and milliseconds per call, compared against quality. A prompt change that adds two points of judge score but doubles cost and TTFT may be a net loss.
• Utilization. How busy the GPUs are behind a self-hosted model, which tells you whether you need more hardware or better batching.

Lab code · runs free

from math import log

def histogram(values, edges):
    counts = [0] * (len(edges) - 1)
    for v in values:
        for i in range(len(edges) - 1):
            hi_ok = v < edges[i+1] or (i == len(edges) - 2 and v == edges[-1])
            if edges[i] <= v and hi_ok:
                counts[i] += 1
                break
    total = sum(counts) or 1
    return [c / total for c in counts]

def psi(expected, actual, edges):
    e, a = histogram(expected, edges), histogram(actual, edges)
    return sum((ai - ei) * log((ai + 1e-6) / (ei + 1e-6)) for ei, ai in zip(e, a))

edges     = [0, 0.2, 0.4, 0.6, 0.8, 1.0]
baseline  = [0.90, 0.85, 0.80, 0.95, 0.70, 0.88, 0.92, 0.78, 0.83, 0.91]  # last month
this_week = [0.50, 0.55, 0.40, 0.60, 0.45, 0.70, 0.30, 0.65, 0.50, 0.42]  # quality slipped

d = psi(baseline, this_week, edges)
print(f"PSI = {d:.2f}  ->  {'DRIFT — investigate' if d > 0.2 else 'stable'}")

PSI = 1.93  ->  DRIFT — investigate