Production Evaluation: Monitoring & Online Evals

Offline evals tell you about the cases you thought of. Production tells you the truth. Online evaluation closes the loop — measuring real behavior on real traffic and feeding problems back into your offline suite.

Signals, guardrails, and the silent regression

Live you can collect implicit signals (thumbs, copy, regenerate, edits, abandonment), run an online judge on sampled traffic, and enforce guardrails — real-time checks that block or repair before a response ships. Note the distinction: guardrails enforce in the moment; evals measure over time. The nightmare is the silent regression: quality drops with no crash and no error — the input distribution shifts, or the provider silently updates the model — and you find out only when users churn. Drift detection makes silent failures loud.

What to measure live: explicit vs. implicit signals

Two kinds of signal come off real traffic, and mature teams instrument both. Explicit feedback is what users deliberately give — thumbs up/down, star ratings, a "report" flag. It's clean but rare: on most products only a percent or two of responses get rated, and the raters skew angry or delighted. Implicit feedback is what their behavior reveals, and it's where the volume is:

• Acceptance rate — for an inline suggestion (code completion, reply draft, autocomplete), did the user keep it? A completion product's headline quality number is often "% of suggestions accepted," not any offline score.
• Edit distance — how much did they change an accepted output before shipping it? A suggestion accepted and then rewritten word-for-word wasn't really a win; low post-accept edit distance is strong tacit approval.
• Copy / retry / regenerate / abandon — a copy is tacit approval; a regenerate or a rephrased re-ask is tacit rejection; abandoning mid-conversation is the loudest quiet signal of all.

Implicit signals are noisy per event but overwhelming in aggregate, and they cost no annotation budget — which is why they anchor production quality dashboards. Pair them with an online judge on sampled traffic for the "why," and explicit feedback as ground-truth spot-checks.

Operational metrics: quality isn't free

An eval that reports only quality is half an answer. In production, quality trades off against latency and cost, and at scale that trade-off is the engineering. Track these next to your quality signals — an interviewer at a company serving models at scale will expect them by name:

• TTFT (time to first token) — how long until the user sees anything. For streaming UIs, this (not total time) is perceived responsiveness: sub-second feels instant, multi-second feels broken.
• Tokens/second (throughput) — the streaming rate once output starts, and how many concurrent requests a deployment sustains before it degrades.
• Cost & latency per request — dollars and milliseconds per call, watched against quality. A prompt change that lifts a judge score two points but doubles cost and TTFT can be a net loss.
• Utilization — GPU/accelerator saturation behind self-hosted models; the number that decides whether you need more hardware or just better batching.

Lab code — runs free

from math import log

def histogram(values, edges):
    counts = [0] * (len(edges) - 1)
    for v in values:
        for i in range(len(edges) - 1):
            hi_ok = v < edges[i+1] or (i == len(edges) - 2 and v == edges[-1])
            if edges[i] <= v and hi_ok:
                counts[i] += 1
                break
    total = sum(counts) or 1
    return [c / total for c in counts]

def psi(expected, actual, edges):
    e, a = histogram(expected, edges), histogram(actual, edges)
    return sum((ai - ei) * log((ai + 1e-6) / (ei + 1e-6)) for ei, ai in zip(e, a))

edges     = [0, 0.2, 0.4, 0.6, 0.8, 1.0]
baseline  = [0.90, 0.85, 0.80, 0.95, 0.70, 0.88, 0.92, 0.78, 0.83, 0.91]  # last month
this_week = [0.50, 0.55, 0.40, 0.60, 0.45, 0.70, 0.30, 0.65, 0.50, 0.42]  # quality slipped

d = psi(baseline, this_week, edges)
print(f"PSI = {d:.2f}  ->  {'DRIFT — investigate' if d > 0.2 else 'stable'}")

PSI = 1.93  ->  DRIFT — investigate