AI Engineering: Evaluation Before Production Deployment

A working demo of an LLM-powered feature is one of the easiest things to produce in software engineering today, and one of the least reliable signals of production readiness. A demo shows that a model can produce a plausible answer to a handful of hand-picked prompts, run by the person who built the system and knows exactly what phrasing works. Production traffic looks nothing like that: it's adversarial, ambiguous, out-of-distribution, and evaluated by users who have no reason to phrase things the way your test prompts did. The gap between 'it worked when I tried it' and 'it works reliably enough to ship' is where evaluation belongs, and skipping it is the single most common reason AI features underperform or embarrass their owners after launch.

Why demos mislead

Demos are typically run against a small, favorable set of inputs, evaluated subjectively by someone with context on how the system is supposed to work, and judged on a single pass rather than the distribution of outcomes across many runs (LLM outputs are non-deterministic by default). None of that resembles how the feature will be judged once real users depend on it. Evaluation exists to replace 'I tried it and it seemed good' with a repeatable, quantified answer to 'how often does it meet the bar we defined, across the range of inputs we expect to see.'

Define task-level success criteria first

Before building an eval dataset, define what success means for the specific task, in terms specific enough that two different reviewers would judge the same output the same way. Vague criteria like 'the answer should be helpful and accurate' produce inconsistent evaluations. Better criteria are decomposed and checkable:

• For a support-answering assistant: does the answer address the actual question, is every factual claim traceable to a retrieved source, does it avoid recommending an action outside policy, is the tone appropriate.
• For a data-extraction task: does the output match the expected schema, are required fields populated, is the extracted value correct against the source document.
• For a code-generation assistant: does the generated code compile/run, does it pass the relevant test cases, does it avoid disallowed patterns (e.g., hardcoded secrets).

Each criterion should be specific enough to be scored consistently, ideally by an automated check, and only judged subjectively where automation genuinely cannot cover it.

Building an evaluation dataset

An eval dataset is a curated, versioned set of representative inputs (and, where possible, expected or reference outputs) that reflects the real distribution of what the system will face — including the awkward and adversarial cases, not just the clean ones.

• Source real inputs where you can: sampled historical queries, support tickets, or documents, rather than only synthetic ones you generate yourself, which tend to reflect your own assumptions about what users will ask.
• Deliberately include edge cases: ambiguous questions, out-of-scope requests, adversarial prompts attempting to bypass instructions, inputs in unexpected formats or languages if relevant.
• Label a subset with expected outputs or acceptance criteria, ideally reviewed by someone with domain expertise, not just the engineer building the feature.
• Version the dataset like code, so you can tell whether a regression is due to a model or prompt change versus a dataset change.

Offline vs online evaluation

Offline evaluation runs your system against the curated dataset before deployment, giving you a repeatable score to compare across model or prompt versions. Online evaluation observes real production traffic and outcomes — user feedback signals, downstream conversion or resolution rates, explicit thumbs up/down — and tells you whether offline gains translate to real-world usage, which they do not always do, because production traffic drifts from any fixed dataset over time.

Both are necessary. Offline evaluation without online monitoring will miss drift and edge cases you didn't anticipate. Online monitoring without an offline suite means you can't safely test a change before it's already affecting users.

Retrieval grounding and citation checks

For retrieval-augmented systems, evaluating the generated answer alone is not enough — you need to separately evaluate retrieval quality (did the system fetch the right source documents for the question) and grounding (is every claim in the answer actually supported by the retrieved content, or did the model add unsupported information).

• Check retrieval quality independently: for a labeled set of questions, does the retrieval step return the document(s) that actually contain the answer.
• Check grounding by verifying that specific claims in the generated answer can be traced to specific passages in the retrieved context; flag unsupported claims as failures even if the answer happens to be factually correct by coincidence.
• Track citation accuracy where the system presents sources to users — a citation pointing to a source that does not actually support the claim is arguably worse than no citation, because it creates false confidence.

Using an LLM as a judge, carefully

Using a second LLM call to score outputs against a rubric ('LLM-as-judge') is a practical way to scale evaluation beyond what human review can cover, but it has real limitations that are easy to underestimate.

• Judge models have their own biases — they tend to favor longer, more confident-sounding answers regardless of correctness, and can be inconsistent across repeated runs.
• Validate the judge itself: periodically sample its scores and have a human check agreement, and treat disagreement as a signal to refine the rubric or the judge prompt.
• Use LLM-as-judge for high-volume, lower-stakes triage, and reserve human review for high-stakes decisions, ambiguous cases, and periodic calibration.
• Never rely on a single overall 'quality score' from a judge model as the sole gate for shipping — decompose scoring into the specific criteria defined earlier so failures are diagnosable.

Regression suites in CI

Prompt changes, model version upgrades, and retrieval pipeline changes should all run against the eval suite in CI before merging, the same way code changes run against unit tests. This catches the common failure mode where a prompt tweak fixes one reported issue but silently degrades performance on a different class of input. Track eval scores over time per category of input, not just an aggregate score, so a regression in a specific area (e.g., multi-turn conversations, or a particular language) doesn't get averaged away by improvements elsewhere.

Guardrails and human-in-the-loop

Evaluation reduces risk; it does not eliminate the need for runtime safeguards. Input and output guardrails (content filters, schema validators, policy checks on generated actions) catch failures that evaluation didn't anticipate. For high-stakes or irreversible actions — sending an email on a user's behalf, executing a financial transaction, making a medical or legal recommendation — keep a human-in-the-loop approval step rather than fully automating the decision, at least until the system has a substantial track record and the cost of an error is well understood.

Observability and tracing

Once deployed, you need visibility into what the system actually did for a given request: the prompt sent, the retrieved context, the model response, any tool calls, latency at each step, and the final output shown to the user. Without this trace, debugging a user-reported bad answer after the fact is close to impossible. Structured tracing also feeds back into your eval dataset — production failures, once understood, should become new eval cases so the same failure mode is tested for going forward.

Cost and latency budgets

Model quality, cost, and latency trade against each other, and evaluation should account for all three rather than optimizing quality in isolation. Define an acceptable latency budget and cost-per-request ceiling for the feature up front, and evaluate candidate models or prompt strategies against all three dimensions together — a marginally more accurate model that doubles latency or cost may not be the right choice for a real-time user-facing feature, even if it wins on the quality metric alone.

Rollout strategy

Treat AI feature rollout the same way you would treat any risky software change: start with a limited audience or shadow mode (running the new system alongside the old one without showing its output to users, to compare behavior), expand gradually while monitoring online metrics and user feedback, and keep a fast rollback path — ideally a feature flag or prompt/model version toggle rather than a full redeploy.

Common mistakes

• Shipping based on a small set of manually tried prompts, with no versioned eval dataset.
• Evaluating only the final answer for RAG systems, without separately checking retrieval quality and grounding.
• Trusting a single LLM-as-judge score without periodically validating it against human judgment.
• No regression suite in CI, so prompt or model changes silently reintroduce old failure modes.
• No tracing in production, making reported failures nearly impossible to reproduce or diagnose.
• Automating irreversible or high-stakes actions before the system has a track record and appropriate guardrails.
• Optimizing for quality alone and discovering latency or cost problems only after launch.

Checklist

• Task-level, checkable success criteria defined before building the eval set.
• A versioned eval dataset built from real, representative inputs, including edge cases.
• Separate retrieval-quality and grounding checks for RAG systems.
• LLM-as-judge usage validated periodically against human review.
• Eval suite wired into CI, with per-category (not just aggregate) score tracking.
• Runtime guardrails and human-in-the-loop approval for high-stakes actions.
• Production tracing that captures prompts, context, outputs, and latency per request.
• Defined cost and latency budgets evaluated alongside quality.
• A staged rollout plan with a fast rollback mechanism.

The organizations that get the most value out of AI features are rarely the ones with the most impressive demo; they're the ones that built the unglamorous evaluation and monitoring infrastructure to know, continuously, whether the system is actually working — and to catch it quickly when it isn't.