How to build LLM evals

What an LLM eval is — and why you need one

An eval is a repeatable measurement of how well an LLM system does a task, computed over a fixed set of inputs. Concretely it is three things: a dataset of cases, a way to produce the system's output for each case, and a grader that scores each output. Run all three and you get a number — an aggregate score across the set — that means the same thing today as it did last week. That reproducibility is the entire point. It lets you compare two versions of your system on equal footing instead of arguing about vibes.

The word 'evaluation' in the LLM world is used loosely, so it is worth being precise. It is not the same as unit testing (deterministic code with one right answer), and it is not the same as a public leaderboard benchmark (which measures a base model on generic tasks, not your system on your task). An eval measures your pipeline — prompt, model, retrieval, tools, and all — on inputs drawn from your problem. A model that tops every public benchmark can still fail badly on your specific use case, which is exactly why you build your own.

The reason to invest here is leverage. Once an eval suite exists, model upgrades become a measurement rather than a leap of faith, prompt tweaks stop silently regressing edge cases, and cost-reduction work (cheaper models, shorter context) can be pushed exactly until quality drops. Evals are the safety rail that lets a team move fast on an inherently non-deterministic system.

The four approaches to LLM evaluation

There is no official taxonomy, but a useful way to categorise evals is by how the grader decides whether an output is good. Four broad approaches cover most of what teams actually do, and real systems usually combine several.

1. Reference-based (ground-truth) evals. You have a known-correct answer for each case and compare the output against it. When the task has a crisp answer — a classification label, an extracted field, a number, a SQL query that returns a specific result — this is the strongest signal you can get, because it is objective and cheap to run. Graders range from exact match and normalised string match, through set-overlap metrics like F1 for extraction, to overlap metrics like BLEU/ROUGE for tasks with a canonical target text. The catch: most open-ended generation has no single correct answer, and surface-overlap metrics reward matching words rather than matching meaning, so they degrade fast on free-form output.

2. Human evaluation. A person reads the output and judges it, ideally against a written rubric (correct? grounded? appropriate tone?). This is the gold standard for quality and the thing every other method is ultimately trying to approximate — humans catch nuance nothing else does. It is also slow, expensive, and hard to keep consistent between raters, so you cannot run it on every commit. Its real job is to produce the labelled data that validates your cheaper automated graders and to adjudicate the cases those graders disagree on.

3. Model-graded (LLM-as-judge). You use an LLM to score the output, given the input, a rubric, and often a reference answer. This scales human-style qualitative judgement to thousands of cases at low cost, which is why it has become the default for open-ended tasks. It is also the approach with the most failure modes — it needs its own validation, covered below — so treat a judge model as a component you test, not an oracle you trust.

4. Task / behavioural (outcome-based) evals. Instead of grading the text, you check whether the end task actually succeeded. Did the generated code compile and pass tests? Did the agent's tool calls produce the right final state? Did the SQL return the expected rows? Did the multi-step business task reach the correct outcome? This is the most honest measure for agents and tool-using systems because it rewards the thing you actually care about — the result — and ignores whether the prose along the way looked nice. It is harder to set up (you need an executable environment and a definition of 'succeeded'), but for anything long-horizon it is the approach that best predicts real-world usefulness.

Choosing metrics for your task

The right metric follows from the shape of your task, not the other way round. Start by asking: does a case have a single correct answer? If yes — classification, extraction, routing, structured output — use reference-based metrics and be done; they are cheap, objective, and you should prefer them wherever they apply. If the output is open-ended, you are choosing between an LLM-as-judge rubric and, where possible, an outcome check.

Break a compound feature into sub-tasks and evaluate each with the metric that fits. A RAG system, for example, is really two evals: retrieval (did we fetch the right documents — a reference-based set-overlap question) and generation (is the answer grounded in what we retrieved and actually correct — a judge or human question). Averaging them into one score hides which half is failing. Keep them separate so a regression points at a cause.

Define failure modes explicitly and make sure at least one metric would catch each. Common ones worth their own check: hallucination (claims not supported by the source), format violations (broken JSON, wrong schema — often a cheap deterministic check), refusals on valid requests, instruction-following lapses, and unsafe or off-policy content. A single 'quality' number that blends all of these tells you something moved but not what, so track a small panel of specific metrics rather than one grand average.

Finally, resist metric theatre. Ten decimal-place scores that no one acts on are worse than three metrics wired to a pass/fail threshold you have agreed on. Every metric should answer the question 'would this block a deploy?' — if the answer is no, it is dashboard decoration, not an eval.

LLM-as-judge: where it shines and where it bites

Using a model to grade another model's output is the only practical way to scale nuanced, open-ended evaluation, and when it is done carefully it correlates well with human judgement. It works best as a focused rubric grader: give the judge the input, the output, a reference answer where you have one, and a narrow, concrete question — 'Is every factual claim in the answer supported by the provided context? Answer yes or no with the unsupported claim if no.' Narrow, well-specified questions get reliable answers; vague ones ('rate the quality 1–10') get noise.

The pitfalls are real and you must design around them. Judges show position bias (favouring the first option in a pairwise comparison — mitigate by running both orderings and averaging), verbosity and self-preference bias (preferring longer answers, and preferring outputs from the same model family), and rubric drift where the judge quietly reinterprets vague criteria across runs. A judge is also blind to anything outside its own knowledge, so it can confidently pass a subtly wrong answer.

The rule that keeps you honest: a judge is itself a model that must be validated against human labels before you trust it. Take a sample of cases, have humans grade them against the same rubric, and measure how often the judge agrees. If agreement is high, run the judge at scale and re-check periodically. If it is low, tighten the rubric, add few-shot examples, or fall back to human grading for that slice. Never let an unvalidated judge gate a deploy — you would be trusting one non-deterministic system to police another with no ground truth in the loop.

Two practical guards: prefer a different, strong model as the judge than the one under test to reduce self-preference, and where you can, prefer a cheap deterministic or reference check over a judge — do not reach for an LLM to grade something that a regex or an equality check would settle.

Building the eval set from real traffic and failures

The dataset is where evals live or die, and the single most common mistake is inventing tidy examples at your desk. Those cases are drawn from your imagination of the problem, not the problem — they are systematically easier and cleaner than reality, so they pass while production breaks. Build the set from real inputs your system actually sees.

Start by mining logs. Sample real production or beta traffic to capture the true distribution of inputs, including the messy, ambiguous, and adversarial ones you would never think to write. Then mine your failures deliberately: every bug report, every thumbs-down, every 'the model did something weird' Slack message is a case that belongs in the set. A regression that reaches production once should never reach it twice — turn it into an eval case the moment you fix it, exactly as you would add a regression test after fixing a bug.

Label the cases. For reference-based and outcome evals you need the correct answer or success condition; for judged evals you need at least a human-graded subset to validate the judge. This labelling is the expensive part and it is unavoidable — budget for it. A few hundred well-chosen, correctly labelled cases that span your real distribution and known failure modes beat tens of thousands of synthetic ones. Cover the ordinary path, the edge cases, and the adversarial inputs; note how each was created so future maintainers can trust it.

Treat the eval set as a living asset under version control, not a one-off. It grows as you find new failure modes, and you should periodically audit it for stale labels and cases that no longer reflect the product. A frozen eval set slowly stops measuring the system you actually run.

Wiring evals into CI and online monitoring

An eval that a human remembers to run by hand is an eval that stops getting run. The value comes from automation on two fronts: offline in CI before a change ships, and online against live traffic after it does.

In CI: run the eval suite on every change that can affect model behaviour — prompt edits, model swaps, retrieval or tool changes, pipeline refactors — and gate the merge on it. Set explicit thresholds (a minimum aggregate score, or 'no metric drops more than X versus the current baseline') and fail the build when they are not met, exactly like a failing test. Keep the suite fast and cheap enough to run per-PR; if the full set is too slow or expensive, run a representative subset on every commit and the full set nightly. Store each run's scores so you can see the trend, not just today's pass/fail, and so a slow drift is visible before it becomes a cliff.

Online: the offline set can never cover everything real users will do, and input distributions shift over time, so keep measuring in production. Log inputs and outputs, sample live traffic, and run the same graders (especially the automated ones — deterministic checks and validated judges) against real interactions. Watch proxy signals too: format-violation rate, refusal rate, latency, user thumbs-up/down, and downstream task success. When an online metric degrades, that sampled traffic is your richest source of new eval cases — feed the failures back into the offline set, closing the loop between what you catch before shipping and what you learn after.

Done well, the two loops reinforce each other: online monitoring finds the cases you did not anticipate, those cases harden the CI suite, and the hardened suite stops the same failure from ever shipping again. That feedback loop, more than any single metric, is what makes an LLM system dependable over time.