LLM evals
Last verified: June 2026· definition
An eval (evaluation) is a repeatable test for an LLM-powered feature. It pairs representative inputs with a way to score the outputs — exact match, a rubric, or an LLM-as-judge — so you can tell whether a prompt change, model swap, or retrieval tweak made quality go up or down.
Evals run offline (against a fixed dataset, ideally in CI) and online (sampling real production traffic). Together they turn quality from a vibe into a number you can gate deploys on.
Most teams shipped AI features with no evals, which makes every change a blind deploy. Adding even a small eval suite to the highest-stakes route is the highest-leverage reliability investment most teams can make.
Start with the route that would hurt most
Comprehensive eval coverage is a good end state and a bad starting point, because the effort required stops most teams before they have anything. The productive move is picking the single route where a quality regression would cause the most damage and building a small suite for it — a few dozen representative inputs with a scoring method. That is enough to catch the failure that matters and small enough to actually finish, and it establishes the pattern for everything after.
Choose the cheapest scoring method that works
Not everything needs a sophisticated judge. Exact match or schema validation handles classification, extraction, and structured output at near-zero cost and with perfect reliability. Rubric-based checks handle constrained generation. Reserve LLM-as-judge for genuinely open-ended output where no cheaper method applies, since it is the slowest, most expensive option and the only one that itself needs validating.
Offline and online evals answer different questions
Offline evals run against a fixed dataset and tell you whether a change is safe to ship — they belong in CI, gating deploys. Online evals sample real production traffic and tell you whether quality is holding against inputs you did not anticipate, which is where drift and unexpected usage patterns surface. A system with only offline evals passes its tests while degrading in production; one with only online evals discovers regressions after users do.
Common misconceptions
MythYou need a large eval set for it to be worth building.
RealityA few dozen well-chosen cases on a high-stakes route catches most regressions. The gap between no evals and a small suite is far larger than the gap between a small suite and a comprehensive one.
MythEvals are a testing concern you add before launch.
RealityThey are what makes iteration safe, so their value is highest during development. Without them every prompt change, model swap, and retrieval tweak is an unmeasured guess.
MythAn LLM judge is the standard way to score.
RealityIt is one method, and the most expensive. Exact match and schema validation are cheaper, faster, and more reliable wherever the task admits them, which is more often than teams assume.
Frequently asked questions
How do I build an eval set if I have no labelled data?
Take real production inputs and have someone who understands the domain judge the outputs. You do not need pre-existing labels — you need agreement on what good looks like for a few dozen representative cases, including the awkward ones. Sampling actual traffic matters more than volume, because a set built from imagined inputs tests a distribution your system does not encounter.
How often should evals run?
Offline evals belong in CI on every change touching prompts, models, or retrieval, since those are exactly the changes that silently alter quality. Online evals should sample continuously at a rate you can afford to review. The important property is that a regression is caught by the system rather than by a user, which requires evals to run on the same cadence as the changes that cause regressions.
How do I know if my LLM judge is trustworthy?
Validate it against human judgement before relying on it. Score a sample with the judge and independently with a person, then compare — if they disagree often, you are measuring the judge’s opinion rather than your quality bar. This calibration step is frequently skipped, which produces eval suites that report stable numbers while actual quality moves underneath them.
Go deeper
How to reduce LLM API costs
Most production LLM spend is avoidable: it comes from sending the wrong model too many tokens, too many times. Cutting it is an engineering problem — routing, caching, batching, and context control — not a matter of waiting for prices to drop.
LLM evalsHow to build LLM evals
Evals turn 'the model feels better' into a number you can gate a deploy on — they are the test suite that makes every prompt, model, and pipeline change safe to ship.
Hiring & partnersHow to hire an LLM cost optimization consultant
Hire an LLM cost consultant when your bill is large enough to matter but your team is too busy to instrument, route, and eval their way out of it themselves — and only hire one who leaves you owning the result.