LLM-as-judge
Last verified: June 2026· definition
LLM-as-judge uses a capable model to grade outputs that have no single correct answer — summaries, explanations, generated code review. You give the judge a rubric, and it returns a score or pass/fail per criterion.
It’s powerful for nuanced, open-ended quality, but the judge itself must be validated against human labels so you’re not just measuring one model’s opinion of another. Calibrating the judge is part of building a trustworthy eval.
In production eval systems, LLM-as-judge is usually combined with cheaper exact-match and rubric checks, each applied to the routes where it fits best.
The judge must be calibrated against humans
An unvalidated judge measures one model’s opinion of another, which is not the same as measuring quality. Calibration means scoring a sample with the judge and independently with a person who understands the domain, then comparing. Where they disagree systematically, the rubric needs sharpening or the task is not suited to automated judging. Teams frequently skip this and end up with an eval that reports stable numbers while actual quality drifts underneath.
Rubrics beat general quality prompts
Asking a judge whether an answer is good produces inconsistent scores because the criterion is undefined. Breaking quality into specific checkable dimensions — is it grounded in the source, does it answer the question asked, is the format correct, does it avoid unsupported claims — produces far more stable judgements. It also makes disagreement diagnosable: you can see which dimension the judge and a human read differently, rather than staring at two different overall scores.
Known biases to design around
Judges tend to prefer longer answers, favour responses stylistically similar to their own output, and are sensitive to presentation order when comparing candidates. None of these correlate with quality. Practical mitigations include randomising order in pairwise comparisons, scoring against a rubric rather than picking a winner, and checking whether your judge’s scores correlate suspiciously well with response length.
Use it only where cheaper methods fail
LLM-as-judge is the slowest and most expensive scoring method and the only one requiring its own validation. Exact match, schema validation, and deterministic rubric checks are faster, cheaper, and perfectly reliable wherever the task admits them. Reserving the judge for genuinely open-ended output keeps eval suites affordable enough to run on every change, which is the property that makes them useful.
Common misconceptions
MythA capable model makes a reliable judge by default.
RealityCapability is necessary and not sufficient. Without calibration against human judgement you do not know whether its scores track your quality bar or something else.
MythLLM-as-judge is the standard way to evaluate.
RealityIt is one method and the most expensive. Deterministic checks cover more evaluation work than teams expect and should be preferred wherever they apply.
MythA consistent score means an accurate score.
RealityA judge can be reliably consistent and consistently wrong. Consistency is a prerequisite for a useful metric, not evidence that it measures what you intended.
Frequently asked questions
How do I validate an LLM judge?
Take a sample of outputs, score them with the judge, and have a domain expert score them independently without seeing the judge’s verdict. Compare the two. Broad agreement means the judge is usable for that task; systematic divergence means the rubric is ambiguous or the criterion is not one a model can assess reliably. Repeat this whenever you change the rubric or the judge model, since both invalidate prior calibration.
Should the judge be the same model being evaluated?
Preferably not. Models show some tendency to favour output resembling their own, which biases the evaluation in a direction that is hard to detect from the scores alone. Using a different model, ideally from another family, reduces this. Where using the same model is unavoidable, calibration against human judgement matters even more than usual.
What tasks are unsuitable for LLM-as-judge?
Anything with a deterministic correct answer, where exact match is cheaper and perfectly reliable. Also anything requiring knowledge the judge lacks — factual accuracy against private data, for instance, where the judge cannot verify the claim any better than the model that made it. Judging works best on qualities a careful reader could assess from the output and its source material alone.