The argument for LLM-as-judge is honest and short: most of what makes an output good is subjective, humans who can label it are expensive and slow, and deterministic checks cannot tell helpful from hollow. So you point a model at the outputs of another model, hand it a grading prompt, and get back quality scores at a thousand per hour for pennies. The practice is now universal, and so is the mistake that comes with it: teams adopt the judge and skip the part where they find out whether it judges anything.

I wrote in the evals essay that a judge is useful, biased, and not a substitute for ground truth. This is the full version of that paragraph: what the biases actually are, the engineering that tames them, and the arithmetic that tells you what an imperfect judge costs. The one-sentence summary is the title. A judge without measured agreement against human labels is not an eval, it is a mood.

The Biases Are Systematic, Not Noise

If judge errors were random, scale would wash them out. They are not random. They point in consistent directions, which means more judging produces more confident wrongness, and every one of these has been reproduced enough times that you should assume your judge has it until shown otherwise.

Position bias. In pairwise comparisons, judges favor one slot, often the first. Run A-then-B and B-then-A and watch a measurable fraction of verdicts flip. Any pairwise result produced without randomized or mirrored ordering has this baked in.

Verbosity bias. Longer answers score higher at equal correctness. Your judge is partly measuring word count, which is a problem, because your users mostly want the opposite.

Style and self-preference. Judges prefer answers that sound like them: same hedging, same structure, same register. A model grading its own output family is a proud parent, which is why the judge should usually come from a different family than the system under test.

Confidence bias. Assertive wrong answers outscore hedged right ones. The judge reads fluency as competence, exactly the failure that makes human graders unreliable, now at scale.

Score compression. Ask for 1 to 10 and you get a distribution that huddles between 6 and 8. The scale looks continuous; the information content is roughly one bit and a half. This is not a bias so much as a confession: the judge does not have ten distinguishable quality levels, and pretending it does just hides the signal.

Rubric Engineering: Make the Question Answerable

Most judge failures are prompt failures. "Rate this response's quality from 1 to 10" is not a question a careful human could answer consistently, and the judge inherits every ambiguity you hand it. The fixes are the same ones you would apply to a human grading team, applied with more discipline because the grader cannot push back.

Score criteria separately. One overall number blends correctness, tone, completeness, and format into something unexplainable. Judge each criterion on its own, and your eval results become diagnoses instead of grades: correctness fell, tone held.

Use the smallest scale that carries the decision. Binary per criterion, pass or fail against a stated bar, is where I start, with a three-point scale as the ceiling. Every anchor needs a concrete description of what qualifies, because an anchor the rubric does not define is a number the judge invents.

Demand evidence. Require the judge to quote the specific text that justifies each verdict before scoring, and emit it as a field you can read. Judges forced to cite are measurably harder to drift into vibes, and when you audit disagreements later, the quote tells you instantly whether the judge was looking at the right thing.

Make the output a contract. Verdicts, quotes, and scores come back as structured output against a schema, with the reasoning field ordered before the verdict field so the judge reasons its way to the answer instead of rationalizing one it already emitted. Temperature zero, pinned model version, pinned rubric version, both recorded with every run.

Randomize what can be randomized. Pairwise order, option labels, anything positional. Pairwise comparison with mirrored ordering is generally more reliable than absolute scoring and I reach for it whenever the question is "did this change help," which is most questions.

Calibration: The Step That Makes It an Instrument

Everything above improves the judge. None of it tells you whether to trust it. Only one thing does, and it is the step teams skip because it involves the manual labor they bought the judge to avoid: label a few hundred cases yourself, run the judge on the same cases, and measure the agreement.

The protocol fits in a paragraph. Pull 200 or so real cases spanning your quality range, including the ugly middle, not just clean passes and clean failures. Have the people who actually know what good looks like label them against the same rubric the judge uses. Compare. Raw percent agreement is the headline, but check it against the base rate, because on a set that is 90 percent passes, a judge that always says pass scores 90 percent agreement while measuring nothing. Where the judge and humans disagree, read every case: half the time the finding is that your rubric is ambiguous, which is worth the exercise on its own. And do not treat calibration as permanent. The judge gets recalibrated when the judge model changes, when the rubric changes, or when the task drifts, and a small judge regression set, human-labeled cases the judge must keep agreeing on, catches the silent version of that drift.

Then price the agreement into your eval math, because an imperfect judge does not just add noise, it dilutes every measurement you make through it. The arithmetic is brutal and most teams have never run it: a judge that agrees with your human labels 85 percent of the time roughly doubles the number of eval cases you need to detect the same quality difference, and the multiplier grows fast from there. I built that penalty into the sample size calculator because it changes decisions: a fifty-case eval read through an 85 percent judge can only see quality swings so large you would have noticed them without an eval.

Where a Judge Does Not Belong

The judge earns its place in the middle layer of the eval stack: scaled scoring of subjective quality, triaging which outputs deserve human eyes, tracking drift week over week. It does not belong everywhere, and the discipline of keeping it out of the wrong jobs matters as much as tuning it for the right ones.

Anything a deterministic check can decide, a deterministic check decides. Format, schema, required fields, citation-exists, length bounds: these are assertions, they never lie, and routing them through a judge trades certainty for a probability at higher cost. Anything that gates a consequential action, a refund issued, a record deleted, content published, gets decided by rules and scoped capabilities, not by a model's opinion of another model's output. And the final word on whether the system is actually good belongs to ground truth: user behavior, task completion, the production signals the whole eval stack exists to approximate. The judge approximates the approximation. It is a powerful instrument once calibrated, and like any instrument, the calibration sticker is the difference between measurement and decoration.

FAQ

Is LLM-as-judge reliable?
Only after engineering and calibration. Uncalibrated judges carry position, verbosity, style, and confidence biases, with unknown agreement against careful humans. With concrete per-criterion rubrics, randomized ordering, a pinned version, and measured agreement on a few hundred labels, a judge becomes an instrument with a known error rate.

Which model should judge?
A strong model, pinned to an exact version, usually from a different family than the system under test. Discipline beats capability: a mid-tier judge with a tight rubric and calibration outperforms a frontier judge scoring 1 to 10 on vibes.

Can a model judge its own outputs?
It will flatter its own style. For decisions that matter, judge with a different family, prefer mirrored pairwise comparisons, and verify against human labels on your task.

How many human labels for calibration?
A few hundred is the practical floor. Then price the result into eval sizes: 85 percent agreement roughly doubles the cases needed to detect the same difference, and the penalty grows fast below that.