LLM-as-judge, and where it fails
A model grading a model is cheap enough to run on every commit and biased in named, measured ways — position, verbosity, self-preference — so knowing the failure modes is the price of being allowed to cite the score.
Once evaluation moves onto live traffic there is no reference answer, so something has to stand in for one. In practice that something is another model, prompted to grade the output. It costs cents, it runs on every commit, and it will produce a number for any question you can phrase.
Which is the problem. A judge score is the cheapest authoritative-looking number in the building, and you are going to be in rooms where one is used to close an argument about your surface. The price of being allowed to cite it — or to push back on it — is knowing which of its failure modes have actually been measured, and by whom.
Position bias, measured
Show a judge two answers and ask which is better. Then swap them and ask again. A judge with no position bias returns the same verdict both times. Zheng and colleagues ran exactly that and reported consistency under answer-order swapping of 65.0% for GPT-4, 46.2% for GPT-3.5 and 23.8% for Claude-v1. Adding few-shot examples to the judge prompt raised GPT-4 to 77.5%.
Read the middle number again. GPT-3.5, as a judge, agreed with itself under a swap less than half the time. Those are 2023 model versions and the numbers will not transfer to whatever you are running; the transferable thing is the test, which takes an afternoon and which almost nobody runs on their own scorer.
For scale, Shi and colleagues studied position bias across 15 judges over 22 tasks and more than 150,000 evaluation instances, and report that it is not random: it varies by judge and by task, and is strongly affected by the quality gap between the two answers. Only that paper’s abstract page was opened for this course, so no per-judge figure from it appears here.
Verbosity bias, measured
The attack is blunt. Take an answer, pad it into a repetitive list that adds no information, and ask the judge to choose. On that construction, Zheng and colleagues report the padded answer winning against Claude-v1 and GPT-3.5 91.3% of the time, and against GPT-4 8.7% of the time.
That is a fourteen-fold difference between judges on the same attack, which is the point worth carrying rather than either number. “We use an LLM judge” is not a specification. Two judges given the same rubric disagree about length by an order of magnitude.
The bias the famous paper could not confirm
It is common to see position, verbosity and self-preference listed together as the three measured biases of LLM judges, all attributed to Zheng and colleagues. Two of those three are theirs. The third is not.
The paper studies what it calls self-enhancement bias — a judge favouring its own generations — and states plainly that it does not find significant evidence that GPT-4 or Claude favour their own responses. It observed a small effect it describes as one it cannot confirm with its data, and said so. Citing that paper as the source for “self-preference, measured” inverts what it actually reports.
Self-preference, measured — by a different paper
The bias is real and it does have a source. Arjun Panickssery, Samuel R. Bowman and Shi Feng measured it directly in “LLM Evaluators Recognize and Favor Their Own Generations”, at NeurIPS 2024. Their definition is the sharp one to carry: self-preference is where a model evaluator “scores its own outputs higher than others’ while human annotators consider them of equal quality.” Two findings from the abstract:
- Out of the box, models including GPT-4 and Llama 2 have non-trivial accuracy at distinguishing their own outputs from those of other models and of humans.
- By finetuning models, the authors find a linear correlation between that self-recognition capability and the strength of self-preference bias, and report controlled experiments showing the causal explanation resists straightforward confounders.
Only the proceedings abstract was opened for this course, so no accuracy or correlation figure from the full paper is quoted anywhere in it. The finding as stated is enough for the design conclusion, which is narrow and practical: a judge grading output from the model family it belongs to is not a neutral instrument, and the better it is at recognising that output, the less neutral it gets.
So the correct sentence in the room is not “the three known biases are position, verbosity and self-preference.” It is: position and verbosity were measured by Zheng and colleagues in 2023, that same paper could not confirm self-preference, and self-preference was measured by Panickssery, Bowman and Feng in 2024. Attribution at that resolution is the difference between sounding well-read and being useful.
The agreement number, and how it gets misquoted
The figure everybody repeats from Zheng and colleagues is that a judge agreed with humans 85% of the time. Here is the whole claim, with the qualifiers that never survive the retelling: on MT-Bench first-turn questions, counting only non-tie votes, GPT-4 agreed with human preferences 85% of the time, against 81% agreement between two humans on the same comparisons; on Chatbot Arena the figure was 87%.
Three things that number is not.
- It is not general. It is one benchmark, one turn, one judge, one vote-filtering rule. There is no such fact as “LLM judges agree with humans 85% of the time.”
- It is not chance-corrected. Raw percent agreement rewards a judge for guessing the majority label. A reliability statistic that accounts for agreement by chance, such as Cohen’s kappa, is a different and lower number, and this is not it.
- It is not separable from the bias numbers. The same paper that produced the 85% produced the 23.8% position consistency and the 91.3% verbosity result. Quoting the first without the others is the move a vendor makes.
Whenever you cite an agreement rate, name the dataset and the task in the same sentence. A bare agreement percentage with no dataset attached is not a weak citation. It is a fabricated figure.
What this changes about your gate
Three concrete consequences for the instrumentation spec, and they are the reason an interface designer needs any of this.
A judge score is a metric with a second reading, like the others. A rising judge score means the output improved, or the output got longer, or the prompt template started putting the new variant second. The same discipline the metric set applies to correction rate applies here, and it applies to a number produced by someone else’s system that will be quoted alongside yours.
Human corrections are the only unjudged signal in the loop. When online scoring substitutes a judge for a missing reference, the review gate is the one place a human verdict still enters the system. That raises the value of capturing corrections well, and it is why the corrections lesson closes this module.
Judge scores belong in the trace, not in product analytics. They are per-run engineering artefacts about a model, not aggregate behavioural facts about people. The routing lesson in the instrumentation module argues that split in full.
Check your recall
Answer from memory — no scrolling back.
Retrieval check
Someone shows you an online eval dashboard where the judge score rose 8% after a prompt change. What do you ask before agreeing that the output improved?
Check your answer
Whether the outputs got longer. Verbosity bias is measured, it is large on some judges, and a prompt change that produces fuller answers will move a judge score without moving quality.
Then, whether the judge belongs to the same model family as the system under test, because Panickssery and colleagues found self-preference rising with a model’s ability to recognise its own generations. And whether the comparison is pairwise and, if so, whether the order was randomised, because position consistency under a swap has been measured below 50% on some judges.
None of those three questions is an accusation. Each one has a cheap answer, and asking them is the reason the score gets to be cited at all.
Hands on
Write the judge caveats into the spec, not into your memory
Done when: Section 4 of INSTRUMENTATION.md contains a judge row naming the scorer, its model family, whether order is randomised, and who owns the answer — and you can state the three biases with the right paper attached to each, out loud, without notes.
- Add a row to section 4 for the judge itself: which model runs the scorer, whether it is from the same family as the model being scored, and whether pairwise comparisons randomise order. Owner: whoever runs the eval stack.
- Write the three-sentence attribution and say it aloud until it is fluent. Position and verbosity, Zheng and colleagues, 2023. The same paper could not confirm self-enhancement. Self-preference, Panickssery, Bowman and Feng, 2024.
- If you have a judge prompt within reach, run the swap test on ten pairs and write down the consistency. Ten is too few to conclude anything and enough to find out whether you can run the test at all.
- Add a line to section 1 reserving an event for a judge score attached to a reviewed item, routed to the trace. You will define its properties in the corrections lesson; for now it is a placeholder with a stated destination.
- Bring the attribution into the chat and I will misquote it at you in three different ways. Your job is to name which qualifier went missing.
What this does not cover
Nothing here tells you how to write a grader, choose a scoring model or tune a judge prompt. That is eval ownership, and the mission puts it out of scope deliberately: you need enough to interrogate a score, not enough to produce one.
The tooling lesson comes next and is where the abstraction stops. It puts LangSmith, promptfoo and Braintrust side by side, names what each one actually answers, and asks you to run one against a small dataset so that the opinion you carry into an interview came from a terminal rather than a landing page. The corrections lesson then closes the loop, turning what reviewers fix at the gate into the dataset an eval could run against.
Read this next — primary source
Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaLianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez & Ion Stoica, NeurIPS 2023 Datasets and Benchmarks — open access, fetched 2026-09-05
The paper that both introduced the practice at scale and measured its failure modes, which is why this lesson cites it rather than any vendor’s claim about its own scorers. Read it in full for two reasons. The attack constructions are the transferable part: position bias is measured by swapping the two answers and checking whether the verdict survives, verbosity bias by padding an answer with no new information. And the paper is unusually honest about the bias it could not establish, which is the exact discipline this course is trying to teach.
Stuck, curious, or think this lesson is wrong? Ask your teaching agent. The lessons are the scaffold; the conversation is where the learning gets unstuck.