LLM-as-judge pitfalls
“What are the pitfalls of using an LLM as a judge, and how do you mitigate them?”
What this tests
- Knowledge of concrete biases: position, length, self-preference, verbosity
- Calibration against human labels
- Rubric design and pairwise versus absolute scoring
- Knowing when not to use a judge
Answers by level
Read the beginner answer first and notice what is missing.
Known biases: position bias (prefers the first or second option in pairwise comparisons), length and verbosity bias (longer looks better), self-preference (rates outputs from its own model family higher), sycophancy toward confident tone, and insensitivity to factual errors when the text reads well. Rubric-free 1–10 scores are also poorly calibrated and drift between runs. See LLM-as-Judge.
Mitigations: a specific rubric with observable criteria per score level; pairwise judgements with order swapped and results averaged; a different model family than the one under test; provide the reference answer or the source documents so the judge checks facts rather than fluency; and calibrate by measuring agreement with a human-labeled set, reporting it, and re-checking when the judge model or prompt changes.
And use the judge only where a deterministic check cannot work. Format, presence of ids, forbidden actions, numeric results are all coded checks. See Deterministic Evaluators.
Green flags · Red flags
- Names position, length, self-preference, fluency bias
- Order swapping, different model family, reference-grounded judging
- Binary rubric criteria over holistic scores
- Measures judge agreement with human labels and tracks it over time
- Versioned judge prompts and models
- Uses deterministic checks wherever possible
- Trusts a strong model with a vague prompt
- No calibration against humans
- Holistic 1–10 scores as the primary metric
- Unaware of injection into judged content