LLM-as-Judge

LLM-as-judge evaluation uses a model to score, classify, critique, or compare outputs from another system. It can scale qualitative review for RAG evaluation, but it is not ground truth. The judge prompt, candidate order, rubric, model version, and sampling settings are part of the evaluation artifact.

Judge-human agreement

For pairwise judging, compare the judge label $J_i\in \{A,B,\text{tie}\}$ with a human label $H_i$ on an audit sample. Agreement and kappa measure whether judge outputs are usable as a proxy:

$$\text{agreement}=\frac{1}{n}\sum _{i}1(J_i=H_i).$$

This audit should sit next to human evaluation, not replace it. If the judge chooses the first answer too often, randomize answer order and estimate the bias.

Worked calculation

On a twelve-example audit, the human and judge labels compare as follows:

audit statistic	value
exact agreements	10 of 12
raw agreement	0.833
Cohen’s kappa	0.733
judge A-share after removing ties	0.500

The judge agrees with the human labels on 10 of 12 examples and shows no A-side preference in this small audit. That is promising, but too small for a production claim; use repeated sampling and slice audits before trusting automated scores.

Caveats

LLM judges can reward verbosity, miss subtle factual errors, prefer outputs from similar models, and leak rubric assumptions into prompts. For abstention, the rubric must penalize both unsafe answers and unnecessary refusals. Keep a blind human audit set to detect drift in judge behavior.

References

• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• scikit-learn documentation: cohen_kappa_score

Nav

Section — Experimentation and Evaluation

← Human Evaluation Comparing Generative AI and Classical ML Systems →