Who Judges the Judge? Calibrating LLM Evaluators Against Human Labels.

LLM-as-judge has become the default way to evaluate LLM applications at scale. Most teams adopt it with a plausible-sounding prompt, a handful of spot checks, and a dashboard number nobody can put an error bar on. In this workshop we treat the judge as what it is -- a measuring instrument -- and calibrate it the way any instrument should be calibrated. Participants work through the full loop: 1. Error analysis We read real model outputs, cluster the failures, and turn them into a small set of binary, observable criteria. This replaces "rate helpfulness from 1 to 10" with criteria two people could agree on. 2. Human judges Participants label a sample in pairs and measure their own agreement. They usually disagree more than they expect. We adjudicate the disagreements, sharpen the criteria, and split the labels into development and held-out test sets. 3. LLM judges We implement one judge per criterion and evaluate it against the adjudicated labels. We report true positive and true negative rates separately and add confidence intervals that are honest about small samples. A single accuracy figure can hide a judge that misses most real failures. 4. Probe for bias We test position bias, verbosity bias, and sensitivity to prompt wording, then iterate on the judge without overfitting to the development set. 5. LLM judge calibration Every judge makes errors. We show how to correct an observed pass rate using the judge's measured error rates, how to bootstrap an interval around the corrected estimate, and how to decide whether the judge is good enough to guard a release. --- Participants leave with a working, framework-light calibration pipeline, a measured judge with known error rates, and a procedure they can apply to their own evaluation criteria the following week. The workshop draws on experience building and validating LLM-judge pipelines for e.g. a production customer support agents, and on many years of adjudicating competitive debates. Human judging communities were solving rubric design, calibration, and inter-rater disagreement long before LLM judges existed. Prerequisites: comfortable Python (pandas level); a laptop with Python 3.11+ and an OpenRouter API key with credit on the account. No GPU and no ML training background needed. Basic familiarity with precision and recall helps.