A team is about to deploy LLM-as-Judge scores into production KPI reporting — the VP will see the judge's monthly quality scores in a dashboard. The team has tested the judge in development and it "seems to work."
What should they do before deploying?
Why did you pick that answer? Two or three sentences. The act of articulating it is what builds the judgment — not the click that follows.
An LLM-as-Judge is a measurement instrument; a measurement instrument has to be calibrated. Comparing the judge's scores to human-labeled scores on a representative test set tells you whether the judge agrees with the humans whose role it's replacing. If agreement is high, deploy. If it's low or systematically biased (always higher on certain response types, always lower on others), refine the prompt and re-calibrate. Skipping calibration deploys a measurement system you can't defend.
Claude is capable; that doesn't make it a calibrated judge for your specific quality criteria. "Out-of-box" thinking is exactly how vague-rubric drift goes undetected for months.
A quarter of production scores with no ground truth produces no calibration — you have data but nothing to compare it to. Calibrate before, not during.
This is the genuinely tempting answer because it is a real measurement — the judge agreeing with itself across runs is consistency, which is necessary but not sufficient. The problem: a judge can be perfectly consistent and systematically wrong (always scoring helpfulness too high, always undervaluing accuracy). Self-agreement measures whether the same bias appears twice, not whether the bias is correct. Calibration requires comparison to an external ground truth — which is what human scoring on a labeled set provides.