A team uses Claude as an LLM-as-Judge to score customer support agent responses on a 1–10 scale. The judge prompt is: "How good is this response? Score 1–10." After a month, scores have drifted upward — the judge gives 8s and 9s where it used to give 6s and 7s. Nothing about the agent has changed.
What's the most likely cause?
Why did you pick that answer? Two or three sentences. The act of articulating it is what builds the judgment — not the click that follows.
Vague rubrics produce vague (and drifting) scores. "How good is this response?" gives the judge no anchor — its sense of "good" floats based on what it has seen recently in context. Replacing the prompt with explicit anchor descriptions ("a 9 has all of X, Y, Z; a 5 has at most X; a 3 fails on Z") and concrete examples (one sample response per score level) calibrates the judge to definitions rather than to its own internal sense of quality. Replace adjectives with categorical criteria — one of the most reliable moves in prompt engineering, applied here to evaluation.
There's no described mechanism by which the judge sees its own past scores. Reaching for "feedback loops" when the simpler explanation (vague rubric) is sitting right there is a debugging anti-pattern.
Provider model upgrades happen, but blaming an unannounced upgrade before checking your own rubric is misallocating diagnostic effort. The rubric is something you can inspect and fix.
Drift in a structured measurement system is rarely random — it's usually pointing to under-specified evaluation criteria. Averaging across runs smooths the symptom while leaving the cause in place.