A team built a RAG agent for customer support. To know whether it works, they've been reading the agent's final answers and judging whether each "looks right." They want to deploy it more broadly but the VP is asking "how do you know retrieval is actually working?"
What's the right evaluation approach?
Why did you pick that answer? Two or three sentences. The act of articulating it is what builds the judgment — not the click that follows.
Component-level evaluation beats outcome-only evaluation. When you measure only the final answer, you can't tell whether bad answers are due to retrieval (the system found the wrong documents) or generation (the model misread good documents) — and you can't fix what you can't isolate. Direct retrieval metrics on a labeled test set tell you "the retriever is/isn't finding the right documents" as a separate question from "the model is/isn't using them well." This is one of the most important moves in RAG evaluation.
User satisfaction is one signal — but it bundles retrieval, generation, latency, UX, and topic coverage into a single number. You can't debug RAG with bundled metrics; you need to know which component is failing.
Reading 100 conversations is non-systematic. Different reviewers will disagree, edge cases won't be caught, and you can't produce a reproducible "retrieval recall" number. Useful as a complement, not as the primary measurement.
LLM-as-Judge is useful — but for answer evaluation, not for retrieval evaluation. Asking an LLM to judge whether an answer is good doesn't tell you whether retrieval found the right documents. Component metrics first; LLM-as-judge for the parts where it's actually informative.