A RAG-based legal research agent is grounded in a firm's library of past memos. A junior attorney asks about a niche area of cryptocurrency tax law — an area where the firm has no past memos. The retriever returns three passages with low relevance scores (the highest is 0.31, well below the firm's typical threshold of 0.65). The agent confidently produces an answer based on... nothing relevant.
What's the right behavior?
Why did you pick that answer? Two or three sentences. The act of articulating it is what builds the judgment — not the click that follows.
Detecting and surfacing retrieval failures is one of the most important behaviors in a RAG agent. When relevance scores are below threshold, the responsible move is to say so — distinguish base-model knowledge ("here's general background") from firm-grounded answers ("here's what we've done before"). That distinction is exactly what makes a RAG agent trustworthy. The "always confidently answer" pattern is what makes RAG agents fail expensively.
A confident answer from the base model defeats the entire RAG architecture — the user thought they were getting firm-grounded guidance and got generic guidance presented identically. The fix is honest signaling, not silent fallback.
A 0.31-relevance passage is noise. Including it pretends the answer is grounded when it isn't. Worse, the model will partially attend to whatever is there, which can produce subtly wrong answers framed as if they're from firm precedent.
A refusal helps no one. Honest signaling ("I don't have firm-specific guidance, but here's general background — verify with a partner") is dramatically more useful than either confident hallucination or silent refusal.