06
Domain 06
Knowledge Agents & RAG
When to ground agents in your data, retrieval failure modes, "I don't know" patterns, and how to evaluate retrieval quality.
5 questions
- Q76What's the right architecture?A consulting firm's assistant doesn't know the firm's own past engagements. 12 years of memos sit in SharePoint.
- Q77What's the right behavior?A RAG-based legal agent is asked about a niche area where the firm has no past memos. It confidently answers anyway.
- Q78What's the architectural fix?A policy agent answers based on a 2022 version of a policy that's been superseded twice — and reps make commitments management overrides.
- Q79What's the architectural fix?Two versions of the remote-work policy exist in the index. The retriever returns one. The agent never sees the conflict.
- Q80What's the right evaluation approach?A team built a RAG agent and judges quality by reading final answers. The VP wants to know if retrieval is actually working.