A consulting firm's Claude-based assistant gets asked questions about the firm's past work — "What was our recommendation for that retail client last year?", "Have we ever helped a healthcare company with a similar problem?" The assistant currently either makes things up or says it can't access that data. The firm has 12 years of engagement memos, slide decks, and post-mortems sitting in a SharePoint archive.
What's the right architecture?
Why did you pick that answer? Two or three sentences. The act of articulating it is what builds the judgment — not the click that follows.
**RAG is the right pattern when an agent needs to ground its responses in your data** — data that's too large to fit in context, too dynamic to bake into a model, and too specific for the base model to know. The retrieval step finds the most relevant past engagements; the generation step grounds the assistant's answer in those passages with attribution. This is the architectural answer to the "the agent doesn't know our business" problem.
Fine-tuning is expensive, slow, and produces a static snapshot. New engagements would require re-training. RAG handles new data automatically — index it and it's retrievable. For dynamic knowledge bases, retrieval beats training every time.
12 years of engagement summaries is hundreds of thousands of tokens. Stuffing them into every system prompt burns those tokens on every query, including queries that have nothing to do with most of the engagements. Lost-in-the-middle effects mean the assistant won't reliably use what's loaded anyway. Retrieve only what's relevant per query.
If the user has to find and paste the relevant past memo, you've built a search interface, not an assistant. The whole point is that the user shouldn't need to know which past engagement is relevant before asking the question.