ISOM 260 · Suffolk University

Knowledge Agents —
RAG

Your agent is smart but doesn't know YOUR business — add company knowledge with Retrieval-Augmented Generation

RAG Knowledge Agents Document Search Long Context
Use arrow keys to navigate
Part 1 · 5:00–5:15 PM

The Knowledge Problem

Claude knows the internet. It doesn't know your company's SOPs, product catalog, or HR policies. That's a problem — and RAG is the solution.

Demo: "What's CloudSync's refund policy?"

Without RAG
> What's CloudSync's refund policy?
"CloudSync offers a 60-day money-back guarantee on all plans. Simply contact billing@cloudsync.com for a full refund, no questions asked."
❌ Hallucinated — No Document Access
With RAG
> What's CloudSync's refund policy?
"CloudSync's annual plans have a 30-day full refund window. After 30 days, you receive a pro-rated refund. Contact support@cloudsync.com." refund-policy-2024.md billing-faq.md
✅ Grounded — Cited from Real Documents

The RAG Pattern

User Question
"What's our refund policy?"
→
Search Documents
find relevant passages
→
Insert Passages
add to prompt context
→
Claude Answers
from YOUR data
Every enterprise AI deployment needs RAG. This is the #1 pattern in production AI. If your agent answers questions about company data, you need this.
Part 2 · 5:15–5:40 PM

RAG Architecture Deep Dive

How RAG works under the hood — from document loading to retrieval to generation. Three approaches, one winner for today.

Keyword Search
Simplest approach. Fast and familiar. Limited understanding — misses synonyms and context.
grep "refund" docs/*.md
Long Context Window
Claude reads 200K tokens at once. No database needed. Works great for smaller document sets.
Our approach today
Vector Embeddings
Semantic search at scale. Needs infrastructure (Pinecone, Weaviate). The enterprise standard.
Production at scale
Our approach today: Long context + smart search. No vector DB needed — Claude can read 200K tokens at once. Start simple, scale later.

Document Processing Pipeline

Raw Documents
PDFs, docs, FAQs
→
Chunking
split into passages
→
Searchable Index
tagged & organized
→
Agent Tool
search_documents()

How Document Chunking Works

refund-policy-2024.md (Full Document)
§1  General Policy
§2  Monthly Plans
§3  Annual Plans
§4  Enterprise Plans
§5  Exceptions
↓ Chunking (with overlap)
Chunk 1
§1 + §2
§2 overlap
~500 tokens
Chunk 2
§2 + §3
§2 §3 overlap
~500 tokens
Chunk 3
§3 + §4
§3 §4 overlap
~500 tokens
Chunk 4
§4 + §5
§4 §5 overlap
~500 tokens
Chunks overlap so context isn't lost at boundaries. If a section gets split between two chunks, the overlap ensures the full meaning is captured in at least one chunk.
Document Chunk Example
{ "id": "policy-returns-001", "source": "policies/returns.md", "content": "Customers may return items within 14 days of purchase for store credit. Original receipt required. Sale items are final sale.", "metadata": { "category": "policy", "last_updated": "2024-11-15" } }

🎬 See It In Action

Watch a RAG agent search, rank, verify, and generate — step by step in our interactive trace demo.

→ Open RAG Agent: Live Trace
Three Levels

Three Levels of Knowledge

Same question, three approaches. Watch how adding documents — and how you add them — changes everything.

L0 No Documents
Claude hallucinates company info. Sounds confident, totally wrong.
# No company data at all "Your refund policy is 30 days with full money back..." # Hallucinated. Dangerous.
❌ Confident hallucination.
L1 Stuff All Docs
Paste ALL documents into the prompt. Works, but prompt is enormous. Doesn't scale.
# Dump everything in the prompt system = "Here are ALL 200 company docs: [50,000 words]..." # Correct but expensive. # Won't scale to 1000+ docs.
⚠️ Works but doesn't scale.
L2 RAG Agent
Searches relevant docs, reads only what matters. Scales to millions of documents.
THINK: Need refund policy search_documents("refund") FOUND: policies/returns.md ANSWER: From returns.md: "14 days, store credit only." SOURCE: policies/returns.md
✓ Smart retrieval. Cited. Scalable.
Same question. Same model. The data pipeline changed everything.
Hands-On · 5:40–6:25 PM

Build a Knowledge Agent

45 minutes in Google Colab. Load company documents, build a search tool, wire it into your agent loop.

Open Colab Now
Build an agent that answers questions from YOUR company documents. No vector DB needed.
Open in Google Colab

Workshop Phases

Part 5 · 6:35–7:05 PM

RAG Failure Modes & Trust

Your agent is only as trustworthy as its data. What happens when documents are wrong, outdated, or contradictory?

The Air Canada Case — A Real-World Warning

2022
Customer Asks Chatbot About Bereavement Fare
Customer asks Air Canada's chatbot about bereavement fares. The chatbot responds: "You can book now and request a retroactive bereavement discount within 90 days."
2023
Air Canada Denies the Discount
Customer requests the retroactive discount. Air Canada responds: "No, that's not our policy. The chatbot was wrong." Air Canada argues the chatbot is a separate entity and they aren't liable for its answers.
2024
Civil Resolution Tribunal Rules
The tribunal finds: "Air Canada is responsible for information provided by its chatbot. The customer relied on it in good faith." The company cannot disclaim its own agent's answers.
Outcome
Air Canada Pays Damages
💰 Air Canada pays ~$800 in damages + tribunal fees
Your agent's answers have real consequences. If your chatbot says it, your company owns it. Outdated documents create legal liability.

Common Failure Modes

Outdated Data
Agent quotes last year's pricing confidently. The document hasn't been updated but the policy changed 3 months ago. Users trust the answer.
Agent Says
"CloudSync's refund policy allows 60-day full refunds." (from 2022 doc)
Should Say
"CloudSync's current policy (2024) allows 30-day refunds. Note: an older policy referenced 60 days but has been superseded."
Contradicting Documents
Two policies say different things about the same topic. Which one does the agent choose? Does it flag the contradiction or silently pick one?
Agent Says
"You can return the product within 14 days." (picked one randomly)
Should Say
"I found conflicting information: Document A says 14 days, Document B says 30 days. Please check with support for the current policy."
No Relevant Data
Agent should say "I don't have information on that" — not hallucinate a plausible answer. The hardest failure mode to catch.
Agent Says
"CloudSync's enterprise tier costs $999/month with unlimited users." (hallucinated)
Should Say
"I don't have information about enterprise pricing in my available documents. Please contact sales@cloudsync.com for enterprise pricing."
Wrong Retrieval
Search returns irrelevant docs (e.g., finds "refund" in an unrelated context). Agent answers confidently from the wrong source.
Building Trust
Citations, confidence scores, and "I don't know" as a feature, not a bug. The best knowledge agents know the limits of their data.
Part 6 · 7:05–7:20 PM

RAG in the Real World

How companies actually deploy RAG — from customer support to legal compliance to medical information retrieval.

Customer Support
Auto-answer FAQs from knowledge base. Deflect tickets, reduce wait times.
Internal Wiki
Search company docs with natural language. No more digging through Confluence.
Legal / Compliance
Find relevant regulations across thousands of documents in seconds.
Medical Info
Look up drug interactions, treatment protocols from verified medical databases.

Build vs Buy

Build Your Own RAG
Full control over data pipeline, custom chunking, proprietary data stays in-house. Requires engineering investment and maintenance.
Use Platforms
Glean, Notion AI, Perplexity Enterprise — faster deployment, less control. Great for common use cases. Less customization.
Scaling Insight
From 10 documents to 10 million — when long context isn't enough and you need vector search. Start with long context, graduate to embeddings when your data demands it.
Trust Equation

The Trust Equation for Knowledge Agents

What makes a knowledge agent trustworthy? A checklist for production-grade RAG systems.

Accurate Retrieval + Honest Generation + Source Citation = Trustworthy RAG
Part 7 · 7:20–7:30 PM

The Stack So Far

Where we've been, where we're going. Your agents now have tools, memory, reasoning, and knowledge.

S1-2
AI
Fundamentals
S3-4
Prompts &
Tools
S5
Real APIs
+ Memory
S6
ReAct
Patterns
S7
Knowledge
RAG
S8
Multi-Agent
Collab
S9
Industry
Agents
S10
Guard-
rails
S11
Eval
S12+
Final
Project

What Each Session Added

S8 Multi-Agent Collaboration NEXT
S7 Knowledge / RAG ← YOU ARE HERE
S6 ReAct Reasoning
S5 Real APIs + Memory
S4 Tools (Function Calling)
The pattern: Tools (S4) → Real APIs + Memory (S5) → ReAct Patterns (S6) → Knowledge/RAG (TODAY). Each session adds a capability. Next: agents that work in teams.
Takeaways

Key Takeaways

Four things to remember from today's session on Knowledge Agents and RAG.

RAG grounds agents in reality. Without RAG, your agent hallucinates about your company. With RAG, it answers from real documents. This is the #1 pattern in enterprise AI.
Your agent is only as trustworthy as its data. Outdated documents, contradictions, and gaps are the real-world challenges. Building trust means handling these gracefully.
Long context is the accessible RAG. Claude's 200K token window means business students can build RAG without vector databases. Start simple, scale later.
"I don't know" is a feature. The best knowledge agents know the limits of their data. An agent that refuses to answer is more trustworthy than one that hallucinates.
"Your agent is smart. But it doesn't know YOUR business.
Now it does."
Homework

Homework & Resources

Build a RAG agent for your domain, break it, evaluate it, and design a real-world deployment.

Build
Create a RAG agent for a domain you care about — load 5+ documents (articles, policies, product pages) and build a search tool. Test with 10 questions.
Break
Deliberately feed your agent contradicting documents and outdated data. Document 3 failure modes and how you'd fix each one.
Evaluate
Score your agent's accuracy: For 10 test questions, record whether it retrieved the right document, gave a correct answer, and cited sources properly.
Design
Write a 1-page proposal for a knowledge agent at a real company. What documents would it need? Who would use it? What's the risk if it's wrong?

Resources

Knowledge Agent Notebook
Build a company knowledge agent with document loading, search, and RAG in Colab.
Anthropic RAG Docs
Official guide to building RAG systems with Claude. Chunking, retrieval, best practices.
Long Context Window
Claude processes 200K tokens at once. The simpler alternative to vector search for smaller datasets.
Building Effective Agents
Anthropic's guide to agent patterns — including RAG as a core capability.
Next Session
Session 8: Multi-Agent Systems in Practice
One agent is useful. A team of agents is powerful. Build a researcher → analyst pipeline and learn coordination patterns.