Suffolk University • Sawyer Business School

Knowledge Agents — RAG

Your agent is smart but doesn't know YOUR business — add company knowledge with Retrieval-Augmented Generation

Session 7 of 13
150 Minutes
40 Students
Hands-On Learning
01 // LEARNING OBJECTIVES

What You'll Learn Today

By the end of this session, you'll have both conceptual understanding and hands-on experience with the foundations of AI.

01

Understand the Knowledge Gap

See why general-purpose LLMs fail on company-specific questions and how Retrieval-Augmented Generation (RAG) solves this by grounding agents in your own data.

02

Build a Company Knowledge Agent

Load real documents, build a search tool that finds relevant passages, and wire it into your agent loop so Claude answers from YOUR data.

03

Handle RAG Failure Modes

Learn what happens when documents contradict each other, data is outdated, or the retriever returns irrelevant results. Build agents that say 'I don't know' when they should.

04

Evaluate Retrieval Quality

Measure whether your agent finds the right documents and gives accurate answers. Introduction to retrieval metrics that matter for business deployment.

02 // SESSION AGENDA

Today's Journey

A carefully crafted progression from concepts to hands-on building.

5:00 PM - 5:15 PM

Part 1: The Knowledge Problem

15 minutes

Claude knows the internet. It doesn't know your company's SOPs, product catalog, or HR policies. That's a problem — and RAG is the solution.

  • Demo: Ask Claude about your company's refund policy. It hallucinates confidently. This is the problem RAG solves.
  • RAG = Retrieval-Augmented Generation: search your documents first, then generate an answer grounded in real data
  • The pattern: User question → Search documents → Insert relevant passages into prompt → Claude answers from YOUR data
  • Why this matters: Every enterprise AI deployment needs RAG. This is the #1 pattern in production AI.
5:15 PM - 5:40 PM

Part 2: RAG Architecture Deep Dive

25 minutes

How RAG works under the hood — from document loading to retrieval to generation.

  • Three approaches: Simple keyword search, Claude's long context window (200K tokens), and vector embeddings
  • Our approach today: Long context + smart chunking. No vector DB needed — Claude can read 200K tokens at once.
  • Document processing: Loading, chunking, and creating a searchable index. Why chunk size matters.
  • The retrieval tool: A search function your agent calls to find relevant document passages before answering.
5:40 PM - 6:25 PM

Part 3: Hands-On — Build a Knowledge Agent

45 minutes

Build an agent that answers questions about a company using its own documents.

  • Phase 1 (15 min): Load sample company documents (policies, FAQs, product info). Build a search_documents tool.
  • Phase 2 (15 min): Wire the search tool into your agent loop. Ask company-specific questions — watch it retrieve and answer.
  • Phase 3 (15 min): Test edge cases — contradicting documents, outdated info, questions with no answer. Does your agent handle them gracefully?
6:25 PM - 6:35 PM

Break

10 minutes

Your agent now knows YOUR business. Next: what happens when the data is wrong?

6:35 PM - 7:05 PM

Part 5: RAG Failure Modes & Trust

30 minutes

Your agent is only as trustworthy as its data. What happens when documents are wrong, outdated, or contradictory?

  • Ethics moment: The Air Canada chatbot case — held legally liable for answers based on outdated policy documents
  • Failure mode 1: Outdated data. Your agent confidently quotes last year's pricing.
  • Failure mode 2: Contradicting documents. Two SOPs say different things. What does your agent do?
  • Failure mode 3: No relevant data. The agent should say 'I don't have information on that' — not hallucinate.
  • Building trust: Citation, confidence scores, and 'I don't know' as a feature, not a bug.
7:05 PM - 7:20 PM

Part 6: RAG in the Real World

15 minutes

How companies actually deploy RAG — from customer support to internal knowledge bases to compliance.

  • Case studies: Customer support bots, internal wikis, legal document search, medical information retrieval
  • The build-vs-buy decision: When to build your own RAG vs. use platforms like Glean, Notion AI, or Perplexity Enterprise
  • Scaling: From 10 documents to 10 million. When long context isn't enough and you need vector search.
7:20 PM - 7:30 PM

Part 7: Wrap-Up + Homework

10 minutes

Your agents now have memory (Session 5), reasoning (Session 6), and knowledge (today). Next: make them work in teams.

  • The stack so far: Tools (S4) → Real APIs + Memory (S5) → ReAct Patterns (S6) → Knowledge/RAG (TODAY)
  • Next session: Multi-Agent Systems in Practice — build a team of agents that collaborate
  • Homework preview: Build a RAG agent for a domain you care about, test with edge cases, evaluate retrieval quality
03 // INTERACTIVE RESOURCES

Tools & Learning Materials

Everything you need to explore, experiment, and build.

Start Here

RAG Agent: Live Walkthrough

Watch a RAG agent search company documents, rank results, detect outdated data, and generate a cited answer. Interactive animated trace with real CloudSync data.

Watch the agent
Lecture

Session 7 Lecture Guide

Projectable lecture companion — visual slides for the full Session 7 lecture with RAG architecture diagrams, failure mode analysis, and interactive navigation.

Open lecture guide
Hands-On

Knowledge Agent Workshop Notebook

Build a company knowledge agent with document loading, search, and RAG. Uses Claude's long context window — no vector DB needed.

Open in Colab
Docs ?

Anthropic RAG Documentation

Official guide to building RAG systems with Claude. Covers chunking strategies, retrieval patterns, and best practices.

Read docs
Docs

Claude's Long Context Window

Claude can process 200K tokens at once. Learn how to use long context as a simpler alternative to traditional vector search for smaller document sets.

Read docs
Guide

Building Effective Agents (Anthropic)

Anthropic's guide to agent patterns — including RAG as a core capability for grounding agents in real data.

Read guide
Instructor ?

Speaker Notes — Professor's Command Center

Private dashboard with talking points, demo cues, discussion prompts, and live timers for each lecture section. Open on your laptop while projecting the lecture guide.

Open speaker notes
Workshop ?

Workshop Timer

Full-screen projectable 45-minute countdown with 3 phases, auto-transitions, audio chimes, and troubleshooting tips for the hands-on Colab session.

Start timer
Demo

RAG Failure Modes Demo

Interactive demo of three RAG failure scenarios — outdated documents, contradictions, and unanswerable questions. Toggle between broken and fixed agent behaviors.

Explore failures
Homework ?

Homework Guide — Build Your Knowledge Agent

Step-by-step walkthrough for the Session 7 homework: build a RAG agent, stress-test it, evaluate accuracy, and design a real-world proposal. Includes starter code and rubrics.

Start homework
Homework ?

Homework Notebook — Build Your Own Knowledge Agent

Colab notebook for the Session 7 homework. All RAG infrastructure is pre-built — pick your domain, load your documents, test your agent, break it, and evaluate it.

Open in Colab
04 // HANDS-ON ACTIVITIES

Learning by Doing

Three interactive challenges to build your intuition.

Activity 1

Build a Company Knowledge Agent

45 minutes in Google Colab. Build an agent grounded in company documents:

1

Load Documents (10 min)

Load sample company documents — policies, FAQs, product catalogs. Process them into searchable chunks.

2

Build the Search Tool (10 min)

Create a search_documents tool that finds relevant passages for any query. Wire it into your agent loop.

3

Test with Real Questions (15 min)

Ask company-specific questions. Compare: does the agent answer from documents or hallucinate? Test edge cases.

4

Handle Failures Gracefully (10 min)

Test with contradicting docs, outdated info, and unanswerable questions. Make your agent say 'I don't know' when appropriate.

Activity 2

RAG Trust Assessment

Evaluate your knowledge agent's trustworthiness:

1

Create 10 Test Questions

Write 10 questions: 5 answerable from the documents, 3 partially answerable, 2 unanswerable. Run all through your agent.

2

Score Accuracy

For each answer: Did it use the right document? Is the answer correct? Did it hallucinate? Did it cite sources?

3

Identify Failure Patterns

Where does your agent fail? Wrong retrieval? Right retrieval but wrong answer? No retrieval when it should have searched?

4

Propose Improvements

List 3 specific improvements: better chunking, better search queries, better system prompt instructions, or additional guardrails.

05 // KEY TAKEAWAYS

What to Remember

RAG grounds agents in reality. Without RAG, your agent hallucinates about your company. With RAG, it answers from real documents. This is the #1 pattern in enterprise AI.

Your agent is only as trustworthy as its data. Outdated documents, contradictions, and gaps are the real-world challenges. Building trust means handling these gracefully.

Long context is the accessible RAG. Claude's 200K token window means business students can build RAG without vector databases. Start simple, scale later.

'I don't know' is a feature. The best knowledge agents know the limits of their data. An agent that refuses to answer is more trustworthy than one that hallucinates.

06 // HOMEWORK

Before Next Session

Complete these tasks before our next class to prepare for prompt engineering.

Build: Create a RAG agent for a domain you care about — load 5+ documents (articles, policies, product pages) and build a search tool. Test with 10 questions.
Break: Deliberately feed your agent contradicting documents and outdated data. Document 3 failure modes and how you'd fix each one.
Evaluate: Score your agent's accuracy: For 10 test questions, record whether it retrieved the right document, gave a correct answer, and cited sources properly.
Design: Write a 1-page proposal for a knowledge agent at a real company. What documents would it need? Who would use it? What's the risk if it's wrong?
Coming Next

Session 8: Multi-Agent Systems in Practice

One agent is useful. A team of agents is powerful. Build a researcher→analyst pipeline and learn coordination patterns.

Multi-AgentPipelinesCoordinationSpecialization