Session 7 — Homework

Build Your Own Knowledge Agent

From loading documents to handling failures — your complete guide to the RAG homework. Follow each task in order and you’ll have a working knowledge agent by the end.

4 tasks • Estimated time: 3–4 hours • Due before Session 8

Task 1 — Build

Create a RAG Agent for Your Domain

Choose a domain you care about, gather documents, and wire up a complete knowledge agent that retrieves information before answering questions.

Your build roadmap

Choose Your
Domain

→

Gather 5+
Documents

→

Process
& Chunk

→

Build Search
Tool

→

Wire Agent
Loop

→

Test 10
Questions

Pick a domain

Choose something you know well so you can evaluate accuracy. Here are some ideas:

🍽️

Restaurant

Menu items, allergens, wine pairings, reservation policies

📚

Student Handbook

Academic policies, registration deadlines, grading systems

🛍️

Product Catalog

Product specs, pricing, compatibility, warranty info

⚖️

Legal

Contract templates, compliance checklists, regulation summaries

🏥

Medical

Patient guidelines, medication info, procedure protocols

Complete starter code

This is the restaurant worked example. Replace the documents with your domain content.

starter-code.py

import anthropic
import json
client = anthropic.Anthropic()
# ====== STEP 1: Your Documents ======
# Replace these with YOUR domain documents
documents = [
    {
        "title": "Menu - Main Courses",
        "content": """
        Grilled Salmon - $24.99
        Pan-seared with lemon butter sauce. Contains: fish, dairy.
        Pairs well with: Sauvignon Blanc, Pinot Grigio.
        Ribeye Steak - $34.99
        12oz USDA Prime, chargrilled. Contains: none (GF option available).
        Pairs well with: Cabernet Sauvignon, Malbec.
        Mushroom Risotto - $18.99
        Arborio rice with wild mushrooms and parmesan. Contains: dairy, gluten.
        Pairs well with: Chardonnay, Pinot Noir.
        """,
        "source": "menu_main_courses.md",
        "date": "2024-11-01"
    },
    {
        "title": "Reservation & Dining Policies",
        "content": """
        Reservations: Required for parties of 6+. Walk-ins welcome for smaller groups.
        Cancellation: 24-hour notice required or $25 per person fee.
        Dress code: Smart casual. No athletic wear.
        Private dining: Available for 12-40 guests. $500 minimum spend.
        Happy hour: Mon-Fri 4-6 PM, 50% off select appetizers and house wine.
        """,
        "source": "policies.md",
        "date": "2024-10-15"
    }
    # ADD 3+ MORE DOCUMENTS FOR YOUR DOMAIN
]
# ====== STEP 2: Search Tool ======
def search_documents(query):
    """Search documents for relevant information."""
    results = []
    query_words = query.lower().split()
    for doc in documents:
        content_lower = doc["content"].lower()
        # Count matching words
        score = sum(1 for word in query_words if word in content_lower)
        if score > 0:
            results.append({
                "title": doc["title"],
                "content": doc["content"][:500],
                "source": doc["source"],
                "relevance": score
            })
    results.sort(key=lambda x: x["relevance"], reverse=True)
    return results[:3]  # Top 3 results
# ====== STEP 3: Agent Loop ======
tools = [{
    "name": "search_documents",
    "description": "Search the knowledge base for relevant information. "
                  "Use this for ANY question about our domain.",
    "input_schema": {
        "type": "object",
        "properties": {
            "query": {
                "type": "string",
                "description": "The search query"
            }
        },
        "required": ["query"]
    }
}]
def run_agent(question):
    print(f"\n{'='*50}")
    print(f"Question: {question}")
    print(f"{'='*50}")
    messages = [{"role": "user", "content": question}]
    while True:
        response = client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=1024,
            system="You are a helpful assistant. ALWAYS use the "
                   "search_documents tool before answering questions. "
                   "Base your answers ONLY on the search results. If the "
                   "search results don't contain relevant information, say "
                   "'I don't have information about that in our "
                   "knowledge base.'",
            tools=tools,
            messages=messages
        )
        # Check if done
        if response.stop_reason == "end_turn":
            for block in response.content:
                if hasattr(block, 'text'):
                    print(f"\nAnswer: {block.text}")
            return response
        # Handle tool calls
        if response.stop_reason == "tool_use":
            tool_results = []
            for block in response.content:
                if block.type == "tool_use":
                    print(f"\nSearching: {block.input['query']}")
                    result = search_documents(block.input["query"])
                    print(f"   Found {len(result)} results")
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": json.dumps(result)
                    })
            messages.append({"role": "assistant", "content": response.content})
            messages.append({"role": "user", "content": tool_results})
# ====== STEP 4: Test It! ======
test_questions = [
    "What fish dishes do you have?",
    "Do you have any gluten-free options?",
    "What's your cancellation policy?",
    "What wine goes with steak?",
    "Do you have outdoor seating?",  # Not in docs!
]
for q in test_questions:
    run_agent(q)

→ 4 color-coded sections. Documents (cyan) — replace with your domain. Search tool (yellow) — finds relevant passages. Tool definition (magenta) — tells Claude when to search. Agent loop (green) — the retrieval-generation cycle.

import anthropic
import json

client = anthropic.Anthropic()

# ====== STEP 1: Your Documents ======
# Replace these with YOUR domain documents
documents = [
    {
        "title": "Menu - Main Courses",
        "content": """
        Grilled Salmon - $24.99
        Pan-seared with lemon butter sauce. Contains: fish, dairy.
        Pairs well with: Sauvignon Blanc, Pinot Grigio.

Ribeye Steak - $34.99
        12oz USDA Prime, chargrilled. Contains: none (GF option available).
        Pairs well with: Cabernet Sauvignon, Malbec.

Mushroom Risotto - $18.99
        Arborio rice with wild mushrooms and parmesan. Contains: dairy, gluten.
        Pairs well with: Chardonnay, Pinot Noir.
        """,
        "source": "menu_main_courses.md",
        "date": "2024-11-01"
    },
    {
        "title": "Reservation & Dining Policies",
        "content": """
        Reservations: Required for parties of 6+. Walk-ins welcome for smaller groups.
        Cancellation: 24-hour notice required or $25 per person fee.
        Dress code: Smart casual. No athletic wear.
        Private dining: Available for 12-40 guests. $500 minimum spend.
        Happy hour: Mon-Fri 4-6 PM, 50% off select appetizers and house wine.
        """,
        "source": "policies.md",
        "date": "2024-10-15"
    }
    # ADD 3+ MORE DOCUMENTS FOR YOUR DOMAIN
]

# ====== STEP 2: Search Tool ======
def search_documents(query):
    """Search documents for relevant information."""
    results = []
    query_words = query.lower().split()

for doc in documents:
        content_lower = doc["content"].lower()
        # Count matching words
        score = sum(1 for word in query_words if word in content_lower)
        if score > 0:
            results.append({
                "title": doc["title"],
                "content": doc["content"][:500],
                "source": doc["source"],
                "relevance": score
            })

results.sort(key=lambda x: x["relevance"], reverse=True)
    return results[:3]  # Top 3 results

# ====== STEP 3: Agent Loop ======
tools = [{
    "name": "search_documents",
    "description": "Search the knowledge base for relevant information. Use this for ANY question about our domain.",
    "input_schema": {
        "type": "object",
        "properties": {
            "query": {
                "type": "string",
                "description": "The search query"
            }
        },
        "required": ["query"]
    }
}]

def run_agent(question):
    print(f"\n{'='*50}")
    print(f"Question: {question}")
    print(f"{'='*50}")

messages = [{"role": "user", "content": question}]

while True:
        response = client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=1024,
            system="You are a helpful assistant. ALWAYS use the search_documents tool before answering questions. Base your answers ONLY on the search results. If the search results don't contain relevant information, say 'I don't have information about that in our knowledge base.'",
            tools=tools,
            messages=messages
        )

# Check if done
        if response.stop_reason == "end_turn":
            for block in response.content:
                if hasattr(block, 'text'):
                    print(f"\nAnswer: {block.text}")
            return response

# Handle tool calls
        if response.stop_reason == "tool_use":
            tool_results = []
            for block in response.content:
                if block.type == "tool_use":
                    print(f"\nSearching: {block.input['query']}")
                    result = search_documents(block.input["query"])
                    print(f"   Found {len(result)} results")
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": json.dumps(result)
                    })

messages.append({"role": "assistant", "content": response.content})
            messages.append({"role": "user", "content": tool_results})

# ====== STEP 4: Test It! ======
test_questions = [
    "What fish dishes do you have?",
    "Do you have any gluten-free options?",
    "What's your cancellation policy?",
    "What wine goes with steak?",
    "Do you have outdoor seating?",  # Not in docs!
]

for q in test_questions:
    run_agent(q)

Deliverable checklist

5+ documents loaded for your chosen domain
search_documents tool finds relevant passages
Agent uses tool before answering (check the logs — you should see “Searching: ...” output)
Tested with 10 questions (mix of answerable and unanswerable)
Agent says “I don’t know” for questions outside its knowledge

💡 Pro tip: Quality over quantity

Your documents don’t need to be long. A paragraph per document is fine. What matters is that they cover different aspects of your domain — the agent should be able to tell them apart when searching.

Task 2 — Break

Stress-Test with Bad Data

Deliberately feed your agent bad data and document what breaks. This is how you learn where RAG systems fail — and every production system has these failure modes.

Outdated Data

Add a document with old pricing or policies. Does the agent quote the outdated info?

Add a 2022 menu with different prices alongside your 2024 menu

Observe: Which document does the agent cite? Does it notice the date difference?

Contradicting Documents

Add two documents that say different things about the same topic.

One doc says “No refunds after 30 days”, another says “Full refund within 60 days”

Observe: Does the agent blend the answers? Flag the conflict? Pick one?

Knowledge Gaps

Ask questions your documents DON’T cover.

Ask about competitor products when you only have your own catalog

Observe: Does the agent hallucinate? Admit it doesn’t know? Make something up?

Document your findings

Use this format (a Markdown table in your notebook or a simple list):

Test	What You Tried	What Happened	Why It Happened	How to Fix
Outdated data	Added 2022 menu	Agent quoted old prices	No date prioritization	Add date metadata, sort by recency
Contradiction	Conflicting refund policies	Agent blended both	No conflict detection	Add contradiction check before answering
Knowledge gap	Asked about competitors	Agent hallucinated pricing	No guardrails on search results	Add confidence threshold, refuse if no relevant docs

⚠ The point is to find failures

If your agent handles everything perfectly, you’re not trying hard enough. Real-world RAG systems fail in exactly these ways — that’s why companies spend millions on guardrails.

Task 3 — Evaluate

Score Accuracy on 10 Questions

Measure how well your agent actually performs. This is the same evaluation framework real companies use — just on a smaller scale.

Scoring rubric

Criteria	Score	Description
Retrieved right doc?	Yes / No	Did the search tool return the correct document?
Answer correct?	Yes / Partial / No	Is the answer factually accurate based on the documents?
Hallucinated?	No / Yes	Did the agent add information not in the documents?
Cited sources?	Yes / No	Did the agent mention which document it used?

Example evaluation

Question	Right Doc?	Correct?	Hallucinated?	Cited?	Notes
“What’s the salmon price?”	Yes	Yes	No	Yes	Found in menu_main_courses.md
“Do you have vegan options?”	No	No	Yes	No	Agent invented vegan dishes not in our menu
“What are your hours?”	No	No	Yes	No	Not in docs — agent should have said “I don’t know”

Your Agent’s Score Card

After testing 10 questions, tally your results:

X/10 correct • Y/10 cited sources • Z/10 no hallucination

Grading guidance:

7+ correct = Good

5–6 = Needs work

<5 = Major issues to fix

✅ A low score is not a bad grade

Your grade is based on the quality of your evaluation, not on how well your agent performs. A score of 4/10 with a thoughtful analysis of why it failed is worth more than 10/10 with no explanation.

Task 4 — Design

1-Page Proposal for a Real Company

Write a 1-page proposal for a knowledge agent at a real company. This is the kind of deliverable consultants charge $50K+ to produce.

Proposal Template

Company & Use Case

What company? What problem does the knowledge agent solve?

Documents Needed

What documents would you load? List 5–10 specific document types.

Target Users

Who would use this agent? How often? What questions would they ask?

Risk Assessment

What happens if the agent is wrong? What’s the worst case?

Build vs Buy

Would you build custom RAG or use a platform (Glean, Notion AI)? Why?

Success Metrics

How would you measure if the agent is working? What’s the ROI?

Example proposal

🏨 Marriott Hotels — Front Desk Knowledge Agent

Company:

Marriott Hotels

Use Case:

Customer service agent for front desk staff answering guest questions

Documents:

Room types & rates, hotel amenities, cancellation policies, local attractions guide, loyalty program rules, accessibility info

Users:

500+ front desk staff across 50 properties, answering guest questions 24/7

Risk:

Wrong room rate quoted → revenue loss. Wrong cancellation policy → legal exposure. Wrong accessibility info → ADA compliance issue.

Build vs Buy:

Buy (Glean or similar) — 50 properties means thousands of documents across multiple systems. Too complex for simple RAG.

Success:

Reduce average call handling time by 30%, improve first-call resolution from 72% to 90%, eliminate policy misquotes.

💡 Pick a company you know

The best proposals come from personal experience. If you worked at Starbucks, write about Starbucks. If you follow Tesla, write about Tesla. Domain knowledge makes your risk assessment and success metrics more realistic.

Troubleshooting

FAQ & Common Issues

Click any question to expand the answer.

📄 How many documents do I need? ▶

Minimum 5, but more is better. Aim for documents that cover different aspects of your domain. Quality matters more than quantity — a few well-structured documents beat dozens of messy ones.

📎 Can I use PDFs? ▶

For this homework, convert PDFs to text and paste into your documents list. In production, you’d use PDF parsing libraries like PyPDF2 or pdfplumber. The point of this assignment is the agent architecture, not document parsing.

🚨 What if my agent never says “I don’t know”? ▶

Update your system prompt to be more explicit:

"If the search results don't contain relevant information, you MUST say: I don't have information about that in our knowledge base."

You can also add: "NEVER make up information that is not in the search results."

🔍 How do I improve search accuracy? ▶

Three things to try: (1) Better chunking — split docs by topic so each chunk has a clear subject. (2) Better queries — extract keywords from the question before searching. (3) Better scoring — weight title matches higher than content matches, or check for phrase matches instead of individual words.

🤔 What’s a good domain to choose? ▶

Pick something you know well, so you can evaluate accuracy. Restaurant menus, course catalogs, and product specs work great because the information is structured and you can easily check if the agent’s answers are correct. Avoid domains where you can’t verify the output.

What You Just Built

You built a complete knowledge agent — from document loading to failure handling to business proposal. This is exactly what companies pay $100K+ for consultants to build. The architecture is identical; only the scale is different.

RAG is the #1 enterprise AI pattern. Every company deploying AI today — from Slack to Salesforce to internal tools at Goldman Sachs — uses retrieval-augmented generation. You now understand how it works, why it breaks, and how to evaluate it.

The proposal matters. Being able to articulate what to build, who it’s for, and what could go wrong is the difference between an engineer and a leader. That’s the skill companies pay for.

docs loaded

questions tested

failures found

proposal written

Back to Session 7 Open in Colab