Session 7: RAG Agent Live Trace

Live Agent Trace

Step 0 of 14

Space Play / Pause → Step Forward ← Step Back R Reset

Docs Searched

Chunks Retrieved

Sources Used

Conflicts Found

Confidence

Understand Step 1 of 14

User query received:

User Input

Understand Step 2 of 14

Decomposing query into search terms:

refund policy annual plans CloudSync cancellation terms

I need to search company documents for refund and cancellation policies specific to annual subscriptions.

Retrieve Step 3 of 14

Searching 7 documents...

refund-policy-2024.md

pricing-tiers.md

refund-policy-2022.md

customer-faq.md

terms-of-service.md

employee-handbook.md

onboarding-guide.md

Retrieve Step 4 of 14

Retrieved 6 relevant chunks:

refund-policy-2024.md 0.92

Annual plan subscribers may request a full refund within 30 days of purchase. After the 30-day window, cancellations receive a pro-rated refund for the remaining subscription period.

pricing-tiers.md 0.71

CloudSync Annual Plan: $299/year (billed annually). Includes all Pro features, priority support, and 500GB cloud storage. Enterprise plans available for teams of 10+.

refund-policy-2022.md 0.89

All annual plan customers are entitled to a full 60-day money-back guarantee. No questions asked. Contact support to initiate the refund process within the guarantee period.

customer-faq.md 0.78

To request a refund, email support@cloudsync.com with your account ID and reason for cancellation. Refunds are processed within 5-7 business days to your original payment method.

terms-of-service.md 0.34

CloudSync reserves the right to modify, suspend, or discontinue any part of the Service at any time. We will provide at least 30 days notice for material changes.

employee-handbook.md 0.12

Full-time employees are eligible for health insurance, 401(k) matching, and annual performance bonuses. Part-time employees receive prorated benefits after 6 months.

Rank Step 5 of 14

Ranking by relevance score...

Rank Step 6 of 14

Selection complete

Verify Step 7 of 14

Cross-referencing sources...

Examining the 3 selected chunks for consistency, date conflicts, and factual alignment.

Verify Step 8 of 14

Conflict: The 2024 policy states a 30-day refund window. The 2022 policy states a 60-day refund window. These cannot both be current.

Verify Step 9 of 14

The 2022 document is outdated. The 2024 document supersedes it based on date metadata. I'll use the 2024 policy but note the discrepancy for transparency.

Verify Step 10 of 14

1 conflict resolved

Generate Step 11 of 14

Generating answer from verified sources...

Context window 0 / 200,000 tokens

Generate Step 12 of 14

Generate Step 13 of 14

Generate Step 14 of 14

Confidence 0%

Why RAG Matters

Same Question, Two Approaches

What happens when an LLM answers from memory vs. with RAG-grounded knowledge?

Without RAG

"CloudSync offers a standard ~~60-day~~ money-back guarantee on all plans, including annual subscriptions. Simply contact support for a full refund within ~~60 days~~ of purchase."

No citations. No sources.

Hallucinated — wrong policy, no sources

With RAG

"CloudSync's refund policy for annual plans provides a 30-day full refund window from the date of purchase. After 30 days, cancellations receive a pro-rated refund for the remaining subscription period." refund-policy-2024.md customer-faq.md

"Note: An older 2022 policy referenced a 60-day window, but this has been superseded by the current 2024 policy." refund-policy-2024.md §3.2

Accurate — correct policy, cited sources, conflict flagged

This is what you're about to build. Open the Colab notebook and you'll wire up a RAG pipeline — loading documents, searching with TF-IDF, ranking results, and generating grounded answers. By the end of class, your agent will retrieve real documents and generate cited answers like this one.

RAG Agent: Live Trace

Without RAG

With RAG