If you can't measure it, you can't trust it — build test suites, run evaluations, and prove your agent works
By the end of this session, you'll have both conceptual understanding and hands-on experience with the foundations of AI.
Create a 20-case test suite with expected outputs, edge cases, and adversarial inputs. Move from 'vibes-based' to systematic evaluation.
Use Claude to evaluate Claude: build an automated judge that scores your agent's outputs on accuracy, helpfulness, and safety.
Run the same query 3 times and measure output variance. Understand why non-determinism matters for business deployment.
When a VP asks 'does your agent work?', answer with metrics, not feelings. Build dashboards that show accuracy, coverage, and failure rates.
A carefully crafted progression from concepts to hands-on building.
You've been evaluating your agent by trying a few queries and seeing if the output 'looks right.' That's not evaluation — that's vibes.
How to write test cases for AI agents — it's not like testing normal software.
Build a complete evaluation pipeline for your agent — test suite, automated runner, and LLM-as-judge.
You now have a systematic evaluation pipeline. Next: what the metrics tell you.
Run your evaluation, analyze results, and systematically improve your agent.
Your final project requires a test suite. Here's exactly what's expected.
You can now build agents AND prove they work. Next: put it all together for your final project.
Everything you need to explore, experiment, and build.
Build a 20-case test suite, automated runner, and LLM-as-judge evaluator. Includes consistency testing and failure analysis.
Open in ColabOfficial guide to testing and evaluating Claude-based applications. Covers test design, automated scoring, and quality metrics.
Read docsUsing one LLM to evaluate another. The pattern behind automated agent evaluation at scale.
Read guideThree interactive challenges to build your intuition.
45 minutes. Create a complete testing and evaluation system for your agent:
10 happy path, 5 edge cases, 3 adversarial, 2 domain-specific risks. Each case: input, expected behavior, evaluation criteria.
Automated test runner that executes all cases. LLM-as-judge that scores each output on accuracy (1-5), helpfulness (1-5), and safety (pass/fail).
Run 5 queries 3 times each. Calculate output variance. Flag any query where the agent gives meaningfully different answers.
Analyze your evaluation results and systematically improve:
Group failures by type: wrong tool, bad reasoning, hallucination, safety issue. Which type is most common?
Pick the most impactful failure. Fix it (usually a system prompt or tool improvement). Re-run that test case.
After the fix, re-run all 20 cases. Did overall scores improve? Did the fix break anything else?
Vibes are not evaluation. 'It seems to work' isn't a deployment-ready answer. Systematic test suites, automated scoring, and consistency checks are what production agents need.
LLM-as-judge scales evaluation. You can't manually review every agent output. Using Claude to evaluate Claude gives you automated, consistent scoring at scale.
Answer with data. When someone asks 'does it work?', show them: 85% accuracy on 20 test cases, 92% consistency across 3 runs, 0 safety failures. That's deployable.
Test for bias and fairness. Your agent might perform differently for different inputs. Test across demographics, edge cases, and adversarial scenarios to catch hidden biases.
Complete these tasks before our next class to prepare for prompt engineering.
From notebook to product. From prototype to pitch. Teams form, scope locks, and building begins.