Does it behave as expected?
Compare behavior with requirements.
2 × 10 + 3 × 2 = 26
Evals 10101 / 14
How do we know our AI is working?
Evals are a form of testing.
AI makes repeatability and judging correctness harder.
40 minutesNo coding requiredTwo simple examples
Today’s session · 40 minutes02 / 14
Table of contents
Start here03 / 14
Testing and AI evals overlap.
How reliably does it meet the task?
Structured tests of answers or actions across examples.
Evals are testing adapted to measuring AI task performance.
What changes?04 / 14
Two challenges come with AI.
Challenge 01
Behavior can vary.
The same request can produce different outputs.
Repeat cases. Check reliability.
Challenge 02
“Good” needs a definition.
Different answers can be valid.
Define accuracy, coverage, and clarity.
A repeatable answer can still be wrong.
Multiple acceptable answers05 / 14
Different words. Same correct meaning.
Source
The launch has moved from Thursday to Friday.
“Friday is the new launch date.”Valid answer A
“The launch was postponed by one day, to Friday.”Valid answer B
Friday is the new launch date.
Evaluate the meaning. Exact wording does not define correctness.
Choose the check that matches the behavior.06 / 14
When do we use each?
Does arithmetic follow the rounding rule?
Use a software test with the exact expected value.
Does a missing document produce the right error?
Use software tests for loading and error handling.
Is the summary accurate and complete?
Use an AI eval with a factual and quality rubric.
Are the extracted invoice fields correct?
Use an AI eval with exact comparisons to reviewed fields.
Did the agent supply the right tool arguments?
Inspect recorded tool calls against the source invoice.
Did a prompt or model change improve the system?
Run relevant software tests and the same AI eval cases.
An AI application usually needs both. Exact assertions can also be eval checks.
Example 01 · Summaries07 / 14
Summarize this update.
For a teammate who missed the meeting. Maximum 50 words.
The team moved the product launch from 12 May to 19 May after finding a payment bug. Priya will fix the bug by 18 May. The signup flow is ready. The team will review progress on 17 May. The launch remains dependent on the payment fix being completed.
19 MayPayment bugPriya · 18 MayLaunch depends on fix
Compare the outputs08 / 14
Which summary would you send?
The launch moved to 19 May because of a payment bug. Priya will fix it by 18 May, and launch depends on that fix being completed.
The product will launch on 12 May. Payment testing is complete, and the team is ready to release.
The team discussed launch plans and a bug. Some work is ready, and progress will be reviewed.
What specifically makes the others weaker?
A simple rubric09 / 14
Define “good” before scoring.
1 = poor · 2 = partly meets · 3 = meets
Reveal the worked assessment
| Criterion | A | B | C |
|---|---|---|---|
| Consistency | 3 | 1 | 3 |
| Relevance | 3 | 1 | 1 |
| Coherence + fluency | 3 / 3 | 3 / 3 | 3 / 3 |
| Result | Pass | Fail: changed facts | Fail: omitted essentials |
Passing rule: consistency and relevance = 3; coherence and fluency ≥ 2; ≤ 50 words. Prepared judgments; simplified classroom scale.
Example 02 · One agent, one tool10 / 14
A simple invoice agent.
InvoiceText + line items
→
ModelExtracts + calls tool
→
CalculatorComputes item total
→
ModelReturns fields
→
ApplicationValidates mismatch
One agentOne calculator toolRecorded calls + final result
An invoice with a mismatch11 / 14
Is “needs review” a successful result?
- 2 notebooks × 10.00
- 20.00
- 3 pens × 2.00
- 6.00
- Printed total
- 29.00
The items sum to 26.00.
What should a good agent do?
A correct result can be “needs review.”
printed_total: 29.00calculated_total: 26.00status: needs_review
Three evaluation layers12 / 14
Check the result and the actions.
Correct fields?
Printed total stays 29.00.
Correct arguments?
2 × 10.00 and 3 × 2.00.
Supported facts?
Missing invoice number stays null.
A correct total can come from wrong arguments. Inspect the recorded tool calls.
Build with evidence13 / 14
Eval-driven development.
- 1
Define success
Write passing rules.
- 2
Build cases
Normal, tricky, missing information.
- 3
Run baseline
Save outputs and scores.
- 4
Make a change
Change one thing.
- 5
Rerun + inspect
Same cases. Read failures.
Check improvements and regressions. Add useful failures.
Same cases and settings. Repeat when outputs vary. Keep some cases aside from tuning.
Take it into your next change14 / 14
Make “good” something we can check.
- Use exact checks for precise answers; rubrics for multiple valid answers.
- Evaluate agent actions as well as the final output.
- Run the same cases before and after a change. Read failures.
Next session: build a small dataset and compare two runs.