Test decisions and failure paths independently of model variability.
1. Deterministic tool tests
Unit-test validation, authorization and result formatting with fake dependencies. These tests should pass without calling a live model.
2. Behavior evaluation
Use labeled tasks and inspect selected tools, arguments and final outcomes. Exact text matching is often too brittle; define task-specific acceptance properties.
3. Operational tracing
Record run and tool identifiers, timings and failure categories with redacted payloads. Make a failed task reconstructable without storing unnecessary private content.
Worked scenario
An answer sounds correct but used the wrong customer record. Outcome-only prose scoring would miss the authorization failure.
Apply it
Create a held-out triage set including missing data, repeated calls and malicious user instructions.
Check your understanding
Your report separates model decision errors from tool implementation errors. Explain the decision and show evidence from your implementation or design. If you cannot demonstrate it yet, revisit the relevant section before continuing.