What you will learn
Golden datasets, LLM judges, Human review, Regression suites, Error analysis.
Engineering the capability
A useful rubric describes observable behavior. “Good answer” is difficult to reproduce; “states the correct eligibility condition and cites the current policy” can be reviewed. Include examples of passing, borderline, and failing answers so reviewers interpret the rubric consistently.
Model-based judges can scale review, but their preferences can differ from your users. Calibrate a judge against human-labeled cases, inspect disagreements, and keep deterministic checks for things such as missing citations and forbidden actions. A judge can be fooled by polished language or instructions embedded in the answer being graded.
Treat a release comparison as paired evidence: run old and new versions on the same cases. Separate development data from a protected test set. Report the number of cases, important subgroups, and uncertainty; a tiny improvement on a tiny dataset is not strong evidence of a durable gain.
Worked case
Version A passes 17 of 20 cases and B passes 18. B also breaks the only permission-boundary case. An aggregate score favors B, but the release gate should block it. Some criteria are hard constraints, while others are optimization targets. Write that distinction before looking at the scores.
Put it into practice
Continue with the next lab: create an AI release scorecard. Build the artifact, record the failure cases, and explain the tradeoff before moving on.