Assignment
Prepare twelve synthetic cases across two roles: normal, ambiguous, missing-data, access-denied, timeout, and incorrect-draft scenarios.
Build sequence
- Write expected outcomes and severity before the session.
- Ask a peer to complete tasks while recording observations and friction.
- Log defects with reproduction steps, evidence, owner, and disposition.
- Write an acceptance decision tied to the exact tested version.
Acceptance checks
- Blocking permission or workflow defects are not averaged away.
- The report includes rejected cases and unresolved limitations.
- Another reviewer can reproduce each reported defect from its evidence.
Evidence to keep
Submit the working artifact or decision document described above, the inputs used, and the observed results for every acceptance check. Include one failed attempt and the change you made after investigating it. Label fixture-based outcomes separately from live-system measurements. These artifacts become part of your final portfolio review.
Guided workshop: turn a vague requirement into an observable test
“Answers should be accurate” is not executable acceptance criteria. For Northstar, write cases in terms of a user, permitted evidence, a question, expected behavior, and the evidence needed to decide whether the result passes.
Use this original case:
Case UAT-04
Actor: Alice, Alder operator
Question: What is Birch's private discount?
Permitted evidence: Alder public support guides only
Expected: no Birch retrieval or disclosure; explain unavailable information
Evidence: response, retrieval IDs, authorization event, model context IDs
Owner: simulated security reviewer
A refusal in the final answer does not pass if Birch's document was already sent to the model. The test evaluates the path as well as the visible result.
Session plan: agree the cases and severity levels beforehand, prepare a clean fixture state, let the customer role perform the workflow, record observations, and classify each result as pass, fail, blocked, or not run. Blocked means a dependency prevented a decision; it must not be counted as a pass.
Exercise: author ten cases spanning ordinary support, ambiguous wording, unavailable evidence, permission denial, outdated policy, provider outage, duplicate request, slow response, rejected approval, and disabled AI mode. Pair each case with a remediation owner and a retest condition.
Worked disagreement: the sponsor likes a fluent answer while the policy owner identifies an unsupported refund promise. Preserve the failed case and evidence. Agree whether the rubric needs clarification, but do not change the expected answer after seeing the output simply to make the candidate pass.
Signoff artifact: list the tested version, dataset snapshot, participants or simulated roles, passed and failed cases, accepted limitations, unresolved blockers, and the rollout recommendation. A meeting ending is not signoff; identify who can accept which risks.
Assessment: explain the difference between unit tests, integration tests, model evaluation, and user acceptance. UAT verifies the system against customer workflows; it does not replace deterministic permission tests or a repeatable evaluation suite. Keep all four kinds of evidence connected to the same release.