Assignment
Use 30 saved question-answer pairs across grounded, unsupported, stale-policy, and access-boundary categories. Keep ten hidden during prompt development.
Build sequence
- Write a rubric with separate groundedness, completeness, and policy checks.
- Have two reviewers label a small overlapping subset and resolve disagreements.
- Compare two prompt versions with paired case results.
- Define hard release gates and a manual-review queue for uncertain scores.
Acceptance checks
- A permission regression blocks release regardless of average quality.
- Judge agreement is reported against human labels, not assumed.
- The scorecard includes regressions, sample sizes, and a reproducible version manifest.
Evidence to keep
Submit the working artifact or decision document described above, the inputs used, and the observed results for every acceptance check. Include one failed attempt and the change you made after investigating it. Label fixture-based outcomes separately from live-system measurements. These artifacts become part of your final portfolio review.