Assignment
Use 30 saved question-answer pairs across grounded, unsupported, stale-policy, and access-boundary categories. Keep ten hidden during prompt development.
Build sequence
- Write a rubric with separate groundedness, completeness, and policy checks.
- Have two reviewers label a small overlapping subset and resolve disagreements.
- Compare two prompt versions with paired case results.
- Define hard release gates and a manual-review queue for uncertain scores.
Acceptance checks
- A permission regression blocks release regardless of average quality.
- Judge agreement is reported against human labels, not assumed.
- The scorecard includes regressions, sample sizes, and a reproducible version manifest.
Evidence to keep
Submit the working artifact or decision document described above, the inputs used, and the observed results for every acceptance check. Include one failed attempt and the change you made after investigating it. Label fixture-based outcomes separately from live-system measurements. These artifacts become part of your final portfolio review.
Guided workshop: separate a quality score from a release gate
An average can hide a critical failure. Track answer correctness, unsupported-answer behavior, citation validity, access control, latency, and cost separately. A security boundary failure must not be compensated for by high answer accuracy.
# academy-check
def release_allowed(rows):
if not rows:
return False
if any(not row["authorized"] or not row["citations_valid"] for row in rows):
return False
correct = sum(row["correct"] for row in rows) / len(rows)
return correct >= .9
clean = [{"authorized": True, "citations_valid": True, "correct": True} for _ in range(10)]
assert release_allowed(clean)
unsafe = clean + [{"authorized": False, "citations_valid": True, "correct": True}]
assert not release_allowed(unsafe)
assert not release_allowed([])
The 90% threshold is an exercise policy, not a universal production target. Specify targets before evaluating and justify them using the task's risk and manual fallback. This tiny example treats all correctness cases equally; your actual report should retain category and difficulty slices.
Build the dataset: author supported, unsupported, ambiguous, permission-restricted, and adversarial cases. Each needs an ID, input, allowed evidence, expected behavior, and scoring notes. Keep a development set and a final held-out set. Record who labelled uncertain cases and how disagreement was resolved.
Judge calibration: independently human-score a sample before using a model judge. Ask the judge to assess claims against evidence using a fixed rubric. Compare agreement and disagreement cases, blind the judge to candidate names, and check whether answer order or verbosity changes scores. A judge's explanation is not ground truth.
Exercise: compare two prompt versions on the same examples. Report the count improved, regressed, unchanged, and invalid, rather than just a net average. Rerun stochastic cases and report the variation. Do not silently drop provider failures from the denominator.
Worked decision: candidate B raises correctness from 88% to 93% but exposes one forbidden passage. Block B, retain the safe baseline, and investigate authorization before tuning the prompt. Candidate C improves correctness but doubles latency beyond the agreed budget; it needs another tradeoff decision.
Deliverable: a machine-readable scorecard with dataset, model, prompt, retrieval, and application versions, plus a decision record. Keep deterministic boundary tests in CI and live-model evaluations separately scheduled or explicitly invoked so cost and variability are visible.