Your assignment
Define release thresholds before comparing two configurations on the held-out set.
Work through it
- Start from the scenario below and write the expected behavior before changing code. If this is a design exercise, produce a diagram or decision table; for an implementation exercise, create a small reproducible project or test fixture.
- Address each of these decisions explicitly: Representative datasets, Task-level scoring, Change attribution. Record the assumptions that influence your choices.
- Reproduce the ordinary case and the failure described in the scenario. Use controlled inputs or fake dependencies where a live service would make the result unpredictable.
- Compare the observed result with your expected behavior. Fix the underlying decision or implementation when they differ; do not change the expected outcome merely to hide a failure.
- Save the evidence and explain what remains unverified. A simulated service failure demonstrates application handling, but does not establish real-device or provider compatibility.
Scenario to test
A new retrieval setting improves average answer quality but exposes stale restricted documents. A segmented evaluation catches the unacceptable failure.
Completion criteria
Your release gate can reject an attractive demo that fails important edge cases.
Submit the diagram or relevant code, the inputs used, the observed result and a short explanation of the tradeoff. Include at least one ordinary case and one boundary or failure case.
Review your work
- Could another learner reproduce the result from your notes?
- Does the evidence prove the completion criterion, or only that a tool ran?
- Which assumption would change your design if it proved false?
Continue only when you can explain the result independently. The final project will combine these decisions.