Assignment
Collect or synthesize 40 examples for a narrow classification or formatting task. The required exercise is dataset preparation and a decision report; a training run is optional if compute is available.
Build sequence
- Label failures as knowledge, instruction, format, or reasoning failures.
- Build deduplicated development and held-out splits with provenance.
- Compare prompt-only and retrieval-based alternatives where appropriate.
- Estimate training and serving requirements; specify a pass/fail rubric for any later tuned model.
Acceptance checks
- The report can recommend no tuning when evidence does not justify it.
- Held-out examples are not copied into demonstrations or training data.
- Any actual training results are clearly separated from a proposed experiment.
Evidence to keep
Submit the working artifact or decision document described above, the inputs used, and the observed results for every acceptance check. Include one failed attempt and the change you made after investigating it. Label fixture-based outcomes separately from live-system measurements. These artifacts become part of your final portfolio review.
Guided workshop: decide whether training is the right intervention
Start with a failure taxonomy. Missing current product facts suggests retrieval or data repair. Inconsistent formatting suggests a schema contract and validation. Repeated domain-specific behavior failures may justify adaptation after simpler methods are evaluated. Fine-tuning is not a way to make stale private facts reliably current.
Use this original exercise dataset of measured outcomes; the figures are fictional:
| Candidate | Correct / 100 | Invalid format | Median latency | Cost / 100 requests |
|---|---|---|---|---|
| Prompt baseline | 78 | 8 | 1.0 s | 0.20 units |
| Prompt + schema | 82 | 0 | 1.1 s | 0.21 units |
| Schema + retrieval | 91 | 0 | 1.4 s | 0.30 units |
| Adapted candidate | 92 | 0 | 1.3 s | 0.28 units |
Worked decision: a one-case improvement by the adapted candidate does not establish a reliable benefit. Account for training data creation, training runs, evaluation, hosting, rollback, and maintenance. If most remaining errors are outdated facts, improve the retrieval pipeline before committing to training.
Build your decision record: define the target behavior, baseline, error categories, dataset provenance, allowed use, train/validation/test split, proposed method, operational costs, and a go/no-go rule. Specify how you will prevent examples from the same source or customer leaking across splits.
Explain the distinction among prompting, retrieval, and parameter adaptation. Prompting changes instructions and examples at inference time. Retrieval supplies external context. Fine-tuning changes model parameters using training examples; parameter-efficient techniques update a smaller set of additional or selected parameters. These can be combined, but each adds a different maintenance obligation.
Challenge: assume the adapted model improves specialized extraction but loses performance on general support questions. Design a router or reject the model as a general replacement. Test the router too: perfect specialist performance is useless if the wrong requests reach it.
Assessment: deliver the table, failure analysis, proposed experiment, and rollback policy. If you have not actually trained and evaluated a model, label the result a decision simulation. For a real experiment, pin the model revision, training code, dataset snapshot, hyperparameters, and evaluation procedure. Do not claim live model results from the supplied fictional table.