The project
Build a visual field-note assistant that accepts a user-selected photo and optional dictated context, then produces an editable structured note.
Prerequisites
Complete the six teaching modules and their exercises. Keep the design notes and failure cases: they form the starting evidence for this project.
Build in increments
1. What multimodal AI means
Understand how different input types contribute evidence. Apply the module decision to this project. Your checkpoint: Your output does not present unsupported inference as an observed fact.
2. Capture, selection and permissions
Give users control over the media they share. Apply the module decision to this project. Your checkpoint: No image is sent merely because the picker returned it.
3. Image preparation and transport
Convert media without losing meaning or overwhelming the request. Apply the module decision to this project. Your checkpoint: Your adapter receives a supported, bounded and correctly oriented input.
4. Audio and video context
Control temporal information and resource use. Apply the module decision to this project. Your checkpoint: Asynchronous media processing cannot attach evidence to the wrong note.
5. Structured results and correction
Turn multimodal interpretation into editable application data. Apply the module decision to this project. Your checkpoint: Generated fields are not treated as confirmed records before review.
6. Evaluation across modalities
Measure the errors specific to each input type. Apply the module decision to this project. Your checkpoint: You can attribute the main errors to the correct part of the pipeline.
Acceptance criteria
- Preserve image orientation and bound upload size
- Handle denied permissions and unsupported media
- Evaluate factual grounding and let the user correct generated fields
Evidence to submit
- A repository or reproducible design artifact with setup instructions and tested tool versions.
- A diagram showing components, data flow, authority and failure boundaries.
- An evaluation table with inputs, expected results, actual results and explanations of failures.
- A demonstration of the normal path and at least two relevant failure paths.
- A short decision record covering alternatives, known limitations and the next improvement.
Review rubric
| Dimension | Ready to demonstrate | Revisit when |
|---|---|---|
| Understanding | Explain each boundary and why it exists | You can only repeat framework terminology |
| Implementation | Reproduce the core behavior from a clean setup | Hidden local state is required |
| Reliability | Show bounded failures and useful recovery | The demo only works on the happy path |
| Evidence | Present observed outcomes and limitations | Success is inferred from a generated answer |
| Ownership | Modify a requirement and explain the consequences | You cannot adapt the implementation independently |
Do not treat a completed reading checklist as proof of project readiness. Revisit any module whose checkpoint you cannot demonstrate.