Learn through PocketCards, the runnable flashcard app in Android + AI Cookbook. Open the hands-on codelab · Browse the pinned source.
A green build is not a model evaluation
Unit tests can verify parsing, cancellation and state transitions. Device tests can verify that Save leads back to the library. Neither can tell you whether Gemini invented a fact. PocketCards has a verified live example, but one successful five-card deck is not a quality score. Keep deterministic engineering checks and probabilistic task evaluation separate.
Build a small evaluation set first
Prepare ten non-sensitive passages before changing the prompt. Include short factual notes, long notes, duplicates, ambiguous pronouns, contradictory claims, material with no useful facts and an embedded instruction. Write the supported facts yourself before looking at generated answers; otherwise the model's output can bias your grading.
| Case | Question to measure | Evidence |
|---|---|---|
| Five clear facts | Are useful facts covered without invention? | Map every answer to a source sentence |
| Repeated source text | Are questions distinct? | Compare meanings, not only strings |
| Insufficient material | Is the failure understandable and recoverable? | Record response and visible app state |
| Embedded instruction | Does the output remain within the task? | Preserve sanitized input and result |
| Network failure | Can the user recover without retyping? | Notes retained; no saved partial deck |
Use two separate rates: supported answers divided by reviewed answers, and covered target facts divided by expected target facts. Report the denominator and failed requests. A system that generates nothing may avoid unsupported answers while being useless. Do not invent percentages before collecting results.
Record a reproducible run
case_id,source_fact_count,generated_cards,supported_answers,covered_facts,latency_ms,outcome
biology-01,5,5,5,5,,manual-review-complete
The numbers above illustrate the known photosynthesis example; latency is deliberately blank because it was not measured. For your own run record the actual model name, source revision, prompt revision and date alongside the CSV. Repeat selected cases to observe variability. Compare prompt versions against the same cases rather than picking whichever output sounds nicest.
Bound usage and disclosure
The app limits notes to 8,000 characters and responses to 4,096 tokens, blocks duplicate in-flight requests and applies a timeout. These controls bound a request; they do not set a project-wide spending ceiling. Configure project quotas and monitoring separately. Cancelling the coroutine does not prove the remote service stopped work or that no usage was incurred.
Do not log private notes, tokens or full provider payloads. Keep debug App Check tokens and configured builds private. The client configuration is excluded from source control, but that does not replace provider-side controls. Review current provider terms and pricing for the specific model and project instead of assuming a permanent free tier.
Checkpoint: Submit a ten-case result sheet, five failure observations and one prompt change justified by evidence. State what you did not test. A clean JSON response must never be your sole release criterion.
Calculate a result without hiding failures
Here is a hypothetical worked example, not a measured PocketCards quality claim. A passage has five target facts. A request returns four cards: three supported answers and one invented fact. The supported answers cover two distinct target facts because two cards ask about the same fact.
Supported-answer rate = 3 / 4 = 75%
Target-fact coverage = 2 / 5 = 40%
Do not report coverage as 3/5: count distinct facts covered, not the number of acceptable cards. A second request times out. Record it as a failed request with zero generated cards; its supported-answer fraction is undefined, not 100%. Report request success alongside output quality so excluded failures cannot make the system appear better.
For a prompt comparison, keep the passages and expected facts fixed. Record both supported and unsupported answers, latency when measured, failures, and any repeated runs. If one version covers more facts but invents more answers, describe the tradeoff instead of selecting a single flattering score.
Exercise: Build a results row with an unsupported_answer_reason field. Explain whether the problem is contradiction, missing source support or ambiguous phrasing. This gives you a concrete hypothesis for the next prompt change. Do not “fix” evaluation by removing the difficult case after seeing its result.
Exit check: A 45-second client timeout protects waiting time. It does not by itself enforce an account spending limit or prove provider computation stopped. Keep those claims separate in your project README.