Assignment
Use a fake service with controllable retrieval delay, model delay, failure rate, and token usage. Define a fixed test workload of 100 requests.
Build sequence
- Record median and tail latency, queue time, and accepted outputs.
- Compare no cache with a correctly scoped exact-match cache.
- Try two concurrency limits and report throughput and rejection rate.
- Compute cost per accepted result including failed calls and repairs.
Acceptance checks
- Cache keys prevent reuse across incompatible tenants or versions.
- The report shows the workload and percentile calculation method.
- An optimization is accepted only when quality gates remain satisfied.
Evidence to keep
Submit the working artifact or decision document described above, the inputs used, and the observed results for every acceptance check. Include one failed attempt and the change you made after investigating it. Label fixture-based outcomes separately from live-system measurements. These artifacts become part of your final portfolio review.