Assignment
Use a fake service with controllable retrieval delay, model delay, failure rate, and token usage. Define a fixed test workload of 100 requests.
Build sequence
- Record median and tail latency, queue time, and accepted outputs.
- Compare no cache with a correctly scoped exact-match cache.
- Try two concurrency limits and report throughput and rejection rate.
- Compute cost per accepted result including failed calls and repairs.
Acceptance checks
- Cache keys prevent reuse across incompatible tenants or versions.
- The report shows the workload and percentile calculation method.
- An optimization is accepted only when quality gates remain satisfied.
Evidence to keep
Submit the working artifact or decision document described above, the inputs used, and the observed results for every acceptance check. Include one failed attempt and the change you made after investigating it. Label fixture-based outcomes separately from live-system measurements. These artifacts become part of your final portfolio review.
Guided workshop: account for successful and wasted work
The useful unit is a completed acceptable task, not just one model call. Retries, retrieval, reranking, and failed generations all consume resources. Separate variable request cost from fixed infrastructure and human review cost.
# academy-check
attempts = [
{"cost": .02, "useful": True, "ms": 300},
{"cost": .03, "useful": False, "ms": 900},
{"cost": .02, "useful": True, "ms": 400},
]
total_cost = sum(row["cost"] for row in attempts)
useful = sum(row["useful"] for row in attempts)
assert abs(total_cost / useful - .035) < 1e-9
These are fictional cost units. The failed attempt stays in the numerator. If no task succeeds, report that no finite cost per useful result is established; do not divide by one to make the dashboard look normal.
Budget exercise: allocate a two-second request budget across authentication, retrieval, generation, and response validation. Example allocations are 100 ms, 300 ms, 1,400 ms, and 200 ms. Measure actual stages and their overlap rather than assuming the sum of independent p95 values equals the end-to-end p95.
Caching experiment: compare no cache, retrieval cache, and answer cache on a fixed request set. Record hit rate, quality, latency, and stale-answer cases. An answer cache must incorporate authorization scope, source version, prompt version, and model behavior where relevant. A cache hit that exposes another user's result is a failed request even if it is fast.
Capacity challenge: increase concurrent requests gradually while monitoring queue time, active work, timeouts, throughput, and memory. State an admission limit and what callers see when it is exceeded. Batching can improve throughput while increasing latency for the first request waiting in a batch.
Worked decision: a cheaper model with more retries and manual review may have a higher cost per useful task. Compare complete workflows using the same acceptance criteria. A faster retrieval stage may not affect total latency if generation dominates; optimize measured bottlenecks.
Deliverable: preserve raw observations, a workload description, quality thresholds, capacity settings, and a sensitivity table for traffic, token count, failure rate, and human review. Use current provider prices only after recording the source and date; the arithmetic in this exercise intentionally avoids a pricing promise.