What you will learn
Two of the most effective ways to cut cost and latency in production: reusing the provider's prompt cache, and sending each request to the cheapest model that can handle it — with a safe fallback.
Prompt caching: reuse the repeated prefix
Many requests share a long, identical beginning: system instructions, tool definitions, output schemas, or a large reference document. Major model providers can cache that repeated prefix so later requests process it faster and, depending on the provider, at a lower price. Some providers do this automatically; others need you to mark what to cache or create a cache explicitly. Minimum lengths, lifetimes, and pricing differ, so check your provider's current documentation.
The application work is the same everywhere: put stable content first and variable content last.
[ instructions ][ tool definitions ][ schema ][ shared reference doc ] ← stable, cacheable
[ retrieved evidence ][ conversation ][ user question ] ← changes per request
A timestamp, request ID, or user name near the top of the prompt changes the prefix on every call and quietly defeats the cache. Keep those in the variable part.
Prompt caching is different from the answer cache in the earlier lesson. It does not reuse an answer — the model still generates a fresh response — so it does not create the stale-answer or cross-tenant risks of an answer cache. Still never place one tenant's private data in a "shared" prefix.
Measure it: providers report cached input tokens in their usage data. Record cached and uncached input tokens per request, and compare cost and time-to-first-token before and after reordering your prompt.
Model routing: the cheapest model that passes
Not every request needs your most capable model. A router sends easy, well-defined tasks to a smaller, cheaper model and harder ones to a larger model. Rules can be simple — task type, input length, or whether retrieval found strong evidence — and you can add a fallback when the first model's output fails validation.
# academy-check
MODELS = {"small": {"cost": 1}, "large": {"cost": 8}}
def route(task):
if task["type"] in {"classify", "extract"} and task["chars"] < 4000:
return "small"
return "large"
def run(task, call, validate):
attempts = []
first = route(task)
for model in [first, "large"] if first == "small" else ["large"]:
output = call(model, task)
attempts.append(model)
if validate(output):
return {"model": model, "output": output, "attempts": attempts,
"cost": sum(MODELS[m]["cost"] for m in attempts)}
return {"model": None, "output": None, "attempts": attempts,
"cost": sum(MODELS[m]["cost"] for m in attempts)}
ok = lambda output: output is not None
small_fails = lambda model, task: "label" if model == "large" else None
assert run({"type": "classify", "chars": 300}, lambda m, t: "label", ok)["cost"] == 1
assert run({"type": "classify", "chars": 300}, small_fails, ok)["attempts"] == ["small", "large"]
assert run({"type": "answer", "chars": 300}, lambda m, t: "text", ok)["model"] == "large"
assert run({"type": "classify", "chars": 300}, lambda m, t: None, ok)["model"] is None
The fallback cost stays in the total. A router that sends most traffic to the small model but falls back half the time may cost more than always using the large one. Measure cost per accepted result, as in the latency and cost lesson.
Evaluate the router, not just the models
Run your frozen evaluation set through the router and report, per route: quality, fallback rate, latency, and cost. Check the cases the router sends to the small model — a router that misclassifies hard cases as easy will quietly lower quality. Hard gates still apply: a cheaper route that breaks a permission or grounding case is rejected.
Exercise
Reorder your knowledge service's prompt so the stable parts come first, and record cached input tokens over twenty requests. Then add a two-model router with validation-based fallback. Compare three setups on the same cases: large only, small only, and routed.
Check: the stable prefix contains no per-request or per-tenant values; the report includes fallback rate and cost per accepted result; the routed setup passes the same quality gates as the large-only baseline, or you explain the tradeoff you are accepting.