Deployment turns a local experiment into a service with users, resource limits, and operational ownership. Define how the service behaves when a dependency fails before allowing other applications to depend on it.
Trace a request
A request moves through authentication, input validation, retrieval, model generation, output validation, and response delivery. Give it a request ID and record safe timing and status information for each stage. Record model and prompt versions so a regression can be associated with a change.
| Failure | Application behavior |
|---|---|
| Invalid input | Reject before expensive work |
| Retrieval timeout | Return a controlled fallback |
| Model rate limit | Bounded retry within the request budget |
| Invalid generated output | Reject or use a bounded recovery path |
| Client cancellation | Stop unnecessary downstream work where supported |
Keep secrets in server configuration. Limit request sizes, concurrency, and spending. Redact sensitive content rather than logging full prompts by default. Measure tail latency as well as the average: a minority of very slow requests can dominate user complaints.
Exercise
Write a runbook covering configuration, a sample request, expected metrics, common failures, and how to disable model calls. Inject a timeout and a malformed response in a staging environment.
Check: a new operator can identify the failed stage from a request ID, turn off the feature, and restore the previous version. Document which components remain usable during an outage.