Assignment
Choose three failures: expired credential, incompatible document schema, and model timeout. Provide a peer with only your runbook and synthetic logs.
Build sequence
- Define a workflow-level service objective and two actionable alerts.
- Create traces that locate each failure stage without exposing sensitive content.
- Time the peer’s diagnosis and recovery steps.
- Write a postmortem with detection, impact, recovery, and assigned follow-up work.
Acceptance checks
- The peer can identify the correct owner and recovery action.
- Recovery includes a representative successful task, not only a green process status.
- Each follow-up has a specific expected improvement and verification method.
Evidence to keep
Submit the working artifact or decision document described above, the inputs used, and the observed results for every acceptance check. Include one failed attempt and the change you made after investigating it. Label fixture-based outcomes separately from live-system measurements. These artifacts become part of your final portfolio review.
Guided workshop: lead a customer incident
At 10:00 in the simulation, Northstar operators report that drafts cite yesterday's invoice policy. At 10:03, one operator reports a timeout. You are incident lead. The purpose is to practice containment, diagnosis, communication, and recovery without inventing certainty.
Timeline injects: at 10:05 the index job is found to have failed halfway; at 10:08 the previous index is still available; at 10:12 someone proposes deleting all caches; at 10:15 the sponsor asks when service will be fully restored. Give these facts to a peer one at a time or reveal them sequentially in your own written exercise.
Worked response: pause the affected drafting path, preserve manual support, record the incident owner and scope, and switch to the last verified index if its freshness and compatibility meet the fallback policy. Do not delete diagnostic evidence reflexively. Check whether answer caches include source versions before choosing invalidation.
Write a customer update: “Drafting for invoice questions is paused while we investigate stale policy references. Manual ticket handling remains available. We have identified an interrupted index update and are validating the previous version. Next update at 10:30.” It states impact, action, uncertainty, and the next communication point without promising an unverified fix time.
Exercise: separate hypotheses from observations. Trace one failed request through application version, index version, cache key, and provider timing. Verify recovery with supported, unsupported, and tenant-denied requests. Confirm that queued work cannot republish stale answers after the system resumes.
Post-incident review: record contributing conditions, detection gap, user impact, effective actions, and preventive work with owners. An SLO is a measurable reliability objective; an SLA is a contractual commitment. Do not invent either during the incident or promise contractual terms as part of a technical update.
Assessment: hand the runbook to a learner who did not build the system. They should identify the active version, disable drafting, inspect a failed request, restore a verified version, and know when to escalate. A runbook that depends on its author's memory has not passed the handoff test.