diff --git a/wizard-work/quill/runbook-incident-response.md b/wizard-work/quill/runbook-incident-response.md new file mode 100644 index 0000000..b0ff7e7 --- /dev/null +++ b/wizard-work/quill/runbook-incident-response.md @@ -0,0 +1,49 @@ +# Incident Response Runbook + +## Purpose +This runbook defines the minimum steps for a first responder during a service incident. It is not exhaustive; it is a checklist to prevent panic and ensure consistency. + +## Severity Classification + +| Severity | Definition | Response SLA | Notification | +|----------|------------|--------------|--------------| +| P1 | Total outage or data loss | 15 min | Page on-call + manager | +| P2 | Major feature broken for all users | 30 min | Page on-call | +| P3 | Minor feature broken or major feature degraded for some users | 2 hrs | Slack #incidents | +| P4 | Cosmetic or low-impact | Next business day | Ticket | + +## First 5 Minutes + +1. **Acknowledge the alert.** Do not dismiss until a human is actively investigating. +2. **Open a dedicated incident channel.** Name it `#incident--`. +3. **Assign roles.** Incident Commander (IC), Communications Lead (CL), and Technical Lead (TL). +4. **Assess impact.** Confirm affected services, user count, and error symptoms. +5. **Declare severity** using the table above and communicate it to the channel. + +## Investigation Loop (repeat until mitigated) + +1. **Gather data.** Check dashboards, recent deploys, config changes, dependency health. +2. **Form a hypothesis.** State it as: "If X, then Y should happen." +3. **Test the hypothesis.** Make a single, reversible change or run a targeted query. +4. **Update status.** Post a one-line summary every 10 minutes. + +## Communication Cadence + +- Internal: update the incident channel every 10 minutes, even if no change. +- External: draft a status page update within 30 minutes of P1/P2 declaration. + +## Mitigation vs. Resolution + +- Mitigation: service health restored, monitoring stable for 15 minutes. +- Resolution: root cause fixed, not just masked. Timeline documented. + +## Post-Incident + +1. Schedule a blameless postmortem within 48 hours for P1/P2, 5 days for P3. +2. Create follow-up tickets with owners and due dates. +3. Update this runbook with any missing step discovered during the incident. + +## Escalation + +- If you cannot identify the affected service within 10 minutes, escalate to senior on-call. +- If you are unsure of severity, default to one level higher.