diff --git a/wizard-work/quill/incident-response-runbook.md b/wizard-work/quill/incident-response-runbook.md new file mode 100644 index 0000000..b847d9e --- /dev/null +++ b/wizard-work/quill/incident-response-runbook.md @@ -0,0 +1,83 @@ +# Incident Response Runbook — Web Service + +Version: 0.1 (draft) +Author: quill +Date: 2026-10-26 + +--- + +## 1. Purpose + +Define a repeatable, step-by-step process for responding to service incidents so every team member knows exactly what to do in the first 15 minutes. + +## 2. Severity Levels + +| Level | Definition | Response Time | Examples | +|---|---|---|---| +| SEV-1 | Complete outage or data breach | 15 min | All API endpoints return 500; PII exposed | +| SEV-2 | Major degradation | 30 min | >50% errors on a critical path; DB replication lag > 60s | +| SEV-3 | Minor issue, no customer impact | 2 hours | Non-critical feature misbehaving; internal tooling down | +| SEV-4 | Monitoring/follow-up | Next business day | Capacity planning, backlogged tech debt | + +## 3. Roles + +- Incident Commander (IC): owns timeline, comms, and escalation +- Tech Lead: owns diagnosis and fix +- Comms Lead: drafts customer-facing updates +- Scribe: records timeline and decisions + +## 4. Immediate Response (First 15 Minutes) + +1. Acknowledge the alert. +2. Declare severity and assign IC. +3. Open the incident channel (Slack #incident--NN). +4. Create the incident ticket with title format: `[SEV-N] `. +5. IC posts a 1-line status update in #status. + +## 5. Triage Checklist + +- [ ] Check dashboards: latency p99, error rate, saturation (CPU/RAM/DB connections). +- [ ] Check deploy log: was there a deploy in the last 24h? +- [ ] Check dependency status pages (DB, CDN, auth provider). +- [ ] Check recent config changes. +- [ ] Review error logs for the dominant error signature. + +## 6. Common Playbooks + +### 6.1 Database Connection Exhaustion +1. Show current connections: `SELECT count(*) FROM pg_stat_activity;` +2. Kill idle-in-transaction sessions older than 5 minutes. +3. If pooler is full, scale read replicas. +4. If root cause is a runaway query, cancel it and deploy a hotfix or rollback. + +### 6.2 Deployment Regression +1. Rollback to previous known-good release. +2. Verify error rate drops within 5 minutes. +3. Keep the previous build artifact for post-mortem. + +### 6.3 Third-Party Outage +1. Enable feature flag to disable the dependency path. +2. Serve degraded mode (cached responses or static fallback). +3. Monitor for dependency recovery; re-enable when healthy. + +## 7. Communication Cadence + +- Update #status every 15 minutes until resolved. +- Final customer note within 30 minutes of resolution. +- Post-mortem draft due within 48 hours. + +## 8. Post-Mortem Template + +1. Summary (what happened, who was affected, for how long) +2. Timeline (UTC timestamps, key decisions) +3. Root cause (5-whys or equivalent) +4. Remediation steps (immediate + longer-term) +5. Action items (owner, due date) + +## 9. Escalation + +If the on-call engineer cannot restore service within 30 minutes for SEV-1 or 60 minutes for SEV-2, the IC must escalate to the engineering manager and on-call director. + +--- + +End of runbook.