quill: add incident response runbook draft

wizard-work/quill/incident-response-runbook.md

@@ -0,0 +1,83 @@
+# Incident Response Runbook — Web Service
+
+Version: 0.1 (draft)
+Author: quill
+Date: 2026-10-26
+
+---
+
+## 1. Purpose
+
+Define a repeatable, step-by-step process for responding to service incidents so every team member knows exactly what to do in the first 15 minutes.
+
+## 2. Severity Levels
+
+| Level | Definition | Response Time | Examples |
+|---|---|---|---|
+| SEV-1 | Complete outage or data breach | 15 min | All API endpoints return 500; PII exposed |
+| SEV-2 | Major degradation | 30 min | >50% errors on a critical path; DB replication lag > 60s |
+| SEV-3 | Minor issue, no customer impact | 2 hours | Non-critical feature misbehaving; internal tooling down |
+| SEV-4 | Monitoring/follow-up | Next business day | Capacity planning, backlogged tech debt |
+
+## 3. Roles
+
+- Incident Commander (IC): owns timeline, comms, and escalation
+- Tech Lead: owns diagnosis and fix
+- Comms Lead: drafts customer-facing updates
+- Scribe: records timeline and decisions
+
+## 4. Immediate Response (First 15 Minutes)
+
+1. Acknowledge the alert.
+2. Declare severity and assign IC.
+3. Open the incident channel (Slack #incident-<YYYYMMDD>-NN).
+4. Create the incident ticket with title format: `[SEV-N] <short description>`.
+5. IC posts a 1-line status update in #status.
+
+## 5. Triage Checklist
+
+- [ ] Check dashboards: latency p99, error rate, saturation (CPU/RAM/DB connections).
+- [ ] Check deploy log: was there a deploy in the last 24h?
+- [ ] Check dependency status pages (DB, CDN, auth provider).
+- [ ] Check recent config changes.
+- [ ] Review error logs for the dominant error signature.
+
+## 6. Common Playbooks
+
+### 6.1 Database Connection Exhaustion
+1. Show current connections: `SELECT count(*) FROM pg_stat_activity;`
+2. Kill idle-in-transaction sessions older than 5 minutes.
+3. If pooler is full, scale read replicas.
+4. If root cause is a runaway query, cancel it and deploy a hotfix or rollback.
+
+### 6.2 Deployment Regression
+1. Rollback to previous known-good release.
+2. Verify error rate drops within 5 minutes.
+3. Keep the previous build artifact for post-mortem.
+
+### 6.3 Third-Party Outage
+1. Enable feature flag to disable the dependency path.
+2. Serve degraded mode (cached responses or static fallback).
+3. Monitor for dependency recovery; re-enable when healthy.
+
+## 7. Communication Cadence
+
+- Update #status every 15 minutes until resolved.
+- Final customer note within 30 minutes of resolution.
+- Post-mortem draft due within 48 hours.
+
+## 8. Post-Mortem Template
+
+1. Summary (what happened, who was affected, for how long)
+2. Timeline (UTC timestamps, key decisions)
+3. Root cause (5-whys or equivalent)
+4. Remediation steps (immediate + longer-term)
+5. Action items (owner, due date)
+
+## 9. Escalation
+
+If the on-call engineer cannot restore service within 30 minutes for SEV-1 or 60 minutes for SEV-2, the IC must escalate to the engineering manager and on-call director.
+
+---
+
+End of runbook.