quill: draft incident response runbook

wizard-work/quill/incident-runbook.md

@@ -1,29 +1,30 @@
-# Incident Response Runbook Template
+# Incident Response Runbook
 
-## Purpose
-Standardize response steps for severity-1 through severity-3 incidents.
+## 1. Overview
+This runbook defines the standard operating procedure for responding to security and availability incidents. It is intended for on-call engineers and security staff.
 
-## Severity Levels
-- **SEV-1**: Complete outage or data loss. Page on-call lead. Target resolution: 1 hour.
-- **SEV-2**: Major degradation. Page on-call. Target resolution: 4 hours.
-- **SEV-3**: Minor impact. Ticket and triage within business hours. Target resolution: 24 hours.
+## 2. Severity Levels
+- **SEV1 (Critical):** Service down for all users; data breach confirmed.
+- **SEV2 (High):** Major functionality impaired; potential security issue.
+- **SEV3 (Medium):** Minor degradation; limited user impact.
+- **SEV4 (Low):** Cosmetic or non-urgent bugs.
 
-## Initial Actions
-1. Acknowledge alert and assign incident commander.
-2. Open incident channel (#incident-YYYY-MM-DD).
-3. Declare severity and start timeline doc.
+## 3. Detection & Alerting
+- Alerts are fired via PagerDuty.
+- Acknowledge within 5 minutes for SEV1/SEV2.
+- Always check #incident Slack channel for context.
 
-## Communication Plan
-- Internal: 15-min updates in incident channel.
-- External: Status page update within 30 min for SEV-1/2.
-- Leadership: Email summary every hour for SEV-1.
+## 4. Response Steps
+1. **Assess:** Determine scope, impact, and severity.
+2. **Contain:** Isolate affected systems (e.g., disable account, block IP).
+3. **Eradicate:** Remove root cause (patch, rotate keys, revoke tokens).
+4. **Recover:** Restore service from clean backups; validate integrity.
+5. **Post-mortem:** Within 48 hours, document timeline, root cause, and remediation actions.
 
-## Mitigation Steps
-- Roll back recent deploy if cause unknown.
-- Enable circuit breakers / rate limits.
-- Switch to read-replica or backup region.
+## 5. Communication
+- Update status page within 15 minutes for SEV1/SEV2.
+- Notify legal/compliance if personal data involved.
+- Keep internal stakeholders informed every 30 minutes until resolved.
 
-## Post-Incident Review
-- Draft RCA within 24 hours.
-- Schedule review meeting within 48 hours.
-- Track remediation tickets with owners and due dates.
+## 6. Runbook Maintenance
+Review quarterly or after any major incident.