quill: draft incident response runbook

wizard-work/quill/incident-runbook.md

@@ -1,29 +1,30 @@
-# Incident Response Runbook Template
+# Incident Response Runbook
 
-## Purpose
+## 1. Overview
-Standardize response steps for severity-1 through severity-3 incidents.
+This runbook defines the standard operating procedure for responding to security and availability incidents. It is intended for on-call engineers and security staff.
 
-## Severity Levels
+## 2. Severity Levels
-- **SEV-1**: Complete outage or data loss. Page on-call lead. Target resolution: 1 hour.
+- **SEV1 (Critical):** Service down for all users; data breach confirmed.
-- **SEV-2**: Major degradation. Page on-call. Target resolution: 4 hours.
+- **SEV2 (High):** Major functionality impaired; potential security issue.
-- **SEV-3**: Minor impact. Ticket and triage within business hours. Target resolution: 24 hours.
+- **SEV3 (Medium):** Minor degradation; limited user impact.
+- **SEV4 (Low):** Cosmetic or non-urgent bugs.
 
-## Initial Actions
+## 3. Detection & Alerting
-1. Acknowledge alert and assign incident commander.
+- Alerts are fired via PagerDuty.
-2. Open incident channel (#incident-YYYY-MM-DD).
+- Acknowledge within 5 minutes for SEV1/SEV2.
-3. Declare severity and start timeline doc.
+- Always check #incident Slack channel for context.
 
-## Communication Plan
+## 4. Response Steps
-- Internal: 15-min updates in incident channel.
+1. **Assess:** Determine scope, impact, and severity.
-- External: Status page update within 30 min for SEV-1/2.
+2. **Contain:** Isolate affected systems (e.g., disable account, block IP).
-- Leadership: Email summary every hour for SEV-1.
+3. **Eradicate:** Remove root cause (patch, rotate keys, revoke tokens).
+4. **Recover:** Restore service from clean backups; validate integrity.
+5. **Post-mortem:** Within 48 hours, document timeline, root cause, and remediation actions.
 
-## Mitigation Steps
+## 5. Communication
-- Roll back recent deploy if cause unknown.
+- Update status page within 15 minutes for SEV1/SEV2.
-- Enable circuit breakers / rate limits.
+- Notify legal/compliance if personal data involved.
-- Switch to read-replica or backup region.
+- Keep internal stakeholders informed every 30 minutes until resolved.
 
-## Post-Incident Review
+## 6. Runbook Maintenance
-- Draft RCA within 24 hours.
+Review quarterly or after any major incident.
-- Schedule review meeting within 48 hours.
-- Track remediation tickets with owners and due dates.