From 06697f7aff3356ff9481b13738ec78cae1a01a10 Mon Sep 17 00:00:00 2001 From: quill Date: Mon, 2 Nov 2026 18:44:25 +0000 Subject: [PATCH] quill: draft incident response runbook --- wizard-work/quill/incident-runbook.md | 47 ++++++++++++++------------- 1 file changed, 24 insertions(+), 23 deletions(-) diff --git a/wizard-work/quill/incident-runbook.md b/wizard-work/quill/incident-runbook.md index fb85f13..e93ed88 100644 --- a/wizard-work/quill/incident-runbook.md +++ b/wizard-work/quill/incident-runbook.md @@ -1,29 +1,30 @@ -# Incident Response Runbook Template +# Incident Response Runbook -## Purpose -Standardize response steps for severity-1 through severity-3 incidents. +## 1. Overview +This runbook defines the standard operating procedure for responding to security and availability incidents. It is intended for on-call engineers and security staff. -## Severity Levels -- **SEV-1**: Complete outage or data loss. Page on-call lead. Target resolution: 1 hour. -- **SEV-2**: Major degradation. Page on-call. Target resolution: 4 hours. -- **SEV-3**: Minor impact. Ticket and triage within business hours. Target resolution: 24 hours. +## 2. Severity Levels +- **SEV1 (Critical):** Service down for all users; data breach confirmed. +- **SEV2 (High):** Major functionality impaired; potential security issue. +- **SEV3 (Medium):** Minor degradation; limited user impact. +- **SEV4 (Low):** Cosmetic or non-urgent bugs. -## Initial Actions -1. Acknowledge alert and assign incident commander. -2. Open incident channel (#incident-YYYY-MM-DD). -3. Declare severity and start timeline doc. +## 3. Detection & Alerting +- Alerts are fired via PagerDuty. +- Acknowledge within 5 minutes for SEV1/SEV2. +- Always check #incident Slack channel for context. -## Communication Plan -- Internal: 15-min updates in incident channel. -- External: Status page update within 30 min for SEV-1/2. -- Leadership: Email summary every hour for SEV-1. +## 4. Response Steps +1. **Assess:** Determine scope, impact, and severity. +2. **Contain:** Isolate affected systems (e.g., disable account, block IP). +3. **Eradicate:** Remove root cause (patch, rotate keys, revoke tokens). +4. **Recover:** Restore service from clean backups; validate integrity. +5. **Post-mortem:** Within 48 hours, document timeline, root cause, and remediation actions. -## Mitigation Steps -- Roll back recent deploy if cause unknown. -- Enable circuit breakers / rate limits. -- Switch to read-replica or backup region. +## 5. Communication +- Update status page within 15 minutes for SEV1/SEV2. +- Notify legal/compliance if personal data involved. +- Keep internal stakeholders informed every 30 minutes until resolved. -## Post-Incident Review -- Draft RCA within 24 hours. -- Schedule review meeting within 48 hours. -- Track remediation tickets with owners and due dates. +## 6. Runbook Maintenance +Review quarterly or after any major incident.