troupe-standup/wizard-work/quill/incident-runbook.md

Incident Response Runbook

1. Overview

This runbook defines the standard operating procedure for responding to security and availability incidents. It is intended for on-call engineers and security staff.

2. Severity Levels

• SEV1 (Critical): Service down for all users; data breach confirmed.
• SEV2 (High): Major functionality impaired; potential security issue.
• SEV3 (Medium): Minor degradation; limited user impact.
• SEV4 (Low): Cosmetic or non-urgent bugs.

3. Detection & Alerting

• Alerts are fired via PagerDuty.
• Acknowledge within 5 minutes for SEV1/SEV2.
• Always check #incident Slack channel for context.

4. Response Steps

1. Assess: Determine scope, impact, and severity.
2. Contain: Isolate affected systems (e.g., disable account, block IP).
3. Eradicate: Remove root cause (patch, rotate keys, revoke tokens).
4. Recover: Restore service from clean backups; validate integrity.
5. Post-mortem: Within 48 hours, document timeline, root cause, and remediation actions.

5. Communication

• Update status page within 15 minutes for SEV1/SEV2.
• Notify legal/compliance if personal data involved.
• Keep internal stakeholders informed every 30 minutes until resolved.

6. Runbook Maintenance

Review quarterly or after any major incident.