Incident Response Runbook — Web Service

Version: 0.1 (draft) Author: quill Date: 2026-10-26

1. Purpose

Define a repeatable, step-by-step process for responding to service incidents so every team member knows exactly what to do in the first 15 minutes.

2. Severity Levels

Level	Definition	Response Time	Examples
SEV-1	Complete outage or data breach	15 min	All API endpoints return 500; PII exposed
SEV-2	Major degradation	30 min	>50% errors on a critical path; DB replication lag > 60s
SEV-3	Minor issue, no customer impact	2 hours	Non-critical feature misbehaving; internal tooling down
SEV-4	Monitoring/follow-up	Next business day	Capacity planning, backlogged tech debt

3. Roles

Incident Commander (IC): owns timeline, comms, and escalation
Tech Lead: owns diagnosis and fix
Comms Lead: drafts customer-facing updates
Scribe: records timeline and decisions

4. Immediate Response (First 15 Minutes)

Acknowledge the alert.
Declare severity and assign IC.
Open the incident channel (Slack #incident--NN).
Create the incident ticket with title format: [SEV-N] <short description>.
IC posts a 1-line status update in #status.

5. Triage Checklist

Check dashboards: latency p99, error rate, saturation (CPU/RAM/DB connections).
Check deploy log: was there a deploy in the last 24h?
Check dependency status pages (DB, CDN, auth provider).
Check recent config changes.
Review error logs for the dominant error signature.

6. Common Playbooks

6.1 Database Connection Exhaustion

Show current connections: SELECT count(*) FROM pg_stat_activity;
Kill idle-in-transaction sessions older than 5 minutes.
If pooler is full, scale read replicas.
If root cause is a runaway query, cancel it and deploy a hotfix or rollback.

6.2 Deployment Regression

Rollback to previous known-good release.
Verify error rate drops within 5 minutes.
Keep the previous build artifact for post-mortem.

6.3 Third-Party Outage

Enable feature flag to disable the dependency path.
Serve degraded mode (cached responses or static fallback).
Monitor for dependency recovery; re-enable when healthy.

7. Communication Cadence

Update #status every 15 minutes until resolved.
Final customer note within 30 minutes of resolution.
Post-mortem draft due within 48 hours.

8. Post-Mortem Template

Summary (what happened, who was affected, for how long)
Timeline (UTC timestamps, key decisions)
Root cause (5-whys or equivalent)
Remediation steps (immediate + longer-term)
Action items (owner, due date)

9. Escalation

If the on-call engineer cannot restore service within 30 minutes for SEV-1 or 60 minutes for SEV-2, the IC must escalate to the engineering manager and on-call director.

End of runbook.

2.8 KiB Raw Permalink Blame History