Incident Response Runbook — Draft v0.1

Purpose

Provide a repeatable, evidence-based workflow for responding to security incidents so every handler knows the exact steps, who to page, and what to preserve. Reduces mean-time-to-detect (MTTD) and mean-time-to-respond (MTTR) by removing guesswork.

Scope

Applies to all production systems, managed endpoints, and cloud identities under *.troupe.internal. Excludes physical access controls and paper records.

Detection & Alerting

Alert sources: PagerDuty, CloudWatch Alarms, Sigma rules, EDRE telemetry, user-reported phishing.
Triage criteria (P1): data exfiltration path confirmed, customer PII exposure, or ransomware encryption on >1 host.
Initial evidence to collect: alert payload, affected asset list, last-known-good state, CloudTrail event IDs for the past 6 hours.
Tag every new detection as an incident only after triage confirms malicious intent.

Containment

Isolate affected hosts at the network layer (nftables drop rule or AWS SG deny-all).
Rotate all credentials for accounts that touched the affected asset during the incident window.
Do NOT power off hosts before volatile memory acquisition; capture LiME or osquery process/memory tables first.
Record containment actions in the incident ticket with timestamps for audit.

Eradication

Verify removal: rootkit scan (rkhunter --check --skip-keypress), persistence audit (crontab -l, systemctl list-units --type=service --state=failed, ~/.ssh/authorized_keys diffs).
Re-image from known-good templates; do not patch in place.
Revoke every API key, OAuth token, and SSH key created or used in the incident window.
Confirm eradication with a second handler before closing the incident.

Recovery

Restore from snapshots taken before the first indicator of compromise (IOC).
Validate integrity: sha256sum of critical binaries, checksum of config files against git HEAD.
Enable enhanced monitoring on recovered assets for 72 hours: auditd syscall logging, bpftrace exec tracing.
If the incident affected customer data, notify the DPO before resuming normal traffic.

Post-Incident Review

Draft timeline within 24 hours using verified evidence only; mark speculative inferences with [UNCONFIRMED].
Document root cause, blast radius, and three concrete remediation actions with owners and deadlines.
Update detection rules and this runbook within 48 hours.

Communication Templates

Internal #incidents: timestamp, host(s), impact, current action, ETA.
Executive summary: 3 bullets — what happened, what we did, what changes.
Customer notice (if required): exposure type, affected record count, remediation steps, contact.

2.7 KiB Raw Blame History