quill: add incident postmortem template with example
This commit is contained in:
1 parent
6565ecd0f9
commit
40a7c6cf9d
1 file changed
+40
@@ -0,0 +1,40 @@
|
||||
# Incident Postmortem — [Service Name]
|
||||
|
||||
## Metadata
|
||||
- Incident ID: INC-2026-001
|
||||
- Severity: SEV-2
|
||||
- Date: 2026-10-24
|
||||
- Duration: 42 minutes
|
||||
- Author: quill
|
||||
|
||||
## Summary
|
||||
A dependency upgrade introduced a breaking change in the authentication flow, causing a 42-minute outage affecting 18% of user sessions.
|
||||
|
||||
## Timeline (all times UTC)
|
||||
- 14:03 — v2.4.1 deployed to prod
|
||||
- 14:05 — PagerDuty alert: auth failure rate > 5%
|
||||
- 14:07 — on-call confirmed spike, began rollback
|
||||
- 14:22 — rollback completed to v2.4.0
|
||||
- 14:35 — verification passed, all-clear sent
|
||||
- 14:45 — postmortem scheduled
|
||||
|
||||
## Root Cause
|
||||
`auth-client` v3.1.0 changed the token refresh endpoint path from `/oauth/token` to `/v2/oauth/token`. The deploy updated the dependency but did not update the hardcoded path. All refresh attempts 404'd until rollback.
|
||||
|
||||
## Blast Radius
|
||||
- 18% of active sessions affected
|
||||
- ~12,000 failed login/refresh attempts
|
||||
- 3 downstream services degraded (notifications, recommendations, search indexing)
|
||||
|
||||
## Resolution
|
||||
Rollback to v2.4.0 restored service. Post-rollback, we patched the path constant and re-deployed v2.4.2 with the fix plus a regression test.
|
||||
|
||||
## Action Items
|
||||
1. Add contract test for auth endpoint paths (owner: @alex, due: 2026-10-31)
|
||||
2. Pin auth-client in CI until path test is green (owner: @quill, due: 2026-10-27)
|
||||
3. Add dependency-change alert in deploy pipeline (owner: @bot-crew, due: 2026-11-03)
|
||||
|
||||
## Lessons Learned
|
||||
- Do not silently bump major versions in patch deploys.
|
||||
- Path constants should be discovered, not hardcoded.
|
||||
- Rollback runbook needs a <5-min verification gate.
|
||||
Reference in new issue
Block a user