diff --git a/wizard-work/quill/incident-postmortem-template.md b/wizard-work/quill/incident-postmortem-template.md new file mode 100644 index 0000000..b5f78d4 --- /dev/null +++ b/wizard-work/quill/incident-postmortem-template.md @@ -0,0 +1,40 @@ +# Incident Postmortem — [Service Name] + +## Metadata +- Incident ID: INC-2026-001 +- Severity: SEV-2 +- Date: 2026-10-24 +- Duration: 42 minutes +- Author: quill + +## Summary +A dependency upgrade introduced a breaking change in the authentication flow, causing a 42-minute outage affecting 18% of user sessions. + +## Timeline (all times UTC) +- 14:03 — v2.4.1 deployed to prod +- 14:05 — PagerDuty alert: auth failure rate > 5% +- 14:07 — on-call confirmed spike, began rollback +- 14:22 — rollback completed to v2.4.0 +- 14:35 — verification passed, all-clear sent +- 14:45 — postmortem scheduled + +## Root Cause +`auth-client` v3.1.0 changed the token refresh endpoint path from `/oauth/token` to `/v2/oauth/token`. The deploy updated the dependency but did not update the hardcoded path. All refresh attempts 404'd until rollback. + +## Blast Radius +- 18% of active sessions affected +- ~12,000 failed login/refresh attempts +- 3 downstream services degraded (notifications, recommendations, search indexing) + +## Resolution +Rollback to v2.4.0 restored service. Post-rollback, we patched the path constant and re-deployed v2.4.2 with the fix plus a regression test. + +## Action Items +1. Add contract test for auth endpoint paths (owner: @alex, due: 2026-10-31) +2. Pin auth-client in CI until path test is green (owner: @quill, due: 2026-10-27) +3. Add dependency-change alert in deploy pipeline (owner: @bot-crew, due: 2026-11-03) + +## Lessons Learned +- Do not silently bump major versions in patch deploys. +- Path constants should be discovered, not hardcoded. +- Rollback runbook needs a <5-min verification gate.