troupe-standup/wizard-work/quill/incident-postmortem-template.md

Incident Postmortem — [Service Name]

Metadata

• Incident ID: INC-2026-001
• Severity: SEV-2
• Date: 2026-10-24
• Duration: 42 minutes
• Author: quill

Summary

A dependency upgrade introduced a breaking change in the authentication flow, causing a 42-minute outage affecting 18% of user sessions.

Timeline (all times UTC)

• 14:03 — v2.4.1 deployed to prod
• 14:05 — PagerDuty alert: auth failure rate > 5%
• 14:07 — on-call confirmed spike, began rollback
• 14:22 — rollback completed to v2.4.0
• 14:35 — verification passed, all-clear sent
• 14:45 — postmortem scheduled

Root Cause

auth-client v3.1.0 changed the token refresh endpoint path from /oauth/token to /v2/oauth/token. The deploy updated the dependency but did not update the hardcoded path. All refresh attempts 404'd until rollback.

Blast Radius

• 18% of active sessions affected
• ~12,000 failed login/refresh attempts
• 3 downstream services degraded (notifications, recommendations, search indexing)

Resolution

Rollback to v2.4.0 restored service. Post-rollback, we patched the path constant and re-deployed v2.4.2 with the fix plus a regression test.

Action Items

1. Add contract test for auth endpoint paths (owner: @alex, due: 2026-10-31)
2. Pin auth-client in CI until path test is green (owner: @quill, due: 2026-10-27)
3. Add dependency-change alert in deploy pipeline (owner: @bot-crew, due: 2026-11-03)

Lessons Learned

• Do not silently bump major versions in patch deploys.
• Path constants should be discovered, not hardcoded.
• Rollback runbook needs a <5-min verification gate.