From 3785683a915160420e7e4d0ba50ab34fc133fd56 Mon Sep 17 00:00:00 2001 From: quill Date: Thu, 22 Oct 2026 11:09:56 +0000 Subject: [PATCH] quill: draft runbook for API service restart --- wizard-work/quill/incident-runbook.md | 29 ++++++++++++++++++++++++ wizard-work/quill/runbook-draft.md | 32 +++++++++++++++++++++++++++ 2 files changed, 61 insertions(+) create mode 100644 wizard-work/quill/incident-runbook.md create mode 100644 wizard-work/quill/runbook-draft.md diff --git a/wizard-work/quill/incident-runbook.md b/wizard-work/quill/incident-runbook.md new file mode 100644 index 0000000..fb85f13 --- /dev/null +++ b/wizard-work/quill/incident-runbook.md @@ -0,0 +1,29 @@ +# Incident Response Runbook Template + +## Purpose +Standardize response steps for severity-1 through severity-3 incidents. + +## Severity Levels +- **SEV-1**: Complete outage or data loss. Page on-call lead. Target resolution: 1 hour. +- **SEV-2**: Major degradation. Page on-call. Target resolution: 4 hours. +- **SEV-3**: Minor impact. Ticket and triage within business hours. Target resolution: 24 hours. + +## Initial Actions +1. Acknowledge alert and assign incident commander. +2. Open incident channel (#incident-YYYY-MM-DD). +3. Declare severity and start timeline doc. + +## Communication Plan +- Internal: 15-min updates in incident channel. +- External: Status page update within 30 min for SEV-1/2. +- Leadership: Email summary every hour for SEV-1. + +## Mitigation Steps +- Roll back recent deploy if cause unknown. +- Enable circuit breakers / rate limits. +- Switch to read-replica or backup region. + +## Post-Incident Review +- Draft RCA within 24 hours. +- Schedule review meeting within 48 hours. +- Track remediation tickets with owners and due dates. diff --git a/wizard-work/quill/runbook-draft.md b/wizard-work/quill/runbook-draft.md new file mode 100644 index 0000000..cf50248 --- /dev/null +++ b/wizard-work/quill/runbook-draft.md @@ -0,0 +1,32 @@ +# Service Restart Runbook (Draft v0.1) + +## Purpose +Provide a repeatable, low-risk procedure for restarting the API service without dropping in-flight requests. + +## Scope +Applies to api-svc (primary and canary). Excludes database migrations. + +## Prerequisites +- Access to the ops bastion +- PagerDuty duty-handoff confirmed +- Deployment queue is clear (check via `kubectl get events -n deploy`) + +## Steps + +1. Verify current health: `kubectl get pods -n api -l app=api-svc` +2. Confirm no in-flight requests: `kubectl exec -n api deploy/api-svc -- curl -s localhost:8080/health | jq .in_flight` +3. Drain one pod: `kubectl drain --ignore-daemonsets --delete-emptydir-data` +4. Wait for pod reschedule: `kubectl rollout status deployment/api-svc -n api --timeout=120s` +5. Repeat for remaining pods, one at a time +6. Run smoke test: `curl -s https://api.internal/healthz | jq .status` +7. Page on-call if smoke test fails + +## Rollback +If health check fails after 2 pods, halt and execute: `kubectl rollout undo deployment/api-svc -n api` + +## Evidence Required +- Screenshot of `kubectl get pods` post-restart +- Smoke test output showing `"status":"ok"` + +--- +*Draft prepared by Quill — verify commands against staging before production use.*