quill: draft runbook for API service restart

wizard-work/quill/incident-runbook.md

@@ -0,0 +1,29 @@
+# Incident Response Runbook Template
+
+## Purpose
+Standardize response steps for severity-1 through severity-3 incidents.
+
+## Severity Levels
+- **SEV-1**: Complete outage or data loss. Page on-call lead. Target resolution: 1 hour.
+- **SEV-2**: Major degradation. Page on-call. Target resolution: 4 hours.
+- **SEV-3**: Minor impact. Ticket and triage within business hours. Target resolution: 24 hours.
+
+## Initial Actions
+1. Acknowledge alert and assign incident commander.
+2. Open incident channel (#incident-YYYY-MM-DD).
+3. Declare severity and start timeline doc.
+
+## Communication Plan
+- Internal: 15-min updates in incident channel.
+- External: Status page update within 30 min for SEV-1/2.
+- Leadership: Email summary every hour for SEV-1.
+
+## Mitigation Steps
+- Roll back recent deploy if cause unknown.
+- Enable circuit breakers / rate limits.
+- Switch to read-replica or backup region.
+
+## Post-Incident Review
+- Draft RCA within 24 hours.
+- Schedule review meeting within 48 hours.
+- Track remediation tickets with owners and due dates.

wizard-work/quill/runbook-draft.md

@@ -0,0 +1,32 @@
+# Service Restart Runbook (Draft v0.1)
+
+## Purpose
+Provide a repeatable, low-risk procedure for restarting the API service without dropping in-flight requests.
+
+## Scope
+Applies to api-svc (primary and canary). Excludes database migrations.
+
+## Prerequisites
+- Access to the ops bastion
+- PagerDuty duty-handoff confirmed
+- Deployment queue is clear (check via `kubectl get events -n deploy`)
+
+## Steps
+
+1. Verify current health: `kubectl get pods -n api -l app=api-svc`
+2. Confirm no in-flight requests: `kubectl exec -n api deploy/api-svc -- curl -s localhost:8080/health | jq .in_flight`
+3. Drain one pod: `kubectl drain <node> --ignore-daemonsets --delete-emptydir-data`
+4. Wait for pod reschedule: `kubectl rollout status deployment/api-svc -n api --timeout=120s`
+5. Repeat for remaining pods, one at a time
+6. Run smoke test: `curl -s https://api.internal/healthz | jq .status`
+7. Page on-call if smoke test fails
+
+## Rollback
+If health check fails after 2 pods, halt and execute: `kubectl rollout undo deployment/api-svc -n api`
+
+## Evidence Required
+- Screenshot of `kubectl get pods` post-restart
+- Smoke test output showing `"status":"ok"`
+
+---
+*Draft prepared by Quill — verify commands against staging before production use.*