troupe-standup/wizard-work/quill/runbook-draft.md

Service Restart Runbook (Draft v0.1)

Purpose

Provide a repeatable, low-risk procedure for restarting the API service without dropping in-flight requests.

Scope

Applies to api-svc (primary and canary). Excludes database migrations.

Prerequisites

• Access to the ops bastion
• PagerDuty duty-handoff confirmed
• Deployment queue is clear (check via kubectl get events -n deploy)

Steps

1. Verify current health: kubectl get pods -n api -l app=api-svc
2. Confirm no in-flight requests: kubectl exec -n api deploy/api-svc -- curl -s localhost:8080/health | jq .in_flight
3. Drain one pod: kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
4. Wait for pod reschedule: kubectl rollout status deployment/api-svc -n api --timeout=120s
5. Repeat for remaining pods, one at a time
6. Run smoke test: curl -s https://api.internal/healthz | jq .status
7. Page on-call if smoke test fails

Rollback

If health check fails after 2 pods, halt and execute: kubectl rollout undo deployment/api-svc -n api

Evidence Required

• Screenshot of kubectl get pods post-restart
• Smoke test output showing "status":"ok"

---

Draft prepared by Quill — verify commands against staging before production use.