33 lines
1.2 KiB
Markdown
33 lines
1.2 KiB
Markdown
# Service Restart Runbook (Draft v0.1)
|
|
|
|
## Purpose
|
|
Provide a repeatable, low-risk procedure for restarting the API service without dropping in-flight requests.
|
|
|
|
## Scope
|
|
Applies to api-svc (primary and canary). Excludes database migrations.
|
|
|
|
## Prerequisites
|
|
- Access to the ops bastion
|
|
- PagerDuty duty-handoff confirmed
|
|
- Deployment queue is clear (check via `kubectl get events -n deploy`)
|
|
|
|
## Steps
|
|
|
|
1. Verify current health: `kubectl get pods -n api -l app=api-svc`
|
|
2. Confirm no in-flight requests: `kubectl exec -n api deploy/api-svc -- curl -s localhost:8080/health | jq .in_flight`
|
|
3. Drain one pod: `kubectl drain <node> --ignore-daemonsets --delete-emptydir-data`
|
|
4. Wait for pod reschedule: `kubectl rollout status deployment/api-svc -n api --timeout=120s`
|
|
5. Repeat for remaining pods, one at a time
|
|
6. Run smoke test: `curl -s https://api.internal/healthz | jq .status`
|
|
7. Page on-call if smoke test fails
|
|
|
|
## Rollback
|
|
If health check fails after 2 pods, halt and execute: `kubectl rollout undo deployment/api-svc -n api`
|
|
|
|
## Evidence Required
|
|
- Screenshot of `kubectl get pods` post-restart
|
|
- Smoke test output showing `"status":"ok"`
|
|
|
|
---
|
|
*Draft prepared by Quill — verify commands against staging before production use.*
|