troupe-standup/wizard-work/quill/draft-incident-response-runbook-v0.2.md

Draft Incident Response Runbook v0.2

Purpose

Fast, repeatable incident response for the sovereign AI stack. This runbook covers detection, triage, mitigation, and post-incident review.

Detection Signals

1. Hermes Agent health-check endpoint returns non-200 or times out (>30 s).
2. cronjob action='list' shows missed executions for any job tagged critical.
3. Disk usage on /root exceeds 90 % (df -h /root).
4. Outbound API error rate for the nous provider exceeds 5 % over a 5-minute window.

Triage Steps

1. Run hermes-agent health (or the equivalent stack health probe) to scope blast radius.
2. Identify the failing component: provider, cron, disk, or network.
3. Check the last 200 lines of the Hermes runtime log: journalctl -u hermes -n 200 or tail -n 200 ~/.hermes/logs/runtime.log.
4. Record the timeline in a scratch note under ~/.hermes/profiles/quill/cache/scratch/.

Mitigation Playbook

Provider outage (nous / inference backend)

• Switch to fallback provider in config.yaml if one is configured.
• Re-queue failed jobs: hermes-agent retry --since <ISO8601>.
• Notify operator via the configured messaging channel.

Cron job missed / stuck

• Inspect: cronjob action='list'.
• Stop runaway job: cronjob action='stop' --id <job_id>.
• Re-run manually with backfill: cronjob action='run' --id <job_id> --backfill.

Disk full

• Identify large files: du -ah /root/.hermes | sort -rh | head -n 20.
• Purge cache older than 24 h: find /root/.hermes/cache -type f -mmin +1440 -delete.
• If still critical, rotate logs: logrotate -f /etc/logrotate.conf.

Network / DNS

• Test resolver: dig @1.1.1.1 api.nousresearch.com +short.
• Flush DNS cache if applicable: resolvectl flush-caches.
• Switch to fallback DNS in /etc/resolv.conf temporarily.

Escalation

• Page operator if the incident is P1 (customer-facing or data-loss risk).
• Use browser_exec to send a desktop notification if the messaging channel is down.
• Create an incident ticket immediately with timeline, impact, and mitigation steps.

Post-Incident Review

1. Write a postmortem within 24 h.
2. Answer: What failed, why, how we knew, how we fixed it, how we prevent recurrence.
3. Update this runbook with any new signals or playbook steps discovered.

Verification

After mitigation, run these commands to confirm recovery:

• hermes-agent health exits 0.
• df -h /root shows < 80 % usage.
• cronjob action='list' shows no jobs in a failed state.