# Draft Incident Response Runbook v0.2

## Purpose
Fast, repeatable incident response for the sovereign AI stack. This runbook
covers detection, triage, mitigation, and post-incident review.

## Detection Signals
1. Hermes Agent health-check endpoint returns non-200 or times out (>30 s).
2. `cronjob` action='list' shows missed executions for any job tagged `critical`.
3. Disk usage on `/root` exceeds 90 % (`df -h /root`).
4. Outbound API error rate for the `nous` provider exceeds 5 % over a 5-minute window.

## Triage Steps
1. Run `hermes-agent health` (or the equivalent stack health probe) to scope blast radius.
2. Identify the failing component: provider, cron, disk, or network.
3. Check the last 200 lines of the Hermes runtime log: `journalctl -u hermes -n 200` or `tail -n 200 ~/.hermes/logs/runtime.log`.
4. Record the timeline in a scratch note under `~/.hermes/profiles/quill/cache/scratch/`.

## Mitigation Playbook

### Provider outage (nous / inference backend)
- Switch to fallback provider in `config.yaml` if one is configured.
- Re-queue failed jobs: `hermes-agent retry --since <ISO8601>`.
- Notify operator via the configured messaging channel.

### Cron job missed / stuck
- Inspect: `cronjob action='list'`.
- Stop runaway job: `cronjob action='stop' --id <job_id>`.
- Re-run manually with backfill: `cronjob action='run' --id <job_id> --backfill`.

### Disk full
- Identify large files: `du -ah /root/.hermes | sort -rh | head -n 20`.
- Purge cache older than 24 h: `find /root/.hermes/cache -type f -mmin +1440 -delete`.
- If still critical, rotate logs: `logrotate -f /etc/logrotate.conf`.

### Network / DNS
- Test resolver: `dig @1.1.1.1 api.nousresearch.com +short`.
- Flush DNS cache if applicable: `resolvectl flush-caches`.
- Switch to fallback DNS in `/etc/resolv.conf` temporarily.

## Escalation
- Page operator if the incident is P1 (customer-facing or data-loss risk).
- Use `browser_exec` to send a desktop notification if the messaging channel is down.
- Create an incident ticket immediately with timeline, impact, and mitigation steps.

## Post-Incident Review
1. Write a postmortem within 24 h.
2. Answer: What failed, why, how we knew, how we fixed it, how we prevent recurrence.
3. Update this runbook with any new signals or playbook steps discovered.

## Verification
After mitigation, run these commands to confirm recovery:
- `hermes-agent health` exits 0.
- `df -h /root` shows < 80 % usage.
- `cronjob action='list'` shows no jobs in a failed state.