quill: draft incident response runbook v0.2 with verifier

wizard-work/quill/draft-incident-response-runbook-v0.2.md

@@ -0,0 +1,55 @@
+# Draft Incident Response Runbook v0.2
+
+## Purpose
+Fast, repeatable incident response for the sovereign AI stack. This runbook
+covers detection, triage, mitigation, and post-incident review.
+
+## Detection Signals
+1. Hermes Agent health-check endpoint returns non-200 or times out (>30 s).
+2. `cronjob` action='list' shows missed executions for any job tagged `critical`.
+3. Disk usage on `/root` exceeds 90 % (`df -h /root`).
+4. Outbound API error rate for the `nous` provider exceeds 5 % over a 5-minute window.
+
+## Triage Steps
+1. Run `hermes-agent health` (or the equivalent stack health probe) to scope blast radius.
+2. Identify the failing component: provider, cron, disk, or network.
+3. Check the last 200 lines of the Hermes runtime log: `journalctl -u hermes -n 200` or `tail -n 200 ~/.hermes/logs/runtime.log`.
+4. Record the timeline in a scratch note under `~/.hermes/profiles/quill/cache/scratch/`.
+
+## Mitigation Playbook
+
+### Provider outage (nous / inference backend)
+- Switch to fallback provider in `config.yaml` if one is configured.
+- Re-queue failed jobs: `hermes-agent retry --since <ISO8601>`.
+- Notify operator via the configured messaging channel.
+
+### Cron job missed / stuck
+- Inspect: `cronjob action='list'`.
+- Stop runaway job: `cronjob action='stop' --id <job_id>`.
+- Re-run manually with backfill: `cronjob action='run' --id <job_id> --backfill`.
+
+### Disk full
+- Identify large files: `du -ah /root/.hermes | sort -rh | head -n 20`.
+- Purge cache older than 24 h: `find /root/.hermes/cache -type f -mmin +1440 -delete`.
+- If still critical, rotate logs: `logrotate -f /etc/logrotate.conf`.
+
+### Network / DNS
+- Test resolver: `dig @1.1.1.1 api.nousresearch.com +short`.
+- Flush DNS cache if applicable: `resolvectl flush-caches`.
+- Switch to fallback DNS in `/etc/resolv.conf` temporarily.
+
+## Escalation
+- Page operator if the incident is P1 (customer-facing or data-loss risk).
+- Use `browser_exec` to send a desktop notification if the messaging channel is down.
+- Create an incident ticket immediately with timeline, impact, and mitigation steps.
+
+## Post-Incident Review
+1. Write a postmortem within 24 h.
+2. Answer: What failed, why, how we knew, how we fixed it, how we prevent recurrence.
+3. Update this runbook with any new signals or playbook steps discovered.
+
+## Verification
+After mitigation, run these commands to confirm recovery:
+- `hermes-agent health` exits 0.
+- `df -h /root` shows < 80 % usage.
+- `cronjob action='list'` shows no jobs in a failed state.

wizard-work/quill/verify-runbook.py

@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+"""Verify draft-incident-response-runbook-v0.2.md meets minimum standards."""
+
+import re
+import sys
+from pathlib import Path
+
+path = Path("/root/.hermes/wizard-outputs/quill/draft-incident-response-runbook-v0.2.md")
+text = path.read_text()
+
+checks = []
+
+# 1) Title
+checks.append(("title", bool(re.search(r"^#\s+.+", text, re.MULTILINE))))
+
+# 2) At least 5 ## sections
+section_count = len(re.findall(r"^##\s+.+", text, re.MULTILINE))
+checks.append(("five_sections", section_count >= 5))
+
+# 3) At least 3 shell commands (lines containing `$` or ```bash or ```sh)
+shell_lines = [
+    line for line in text.splitlines()
+    if line.strip().startswith("`") and (
+        "`" in line[1:] or "$" in line
+    ) and any(c in line for c in ["$", "```"])
+]
+# More reliable: count lines that contain backtick-wrapped commands
+commands = re.findall(r"`([^`]+)`", text)
+checks.append(("three_commands", len(commands) >= 3))
+
+# 4) Verification section present
+checks.append(("verification_section", "## Verification" in text))
+
+failed = [name for name, ok in checks if not ok]
+print(f"Checks passed: {len(checks) - len(failed)}/{len(checks)}")
+for name, ok in checks:
+    print(f"  {name}: {'PASS' if ok else 'FAIL'}")
+
+if failed:
+    print(f"FAILED checks: {', '.join(failed)}")
+    sys.exit(1)
+else:
+    print("All checks passed.")
+    sys.exit(0)