timmy-talking-turd/docs/RELEASE-OBSERVABILITY.md

Release observability and incident-response runbook

This runbook defines the privacy-safe release health surface for Timmy staging and release review. It is a reviewed operator document; it does not authorize changes to any live host. Everything here consumes sanitized release/health evidence only.

Privacy boundary (non-negotiable)

Telemetry in this system is release telemetry, not user telemetry. Evidence files, dashboard output, alerts, and drill transcripts must never contain:

• photos or medical imagery of any kind;
• stool records or journal entry content;
• session identifiers, cookies, tokens, credentials, or environment dumps;
• infrastructure addresses beyond the loopback 127.0.0.1 drill fixture;
• user identity of any kind (no emails, usernames, device IDs).

src/release-observability.js enforces this mechanically: sanitizeEvidence() allowlists bounded fields (release tag, full commit, UTC timestamp, per-check id/boundary/status/failure-class/latency/counters) and drops anything else — including any check whose extra fields look like keys, sessions, base64 payloads, image hashes, notes, or free text. If sanitization fails, the tool exits nonzero and echoes nothing from the input. The sanitizer itself is tested with hostile payloads (tests/release-observability.test.js) and the CLI is tested to never echo rejected content (tests/release-dashboard-cli.test.js).

Dashboard

Build a sanitized evidence file (schema below) and render:

node scripts/release_dashboard.mjs --evidence /secure/inbox/evidence.json

Output groups checks under four boundaries — app, api, queue, model — with pass/fail/degraded counts, max latency, bounded counters, failure-class totals, active alerts, and manual-fallback state. Exit code is 0 when no page-severity alert fires, 1 when one does, 2 on bad input. The CLI never contacts a live host; it reads local files or a loopback drill fixture only.

Evidence schema (v1)

{
  "schemaVersion": 1,
  "releaseTag": "daily-2026-08-22.1",
  "commit": "40 lowercase hex characters (12+ accepted)",
  "generatedAtUtc": "YYYY-MM-DDTHH:MM:SSZ",
  "checks": [
    {
      "id": "lowercase-dot-or-dash id",
      "boundary": "app | api | queue | model",
      "status": "pass | fail | degraded | unknown",
      "failureClass": "dotted.class.name or null",
      "latencyMs": integer milliseconds or omitted,
      "counters": { "ok|fail|abstain|retry|timeout|rejected|fallback|depth": non-negative integer }
    }
  ]
}

Any other field is dropped, and checks carrying forbidden-shaped fields are discarded individually. Counters are capped to the fixed vocabulary above so free text can never smuggle in through a label. Issue #19 owns producing these signals from the worker; until that lands, evidence can be hand-authored for drills.

Alert inventory

Alerts are evaluated by evaluateAlerts() over the sanitized dashboard. Every rule carries an owner, a threshold, severity, and a runbook anchor in this file. Page-severity alerts make the CLI exit 1.

Alert id	Severity	Owner	Threshold (observed >=)	Condition	Runbook
worker.outage	page	release-operator	1	any check reports worker.unavailable, or both model and queue boundaries report failures	Simulate a worker outage
queue.backlog	warn	release-operator	depth >= 10	queue depth counter at or above 10	Alert inventory
vision.degraded	warn	vision-owner	3	three or more vision.timeout failure classes	Alert inventory
app.unhealthy	page	release-operator	1	app boundary healthz reports fail	Alert inventory

Escalation: pages go to the release operator on call (currently Alexander as release owner); warns are batched into the next release-review pulse. An unresolved page after one manual fallback verification is raised in the Gitea issue for the affected release epic rather than paged repeatedly.

Simulate a worker outage

This drill proves the issue #41 acceptance criterion: one simulated outage produces exactly one actionable page alert while the graceful manual fallback stays available. It runs entirely against a throwaway loopback fixture — never against staging or production.

1. Start any loopback fixture that serves:
   • GET /api/drill/checks returning { "checks": [...] } (healthy first);
   • POST /drill/outage flipping the fixture's simulated worker off (subsequent /api/drill/checks responses then report status: "fail", failureClass: "worker.unavailable" across the api, queue, and model checks).
2. Run the drill:

node scripts/release_dashboard.mjs --drill-origin http://127.0.0.1:<port>

3. Required outcome (the automated test asserts all of it):

   • pre-drill baseline renders zero alerts;
   • after the flip, exactly one page alert fires — worker.outage;
   • the alert names owner release-operator, its threshold, and this runbook section;
   • manual fallback: available — local journal remains usable;
   • exit code 0 (DRILL PASS).

4. Manual fallback procedure if a real outage ever matches this signature: users keep logging locally (the browser journal never depends on the vision worker), operators capture one sanitized evidence file, run the dashboard once, record the single page alert, and follow rollback via the staging runbook. No user data leaves the device.

The drill is exercised end-to-end by tests/release-dashboard-cli.test.js ("outage drill flips one simulated switch…") using an in-test loopback fixture; CI runs it on every PR without touching any real host.

Incident flow (documented, local-first)

1. Detect — dashboard run or drill shows a page alert.
2. Triage — read the alert's failure class and boundary; open the linked runbook anchor above.
3. Contain — confirm the manual fallback line says available; if it does not, treat the app boundary as down too and escalate immediately.
4. Verify scope — re-run the dashboard on fresh sanitized evidence; never attach raw logs or user content to the incident.
5. Recover — use scripts/deploy_staging.py rollback --commit <known-good> inside an approved window (see docs/STAGING-RUNBOOK.md), or fix forward through a reviewed PR.
6. Review — file the sanitized dashboard output (text only) plus commit/tag identity as evidence in the release issue; delete stale evidence files from the inbox.

Receipts

• npm test — includes tests/release-observability.test.js, tests/release-dashboard.test.js, tests/release-dashboard-cli.test.js.
• Sanitizer hostile-payload test proves session tokens, cookies, environment dumps, private keys, base64 payloads, image hashes, note text, and emails cannot survive into dashboard output.
• CLI tests prove rejected evidence exits nonzero without echoing contents, and the outage drill yields exactly one actionable alert with graceful manual fallback.