# Release observability and incident-response runbook

This runbook defines the privacy-safe release health surface for Timmy staging and release review. It is a reviewed operator document; it does not authorize changes to any live host. Everything here consumes sanitized release/health evidence only.

## Privacy boundary (non-negotiable)

Telemetry in this system is release telemetry, not user telemetry. Evidence files, dashboard output, alerts, and drill transcripts must never contain:

- photos or medical imagery of any kind;
- stool records or journal entry content;
- session identifiers, cookies, tokens, credentials, or environment dumps;
- infrastructure addresses beyond the loopback `127.0.0.1` drill fixture;
- user identity of any kind (no emails, usernames, device IDs).

`src/release-observability.js` enforces this mechanically: `sanitizeEvidence()` allowlists bounded fields (release tag, full commit, UTC timestamp, per-check id/boundary/status/failure-class/latency/counters) and drops anything else — including any check whose extra fields look like keys, sessions, base64 payloads, image hashes, notes, or free text. Failure classes are a closed vocabulary (`worker.unavailable`, `vision.timeout`, `model.error`) matched against a strict dotted lowercase slug grammar; anything else — newlines, ANSI/control characters, secrets, medical text, unknown classes — fails the whole evidence file closed rather than being rendered. Counters are bounded non-negative integers capped at 1,000,000. Every one of the four boundaries (app, api, queue, model) must be present; partial telemetry is refused instead of rendered as silent health. If sanitization fails, the tool exits nonzero and echoes nothing from the input, and all CLI output is scrubbed of control characters so hostile file paths can never inject terminal escapes. The sanitizer itself is tested with hostile payloads (`tests/release-observability.test.js`) and the CLI is tested to never echo rejected content (`tests/release-dashboard-cli.test.js`).

## Dashboard

Build a sanitized evidence file (schema below) and render:

```bash
node scripts/release_dashboard.mjs --evidence /secure/inbox/evidence.json
```

Output groups checks under four boundaries — app, api, queue, model — with pass/fail/degraded counts, max latency, bounded counters, failure-class totals, active alerts, and manual-fallback state. Exit code is `0` when no page-severity alert fires, `1` when one does, `2` on bad input. The CLI never contacts a live host; it reads local files or a loopback drill fixture only.

### Evidence schema (v1)

```json
{
  "schemaVersion": 1,
  "releaseTag": "daily-2026-08-22.1",
  "commit": "40 lowercase hex characters (12+ accepted)",
  "generatedAtUtc": "YYYY-MM-DDTHH:MM:SSZ",
  "checks": [
    {
      "id": "lowercase-dot-or-dash id",
      "boundary": "app | api | queue | model",
      "status": "pass | fail | degraded | unknown",
      "failureClass": "dotted.class.name or null",
      "latencyMs": integer milliseconds or omitted,
      "counters": { "ok|fail|abstain|retry|timeout|rejected|fallback|depth": non-negative integer }
    }
  ]
}
```

Any other field is dropped, and checks carrying forbidden-shaped fields are discarded individually. Counters are capped to the fixed vocabulary above so free text can never smuggle in through a label. Issue #19 owns producing these signals from the worker; until that lands, evidence can be hand-authored for drills.

## Alert inventory

Alerts are evaluated by `evaluateAlerts()` over the sanitized dashboard. Every rule carries an owner, a threshold, severity, and a runbook anchor in this file. Page-severity alerts make the CLI exit `1`.

| Alert id        | Severity | Owner            | Threshold (observed >=) | Condition                                        | Runbook |
|-----------------|----------|------------------|-------------------------|--------------------------------------------------|---------|
| `worker.outage` | page     | release-operator | 1                       | any check reports `worker.unavailable`, or both model and queue boundaries report failures | [Simulate a worker outage](#simulate-a-worker-outage) |
| `telemetry.gap` | warn     | release-operator | 1                       | one or more checks report status `unknown` (missing telemetry fails closed as a warning, never as health) | [Alert inventory](#alert-inventory) |
| `queue.backlog` | warn     | release-operator | depth >= 10             | queue depth counter at or above 10; suppressed while `worker.outage` pages (depth is residual from the same incident) | [Alert inventory](#alert-inventory) |
| `vision.degraded` | warn   | vision-owner     | 3                       | three or more `vision.timeout` failure classes   | [Alert inventory](#alert-inventory) |
| `app.unhealthy` | page     | release-operator | 1                       | app boundary healthz reports fail; this is also the only condition that makes the manual fallback unavailable (`reason: app-down`) | [Alert inventory](#alert-inventory) |

Escalation: pages go to the release operator on call (currently Alexander as release owner); warns are batched into the next release-review pulse. An unresolved page after one manual fallback verification is raised in the Gitea issue for the affected release epic rather than paged repeatedly.

## Simulate a worker outage

This drill proves the issue #41 acceptance criterion: one simulated outage produces exactly one actionable page alert while the graceful manual fallback stays available. It runs entirely against a throwaway loopback fixture — never against staging or production.

1. Start any loopback fixture that serves:
   - `GET /api/drill/checks` returning `{ "checks": [...] }`, healthy first, with at least one check for each boundary (app, api, queue, model);
   - `POST /drill/outage` flipping the fixture's simulated worker off (subsequent `/api/drill/checks` responses then report `status: "fail"`, `failureClass: "worker.unavailable"` across the api, queue, and model checks).
2. Run the drill:

```bash
node scripts/release_dashboard.mjs --drill-origin http://127.0.0.1:<port>
```

   The origin must be a bare `http://127.0.0.1:<port>` URL. Credentials (`user:pass@`), DNS names, alternative IP encodings (hex octets, decimal integers, percent-encoding), IPv6 forms, non-http schemes, and any path or query are rejected before the CLI contacts anything.

3. Required outcome (the automated tests assert all of it):
   - the pre-drill baseline is read from the fixture itself and must be genuinely healthy — an already-outaged fixture is refused with exit code 2 and its switch is never touched;
   - after the flip, exactly one page alert fires — `worker.outage`; if the flip produced no real healthy-to-outage transition, the drill reports `DRILL FAIL` with exit code 1 instead of passing vacuously;
   - the alert names owner `release-operator`, its threshold, and this runbook section;
   - `manual fallback: available — local journal remains usable`;
   - exit code `0` (`DRILL PASS`).

4. Manual fallback procedure if a real outage ever matches this signature: users keep logging locally (the browser journal never depends on the vision worker), operators capture one sanitized evidence file, run the dashboard once, record the single page alert, and follow rollback via the staging runbook. No user data leaves the device.

The drill is exercised end-to-end by `tests/release-dashboard-cli.test.js` ("outage drill flips one simulated switch…") using an in-test loopback fixture; CI runs it on every PR without touching any real host.

## Incident flow (documented, local-first)

1. **Detect** — dashboard run or drill shows a page alert.
2. **Triage** — read the alert's failure class and boundary; open the linked runbook anchor above.
3. **Contain** — confirm the manual fallback line says `available`; if it does not, treat the app boundary as down too and escalate immediately.
4. **Verify scope** — re-run the dashboard on fresh sanitized evidence; never attach raw logs or user content to the incident.
5. **Recover** — use `scripts/deploy_staging.py rollback --commit <known-good>` inside an approved window (see `docs/STAGING-RUNBOOK.md`), or fix forward through a reviewed PR.
6. **Review** — file the sanitized dashboard output (text only) plus commit/tag identity as evidence in the release issue; delete stale evidence files from the inbox.

## Receipts

- `npm test` — includes `tests/release-observability.test.js`, `tests/release-dashboard.test.js`, `tests/release-dashboard-cli.test.js`.
- Sanitizer hostile-payload test proves session tokens, cookies, environment dumps, private keys, base64 payloads, image hashes, note text, and emails cannot survive into dashboard output.
- Hostile failure-class test proves newlines, carriage returns, ANSI escapes, control characters, secrets, medical text, SQL, oversized values, wrong types, and off-vocabulary classes all fail closed.
- Depth-counter tests prove the `queue.backlog` warn stays silent at 9, fires at exactly 10, reports observed counts above threshold, sums across queue checks, is suppressed while `worker.outage` pages, and cannot be smuggled in above the bounded counter ceiling.
- Boundary-completeness tests prove evidence missing any of app, api, queue, or model fails closed; unknown statuses surface as a `telemetry.gap` warn instead of passing as healthy.
- Manual-fallback tests prove a deterministic contract: journal available with no degradation, active (`local-journal`) under any non-fatal degradation, and unavailable only when the app boundary itself is down.
- CLI tests prove rejected evidence exits nonzero without echoing contents, drill origins are validated to bare loopback http before any network contact (credentials, DNS names, hex/decimal/percent-encoded IP encodings, IPv6, paths, queries, and foreign schemes are refused), the outage drill reads a genuine fixture baseline, refuses already-down fixtures without touching their switch, fails when no real transition occurs, and yields exactly one actionable alert with graceful manual fallback.