timmy-talking-turd/docs/RELEASE-OBSERVABILITY.md

Release observability and incident-response runbook

This runbook defines the privacy-safe release health surface for Timmy staging and release review. It is a reviewed operator document; it does not authorize changes to any live host. Everything here consumes sanitized release/health evidence only.

Privacy boundary (non-negotiable)

Telemetry in this system is release telemetry, not user telemetry. Evidence files, dashboard output, alerts, and drill transcripts must never contain:

• photos or medical imagery of any kind;
• stool records or journal entry content;
• session identifiers, cookies, tokens, credentials, or environment dumps;
• infrastructure addresses beyond the loopback 127.0.0.1 drill fixture;
• user identity of any kind (no emails, usernames, device IDs).

src/release-observability.js enforces this mechanically: sanitizeEvidence() allowlists bounded fields (release tag, full commit, UTC timestamp, per-check id/boundary/status/failure-class/latency/counters) and rejects the whole evidence object when anything else appears — forbidden keys (sessions, cookies, base64 payloads, image hashes, notes, free text), values of the wrong JSON type, numbers outside their bounded ranges, or a single malformed check among healthy ones. A hostile file is refused loudly; it is never silently censored into apparent health. Failure classes are a closed vocabulary (worker.unavailable, vision.timeout, model.error) matched against a strict dotted lowercase slug grammar; anything else — newlines, ANSI/control characters, secrets, medical text, unknown classes — fails the whole evidence file closed rather than being rendered. Counters are bounded non-negative integers capped at 1,000,000. Every one of the four boundaries (app, api, queue, model) must be present; partial telemetry is refused instead of rendered as silent health. If sanitization fails, the tool exits nonzero and echoes nothing from the input, and all CLI output is scrubbed of control characters so hostile file paths can never inject terminal escapes. The sanitizer itself is tested with hostile payloads (tests/release-observability.test.js) and the CLI is tested to never echo rejected content (tests/release-dashboard-cli.test.js).

Dashboard

Build a sanitized evidence file (schema below) and render:

node scripts/release_dashboard.mjs --evidence /secure/inbox/evidence.json

Output groups checks under four boundaries — app, api, queue, model — with pass/fail/degraded counts, max latency, bounded counters, failure-class totals, active alerts, and manual-fallback state. Exit code is 0 when no page-severity alert fires, 1 when one does, 2 on bad input. The CLI never contacts a live host; it reads local files or a loopback drill fixture only.

Manual fallback states (truthful by construction)

The manual-fallback line is a safety statement, so it only ever claims what the evidence proves:

Evidence state	Rendered line
every check explicitly pass	manual fallback: available — all checks explicitly passing; no degradation observed; fallback not required
any degradation away from the app boundary	manual fallback: available — degraded elsewhere; local journal remains usable
app boundary fail	manual fallback: unavailable — app boundary is down; local journal cannot be served
any check (including the app boundary itself) unknown	manual fallback: unknown — app state unknown; verify the local journal by hand before relying on it

Unknown telemetry is never rendered as proven availability or as "no degradation": an unknown app state forces the unknown wording and demands manual verification before anyone relies on the journal.

Evidence schema (v1)

{
  "schemaVersion": 1,
  "releaseTag": "string, 1-80 chars: letters, digits, dot, underscore, dash",
  "commit": "40 lowercase hex characters (12+ accepted)",
  "generatedAtUtc": "canonical real UTC, YYYY-MM-DDTHH:MM:SSZ (must round-trip as a real instant)",
  "checks": [
    {
      "id": "lowercase-dot-or-dash id",
      "boundary": "app | api | queue | model",
      "status": "pass | fail | degraded | unknown",
      "failureClass": "vocabulary class on fail/degraded only; omitted or null otherwise",
      "latencyMs": "optional integer 0..3600000; never a string, boolean, null, or fraction",
      "counters": { "ok|fail|abstain|retry|timeout|rejected|fallback|depth": "optional integer 0..1000000" }
    }
  ]
}

Every field above is type-checked against the exact JSON type: schemaVersion must be the number 1 (not "1"), counters and latency must be actual JSON numbers within their bounds, and timestamps must survive a UTC round-trip. A pass or unknown check must never carry a failureClass; a fail or degraded check may carry one of the closed-vocabulary classes or stay classless. Any other field — at the top level or inside a check — rejects the entire evidence object.

Alert inventory

Alerts are evaluated by evaluateAlerts() over the sanitized dashboard. Every rule carries an owner, a threshold, severity, and a runbook anchor in this file. Page-severity alerts make the CLI exit 1.

Alert id	Severity	Owner	Threshold (observed >=)	Condition	Runbook
worker.outage	page	release-operator	1	any check reports worker.unavailable, or both model and queue boundaries report failures	Simulate a worker outage
telemetry.gap	warn	release-operator	1	one or more checks report status unknown (missing telemetry fails closed as a warning, never as health)	Alert inventory
queue.backlog	warn	release-operator	10	queue depth at or above threshold (depth >= 10); suppressed while worker.outage pages (depth is residual from the same incident)	Alert inventory
vision.degraded	warn	vision-owner	3	three or more vision.timeout failure classes	Alert inventory
app.unhealthy	page	release-operator	1	app boundary healthz reports fail; this is also the only condition that makes the manual fallback unavailable (reason: app-down)	Alert inventory

Escalation: pages go to the release operator on call (currently Alexander as release owner); warns are batched into the next release-review pulse. An unresolved page after one manual fallback verification is raised in the Gitea issue for the affected release epic rather than paged repeatedly.

Simulate a worker outage

This drill proves the issue #41 acceptance criterion: one simulated outage produces exactly one actionable page alert while the graceful manual fallback stays available. It runs entirely against a throwaway loopback fixture — never against staging or production.

1. Start any loopback fixture that serves:
   • GET /api/drill/checks returning { "checks": [...] }, healthy first, with at least one check for each boundary (app, api, queue, model);
   • POST /drill/outage flipping the fixture's simulated worker off (subsequent /api/drill/checks responses then report status: "fail", failureClass: "worker.unavailable" across the api, queue, and model checks).
2. Run the drill:

node scripts/release_dashboard.mjs --drill-origin http://127.0.0.1:<port>

The origin must be a bare http://127.0.0.1:<port> URL. Credentials (user:pass@), DNS names, alternative IP encodings (hex octets, decimal integers, percent-encoding), IPv6 forms, non-http schemes, and any path or query are rejected before the CLI contacts anything.

3. Required outcome (the automated tests assert all of it):

   • the pre-drill baseline is read from the fixture itself and must be genuinely healthy — every required boundary check explicitly pass and zero alerts of any severity; unknown, degraded, malformed, missing-boundary, or backlog-warning baselines are refused with exit code 2 before anything is announced or flipped, and the switch is never touched;
   • after the flip, exactly one page alert fires — worker.outage; if the flip produced no real healthy-to-outage transition, the drill reports DRILL FAIL with exit code 1 instead of passing vacuously;
   • the alert names owner release-operator, its threshold, and this runbook section;
   • manual fallback: AVAILABLE — degraded elsewhere; local journal remains usable;
   • exit code 0 (DRILL PASS).

4. Manual fallback procedure if a real outage ever matches this signature: users keep logging locally (the browser journal never depends on the vision worker), operators capture one sanitized evidence file, run the dashboard once, record the single page alert, and follow rollback via the staging runbook. No user data leaves the device.

The drill is exercised end-to-end by tests/release-dashboard-cli.test.js ("outage drill flips one simulated switch…") using an in-test loopback fixture; CI runs it on every PR without touching any real host.

Incident flow (documented, local-first)

1. Detect — dashboard run or drill shows a page alert.
2. Triage — read the alert's failure class and boundary; open the linked runbook anchor above.
3. Contain — confirm the manual fallback line says available; if it does not, treat the app boundary as down too and escalate immediately.
4. Verify scope — re-run the dashboard on fresh sanitized evidence; never attach raw logs or user content to the incident.
5. Recover — use scripts/deploy_staging.py rollback --commit <known-good> inside an approved window (see docs/STAGING-RUNBOOK.md), or fix forward through a reviewed PR.
6. Review — file the sanitized dashboard output (text only) plus commit/tag identity as evidence in the release issue; delete stale evidence files from the inbox.

Receipts

• npm test — includes tests/release-observability.test.js, tests/release-dashboard.test.js, tests/release-dashboard-cli.test.js.
• Strict-type tests prove schemaVersion must be the JSON number 1, counters and latency must be actual bounded JSON numbers (never strings, booleans, nulls, fractions, or out-of-range values), and generatedAtUtc must be canonical real UTC that survives a round-trip.
• Fail-closed tests prove a single malformed check — bad id, missing boundary, unknown status, wrong-typed counters, or an off-vocabulary class on any status — rejects the whole evidence object instead of silently disappearing beside healthy checks.
• Consistency tests prove a pass or unknown check can never carry a failureClass, so all-pass evidence can never page worker.outage; sanitization is proven idempotent, so omitted latency stays omitted instead of becoming 0ms on a second pass.
• Sanitizer hostile-payload test proves session tokens, cookies, environment dumps, private keys, base64 payloads, image hashes, note text, and emails reject the entire file rather than surviving into dashboard output.
• Hostile failure-class test proves newlines, carriage returns, ANSI escapes, control characters, secrets, medical text, SQL, oversized values, wrong types, and off-vocabulary classes all fail closed.
• Depth-counter tests prove the queue.backlog warn stays silent at 9, fires at exactly 10, reports observed counts above threshold, sums across queue checks, is suppressed while worker.outage pages, and cannot be smuggled in above the bounded counter ceiling.
• Boundary-completeness tests prove evidence missing any of app, api, queue, or model fails closed; unknown statuses surface as a telemetry.gap warn instead of passing as healthy.
• Manual-fallback tests prove a truthful deterministic contract: available with no degradation, active (local-journal) under any non-fatal degradation, unavailable only when the app boundary itself is down, and unknown (app-state-unknown, availability null) whenever telemetry is unknown — never claimed as healthy from unknown state.
• CLI tests prove rejected evidence exits nonzero without echoing contents, hostile paths cannot forge terminal lines with LF/CR (only program-authored line breaks reach stdout), drill origins are validated to bare loopback http before any network contact (credentials, DNS names, hex/decimal/percent-encoded IP encodings, IPv6, paths, queries, and foreign schemes are refused), the outage drill requires a genuine baseline — every required check explicitly passing and zero alerts — before announcing anything or touching its switch, refuses already-down fixtures, fails when no real transition occurs, and yields exactly one actionable alert with graceful manual fallback.