Release observability dashboard and incident-response flow (#41) #60

Open

rockachopa wants to merge 3 commits from timmy/41-release-observability-dashboard into main

Author	SHA1	Message	Date
Timmy	d547b4887b	fix: close hostile-review blockers in release observability All checks were successful Quality gates / quality (pull_request) Successful in 2m9s Details Strict vertical RED-GREEN TDD across eight review blockers: - drill baseline: requires every required boundary check explicitly passing plus zero alerts of any severity before anything is announced or flipped; unknown, degraded, malformed, missing-boundary, and backlog-warning baselines exit 2 with switch count zero, and stdout stays empty whenever no drill actually ran - manual fallback: deterministic truthful states — app-down, local-journal, none-required, and new app-state-unknown (availability null) so unknown telemetry can never render as proven availability or 'no degradation'; runbook gains a state table with exact wording - exact schema types: schemaVersion must be JSON number 1; counters and latency must be finite bounded integers (0..1e6, 0..3600000); generatedAtUtc must be canonical real UTC surviving round-trip; releaseTag/commit keep exact safe types; coercions reject - control-safety: controlSafe strips LF/CR too, so hostile paths can no longer forge terminal lines (regression test ships a path embedding LF + forged PAGE text); read errors use fixed wording instead of echoing attacker-derived message bytes - malformed checks reject the whole evidence object instead of silently disappearing beside healthy checks; forbidden top-level keys now fail closed rather than being ignored - status/failureClass consistency: pass/unknown checks carrying any failureClass reject the file, so all-pass evidence can never page worker.outage (fail/degraded may stay classless, preserving the closed vocabulary and nullable schema) - sanitization idempotence: omitted latency stays omitted on a second pass; explicit null latency rejects since the schema forbids it - removed the always-false queue-depth tautology in the shared fixture and made the runbook contract test verify each rule's real numeric threshold against DEFAULT_ALERT_RULES Preserved: closed failure-class vocabulary, queue depth 9/10/11 edges, raw loopback-origin validation, app-down fallback, missing-boundary rejection, transition semantics, and every independent drill case.	2026-08-22 23:40:23 +00:00
Timmy	9c9286b59f	fix(ops): harden release observability against hostile review findings All checks were successful Quality gates / quality (pull_request) Successful in 1m53s Details Strict RED-GREEN TDD over PR review blockers; every fix landed test-first with the failing run observed before implementation. - failureClass: closed privacy-safe vocabulary (worker.unavailable, vision.timeout, model.error) under strict dotted slug grammar. Newlines, carriage returns, ANSI/control characters, secrets, medical text, SQL, oversized values, and off-vocabulary classes fail the entire evidence file closed; nothing hostile can reach rendering. - depth: added to the counter vocabulary so queue.backlog can fire at all; bounded counters (0..1,000,000, integer) fail closed above the ceiling. 9/10/11 edge tests pin silent/at-threshold/above-threshold; backlog is suppressed while worker.outage pages (depth is residual from the same incident) and returns the moment the page clears. - drill integrity: the pre-drill baseline is now read from the fixture itself and must be genuinely healthy; already-outaged fixtures are refused with exit 2 without touching their switch, and a flip that produces no real healthy-to-outage transition reports DRILL FAIL instead of passing vacuously. - manual fallback: deterministic contract replaces the tautology. available+none-required when healthy, available+local-journal under any degradation, unavailable+app-down only when the app boundary itself is down. - fail-closed telemetry: evidence missing any of the four boundaries is rejected; unknown statuses surface as a warn telemetry.gap alert with owner/threshold/runbook instead of passing as healthy (documented in the runbook inventory). - drill origin: validateLoopbackOrigin gates every network path before any fetch. Only a bare http://127.0.0.1:<port> URL passes; credentials, DNS names, hex/decimal/percent-encoded IP encodings, IPv6 forms, paths, queries, fragments, and non-http schemes are refused pre-contact (raw-string grammar gate plus parse round-trip, because the URL parser canonicalizes hostile encodings). - terminal safety: controlSafe() strips C0/C1 control characters from all dynamically produced CLI output so hostile evidence paths cannot inject ANSI escapes into a terminal. Gates: npm test 98/98, check:syntax, npm audit (0 vulns), check:diff, deploy_staging status read-only; 30 adversarial probes against sanitizer, alert edges, and live loopback CLI all pass. No merge, no deploy.	2026-08-22 21:43:18 +00:00
Timmy	ad8c8a9f4c	feat(ops): release observability dashboard and incident-response flow All checks were successful Quality gates / quality (pull_request) Successful in 1m44s Details Implements issue #41 acceptance criteria without touching the live host: - src/release-observability.js: sanitizeEvidence() allowlists bounded, privacy-safe evidence (release tag, commit, UTC time, per-check id/boundary/status/failure-class/latency/counters) and drops session identifiers, credentials, environment dumps, photo payloads, base64, image hashes, note text, emails, and any oversized/suspicious value. - buildDashboard(): app/api/queue/model boundary rollups, failure-class counts, manual-fallback state; evaluateAlerts() with owner, threshold, severity, and runbook anchor per rule. - scripts/release_dashboard.mjs: local operator CLI over sanitized evidence files or a loopback drill fixture; never contacts a live host and exits nonzero without echoing rejected input. - docs/RELEASE-OBSERVABILITY.md: alert inventory, evidence schema, simulated worker-outage drill, incident flow, privacy boundary. - Tests: sanitizer hostile-payload coverage, dashboard/alert rules, end-to-end loopback outage drill asserting exactly one actionable page alert plus graceful manual fallback, runbook/package contract.	2026-08-22 20:33:10 +00:00

Author

SHA1

Message

Date

Timmy

d547b4887b

fix: close hostile-review blockers in release observability

Quality gates / quality (pull_request) Successful in 2m9s

Details

Strict vertical RED-GREEN TDD across eight review blockers:

- drill baseline: requires every required boundary check explicitly
  passing plus zero alerts of any severity before anything is announced
  or flipped; unknown, degraded, malformed, missing-boundary, and
  backlog-warning baselines exit 2 with switch count zero, and stdout
  stays empty whenever no drill actually ran
- manual fallback: deterministic truthful states — app-down,
  local-journal, none-required, and new app-state-unknown (availability
  null) so unknown telemetry can never render as proven availability or
  'no degradation'; runbook gains a state table with exact wording
- exact schema types: schemaVersion must be JSON number 1; counters and
  latency must be finite bounded integers (0..1e6, 0..3600000);
  generatedAtUtc must be canonical real UTC surviving round-trip;
  releaseTag/commit keep exact safe types; coercions reject
- control-safety: controlSafe strips LF/CR too, so hostile paths can no
  longer forge terminal lines (regression test ships a path embedding
  LF + forged PAGE text); read errors use fixed wording instead of
  echoing attacker-derived message bytes
- malformed checks reject the whole evidence object instead of silently
  disappearing beside healthy checks; forbidden top-level keys now fail
  closed rather than being ignored
- status/failureClass consistency: pass/unknown checks carrying any
  failureClass reject the file, so all-pass evidence can never page
  worker.outage (fail/degraded may stay classless, preserving the
  closed vocabulary and nullable schema)
- sanitization idempotence: omitted latency stays omitted on a second
  pass; explicit null latency rejects since the schema forbids it
- removed the always-false queue-depth tautology in the shared fixture
  and made the runbook contract test verify each rule's real numeric
  threshold against DEFAULT_ALERT_RULES

Preserved: closed failure-class vocabulary, queue depth 9/10/11 edges,
raw loopback-origin validation, app-down fallback, missing-boundary
rejection, transition semantics, and every independent drill case.

2026-08-22 23:40:23 +00:00

Timmy

9c9286b59f

fix(ops): harden release observability against hostile review findings

Quality gates / quality (pull_request) Successful in 1m53s

Details

Strict RED-GREEN TDD over PR review blockers; every fix landed test-first
with the failing run observed before implementation.

- failureClass: closed privacy-safe vocabulary (worker.unavailable,
  vision.timeout, model.error) under strict dotted slug grammar. Newlines,
  carriage returns, ANSI/control characters, secrets, medical text, SQL,
  oversized values, and off-vocabulary classes fail the entire evidence
  file closed; nothing hostile can reach rendering.
- depth: added to the counter vocabulary so queue.backlog can fire at all;
  bounded counters (0..1,000,000, integer) fail closed above the ceiling.
  9/10/11 edge tests pin silent/at-threshold/above-threshold; backlog is
  suppressed while worker.outage pages (depth is residual from the same
  incident) and returns the moment the page clears.
- drill integrity: the pre-drill baseline is now read from the fixture
  itself and must be genuinely healthy; already-outaged fixtures are
  refused with exit 2 without touching their switch, and a flip that
  produces no real healthy-to-outage transition reports DRILL FAIL
  instead of passing vacuously.
- manual fallback: deterministic contract replaces the tautology.
  available+none-required when healthy, available+local-journal under any
  degradation, unavailable+app-down only when the app boundary itself is
  down.
- fail-closed telemetry: evidence missing any of the four boundaries is
  rejected; unknown statuses surface as a warn telemetry.gap alert with
  owner/threshold/runbook instead of passing as healthy (documented in
  the runbook inventory).
- drill origin: validateLoopbackOrigin gates every network path before
  any fetch. Only a bare http://127.0.0.1:<port> URL passes; credentials,
  DNS names, hex/decimal/percent-encoded IP encodings, IPv6 forms, paths,
  queries, fragments, and non-http schemes are refused pre-contact
  (raw-string grammar gate plus parse round-trip, because the URL parser
  canonicalizes hostile encodings).
- terminal safety: controlSafe() strips C0/C1 control characters from all
  dynamically produced CLI output so hostile evidence paths cannot inject
  ANSI escapes into a terminal.

Gates: npm test 98/98, check:syntax, npm audit (0 vulns), check:diff,
deploy_staging status read-only; 30 adversarial probes against sanitizer,
alert edges, and live loopback CLI all pass. No merge, no deploy.

2026-08-22 21:43:18 +00:00

Timmy

ad8c8a9f4c

feat(ops): release observability dashboard and incident-response flow

Quality gates / quality (pull_request) Successful in 1m44s

Details

Implements issue #41 acceptance criteria without touching the live host:

- src/release-observability.js: sanitizeEvidence() allowlists bounded,
  privacy-safe evidence (release tag, commit, UTC time, per-check
  id/boundary/status/failure-class/latency/counters) and drops session
  identifiers, credentials, environment dumps, photo payloads, base64,
  image hashes, note text, emails, and any oversized/suspicious value.
- buildDashboard(): app/api/queue/model boundary rollups, failure-class
  counts, manual-fallback state; evaluateAlerts() with owner, threshold,
  severity, and runbook anchor per rule.
- scripts/release_dashboard.mjs: local operator CLI over sanitized
  evidence files or a loopback drill fixture; never contacts a live
  host and exits nonzero without echoing rejected input.
- docs/RELEASE-OBSERVABILITY.md: alert inventory, evidence schema,
  simulated worker-outage drill, incident flow, privacy boundary.
- Tests: sanitizer hostile-payload coverage, dashboard/alert rules,
  end-to-end loopback outage drill asserting exactly one actionable
  page alert plus graceful manual fallback, runbook/package contract.

2026-08-22 20:33:10 +00:00

Release observability dashboard and incident-response flow (#41) #60

3 Commits