Release observability dashboard and incident-response flow (#41) #60
123
docs/RELEASE-OBSERVABILITY.md
Normal file
123
docs/RELEASE-OBSERVABILITY.md
Normal file
|
|
@ -0,0 +1,123 @@
|
||||||
|
# Release observability and incident-response runbook
|
||||||
|
|
||||||
|
This runbook defines the privacy-safe release health surface for Timmy staging and release review. It is a reviewed operator document; it does not authorize changes to any live host. Everything here consumes sanitized release/health evidence only.
|
||||||
|
|
||||||
|
## Privacy boundary (non-negotiable)
|
||||||
|
|
||||||
|
Telemetry in this system is release telemetry, not user telemetry. Evidence files, dashboard output, alerts, and drill transcripts must never contain:
|
||||||
|
|
||||||
|
- photos or medical imagery of any kind;
|
||||||
|
- stool records or journal entry content;
|
||||||
|
- session identifiers, cookies, tokens, credentials, or environment dumps;
|
||||||
|
- infrastructure addresses beyond the loopback `127.0.0.1` drill fixture;
|
||||||
|
- user identity of any kind (no emails, usernames, device IDs).
|
||||||
|
|
||||||
|
`src/release-observability.js` enforces this mechanically: `sanitizeEvidence()` allowlists bounded fields (release tag, full commit, UTC timestamp, per-check id/boundary/status/failure-class/latency/counters) and rejects the whole evidence object when anything else appears — forbidden keys (sessions, cookies, base64 payloads, image hashes, notes, free text), values of the wrong JSON type, numbers outside their bounded ranges, or a single malformed check among healthy ones. A hostile file is refused loudly; it is never silently censored into apparent health. Failure classes are a closed vocabulary (`worker.unavailable`, `vision.timeout`, `model.error`) matched against a strict dotted lowercase slug grammar; anything else — newlines, ANSI/control characters, secrets, medical text, unknown classes — fails the whole evidence file closed rather than being rendered. Counters are bounded non-negative integers capped at 1,000,000. Every one of the four boundaries (app, api, queue, model) must be present; partial telemetry is refused instead of rendered as silent health. If sanitization fails, the tool exits nonzero and echoes nothing from the input, and all CLI output is scrubbed of control characters so hostile file paths can never inject terminal escapes. The sanitizer itself is tested with hostile payloads (`tests/release-observability.test.js`) and the CLI is tested to never echo rejected content (`tests/release-dashboard-cli.test.js`).
|
||||||
|
|
||||||
|
## Dashboard
|
||||||
|
|
||||||
|
Build a sanitized evidence file (schema below) and render:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
node scripts/release_dashboard.mjs --evidence /secure/inbox/evidence.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Output groups checks under four boundaries — app, api, queue, model — with pass/fail/degraded counts, max latency, bounded counters, failure-class totals, active alerts, and manual-fallback state. Exit code is `0` when no page-severity alert fires, `1` when one does, `2` on bad input. The CLI never contacts a live host; it reads local files or a loopback drill fixture only.
|
||||||
|
|
||||||
|
### Manual fallback states (truthful by construction)
|
||||||
|
|
||||||
|
The manual-fallback line is a safety statement, so it only ever claims what the evidence proves:
|
||||||
|
|
||||||
|
| Evidence state | Rendered line |
|
||||||
|
|---|---|
|
||||||
|
| every check explicitly `pass` | `manual fallback: available — all checks explicitly passing; no degradation observed; fallback not required` |
|
||||||
|
| any degradation away from the app boundary | `manual fallback: available — degraded elsewhere; local journal remains usable` |
|
||||||
|
| app boundary `fail` | `manual fallback: unavailable — app boundary is down; local journal cannot be served` |
|
||||||
|
| any check (including the app boundary itself) `unknown` | `manual fallback: unknown — app state unknown; verify the local journal by hand before relying on it` |
|
||||||
|
|
||||||
|
Unknown telemetry is never rendered as proven availability or as "no degradation": an unknown app state forces the `unknown` wording and demands manual verification before anyone relies on the journal.
|
||||||
|
|
||||||
|
### Evidence schema (v1)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"schemaVersion": 1,
|
||||||
|
"releaseTag": "string, 1-80 chars: letters, digits, dot, underscore, dash",
|
||||||
|
"commit": "40 lowercase hex characters (12+ accepted)",
|
||||||
|
"generatedAtUtc": "canonical real UTC, YYYY-MM-DDTHH:MM:SSZ (must round-trip as a real instant)",
|
||||||
|
"checks": [
|
||||||
|
{
|
||||||
|
"id": "lowercase-dot-or-dash id",
|
||||||
|
"boundary": "app | api | queue | model",
|
||||||
|
"status": "pass | fail | degraded | unknown",
|
||||||
|
"failureClass": "vocabulary class on fail/degraded only; omitted or null otherwise",
|
||||||
|
"latencyMs": "optional integer 0..3600000; never a string, boolean, null, or fraction",
|
||||||
|
"counters": { "ok|fail|abstain|retry|timeout|rejected|fallback|depth": "optional integer 0..1000000" }
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Every field above is type-checked against the exact JSON type: `schemaVersion` must be the number `1` (not `"1"`), counters and latency must be actual JSON numbers within their bounds, and timestamps must survive a UTC round-trip. A `pass` or `unknown` check must never carry a failureClass; a `fail` or `degraded` check may carry one of the closed-vocabulary classes or stay classless. Any other field — at the top level or inside a check — rejects the entire evidence object.
|
||||||
|
|
||||||
|
## Alert inventory
|
||||||
|
|
||||||
|
Alerts are evaluated by `evaluateAlerts()` over the sanitized dashboard. Every rule carries an owner, a threshold, severity, and a runbook anchor in this file. Page-severity alerts make the CLI exit `1`.
|
||||||
|
|
||||||
|
| Alert id | Severity | Owner | Threshold (observed >=) | Condition | Runbook |
|
||||||
|
|-----------------|----------|------------------|-------------------------|--------------------------------------------------|---------|
|
||||||
|
| `worker.outage` | page | release-operator | 1 | any check reports `worker.unavailable`, or both model and queue boundaries report failures | [Simulate a worker outage](#simulate-a-worker-outage) |
|
||||||
|
| `telemetry.gap` | warn | release-operator | 1 | one or more checks report status `unknown` (missing telemetry fails closed as a warning, never as health) | [Alert inventory](#alert-inventory) |
|
||||||
|
| `queue.backlog` | warn | release-operator | 10 | queue depth at or above threshold (`depth >= 10`); suppressed while `worker.outage` pages (depth is residual from the same incident) | [Alert inventory](#alert-inventory) |
|
||||||
|
| `vision.degraded` | warn | vision-owner | 3 | three or more `vision.timeout` failure classes | [Alert inventory](#alert-inventory) |
|
||||||
|
| `app.unhealthy` | page | release-operator | 1 | app boundary healthz reports fail; this is also the only condition that makes the manual fallback unavailable (`reason: app-down`) | [Alert inventory](#alert-inventory) |
|
||||||
|
|
||||||
|
Escalation: pages go to the release operator on call (currently Alexander as release owner); warns are batched into the next release-review pulse. An unresolved page after one manual fallback verification is raised in the Gitea issue for the affected release epic rather than paged repeatedly.
|
||||||
|
|
||||||
|
## Simulate a worker outage
|
||||||
|
|
||||||
|
This drill proves the issue #41 acceptance criterion: one simulated outage produces exactly one actionable page alert while the graceful manual fallback stays available. It runs entirely against a throwaway loopback fixture — never against staging or production.
|
||||||
|
|
||||||
|
1. Start any loopback fixture that serves:
|
||||||
|
- `GET /api/drill/checks` returning `{ "checks": [...] }`, healthy first, with at least one check for each boundary (app, api, queue, model);
|
||||||
|
- `POST /drill/outage` flipping the fixture's simulated worker off (subsequent `/api/drill/checks` responses then report `status: "fail"`, `failureClass: "worker.unavailable"` across the api, queue, and model checks).
|
||||||
|
2. Run the drill:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
node scripts/release_dashboard.mjs --drill-origin http://127.0.0.1:<port>
|
||||||
|
```
|
||||||
|
|
||||||
|
The origin must be a bare `http://127.0.0.1:<port>` URL. Credentials (`user:pass@`), DNS names, alternative IP encodings (hex octets, decimal integers, percent-encoding), IPv6 forms, non-http schemes, and any path or query are rejected before the CLI contacts anything.
|
||||||
|
|
||||||
|
3. Required outcome (the automated tests assert all of it):
|
||||||
|
- the pre-drill baseline is read from the fixture itself and must be genuinely healthy — every required boundary check explicitly `pass` and zero alerts of any severity; unknown, degraded, malformed, missing-boundary, or backlog-warning baselines are refused with exit code 2 before anything is announced or flipped, and the switch is never touched;
|
||||||
|
- after the flip, exactly one page alert fires — `worker.outage`; if the flip produced no real healthy-to-outage transition, the drill reports `DRILL FAIL` with exit code 1 instead of passing vacuously;
|
||||||
|
- the alert names owner `release-operator`, its threshold, and this runbook section;
|
||||||
|
- `manual fallback: AVAILABLE — degraded elsewhere; local journal remains usable`;
|
||||||
|
- exit code `0` (`DRILL PASS`).
|
||||||
|
|
||||||
|
4. Manual fallback procedure if a real outage ever matches this signature: users keep logging locally (the browser journal never depends on the vision worker), operators capture one sanitized evidence file, run the dashboard once, record the single page alert, and follow rollback via the staging runbook. No user data leaves the device.
|
||||||
|
|
||||||
|
The drill is exercised end-to-end by `tests/release-dashboard-cli.test.js` ("outage drill flips one simulated switch…") using an in-test loopback fixture; CI runs it on every PR without touching any real host.
|
||||||
|
|
||||||
|
## Incident flow (documented, local-first)
|
||||||
|
|
||||||
|
1. **Detect** — dashboard run or drill shows a page alert.
|
||||||
|
2. **Triage** — read the alert's failure class and boundary; open the linked runbook anchor above.
|
||||||
|
3. **Contain** — confirm the manual fallback line says `available`; if it does not, treat the app boundary as down too and escalate immediately.
|
||||||
|
4. **Verify scope** — re-run the dashboard on fresh sanitized evidence; never attach raw logs or user content to the incident.
|
||||||
|
5. **Recover** — use `scripts/deploy_staging.py rollback --commit <known-good>` inside an approved window (see `docs/STAGING-RUNBOOK.md`), or fix forward through a reviewed PR.
|
||||||
|
6. **Review** — file the sanitized dashboard output (text only) plus commit/tag identity as evidence in the release issue; delete stale evidence files from the inbox.
|
||||||
|
|
||||||
|
## Receipts
|
||||||
|
|
||||||
|
- `npm test` — includes `tests/release-observability.test.js`, `tests/release-dashboard.test.js`, `tests/release-dashboard-cli.test.js`.
|
||||||
|
- Strict-type tests prove `schemaVersion` must be the JSON number 1, counters and latency must be actual bounded JSON numbers (never strings, booleans, nulls, fractions, or out-of-range values), and `generatedAtUtc` must be canonical real UTC that survives a round-trip.
|
||||||
|
- Fail-closed tests prove a single malformed check — bad id, missing boundary, unknown status, wrong-typed counters, or an off-vocabulary class on any status — rejects the whole evidence object instead of silently disappearing beside healthy checks.
|
||||||
|
- Consistency tests prove a `pass` or `unknown` check can never carry a failureClass, so all-pass evidence can never page `worker.outage`; sanitization is proven idempotent, so omitted latency stays omitted instead of becoming 0ms on a second pass.
|
||||||
|
- Sanitizer hostile-payload test proves session tokens, cookies, environment dumps, private keys, base64 payloads, image hashes, note text, and emails reject the entire file rather than surviving into dashboard output.
|
||||||
|
- Hostile failure-class test proves newlines, carriage returns, ANSI escapes, control characters, secrets, medical text, SQL, oversized values, wrong types, and off-vocabulary classes all fail closed.
|
||||||
|
- Depth-counter tests prove the `queue.backlog` warn stays silent at 9, fires at exactly 10, reports observed counts above threshold, sums across queue checks, is suppressed while `worker.outage` pages, and cannot be smuggled in above the bounded counter ceiling.
|
||||||
|
- Boundary-completeness tests prove evidence missing any of app, api, queue, or model fails closed; unknown statuses surface as a `telemetry.gap` warn instead of passing as healthy.
|
||||||
|
- Manual-fallback tests prove a truthful deterministic contract: available with no degradation, active (`local-journal`) under any non-fatal degradation, unavailable only when the app boundary itself is down, and `unknown` (`app-state-unknown`, availability `null`) whenever telemetry is unknown — never claimed as healthy from unknown state.
|
||||||
|
- CLI tests prove rejected evidence exits nonzero without echoing contents, hostile paths cannot forge terminal lines with LF/CR (only program-authored line breaks reach stdout), drill origins are validated to bare loopback http before any network contact (credentials, DNS names, hex/decimal/percent-encoded IP encodings, IPv6, paths, queries, and foreign schemes are refused), the outage drill requires a genuine baseline — every required check explicitly passing and zero alerts — before announcing anything or touching its switch, refuses already-down fixtures, fails when no real transition occurs, and yields exactly one actionable alert with graceful manual fallback.
|
||||||
|
|
@ -4,12 +4,13 @@
|
||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js",
|
"test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/release-observability.test.js tests/release-dashboard.test.js tests/release-dashboard-cli.test.js tests/release-observability-runbook.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js",
|
||||||
"test:ui": "node tests/ui.acceptance.mjs",
|
"test:ui": "node tests/ui.acceptance.mjs",
|
||||||
"test:photo": "node tests/photo-first.acceptance.mjs",
|
"test:photo": "node tests/photo-first.acceptance.mjs",
|
||||||
"test:sleek": "node tests/sleek-chat.acceptance.mjs",
|
"test:sleek": "node tests/sleek-chat.acceptance.mjs",
|
||||||
"test:staging-smoke": "node tests/staging.acceptance.mjs",
|
"test:staging-smoke": "node tests/staging.acceptance.mjs",
|
||||||
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py",
|
"test:observability-drill": "node scripts/release_dashboard.mjs --self-check",
|
||||||
|
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check scripts/record_release_demo.mjs && node --check scripts/release_dashboard.mjs && node --check tests/staging.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py",
|
||||||
"check:diff": "bash scripts/check_diff.sh",
|
"check:diff": "bash scripts/check_diff.sh",
|
||||||
"start": "node server.mjs"
|
"start": "node server.mjs"
|
||||||
},
|
},
|
||||||
|
|
|
||||||
265
scripts/release_dashboard.mjs
Normal file
265
scripts/release_dashboard.mjs
Normal file
|
|
@ -0,0 +1,265 @@
|
||||||
|
#!/usr/bin/env node
|
||||||
|
// Local operator surface for release observability (issue #41).
|
||||||
|
// Consumes only sanitized evidence files or a loopback drill fixture.
|
||||||
|
// Never contacts a live host, never accepts credentials, never prints secrets.
|
||||||
|
import { readFile } from 'node:fs/promises';
|
||||||
|
import { sanitizeEvidence, buildDashboard, evaluateAlerts } from '../src/release-observability.js';
|
||||||
|
|
||||||
|
function argValue(args, flag) {
|
||||||
|
const index = args.indexOf(flag);
|
||||||
|
if (index === -1) return null;
|
||||||
|
return args[index + 1] || null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Terminal safety: strip C0/C1 control characters — newline and carriage
|
||||||
|
// return included — from any dynamically produced text before it reaches
|
||||||
|
// stdout/stderr. Hostile file paths or fixture payloads can therefore never
|
||||||
|
// forge additional terminal lines (e.g. a fake "PAGE worker.outage"): the
|
||||||
|
// only line boundaries in CLI output are the ones this program authors.
|
||||||
|
function controlSafe(text) {
|
||||||
|
return String(text)
|
||||||
|
.replace(/[\u0000-\u001F\u007F-\u009F]/g, '');
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadEvidence(source) {
|
||||||
|
if (source.drillOrigin) {
|
||||||
|
const response = await fetch(`${source.drillOrigin.replace(/\/$/, '')}/api/drill/checks`, { signal: AbortSignal.timeout(5_000) });
|
||||||
|
if (!response.ok) throw new Error(`drill fixture returned ${response.status}`);
|
||||||
|
const body = await response.json();
|
||||||
|
return {
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: controlSafe('daily-2026-08-22.1'),
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: new Date().toISOString().replace(/\.\d{3}Z$/, 'Z'),
|
||||||
|
checks: Array.isArray(body.checks) ? body.checks : [],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
const raw = JSON.parse(await readFile(source.evidencePath, 'utf8'));
|
||||||
|
return raw;
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderDashboard(dashboard, alerts) {
|
||||||
|
const lines = [];
|
||||||
|
lines.push('RELEASE OBSERVABILITY DASHBOARD');
|
||||||
|
lines.push(`release ${dashboard.identity.releaseTag} · commit ${dashboard.identity.commit.slice(0, 12)} · evidence ${dashboard.identity.generatedAtUtc}`);
|
||||||
|
lines.push('');
|
||||||
|
for (const boundary of ['app', 'api', 'queue', 'model']) {
|
||||||
|
const stats = dashboard.boundaries[boundary];
|
||||||
|
const latency = stats.latencyMs == null ? '' : ` · max latency ${stats.latencyMs}ms`;
|
||||||
|
const counterBits = Object.keys(stats.counters).sort()
|
||||||
|
.map(key => `${key}=${stats.counters[key]}`)
|
||||||
|
.join(' ');
|
||||||
|
const counterText = counterBits ? ` · ${counterBits}` : '';
|
||||||
|
lines.push(`${boundary.padEnd(6)}${stats.pass}/${stats.checks} pass · fail ${stats.fail} · degraded ${stats.degraded}${latency}${counterText}`);
|
||||||
|
}
|
||||||
|
lines.push('');
|
||||||
|
const failureEntries = Object.entries(dashboard.failureClasses);
|
||||||
|
if (failureEntries.length === 0) {
|
||||||
|
lines.push('failure classes: none');
|
||||||
|
} else {
|
||||||
|
for (const [failureClass, count] of failureEntries.sort((a, b) => b[1] - a[1])) {
|
||||||
|
lines.push(`failure class ${failureClass}: ${count}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
lines.push('');
|
||||||
|
lines.push(`alerts (${alerts.length})`);
|
||||||
|
for (const alert of alerts) {
|
||||||
|
lines.push(`[${alert.severity.toUpperCase()}] ${alert.id} — owner ${alert.owner}, observed ${alert.count} >= threshold ${alert.threshold}`);
|
||||||
|
lines.push(` ${alert.message}`);
|
||||||
|
lines.push(` runbook: ${alert.runbook}`);
|
||||||
|
}
|
||||||
|
if (alerts.length === 0) lines.push('no alert conditions met');
|
||||||
|
return lines.join('\n');
|
||||||
|
}
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
const args = process.argv.slice(2);
|
||||||
|
|
||||||
|
if (args.includes('--self-check')) {
|
||||||
|
console.log('release-observability module self-check: exports OK');
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const drillOrigin = argValue(args, '--drill-origin');
|
||||||
|
const evidencePath = argValue(args, '--evidence');
|
||||||
|
if (Boolean(drillOrigin) === Boolean(evidencePath)) {
|
||||||
|
console.error('usage: node scripts/release_dashboard.mjs (--evidence <sanitized-evidence.json> | --drill-origin <http://127.0.0.1:port>)');
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Gate every network path before any fetch can happen: a drill may only
|
||||||
|
// ever address a validated bare loopback origin.
|
||||||
|
let validatedOrigin = null;
|
||||||
|
if (drillOrigin) {
|
||||||
|
try {
|
||||||
|
validatedOrigin = validateLoopbackOrigin(drillOrigin);
|
||||||
|
} catch (error) {
|
||||||
|
console.error(controlSafe(error.message));
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let rawEvidence;
|
||||||
|
try {
|
||||||
|
rawEvidence = await loadEvidence({ drillOrigin: validatedOrigin, evidencePath });
|
||||||
|
} catch {
|
||||||
|
// Fixed wording on purpose: interpolating filesystem error messages would
|
||||||
|
// echo attacker-controlled path bytes (even sanitized) into the terminal.
|
||||||
|
console.error('release_dashboard: cannot read sanitized evidence (unreadable file or invalid JSON). No contents are echoed.');
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Drill mode owns its whole flow: the baseline gate must speak with one
|
||||||
|
// voice ("not a genuine drill baseline") whether the fixture served
|
||||||
|
// unsanitizable evidence or merely unhealthy checks, and nothing may be
|
||||||
|
// rendered or flipped until the baseline proves genuinely healthy.
|
||||||
|
if (drillOrigin) return runDrill({ drillOrigin: validatedOrigin });
|
||||||
|
|
||||||
|
const validated = sanitizeEvidence(rawEvidence);
|
||||||
|
if (!validated.ok) {
|
||||||
|
console.error('release_dashboard: evidence failed sanitization (schema mismatch, forbidden fields, or unbounded values). Nothing was rendered.');
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
const dashboard = buildDashboard(validated.evidence);
|
||||||
|
const alerts = evaluateAlerts(dashboard);
|
||||||
|
console.log(renderDashboard(dashboard, alerts));
|
||||||
|
console.log('');
|
||||||
|
console.log(manualFallbackLine(dashboard.manualFallback));
|
||||||
|
|
||||||
|
return alerts.some(alert => alert.severity === 'page') ? 1 : 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Truthful operator wording: the manual-fallback line is a safety statement,
|
||||||
|
// so each deterministic state says exactly what is known. Availability is
|
||||||
|
// only ever claimed from explicit passes; an unknown app boundary reads as
|
||||||
|
// unknown and demands hand verification — never as quiet health.
|
||||||
|
const FALLBACK_PHRASES = {
|
||||||
|
'none-required': 'all checks explicitly passing; no degradation observed; fallback not required',
|
||||||
|
'local-journal': 'degraded elsewhere; local journal remains usable',
|
||||||
|
'app-down': 'app boundary is down; local journal cannot be served',
|
||||||
|
'app-state-unknown': 'app state unknown; verify the local journal by hand before relying on it',
|
||||||
|
};
|
||||||
|
|
||||||
|
function manualFallbackLine(fallback, { shout = false } = {}) {
|
||||||
|
const label = fallback.available === null
|
||||||
|
? (shout ? 'UNKNOWN' : 'unknown')
|
||||||
|
: fallback.available
|
||||||
|
? (shout ? 'AVAILABLE' : 'available')
|
||||||
|
: (shout ? 'UNAVAILABLE' : 'unavailable');
|
||||||
|
return `manual fallback: ${label} — ${FALLBACK_PHRASES[fallback.reason] ?? fallback.reason}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// A drill may only ever talk to a throwaway fixture on 127.0.0.1. The origin
|
||||||
|
// must be a bare http URL whose host is exactly the loopback address: no
|
||||||
|
// credentials, no DNS names, no alternative IP encodings, no IPv6, no path,
|
||||||
|
// query, or fragment, and no scheme other than http.
|
||||||
|
const LOOPBACK_ORIGIN_PATTERN = /^http:\/\/127\.0\.0\.1:[0-9]{1,5}$/;
|
||||||
|
|
||||||
|
function validateLoopbackOrigin(rawOrigin) {
|
||||||
|
if (typeof rawOrigin !== 'string') {
|
||||||
|
throw new Error('drill origin rejected: expected an http://127.0.0.1:<port> URL');
|
||||||
|
}
|
||||||
|
// Grammar gate runs on the RAW string first: the WHATWG URL parser
|
||||||
|
// canonicalizes hostile host encodings (hex octets, decimal IP integers,
|
||||||
|
// percent-encoded dots) into a clean-looking 127.0.0.1, so a normalized
|
||||||
|
// string can never be trusted on its own.
|
||||||
|
if (!LOOPBACK_ORIGIN_PATTERN.test(rawOrigin)) {
|
||||||
|
throw new Error('drill origin rejected: only a bare http://127.0.0.1:<port> loopback origin is permitted');
|
||||||
|
}
|
||||||
|
let parsed;
|
||||||
|
try {
|
||||||
|
parsed = new URL(rawOrigin);
|
||||||
|
} catch {
|
||||||
|
throw new Error('drill origin rejected: not a valid URL');
|
||||||
|
}
|
||||||
|
// Round-trip defense in depth: the parsed form must be byte-identical to
|
||||||
|
// what was supplied, proving no hidden credentials, path, query, fragment,
|
||||||
|
// or alternative host encoding was smuggled past the grammar.
|
||||||
|
if (parsed.toString().replace(/\/$/, '') !== rawOrigin) {
|
||||||
|
throw new Error('drill origin rejected: URL did not round-trip as a bare loopback origin');
|
||||||
|
}
|
||||||
|
if (parsed.protocol !== 'http:' || parsed.hostname !== '127.0.0.1') {
|
||||||
|
throw new Error('drill origin rejected: host must be exactly http://127.0.0.1');
|
||||||
|
}
|
||||||
|
const port = Number(parsed.port);
|
||||||
|
if (!Number.isInteger(port) || port < 1 || port > 65535) {
|
||||||
|
throw new Error('drill origin rejected: port must be between 1 and 65535');
|
||||||
|
}
|
||||||
|
return rawOrigin;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function runDrill({ drillOrigin }) {
|
||||||
|
let origin;
|
||||||
|
try {
|
||||||
|
origin = validateLoopbackOrigin(drillOrigin);
|
||||||
|
} catch (error) {
|
||||||
|
console.error(controlSafe(error.message));
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Genuine baseline: read the fixture's own current state BEFORE anything is
|
||||||
|
// announced or flipped. A drill may only start from proven health — every
|
||||||
|
// required boundary check must explicitly pass and zero alerts may fire.
|
||||||
|
// Unknown, degraded, malformed, missing-boundary, or warning baselines are
|
||||||
|
// refused here, so stdout stays empty whenever no drill actually ran.
|
||||||
|
const preRaw = await loadEvidence({ drillOrigin });
|
||||||
|
const preValidated = sanitizeEvidence(preRaw);
|
||||||
|
if (!preValidated.ok) {
|
||||||
|
console.error('release_dashboard: fixture evidence is unsanitizable: not a genuine drill baseline (schema mismatch, forbidden fields, or unbounded values). Nothing will be rendered or flipped.');
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
const preDashboard = buildDashboard(preValidated.evidence);
|
||||||
|
const preAlerts = evaluateAlerts(preDashboard);
|
||||||
|
const allExplicitlyPassing = Object.values(preDashboard.boundaries).every(
|
||||||
|
boundary => boundary.checks > 0 && boundary.pass === boundary.checks,
|
||||||
|
);
|
||||||
|
if (!allExplicitlyPassing || preAlerts.length > 0) {
|
||||||
|
console.error('release_dashboard: fixture is not healthy before the drill: not a genuine drill baseline (every required boundary check must explicitly pass with zero alerts). Refusing to flip an already-down or uncertain worker.');
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log('');
|
||||||
|
console.log('WORKER OUTAGE DRILL (simulated fixture on loopback; no live host is contacted)');
|
||||||
|
console.log('pre-drill baseline: every required boundary check explicitly passing; alerts: none');
|
||||||
|
console.log('pre-drill: healthy (0 page alerts)');
|
||||||
|
|
||||||
|
try {
|
||||||
|
const response = await fetch(`${origin}/drill/outage`, { method: 'POST', signal: AbortSignal.timeout(5_000) });
|
||||||
|
if (!response.ok) throw new Error(`fixture returned ${response.status}`);
|
||||||
|
} catch {
|
||||||
|
// Fixed wording on purpose: fetch failures can embed attacker-chosen URL
|
||||||
|
// text; echoing it (even sanitized) has no operational value.
|
||||||
|
console.error('release_dashboard: drill switch failed. The fixture switch was not verifiably flipped.');
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
const postRaw = await loadEvidence({ drillOrigin });
|
||||||
|
const postValidated = sanitizeEvidence(postRaw);
|
||||||
|
if (!postValidated.ok) {
|
||||||
|
console.error('release_dashboard: post-outage evidence failed sanitization.');
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
const postDashboard = buildDashboard(postValidated.evidence);
|
||||||
|
const postAlerts = evaluateAlerts(postDashboard);
|
||||||
|
const pageAlerts = postAlerts.filter(alert => alert.severity === 'page');
|
||||||
|
|
||||||
|
if (pageAlerts.length === 0) {
|
||||||
|
console.log('no outage observed after flip: the fixture never transitioned from healthy to outage.');
|
||||||
|
console.log('DRILL FAIL');
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
console.log(`post-outage: exactly 1 alert expected, found ${pageAlerts.length}`);
|
||||||
|
for (const alert of pageAlerts) {
|
||||||
|
console.log(` ${alert.severity.toUpperCase()} ${alert.message.replace(/^PAGE /, '')}`);
|
||||||
|
console.log(` owner ${alert.owner} · threshold ${alert.threshold} · runbook ${alert.runbook}`);
|
||||||
|
}
|
||||||
|
console.log(manualFallbackLine(postDashboard.manualFallback, { shout: true }));
|
||||||
|
const pass = pageAlerts.length === 1
|
||||||
|
&& pageAlerts[0].id === 'worker.outage'
|
||||||
|
&& postDashboard.manualFallback.available === true;
|
||||||
|
console.log(pass ? 'DRILL PASS' : 'DRILL FAIL');
|
||||||
|
return pass ? 0 : 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
process.exitCode = await main();
|
||||||
322
src/release-observability.js
Normal file
322
src/release-observability.js
Normal file
|
|
@ -0,0 +1,322 @@
|
||||||
|
const SCHEMA_VERSION = 1;
|
||||||
|
const BOUNDARIES = new Set(['app', 'api', 'queue', 'model']);
|
||||||
|
const STATUSES = new Set(['pass', 'fail', 'degraded', 'unknown']);
|
||||||
|
|
||||||
|
// Closed privacy-safe vocabulary: a failure class can never carry free text,
|
||||||
|
// secrets, medical notes, control characters, or terminal escapes because only
|
||||||
|
// these dotted lowercase slugs may ever reach rendering.
|
||||||
|
const FAILURE_CLASSES = new Set(['worker.unavailable', 'vision.timeout', 'model.error']);
|
||||||
|
const FAILURE_CLASS_PATTERN = /^[a-z][a-z0-9]*(?:\.[a-z][a-z0-9]*){1,2}$/;
|
||||||
|
|
||||||
|
const FORBIDDEN_TOP_KEYS = new Set([
|
||||||
|
'session', 'sessionid', 'sessionidtoken', 'sessiontoken', 'cookie', 'cookies',
|
||||||
|
'authorization', 'authheader', 'apikey', 'api_key', 'password', 'credential',
|
||||||
|
'credentials', 'environment', 'envdump', 'photos', 'photo', 'image', 'images',
|
||||||
|
'base64', 'note', 'notetext', 'notes', 'user', 'username', 'email', 'identity',
|
||||||
|
]);
|
||||||
|
|
||||||
|
const FORBIDDEN_CHECK_KEYS = new Set([
|
||||||
|
...FORBIDDEN_TOP_KEYS,
|
||||||
|
'imagehash', 'base64payload', 'payload', 'blob', 'rawoutput',
|
||||||
|
]);
|
||||||
|
|
||||||
|
const COUNTER_KEYS = new Set(['ok', 'fail', 'abstain', 'retry', 'timeout', 'rejected', 'fallback', 'depth']);
|
||||||
|
const COUNTER_CEILING = 1_000_000;
|
||||||
|
const LATENCY_CEILING_MS = 3_600_000;
|
||||||
|
const CHECK_KEYS = new Set(['id', 'boundary', 'status', 'failureClass', 'latencyMs', 'counters']);
|
||||||
|
|
||||||
|
// Strict JSON-type gates: evidence crosses a JSON boundary, so only real JSON
|
||||||
|
// types may pass. Strings that merely look like numbers, booleans, nulls,
|
||||||
|
// non-finite values, and fractions are coercion attempts and must fail closed
|
||||||
|
// instead of being silently converted by Number().
|
||||||
|
function isPlainObject(value) {
|
||||||
|
return typeof value === 'object' && value !== null && !Array.isArray(value);
|
||||||
|
}
|
||||||
|
|
||||||
|
function isBoundedInteger(value, { min, max }) {
|
||||||
|
return typeof value === 'number' && Number.isInteger(value) && value >= min && value <= max;
|
||||||
|
}
|
||||||
|
|
||||||
|
function boundedCounters(raw) {
|
||||||
|
if (raw === undefined) return {};
|
||||||
|
if (!isPlainObject(raw)) return null;
|
||||||
|
const counters = {};
|
||||||
|
for (const key of Object.keys(raw).sort()) {
|
||||||
|
if (!COUNTER_KEYS.has(key)) return null;
|
||||||
|
if (!isBoundedInteger(raw[key], { min: 0, max: COUNTER_CEILING })) return null;
|
||||||
|
counters[key] = raw[key];
|
||||||
|
}
|
||||||
|
return counters;
|
||||||
|
}
|
||||||
|
|
||||||
|
const GENERATED_AT_PATTERN = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$/;
|
||||||
|
|
||||||
|
// Canonical real UTC: the string must match the exact grammar AND survive a
|
||||||
|
// Date round-trip byte-identically, so impossible dates (Feb 30, hour 25,
|
||||||
|
// month 13) and non-canonical spellings (offsets, lowercase designators, a
|
||||||
|
// space separator) are rejected rather than normalized into something else.
|
||||||
|
function isCanonicalUtc(value) {
|
||||||
|
if (typeof value !== 'string' || !GENERATED_AT_PATTERN.test(value)) return false;
|
||||||
|
const parsed = new Date(value);
|
||||||
|
if (Number.isNaN(parsed.getTime())) return false;
|
||||||
|
// toISOString() always carries milliseconds, so strip them before comparing
|
||||||
|
// to the canonical seconds-precision spelling the grammar requires.
|
||||||
|
return parsed.toISOString().replace(/\.\d{3}Z$/, 'Z') === value;
|
||||||
|
}
|
||||||
|
|
||||||
|
// A check is either fully valid or the whole evidence object is rejected.
|
||||||
|
// Malformed checks may never silently disappear beside healthy ones: censoring
|
||||||
|
// a failure would dress a gap up as health, so every field is type-gated and
|
||||||
|
// any violation fails closed for the entire file.
|
||||||
|
function buildCheck(raw) {
|
||||||
|
if (!isPlainObject(raw)) return { rejected: true };
|
||||||
|
const id = raw.id;
|
||||||
|
if (typeof id !== 'string' || !/^[a-z][a-z0-9.-]{0,63}$/.test(id)) return { rejected: true };
|
||||||
|
if (!Object.hasOwn(raw, 'boundary') || !BOUNDARIES.has(raw.boundary)) return { rejected: true };
|
||||||
|
const status = raw.status;
|
||||||
|
if (typeof status !== 'string' || !STATUSES.has(status)) return { rejected: true };
|
||||||
|
|
||||||
|
let failureClass = null;
|
||||||
|
if (raw.failureClass !== undefined && raw.failureClass !== null) {
|
||||||
|
failureClass = raw.failureClass;
|
||||||
|
if (
|
||||||
|
typeof failureClass !== 'string'
|
||||||
|
|| !FAILURE_CLASSES.has(failureClass)
|
||||||
|
|| !FAILURE_CLASS_PATTERN.test(failureClass)
|
||||||
|
) {
|
||||||
|
return { rejected: true };
|
||||||
|
}
|
||||||
|
if (status === 'pass' || status === 'unknown') {
|
||||||
|
// Consistency contract: only failing/degraded checks may classify their
|
||||||
|
// failures. A pass or unknown can therefore never contribute to
|
||||||
|
// worker.outage, and all-pass evidence can never page anyone.
|
||||||
|
// A fail/degraded may stay classless (the schema keeps failureClass
|
||||||
|
// nullable); carrying a class is optional, never mandatory.
|
||||||
|
return { rejected: true };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const counters = boundedCounters(raw.counters);
|
||||||
|
if (counters === null) return { rejected: true };
|
||||||
|
|
||||||
|
const latencyMs = raw.latencyMs;
|
||||||
|
if (latencyMs !== undefined) {
|
||||||
|
// The schema allows an integer or omission — explicit null is neither,
|
||||||
|
// and accepting it here would let a second pass rewrite the shape.
|
||||||
|
if (!isBoundedInteger(latencyMs, { min: 0, max: LATENCY_CEILING_MS })) {
|
||||||
|
return { rejected: true };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const key of Object.keys(raw)) {
|
||||||
|
if (!CHECK_KEYS.has(key)) return { rejected: true };
|
||||||
|
}
|
||||||
|
|
||||||
|
const check = {
|
||||||
|
id,
|
||||||
|
boundary: raw.boundary,
|
||||||
|
status,
|
||||||
|
failureClass,
|
||||||
|
counters,
|
||||||
|
};
|
||||||
|
if (latencyMs !== undefined && latencyMs !== null) check.latencyMs = latencyMs;
|
||||||
|
return { check, dropped: false };
|
||||||
|
}
|
||||||
|
|
||||||
|
export function sanitizeEvidence(input) {
|
||||||
|
if (!isPlainObject(input)) return { ok: false, evidence: null };
|
||||||
|
// Forbidden-shaped top-level fields reject the whole evidence object.
|
||||||
|
// Silently ignoring them would let a censored file pass as healthy; a
|
||||||
|
// hostile file must be refused loudly instead of laundered.
|
||||||
|
for (const key of Object.keys(input)) {
|
||||||
|
if (FORBIDDEN_TOP_KEYS.has(key.replace(/[^a-z0-9]/gi, '').toLowerCase())) {
|
||||||
|
return { ok: false, evidence: null };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Identity fields keep their exact safe JSON types; anything else is a
|
||||||
|
// type-confusion attempt and fails closed.
|
||||||
|
const commit = input.commit;
|
||||||
|
if (typeof commit !== 'string' || !/^[0-9a-f]{12,40}$/.test(commit)) return { ok: false, evidence: null };
|
||||||
|
const releaseTag = input.releaseTag;
|
||||||
|
if (
|
||||||
|
typeof releaseTag !== 'string'
|
||||||
|
|| !/^[A-Za-z0-9][A-Za-z0-9._-]{0,79}$/.test(releaseTag)
|
||||||
|
) {
|
||||||
|
return { ok: false, evidence: null };
|
||||||
|
}
|
||||||
|
const generatedAtUtc = input.generatedAtUtc;
|
||||||
|
if (!isCanonicalUtc(generatedAtUtc)) return { ok: false, evidence: null };
|
||||||
|
// schemaVersion is the JSON number 1 exactly — "1", true, null, 1.0-style
|
||||||
|
// fractions, and every other lookalike are refused.
|
||||||
|
if (input.schemaVersion !== SCHEMA_VERSION) return { ok: false, evidence: null };
|
||||||
|
if (!Array.isArray(input.checks) || input.checks.length === 0 || input.checks.length > 100) {
|
||||||
|
return { ok: false, evidence: null };
|
||||||
|
}
|
||||||
|
|
||||||
|
const checks = [];
|
||||||
|
for (const raw of input.checks) {
|
||||||
|
const built = buildCheck(raw);
|
||||||
|
if (built.rejected) return { ok: false, evidence: null };
|
||||||
|
checks.push(built.check);
|
||||||
|
}
|
||||||
|
if (checks.length === 0) return { ok: false, evidence: null };
|
||||||
|
// Fail closed on partial telemetry: an evidence set that is missing any of
|
||||||
|
// the four boundaries cannot be distinguished from a censored one, so the
|
||||||
|
// dashboard refuses it instead of rendering silent gaps as healthy.
|
||||||
|
const presentBoundaries = new Set(checks.map(check => check.boundary));
|
||||||
|
for (const boundary of BOUNDARIES) {
|
||||||
|
if (!presentBoundaries.has(boundary)) return { ok: false, evidence: null };
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
ok: true,
|
||||||
|
evidence: {
|
||||||
|
schemaVersion: SCHEMA_VERSION,
|
||||||
|
releaseTag,
|
||||||
|
commit,
|
||||||
|
generatedAtUtc,
|
||||||
|
checks,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function emptyBoundary() {
|
||||||
|
return { checks: 0, pass: 0, degraded: 0, fail: 0, unknown: 0, latencyMs: null, counters: {} };
|
||||||
|
}
|
||||||
|
|
||||||
|
export function buildDashboard(evidence) {
|
||||||
|
const validated = sanitizeEvidence(evidence);
|
||||||
|
if (!validated.ok) return { ok: false, boundaries: null, totals: null, failureClasses: null, manualFallback: null };
|
||||||
|
const clean = validated.evidence;
|
||||||
|
|
||||||
|
const boundaries = { app: emptyBoundary(), api: emptyBoundary(), queue: emptyBoundary(), model: emptyBoundary() };
|
||||||
|
const failureClasses = {};
|
||||||
|
const totals = { checks: 0, pass: 0, degraded: 0, fail: 0, unknown: 0 };
|
||||||
|
|
||||||
|
for (const check of clean.checks) {
|
||||||
|
const bucket = boundaries[check.boundary];
|
||||||
|
bucket.checks += 1;
|
||||||
|
bucket[check.status] += 1;
|
||||||
|
if (check.latencyMs != null && (bucket.latencyMs == null || check.latencyMs > bucket.latencyMs)) {
|
||||||
|
bucket.latencyMs = check.latencyMs;
|
||||||
|
}
|
||||||
|
for (const key of Object.keys(check.counters)) {
|
||||||
|
bucket.counters[key] = (bucket.counters[key] || 0) + check.counters[key];
|
||||||
|
}
|
||||||
|
if (check.failureClass) {
|
||||||
|
failureClasses[check.failureClass] = (failureClasses[check.failureClass] || 0) + 1;
|
||||||
|
}
|
||||||
|
totals.checks += 1;
|
||||||
|
totals[check.status] += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Deterministic manual-fallback contract, truthful under every status:
|
||||||
|
// - app-boundary fail -> { available: false, reason: 'app-down' }
|
||||||
|
// the journal is served by the app, so a dead app takes it down too;
|
||||||
|
// - app-boundary unknown -> { available: null, reason: 'app-state-unknown' }
|
||||||
|
// availability cannot be claimed from unknown telemetry;
|
||||||
|
// - any other degradation -> { available: true, reason: 'local-journal' };
|
||||||
|
// - everything explicitly passing -> { available: true, reason: 'none-required' }.
|
||||||
|
const appBoundary = boundaries.app;
|
||||||
|
const anyDegradation = totals.fail > 0 || totals.degraded > 0;
|
||||||
|
const anyUnknown = totals.unknown > 0;
|
||||||
|
let manualFallback;
|
||||||
|
if (appBoundary.fail > 0) {
|
||||||
|
manualFallback = { available: false, reason: 'app-down' };
|
||||||
|
} else if (anyUnknown) {
|
||||||
|
manualFallback = { available: null, reason: 'app-state-unknown' };
|
||||||
|
} else if (anyDegradation) {
|
||||||
|
manualFallback = { available: true, reason: 'local-journal' };
|
||||||
|
} else {
|
||||||
|
manualFallback = { available: true, reason: 'none-required' };
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
ok: true,
|
||||||
|
identity: { releaseTag: clean.releaseTag, commit: clean.commit, generatedAtUtc: clean.generatedAtUtc },
|
||||||
|
boundaries,
|
||||||
|
totals,
|
||||||
|
failureClasses,
|
||||||
|
manualFallback,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export const DEFAULT_ALERT_RULES = [
|
||||||
|
{
|
||||||
|
id: 'worker.outage',
|
||||||
|
severity: 'page',
|
||||||
|
owner: 'release-operator',
|
||||||
|
threshold: 1,
|
||||||
|
runbook: 'docs/RELEASE-OBSERVABILITY.md#simulate-a-worker-outage',
|
||||||
|
count: dashboard =>
|
||||||
|
Math.max(
|
||||||
|
dashboard.failureClasses['worker.unavailable'] || 0,
|
||||||
|
dashboard.boundaries.model.fail > 0 && dashboard.boundaries.queue.fail > 0
|
||||||
|
? dashboard.boundaries.model.fail + dashboard.boundaries.queue.fail + dashboard.boundaries.api.fail
|
||||||
|
: 0,
|
||||||
|
),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'telemetry.gap',
|
||||||
|
severity: 'warn',
|
||||||
|
owner: 'release-operator',
|
||||||
|
threshold: 1,
|
||||||
|
runbook: 'docs/RELEASE-OBSERVABILITY.md#alert-inventory',
|
||||||
|
count: dashboard => dashboard.totals.unknown,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'queue.backlog',
|
||||||
|
severity: 'warn',
|
||||||
|
owner: 'release-operator',
|
||||||
|
threshold: 10,
|
||||||
|
runbook: 'docs/RELEASE-OBSERVABILITY.md#alert-inventory',
|
||||||
|
count: dashboard => dashboard.boundaries.queue.counters.depth || 0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'vision.degraded',
|
||||||
|
severity: 'warn',
|
||||||
|
owner: 'vision-owner',
|
||||||
|
threshold: 3,
|
||||||
|
runbook: 'docs/RELEASE-OBSERVABILITY.md#alert-inventory',
|
||||||
|
count: dashboard => dashboard.failureClasses['vision.timeout'] || 0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'app.unhealthy',
|
||||||
|
severity: 'page',
|
||||||
|
owner: 'release-operator',
|
||||||
|
threshold: 1,
|
||||||
|
runbook: 'docs/RELEASE-OBSERVABILITY.md#alert-inventory',
|
||||||
|
count: dashboard => dashboard.boundaries.app.fail,
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
export function evaluateAlerts(dashboard, rules = DEFAULT_ALERT_RULES) {
|
||||||
|
if (!dashboard || !dashboard.ok) return [];
|
||||||
|
const alerts = [];
|
||||||
|
for (const rule of rules) {
|
||||||
|
let count = 0;
|
||||||
|
try {
|
||||||
|
count = Number(rule.count(dashboard));
|
||||||
|
} catch {
|
||||||
|
count = 0;
|
||||||
|
}
|
||||||
|
if (!Number.isFinite(count) || count <= 0 || count < rule.threshold) continue;
|
||||||
|
// Correlation: while a worker outage is paging, queue depth is residual
|
||||||
|
// noise from the same incident, so the backlog warn would only duplicate
|
||||||
|
// the page. It stays active the moment the page clears.
|
||||||
|
if (rule.id === 'queue.backlog' && alerts.some(alert => alert.id === 'worker.outage')) continue;
|
||||||
|
alerts.push({
|
||||||
|
id: rule.id,
|
||||||
|
severity: rule.severity,
|
||||||
|
owner: rule.owner,
|
||||||
|
threshold: rule.threshold,
|
||||||
|
count,
|
||||||
|
runbook: rule.runbook,
|
||||||
|
message:
|
||||||
|
rule.id === 'worker.outage'
|
||||||
|
? `${rule.severity.toUpperCase()} worker.unavailable on ${dashboard.identity.releaseTag} (${dashboard.identity.commit.slice(0, 12)}): ${count} failing boundary checks — follow ${rule.runbook} and keep the local journal as fallback.`
|
||||||
|
: `${rule.severity.toUpperCase()} ${rule.id} on ${dashboard.identity.releaseTag}: observed ${count} (threshold ${rule.threshold}) — see ${rule.runbook}.`,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return alerts;
|
||||||
|
}
|
||||||
338
tests/release-dashboard-cli.test.js
Normal file
338
tests/release-dashboard-cli.test.js
Normal file
|
|
@ -0,0 +1,338 @@
|
||||||
|
import test from 'node:test';
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import { spawn } from 'node:child_process';
|
||||||
|
import { mkdtemp, rm, writeFile } from 'node:fs/promises';
|
||||||
|
import { tmpdir } from 'node:os';
|
||||||
|
import { join } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import http from 'node:http';
|
||||||
|
|
||||||
|
const root = fileURLToPath(new URL('..', import.meta.url));
|
||||||
|
let nextPort = 43600;
|
||||||
|
|
||||||
|
const healthyEvidence = {
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: 'daily-2026-08-22.1',
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'pass', counters: { ok: 40, fail: 0 } },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 2, ok: 30, fail: 0 } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 900, counters: { ok: 25, fail: 0, abstain: 3 } },
|
||||||
|
],
|
||||||
|
};
|
||||||
|
|
||||||
|
function runCli(args) {
|
||||||
|
return new Promise(resolve => {
|
||||||
|
const child = spawn(process.execPath, ['scripts/release_dashboard.mjs', ...args], {
|
||||||
|
cwd: root,
|
||||||
|
stdio: ['ignore', 'pipe', 'pipe'],
|
||||||
|
env: process.env,
|
||||||
|
});
|
||||||
|
let stdout = '';
|
||||||
|
let stderr = '';
|
||||||
|
child.stdout.on('data', chunk => { stdout += chunk; });
|
||||||
|
child.stderr.on('data', chunk => { stderr += chunk; });
|
||||||
|
const timer = setTimeout(() => child.kill('SIGKILL'), 15_000);
|
||||||
|
child.on('close', code => {
|
||||||
|
clearTimeout(timer);
|
||||||
|
resolve({ code, stdout, stderr });
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
test('cli renders boundary dashboard with zero alerts from sanitized evidence file', async () => {
|
||||||
|
const dir = await mkdtemp(join(tmpdir(), 'timmy-obs-'));
|
||||||
|
try {
|
||||||
|
const evidencePath = join(dir, 'evidence.json');
|
||||||
|
await writeFile(evidencePath, JSON.stringify(healthyEvidence));
|
||||||
|
|
||||||
|
const result = await runCli(['--evidence', evidencePath]);
|
||||||
|
assert.equal(result.code, 0, result.stderr);
|
||||||
|
assert.match(result.stdout, /RELEASE OBSERVABILITY DASHBOARD/);
|
||||||
|
assert.match(result.stdout, /release daily-2026-08-22\.1 · commit ca31e6d38bec/);
|
||||||
|
assert.match(result.stdout, /app\s+1\/1 pass/);
|
||||||
|
assert.match(result.stdout, /queue\s+1\/1 pass/);
|
||||||
|
assert.match(result.stdout, /model\s+1\/1 pass/);
|
||||||
|
assert.match(result.stdout, /manual fallback: available/);
|
||||||
|
assert.match(result.stdout, /alerts \(0\)/);
|
||||||
|
assert.doesNotMatch(result.stdout, /token|cookie|session|password/i);
|
||||||
|
} finally {
|
||||||
|
await rm(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('cli exits nonzero on unsanitizable evidence and never echoes its contents', async () => {
|
||||||
|
const dir = await mkdtemp(join(tmpdir(), 'timmy-obs-'));
|
||||||
|
try {
|
||||||
|
const hostile = {
|
||||||
|
...healthyEvidence,
|
||||||
|
sessionToken: 'sess_live_supersecret99',
|
||||||
|
checks: [{ id: 'x', boundary: 'nope-boundary', status: 'weird' }],
|
||||||
|
};
|
||||||
|
const evidencePath = join(dir, 'hostile.json');
|
||||||
|
await writeFile(evidencePath, JSON.stringify(hostile));
|
||||||
|
|
||||||
|
const result = await runCli(['--evidence', evidencePath]);
|
||||||
|
assert.equal(result.code, 2);
|
||||||
|
assert.doesNotMatch(result.stderr + result.stdout, /supersecret/);
|
||||||
|
} finally {
|
||||||
|
await rm(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('cli output stays control-safe when rejected evidence paths carry ANSI escapes', async () => {
|
||||||
|
const evilPath = join(tmpdir(), `timmy-\x1b[31mred\rbell\x07.json`);
|
||||||
|
const result = await runCli(['--evidence', evilPath]);
|
||||||
|
assert.equal(result.code, 2);
|
||||||
|
const emitted = result.stdout + result.stderr;
|
||||||
|
assert.doesNotMatch(emitted, /\x1b/, 'ANSI escape reached the terminal');
|
||||||
|
assert.doesNotMatch(emitted, /\x07/, 'bell control character reached the terminal');
|
||||||
|
assert.doesNotMatch(emitted, /\r/, 'carriage return reached the terminal');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('cli output stays control-safe when hostile paths carry LF-forged PAGE lines', async () => {
|
||||||
|
// The path embeds a real newline followed by a forged terminal line, as if
|
||||||
|
// an attacker-supplied filename tried to end the error line early and page
|
||||||
|
// the operator with fake drill output.
|
||||||
|
const forgedPath = join(tmpdir(), `timmy-legit.json\nPAGE worker.outage — ALL SYSTEMS DOWN\r\nEXIT 1 DRILL FAIL`);
|
||||||
|
const result = await runCli(['--evidence', forgedPath]);
|
||||||
|
assert.equal(result.code, 2);
|
||||||
|
const emitted = result.stdout + result.stderr;
|
||||||
|
const lines = emitted.split('\n').filter(line => line.length > 0);
|
||||||
|
for (const line of lines) {
|
||||||
|
assert.match(line, /^release_dashboard:/, `forged line escaped into the terminal: ${JSON.stringify(line)}`);
|
||||||
|
}
|
||||||
|
assert.doesNotMatch(emitted, /ALL SYSTEMS DOWN/, 'LF-forged PAGE text reached the terminal');
|
||||||
|
assert.doesNotMatch(emitted, /DRILL FAIL/, 'LF-forged drill verdict reached the terminal');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('successful dashboard rendering emits no control characters besides newlines', async () => {
|
||||||
|
const dir = await mkdtemp(join(tmpdir(), 'timmy-obs-'));
|
||||||
|
try {
|
||||||
|
const evidencePath = join(dir, 'evidence.json');
|
||||||
|
await writeFile(evidencePath, JSON.stringify(healthyEvidence));
|
||||||
|
const result = await runCli(['--evidence', evidencePath]);
|
||||||
|
assert.equal(result.code, 0, result.stderr);
|
||||||
|
assert.doesNotMatch(result.stdout, /[\u0000-\u0009\u000B-\u001F\u007F-\u009F]/, 'control character reached the terminal');
|
||||||
|
} finally {
|
||||||
|
await rm(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('cli never claims proven availability or no degradation from unknown telemetry', async () => {
|
||||||
|
const dir = await mkdtemp(join(tmpdir(), 'timmy-obs-'));
|
||||||
|
try {
|
||||||
|
const unknownEvidence = {
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: 'daily-2026-08-22.1',
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'unknown', latencyMs: 12 },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'pass', counters: { ok: 40, fail: 0 } },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 2, ok: 30, fail: 0 } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 900, counters: { ok: 25, fail: 0 } },
|
||||||
|
],
|
||||||
|
};
|
||||||
|
const evidencePath = join(dir, 'unknown.json');
|
||||||
|
await writeFile(evidencePath, JSON.stringify(unknownEvidence));
|
||||||
|
|
||||||
|
const result = await runCli(['--evidence', evidencePath]);
|
||||||
|
assert.equal(result.code, 0, result.stderr);
|
||||||
|
assert.match(result.stdout, /manual fallback: unknown — app state unknown; verify the local journal by hand before relying on it/);
|
||||||
|
assert.doesNotMatch(result.stdout, /no degradation detected/, 'unknown app state must never read as proven health');
|
||||||
|
assert.doesNotMatch(result.stdout, /manual fallback: available/, 'unknown app state must never read as proven availability');
|
||||||
|
} finally {
|
||||||
|
await rm(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
async function startDrillServer({ startDown = false, flipHasNoEffect = false, preDrillChecks = null } = {}) {
|
||||||
|
const port = nextPort++;
|
||||||
|
const origin = `http://127.0.0.1:${port}`;
|
||||||
|
let workerUp = !startDown;
|
||||||
|
let outageSwitchCount = 0;
|
||||||
|
const app = http.createServer((req, res) => {
|
||||||
|
const url = new URL(req.url, origin);
|
||||||
|
if (url.pathname === '/drill/outage' && req.method === 'POST') {
|
||||||
|
outageSwitchCount += 1;
|
||||||
|
if (!flipHasNoEffect) workerUp = false;
|
||||||
|
res.writeHead(200, { 'content-type': 'application/json' });
|
||||||
|
res.end(JSON.stringify({ ok: true }));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (url.pathname === '/api/healthz') {
|
||||||
|
res.writeHead(200, { 'content-type': 'application/json' });
|
||||||
|
res.end(JSON.stringify({ ok: true, release: 'daily-2026-08-22.1', commit: 'ca31e6d38bec649407f63880504554c59f2878ae', visionEnabled: false, agentEnabled: false }));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (url.pathname === '/api/drill/checks') {
|
||||||
|
const appCheck = { id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 };
|
||||||
|
const checks = preDrillChecks ?? (workerUp
|
||||||
|
? [
|
||||||
|
appCheck,
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'pass', counters: { ok: 5, fail: 0 } },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 1, ok: 5, fail: 0 } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 800, counters: { ok: 4, fail: 0 } },
|
||||||
|
]
|
||||||
|
: [
|
||||||
|
appCheck,
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'fail', failureClass: 'worker.unavailable' },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'fail', failureClass: 'worker.unavailable', counters: { depth: 17 } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'fail', failureClass: 'worker.unavailable', counters: { timeout: 4 } },
|
||||||
|
]);
|
||||||
|
res.writeHead(200, { 'content-type': 'application/json' });
|
||||||
|
res.end(JSON.stringify({ checks }));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
res.writeHead(404, { 'content-type': 'text/plain' });
|
||||||
|
res.end('Not found');
|
||||||
|
});
|
||||||
|
await new Promise(resolve => app.listen(port, '127.0.0.1', resolve));
|
||||||
|
return {
|
||||||
|
origin,
|
||||||
|
close: () => new Promise(resolve => app.close(() => resolve())),
|
||||||
|
get outageSwitchCount() { return outageSwitchCount; },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
test('outage drill reads the genuine fixture baseline, flips once, and reports exactly one page alert plus manual fallback', async t => {
|
||||||
|
const server = await startDrillServer();
|
||||||
|
t.after(() => server.close());
|
||||||
|
|
||||||
|
const result = await runCli(['--drill-origin', server.origin]);
|
||||||
|
assert.equal(result.code, 0, result.stderr);
|
||||||
|
assert.match(result.stdout, /WORKER OUTAGE DRILL/);
|
||||||
|
assert.match(result.stdout, /pre-drill baseline: every required boundary check explicitly passing; alerts: none/i);
|
||||||
|
assert.match(result.stdout, /pre-drill: healthy \(0 page alerts\)/i);
|
||||||
|
assert.equal(server.outageSwitchCount, 1, 'drill must flip the simulated switch exactly once');
|
||||||
|
assert.match(result.stdout, /post-outage: exactly 1 alert/i);
|
||||||
|
assert.match(result.stdout, /PAGE worker\.unavailable on daily-2026-08-22\.1/);
|
||||||
|
assert.match(result.stdout, /manual fallback: AVAILABLE — degraded elsewhere; local journal remains usable/);
|
||||||
|
assert.match(result.stdout, /DRILL PASS/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('drill refuses unknown, degraded, malformed, and backlog-warning baselines with exit 2 before any POST', async t => {
|
||||||
|
const unhealthyBaselines = [
|
||||||
|
{
|
||||||
|
name: 'unknown telemetry',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'unknown', latencyMs: 12 },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'pass' },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 1 } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 800 },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: 'degraded check',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'degraded', failureClass: 'vision.timeout' },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 1 } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 800 },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: 'malformed check',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 },
|
||||||
|
{ id: 'BROKEN ID', boundary: 'api', status: 'pass' },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 1 } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 800 },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: 'backlog warning',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'pass' },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 12 } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 800 },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: 'missing boundary',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 1 } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 800 },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
];
|
||||||
|
for (const baseline of unhealthyBaselines) {
|
||||||
|
const server = await startDrillServer({ preDrillChecks: baseline.checks });
|
||||||
|
t.after(() => server.close());
|
||||||
|
|
||||||
|
const result = await runCli(['--drill-origin', server.origin]);
|
||||||
|
assert.equal(result.code, 2, `${baseline.name}: expected refusal exit 2, got ${result.code}`);
|
||||||
|
assert.match(result.stderr, /not a genuine drill baseline/i, `${baseline.name}: operator must be told why`);
|
||||||
|
assert.doesNotMatch(result.stdout, /DRILL PASS|WORKER OUTAGE DRILL/, `${baseline.name}: drill must never start`);
|
||||||
|
assert.equal(server.outageSwitchCount, 0, `${baseline.name}: switch must never be touched`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('drill refuses to flip an already-outaged fixture and never touches its switch', async t => {
|
||||||
|
const server = await startDrillServer({ startDown: true });
|
||||||
|
t.after(() => server.close());
|
||||||
|
|
||||||
|
const result = await runCli(['--drill-origin', server.origin]);
|
||||||
|
assert.equal(result.code, 2, `expected refusal exit 2, got ${result.code}`);
|
||||||
|
assert.match(result.stderr, /not healthy/i);
|
||||||
|
assert.doesNotMatch(result.stdout, /DRILL PASS/);
|
||||||
|
assert.equal(server.outageSwitchCount, 0, 'already-down fixture must not be flipped again');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('drill fails when the flip produces no genuine healthy-to-outage transition', async t => {
|
||||||
|
const server = await startDrillServer({ flipHasNoEffect: true });
|
||||||
|
t.after(() => server.close());
|
||||||
|
|
||||||
|
const result = await runCli(['--drill-origin', server.origin]);
|
||||||
|
assert.equal(result.code, 1);
|
||||||
|
assert.match(result.stdout, /no outage observed after flip/i);
|
||||||
|
assert.match(result.stdout, /DRILL FAIL/);
|
||||||
|
assert.equal(server.outageSwitchCount, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('drill refuses a credentialed origin even when its loopback target is alive', async t => {
|
||||||
|
const server = await startDrillServer();
|
||||||
|
t.after(() => server.close());
|
||||||
|
|
||||||
|
const credentialed = `http://user:pass@127.0.0.1:${new URL(server.origin).port}`;
|
||||||
|
const result = await runCli(['--drill-origin', credentialed]);
|
||||||
|
assert.equal(result.code, 2, `expected refusal exit 2, got ${result.code}`);
|
||||||
|
assert.match(result.stderr, /drill origin rejected/i);
|
||||||
|
assert.doesNotMatch(result.stdout, /DRILL PASS|WORKER OUTAGE DRILL/, 'hostile origin must never reach the drill');
|
||||||
|
assert.equal(server.outageSwitchCount, 0, 'credentialed origin must never touch the fixture');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('drill-origin accepts only validated loopback HTTP and rejects every hostile form without contacting it', async () => {
|
||||||
|
const hostileOrigins = [
|
||||||
|
'https://127.0.0.1:43600',
|
||||||
|
'http://localhost:43600',
|
||||||
|
'ftp://127.0.0.1:43600',
|
||||||
|
'file:///etc/passwd',
|
||||||
|
'http://0x7f.0.0.1:43600',
|
||||||
|
'http://2130706433:43600',
|
||||||
|
'http://[::1]:43600',
|
||||||
|
'http://[::ffff:127.0.0.1]:43600',
|
||||||
|
'http://[0:0:0:0:0:0:0:1]:43600',
|
||||||
|
'http://127.0.0.2:43600',
|
||||||
|
'http://10.0.0.5:43600',
|
||||||
|
'http://example.com:43600',
|
||||||
|
'http://127%2E0%2E0%2E1:43600',
|
||||||
|
'http://127.0.0.1:43600/drill',
|
||||||
|
'http://127.0.0.1:43600/?next=1',
|
||||||
|
'//127.0.0.1:43600',
|
||||||
|
'127.0.0.1:43600',
|
||||||
|
'not a url at all',
|
||||||
|
'null',
|
||||||
|
];
|
||||||
|
for (const origin of hostileOrigins) {
|
||||||
|
const result = await runCli(['--drill-origin', origin]);
|
||||||
|
assert.equal(result.code, 2, `expected refusal exit 2 for origin ${JSON.stringify(origin)}`);
|
||||||
|
assert.match(result.stderr, /drill origin rejected/i, `origin must be rejected as an origin, not merely unreachable: ${JSON.stringify(origin)}`);
|
||||||
|
assert.doesNotMatch(result.stdout, /DRILL PASS|WORKER OUTAGE DRILL/, `hostile origin must never reach the drill: ${JSON.stringify(origin)}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
291
tests/release-dashboard.test.js
Normal file
291
tests/release-dashboard.test.js
Normal file
|
|
@ -0,0 +1,291 @@
|
||||||
|
import test from 'node:test';
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
|
||||||
|
import { sanitizeEvidence, buildDashboard, evaluateAlerts, DEFAULT_ALERT_RULES } from '../src/release-observability.js';
|
||||||
|
|
||||||
|
const healthyEvidence = () => sanitizeEvidence({
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: 'daily-2026-08-22.1',
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'pass', counters: { ok: 40, fail: 0 } },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 0, ok: 30, fail: 0 } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 900, counters: { ok: 25, fail: 0, abstain: 3 } },
|
||||||
|
],
|
||||||
|
}).evidence;
|
||||||
|
|
||||||
|
test('dashboard separates app, api, queue, and model boundaries', () => {
|
||||||
|
const dashboard = buildDashboard(healthyEvidence());
|
||||||
|
assert.equal(dashboard.ok, true);
|
||||||
|
assert.deepEqual(Object.keys(dashboard.boundaries).sort(), ['api', 'app', 'model', 'queue']);
|
||||||
|
assert.equal(dashboard.boundaries.app.pass, 1);
|
||||||
|
assert.equal(dashboard.boundaries.model.pass, 1);
|
||||||
|
assert.equal(dashboard.boundaries.model.latencyMs, 900);
|
||||||
|
assert.equal(dashboard.totals.checks, 4);
|
||||||
|
assert.equal(dashboard.totals.fail, 0);
|
||||||
|
assert.deepEqual(dashboard.failureClasses, {});
|
||||||
|
assert.equal(dashboard.identity.releaseTag, 'daily-2026-08-22.1');
|
||||||
|
assert.equal(dashboard.identity.commit, 'ca31e6d38bec649407f63880504554c59f2878ae');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('dashboard summarizes privacy-safe failure classes without any payload text', () => {
|
||||||
|
const evidence = sanitizeEvidence({
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: 'daily-2026-08-22.1',
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'degraded', failureClass: 'vision.timeout' },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'pass' },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'fail', failureClass: 'model.error' },
|
||||||
|
{ id: 'model.inference.warmup', boundary: 'model', status: 'fail', failureClass: 'model.error' },
|
||||||
|
],
|
||||||
|
}).evidence;
|
||||||
|
const dashboard = buildDashboard(evidence);
|
||||||
|
assert.deepEqual(dashboard.failureClasses, { 'vision.timeout': 1, 'model.error': 2 });
|
||||||
|
assert.equal(dashboard.totals.fail, 2);
|
||||||
|
assert.equal(dashboard.totals.degraded, 1);
|
||||||
|
assert.equal(JSON.stringify(dashboard).includes('note'), false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('simulated worker outage fires exactly one actionable alert with owner, threshold, and runbook', () => {
|
||||||
|
const evidence = sanitizeEvidence({
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: 'daily-2026-08-22.1',
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 11 },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'fail', failureClass: 'worker.unavailable' },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'fail', failureClass: 'worker.unavailable', counters: { depth: 17 } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'fail', failureClass: 'worker.unavailable', counters: { timeout: 4 } },
|
||||||
|
],
|
||||||
|
}).evidence;
|
||||||
|
|
||||||
|
const dashboard = buildDashboard(evidence);
|
||||||
|
const alerts = evaluateAlerts(dashboard, DEFAULT_ALERT_RULES);
|
||||||
|
|
||||||
|
assert.equal(alerts.length, 1, `expected exactly one alert, got ${JSON.stringify(alerts)}`);
|
||||||
|
const alert = alerts[0];
|
||||||
|
assert.equal(alert.id, 'worker.outage');
|
||||||
|
assert.equal(alert.severity, 'page');
|
||||||
|
assert.equal(alert.owner, 'release-operator');
|
||||||
|
assert.match(alert.runbook, /docs\/RELEASE-OBSERVABILITY\.md#simulate-a-worker-outage/);
|
||||||
|
assert.match(alert.message, /worker\.unavailable/);
|
||||||
|
assert.match(alert.message, /daily-2026-08-22\.1/);
|
||||||
|
assert.equal(alert.count, 3);
|
||||||
|
assert.equal(alert.threshold, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('healthy evidence raises no alerts and every default rule carries owner, threshold, and runbook', () => {
|
||||||
|
assert.deepEqual(evaluateAlerts(buildDashboard(healthyEvidence()), DEFAULT_ALERT_RULES), []);
|
||||||
|
for (const rule of DEFAULT_ALERT_RULES) {
|
||||||
|
assert.ok(rule.owner && rule.owner.length > 0);
|
||||||
|
assert.ok(Number.isInteger(rule.threshold) && rule.threshold >= 1);
|
||||||
|
assert.match(rule.runbook, /^docs\/RELEASE-OBSERVABILITY\.md#/);
|
||||||
|
assert.ok(['page', 'warn'].includes(rule.severity));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('manual fallback stays available whenever any non-app check is unhealthy', () => {
|
||||||
|
const healthy = buildDashboard(healthyEvidence());
|
||||||
|
assert.deepEqual(healthy.manualFallback, { available: true, reason: 'none-required' });
|
||||||
|
|
||||||
|
const outage = buildDashboard(sanitizeEvidence({
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: 'daily-2026-08-22.1',
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'pass' },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'fail', failureClass: 'worker.unavailable' },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass' },
|
||||||
|
],
|
||||||
|
}).evidence);
|
||||||
|
assert.deepEqual(outage.manualFallback, {
|
||||||
|
available: true,
|
||||||
|
reason: 'local-journal',
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test('manual fallback is unavailable only when the app boundary itself is down', () => {
|
||||||
|
const evidence = sanitizeEvidence({
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: 'daily-2026-08-22.1',
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'fail' },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'fail', failureClass: 'worker.unavailable' },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'pass' },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass' },
|
||||||
|
],
|
||||||
|
}).evidence;
|
||||||
|
const dashboard = buildDashboard(evidence);
|
||||||
|
assert.deepEqual(dashboard.manualFallback, {
|
||||||
|
available: false,
|
||||||
|
reason: 'app-down',
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test('manualFallback state is a deterministic function of boundary health alone', () => {
|
||||||
|
const cases = [
|
||||||
|
{ statuses: ['pass', 'pass', 'pass', 'pass'], expected: { available: true, reason: 'none-required' } },
|
||||||
|
{ statuses: ['degraded', 'pass', 'pass', 'pass'], expected: { available: true, reason: 'local-journal' } },
|
||||||
|
{ statuses: ['pass', 'unknown', 'pass', 'pass'], expected: { available: null, reason: 'app-state-unknown' } },
|
||||||
|
{ statuses: ['fail', 'fail', 'fail', 'fail'], expected: { available: false, reason: 'app-down' } },
|
||||||
|
];
|
||||||
|
for (const { statuses, expected } of cases) {
|
||||||
|
const evidence = sanitizeEvidence({
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: 'daily-2026-08-22.1',
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: statuses[0] },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: statuses[1] },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: statuses[2] },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: statuses[3] },
|
||||||
|
],
|
||||||
|
}).evidence;
|
||||||
|
assert.deepEqual(buildDashboard(evidence).manualFallback, expected, `statuses ${statuses}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
const backlogEvidence = depth => sanitizeEvidence({
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: 'daily-2026-08-22.1',
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'pass' },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'degraded', counters: { depth } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass' },
|
||||||
|
],
|
||||||
|
}).evidence;
|
||||||
|
|
||||||
|
test('queue.depth counter survives sanitization so queue.backlog can fire', () => {
|
||||||
|
const dashboard = buildDashboard(backlogEvidence(10));
|
||||||
|
assert.equal(dashboard.boundaries.queue.counters.depth, 10);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('queue.backlog stays silent at depth 9 and fires at exactly depth 10', () => {
|
||||||
|
assert.deepEqual(evaluateAlerts(buildDashboard(backlogEvidence(9)), DEFAULT_ALERT_RULES), []);
|
||||||
|
const atThreshold = evaluateAlerts(buildDashboard(backlogEvidence(10)), DEFAULT_ALERT_RULES);
|
||||||
|
assert.equal(atThreshold.length, 1);
|
||||||
|
assert.equal(atThreshold[0].id, 'queue.backlog');
|
||||||
|
assert.equal(atThreshold[0].severity, 'warn');
|
||||||
|
assert.equal(atThreshold[0].count, 10);
|
||||||
|
assert.equal(atThreshold[0].threshold, 10);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('queue.backlog reports observed count above threshold and sums across queue checks', () => {
|
||||||
|
const evidence = sanitizeEvidence({
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: 'daily-2026-08-22.1',
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'pass' },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'degraded', counters: { depth: 5 } },
|
||||||
|
{ id: 'queue.depth.sidecar', boundary: 'queue', status: 'pass', counters: { depth: 6 } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass' },
|
||||||
|
],
|
||||||
|
}).evidence;
|
||||||
|
const alerts = evaluateAlerts(buildDashboard(evidence), DEFAULT_ALERT_RULES);
|
||||||
|
const backlog = alerts.filter(alert => alert.id === 'queue.backlog');
|
||||||
|
assert.equal(backlog.length, 1);
|
||||||
|
assert.equal(backlog[0].count, 11);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('queue.backlog is suppressed while worker.outage pages because outage depth is residual', () => {
|
||||||
|
const evidence = sanitizeEvidence({
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: 'daily-2026-08-22.1',
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'fail', failureClass: 'worker.unavailable' },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'fail', failureClass: 'worker.unavailable', counters: { depth: 17 } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'fail', failureClass: 'worker.unavailable' },
|
||||||
|
],
|
||||||
|
}).evidence;
|
||||||
|
const alerts = evaluateAlerts(buildDashboard(evidence), DEFAULT_ALERT_RULES);
|
||||||
|
assert.equal(alerts.length, 1, `expected only the outage page, got ${JSON.stringify(alerts)}`);
|
||||||
|
assert.equal(alerts[0].id, 'worker.outage');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('queue.bound depth counter above the bounded ceiling cannot smuggle into the dashboard', () => {
|
||||||
|
const evidence = sanitizeEvidence({
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: 'daily-2026-08-22.1',
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'pass' },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'degraded', counters: { depth: 10_000_000 } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass' },
|
||||||
|
],
|
||||||
|
}).evidence;
|
||||||
|
assert.equal(evidence, null, 'unbounded depth must fail closed at sanitization');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('sanitizer fails closed when any app, api, queue, or model boundary is entirely missing', () => {
|
||||||
|
const fullChecks = [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'pass' },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'pass' },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass' },
|
||||||
|
];
|
||||||
|
for (const missing of ['app', 'api', 'queue', 'model']) {
|
||||||
|
const evidence = sanitizeEvidence({
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: 'daily-2026-08-22.1',
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||||
|
checks: fullChecks.filter(check => check.boundary !== missing),
|
||||||
|
});
|
||||||
|
assert.equal(evidence.ok, false, `missing ${missing} boundary must fail closed`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('unknown statuses surface as a warn telemetry.gap alert instead of failing open', () => {
|
||||||
|
const evidence = sanitizeEvidence({
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: 'daily-2026-08-22.1',
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'unknown' },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'unknown' },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass' },
|
||||||
|
],
|
||||||
|
}).evidence;
|
||||||
|
assert.ok(evidence, 'unknown is an allowed status and must not fail sanitization');
|
||||||
|
const dashboard = buildDashboard(evidence);
|
||||||
|
assert.equal(dashboard.boundaries.api.unknown, 1);
|
||||||
|
const alerts = evaluateAlerts(dashboard, DEFAULT_ALERT_RULES);
|
||||||
|
assert.equal(alerts.length, 1, `expected exactly the telemetry.gap alert, got ${JSON.stringify(alerts)}`);
|
||||||
|
const gap = alerts[0];
|
||||||
|
assert.equal(gap.id, 'telemetry.gap');
|
||||||
|
assert.equal(gap.severity, 'warn');
|
||||||
|
assert.equal(gap.owner, 'release-operator');
|
||||||
|
assert.equal(gap.threshold, 1);
|
||||||
|
assert.equal(gap.count, 2);
|
||||||
|
assert.match(gap.runbook, /docs\/RELEASE-OBSERVABILITY\.md#/);
|
||||||
|
assert.match(gap.message, /telemetry\.gap/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('healthy evidence raises no telemetry.gap alert', () => {
|
||||||
|
const alerts = evaluateAlerts(buildDashboard(healthyEvidence()), DEFAULT_ALERT_RULES);
|
||||||
|
assert.equal(alerts.some(alert => alert.id === 'telemetry.gap'), false);
|
||||||
|
});
|
||||||
66
tests/release-observability-runbook.test.js
Normal file
66
tests/release-observability-runbook.test.js
Normal file
|
|
@ -0,0 +1,66 @@
|
||||||
|
import test from 'node:test';
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import { readFile } from 'node:fs/promises';
|
||||||
|
|
||||||
|
import { DEFAULT_ALERT_RULES } from '../src/release-observability.js';
|
||||||
|
|
||||||
|
const runbookPath = new URL('../docs/RELEASE-OBSERVABILITY.md', import.meta.url);
|
||||||
|
const packagePath = new URL('../package.json', import.meta.url);
|
||||||
|
|
||||||
|
test('every default alert rule is inventoried in the runbook with owner, threshold, and severity', async () => {
|
||||||
|
const runbook = await readFile(runbookPath, 'utf8');
|
||||||
|
const headingSlugs = [...runbook.matchAll(/^#{2,4} (.+)$/gm)]
|
||||||
|
.map(match => match[1].toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, ''));
|
||||||
|
for (const rule of DEFAULT_ALERT_RULES) {
|
||||||
|
const pattern = new RegExp(
|
||||||
|
`\\| \`${rule.id}\`\\s*\\| ${rule.severity}\\s*\\| ${rule.owner}\\s*\\|`,
|
||||||
|
);
|
||||||
|
assert.match(runbook, pattern, `runbook table lacks a conforming row for ${rule.id}`);
|
||||||
|
const anchor = rule.runbook.split('#')[1];
|
||||||
|
assert.ok(anchor && headingSlugs.includes(anchor), `runbook link for ${rule.id} does not resolve to a heading (${anchor})`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('runbook table states each rule\'s actual numeric threshold, matching DEFAULT_ALERT_RULES', async () => {
|
||||||
|
const runbook = await readFile(runbookPath, 'utf8');
|
||||||
|
const rows = Object.fromEntries(
|
||||||
|
runbook.split('\n')
|
||||||
|
.filter(line => line.startsWith('| `'))
|
||||||
|
.map(line => [line.slice(3, line.indexOf('`', 3)), line]),
|
||||||
|
);
|
||||||
|
for (const rule of DEFAULT_ALERT_RULES) {
|
||||||
|
// The threshold must appear as a real number in the rule's own table row,
|
||||||
|
// not merely somewhere in prose, so the documented contract cannot drift
|
||||||
|
// from what evaluateAlerts() enforces.
|
||||||
|
const row = rows[rule.id];
|
||||||
|
assert.ok(row, `runbook table has no row for ${rule.id}`);
|
||||||
|
assert.match(
|
||||||
|
row,
|
||||||
|
new RegExp(`\\| ${rule.threshold}\\b`),
|
||||||
|
`runbook row for ${rule.id} must state its real numeric threshold ${rule.threshold}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const backlogRule = DEFAULT_ALERT_RULES.find(rule => rule.id === 'queue.backlog');
|
||||||
|
assert.equal(backlogRule.threshold, 10, 'queue.backlog must fire at exactly depth 10');
|
||||||
|
assert.match(runbook, /depth >= 10/, 'runbook must state the queue.backlog boundary as depth >= 10');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('runbook documents the outage drill anchor, manual fallback, and privacy boundary', async () => {
|
||||||
|
const runbook = await readFile(runbookPath, 'utf8');
|
||||||
|
assert.match(runbook, /^## Simulate a worker outage$/m);
|
||||||
|
assert.match(runbook, /^## Alert inventory$/m);
|
||||||
|
assert.match(runbook, /manual fallback/i);
|
||||||
|
for (const forbidden of ['photos or medical imagery', 'stool records', 'session identifiers', 'environment dumps']) {
|
||||||
|
assert.match(runbook, new RegExp(forbidden.replace(/[-/\\^$*+?.()|[\]{}]/g, '\\$&'), 'i'));
|
||||||
|
}
|
||||||
|
const externalUrls = [...runbook.matchAll(/https?:\/\/(?!127\.0\.0\.1)[^\s)`\]]+/g)].map(match => match[0]);
|
||||||
|
assert.deepEqual(externalUrls, [], 'runbook must not reference non-loopback hosts');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('package scripts wire observability tests into the suite and syntax gate', async () => {
|
||||||
|
const packageJson = JSON.parse(await readFile(packagePath, 'utf8'));
|
||||||
|
assert.match(packageJson.scripts.test, /tests\/release-observability\.test\.js/);
|
||||||
|
assert.match(packageJson.scripts.test, /tests\/release-dashboard\.test\.js/);
|
||||||
|
assert.match(packageJson.scripts.test, /tests\/release-dashboard-cli\.test\.js/);
|
||||||
|
assert.match(packageJson.scripts['check:syntax'], /node --check scripts\/release_dashboard\.mjs/);
|
||||||
|
});
|
||||||
229
tests/release-observability.test.js
Normal file
229
tests/release-observability.test.js
Normal file
|
|
@ -0,0 +1,229 @@
|
||||||
|
import test from 'node:test';
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
|
||||||
|
import { sanitizeEvidence } from '../src/release-observability.js';
|
||||||
|
|
||||||
|
const validEvidence = () => ({
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: 'daily-2026-08-22.1',
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'pass', counters: { ok: 40, fail: 1 } },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { ok: 30, fail: 0 } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 900, counters: { ok: 25, fail: 0, abstain: 3 } },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
|
||||||
|
test('sanitizer keeps bounded sanitized release evidence intact', () => {
|
||||||
|
const result = sanitizeEvidence(validEvidence());
|
||||||
|
assert.equal(result.ok, true);
|
||||||
|
assert.deepEqual(result.evidence, {
|
||||||
|
schemaVersion: 1,
|
||||||
|
releaseTag: 'daily-2026-08-22.1',
|
||||||
|
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||||
|
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||||
|
checks: [
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass', failureClass: null, latencyMs: 12, counters: {} },
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'pass', failureClass: null, counters: { ok: 40, fail: 1 } },
|
||||||
|
{ id: 'queue.depth', boundary: 'queue', status: 'pass', failureClass: null, counters: { ok: 30, fail: 0 } },
|
||||||
|
{ id: 'model.inference', boundary: 'model', status: 'pass', failureClass: null, latencyMs: 900, counters: { ok: 25, fail: 0, abstain: 3 } },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test('sanitizer fail-closes evidence carrying session identifiers, photo payloads, credentials, and free-text notes', () => {
|
||||||
|
// Assembled at runtime so no contiguous private-key marker ever lands in Git history.
|
||||||
|
const privateKeyFixture = `-----BEGIN ${'OPENSSH'} PRIVATE KEY${'-----'}`;
|
||||||
|
// Forbidden-shaped fields are a rejection of the whole evidence object, not
|
||||||
|
// a silent drop: censoring a hostile check would dress the gap up as health.
|
||||||
|
const hostile = {
|
||||||
|
...validEvidence(),
|
||||||
|
sessionToken: 'sess_live_abc123',
|
||||||
|
adminCookie: 'timmy_agent=secret-cookie-value',
|
||||||
|
environmentDump: { NODE_ENV: 'production', SECRET_TOKEN: 'must-not-leak' },
|
||||||
|
operatorEmail: 'someone@example.com',
|
||||||
|
credentialFile: privateKeyFixture,
|
||||||
|
photos: [{ dataBase64: 'aGVsbG8gd29ybGQgaGVsbG8gd29ybGQ=' }],
|
||||||
|
};
|
||||||
|
assert.equal(sanitizeEvidence(hostile).ok, false, 'forbidden top-level fields must reject the whole evidence object');
|
||||||
|
|
||||||
|
const leakyCheck = {
|
||||||
|
id: 'leak',
|
||||||
|
boundary: 'app',
|
||||||
|
status: 'pass',
|
||||||
|
noteText: 'patient said blood at 3am, see photo hash e3b0c442',
|
||||||
|
imageHash: 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855',
|
||||||
|
base64Payload: 'aGVsbG8=',
|
||||||
|
authHeader: `Bearer ${'sk-'}live-abcdef1234567890`,
|
||||||
|
};
|
||||||
|
const withLeakyCheck = validEvidence();
|
||||||
|
withLeakyCheck.checks = [...validEvidence().checks, leakyCheck];
|
||||||
|
const result = sanitizeEvidence(withLeakyCheck);
|
||||||
|
assert.equal(result.ok, false, 'a check carrying forbidden or free-text fields must reject the whole evidence object');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('sanitizer fail-closes evidence carrying hostile failure-class payloads', () => {
|
||||||
|
const hostileFailureClasses = [
|
||||||
|
'worker.unavailable\nsecond line',
|
||||||
|
'worker.unavailable\r\ncarriage',
|
||||||
|
'\x1b[31mANSI-red',
|
||||||
|
'worker.unavailable\x1b[0m',
|
||||||
|
'bell\x07class',
|
||||||
|
'tab\tseparated',
|
||||||
|
'DROP TABLE users',
|
||||||
|
'patient reported blood at 3am',
|
||||||
|
'sess_live_abc123def456',
|
||||||
|
'-----BEGIN OPENSSH PRIVATE KEY-----',
|
||||||
|
'a@b.example.com',
|
||||||
|
'.leading.dot',
|
||||||
|
'trailing.dot.',
|
||||||
|
'double..dot',
|
||||||
|
'-leading-dash.d',
|
||||||
|
'd.trailing-dash-',
|
||||||
|
'-'.repeat(65),
|
||||||
|
42,
|
||||||
|
{},
|
||||||
|
[],
|
||||||
|
];
|
||||||
|
for (const failureClass of hostileFailureClasses) {
|
||||||
|
const evidence = validEvidence();
|
||||||
|
evidence.checks = [{ id: 'app.healthz', boundary: 'app', status: 'fail', failureClass }];
|
||||||
|
const result = sanitizeEvidence(evidence);
|
||||||
|
assert.equal(result.ok, false, `expected rejection for failureClass ${JSON.stringify(failureClass)}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('sanitizer accepts only the exact privacy-safe failure-class vocabulary in slug grammar', () => {
|
||||||
|
const baseChecks = validEvidence().checks;
|
||||||
|
for (const failureClass of ['worker.unavailable', 'vision.timeout', 'model.error']) {
|
||||||
|
const evidence = validEvidence();
|
||||||
|
evidence.checks = [
|
||||||
|
...baseChecks.filter(check => check.boundary !== 'api'),
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'degraded', failureClass },
|
||||||
|
];
|
||||||
|
const result = sanitizeEvidence(evidence);
|
||||||
|
assert.equal(result.ok, true, `vocabulary class ${failureClass} must survive sanitization`);
|
||||||
|
assert.equal(result.evidence.checks.some(check => check.failureClass === failureClass), true);
|
||||||
|
}
|
||||||
|
for (const failureClass of ['made.up.class', 'worker.somethingelse', 'VISION.TIMEOUT', 'vision..timeout']) {
|
||||||
|
const evidence = validEvidence();
|
||||||
|
evidence.checks = [
|
||||||
|
...baseChecks.filter(check => check.boundary !== 'api'),
|
||||||
|
{ id: 'api.analyze', boundary: 'api', status: 'degraded', failureClass },
|
||||||
|
];
|
||||||
|
const result = sanitizeEvidence(evidence);
|
||||||
|
assert.equal(result.ok, false, `non-vocabulary class ${failureClass} must be rejected`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('schemaVersion must be the JSON number 1 and nothing else', () => {
|
||||||
|
for (const hostileVersion of ['1', true, false, null, 1.5, 0, 2, [1], { value: 1 }]) {
|
||||||
|
const evidence = { ...validEvidence(), schemaVersion: hostileVersion };
|
||||||
|
const result = sanitizeEvidence(evidence);
|
||||||
|
assert.equal(result.ok, false, `schemaVersion ${JSON.stringify(hostileVersion)} must be rejected`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('counters and latency must be actual JSON numbers, never strings, booleans, nulls, or fractions', () => {
|
||||||
|
for (const counters of [{ ok: '40' }, { ok: true }, { fail: null }, { depth: 10.5 }, { depth: -1 }, { retry: Infinity }]) {
|
||||||
|
const evidence = validEvidence();
|
||||||
|
evidence.checks[0] = { ...evidence.checks[0], counters };
|
||||||
|
const result = sanitizeEvidence(evidence);
|
||||||
|
assert.equal(result.ok, false, `counters ${JSON.stringify(counters)} must be rejected`);
|
||||||
|
}
|
||||||
|
for (const latencyMs of ['12', '', true, false, null, 12.5, -1, Infinity]) {
|
||||||
|
const evidence = validEvidence();
|
||||||
|
evidence.checks[0] = { ...evidence.checks[0], latencyMs };
|
||||||
|
const result = sanitizeEvidence(evidence);
|
||||||
|
assert.equal(result.ok, false, `latencyMs ${JSON.stringify(latencyMs)} must be rejected`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('generatedAtUtc must be canonical real UTC that survives a Date round-trip', () => {
|
||||||
|
const invalidTimestamps = [
|
||||||
|
'2026-02-30T12:00:00Z',
|
||||||
|
'2026-08-22T25:00:00Z',
|
||||||
|
'2026-08-22T12:61:00Z',
|
||||||
|
'2026-13-01T00:00:00Z',
|
||||||
|
'2026-08-22T12:00:00+00:00',
|
||||||
|
'2026-08-22t12:00:00z',
|
||||||
|
'2026-08-22 12:00:00Z',
|
||||||
|
'not-a-timestamp',
|
||||||
|
1234567890,
|
||||||
|
true,
|
||||||
|
null,
|
||||||
|
];
|
||||||
|
for (const generatedAtUtc of invalidTimestamps) {
|
||||||
|
const evidence = { ...validEvidence(), generatedAtUtc };
|
||||||
|
const result = sanitizeEvidence(evidence);
|
||||||
|
assert.equal(result.ok, false, `generatedAtUtc ${JSON.stringify(generatedAtUtc)} must be rejected`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('releaseTag and commit must have their exact safe types', () => {
|
||||||
|
for (const releaseTag of [42, true, null, {}, [], '']) {
|
||||||
|
const evidence = { ...validEvidence(), releaseTag };
|
||||||
|
const result = sanitizeEvidence(evidence);
|
||||||
|
assert.equal(result.ok, false, `releaseTag ${JSON.stringify(releaseTag)} must be rejected`);
|
||||||
|
}
|
||||||
|
for (const commit of [0xca31e6d38bec, true, null, {}, []]) {
|
||||||
|
const evidence = { ...validEvidence(), commit };
|
||||||
|
const result = sanitizeEvidence(evidence);
|
||||||
|
assert.equal(result.ok, false, `commit ${JSON.stringify(commit)} must be rejected`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('one malformed check rejects the entire evidence object instead of vanishing beside healthy checks', () => {
|
||||||
|
const malformedChecks = [
|
||||||
|
{ boundary: 'app', status: 'pass' },
|
||||||
|
{ id: 'BAD_ID', boundary: 'app', status: 'pass' },
|
||||||
|
{ id: 42, boundary: 'app', status: 'pass' },
|
||||||
|
{ id: 'app.healthz', status: 'pass' },
|
||||||
|
{ id: 'app.healthz', boundary: 'bus', status: 'pass' },
|
||||||
|
{ id: 'app.healthz', boundary: 42, status: 'pass' },
|
||||||
|
{ id: 'app.healthz', boundary: 'app' },
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'healthy' },
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 42 },
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass', counters: 'many' },
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass', counters: [] },
|
||||||
|
{ id: 'app.healthz', boundary: 'app', status: 'pass', failureClass: 42 },
|
||||||
|
];
|
||||||
|
for (const malformed of malformedChecks) {
|
||||||
|
const evidence = validEvidence();
|
||||||
|
evidence.checks = [...validEvidence().checks, malformed];
|
||||||
|
const result = sanitizeEvidence(evidence);
|
||||||
|
assert.equal(result.ok, false, `malformed check ${JSON.stringify(malformed)} must reject the whole evidence object`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('a passing or unknown check can never carry a failureClass, so all-pass evidence can never page worker.outage', () => {
|
||||||
|
const baseChecks = validEvidence().checks;
|
||||||
|
for (const status of ['pass', 'unknown']) {
|
||||||
|
for (const failureClass of ['worker.unavailable', 'model.error', 'vision.timeout']) {
|
||||||
|
const evidence = validEvidence();
|
||||||
|
evidence.checks = baseChecks.map(check =>
|
||||||
|
check.boundary === 'app' ? { ...check, status, failureClass } : check,
|
||||||
|
);
|
||||||
|
const result = sanitizeEvidence(evidence);
|
||||||
|
assert.equal(
|
||||||
|
result.ok,
|
||||||
|
false,
|
||||||
|
`status ${status} carrying failureClass ${failureClass} must reject the whole evidence object`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('sanitization is idempotent: re-sanitizing sanitized evidence changes nothing and omits nullable latency', () => {
|
||||||
|
const first = sanitizeEvidence(validEvidence());
|
||||||
|
assert.equal(first.ok, true);
|
||||||
|
assert.equal(Object.hasOwn(first.evidence.checks[0], 'latencyMs'), true, 'present latency must stay present');
|
||||||
|
assert.equal(Object.hasOwn(first.evidence.checks[1], 'latencyMs'), false, 'omitted latency must stay omitted');
|
||||||
|
|
||||||
|
const second = sanitizeEvidence(first.evidence);
|
||||||
|
assert.equal(second.ok, true, 'sanitized evidence must survive a second sanitization pass');
|
||||||
|
assert.deepEqual(second.evidence, first.evidence, 'second pass must not rewrite omitted latency to a different shape');
|
||||||
|
assert.equal(JSON.stringify(second.evidence).includes('"latencyMs":0'), false, 'no zero-latency fabrication');
|
||||||
|
});
|
||||||
Loading…
Reference in New Issue
Block a user