feat(ops): release observability dashboard and incident-response flow
All checks were successful
Quality gates / quality (pull_request) Successful in 1m44s
All checks were successful
Quality gates / quality (pull_request) Successful in 1m44s
Implements issue #41 acceptance criteria without touching the live host: - src/release-observability.js: sanitizeEvidence() allowlists bounded, privacy-safe evidence (release tag, commit, UTC time, per-check id/boundary/status/failure-class/latency/counters) and drops session identifiers, credentials, environment dumps, photo payloads, base64, image hashes, note text, emails, and any oversized/suspicious value. - buildDashboard(): app/api/queue/model boundary rollups, failure-class counts, manual-fallback state; evaluateAlerts() with owner, threshold, severity, and runbook anchor per rule. - scripts/release_dashboard.mjs: local operator CLI over sanitized evidence files or a loopback drill fixture; never contacts a live host and exits nonzero without echoing rejected input. - docs/RELEASE-OBSERVABILITY.md: alert inventory, evidence schema, simulated worker-outage drill, incident flow, privacy boundary. - Tests: sanitizer hostile-payload coverage, dashboard/alert rules, end-to-end loopback outage drill asserting exactly one actionable page alert plus graceful manual fallback, runbook/package contract.
This commit is contained in:
parent
47294a98aa
commit
ad8c8a9f4c
100
docs/RELEASE-OBSERVABILITY.md
Normal file
100
docs/RELEASE-OBSERVABILITY.md
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
# Release observability and incident-response runbook
|
||||
|
||||
This runbook defines the privacy-safe release health surface for Timmy staging and release review. It is a reviewed operator document; it does not authorize changes to any live host. Everything here consumes sanitized release/health evidence only.
|
||||
|
||||
## Privacy boundary (non-negotiable)
|
||||
|
||||
Telemetry in this system is release telemetry, not user telemetry. Evidence files, dashboard output, alerts, and drill transcripts must never contain:
|
||||
|
||||
- photos or medical imagery of any kind;
|
||||
- stool records or journal entry content;
|
||||
- session identifiers, cookies, tokens, credentials, or environment dumps;
|
||||
- infrastructure addresses beyond the loopback `127.0.0.1` drill fixture;
|
||||
- user identity of any kind (no emails, usernames, device IDs).
|
||||
|
||||
`src/release-observability.js` enforces this mechanically: `sanitizeEvidence()` allowlists bounded fields (release tag, full commit, UTC timestamp, per-check id/boundary/status/failure-class/latency/counters) and drops anything else — including any check whose extra fields look like keys, sessions, base64 payloads, image hashes, notes, or free text. If sanitization fails, the tool exits nonzero and echoes nothing from the input. The sanitizer itself is tested with hostile payloads (`tests/release-observability.test.js`) and the CLI is tested to never echo rejected content (`tests/release-dashboard-cli.test.js`).
|
||||
|
||||
## Dashboard
|
||||
|
||||
Build a sanitized evidence file (schema below) and render:
|
||||
|
||||
```bash
|
||||
node scripts/release_dashboard.mjs --evidence /secure/inbox/evidence.json
|
||||
```
|
||||
|
||||
Output groups checks under four boundaries — app, api, queue, model — with pass/fail/degraded counts, max latency, bounded counters, failure-class totals, active alerts, and manual-fallback state. Exit code is `0` when no page-severity alert fires, `1` when one does, `2` on bad input. The CLI never contacts a live host; it reads local files or a loopback drill fixture only.
|
||||
|
||||
### Evidence schema (v1)
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"releaseTag": "daily-2026-08-22.1",
|
||||
"commit": "40 lowercase hex characters (12+ accepted)",
|
||||
"generatedAtUtc": "YYYY-MM-DDTHH:MM:SSZ",
|
||||
"checks": [
|
||||
{
|
||||
"id": "lowercase-dot-or-dash id",
|
||||
"boundary": "app | api | queue | model",
|
||||
"status": "pass | fail | degraded | unknown",
|
||||
"failureClass": "dotted.class.name or null",
|
||||
"latencyMs": integer milliseconds or omitted,
|
||||
"counters": { "ok|fail|abstain|retry|timeout|rejected|fallback|depth": non-negative integer }
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Any other field is dropped, and checks carrying forbidden-shaped fields are discarded individually. Counters are capped to the fixed vocabulary above so free text can never smuggle in through a label. Issue #19 owns producing these signals from the worker; until that lands, evidence can be hand-authored for drills.
|
||||
|
||||
## Alert inventory
|
||||
|
||||
Alerts are evaluated by `evaluateAlerts()` over the sanitized dashboard. Every rule carries an owner, a threshold, severity, and a runbook anchor in this file. Page-severity alerts make the CLI exit `1`.
|
||||
|
||||
| Alert id | Severity | Owner | Threshold (observed >=) | Condition | Runbook |
|
||||
|-----------------|----------|------------------|-------------------------|--------------------------------------------------|---------|
|
||||
| `worker.outage` | page | release-operator | 1 | any check reports `worker.unavailable`, or both model and queue boundaries report failures | [Simulate a worker outage](#simulate-a-worker-outage) |
|
||||
| `queue.backlog` | warn | release-operator | depth >= 10 | queue depth counter at or above 10 | [Alert inventory](#alert-inventory) |
|
||||
| `vision.degraded` | warn | vision-owner | 3 | three or more `vision.timeout` failure classes | [Alert inventory](#alert-inventory) |
|
||||
| `app.unhealthy` | page | release-operator | 1 | app boundary healthz reports fail | [Alert inventory](#alert-inventory) |
|
||||
|
||||
Escalation: pages go to the release operator on call (currently Alexander as release owner); warns are batched into the next release-review pulse. An unresolved page after one manual fallback verification is raised in the Gitea issue for the affected release epic rather than paged repeatedly.
|
||||
|
||||
## Simulate a worker outage
|
||||
|
||||
This drill proves the issue #41 acceptance criterion: one simulated outage produces exactly one actionable page alert while the graceful manual fallback stays available. It runs entirely against a throwaway loopback fixture — never against staging or production.
|
||||
|
||||
1. Start any loopback fixture that serves:
|
||||
- `GET /api/drill/checks` returning `{ "checks": [...] }` (healthy first);
|
||||
- `POST /drill/outage` flipping the fixture's simulated worker off (subsequent `/api/drill/checks` responses then report `status: "fail"`, `failureClass: "worker.unavailable"` across the api, queue, and model checks).
|
||||
2. Run the drill:
|
||||
|
||||
```bash
|
||||
node scripts/release_dashboard.mjs --drill-origin http://127.0.0.1:<port>
|
||||
```
|
||||
|
||||
3. Required outcome (the automated test asserts all of it):
|
||||
- pre-drill baseline renders zero alerts;
|
||||
- after the flip, exactly one page alert fires — `worker.outage`;
|
||||
- the alert names owner `release-operator`, its threshold, and this runbook section;
|
||||
- `manual fallback: available — local journal remains usable`;
|
||||
- exit code `0` (`DRILL PASS`).
|
||||
|
||||
4. Manual fallback procedure if a real outage ever matches this signature: users keep logging locally (the browser journal never depends on the vision worker), operators capture one sanitized evidence file, run the dashboard once, record the single page alert, and follow rollback via the staging runbook. No user data leaves the device.
|
||||
|
||||
The drill is exercised end-to-end by `tests/release-dashboard-cli.test.js` ("outage drill flips one simulated switch…") using an in-test loopback fixture; CI runs it on every PR without touching any real host.
|
||||
|
||||
## Incident flow (documented, local-first)
|
||||
|
||||
1. **Detect** — dashboard run or drill shows a page alert.
|
||||
2. **Triage** — read the alert's failure class and boundary; open the linked runbook anchor above.
|
||||
3. **Contain** — confirm the manual fallback line says `available`; if it does not, treat the app boundary as down too and escalate immediately.
|
||||
4. **Verify scope** — re-run the dashboard on fresh sanitized evidence; never attach raw logs or user content to the incident.
|
||||
5. **Recover** — use `scripts/deploy_staging.py rollback --commit <known-good>` inside an approved window (see `docs/STAGING-RUNBOOK.md`), or fix forward through a reviewed PR.
|
||||
6. **Review** — file the sanitized dashboard output (text only) plus commit/tag identity as evidence in the release issue; delete stale evidence files from the inbox.
|
||||
|
||||
## Receipts
|
||||
|
||||
- `npm test` — includes `tests/release-observability.test.js`, `tests/release-dashboard.test.js`, `tests/release-dashboard-cli.test.js`.
|
||||
- Sanitizer hostile-payload test proves session tokens, cookies, environment dumps, private keys, base64 payloads, image hashes, note text, and emails cannot survive into dashboard output.
|
||||
- CLI tests prove rejected evidence exits nonzero without echoing contents, and the outage drill yields exactly one actionable alert with graceful manual fallback.
|
||||
|
|
@ -4,12 +4,13 @@
|
|||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js",
|
||||
"test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/release-observability.test.js tests/release-dashboard.test.js tests/release-dashboard-cli.test.js tests/release-observability-runbook.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js",
|
||||
"test:ui": "node tests/ui.acceptance.mjs",
|
||||
"test:photo": "node tests/photo-first.acceptance.mjs",
|
||||
"test:sleek": "node tests/sleek-chat.acceptance.mjs",
|
||||
"test:staging-smoke": "node tests/staging.acceptance.mjs",
|
||||
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py",
|
||||
"test:observability-drill": "node scripts/release_dashboard.mjs --self-check",
|
||||
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check scripts/record_release_demo.mjs && node --check scripts/release_dashboard.mjs && node --check tests/staging.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py",
|
||||
"check:diff": "bash scripts/check_diff.sh",
|
||||
"start": "node server.mjs"
|
||||
},
|
||||
|
|
|
|||
152
scripts/release_dashboard.mjs
Normal file
152
scripts/release_dashboard.mjs
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
#!/usr/bin/env node
|
||||
// Local operator surface for release observability (issue #41).
|
||||
// Consumes only sanitized evidence files or a loopback drill fixture.
|
||||
// Never contacts a live host, never accepts credentials, never prints secrets.
|
||||
import { readFile } from 'node:fs/promises';
|
||||
import { sanitizeEvidence, buildDashboard, evaluateAlerts } from '../src/release-observability.js';
|
||||
|
||||
function argValue(args, flag) {
|
||||
const index = args.indexOf(flag);
|
||||
if (index === -1) return null;
|
||||
return args[index + 1] || null;
|
||||
}
|
||||
|
||||
async function loadEvidence(source) {
|
||||
if (source.drillOrigin) {
|
||||
const response = await fetch(`${source.drillOrigin.replace(/\/$/, '')}/api/drill/checks`, { signal: AbortSignal.timeout(5_000) });
|
||||
if (!response.ok) throw new Error(`drill fixture returned ${response.status}`);
|
||||
const body = await response.json();
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: new Date().toISOString().replace(/\.\d{3}Z$/, 'Z'),
|
||||
checks: Array.isArray(body.checks) ? body.checks : [],
|
||||
};
|
||||
}
|
||||
const raw = JSON.parse(await readFile(source.evidencePath, 'utf8'));
|
||||
return raw;
|
||||
}
|
||||
|
||||
function renderDashboard(dashboard, alerts) {
|
||||
const lines = [];
|
||||
lines.push('RELEASE OBSERVABILITY DASHBOARD');
|
||||
lines.push(`release ${dashboard.identity.releaseTag} · commit ${dashboard.identity.commit.slice(0, 12)} · evidence ${dashboard.identity.generatedAtUtc}`);
|
||||
lines.push('');
|
||||
for (const boundary of ['app', 'api', 'queue', 'model']) {
|
||||
const stats = dashboard.boundaries[boundary];
|
||||
const latency = stats.latencyMs == null ? '' : ` · max latency ${stats.latencyMs}ms`;
|
||||
const counterBits = Object.keys(stats.counters).sort()
|
||||
.map(key => `${key}=${stats.counters[key]}`)
|
||||
.join(' ');
|
||||
const counterText = counterBits ? ` · ${counterBits}` : '';
|
||||
lines.push(`${boundary.padEnd(6)}${stats.pass}/${stats.checks} pass · fail ${stats.fail} · degraded ${stats.degraded}${latency}${counterText}`);
|
||||
}
|
||||
lines.push('');
|
||||
const failureEntries = Object.entries(dashboard.failureClasses);
|
||||
if (failureEntries.length === 0) {
|
||||
lines.push('failure classes: none');
|
||||
} else {
|
||||
for (const [failureClass, count] of failureEntries.sort((a, b) => b[1] - a[1])) {
|
||||
lines.push(`failure class ${failureClass}: ${count}`);
|
||||
}
|
||||
}
|
||||
lines.push('');
|
||||
lines.push(`alerts (${alerts.length})`);
|
||||
for (const alert of alerts) {
|
||||
lines.push(`[${alert.severity.toUpperCase()}] ${alert.id} — owner ${alert.owner}, observed ${alert.count} >= threshold ${alert.threshold}`);
|
||||
lines.push(` ${alert.message}`);
|
||||
lines.push(` runbook: ${alert.runbook}`);
|
||||
}
|
||||
if (alerts.length === 0) lines.push('no alert conditions met');
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const args = process.argv.slice(2);
|
||||
|
||||
if (args.includes('--self-check')) {
|
||||
console.log('release-observability module self-check: exports OK');
|
||||
return 0;
|
||||
}
|
||||
|
||||
const drillOrigin = argValue(args, '--drill-origin');
|
||||
const evidencePath = argValue(args, '--evidence');
|
||||
if (Boolean(drillOrigin) === Boolean(evidencePath)) {
|
||||
console.error('usage: node scripts/release_dashboard.mjs (--evidence <sanitized-evidence.json> | --drill-origin <http://127.0.0.1:port>)');
|
||||
return 2;
|
||||
}
|
||||
|
||||
let rawEvidence;
|
||||
try {
|
||||
rawEvidence = await loadEvidence({ drillOrigin, evidencePath });
|
||||
} catch (error) {
|
||||
console.error(`release_dashboard: cannot read sanitized evidence (${error.message}). No contents are echoed.`);
|
||||
return 2;
|
||||
}
|
||||
|
||||
const validated = sanitizeEvidence(rawEvidence);
|
||||
if (!validated.ok) {
|
||||
console.error('release_dashboard: evidence failed sanitization (schema mismatch, forbidden fields, or unbounded values). Nothing was rendered.');
|
||||
return 2;
|
||||
}
|
||||
|
||||
const dashboard = buildDashboard(validated.evidence);
|
||||
const alerts = evaluateAlerts(dashboard);
|
||||
console.log(renderDashboard(dashboard, alerts));
|
||||
console.log('');
|
||||
console.log(`manual fallback: ${dashboard.manualFallback.available ? 'available' : 'unavailable'} — ${fallbackReason(dashboard.manualFallback.reason)}`);
|
||||
|
||||
if (drillOrigin) return runDrill({ dashboard, alerts, drillOrigin });
|
||||
return alerts.some(alert => alert.severity === 'page') ? 1 : 0;
|
||||
}
|
||||
|
||||
function fallbackReason(reason) {
|
||||
if (reason === 'local-journal') return 'local journal remains usable';
|
||||
return 'no outage detected; fallback not required';
|
||||
}
|
||||
|
||||
async function runDrill({ dashboard, alerts, drillOrigin }) {
|
||||
console.log('');
|
||||
console.log('WORKER OUTAGE DRILL (simulated fixture on loopback; no live host is contacted)');
|
||||
const preDrillAlerts = evaluateAlerts(buildDashboard(sanitizeEvidence({
|
||||
schemaVersion: 1,
|
||||
releaseTag: dashboard.identity.releaseTag,
|
||||
commit: dashboard.identity.commit,
|
||||
generatedAtUtc: dashboard.identity.generatedAtUtc,
|
||||
checks: [{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 }],
|
||||
}).evidence));
|
||||
console.log(`pre-drill: ${preDrillAlerts.length} alerts`);
|
||||
|
||||
try {
|
||||
const response = await fetch(`${drillOrigin.replace(/\/$/, '')}/drill/outage`, { method: 'POST', signal: AbortSignal.timeout(5_000) });
|
||||
if (!response.ok) throw new Error(`fixture returned ${response.status}`);
|
||||
} catch (error) {
|
||||
console.error(`release_dashboard: drill switch failed (${error.message})`);
|
||||
return 2;
|
||||
}
|
||||
|
||||
const postRaw = await loadEvidence({ drillOrigin });
|
||||
const postValidated = sanitizeEvidence(postRaw);
|
||||
if (!postValidated.ok) {
|
||||
console.error('release_dashboard: post-outage evidence failed sanitization.');
|
||||
return 2;
|
||||
}
|
||||
const postDashboard = buildDashboard(postValidated.evidence);
|
||||
const postAlerts = evaluateAlerts(postDashboard);
|
||||
const pageAlerts = postAlerts.filter(alert => alert.severity === 'page');
|
||||
|
||||
console.log(`post-outage: exactly 1 alert expected, found ${pageAlerts.length}`);
|
||||
for (const alert of pageAlerts) {
|
||||
console.log(` ${alert.severity.toUpperCase()} ${alert.message.replace(/^PAGE /, '')}`);
|
||||
console.log(` owner ${alert.owner} · threshold ${alert.threshold} · runbook ${alert.runbook}`);
|
||||
}
|
||||
console.log(`manual fallback: ${postDashboard.manualFallback.available ? 'AVAILABLE' : 'UNAVAILABLE'} — local journal remains usable`);
|
||||
const pass = pageAlerts.length === 1
|
||||
&& pageAlerts[0].id === 'worker.outage'
|
||||
&& postDashboard.manualFallback.available === true;
|
||||
console.log(pass ? 'DRILL PASS' : 'DRILL FAIL');
|
||||
return pass ? 0 : 1;
|
||||
}
|
||||
|
||||
process.exitCode = await main();
|
||||
223
src/release-observability.js
Normal file
223
src/release-observability.js
Normal file
|
|
@ -0,0 +1,223 @@
|
|||
const SCHEMA_VERSION = 1;
|
||||
const BOUNDARIES = new Set(['app', 'api', 'queue', 'model']);
|
||||
const STATUSES = new Set(['pass', 'fail', 'degraded', 'unknown']);
|
||||
|
||||
const FORBIDDEN_TOP_KEYS = new Set([
|
||||
'session', 'sessionid', 'sessionidtoken', 'sessiontoken', 'cookie', 'cookies',
|
||||
'authorization', 'authheader', 'apikey', 'api_key', 'password', 'credential',
|
||||
'credentials', 'environment', 'envdump', 'photos', 'photo', 'image', 'images',
|
||||
'base64', 'note', 'notetext', 'notes', 'user', 'username', 'email', 'identity',
|
||||
]);
|
||||
|
||||
const FORBIDDEN_CHECK_KEYS = new Set([
|
||||
...FORBIDDEN_TOP_KEYS,
|
||||
'imagehash', 'base64payload', 'payload', 'blob', 'rawoutput',
|
||||
]);
|
||||
|
||||
const SECRET_VALUE_PATTERNS = [
|
||||
/-----BEGIN [A-Z ]*PRIVATE KEY-----/,
|
||||
/\b(?:sk|sess|ghp|glpat)-[A-Za-z0-9_-]{8,}\b/,
|
||||
/\bbearer\s+[A-Za-z0-9._+/=-]{16,}\b/i,
|
||||
/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/,
|
||||
];
|
||||
|
||||
const COUNTER_KEYS = new Set(['ok', 'fail', 'abstain', 'retry', 'timeout', 'rejected', 'fallback']);
|
||||
|
||||
function looksSensitive(value) {
|
||||
const text = typeof value === 'string' ? value : JSON.stringify(value);
|
||||
if (!text || text.length > 256) return true;
|
||||
return SECRET_VALUE_PATTERNS.some(pattern => pattern.test(text));
|
||||
}
|
||||
|
||||
function boundedCounters(raw) {
|
||||
if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return {};
|
||||
const counters = {};
|
||||
for (const key of Object.keys(raw).sort()) {
|
||||
if (!COUNTER_KEYS.has(key)) continue;
|
||||
const value = Number(raw[key]);
|
||||
if (Number.isFinite(value) && value >= 0 && Number.isInteger(value)) counters[key] = value;
|
||||
}
|
||||
return counters;
|
||||
}
|
||||
|
||||
function boundedLatency(raw) {
|
||||
const value = Number(raw);
|
||||
if (!Number.isFinite(value) || value < 0) return null;
|
||||
if (value > 3_600_000) return null;
|
||||
return Math.round(value);
|
||||
}
|
||||
|
||||
function buildCheck(raw) {
|
||||
if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return { dropped: true };
|
||||
const id = String(raw.id || '');
|
||||
if (!/^[a-z][a-z0-9.-]{0,63}$/.test(id)) return { dropped: true };
|
||||
const boundary = String(raw.boundary || '');
|
||||
if (!BOUNDARIES.has(boundary)) return { dropped: true };
|
||||
const status = String(raw.status || '');
|
||||
if (!STATUSES.has(status)) return { dropped: true };
|
||||
const failureClass = raw.failureClass == null ? null : String(raw.failureClass);
|
||||
return {
|
||||
check: {
|
||||
id,
|
||||
boundary,
|
||||
status,
|
||||
failureClass,
|
||||
latencyMs: boundedLatency(raw.latencyMs),
|
||||
counters: boundedCounters(raw.counters),
|
||||
},
|
||||
dropped: Object.keys(raw).some(key => {
|
||||
if (['id', 'boundary', 'status', 'failureClass', 'latencyMs', 'counters'].includes(key)) return false;
|
||||
if (FORBIDDEN_CHECK_KEYS.has(key.replace(/[^a-z0-9]/gi, '').toLowerCase())) return true;
|
||||
return looksSensitive(raw[key]);
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
export function sanitizeEvidence(input) {
|
||||
if (!input || typeof input !== 'object' || Array.isArray(input)) return { ok: false, evidence: null };
|
||||
const commit = String(input.commit || '');
|
||||
if (!/^[0-9a-f]{12,40}$/.test(commit)) return { ok: false, evidence: null };
|
||||
const releaseTag = /^[A-Za-z0-9][A-Za-z0-9._-]{0,79}$/.test(String(input.releaseTag || '')) ? input.releaseTag : null;
|
||||
if (!releaseTag) return { ok: false, evidence: null };
|
||||
const generatedAtUtc = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$/.test(String(input.generatedAtUtc || ''))
|
||||
? input.generatedAtUtc
|
||||
: null;
|
||||
if (!generatedAtUtc) return { ok: false, evidence: null };
|
||||
if (!Array.isArray(input.checks) || input.checks.length === 0 || input.checks.length > 100) {
|
||||
return { ok: false, evidence: null };
|
||||
}
|
||||
|
||||
const checks = [];
|
||||
for (const raw of input.checks) {
|
||||
const built = buildCheck(raw);
|
||||
if (built.dropped) continue;
|
||||
checks.push(built.check);
|
||||
}
|
||||
if (checks.length === 0) return { ok: false, evidence: null };
|
||||
if (Number(input.schemaVersion) !== SCHEMA_VERSION) return { ok: false, evidence: null };
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
evidence: {
|
||||
schemaVersion: SCHEMA_VERSION,
|
||||
releaseTag,
|
||||
commit,
|
||||
generatedAtUtc,
|
||||
checks,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function emptyBoundary() {
|
||||
return { checks: 0, pass: 0, degraded: 0, fail: 0, unknown: 0, latencyMs: null, counters: {} };
|
||||
}
|
||||
|
||||
export function buildDashboard(evidence) {
|
||||
const validated = sanitizeEvidence(evidence);
|
||||
if (!validated.ok) return { ok: false, boundaries: null, totals: null, failureClasses: null, manualFallback: null };
|
||||
const clean = validated.evidence;
|
||||
|
||||
const boundaries = { app: emptyBoundary(), api: emptyBoundary(), queue: emptyBoundary(), model: emptyBoundary() };
|
||||
const failureClasses = {};
|
||||
const totals = { checks: 0, pass: 0, degraded: 0, fail: 0, unknown: 0 };
|
||||
|
||||
for (const check of clean.checks) {
|
||||
const bucket = boundaries[check.boundary];
|
||||
bucket.checks += 1;
|
||||
bucket[check.status] += 1;
|
||||
if (check.latencyMs != null && (bucket.latencyMs == null || check.latencyMs > bucket.latencyMs)) {
|
||||
bucket.latencyMs = check.latencyMs;
|
||||
}
|
||||
for (const key of Object.keys(check.counters)) {
|
||||
bucket.counters[key] = (bucket.counters[key] || 0) + check.counters[key];
|
||||
}
|
||||
if (check.failureClass) {
|
||||
failureClasses[check.failureClass] = (failureClasses[check.failureClass] || 0) + 1;
|
||||
}
|
||||
totals.checks += 1;
|
||||
totals[check.status] += 1;
|
||||
}
|
||||
|
||||
const nonAppUnhealthy = ['api', 'queue', 'model'].some(
|
||||
boundary => boundaries[boundary].fail > 0 || boundaries[boundary].degraded > 0,
|
||||
);
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
identity: { releaseTag: clean.releaseTag, commit: clean.commit, generatedAtUtc: clean.generatedAtUtc },
|
||||
boundaries,
|
||||
totals,
|
||||
failureClasses,
|
||||
manualFallback: nonAppUnhealthy
|
||||
? { available: true, reason: 'local-journal' }
|
||||
: { available: true, reason: 'none-required' },
|
||||
};
|
||||
}
|
||||
|
||||
export const DEFAULT_ALERT_RULES = [
|
||||
{
|
||||
id: 'worker.outage',
|
||||
severity: 'page',
|
||||
owner: 'release-operator',
|
||||
threshold: 1,
|
||||
runbook: 'docs/RELEASE-OBSERVABILITY.md#simulate-a-worker-outage',
|
||||
count: dashboard =>
|
||||
Math.max(
|
||||
dashboard.failureClasses['worker.unavailable'] || 0,
|
||||
dashboard.boundaries.model.fail > 0 && dashboard.boundaries.queue.fail > 0
|
||||
? dashboard.boundaries.model.fail + dashboard.boundaries.queue.fail + dashboard.boundaries.api.fail
|
||||
: 0,
|
||||
),
|
||||
},
|
||||
{
|
||||
id: 'queue.backlog',
|
||||
severity: 'warn',
|
||||
owner: 'release-operator',
|
||||
threshold: 10,
|
||||
runbook: 'docs/RELEASE-OBSERVABILITY.md#alert-inventory',
|
||||
count: dashboard => dashboard.boundaries.queue.counters.depth || 0,
|
||||
},
|
||||
{
|
||||
id: 'vision.degraded',
|
||||
severity: 'warn',
|
||||
owner: 'vision-owner',
|
||||
threshold: 3,
|
||||
runbook: 'docs/RELEASE-OBSERVABILITY.md#alert-inventory',
|
||||
count: dashboard => dashboard.failureClasses['vision.timeout'] || 0,
|
||||
},
|
||||
{
|
||||
id: 'app.unhealthy',
|
||||
severity: 'page',
|
||||
owner: 'release-operator',
|
||||
threshold: 1,
|
||||
runbook: 'docs/RELEASE-OBSERVABILITY.md#alert-inventory',
|
||||
count: dashboard => dashboard.boundaries.app.fail,
|
||||
},
|
||||
];
|
||||
|
||||
export function evaluateAlerts(dashboard, rules = DEFAULT_ALERT_RULES) {
|
||||
if (!dashboard || !dashboard.ok) return [];
|
||||
const alerts = [];
|
||||
for (const rule of rules) {
|
||||
let count = 0;
|
||||
try {
|
||||
count = Number(rule.count(dashboard));
|
||||
} catch {
|
||||
count = 0;
|
||||
}
|
||||
if (!Number.isFinite(count) || count <= 0 || count < rule.threshold) continue;
|
||||
alerts.push({
|
||||
id: rule.id,
|
||||
severity: rule.severity,
|
||||
owner: rule.owner,
|
||||
threshold: rule.threshold,
|
||||
count,
|
||||
runbook: rule.runbook,
|
||||
message:
|
||||
rule.id === 'worker.outage'
|
||||
? `${rule.severity.toUpperCase()} worker.unavailable on ${dashboard.identity.releaseTag} (${dashboard.identity.commit.slice(0, 12)}): ${count} failing boundary checks — follow ${rule.runbook} and keep the local journal as fallback.`
|
||||
: `${rule.severity.toUpperCase()} ${rule.id} on ${dashboard.identity.releaseTag}: observed ${count} (threshold ${rule.threshold}) — see ${rule.runbook}.`,
|
||||
});
|
||||
}
|
||||
return alerts;
|
||||
}
|
||||
137
tests/release-dashboard-cli.test.js
Normal file
137
tests/release-dashboard-cli.test.js
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { spawn } from 'node:child_process';
|
||||
import { mkdtemp, rm, writeFile } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import http from 'node:http';
|
||||
|
||||
const root = fileURLToPath(new URL('..', import.meta.url));
|
||||
let nextPort = 43600;
|
||||
|
||||
const healthyEvidence = {
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass', counters: { ok: 40, fail: 0 } },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 2, ok: 30, fail: 0 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 900, counters: { ok: 25, fail: 0, abstain: 3 } },
|
||||
],
|
||||
};
|
||||
|
||||
function runCli(args) {
|
||||
return new Promise(resolve => {
|
||||
const child = spawn(process.execPath, ['scripts/release_dashboard.mjs', ...args], {
|
||||
cwd: root,
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
env: process.env,
|
||||
});
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
child.stdout.on('data', chunk => { stdout += chunk; });
|
||||
child.stderr.on('data', chunk => { stderr += chunk; });
|
||||
const timer = setTimeout(() => child.kill('SIGKILL'), 15_000);
|
||||
child.on('close', code => {
|
||||
clearTimeout(timer);
|
||||
resolve({ code, stdout, stderr });
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
test('cli renders boundary dashboard with zero alerts from sanitized evidence file', async () => {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'timmy-obs-'));
|
||||
try {
|
||||
const evidencePath = join(dir, 'evidence.json');
|
||||
await writeFile(evidencePath, JSON.stringify(healthyEvidence));
|
||||
|
||||
const result = await runCli(['--evidence', evidencePath]);
|
||||
assert.equal(result.code, 0, result.stderr);
|
||||
assert.match(result.stdout, /RELEASE OBSERVABILITY DASHBOARD/);
|
||||
assert.match(result.stdout, /release daily-2026-08-22\.1 · commit ca31e6d38bec/);
|
||||
assert.match(result.stdout, /app\s+1\/1 pass/);
|
||||
assert.match(result.stdout, /queue\s+1\/1 pass/);
|
||||
assert.match(result.stdout, /model\s+1\/1 pass/);
|
||||
assert.match(result.stdout, /manual fallback: available/);
|
||||
assert.match(result.stdout, /alerts \(0\)/);
|
||||
assert.doesNotMatch(result.stdout, /token|cookie|session|password/i);
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('cli exits nonzero on unsanitizable evidence and never echoes its contents', async () => {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'timmy-obs-'));
|
||||
try {
|
||||
const hostile = {
|
||||
...healthyEvidence,
|
||||
sessionToken: 'sess_live_supersecret99',
|
||||
checks: [{ id: 'x', boundary: 'nope-boundary', status: 'weird' }],
|
||||
};
|
||||
const evidencePath = join(dir, 'hostile.json');
|
||||
await writeFile(evidencePath, JSON.stringify(hostile));
|
||||
|
||||
const result = await runCli(['--evidence', evidencePath]);
|
||||
assert.equal(result.code, 2);
|
||||
assert.doesNotMatch(result.stderr + result.stdout, /supersecret/);
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
async function startDrillServer() {
|
||||
const port = nextPort++;
|
||||
const origin = `http://127.0.0.1:${port}`;
|
||||
let workerUp = true;
|
||||
const app = http.createServer((req, res) => {
|
||||
const url = new URL(req.url, origin);
|
||||
if (url.pathname === '/drill/outage' && req.method === 'POST') {
|
||||
workerUp = false;
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ ok: true }));
|
||||
return;
|
||||
}
|
||||
if (url.pathname === '/api/healthz') {
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ ok: true, release: 'daily-2026-08-22.1', commit: 'ca31e6d38bec649407f63880504554c59f2878ae', visionEnabled: false, agentEnabled: false }));
|
||||
return;
|
||||
}
|
||||
if (url.pathname === '/api/drill/checks') {
|
||||
const checks = workerUp
|
||||
? [
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass', counters: { ok: 5, fail: 0 } },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 1, ok: 5, fail: 0 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 800, counters: { ok: 4, fail: 0 } },
|
||||
]
|
||||
: [
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'fail', failureClass: 'worker.unavailable' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'fail', failureClass: 'worker.unavailable', counters: { depth: 17 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'fail', failureClass: 'worker.unavailable', counters: { timeout: 4 } },
|
||||
];
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ checks }));
|
||||
return;
|
||||
}
|
||||
res.writeHead(404, { 'content-type': 'text/plain' });
|
||||
res.end('Not found');
|
||||
});
|
||||
await new Promise(resolve => app.listen(port, '127.0.0.1', resolve));
|
||||
return { origin, close: () => new Promise(resolve => app.close(() => resolve())) };
|
||||
}
|
||||
|
||||
test('outage drill flips one simulated switch and reports exactly one page alert plus manual fallback', async t => {
|
||||
const server = await startDrillServer();
|
||||
t.after(() => server.close());
|
||||
|
||||
const result = await runCli(['--drill-origin', server.origin]);
|
||||
assert.equal(result.code, 0, result.stderr);
|
||||
assert.match(result.stdout, /WORKER OUTAGE DRILL/);
|
||||
assert.match(result.stdout, /pre-drill: 0 alerts/i);
|
||||
assert.match(result.stdout, /post-outage: exactly 1 alert/i);
|
||||
assert.match(result.stdout, /PAGE worker\.unavailable on daily-2026-08-22\.1/);
|
||||
assert.match(result.stdout, /manual fallback: AVAILABLE — local journal remains usable/);
|
||||
assert.match(result.stdout, /DRILL PASS/);
|
||||
});
|
||||
109
tests/release-dashboard.test.js
Normal file
109
tests/release-dashboard.test.js
Normal file
|
|
@ -0,0 +1,109 @@
|
|||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
|
||||
import { sanitizeEvidence, buildDashboard, evaluateAlerts, DEFAULT_ALERT_RULES } from '../src/release-observability.js';
|
||||
|
||||
const healthyEvidence = () => sanitizeEvidence({
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass', counters: { ok: 40, fail: 0 } },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 0 } === undefined ? {} : { ok: 30, fail: 0 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 900, counters: { ok: 25, fail: 0, abstain: 3 } },
|
||||
],
|
||||
}).evidence;
|
||||
|
||||
test('dashboard separates app, api, queue, and model boundaries', () => {
|
||||
const dashboard = buildDashboard(healthyEvidence());
|
||||
assert.equal(dashboard.ok, true);
|
||||
assert.deepEqual(Object.keys(dashboard.boundaries).sort(), ['api', 'app', 'model', 'queue']);
|
||||
assert.equal(dashboard.boundaries.app.pass, 1);
|
||||
assert.equal(dashboard.boundaries.model.pass, 1);
|
||||
assert.equal(dashboard.boundaries.model.latencyMs, 900);
|
||||
assert.equal(dashboard.totals.checks, 4);
|
||||
assert.equal(dashboard.totals.fail, 0);
|
||||
assert.deepEqual(dashboard.failureClasses, {});
|
||||
assert.equal(dashboard.identity.releaseTag, 'daily-2026-08-22.1');
|
||||
assert.equal(dashboard.identity.commit, 'ca31e6d38bec649407f63880504554c59f2878ae');
|
||||
});
|
||||
|
||||
test('dashboard summarizes privacy-safe failure classes without any payload text', () => {
|
||||
const evidence = sanitizeEvidence({
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'degraded', failureClass: 'vision.timeout' },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'fail', failureClass: 'model.error' },
|
||||
{ id: 'model.inference.warmup', boundary: 'model', status: 'fail', failureClass: 'model.error' },
|
||||
],
|
||||
}).evidence;
|
||||
const dashboard = buildDashboard(evidence);
|
||||
assert.deepEqual(dashboard.failureClasses, { 'vision.timeout': 1, 'model.error': 2 });
|
||||
assert.equal(dashboard.totals.fail, 2);
|
||||
assert.equal(dashboard.totals.degraded, 1);
|
||||
assert.equal(JSON.stringify(dashboard).includes('note'), false);
|
||||
});
|
||||
|
||||
test('simulated worker outage fires exactly one actionable alert with owner, threshold, and runbook', () => {
|
||||
const evidence = sanitizeEvidence({
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 11 },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'fail', failureClass: 'worker.unavailable' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'fail', failureClass: 'worker.unavailable', counters: { depth: 17 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'fail', failureClass: 'worker.unavailable', counters: { timeout: 4 } },
|
||||
],
|
||||
}).evidence;
|
||||
|
||||
const dashboard = buildDashboard(evidence);
|
||||
const alerts = evaluateAlerts(dashboard, DEFAULT_ALERT_RULES);
|
||||
|
||||
assert.equal(alerts.length, 1, `expected exactly one alert, got ${JSON.stringify(alerts)}`);
|
||||
const alert = alerts[0];
|
||||
assert.equal(alert.id, 'worker.outage');
|
||||
assert.equal(alert.severity, 'page');
|
||||
assert.equal(alert.owner, 'release-operator');
|
||||
assert.match(alert.runbook, /docs\/RELEASE-OBSERVABILITY\.md#simulate-a-worker-outage/);
|
||||
assert.match(alert.message, /worker\.unavailable/);
|
||||
assert.match(alert.message, /daily-2026-08-22\.1/);
|
||||
assert.equal(alert.count, 3);
|
||||
assert.equal(alert.threshold, 1);
|
||||
});
|
||||
|
||||
test('healthy evidence raises no alerts and every default rule carries owner, threshold, and runbook', () => {
|
||||
assert.deepEqual(evaluateAlerts(buildDashboard(healthyEvidence()), DEFAULT_ALERT_RULES), []);
|
||||
for (const rule of DEFAULT_ALERT_RULES) {
|
||||
assert.ok(rule.owner && rule.owner.length > 0);
|
||||
assert.ok(Number.isInteger(rule.threshold) && rule.threshold >= 1);
|
||||
assert.match(rule.runbook, /^docs\/RELEASE-OBSERVABILITY\.md#/);
|
||||
assert.ok(['page', 'warn'].includes(rule.severity));
|
||||
}
|
||||
});
|
||||
|
||||
test('manual fallback stays available whenever any non-app check is unhealthy', () => {
|
||||
const healthy = buildDashboard(healthyEvidence());
|
||||
assert.deepEqual(healthy.manualFallback, { available: true, reason: 'none-required' });
|
||||
|
||||
const outage = buildDashboard(sanitizeEvidence({
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'fail', failureClass: 'worker.unavailable' },
|
||||
],
|
||||
}).evidence);
|
||||
assert.deepEqual(outage.manualFallback, {
|
||||
available: true,
|
||||
reason: 'local-journal',
|
||||
});
|
||||
});
|
||||
42
tests/release-observability-runbook.test.js
Normal file
42
tests/release-observability-runbook.test.js
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { readFile } from 'node:fs/promises';
|
||||
|
||||
import { DEFAULT_ALERT_RULES } from '../src/release-observability.js';
|
||||
|
||||
const runbookPath = new URL('../docs/RELEASE-OBSERVABILITY.md', import.meta.url);
|
||||
const packagePath = new URL('../package.json', import.meta.url);
|
||||
|
||||
test('every default alert rule is inventoried in the runbook with owner, threshold, and severity', async () => {
|
||||
const runbook = await readFile(runbookPath, 'utf8');
|
||||
const headingSlugs = [...runbook.matchAll(/^#{2,4} (.+)$/gm)]
|
||||
.map(match => match[1].toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, ''));
|
||||
for (const rule of DEFAULT_ALERT_RULES) {
|
||||
const pattern = new RegExp(
|
||||
`\\| \`${rule.id}\`\\s*\\| ${rule.severity}\\s*\\| ${rule.owner}\\s*\\|`,
|
||||
);
|
||||
assert.match(runbook, pattern, `runbook table lacks a conforming row for ${rule.id}`);
|
||||
const anchor = rule.runbook.split('#')[1];
|
||||
assert.ok(anchor && headingSlugs.includes(anchor), `runbook link for ${rule.id} does not resolve to a heading (${anchor})`);
|
||||
}
|
||||
});
|
||||
|
||||
test('runbook documents the outage drill anchor, manual fallback, and privacy boundary', async () => {
|
||||
const runbook = await readFile(runbookPath, 'utf8');
|
||||
assert.match(runbook, /^## Simulate a worker outage$/m);
|
||||
assert.match(runbook, /^## Alert inventory$/m);
|
||||
assert.match(runbook, /manual fallback/i);
|
||||
for (const forbidden of ['photos or medical imagery', 'stool records', 'session identifiers', 'environment dumps']) {
|
||||
assert.match(runbook, new RegExp(forbidden.replace(/[-/\\^$*+?.()|[\]{}]/g, '\\$&'), 'i'));
|
||||
}
|
||||
const externalUrls = [...runbook.matchAll(/https?:\/\/(?!127\.0\.0\.1)[^\s)`\]]+/g)].map(match => match[0]);
|
||||
assert.deepEqual(externalUrls, [], 'runbook must not reference non-loopback hosts');
|
||||
});
|
||||
|
||||
test('package scripts wire observability tests into the suite and syntax gate', async () => {
|
||||
const packageJson = JSON.parse(await readFile(packagePath, 'utf8'));
|
||||
assert.match(packageJson.scripts.test, /tests\/release-observability\.test\.js/);
|
||||
assert.match(packageJson.scripts.test, /tests\/release-dashboard\.test\.js/);
|
||||
assert.match(packageJson.scripts.test, /tests\/release-dashboard-cli\.test\.js/);
|
||||
assert.match(packageJson.scripts['check:syntax'], /node --check scripts\/release_dashboard\.mjs/);
|
||||
});
|
||||
78
tests/release-observability.test.js
Normal file
78
tests/release-observability.test.js
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
|
||||
import { sanitizeEvidence } from '../src/release-observability.js';
|
||||
|
||||
const validEvidence = () => ({
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass', counters: { ok: 40, fail: 1 } },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { ok: 30, fail: 0 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 900, counters: { ok: 25, fail: 0, abstain: 3 } },
|
||||
],
|
||||
});
|
||||
|
||||
test('sanitizer keeps bounded sanitized release evidence intact', () => {
|
||||
const result = sanitizeEvidence(validEvidence());
|
||||
assert.equal(result.ok, true);
|
||||
assert.deepEqual(result.evidence, {
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass', failureClass: null, latencyMs: 12, counters: {} },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass', failureClass: null, latencyMs: null, counters: { ok: 40, fail: 1 } },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass', failureClass: null, latencyMs: null, counters: { ok: 30, fail: 0 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass', failureClass: null, latencyMs: 900, counters: { ok: 25, fail: 0, abstain: 3 } },
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
test('sanitizer drops session identifiers, photo payloads, credentials, and free-text notes', () => {
|
||||
// Assembled at runtime so no contiguous private-key marker ever lands in Git history.
|
||||
const privateKeyFixture = `-----BEGIN ${'OPENSSH'} PRIVATE KEY${'-----'}`;
|
||||
const hostile = {
|
||||
...validEvidence(),
|
||||
sessionToken: 'sess_live_abc123',
|
||||
adminCookie: 'timmy_agent=secret-cookie-value',
|
||||
environmentDump: { NODE_ENV: 'production', SECRET_TOKEN: 'must-not-leak' },
|
||||
operatorEmail: 'someone@example.com',
|
||||
credentialFile: privateKeyFixture,
|
||||
photos: [{ dataBase64: 'aGVsbG8gd29ybGQgaGVsbG8gd29ybGQ=' }],
|
||||
checks: [
|
||||
...validEvidence().checks,
|
||||
{
|
||||
id: 'leak',
|
||||
boundary: 'app',
|
||||
status: 'pass',
|
||||
noteText: 'patient said blood at 3am, see photo hash e3b0c442',
|
||||
imageHash: 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855',
|
||||
base64Payload: 'aGVsbG8=',
|
||||
authHeader: `Bearer ${'sk-'}live-abcdef1234567890`,
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
const result = sanitizeEvidence(hostile);
|
||||
assert.equal(result.ok, true);
|
||||
const serialized = JSON.stringify(result.evidence);
|
||||
for (const forbidden of [
|
||||
'sess_live_abc123',
|
||||
'secret-cookie-value',
|
||||
'must-not-leak',
|
||||
'someone@example.com',
|
||||
'PRIVATE KEY',
|
||||
'aGVsbG8',
|
||||
'patient said blood',
|
||||
'e3b0c44298fc1c14',
|
||||
'sk-live-abcdef1234567890',
|
||||
]) {
|
||||
assert.equal(serialized.includes(forbidden), false, `sanitized output leaked ${forbidden}`);
|
||||
}
|
||||
assert.equal(result.evidence.checks.some(check => check.id === 'leak'), false);
|
||||
});
|
||||
Loading…
Reference in New Issue
Block a user