fix: close hostile-review blockers in release observability
All checks were successful
Quality gates / quality (pull_request) Successful in 2m9s
All checks were successful
Quality gates / quality (pull_request) Successful in 2m9s
Strict vertical RED-GREEN TDD across eight review blockers: - drill baseline: requires every required boundary check explicitly passing plus zero alerts of any severity before anything is announced or flipped; unknown, degraded, malformed, missing-boundary, and backlog-warning baselines exit 2 with switch count zero, and stdout stays empty whenever no drill actually ran - manual fallback: deterministic truthful states — app-down, local-journal, none-required, and new app-state-unknown (availability null) so unknown telemetry can never render as proven availability or 'no degradation'; runbook gains a state table with exact wording - exact schema types: schemaVersion must be JSON number 1; counters and latency must be finite bounded integers (0..1e6, 0..3600000); generatedAtUtc must be canonical real UTC surviving round-trip; releaseTag/commit keep exact safe types; coercions reject - control-safety: controlSafe strips LF/CR too, so hostile paths can no longer forge terminal lines (regression test ships a path embedding LF + forged PAGE text); read errors use fixed wording instead of echoing attacker-derived message bytes - malformed checks reject the whole evidence object instead of silently disappearing beside healthy checks; forbidden top-level keys now fail closed rather than being ignored - status/failureClass consistency: pass/unknown checks carrying any failureClass reject the file, so all-pass evidence can never page worker.outage (fail/degraded may stay classless, preserving the closed vocabulary and nullable schema) - sanitization idempotence: omitted latency stays omitted on a second pass; explicit null latency rejects since the schema forbids it - removed the always-false queue-depth tautology in the shared fixture and made the runbook contract test verify each rule's real numeric threshold against DEFAULT_ALERT_RULES Preserved: closed failure-class vocabulary, queue depth 9/10/11 edges, raw loopback-origin validation, app-down fallback, missing-boundary rejection, transition semantics, and every independent drill case.
This commit is contained in:
parent
9c9286b59f
commit
d547b4887b
|
|
@ -12,7 +12,7 @@ Telemetry in this system is release telemetry, not user telemetry. Evidence file
|
|||
- infrastructure addresses beyond the loopback `127.0.0.1` drill fixture;
|
||||
- user identity of any kind (no emails, usernames, device IDs).
|
||||
|
||||
`src/release-observability.js` enforces this mechanically: `sanitizeEvidence()` allowlists bounded fields (release tag, full commit, UTC timestamp, per-check id/boundary/status/failure-class/latency/counters) and drops anything else — including any check whose extra fields look like keys, sessions, base64 payloads, image hashes, notes, or free text. Failure classes are a closed vocabulary (`worker.unavailable`, `vision.timeout`, `model.error`) matched against a strict dotted lowercase slug grammar; anything else — newlines, ANSI/control characters, secrets, medical text, unknown classes — fails the whole evidence file closed rather than being rendered. Counters are bounded non-negative integers capped at 1,000,000. Every one of the four boundaries (app, api, queue, model) must be present; partial telemetry is refused instead of rendered as silent health. If sanitization fails, the tool exits nonzero and echoes nothing from the input, and all CLI output is scrubbed of control characters so hostile file paths can never inject terminal escapes. The sanitizer itself is tested with hostile payloads (`tests/release-observability.test.js`) and the CLI is tested to never echo rejected content (`tests/release-dashboard-cli.test.js`).
|
||||
`src/release-observability.js` enforces this mechanically: `sanitizeEvidence()` allowlists bounded fields (release tag, full commit, UTC timestamp, per-check id/boundary/status/failure-class/latency/counters) and rejects the whole evidence object when anything else appears — forbidden keys (sessions, cookies, base64 payloads, image hashes, notes, free text), values of the wrong JSON type, numbers outside their bounded ranges, or a single malformed check among healthy ones. A hostile file is refused loudly; it is never silently censored into apparent health. Failure classes are a closed vocabulary (`worker.unavailable`, `vision.timeout`, `model.error`) matched against a strict dotted lowercase slug grammar; anything else — newlines, ANSI/control characters, secrets, medical text, unknown classes — fails the whole evidence file closed rather than being rendered. Counters are bounded non-negative integers capped at 1,000,000. Every one of the four boundaries (app, api, queue, model) must be present; partial telemetry is refused instead of rendered as silent health. If sanitization fails, the tool exits nonzero and echoes nothing from the input, and all CLI output is scrubbed of control characters so hostile file paths can never inject terminal escapes. The sanitizer itself is tested with hostile payloads (`tests/release-observability.test.js`) and the CLI is tested to never echo rejected content (`tests/release-dashboard-cli.test.js`).
|
||||
|
||||
## Dashboard
|
||||
|
||||
|
|
@ -24,28 +24,41 @@ node scripts/release_dashboard.mjs --evidence /secure/inbox/evidence.json
|
|||
|
||||
Output groups checks under four boundaries — app, api, queue, model — with pass/fail/degraded counts, max latency, bounded counters, failure-class totals, active alerts, and manual-fallback state. Exit code is `0` when no page-severity alert fires, `1` when one does, `2` on bad input. The CLI never contacts a live host; it reads local files or a loopback drill fixture only.
|
||||
|
||||
### Manual fallback states (truthful by construction)
|
||||
|
||||
The manual-fallback line is a safety statement, so it only ever claims what the evidence proves:
|
||||
|
||||
| Evidence state | Rendered line |
|
||||
|---|---|
|
||||
| every check explicitly `pass` | `manual fallback: available — all checks explicitly passing; no degradation observed; fallback not required` |
|
||||
| any degradation away from the app boundary | `manual fallback: available — degraded elsewhere; local journal remains usable` |
|
||||
| app boundary `fail` | `manual fallback: unavailable — app boundary is down; local journal cannot be served` |
|
||||
| any check (including the app boundary itself) `unknown` | `manual fallback: unknown — app state unknown; verify the local journal by hand before relying on it` |
|
||||
|
||||
Unknown telemetry is never rendered as proven availability or as "no degradation": an unknown app state forces the `unknown` wording and demands manual verification before anyone relies on the journal.
|
||||
|
||||
### Evidence schema (v1)
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"releaseTag": "daily-2026-08-22.1",
|
||||
"releaseTag": "string, 1-80 chars: letters, digits, dot, underscore, dash",
|
||||
"commit": "40 lowercase hex characters (12+ accepted)",
|
||||
"generatedAtUtc": "YYYY-MM-DDTHH:MM:SSZ",
|
||||
"generatedAtUtc": "canonical real UTC, YYYY-MM-DDTHH:MM:SSZ (must round-trip as a real instant)",
|
||||
"checks": [
|
||||
{
|
||||
"id": "lowercase-dot-or-dash id",
|
||||
"boundary": "app | api | queue | model",
|
||||
"status": "pass | fail | degraded | unknown",
|
||||
"failureClass": "dotted.class.name or null",
|
||||
"latencyMs": integer milliseconds or omitted,
|
||||
"counters": { "ok|fail|abstain|retry|timeout|rejected|fallback|depth": non-negative integer }
|
||||
"failureClass": "vocabulary class on fail/degraded only; omitted or null otherwise",
|
||||
"latencyMs": "optional integer 0..3600000; never a string, boolean, null, or fraction",
|
||||
"counters": { "ok|fail|abstain|retry|timeout|rejected|fallback|depth": "optional integer 0..1000000" }
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Any other field is dropped, and checks carrying forbidden-shaped fields are discarded individually. Counters are capped to the fixed vocabulary above so free text can never smuggle in through a label. Issue #19 owns producing these signals from the worker; until that lands, evidence can be hand-authored for drills.
|
||||
Every field above is type-checked against the exact JSON type: `schemaVersion` must be the number `1` (not `"1"`), counters and latency must be actual JSON numbers within their bounds, and timestamps must survive a UTC round-trip. A `pass` or `unknown` check must never carry a failureClass; a `fail` or `degraded` check may carry one of the closed-vocabulary classes or stay classless. Any other field — at the top level or inside a check — rejects the entire evidence object.
|
||||
|
||||
## Alert inventory
|
||||
|
||||
|
|
@ -55,7 +68,7 @@ Alerts are evaluated by `evaluateAlerts()` over the sanitized dashboard. Every r
|
|||
|-----------------|----------|------------------|-------------------------|--------------------------------------------------|---------|
|
||||
| `worker.outage` | page | release-operator | 1 | any check reports `worker.unavailable`, or both model and queue boundaries report failures | [Simulate a worker outage](#simulate-a-worker-outage) |
|
||||
| `telemetry.gap` | warn | release-operator | 1 | one or more checks report status `unknown` (missing telemetry fails closed as a warning, never as health) | [Alert inventory](#alert-inventory) |
|
||||
| `queue.backlog` | warn | release-operator | depth >= 10 | queue depth counter at or above 10; suppressed while `worker.outage` pages (depth is residual from the same incident) | [Alert inventory](#alert-inventory) |
|
||||
| `queue.backlog` | warn | release-operator | 10 | queue depth at or above threshold (`depth >= 10`); suppressed while `worker.outage` pages (depth is residual from the same incident) | [Alert inventory](#alert-inventory) |
|
||||
| `vision.degraded` | warn | vision-owner | 3 | three or more `vision.timeout` failure classes | [Alert inventory](#alert-inventory) |
|
||||
| `app.unhealthy` | page | release-operator | 1 | app boundary healthz reports fail; this is also the only condition that makes the manual fallback unavailable (`reason: app-down`) | [Alert inventory](#alert-inventory) |
|
||||
|
||||
|
|
@ -77,10 +90,10 @@ node scripts/release_dashboard.mjs --drill-origin http://127.0.0.1:<port>
|
|||
The origin must be a bare `http://127.0.0.1:<port>` URL. Credentials (`user:pass@`), DNS names, alternative IP encodings (hex octets, decimal integers, percent-encoding), IPv6 forms, non-http schemes, and any path or query are rejected before the CLI contacts anything.
|
||||
|
||||
3. Required outcome (the automated tests assert all of it):
|
||||
- the pre-drill baseline is read from the fixture itself and must be genuinely healthy — an already-outaged fixture is refused with exit code 2 and its switch is never touched;
|
||||
- the pre-drill baseline is read from the fixture itself and must be genuinely healthy — every required boundary check explicitly `pass` and zero alerts of any severity; unknown, degraded, malformed, missing-boundary, or backlog-warning baselines are refused with exit code 2 before anything is announced or flipped, and the switch is never touched;
|
||||
- after the flip, exactly one page alert fires — `worker.outage`; if the flip produced no real healthy-to-outage transition, the drill reports `DRILL FAIL` with exit code 1 instead of passing vacuously;
|
||||
- the alert names owner `release-operator`, its threshold, and this runbook section;
|
||||
- `manual fallback: available — local journal remains usable`;
|
||||
- `manual fallback: AVAILABLE — degraded elsewhere; local journal remains usable`;
|
||||
- exit code `0` (`DRILL PASS`).
|
||||
|
||||
4. Manual fallback procedure if a real outage ever matches this signature: users keep logging locally (the browser journal never depends on the vision worker), operators capture one sanitized evidence file, run the dashboard once, record the single page alert, and follow rollback via the staging runbook. No user data leaves the device.
|
||||
|
|
@ -99,9 +112,12 @@ The drill is exercised end-to-end by `tests/release-dashboard-cli.test.js` ("out
|
|||
## Receipts
|
||||
|
||||
- `npm test` — includes `tests/release-observability.test.js`, `tests/release-dashboard.test.js`, `tests/release-dashboard-cli.test.js`.
|
||||
- Sanitizer hostile-payload test proves session tokens, cookies, environment dumps, private keys, base64 payloads, image hashes, note text, and emails cannot survive into dashboard output.
|
||||
- Strict-type tests prove `schemaVersion` must be the JSON number 1, counters and latency must be actual bounded JSON numbers (never strings, booleans, nulls, fractions, or out-of-range values), and `generatedAtUtc` must be canonical real UTC that survives a round-trip.
|
||||
- Fail-closed tests prove a single malformed check — bad id, missing boundary, unknown status, wrong-typed counters, or an off-vocabulary class on any status — rejects the whole evidence object instead of silently disappearing beside healthy checks.
|
||||
- Consistency tests prove a `pass` or `unknown` check can never carry a failureClass, so all-pass evidence can never page `worker.outage`; sanitization is proven idempotent, so omitted latency stays omitted instead of becoming 0ms on a second pass.
|
||||
- Sanitizer hostile-payload test proves session tokens, cookies, environment dumps, private keys, base64 payloads, image hashes, note text, and emails reject the entire file rather than surviving into dashboard output.
|
||||
- Hostile failure-class test proves newlines, carriage returns, ANSI escapes, control characters, secrets, medical text, SQL, oversized values, wrong types, and off-vocabulary classes all fail closed.
|
||||
- Depth-counter tests prove the `queue.backlog` warn stays silent at 9, fires at exactly 10, reports observed counts above threshold, sums across queue checks, is suppressed while `worker.outage` pages, and cannot be smuggled in above the bounded counter ceiling.
|
||||
- Boundary-completeness tests prove evidence missing any of app, api, queue, or model fails closed; unknown statuses surface as a `telemetry.gap` warn instead of passing as healthy.
|
||||
- Manual-fallback tests prove a deterministic contract: journal available with no degradation, active (`local-journal`) under any non-fatal degradation, and unavailable only when the app boundary itself is down.
|
||||
- CLI tests prove rejected evidence exits nonzero without echoing contents, drill origins are validated to bare loopback http before any network contact (credentials, DNS names, hex/decimal/percent-encoded IP encodings, IPv6, paths, queries, and foreign schemes are refused), the outage drill reads a genuine fixture baseline, refuses already-down fixtures without touching their switch, fails when no real transition occurs, and yields exactly one actionable alert with graceful manual fallback.
|
||||
- Manual-fallback tests prove a truthful deterministic contract: available with no degradation, active (`local-journal`) under any non-fatal degradation, unavailable only when the app boundary itself is down, and `unknown` (`app-state-unknown`, availability `null`) whenever telemetry is unknown — never claimed as healthy from unknown state.
|
||||
- CLI tests prove rejected evidence exits nonzero without echoing contents, hostile paths cannot forge terminal lines with LF/CR (only program-authored line breaks reach stdout), drill origins are validated to bare loopback http before any network contact (credentials, DNS names, hex/decimal/percent-encoded IP encodings, IPv6, paths, queries, and foreign schemes are refused), the outage drill requires a genuine baseline — every required check explicitly passing and zero alerts — before announcing anything or touching its switch, refuses already-down fixtures, fails when no real transition occurs, and yields exactly one actionable alert with graceful manual fallback.
|
||||
|
|
|
|||
|
|
@ -11,12 +11,14 @@ function argValue(args, flag) {
|
|||
return args[index + 1] || null;
|
||||
}
|
||||
|
||||
// Terminal safety: strip C0/C1 control characters (newline excepted) from any
|
||||
// dynamically produced text before it reaches stdout/stderr, so hostile file
|
||||
// paths or fixture payloads can never smuggle ANSI escapes into a terminal.
|
||||
// Terminal safety: strip C0/C1 control characters — newline and carriage
|
||||
// return included — from any dynamically produced text before it reaches
|
||||
// stdout/stderr. Hostile file paths or fixture payloads can therefore never
|
||||
// forge additional terminal lines (e.g. a fake "PAGE worker.outage"): the
|
||||
// only line boundaries in CLI output are the ones this program authors.
|
||||
function controlSafe(text) {
|
||||
return String(text)
|
||||
.replace(/[\u0000-\u0009\u000B-\u001F\u007F-\u009F]/g, '');
|
||||
.replace(/[\u0000-\u001F\u007F-\u009F]/g, '');
|
||||
}
|
||||
|
||||
async function loadEvidence(source) {
|
||||
|
|
@ -92,7 +94,7 @@ async function main() {
|
|||
try {
|
||||
validatedOrigin = validateLoopbackOrigin(drillOrigin);
|
||||
} catch (error) {
|
||||
console.error(error.message);
|
||||
console.error(controlSafe(error.message));
|
||||
return 2;
|
||||
}
|
||||
}
|
||||
|
|
@ -100,11 +102,19 @@ async function main() {
|
|||
let rawEvidence;
|
||||
try {
|
||||
rawEvidence = await loadEvidence({ drillOrigin: validatedOrigin, evidencePath });
|
||||
} catch (error) {
|
||||
console.error(`release_dashboard: cannot read sanitized evidence (${controlSafe(error.message)}). No contents are echoed.`);
|
||||
} catch {
|
||||
// Fixed wording on purpose: interpolating filesystem error messages would
|
||||
// echo attacker-controlled path bytes (even sanitized) into the terminal.
|
||||
console.error('release_dashboard: cannot read sanitized evidence (unreadable file or invalid JSON). No contents are echoed.');
|
||||
return 2;
|
||||
}
|
||||
|
||||
// Drill mode owns its whole flow: the baseline gate must speak with one
|
||||
// voice ("not a genuine drill baseline") whether the fixture served
|
||||
// unsanitizable evidence or merely unhealthy checks, and nothing may be
|
||||
// rendered or flipped until the baseline proves genuinely healthy.
|
||||
if (drillOrigin) return runDrill({ drillOrigin: validatedOrigin });
|
||||
|
||||
const validated = sanitizeEvidence(rawEvidence);
|
||||
if (!validated.ok) {
|
||||
console.error('release_dashboard: evidence failed sanitization (schema mismatch, forbidden fields, or unbounded values). Nothing was rendered.');
|
||||
|
|
@ -115,16 +125,29 @@ async function main() {
|
|||
const alerts = evaluateAlerts(dashboard);
|
||||
console.log(renderDashboard(dashboard, alerts));
|
||||
console.log('');
|
||||
console.log(`manual fallback: ${dashboard.manualFallback.available ? 'available' : 'unavailable'} — ${fallbackReason(dashboard.manualFallback.reason)}`);
|
||||
console.log(manualFallbackLine(dashboard.manualFallback));
|
||||
|
||||
if (drillOrigin) return runDrill({ drillOrigin: validatedOrigin });
|
||||
return alerts.some(alert => alert.severity === 'page') ? 1 : 0;
|
||||
}
|
||||
|
||||
function fallbackReason(reason) {
|
||||
if (reason === 'local-journal') return 'local journal remains usable';
|
||||
if (reason === 'app-down') return 'app boundary is down; local journal cannot be served';
|
||||
return 'no degradation detected; fallback not required';
|
||||
// Truthful operator wording: the manual-fallback line is a safety statement,
|
||||
// so each deterministic state says exactly what is known. Availability is
|
||||
// only ever claimed from explicit passes; an unknown app boundary reads as
|
||||
// unknown and demands hand verification — never as quiet health.
|
||||
const FALLBACK_PHRASES = {
|
||||
'none-required': 'all checks explicitly passing; no degradation observed; fallback not required',
|
||||
'local-journal': 'degraded elsewhere; local journal remains usable',
|
||||
'app-down': 'app boundary is down; local journal cannot be served',
|
||||
'app-state-unknown': 'app state unknown; verify the local journal by hand before relying on it',
|
||||
};
|
||||
|
||||
function manualFallbackLine(fallback, { shout = false } = {}) {
|
||||
const label = fallback.available === null
|
||||
? (shout ? 'UNKNOWN' : 'unknown')
|
||||
: fallback.available
|
||||
? (shout ? 'AVAILABLE' : 'available')
|
||||
: (shout ? 'UNAVAILABLE' : 'unavailable');
|
||||
return `manual fallback: ${label} — ${FALLBACK_PHRASES[fallback.reason] ?? fallback.reason}`;
|
||||
}
|
||||
|
||||
// A drill may only ever talk to a throwaway fixture on 127.0.0.1. The origin
|
||||
|
|
@ -167,37 +190,47 @@ function validateLoopbackOrigin(rawOrigin) {
|
|||
}
|
||||
|
||||
async function runDrill({ drillOrigin }) {
|
||||
console.log('');
|
||||
console.log('WORKER OUTAGE DRILL (simulated fixture on loopback; no live host is contacted)');
|
||||
let origin;
|
||||
try {
|
||||
origin = validateLoopbackOrigin(drillOrigin);
|
||||
} catch (error) {
|
||||
console.error(error.message);
|
||||
console.error(controlSafe(error.message));
|
||||
return 2;
|
||||
}
|
||||
|
||||
// Genuine baseline: read the fixture's own current state. A drill may only
|
||||
// start from proven health — never from an assumed or already-outaged state.
|
||||
// Genuine baseline: read the fixture's own current state BEFORE anything is
|
||||
// announced or flipped. A drill may only start from proven health — every
|
||||
// required boundary check must explicitly pass and zero alerts may fire.
|
||||
// Unknown, degraded, malformed, missing-boundary, or warning baselines are
|
||||
// refused here, so stdout stays empty whenever no drill actually ran.
|
||||
const preRaw = await loadEvidence({ drillOrigin });
|
||||
const preValidated = sanitizeEvidence(preRaw);
|
||||
if (!preValidated.ok) {
|
||||
console.error('release_dashboard: pre-drill fixture evidence failed sanitization. Nothing was rendered.');
|
||||
console.error('release_dashboard: fixture evidence is unsanitizable: not a genuine drill baseline (schema mismatch, forbidden fields, or unbounded values). Nothing will be rendered or flipped.');
|
||||
return 2;
|
||||
}
|
||||
const preAlerts = evaluateAlerts(buildDashboard(preValidated.evidence));
|
||||
const prePages = preAlerts.filter(alert => alert.severity === 'page');
|
||||
if (prePages.length > 0) {
|
||||
console.error(`release_dashboard: fixture is not healthy before the drill (${prePages.length} page alert[s]); refusing to flip an already-down worker.`);
|
||||
const preDashboard = buildDashboard(preValidated.evidence);
|
||||
const preAlerts = evaluateAlerts(preDashboard);
|
||||
const allExplicitlyPassing = Object.values(preDashboard.boundaries).every(
|
||||
boundary => boundary.checks > 0 && boundary.pass === boundary.checks,
|
||||
);
|
||||
if (!allExplicitlyPassing || preAlerts.length > 0) {
|
||||
console.error('release_dashboard: fixture is not healthy before the drill: not a genuine drill baseline (every required boundary check must explicitly pass with zero alerts). Refusing to flip an already-down or uncertain worker.');
|
||||
return 2;
|
||||
}
|
||||
|
||||
console.log('');
|
||||
console.log('WORKER OUTAGE DRILL (simulated fixture on loopback; no live host is contacted)');
|
||||
console.log('pre-drill baseline: every required boundary check explicitly passing; alerts: none');
|
||||
console.log('pre-drill: healthy (0 page alerts)');
|
||||
|
||||
try {
|
||||
const response = await fetch(`${origin}/drill/outage`, { method: 'POST', signal: AbortSignal.timeout(5_000) });
|
||||
if (!response.ok) throw new Error(`fixture returned ${response.status}`);
|
||||
} catch (error) {
|
||||
console.error(`release_dashboard: drill switch failed (${error.message})`);
|
||||
} catch {
|
||||
// Fixed wording on purpose: fetch failures can embed attacker-chosen URL
|
||||
// text; echoing it (even sanitized) has no operational value.
|
||||
console.error('release_dashboard: drill switch failed. The fixture switch was not verifiably flipped.');
|
||||
return 2;
|
||||
}
|
||||
|
||||
|
|
@ -221,7 +254,7 @@ async function runDrill({ drillOrigin }) {
|
|||
console.log(` ${alert.severity.toUpperCase()} ${alert.message.replace(/^PAGE /, '')}`);
|
||||
console.log(` owner ${alert.owner} · threshold ${alert.threshold} · runbook ${alert.runbook}`);
|
||||
}
|
||||
console.log(`manual fallback: ${postDashboard.manualFallback.available ? 'AVAILABLE' : 'UNAVAILABLE'} — ${fallbackReason(postDashboard.manualFallback.reason)}`);
|
||||
console.log(manualFallbackLine(postDashboard.manualFallback, { shout: true }));
|
||||
const pass = pageAlerts.length === 1
|
||||
&& pageAlerts[0].id === 'worker.outage'
|
||||
&& postDashboard.manualFallback.available === true;
|
||||
|
|
|
|||
|
|
@ -20,82 +20,135 @@ const FORBIDDEN_CHECK_KEYS = new Set([
|
|||
'imagehash', 'base64payload', 'payload', 'blob', 'rawoutput',
|
||||
]);
|
||||
|
||||
const SECRET_VALUE_PATTERNS = [
|
||||
/-----BEGIN [A-Z ]*PRIVATE KEY-----/,
|
||||
/\b(?:sk|sess|ghp|glpat)-[A-Za-z0-9_-]{8,}\b/,
|
||||
/\bbearer\s+[A-Za-z0-9._+/=-]{16,}\b/i,
|
||||
/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/,
|
||||
];
|
||||
|
||||
const COUNTER_KEYS = new Set(['ok', 'fail', 'abstain', 'retry', 'timeout', 'rejected', 'fallback', 'depth']);
|
||||
const COUNTER_CEILING = 1_000_000;
|
||||
const LATENCY_CEILING_MS = 3_600_000;
|
||||
const CHECK_KEYS = new Set(['id', 'boundary', 'status', 'failureClass', 'latencyMs', 'counters']);
|
||||
|
||||
function looksSensitive(value) {
|
||||
const text = typeof value === 'string' ? value : JSON.stringify(value);
|
||||
if (!text || text.length > 256) return true;
|
||||
return SECRET_VALUE_PATTERNS.some(pattern => pattern.test(text));
|
||||
// Strict JSON-type gates: evidence crosses a JSON boundary, so only real JSON
|
||||
// types may pass. Strings that merely look like numbers, booleans, nulls,
|
||||
// non-finite values, and fractions are coercion attempts and must fail closed
|
||||
// instead of being silently converted by Number().
|
||||
function isPlainObject(value) {
|
||||
return typeof value === 'object' && value !== null && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function isBoundedInteger(value, { min, max }) {
|
||||
return typeof value === 'number' && Number.isInteger(value) && value >= min && value <= max;
|
||||
}
|
||||
|
||||
function boundedCounters(raw) {
|
||||
if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return {};
|
||||
if (raw === undefined) return {};
|
||||
if (!isPlainObject(raw)) return null;
|
||||
const counters = {};
|
||||
for (const key of Object.keys(raw).sort()) {
|
||||
if (!COUNTER_KEYS.has(key)) continue;
|
||||
const value = Number(raw[key]);
|
||||
if (!Number.isFinite(value) || value < 0 || !Number.isInteger(value) || value > COUNTER_CEILING) return null;
|
||||
counters[key] = value;
|
||||
if (!COUNTER_KEYS.has(key)) return null;
|
||||
if (!isBoundedInteger(raw[key], { min: 0, max: COUNTER_CEILING })) return null;
|
||||
counters[key] = raw[key];
|
||||
}
|
||||
return counters;
|
||||
}
|
||||
|
||||
function boundedLatency(raw) {
|
||||
const value = Number(raw);
|
||||
if (!Number.isFinite(value) || value < 0) return null;
|
||||
if (value > 3_600_000) return null;
|
||||
return Math.round(value);
|
||||
const GENERATED_AT_PATTERN = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$/;
|
||||
|
||||
// Canonical real UTC: the string must match the exact grammar AND survive a
|
||||
// Date round-trip byte-identically, so impossible dates (Feb 30, hour 25,
|
||||
// month 13) and non-canonical spellings (offsets, lowercase designators, a
|
||||
// space separator) are rejected rather than normalized into something else.
|
||||
function isCanonicalUtc(value) {
|
||||
if (typeof value !== 'string' || !GENERATED_AT_PATTERN.test(value)) return false;
|
||||
const parsed = new Date(value);
|
||||
if (Number.isNaN(parsed.getTime())) return false;
|
||||
// toISOString() always carries milliseconds, so strip them before comparing
|
||||
// to the canonical seconds-precision spelling the grammar requires.
|
||||
return parsed.toISOString().replace(/\.\d{3}Z$/, 'Z') === value;
|
||||
}
|
||||
|
||||
// A check is either fully valid or the whole evidence object is rejected.
|
||||
// Malformed checks may never silently disappear beside healthy ones: censoring
|
||||
// a failure would dress a gap up as health, so every field is type-gated and
|
||||
// any violation fails closed for the entire file.
|
||||
function buildCheck(raw) {
|
||||
if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return { dropped: true };
|
||||
const id = String(raw.id || '');
|
||||
if (!/^[a-z][a-z0-9.-]{0,63}$/.test(id)) return { dropped: true };
|
||||
const boundary = String(raw.boundary || '');
|
||||
if (!BOUNDARIES.has(boundary)) return { dropped: true };
|
||||
const status = String(raw.status || '');
|
||||
if (!STATUSES.has(status)) return { dropped: true };
|
||||
const failureClass = raw.failureClass == null ? null : String(raw.failureClass);
|
||||
if (failureClass !== null && (!FAILURE_CLASSES.has(failureClass) || !FAILURE_CLASS_PATTERN.test(failureClass))) {
|
||||
return { rejected: true };
|
||||
if (!isPlainObject(raw)) return { rejected: true };
|
||||
const id = raw.id;
|
||||
if (typeof id !== 'string' || !/^[a-z][a-z0-9.-]{0,63}$/.test(id)) return { rejected: true };
|
||||
if (!Object.hasOwn(raw, 'boundary') || !BOUNDARIES.has(raw.boundary)) return { rejected: true };
|
||||
const status = raw.status;
|
||||
if (typeof status !== 'string' || !STATUSES.has(status)) return { rejected: true };
|
||||
|
||||
let failureClass = null;
|
||||
if (raw.failureClass !== undefined && raw.failureClass !== null) {
|
||||
failureClass = raw.failureClass;
|
||||
if (
|
||||
typeof failureClass !== 'string'
|
||||
|| !FAILURE_CLASSES.has(failureClass)
|
||||
|| !FAILURE_CLASS_PATTERN.test(failureClass)
|
||||
) {
|
||||
return { rejected: true };
|
||||
}
|
||||
if (status === 'pass' || status === 'unknown') {
|
||||
// Consistency contract: only failing/degraded checks may classify their
|
||||
// failures. A pass or unknown can therefore never contribute to
|
||||
// worker.outage, and all-pass evidence can never page anyone.
|
||||
// A fail/degraded may stay classless (the schema keeps failureClass
|
||||
// nullable); carrying a class is optional, never mandatory.
|
||||
return { rejected: true };
|
||||
}
|
||||
}
|
||||
|
||||
const counters = boundedCounters(raw.counters);
|
||||
if (counters === null) return { rejected: true };
|
||||
return {
|
||||
check: {
|
||||
id,
|
||||
boundary,
|
||||
status,
|
||||
failureClass,
|
||||
latencyMs: boundedLatency(raw.latencyMs),
|
||||
counters,
|
||||
},
|
||||
dropped: Object.keys(raw).some(key => {
|
||||
if (['id', 'boundary', 'status', 'failureClass', 'latencyMs', 'counters'].includes(key)) return false;
|
||||
if (FORBIDDEN_CHECK_KEYS.has(key.replace(/[^a-z0-9]/gi, '').toLowerCase())) return true;
|
||||
return looksSensitive(raw[key]);
|
||||
}),
|
||||
|
||||
const latencyMs = raw.latencyMs;
|
||||
if (latencyMs !== undefined) {
|
||||
// The schema allows an integer or omission — explicit null is neither,
|
||||
// and accepting it here would let a second pass rewrite the shape.
|
||||
if (!isBoundedInteger(latencyMs, { min: 0, max: LATENCY_CEILING_MS })) {
|
||||
return { rejected: true };
|
||||
}
|
||||
}
|
||||
|
||||
for (const key of Object.keys(raw)) {
|
||||
if (!CHECK_KEYS.has(key)) return { rejected: true };
|
||||
}
|
||||
|
||||
const check = {
|
||||
id,
|
||||
boundary: raw.boundary,
|
||||
status,
|
||||
failureClass,
|
||||
counters,
|
||||
};
|
||||
if (latencyMs !== undefined && latencyMs !== null) check.latencyMs = latencyMs;
|
||||
return { check, dropped: false };
|
||||
}
|
||||
|
||||
export function sanitizeEvidence(input) {
|
||||
if (!input || typeof input !== 'object' || Array.isArray(input)) return { ok: false, evidence: null };
|
||||
const commit = String(input.commit || '');
|
||||
if (!/^[0-9a-f]{12,40}$/.test(commit)) return { ok: false, evidence: null };
|
||||
const releaseTag = /^[A-Za-z0-9][A-Za-z0-9._-]{0,79}$/.test(String(input.releaseTag || '')) ? input.releaseTag : null;
|
||||
if (!releaseTag) return { ok: false, evidence: null };
|
||||
const generatedAtUtc = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$/.test(String(input.generatedAtUtc || ''))
|
||||
? input.generatedAtUtc
|
||||
: null;
|
||||
if (!generatedAtUtc) return { ok: false, evidence: null };
|
||||
if (!isPlainObject(input)) return { ok: false, evidence: null };
|
||||
// Forbidden-shaped top-level fields reject the whole evidence object.
|
||||
// Silently ignoring them would let a censored file pass as healthy; a
|
||||
// hostile file must be refused loudly instead of laundered.
|
||||
for (const key of Object.keys(input)) {
|
||||
if (FORBIDDEN_TOP_KEYS.has(key.replace(/[^a-z0-9]/gi, '').toLowerCase())) {
|
||||
return { ok: false, evidence: null };
|
||||
}
|
||||
}
|
||||
// Identity fields keep their exact safe JSON types; anything else is a
|
||||
// type-confusion attempt and fails closed.
|
||||
const commit = input.commit;
|
||||
if (typeof commit !== 'string' || !/^[0-9a-f]{12,40}$/.test(commit)) return { ok: false, evidence: null };
|
||||
const releaseTag = input.releaseTag;
|
||||
if (
|
||||
typeof releaseTag !== 'string'
|
||||
|| !/^[A-Za-z0-9][A-Za-z0-9._-]{0,79}$/.test(releaseTag)
|
||||
) {
|
||||
return { ok: false, evidence: null };
|
||||
}
|
||||
const generatedAtUtc = input.generatedAtUtc;
|
||||
if (!isCanonicalUtc(generatedAtUtc)) return { ok: false, evidence: null };
|
||||
// schemaVersion is the JSON number 1 exactly — "1", true, null, 1.0-style
|
||||
// fractions, and every other lookalike are refused.
|
||||
if (input.schemaVersion !== SCHEMA_VERSION) return { ok: false, evidence: null };
|
||||
if (!Array.isArray(input.checks) || input.checks.length === 0 || input.checks.length > 100) {
|
||||
return { ok: false, evidence: null };
|
||||
}
|
||||
|
|
@ -104,7 +157,6 @@ export function sanitizeEvidence(input) {
|
|||
for (const raw of input.checks) {
|
||||
const built = buildCheck(raw);
|
||||
if (built.rejected) return { ok: false, evidence: null };
|
||||
if (built.dropped) continue;
|
||||
checks.push(built.check);
|
||||
}
|
||||
if (checks.length === 0) return { ok: false, evidence: null };
|
||||
|
|
@ -115,7 +167,6 @@ export function sanitizeEvidence(input) {
|
|||
for (const boundary of BOUNDARIES) {
|
||||
if (!presentBoundaries.has(boundary)) return { ok: false, evidence: null };
|
||||
}
|
||||
if (Number(input.schemaVersion) !== SCHEMA_VERSION) return { ok: false, evidence: null };
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
|
|
@ -159,16 +210,26 @@ export function buildDashboard(evidence) {
|
|||
totals[check.status] += 1;
|
||||
}
|
||||
|
||||
// Deterministic manual-fallback contract: the local journal is a usable
|
||||
// fallback exactly while the app boundary itself can still serve it. Any
|
||||
// degradation anywhere makes the journal the active fallback ('local-journal');
|
||||
// only an app-boundary failure removes it ('app-down').
|
||||
// Deterministic manual-fallback contract, truthful under every status:
|
||||
// - app-boundary fail -> { available: false, reason: 'app-down' }
|
||||
// the journal is served by the app, so a dead app takes it down too;
|
||||
// - app-boundary unknown -> { available: null, reason: 'app-state-unknown' }
|
||||
// availability cannot be claimed from unknown telemetry;
|
||||
// - any other degradation -> { available: true, reason: 'local-journal' };
|
||||
// - everything explicitly passing -> { available: true, reason: 'none-required' }.
|
||||
const appBoundary = boundaries.app;
|
||||
const anyDegradation = totals.fail > 0 || totals.degraded > 0;
|
||||
const manualFallback = boundaries.app.fail > 0
|
||||
? { available: false, reason: 'app-down' }
|
||||
: anyDegradation
|
||||
? { available: true, reason: 'local-journal' }
|
||||
: { available: true, reason: 'none-required' };
|
||||
const anyUnknown = totals.unknown > 0;
|
||||
let manualFallback;
|
||||
if (appBoundary.fail > 0) {
|
||||
manualFallback = { available: false, reason: 'app-down' };
|
||||
} else if (anyUnknown) {
|
||||
manualFallback = { available: null, reason: 'app-state-unknown' };
|
||||
} else if (anyDegradation) {
|
||||
manualFallback = { available: true, reason: 'local-journal' };
|
||||
} else {
|
||||
manualFallback = { available: true, reason: 'none-required' };
|
||||
}
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
|
|
|
|||
|
|
@ -92,6 +92,22 @@ test('cli output stays control-safe when rejected evidence paths carry ANSI esca
|
|||
assert.doesNotMatch(emitted, /\r/, 'carriage return reached the terminal');
|
||||
});
|
||||
|
||||
test('cli output stays control-safe when hostile paths carry LF-forged PAGE lines', async () => {
|
||||
// The path embeds a real newline followed by a forged terminal line, as if
|
||||
// an attacker-supplied filename tried to end the error line early and page
|
||||
// the operator with fake drill output.
|
||||
const forgedPath = join(tmpdir(), `timmy-legit.json\nPAGE worker.outage — ALL SYSTEMS DOWN\r\nEXIT 1 DRILL FAIL`);
|
||||
const result = await runCli(['--evidence', forgedPath]);
|
||||
assert.equal(result.code, 2);
|
||||
const emitted = result.stdout + result.stderr;
|
||||
const lines = emitted.split('\n').filter(line => line.length > 0);
|
||||
for (const line of lines) {
|
||||
assert.match(line, /^release_dashboard:/, `forged line escaped into the terminal: ${JSON.stringify(line)}`);
|
||||
}
|
||||
assert.doesNotMatch(emitted, /ALL SYSTEMS DOWN/, 'LF-forged PAGE text reached the terminal');
|
||||
assert.doesNotMatch(emitted, /DRILL FAIL/, 'LF-forged drill verdict reached the terminal');
|
||||
});
|
||||
|
||||
test('successful dashboard rendering emits no control characters besides newlines', async () => {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'timmy-obs-'));
|
||||
try {
|
||||
|
|
@ -105,7 +121,35 @@ test('successful dashboard rendering emits no control characters besides newline
|
|||
}
|
||||
});
|
||||
|
||||
async function startDrillServer({ startDown = false, flipHasNoEffect = false } = {}) {
|
||||
test('cli never claims proven availability or no degradation from unknown telemetry', async () => {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'timmy-obs-'));
|
||||
try {
|
||||
const unknownEvidence = {
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'unknown', latencyMs: 12 },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass', counters: { ok: 40, fail: 0 } },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 2, ok: 30, fail: 0 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 900, counters: { ok: 25, fail: 0 } },
|
||||
],
|
||||
};
|
||||
const evidencePath = join(dir, 'unknown.json');
|
||||
await writeFile(evidencePath, JSON.stringify(unknownEvidence));
|
||||
|
||||
const result = await runCli(['--evidence', evidencePath]);
|
||||
assert.equal(result.code, 0, result.stderr);
|
||||
assert.match(result.stdout, /manual fallback: unknown — app state unknown; verify the local journal by hand before relying on it/);
|
||||
assert.doesNotMatch(result.stdout, /no degradation detected/, 'unknown app state must never read as proven health');
|
||||
assert.doesNotMatch(result.stdout, /manual fallback: available/, 'unknown app state must never read as proven availability');
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
async function startDrillServer({ startDown = false, flipHasNoEffect = false, preDrillChecks = null } = {}) {
|
||||
const port = nextPort++;
|
||||
const origin = `http://127.0.0.1:${port}`;
|
||||
let workerUp = !startDown;
|
||||
|
|
@ -126,7 +170,7 @@ async function startDrillServer({ startDown = false, flipHasNoEffect = false } =
|
|||
}
|
||||
if (url.pathname === '/api/drill/checks') {
|
||||
const appCheck = { id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 };
|
||||
const checks = workerUp
|
||||
const checks = preDrillChecks ?? (workerUp
|
||||
? [
|
||||
appCheck,
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass', counters: { ok: 5, fail: 0 } },
|
||||
|
|
@ -138,7 +182,7 @@ async function startDrillServer({ startDown = false, flipHasNoEffect = false } =
|
|||
{ id: 'api.analyze', boundary: 'api', status: 'fail', failureClass: 'worker.unavailable' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'fail', failureClass: 'worker.unavailable', counters: { depth: 17 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'fail', failureClass: 'worker.unavailable', counters: { timeout: 4 } },
|
||||
];
|
||||
]);
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ checks }));
|
||||
return;
|
||||
|
|
@ -161,14 +205,74 @@ test('outage drill reads the genuine fixture baseline, flips once, and reports e
|
|||
const result = await runCli(['--drill-origin', server.origin]);
|
||||
assert.equal(result.code, 0, result.stderr);
|
||||
assert.match(result.stdout, /WORKER OUTAGE DRILL/);
|
||||
assert.match(result.stdout, /pre-drill baseline: every required boundary check explicitly passing; alerts: none/i);
|
||||
assert.match(result.stdout, /pre-drill: healthy \(0 page alerts\)/i);
|
||||
assert.equal(server.outageSwitchCount, 1, 'drill must flip the simulated switch exactly once');
|
||||
assert.match(result.stdout, /post-outage: exactly 1 alert/i);
|
||||
assert.match(result.stdout, /PAGE worker\.unavailable on daily-2026-08-22\.1/);
|
||||
assert.match(result.stdout, /manual fallback: AVAILABLE — local journal remains usable/);
|
||||
assert.match(result.stdout, /manual fallback: AVAILABLE — degraded elsewhere; local journal remains usable/);
|
||||
assert.match(result.stdout, /DRILL PASS/);
|
||||
});
|
||||
|
||||
test('drill refuses unknown, degraded, malformed, and backlog-warning baselines with exit 2 before any POST', async t => {
|
||||
const unhealthyBaselines = [
|
||||
{
|
||||
name: 'unknown telemetry',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'unknown', latencyMs: 12 },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 1 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 800 },
|
||||
],
|
||||
},
|
||||
{
|
||||
name: 'degraded check',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'degraded', failureClass: 'vision.timeout' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 1 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 800 },
|
||||
],
|
||||
},
|
||||
{
|
||||
name: 'malformed check',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 },
|
||||
{ id: 'BROKEN ID', boundary: 'api', status: 'pass' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 1 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 800 },
|
||||
],
|
||||
},
|
||||
{
|
||||
name: 'backlog warning',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 12 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 800 },
|
||||
],
|
||||
},
|
||||
{
|
||||
name: 'missing boundary',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 1 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 800 },
|
||||
],
|
||||
},
|
||||
];
|
||||
for (const baseline of unhealthyBaselines) {
|
||||
const server = await startDrillServer({ preDrillChecks: baseline.checks });
|
||||
t.after(() => server.close());
|
||||
|
||||
const result = await runCli(['--drill-origin', server.origin]);
|
||||
assert.equal(result.code, 2, `${baseline.name}: expected refusal exit 2, got ${result.code}`);
|
||||
assert.match(result.stderr, /not a genuine drill baseline/i, `${baseline.name}: operator must be told why`);
|
||||
assert.doesNotMatch(result.stdout, /DRILL PASS|WORKER OUTAGE DRILL/, `${baseline.name}: drill must never start`);
|
||||
assert.equal(server.outageSwitchCount, 0, `${baseline.name}: switch must never be touched`);
|
||||
}
|
||||
});
|
||||
|
||||
test('drill refuses to flip an already-outaged fixture and never touches its switch', async t => {
|
||||
const server = await startDrillServer({ startDown: true });
|
||||
t.after(() => server.close());
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ const healthyEvidence = () => sanitizeEvidence({
|
|||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass', counters: { ok: 40, fail: 0 } },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 0 } === undefined ? {} : { ok: 30, fail: 0 } },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 0, ok: 30, fail: 0 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 900, counters: { ok: 25, fail: 0, abstain: 3 } },
|
||||
],
|
||||
}).evidence;
|
||||
|
|
@ -136,7 +136,7 @@ test('manualFallback state is a deterministic function of boundary health alone'
|
|||
const cases = [
|
||||
{ statuses: ['pass', 'pass', 'pass', 'pass'], expected: { available: true, reason: 'none-required' } },
|
||||
{ statuses: ['degraded', 'pass', 'pass', 'pass'], expected: { available: true, reason: 'local-journal' } },
|
||||
{ statuses: ['pass', 'unknown', 'pass', 'pass'], expected: { available: true, reason: 'none-required' } },
|
||||
{ statuses: ['pass', 'unknown', 'pass', 'pass'], expected: { available: null, reason: 'app-state-unknown' } },
|
||||
{ statuses: ['fail', 'fail', 'fail', 'fail'], expected: { available: false, reason: 'app-down' } },
|
||||
];
|
||||
for (const { statuses, expected } of cases) {
|
||||
|
|
|
|||
|
|
@ -21,6 +21,30 @@ test('every default alert rule is inventoried in the runbook with owner, thresho
|
|||
}
|
||||
});
|
||||
|
||||
test('runbook table states each rule\'s actual numeric threshold, matching DEFAULT_ALERT_RULES', async () => {
|
||||
const runbook = await readFile(runbookPath, 'utf8');
|
||||
const rows = Object.fromEntries(
|
||||
runbook.split('\n')
|
||||
.filter(line => line.startsWith('| `'))
|
||||
.map(line => [line.slice(3, line.indexOf('`', 3)), line]),
|
||||
);
|
||||
for (const rule of DEFAULT_ALERT_RULES) {
|
||||
// The threshold must appear as a real number in the rule's own table row,
|
||||
// not merely somewhere in prose, so the documented contract cannot drift
|
||||
// from what evaluateAlerts() enforces.
|
||||
const row = rows[rule.id];
|
||||
assert.ok(row, `runbook table has no row for ${rule.id}`);
|
||||
assert.match(
|
||||
row,
|
||||
new RegExp(`\\| ${rule.threshold}\\b`),
|
||||
`runbook row for ${rule.id} must state its real numeric threshold ${rule.threshold}`,
|
||||
);
|
||||
}
|
||||
const backlogRule = DEFAULT_ALERT_RULES.find(rule => rule.id === 'queue.backlog');
|
||||
assert.equal(backlogRule.threshold, 10, 'queue.backlog must fire at exactly depth 10');
|
||||
assert.match(runbook, /depth >= 10/, 'runbook must state the queue.backlog boundary as depth >= 10');
|
||||
});
|
||||
|
||||
test('runbook documents the outage drill anchor, manual fallback, and privacy boundary', async () => {
|
||||
const runbook = await readFile(runbookPath, 'utf8');
|
||||
assert.match(runbook, /^## Simulate a worker outage$/m);
|
||||
|
|
|
|||
|
|
@ -26,16 +26,18 @@ test('sanitizer keeps bounded sanitized release evidence intact', () => {
|
|||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass', failureClass: null, latencyMs: 12, counters: {} },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass', failureClass: null, latencyMs: null, counters: { ok: 40, fail: 1 } },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass', failureClass: null, latencyMs: null, counters: { ok: 30, fail: 0 } },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass', failureClass: null, counters: { ok: 40, fail: 1 } },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass', failureClass: null, counters: { ok: 30, fail: 0 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass', failureClass: null, latencyMs: 900, counters: { ok: 25, fail: 0, abstain: 3 } },
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
test('sanitizer drops session identifiers, photo payloads, credentials, and free-text notes', () => {
|
||||
test('sanitizer fail-closes evidence carrying session identifiers, photo payloads, credentials, and free-text notes', () => {
|
||||
// Assembled at runtime so no contiguous private-key marker ever lands in Git history.
|
||||
const privateKeyFixture = `-----BEGIN ${'OPENSSH'} PRIVATE KEY${'-----'}`;
|
||||
// Forbidden-shaped fields are a rejection of the whole evidence object, not
|
||||
// a silent drop: censoring a hostile check would dress the gap up as health.
|
||||
const hostile = {
|
||||
...validEvidence(),
|
||||
sessionToken: 'sess_live_abc123',
|
||||
|
|
@ -44,37 +46,22 @@ test('sanitizer drops session identifiers, photo payloads, credentials, and free
|
|||
operatorEmail: 'someone@example.com',
|
||||
credentialFile: privateKeyFixture,
|
||||
photos: [{ dataBase64: 'aGVsbG8gd29ybGQgaGVsbG8gd29ybGQ=' }],
|
||||
checks: [
|
||||
...validEvidence().checks,
|
||||
{
|
||||
id: 'leak',
|
||||
boundary: 'app',
|
||||
status: 'pass',
|
||||
noteText: 'patient said blood at 3am, see photo hash e3b0c442',
|
||||
imageHash: 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855',
|
||||
base64Payload: 'aGVsbG8=',
|
||||
authHeader: `Bearer ${'sk-'}live-abcdef1234567890`,
|
||||
},
|
||||
],
|
||||
};
|
||||
assert.equal(sanitizeEvidence(hostile).ok, false, 'forbidden top-level fields must reject the whole evidence object');
|
||||
|
||||
const result = sanitizeEvidence(hostile);
|
||||
assert.equal(result.ok, true);
|
||||
const serialized = JSON.stringify(result.evidence);
|
||||
for (const forbidden of [
|
||||
'sess_live_abc123',
|
||||
'secret-cookie-value',
|
||||
'must-not-leak',
|
||||
'someone@example.com',
|
||||
'PRIVATE KEY',
|
||||
'aGVsbG8',
|
||||
'patient said blood',
|
||||
'e3b0c44298fc1c14',
|
||||
'sk-live-abcdef1234567890',
|
||||
]) {
|
||||
assert.equal(serialized.includes(forbidden), false, `sanitized output leaked ${forbidden}`);
|
||||
}
|
||||
assert.equal(result.evidence.checks.some(check => check.id === 'leak'), false);
|
||||
const leakyCheck = {
|
||||
id: 'leak',
|
||||
boundary: 'app',
|
||||
status: 'pass',
|
||||
noteText: 'patient said blood at 3am, see photo hash e3b0c442',
|
||||
imageHash: 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855',
|
||||
base64Payload: 'aGVsbG8=',
|
||||
authHeader: `Bearer ${'sk-'}live-abcdef1234567890`,
|
||||
};
|
||||
const withLeakyCheck = validEvidence();
|
||||
withLeakyCheck.checks = [...validEvidence().checks, leakyCheck];
|
||||
const result = sanitizeEvidence(withLeakyCheck);
|
||||
assert.equal(result.ok, false, 'a check carrying forbidden or free-text fields must reject the whole evidence object');
|
||||
});
|
||||
|
||||
test('sanitizer fail-closes evidence carrying hostile failure-class payloads', () => {
|
||||
|
|
@ -130,3 +117,113 @@ test('sanitizer accepts only the exact privacy-safe failure-class vocabulary in
|
|||
assert.equal(result.ok, false, `non-vocabulary class ${failureClass} must be rejected`);
|
||||
}
|
||||
});
|
||||
|
||||
test('schemaVersion must be the JSON number 1 and nothing else', () => {
|
||||
for (const hostileVersion of ['1', true, false, null, 1.5, 0, 2, [1], { value: 1 }]) {
|
||||
const evidence = { ...validEvidence(), schemaVersion: hostileVersion };
|
||||
const result = sanitizeEvidence(evidence);
|
||||
assert.equal(result.ok, false, `schemaVersion ${JSON.stringify(hostileVersion)} must be rejected`);
|
||||
}
|
||||
});
|
||||
|
||||
test('counters and latency must be actual JSON numbers, never strings, booleans, nulls, or fractions', () => {
|
||||
for (const counters of [{ ok: '40' }, { ok: true }, { fail: null }, { depth: 10.5 }, { depth: -1 }, { retry: Infinity }]) {
|
||||
const evidence = validEvidence();
|
||||
evidence.checks[0] = { ...evidence.checks[0], counters };
|
||||
const result = sanitizeEvidence(evidence);
|
||||
assert.equal(result.ok, false, `counters ${JSON.stringify(counters)} must be rejected`);
|
||||
}
|
||||
for (const latencyMs of ['12', '', true, false, null, 12.5, -1, Infinity]) {
|
||||
const evidence = validEvidence();
|
||||
evidence.checks[0] = { ...evidence.checks[0], latencyMs };
|
||||
const result = sanitizeEvidence(evidence);
|
||||
assert.equal(result.ok, false, `latencyMs ${JSON.stringify(latencyMs)} must be rejected`);
|
||||
}
|
||||
});
|
||||
|
||||
test('generatedAtUtc must be canonical real UTC that survives a Date round-trip', () => {
|
||||
const invalidTimestamps = [
|
||||
'2026-02-30T12:00:00Z',
|
||||
'2026-08-22T25:00:00Z',
|
||||
'2026-08-22T12:61:00Z',
|
||||
'2026-13-01T00:00:00Z',
|
||||
'2026-08-22T12:00:00+00:00',
|
||||
'2026-08-22t12:00:00z',
|
||||
'2026-08-22 12:00:00Z',
|
||||
'not-a-timestamp',
|
||||
1234567890,
|
||||
true,
|
||||
null,
|
||||
];
|
||||
for (const generatedAtUtc of invalidTimestamps) {
|
||||
const evidence = { ...validEvidence(), generatedAtUtc };
|
||||
const result = sanitizeEvidence(evidence);
|
||||
assert.equal(result.ok, false, `generatedAtUtc ${JSON.stringify(generatedAtUtc)} must be rejected`);
|
||||
}
|
||||
});
|
||||
|
||||
test('releaseTag and commit must have their exact safe types', () => {
|
||||
for (const releaseTag of [42, true, null, {}, [], '']) {
|
||||
const evidence = { ...validEvidence(), releaseTag };
|
||||
const result = sanitizeEvidence(evidence);
|
||||
assert.equal(result.ok, false, `releaseTag ${JSON.stringify(releaseTag)} must be rejected`);
|
||||
}
|
||||
for (const commit of [0xca31e6d38bec, true, null, {}, []]) {
|
||||
const evidence = { ...validEvidence(), commit };
|
||||
const result = sanitizeEvidence(evidence);
|
||||
assert.equal(result.ok, false, `commit ${JSON.stringify(commit)} must be rejected`);
|
||||
}
|
||||
});
|
||||
|
||||
test('one malformed check rejects the entire evidence object instead of vanishing beside healthy checks', () => {
|
||||
const malformedChecks = [
|
||||
{ boundary: 'app', status: 'pass' },
|
||||
{ id: 'BAD_ID', boundary: 'app', status: 'pass' },
|
||||
{ id: 42, boundary: 'app', status: 'pass' },
|
||||
{ id: 'app.healthz', status: 'pass' },
|
||||
{ id: 'app.healthz', boundary: 'bus', status: 'pass' },
|
||||
{ id: 'app.healthz', boundary: 42, status: 'pass' },
|
||||
{ id: 'app.healthz', boundary: 'app' },
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'healthy' },
|
||||
{ id: 'app.healthz', boundary: 'app', status: 42 },
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass', counters: 'many' },
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass', counters: [] },
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass', failureClass: 42 },
|
||||
];
|
||||
for (const malformed of malformedChecks) {
|
||||
const evidence = validEvidence();
|
||||
evidence.checks = [...validEvidence().checks, malformed];
|
||||
const result = sanitizeEvidence(evidence);
|
||||
assert.equal(result.ok, false, `malformed check ${JSON.stringify(malformed)} must reject the whole evidence object`);
|
||||
}
|
||||
});
|
||||
|
||||
test('a passing or unknown check can never carry a failureClass, so all-pass evidence can never page worker.outage', () => {
|
||||
const baseChecks = validEvidence().checks;
|
||||
for (const status of ['pass', 'unknown']) {
|
||||
for (const failureClass of ['worker.unavailable', 'model.error', 'vision.timeout']) {
|
||||
const evidence = validEvidence();
|
||||
evidence.checks = baseChecks.map(check =>
|
||||
check.boundary === 'app' ? { ...check, status, failureClass } : check,
|
||||
);
|
||||
const result = sanitizeEvidence(evidence);
|
||||
assert.equal(
|
||||
result.ok,
|
||||
false,
|
||||
`status ${status} carrying failureClass ${failureClass} must reject the whole evidence object`,
|
||||
);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('sanitization is idempotent: re-sanitizing sanitized evidence changes nothing and omits nullable latency', () => {
|
||||
const first = sanitizeEvidence(validEvidence());
|
||||
assert.equal(first.ok, true);
|
||||
assert.equal(Object.hasOwn(first.evidence.checks[0], 'latencyMs'), true, 'present latency must stay present');
|
||||
assert.equal(Object.hasOwn(first.evidence.checks[1], 'latencyMs'), false, 'omitted latency must stay omitted');
|
||||
|
||||
const second = sanitizeEvidence(first.evidence);
|
||||
assert.equal(second.ok, true, 'sanitized evidence must survive a second sanitization pass');
|
||||
assert.deepEqual(second.evidence, first.evidence, 'second pass must not rewrite omitted latency to a different shape');
|
||||
assert.equal(JSON.stringify(second.evidence).includes('"latencyMs":0'), false, 'no zero-latency fabrication');
|
||||
});
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user