fix(ops): harden release observability against hostile review findings
All checks were successful
Quality gates / quality (pull_request) Successful in 1m53s
All checks were successful
Quality gates / quality (pull_request) Successful in 1m53s
Strict RED-GREEN TDD over PR review blockers; every fix landed test-first with the failing run observed before implementation. - failureClass: closed privacy-safe vocabulary (worker.unavailable, vision.timeout, model.error) under strict dotted slug grammar. Newlines, carriage returns, ANSI/control characters, secrets, medical text, SQL, oversized values, and off-vocabulary classes fail the entire evidence file closed; nothing hostile can reach rendering. - depth: added to the counter vocabulary so queue.backlog can fire at all; bounded counters (0..1,000,000, integer) fail closed above the ceiling. 9/10/11 edge tests pin silent/at-threshold/above-threshold; backlog is suppressed while worker.outage pages (depth is residual from the same incident) and returns the moment the page clears. - drill integrity: the pre-drill baseline is now read from the fixture itself and must be genuinely healthy; already-outaged fixtures are refused with exit 2 without touching their switch, and a flip that produces no real healthy-to-outage transition reports DRILL FAIL instead of passing vacuously. - manual fallback: deterministic contract replaces the tautology. available+none-required when healthy, available+local-journal under any degradation, unavailable+app-down only when the app boundary itself is down. - fail-closed telemetry: evidence missing any of the four boundaries is rejected; unknown statuses surface as a warn telemetry.gap alert with owner/threshold/runbook instead of passing as healthy (documented in the runbook inventory). - drill origin: validateLoopbackOrigin gates every network path before any fetch. Only a bare http://127.0.0.1:<port> URL passes; credentials, DNS names, hex/decimal/percent-encoded IP encodings, IPv6 forms, paths, queries, fragments, and non-http schemes are refused pre-contact (raw-string grammar gate plus parse round-trip, because the URL parser canonicalizes hostile encodings). - terminal safety: controlSafe() strips C0/C1 control characters from all dynamically produced CLI output so hostile evidence paths cannot inject ANSI escapes into a terminal. Gates: npm test 98/98, check:syntax, npm audit (0 vulns), check:diff, deploy_staging status read-only; 30 adversarial probes against sanitizer, alert edges, and live loopback CLI all pass. No merge, no deploy.
This commit is contained in:
parent
ad8c8a9f4c
commit
9c9286b59f
|
|
@ -12,7 +12,7 @@ Telemetry in this system is release telemetry, not user telemetry. Evidence file
|
|||
- infrastructure addresses beyond the loopback `127.0.0.1` drill fixture;
|
||||
- user identity of any kind (no emails, usernames, device IDs).
|
||||
|
||||
`src/release-observability.js` enforces this mechanically: `sanitizeEvidence()` allowlists bounded fields (release tag, full commit, UTC timestamp, per-check id/boundary/status/failure-class/latency/counters) and drops anything else — including any check whose extra fields look like keys, sessions, base64 payloads, image hashes, notes, or free text. If sanitization fails, the tool exits nonzero and echoes nothing from the input. The sanitizer itself is tested with hostile payloads (`tests/release-observability.test.js`) and the CLI is tested to never echo rejected content (`tests/release-dashboard-cli.test.js`).
|
||||
`src/release-observability.js` enforces this mechanically: `sanitizeEvidence()` allowlists bounded fields (release tag, full commit, UTC timestamp, per-check id/boundary/status/failure-class/latency/counters) and drops anything else — including any check whose extra fields look like keys, sessions, base64 payloads, image hashes, notes, or free text. Failure classes are a closed vocabulary (`worker.unavailable`, `vision.timeout`, `model.error`) matched against a strict dotted lowercase slug grammar; anything else — newlines, ANSI/control characters, secrets, medical text, unknown classes — fails the whole evidence file closed rather than being rendered. Counters are bounded non-negative integers capped at 1,000,000. Every one of the four boundaries (app, api, queue, model) must be present; partial telemetry is refused instead of rendered as silent health. If sanitization fails, the tool exits nonzero and echoes nothing from the input, and all CLI output is scrubbed of control characters so hostile file paths can never inject terminal escapes. The sanitizer itself is tested with hostile payloads (`tests/release-observability.test.js`) and the CLI is tested to never echo rejected content (`tests/release-dashboard-cli.test.js`).
|
||||
|
||||
## Dashboard
|
||||
|
||||
|
|
@ -54,9 +54,10 @@ Alerts are evaluated by `evaluateAlerts()` over the sanitized dashboard. Every r
|
|||
| Alert id | Severity | Owner | Threshold (observed >=) | Condition | Runbook |
|
||||
|-----------------|----------|------------------|-------------------------|--------------------------------------------------|---------|
|
||||
| `worker.outage` | page | release-operator | 1 | any check reports `worker.unavailable`, or both model and queue boundaries report failures | [Simulate a worker outage](#simulate-a-worker-outage) |
|
||||
| `queue.backlog` | warn | release-operator | depth >= 10 | queue depth counter at or above 10 | [Alert inventory](#alert-inventory) |
|
||||
| `telemetry.gap` | warn | release-operator | 1 | one or more checks report status `unknown` (missing telemetry fails closed as a warning, never as health) | [Alert inventory](#alert-inventory) |
|
||||
| `queue.backlog` | warn | release-operator | depth >= 10 | queue depth counter at or above 10; suppressed while `worker.outage` pages (depth is residual from the same incident) | [Alert inventory](#alert-inventory) |
|
||||
| `vision.degraded` | warn | vision-owner | 3 | three or more `vision.timeout` failure classes | [Alert inventory](#alert-inventory) |
|
||||
| `app.unhealthy` | page | release-operator | 1 | app boundary healthz reports fail | [Alert inventory](#alert-inventory) |
|
||||
| `app.unhealthy` | page | release-operator | 1 | app boundary healthz reports fail; this is also the only condition that makes the manual fallback unavailable (`reason: app-down`) | [Alert inventory](#alert-inventory) |
|
||||
|
||||
Escalation: pages go to the release operator on call (currently Alexander as release owner); warns are batched into the next release-review pulse. An unresolved page after one manual fallback verification is raised in the Gitea issue for the affected release epic rather than paged repeatedly.
|
||||
|
||||
|
|
@ -65,7 +66,7 @@ Escalation: pages go to the release operator on call (currently Alexander as rel
|
|||
This drill proves the issue #41 acceptance criterion: one simulated outage produces exactly one actionable page alert while the graceful manual fallback stays available. It runs entirely against a throwaway loopback fixture — never against staging or production.
|
||||
|
||||
1. Start any loopback fixture that serves:
|
||||
- `GET /api/drill/checks` returning `{ "checks": [...] }` (healthy first);
|
||||
- `GET /api/drill/checks` returning `{ "checks": [...] }`, healthy first, with at least one check for each boundary (app, api, queue, model);
|
||||
- `POST /drill/outage` flipping the fixture's simulated worker off (subsequent `/api/drill/checks` responses then report `status: "fail"`, `failureClass: "worker.unavailable"` across the api, queue, and model checks).
|
||||
2. Run the drill:
|
||||
|
||||
|
|
@ -73,9 +74,11 @@ This drill proves the issue #41 acceptance criterion: one simulated outage produ
|
|||
node scripts/release_dashboard.mjs --drill-origin http://127.0.0.1:<port>
|
||||
```
|
||||
|
||||
3. Required outcome (the automated test asserts all of it):
|
||||
- pre-drill baseline renders zero alerts;
|
||||
- after the flip, exactly one page alert fires — `worker.outage`;
|
||||
The origin must be a bare `http://127.0.0.1:<port>` URL. Credentials (`user:pass@`), DNS names, alternative IP encodings (hex octets, decimal integers, percent-encoding), IPv6 forms, non-http schemes, and any path or query are rejected before the CLI contacts anything.
|
||||
|
||||
3. Required outcome (the automated tests assert all of it):
|
||||
- the pre-drill baseline is read from the fixture itself and must be genuinely healthy — an already-outaged fixture is refused with exit code 2 and its switch is never touched;
|
||||
- after the flip, exactly one page alert fires — `worker.outage`; if the flip produced no real healthy-to-outage transition, the drill reports `DRILL FAIL` with exit code 1 instead of passing vacuously;
|
||||
- the alert names owner `release-operator`, its threshold, and this runbook section;
|
||||
- `manual fallback: available — local journal remains usable`;
|
||||
- exit code `0` (`DRILL PASS`).
|
||||
|
|
@ -97,4 +100,8 @@ The drill is exercised end-to-end by `tests/release-dashboard-cli.test.js` ("out
|
|||
|
||||
- `npm test` — includes `tests/release-observability.test.js`, `tests/release-dashboard.test.js`, `tests/release-dashboard-cli.test.js`.
|
||||
- Sanitizer hostile-payload test proves session tokens, cookies, environment dumps, private keys, base64 payloads, image hashes, note text, and emails cannot survive into dashboard output.
|
||||
- CLI tests prove rejected evidence exits nonzero without echoing contents, and the outage drill yields exactly one actionable alert with graceful manual fallback.
|
||||
- Hostile failure-class test proves newlines, carriage returns, ANSI escapes, control characters, secrets, medical text, SQL, oversized values, wrong types, and off-vocabulary classes all fail closed.
|
||||
- Depth-counter tests prove the `queue.backlog` warn stays silent at 9, fires at exactly 10, reports observed counts above threshold, sums across queue checks, is suppressed while `worker.outage` pages, and cannot be smuggled in above the bounded counter ceiling.
|
||||
- Boundary-completeness tests prove evidence missing any of app, api, queue, or model fails closed; unknown statuses surface as a `telemetry.gap` warn instead of passing as healthy.
|
||||
- Manual-fallback tests prove a deterministic contract: journal available with no degradation, active (`local-journal`) under any non-fatal degradation, and unavailable only when the app boundary itself is down.
|
||||
- CLI tests prove rejected evidence exits nonzero without echoing contents, drill origins are validated to bare loopback http before any network contact (credentials, DNS names, hex/decimal/percent-encoded IP encodings, IPv6, paths, queries, and foreign schemes are refused), the outage drill reads a genuine fixture baseline, refuses already-down fixtures without touching their switch, fails when no real transition occurs, and yields exactly one actionable alert with graceful manual fallback.
|
||||
|
|
|
|||
|
|
@ -11,6 +11,14 @@ function argValue(args, flag) {
|
|||
return args[index + 1] || null;
|
||||
}
|
||||
|
||||
// Terminal safety: strip C0/C1 control characters (newline excepted) from any
|
||||
// dynamically produced text before it reaches stdout/stderr, so hostile file
|
||||
// paths or fixture payloads can never smuggle ANSI escapes into a terminal.
|
||||
function controlSafe(text) {
|
||||
return String(text)
|
||||
.replace(/[\u0000-\u0009\u000B-\u001F\u007F-\u009F]/g, '');
|
||||
}
|
||||
|
||||
async function loadEvidence(source) {
|
||||
if (source.drillOrigin) {
|
||||
const response = await fetch(`${source.drillOrigin.replace(/\/$/, '')}/api/drill/checks`, { signal: AbortSignal.timeout(5_000) });
|
||||
|
|
@ -18,7 +26,7 @@ async function loadEvidence(source) {
|
|||
const body = await response.json();
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
releaseTag: controlSafe('daily-2026-08-22.1'),
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: new Date().toISOString().replace(/\.\d{3}Z$/, 'Z'),
|
||||
checks: Array.isArray(body.checks) ? body.checks : [],
|
||||
|
|
@ -77,11 +85,23 @@ async function main() {
|
|||
return 2;
|
||||
}
|
||||
|
||||
// Gate every network path before any fetch can happen: a drill may only
|
||||
// ever address a validated bare loopback origin.
|
||||
let validatedOrigin = null;
|
||||
if (drillOrigin) {
|
||||
try {
|
||||
validatedOrigin = validateLoopbackOrigin(drillOrigin);
|
||||
} catch (error) {
|
||||
console.error(error.message);
|
||||
return 2;
|
||||
}
|
||||
}
|
||||
|
||||
let rawEvidence;
|
||||
try {
|
||||
rawEvidence = await loadEvidence({ drillOrigin, evidencePath });
|
||||
rawEvidence = await loadEvidence({ drillOrigin: validatedOrigin, evidencePath });
|
||||
} catch (error) {
|
||||
console.error(`release_dashboard: cannot read sanitized evidence (${error.message}). No contents are echoed.`);
|
||||
console.error(`release_dashboard: cannot read sanitized evidence (${controlSafe(error.message)}). No contents are echoed.`);
|
||||
return 2;
|
||||
}
|
||||
|
||||
|
|
@ -97,29 +117,84 @@ async function main() {
|
|||
console.log('');
|
||||
console.log(`manual fallback: ${dashboard.manualFallback.available ? 'available' : 'unavailable'} — ${fallbackReason(dashboard.manualFallback.reason)}`);
|
||||
|
||||
if (drillOrigin) return runDrill({ dashboard, alerts, drillOrigin });
|
||||
if (drillOrigin) return runDrill({ drillOrigin: validatedOrigin });
|
||||
return alerts.some(alert => alert.severity === 'page') ? 1 : 0;
|
||||
}
|
||||
|
||||
function fallbackReason(reason) {
|
||||
if (reason === 'local-journal') return 'local journal remains usable';
|
||||
return 'no outage detected; fallback not required';
|
||||
if (reason === 'app-down') return 'app boundary is down; local journal cannot be served';
|
||||
return 'no degradation detected; fallback not required';
|
||||
}
|
||||
|
||||
async function runDrill({ dashboard, alerts, drillOrigin }) {
|
||||
// A drill may only ever talk to a throwaway fixture on 127.0.0.1. The origin
|
||||
// must be a bare http URL whose host is exactly the loopback address: no
|
||||
// credentials, no DNS names, no alternative IP encodings, no IPv6, no path,
|
||||
// query, or fragment, and no scheme other than http.
|
||||
const LOOPBACK_ORIGIN_PATTERN = /^http:\/\/127\.0\.0\.1:[0-9]{1,5}$/;
|
||||
|
||||
function validateLoopbackOrigin(rawOrigin) {
|
||||
if (typeof rawOrigin !== 'string') {
|
||||
throw new Error('drill origin rejected: expected an http://127.0.0.1:<port> URL');
|
||||
}
|
||||
// Grammar gate runs on the RAW string first: the WHATWG URL parser
|
||||
// canonicalizes hostile host encodings (hex octets, decimal IP integers,
|
||||
// percent-encoded dots) into a clean-looking 127.0.0.1, so a normalized
|
||||
// string can never be trusted on its own.
|
||||
if (!LOOPBACK_ORIGIN_PATTERN.test(rawOrigin)) {
|
||||
throw new Error('drill origin rejected: only a bare http://127.0.0.1:<port> loopback origin is permitted');
|
||||
}
|
||||
let parsed;
|
||||
try {
|
||||
parsed = new URL(rawOrigin);
|
||||
} catch {
|
||||
throw new Error('drill origin rejected: not a valid URL');
|
||||
}
|
||||
// Round-trip defense in depth: the parsed form must be byte-identical to
|
||||
// what was supplied, proving no hidden credentials, path, query, fragment,
|
||||
// or alternative host encoding was smuggled past the grammar.
|
||||
if (parsed.toString().replace(/\/$/, '') !== rawOrigin) {
|
||||
throw new Error('drill origin rejected: URL did not round-trip as a bare loopback origin');
|
||||
}
|
||||
if (parsed.protocol !== 'http:' || parsed.hostname !== '127.0.0.1') {
|
||||
throw new Error('drill origin rejected: host must be exactly http://127.0.0.1');
|
||||
}
|
||||
const port = Number(parsed.port);
|
||||
if (!Number.isInteger(port) || port < 1 || port > 65535) {
|
||||
throw new Error('drill origin rejected: port must be between 1 and 65535');
|
||||
}
|
||||
return rawOrigin;
|
||||
}
|
||||
|
||||
async function runDrill({ drillOrigin }) {
|
||||
console.log('');
|
||||
console.log('WORKER OUTAGE DRILL (simulated fixture on loopback; no live host is contacted)');
|
||||
const preDrillAlerts = evaluateAlerts(buildDashboard(sanitizeEvidence({
|
||||
schemaVersion: 1,
|
||||
releaseTag: dashboard.identity.releaseTag,
|
||||
commit: dashboard.identity.commit,
|
||||
generatedAtUtc: dashboard.identity.generatedAtUtc,
|
||||
checks: [{ id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 }],
|
||||
}).evidence));
|
||||
console.log(`pre-drill: ${preDrillAlerts.length} alerts`);
|
||||
let origin;
|
||||
try {
|
||||
origin = validateLoopbackOrigin(drillOrigin);
|
||||
} catch (error) {
|
||||
console.error(error.message);
|
||||
return 2;
|
||||
}
|
||||
|
||||
// Genuine baseline: read the fixture's own current state. A drill may only
|
||||
// start from proven health — never from an assumed or already-outaged state.
|
||||
const preRaw = await loadEvidence({ drillOrigin });
|
||||
const preValidated = sanitizeEvidence(preRaw);
|
||||
if (!preValidated.ok) {
|
||||
console.error('release_dashboard: pre-drill fixture evidence failed sanitization. Nothing was rendered.');
|
||||
return 2;
|
||||
}
|
||||
const preAlerts = evaluateAlerts(buildDashboard(preValidated.evidence));
|
||||
const prePages = preAlerts.filter(alert => alert.severity === 'page');
|
||||
if (prePages.length > 0) {
|
||||
console.error(`release_dashboard: fixture is not healthy before the drill (${prePages.length} page alert[s]); refusing to flip an already-down worker.`);
|
||||
return 2;
|
||||
}
|
||||
console.log('pre-drill: healthy (0 page alerts)');
|
||||
|
||||
try {
|
||||
const response = await fetch(`${drillOrigin.replace(/\/$/, '')}/drill/outage`, { method: 'POST', signal: AbortSignal.timeout(5_000) });
|
||||
const response = await fetch(`${origin}/drill/outage`, { method: 'POST', signal: AbortSignal.timeout(5_000) });
|
||||
if (!response.ok) throw new Error(`fixture returned ${response.status}`);
|
||||
} catch (error) {
|
||||
console.error(`release_dashboard: drill switch failed (${error.message})`);
|
||||
|
|
@ -136,12 +211,17 @@ async function runDrill({ dashboard, alerts, drillOrigin }) {
|
|||
const postAlerts = evaluateAlerts(postDashboard);
|
||||
const pageAlerts = postAlerts.filter(alert => alert.severity === 'page');
|
||||
|
||||
if (pageAlerts.length === 0) {
|
||||
console.log('no outage observed after flip: the fixture never transitioned from healthy to outage.');
|
||||
console.log('DRILL FAIL');
|
||||
return 1;
|
||||
}
|
||||
console.log(`post-outage: exactly 1 alert expected, found ${pageAlerts.length}`);
|
||||
for (const alert of pageAlerts) {
|
||||
console.log(` ${alert.severity.toUpperCase()} ${alert.message.replace(/^PAGE /, '')}`);
|
||||
console.log(` owner ${alert.owner} · threshold ${alert.threshold} · runbook ${alert.runbook}`);
|
||||
}
|
||||
console.log(`manual fallback: ${postDashboard.manualFallback.available ? 'AVAILABLE' : 'UNAVAILABLE'} — local journal remains usable`);
|
||||
console.log(`manual fallback: ${postDashboard.manualFallback.available ? 'AVAILABLE' : 'UNAVAILABLE'} — ${fallbackReason(postDashboard.manualFallback.reason)}`);
|
||||
const pass = pageAlerts.length === 1
|
||||
&& pageAlerts[0].id === 'worker.outage'
|
||||
&& postDashboard.manualFallback.available === true;
|
||||
|
|
|
|||
|
|
@ -2,6 +2,12 @@ const SCHEMA_VERSION = 1;
|
|||
const BOUNDARIES = new Set(['app', 'api', 'queue', 'model']);
|
||||
const STATUSES = new Set(['pass', 'fail', 'degraded', 'unknown']);
|
||||
|
||||
// Closed privacy-safe vocabulary: a failure class can never carry free text,
|
||||
// secrets, medical notes, control characters, or terminal escapes because only
|
||||
// these dotted lowercase slugs may ever reach rendering.
|
||||
const FAILURE_CLASSES = new Set(['worker.unavailable', 'vision.timeout', 'model.error']);
|
||||
const FAILURE_CLASS_PATTERN = /^[a-z][a-z0-9]*(?:\.[a-z][a-z0-9]*){1,2}$/;
|
||||
|
||||
const FORBIDDEN_TOP_KEYS = new Set([
|
||||
'session', 'sessionid', 'sessionidtoken', 'sessiontoken', 'cookie', 'cookies',
|
||||
'authorization', 'authheader', 'apikey', 'api_key', 'password', 'credential',
|
||||
|
|
@ -21,7 +27,8 @@ const SECRET_VALUE_PATTERNS = [
|
|||
/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/,
|
||||
];
|
||||
|
||||
const COUNTER_KEYS = new Set(['ok', 'fail', 'abstain', 'retry', 'timeout', 'rejected', 'fallback']);
|
||||
const COUNTER_KEYS = new Set(['ok', 'fail', 'abstain', 'retry', 'timeout', 'rejected', 'fallback', 'depth']);
|
||||
const COUNTER_CEILING = 1_000_000;
|
||||
|
||||
function looksSensitive(value) {
|
||||
const text = typeof value === 'string' ? value : JSON.stringify(value);
|
||||
|
|
@ -35,7 +42,8 @@ function boundedCounters(raw) {
|
|||
for (const key of Object.keys(raw).sort()) {
|
||||
if (!COUNTER_KEYS.has(key)) continue;
|
||||
const value = Number(raw[key]);
|
||||
if (Number.isFinite(value) && value >= 0 && Number.isInteger(value)) counters[key] = value;
|
||||
if (!Number.isFinite(value) || value < 0 || !Number.isInteger(value) || value > COUNTER_CEILING) return null;
|
||||
counters[key] = value;
|
||||
}
|
||||
return counters;
|
||||
}
|
||||
|
|
@ -56,6 +64,11 @@ function buildCheck(raw) {
|
|||
const status = String(raw.status || '');
|
||||
if (!STATUSES.has(status)) return { dropped: true };
|
||||
const failureClass = raw.failureClass == null ? null : String(raw.failureClass);
|
||||
if (failureClass !== null && (!FAILURE_CLASSES.has(failureClass) || !FAILURE_CLASS_PATTERN.test(failureClass))) {
|
||||
return { rejected: true };
|
||||
}
|
||||
const counters = boundedCounters(raw.counters);
|
||||
if (counters === null) return { rejected: true };
|
||||
return {
|
||||
check: {
|
||||
id,
|
||||
|
|
@ -63,7 +76,7 @@ function buildCheck(raw) {
|
|||
status,
|
||||
failureClass,
|
||||
latencyMs: boundedLatency(raw.latencyMs),
|
||||
counters: boundedCounters(raw.counters),
|
||||
counters,
|
||||
},
|
||||
dropped: Object.keys(raw).some(key => {
|
||||
if (['id', 'boundary', 'status', 'failureClass', 'latencyMs', 'counters'].includes(key)) return false;
|
||||
|
|
@ -90,10 +103,18 @@ export function sanitizeEvidence(input) {
|
|||
const checks = [];
|
||||
for (const raw of input.checks) {
|
||||
const built = buildCheck(raw);
|
||||
if (built.rejected) return { ok: false, evidence: null };
|
||||
if (built.dropped) continue;
|
||||
checks.push(built.check);
|
||||
}
|
||||
if (checks.length === 0) return { ok: false, evidence: null };
|
||||
// Fail closed on partial telemetry: an evidence set that is missing any of
|
||||
// the four boundaries cannot be distinguished from a censored one, so the
|
||||
// dashboard refuses it instead of rendering silent gaps as healthy.
|
||||
const presentBoundaries = new Set(checks.map(check => check.boundary));
|
||||
for (const boundary of BOUNDARIES) {
|
||||
if (!presentBoundaries.has(boundary)) return { ok: false, evidence: null };
|
||||
}
|
||||
if (Number(input.schemaVersion) !== SCHEMA_VERSION) return { ok: false, evidence: null };
|
||||
|
||||
return {
|
||||
|
|
@ -138,9 +159,16 @@ export function buildDashboard(evidence) {
|
|||
totals[check.status] += 1;
|
||||
}
|
||||
|
||||
const nonAppUnhealthy = ['api', 'queue', 'model'].some(
|
||||
boundary => boundaries[boundary].fail > 0 || boundaries[boundary].degraded > 0,
|
||||
);
|
||||
// Deterministic manual-fallback contract: the local journal is a usable
|
||||
// fallback exactly while the app boundary itself can still serve it. Any
|
||||
// degradation anywhere makes the journal the active fallback ('local-journal');
|
||||
// only an app-boundary failure removes it ('app-down').
|
||||
const anyDegradation = totals.fail > 0 || totals.degraded > 0;
|
||||
const manualFallback = boundaries.app.fail > 0
|
||||
? { available: false, reason: 'app-down' }
|
||||
: anyDegradation
|
||||
? { available: true, reason: 'local-journal' }
|
||||
: { available: true, reason: 'none-required' };
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
|
|
@ -148,9 +176,7 @@ export function buildDashboard(evidence) {
|
|||
boundaries,
|
||||
totals,
|
||||
failureClasses,
|
||||
manualFallback: nonAppUnhealthy
|
||||
? { available: true, reason: 'local-journal' }
|
||||
: { available: true, reason: 'none-required' },
|
||||
manualFallback,
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -169,6 +195,14 @@ export const DEFAULT_ALERT_RULES = [
|
|||
: 0,
|
||||
),
|
||||
},
|
||||
{
|
||||
id: 'telemetry.gap',
|
||||
severity: 'warn',
|
||||
owner: 'release-operator',
|
||||
threshold: 1,
|
||||
runbook: 'docs/RELEASE-OBSERVABILITY.md#alert-inventory',
|
||||
count: dashboard => dashboard.totals.unknown,
|
||||
},
|
||||
{
|
||||
id: 'queue.backlog',
|
||||
severity: 'warn',
|
||||
|
|
@ -206,6 +240,10 @@ export function evaluateAlerts(dashboard, rules = DEFAULT_ALERT_RULES) {
|
|||
count = 0;
|
||||
}
|
||||
if (!Number.isFinite(count) || count <= 0 || count < rule.threshold) continue;
|
||||
// Correlation: while a worker outage is paging, queue depth is residual
|
||||
// noise from the same incident, so the backlog warn would only duplicate
|
||||
// the page. It stays active the moment the page clears.
|
||||
if (rule.id === 'queue.backlog' && alerts.some(alert => alert.id === 'worker.outage')) continue;
|
||||
alerts.push({
|
||||
id: rule.id,
|
||||
severity: rule.severity,
|
||||
|
|
|
|||
|
|
@ -82,14 +82,39 @@ test('cli exits nonzero on unsanitizable evidence and never echoes its contents'
|
|||
}
|
||||
});
|
||||
|
||||
async function startDrillServer() {
|
||||
test('cli output stays control-safe when rejected evidence paths carry ANSI escapes', async () => {
|
||||
const evilPath = join(tmpdir(), `timmy-\x1b[31mred\rbell\x07.json`);
|
||||
const result = await runCli(['--evidence', evilPath]);
|
||||
assert.equal(result.code, 2);
|
||||
const emitted = result.stdout + result.stderr;
|
||||
assert.doesNotMatch(emitted, /\x1b/, 'ANSI escape reached the terminal');
|
||||
assert.doesNotMatch(emitted, /\x07/, 'bell control character reached the terminal');
|
||||
assert.doesNotMatch(emitted, /\r/, 'carriage return reached the terminal');
|
||||
});
|
||||
|
||||
test('successful dashboard rendering emits no control characters besides newlines', async () => {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'timmy-obs-'));
|
||||
try {
|
||||
const evidencePath = join(dir, 'evidence.json');
|
||||
await writeFile(evidencePath, JSON.stringify(healthyEvidence));
|
||||
const result = await runCli(['--evidence', evidencePath]);
|
||||
assert.equal(result.code, 0, result.stderr);
|
||||
assert.doesNotMatch(result.stdout, /[\u0000-\u0009\u000B-\u001F\u007F-\u009F]/, 'control character reached the terminal');
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
async function startDrillServer({ startDown = false, flipHasNoEffect = false } = {}) {
|
||||
const port = nextPort++;
|
||||
const origin = `http://127.0.0.1:${port}`;
|
||||
let workerUp = true;
|
||||
let workerUp = !startDown;
|
||||
let outageSwitchCount = 0;
|
||||
const app = http.createServer((req, res) => {
|
||||
const url = new URL(req.url, origin);
|
||||
if (url.pathname === '/drill/outage' && req.method === 'POST') {
|
||||
workerUp = false;
|
||||
outageSwitchCount += 1;
|
||||
if (!flipHasNoEffect) workerUp = false;
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ ok: true }));
|
||||
return;
|
||||
|
|
@ -100,13 +125,16 @@ async function startDrillServer() {
|
|||
return;
|
||||
}
|
||||
if (url.pathname === '/api/drill/checks') {
|
||||
const appCheck = { id: 'app.healthz', boundary: 'app', status: 'pass', latencyMs: 12 };
|
||||
const checks = workerUp
|
||||
? [
|
||||
appCheck,
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass', counters: { ok: 5, fail: 0 } },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass', counters: { depth: 1, ok: 5, fail: 0 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass', latencyMs: 800, counters: { ok: 4, fail: 0 } },
|
||||
]
|
||||
: [
|
||||
appCheck,
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'fail', failureClass: 'worker.unavailable' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'fail', failureClass: 'worker.unavailable', counters: { depth: 17 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'fail', failureClass: 'worker.unavailable', counters: { timeout: 4 } },
|
||||
|
|
@ -119,19 +147,88 @@ async function startDrillServer() {
|
|||
res.end('Not found');
|
||||
});
|
||||
await new Promise(resolve => app.listen(port, '127.0.0.1', resolve));
|
||||
return { origin, close: () => new Promise(resolve => app.close(() => resolve())) };
|
||||
return {
|
||||
origin,
|
||||
close: () => new Promise(resolve => app.close(() => resolve())),
|
||||
get outageSwitchCount() { return outageSwitchCount; },
|
||||
};
|
||||
}
|
||||
|
||||
test('outage drill flips one simulated switch and reports exactly one page alert plus manual fallback', async t => {
|
||||
test('outage drill reads the genuine fixture baseline, flips once, and reports exactly one page alert plus manual fallback', async t => {
|
||||
const server = await startDrillServer();
|
||||
t.after(() => server.close());
|
||||
|
||||
const result = await runCli(['--drill-origin', server.origin]);
|
||||
assert.equal(result.code, 0, result.stderr);
|
||||
assert.match(result.stdout, /WORKER OUTAGE DRILL/);
|
||||
assert.match(result.stdout, /pre-drill: 0 alerts/i);
|
||||
assert.match(result.stdout, /pre-drill: healthy \(0 page alerts\)/i);
|
||||
assert.equal(server.outageSwitchCount, 1, 'drill must flip the simulated switch exactly once');
|
||||
assert.match(result.stdout, /post-outage: exactly 1 alert/i);
|
||||
assert.match(result.stdout, /PAGE worker\.unavailable on daily-2026-08-22\.1/);
|
||||
assert.match(result.stdout, /manual fallback: AVAILABLE — local journal remains usable/);
|
||||
assert.match(result.stdout, /DRILL PASS/);
|
||||
});
|
||||
|
||||
test('drill refuses to flip an already-outaged fixture and never touches its switch', async t => {
|
||||
const server = await startDrillServer({ startDown: true });
|
||||
t.after(() => server.close());
|
||||
|
||||
const result = await runCli(['--drill-origin', server.origin]);
|
||||
assert.equal(result.code, 2, `expected refusal exit 2, got ${result.code}`);
|
||||
assert.match(result.stderr, /not healthy/i);
|
||||
assert.doesNotMatch(result.stdout, /DRILL PASS/);
|
||||
assert.equal(server.outageSwitchCount, 0, 'already-down fixture must not be flipped again');
|
||||
});
|
||||
|
||||
test('drill fails when the flip produces no genuine healthy-to-outage transition', async t => {
|
||||
const server = await startDrillServer({ flipHasNoEffect: true });
|
||||
t.after(() => server.close());
|
||||
|
||||
const result = await runCli(['--drill-origin', server.origin]);
|
||||
assert.equal(result.code, 1);
|
||||
assert.match(result.stdout, /no outage observed after flip/i);
|
||||
assert.match(result.stdout, /DRILL FAIL/);
|
||||
assert.equal(server.outageSwitchCount, 1);
|
||||
});
|
||||
|
||||
test('drill refuses a credentialed origin even when its loopback target is alive', async t => {
|
||||
const server = await startDrillServer();
|
||||
t.after(() => server.close());
|
||||
|
||||
const credentialed = `http://user:pass@127.0.0.1:${new URL(server.origin).port}`;
|
||||
const result = await runCli(['--drill-origin', credentialed]);
|
||||
assert.equal(result.code, 2, `expected refusal exit 2, got ${result.code}`);
|
||||
assert.match(result.stderr, /drill origin rejected/i);
|
||||
assert.doesNotMatch(result.stdout, /DRILL PASS|WORKER OUTAGE DRILL/, 'hostile origin must never reach the drill');
|
||||
assert.equal(server.outageSwitchCount, 0, 'credentialed origin must never touch the fixture');
|
||||
});
|
||||
|
||||
test('drill-origin accepts only validated loopback HTTP and rejects every hostile form without contacting it', async () => {
|
||||
const hostileOrigins = [
|
||||
'https://127.0.0.1:43600',
|
||||
'http://localhost:43600',
|
||||
'ftp://127.0.0.1:43600',
|
||||
'file:///etc/passwd',
|
||||
'http://0x7f.0.0.1:43600',
|
||||
'http://2130706433:43600',
|
||||
'http://[::1]:43600',
|
||||
'http://[::ffff:127.0.0.1]:43600',
|
||||
'http://[0:0:0:0:0:0:0:1]:43600',
|
||||
'http://127.0.0.2:43600',
|
||||
'http://10.0.0.5:43600',
|
||||
'http://example.com:43600',
|
||||
'http://127%2E0%2E0%2E1:43600',
|
||||
'http://127.0.0.1:43600/drill',
|
||||
'http://127.0.0.1:43600/?next=1',
|
||||
'//127.0.0.1:43600',
|
||||
'127.0.0.1:43600',
|
||||
'not a url at all',
|
||||
'null',
|
||||
];
|
||||
for (const origin of hostileOrigins) {
|
||||
const result = await runCli(['--drill-origin', origin]);
|
||||
assert.equal(result.code, 2, `expected refusal exit 2 for origin ${JSON.stringify(origin)}`);
|
||||
assert.match(result.stderr, /drill origin rejected/i, `origin must be rejected as an origin, not merely unreachable: ${JSON.stringify(origin)}`);
|
||||
assert.doesNotMatch(result.stdout, /DRILL PASS|WORKER OUTAGE DRILL/, `hostile origin must never reach the drill: ${JSON.stringify(origin)}`);
|
||||
}
|
||||
});
|
||||
|
|
|
|||
|
|
@ -37,7 +37,9 @@ test('dashboard summarizes privacy-safe failure classes without any payload text
|
|||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'degraded', failureClass: 'vision.timeout' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass' },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'fail', failureClass: 'model.error' },
|
||||
{ id: 'model.inference.warmup', boundary: 'model', status: 'fail', failureClass: 'model.error' },
|
||||
],
|
||||
|
|
@ -99,7 +101,9 @@ test('manual fallback stays available whenever any non-app check is unhealthy',
|
|||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'fail', failureClass: 'worker.unavailable' },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass' },
|
||||
],
|
||||
}).evidence);
|
||||
assert.deepEqual(outage.manualFallback, {
|
||||
|
|
@ -107,3 +111,181 @@ test('manual fallback stays available whenever any non-app check is unhealthy',
|
|||
reason: 'local-journal',
|
||||
});
|
||||
});
|
||||
|
||||
test('manual fallback is unavailable only when the app boundary itself is down', () => {
|
||||
const evidence = sanitizeEvidence({
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'fail' },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'fail', failureClass: 'worker.unavailable' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass' },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass' },
|
||||
],
|
||||
}).evidence;
|
||||
const dashboard = buildDashboard(evidence);
|
||||
assert.deepEqual(dashboard.manualFallback, {
|
||||
available: false,
|
||||
reason: 'app-down',
|
||||
});
|
||||
});
|
||||
|
||||
test('manualFallback state is a deterministic function of boundary health alone', () => {
|
||||
const cases = [
|
||||
{ statuses: ['pass', 'pass', 'pass', 'pass'], expected: { available: true, reason: 'none-required' } },
|
||||
{ statuses: ['degraded', 'pass', 'pass', 'pass'], expected: { available: true, reason: 'local-journal' } },
|
||||
{ statuses: ['pass', 'unknown', 'pass', 'pass'], expected: { available: true, reason: 'none-required' } },
|
||||
{ statuses: ['fail', 'fail', 'fail', 'fail'], expected: { available: false, reason: 'app-down' } },
|
||||
];
|
||||
for (const { statuses, expected } of cases) {
|
||||
const evidence = sanitizeEvidence({
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: statuses[0] },
|
||||
{ id: 'api.analyze', boundary: 'api', status: statuses[1] },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: statuses[2] },
|
||||
{ id: 'model.inference', boundary: 'model', status: statuses[3] },
|
||||
],
|
||||
}).evidence;
|
||||
assert.deepEqual(buildDashboard(evidence).manualFallback, expected, `statuses ${statuses}`);
|
||||
}
|
||||
});
|
||||
|
||||
const backlogEvidence = depth => sanitizeEvidence({
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'degraded', counters: { depth } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass' },
|
||||
],
|
||||
}).evidence;
|
||||
|
||||
test('queue.depth counter survives sanitization so queue.backlog can fire', () => {
|
||||
const dashboard = buildDashboard(backlogEvidence(10));
|
||||
assert.equal(dashboard.boundaries.queue.counters.depth, 10);
|
||||
});
|
||||
|
||||
test('queue.backlog stays silent at depth 9 and fires at exactly depth 10', () => {
|
||||
assert.deepEqual(evaluateAlerts(buildDashboard(backlogEvidence(9)), DEFAULT_ALERT_RULES), []);
|
||||
const atThreshold = evaluateAlerts(buildDashboard(backlogEvidence(10)), DEFAULT_ALERT_RULES);
|
||||
assert.equal(atThreshold.length, 1);
|
||||
assert.equal(atThreshold[0].id, 'queue.backlog');
|
||||
assert.equal(atThreshold[0].severity, 'warn');
|
||||
assert.equal(atThreshold[0].count, 10);
|
||||
assert.equal(atThreshold[0].threshold, 10);
|
||||
});
|
||||
|
||||
test('queue.backlog reports observed count above threshold and sums across queue checks', () => {
|
||||
const evidence = sanitizeEvidence({
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'degraded', counters: { depth: 5 } },
|
||||
{ id: 'queue.depth.sidecar', boundary: 'queue', status: 'pass', counters: { depth: 6 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass' },
|
||||
],
|
||||
}).evidence;
|
||||
const alerts = evaluateAlerts(buildDashboard(evidence), DEFAULT_ALERT_RULES);
|
||||
const backlog = alerts.filter(alert => alert.id === 'queue.backlog');
|
||||
assert.equal(backlog.length, 1);
|
||||
assert.equal(backlog[0].count, 11);
|
||||
});
|
||||
|
||||
test('queue.backlog is suppressed while worker.outage pages because outage depth is residual', () => {
|
||||
const evidence = sanitizeEvidence({
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'fail', failureClass: 'worker.unavailable' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'fail', failureClass: 'worker.unavailable', counters: { depth: 17 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'fail', failureClass: 'worker.unavailable' },
|
||||
],
|
||||
}).evidence;
|
||||
const alerts = evaluateAlerts(buildDashboard(evidence), DEFAULT_ALERT_RULES);
|
||||
assert.equal(alerts.length, 1, `expected only the outage page, got ${JSON.stringify(alerts)}`);
|
||||
assert.equal(alerts[0].id, 'worker.outage');
|
||||
});
|
||||
|
||||
test('queue.bound depth counter above the bounded ceiling cannot smuggle into the dashboard', () => {
|
||||
const evidence = sanitizeEvidence({
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'degraded', counters: { depth: 10_000_000 } },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass' },
|
||||
],
|
||||
}).evidence;
|
||||
assert.equal(evidence, null, 'unbounded depth must fail closed at sanitization');
|
||||
});
|
||||
|
||||
test('sanitizer fails closed when any app, api, queue, or model boundary is entirely missing', () => {
|
||||
const fullChecks = [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'pass' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'pass' },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass' },
|
||||
];
|
||||
for (const missing of ['app', 'api', 'queue', 'model']) {
|
||||
const evidence = sanitizeEvidence({
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: fullChecks.filter(check => check.boundary !== missing),
|
||||
});
|
||||
assert.equal(evidence.ok, false, `missing ${missing} boundary must fail closed`);
|
||||
}
|
||||
});
|
||||
|
||||
test('unknown statuses surface as a warn telemetry.gap alert instead of failing open', () => {
|
||||
const evidence = sanitizeEvidence({
|
||||
schemaVersion: 1,
|
||||
releaseTag: 'daily-2026-08-22.1',
|
||||
commit: 'ca31e6d38bec649407f63880504554c59f2878ae',
|
||||
generatedAtUtc: '2026-08-22T12:00:00Z',
|
||||
checks: [
|
||||
{ id: 'app.healthz', boundary: 'app', status: 'pass' },
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'unknown' },
|
||||
{ id: 'queue.depth', boundary: 'queue', status: 'unknown' },
|
||||
{ id: 'model.inference', boundary: 'model', status: 'pass' },
|
||||
],
|
||||
}).evidence;
|
||||
assert.ok(evidence, 'unknown is an allowed status and must not fail sanitization');
|
||||
const dashboard = buildDashboard(evidence);
|
||||
assert.equal(dashboard.boundaries.api.unknown, 1);
|
||||
const alerts = evaluateAlerts(dashboard, DEFAULT_ALERT_RULES);
|
||||
assert.equal(alerts.length, 1, `expected exactly the telemetry.gap alert, got ${JSON.stringify(alerts)}`);
|
||||
const gap = alerts[0];
|
||||
assert.equal(gap.id, 'telemetry.gap');
|
||||
assert.equal(gap.severity, 'warn');
|
||||
assert.equal(gap.owner, 'release-operator');
|
||||
assert.equal(gap.threshold, 1);
|
||||
assert.equal(gap.count, 2);
|
||||
assert.match(gap.runbook, /docs\/RELEASE-OBSERVABILITY\.md#/);
|
||||
assert.match(gap.message, /telemetry\.gap/);
|
||||
});
|
||||
|
||||
test('healthy evidence raises no telemetry.gap alert', () => {
|
||||
const alerts = evaluateAlerts(buildDashboard(healthyEvidence()), DEFAULT_ALERT_RULES);
|
||||
assert.equal(alerts.some(alert => alert.id === 'telemetry.gap'), false);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -76,3 +76,57 @@ test('sanitizer drops session identifiers, photo payloads, credentials, and free
|
|||
}
|
||||
assert.equal(result.evidence.checks.some(check => check.id === 'leak'), false);
|
||||
});
|
||||
|
||||
test('sanitizer fail-closes evidence carrying hostile failure-class payloads', () => {
|
||||
const hostileFailureClasses = [
|
||||
'worker.unavailable\nsecond line',
|
||||
'worker.unavailable\r\ncarriage',
|
||||
'\x1b[31mANSI-red',
|
||||
'worker.unavailable\x1b[0m',
|
||||
'bell\x07class',
|
||||
'tab\tseparated',
|
||||
'DROP TABLE users',
|
||||
'patient reported blood at 3am',
|
||||
'sess_live_abc123def456',
|
||||
'-----BEGIN OPENSSH PRIVATE KEY-----',
|
||||
'a@b.example.com',
|
||||
'.leading.dot',
|
||||
'trailing.dot.',
|
||||
'double..dot',
|
||||
'-leading-dash.d',
|
||||
'd.trailing-dash-',
|
||||
'-'.repeat(65),
|
||||
42,
|
||||
{},
|
||||
[],
|
||||
];
|
||||
for (const failureClass of hostileFailureClasses) {
|
||||
const evidence = validEvidence();
|
||||
evidence.checks = [{ id: 'app.healthz', boundary: 'app', status: 'fail', failureClass }];
|
||||
const result = sanitizeEvidence(evidence);
|
||||
assert.equal(result.ok, false, `expected rejection for failureClass ${JSON.stringify(failureClass)}`);
|
||||
}
|
||||
});
|
||||
|
||||
test('sanitizer accepts only the exact privacy-safe failure-class vocabulary in slug grammar', () => {
|
||||
const baseChecks = validEvidence().checks;
|
||||
for (const failureClass of ['worker.unavailable', 'vision.timeout', 'model.error']) {
|
||||
const evidence = validEvidence();
|
||||
evidence.checks = [
|
||||
...baseChecks.filter(check => check.boundary !== 'api'),
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'degraded', failureClass },
|
||||
];
|
||||
const result = sanitizeEvidence(evidence);
|
||||
assert.equal(result.ok, true, `vocabulary class ${failureClass} must survive sanitization`);
|
||||
assert.equal(result.evidence.checks.some(check => check.failureClass === failureClass), true);
|
||||
}
|
||||
for (const failureClass of ['made.up.class', 'worker.somethingelse', 'VISION.TIMEOUT', 'vision..timeout']) {
|
||||
const evidence = validEvidence();
|
||||
evidence.checks = [
|
||||
...baseChecks.filter(check => check.boundary !== 'api'),
|
||||
{ id: 'api.analyze', boundary: 'api', status: 'degraded', failureClass },
|
||||
];
|
||||
const result = sanitizeEvidence(evidence);
|
||||
assert.equal(result.ok, false, `non-vocabulary class ${failureClass} must be rejected`);
|
||||
}
|
||||
});
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user