diff --git a/README.md b/README.md index 54805b5..b40d32e 100644 --- a/README.md +++ b/README.md @@ -110,7 +110,7 @@ npm audit --audit-level=high - Client-side image compression before transfer - Explicit consent before one-time provider analysis - Structured AI suggestions for **visible Bristol form and color only** -- Confidence display, low-confidence abstention, and user confirmation +- Confidence display, low-confidence abstention, and user confirmation, governed by one versioned policy: `poor` image quality abstains even at high confidence, and a confident reply is never enough on its own — see [docs/ABSTENTION-POLICY.md](docs/ABSTENTION-POLICY.md) - Urgency, discomfort, notes, and symptoms remain strictly user-reported - Manual logging that never uploads - Sleek three-destination mobile shell with one dominant photo action diff --git a/docs/ABSTENTION-POLICY.md b/docs/ABSTENTION-POLICY.md new file mode 100644 index 0000000..013936b --- /dev/null +++ b/docs/ABSTENTION-POLICY.md @@ -0,0 +1,98 @@ +# Abstention policy + +Issue #32 asks for raw model scores to be turned into a conservative suggestion policy. This is that +policy. It is the only thing that decides whether Timmy offers a visual prefill, and it answers two +different questions that are easy to confuse: + +1. **Per photo** — is this individual answer good enough to show? +2. **Per release** — has a candidate model earned the right to prefill at all? + +Both are implemented in [`src/abstention-policy.js`](../src/abstention-policy.js). The parser in +[`src/analysis.js`](../src/analysis.js) reads its limits from that module rather than carrying its own +copy, so a policy change is one edit, not three. + +A suggestion is never a diagnosis, never fills in urgency, discomfort, symptoms, or notes, and is +never saved until the user confirms it. Abstention is the default; a prefill is the exception. + +## Why quality is a gate, not a footnote + +`imageQuality` is something the model reports about its own photo. A confident form and colour read +from a blurry photo is exactly the answer a user is most likely to accept without thinking, so +confidence alone cannot authorise a prefill. The policy accepts `good` and `fair`; `poor` abstains +regardless of how confident the reply looked. This is proven end to end by the committed corpus case +`poor-quality-confident`, which replays a 0.94-confidence answer about the synthetic motion-blur +fixture and expects an abstention. + +## Per-photo gates + +Evaluated in this order. The first gate that fails names the refusal, so the user is told the real +reason rather than a generic one. + +| Order | Gate | Refusal (`abstainedFor`) | User-facing copy | +|---|---|---|---| +| 1 | Model says the photo is not clearly stool | `not_stool` | The image does not clearly show stool. | +| 2 | Confidence below `minConfidence` (0.55) | `low_confidence` | The image is too uncertain to prefill safely. | +| 3 | Form or colour came back blank | `unusable_visual_fields` | The photo does not show a form Timmy can read safely. | +| 4 | `imageQuality` is below the accepted levels | `low_quality` | The photo is too blurry or poorly lit to suggest a form safely. | + +A filled-in field outside the contract (`bristolType` outside 1–7, an unknown `color`, an unknown +`imageQuality`) is not an abstention — it **fails closed** with no provider detail shown, because a +caller must never be told "the photo was unreadable" when the model actually returned an impossible +value. That ordering is covered by the corpus escape cases (`out-of-range-fields`, +`blank-plus-out-of-range`). + +Unknown and out-of-distribution images can only be judged from what the schema lets the model say: it +must declare `isStool`, a confidence, and a quality. There is no separate OOD channel, and this policy +does not invent one. A photo the model cannot place therefore lands on `not_stool`, `low_confidence`, +or `low_quality` — all three abstain. + +## Per-release gate + +The calibration harness ([`scripts/calibration_report.py`](../scripts/calibration_report.py), issue +#29) scores candidates on a held-out test split with a threshold selected on validation only. The +policy turns that report into a single decision with `evaluatePrefillGate(report)`: + +| Requirement | Blocked reason if unmet | +|---|---| +| Report verdict is `GO` | `calibration-verdict-not-go` | +| The report names a recommended candidate | `no-recommended-calibration` | +| That candidate's threshold was selected on `validation` | `threshold-not-selected-on-validation` | +| That candidate was scored on the held-out `test` split | `gate-not-scored-on-held-out-test` | +| That candidate passed every release gate | `candidate-failed-release-gates` | + +A missing or unreadable report is a block, never a default allow. + +Run it as a receipt: + +```bash +python3 scripts/calibration_report.py \ + --predictions tests/fixtures/calibration/predictions.jsonl \ + --min-coverage 0.5 --json /tmp/calibration-report.json + +node scripts/abstention_policy_receipt.mjs --report /tmp/calibration-report.json +# exit 0 = prefill authorised, 3 = blocked, 2 = the report could not be read +``` + +**Current state:** the committed synthetic predictions produce `BLOCKED`, so the receipt exits 3 and +names why. No production candidate exists yet — issue #31 (fine-tuned classifier) is the dependency — +and the policy exists so that candidate is held to one stated contract instead of whatever it happens +to score. The release builder does not gate on this receipt yet; it records the quality gate on camera +through the demo instead. Wiring the receipt into `scripts/build_release.py` as a hard prefill gate +belongs with #31, when there is a candidate to judge. + +## Versioning + +`ABSTENTION_POLICY.version` is an integer that changes whenever a limit changes. A policy change is a +behaviour change: the release notes must be able to say which policy produced a suggestion. The +shipped object is frozen so nothing can tighten or loosen it at runtime; callers that need a +different policy pass one explicitly (`parseVisionResponse(raw, policy)`), which is how the tests +prove the parser has no hidden limits of its own. + +## What this policy does not do + +- It does not decide the Bristol type, the colour, or the quality. It only decides whether to show + them. +- It does not touch deterministic red-flag escalation, which runs independently of model state. +- It does not read, store, or log an image, and it never inspects inference bodies. +- It does not set the numeric thresholds from anything but validation data, and it does not convert a + `BLOCKED` calibration into a pass. diff --git a/docs/EVALUATION-HARNESS.md b/docs/EVALUATION-HARNESS.md index 3bb78f5..efbb38e 100644 --- a/docs/EVALUATION-HARNESS.md +++ b/docs/EVALUATION-HARNESS.md @@ -7,6 +7,10 @@ splits that are contributor-held-out and reproducible, comparing candidate model The implementation is [`scripts/calibration_report.py`](../scripts/calibration_report.py). It reads predictions something else produced, and it refuses to compute gates it cannot honestly compute. +The GO/BLOCKED verdict this report produces is consumed by the production abstention policy in +[`docs/ABSTENTION-POLICY.md`](ABSTENTION-POLICY.md): a candidate that has not cleared these gates +cannot authorise a visual prefill. + **This harness does not train, does not diagnose, and never reads an image.** It only sees Bristol labels, predictions, and confidences. diff --git a/package.json b/package.json index cdbc19c..b5acb73 100644 --- a/package.json +++ b/package.json @@ -4,7 +4,7 @@ "private": true, "type": "module", "scripts": { - "test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/draft.test.js tests/suggestion.test.js tests/followup.test.js tests/vision-readiness.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js tests/threat-model.test.js tests/vision-corpus.test.js tests/vision-boundary.test.js tests/worker-benchmark.test.js tests/worker-ready.test.js tests/worker-container.test.js tests/summary.test.js tests/escalation-regression.test.js tests/grounded-chat.test.js tests/photo-lifecycle.test.js tests/training-contribution.test.js tests/portability.test.js", + "test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/draft.test.js tests/suggestion.test.js tests/followup.test.js tests/vision-readiness.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js tests/threat-model.test.js tests/vision-corpus.test.js tests/vision-boundary.test.js tests/worker-benchmark.test.js tests/worker-ready.test.js tests/worker-container.test.js tests/summary.test.js tests/escalation-regression.test.js tests/grounded-chat.test.js tests/photo-lifecycle.test.js tests/training-contribution.test.js tests/portability.test.js tests/abstention-policy.test.js", "test:ui": "node tests/ui.acceptance.mjs", "test:photo": "node tests/photo-first.acceptance.mjs", "test:mobile-capture": "node tests/mobile-capture.acceptance.mjs", @@ -18,13 +18,14 @@ "test:grounded-chat": "node tests/grounded-chat.acceptance.mjs", "test:photo-lifecycle": "node tests/photo-lifecycle.acceptance.mjs", "test:training-contribution": "node tests/training-contribution.acceptance.mjs", - "check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py && node --check scripts/record_worker_benchmark.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh", + "check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh", "check:diff": "bash scripts/check_diff.sh", "start": "node server.mjs", "test:worker-readiness": "node tests/worker-readiness.acceptance.mjs", "test:portability": "node tests/portability.acceptance.mjs", "test:corpus": "python3 tests/corpus-manifest.test.py", - "test:calibration": "python3 tests/calibration-harness.test.py" + "test:calibration": "python3 tests/calibration-harness.test.py", + "test:policy": "node --test tests/abstention-policy.test.js" }, "devDependencies": { "playwright": "^1.62.1" diff --git a/scripts/abstention_policy_receipt.mjs b/scripts/abstention_policy_receipt.mjs new file mode 100644 index 0000000..b4a2a95 --- /dev/null +++ b/scripts/abstention_policy_receipt.mjs @@ -0,0 +1,49 @@ +#!/usr/bin/env node +// Executable receipt for the production abstention policy (issue #32). +// +// The calibration harness decides GO/BLOCKED for a candidate model. This reads that report and +// answers the only question the app cares about: may a suggestion be shipped at all? It exits 3 when +// prefill is blocked, so a release step can gate on it instead of reading prose. +// +// There is no model candidate yet (#31), so in practice this receipt reports BLOCKED today. That is +// the honest state: the policy is ready, the evidence is not. +import { readFile } from 'node:fs/promises'; +import { ABSTENTION_POLICY, evaluatePrefillGate } from '../src/abstention-policy.js'; + +function parseArgs(argv) { + const args = { report: '' }; + for (let i = 0; i < argv.length; i += 1) { + if (argv[i] === '--report') { + args.report = argv[i + 1] || ''; + i += 1; + } else throw new Error(`unknown argument: ${argv[i]}`); + } + if (!args.report) throw new Error('--report is required'); + return args; +} + +let args; +try { + args = parseArgs(process.argv.slice(2)); +} catch (error) { + console.error(error.message); + process.exit(2); +} + +let report; +try { + report = JSON.parse(await readFile(args.report, 'utf8')); +} catch (error) { + console.error(`could not read the calibration report at ${args.report}: ${error.message}`); + process.exit(2); +} + +const gate = evaluatePrefillGate(report); +const policy = ABSTENTION_POLICY; +console.log('TIMMY ABSTENTION POLICY RECEIPT'); +console.log(`policy: version ${policy.version} · confidence ≥ ${policy.minConfidence.toFixed(2)} · quality ${policy.acceptedImageQuality.join('|')}`); +console.log(`calibration: verdict ${gate.verdict} · model ${gate.calibrationModel || 'none'}`); +console.log(gate.prefillAllowed + ? `prefill: AUTHORISED for ${gate.calibrationModel}` + : `prefill: BLOCKED — ${gate.blockedBy.join(', ')}`); +process.exit(gate.prefillAllowed ? 0 : 3); diff --git a/scripts/build_release.py b/scripts/build_release.py index 27bf365..a057777 100755 --- a/scripts/build_release.py +++ b/scripts/build_release.py @@ -46,6 +46,7 @@ DEMO_REQUIRED_CAPTIONS = ( "A tarry note already saved in the journal escalates the same way, before any provider call", "An out-of-range answer fails closed: nothing is applied and no provider detail reaches you", "A filled-in field that is out of range still fails closed, even when the other field came back blank", + "A confident reply about a blurry photo still abstains — photo quality is part of the policy, not just confidence", "Reachable is not loaded: no AI control, no consent, and no upload until the pinned model is resident", "Every pattern line names the confirmed logs behind it, and following one shows exactly those records", "Two logs is not a pattern: the summary refuses to describe a trend it cannot support", @@ -340,7 +341,7 @@ def main() -> int: "codec": "h264", "pixel_format": "yuv420p", "recorded_from_working_app": True, - "fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything", + "fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything", }, "gates": { "unit_security": "passed", @@ -351,6 +352,7 @@ def main() -> int: "provisional_suggestion_state": "passed", "hard_stool_followup": "passed", "vision_corpus_refusal": "passed", + "abstention_quality_gate": "passed", "worker_container_receipts": "passed", "worker_post_load_readiness": "passed", "longitudinal_pattern_summary": "passed", diff --git a/scripts/record_release_demo.mjs b/scripts/record_release_demo.mjs index d8f80c3..1274ddd 100644 --- a/scripts/record_release_demo.mjs +++ b/scripts/record_release_demo.mjs @@ -341,6 +341,20 @@ await tap('#analyze-photo', 600); await page.getByText(/does not show a form Timmy can read safely/i).waitFor(); await caption('Unreadable visual fields abstain as well: a valid reply with nothing to prefill is not an error', 2700); await indicate('.scan-result'); +// Issue #32's quality gate: the model can be highly confident about a photo it also calls poor, and +// that confident answer must still abstain. Replayed from the committed corpus, and asserted in the +// browser so the scene is a check rather than a claim. +analyzeOverride = await corpusAnalyzeResult(corpusCase('poor-quality-confident')); +await tap('#retake-photo', 400); +await page.locator('#gallery-photo').setInputFiles('tests/fixtures/vision-corpus/blur-low-detail.png'); +await page.locator('#ai-consent').check(); +await tap('#analyze-photo', 600); +await page.getByText(/too blurry or poorly lit/i).waitFor(); +if (await page.getByText(/BRISTOL FORM/i).count() !== 0) { + throw new Error('A low-quality photo must not prefill a form, however confident the reply looked'); +} +await caption('A confident reply about a blurry photo still abstains — photo quality is part of the policy, not just confidence', 3100); +await indicate('.scan-result'); analyzeOverride = await corpusAnalyzeResult(corpusCase('out-of-range-fields')); await tap('#retake-photo', 400); await page.locator('#gallery-photo').setInputFiles('tests/fixtures/vision-corpus/out-of-range-fields.png'); diff --git a/src/abstention-policy.js b/src/abstention-policy.js new file mode 100644 index 0000000..f45d68d --- /dev/null +++ b/src/abstention-policy.js @@ -0,0 +1,69 @@ +// The production abstention policy (issue #32). +// +// A suggestion is only ever a visual prefill, and this module is the single place that decides when +// Timmy is allowed to offer one. Two different questions live here, and they are deliberately +// separate: +// +// 1. Per-photo: is this individual answer good enough to show? Confidence, image quality, and +// whether the model actually read a form and a colour. +// 2. Per-release: has a candidate cleared the calibration gates on a held-out split, with its +// threshold chosen on validation only? Until it has, no prefill may be shipped at all. +// +// The copy is fixed here so the browser, the corpus, and the server cannot drift into three +// different explanations of the same refusal. +export const ABSTENTION_COPY = Object.freeze({ + not_stool: 'The image does not clearly show stool.', + low_confidence: 'The image is too uncertain to prefill safely.', + unusable_visual_fields: 'The photo does not show a form Timmy can read safely.', + low_quality: 'The photo is too blurry or poorly lit to suggest a form safely.', +}); + +// Bump `version` whenever a limit below changes: a policy change is a behaviour change, and the +// release notes have to be able to name which policy a suggestion was produced under. +export const ABSTENTION_POLICY = Object.freeze({ + version: 1, + minConfidence: 0.55, + // "fair" may prefill; "poor" never does. The model already returns its own quality judgement, so + // a confident answer about an unreadable photo is exactly the case this gate exists for. + acceptedImageQuality: Object.freeze(['good', 'fair']), +}); + +export function abstainFor(reason, isStool = true) { + if (!Object.prototype.hasOwnProperty.call(ABSTENTION_COPY, reason)) { + throw new Error(`unknown abstention reason: ${reason}`); + } + return { + status: 'needs_user_input', + isStool, + abstainedFor: reason, + reason: ABSTENTION_COPY[reason], + }; +} + +export function isLowQuality(imageQuality, policy = ABSTENTION_POLICY) { + return !policy.acceptedImageQuality.includes(String(imageQuality || '').toLowerCase()); +} + +// Release-level gate. The calibration harness already refuses to pick a threshold anywhere but the +// validation split; this decides whether a report authorises prefill at all, and names every reason +// it does not. A missing or malformed report is a block, never a default-allow. +export function evaluatePrefillGate(report) { + const blockedBy = []; + const verdict = report?.verdict || 'UNKNOWN'; + if (verdict !== 'GO') blockedBy.push('calibration-verdict-not-go'); + const models = Array.isArray(report?.models) ? report.models : []; + const recommended = models.find(row => row?.model === report?.recommended) || null; + if (!recommended) { + blockedBy.push('no-recommended-calibration'); + } else { + if (recommended.selectionSplit !== 'validation') blockedBy.push('threshold-not-selected-on-validation'); + if (recommended.evaluatedSplit !== 'test') blockedBy.push('gate-not-scored-on-held-out-test'); + if (recommended.gate?.passed !== true) blockedBy.push('candidate-failed-release-gates'); + } + return { + prefillAllowed: blockedBy.length === 0, + verdict, + calibrationModel: recommended?.model || null, + blockedBy, + }; +} diff --git a/src/analysis.js b/src/analysis.js index f36db42..60aa53c 100644 --- a/src/analysis.js +++ b/src/analysis.js @@ -1,3 +1,5 @@ +import { ABSTENTION_POLICY, abstainFor, isLowQuality } from './abstention-policy.js'; + const COLORS = new Set(['brown', 'green', 'yellow', 'pale', 'red', 'black']); const QUALITIES = new Set(['good', 'fair', 'poor']); const MAX_IMAGE_BYTES = 4 * 1024 * 1024; @@ -26,13 +28,13 @@ function asObject(raw) { } } -export function parseVisionResponse(raw) { +export function parseVisionResponse(raw, policy = ABSTENTION_POLICY) { const value = asObject(raw); if (typeof value.isStool !== 'boolean') throw visionResponseError(VISION_FAILURE_CODES.invalidFields, 'Invalid AI response: isStool is required.'); const confidence = Number(value.confidence); if (!Number.isFinite(confidence) || confidence < 0 || confidence > 1) throw visionResponseError(VISION_FAILURE_CODES.invalidFields, 'Invalid AI response: confidence is required.'); - if (!value.isStool) return { status: 'needs_user_input', isStool: false, abstainedFor: 'not_stool', reason: 'The image does not clearly show stool.' }; - if (confidence < 0.55) return { status: 'needs_user_input', isStool: true, abstainedFor: 'low_confidence', reason: 'The image is too uncertain to prefill safely.' }; + if (!value.isStool) return abstainFor('not_stool', false); + if (confidence < policy.minConfidence) return abstainFor('low_confidence'); // The request schema lets the model return a null form and color when it cannot read them. // A blank field means the model could not read it, so a schema-valid answer with nothing usable // abstains and nothing is prefilled. A filled-in field is a claim, though: if it sits outside the @@ -47,15 +49,12 @@ export function parseVisionResponse(raw) { throw visionResponseError(VISION_FAILURE_CODES.invalidFields, 'Invalid AI response: visual fields are out of range.'); } if (bristolType === null || color === null) { - return { - status: 'needs_user_input', - isStool: true, - abstainedFor: 'unusable_visual_fields', - reason: 'The photo does not show a form Timmy can read safely.', - }; + return abstainFor('unusable_visual_fields'); } const imageQuality = String(value.imageQuality || '').toLowerCase(); if (!QUALITIES.has(imageQuality)) throw visionResponseError(VISION_FAILURE_CODES.invalidFields, 'Invalid AI response: visual fields are out of range.'); + // A confident reading of a photo the model itself calls poor is still not a safe prefill. + if (isLowQuality(imageQuality, policy)) return abstainFor('low_quality'); return { status: 'suggestion', isStool: true, diff --git a/tests/abstention-policy.test.js b/tests/abstention-policy.test.js new file mode 100644 index 0000000..e2c7bea --- /dev/null +++ b/tests/abstention-policy.test.js @@ -0,0 +1,117 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { execFile } from 'node:child_process'; +import { mkdtemp, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { promisify } from 'node:util'; +import { ABSTENTION_COPY, ABSTENTION_POLICY, abstainFor, evaluatePrefillGate } from '../src/abstention-policy.js'; + +test('every abstention reason has one fixed piece of copy the user sees', () => { + assert.equal(abstainFor('not_stool').reason, 'The image does not clearly show stool.'); + assert.equal(abstainFor('low_confidence').reason, 'The image is too uncertain to prefill safely.'); + assert.equal(abstainFor('unusable_visual_fields').reason, 'The photo does not show a form Timmy can read safely.'); + assert.equal(abstainFor('low_quality').reason, 'The photo is too blurry or poorly lit to suggest a form safely.'); + assert.deepEqual(abstainFor('low_quality'), { + status: 'needs_user_input', + isStool: true, + abstainedFor: 'low_quality', + reason: ABSTENTION_COPY.low_quality, + }); + assert.equal(abstainFor('not_stool', false).isStool, false); + assert.throws(() => abstainFor('looked_wrong'), /unknown abstention reason/); +}); + +test('the policy states the confidence floor and the quality levels that may prefill', () => { + assert.ok(Number.isInteger(ABSTENTION_POLICY.version)); + assert.equal(ABSTENTION_POLICY.minConfidence, 0.55); + assert.deepEqual([...ABSTENTION_POLICY.acceptedImageQuality], ['good', 'fair']); + assert.equal(ABSTENTION_POLICY.acceptedImageQuality.includes('poor'), false); + assert.equal(Object.isFrozen(ABSTENTION_POLICY), true, 'the shipped policy must not be mutated at runtime'); +}); + +test('prefill is authorised only by a GO report whose threshold was selected on validation', () => { + const passing = { + verdict: 'GO', + recommended: 'specialist-candidate', + models: [{ + model: 'specialist-candidate', + selectionSplit: 'validation', + evaluatedSplit: 'test', + gate: { passed: true, failed: [] }, + }], + }; + assert.deepEqual(evaluatePrefillGate(passing), { + prefillAllowed: true, + verdict: 'GO', + calibrationModel: 'specialist-candidate', + blockedBy: [], + }); + + const blocked = evaluatePrefillGate({ + verdict: 'BLOCKED', + recommended: null, + models: [{ model: 'specialist-candidate', selectionSplit: 'validation', evaluatedSplit: 'test', gate: { passed: false, failed: ['precisionAtCoverage'] } }], + }); + assert.equal(blocked.prefillAllowed, false); + assert.deepEqual(blocked.blockedBy, ['calibration-verdict-not-go', 'no-recommended-calibration']); + + const selectedOnTest = evaluatePrefillGate({ + verdict: 'GO', + recommended: 'opportunist', + models: [{ model: 'opportunist', selectionSplit: 'test', evaluatedSplit: 'test', gate: { passed: true, failed: [] } }], + }); + assert.equal(selectedOnTest.prefillAllowed, false, 'a threshold chosen on the split it is scored on is not evidence'); + assert.deepEqual(selectedOnTest.blockedBy, ['threshold-not-selected-on-validation']); + + const noReport = evaluatePrefillGate(undefined); + assert.equal(noReport.prefillAllowed, false); + assert.match(noReport.blockedBy.join(','), /calibration-verdict-not-go/); +}); + +const execFileAsync = promisify(execFile); +const receiptScript = fileURLToPath(new URL('../scripts/abstention_policy_receipt.mjs', import.meta.url)); + +async function runReceipt(report) { + const directory = await mkdtemp(join(tmpdir(), 'timmy-policy-')); + const reportPath = join(directory, 'calibration-report.json'); + await writeFile(reportPath, JSON.stringify(report)); + try { + const { stdout } = await execFileAsync(process.execPath, [receiptScript, '--report', reportPath]); + return { code: 0, stdout }; + } catch (error) { + return { code: error.code, stdout: error.stdout || '', stderr: error.stderr || '' }; + } +} + +test('the receipt refuses to authorise prefill without a passing held-out calibration', async () => { + const blocked = await runReceipt({ + verdict: 'BLOCKED', + recommended: null, + models: [{ model: 'specialist-candidate', selectionSplit: 'validation', evaluatedSplit: 'test', gate: { passed: false, failed: ['precisionAtCoverage'] } }], + }); + assert.equal(blocked.code, 3, 'a blocked calibration must exit non-zero instead of shipping prefill'); + assert.match(blocked.stdout, /prefill: BLOCKED/); + assert.match(blocked.stdout, /calibration-verdict-not-go/); + assert.match(blocked.stdout, /policy: version 1 · confidence ≥ 0\.55 · quality good\|fair/); + + const passing = await runReceipt({ + verdict: 'GO', + recommended: 'specialist-candidate', + models: [{ model: 'specialist-candidate', selectionSplit: 'validation', evaluatedSplit: 'test', gate: { passed: true, failed: [] } }], + }); + assert.equal(passing.code, 0); + assert.match(passing.stdout, /prefill: AUTHORISED for specialist-candidate/); +}); + +test('the receipt treats a missing report as a block and never as a default allow', async () => { + const directory = await mkdtemp(join(tmpdir(), 'timmy-policy-')); + try { + await execFileAsync(process.execPath, [receiptScript, '--report', join(directory, 'absent.json')]); + assert.fail('a missing calibration report must not exit 0'); + } catch (error) { + assert.equal(error.code, 2); + assert.match(String(error.stderr), /could not read|absent/i); + } +}); diff --git a/tests/analysis.test.js b/tests/analysis.test.js index 2961a29..cccca5b 100644 --- a/tests/analysis.test.js +++ b/tests/analysis.test.js @@ -1,5 +1,6 @@ import test from 'node:test'; import assert from 'node:assert/strict'; +import { ABSTENTION_COPY, ABSTENTION_POLICY } from '../src/abstention-policy.js'; import { buildVisionRequest, @@ -131,3 +132,39 @@ test('abstains instead of failing when a schema-valid response carries no readab reason: 'The photo does not show a form Timmy can read safely.', }); }); + +test('a confident answer about a low-quality photo still abstains', () => { + // Issue #32: confidence is not the only gate. A blurry or badly lit photo can produce a + // confident-looking form and colour, and prefilling it would show the user a claim the photo + // cannot support. Quality is part of the policy, so this abstains instead of prefilling. + assert.deepEqual(parseVisionResponse({ + isStool: true, + bristolType: 4, + color: 'brown', + confidence: 0.94, + imageQuality: 'poor', + observations: 'Form barely visible through motion blur.', + }), { + status: 'needs_user_input', + isStool: true, + abstainedFor: 'low_quality', + reason: 'The photo is too blurry or poorly lit to suggest a form safely.', + }); +}); + +test('the parse path reads the shipped policy instead of hardcoding its own limits', () => { + // Issue #32: policy numbers and copy live in one module. A caller may tighten the policy for a + // release without editing the parser, and a photo that the default policy would accept has to + // abstain under the stricter one — otherwise the parser is carrying its own hidden policy. + const fields = { isStool: true, bristolType: 4, color: 'brown', confidence: 0.8, imageQuality: 'fair', observations: 'Form visible but soft.' }; + assert.equal(parseVisionResponse(fields).status, 'suggestion', 'fair quality may prefill under the shipped policy'); + const strict = { ...ABSTENTION_POLICY, minConfidence: 0.9, acceptedImageQuality: ['good'] }; + assert.equal(parseVisionResponse(fields, strict).abstainedFor, 'low_confidence'); + assert.equal(parseVisionResponse({ ...fields, confidence: 0.95 }, strict).abstainedFor, 'low_quality'); + assert.equal(parseVisionResponse({ ...fields, confidence: 0.95, imageQuality: 'good' }, strict).status, 'suggestion'); + assert.equal( + parseVisionResponse({ ...fields, imageQuality: 'poor' }).reason, + ABSTENTION_COPY.low_quality, + 'the refusal copy must come from the policy module, not a second copy in the parser', + ); +}); diff --git a/tests/fixtures/vision-corpus/manifest.json b/tests/fixtures/vision-corpus/manifest.json index 33dd95f..151888e 100644 --- a/tests/fixtures/vision-corpus/manifest.json +++ b/tests/fixtures/vision-corpus/manifest.json @@ -131,6 +131,22 @@ "expectedOutcome": "fail_closed", "expectedReason": "out_of_range_or_missing_fields", "providerContent": "{\"isStool\":true,\"bristolType\":9,\"color\":null,\"confidence\":0.9,\"imageQuality\":\"good\",\"observations\":\"Form read, colour not.\"}" + }, + { + "id": "poor-quality-confident", + "caseClass": "blur", + "file": "blur-low-detail.png", + "bytes": 21598, + "sha256": "17000163662a42fd434bb2b2428d9e5a2156da08cbc82210e41dfc6fa7e24852", + "width": 640, + "height": 480, + "format": "png", + "synthetic": true, + "license": "CC0-1.0", + "provenance": "The same synthetic motion-blur fixture that blur-low-detail replays, replayed again with a confident provider answer; drawn deterministically by scripts/build_vision_corpus.py, no camera, no human subject, no third-party content.", + "expectedOutcome": "abstained", + "expectedReason": "low_quality", + "providerContent": "{\"isStool\":true,\"bristolType\":4,\"color\":\"brown\",\"confidence\":0.94,\"imageQuality\":\"poor\",\"observations\":\"Form barely visible through motion blur.\"}" } ] } diff --git a/tests/release-demo.test.js b/tests/release-demo.test.js index 7ed1b00..f2e5309 100644 --- a/tests/release-demo.test.js +++ b/tests/release-demo.test.js @@ -272,6 +272,14 @@ test('release demo visibly exercises the corpus refusal paths instead of only th assert.match(demo, /A non-stool photo gets a refusal, never a guess/); assert.match(demo, /Unreadable visual fields abstain as well/); assert.match(demo, /An out-of-range answer fails closed/); + // Issue #32's quality gate: a confident answer about a low-quality photo must abstain on camera. + assert.match(demo, /corpusCase\('poor-quality-confident'\)/); + assert.match(demo, /tests\/fixtures\/vision-corpus\/blur-low-detail\.png/); + assert.match(demo, /too blurry or poorly lit/); + assert.match(demo, /must not prefill a form, however confident the reply looked/); + assert.match(demo, /A confident reply about a blurry photo still abstains/); + assert.match(builder, /A confident reply about a blurry photo still abstains — photo quality is part of the policy, not just confidence/); + assert.match(builder, /"abstention_quality_gate": "passed"/); assert.match(demo, /no real photos in the corpus|no real photos in the recording/); // The refusal copy the user sees is gated on camera, so a release cannot drop it silently. assert.match(builder, /An out-of-range answer fails closed/);