Compare commits

...
Author SHA1 Message Date
timmy 209e21a951 Gate visual prefill on a versioned abstention policy (#32)
Quality gates / quality (pull_request) Successful in 13m29s
Confidence was the only thing standing between a model answer and a
prefilled journal entry, so a confident reading of a blurry photo could
still be offered as fact. Quality is now part of the policy: `poor`
abstains however confident the reply looked, with its own fixed copy.

The confidence floor, accepted quality levels, and refusal copy move into
src/abstention-policy.js so the parser, the corpus, and the browser cannot
drift into three explanations of the same refusal. parseVisionResponse
takes an optional policy, which is how the tests prove the parser carries
no hidden limits of its own.

evaluatePrefillGate turns the #29 calibration report into one release
decision: GO plus a validation-selected threshold scored on the held-out
test split, or prefill is blocked with each reason named. A missing report
is a block, never a default allow. scripts/abstention_policy_receipt.mjs
exposes that as a receipt (exit 0 / 3 / 2).

The committed synthetic predictions currently report BLOCKED and the
receipt says so. No candidate exists until #31, and the builder does not
hard-gate on the receipt yet; the on-camera quality gate is the evidence
this release carries.

The corpus gains a case that replays a 0.94-confidence answer about the
synthetic motion-blur fixture, and the release demo asserts in the browser
that nothing is prefilled before captioning it.

Verified: 226 unit tests, calibration harness (19), corpus manifest (8),
vision-refusal, photo-first, and provisional acceptance, check:syntax,
check:diff, and a full real-browser demo recording whose caption receipt
confirms the new scene painted on screen.
2026-10-26 10:58:38 +00:00
rockachopa 2a10a20518 fix(release): give each calibration candidate its own row so the name is not split at its hyphen
Quality gates / quality (push) Successful in 12m51s
2026-10-25 12:23:56 +00:00
rockachopa a2293c6e34 Merge pull request #86: Build contributor-held-out evaluation harness and calibration report (#29)
Quality gates / quality (push) Successful in 12m54s
Closes #29
2026-10-25 12:02:11 +00:00
13 changed files with 432 additions and 15 deletions

No files matched your search

+1 -1
View File
@@ -110,7 +110,7 @@ npm audit --audit-level=high
- Client-side image compression before transfer
- Explicit consent before one-time provider analysis
- Structured AI suggestions for **visible Bristol form and color only**
- Confidence display, low-confidence abstention, and user confirmation
- Confidence display, low-confidence abstention, and user confirmation, governed by one versioned policy: `poor` image quality abstains even at high confidence, and a confident reply is never enough on its own — see [docs/ABSTENTION-POLICY.md](docs/ABSTENTION-POLICY.md)
- Urgency, discomfort, notes, and symptoms remain strictly user-reported
- Manual logging that never uploads
- Sleek three-destination mobile shell with one dominant photo action
+98
View File
@@ -0,0 +1,98 @@
# Abstention policy
Issue #32 asks for raw model scores to be turned into a conservative suggestion policy. This is that
policy. It is the only thing that decides whether Timmy offers a visual prefill, and it answers two
different questions that are easy to confuse:
1. **Per photo** — is this individual answer good enough to show?
2. **Per release** — has a candidate model earned the right to prefill at all?
Both are implemented in [`src/abstention-policy.js`](../src/abstention-policy.js). The parser in
[`src/analysis.js`](../src/analysis.js) reads its limits from that module rather than carrying its own
copy, so a policy change is one edit, not three.
A suggestion is never a diagnosis, never fills in urgency, discomfort, symptoms, or notes, and is
never saved until the user confirms it. Abstention is the default; a prefill is the exception.
## Why quality is a gate, not a footnote
`imageQuality` is something the model reports about its own photo. A confident form and colour read
from a blurry photo is exactly the answer a user is most likely to accept without thinking, so
confidence alone cannot authorise a prefill. The policy accepts `good` and `fair`; `poor` abstains
regardless of how confident the reply looked. This is proven end to end by the committed corpus case
`poor-quality-confident`, which replays a 0.94-confidence answer about the synthetic motion-blur
fixture and expects an abstention.
## Per-photo gates
Evaluated in this order. The first gate that fails names the refusal, so the user is told the real
reason rather than a generic one.
| Order | Gate | Refusal (`abstainedFor`) | User-facing copy |
|---|---|---|---|
| 1 | Model says the photo is not clearly stool | `not_stool` | The image does not clearly show stool. |
| 2 | Confidence below `minConfidence` (0.55) | `low_confidence` | The image is too uncertain to prefill safely. |
| 3 | Form or colour came back blank | `unusable_visual_fields` | The photo does not show a form Timmy can read safely. |
| 4 | `imageQuality` is below the accepted levels | `low_quality` | The photo is too blurry or poorly lit to suggest a form safely. |
A filled-in field outside the contract (`bristolType` outside 1–7, an unknown `color`, an unknown
`imageQuality`) is not an abstention — it **fails closed** with no provider detail shown, because a
caller must never be told "the photo was unreadable" when the model actually returned an impossible
value. That ordering is covered by the corpus escape cases (`out-of-range-fields`,
`blank-plus-out-of-range`).
Unknown and out-of-distribution images can only be judged from what the schema lets the model say: it
must declare `isStool`, a confidence, and a quality. There is no separate OOD channel, and this policy
does not invent one. A photo the model cannot place therefore lands on `not_stool`, `low_confidence`,
or `low_quality` — all three abstain.
## Per-release gate
The calibration harness ([`scripts/calibration_report.py`](../scripts/calibration_report.py), issue
#29) scores candidates on a held-out test split with a threshold selected on validation only. The
policy turns that report into a single decision with `evaluatePrefillGate(report)`:
| Requirement | Blocked reason if unmet |
|---|---|
| Report verdict is `GO` | `calibration-verdict-not-go` |
| The report names a recommended candidate | `no-recommended-calibration` |
| That candidate's threshold was selected on `validation` | `threshold-not-selected-on-validation` |
| That candidate was scored on the held-out `test` split | `gate-not-scored-on-held-out-test` |
| That candidate passed every release gate | `candidate-failed-release-gates` |
A missing or unreadable report is a block, never a default allow.
Run it as a receipt:
```bash
python3 scripts/calibration_report.py \
--predictions tests/fixtures/calibration/predictions.jsonl \
--min-coverage 0.5 --json /tmp/calibration-report.json
node scripts/abstention_policy_receipt.mjs --report /tmp/calibration-report.json
# exit 0 = prefill authorised, 3 = blocked, 2 = the report could not be read
```
**Current state:** the committed synthetic predictions produce `BLOCKED`, so the receipt exits 3 and
names why. No production candidate exists yet — issue #31 (fine-tuned classifier) is the dependency —
and the policy exists so that candidate is held to one stated contract instead of whatever it happens
to score. The release builder does not gate on this receipt yet; it records the quality gate on camera
through the demo instead. Wiring the receipt into `scripts/build_release.py` as a hard prefill gate
belongs with #31, when there is a candidate to judge.
## Versioning
`ABSTENTION_POLICY.version` is an integer that changes whenever a limit changes. A policy change is a
behaviour change: the release notes must be able to say which policy produced a suggestion. The
shipped object is frozen so nothing can tighten or loosen it at runtime; callers that need a
different policy pass one explicitly (`parseVisionResponse(raw, policy)`), which is how the tests
prove the parser has no hidden limits of its own.
## What this policy does not do
- It does not decide the Bristol type, the colour, or the quality. It only decides whether to show
them.
- It does not touch deterministic red-flag escalation, which runs independently of model state.
- It does not read, store, or log an image, and it never inspects inference bodies.
- It does not set the numeric thresholds from anything but validation data, and it does not convert a
`BLOCKED` calibration into a pass.
+4
View File
@@ -7,6 +7,10 @@ splits that are contributor-held-out and reproducible, comparing candidate model
The implementation is [`scripts/calibration_report.py`](../scripts/calibration_report.py). It reads
predictions something else produced, and it refuses to compute gates it cannot honestly compute.
The GO/BLOCKED verdict this report produces is consumed by the production abstention policy in
[`docs/ABSTENTION-POLICY.md`](ABSTENTION-POLICY.md): a candidate that has not cleared these gates
cannot authorise a visual prefill.
**This harness does not train, does not diagnose, and never reads an image.** It only sees Bristol
labels, predictions, and confidences.
+4 -3
View File
@@ -4,7 +4,7 @@
"private": true,
"type": "module",
"scripts": {
"test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/draft.test.js tests/suggestion.test.js tests/followup.test.js tests/vision-readiness.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js tests/threat-model.test.js tests/vision-corpus.test.js tests/vision-boundary.test.js tests/worker-benchmark.test.js tests/worker-ready.test.js tests/worker-container.test.js tests/summary.test.js tests/escalation-regression.test.js tests/grounded-chat.test.js tests/photo-lifecycle.test.js tests/training-contribution.test.js tests/portability.test.js",
"test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/draft.test.js tests/suggestion.test.js tests/followup.test.js tests/vision-readiness.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js tests/threat-model.test.js tests/vision-corpus.test.js tests/vision-boundary.test.js tests/worker-benchmark.test.js tests/worker-ready.test.js tests/worker-container.test.js tests/summary.test.js tests/escalation-regression.test.js tests/grounded-chat.test.js tests/photo-lifecycle.test.js tests/training-contribution.test.js tests/portability.test.js tests/abstention-policy.test.js",
"test:ui": "node tests/ui.acceptance.mjs",
"test:photo": "node tests/photo-first.acceptance.mjs",
"test:mobile-capture": "node tests/mobile-capture.acceptance.mjs",
@@ -18,13 +18,14 @@
"test:grounded-chat": "node tests/grounded-chat.acceptance.mjs",
"test:photo-lifecycle": "node tests/photo-lifecycle.acceptance.mjs",
"test:training-contribution": "node tests/training-contribution.acceptance.mjs",
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py && node --check scripts/record_worker_benchmark.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh",
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh",
"check:diff": "bash scripts/check_diff.sh",
"start": "node server.mjs",
"test:worker-readiness": "node tests/worker-readiness.acceptance.mjs",
"test:portability": "node tests/portability.acceptance.mjs",
"test:corpus": "python3 tests/corpus-manifest.test.py",
"test:calibration": "python3 tests/calibration-harness.test.py"
"test:calibration": "python3 tests/calibration-harness.test.py",
"test:policy": "node --test tests/abstention-policy.test.js"
},
"devDependencies": {
"playwright": "^1.62.1"
+49
View File
@@ -0,0 +1,49 @@
#!/usr/bin/env node
// Executable receipt for the production abstention policy (issue #32).
//
// The calibration harness decides GO/BLOCKED for a candidate model. This reads that report and
// answers the only question the app cares about: may a suggestion be shipped at all? It exits 3 when
// prefill is blocked, so a release step can gate on it instead of reading prose.
//
// There is no model candidate yet (#31), so in practice this receipt reports BLOCKED today. That is
// the honest state: the policy is ready, the evidence is not.
import { readFile } from 'node:fs/promises';
import { ABSTENTION_POLICY, evaluatePrefillGate } from '../src/abstention-policy.js';
function parseArgs(argv) {
const args = { report: '' };
for (let i = 0; i < argv.length; i += 1) {
if (argv[i] === '--report') {
args.report = argv[i + 1] || '';
i += 1;
} else throw new Error(`unknown argument: ${argv[i]}`);
}
if (!args.report) throw new Error('--report <calibration-report.json> is required');
return args;
}
let args;
try {
args = parseArgs(process.argv.slice(2));
} catch (error) {
console.error(error.message);
process.exit(2);
}
let report;
try {
report = JSON.parse(await readFile(args.report, 'utf8'));
} catch (error) {
console.error(`could not read the calibration report at ${args.report}: ${error.message}`);
process.exit(2);
}
const gate = evaluatePrefillGate(report);
const policy = ABSTENTION_POLICY;
console.log('TIMMY ABSTENTION POLICY RECEIPT');
console.log(`policy: version ${policy.version} · confidence ≥ ${policy.minConfidence.toFixed(2)} · quality ${policy.acceptedImageQuality.join('|')}`);
console.log(`calibration: verdict ${gate.verdict} · model ${gate.calibrationModel || 'none'}`);
console.log(gate.prefillAllowed
? `prefill: AUTHORISED for ${gate.calibrationModel}`
: `prefill: BLOCKED — ${gate.blockedBy.join(', ')}`);
process.exit(gate.prefillAllowed ? 0 : 3);
+3 -1
View File
@@ -46,6 +46,7 @@ DEMO_REQUIRED_CAPTIONS = (
"A tarry note already saved in the journal escalates the same way, before any provider call",
"An out-of-range answer fails closed: nothing is applied and no provider detail reaches you",
"A filled-in field that is out of range still fails closed, even when the other field came back blank",
"A confident reply about a blurry photo still abstains — photo quality is part of the policy, not just confidence",
"Reachable is not loaded: no AI control, no consent, and no upload until the pinned model is resident",
"Every pattern line names the confirmed logs behind it, and following one shows exactly those records",
"Two logs is not a pattern: the summary refuses to describe a trend it cannot support",
@@ -340,7 +341,7 @@ def main() -> int:
"codec": "h264",
"pixel_format": "yuv420p",
"recorded_from_working_app": True,
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything",
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything",
},
"gates": {
"unit_security": "passed",
@@ -351,6 +352,7 @@ def main() -> int:
"provisional_suggestion_state": "passed",
"hard_stool_followup": "passed",
"vision_corpus_refusal": "passed",
"abstention_quality_gate": "passed",
"worker_container_receipts": "passed",
"worker_post_load_readiness": "passed",
"longitudinal_pattern_summary": "passed",
+18 -1
View File
@@ -341,6 +341,20 @@ await tap('#analyze-photo', 600);
await page.getByText(/does not show a form Timmy can read safely/i).waitFor();
await caption('Unreadable visual fields abstain as well: a valid reply with nothing to prefill is not an error', 2700);
await indicate('.scan-result');
// Issue #32's quality gate: the model can be highly confident about a photo it also calls poor, and
// that confident answer must still abstain. Replayed from the committed corpus, and asserted in the
// browser so the scene is a check rather than a claim.
analyzeOverride = await corpusAnalyzeResult(corpusCase('poor-quality-confident'));
await tap('#retake-photo', 400);
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/vision-corpus/blur-low-detail.png');
await page.locator('#ai-consent').check();
await tap('#analyze-photo', 600);
await page.getByText(/too blurry or poorly lit/i).waitFor();
if (await page.getByText(/BRISTOL FORM/i).count() !== 0) {
throw new Error('A low-quality photo must not prefill a form, however confident the reply looked');
}
await caption('A confident reply about a blurry photo still abstains — photo quality is part of the policy, not just confidence', 3100);
await indicate('.scan-result');
analyzeOverride = await corpusAnalyzeResult(corpusCase('out-of-range-fields'));
await tap('#retake-photo', 400);
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/vision-corpus/out-of-range-fields.png');
@@ -769,8 +783,11 @@ const gateRows = [
['—', ''],
];
for (const row of gate.report.models) {
// The model id gets its own row: a 74px key column broke "specialist-candidate" at its hyphen,
// which reads as a label with no value and a stray line under it. The name is the value here.
gateRows.push(['model', row.model]);
gateRows.push([
row.model,
'metrics',
`thr ${row.threshold.toFixed(2)} · F1 ${row.macroF1.toFixed(3)} · P@C ${row.precisionAtCoverage.toFixed(3)} · cov ${row.coverage.toFixed(2)}`,
]);
gateRows.push(['gate', `FAIL: ${row.gate.failed.join(', ')} · ECE ${row.ece.toFixed(3)}`]);
+69
View File
@@ -0,0 +1,69 @@
// The production abstention policy (issue #32).
//
// A suggestion is only ever a visual prefill, and this module is the single place that decides when
// Timmy is allowed to offer one. Two different questions live here, and they are deliberately
// separate:
//
// 1. Per-photo: is this individual answer good enough to show? Confidence, image quality, and
// whether the model actually read a form and a colour.
// 2. Per-release: has a candidate cleared the calibration gates on a held-out split, with its
// threshold chosen on validation only? Until it has, no prefill may be shipped at all.
//
// The copy is fixed here so the browser, the corpus, and the server cannot drift into three
// different explanations of the same refusal.
export const ABSTENTION_COPY = Object.freeze({
not_stool: 'The image does not clearly show stool.',
low_confidence: 'The image is too uncertain to prefill safely.',
unusable_visual_fields: 'The photo does not show a form Timmy can read safely.',
low_quality: 'The photo is too blurry or poorly lit to suggest a form safely.',
});
// Bump `version` whenever a limit below changes: a policy change is a behaviour change, and the
// release notes have to be able to name which policy a suggestion was produced under.
export const ABSTENTION_POLICY = Object.freeze({
version: 1,
minConfidence: 0.55,
// "fair" may prefill; "poor" never does. The model already returns its own quality judgement, so
// a confident answer about an unreadable photo is exactly the case this gate exists for.
acceptedImageQuality: Object.freeze(['good', 'fair']),
});
export function abstainFor(reason, isStool = true) {
if (!Object.prototype.hasOwnProperty.call(ABSTENTION_COPY, reason)) {
throw new Error(`unknown abstention reason: ${reason}`);
}
return {
status: 'needs_user_input',
isStool,
abstainedFor: reason,
reason: ABSTENTION_COPY[reason],
};
}
export function isLowQuality(imageQuality, policy = ABSTENTION_POLICY) {
return !policy.acceptedImageQuality.includes(String(imageQuality || '').toLowerCase());
}
// Release-level gate. The calibration harness already refuses to pick a threshold anywhere but the
// validation split; this decides whether a report authorises prefill at all, and names every reason
// it does not. A missing or malformed report is a block, never a default-allow.
export function evaluatePrefillGate(report) {
const blockedBy = [];
const verdict = report?.verdict || 'UNKNOWN';
if (verdict !== 'GO') blockedBy.push('calibration-verdict-not-go');
const models = Array.isArray(report?.models) ? report.models : [];
const recommended = models.find(row => row?.model === report?.recommended) || null;
if (!recommended) {
blockedBy.push('no-recommended-calibration');
} else {
if (recommended.selectionSplit !== 'validation') blockedBy.push('threshold-not-selected-on-validation');
if (recommended.evaluatedSplit !== 'test') blockedBy.push('gate-not-scored-on-held-out-test');
if (recommended.gate?.passed !== true) blockedBy.push('candidate-failed-release-gates');
}
return {
prefillAllowed: blockedBy.length === 0,
verdict,
calibrationModel: recommended?.model || null,
blockedBy,
};
}
+8 -9
View File
@@ -1,3 +1,5 @@
import { ABSTENTION_POLICY, abstainFor, isLowQuality } from './abstention-policy.js';
const COLORS = new Set(['brown', 'green', 'yellow', 'pale', 'red', 'black']);
const QUALITIES = new Set(['good', 'fair', 'poor']);
const MAX_IMAGE_BYTES = 4 * 1024 * 1024;
@@ -26,13 +28,13 @@ function asObject(raw) {
}
}
export function parseVisionResponse(raw) {
export function parseVisionResponse(raw, policy = ABSTENTION_POLICY) {
const value = asObject(raw);
if (typeof value.isStool !== 'boolean') throw visionResponseError(VISION_FAILURE_CODES.invalidFields, 'Invalid AI response: isStool is required.');
const confidence = Number(value.confidence);
if (!Number.isFinite(confidence) || confidence < 0 || confidence > 1) throw visionResponseError(VISION_FAILURE_CODES.invalidFields, 'Invalid AI response: confidence is required.');
if (!value.isStool) return { status: 'needs_user_input', isStool: false, abstainedFor: 'not_stool', reason: 'The image does not clearly show stool.' };
if (confidence < 0.55) return { status: 'needs_user_input', isStool: true, abstainedFor: 'low_confidence', reason: 'The image is too uncertain to prefill safely.' };
if (!value.isStool) return abstainFor('not_stool', false);
if (confidence < policy.minConfidence) return abstainFor('low_confidence');
// The request schema lets the model return a null form and color when it cannot read them.
// A blank field means the model could not read it, so a schema-valid answer with nothing usable
// abstains and nothing is prefilled. A filled-in field is a claim, though: if it sits outside the
@@ -47,15 +49,12 @@ export function parseVisionResponse(raw) {
throw visionResponseError(VISION_FAILURE_CODES.invalidFields, 'Invalid AI response: visual fields are out of range.');
}
if (bristolType === null || color === null) {
return {
status: 'needs_user_input',
isStool: true,
abstainedFor: 'unusable_visual_fields',
reason: 'The photo does not show a form Timmy can read safely.',
};
return abstainFor('unusable_visual_fields');
}
const imageQuality = String(value.imageQuality || '').toLowerCase();
if (!QUALITIES.has(imageQuality)) throw visionResponseError(VISION_FAILURE_CODES.invalidFields, 'Invalid AI response: visual fields are out of range.');
// A confident reading of a photo the model itself calls poor is still not a safe prefill.
if (isLowQuality(imageQuality, policy)) return abstainFor('low_quality');
return {
status: 'suggestion',
isStool: true,
+117
View File
@@ -0,0 +1,117 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { execFile } from 'node:child_process';
import { mkdtemp, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { promisify } from 'node:util';
import { ABSTENTION_COPY, ABSTENTION_POLICY, abstainFor, evaluatePrefillGate } from '../src/abstention-policy.js';
test('every abstention reason has one fixed piece of copy the user sees', () => {
assert.equal(abstainFor('not_stool').reason, 'The image does not clearly show stool.');
assert.equal(abstainFor('low_confidence').reason, 'The image is too uncertain to prefill safely.');
assert.equal(abstainFor('unusable_visual_fields').reason, 'The photo does not show a form Timmy can read safely.');
assert.equal(abstainFor('low_quality').reason, 'The photo is too blurry or poorly lit to suggest a form safely.');
assert.deepEqual(abstainFor('low_quality'), {
status: 'needs_user_input',
isStool: true,
abstainedFor: 'low_quality',
reason: ABSTENTION_COPY.low_quality,
});
assert.equal(abstainFor('not_stool', false).isStool, false);
assert.throws(() => abstainFor('looked_wrong'), /unknown abstention reason/);
});
test('the policy states the confidence floor and the quality levels that may prefill', () => {
assert.ok(Number.isInteger(ABSTENTION_POLICY.version));
assert.equal(ABSTENTION_POLICY.minConfidence, 0.55);
assert.deepEqual([...ABSTENTION_POLICY.acceptedImageQuality], ['good', 'fair']);
assert.equal(ABSTENTION_POLICY.acceptedImageQuality.includes('poor'), false);
assert.equal(Object.isFrozen(ABSTENTION_POLICY), true, 'the shipped policy must not be mutated at runtime');
});
test('prefill is authorised only by a GO report whose threshold was selected on validation', () => {
const passing = {
verdict: 'GO',
recommended: 'specialist-candidate',
models: [{
model: 'specialist-candidate',
selectionSplit: 'validation',
evaluatedSplit: 'test',
gate: { passed: true, failed: [] },
}],
};
assert.deepEqual(evaluatePrefillGate(passing), {
prefillAllowed: true,
verdict: 'GO',
calibrationModel: 'specialist-candidate',
blockedBy: [],
});
const blocked = evaluatePrefillGate({
verdict: 'BLOCKED',
recommended: null,
models: [{ model: 'specialist-candidate', selectionSplit: 'validation', evaluatedSplit: 'test', gate: { passed: false, failed: ['precisionAtCoverage'] } }],
});
assert.equal(blocked.prefillAllowed, false);
assert.deepEqual(blocked.blockedBy, ['calibration-verdict-not-go', 'no-recommended-calibration']);
const selectedOnTest = evaluatePrefillGate({
verdict: 'GO',
recommended: 'opportunist',
models: [{ model: 'opportunist', selectionSplit: 'test', evaluatedSplit: 'test', gate: { passed: true, failed: [] } }],
});
assert.equal(selectedOnTest.prefillAllowed, false, 'a threshold chosen on the split it is scored on is not evidence');
assert.deepEqual(selectedOnTest.blockedBy, ['threshold-not-selected-on-validation']);
const noReport = evaluatePrefillGate(undefined);
assert.equal(noReport.prefillAllowed, false);
assert.match(noReport.blockedBy.join(','), /calibration-verdict-not-go/);
});
const execFileAsync = promisify(execFile);
const receiptScript = fileURLToPath(new URL('../scripts/abstention_policy_receipt.mjs', import.meta.url));
async function runReceipt(report) {
const directory = await mkdtemp(join(tmpdir(), 'timmy-policy-'));
const reportPath = join(directory, 'calibration-report.json');
await writeFile(reportPath, JSON.stringify(report));
try {
const { stdout } = await execFileAsync(process.execPath, [receiptScript, '--report', reportPath]);
return { code: 0, stdout };
} catch (error) {
return { code: error.code, stdout: error.stdout || '', stderr: error.stderr || '' };
}
}
test('the receipt refuses to authorise prefill without a passing held-out calibration', async () => {
const blocked = await runReceipt({
verdict: 'BLOCKED',
recommended: null,
models: [{ model: 'specialist-candidate', selectionSplit: 'validation', evaluatedSplit: 'test', gate: { passed: false, failed: ['precisionAtCoverage'] } }],
});
assert.equal(blocked.code, 3, 'a blocked calibration must exit non-zero instead of shipping prefill');
assert.match(blocked.stdout, /prefill: BLOCKED/);
assert.match(blocked.stdout, /calibration-verdict-not-go/);
assert.match(blocked.stdout, /policy: version 1 · confidence ≥ 0\.55 · quality good\|fair/);
const passing = await runReceipt({
verdict: 'GO',
recommended: 'specialist-candidate',
models: [{ model: 'specialist-candidate', selectionSplit: 'validation', evaluatedSplit: 'test', gate: { passed: true, failed: [] } }],
});
assert.equal(passing.code, 0);
assert.match(passing.stdout, /prefill: AUTHORISED for specialist-candidate/);
});
test('the receipt treats a missing report as a block and never as a default allow', async () => {
const directory = await mkdtemp(join(tmpdir(), 'timmy-policy-'));
try {
await execFileAsync(process.execPath, [receiptScript, '--report', join(directory, 'absent.json')]);
assert.fail('a missing calibration report must not exit 0');
} catch (error) {
assert.equal(error.code, 2);
assert.match(String(error.stderr), /could not read|absent/i);
}
});
+37
View File
@@ -1,5 +1,6 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { ABSTENTION_COPY, ABSTENTION_POLICY } from '../src/abstention-policy.js';
import {
buildVisionRequest,
@@ -131,3 +132,39 @@ test('abstains instead of failing when a schema-valid response carries no readab
reason: 'The photo does not show a form Timmy can read safely.',
});
});
test('a confident answer about a low-quality photo still abstains', () => {
// Issue #32: confidence is not the only gate. A blurry or badly lit photo can produce a
// confident-looking form and colour, and prefilling it would show the user a claim the photo
// cannot support. Quality is part of the policy, so this abstains instead of prefilling.
assert.deepEqual(parseVisionResponse({
isStool: true,
bristolType: 4,
color: 'brown',
confidence: 0.94,
imageQuality: 'poor',
observations: 'Form barely visible through motion blur.',
}), {
status: 'needs_user_input',
isStool: true,
abstainedFor: 'low_quality',
reason: 'The photo is too blurry or poorly lit to suggest a form safely.',
});
});
test('the parse path reads the shipped policy instead of hardcoding its own limits', () => {
// Issue #32: policy numbers and copy live in one module. A caller may tighten the policy for a
// release without editing the parser, and a photo that the default policy would accept has to
// abstain under the stricter one — otherwise the parser is carrying its own hidden policy.
const fields = { isStool: true, bristolType: 4, color: 'brown', confidence: 0.8, imageQuality: 'fair', observations: 'Form visible but soft.' };
assert.equal(parseVisionResponse(fields).status, 'suggestion', 'fair quality may prefill under the shipped policy');
const strict = { ...ABSTENTION_POLICY, minConfidence: 0.9, acceptedImageQuality: ['good'] };
assert.equal(parseVisionResponse(fields, strict).abstainedFor, 'low_confidence');
assert.equal(parseVisionResponse({ ...fields, confidence: 0.95 }, strict).abstainedFor, 'low_quality');
assert.equal(parseVisionResponse({ ...fields, confidence: 0.95, imageQuality: 'good' }, strict).status, 'suggestion');
assert.equal(
parseVisionResponse({ ...fields, imageQuality: 'poor' }).reason,
ABSTENTION_COPY.low_quality,
'the refusal copy must come from the policy module, not a second copy in the parser',
);
});
+16
View File
@@ -131,6 +131,22 @@
"expectedOutcome": "fail_closed",
"expectedReason": "out_of_range_or_missing_fields",
"providerContent": "{\"isStool\":true,\"bristolType\":9,\"color\":null,\"confidence\":0.9,\"imageQuality\":\"good\",\"observations\":\"Form read, colour not.\"}"
},
{
"id": "poor-quality-confident",
"caseClass": "blur",
"file": "blur-low-detail.png",
"bytes": 21598,
"sha256": "17000163662a42fd434bb2b2428d9e5a2156da08cbc82210e41dfc6fa7e24852",
"width": 640,
"height": 480,
"format": "png",
"synthetic": true,
"license": "CC0-1.0",
"provenance": "The same synthetic motion-blur fixture that blur-low-detail replays, replayed again with a confident provider answer; drawn deterministically by scripts/build_vision_corpus.py, no camera, no human subject, no third-party content.",
"expectedOutcome": "abstained",
"expectedReason": "low_quality",
"providerContent": "{\"isStool\":true,\"bristolType\":4,\"color\":\"brown\",\"confidence\":0.94,\"imageQuality\":\"poor\",\"observations\":\"Form barely visible through motion blur.\"}"
}
]
}
+8
View File
@@ -272,6 +272,14 @@ test('release demo visibly exercises the corpus refusal paths instead of only th
assert.match(demo, /A non-stool photo gets a refusal, never a guess/);
assert.match(demo, /Unreadable visual fields abstain as well/);
assert.match(demo, /An out-of-range answer fails closed/);
// Issue #32's quality gate: a confident answer about a low-quality photo must abstain on camera.
assert.match(demo, /corpusCase\('poor-quality-confident'\)/);
assert.match(demo, /tests\/fixtures\/vision-corpus\/blur-low-detail\.png/);
assert.match(demo, /too blurry or poorly lit/);
assert.match(demo, /must not prefill a form, however confident the reply looked/);
assert.match(demo, /A confident reply about a blurry photo still abstains/);
assert.match(builder, /A confident reply about a blurry photo still abstains — photo quality is part of the policy, not just confidence/);
assert.match(builder, /"abstention_quality_gate": "passed"/);
assert.match(demo, /no real photos in the corpus|no real photos in the recording/);
// The refusal copy the user sees is gated on camera, so a release cannot drop it silently.
assert.match(builder, /An out-of-range answer fails closed/);