Gate visual prefill on a versioned abstention policy (#32)
Quality gates / quality (pull_request) Successful in 13m29s

Confidence was the only thing standing between a model answer and a
prefilled journal entry, so a confident reading of a blurry photo could
still be offered as fact. Quality is now part of the policy: `poor`
abstains however confident the reply looked, with its own fixed copy.

The confidence floor, accepted quality levels, and refusal copy move into
src/abstention-policy.js so the parser, the corpus, and the browser cannot
drift into three explanations of the same refusal. parseVisionResponse
takes an optional policy, which is how the tests prove the parser carries
no hidden limits of its own.

evaluatePrefillGate turns the #29 calibration report into one release
decision: GO plus a validation-selected threshold scored on the held-out
test split, or prefill is blocked with each reason named. A missing report
is a block, never a default allow. scripts/abstention_policy_receipt.mjs
exposes that as a receipt (exit 0 / 3 / 2).

The committed synthetic predictions currently report BLOCKED and the
receipt says so. No candidate exists until #31, and the builder does not
hard-gate on the receipt yet; the on-camera quality gate is the evidence
this release carries.

The corpus gains a case that replays a 0.94-confidence answer about the
synthetic motion-blur fixture, and the release demo asserts in the browser
that nothing is prefilled before captioning it.

Verified: 226 unit tests, calibration harness (19), corpus manifest (8),
vision-refusal, photo-first, and provisional acceptance, check:syntax,
check:diff, and a full real-browser demo recording whose caption receipt
confirms the new scene painted on screen.
This commit is contained in:
timmy committed 2026-10-26 10:58:38 +00:00
1 parent 2a10a20518
commit 209e21a951
13 files changed
+428 -14

No files matched your search

+49
View File
@@ -0,0 +1,49 @@
#!/usr/bin/env node
// Executable receipt for the production abstention policy (issue #32).
//
// The calibration harness decides GO/BLOCKED for a candidate model. This reads that report and
// answers the only question the app cares about: may a suggestion be shipped at all? It exits 3 when
// prefill is blocked, so a release step can gate on it instead of reading prose.
//
// There is no model candidate yet (#31), so in practice this receipt reports BLOCKED today. That is
// the honest state: the policy is ready, the evidence is not.
import { readFile } from 'node:fs/promises';
import { ABSTENTION_POLICY, evaluatePrefillGate } from '../src/abstention-policy.js';
function parseArgs(argv) {
const args = { report: '' };
for (let i = 0; i < argv.length; i += 1) {
if (argv[i] === '--report') {
args.report = argv[i + 1] || '';
i += 1;
} else throw new Error(`unknown argument: ${argv[i]}`);
}
if (!args.report) throw new Error('--report <calibration-report.json> is required');
return args;
}
let args;
try {
args = parseArgs(process.argv.slice(2));
} catch (error) {
console.error(error.message);
process.exit(2);
}
let report;
try {
report = JSON.parse(await readFile(args.report, 'utf8'));
} catch (error) {
console.error(`could not read the calibration report at ${args.report}: ${error.message}`);
process.exit(2);
}
const gate = evaluatePrefillGate(report);
const policy = ABSTENTION_POLICY;
console.log('TIMMY ABSTENTION POLICY RECEIPT');
console.log(`policy: version ${policy.version} · confidence ≥ ${policy.minConfidence.toFixed(2)} · quality ${policy.acceptedImageQuality.join('|')}`);
console.log(`calibration: verdict ${gate.verdict} · model ${gate.calibrationModel || 'none'}`);
console.log(gate.prefillAllowed
? `prefill: AUTHORISED for ${gate.calibrationModel}`
: `prefill: BLOCKED — ${gate.blockedBy.join(', ')}`);
process.exit(gate.prefillAllowed ? 0 : 3);
+3 -1
View File
@@ -46,6 +46,7 @@ DEMO_REQUIRED_CAPTIONS = (
"A tarry note already saved in the journal escalates the same way, before any provider call",
"An out-of-range answer fails closed: nothing is applied and no provider detail reaches you",
"A filled-in field that is out of range still fails closed, even when the other field came back blank",
"A confident reply about a blurry photo still abstains — photo quality is part of the policy, not just confidence",
"Reachable is not loaded: no AI control, no consent, and no upload until the pinned model is resident",
"Every pattern line names the confirmed logs behind it, and following one shows exactly those records",
"Two logs is not a pattern: the summary refuses to describe a trend it cannot support",
@@ -340,7 +341,7 @@ def main() -> int:
"codec": "h264",
"pixel_format": "yuv420p",
"recorded_from_working_app": True,
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything",
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything",
},
"gates": {
"unit_security": "passed",
@@ -351,6 +352,7 @@ def main() -> int:
"provisional_suggestion_state": "passed",
"hard_stool_followup": "passed",
"vision_corpus_refusal": "passed",
"abstention_quality_gate": "passed",
"worker_container_receipts": "passed",
"worker_post_load_readiness": "passed",
"longitudinal_pattern_summary": "passed",
+14
View File
@@ -341,6 +341,20 @@ await tap('#analyze-photo', 600);
await page.getByText(/does not show a form Timmy can read safely/i).waitFor();
await caption('Unreadable visual fields abstain as well: a valid reply with nothing to prefill is not an error', 2700);
await indicate('.scan-result');
// Issue #32's quality gate: the model can be highly confident about a photo it also calls poor, and
// that confident answer must still abstain. Replayed from the committed corpus, and asserted in the
// browser so the scene is a check rather than a claim.
analyzeOverride = await corpusAnalyzeResult(corpusCase('poor-quality-confident'));
await tap('#retake-photo', 400);
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/vision-corpus/blur-low-detail.png');
await page.locator('#ai-consent').check();
await tap('#analyze-photo', 600);
await page.getByText(/too blurry or poorly lit/i).waitFor();
if (await page.getByText(/BRISTOL FORM/i).count() !== 0) {
throw new Error('A low-quality photo must not prefill a form, however confident the reply looked');
}
await caption('A confident reply about a blurry photo still abstains — photo quality is part of the policy, not just confidence', 3100);
await indicate('.scan-result');
analyzeOverride = await corpusAnalyzeResult(corpusCase('out-of-range-fields'));
await tap('#retake-photo', 400);
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/vision-corpus/out-of-range-fields.png');