timmy
209e21a951
Quality gates / quality (pull_request) Successful in 13m29s
Confidence was the only thing standing between a model answer and a prefilled journal entry, so a confident reading of a blurry photo could still be offered as fact. Quality is now part of the policy: `poor` abstains however confident the reply looked, with its own fixed copy. The confidence floor, accepted quality levels, and refusal copy move into src/abstention-policy.js so the parser, the corpus, and the browser cannot drift into three explanations of the same refusal. parseVisionResponse takes an optional policy, which is how the tests prove the parser carries no hidden limits of its own. evaluatePrefillGate turns the #29 calibration report into one release decision: GO plus a validation-selected threshold scored on the held-out test split, or prefill is blocked with each reason named. A missing report is a block, never a default allow. scripts/abstention_policy_receipt.mjs exposes that as a receipt (exit 0 / 3 / 2). The committed synthetic predictions currently report BLOCKED and the receipt says so. No candidate exists until #31, and the builder does not hard-gate on the receipt yet; the on-camera quality gate is the evidence this release carries. The corpus gains a case that replays a 0.94-confidence answer about the synthetic motion-blur fixture, and the release demo asserts in the browser that nothing is prefilled before captioning it. Verified: 226 unit tests, calibration harness (19), corpus manifest (8), vision-refusal, photo-first, and provisional acceptance, check:syntax, check:diff, and a full real-browser demo recording whose caption receipt confirms the new scene painted on screen.
50 lines
1.9 KiB
JavaScript
50 lines
1.9 KiB
JavaScript
#!/usr/bin/env node
|
|
// Executable receipt for the production abstention policy (issue #32).
|
|
//
|
|
// The calibration harness decides GO/BLOCKED for a candidate model. This reads that report and
|
|
// answers the only question the app cares about: may a suggestion be shipped at all? It exits 3 when
|
|
// prefill is blocked, so a release step can gate on it instead of reading prose.
|
|
//
|
|
// There is no model candidate yet (#31), so in practice this receipt reports BLOCKED today. That is
|
|
// the honest state: the policy is ready, the evidence is not.
|
|
import { readFile } from 'node:fs/promises';
|
|
import { ABSTENTION_POLICY, evaluatePrefillGate } from '../src/abstention-policy.js';
|
|
|
|
function parseArgs(argv) {
|
|
const args = { report: '' };
|
|
for (let i = 0; i < argv.length; i += 1) {
|
|
if (argv[i] === '--report') {
|
|
args.report = argv[i + 1] || '';
|
|
i += 1;
|
|
} else throw new Error(`unknown argument: ${argv[i]}`);
|
|
}
|
|
if (!args.report) throw new Error('--report <calibration-report.json> is required');
|
|
return args;
|
|
}
|
|
|
|
let args;
|
|
try {
|
|
args = parseArgs(process.argv.slice(2));
|
|
} catch (error) {
|
|
console.error(error.message);
|
|
process.exit(2);
|
|
}
|
|
|
|
let report;
|
|
try {
|
|
report = JSON.parse(await readFile(args.report, 'utf8'));
|
|
} catch (error) {
|
|
console.error(`could not read the calibration report at ${args.report}: ${error.message}`);
|
|
process.exit(2);
|
|
}
|
|
|
|
const gate = evaluatePrefillGate(report);
|
|
const policy = ABSTENTION_POLICY;
|
|
console.log('TIMMY ABSTENTION POLICY RECEIPT');
|
|
console.log(`policy: version ${policy.version} · confidence ≥ ${policy.minConfidence.toFixed(2)} · quality ${policy.acceptedImageQuality.join('|')}`);
|
|
console.log(`calibration: verdict ${gate.verdict} · model ${gate.calibrationModel || 'none'}`);
|
|
console.log(gate.prefillAllowed
|
|
? `prefill: AUTHORISED for ${gate.calibrationModel}`
|
|
: `prefill: BLOCKED — ${gate.blockedBy.join(', ')}`);
|
|
process.exit(gate.prefillAllowed ? 0 : 3);
|