Gate visual prefill on a versioned abstention policy (#32) #87

2026-10-26T10:58:55Z

rockachopa commented

2026-10-26 10:58:55 +00:00

Closes #32

What changed

Confidence was the only gate between a model answer and a prefilled entry, so a confident reading of a blurry photo could still be offered as fact. Photo quality is now part of the policy.

src/abstention-policy.js — one versioned policy: confidence floor (0.55), accepted quality (good, fair), fixed refusal copy for every reason, and evaluatePrefillGate(report).
src/analysis.js — parseVisionResponse(raw, policy) reads its limits and copy from that module. imageQuality: "poor" now abstains (low_quality) regardless of confidence. Release-level gate: a calibration report authorises prefill only with GO, a recommended candidate, a threshold selected on validation, scored on the held-out test split, gates passed. A missing report is a block, never a default allow.
scripts/abstention_policy_receipt.mjs — executable receipt over the #29 report (exit 0 authorised / 3 blocked / 2 unreadable).
Corpus — new synthetic case poor-quality-confident: a 0.94-confidence answer about the committed motion-blur fixture, expected to abstain with low_quality. Reuses the existing CC0 fixture; no new or real imagery.
Release demo — new scene replays that corpus case through the real analysis path, asserts in the browser that no Bristol form is prefilled, then captions it. Caption added to the builder's required-caption gate and abstention_quality_gate.
docs/ABSTENTION-POLICY.md — gate order, refusal copy, release-gate table, versioning, and what the policy deliberately does not do.

Evidence

npm test — 226/226 pass (219 before this change)
npm run test:calibration — 19 pass; npm run test:corpus — 8 pass
npm run test:vision-refusal, test:photo, test:provisional — pass
npm run check:syntax, bash scripts/check_diff.sh — pass
Full real-browser demo recorded locally: exit 0, 78/78 captions painted, new caption verified on screen
Receipt against the committed synthetic predictions: prefill: BLOCKED — calibration-verdict-not-go, no-recommended-calibration (exit 3)

Deferred (stated, not implied)

No candidate model exists until #31. The receipt reports BLOCKED today because the synthetic predictions fail the gates. That is the honest state; this PR ships the contract the candidate will be judged against, not a passing model.
build_release.py does not hard-gate on the receipt yet. Wiring it in belongs with #31, when there is a candidate whose verdict is meaningful. The on-camera quality gate is the release evidence here.
Unknown/OOD coverage is limited to what the schema can express: not_stool, low_confidence, and now low_quality. There is no separate OOD channel and this PR does not invent one.
The abstention card's heading is still the shared "No confident match." for every reason; the sub-copy names quality. A per-reason heading would touch the existing refusal selectors and is left out of this bounded change.

Safety boundary

No diagnosis, no new inference, no image storage or logging, no provider detail in refusal copy, and deterministic red-flag escalation is untouched. The change only makes prefill harder.

Closes #32 ## What changed Confidence was the only gate between a model answer and a prefilled entry, so a confident reading of a blurry photo could still be offered as fact. Photo quality is now part of the policy. - **`src/abstention-policy.js`** — one versioned policy: confidence floor (0.55), accepted quality (`good`, `fair`), fixed refusal copy for every reason, and `evaluatePrefillGate(report)`. - **`src/analysis.js`** — `parseVisionResponse(raw, policy)` reads its limits and copy from that module. `imageQuality: "poor"` now abstains (`low_quality`) regardless of confidence. Release-level gate: a calibration report authorises prefill only with `GO`, a recommended candidate, a threshold selected on `validation`, scored on the held-out `test` split, gates passed. A missing report is a block, never a default allow. - **`scripts/abstention_policy_receipt.mjs`** — executable receipt over the #29 report (exit 0 authorised / 3 blocked / 2 unreadable). - **Corpus** — new synthetic case `poor-quality-confident`: a 0.94-confidence answer about the committed motion-blur fixture, expected to abstain with `low_quality`. Reuses the existing CC0 fixture; no new or real imagery. - **Release demo** — new scene replays that corpus case through the real analysis path, asserts in the browser that no Bristol form is prefilled, then captions it. Caption added to the builder's required-caption gate and `abstention_quality_gate`. - **`docs/ABSTENTION-POLICY.md`** — gate order, refusal copy, release-gate table, versioning, and what the policy deliberately does not do. ## Evidence - `npm test` — 226/226 pass (219 before this change) - `npm run test:calibration` — 19 pass; `npm run test:corpus` — 8 pass - `npm run test:vision-refusal`, `test:photo`, `test:provisional` — pass - `npm run check:syntax`, `bash scripts/check_diff.sh` — pass - Full real-browser demo recorded locally: exit 0, 78/78 captions painted, new caption verified on screen - Receipt against the committed synthetic predictions: `prefill: BLOCKED — calibration-verdict-not-go, no-recommended-calibration` (exit 3) ## Deferred (stated, not implied) - **No candidate model exists until #31.** The receipt reports BLOCKED today because the synthetic predictions fail the gates. That is the honest state; this PR ships the contract the candidate will be judged against, not a passing model. - **`build_release.py` does not hard-gate on the receipt yet.** Wiring it in belongs with #31, when there is a candidate whose verdict is meaningful. The on-camera quality gate is the release evidence here. - **Unknown/OOD coverage** is limited to what the schema can express: `not_stool`, `low_confidence`, and now `low_quality`. There is no separate OOD channel and this PR does not invent one. - The abstention card's heading is still the shared "No confident match." for every reason; the sub-copy names quality. A per-reason heading would touch the existing refusal selectors and is left out of this bounded change. ## Safety boundary No diagnosis, no new inference, no image storage or logging, no provider detail in refusal copy, and deterministic red-flag escalation is untouched. The change only makes prefill harder.

rockachopa added 1 commit 2026-10-26 10:58:55 +00:00

Gate visual prefill on a versioned abstention policy (#32 )

Quality gates / quality (pull_request) Successful in 13m29s

Details

209e21a951

Confidence was the only thing standing between a model answer and a
prefilled journal entry, so a confident reading of a blurry photo could
still be offered as fact. Quality is now part of the policy: `poor`
abstains however confident the reply looked, with its own fixed copy.

The confidence floor, accepted quality levels, and refusal copy move into
src/abstention-policy.js so the parser, the corpus, and the browser cannot
drift into three explanations of the same refusal. parseVisionResponse
takes an optional policy, which is how the tests prove the parser carries
no hidden limits of its own.

evaluatePrefillGate turns the #29 calibration report into one release
decision: GO plus a validation-selected threshold scored on the held-out
test split, or prefill is blocked with each reason named. A missing report
is a block, never a default allow. scripts/abstention_policy_receipt.mjs
exposes that as a receipt (exit 0 / 3 / 2).

The committed synthetic predictions currently report BLOCKED and the
receipt says so. No candidate exists until #31, and the builder does not
hard-gate on the receipt yet; the on-camera quality gate is the evidence
this release carries.

The corpus gains a case that replays a 0.94-confidence answer about the
synthetic motion-blur fixture, and the release demo asserts in the browser
that nothing is prefilled before captioning it.

Verified: 226 unit tests, calibration harness (19), corpus manifest (8),
vision-refusal, photo-first, and provisional acceptance, check:syntax,
check:diff, and a full real-browser demo recording whose caption receipt
confirms the new scene painted on screen.

rockachopa merged commit 2533f83af3 into main

2026-10-26 12:03:48 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: stackchain/timmy-talking-turd#87