timmy
209e21a951
Confidence was the only thing standing between a model answer and a prefilled journal entry, so a confident reading of a blurry photo could still be offered as fact. Quality is now part of the policy: `poor` abstains however confident the reply looked, with its own fixed copy. The confidence floor, accepted quality levels, and refusal copy move into src/abstention-policy.js so the parser, the corpus, and the browser cannot drift into three explanations of the same refusal. parseVisionResponse takes an optional policy, which is how the tests prove the parser carries no hidden limits of its own. evaluatePrefillGate turns the #29 calibration report into one release decision: GO plus a validation-selected threshold scored on the held-out test split, or prefill is blocked with each reason named. A missing report is a block, never a default allow. scripts/abstention_policy_receipt.mjs exposes that as a receipt (exit 0 / 3 / 2). The committed synthetic predictions currently report BLOCKED and the receipt says so. No candidate exists until #31, and the builder does not hard-gate on the receipt yet; the on-camera quality gate is the evidence this release carries. The corpus gains a case that replays a 0.94-confidence answer about the synthetic motion-blur fixture, and the release demo asserts in the browser that nothing is prefilled before captioning it. Verified: 226 unit tests, calibration harness (19), corpus manifest (8), vision-refusal, photo-first, and provisional acceptance, check:syntax, check:diff, and a full real-browser demo recording whose caption receipt confirms the new scene painted on screen.
5.4 KiB
Abstention policy
Issue #32 asks for raw model scores to be turned into a conservative suggestion policy. This is that policy. It is the only thing that decides whether Timmy offers a visual prefill, and it answers two different questions that are easy to confuse:
- Per photo — is this individual answer good enough to show?
- Per release — has a candidate model earned the right to prefill at all?
Both are implemented in src/abstention-policy.js. The parser in
src/analysis.js reads its limits from that module rather than carrying its own
copy, so a policy change is one edit, not three.
A suggestion is never a diagnosis, never fills in urgency, discomfort, symptoms, or notes, and is never saved until the user confirms it. Abstention is the default; a prefill is the exception.
Why quality is a gate, not a footnote
imageQuality is something the model reports about its own photo. A confident form and colour read
from a blurry photo is exactly the answer a user is most likely to accept without thinking, so
confidence alone cannot authorise a prefill. The policy accepts good and fair; poor abstains
regardless of how confident the reply looked. This is proven end to end by the committed corpus case
poor-quality-confident, which replays a 0.94-confidence answer about the synthetic motion-blur
fixture and expects an abstention.
Per-photo gates
Evaluated in this order. The first gate that fails names the refusal, so the user is told the real reason rather than a generic one.
| Order | Gate | Refusal (abstainedFor) |
User-facing copy |
|---|---|---|---|
| 1 | Model says the photo is not clearly stool | not_stool |
The image does not clearly show stool. |
| 2 | Confidence below minConfidence (0.55) |
low_confidence |
The image is too uncertain to prefill safely. |
| 3 | Form or colour came back blank | unusable_visual_fields |
The photo does not show a form Timmy can read safely. |
| 4 | imageQuality is below the accepted levels |
low_quality |
The photo is too blurry or poorly lit to suggest a form safely. |
A filled-in field outside the contract (bristolType outside 1–7, an unknown color, an unknown
imageQuality) is not an abstention — it fails closed with no provider detail shown, because a
caller must never be told "the photo was unreadable" when the model actually returned an impossible
value. That ordering is covered by the corpus escape cases (out-of-range-fields,
blank-plus-out-of-range).
Unknown and out-of-distribution images can only be judged from what the schema lets the model say: it
must declare isStool, a confidence, and a quality. There is no separate OOD channel, and this policy
does not invent one. A photo the model cannot place therefore lands on not_stool, low_confidence,
or low_quality — all three abstain.
Per-release gate
The calibration harness (scripts/calibration_report.py, issue
#29) scores candidates on a held-out test split with a threshold selected on validation only. The
policy turns that report into a single decision with evaluatePrefillGate(report):
| Requirement | Blocked reason if unmet |
|---|---|
Report verdict is GO |
calibration-verdict-not-go |
| The report names a recommended candidate | no-recommended-calibration |
That candidate's threshold was selected on validation |
threshold-not-selected-on-validation |
That candidate was scored on the held-out test split |
gate-not-scored-on-held-out-test |
| That candidate passed every release gate | candidate-failed-release-gates |
A missing or unreadable report is a block, never a default allow.
Run it as a receipt:
python3 scripts/calibration_report.py \
--predictions tests/fixtures/calibration/predictions.jsonl \
--min-coverage 0.5 --json /tmp/calibration-report.json
node scripts/abstention_policy_receipt.mjs --report /tmp/calibration-report.json
# exit 0 = prefill authorised, 3 = blocked, 2 = the report could not be read
Current state: the committed synthetic predictions produce BLOCKED, so the receipt exits 3 and
names why. No production candidate exists yet — issue #31 (fine-tuned classifier) is the dependency —
and the policy exists so that candidate is held to one stated contract instead of whatever it happens
to score. The release builder does not gate on this receipt yet; it records the quality gate on camera
through the demo instead. Wiring the receipt into scripts/build_release.py as a hard prefill gate
belongs with #31, when there is a candidate to judge.
Versioning
ABSTENTION_POLICY.version is an integer that changes whenever a limit changes. A policy change is a
behaviour change: the release notes must be able to say which policy produced a suggestion. The
shipped object is frozen so nothing can tighten or loosen it at runtime; callers that need a
different policy pass one explicitly (parseVisionResponse(raw, policy)), which is how the tests
prove the parser has no hidden limits of its own.
What this policy does not do
- It does not decide the Bristol type, the colour, or the quality. It only decides whether to show them.
- It does not touch deterministic red-flag escalation, which runs independently of model state.
- It does not read, store, or log an image, and it never inspects inference bodies.
- It does not set the numeric thresholds from anything but validation data, and it does not convert a
BLOCKEDcalibration into a pass.