timmy
209e21a951
Quality gates / quality (pull_request) Successful in 13m29s
Confidence was the only thing standing between a model answer and a prefilled journal entry, so a confident reading of a blurry photo could still be offered as fact. Quality is now part of the policy: `poor` abstains however confident the reply looked, with its own fixed copy. The confidence floor, accepted quality levels, and refusal copy move into src/abstention-policy.js so the parser, the corpus, and the browser cannot drift into three explanations of the same refusal. parseVisionResponse takes an optional policy, which is how the tests prove the parser carries no hidden limits of its own. evaluatePrefillGate turns the #29 calibration report into one release decision: GO plus a validation-selected threshold scored on the held-out test split, or prefill is blocked with each reason named. A missing report is a block, never a default allow. scripts/abstention_policy_receipt.mjs exposes that as a receipt (exit 0 / 3 / 2). The committed synthetic predictions currently report BLOCKED and the receipt says so. No candidate exists until #31, and the builder does not hard-gate on the receipt yet; the on-camera quality gate is the evidence this release carries. The corpus gains a case that replays a 0.94-confidence answer about the synthetic motion-blur fixture, and the release demo asserts in the browser that nothing is prefilled before captioning it. Verified: 226 unit tests, calibration harness (19), corpus manifest (8), vision-refusal, photo-first, and provisional acceptance, check:syntax, check:diff, and a full real-browser demo recording whose caption receipt confirms the new scene painted on screen.
Vision corpus (issue #18)
A committed, deterministic corpus that proves one claim on every run: every reply Timmy receives is either a schema-valid visible-field suggestion or it fails closed. Nothing is guessed, and a refusal never carries a form or a colour.
Licence and provenance
- All fixtures are licensed CC0-1.0.
- Every fixture is a synthetic image drawn from fixed integer math by
scripts/build_vision_corpus.py. There is no camera, no human subject, no third-party content, and no real medical imagery anywhere in this directory. manifest.jsonrecords each fixture'ssha256, byte length, provenance, licence, case class, the provider reply it replays, and the outcome that reply must produce.- The hashes are an integrity receipt:
loadVisionCorpus()refuses a fixture whose bytes no longer match the manifest, so the corpus cannot be quietly edited into a different test.
Coverage
| Case | Class | Replayed provider reply | Required outcome |
|---|---|---|---|
stool-like-formed |
stool | valid suggestion, confidence 0.83 | suggestion |
non-stool-scene |
non-stool | isStool: false |
abstained (not_stool) |
blur-low-detail |
blur | confidence 0.41 | abstained (low_confidence) |
glare-overexposed |
glare | schema-valid with null form and colour | abstained (unusable_visual_fields) |
empty-bowl |
empty-bowl | isStool: false |
abstained (not_stool) |
malformed-response |
malformed-response | plain prose, not JSON | fail_closed (unparsable_content) |
out-of-range-fields |
malformed-response | Bristol 9, colour purple, confidence 0.9 |
fail_closed (out_of_range_or_missing_fields) |
blank-plus-out-of-range |
malformed-response | Bristol 9 with a null colour, confidence 0.9 |
fail_closed (out_of_range_or_missing_fields) |
Both out-of-range cases keep confidence inside the contract on purpose: at 1.4 the reply died at the
confidence gate, so the out_of_range_or_missing_fields label was credited to a guard the case never
reached. blank-plus-out-of-range covers the reply shape that the blank-first ordering used to abstain on
(issue #73) — a filled-in field is a claim, and a claim outside the contract fails closed even when the
other field came back blank.
Where the corpus is enforced
tests/vision-corpus.test.js— corpus contract, hash integrity, and conformance of every reply.tests/vision-boundary.test.js— the corpus replayed through the real provider call path and through a runningserver.mjs, asserting status codes and that no image bytes or credentials appear in a fail-closed body.tests/vision-refusal.acceptance.mjs— the same refusals in a real browser.scripts/record_release_demo.mjs— the release demo replays corpus cases through the real analysis code path, so the video shows the status the server would actually send.
Regenerating
python3 scripts/build_vision_corpus.py # rewrite fixtures and manifest.json
python3 scripts/build_vision_corpus.py --verify # verify committed fixtures against the manifest
The release builder runs --verify before anything is packaged. Regenerate deliberately and commit
the new hashes; never hand-edit a fixture or a recorded reply.