timmy
5c759078c4
The blank-first ordering let one blank field short-circuit the whole reply, so a response carrying an impossible value came back as `unusable_visual_fields`: the user was told the photo was unreadable when the model had in fact returned something Timmy must never apply. Blank fields still abstain and still prefill nothing. A filled-in field is a claim, and a claim outside the contract now fails closed even when the other field came back blank. The corpus earns its label too. `out-of-range-fields` drops from confidence 1.4 to 0.9 so the case reaches the field-range check instead of dying at the confidence gate, and a new `blank-plus-out-of-range` case covers the class the old ordering let through. The release demo replays that case on camera and the fail-closed 503 budget moves from one to two, so the scene cannot be dropped silently. Closes #73
3.3 KiB
Vision corpus (issue #18)
A committed, deterministic corpus that proves one claim on every run: every reply Timmy receives is either a schema-valid visible-field suggestion or it fails closed. Nothing is guessed, and a refusal never carries a form or a colour.
Licence and provenance
- All fixtures are licensed CC0-1.0.
- Every fixture is a synthetic image drawn from fixed integer math by
scripts/build_vision_corpus.py. There is no camera, no human subject, no third-party content, and no real medical imagery anywhere in this directory. manifest.jsonrecords each fixture'ssha256, byte length, provenance, licence, case class, the provider reply it replays, and the outcome that reply must produce.- The hashes are an integrity receipt:
loadVisionCorpus()refuses a fixture whose bytes no longer match the manifest, so the corpus cannot be quietly edited into a different test.
Coverage
| Case | Class | Replayed provider reply | Required outcome |
|---|---|---|---|
stool-like-formed |
stool | valid suggestion, confidence 0.83 | suggestion |
non-stool-scene |
non-stool | isStool: false |
abstained (not_stool) |
blur-low-detail |
blur | confidence 0.41 | abstained (low_confidence) |
glare-overexposed |
glare | schema-valid with null form and colour | abstained (unusable_visual_fields) |
empty-bowl |
empty-bowl | isStool: false |
abstained (not_stool) |
malformed-response |
malformed-response | plain prose, not JSON | fail_closed (unparsable_content) |
out-of-range-fields |
malformed-response | Bristol 9, colour purple, confidence 0.9 |
fail_closed (out_of_range_or_missing_fields) |
blank-plus-out-of-range |
malformed-response | Bristol 9 with a null colour, confidence 0.9 |
fail_closed (out_of_range_or_missing_fields) |
Both out-of-range cases keep confidence inside the contract on purpose: at 1.4 the reply died at the
confidence gate, so the out_of_range_or_missing_fields label was credited to a guard the case never
reached. blank-plus-out-of-range covers the reply shape that the blank-first ordering used to abstain on
(issue #73) — a filled-in field is a claim, and a claim outside the contract fails closed even when the
other field came back blank.
Where the corpus is enforced
tests/vision-corpus.test.js— corpus contract, hash integrity, and conformance of every reply.tests/vision-boundary.test.js— the corpus replayed through the real provider call path and through a runningserver.mjs, asserting status codes and that no image bytes or credentials appear in a fail-closed body.tests/vision-refusal.acceptance.mjs— the same refusals in a real browser.scripts/record_release_demo.mjs— the release demo replays corpus cases through the real analysis code path, so the video shows the status the server would actually send.
Regenerating
python3 scripts/build_vision_corpus.py # rewrite fixtures and manifest.json
python3 scripts/build_vision_corpus.py --verify # verify committed fixtures against the manifest
The release builder runs --verify before anything is packaged. Regenerate deliberately and commit
the new hashes; never hand-edit a fixture or a recorded reply.