fix: fail closed when a filled-in visual field is out of range (#73) #74

2026-10-12T08:07:58Z

rockachopa commented

2026-10-12 08:07:58 +00:00

Closes #73

What changed

parseVisionResponse used to check for blank fields before validating them, so a single blank field short-circuited the whole reply. A response that filled in one field and left the other blank therefore abstained as unusable_visual_fields even when the filled-in field was out of range — telling the caller the photo was unreadable when the model had actually returned a value Timmy must never apply.

Decision taken: option 1 from the control-plane call recorded on the issue (2026-10-11T13:45Z). Blank-first abstention is kept; the range clause now fails closed only for non-blank fields.

A blank field is still the model saying it could not read it → abstains, 200, nothing prefilled, unusable_visual_fields.
A filled-in field is a claim → if it is out of range, the response fails closed even when the other field is blank.
A partial read that is entirely valid (e.g. Bristol 4 with a blank colour) still abstains. The glare path is unchanged.

Corpus earns its label

The issue found that no corpus fixture failed at the range check:

out-of-range-fields had confidence: 1.4, so it died at the confidence gate (line 33) and never reached the field-range guard, while carrying the out_of_range_or_missing_fields label. Confidence is now 0.9, so it reaches the range check it is named for.
New case blank-plus-out-of-range: bristolType: 9 with a null colour, confidence 0.9 → fail_closed / out_of_range_or_missing_fields. This is the exact reply shape the old ordering let through. Fixture is synthetic, drawn deterministically by scripts/build_vision_corpus.py; manifest regenerated (--verify passes over 8 fixtures, and every pre-existing fixture hash is byte-identical).

Evidence (real runs, this working tree at `5c75907`)

Command	Result
`node --test tests/analysis.test.js`	RED first: `not ok 7 - fails closed when a field the model did fill in is out of range while another field is blank` → `Missing expected exception`. Then 9/9 pass
`node --test tests/vision-corpus.test.js tests/vision-boundary.test.js tests/release-demo.test.js`	RED first: `corpus case blank-plus-out-of-range is missing`, and `at least one out-of-range case must clear the confidence gate`. Then 30/30 pass
`npm test`	126/126 pass (baseline before the change: 122/122)
`npm run test:vision-refusal`	PASS — browser scene 4 asserts the fail-closed copy, asserts `No confident match` count is 0, and requires exactly two 503s
`npm run test:ui \| test:photo \| test:mobile-capture \| test:sleek \| test:manual-fallback \| test:provisional \| test:followup`	all PASS
`node scripts/record_release_demo.mjs`	recorded end-to-end; caption receipt contains both new captions
`python3 tests/staging-deploy.test.py -v`	20 tests OK
`npm run check:syntax` / `check:diff` / `build_vision_corpus.py --verify`	PASS

Demonstration path

scripts/record_release_demo.mjs now replays blank-plus-out-of-range through the real analysis code path and asserts on camera that the app shows the fail-closed copy and not the abstention copy. Two captions:

A filled-in field that is out of range still fails closed, even when the other field came back blank
You are no longer told the photo was unreadable when the model returned an impossible value

The fail-closed 503 budget is raised from one to two so the added scene cannot be dropped silently, and the new caption is a required DEMO_REQUIRED_CAPTIONS gate in scripts/build_release.py. Release notes and feature_demo.fixture_data mention the new case. New visual evidence: artifacts/vision-refusal-blank-plus-out-of-range-mobile.png (follows the existing committed-screenshot convention).

Boundary

Unchanged: no diagnosis, no symptom inference from images, no provider detail or image bytes in a fail-closed body, no real medical imagery (all fixtures synthetic, CC0), no model weights, no infrastructure or production-service changes. A schema-valid suggestion is still the only 200-with-fields outcome.

Not merged — opened for review.

🤖 Generated with Hermes Agent (deepseek-flash)

Closes #73 ## What changed `parseVisionResponse` used to check for blank fields *before* validating them, so a single blank field short-circuited the whole reply. A response that filled in one field and left the other blank therefore abstained as `unusable_visual_fields` even when the filled-in field was out of range — telling the caller the photo was unreadable when the model had actually returned a value Timmy must never apply. **Decision taken: option 1** from the control-plane call recorded on the issue (2026-10-11T13:45Z). Blank-first abstention is kept; the range clause now fails closed only for non-blank fields. - A **blank** field is still the model saying it could not read it → abstains, 200, nothing prefilled, `unusable_visual_fields`. - A **filled-in** field is a claim → if it is out of range, the response fails closed even when the other field is blank. - A partial read that is entirely valid (e.g. Bristol 4 with a blank colour) still abstains. The glare path is unchanged. ## Corpus earns its label The issue found that **no corpus fixture failed at the range check**: - `out-of-range-fields` had `confidence: 1.4`, so it died at the confidence gate (line 33) and never reached the field-range guard, while carrying the `out_of_range_or_missing_fields` label. Confidence is now `0.9`, so it reaches the range check it is named for. - New case `blank-plus-out-of-range`: `bristolType: 9` with a `null` colour, confidence `0.9` → `fail_closed` / `out_of_range_or_missing_fields`. This is the exact reply shape the old ordering let through. Fixture is synthetic, drawn deterministically by `scripts/build_vision_corpus.py`; manifest regenerated (`--verify` passes over 8 fixtures, and every pre-existing fixture hash is byte-identical). ## Evidence (real runs, this working tree at `5c75907`) | Command | Result | |---|---| | `node --test tests/analysis.test.js` | RED first: `not ok 7 - fails closed when a field the model did fill in is out of range while another field is blank` → `Missing expected exception`. Then 9/9 pass | | `node --test tests/vision-corpus.test.js tests/vision-boundary.test.js tests/release-demo.test.js` | RED first: `corpus case blank-plus-out-of-range is missing`, and `at least one out-of-range case must clear the confidence gate`. Then 30/30 pass | | `npm test` | 126/126 pass (baseline before the change: 122/122) | | `npm run test:vision-refusal` | PASS — browser scene 4 asserts the fail-closed copy, asserts `No confident match` count is 0, and requires exactly two 503s | | `npm run test:ui \| test:photo \| test:mobile-capture \| test:sleek \| test:manual-fallback \| test:provisional \| test:followup` | all PASS | | `node scripts/record_release_demo.mjs` | recorded end-to-end; caption receipt contains both new captions | | `python3 tests/staging-deploy.test.py -v` | 20 tests OK | | `npm run check:syntax` / `check:diff` / `build_vision_corpus.py --verify` | PASS | ## Demonstration path `scripts/record_release_demo.mjs` now replays `blank-plus-out-of-range` through the real analysis code path and asserts on camera that the app shows the fail-closed copy and **not** the abstention copy. Two captions: - `A filled-in field that is out of range still fails closed, even when the other field came back blank` - `You are no longer told the photo was unreadable when the model returned an impossible value` The fail-closed 503 budget is raised from one to two so the added scene cannot be dropped silently, and the new caption is a required `DEMO_REQUIRED_CAPTIONS` gate in `scripts/build_release.py`. Release notes and `feature_demo.fixture_data` mention the new case. New visual evidence: `artifacts/vision-refusal-blank-plus-out-of-range-mobile.png` (follows the existing committed-screenshot convention). ## Boundary Unchanged: no diagnosis, no symptom inference from images, no provider detail or image bytes in a fail-closed body, no real medical imagery (all fixtures synthetic, CC0), no model weights, no infrastructure or production-service changes. A schema-valid suggestion is still the only 200-with-fields outcome. Not merged — opened for review. 🤖 Generated with Hermes Agent (deepseek-flash)

rockachopa added 1 commit 2026-10-12 08:07:58 +00:00

fix: fail closed when a filled-in visual field is out of range (#73 )

Quality gates / quality (pull_request) Failing after 14s

Details

5c759078c4

The blank-first ordering let one blank field short-circuit the whole reply, so a
response carrying an impossible value came back as `unusable_visual_fields`:
the user was told the photo was unreadable when the model had in fact returned
something Timmy must never apply.

Blank fields still abstain and still prefill nothing. A filled-in field is a
claim, and a claim outside the contract now fails closed even when the other
field came back blank.

The corpus earns its label too. `out-of-range-fields` drops from confidence 1.4
to 0.9 so the case reaches the field-range check instead of dying at the
confidence gate, and a new `blank-plus-out-of-range` case covers the class the
old ordering let through. The release demo replays that case on camera and the
fail-closed 503 budget moves from one to two, so the scene cannot be dropped
silently.

Closes #73

rockachopa closed this pull request

2026-10-12 08:10:17 +00:00

rockachopa reopened this pull request

2026-10-12 08:10:21 +00:00

rockachopa closed this pull request

2026-10-12 12:02:39 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: stackchain/timmy-talking-turd#74