test: prove schema conformance and refusal with a committed vision corpus (#18) #72

2026-10-11T08:15:17Z

rockachopa commented

2026-10-11 08:15:17 +00:00

Closes #18

What changed

The worker contract was only proven by one happy-path fixture plus a few inline assertions, so nothing continuously checked that a reply Timmy cannot use is refused rather than guessed at.

Committed corpus (tests/fixtures/vision-corpus/, CC0-1.0): stool, non-stool, blur, glare, empty bowl, and malformed-response cases, each a synthetic fixture drawn deterministically by scripts/build_vision_corpus.py. Every entry records its sha256, provenance, licence, case class, the provider reply it replays, and the outcome that reply must produce. loadVisionCorpus() refuses a fixture whose bytes no longer match the manifest, so the corpus cannot be quietly edited into a different test. No camera, no human subject, no real medical imagery, no third-party content.
Two contract fixes the corpus exposed:
1. A schema-valid reply with a null form and colour was being thrown as Invalid AI response: visual fields are out of range and surfaced to the user as a worker failure. The declared response schema permits those nulls, so it now abstains (abstainedFor: unusable_visual_fields, "No confident match") while still prefilling nothing. This is the visible delta for this release.
2. Fail-closed replies now carry a machine-readable code (invalid_json, invalid_fields) instead of prose matching, and visionFailureResponse() maps them to plain user-safe copy — "No suggestion was applied. Continue manually, or try a clearer photo." No provider text, image bytes, or credentials are echoed. server.mjs uses it, so unparsable and out-of-range replies return 503 with that copy.

Evidence

npm test → 122 tests, 122 pass, 0 fail (was 112; +10 covering the corpus, hash integrity, conformance, the abstention, and the boundary).
tests/vision-corpus.test.js — corpus contract, licence/provenance, hash + byte-length integrity, coverage of every required case class, and per-case outcome/reason conformance.
tests/vision-boundary.test.js — the corpus replayed through the real analyzePhoto path, then through a running server.mjs with a stub worker on a real socket: 200 for the only schema-valid suggestion, 200 abstention for unreadable fields, 503 with plain copy for unparsable and out-of-range replies, 400 without consent, and no base64/data:image/credential in any body.
npm run test:vision-refusal → browser acceptance against a real server: non-stool refusal, unreadable-fields abstention (this is the regression test for fix 1 — it fails before the fix), out-of-range fail-closed, plus a positive control that a usable reply still prefill. Screenshots in artifacts/vision-refusal-*.png and artifacts/vision-corpus-suggestion-mobile.png.
Other suites unchanged and re-run green: test:ui, test:photo, test:mobile-capture, test:sleek, test:manual-fallback, test:provisional, test:followup; python3 tests/staging-deploy.test.py → 20/20 OK; npm audit --audit-level=high → 0 vulnerabilities; npm run check:syntax; npm run check:diff.
python3 scripts/build_vision_corpus.py --verify → verified 7 fixtures against manifest.json.

Demonstration path

scripts/record_release_demo.mjs now replays three corpus cases through the real analysis code path (the recorded response is the status and body the server would actually send, not copy written for the video) and shows the refusal, the new abstention, and the fail-closed reply on camera, with honest captions and no real photos. The demo was run end to end against server.mjs and its on-camera caption receipt lists all 36 captions.

Gate wiring: package.json (test, test:vision-refusal, check:syntax), .gitea/workflows/quality.yml, scripts/build_release.py (corpus --verify, the refusal suite, a new required on-camera caption, vision_corpus_refusal: passed), with meta-tests in tests/ci-workflow.test.js and tests/release-demo.test.js so the wiring cannot silently drop.

Safety boundary

Unchanged and deliberately more conservative: the corpus adds no new claims, a refusal never carries a form or colour, and nothing diagnoses. Only visible form, colour, and image quality remain suggestible.

Not merged: awaiting review.

Closes #18 ## What changed The worker contract was only proven by one happy-path fixture plus a few inline assertions, so nothing continuously checked that a reply Timmy cannot use is refused rather than guessed at. - **Committed corpus** (`tests/fixtures/vision-corpus/`, CC0-1.0): stool, non-stool, blur, glare, empty bowl, and malformed-response cases, each a synthetic fixture drawn deterministically by `scripts/build_vision_corpus.py`. Every entry records its `sha256`, provenance, licence, case class, the provider reply it replays, and the outcome that reply must produce. `loadVisionCorpus()` refuses a fixture whose bytes no longer match the manifest, so the corpus cannot be quietly edited into a different test. No camera, no human subject, no real medical imagery, no third-party content. - **Two contract fixes the corpus exposed**: 1. A schema-valid reply with a null form and colour was being thrown as `Invalid AI response: visual fields are out of range` and surfaced to the user as a worker failure. The declared response schema permits those nulls, so it now abstains (`abstainedFor: unusable_visual_fields`, "No confident match") while still prefilling nothing. This is the visible delta for this release. 2. Fail-closed replies now carry a machine-readable code (`invalid_json`, `invalid_fields`) instead of prose matching, and `visionFailureResponse()` maps them to plain user-safe copy — "No suggestion was applied. Continue manually, or try a clearer photo." No provider text, image bytes, or credentials are echoed. `server.mjs` uses it, so unparsable and out-of-range replies return 503 with that copy. ## Evidence - `npm test` → **122 tests, 122 pass, 0 fail** (was 112; +10 covering the corpus, hash integrity, conformance, the abstention, and the boundary). - `tests/vision-corpus.test.js` — corpus contract, licence/provenance, hash + byte-length integrity, coverage of every required case class, and per-case outcome/reason conformance. - `tests/vision-boundary.test.js` — the corpus replayed through the real `analyzePhoto` path, then through a **running `server.mjs`** with a stub worker on a real socket: 200 for the only schema-valid suggestion, 200 abstention for unreadable fields, 503 with plain copy for unparsable and out-of-range replies, 400 without consent, and no `base64`/`data:image`/credential in any body. - `npm run test:vision-refusal` → browser acceptance against a real server: non-stool refusal, unreadable-fields abstention (this is the regression test for fix 1 — it fails before the fix), out-of-range fail-closed, plus a positive control that a usable reply still prefill. Screenshots in `artifacts/vision-refusal-*.png` and `artifacts/vision-corpus-suggestion-mobile.png`. - Other suites unchanged and re-run green: `test:ui`, `test:photo`, `test:mobile-capture`, `test:sleek`, `test:manual-fallback`, `test:provisional`, `test:followup`; `python3 tests/staging-deploy.test.py` → 20/20 OK; `npm audit --audit-level=high` → 0 vulnerabilities; `npm run check:syntax`; `npm run check:diff`. - `python3 scripts/build_vision_corpus.py --verify` → verified 7 fixtures against `manifest.json`. ## Demonstration path `scripts/record_release_demo.mjs` now replays three corpus cases through the **real analysis code path** (the recorded response is the status and body the server would actually send, not copy written for the video) and shows the refusal, the new abstention, and the fail-closed reply on camera, with honest captions and no real photos. The demo was run end to end against `server.mjs` and its on-camera caption receipt lists all 36 captions. Gate wiring: `package.json` (`test`, `test:vision-refusal`, `check:syntax`), `.gitea/workflows/quality.yml`, `scripts/build_release.py` (corpus `--verify`, the refusal suite, a new required on-camera caption, `vision_corpus_refusal: passed`), with meta-tests in `tests/ci-workflow.test.js` and `tests/release-demo.test.js` so the wiring cannot silently drop. ## Safety boundary Unchanged and deliberately more conservative: the corpus adds no new claims, a refusal never carries a form or colour, and nothing diagnoses. Only visible form, colour, and image quality remain suggestible. Not merged: awaiting review.

rockachopa added 1 commit 2026-10-11 08:15:17 +00:00

test: prove schema conformance and refusal with a committed vision corpus (#18 )

Quality gates / quality (pull_request) Successful in 2m26s

Details

d9f82299d0

The worker contract was only proven by one happy-path fixture and a handful of
inline assertions, so nothing continuously checked that a reply Timmy cannot use
is refused instead of guessed at.

Corpus: a CC0 synthetic corpus (stool, non-stool, blur, glare, empty bowl,
malformed response) generated deterministically by scripts/build_vision_corpus.py.
Every fixture records its sha256, provenance, licence, case class, the provider
reply it replays, and the outcome that reply must produce. loadVisionCorpus
refuses a fixture whose bytes no longer match the manifest, so the corpus cannot
be edited into a different test. No real medical imagery, camera, or third-party
content is involved.

Contract fixes the corpus exposed:
- A schema-valid reply with a null form and colour is now an abstention
  (abstainedFor: unusable_visual_fields) instead of a hard provider failure. The
  declared response schema permits those nulls, so treating the reply as an error
  was wrong; it now abstains calmly and still prefill nothing.
- Fail-closed replies carry a machine-readable code (invalid_json, invalid_fields)
  instead of prose matching, and visionFailureResponse() maps them to plain,
  user-safe copy with no provider text, image bytes, or credentials echoed. A
  running server now returns that copy for unparsable and out-of-range replies.

Evidence: unit corpus contract and hash integrity, corpus-wide conformance
through the real analysis path, the same corpus through a running server.mjs with
a stub worker (200 abstain vs 503 fail-closed, no leaked bytes), and a real
browser acceptance run for the refusal UI. The release demo replays three corpus
cases through the real code path so the video shows the status the server would
actually send, with the on-camera caption gated by the release builder.

rockachopa closed this pull request

2026-10-11 12:02:51 +00:00

rockachopa referenced this pull request

2026-10-11 12:28:07 +00:00

analysis.js: the blank-first ordering routes an out-of-range reply to abstain, against its own comment and the corpus's fail_closed label #73

rockachopa referenced this pull request

2026-10-11 13:15:58 +00:00

Build schema-conformance and real-stool refusal regression corpus #18