Gate visual prefill on a versioned abstention policy (#32) #87
No files matched your search
@@ -110,7 +110,7 @@ npm audit --audit-level=high
|
|||||||
- Client-side image compression before transfer
|
- Client-side image compression before transfer
|
||||||
- Explicit consent before one-time provider analysis
|
- Explicit consent before one-time provider analysis
|
||||||
- Structured AI suggestions for **visible Bristol form and color only**
|
- Structured AI suggestions for **visible Bristol form and color only**
|
||||||
- Confidence display, low-confidence abstention, and user confirmation
|
- Confidence display, low-confidence abstention, and user confirmation, governed by one versioned policy: `poor` image quality abstains even at high confidence, and a confident reply is never enough on its own — see [docs/ABSTENTION-POLICY.md](docs/ABSTENTION-POLICY.md)
|
||||||
- Urgency, discomfort, notes, and symptoms remain strictly user-reported
|
- Urgency, discomfort, notes, and symptoms remain strictly user-reported
|
||||||
- Manual logging that never uploads
|
- Manual logging that never uploads
|
||||||
- Sleek three-destination mobile shell with one dominant photo action
|
- Sleek three-destination mobile shell with one dominant photo action
|
||||||
|
|||||||
@@ -0,0 +1,98 @@
|
|||||||
|
# Abstention policy
|
||||||
|
|
||||||
|
Issue #32 asks for raw model scores to be turned into a conservative suggestion policy. This is that
|
||||||
|
policy. It is the only thing that decides whether Timmy offers a visual prefill, and it answers two
|
||||||
|
different questions that are easy to confuse:
|
||||||
|
|
||||||
|
1. **Per photo** — is this individual answer good enough to show?
|
||||||
|
2. **Per release** — has a candidate model earned the right to prefill at all?
|
||||||
|
|
||||||
|
Both are implemented in [`src/abstention-policy.js`](../src/abstention-policy.js). The parser in
|
||||||
|
[`src/analysis.js`](../src/analysis.js) reads its limits from that module rather than carrying its own
|
||||||
|
copy, so a policy change is one edit, not three.
|
||||||
|
|
||||||
|
A suggestion is never a diagnosis, never fills in urgency, discomfort, symptoms, or notes, and is
|
||||||
|
never saved until the user confirms it. Abstention is the default; a prefill is the exception.
|
||||||
|
|
||||||
|
## Why quality is a gate, not a footnote
|
||||||
|
|
||||||
|
`imageQuality` is something the model reports about its own photo. A confident form and colour read
|
||||||
|
from a blurry photo is exactly the answer a user is most likely to accept without thinking, so
|
||||||
|
confidence alone cannot authorise a prefill. The policy accepts `good` and `fair`; `poor` abstains
|
||||||
|
regardless of how confident the reply looked. This is proven end to end by the committed corpus case
|
||||||
|
`poor-quality-confident`, which replays a 0.94-confidence answer about the synthetic motion-blur
|
||||||
|
fixture and expects an abstention.
|
||||||
|
|
||||||
|
## Per-photo gates
|
||||||
|
|
||||||
|
Evaluated in this order. The first gate that fails names the refusal, so the user is told the real
|
||||||
|
reason rather than a generic one.
|
||||||
|
|
||||||
|
| Order | Gate | Refusal (`abstainedFor`) | User-facing copy |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | Model says the photo is not clearly stool | `not_stool` | The image does not clearly show stool. |
|
||||||
|
| 2 | Confidence below `minConfidence` (0.55) | `low_confidence` | The image is too uncertain to prefill safely. |
|
||||||
|
| 3 | Form or colour came back blank | `unusable_visual_fields` | The photo does not show a form Timmy can read safely. |
|
||||||
|
| 4 | `imageQuality` is below the accepted levels | `low_quality` | The photo is too blurry or poorly lit to suggest a form safely. |
|
||||||
|
|
||||||
|
A filled-in field outside the contract (`bristolType` outside 1–7, an unknown `color`, an unknown
|
||||||
|
`imageQuality`) is not an abstention — it **fails closed** with no provider detail shown, because a
|
||||||
|
caller must never be told "the photo was unreadable" when the model actually returned an impossible
|
||||||
|
value. That ordering is covered by the corpus escape cases (`out-of-range-fields`,
|
||||||
|
`blank-plus-out-of-range`).
|
||||||
|
|
||||||
|
Unknown and out-of-distribution images can only be judged from what the schema lets the model say: it
|
||||||
|
must declare `isStool`, a confidence, and a quality. There is no separate OOD channel, and this policy
|
||||||
|
does not invent one. A photo the model cannot place therefore lands on `not_stool`, `low_confidence`,
|
||||||
|
or `low_quality` — all three abstain.
|
||||||
|
|
||||||
|
## Per-release gate
|
||||||
|
|
||||||
|
The calibration harness ([`scripts/calibration_report.py`](../scripts/calibration_report.py), issue
|
||||||
|
#29) scores candidates on a held-out test split with a threshold selected on validation only. The
|
||||||
|
policy turns that report into a single decision with `evaluatePrefillGate(report)`:
|
||||||
|
|
||||||
|
| Requirement | Blocked reason if unmet |
|
||||||
|
|---|---|
|
||||||
|
| Report verdict is `GO` | `calibration-verdict-not-go` |
|
||||||
|
| The report names a recommended candidate | `no-recommended-calibration` |
|
||||||
|
| That candidate's threshold was selected on `validation` | `threshold-not-selected-on-validation` |
|
||||||
|
| That candidate was scored on the held-out `test` split | `gate-not-scored-on-held-out-test` |
|
||||||
|
| That candidate passed every release gate | `candidate-failed-release-gates` |
|
||||||
|
|
||||||
|
A missing or unreadable report is a block, never a default allow.
|
||||||
|
|
||||||
|
Run it as a receipt:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/calibration_report.py \
|
||||||
|
--predictions tests/fixtures/calibration/predictions.jsonl \
|
||||||
|
--min-coverage 0.5 --json /tmp/calibration-report.json
|
||||||
|
|
||||||
|
node scripts/abstention_policy_receipt.mjs --report /tmp/calibration-report.json
|
||||||
|
# exit 0 = prefill authorised, 3 = blocked, 2 = the report could not be read
|
||||||
|
```
|
||||||
|
|
||||||
|
**Current state:** the committed synthetic predictions produce `BLOCKED`, so the receipt exits 3 and
|
||||||
|
names why. No production candidate exists yet — issue #31 (fine-tuned classifier) is the dependency —
|
||||||
|
and the policy exists so that candidate is held to one stated contract instead of whatever it happens
|
||||||
|
to score. The release builder does not gate on this receipt yet; it records the quality gate on camera
|
||||||
|
through the demo instead. Wiring the receipt into `scripts/build_release.py` as a hard prefill gate
|
||||||
|
belongs with #31, when there is a candidate to judge.
|
||||||
|
|
||||||
|
## Versioning
|
||||||
|
|
||||||
|
`ABSTENTION_POLICY.version` is an integer that changes whenever a limit changes. A policy change is a
|
||||||
|
behaviour change: the release notes must be able to say which policy produced a suggestion. The
|
||||||
|
shipped object is frozen so nothing can tighten or loosen it at runtime; callers that need a
|
||||||
|
different policy pass one explicitly (`parseVisionResponse(raw, policy)`), which is how the tests
|
||||||
|
prove the parser has no hidden limits of its own.
|
||||||
|
|
||||||
|
## What this policy does not do
|
||||||
|
|
||||||
|
- It does not decide the Bristol type, the colour, or the quality. It only decides whether to show
|
||||||
|
them.
|
||||||
|
- It does not touch deterministic red-flag escalation, which runs independently of model state.
|
||||||
|
- It does not read, store, or log an image, and it never inspects inference bodies.
|
||||||
|
- It does not set the numeric thresholds from anything but validation data, and it does not convert a
|
||||||
|
`BLOCKED` calibration into a pass.
|
||||||
@@ -7,6 +7,10 @@ splits that are contributor-held-out and reproducible, comparing candidate model
|
|||||||
The implementation is [`scripts/calibration_report.py`](../scripts/calibration_report.py). It reads
|
The implementation is [`scripts/calibration_report.py`](../scripts/calibration_report.py). It reads
|
||||||
predictions something else produced, and it refuses to compute gates it cannot honestly compute.
|
predictions something else produced, and it refuses to compute gates it cannot honestly compute.
|
||||||
|
|
||||||
|
The GO/BLOCKED verdict this report produces is consumed by the production abstention policy in
|
||||||
|
[`docs/ABSTENTION-POLICY.md`](ABSTENTION-POLICY.md): a candidate that has not cleared these gates
|
||||||
|
cannot authorise a visual prefill.
|
||||||
|
|
||||||
**This harness does not train, does not diagnose, and never reads an image.** It only sees Bristol
|
**This harness does not train, does not diagnose, and never reads an image.** It only sees Bristol
|
||||||
labels, predictions, and confidences.
|
labels, predictions, and confidences.
|
||||||
|
|
||||||
|
|||||||
+4
-3
@@ -4,7 +4,7 @@
|
|||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/draft.test.js tests/suggestion.test.js tests/followup.test.js tests/vision-readiness.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js tests/threat-model.test.js tests/vision-corpus.test.js tests/vision-boundary.test.js tests/worker-benchmark.test.js tests/worker-ready.test.js tests/worker-container.test.js tests/summary.test.js tests/escalation-regression.test.js tests/grounded-chat.test.js tests/photo-lifecycle.test.js tests/training-contribution.test.js tests/portability.test.js",
|
"test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/draft.test.js tests/suggestion.test.js tests/followup.test.js tests/vision-readiness.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js tests/threat-model.test.js tests/vision-corpus.test.js tests/vision-boundary.test.js tests/worker-benchmark.test.js tests/worker-ready.test.js tests/worker-container.test.js tests/summary.test.js tests/escalation-regression.test.js tests/grounded-chat.test.js tests/photo-lifecycle.test.js tests/training-contribution.test.js tests/portability.test.js tests/abstention-policy.test.js",
|
||||||
"test:ui": "node tests/ui.acceptance.mjs",
|
"test:ui": "node tests/ui.acceptance.mjs",
|
||||||
"test:photo": "node tests/photo-first.acceptance.mjs",
|
"test:photo": "node tests/photo-first.acceptance.mjs",
|
||||||
"test:mobile-capture": "node tests/mobile-capture.acceptance.mjs",
|
"test:mobile-capture": "node tests/mobile-capture.acceptance.mjs",
|
||||||
@@ -18,13 +18,14 @@
|
|||||||
"test:grounded-chat": "node tests/grounded-chat.acceptance.mjs",
|
"test:grounded-chat": "node tests/grounded-chat.acceptance.mjs",
|
||||||
"test:photo-lifecycle": "node tests/photo-lifecycle.acceptance.mjs",
|
"test:photo-lifecycle": "node tests/photo-lifecycle.acceptance.mjs",
|
||||||
"test:training-contribution": "node tests/training-contribution.acceptance.mjs",
|
"test:training-contribution": "node tests/training-contribution.acceptance.mjs",
|
||||||
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py && node --check scripts/record_worker_benchmark.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh",
|
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh",
|
||||||
"check:diff": "bash scripts/check_diff.sh",
|
"check:diff": "bash scripts/check_diff.sh",
|
||||||
"start": "node server.mjs",
|
"start": "node server.mjs",
|
||||||
"test:worker-readiness": "node tests/worker-readiness.acceptance.mjs",
|
"test:worker-readiness": "node tests/worker-readiness.acceptance.mjs",
|
||||||
"test:portability": "node tests/portability.acceptance.mjs",
|
"test:portability": "node tests/portability.acceptance.mjs",
|
||||||
"test:corpus": "python3 tests/corpus-manifest.test.py",
|
"test:corpus": "python3 tests/corpus-manifest.test.py",
|
||||||
"test:calibration": "python3 tests/calibration-harness.test.py"
|
"test:calibration": "python3 tests/calibration-harness.test.py",
|
||||||
|
"test:policy": "node --test tests/abstention-policy.test.js"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"playwright": "^1.62.1"
|
"playwright": "^1.62.1"
|
||||||
|
|||||||
@@ -0,0 +1,49 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
// Executable receipt for the production abstention policy (issue #32).
|
||||||
|
//
|
||||||
|
// The calibration harness decides GO/BLOCKED for a candidate model. This reads that report and
|
||||||
|
// answers the only question the app cares about: may a suggestion be shipped at all? It exits 3 when
|
||||||
|
// prefill is blocked, so a release step can gate on it instead of reading prose.
|
||||||
|
//
|
||||||
|
// There is no model candidate yet (#31), so in practice this receipt reports BLOCKED today. That is
|
||||||
|
// the honest state: the policy is ready, the evidence is not.
|
||||||
|
import { readFile } from 'node:fs/promises';
|
||||||
|
import { ABSTENTION_POLICY, evaluatePrefillGate } from '../src/abstention-policy.js';
|
||||||
|
|
||||||
|
function parseArgs(argv) {
|
||||||
|
const args = { report: '' };
|
||||||
|
for (let i = 0; i < argv.length; i += 1) {
|
||||||
|
if (argv[i] === '--report') {
|
||||||
|
args.report = argv[i + 1] || '';
|
||||||
|
i += 1;
|
||||||
|
} else throw new Error(`unknown argument: ${argv[i]}`);
|
||||||
|
}
|
||||||
|
if (!args.report) throw new Error('--report <calibration-report.json> is required');
|
||||||
|
return args;
|
||||||
|
}
|
||||||
|
|
||||||
|
let args;
|
||||||
|
try {
|
||||||
|
args = parseArgs(process.argv.slice(2));
|
||||||
|
} catch (error) {
|
||||||
|
console.error(error.message);
|
||||||
|
process.exit(2);
|
||||||
|
}
|
||||||
|
|
||||||
|
let report;
|
||||||
|
try {
|
||||||
|
report = JSON.parse(await readFile(args.report, 'utf8'));
|
||||||
|
} catch (error) {
|
||||||
|
console.error(`could not read the calibration report at ${args.report}: ${error.message}`);
|
||||||
|
process.exit(2);
|
||||||
|
}
|
||||||
|
|
||||||
|
const gate = evaluatePrefillGate(report);
|
||||||
|
const policy = ABSTENTION_POLICY;
|
||||||
|
console.log('TIMMY ABSTENTION POLICY RECEIPT');
|
||||||
|
console.log(`policy: version ${policy.version} · confidence ≥ ${policy.minConfidence.toFixed(2)} · quality ${policy.acceptedImageQuality.join('|')}`);
|
||||||
|
console.log(`calibration: verdict ${gate.verdict} · model ${gate.calibrationModel || 'none'}`);
|
||||||
|
console.log(gate.prefillAllowed
|
||||||
|
? `prefill: AUTHORISED for ${gate.calibrationModel}`
|
||||||
|
: `prefill: BLOCKED — ${gate.blockedBy.join(', ')}`);
|
||||||
|
process.exit(gate.prefillAllowed ? 0 : 3);
|
||||||
@@ -46,6 +46,7 @@ DEMO_REQUIRED_CAPTIONS = (
|
|||||||
"A tarry note already saved in the journal escalates the same way, before any provider call",
|
"A tarry note already saved in the journal escalates the same way, before any provider call",
|
||||||
"An out-of-range answer fails closed: nothing is applied and no provider detail reaches you",
|
"An out-of-range answer fails closed: nothing is applied and no provider detail reaches you",
|
||||||
"A filled-in field that is out of range still fails closed, even when the other field came back blank",
|
"A filled-in field that is out of range still fails closed, even when the other field came back blank",
|
||||||
|
"A confident reply about a blurry photo still abstains — photo quality is part of the policy, not just confidence",
|
||||||
"Reachable is not loaded: no AI control, no consent, and no upload until the pinned model is resident",
|
"Reachable is not loaded: no AI control, no consent, and no upload until the pinned model is resident",
|
||||||
"Every pattern line names the confirmed logs behind it, and following one shows exactly those records",
|
"Every pattern line names the confirmed logs behind it, and following one shows exactly those records",
|
||||||
"Two logs is not a pattern: the summary refuses to describe a trend it cannot support",
|
"Two logs is not a pattern: the summary refuses to describe a trend it cannot support",
|
||||||
@@ -340,7 +341,7 @@ def main() -> int:
|
|||||||
"codec": "h264",
|
"codec": "h264",
|
||||||
"pixel_format": "yuv420p",
|
"pixel_format": "yuv420p",
|
||||||
"recorded_from_working_app": True,
|
"recorded_from_working_app": True,
|
||||||
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything",
|
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything",
|
||||||
},
|
},
|
||||||
"gates": {
|
"gates": {
|
||||||
"unit_security": "passed",
|
"unit_security": "passed",
|
||||||
@@ -351,6 +352,7 @@ def main() -> int:
|
|||||||
"provisional_suggestion_state": "passed",
|
"provisional_suggestion_state": "passed",
|
||||||
"hard_stool_followup": "passed",
|
"hard_stool_followup": "passed",
|
||||||
"vision_corpus_refusal": "passed",
|
"vision_corpus_refusal": "passed",
|
||||||
|
"abstention_quality_gate": "passed",
|
||||||
"worker_container_receipts": "passed",
|
"worker_container_receipts": "passed",
|
||||||
"worker_post_load_readiness": "passed",
|
"worker_post_load_readiness": "passed",
|
||||||
"longitudinal_pattern_summary": "passed",
|
"longitudinal_pattern_summary": "passed",
|
||||||
|
|||||||
@@ -341,6 +341,20 @@ await tap('#analyze-photo', 600);
|
|||||||
await page.getByText(/does not show a form Timmy can read safely/i).waitFor();
|
await page.getByText(/does not show a form Timmy can read safely/i).waitFor();
|
||||||
await caption('Unreadable visual fields abstain as well: a valid reply with nothing to prefill is not an error', 2700);
|
await caption('Unreadable visual fields abstain as well: a valid reply with nothing to prefill is not an error', 2700);
|
||||||
await indicate('.scan-result');
|
await indicate('.scan-result');
|
||||||
|
// Issue #32's quality gate: the model can be highly confident about a photo it also calls poor, and
|
||||||
|
// that confident answer must still abstain. Replayed from the committed corpus, and asserted in the
|
||||||
|
// browser so the scene is a check rather than a claim.
|
||||||
|
analyzeOverride = await corpusAnalyzeResult(corpusCase('poor-quality-confident'));
|
||||||
|
await tap('#retake-photo', 400);
|
||||||
|
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/vision-corpus/blur-low-detail.png');
|
||||||
|
await page.locator('#ai-consent').check();
|
||||||
|
await tap('#analyze-photo', 600);
|
||||||
|
await page.getByText(/too blurry or poorly lit/i).waitFor();
|
||||||
|
if (await page.getByText(/BRISTOL FORM/i).count() !== 0) {
|
||||||
|
throw new Error('A low-quality photo must not prefill a form, however confident the reply looked');
|
||||||
|
}
|
||||||
|
await caption('A confident reply about a blurry photo still abstains — photo quality is part of the policy, not just confidence', 3100);
|
||||||
|
await indicate('.scan-result');
|
||||||
analyzeOverride = await corpusAnalyzeResult(corpusCase('out-of-range-fields'));
|
analyzeOverride = await corpusAnalyzeResult(corpusCase('out-of-range-fields'));
|
||||||
await tap('#retake-photo', 400);
|
await tap('#retake-photo', 400);
|
||||||
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/vision-corpus/out-of-range-fields.png');
|
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/vision-corpus/out-of-range-fields.png');
|
||||||
|
|||||||
@@ -0,0 +1,69 @@
|
|||||||
|
// The production abstention policy (issue #32).
|
||||||
|
//
|
||||||
|
// A suggestion is only ever a visual prefill, and this module is the single place that decides when
|
||||||
|
// Timmy is allowed to offer one. Two different questions live here, and they are deliberately
|
||||||
|
// separate:
|
||||||
|
//
|
||||||
|
// 1. Per-photo: is this individual answer good enough to show? Confidence, image quality, and
|
||||||
|
// whether the model actually read a form and a colour.
|
||||||
|
// 2. Per-release: has a candidate cleared the calibration gates on a held-out split, with its
|
||||||
|
// threshold chosen on validation only? Until it has, no prefill may be shipped at all.
|
||||||
|
//
|
||||||
|
// The copy is fixed here so the browser, the corpus, and the server cannot drift into three
|
||||||
|
// different explanations of the same refusal.
|
||||||
|
export const ABSTENTION_COPY = Object.freeze({
|
||||||
|
not_stool: 'The image does not clearly show stool.',
|
||||||
|
low_confidence: 'The image is too uncertain to prefill safely.',
|
||||||
|
unusable_visual_fields: 'The photo does not show a form Timmy can read safely.',
|
||||||
|
low_quality: 'The photo is too blurry or poorly lit to suggest a form safely.',
|
||||||
|
});
|
||||||
|
|
||||||
|
// Bump `version` whenever a limit below changes: a policy change is a behaviour change, and the
|
||||||
|
// release notes have to be able to name which policy a suggestion was produced under.
|
||||||
|
export const ABSTENTION_POLICY = Object.freeze({
|
||||||
|
version: 1,
|
||||||
|
minConfidence: 0.55,
|
||||||
|
// "fair" may prefill; "poor" never does. The model already returns its own quality judgement, so
|
||||||
|
// a confident answer about an unreadable photo is exactly the case this gate exists for.
|
||||||
|
acceptedImageQuality: Object.freeze(['good', 'fair']),
|
||||||
|
});
|
||||||
|
|
||||||
|
export function abstainFor(reason, isStool = true) {
|
||||||
|
if (!Object.prototype.hasOwnProperty.call(ABSTENTION_COPY, reason)) {
|
||||||
|
throw new Error(`unknown abstention reason: ${reason}`);
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
status: 'needs_user_input',
|
||||||
|
isStool,
|
||||||
|
abstainedFor: reason,
|
||||||
|
reason: ABSTENTION_COPY[reason],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function isLowQuality(imageQuality, policy = ABSTENTION_POLICY) {
|
||||||
|
return !policy.acceptedImageQuality.includes(String(imageQuality || '').toLowerCase());
|
||||||
|
}
|
||||||
|
|
||||||
|
// Release-level gate. The calibration harness already refuses to pick a threshold anywhere but the
|
||||||
|
// validation split; this decides whether a report authorises prefill at all, and names every reason
|
||||||
|
// it does not. A missing or malformed report is a block, never a default-allow.
|
||||||
|
export function evaluatePrefillGate(report) {
|
||||||
|
const blockedBy = [];
|
||||||
|
const verdict = report?.verdict || 'UNKNOWN';
|
||||||
|
if (verdict !== 'GO') blockedBy.push('calibration-verdict-not-go');
|
||||||
|
const models = Array.isArray(report?.models) ? report.models : [];
|
||||||
|
const recommended = models.find(row => row?.model === report?.recommended) || null;
|
||||||
|
if (!recommended) {
|
||||||
|
blockedBy.push('no-recommended-calibration');
|
||||||
|
} else {
|
||||||
|
if (recommended.selectionSplit !== 'validation') blockedBy.push('threshold-not-selected-on-validation');
|
||||||
|
if (recommended.evaluatedSplit !== 'test') blockedBy.push('gate-not-scored-on-held-out-test');
|
||||||
|
if (recommended.gate?.passed !== true) blockedBy.push('candidate-failed-release-gates');
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
prefillAllowed: blockedBy.length === 0,
|
||||||
|
verdict,
|
||||||
|
calibrationModel: recommended?.model || null,
|
||||||
|
blockedBy,
|
||||||
|
};
|
||||||
|
}
|
||||||
+8
-9
@@ -1,3 +1,5 @@
|
|||||||
|
import { ABSTENTION_POLICY, abstainFor, isLowQuality } from './abstention-policy.js';
|
||||||
|
|
||||||
const COLORS = new Set(['brown', 'green', 'yellow', 'pale', 'red', 'black']);
|
const COLORS = new Set(['brown', 'green', 'yellow', 'pale', 'red', 'black']);
|
||||||
const QUALITIES = new Set(['good', 'fair', 'poor']);
|
const QUALITIES = new Set(['good', 'fair', 'poor']);
|
||||||
const MAX_IMAGE_BYTES = 4 * 1024 * 1024;
|
const MAX_IMAGE_BYTES = 4 * 1024 * 1024;
|
||||||
@@ -26,13 +28,13 @@ function asObject(raw) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
export function parseVisionResponse(raw) {
|
export function parseVisionResponse(raw, policy = ABSTENTION_POLICY) {
|
||||||
const value = asObject(raw);
|
const value = asObject(raw);
|
||||||
if (typeof value.isStool !== 'boolean') throw visionResponseError(VISION_FAILURE_CODES.invalidFields, 'Invalid AI response: isStool is required.');
|
if (typeof value.isStool !== 'boolean') throw visionResponseError(VISION_FAILURE_CODES.invalidFields, 'Invalid AI response: isStool is required.');
|
||||||
const confidence = Number(value.confidence);
|
const confidence = Number(value.confidence);
|
||||||
if (!Number.isFinite(confidence) || confidence < 0 || confidence > 1) throw visionResponseError(VISION_FAILURE_CODES.invalidFields, 'Invalid AI response: confidence is required.');
|
if (!Number.isFinite(confidence) || confidence < 0 || confidence > 1) throw visionResponseError(VISION_FAILURE_CODES.invalidFields, 'Invalid AI response: confidence is required.');
|
||||||
if (!value.isStool) return { status: 'needs_user_input', isStool: false, abstainedFor: 'not_stool', reason: 'The image does not clearly show stool.' };
|
if (!value.isStool) return abstainFor('not_stool', false);
|
||||||
if (confidence < 0.55) return { status: 'needs_user_input', isStool: true, abstainedFor: 'low_confidence', reason: 'The image is too uncertain to prefill safely.' };
|
if (confidence < policy.minConfidence) return abstainFor('low_confidence');
|
||||||
// The request schema lets the model return a null form and color when it cannot read them.
|
// The request schema lets the model return a null form and color when it cannot read them.
|
||||||
// A blank field means the model could not read it, so a schema-valid answer with nothing usable
|
// A blank field means the model could not read it, so a schema-valid answer with nothing usable
|
||||||
// abstains and nothing is prefilled. A filled-in field is a claim, though: if it sits outside the
|
// abstains and nothing is prefilled. A filled-in field is a claim, though: if it sits outside the
|
||||||
@@ -47,15 +49,12 @@ export function parseVisionResponse(raw) {
|
|||||||
throw visionResponseError(VISION_FAILURE_CODES.invalidFields, 'Invalid AI response: visual fields are out of range.');
|
throw visionResponseError(VISION_FAILURE_CODES.invalidFields, 'Invalid AI response: visual fields are out of range.');
|
||||||
}
|
}
|
||||||
if (bristolType === null || color === null) {
|
if (bristolType === null || color === null) {
|
||||||
return {
|
return abstainFor('unusable_visual_fields');
|
||||||
status: 'needs_user_input',
|
|
||||||
isStool: true,
|
|
||||||
abstainedFor: 'unusable_visual_fields',
|
|
||||||
reason: 'The photo does not show a form Timmy can read safely.',
|
|
||||||
};
|
|
||||||
}
|
}
|
||||||
const imageQuality = String(value.imageQuality || '').toLowerCase();
|
const imageQuality = String(value.imageQuality || '').toLowerCase();
|
||||||
if (!QUALITIES.has(imageQuality)) throw visionResponseError(VISION_FAILURE_CODES.invalidFields, 'Invalid AI response: visual fields are out of range.');
|
if (!QUALITIES.has(imageQuality)) throw visionResponseError(VISION_FAILURE_CODES.invalidFields, 'Invalid AI response: visual fields are out of range.');
|
||||||
|
// A confident reading of a photo the model itself calls poor is still not a safe prefill.
|
||||||
|
if (isLowQuality(imageQuality, policy)) return abstainFor('low_quality');
|
||||||
return {
|
return {
|
||||||
status: 'suggestion',
|
status: 'suggestion',
|
||||||
isStool: true,
|
isStool: true,
|
||||||
|
|||||||
@@ -0,0 +1,117 @@
|
|||||||
|
import test from 'node:test';
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import { execFile } from 'node:child_process';
|
||||||
|
import { mkdtemp, writeFile } from 'node:fs/promises';
|
||||||
|
import { tmpdir } from 'node:os';
|
||||||
|
import { join } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import { promisify } from 'node:util';
|
||||||
|
import { ABSTENTION_COPY, ABSTENTION_POLICY, abstainFor, evaluatePrefillGate } from '../src/abstention-policy.js';
|
||||||
|
|
||||||
|
test('every abstention reason has one fixed piece of copy the user sees', () => {
|
||||||
|
assert.equal(abstainFor('not_stool').reason, 'The image does not clearly show stool.');
|
||||||
|
assert.equal(abstainFor('low_confidence').reason, 'The image is too uncertain to prefill safely.');
|
||||||
|
assert.equal(abstainFor('unusable_visual_fields').reason, 'The photo does not show a form Timmy can read safely.');
|
||||||
|
assert.equal(abstainFor('low_quality').reason, 'The photo is too blurry or poorly lit to suggest a form safely.');
|
||||||
|
assert.deepEqual(abstainFor('low_quality'), {
|
||||||
|
status: 'needs_user_input',
|
||||||
|
isStool: true,
|
||||||
|
abstainedFor: 'low_quality',
|
||||||
|
reason: ABSTENTION_COPY.low_quality,
|
||||||
|
});
|
||||||
|
assert.equal(abstainFor('not_stool', false).isStool, false);
|
||||||
|
assert.throws(() => abstainFor('looked_wrong'), /unknown abstention reason/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('the policy states the confidence floor and the quality levels that may prefill', () => {
|
||||||
|
assert.ok(Number.isInteger(ABSTENTION_POLICY.version));
|
||||||
|
assert.equal(ABSTENTION_POLICY.minConfidence, 0.55);
|
||||||
|
assert.deepEqual([...ABSTENTION_POLICY.acceptedImageQuality], ['good', 'fair']);
|
||||||
|
assert.equal(ABSTENTION_POLICY.acceptedImageQuality.includes('poor'), false);
|
||||||
|
assert.equal(Object.isFrozen(ABSTENTION_POLICY), true, 'the shipped policy must not be mutated at runtime');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('prefill is authorised only by a GO report whose threshold was selected on validation', () => {
|
||||||
|
const passing = {
|
||||||
|
verdict: 'GO',
|
||||||
|
recommended: 'specialist-candidate',
|
||||||
|
models: [{
|
||||||
|
model: 'specialist-candidate',
|
||||||
|
selectionSplit: 'validation',
|
||||||
|
evaluatedSplit: 'test',
|
||||||
|
gate: { passed: true, failed: [] },
|
||||||
|
}],
|
||||||
|
};
|
||||||
|
assert.deepEqual(evaluatePrefillGate(passing), {
|
||||||
|
prefillAllowed: true,
|
||||||
|
verdict: 'GO',
|
||||||
|
calibrationModel: 'specialist-candidate',
|
||||||
|
blockedBy: [],
|
||||||
|
});
|
||||||
|
|
||||||
|
const blocked = evaluatePrefillGate({
|
||||||
|
verdict: 'BLOCKED',
|
||||||
|
recommended: null,
|
||||||
|
models: [{ model: 'specialist-candidate', selectionSplit: 'validation', evaluatedSplit: 'test', gate: { passed: false, failed: ['precisionAtCoverage'] } }],
|
||||||
|
});
|
||||||
|
assert.equal(blocked.prefillAllowed, false);
|
||||||
|
assert.deepEqual(blocked.blockedBy, ['calibration-verdict-not-go', 'no-recommended-calibration']);
|
||||||
|
|
||||||
|
const selectedOnTest = evaluatePrefillGate({
|
||||||
|
verdict: 'GO',
|
||||||
|
recommended: 'opportunist',
|
||||||
|
models: [{ model: 'opportunist', selectionSplit: 'test', evaluatedSplit: 'test', gate: { passed: true, failed: [] } }],
|
||||||
|
});
|
||||||
|
assert.equal(selectedOnTest.prefillAllowed, false, 'a threshold chosen on the split it is scored on is not evidence');
|
||||||
|
assert.deepEqual(selectedOnTest.blockedBy, ['threshold-not-selected-on-validation']);
|
||||||
|
|
||||||
|
const noReport = evaluatePrefillGate(undefined);
|
||||||
|
assert.equal(noReport.prefillAllowed, false);
|
||||||
|
assert.match(noReport.blockedBy.join(','), /calibration-verdict-not-go/);
|
||||||
|
});
|
||||||
|
|
||||||
|
const execFileAsync = promisify(execFile);
|
||||||
|
const receiptScript = fileURLToPath(new URL('../scripts/abstention_policy_receipt.mjs', import.meta.url));
|
||||||
|
|
||||||
|
async function runReceipt(report) {
|
||||||
|
const directory = await mkdtemp(join(tmpdir(), 'timmy-policy-'));
|
||||||
|
const reportPath = join(directory, 'calibration-report.json');
|
||||||
|
await writeFile(reportPath, JSON.stringify(report));
|
||||||
|
try {
|
||||||
|
const { stdout } = await execFileAsync(process.execPath, [receiptScript, '--report', reportPath]);
|
||||||
|
return { code: 0, stdout };
|
||||||
|
} catch (error) {
|
||||||
|
return { code: error.code, stdout: error.stdout || '', stderr: error.stderr || '' };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
test('the receipt refuses to authorise prefill without a passing held-out calibration', async () => {
|
||||||
|
const blocked = await runReceipt({
|
||||||
|
verdict: 'BLOCKED',
|
||||||
|
recommended: null,
|
||||||
|
models: [{ model: 'specialist-candidate', selectionSplit: 'validation', evaluatedSplit: 'test', gate: { passed: false, failed: ['precisionAtCoverage'] } }],
|
||||||
|
});
|
||||||
|
assert.equal(blocked.code, 3, 'a blocked calibration must exit non-zero instead of shipping prefill');
|
||||||
|
assert.match(blocked.stdout, /prefill: BLOCKED/);
|
||||||
|
assert.match(blocked.stdout, /calibration-verdict-not-go/);
|
||||||
|
assert.match(blocked.stdout, /policy: version 1 · confidence ≥ 0\.55 · quality good\|fair/);
|
||||||
|
|
||||||
|
const passing = await runReceipt({
|
||||||
|
verdict: 'GO',
|
||||||
|
recommended: 'specialist-candidate',
|
||||||
|
models: [{ model: 'specialist-candidate', selectionSplit: 'validation', evaluatedSplit: 'test', gate: { passed: true, failed: [] } }],
|
||||||
|
});
|
||||||
|
assert.equal(passing.code, 0);
|
||||||
|
assert.match(passing.stdout, /prefill: AUTHORISED for specialist-candidate/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('the receipt treats a missing report as a block and never as a default allow', async () => {
|
||||||
|
const directory = await mkdtemp(join(tmpdir(), 'timmy-policy-'));
|
||||||
|
try {
|
||||||
|
await execFileAsync(process.execPath, [receiptScript, '--report', join(directory, 'absent.json')]);
|
||||||
|
assert.fail('a missing calibration report must not exit 0');
|
||||||
|
} catch (error) {
|
||||||
|
assert.equal(error.code, 2);
|
||||||
|
assert.match(String(error.stderr), /could not read|absent/i);
|
||||||
|
}
|
||||||
|
});
|
||||||
@@ -1,5 +1,6 @@
|
|||||||
import test from 'node:test';
|
import test from 'node:test';
|
||||||
import assert from 'node:assert/strict';
|
import assert from 'node:assert/strict';
|
||||||
|
import { ABSTENTION_COPY, ABSTENTION_POLICY } from '../src/abstention-policy.js';
|
||||||
|
|
||||||
import {
|
import {
|
||||||
buildVisionRequest,
|
buildVisionRequest,
|
||||||
@@ -131,3 +132,39 @@ test('abstains instead of failing when a schema-valid response carries no readab
|
|||||||
reason: 'The photo does not show a form Timmy can read safely.',
|
reason: 'The photo does not show a form Timmy can read safely.',
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('a confident answer about a low-quality photo still abstains', () => {
|
||||||
|
// Issue #32: confidence is not the only gate. A blurry or badly lit photo can produce a
|
||||||
|
// confident-looking form and colour, and prefilling it would show the user a claim the photo
|
||||||
|
// cannot support. Quality is part of the policy, so this abstains instead of prefilling.
|
||||||
|
assert.deepEqual(parseVisionResponse({
|
||||||
|
isStool: true,
|
||||||
|
bristolType: 4,
|
||||||
|
color: 'brown',
|
||||||
|
confidence: 0.94,
|
||||||
|
imageQuality: 'poor',
|
||||||
|
observations: 'Form barely visible through motion blur.',
|
||||||
|
}), {
|
||||||
|
status: 'needs_user_input',
|
||||||
|
isStool: true,
|
||||||
|
abstainedFor: 'low_quality',
|
||||||
|
reason: 'The photo is too blurry or poorly lit to suggest a form safely.',
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test('the parse path reads the shipped policy instead of hardcoding its own limits', () => {
|
||||||
|
// Issue #32: policy numbers and copy live in one module. A caller may tighten the policy for a
|
||||||
|
// release without editing the parser, and a photo that the default policy would accept has to
|
||||||
|
// abstain under the stricter one — otherwise the parser is carrying its own hidden policy.
|
||||||
|
const fields = { isStool: true, bristolType: 4, color: 'brown', confidence: 0.8, imageQuality: 'fair', observations: 'Form visible but soft.' };
|
||||||
|
assert.equal(parseVisionResponse(fields).status, 'suggestion', 'fair quality may prefill under the shipped policy');
|
||||||
|
const strict = { ...ABSTENTION_POLICY, minConfidence: 0.9, acceptedImageQuality: ['good'] };
|
||||||
|
assert.equal(parseVisionResponse(fields, strict).abstainedFor, 'low_confidence');
|
||||||
|
assert.equal(parseVisionResponse({ ...fields, confidence: 0.95 }, strict).abstainedFor, 'low_quality');
|
||||||
|
assert.equal(parseVisionResponse({ ...fields, confidence: 0.95, imageQuality: 'good' }, strict).status, 'suggestion');
|
||||||
|
assert.equal(
|
||||||
|
parseVisionResponse({ ...fields, imageQuality: 'poor' }).reason,
|
||||||
|
ABSTENTION_COPY.low_quality,
|
||||||
|
'the refusal copy must come from the policy module, not a second copy in the parser',
|
||||||
|
);
|
||||||
|
});
|
||||||
+16
@@ -131,6 +131,22 @@
|
|||||||
"expectedOutcome": "fail_closed",
|
"expectedOutcome": "fail_closed",
|
||||||
"expectedReason": "out_of_range_or_missing_fields",
|
"expectedReason": "out_of_range_or_missing_fields",
|
||||||
"providerContent": "{\"isStool\":true,\"bristolType\":9,\"color\":null,\"confidence\":0.9,\"imageQuality\":\"good\",\"observations\":\"Form read, colour not.\"}"
|
"providerContent": "{\"isStool\":true,\"bristolType\":9,\"color\":null,\"confidence\":0.9,\"imageQuality\":\"good\",\"observations\":\"Form read, colour not.\"}"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "poor-quality-confident",
|
||||||
|
"caseClass": "blur",
|
||||||
|
"file": "blur-low-detail.png",
|
||||||
|
"bytes": 21598,
|
||||||
|
"sha256": "17000163662a42fd434bb2b2428d9e5a2156da08cbc82210e41dfc6fa7e24852",
|
||||||
|
"width": 640,
|
||||||
|
"height": 480,
|
||||||
|
"format": "png",
|
||||||
|
"synthetic": true,
|
||||||
|
"license": "CC0-1.0",
|
||||||
|
"provenance": "The same synthetic motion-blur fixture that blur-low-detail replays, replayed again with a confident provider answer; drawn deterministically by scripts/build_vision_corpus.py, no camera, no human subject, no third-party content.",
|
||||||
|
"expectedOutcome": "abstained",
|
||||||
|
"expectedReason": "low_quality",
|
||||||
|
"providerContent": "{\"isStool\":true,\"bristolType\":4,\"color\":\"brown\",\"confidence\":0.94,\"imageQuality\":\"poor\",\"observations\":\"Form barely visible through motion blur.\"}"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
@@ -272,6 +272,14 @@ test('release demo visibly exercises the corpus refusal paths instead of only th
|
|||||||
assert.match(demo, /A non-stool photo gets a refusal, never a guess/);
|
assert.match(demo, /A non-stool photo gets a refusal, never a guess/);
|
||||||
assert.match(demo, /Unreadable visual fields abstain as well/);
|
assert.match(demo, /Unreadable visual fields abstain as well/);
|
||||||
assert.match(demo, /An out-of-range answer fails closed/);
|
assert.match(demo, /An out-of-range answer fails closed/);
|
||||||
|
// Issue #32's quality gate: a confident answer about a low-quality photo must abstain on camera.
|
||||||
|
assert.match(demo, /corpusCase\('poor-quality-confident'\)/);
|
||||||
|
assert.match(demo, /tests\/fixtures\/vision-corpus\/blur-low-detail\.png/);
|
||||||
|
assert.match(demo, /too blurry or poorly lit/);
|
||||||
|
assert.match(demo, /must not prefill a form, however confident the reply looked/);
|
||||||
|
assert.match(demo, /A confident reply about a blurry photo still abstains/);
|
||||||
|
assert.match(builder, /A confident reply about a blurry photo still abstains — photo quality is part of the policy, not just confidence/);
|
||||||
|
assert.match(builder, /"abstention_quality_gate": "passed"/);
|
||||||
assert.match(demo, /no real photos in the corpus|no real photos in the recording/);
|
assert.match(demo, /no real photos in the corpus|no real photos in the recording/);
|
||||||
// The refusal copy the user sees is gated on camera, so a release cannot drop it silently.
|
// The refusal copy the user sees is gated on camera, so a release cannot drop it silently.
|
||||||
assert.match(builder, /An out-of-range answer fails closed/);
|
assert.match(builder, /An out-of-range answer fails closed/);
|
||||||
|
|||||||
Reference in new issue
Block a user