Add an executable abuse suite that drives the real server with synthetic hostile bytes while a
counting stub worker records every call that reaches the model boundary, plus the verdict rule that
turns those observations into a release decision.
- src/release-security-gate.js: nearest-rank latency percentiles, per-case pass/fail derivation that
names the field that broke, and the rule that any failing P0 case blocks RC approval while a
required class with no verified evidence blocks it too. Reports are redacted and fail closed.
- scripts/security_attack_suite.mjs: oversized body, HTML polyglot as JPEG, undecodable bytes,
missing consent, non-conforming worker answer, byte-identical replay, concurrent burst, captured
log scan, and an out-of-surface deletion case counted only against a real test file. Writes the
report, prints the verdict, exits 1 when blocked.
- docs/RC-SECURITY-EVIDENCE.md: procedure, rule, and the current finding stated plainly.
- Demo: a real run of the suite on camera, printing the verdict it derived instead of a flattering
one; CI regenerates the same evidence on every change.
The suite currently returns VERDICT: BLOCKED: bytes are trusted from the data-URL label, so an HTML
polyglot still reaches the worker and only fails afterwards on schema. That is a finding against
#16, not a passing check, and it is why the gate treats a failing case as a block.
The corpus had an append-only manifest, a reviewer tool, and an evaluation harness, but nothing a
stranger could read: what is in the corpus, what a label means, and how leakage is prevented lived in
code and in reviewers' heads.
scripts/dataset_card.py generates docs/DATASET_CARD.md from manifest.jsonl. Every count is read back
from real records — active samples, contributors, per-split contributor and sample counts, class
balance over active samples only — and every fact the pipeline does not collect (devices, contexts,
licenses) is printed as pending rather than estimated. The generator refuses to render a card for a
corpus where one contributor straddles two splits or where a recorded split is not the one the
subject key derives, and it never publishes a subject key or a withdrawn sample's labels. --check
fails when the committed page is not the generator's own output, so the card cannot drift into a
hand-written document.
docs/LABELING_GUIDE.md is the reviewer's rulebook: visible form, broad visible color, and photo
quality; visible color is not blood; nothing is diagnosed; a model suggestion is provisional input to
a human decision. Exclusions are named, and the split policy is stated where it is enforced.
The demo's new DATASET CARD scene builds a throwaway corpus with the real ingest command from
committed synthetic fixtures, publishes the card, and shows the refusal a planted contributor earns,
then re-checks the committed page. Its captions are release gates in build_release.py, which also runs
tests/dataset-card.test.py before a release exists.
tests/dataset-card.test.py covers the counts, the pending facts, the withdrawal receipt, both
refusals, determinism, the empty-corpus case, the committed-page identity, and both documents'
prohibitions. Every fixture is synthetic.
Closes#27
The ROADMAP gates (macro-F1 >= 0.80, precision@coverage >= 0.90, ECE <= 0.05) were prose with no
tool behind them. scripts/calibration_report.py now computes them from prediction records and
refuses input it cannot honestly score:
- a split is re-derived from the pseudonymous subject key with the ingestion rule, so a corpus
cannot re-label its own splits; a contributor spanning two splits is refused before scoring
- the threshold is selected on validation only, and selecting on test or train is refused
- coverage, precision-at-coverage, macro-F1, per-class precision/recall, confusion, and expected
calibration error are computed on accepted suggestions, with classes that abstained away named
in excludedClasses instead of being scored as a silent zero
- exit codes carry the verdict: 0 GO, 3 BLOCKED, 2 refused
The release demo records the real harness against committed synthetic fixtures: a blocked
three-candidate comparison, a fixture engineered to clear every gate, and a leaked corpus refused.
Every number on the card is read back from the report the run wrote, and the captions state that
no model was measured, because a fixture is not evidence about accuracy.
The harness suite is wired into CI, check:syntax, and the release builder's own gates.
Deterministic escalation only fired on a narrow set of literal phrases, so real reports
like "there is blood when I wipe" or "I have tarry stools" reached the chat provider
without the fixed medical guidance. Expanding the suite found 18 of 18 checked clinical
phrasings missing.
- domain: cover bleeding reported by wiping/toilet paper/anatomy/mixed-in phrasing,
black-tarry-dark-red stool, reported temperature and high temperature, abdominal-scoped
severe pain in every natural word order, and past-tense/slang inability to pass gas.
Near-miss wording that must stay quiet (paper cut, blood pressure, black beans, dark red
wine, a normal temperature, knee or back pain, passed gas, no gas in the car) is pinned.
- domain: a symptom reported in any truthy shape (1, "yes", "true") now counts as reported.
Reading only the literal boolean `true` silently cleared a symptom, which is the one
direction this boundary may never fail in.
- hermes-agent-service: the gateway ledger sanitizer used the same strict check, so a
browser-supplied ledger entry could have its symptom cleared before the escalation check.
- tests/escalation-regression.test.js: per-flag positive and negative language coverage,
model output can neither set nor clear a symptom/urgency/discomfort/note, and the
escalation copy is byte-identical for every flag and under a missing, throwing, or
malicious provider. Registered in the default unit gate.
- scripts/record_release_demo.mjs: the recording now types the new bleeding and tarry
phrasing, and seeds a fictional ledger whose note says "stool was tarry yesterday", then
asserts in the browser that exactly one new Timmy bubble carries the app's own copy and
that the turn never reaches Hermes. Three new captions are required release gates.
Boundary unchanged: no new inference, no diagnosis, no image claim. Escalation only ever
gets broader on user-reported symptoms.
The home insight card now reports the confirmed-log count and form
groupings for a window, every line names the exact confirmed records it
was built from, and following a citation opens those records. A window
below the minimum states that it is too sparse instead of describing a
trend, and a window with both firm and loose logs says it is mixed rather
than picking a winner. No line explains a cause, rules on food, or offers
a health verdict.
- src/summary.js: window filtering, insufficient-data guard, per-form
lines, mixed-pattern line, boundary copy, record citations
- app.js: explainable home card plus a pattern-source sheet listing the
cited records
- tests/summary.test.js, tests/pattern-summary.acceptance.mjs: unit and
real-browser coverage, gated in CI and the release builder
- scripts/record_release_demo.mjs: fictional multi-day sample logs,
citation follow-through, and the sparse-window refusal on camera
A worker that answers /v1/models has not necessarily loaded its weights, so the app
was offering an AI control that could only fail after the upload. Readiness now
passes only once the pinned model is loaded, and the not-loaded state reaches manual
Bristol entry in one tap. Verified against the real pinned llama.cpp build: it serves
503 while loading and the gate reports the distinct not-loaded state.
Adds the reproducible container recipe (digest-pinned bases, pinned llama.cpp commit,
hash-verified read-only weights, unprivileged, loopback-only publish), one shared
post-load readiness gate used by the host bootstrap and the image healthcheck, and a
target-GPU benchmark that refuses to write a receipt it did not measure.
Refs #15
The worker contract was only proven by one happy-path fixture and a handful of
inline assertions, so nothing continuously checked that a reply Timmy cannot use
is refused instead of guessed at.
Corpus: a CC0 synthetic corpus (stool, non-stool, blur, glare, empty bowl,
malformed response) generated deterministically by scripts/build_vision_corpus.py.
Every fixture records its sha256, provenance, licence, case class, the provider
reply it replays, and the outcome that reply must produce. loadVisionCorpus
refuses a fixture whose bytes no longer match the manifest, so the corpus cannot
be edited into a different test. No real medical imagery, camera, or third-party
content is involved.
Contract fixes the corpus exposed:
- A schema-valid reply with a null form and colour is now an abstention
(abstainedFor: unusable_visual_fields) instead of a hard provider failure. The
declared response schema permits those nulls, so treating the reply as an error
was wrong; it now abstains calmly and still prefill nothing.
- Fail-closed replies carry a machine-readable code (invalid_json, invalid_fields)
instead of prose matching, and visionFailureResponse() maps them to plain,
user-safe copy with no provider text, image bytes, or credentials echoed. A
running server now returns that copy for unparsable and out-of-range replies.
Evidence: unit corpus contract and hash integrity, corpus-wide conformance
through the real analysis path, the same corpus through a running server.mjs with
a stub worker (200 abstain vs 503 fail-closed, no leaked bytes), and a real
browser acceptance run for the refusal UI. The release demo replays three corpus
cases through the real code path so the video shows the status the server would
actually send, with the on-camera caption gated by the release builder.
Field finding #67: a hard/lumpy (Bristol 1-2) entry now asks a short follow-up
before saving — difficult to pass, soreness, visible red blood, truly
black/sticky/tarry versus dark brown, and dehydration. Every answer is
user-reported and starts unanswered; nothing is inferred from the photo, and
answering "dark brown" never silently escalates as black stool.
A confirmed red-flag answer writes the same symptom flag manual logging already
uses, so the existing deterministic escalation is the only escalation path.
Mild reports get concise non-diagnostic self-care that states the photo cannot
rule out bleeding, and it is withdrawn whenever a red flag is confirmed.
- src/followup.js: pure follow-up contract (9 unit tests, tests/followup.test.js)
- app.js + styles.css: follow-up block on the safety step for Type 1-2 entries only
- tests/followup.acceptance.mjs: browser acceptance incl. that the AI suggestion
surface still contains no symptom wording
- CI + release builder gates: npm run test:followup, syntax checks, manifest gate
- scripts/record_release_demo.mjs: records the real follow-up, mild guidance, and
the black/tarry alert handoff with synthetic data
Suggested, confirmed, and corrected AI states are now distinct in the log
sheet, and correcting a suggested value clears the AI-confirmed state while
keeping every user-entered field. Screen-reader copy states the AI confidence
and who is in control.
- src/suggestion.js: provenance state machine (pending/confirmed/corrected/manual)
- app.js: renders provenance on the scan result and log steps, tracks corrections
- tests/suggestion.test.js: unit coverage for the state machine and a11y copy
- tests/provisional-suggestion.acceptance.mjs: browser acceptance + screenshots
- record_release_demo.mjs: demo now reviews, confirms, and corrects the suggestion
- CI, release builder, and release notes gate the new contract
Logging must never depend on AI availability, and an unsaved entry must not
disappear when the user navigates away mid-log.
- add src/vision-readiness.js so the photo flow consumes one readiness contract
- when the worker is offline/disabled/unknown, offer no AI action and no
analysis-consent prompt, and keep manual Bristol entry one action away
- add src/draft.js and make the draft own the photo so a chosen photo and
entered context survive a recoverable close or navigation, with a visible
kept-draft summary and an explicit start over
- drafts stay in session memory only: a photo awaiting consent is never
persisted, and reload starts clean
- service worker now refuses to cache image bodies in addition to analysis
requests and no-store responses
- tests: 2 new unit files, 3 service-worker privacy assertions (1 hold-out),
1 browser acceptance file, release-demo and CI/package assertions
- release demo now visibly replays the vision-offline pass and the retained
draft with synthetic data and an in-script assertion
timmy
timmy
timmy