Compare commits

...
Author SHA1 Message Date
rockachopa 2a10a20518 fix(release): give each calibration candidate its own row so the name is not split at its hyphen
Quality gates / quality (push) Successful in 12m51s
2026-10-25 12:23:56 +00:00
rockachopa a2293c6e34 Merge pull request #86: Build contributor-held-out evaluation harness and calibration report (#29)
Quality gates / quality (push) Successful in 12m54s
Closes #29
2026-10-25 12:02:11 +00:00
timmy 179f8fe6e9 feat(eval): score candidates against the release gates on held-out contributors (#29)
Quality gates / quality (pull_request) Successful in 13m25s
The ROADMAP gates (macro-F1 >= 0.80, precision@coverage >= 0.90, ECE <= 0.05) were prose with no
tool behind them. scripts/calibration_report.py now computes them from prediction records and
refuses input it cannot honestly score:

- a split is re-derived from the pseudonymous subject key with the ingestion rule, so a corpus
  cannot re-label its own splits; a contributor spanning two splits is refused before scoring
- the threshold is selected on validation only, and selecting on test or train is refused
- coverage, precision-at-coverage, macro-F1, per-class precision/recall, confusion, and expected
  calibration error are computed on accepted suggestions, with classes that abstained away named
  in excludedClasses instead of being scored as a silent zero
- exit codes carry the verdict: 0 GO, 3 BLOCKED, 2 refused

The release demo records the real harness against committed synthetic fixtures: a blocked
three-candidate comparison, a fixture engineered to clear every gate, and a leaked corpus refused.
Every number on the card is read back from the report the run wrote, and the captions state that
no model was measured, because a fixture is not evidence about accuracy.

The harness suite is wired into CI, check:syntax, and the release builder's own gates.
2026-10-25 11:05:28 +00:00
rockachopa 9f69730780 fix(release): say "any raw copy" so the caption matches what a withdrawal can delete
Quality gates / quality (push) Successful in 12m45s
On the CLI ingest path there is no raw pre-derivation copy at all, so the
caption asserted a deletion the recording never showed. The code deletes a raw
copy only when one exists under the dataset's raw/ directory, matched by digest.
2026-10-24 12:51:37 +00:00
rockachopa fa5f4232dd fix(release): draw the corpus audit card opaque and gate on it
Quality gates / quality (push) Successful in 12m37s
The first recording of the card was legible but translucent, so the bottom nav
and the journal's body copy ghosted through its empty lower rows. The card is
now fully opaque and the on-screen proof rejects a background with any alpha, so
a future edit cannot reintroduce the bleed unnoticed.
2026-10-24 12:38:01 +00:00
rockachopa 1d338ed35b test(release): record the corpus manifest on camera with the real ingest and revoke commands
Quality gates / quality (push) Successful in 12m46s
Issue #25's corpus pipeline is a CLI with no app screen, so the demo runs the
real ingest and revoke commands against the committed synthetic fixture in a
throwaway dataset and draws the on-camera audit card from their actual output:
the provenance record, the deleted derived image, the kept original record, the
tombstone that drops labels and the subject key, zero active samples, and a
second revocation refused without appending anything.

The card is asserted on screen (fixed, topmost, unclipped) and the values on it
are read back from the run, so a build that stops recording provenance or stops
deleting bytes fails instead of recording a claim. build_release.py now runs
tests/corpus-manifest.test.py as a release gate and requires the four new
captions to have painted on camera.
2026-10-24 12:20:25 +00:00
rockachopa 6aacefca65 Merge pull request #85: feat(corpus): auditable manifest provenance, tombstones, and lifecycle guards (#25)
Quality gates / quality (push) Successful in 12m47s
2026-10-24 12:02:52 +00:00
timmy 578cc9f129 style: end the new corpus files with a single newline
Quality gates / quality (pull_request) Successful in 13m1s
CI's diff-hygiene gate rejects a trailing blank line at EOF; the local
check:diff run happened before the first commit existed, so it checked the
previous range and missed it.
2026-10-24 11:12:50 +00:00
timmy 860c84f0fe feat(corpus): auditable manifest provenance, tombstones, and lifecycle guards (#25)
Quality gates / quality (pull_request) Failing after 9m42s
Promote the ingest script into an auditable corpus pipeline. Each sample now
records consent (version + timestamp), source kind and raw-input digest, both
hashes, the derivation pipeline, review status, split, and lifecycle state, with
the caller's path never written down.

Withdrawal is a new `--revoke <contribution-id> --reason` operation: it deletes
the derived image and any raw pre-derivation copy, then appends a tombstone that
addresses each removed artifact by contribution id and digest while dropping
labels, subject key, and paths. History is append-only, so the original sample
record survives the withdrawal it documents.

Illegal moves are refused instead of silently applied: revoked -> active,
revoking an unknown or already-revoked contribution, an unknown state, a
duplicate contribution id, and a second label set for the same derived image.
The duplicate guard runs before any byte is written, so a refused ingest cannot
strand an image in the active dataset.

Tests written first and watched fail (tests/corpus-manifest.test.py): provenance,
tombstone + byte removal, active-record filtering, and the transition guards.
The transition helper itself passed on first write and is reported as a
hold-out check rather than a RED cycle. Wired into the quality workflow and
check:syntax; contract documented in docs/CORPUS-MANIFEST.md.

No browser surface changed, so record_release_demo.mjs is untouched and claims
nothing new: the nearest honest visible effect is the withdrawn-receipt
tombstone already recorded for #24.
2026-10-24 10:58:30 +00:00
mercury 3565ce87a4 fix(release): refuse to gate or record against a foreign acceptance server
Quality gates / quality (push) Successful in 16m30s
A leftover checkout answering on 127.0.0.1:4173 satisfied the readiness probe,
so every acceptance flow and the recorded demo described another tree while the
build reported green. The builder now fails closed if the port already has a
listener, re-checks the spawned server after readiness, and verifies the served
app.js and src/domain.js bytes match the clone under test before the first gate.

Verified against a real stale tree: the occupied-port guard aborts, the
foreign-bytes guard aborts, and the matching tree still passes.
2026-10-22 12:37:57 +00:00
mercury 4ac71915fb fix(portability): report only the upgrade in the import notice (#35)
Quality gates / quality (push) Successful in 29m22s
The import screen already states what it restored, and the migration notice
restated the same count, so the on-screen notice read as two stacked sentences
saying one thing. The notice now only describes the upgrade, and a unit test
fails if it starts restating the restored count again.
2026-10-22 12:20:31 +00:00
rockachopa 5a15fa660e Merge pull request #84: feat(portability): versioned backups, v1 migration, and AI provenance (#35)
Quality gates / quality (push) Successful in 21m49s
2026-10-22 12:02:21 +00:00
timmy 2094a07d9f feat(portability): version the backup, migrate older files, and keep AI provenance (#35)
Quality gates / quality (pull_request) Successful in 18m50s
A backup is the person's own record and has to keep working as the AI fields evolve.

- Export stamps schema version 2 and carries the confirmed logs plus the training
  contribution receipts, including withdrawn tombstones; both are written through an
  allowlist so unknown keys and credentials never leave the device.
- Import accepts version 1 and 2. A version 1 file is upgraded, the notice names the
  version it came from, and entries that predate provenance stay null instead of being
  given a source they never recorded.
- A file from a newer Timmy is refused with an update instruction, malformed, oversized
  and over-limit files fail closed, and a refused import leaves the ledger unchanged.
- A log saved from a confirmed or corrected AI suggestion records its provenance
  (source, suggested visible fields, model, confirmedAt) and the entry sheet states it.
- Service worker shell cache v5 -> v6 so installed clients actually receive the change.

Evidence: tests/portability.test.js (7 unit cases), tests/portability.acceptance.mjs
(real browser: v1 upgrade, future refusal with unchanged ledger, provenance recorded,
export v2, round trip), release demo scenes with on-camera caption proof, and the
release builder gate portable_backup_restore.

Refs #35
2026-10-22 11:08:55 +00:00
timmy baabeb0f4f fix(privacy): give training-contribution rows a readable layout (#24)
Quality gates / quality (push) Successful in 3m52s
The opt-in row and its tombstone rendered as one run-on line ('Type 4 ·
brownReceipt contrib-…', 'Withdrawn Oct 21, 12:09 PMraw image, …') because
nothing styled .contribution-row: the label and the receipt/artifact line
shared a line and the Withdraw control butted against the text.

Split label and detail into their own lines, right-align the row action,
mute withdrawn tombstones, and keep long receipt ids wrapping inside the
card. Verified on a 390px viewport: queued, withdrawn, and offer rows are
separately legible with no horizontal overflow.
2026-10-21 12:15:25 +00:00
timmy 11c63a8d25 Merge pull request #83: feat: separate opt-in training contribution receipts (#24)
Quality gates / quality (push) Successful in 3m47s
2026-10-21 12:01:52 +00:00
timmy ef22ea8970 feat(training): separate opt-in contribution receipts with withdrawable tombstones
Quality gates / quality (pull_request) Successful in 3m44s
A journal save now records no training contribution. A contribution is a second
consent: the privacy page states consent version training-consent-v1 and the
receipt (contribution id, included fields, never-included fields) before the
confirm control is enabled, the receipt is queued on this device only and copies
no photo bytes, note text, or symptom answers, and withdrawal keeps a tombstone
naming every derived artifact by contribution id. Delete-all clears the receipts
with the ledger.

Closes #24
2026-10-21 10:58:50 +00:00
rockachopa d0b3c10aea Merge pull request 'fix(release): paint demo captions and touch rings on camera and verify the pixels' (#81) from fix/demo-caption-pixels into main 2026-10-19 12:35:29 +00:00
rockachopa c2e9553be5 chore: update tests/release-demo.test.js for the caption pixel gate
Quality gates / quality (pull_request) Successful in 3m14s
2026-10-19 12:35:14 +00:00
rockachopa d1c858b940 chore: update scripts/build_release.py for the caption pixel gate 2026-10-19 12:35:12 +00:00
rockachopa fd09282d8d fix(release): paint demo captions and touch rings on camera and verify the pixels
The demo injected its caption CSS with page.addStyleTag(), which writes into the current document
only. The first page.reload() in the recording stripped that stylesheet, so every later caption
rendered as unstyled body text clipped at the bottom of the frame and every touch indicator was
invisible, while the caption receipt still listed each string and the release gate accepted it.

- caption and touch nodes now carry inline styles, which no reload can remove
- caption() proves each caption is a fixed bar at the top of the viewport, on top of what is
  beneath it, fully inside the frame, and that the caption band pixels change; the recording fails
  rather than shipping a caption nobody can read
- the receipt records per-caption evidence and build_release.py requires every required caption,
  including the release delta, to be pixel-verified
- the demo photo is a legible synthetic test card instead of a 1x1 white pixel that rendered on
  camera as an empty box
- tests assert the inline styling and the pixel gate
2026-10-19 12:35:11 +00:00
rockachopa 0e4dde5f81 Merge pull request 'feat: separate saved-photo lifecycle from the confirmed log (#36)' (#80) from timmy/36-photo-lifecycle-controls into main 2026-10-19 12:02:37 +00:00
timmy ebee52fd8a feat: separate saved-photo lifecycle from the confirmed log (#36)
Quality gates / quality (pull_request) Successful in 3m9s
A confirmed log and the image attached to it are separate things the
person owns. The journal now reports a saved photo as a label instead of
a thumbnail, reveals the image only on explicit request, removes the
photo without touching any confirmed field, and deletes a log through a
separate two-step decision that takes its photo with it. Delete-all is
regression tested to remove the ledger and every stored photo together.

- src/photo-lifecycle.js: presence decided by stored image bytes;
  withoutPhoto/withoutEntry return new lists and never mutate the ledger
- app.js: journal rows expose Open-entry controls; entry sheet states
  presence, offers explicit reveal, remove-photo-keeps-log, and a
  two-step log delete; privacy copy names photo removal and delete-all
- tests: 6 unit cases + browser acceptance (presence, no default
  thumbnail, explicit reveal, remove keeps log, two-step delete,
  delete-all leaves no image bytes, Escape dismiss, 44px targets)
- record_release_demo.mjs: on-camera photo lifecycle scenes driven by a
  synthetic 1x1 PNG, with delete-all actually performed and asserted;
  build_release gates the new captions
- CI/release/syntax gates include the new suite

Closes #36
2026-10-19 08:24:44 +00:00
35 changed files with 3598 additions and 73 deletions

No files matched your search

+5
View File
@@ -33,6 +33,8 @@ jobs:
run: |
npm test
python3 tests/staging-deploy.test.py -v
python3 tests/corpus-manifest.test.py -v
python3 tests/calibration-harness.test.py -v
- name: Mobile browser acceptance
run: |
npm start > /tmp/timmy-server.log 2>&1 &
@@ -58,6 +60,9 @@ jobs:
npm run test:vision-refusal
npm run test:worker-readiness
npm run test:pattern-summary
npm run test:photo-lifecycle
npm run test:training-contribution
npm run test:portability
- name: Dependency audit
run: npm audit --audit-level=high
- name: Syntax checks
+6 -1
View File
@@ -99,6 +99,8 @@ npm run test:sleek # simplified 390x844 shell + free-text Hermes chat
npm run test:manual-fallback # vision-offline manual path + in-session draft recovery
npm run test:vision-refusal # corpus refusals through a real server and a real browser
npm run test:grounded-chat # chat answers cite the ledger, or refuse to answer
npm run test:photo-lifecycle # photo presence, explicit reveal, remove-photo-keeps-log, delete-all
npm run test:calibration # the release-gate harness: gates, threshold protocol, refused corpora
npm audit --audit-level=high
```
@@ -120,7 +122,8 @@ npm audit --audit-level=high
- Manual fallback that never depends on AI: when the vision worker is offline or disabled, no AI or consent control is offered and manual Bristol entry stays one action away
- A committed CC0 synthetic vision corpus that continuously proves every reply is schema-valid or fails closed. A schema-valid reply with no readable form or colour now **abstains** ("No confident match") instead of surfacing as a worker failure, and unparsable or out-of-range replies return plain copy — "No suggestion was applied. Continue manually, or try a clearer photo." — with no provider text, image bytes, or credentials echoed. See [tests/fixtures/vision-corpus](tests/fixtures/vision-corpus/README.md)
- Unsaved logging drafts are retained in the session with an explicit start over, and the service worker never caches analysis requests or image bodies
- Local browser ledger, calendar, pattern summary, JSON portability, and delete-all
- Private photo lifecycle: a confirmed log reports a saved photo as a label rather than a thumbnail, showing the image is an explicit request, removing the photo keeps every confirmed field, deleting a log is a separate two-step decision, and delete-all removes the ledger and every stored photo together
- Local browser ledger, calendar, pattern summary, versioned JSON backup (logs, training receipts, and per-log AI provenance) with safe upgrade/refusal on import, and delete-all
- Explainable pattern summaries: the home insight card reports the confirmed-log count and form groupings for the last 28 days, every line names the exact confirmed records behind it, following a citation opens those records, and a window below the minimum says it is too sparse instead of describing a trend. No line explains a cause, rules on food, or offers a health verdict
- Installable/offline PWA shell for the manual and saved-ledger paths
@@ -130,6 +133,8 @@ The positive photo-prefill interface is covered by a deterministic provider fixt
Timmy does not diagnose disease, identify bleeding, infer pain/urgency, recommend treatment, or clear foods. The selected VLM is a bootstrap worker, not the final classifier. See [research/SELF-HOSTED-STOOL-VISION.md](research/SELF-HOSTED-STOOL-VISION.md) for the measured receipts, hardware gate, consented-data pipeline, specialist-classifier plan, and sources.
The calibration harness ([scripts/calibration_report.py](scripts/calibration_report.py), [contract](docs/EVALUATION-HARNESS.md)) computes the ROADMAP release gates — macro-F1, precision-at-coverage, and calibration error — from prediction files, chooses a threshold on validation only, and refuses a corpus that leaks one contributor across two splits. Its committed fixtures are invented numbers that exercise the arithmetic: **no candidate model has been measured against these gates yet, and the harness itself is not evidence about accuracy.**
## Architecture
```text
+128 -7
View File
@@ -1,4 +1,4 @@
import { bucketForBristolType, detectUrgentFlags, detectUrgentText, exportLedger, hasUrgentLedgerContext, importLedger, photoQualityMessage, sanitizeEntry, urgentChatMessage } from './src/domain.js';
import { bucketForBristolType, detectUrgentFlags, detectUrgentText, exportLedgerBackup, LEDGER_EXPORT_VERSION, hasUrgentLedgerContext, importLedgerBackup, photoQualityMessage, sanitizeEntry, urgentChatMessage } from './src/domain.js';
import { mergeVisualSuggestion } from './src/analysis.js';
import { createDraft, draftNotice, isDraftDirty } from './src/draft.js';
import { changeVisualField, confirmSuggestion, createSuggestionState, provenanceCopy } from './src/suggestion.js';
@@ -6,6 +6,8 @@ import { HARD_STOOL_QUESTIONS, createFollowUpAnswers, evaluateHardStoolFollowUp,
import { visionReadiness } from './src/vision-readiness.js';
import { buildPatternSummary } from './src/summary.js';
import { answerFromLedger, buildChatLedger, resolveAgentReply } from './src/grounded-chat.js';
import { photoState, withoutEntry, withoutPhoto } from './src/photo-lifecycle.js';
import { addContribution, consentTerms, contributionEligibility, contributionForEntry, receiptPreview, withdrawContribution } from './src/training-contribution.js';
const runtimeConfig = {
basePath: document.querySelector('meta[name="timmy-base-path"]')?.content || '/',
@@ -15,6 +17,7 @@ const BASE_PATH = runtimeConfig.basePath || '/';
const APP_ROOT = BASE_PATH === '/' ? '/' : `${BASE_PATH}/`;
function appPath(path='') { return `${APP_ROOT}${String(path).replace(/^\/+/, '')}`; }
const STORE = `timmy:${BASE_PATH}:ledger-v1`;
const TRAINING_STORE = `timmy:${BASE_PATH}:training-contributions-v1`;
const LEGACY_STORE = 'timmy-ledger-v1';
if (BASE_PATH === '/') {
const legacyEntries = localStorage.getItem(LEGACY_STORE);
@@ -27,6 +30,7 @@ if (BASE_PATH === '/') {
}
const app = document.querySelector('#app');
let entries = loadEntries();
let contributions = loadContributions();
let view = 'home';
let photoHint = '';
let aiSuggestion = null;
@@ -38,6 +42,10 @@ let chatBusy = false;
let chatError = '';
let chatMessages = [{ role: 'timmy', text: 'Ask me about your confirmed logs, visible patterns, privacy, or how Timmy works.' }];
let draftResumed = false;
// Issue #35 — importing a backup reports which version the file came from and what changed, and a
// refused file leaves the ledger exactly as it was. The notice is state, not a toast, because a
// message that disappears in two seconds cannot tell anyone whether their records were replaced.
let portabilityNotice = null;
let form = createDraft();
let followUpAnswers = createFollowUpAnswers();
@@ -85,6 +93,10 @@ function refreshFollowUp(){
function esc(value='') { return String(value).replace(/[&<>'"]/g, c => ({'&':'&amp;','<':'&lt;','>':'&gt;',"'":'&#39;','"':'&quot;'}[c])); }
function loadEntries() { try { return JSON.parse(localStorage.getItem(STORE) || '[]'); } catch { return []; } }
function saveEntries() { localStorage.setItem(STORE, JSON.stringify(entries)); }
// Training contribution receipts are their own store: a journal save never writes here, and a
// withdrawal keeps a tombstone instead of quietly erasing the audit trail.
function loadContributions() { try { return JSON.parse(localStorage.getItem(TRAINING_STORE) || '[]'); } catch { return []; } }
function saveContributions() { localStorage.setItem(TRAINING_STORE, JSON.stringify(contributions)); }
function formatDate(value) { return new Intl.DateTimeFormat(undefined,{month:'short',day:'numeric',hour:'numeric',minute:'2-digit'}).format(new Date(value)); }
function toast(message) { const node=document.createElement('div');node.className='toast';node.textContent=message;document.body.append(node);setTimeout(()=>node.remove(),2400); }
@@ -99,11 +111,51 @@ function bindGlobal(){
document.querySelectorAll('[data-log]').forEach(btn=>btn.onclick=openLogger);
document.querySelectorAll('[data-scan]').forEach(btn=>btn.onclick=openPhotoFirst);
document.querySelectorAll('[data-statement-link]').forEach(btn=>btn.onclick=()=>showPatternRecords(btn.dataset.statementLink));
document.querySelectorAll('[data-entry-open]').forEach(btn=>btn.onclick=()=>showEntrySheet(btn.dataset.entryOpen));
document.querySelectorAll('[data-contribute-entry]').forEach(btn=>btn.onclick=()=>showContributionConsent(btn.dataset.contributeEntry));
document.querySelectorAll('[data-withdraw]').forEach(btn=>btn.onclick=()=>withdrawTrainingContribution(btn.dataset.withdraw));
}
function recentList(limit=5){
if(!entries.length)return `<div class="empty"><img src="${appPath('assets/timmy.svg')}" alt=""><h3>Quiet bowl, clean slate.</h3><p>Your first log takes about ten seconds.</p></div>`;
return entries.slice().sort((a,b)=>new Date(b.occurredAt)-new Date(a.occurredAt)).slice(0,limit).map(e=>`<article class="entry"><div class="type-dot">T${e.bristolType}</div><div><strong>${formatDate(e.occurredAt)}</strong><span>${esc(e.color)} · urgency ${e.urgency}/4 · discomfort ${e.discomfort}/4${e.note?` · ${esc(e.note)}`:''}</span></div><span class="bucket bucket-${bucketForBristolType(e.bristolType)}">${bucketForBristolType(e.bristolType)}</span></article>`).join('');
return entries.slice().sort((a,b)=>new Date(b.occurredAt)-new Date(a.occurredAt)).slice(0,limit).map(e=>{
const photo=photoState(e);
return `<article class="entry"><div class="type-dot">T${e.bristolType}</div><div><strong>${formatDate(e.occurredAt)}</strong><span>${esc(e.color)} · urgency ${e.urgency}/4 · discomfort ${e.discomfort}/4${e.note?` · ${esc(e.note)}`:''}</span>${photo.hasPhoto?`<em class="photo-chip" data-photo-indicator="present">${esc(photo.label)}</em>`:''}</div><span class="entry-tail"><span class="bucket bucket-${bucketForBristolType(e.bristolType)}">${bucketForBristolType(e.bristolType)}</span><button class="entry-open" data-entry-open="${esc(e.id)}" aria-label="Open confirmed log from ${esc(formatDate(e.occurredAt))}, Type ${e.bristolType}"><span aria-hidden="true">›</span></button></span></article>`;
}).join('');
}
// Issue #36 — the saved photo and the confirmed log are independent. Presence is stated without
// rendering a thumbnail, showing the image is an explicit request, removing it keeps every confirmed
// field, and deleting the log is a separate two-step decision that takes the photo with it.
let entrySheetKeyHandler=null;
function showEntrySheet(id, armed=false){
const entry=entries.find(item=>item.id===id);
if(!entry)return;
const photo=photoState(entry);
const urgent=detectUrgentFlags(entry.symptoms);
document.querySelector('.sheet-backdrop')?.remove();
const wrap=document.createElement('div');wrap.className='sheet-backdrop';
wrap.innerHTML=`<section class="sheet" role="dialog" aria-modal="true" aria-labelledby="entry-sheet-title"><div class="sheet-handle"></div><div class="sheet-header"><div><span class="eyebrow">Confirmed log</span><h2 id="entry-sheet-title">${formatDate(entry.occurredAt)}</h2></div><button class="icon-btn" id="entry-sheet-close" aria-label="Close">×</button></div><div class="entry-facts"><div><span>Form</span><strong>Type ${entry.bristolType}</strong></div><div><span>Color</span><strong>${esc(entry.color)}</strong></div><div><span>Urgency</span><strong>${entry.urgency}/4</strong></div><div><span>Discomfort</span><strong>${entry.discomfort}/4</strong></div></div>${entry.note?`<p class="fine entry-note">${esc(entry.note)}</p>`:''}${entry.provenance?`<p class="fine entry-provenance" data-entry-provenance="${esc(entry.provenance.source)}">${esc(entry.provenance.source==='ai-corrected'?'Saved from an AI suggestion you corrected. Your values are stored, not the model’s.':'Saved from an AI suggestion you confirmed: visible form and color only, never a diagnosis.')}${entry.provenance.model?` Model: ${esc(entry.provenance.model)}.`:''}</p>`:''}${urgent.urgent?alertHtml(urgent.message):''}${photo.hasPhoto?`<section class="photo-lifecycle" data-photo-lifecycle="present"><strong>Photo saved with this log</strong><p class="fine">The image stays in this browser. The journal never shows it as a thumbnail until you ask.</p><div id="entry-photo-slot"></div><div class="row"><button class="btn btn-secondary" id="show-entry-photo">Show photo</button><button class="btn btn-ghost" id="remove-entry-photo">Remove photo, keep log</button></div></section>`:`<section class="photo-lifecycle" data-photo-lifecycle="absent"><strong>No photo saved</strong><p class="fine">This confirmed log is text only. Its record does not depend on an image.</p></section>`}<div class="entry-actions">${armed?`<button class="btn btn-danger btn-wide" id="delete-entry-confirm">Tap again to delete this log and its photo</button><p class="fine">This removes the confirmed log and its saved photo from this browser. It cannot be undone.</p>`:`<button class="btn btn-ghost btn-wide" id="delete-entry">Delete this log and its photo</button>`}</div></section>`;
document.body.append(wrap);
if(entrySheetKeyHandler)document.removeEventListener('keydown',entrySheetKeyHandler);
const close=()=>{wrap.remove();document.removeEventListener('keydown',onKey);if(entrySheetKeyHandler===onKey)entrySheetKeyHandler=null};
const onKey=event=>{if(event.key==='Escape')close()};
entrySheetKeyHandler=onKey;
document.addEventListener('keydown',onKey);
document.querySelector('#entry-sheet-close').onclick=close;
wrap.onclick=event=>{if(event.target===wrap)close()};
document.querySelector('#show-entry-photo')?.addEventListener('click',()=>{
const slot=document.querySelector('#entry-photo-slot');
slot.innerHTML=`<img class="photo-preview" id="entry-photo" src="${esc(entry.photoDataUrl)}" alt="Photo saved with this log">`;
document.querySelector('#show-entry-photo')?.remove();
});
document.querySelector('#remove-entry-photo')?.addEventListener('click',()=>{
entries=withoutPhoto(entries,id);saveEntries();render();showEntrySheet(id);toast('Photo removed. Your log is still here.');
});
document.querySelector('#delete-entry')?.addEventListener('click',()=>showEntrySheet(id,true));
document.querySelector('#delete-entry-confirm')?.addEventListener('click',()=>{
entries=withoutEntry(entries,id);saveEntries();close();render();toast('Log and photo deleted');
});
}
function thisWeek(){const now=Date.now(),week=7*864e5;return entries.filter(e=>now-new Date(e.occurredAt).getTime()<week).length}
function currentStreak(){const dates=new Set(entries.map(e=>e.occurredAt.slice(0,10)));let n=0,d=new Date();while(dates.has(d.toISOString().slice(0,10))){n++;d.setDate(d.getDate()-1)}return n}
@@ -206,13 +258,68 @@ async function sendChat(event){
finally{chatBusy=false;timmy()}
}
// Issue #24 — contributing a confirmed photo to training is a second, separate consent. A journal
// save writes nothing here, the consent version and the receipt are on screen before the confirm
// control can be pressed, and a withdrawal keeps a tombstone naming every derived artifact by
// contribution id. Nothing is transmitted in this build: the receipt is queued on the device.
function newContributionId(){
const raw=globalThis.crypto?.randomUUID?.()||`${Date.now().toString(16)}${Math.random().toString(16).slice(2)}`;
return `contrib-${String(raw).replace(/[^a-z0-9]/gi,'').slice(0,12)}`;
}
function trainingConsentSection(){
const terms=consentTerms();
const queued=contributions.filter(receipt=>receipt.state!=='withdrawn').length;
const active=contributions.filter(receipt=>receipt.state!=='withdrawn').map(receipt=>`<div class="contribution-row" data-contribution-state="${esc(receipt.state)}"><div><strong>Type ${receipt.labels.bristolType} · ${esc(receipt.labels.color)}</strong><span>Receipt ${esc(receipt.contributionId)} · consent ${esc(receipt.consentVersion)}</span></div><button class="btn btn-ghost" data-withdraw="${esc(receipt.contributionId)}">Withdraw</button></div>`);
const offerable=entries.filter(entry=>contributionEligibility(entry).eligible&&!contributions.some(receipt=>receipt.entryId===entry.id)).map(entry=>`<div class="contribution-row"><div><strong>${formatDate(entry.occurredAt)}</strong><span>Type ${entry.bristolType} · photo saved</span></div><button class="btn btn-secondary" data-contribute-entry="${esc(entry.id)}">Review opt-in</button></div>`);
const tombstones=contributions.filter(receipt=>receipt.state==='withdrawn').map(receipt=>`<div class="contribution-row withdrawn" data-contribution-state="withdrawn"><div><strong>Withdrawn ${esc(formatDate(receipt.withdrawnAt))}</strong><span data-withdrawn-artifacts>raw image, confirmed label · receipt ${esc(receipt.contributionId)}</span></div></div>`);
const rows=[...active,...offerable].join('');
return `<section class="section card training-consent" data-training-consent><h2>Training contribution</h2><p class="fine">Off by default. Saving a log never contributes it, and nothing is uploaded in this build — a receipt is queued on this device.</p><p class="fine">Consent version <code data-contribution-consent-version>${esc(terms.version)}</code> · <span data-contribution-queued-count>${queued}</span> queued</p>${rows?`<div class="contribution-list">${rows}</div>`:'<p class="fine">No saved photos are available to contribute yet.</p>'}${tombstones.length?`<h3>Withdrawn receipts</h3>${tombstones.join('')}`:''}<p class="fine">Deleting all local data below also clears these receipts.</p></section>`;
}
let contributionSheetKeyHandler=null;
function showContributionConsent(entryId){
const entry=entries.find(item=>item.id===entryId);
const eligibility=contributionEligibility(entry);
if(!eligibility.eligible){toast(eligibility.reason||'This log cannot be contributed.');return}
if(contributionForEntry(contributions,entryId)){toast('This log already has an active contribution receipt.');return}
const terms=consentTerms();
const contributionId=newContributionId();
const consentedAt=new Date().toISOString();
document.querySelector('.sheet-backdrop')?.remove();
const wrap=document.createElement('div');wrap.className='sheet-backdrop';
wrap.innerHTML=`<section class="sheet" role="dialog" aria-modal="true" aria-labelledby="contribution-title" data-contribution-sheet><div class="sheet-handle"></div><div class="sheet-header"><div><span class="eyebrow">Separate consent</span><h2 id="contribution-title">Contribute one photo?</h2></div><button class="icon-btn" id="contribution-cancel" aria-label="Close">×</button></div><div class="consent-card"><p class="fine">Consent version <code data-contribution-version>${esc(terms.version)}</code></p><p class="fine">Receipt <code data-receipt-id>${esc(contributionId)}</code> · ${esc(formatDate(consentedAt))}</p><p class="fine"><strong>Included:</strong> ${esc(terms.includes.join('; '))}.</p><p class="fine"><strong>Never included:</strong> ${esc(terms.excludes.join('; '))}.</p><p class="fine">${esc(terms.transport)}</p></div><label class="check"><input id="training-consent-check" type="checkbox"><span>I have read consent ${esc(terms.version)} and want this one log contributed.</span></label><button class="btn btn-primary btn-wide" id="contribute-confirm" disabled>Contribute this photo and label</button><button class="btn btn-ghost btn-wide section" id="contribution-cancel-bottom">Not now</button></section>`;
document.body.append(wrap);
if(contributionSheetKeyHandler)document.removeEventListener('keydown',contributionSheetKeyHandler);
const close=()=>{wrap.remove();document.removeEventListener('keydown',onKey);if(contributionSheetKeyHandler===onKey)contributionSheetKeyHandler=null};
const onKey=event=>{if(event.key==='Escape')close()};
contributionSheetKeyHandler=onKey;
document.addEventListener('keydown',onKey);
document.querySelector('#contribution-cancel').onclick=close;
document.querySelector('#contribution-cancel-bottom').onclick=close;
wrap.onclick=event=>{if(event.target===wrap)close()};
const check=document.querySelector('#training-consent-check'),confirm=document.querySelector('#contribute-confirm');
check.onchange=()=>{confirm.disabled=!check.checked};
confirm.onclick=()=>{
try{
contributions=addContribution(contributions,receiptPreview(entry,{contributionId,consentedAt}));
saveContributions();close();render();toast('Queued on this device. Nothing was uploaded.');
}catch(error){toast(error.message||'That receipt could not be recorded.')}
};
}
function withdrawTrainingContribution(contributionId){
const result=withdrawContribution(contributions,contributionId,new Date().toISOString());
if(!result.tombstone){toast('That contribution receipt was not found.');return}
contributions=result.contributions;saveContributions();render();toast('Contribution withdrawn. The receipt is kept as a tombstone.');
return result.tombstone;
}
function privacy(){
shell(`<main><div class="page-title"><span class="eyebrow">Private by design</span><h1>Your poop. Your phone.</h1><p>This prototype has no account, analytics, ad tracker, or server database.</p></div><section class="card privacy-list"><div class="privacy-item"><b>⌂</b><div><h3>Stored locally</h3><p>Saved entries and optional photos live in this browser’s local storage.</p></div></div><div class="privacy-item"><b>⇩</b><div><h3>Portable</h3><p>Export a readable JSON file. Import it in another copy of Timmy.</p></div></div><div class="privacy-item"><b>◎</b><div><h3>AI only when you ask</h3><p>Manual logging never uploads. Analyze a photo sends one compressed copy after consent. Hermes chat may receive up to 20 confirmed text-only entries after you connect; photos and backend credentials never enter chat.</p></div></div></section><section class="section card"><h2>Data controls</h2><div class="row"><button class="btn btn-primary" id="export">Export JSON</button><label class="btn btn-secondary" for="import">Import JSON</label><input class="hidden" type="file" id="import" accept="application/json"></div><p class="fine section">Exports can contain sensitive health notes and photos. Store them somewhere you trust.</p></section><section class="section card source-list"><h2>Health sources</h2><p class="fine">The Bristol groupings and urgent-symptom copy are grounded in public clinical guidance.</p><p><a target="_blank" rel="noreferrer" href="https://www.continence.org.au/about-incontinence/bowel-incontinence/bristol-stool-chart/">Continence Health Australia</a></p><p><a target="_blank" rel="noreferrer" href="https://www.nhs.uk/conditions/bleeding-from-the-bottom-rectal-bleeding/">NHS rectal bleeding guidance</a></p><p><a target="_blank" rel="noreferrer" href="https://www.niddk.nih.gov/health-information/digestive-diseases/constipation/symptoms-causes">NIDDK constipation guidance</a></p></section><section class="section card danger-zone"><h2>Delete everything</h2><p class="fine">Permanently removes Timmy’s local ledger from this browser.</p><button class="btn btn-danger" id="delete-all">Delete all local data</button></section></main>`);
shell(`<main><div class="page-title"><span class="eyebrow">Private by design</span><h1>Your poop. Your phone.</h1><p>This prototype has no account, analytics, ad tracker, or server database.</p></div><section class="card privacy-list"><div class="privacy-item"><b>⌂</b><div><h3>Stored locally</h3><p>Saved entries and optional photos live in this browser’s local storage. You can remove one photo without deleting its confirmed log.</p></div></div><div class="privacy-item"><b>⇩</b><div><h3>Portable</h3><p>Export a readable JSON file. Import it in another copy of Timmy.</p></div></div><div class="privacy-item"><b>◎</b><div><h3>AI only when you ask</h3><p>Manual logging never uploads. Analyze a photo sends one compressed copy after consent. Hermes chat may receive up to 20 confirmed text-only entries after you connect; photos and backend credentials never enter chat.</p></div></div></section><section class="section card"><h2>Data controls</h2><div class="row"><button class="btn btn-primary" id="export">Export JSON</button><label class="btn btn-secondary" for="import">Import JSON</label><input class="hidden" type="file" id="import" accept="application/json"></div><p class="fine section">Exports can contain sensitive health notes and photos. Store them somewhere you trust.</p></section>${portabilityNoticeBlock()}${trainingConsentSection()}<section class="section card source-list"><h2>Health sources</h2><p class="fine">The Bristol groupings and urgent-symptom copy are grounded in public clinical guidance.</p><p><a target="_blank" rel="noreferrer" href="https://www.continence.org.au/about-incontinence/bowel-incontinence/bristol-stool-chart/">Continence Health Australia</a></p><p><a target="_blank" rel="noreferrer" href="https://www.nhs.uk/conditions/bleeding-from-the-bottom-rectal-bleeding/">NHS rectal bleeding guidance</a></p><p><a target="_blank" rel="noreferrer" href="https://www.niddk.nih.gov/health-information/digestive-diseases/constipation/symptoms-causes">NIDDK constipation guidance</a></p></section><section class="section card danger-zone"><h2>Delete everything</h2><p class="fine">Permanently removes Timmy’s local ledger and every saved photo from this browser.</p><button class="btn btn-danger" id="delete-all">Delete all local data</button></section></main>`);
document.querySelector('#export').onclick=exportData;document.querySelector('#import').onchange=importData;document.querySelector('#delete-all').onclick=deleteData;
}
function exportData(){const blob=new Blob([exportLedger(entries)],{type:'application/json'}),a=document.createElement('a');a.href=URL.createObjectURL(blob);a.download='timmy-ledger.json';a.click();URL.revokeObjectURL(a.href);toast('Export created');}
async function importData(e){try{const text=await e.target.files[0].text();entries=importLedger(text);saveEntries();render();toast('Ledger imported')}catch(err){toast(err.message)}}
function deleteData(){if(confirm('Delete every local Timmy entry and photo? This cannot be undone.')){entries=[];localStorage.removeItem(STORE);localStorage.removeItem(LEGACY_STORE);render();toast('Local ledger deleted')}}
function exportData(){const blob=new Blob([exportLedgerBackup({entries,contributions})],{type:'application/json'}),a=document.createElement('a');a.href=URL.createObjectURL(blob);a.download=`timmy-backup-v${LEDGER_EXPORT_VERSION}.json`;a.click();URL.revokeObjectURL(a.href);portabilityNotice={kind:'ok',version:LEDGER_EXPORT_VERSION,text:`Backup written as version ${LEDGER_EXPORT_VERSION}: ${entries.length} log${entries.length===1?'':'s'} and ${contributions.length} training receipt${contributions.length===1?'':'s'}.`};render();toast('Backup exported');}
async function importData(e){const file=e.target.files?.[0];if(!file)return;try{const result=importLedgerBackup(await file.text());entries=result.entries;contributions=result.contributions;saveEntries();saveContributions();portabilityNotice={kind:'ok',version:result.fromVersion,text:[`Restored ${result.entries.length} log${result.entries.length===1?'':'s'} and ${result.contributions.length} training receipt${result.contributions.length===1?'':'s'} from a version ${result.fromVersion} backup.`,...result.notices].join(' ')};e.target.value='';render();toast('Backup imported')}catch(err){portabilityNotice={kind:'error',text:`${err.message||'That backup could not be read.'} Your ledger is unchanged.`};e.target.value='';render();toast('Nothing was imported')}}
function portabilityNoticeBlock(){if(!portabilityNotice)return '';if(portabilityNotice.kind==='error')return `<p class="portability-notice error" role="alert" data-portability-error>${esc(portabilityNotice.text)}</p>`;return `<p class="portability-notice" role="status" data-portability-notice data-portability-version="${esc(String(portabilityNotice.version))}">${esc(portabilityNotice.text)}</p>`}
function deleteData(){if(confirm('Delete every local Timmy entry and photo? This cannot be undone.')){entries=[];contributions=[];portabilityNotice=null;localStorage.removeItem(STORE);localStorage.removeItem(LEGACY_STORE);localStorage.removeItem(TRAINING_STORE);render();toast('Local ledger and photos deleted')}}
function openPhotoFirst(){beginDraft();photoHint='';resetSuggestion();visionStatus=null;showPhotoFirst('pick');loadVisionStatus()}
async function loadVisionStatus(){
@@ -269,7 +376,21 @@ function bindStep(step){
}
async function handlePhoto(e){const file=e.target.files[0];if(!file)return;try{const result=await compressPhoto(file);form.photoDataUrl=result.dataUrl;photoHint=photoQualityMessage(result);showLogStep(2)}catch{photoHint='That image could not be read. Try another photo.';showLogStep(2)}}
function compressPhoto(file){return new Promise((resolve,reject)=>{const img=new Image(),url=URL.createObjectURL(file);img.onload=()=>{const scale=Math.min(1,1200/Math.max(img.width,img.height)),canvas=document.createElement('canvas');canvas.width=Math.round(img.width*scale);canvas.height=Math.round(img.height*scale);const ctx=canvas.getContext('2d');ctx.drawImage(img,0,0,canvas.width,canvas.height);const sample=ctx.getImageData(0,0,Math.min(canvas.width,120),Math.min(canvas.height,120)).data;let total=0;for(let i=0;i<sample.length;i+=4)total+=(sample[i]+sample[i+1]+sample[i+2])/3;URL.revokeObjectURL(url);resolve({dataUrl:canvas.toDataURL('image/jpeg',.7),width:img.width,height:img.height,brightness:total/(sample.length/4)/255})};img.onerror=()=>{URL.revokeObjectURL(url);reject()};img.src=url})}
function saveLog(){const result=detectUrgentFlags(form.symptoms);const entry=sanitizeEntry({...form});try{entries.push(entry);saveEntries()}catch{entry.photoDataUrl='';entries[entries.length-1]=entry;saveEntries();toast('Log saved, but the photo was too large for browser storage')}clearDraft();document.querySelector('.sheet-backdrop')?.remove();view='home';render();toast(result.urgent?'Saved. Please follow the medical-care alert.':'Private log saved')}
// Issue #35 — if this log began as an AI suggestion, the saved record says so. The fields are
// limited to what the app actually knows (which visible fields were suggested, whether they were
// confirmed or corrected, and which model answered), so a backup cannot be read as a diagnosis or a
// broader claim than the visual suggestion the person accepted.
function provenanceForSave(){
const status=suggestionState?.status;
if(status!=='confirmed'&&status!=='corrected')return null;
const suggestion=suggestionState.suggestion;
if(!suggestion)return null;
const provenance={source:status==='confirmed'?'ai-confirmed':'ai-corrected',suggestedBristolType:suggestion.bristolType,suggestedColor:suggestion.color,confirmedAt:new Date().toISOString()};
const model=typeof visionStatus?.model==='string'?visionStatus.model.trim():'';
if(model)provenance.model=model.slice(0,80);
return provenance;
}
function saveLog(){const result=detectUrgentFlags(form.symptoms);const entry=sanitizeEntry({...form,provenance:provenanceForSave()});try{entries.push(entry);saveEntries()}catch{entry.photoDataUrl='';entries[entries.length-1]=entry;saveEntries();toast('Log saved, but the photo was too large for browser storage')}clearDraft();document.querySelector('.sheet-backdrop')?.remove();view='home';render();toast(result.urgent?'Saved. Please follow the medical-care alert.':'Private log saved')}
function render(){({home,calendar,timmy,privacy}[view]||home)()}
render();
+61
View File
@@ -0,0 +1,61 @@
# Corpus manifest contract
The training corpus is described by an **append-only** `manifest.jsonl`. One line is one event;
nothing is ever rewritten in place, so the trail can be audited after a withdrawal. The
implementation lives in [`scripts/corpus_manifest.py`](../scripts/corpus_manifest.py) and is
driven by [`scripts/ingest_training_photo.py`](../scripts/ingest_training_photo.py).
## Sample record (`recordType: "sample"`, schema version 2)
| Field | Meaning |
|---|---|
| `contributionId` | Opaque id minted by the app's contribution receipt. The same id addresses every derived artifact and every later tombstone. |
| `consent.version`, `consent.recordedAt` | The consent version the contributor accepted, and when. Duplicated as `consentVersion` for older readers. |
| `source.kind`, `source.inputSha256` | How the bytes arrived (`cli-ingest`) and the digest of the caller's input. The caller's path is never recorded. |
| `hashes.rawInput`, `hashes.derived` | Digest of the supplied bytes and of the re-encoded image that is actually stored. |
| `derivation` | `format`, `maxDimension`, `metadataStripped`, and the `pipeline` version that produced the derived image. |
| `review.status`, `review.reviewedAt` | `user-confirmed` or `clinician-reviewed`. A label set is never unlabelled-by-default. |
| `split` | `train`, `validation`, or `test`, derived from the salted subject key so one contributor never straddles splits. |
| `subjectKey` | 16-hex pseudonym derived from a salted subject id. Never a name, email, or account. |
| `lifecycle.state`, `lifecycle.updatedAt` | `active` while the sample is usable for training. |
| `imageId`, `relativePath` | Derived image digest and its dataset-relative path. |
## Tombstone record (`recordType: "tombstone"`)
A withdrawal never deletes history; it appends a tombstone that:
- keeps `contributionId` and `imageId` so the withdrawal can be matched to the original sample,
- records `lifecycle.state: "revoked"`, the timestamp, and the reason,
- lists `deletedArtifacts` — every derived copy it removed, addressed by contribution id with the
digest that identifies the bytes,
- deliberately drops `labels`, `subjectKey`, and `relativePath`: a deletion receipt keeps
identifiers and hashes, not the health label or the person.
`revoke()` removes the derived image and any raw pre-derivation copy from the active dataset
before appending the tombstone, and refuses any path that resolves outside the dataset root.
## Lifecycle rules (all covered by `tests/corpus-manifest.test.py`)
- `active` → `revoked` only, and only with a recorded reason.
- `revoked` → `active` is refused: a revoked contribution cannot return to the active dataset.
- Revoking an unknown contribution, or revoking twice, is refused and appends nothing.
- A contribution id can be ingested once; the same derived image cannot carry two label sets.
- `active_records()` returns only samples that are still `active` and were never tombstoned, so a
trainer that reads through it cannot silently train on withdrawn data.
## Usage
```bash
export TIMMY_DATASET_SALT=... # >= 8 chars, never committed
python3 scripts/ingest_training_photo.py \
--input photo.jpg --output-dir /srv/timmy-corpus \
--subject-id subject-123 --consent-version research-v1 \
--contribution-id "$CONTRIBUTION_ID" \
--bristol-type 4 --color brown --quality good --reviewer user-confirmed
python3 scripts/ingest_training_photo.py --revoke "$CONTRIBUTION_ID" \
--output-dir /srv/timmy-corpus --reason "subject withdrawal"
```
`npm run test:corpus` runs the contract suite. Every fixture is synthetic; no real medical image
is read, written, or committed.
+113
View File
@@ -0,0 +1,113 @@
# Evaluation harness contract
Issue #29 asks for the authoritative comparison and release-gate harness: macro-F1, per-class
precision/recall, a confusion matrix, calibration error, coverage, and precision-at-coverage, over
splits that are contributor-held-out and reproducible, comparing candidate models.
The implementation is [`scripts/calibration_report.py`](../scripts/calibration_report.py). It reads
predictions something else produced, and it refuses to compute gates it cannot honestly compute.
**This harness does not train, does not diagnose, and never reads an image.** It only sees Bristol
labels, predictions, and confidences.
## Input
One prediction record per line (JSONL), one file or many:
```json
{"model":"specialist-candidate","subjectKey":"cd1a2b3c4d5e6f01","split":"validation",
"truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.97}}
```
| Field | Rule |
|---|---|
| `model` | Non-empty. Candidates are grouped by this value, so one file may carry several. |
| `subjectKey` | The 16-hex pseudonymous key from the corpus manifest. Never a name, email, or account. |
| `split` | `train`, `validation`, or `test`. |
| `truth.bristolType` | Whole number 1–7. |
| `prediction` | `{bristolType: 1–7, confidence: 0–1}`, or `null` when the candidate offered nothing. |
A record that breaks any row above is refused (exit 2) rather than scored.
## Splits are contributor-held-out and reproducible
The split rule is the ingestion rule from `scripts/ingest_training_photo.py`: the first byte of the
subject key picks the bucket — below `205` is `train`, below `230` is `validation`, otherwise `test`.
Because the rule reads only the pseudonymous key, a split can be re-derived by anyone holding the
manifest, and a corpus cannot quietly re-label itself.
`assert_contributor_held_out()` refuses:
- a `split` that does not match the one the subject key derives (not reproducible), and
- one subject key appearing in more than one split (contributor leak) — a held-out score where the
model has already seen that person is not held out.
Several photos from one contributor inside one split are fine; that is the normal case.
## Metrics
The harness is a *selective* classifier evaluation. A suggestion is **accepted** when it exists and
its confidence is at or above the threshold; everything else is an abstention. Two abstention
reasons are counted separately: `no-suggestion` and `low-confidence`.
| Metric | Definition |
|---|---|
| `coverage` | accepted / total in the scored split. |
| `precisionAtCoverage` | correct / accepted. The gate metric. |
| `perClass[label]` | `precision`, `recall`, `f1`, `support`, `predicted` over accepted suggestions. |
| `macroF1` | Mean of `perClass[*].f1` over the classes present in **accepted** truth. A class whose every photo abstained cannot be scored, so it is named in `excludedClasses` instead of being counted as a silent zero. |
| `confusion` | `"truth->predicted": count` over accepted suggestions. |
| `ece` | Expected calibration error over accepted suggestions in ten equal-width confidence bins, weighted by each bin's share. |
Calibration is measured on the same accepted set as precision, so a candidate cannot look calibrated
by abstaining out of its worst bins.
## Threshold protocol
`select_threshold()` takes the coverage floor and returns the threshold with the highest
precision-at-coverage among the cuts that still clear that floor, breaking ties toward wider coverage.
It reads the **validation split only** and raises otherwise: choosing a threshold on the data the gate
is measured on manufactures whatever precision the report needs.
`compare()` therefore fixes one threshold per candidate on validation, scores that candidate on test,
and never looks back at validation scores.
## Gates
Taken from the ROADMAP release gates. All limits are inclusive — a candidate exactly on the line passes.
| Gate | Limit |
|---|---|
| `macroF1` | ≥ 0.80 |
| `precisionAtCoverage` | ≥ 0.90 |
| `ece` | ≤ 0.05 |
A candidate must clear all three on its held-out split. `verdict` is `GO` only when at least one
candidate does; `recommended` is the passing candidate with the highest macro-F1 (then precision).
Otherwise the verdict is `BLOCKED` and no candidate is recommended.
## Run it
```bash
python3 scripts/calibration_report.py \
--predictions tests/fixtures/calibration/predictions.jsonl \
--min-coverage 0.5 --json /tmp/calibration-report.json
npm run test:calibration # the harness contract suite
```
| Exit code | Meaning |
|---|---|
| 0 | `GO` — a candidate cleared every gate. |
| 3 | `BLOCKED` — the release gate refuses every candidate. |
| 2 | Refused — malformed, leaky, or unreproducible input. Nothing was scored or written. |
`--json PATH` writes the full report (per-class rows, confusion matrix, calibration bins, gate detail).
The report is only written after a successful run.
## Fixtures
[`tests/fixtures/calibration/`](../tests/fixtures/calibration/README.md) holds synthetic predictions
that exercise `BLOCKED`, `GO`, and the leaked-corpus refusal. They are invented numbers on fabricated
subject keys: **no model produced them and they say nothing about the real classifier.** Recording a
video or writing a report from them means saying so.
+18
View File
@@ -34,6 +34,14 @@ The application’s deterministic urgent-symptom guidance runs before chat and r
Logging must never depend on AI availability. When the vision worker is offline, disabled, or still unknown, the photo-first flow offers no AI action and no analysis-consent prompt, and manual Bristol entry remains one action away from every photo step. A chosen photo and any entered context are retained as an in-session draft with a visible summary and an explicit start over; drafts are deliberately not persisted to storage, so a photo awaiting consent is never written to disk. The service worker never caches analysis requests or image bodies.
### Training contribution consent
Contributing a confirmed photo to training is a second consent, not a side effect of logging. A journal save is never a training contribution; a contribution exists only after the person reads the current consent version and its receipt (contribution id, what is included, what is never included) and accepts it explicitly. The receipt carries the consent version it was accepted under, the confirmed Bristol form and broad color, and no note text, symptom answers, urgency/discomfort ratings, identity, or photo bytes.
Withdrawal is first-class: withdrawal keeps a tombstone naming every derived artifact by contribution id (raw image and confirmed label), so a deletion request can be addressed against the corpus rather than trusted to a promise. Delete-all clears the ledger, saved photos, and contribution receipts together.
This build acknowledges that records the intent and does not transmit a contribution until the corpus pipeline and privacy copy are reviewed in #22 and #25; the section says so in the app rather than implying an upload that does not exist.
### Release authority
Hermes/Timmy is the release authority for routine engineering, test, build, deployment, and release-candidate preparation. Automated evidence must remain reproducible and must not overstate model accuracy or safety.
@@ -48,6 +56,16 @@ Human gates are limited to:
Routine provisioning, deployment, label sweeps, and release-candidate assembly are not human-operated gates.
### Backup portability and entry provenance
A backup is the person's own record and must keep working as the AI fields evolve.
- Every export stamps a schema version (currently version 2). The same file carries the confirmed logs and the training contribution receipts, including withdrawn tombstones.
- An older backup is upgraded on import, and the app says which version the file came from. Fields that did not exist when the file was written are left empty rather than invented.
- A backup from a newer version is refused with an instruction to update Timmy, and a refused import leaves the ledger unchanged.
- Malformed, oversized, and over-limit files fail before anything is replaced.
- Provenance is stored on the confirmed log when a saved entry began as a confirmed or corrected AI suggestion: source, suggested visible fields, the model that answered, and when it was confirmed. It never records a diagnosis, a symptom, or provider credentials, and backups are written through an allowlist so unknown keys never leave the device.
## Consequences
- Model output is assistive and fail-closed, never authoritative.
+8 -3
View File
@@ -4,7 +4,7 @@
"private": true,
"type": "module",
"scripts": {
"test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/draft.test.js tests/suggestion.test.js tests/followup.test.js tests/vision-readiness.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js tests/threat-model.test.js tests/vision-corpus.test.js tests/vision-boundary.test.js tests/worker-benchmark.test.js tests/worker-ready.test.js tests/worker-container.test.js tests/summary.test.js tests/escalation-regression.test.js tests/grounded-chat.test.js",
"test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/draft.test.js tests/suggestion.test.js tests/followup.test.js tests/vision-readiness.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js tests/threat-model.test.js tests/vision-corpus.test.js tests/vision-boundary.test.js tests/worker-benchmark.test.js tests/worker-ready.test.js tests/worker-container.test.js tests/summary.test.js tests/escalation-regression.test.js tests/grounded-chat.test.js tests/photo-lifecycle.test.js tests/training-contribution.test.js tests/portability.test.js",
"test:ui": "node tests/ui.acceptance.mjs",
"test:photo": "node tests/photo-first.acceptance.mjs",
"test:mobile-capture": "node tests/mobile-capture.acceptance.mjs",
@@ -16,10 +16,15 @@
"test:staging-smoke": "node tests/staging.acceptance.mjs",
"test:pattern-summary": "node tests/pattern-summary.acceptance.mjs",
"test:grounded-chat": "node tests/grounded-chat.acceptance.mjs",
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/grounded-chat.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py && node --check scripts/record_worker_benchmark.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh",
"test:photo-lifecycle": "node tests/photo-lifecycle.acceptance.mjs",
"test:training-contribution": "node tests/training-contribution.acceptance.mjs",
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py && node --check scripts/record_worker_benchmark.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh",
"check:diff": "bash scripts/check_diff.sh",
"start": "node server.mjs",
"test:worker-readiness": "node tests/worker-readiness.acceptance.mjs"
"test:worker-readiness": "node tests/worker-readiness.acceptance.mjs",
"test:portability": "node tests/portability.acceptance.mjs",
"test:corpus": "python3 tests/corpus-manifest.test.py",
"test:calibration": "python3 tests/calibration-harness.test.py"
},
"devDependencies": {
"playwright": "^1.62.1"
+76 -6
View File
@@ -9,6 +9,7 @@ import os
from pathlib import Path
import re
import shutil
import socket
import subprocess
import sys
import tarfile
@@ -34,7 +35,10 @@ DEMO_REQUIRED_CAPTIONS = (
"Type 2 is hard or lumpy, so Timmy asks about soreness, hydration, blood, and black-or-tarry — never guessed from the photo",
"Vision worker offline — the AI and consent steps disappear instead of failing after upload",
"Every byte-bearing control the model names, in the running app: local ledger, consent-gated upload, portable export, full deletion",
"Delete all local data clears the ledger in this browser; there is no server copy of it to keep",
"Photo presence is a label, not a thumbnail — nothing sensitive is on screen until you ask",
"Showing the photo is your explicit request, from this browser’s own storage",
"Remove just the photo: the confirmed log, form, and note are all still here",
"Delete all local data clears the ledger and every saved photo in this browser; nothing is kept on a server",
"Hermes Agent connected — free-text, contextual, and server-side",
"Urgent language is intercepted deterministically before Hermes",
"Bleeding reported as wiping still escalates, in the app’s own fixed words",
@@ -48,6 +52,21 @@ DEMO_REQUIRED_CAPTIONS = (
"The answer states the count and cites the four confirmed logs it counted",
"Following a citation opens exactly the confirmed log behind that answer",
"Ask something the journal cannot answer and Timmy refuses instead of reassuring",
"Training contribution is off by default: a saved log is never training data",
"The consent version and the receipt are on screen before the contribute button works",
"Withdrawing keeps a tombstone naming every derived artifact by contribution ID",
"The corpus pipeline is a command-line tool, not a screen in the app — this runs the real ingest against the committed synthetic fixture",
"Every ingested sample records consent, source digest, both hashes, derivation, review, and lifecycle — the caller file path is never stored",
"Withdrawing deletes the derived image and any raw copy, keeps the original record, and appends a tombstone — the trainer sees zero active samples",
"Revoking the same contribution twice is refused and appends nothing: the manifest only ever grows",
"The calibration harness is a command-line release gate, not a screen in the app — this runs it against committed synthetic predictions",
"Every score on this card is invented to exercise the arithmetic: no model produced them and none of them describes the real classifier",
"Each candidate is scored on test at a threshold chosen on validation only; picking a threshold on the split you report is refused",
"One contributor spread across two splits is refused before anything is scored, and a passing verdict is proven only by a fixture engineered to clear it",
"A backup is versioned, and it carries the withdrawn receipt as a tombstone rather than silence",
"An older backup is upgraded on import, and the notice says which version the file came from",
"A file from a newer Timmy is refused with an instruction, and the ledger it would have replaced is untouched",
"A restored log still says it began as an AI suggestion you corrected — provenance is part of the record",
)
@@ -66,6 +85,29 @@ def sha256(path: Path) -> str:
return digest.hexdigest()
def assert_port_free(host: str, port: int) -> None:
"""Fail closed when something is already listening where the acceptance server must bind."""
with socket.socket() as sock:
sock.settimeout(1.0)
if sock.connect_ex((host, port)) == 0:
raise SystemExit(
f"Port {port} is already served by another process; refusing to gate or record this "
"release against a tree that is not the one under test."
)
def assert_served_tree(tree: Path, host: str, port: int) -> None:
"""Prove the responding server is the clone under test, not a stale checkout on the same port."""
for relative in ("src/domain.js", "app.js"):
served = urllib.request.urlopen(f"http://{host}:{port}/{relative}", timeout=5).read()
expected = (tree / relative).read_bytes()
if hashlib.sha256(served).hexdigest() != hashlib.sha256(expected).hexdigest():
raise SystemExit(
f"The acceptance server is serving {relative} from a different tree; "
"every gate and the demo would describe a release that is not this commit."
)
def tracked_files(tree: Path) -> list[str]:
names = run(["git", "ls-files"], tree, capture=True).splitlines()
return sorted(name for name in names if name and not name.startswith(EXCLUDED_PREFIXES)
@@ -121,8 +163,17 @@ def main() -> int:
run(["npm", "test"], tree)
# The committed corpus hashes are an integrity receipt: verify them before anything is built.
run(["python3", "scripts/build_vision_corpus.py", "--verify"], tree)
# Issue #25's corpus pipeline is a CLI, so its own suite is a release gate, not just CI.
run(["python3", "tests/corpus-manifest.test.py", "-v"], tree)
# Issue #29's evaluation harness decides GO/BLOCKED for a model, so its refusal and gate
# arithmetic are exercised before this release is allowed to exist.
run(["python3", "tests/calibration-harness.test.py", "-v"], tree)
server_env = dict(os.environ)
server_env.update({"HOST": "127.0.0.1", "PORT": "4173", "TIMMY_VISION_ENABLED": "false"})
# A leftover server from another checkout answers on this port and the readiness probe cannot
# tell it apart from ours, so every gate and the recorded demo would silently describe a
# different tree. Refuse to build over a live listener, and verify the served bytes afterwards.
assert_port_free("127.0.0.1", 4173)
server_log = (release_dir / "acceptance-server.log").open("w", encoding="utf-8")
server = subprocess.Popen(["node", "server.mjs"], cwd=tree, env=server_env,
stdout=server_log, stderr=subprocess.STDOUT, text=True)
@@ -137,6 +188,9 @@ def main() -> int:
time.sleep(0.1)
else:
raise SystemExit("Acceptance server did not become ready")
if server.poll() is not None:
raise SystemExit("Acceptance server exited after reporting ready")
assert_served_tree(tree, "127.0.0.1", 4173)
run(["npm", "run", "test:ui"], tree)
run(["npm", "run", "test:photo"], tree)
run(["npm", "run", "test:mobile-capture"], tree)
@@ -148,6 +202,9 @@ def main() -> int:
run(["npm", "run", "test:worker-readiness"], tree)
run(["npm", "run", "test:pattern-summary"], tree)
run(["npm", "run", "test:grounded-chat"], tree)
run(["npm", "run", "test:photo-lifecycle"], tree)
run(["npm", "run", "test:training-contribution"], tree)
run(["npm", "run", "test:portability"], tree)
demo_raw = release_dir / f"timmy-talking-turd-{version}-demo.raw.webm"
caption_receipt = release_dir / f"timmy-talking-turd-{version}-demo-captions.json"
demo_env = dict(server_env)
@@ -159,11 +216,17 @@ def main() -> int:
raise SystemExit("Release demo produced no on-camera caption receipt")
recorded = json.loads(caption_receipt.read_text(encoding="utf-8"))
onscreen = [str(line) for line in (recorded.get("captions") or [])]
if recorded.get("releaseDelta") != release_delta or not any(release_delta in line for line in onscreen):
# A pushed string is not proof of pixels. Only captions the demo measured as a painted,
# on-top, in-frame bar (style + elementFromPoint + band pixel change) count as on camera.
verified = [str(item.get("text")) for item in (recorded.get("captionsVerified") or [])
if item.get("visible") is True and item.get("bandChanged") is True]
if recorded.get("releaseDelta") != release_delta or not any(release_delta in line for line in verified):
raise SystemExit("Release demo did not state the declared release delta on camera")
for required in DEMO_REQUIRED_CAPTIONS:
if not any(required in line for line in onscreen):
raise SystemExit(f"Release demo is missing a required on-camera caption: {required}")
if not any(required in line for line in verified):
raise SystemExit(f"Release demo caption never painted on screen: {required}")
finally:
server.terminate()
try:
@@ -189,7 +252,7 @@ def main() -> int:
if not contact_sheet.exists() or contact_sheet.stat().st_size == 0:
raise SystemExit("Release demo contact sheet was not produced")
run(["npm", "audit", "--audit-level=high"], tree)
for file in ("app.js", "server.mjs", "service-worker.js", "src/analysis.js", "src/domain.js", "src/draft.js", "src/suggestion.js", "src/followup.js", "src/vision-readiness.js", "src/hermes-agent-service.js", "src/vision-config.js", "src/vision-service.js", "src/vision-corpus.js", "src/summary.js", "scripts/record_worker_benchmark.mjs", "tests/staging.acceptance.mjs", "tests/vision-refusal.acceptance.mjs", "tests/worker-readiness.acceptance.mjs", "tests/pattern-summary.acceptance.mjs"):
for file in ("app.js", "server.mjs", "service-worker.js", "src/analysis.js", "src/domain.js", "src/draft.js", "src/suggestion.js", "src/followup.js", "src/vision-readiness.js", "src/hermes-agent-service.js", "src/vision-config.js", "src/vision-service.js", "src/vision-corpus.js", "src/summary.js", "src/photo-lifecycle.js", "src/training-contribution.js", "scripts/record_worker_benchmark.mjs", "tests/staging.acceptance.mjs", "tests/vision-refusal.acceptance.mjs", "tests/worker-readiness.acceptance.mjs", "tests/pattern-summary.acceptance.mjs", "tests/photo-lifecycle.acceptance.mjs", "tests/training-contribution.acceptance.mjs"):
run(["node", "--check", file], tree)
run(["bash", "-n", "scripts/bootstrap_selfhost_smolvlm.sh"], tree)
run(["bash", "-n", "scripts/run_selfhost_smolvlm.sh"], tree)
@@ -197,7 +260,7 @@ def main() -> int:
run(["bash", "-n", "deploy/worker/entrypoint.sh"], tree)
run(["bash", "-n", "deploy/worker/build_image.sh"], tree)
run(["bash", "-n", "deploy/worker/run_worker_container.sh"], tree)
run(["python3", "-m", "py_compile", "scripts/ingest_training_photo.py", "scripts/build_release.py", "scripts/build_vision_corpus.py"], tree)
run(["python3", "-m", "py_compile", "scripts/ingest_training_photo.py", "scripts/build_release.py", "scripts/build_vision_corpus.py", "scripts/corpus_manifest.py"], tree)
run(["git", "diff", "--check", commit], tree)
names = tracked_files(tree)
@@ -229,7 +292,10 @@ def main() -> int:
"- Post-load readiness: the app offers AI only once the pinned model is actually loaded, so a reachable-but-unloaded worker falls back to manual logging in one tap instead of failing after the upload (issue #15)\n"
"- Target-GPU benchmark: scripts/record_worker_benchmark.mjs records cold start, warm latency, and VRAM on the target GPU and refuses to write a receipt it did not measure; no GPU measurement is bundled with this release\n"
"- Explainable pattern summaries (issue #34): the home insight card reports confirmed-log counts and form groupings per window, every line names the exact confirmed records behind it, following a citation opens those records, and a window below the minimum says it is too sparse instead of describing a trend; no line explains a cause, rules on food, or offers a health verdict\n"
"- Safety: suggestions are observable-field assistance, never diagnosis; user confirmation is required\n"
"- Private photo lifecycle (issue #36): the journal marks a log that holds a saved photo without rendering a thumbnail, showing the image is an explicit request, removing it keeps every confirmed field, deleting a log is a separate two-step decision that takes its photo, and delete-all removes the ledger and every stored photo together\n"
"- Training contribution is a separate opt-in (issue #24): a journal save records no contribution, the consent version and the receipt are on screen before the control can be pressed, the receipt is queued on this device only (this build transmits nothing) and carries no note text, symptom answers, or photo bytes, and a withdrawal keeps a tombstone naming every derived artifact by contribution id\n"
"- Auditable corpus pipeline (issue #25): every ingested training sample now records its consent version and timestamp, source kind and the raw-input digest (the caller's file path is never stored), both the raw and derived hashes, the derivation pipeline, the human review status, the split, and its lifecycle state in an append-only manifest; withdrawing a contribution deletes the derived image and any raw pre-derivation copy from the active dataset, keeps the original record as its own history, and appends a tombstone that names every removed artifact by contribution id and digest while dropping labels and the subject key; illegal lifecycle moves (re-ingesting a contribution, re-labelling a derived image, revoking twice, revoking an unknown contribution, reviving a revoked one, revoking without a reason) are refused before any byte is written\n"
"- Scope note: the corpus pipeline is a command-line tool. There is no corpus screen in the app, and this release does not imply one — the recorded demo runs the real ingest and revoke commands against the committed synthetic fixture and shows their actual output\n"
"- Known limitation: open-weight VLM acceptance is proven, Bristol accuracy is not clinically validated\n"
"- Package excludes model weights, medical-image corpora, raw research pages, generated screenshots, and video production assets\n"
)
@@ -274,7 +340,7 @@ def main() -> int:
"codec": "h264",
"pixel_format": "yuv420p",
"recorded_from_working_app": True,
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources",
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything",
},
"gates": {
"unit_security": "passed",
@@ -289,6 +355,10 @@ def main() -> int:
"worker_post_load_readiness": "passed",
"longitudinal_pattern_summary": "passed",
"grounded_chat_citations": "passed",
"training_contribution_consent": "passed",
"corpus_manifest_provenance": "passed",
"calibration_harness_gates": "passed",
"portable_backup_restore": "passed",
"dependency_audit_high": "passed",
"syntax": "passed",
"secret_scan": "passed",
+378
View File
@@ -0,0 +1,378 @@
#!/usr/bin/env python3
"""Contributor-held-out evaluation harness for the Timmy Bristol classifier.
This is the comparison and release-gate tool for issue #29: it turns candidate predictions into the
numbers the ROADMAP gates are written in — macro-F1, per-class precision/recall, a confusion matrix,
calibration error, coverage, and precision-at-coverage — and it refuses a corpus whose splits are
not contributor-held-out or not reproducible.
Nothing here trains, scores, or diagnoses: it reads predictions that something else produced, and it
reports what those predictions would have done on held-out contributors. Every fixture used to test
it is synthetic.
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from pathlib import Path
BRISTOL_CLASSES = tuple(range(1, 8))
SPLITS = ("train", "validation", "test")
# The split rule is the ingestion rule (scripts/ingest_training_photo.py): the bucket of the first
# byte of the pseudonymous subject key. Keeping it here means a split can be re-derived from the
# key alone, so a corpus cannot quietly re-label its own splits.
_TRAIN_EDGE = 205
_VALIDATION_EDGE = 230
_SHA_LIKE = re.compile(r"^[0-9a-f]{16}$")
# The ROADMAP release gates. A candidate must clear every one on its held-out split; a limit is
# inclusive, so a model exactly on the line passes.
RELEASE_GATES = {"macroF1": 0.80, "precisionAtCoverage": 0.90, "ece": 0.05}
class EvaluationError(Exception):
"""The corpus or the caller asked for a comparison that cannot be trusted."""
def expected_split(subject_key: str) -> str:
"""The split a pseudonymous subject key must belong to, by the ingestion rule."""
if not isinstance(subject_key, str) or not _SHA_LIKE.match(subject_key):
raise EvaluationError("subject key must be the 16-character pseudonymous digest")
bucket = int(subject_key[:2], 16)
if bucket < _TRAIN_EDGE:
return "train"
if bucket < _VALIDATION_EDGE:
return "validation"
return "test"
def _calibration_error(accepted: list[tuple[int, int, float]], bins: int = 10) -> tuple[float, list[dict]]:
"""Expected calibration error over the accepted suggestions, by equal-width confidence bin."""
if not accepted:
return 0.0, []
edges = [index / bins for index in range(bins + 1)]
buckets: list[list[tuple[int, int, float]]] = [[] for _ in range(bins)]
for truth, predicted, confidence in accepted:
index = min(int(confidence * bins), bins - 1)
buckets[index].append((truth, predicted, confidence))
error = 0.0
detail: list[dict] = []
for index, bucket in enumerate(buckets):
if not bucket:
continue
accuracy = sum(1 for truth, predicted, _ in bucket if truth == predicted) / len(bucket)
mean_confidence = sum(confidence for _, _, confidence in bucket) / len(bucket)
share = len(bucket) / len(accepted)
error += share * abs(accuracy - mean_confidence)
detail.append(
{
"lower": edges[index],
"upper": edges[index + 1],
"count": len(bucket),
"accuracy": accuracy,
"meanConfidence": mean_confidence,
}
)
return error, detail
def assert_contributor_held_out(records: list[dict]) -> None:
"""Refuse a corpus whose splits are not contributor-held-out and reproducible.
A split must be re-derivable from the pseudonymous subject key alone (the ingestion rule), and
one contributor's photos must all land in the same split. Otherwise a held-out score is not
held out: the model has already seen that person's body in training.
"""
splits_by_subject: dict[str, set[str]] = {}
for entry in records:
subject = entry.get("subjectKey")
if not isinstance(subject, str) or not subject:
raise EvaluationError("every prediction must carry the pseudonymous subject key")
split = entry.get("split")
if split not in SPLITS:
raise EvaluationError(f"split must be one of {', '.join(SPLITS)}: {split!r}")
splits_by_subject.setdefault(subject, set()).add(split)
for subject, splits in splits_by_subject.items():
if len(splits) > 1:
raise EvaluationError(
f"contributor {subject} leaks across {', '.join(sorted(splits))}: splits must be contributor-held-out"
)
for entry in records:
subject = entry["subjectKey"]
required = expected_split(subject)
if entry["split"] != required:
raise EvaluationError(
f"split {entry['split']!r} for contributor {subject} is not reproducible: the subject key derives {required!r}"
)
def compare(models: dict[str, list[dict]], *, min_coverage: float = 0.5, gates: dict | None = None) -> dict:
"""Compare candidate models against the release gates.
Each candidate gets one threshold, chosen on its own validation split, and is then measured on
its test split. The gates are the ROADMAP release gates; all limits are inclusive.
"""
limits = dict(gates or RELEASE_GATES)
rows: list[dict] = []
for model_id in sorted(models):
records = models[model_id]
assert_contributor_held_out(records)
test = [entry for entry in records if entry.get("split") == "test"]
validation = [entry for entry in records if entry.get("split") == "validation"]
if not test:
raise EvaluationError(f"{model_id} has no test-split predictions to measure a release on")
if not validation:
raise EvaluationError(f"{model_id} has no validation-split predictions to fix a threshold on")
selection = select_threshold(validation, min_coverage=min_coverage)
scored = evaluate(test, threshold=selection["threshold"])
failed = []
if scored["macroF1"] < limits["macroF1"]:
failed.append("macroF1")
if scored["precisionAtCoverage"] < limits["precisionAtCoverage"]:
failed.append("precisionAtCoverage")
if scored["ece"] > limits["ece"]:
failed.append("ece")
rows.append(
{
"model": model_id,
**scored,
"selectionSplit": "validation",
"evaluatedSplit": "test",
"minCoverage": min_coverage,
"gate": {"passed": not failed, "failed": failed},
}
)
ordered = sorted(
rows,
key=lambda row: (
not row["gate"]["passed"],
-row["macroF1"],
-row["precisionAtCoverage"],
row["model"],
),
)
passing = [row for row in ordered if row["gate"]["passed"]]
return {
"gates": limits,
"models": ordered,
"recommended": passing[0]["model"] if passing else None,
"verdict": "GO" if passing else "BLOCKED",
}
def select_threshold(records: list[dict], *, min_coverage: float = 0.5) -> dict:
"""Choose the abstention threshold that maximises precision at a required coverage.
Selection reads the validation split only. Fixing a threshold on the same data the release gate
is measured on would manufacture whatever precision the report needs.
"""
if not records:
raise EvaluationError("threshold selection needs validation predictions")
if not 0.0 <= min_coverage <= 1.0:
raise EvaluationError(f"a coverage floor must be between 0 and 1: {min_coverage}")
if any(entry.get("split") != "validation" for entry in records):
raise EvaluationError("threshold selection must use the validation split only")
assert_contributor_held_out(records)
candidates = {0.0}
candidates.update(
entry["prediction"]["confidence"] for entry in records if entry["prediction"] is not None
)
best: dict | None = None
for threshold in sorted(candidates):
report = evaluate(records, threshold=threshold)
# Float comparison: a coverage exactly on the floor must not be lost to rounding.
if report["coverage"] + 1e-12 < min_coverage:
continue
if best is None:
best = report
continue
more_precise = round(report["precisionAtCoverage"], 12) > round(best["precisionAtCoverage"], 12)
equally_precise_but_wider = (
round(report["precisionAtCoverage"], 12) == round(best["precisionAtCoverage"], 12)
and report["coverage"] > best["coverage"]
)
if more_precise or equally_precise_but_wider:
best = report
if best is None:
raise EvaluationError(f"no threshold reaches the {min_coverage:.0%} coverage floor on validation")
return {**best, "selectionSplit": "validation", "minCoverage": min_coverage}
def evaluate(records: list[dict], *, threshold: float = 0.0) -> dict:
"""Score one model's accepted suggestions at a fixed abstention threshold."""
accepted: list[tuple[int, int, float]] = []
abstentions = {"low-confidence": 0, "no-suggestion": 0}
for entry in records:
truth = entry["truth"]["bristolType"]
prediction = entry["prediction"]
if prediction is None:
abstentions["no-suggestion"] += 1
continue
if prediction["confidence"] < threshold:
abstentions["low-confidence"] += 1
continue
accepted.append((truth, prediction["bristolType"], prediction["confidence"]))
confusion: dict[str, int] = {}
for truth, predicted, _ in accepted:
key = f"{truth}->{predicted}"
confusion[key] = confusion.get(key, 0) + 1
truth_classes = {truth for truth, _, _ in accepted}
per_class: dict[int, dict] = {}
for label in sorted(truth_classes):
true_positive = sum(1 for truth, predicted, _ in accepted if truth == label and predicted == label)
predicted_positive = sum(1 for _, predicted, _ in accepted if predicted == label)
actual_positive = sum(1 for truth, _, _ in accepted if truth == label)
precision = true_positive / predicted_positive if predicted_positive else 0.0
recall = true_positive / actual_positive if actual_positive else 0.0
f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
per_class[label] = {
"precision": precision,
"recall": recall,
"f1": f1,
"support": actual_positive,
"predicted": predicted_positive,
}
macro_f1 = sum(row["f1"] for row in per_class.values()) / len(per_class) if per_class else 0.0
ece, calibration_bins = _calibration_error(accepted)
truth_classes_in_split = {entry["truth"]["bristolType"] for entry in records}
return {
"threshold": threshold,
"counts": {"total": len(records), "accepted": len(accepted), "abstained": len(records) - len(accepted)},
"abstentions": abstentions,
"coverage": len(accepted) / len(records) if records else 0.0,
"precisionAtCoverage": (
sum(1 for truth, predicted, _ in accepted if truth == predicted) / len(accepted) if accepted else 0.0
),
"perClass": per_class,
"excludedClasses": sorted(truth_classes_in_split - set(per_class)),
"macroF1": macro_f1,
"ece": ece,
"calibrationBins": calibration_bins,
"confusion": dict(sorted(confusion.items())),
}
def _validate_prediction(entry: object, where: str) -> dict:
"""Normalise and bound one prediction record, or refuse it."""
if not isinstance(entry, dict):
raise EvaluationError(f"{where} is not a prediction object")
model = entry.get("model")
if not isinstance(model, str) or not model.strip():
raise EvaluationError(f"{where} needs the model that produced the prediction")
truth = entry.get("truth")
if not isinstance(truth, dict) or not isinstance(truth.get("bristolType"), int):
raise EvaluationError(f"{where} needs truth.bristolType as a whole number")
if truth["bristolType"] not in BRISTOL_CLASSES:
raise EvaluationError(f"{where} truth.bristolType {truth['bristolType']} is outside Bristol 1-7")
split = entry.get("split")
if split not in SPLITS:
raise EvaluationError(f"{where} split must be one of {', '.join(SPLITS)}")
subject = entry.get("subjectKey")
if not isinstance(subject, str) or not _SHA_LIKE.match(subject):
raise EvaluationError(f"{where} needs the 16-character pseudonymous subject key")
prediction = entry.get("prediction")
if prediction is not None:
if not isinstance(prediction, dict):
raise EvaluationError(f"{where} prediction must be an object or null")
if prediction.get("bristolType") not in BRISTOL_CLASSES:
raise EvaluationError(f"{where} predicted bristolType must be 1-7")
confidence = prediction.get("confidence")
if not isinstance(confidence, (int, float)) or isinstance(confidence, bool) or not 0.0 <= confidence <= 1.0:
raise EvaluationError(f"{where} confidence must be a number between 0 and 1")
return {
"model": model.strip(),
"subjectKey": subject,
"split": split,
"truth": {"bristolType": truth["bristolType"]},
"prediction": None
if prediction is None
else {"bristolType": prediction["bristolType"], "confidence": float(prediction["confidence"])},
}
def load_predictions(paths: list[str]) -> list[dict]:
"""Read prediction JSONL files, refusing anything the gates could not honestly be computed from."""
records: list[dict] = []
for raw_path in paths:
path = Path(raw_path)
if not path.is_file():
raise EvaluationError(f"predictions file not found: {path}")
for number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
if not line.strip():
continue
try:
entry = json.loads(line)
except json.JSONDecodeError as exc:
raise EvaluationError(f"{path}:{number} is not valid JSON: {exc.msg}") from exc
records.append(_validate_prediction(entry, f"{path}:{number}"))
if not records:
raise EvaluationError("no prediction records were supplied")
return records
def group_by_model(records: list[dict]) -> dict[str, list[dict]]:
models: dict[str, list[dict]] = {}
for entry in records:
models.setdefault(entry["model"], []).append(
{key: value for key, value in entry.items() if key != "model"}
)
return models
def format_report(report: dict, *, source: str) -> str:
"""Render the comparison as a compact receipt a human can read in a terminal."""
lines = [
"TIMMY CALIBRATION REPORT",
f"source: {source}",
f"threshold: chosen on validation only · scored on test · coverage floor {report['models'][0]['minCoverage']:.0%}"
if report["models"]
else "threshold: nothing to score",
"candidates: " + ", ".join(row["model"] for row in report["models"]),
"",
f"{'model':<26}{'thr':>5}{'macroF1':>9}{'prec@cov':>10}{'coverage':>10}{'ECE':>7} gate",
]
for row in report["models"]:
gate = "PASS" if row["gate"]["passed"] else "FAIL: " + ", ".join(row["gate"]["failed"])
lines.append(
f"{row['model']:<26}{row['threshold']:>5.2f}{row['macroF1']:>9.3f}"
f"{row['precisionAtCoverage']:>10.3f}{row['coverage']:>10.2f}{row['ece']:>7.3f} {gate}"
)
limits = report["gates"]
lines += [
"",
"verdict: " + report["verdict"] + (f" — recommended {report['recommended']}" if report["recommended"] else ""),
f"gates: macroF1 >= {limits['macroF1']:.2f} · precision@coverage >= {limits['precisionAtCoverage']:.2f} · ECE <= {limits['ece']:.2f}",
]
return "\n".join(lines)
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Score Bristol candidates against the release gates.")
parser.add_argument("--predictions", action="append", required=True, metavar="PATH", help="prediction JSONL (repeatable)")
parser.add_argument("--min-coverage", type=float, default=0.5, help="coverage floor for threshold selection")
parser.add_argument("--json", metavar="PATH", help="write the full report as JSON here")
parser.add_argument("--source", default=None, help="what the predictions are, recorded in the receipt")
args = parser.parse_args(argv)
try:
records = load_predictions(args.predictions)
report = compare(group_by_model(records), min_coverage=args.min_coverage)
except EvaluationError as exc:
print(f"calibration harness refused: {exc}", file=sys.stderr)
return 2
source = args.source or f"{len(records)} prediction records from {len(args.predictions)} file(s)"
print(format_report(report, source=source))
if args.json:
Path(args.json).write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
return 0 if report["verdict"] == "GO" else 3
if __name__ == "__main__":
sys.exit(main())
+241
View File
@@ -0,0 +1,241 @@
#!/usr/bin/env python3
"""Immutable, auditable corpus manifest for Timmy training contributions.
Every ingested sample is described by one append-only manifest record: who consented, under
which consent version, where the bytes came from, the hashes of the raw input and the derived
image, how the derived image was produced, who reviewed the labels, which split it belongs to,
and where it sits in its lifecycle. Nothing in this module rewrites history: records are only
ever appended, and a withdrawal is a tombstone, not a deletion of the trail.
"""
from __future__ import annotations
import hashlib
import json
import re
from pathlib import Path
SCHEMA_VERSION = 2
PIPELINE = "timmy-ingest/2"
SAMPLE = "sample"
TOMBSTONE = "tombstone"
RECORD_TYPES = frozenset({SAMPLE, TOMBSTONE})
ACTIVE = "active"
REVOKED = "revoked"
STATES = frozenset({ACTIVE, REVOKED})
SPLITS = ("train", "validation", "test")
REVIEWERS = ("user-confirmed", "clinician-reviewed")
COLORS = ("brown", "green", "yellow", "pale", "red", "black")
QUALITIES = ("good", "fair", "poor")
_SHA256 = re.compile(r"^[0-9a-f]{64}$")
_CONTRIBUTION_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$")
class CorpusError(Exception):
"""A record cannot be built or written as asked."""
class IllegalTransition(CorpusError):
"""A lifecycle change that must never happen is being attempted."""
def sha256_hex(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def _require_sha256(value: object, field: str) -> str:
if not isinstance(value, str) or not _SHA256.match(value):
raise CorpusError(f"{field} must be a lowercase hex sha256 digest")
return value
def _require_iso(value: object, field: str) -> str:
if not isinstance(value, str) or not re.match(r"^\d{4}-\d{2}-\d{2}T", value):
raise CorpusError(f"{field} must be an ISO-8601 timestamp")
return value
def build_record(
*,
contribution_id: str,
consent_version: str,
subject_key: str,
split: str,
labels: dict,
review_status: str,
derived_sha256: str,
raw_sha256: str,
ingested_at: str,
source_kind: str = "cli-ingest",
) -> dict:
"""Build one active sample record carrying its full provenance."""
if not isinstance(contribution_id, str) or not _CONTRIBUTION_ID.match(contribution_id):
raise CorpusError("contribution id is required and must be a stable opaque identifier")
if not isinstance(consent_version, str) or not consent_version.strip():
raise CorpusError("consent version is required")
if not isinstance(subject_key, str) or not re.match(r"^[0-9a-f]{16}$", subject_key):
raise CorpusError("subject key must be the 16-character pseudonymous digest")
if split not in SPLITS:
raise CorpusError(f"split must be one of {', '.join(SPLITS)}")
if review_status not in REVIEWERS:
raise CorpusError(f"review status must be one of {', '.join(REVIEWERS)}")
if not isinstance(source_kind, str) or not source_kind.strip():
raise CorpusError("source kind is required")
if not isinstance(labels, dict) or set(labels) != {"bristolType", "color", "quality"}:
raise CorpusError("labels must carry bristolType, color, and quality")
if labels["bristolType"] not in range(1, 8):
raise CorpusError("bristolType must be 1-7")
if labels["color"] not in COLORS or labels["quality"] not in QUALITIES:
raise CorpusError("color and quality must be from the documented vocabularies")
derived = _require_sha256(derived_sha256, "derived sha256")
raw = _require_sha256(raw_sha256, "raw input sha256")
stamped = _require_iso(ingested_at, "ingestedAt")
return {
"schemaVersion": SCHEMA_VERSION,
"recordType": SAMPLE,
"contributionId": contribution_id,
"imageId": derived,
"relativePath": f"images/{derived}.jpg",
"subjectKey": subject_key,
"split": split,
"consentVersion": consent_version.strip(),
"consent": {"version": consent_version.strip(), "recordedAt": stamped},
"source": {"kind": source_kind, "inputSha256": raw},
"hashes": {"rawInput": raw, "derived": derived},
"derivation": {
"format": "jpeg",
"maxDimension": 1200,
"metadataStripped": True,
"pipeline": PIPELINE,
},
"labels": {
"bristolType": labels["bristolType"],
"color": labels["color"],
"quality": labels["quality"],
},
"review": {"status": review_status, "reviewedAt": stamped},
"lifecycle": {"state": ACTIVE, "updatedAt": stamped},
"ingestedAt": stamped,
}
def assert_admissible(manifest_path: Path, record: dict) -> None:
"""Refuse a sample that duplicates a contribution or re-labels the same derived image."""
if record.get("recordType", SAMPLE) != SAMPLE:
return
for prior in load_manifest(manifest_path):
if prior.get("recordType", SAMPLE) != SAMPLE:
continue
if prior.get("contributionId") == record.get("contributionId"):
raise IllegalTransition(f"contribution {record['contributionId']} was already ingested")
if prior.get("imageId") == record.get("imageId") and prior.get("labels") != record.get("labels"):
raise CorpusError("the same derived image cannot carry two different label sets")
def append_record(manifest_path: Path, record: dict) -> None:
"""Append one record to the manifest without touching what is already there."""
assert_admissible(manifest_path, record)
manifest_path.parent.mkdir(parents=True, exist_ok=True)
with manifest_path.open("a", encoding="utf-8") as manifest:
manifest.write(json.dumps(record, separators=(",", ":"), sort_keys=False) + "\n")
def load_manifest(manifest_path: Path) -> list[dict]:
"""Read the append-only manifest in order. A missing manifest is an empty corpus."""
if not manifest_path.exists():
return []
with manifest_path.open("r", encoding="utf-8") as manifest:
return [json.loads(line) for line in manifest.read().splitlines() if line.strip()]
def transition(record: dict, state: str, at: str, reason: str | None = None) -> dict:
"""Return a copy of the record in the next lifecycle state, or refuse the move."""
current = record.get("lifecycle", {}).get("state")
if state not in STATES:
raise IllegalTransition(f"unknown lifecycle state: {state!r}")
if current not in STATES:
raise IllegalTransition(f"record carries no legal lifecycle state: {current!r}")
if current == state:
raise IllegalTransition(f"record is already {state}")
if current == REVOKED:
raise IllegalTransition("a revoked contribution cannot return to the active dataset")
_require_iso(at, "lifecycle timestamp")
if state == REVOKED and (not isinstance(reason, str) or not reason.strip()):
raise IllegalTransition("revocation needs a recorded reason")
updated = {**record, "lifecycle": {"state": state, "updatedAt": at}}
if state == REVOKED:
updated["lifecycle"]["reason"] = reason.strip()
return updated
def active_records(records: list[dict]) -> list[dict]:
"""Samples still usable for training: active, and never revoked afterwards."""
revoked = {
record.get("contributionId")
for record in records
if record.get("recordType") == TOMBSTONE or record.get("lifecycle", {}).get("state") == REVOKED
}
return [
record
for record in records
if record.get("recordType", SAMPLE) == SAMPLE
and record.get("lifecycle", {}).get("state") == ACTIVE
and record.get("contributionId") not in revoked
]
def _safe_dataset_path(dataset_root: Path, relative: str) -> Path:
path = (dataset_root / relative).resolve()
if not path.is_relative_to(dataset_root.resolve()):
raise CorpusError(f"refusing to touch a path outside the dataset: {relative}")
return path
def revoke(dataset_root: Path, contribution_id: str, at: str, reason: str) -> dict:
"""Revoke one contribution: append a tombstone and delete its bytes from the dataset."""
manifest_path = dataset_root / "manifest.jsonl"
records = load_manifest(manifest_path)
samples = [
record
for record in records
if record.get("recordType", SAMPLE) == SAMPLE and record.get("contributionId") == contribution_id
]
if not isinstance(reason, str) or not reason.strip():
raise IllegalTransition("revocation needs a recorded reason")
if any(
record.get("recordType") == TOMBSTONE and record.get("contributionId") == contribution_id
for record in records
):
raise IllegalTransition(f"contribution {contribution_id} is already revoked")
if not samples:
raise IllegalTransition(f"no ingested sample carries contribution {contribution_id}")
sample = samples[-1]
if sample.get("lifecycle", {}).get("state") != ACTIVE:
raise IllegalTransition(f"contribution {contribution_id} is not active")
deleted = []
candidates = [(sample["relativePath"], sample["hashes"]["derived"])]
# A raw pre-derivation copy is identified by the raw input digest, never by a user path.
for path in sorted((dataset_root / "raw").glob(f"{sample['hashes']['rawInput']}*")):
candidates.append((str(path.relative_to(dataset_root)), sample["hashes"]["rawInput"]))
for relative, digest in candidates:
target = _safe_dataset_path(dataset_root, relative)
existed = target.is_file()
if existed:
target.unlink()
deleted.append({"relativePath": relative, "sha256": digest, "deleted": existed})
tombstone = {
"schemaVersion": SCHEMA_VERSION,
"recordType": TOMBSTONE,
"contributionId": contribution_id,
"imageId": sample["imageId"],
"deletedArtifacts": deleted,
"lifecycle": {"state": REVOKED, "updatedAt": _require_iso(at, "lifecycle timestamp"), "reason": reason.strip()},
}
append_record(manifest_path, tombstone)
return tombstone
+69 -35
View File
@@ -12,6 +12,8 @@ from pathlib import Path
from PIL import Image, ImageOps
from corpus_manifest import CorpusError, append_record, assert_admissible, build_record, revoke, sha256_hex
COLORS = ("brown", "green", "yellow", "pale", "red", "black")
QUALITIES = ("good", "fair", "poor")
REVIEWERS = ("user-confirmed", "clinician-reviewed")
@@ -19,27 +21,62 @@ REVIEWERS = ("user-confirmed", "clinician-reviewed")
def parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser()
p.add_argument("--input", required=True)
p.add_argument("--input")
p.add_argument("--output-dir", required=True)
p.add_argument("--subject-id", required=True)
p.add_argument("--consent-version", required=True)
p.add_argument("--bristol-type", required=True, type=int, choices=range(1, 8))
p.add_argument("--color", required=True, choices=COLORS)
p.add_argument("--quality", required=True, choices=QUALITIES)
p.add_argument("--reviewer", required=True, choices=REVIEWERS)
p.add_argument("--subject-id")
p.add_argument("--consent-version")
p.add_argument("--contribution-id")
p.add_argument("--bristol-type", type=int, choices=range(1, 8))
p.add_argument("--color", choices=COLORS)
p.add_argument("--quality", choices=QUALITIES)
p.add_argument("--reviewer", choices=REVIEWERS)
# Revocation is a corpus operation, not an ingestion: it needs the dataset and a reason.
p.add_argument("--revoke", metavar="CONTRIBUTION_ID")
p.add_argument("--reason")
return p
def require_ingest_arguments(args: argparse.Namespace) -> None:
if not args.consent_version or not args.consent_version.strip():
raise SystemExit("consent version is required")
if not args.reviewer:
raise SystemExit("a human review status is required")
missing = [
flag
for flag, value in (
("--input", args.input),
("--subject-id", args.subject_id),
("--contribution-id", args.contribution_id),
("--bristol-type", args.bristol_type),
("--color", args.color),
("--quality", args.quality),
)
if value is None or value == ""
]
if missing:
raise SystemExit(f"ingesting a sample also needs {', '.join(missing)}")
def main() -> int:
args = parser().parse_args()
if args.revoke:
if not args.reason:
raise SystemExit("revoking a contribution needs a recorded reason")
try:
tombstone = revoke(Path(args.output_dir), args.revoke, datetime.now(timezone.utc).isoformat(), args.reason)
except CorpusError as exc:
raise SystemExit(str(exc)) from exc
print(json.dumps(tombstone))
return 0
require_ingest_arguments(args)
salt = os.environ.get("TIMMY_DATASET_SALT", "")
if len(salt) < 8:
raise SystemExit("TIMMY_DATASET_SALT must contain at least 8 characters")
if not args.consent_version.strip():
raise SystemExit("consent version is required")
try:
with Image.open(args.input) as source:
raw_bytes = Path(args.input).read_bytes()
with Image.open(io.BytesIO(raw_bytes)) as source:
image = ImageOps.exif_transpose(source).convert("RGB")
image.thumbnail((1200, 1200))
buffer = io.BytesIO()
@@ -55,37 +92,34 @@ def main() -> int:
output = Path(args.output_dir)
images = output / "images"
images.mkdir(parents=True, exist_ok=True)
relative_path = f"images/{image_id}.jpg"
target = output / relative_path
if not target.exists():
target.write_bytes(data)
record = {
"schemaVersion": 1,
"imageId": image_id,
"relativePath": relative_path,
"subjectKey": subject_key,
"split": split,
"consentVersion": args.consent_version.strip(),
"ingestedAt": datetime.now(timezone.utc).isoformat(),
"labels": {
"bristolType": args.bristol_type,
"color": args.color,
"quality": args.quality,
},
"review": {"status": args.reviewer},
"derivation": {
"format": "jpeg",
"maxDimension": 1200,
"metadataStripped": True,
},
}
with (output / "manifest.jsonl").open("a", encoding="utf-8") as manifest:
manifest.write(json.dumps(record, separators=(",", ":")) + "\n")
now = datetime.now(timezone.utc).isoformat()
record = build_record(
contribution_id=args.contribution_id,
consent_version=args.consent_version,
subject_key=subject_key,
split=split,
labels={"bristolType": args.bristol_type, "color": args.color, "quality": args.quality},
review_status=args.reviewer,
derived_sha256=image_id,
raw_sha256=sha256_hex(raw_bytes),
ingested_at=now,
)
try:
# Refuse a duplicate contribution before any byte lands in the active dataset.
assert_admissible(output / "manifest.jsonl", record)
images.mkdir(parents=True, exist_ok=True)
if not target.exists():
target.write_bytes(data)
append_record(output / "manifest.jsonl", record)
except CorpusError as exc:
raise SystemExit(str(exc)) from exc
print(json.dumps(record))
return 0
if __name__ == "__main__":
sys.exit(main())
+587 -7
View File
@@ -1,10 +1,16 @@
import { chromium } from 'playwright';
import { mkdir, writeFile } from 'node:fs/promises';
import { execFile } from 'node:child_process';
import { mkdir, mkdtemp, readFile, readdir, rm, writeFile } from 'node:fs/promises';
import { createHash } from 'node:crypto';
import os from 'node:os';
import { promisify } from 'node:util';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { analyzePhoto, visionFailureResponse } from '../src/vision-service.js';
import { loadVisionCorpus } from '../src/vision-corpus.js';
const execFileAsync = promisify(execFile);
const output = path.resolve(process.argv[2] || 'release-demo.webm');
const version = process.env.TIMMY_RELEASE_VERSION || 'review build';
// The release delta is supplied by the builder so the recording states, on camera, what actually
@@ -46,6 +52,8 @@ const context = await browser.newContext({
});
const page = await context.newPage();
const errors = [];
// Delete-all asks for confirmation; the recording accepts it so the resulting state is shown, not implied.
page.on('dialog', dialog => dialog.accept());
page.on('console', message => { if (message.type() === 'error') errors.push(message.text()); });
page.on('pageerror', error => errors.push(error.message));
let visionOffline = false;
@@ -111,21 +119,51 @@ await page.route('**/api/agent/chat', route => {
await page.goto('http://127.0.0.1:4173', { waitUntil: 'networkidle' });
await page.evaluate(() => localStorage.clear());
await page.reload({ waitUntil: 'networkidle' });
await page.addStyleTag({ content: `
#demo-caption{position:fixed;left:16px;right:16px;top:14px;z-index:20000;background:rgba(40,33,29,.94);color:#fff;padding:11px 14px;border-radius:14px;font:800 12px/1.25 system-ui;letter-spacing:.02em;text-align:center;box-shadow:0 8px 24px rgba(0,0,0,.2)}
#demo-touch{position:fixed;z-index:20001;width:48px;height:48px;border:4px solid #fff;background:rgba(245,201,91,.55);box-shadow:0 0 0 7px rgba(21,125,120,.28);border-radius:50%;pointer-events:none;transform:translate(-50%,-50%)}
` });
const captionEvidence = [];
// Pixel proof that a caption actually painted: if the caption band looks identical with and without
// the node, the caption did not reach the screen even though the string was pushed.
async function captionBand() {
return page.screenshot({ clip: { x: 0, y: 0, width: 405, height: 68 } });
}
async function caption(text, duration = 1400) {
captionsShown.push(text);
const blankBand = await captionBand().catch(() => null);
await page.evaluate(text => {
document.querySelector('#demo-caption')?.remove();
const node = document.createElement('div');
node.id = 'demo-caption';
node.textContent = text;
// Inline styles, not a stylesheet: a stylesheet belongs to one document and the demo reloads the
// page, which silently stripped caption styling and left every later caption clipped body text.
node.style.cssText = 'position:fixed;left:16px;right:16px;top:14px;z-index:20000;'
+ 'background:rgba(40,33,29,.94);color:#fff;padding:11px 14px;border-radius:14px;'
+ 'font:800 12px/1.25 system-ui;letter-spacing:.02em;text-align:center;'
+ 'box-shadow:0 8px 24px rgba(0,0,0,.2)';
document.body.append(node);
node.animate([{ opacity: 0, transform: 'translateY(-8px)' }, { opacity: 1, transform: 'translateY(0)' }], { duration: 260, fill: 'forwards' });
}, text);
await sleep(320);
// Structural proof: a fixed bar at the top of the viewport, on top of whatever is beneath it, fully
// inside the frame. A caption that lost its stylesheet sits unstyled at the end of the body instead.
const painted = await page.evaluate(() => {
const node = document.querySelector('#demo-caption');
if (!node) return { ok: false, reason: 'caption node missing' };
const rect = node.getBoundingClientRect();
const style = getComputedStyle(node);
const topmost = document.elementFromPoint(Math.round(rect.left + rect.width / 2), Math.round(rect.top + rect.height / 2));
return {
ok: style.position === 'fixed' && style.zIndex === '20000' && rect.top <= 40 && rect.height >= 24
&& rect.width > 200 && rect.bottom <= window.innerHeight && (topmost === node || (topmost && node.contains(topmost))),
top: Math.round(rect.top), height: Math.round(rect.height), position: style.position,
zIndex: style.zIndex, background: style.backgroundColor, topmost: topmost === node,
};
});
const shownBand = await captionBand().catch(() => null);
const bandChanged = Boolean(blankBand && shownBand && !blankBand.equals(shownBand));
if (!painted.ok) throw new Error(`Caption is not an on-screen bar: ${text} ${JSON.stringify(painted)}`);
if (!bandChanged) throw new Error(`Caption painted nothing into the caption band: ${text}`);
captionEvidence.push({ text, visible: true, position: painted.position, top: painted.top, height: painted.height, bandChanged });
await sleep(duration);
// A caption must not outlive its stated duration. It used to stay on screen until the next one
// replaced it, so a view change left the previous screen's claim hanging over a different screen —
@@ -148,6 +186,9 @@ async function tap(selector, after = 650) {
document.querySelector('#demo-touch')?.remove();
const ring = document.createElement('div');
ring.id = 'demo-touch';
ring.style.cssText = 'position:fixed;z-index:20001;width:48px;height:48px;border:4px solid #fff;'
+ 'background:rgba(245,201,91,.55);box-shadow:0 0 0 7px rgba(21,125,120,.28);border-radius:50%;'
+ 'pointer-events:none;transform:translate(-50%,-50%)';
ring.style.left = `${x}px`;
ring.style.top = `${y}px`;
document.body.append(ring);
@@ -168,6 +209,9 @@ async function indicate(selector) {
document.querySelector('#demo-touch')?.remove();
const ring = document.createElement('div');
ring.id = 'demo-touch';
ring.style.cssText = 'position:fixed;z-index:20001;width:48px;height:48px;border:4px solid #fff;'
+ 'background:rgba(245,201,91,.55);box-shadow:0 0 0 7px rgba(21,125,120,.28);border-radius:50%;'
+ 'pointer-events:none;transform:translate(-50%,-50%)';
ring.style.left = `${x}px`;
ring.style.top = `${y}px`;
document.body.append(ring);
@@ -373,6 +417,64 @@ await caption('Two logs is not a pattern: the summary refuses to describe a tren
await page.evaluate(() => localStorage.clear());
await page.reload({ waitUntil: 'networkidle' });
// Issue #36 — a saved photo and its confirmed log are separate things a person owns. The journal
// states that a photo exists without rendering it, the image appears only when asked for, removing
// it keeps every confirmed field, and delete-all takes the ledger and every photo together. The
// only image here is a synthetic 1x1 PNG constant written by this script; no real photo is used.
// A 160x120 synthetic test card (checkerboard + 'SYNTHETIC DEMO PHOTO'), so the reveal is legibly
// an image on camera. It carries no photograph of any person or specimen.
const SYNTHETIC_PHOTO = 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAKAAAAB4CAIAAAD6wG44AAAG0ElEQVR4nO2deVATZxTAXxKSIldVZqyiTEWucMgpyCGggEK96gFYgSKVenCJoYAy1DpaBYpBnEqt1WoVb2071fHCqi2IiFo70gJaT7COWKdYBZExCW7/2HFJm8OFok5f3m/yx+6X9739Jr98m+zb+RKB1MYaCLwIX/UAiBcLCUaOEbfVEh/33OjGtAT+qYeWlvEPpsy9m9ly+w52g2YwckgwckgwckgwckgwckgwckgwckgwckgwckgwckgwcgTc7cILNeXPjUZTqkWf2dsvgt2gGYwcEowcEowcEowcEowcEowcEowcEowcEowcEowcEowcEowcEowcEoycrqUrfG5Robmbhj6z5bMNmsHIIcHIIcHIIcHIIcHIIcHIIcHIIcHIIcHIIcHIIcHIIcHIIcHI6Vp8xv0Q2r3lBa9uPMR/YsBHuewG90NoWlYXmgxwevkjI3qFx/cusRu0utBQIMHIIcHIIcHIIcHIIcHIMXp+CAAANNTVFRcVqFQqkUiUXyRvf9QuS0/57nC5UCgEgKjJEwrkq6Pfnuju6bV15x62i6+7y7na+uPlR8u2bAaAn3867zXCBwDiEhKXLM4+V1vPJWcjAcDT2WG4uwfbGBo+dsgQaz199+3euWvHNlNTMxMT02Ur8wcOsuqdlwQZUhtr9tH+RwP7YLQxZXxEc/MdhmHKjxySpSUzDJO5IPXQgf0Mw/x48gTb4uPmHBc97eyZaraLj5uzegb1XV1P/atdT9/Tpypnxc7o6OhgGKbih5OJsTO0djQ0OImcVr6n6JaWPxVPngBAaPi4uIREAEhJX7j+s7VPnz7dsK40dYGMDUuTfbB2zeoX81b8B5s3fiHLyjE2NgaA4NFjrN8cqlKpXsJx/3fwFSzLWhQXM/3DRdkXzp/z9vEFAFs7O0ep09K83EFWg23t7dkwP/8AADh7pvoFDZfj2tUrTi6u3O7y/EIjI74fNwYF3xdlalR02Nhxx78vL/h4WXhEZFqGDABSFmRMHBd28NgJ9cj0hZmflhSP9A/Qk02pVCbMjFHf1WzPzF7k4eWtK0NnZyfPkRs4vATfv9/S1Njo6eU9LSpmdGjY5MixrGCbYbampqY2w2zVg339/EUiUY3eSSwWi8t27e3q4u6itV0PQ22GXW6od/PwBACGYXKzMwvlJXw6Ghq8TtECgUCWlny3+Q4APPjrgZXVYP3x6QszS0uKe2F0uomNT1hTvEqhUADA4YMH2A1CE14zuF+//svzCzNSk42NjYVC4coiuf74Eb4jxWJxD1509VO0h6dXZs5iXZFvTZzU2Hhz+qTx/S0tLS0tlyxf0d1jGQh0PxgVmveDqZKFHBKMHBKMHL7XwWyVWCAQKBWK6Hdip0ZFg0bpODFpjoeT/Ziw8JLSz9nGHFnGsaOHL166CgDffr13R9kWsViiVCrenTV7yvQozeQqpXLxkqXD3dy56jQLt6uZRGutu739ka5jGRp8BXNXqB2PH6fMmd3HxCRy/ATNy1aJRHLzxo3Ozk6RSMQwzK1bTRKJBACqKiu+2bN7y/bd5hYWba2t85MS3xg40D9w1L+SX/ntcl5O1r79B7WOQWuS8IjI8IhIAPB1d2GTVFVWlG3+UtexDI1un6L7mJhk5eZt+2qTrgBnV9e6X2oB4HJDg6NUyjZu2rA+OzfP3MICAMwtLLJy8zauX6fZ18FRevv277oy80zCM8xA6MlnsKPUqampUdezo4JCqiorAKDqVMWo4BC28cb1a+qlY2cX1+vXrmr2rak+LXVy1pWZZxKeYQZCTwr0KpVKbCQGHaXjwKDgndu2pmbIaqpPz4zX/msxDMMIBAJul83DMIy5ufmKwlWgu1itJ4kueIZhpSeCf6296CCVgo7S8et9+wqEQrauaWZmxjba2tk31Nd5Prt50FBfZ2fvwHXRzKO1WK0/CQfPMAOh26fo1ocP5Z8UJM2drycmKGR0ibwoQO17TdLc+fKClW1tbQDQ1tpaXJj//rzk7h6aZ5JeORYa+M5g9pzJXsnMmZfs6+cPukvHIWNC18iL9h85xnUPDAq+e7d5VmyMRPKaUqmIT3jPLyCwu2PlmaRXjoUGqkWjgmrRBgcJRg6tLkQFrS40OLR8i+beBZqg+bVdw8lMMxg5JBg5JBg5JBg5JBg5JBg5JBg5JBg5JBg5JBg5JBg5JBg59N+FODPTfxcaCiQYOSQYOSQYOSQYOSQYOSQYOSQYOSQYOSQYOSQYOSQYOSQYOSQYOVpWF+oBzd009JlpdaGhQIKRQ4KRQ4KRQ4KRQ4KRQ4KRQ4KRQ4KRQ4KRQ4KR01WLJlBCMxg5JBg5fwMX/GJFpYaDtAAAAABJRU5ErkJggg==';
const photoDay = (id, daysAgo, bristolType = 4) => ({ ...sampleDay(id, daysAgo, bristolType), photoDataUrl: SYNTHETIC_PHOTO });
const storedLedger = async () => {
const key = await page.evaluate(() => `timmy:${document.querySelector('meta[name="timmy-base-path"]')?.content || '/'}:ledger-v1`);
const raw = await page.evaluate(storeKey => localStorage.getItem(storeKey), key);
return { key, raw };
};
await caption('A confirmed log can hold a private photo — the journal says so without showing it', 2300);
await seedSyntheticLedger([photoDay('demo-photo', 1), sampleDay('demo-nophoto', 4, 6)]);
await tap('[data-view="calendar"]', 550);
await page.locator('.entries-card .entry').first().waitFor();
if (await page.locator('.entries-card [data-photo-indicator="present"]').count() !== 1) {
throw new Error('exactly one log must be marked as holding a photo');
}
if (await page.locator('.entries-card img').count() !== 0) {
throw new Error('the journal must not show a thumbnail by default');
}
await caption('Photo presence is a label, not a thumbnail — nothing sensitive is on screen until you ask', 2700);
await indicate('[data-photo-indicator="present"]');
await tap('[data-entry-open="demo-photo"]', 550);
await page.locator('[data-photo-lifecycle="present"]').waitFor();
if (await page.locator('#entry-photo').count() !== 0) {
throw new Error('opening a log must not reveal its photo');
}
await caption('Opening the log states the photo is saved here — and still does not render it', 2500);
await indicate('[data-photo-lifecycle="present"]');
await tap('#show-entry-photo', 600);
await page.locator('#entry-photo').waitFor();
const revealedPhoto = await page.locator('#entry-photo').getAttribute('src');
if (!/^data:image\/png;base64,/.test(revealedPhoto || '')) throw new Error('the revealed image must be the stored bytes');
await caption('Showing the photo is your explicit request, from this browser’s own storage', 2300);
await indicate('#entry-photo');
await tap('#remove-entry-photo', 650);
await page.locator('[data-photo-lifecycle="absent"]').waitFor();
const afterRemoval = JSON.parse((await storedLedger()).raw || '[]');
const keptLog = afterRemoval.find(entry => entry.id === 'demo-photo');
if (afterRemoval.length !== 2 || !keptLog || keptLog.photoDataUrl !== '') {
throw new Error('removing a photo must keep the confirmed log and clear only its image bytes');
}
if (keptLog.bristolType !== 4 || keptLog.color !== 'brown') throw new Error('removing a photo must not alter the confirmed fields');
await caption('Remove just the photo: the confirmed log, form, and note are all still here', 2900);
await indicate('[data-photo-lifecycle="absent"]');
await tap('#entry-sheet-close', 450);
await page.locator('.entries-card .entry').first().waitFor();
if (await page.locator('.entries-card [data-photo-indicator="present"]').count() !== 0) {
throw new Error('the journal must drop the photo label once the bytes are gone');
}
await caption('The label follows the bytes: the log stays, the photo is gone', 2300);
// One photo-bearing log remains for the deletion pass below, so the wipe shows photos leaving with
// the ledger rather than an empty promise.
await seedSyntheticLedger([photoDay('demo-wipe', 2)]);
await caption('The threat model documents this surface, so the recording shows it live rather than describing it', 2200);
await tap('[data-view="calendar"]', 550);
await page.getByText(/Your private journal/i).waitFor();
@@ -381,8 +483,485 @@ await page.getByText(/Your poop\. Your phone\./i).waitFor();
await caption('Every byte-bearing control the model names, in the running app: local ledger, consent-gated upload, portable export, full deletion', 2800);
await indicate('#export');
await caption('Export writes a plain JSON file you hold — the model records it as a data-lifetime boundary, not a server sync', 2600);
// Issue #24 — training contribution is a second, separate consent. The recording shows the section
// reporting nothing queued after ordinary saves, the version and receipt on screen before the
// button can be pressed, one receipt queued on the device only, and a withdrawal tombstone that
// names each derived artifact by contribution id.
await caption('Training contribution is off by default: a saved log is never training data', 2600);
await page.locator('[data-training-consent]').waitFor();
const contributionKey = await page.evaluate(() => `timmy:${document.querySelector('meta[name="timmy-base-path"]')?.content || '/'}:training-contributions-v1`);
const storedContributions = async () => JSON.parse(await page.evaluate(key => localStorage.getItem(key) || '[]', contributionKey));
if ((await page.locator('[data-contribution-queued-count]').innerText()).trim() !== '0') {
throw new Error('a journal save must not queue a training contribution');
}
await indicate('[data-contribute-entry="demo-wipe"]');
await tap('[data-contribute-entry="demo-wipe"]', 600);
await page.locator('[data-contribution-sheet]').waitFor();
const receiptId = (await page.locator('[data-receipt-id]').innerText()).trim();
if (!receiptId) throw new Error('the consent sheet must name the receipt before consent');
const sheetCopy = await page.locator('[data-contribution-sheet]').innerText();
if (!sheetCopy.includes('training-consent-v1')) throw new Error('the consent sheet must state the consent version');
if (!(await page.locator('#contribute-confirm').isDisabled())) throw new Error('the contribute control must stay disabled until the terms are accepted');
if ((await storedContributions()).length !== 0) throw new Error('no receipt may exist before explicit consent');
await caption('The consent version and the receipt are on screen before the contribute button works', 3000);
await page.locator('#training-consent-check').check();
await tap('#contribute-confirm', 700);
await page.locator('[data-contribution-state="queued-local"]').waitFor();
const queuedReceipts = await storedContributions();
if (queuedReceipts.length !== 1 || queuedReceipts[0].consentVersion !== 'training-consent-v1') {
throw new Error('one queued receipt must carry the consent version it was accepted under');
}
if (JSON.stringify(queuedReceipts).includes('data:image')) throw new Error('a receipt must never copy the photo bytes');
await caption('One confirmed log is queued on this device with a receipt you can withdraw — nothing is uploaded', 3200);
await indicate('[data-contribution-state="queued-local"]');
await tap(`[data-withdraw="${receiptId}"]`, 700);
await page.locator('[data-contribution-state="withdrawn"]').waitFor();
const withdrawnReceipts = await storedContributions();
const tombstone = withdrawnReceipts[0];
if (!tombstone || tombstone.state !== 'withdrawn') throw new Error('withdrawal must keep the receipt as a tombstone');
if ((tombstone.artifacts || []).length !== 2) throw new Error('the tombstone must name every derived artifact');
for (const artifact of tombstone.artifacts) {
if (artifact.contributionId !== receiptId) throw new Error('every derived artifact must be addressed by contribution id');
if (artifact.state !== 'withdrawn') throw new Error('every derived artifact must be marked withdrawn');
}
await caption('Withdrawing keeps a tombstone naming every derived artifact by contribution ID', 3000);
await indicate('[data-withdrawn-artifacts]');
// Issue #25 — the corpus pipeline is a command-line tool, not a screen in the app. Rather than
// inventing a UI for it, this scene runs the real ingest and revoke commands against the committed
// synthetic fixture in a throwaway dataset and shows their actual output. Every number, hash prefix,
// and refusal line below is read back from those commands at record time.
async function corpusCli(args) {
return execFileAsync('python3', args, {
cwd: process.cwd(),
env: { ...process.env, TIMMY_DATASET_SALT: 'demo-only-salt-not-a-secret' },
});
}
async function corpusAudit() {
const dataset = await mkdtemp(path.join(os.tmpdir(), 'timmy-corpus-demo-'));
const fixture = 'tests/fixtures/synthetic-type4.jpg';
const contributionId = 'demo-contrib-0001-abcd';
const manifestPath = path.join(dataset, 'manifest.jsonl');
const lines = async () => (await readFile(manifestPath, 'utf8'))
.split('\n').filter(line => line.trim()).map(line => JSON.parse(line));
const images = async () => (await readdir(path.join(dataset, 'images')).catch(() => [])).sort();
try {
const ingest = await corpusCli([
'scripts/ingest_training_photo.py',
'--input', fixture,
'--output-dir', dataset,
'--subject-id', 'demo-subject',
'--consent-version', 'training-consent-v1',
'--contribution-id', contributionId,
'--bristol-type', '4', '--color', 'brown', '--quality', 'good',
'--reviewer', 'user-confirmed',
]);
const record = JSON.parse(ingest.stdout);
const fixtureDigest = createHash('sha256').update(await readFile(fixture)).digest('hex');
if (record.recordType !== 'sample' || record.lifecycle?.state !== 'active') {
throw new Error('ingest must write one active sample record');
}
// The record names the source by digest only: the caller's path must never be stored.
if (record.source?.inputSha256 !== fixtureDigest || record.hashes?.rawInput !== fixtureDigest) {
throw new Error('the record must carry the raw input digest');
}
if (JSON.stringify(record).includes(fixture) || 'inputPath' in record) {
throw new Error('the record must never store the caller path');
}
const afterIngestImages = await images();
if (afterIngestImages.length !== 1 || (await lines()).length !== 1) {
throw new Error('ingest must write exactly one derived image and one manifest record');
}
const revoke = await corpusCli([
'scripts/ingest_training_photo.py', '--output-dir', dataset,
'--revoke', contributionId, '--reason', 'participant withdrew consent',
]);
const tombstone = JSON.parse(revoke.stdout);
const afterRevokeImages = await images();
const history = await lines();
if (tombstone.recordType !== 'tombstone' || history.length !== 2) {
throw new Error('revocation must append exactly one tombstone to the existing history');
}
if (afterRevokeImages.length !== 0) throw new Error('revocation must delete the derived bytes');
if (history[0].recordType !== 'sample' || history[0].lifecycle?.state !== 'active') {
throw new Error('the original record must survive its own withdrawal');
}
if ('labels' in tombstone || 'subjectKey' in tombstone) {
throw new Error('a tombstone must not carry the labels or subject key');
}
const deleted = tombstone.deletedArtifacts || [];
if (!deleted.some(item => item.relativePath === `images/${record.imageId}.jpg` && item.deleted === true)) {
throw new Error('the tombstone must name the derived artifact it removed');
}
const active = await corpusCli([
'-c',
'import sys;sys.path.insert(0,"scripts");import corpus_manifest as c;from pathlib import Path;'
+ 'print(len(c.active_records(c.load_manifest(Path(sys.argv[1])/"manifest.jsonl"))))',
dataset,
]);
if (active.stdout.trim() !== '0') throw new Error('a trainer reading the manifest must see zero active samples');
// A second revocation of the same contribution has to be refused, and refuse without writing.
let refusal = '';
try {
await corpusCli([
'scripts/ingest_training_photo.py', '--output-dir', dataset,
'--revoke', contributionId, '--reason', 'participant withdrew consent again',
]);
throw new Error('a second revocation must be refused');
} catch (error) {
if (error.message === 'a second revocation must be refused') throw error;
refusal = String(error.stderr || error.stdout || '').trim().split('\n').filter(Boolean).pop() || '';
}
if (!/already revoked/.test(refusal)) throw new Error(`second revocation refusal was not legible: ${refusal}`);
if ((await lines()).length !== 2) throw new Error('a refused revocation must append nothing');
return { record, tombstone, deleted, refusal, derived: record.imageId, fixtureDigest };
} finally {
await rm(dataset, { recursive: true, force: true });
}
}
const audit = await corpusAudit();
const short = value => String(value).slice(0, 12);
const deletedName = audit.deleted[0].relativePath.split('/').pop().replace(/\.jpg$/, '');
await tap('[data-view="home"]', 500);
// The card is drawn from the values above, so a run that stops writing provenance or stops deleting
// bytes fails here instead of recording a caption that claims it.
const auditRows = [
['command', 'python3 scripts/ingest_training_photo.py --input <synthetic fixture>'],
['record', `sample · schemaVersion ${audit.record.schemaVersion}`],
['consent', audit.record.consent.version],
['source', `sha256 ${short(audit.record.source.inputSha256)}… (path never stored)`],
['hashes', `raw ${short(audit.record.hashes.rawInput)}… · derived ${short(audit.record.hashes.derived)}…`],
['derived', audit.record.derivation.pipeline + ' · jpeg · ≤1200px · metadata stripped'],
['review', audit.record.review.status],
['lifecycle', audit.record.lifecycle.state],
['—', ''],
['command', `--revoke ${audit.record.contributionId} --reason "participant withdrew consent"`],
['deleted', `images/${short(deletedName)}….jpg → gone from disk`],
['tombstone', 'appended · keeps id + digests · drops labels + subject key'],
['history', '2 lines: the original record is still there'],
['trainer', 'active samples: 0'],
['—', ''],
['command', '--revoke again'],
['refused', audit.refusal],
['manifest', 'unchanged at 2 lines'],
];
const panelOk = await page.evaluate(rows => {
document.querySelector('#corpus-audit')?.remove();
const panel = document.createElement('div');
panel.id = 'corpus-audit';
Object.assign(panel.style, {
position: 'fixed', inset: '96px 14px 40px 14px', zIndex: '15000',
background: '#181412', color: '#f6f1e6', borderRadius: '18px',
border: '1px solid rgba(245,201,91,.45)', padding: '13px 13px 11px',
font: '600 10.5px/1.5 ui-monospace, SFMono-Regular, Menlo, monospace',
display: 'flex', flexDirection: 'column', gap: '1px', overflow: 'hidden',
boxShadow: '0 18px 40px rgba(0,0,0,.42)',
});
const head = document.createElement('div');
head.textContent = 'CORPUS AUDIT';
Object.assign(head.style, { font: '800 14px/1.2 system-ui', letterSpacing: '.14em', color: '#f5c95b', marginBottom: '2px' });
const sub = document.createElement('div');
sub.textContent = 'command line, not app UI — real run, synthetic fixture';
Object.assign(sub.style, { font: '700 9.5px/1.3 system-ui', color: '#c9bfae', marginBottom: '9px' });
panel.append(head, sub);
for (const [key, value] of rows) {
const row = document.createElement('div');
if (key === '—') { row.style.height = '7px'; panel.append(row); continue; }
Object.assign(row.style, { display: 'flex', gap: '7px', alignItems: 'baseline' });
const k = document.createElement('span');
k.textContent = key;
k.style.cssText = 'flex:0 0 62px;color:#8fd7c8;text-transform:uppercase;font-size:9px;letter-spacing:.08em';
const v = document.createElement('span');
v.textContent = value;
v.style.cssText = 'flex:1 1 auto;word-break:break-word;color:#f6f1e6';
row.append(k, v);
panel.append(row);
}
document.body.append(panel);
panel.animate([{ opacity: 0, transform: 'translateY(12px)' }, { opacity: 1, transform: 'translateY(0)' }], { duration: 320, fill: 'forwards' });
return { height: panel.clientHeight, scroll: panel.scrollHeight, text: panel.innerText };
}, auditRows);
await sleep(420);
const panelProof = await page.evaluate(() => {
const panel = document.querySelector('#corpus-audit');
if (!panel) return { ok: false, reason: 'panel missing' };
const rect = panel.getBoundingClientRect();
const topmost = document.elementFromPoint(Math.round(rect.left + rect.width / 2), Math.round(rect.top + rect.height / 2));
const background = getComputedStyle(panel).backgroundColor;
// A translucent card lets the nav bar and body copy ghost through its empty rows, which reads as a
// sloppy overlay on the thinnest part of the frame. The card must be fully opaque.
const opaque = !/^rgba\(.*,\s*(0?\.\d+|0)\)$/.test(background);
return {
ok: opaque && rect.top >= 60 && rect.bottom <= window.innerHeight && rect.width > 300 && rect.height > 300
&& (topmost === panel || panel.contains(topmost)) && panel.scrollHeight <= panel.clientHeight + 1,
top: Math.round(rect.top), height: Math.round(rect.height), width: Math.round(rect.width),
scroll: panel.scrollHeight, client: panel.clientHeight, background, opaque,
covered: topmost === panel || panel.contains(topmost),
};
});
if (!panelProof.ok) throw new Error(`Corpus audit card is not fully readable on screen: ${JSON.stringify(panelProof)}`);
// The card is drawn from live command output; prove the values on it are the ones the run produced.
if (!panelOk.text.includes(audit.record.imageId.slice(0, 12))) throw new Error('the card must show the derived digest it recorded');
if (!panelOk.text.includes('active samples: 0')) throw new Error('the card must show the post-revocation active count');
await caption('The corpus pipeline is a command-line tool, not a screen in the app — this runs the real ingest against the committed synthetic fixture', 3400);
await caption('Every ingested sample records consent, source digest, both hashes, derivation, review, and lifecycle — the caller file path is never stored', 3600);
await caption('Withdrawing deletes the derived image and any raw copy, keeps the original record, and appends a tombstone — the trainer sees zero active samples', 3800);
await caption('Revoking the same contribution twice is refused and appends nothing: the manifest only ever grows', 3200);
await page.evaluate(() => document.querySelector('#corpus-audit')?.remove());
// Issue #29 — the release gate is a command-line harness, not a screen in the app. This scene runs
// the real harness over the committed synthetic prediction fixtures and shows the receipt it wrote:
// a blocked comparison, a passing fixture, and a refused leaked corpus. Every number and every
// refusal line on the card is read back from those runs; the scores are invented fixtures, and the
// captions say so, because a fixture is not a measurement of any model.
async function calibrationCli(args) {
try {
const result = await execFileAsync('python3', ['scripts/calibration_report.py', ...args], { cwd: process.cwd() });
return { code: 0, stdout: result.stdout, stderr: result.stderr };
} catch (error) {
return {
code: typeof error.code === 'number' ? error.code : 1,
stdout: error.stdout || '',
stderr: error.stderr || '',
};
}
}
async function calibrationGate() {
const directory = await mkdtemp(path.join(os.tmpdir(), 'timmy-calibration-demo-'));
const reportPath = path.join(directory, 'report.json');
const source = 'invented scores, not measurements of any model';
try {
const blocked = await calibrationCli([
'--predictions', 'tests/fixtures/calibration/predictions.jsonl',
'--json', reportPath, '--source', source,
]);
if (blocked.code !== 3) throw new Error('the synthetic candidate fixtures must block the release');
const report = JSON.parse(await readFile(reportPath, 'utf8'));
if (report.verdict !== 'BLOCKED' || report.recommended !== null) {
throw new Error('a blocked gate must recommend nobody');
}
if (report.models.length !== 3) throw new Error('the comparison must score all three candidate fixtures');
for (const row of report.models) {
if (!row.gate.failed.length) throw new Error(`${row.model}: a fixture candidate must fail at least one gate`);
if (row.selectionSplit !== 'validation' || row.evaluatedSplit !== 'test') {
throw new Error(`${row.model}: the threshold must come from validation and the score from test`);
}
}
const go = await calibrationCli([
'--predictions', 'tests/fixtures/calibration/predictions-go.jsonl',
'--source', 'engineered fixture — proves the passing branch, not a result',
]);
if (go.code !== 0 || !/verdict: GO/.test(go.stdout)) throw new Error('the engineered fixture must prove the passing branch');
const leak = await calibrationCli(['--predictions', 'tests/fixtures/calibration/predictions-leaky.jsonl']);
if (leak.code !== 2 || !/leaks across/.test(leak.stderr)) throw new Error('a leaked corpus must be refused before scoring');
return { report, source, blocked, go, leak };
} finally {
await rm(directory, { recursive: true, force: true });
}
}
const gate = await calibrationGate();
await tap('[data-view="home"]', 450);
const gateRows = [
['command', 'python3 scripts/calibration_report.py --predictions <fixtures>'],
['split', 'threshold chosen on validation · scored on test'],
['floor', `coverage floor ${Math.round(gate.report.models[0].minCoverage * 100)}%`],
['—', ''],
];
for (const row of gate.report.models) {
// The model id gets its own row: a 74px key column broke "specialist-candidate" at its hyphen,
// which reads as a label with no value and a stray line under it. The name is the value here.
gateRows.push(['model', row.model]);
gateRows.push([
'metrics',
`thr ${row.threshold.toFixed(2)} · F1 ${row.macroF1.toFixed(3)} · P@C ${row.precisionAtCoverage.toFixed(3)} · cov ${row.coverage.toFixed(2)}`,
]);
gateRows.push(['gate', `FAIL: ${row.gate.failed.join(', ')} · ECE ${row.ece.toFixed(3)}`]);
}
gateRows.push(['—', '']);
gateRows.push(['verdict', `${gate.report.verdict} · no candidate clears every gate`]);
gateRows.push(['gates', 'macroF1 ≥ 0.80 · precision@coverage ≥ 0.90 · ECE ≤ 0.05']);
gateRows.push(['—', '']);
gateRows.push(['engineered', 'GO — gate-plumbing-fixture clears every gate (fixture, not a result)']);
gateRows.push(['leaked', gate.leak.stderr.trim().replace(/^calibration harness refused:\s*/, '')]);
gateRows.push(['data', 'synthetic predictions · no model was measured']);
const gateCard = await page.evaluate(rows => {
document.querySelector('#calibration-report')?.remove();
const panel = document.createElement('div');
panel.id = 'calibration-report';
Object.assign(panel.style, {
position: 'fixed', inset: '96px 14px 40px 14px', zIndex: '15000',
background: '#181412', color: '#f6f1e6', borderRadius: '18px',
border: '1px solid rgba(245,201,91,.45)', padding: '13px 13px 11px',
font: '600 9.5px/1.45 ui-monospace, SFMono-Regular, Menlo, monospace',
display: 'flex', flexDirection: 'column', gap: '1px', overflow: 'hidden',
boxShadow: '0 18px 40px rgba(0,0,0,.42)',
});
const head = document.createElement('div');
head.textContent = 'CALIBRATION HARNESS';
Object.assign(head.style, { font: '800 13.5px/1.2 system-ui', letterSpacing: '.12em', color: '#f5c95b', marginBottom: '2px' });
const sub = document.createElement('div');
sub.textContent = 'command line, not app UI — real run, synthetic predictions';
Object.assign(sub.style, { font: '700 9px/1.3 system-ui', color: '#c9bfae', marginBottom: '8px' });
panel.append(head, sub);
for (const [key, value] of rows) {
const row = document.createElement('div');
if (key === '—') { row.style.height = '6px'; panel.append(row); continue; }
Object.assign(row.style, { display: 'flex', gap: '6px', alignItems: 'baseline' });
const k = document.createElement('span');
k.textContent = key;
k.style.cssText = 'flex:0 0 74px;color:#8fd7c8;text-transform:uppercase;font-size:8.5px;letter-spacing:.06em';
const v = document.createElement('span');
v.textContent = value;
v.style.cssText = 'flex:1 1 auto;word-break:break-word;color:#f6f1e6';
row.append(k, v);
panel.append(row);
}
document.body.append(panel);
panel.animate([{ opacity: 0, transform: 'translateY(12px)' }, { opacity: 1, transform: 'translateY(0)' }], { duration: 320, fill: 'forwards' });
return { height: panel.clientHeight, scroll: panel.scrollHeight, text: panel.innerText };
}, gateRows);
await sleep(420);
const gateProof = await page.evaluate(() => {
const panel = document.querySelector('#calibration-report');
if (!panel) return { ok: false, reason: 'panel missing' };
const rect = panel.getBoundingClientRect();
const topmost = document.elementFromPoint(Math.round(rect.left + rect.width / 2), Math.round(rect.top + rect.height / 2));
// Same rule as the corpus card: a translucent card lets the screen behind it ghost through.
const opaque = !/^rgba\(.*,\s*(0?\.\d+|0)\)$/.test(getComputedStyle(panel).backgroundColor);
return {
ok: opaque && rect.top >= 60 && rect.bottom <= window.innerHeight && rect.width > 300 && rect.height > 300
&& (topmost === panel || panel.contains(topmost)) && panel.scrollHeight <= panel.clientHeight + 1,
top: Math.round(rect.top), height: Math.round(rect.height), width: Math.round(rect.width),
scroll: panel.scrollHeight, client: panel.clientHeight, opaque,
covered: topmost === panel || panel.contains(topmost),
};
});
if (!gateProof.ok) throw new Error(`Calibration card is not fully readable on screen: ${JSON.stringify(gateProof)}`);
// The card is drawn from the report; prove the values on it are the ones the harness measured.
for (const row of gate.report.models) {
if (!gateCard.text.includes(row.macroF1.toFixed(3))) {
throw new Error(`the card must show the macro-F1 the harness measured for ${row.model}`);
}
}
if (!gateCard.text.includes('leaks across')) throw new Error('the card must show the refusal the leaked corpus earned');
if (!gateCard.text.includes('GO — gate-plumbing-fixture')) throw new Error('the card must show the passing branch it proved');
await caption('The calibration harness is a command-line release gate, not a screen in the app — this runs it against committed synthetic predictions', 3400);
await caption('Every score on this card is invented to exercise the arithmetic: no model produced them and none of them describes the real classifier', 3600);
await caption('Each candidate is scored on test at a threshold chosen on validation only; picking a threshold on the split you report is refused', 3600);
await caption('One contributor spread across two splits is refused before anything is scored, and a passing verdict is proven only by a fixture engineered to clear it', 3800);
await page.evaluate(() => document.querySelector('#calibration-report')?.remove());
// Back to the privacy view for the portability scene: the privacy entry lives on the journal view.
await tap('[data-view="calendar"]', 550);
await tap('[data-view="privacy"]', 600);
// Issue #35 — a backup is the person's own record, so it carries a version and everything it is
// claimed to hold. The recording exports the real file, imports an older version and a newer one,
// and reopens a restored log. Fictional sample logs are used throughout; nothing here is real data.
const exportDownload = page.waitForEvent('download');
await tap('#export', 600);
const exportedBackup = JSON.parse(await readFile(await (await exportDownload).path(), 'utf8'));
if (exportedBackup.schemaVersion !== 2) throw new Error('the exported backup must declare the current schema version');
if (!Array.isArray(exportedBackup.entries) || !Array.isArray(exportedBackup.contributions)) {
throw new Error('a backup must carry both the confirmed logs and the contribution receipts');
}
const exportedReceipt = exportedBackup.contributions.find(receipt => receipt.contributionId === receiptId);
if (!exportedReceipt || exportedReceipt.state !== 'withdrawn') {
throw new Error('the backup must carry the withdrawn receipt as a tombstone');
}
if (!(exportedReceipt.artifacts || []).length) throw new Error('the backup tombstone must still name its derived artifacts');
await page.locator('[data-portability-notice][data-portability-version="2"]').waitFor();
await caption('A backup is versioned, and it carries the withdrawn receipt as a tombstone rather than silence', 3200);
const fictionalLegacyBackup = JSON.stringify({
product: 'Timmy the Talking Turd',
schemaVersion: 1,
exportedAt: '2026-08-01T09:00:00.000Z',
entries: [{
id: 'demo-legacy-backup',
occurredAt: new Date(Date.now() - 3 * 86400000).toISOString(),
bristolType: 3,
color: 'brown',
urgency: 1,
discomfort: 0,
note: 'fictional version 1 backup',
photoDataUrl: '',
symptoms: {},
}],
});
await page.locator('#import').setInputFiles({ name: 'timmy-backup-v1.json', mimeType: 'application/json', buffer: Buffer.from(fictionalLegacyBackup) });
await page.locator('[data-portability-notice][data-portability-version="1"]').waitFor();
const upgradeNotice = (await page.locator('[data-portability-notice]').innerText()).trim();
if (!/upgraded/i.test(upgradeNotice)) throw new Error('an older backup must be reported as upgraded, not silently rewritten');
await caption('An older backup is upgraded on import, and the notice says which version the file came from', 3400);
const ledgerBeforeFuture = await storedLedger();
await page.locator('#import').setInputFiles({
name: 'timmy-backup-v3.json', mimeType: 'application/json',
buffer: Buffer.from(JSON.stringify({ product: 'Timmy the Talking Turd', schemaVersion: 3, entries: [{ id: 'demo-future', bristolType: 7 }] })),
});
await page.locator('[data-portability-error]').waitFor();
if (!/newer version of Timmy/i.test(await page.locator('[data-portability-error]').innerText())) {
throw new Error('a backup from a newer Timmy must be refused with an instruction to update');
}
if ((await storedLedger()).raw !== ledgerBeforeFuture.raw) {
throw new Error('a refused import must not change the ledger it would have replaced');
}
await caption('A file from a newer Timmy is refused with an instruction, and the ledger it would have replaced is untouched', 3400);
const fictionalAssistedBackup = JSON.stringify({
product: 'Timmy the Talking Turd',
schemaVersion: 2,
exportedAt: new Date().toISOString(),
entries: [{
id: 'demo-restored-provenance',
occurredAt: new Date(Date.now() - 86400000).toISOString(),
bristolType: 2,
color: 'brown',
urgency: 1,
discomfort: 0,
note: 'fictional restored log',
photoDataUrl: '',
symptoms: {},
provenance: {
source: 'ai-corrected',
model: 'SmolVLM2-2.2B-Instruct',
suggestedBristolType: 4,
suggestedColor: 'brown',
confirmedAt: new Date(Date.now() - 86400000).toISOString(),
},
}],
contributions: [],
});
await page.locator('#import').setInputFiles({ name: 'timmy-backup-v2.json', mimeType: 'application/json', buffer: Buffer.from(fictionalAssistedBackup) });
await page.locator('[data-portability-notice][data-portability-version="2"]').waitFor();
await tap('[data-view="calendar"]', 550);
await page.locator('[data-entry-open="demo-restored-provenance"]').waitFor();
await tap('[data-entry-open="demo-restored-provenance"]', 550);
await page.locator('[data-entry-provenance="ai-corrected"]').waitFor();
const provenanceLine = (await page.locator('[data-entry-provenance="ai-corrected"]').innerText()).trim();
if (!/corrected/i.test(provenanceLine) || !/SmolVLM2/.test(provenanceLine)) {
throw new Error('a restored log must still show that it began as a corrected AI suggestion');
}
await caption('A restored log still says it began as an AI suggestion you corrected — provenance is part of the record', 3400);
await indicate('[data-entry-provenance="ai-corrected"]');
await tap('#entry-sheet-close', 450);
await tap('[data-view="privacy"]', 550);
await page.locator('#delete-all').waitFor();
await indicate('#delete-all');
await caption('Delete all local data clears the ledger in this browser; there is no server copy of it to keep', 2600);
await tap('#delete-all', 650);
await page.getByText(/Local ledger and photos deleted/).waitFor();
const wiped = await storedLedger();
if (wiped.raw !== null) throw new Error('delete-all must remove the stored ledger');
if (await page.evaluate(() => JSON.stringify(localStorage).includes('data:image'))) {
throw new Error('no photo bytes may survive delete-all');
}
if (await page.evaluate(key => localStorage.getItem(key), contributionKey) !== null) {
throw new Error('delete-all must clear the contribution receipts as well');
}
await caption('Delete all local data clears the ledger and every saved photo in this browser; nothing is kept on a server', 2900);
await tap('[data-view="home"]', 500);
// Issue #37 — a chat answer about the journal must point at the confirmed records behind it. The
// scenes below run against the app's own local ledger contract with fictional logs, and every claim
@@ -503,7 +1082,8 @@ if (expectedFailClosed.length !== 2 || errors.length !== expectedFailClosed.leng
// Machine-readable receipt of exactly what was rendered on camera, so the release gate can prove
// the recorded delta statement reached the screen instead of trusting the script source.
if (captionsOut) {
await writeFile(captionsOut, JSON.stringify({ version, releaseDelta, captions: captionsShown }, null, 2) + '\n', 'utf8');
await writeFile(captionsOut, JSON.stringify({ version, releaseDelta, captions: captionsShown,
captionsVerified: captionEvidence }, null, 2) + '\n', 'utf8');
}
const recording = page.video();
await page.close();
+1 -1
View File
@@ -1,7 +1,7 @@
const ROOT = new URL(self.registration.scope).pathname;
const appPath = path => `${ROOT}${String(path).replace(/^\/+/, '')}`;
const CACHE_NAMESPACE = `timmy-shell:${ROOT}:`;
const CACHE = `${CACHE_NAMESPACE}v5`;
const CACHE = `${CACHE_NAMESPACE}v6`;
const ASSETS = [
'',
'index.html',
+129 -5
View File
@@ -111,9 +111,32 @@ export function sanitizeEntry(input = {}) {
note: String(input.note || '').trim().slice(0, 500),
photoDataUrl: typeof input.photoDataUrl === 'string' && input.photoDataUrl.startsWith('data:image/') ? input.photoDataUrl : '',
symptoms,
provenance: sanitizeProvenance(input.provenance),
};
}
// Provenance is the only record of whether a confirmed log began as an AI suggestion. It is kept to
// an allowlist so a backup can never smuggle a credential, a prompt, or a raw model reply out of the
// device, and it is `null` for logs that never had one — a missing record is not a manual entry.
const PROVENANCE_SOURCES = Object.freeze(['ai-confirmed', 'ai-corrected']);
function sanitizeProvenance(value) {
if (!value || typeof value !== 'object') return null;
if (!PROVENANCE_SOURCES.includes(value.source)) return null;
const provenance = { source: value.source };
if (typeof value.model === 'string' && value.model.trim()) provenance.model = value.model.trim().slice(0, 80);
const suggestedType = Number(value.suggestedBristolType);
if (Number.isInteger(suggestedType) && suggestedType >= 1 && suggestedType <= 7) {
provenance.suggestedBristolType = suggestedType;
}
if (['brown', 'green', 'yellow', 'pale', 'red', 'black'].includes(value.suggestedColor)) {
provenance.suggestedColor = value.suggestedColor;
}
const confirmedAt = value.confirmedAt ? new Date(value.confirmedAt) : null;
if (confirmedAt && !Number.isNaN(confirmedAt.valueOf())) provenance.confirmedAt = confirmedAt.toISOString();
return provenance;
}
export function photoQualityMessage({ width = 0, height = 0, brightness = 0.5 } = {}) {
if (width < 640 || height < 480) return 'Move a little closer or use a higher-resolution photo. The image stays on this device.';
if (brightness < 0.12) return 'Add more light before saving. Timmy only checks whether the photo is usable.';
@@ -121,19 +144,120 @@ export function photoQualityMessage({ width = 0, height = 0, brightness = 0.5 }
return 'Ready for your review. Choose the matching Bristol form yourself; Timmy does not interpret the picture.';
}
export const LEDGER_EXPORT_VERSION = 2;
export function exportLedger(entries, exportedAt = new Date().toISOString()) {
return exportLedgerBackup({ entries, exportedAt });
}
// A backup is written through an allowlist, so only fields this build understands leave the device.
// Entries are re-sanitized on the way out as well as in: an unknown key on an entry is not part of
// the person's record, and a crafted key must never be laundered through a backup.
export function exportLedgerBackup({ entries = [], contributions = [], exportedAt = new Date().toISOString() } = {}) {
const list = Array.isArray(entries) ? entries.map(sanitizeEntry) : [];
const receipts = (Array.isArray(contributions) ? contributions : []).map(sanitizeContribution).filter(Boolean);
return JSON.stringify({
product: 'Timmy the Talking Turd',
schemaVersion: 1,
schemaVersion: LEDGER_EXPORT_VERSION,
exportedAt,
entries: Array.isArray(entries) ? entries : [],
entries: list,
contributions: receipts,
}, null, 2);
}
const CONTRIBUTION_STATES = Object.freeze(['queued-local', 'withdrawn']);
const CONTRIBUTION_ARTIFACTS = Object.freeze(['raw-image', 'confirmed-label']);
function isoOrNull(value) {
if (!value) return null;
const parsed = new Date(value);
return Number.isNaN(parsed.valueOf()) ? null : parsed.toISOString();
}
export function sanitizeContribution(input = {}) {
const contributionId = String(input?.contributionId || '').trim().slice(0, 120);
if (!contributionId) return null;
const state = CONTRIBUTION_STATES.includes(input.state) ? input.state : 'queued-local';
const receipt = {
contributionId,
entryId: String(input.entryId || '').trim().slice(0, 120),
consentVersion: String(input.consentVersion || '').trim().slice(0, 60),
consentedAt: isoOrNull(input.consentedAt),
labels: {
bristolType: Math.min(7, Math.max(1, Number(input.labels?.bristolType) || 4)),
color: ['brown', 'green', 'yellow', 'pale', 'red', 'black'].includes(input.labels?.color) ? input.labels.color : 'brown',
},
state,
};
const withdrawnAt = isoOrNull(input.withdrawnAt);
if (state === 'withdrawn' && withdrawnAt) receipt.withdrawnAt = withdrawnAt;
if (state === 'withdrawn') {
receipt.artifacts = (Array.isArray(input.artifacts) ? input.artifacts : [])
.filter(artifact => CONTRIBUTION_ARTIFACTS.includes(artifact?.artifact))
.map(artifact => ({
artifact: artifact.artifact,
contributionId: String(artifact.contributionId || contributionId).slice(0, 120),
state: artifact.state === 'withdrawn' ? 'withdrawn' : 'active',
}));
}
return receipt;
}
export function importLedger(text) {
const parsed = JSON.parse(text);
if (parsed?.schemaVersion !== 1 || !Array.isArray(parsed.entries)) throw new Error('This is not a supported Timmy export.');
return parsed.entries.map(sanitizeEntry);
return importLedgerBackup(text).entries;
}
// A backup can be older than this build. Version 1 files record no provenance for their entries, so
// the migration leaves that field empty instead of guessing a source, and the caller is told which
// version the file came from.
export const LEDGER_IMPORT_LIMITS = Object.freeze({ maxTextLength: 16 * 1024 * 1024, maxEntries: 5000 });
export function importLedgerBackup(text) {
const raw = String(text ?? '');
if (raw.length > LEDGER_IMPORT_LIMITS.maxTextLength) {
throw new Error('That backup is too large for Timmy to read safely. Nothing was imported.');
}
let parsed;
try {
parsed = JSON.parse(raw);
} catch {
throw new Error('That file is not readable JSON, so nothing was imported.');
}
if (!parsed || typeof parsed !== 'object' || !Array.isArray(parsed.entries)) {
throw new Error('This is not a supported Timmy backup.');
}
const fromVersion = Number(parsed.schemaVersion);
if (Number.isInteger(fromVersion) && fromVersion > LEDGER_EXPORT_VERSION) {
throw new Error(`This backup was written by a newer version of Timmy (version ${fromVersion}). Update Timmy before importing it; nothing was imported.`);
}
if (!Number.isInteger(fromVersion) || fromVersion < 1) {
throw new Error('This is not a supported Timmy backup.');
}
if (parsed.entries.length > LEDGER_IMPORT_LIMITS.maxEntries) {
throw new Error('That backup holds more logs than Timmy will import at once. Nothing was imported.');
}
const entries = parsed.entries.map(sanitizeEntry);
const rawContributions = Array.isArray(parsed.contributions) ? parsed.contributions : [];
const contributions = rawContributions.map(sanitizeContribution).filter(Boolean);
const migrated = fromVersion < LEDGER_EXPORT_VERSION;
// The caller already says what it restored, so this notice only reports the upgrade. Restating the
// restored count here made the on-screen notice read as two stacked sentences saying one thing.
const notices = migrated
? [`Upgraded from version ${fromVersion} to version ${LEDGER_EXPORT_VERSION}; fields the old file did not record are left empty.`]
: [];
// Skipping an unreadable receipt silently would tell someone their opt-in vanished. Say it.
const skipped = rawContributions.length - contributions.length;
if (skipped > 0) {
notices.push(`${skipped} training contribution ${skipped === 1 ? 'receipt' : 'receipts'} could not be read and ${skipped === 1 ? 'was' : 'were'} skipped.`);
}
return {
entries,
contributions,
fromVersion,
migrated,
notices,
exportedAt: typeof parsed.exportedAt === 'string' ? parsed.exportedAt : null,
};
}
export const urgentSymptomKeys = Object.freeze([...URGENT_KEYS]);
+26
View File
@@ -0,0 +1,26 @@
// Photo lifecycle for saved entries. A confirmed log and the image attached to it are separate
// things a person owns: presence is reported from the stored image bytes, removing the image never
// touches the log, and deleting the log is a separate, explicit decision.
const IMAGE_DATA_URL = /^data:image\/[a-z0-9.+-]+(?:;base64)?,[A-Za-z0-9+/=%.-]+$/i;
export function photoState(entry) {
return IMAGE_DATA_URL.test(String(entry?.photoDataUrl || ''))
? { hasPhoto: true, label: 'Photo saved' }
: { hasPhoto: false, label: '' };
}
export function withoutPhoto(entries, id) {
if (!Array.isArray(entries)) return [];
if (!entries.some(entry => entry.id === id)) return entries;
return entries.map(entry => (entry.id === id ? { ...entry, photoDataUrl: '' } : entry));
}
export function withoutEntry(entries, id) {
if (!Array.isArray(entries)) return [];
if (!entries.some(entry => entry.id === id)) return entries;
return entries.filter(entry => entry.id !== id);
}
export function photoCount(entries) {
return Array.isArray(entries) ? entries.filter(entry => photoState(entry).hasPhoto).length : 0;
}
+103
View File
@@ -0,0 +1,103 @@
// Issue #24 — contributing a confirmed photo to Timmy's training corpus is a separate decision from
// saving a log. A journal save records nothing here; a contribution exists only after someone reads
// the consent terms and accepts them, and it exists as a receipt they can point a withdrawal at.
// Nothing is transmitted: this build queues the receipt on the device so the corpus pipeline (#25)
// has something auditable to consume later.
export const TRAINING_CONSENT_VERSION = 'training-consent-v1';
const IMAGE_DATA_URL = /^data:image\/[a-z0-9.+-]+(?:;base64)?,[A-Za-z0-9+/=%.-]+$/i;
const CONTRIBUTED_ARTIFACTS = ['raw-image', 'confirmed-label'];
export function consentTerms() {
return {
version: TRAINING_CONSENT_VERSION,
includes: [
'The saved photo bytes for this one log',
'The confirmed Bristol form, broad color, and image quality',
],
excludes: [
'Your note text',
'Your symptom answers',
'Urgency and discomfort ratings',
'Anything identifying you or this device',
],
transport: 'Queued on this device only — nothing is uploaded in this build.',
};
}
export function defaultContributionState() {
return { optedIn: false, receipt: null };
}
export function contributionEligibility(entry) {
if (!entry || !entry.id) return { eligible: false, reason: 'This log has no identifier to attach a receipt to.' };
if (!IMAGE_DATA_URL.test(String(entry.photoDataUrl || ''))) {
return { eligible: false, reason: 'This log has no saved photo to contribute.' };
}
if (!Number.isInteger(entry.bristolType) || entry.bristolType < 1 || entry.bristolType > 7) {
return { eligible: false, reason: 'This log has no confirmed Bristol form to contribute.' };
}
if (!entry.color) return { eligible: false, reason: 'This log has no confirmed color to contribute.' };
return { eligible: true, reason: '' };
}
export function receiptPreview(entry, { contributionId, consentedAt } = {}) {
if (!contributionId) throw new Error('A contribution receipt needs its own identifier.');
if (!consentedAt) throw new Error('A contribution receipt needs the moment of consent.');
const eligibility = contributionEligibility(entry);
if (!eligibility.eligible) throw new Error(eligibility.reason);
return {
contributionId,
entryId: entry.id,
consentVersion: TRAINING_CONSENT_VERSION,
consentedAt,
labels: { bristolType: entry.bristolType, color: entry.color },
state: 'queued-local',
};
}
export function contributionForEntry(contributions, entryId) {
if (!Array.isArray(contributions)) return null;
return contributions.find(item => item.entryId === entryId && item.state !== 'withdrawn') || null;
}
export function addContribution(contributions, receipt) {
const list = Array.isArray(contributions) ? contributions : [];
if (!receipt || receipt.consentVersion !== TRAINING_CONSENT_VERSION) {
throw new Error('A contribution needs the current consent version.');
}
if (contributionForEntry(list, receipt.entryId)) return list;
return [...list, receipt];
}
export function derivedArtifacts(receipt) {
const state = receipt?.state === 'withdrawn' ? 'withdrawn' : 'active';
return CONTRIBUTED_ARTIFACTS.map(artifact => ({
artifact,
contributionId: receipt?.contributionId || '',
state,
}));
}
export function withdrawContribution(contributions, contributionId, withdrawnAt) {
const list = Array.isArray(contributions) ? contributions : [];
const receipt = list.find(item => item.contributionId === contributionId);
if (!receipt) return { contributions: list, tombstone: null };
const withdrawn = {
...receipt,
state: 'withdrawn',
withdrawnAt,
artifacts: derivedArtifacts({ ...receipt, state: 'withdrawn' }),
};
return {
contributions: list.map(item => (item.contributionId === contributionId ? withdrawn : item)),
tombstone: {
contributionId: receipt.contributionId,
entryId: receipt.entryId,
consentVersion: receipt.consentVersion,
consentedAt: receipt.consentedAt,
withdrawnAt,
artifacts: withdrawn.artifacts,
},
};
}
+17 -1
View File
@@ -11,7 +11,7 @@ button,input,textarea,select{font:inherit}button{color:inherit}.hidden,.sr-only{
main{display:block}.eyebrow{font-size:11px;font-weight:800;letter-spacing:.1em;text-transform:uppercase;color:var(--teal)}h1,h2,h3,p{margin-top:0}h1{font-size:clamp(32px,9vw,46px);line-height:1.02;letter-spacing:-1.8px;margin:8px 0 12px}h2{font-size:23px;line-height:1.1;letter-spacing:-.65px;margin:4px 0 8px}h3{font-size:16px;margin-bottom:7px}.lead,.page-title p{color:#625a53;line-height:1.5;font-size:15px}
.sleek-hero{padding:25px 2px 16px}.sleek-hero h1{font-size:42px;max-width:390px}.sleek-hero .lead{max-width:460px;margin-bottom:22px}.capture-cta{width:100%;border:0;border-radius:22px;background:var(--ink);color:#fff;display:grid;grid-template-columns:50px 1fr 28px;gap:13px;align-items:center;text-align:left;padding:13px 14px;min-height:78px;box-shadow:0 16px 28px rgba(33,29,26,.2);cursor:pointer}.capture-cta:active{transform:scale(.985)}.capture-cta>span:nth-child(2){display:flex;flex-direction:column;gap:2px}.capture-cta strong{font-size:17px}.capture-cta small{color:rgba(255,255,255,.65);font-size:12px;font-weight:500}.capture-cta i{font-style:normal;font-size:22px}.capture-icon{width:48px;height:48px;border-radius:16px;display:grid;place-items:center;background:var(--gold);color:var(--ink);font-size:25px}.text-action{border:0;background:transparent;color:var(--teal);font-weight:750;min-height:44px;padding:8px 2px;cursor:pointer}.sleek-hero>.text-action{width:100%;margin-top:7px}.text-action.compact{width:auto;min-height:42px;padding:7px 4px;font-size:14px;white-space:nowrap}
.glance{display:flex;align-items:center;justify-content:space-around;background:rgba(255,253,250,.68);border:1px solid var(--line);border-radius:20px;padding:13px 10px;margin-top:7px}.glance div{text-align:center;min-width:68px}.glance strong{font-size:21px;line-height:1;display:block}.glance span{font-size:11px;color:var(--muted)}.glance>i{width:1px;height:28px;background:var(--line)}
.section{margin-top:25px}.section-head{display:flex;justify-content:space-between;align-items:end;gap:12px;margin-bottom:11px}.section-head h2{margin-bottom:0}.insight-card{display:grid;grid-template-columns:54px 1fr;gap:13px;align-items:center;background:var(--teal-soft);border-radius:22px;padding:15px}.insight-card p{margin:3px 0 0;line-height:1.4;font-size:14px}.timmy-orb{width:54px;height:54px;border-radius:18px;background:var(--surface);display:grid;place-items:center}.timmy-orb img{width:45px;height:45px}.latest-card,.entry{display:grid;grid-template-columns:44px 1fr auto;gap:12px;align-items:center}.latest-card{background:var(--surface);border:1px solid var(--line);padding:13px;border-radius:19px}.latest-card>div:nth-child(2),.entry>div:nth-child(2){min-width:0}.latest-card strong,.entry strong{display:block;font-size:14px}.latest-card span:not(.type-dot),.entry span{display:block;color:var(--muted);font-size:12px;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.chevron{font-size:24px!important;color:#aaa!important}.type-dot{width:42px;height:42px;border-radius:14px;display:grid;place-items:center;background:#f0e2d7;color:var(--brown);font-weight:800;font-size:13px}.entries-card{background:var(--surface);border:1px solid var(--line);border-radius:20px;padding:4px 13px}.entry{grid-template-columns:42px minmax(0,1fr) auto;padding:12px 0;border-bottom:1px solid var(--line)}.entry:last-child{border-bottom:0}.bucket{padding:5px 7px;border-radius:999px;background:var(--soft);font-size:10px!important;font-weight:700}.bucket-typical{background:var(--teal-soft);color:var(--teal)}
.section{margin-top:25px}.section-head{display:flex;justify-content:space-between;align-items:end;gap:12px;margin-bottom:11px}.section-head h2{margin-bottom:0}.insight-card{display:grid;grid-template-columns:54px 1fr;gap:13px;align-items:center;background:var(--teal-soft);border-radius:22px;padding:15px}.insight-card p{margin:3px 0 0;line-height:1.4;font-size:14px}.timmy-orb{width:54px;height:54px;border-radius:18px;background:var(--surface);display:grid;place-items:center}.timmy-orb img{width:45px;height:45px}.latest-card,.entry{display:grid;grid-template-columns:44px 1fr auto;gap:12px;align-items:center}.latest-card{background:var(--surface);border:1px solid var(--line);padding:13px;border-radius:19px}.latest-card>div:nth-child(2),.entry>div:nth-child(2){min-width:0}.latest-card strong,.entry strong{display:block;font-size:14px}.latest-card span:not(.type-dot),.entry span{display:block;color:var(--muted);font-size:12px;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.chevron{font-size:24px!important;color:#aaa!important}.type-dot{width:42px;height:42px;border-radius:14px;display:grid;place-items:center;background:#f0e2d7;color:var(--brown);font-weight:800;font-size:13px}.entries-card{background:var(--surface);border:1px solid var(--line);border-radius:20px;padding:4px 13px}.entry{grid-template-columns:42px minmax(0,1fr) auto;padding:12px 0;border-bottom:1px solid var(--line)}.entry:last-child{border-bottom:0}.bucket{padding:5px 7px;border-radius:999px;background:var(--soft);font-size:10px!important;font-weight:700}.entry-open{width:44px;height:44px;margin-right:-6px;border:0;border-radius:50%;background:transparent;color:#aaa;font-size:24px;line-height:1;cursor:pointer;display:grid;place-items:center}.entry-open:active{background:var(--soft)}.entry-open:focus-visible{outline:3px solid var(--teal);outline-offset:-3px}.entry-tail{display:flex!important;align-items:center;gap:8px}.photo-chip{display:inline-flex!important;align-items:center;gap:5px;margin-top:5px;padding:3px 8px;border-radius:999px;background:var(--teal-soft);color:var(--teal)!important;font-size:10px!important;font-style:normal;font-weight:800;letter-spacing:.04em;text-transform:uppercase}.entry-facts{display:grid;grid-template-columns:1fr 1fr;gap:9px;margin:14px 0}.entry-facts>div{background:#f7f3ed;border-radius:14px;padding:10px 12px}.entry-facts span{display:block;font-size:10px;font-weight:800;letter-spacing:.06em;color:var(--muted);text-transform:uppercase}.entry-facts strong{display:block;margin-top:3px;font-size:15px}.entry-note{background:#faf7f1;border-radius:14px;padding:10px 12px}.photo-lifecycle{background:var(--teal-soft);border-radius:16px;padding:13px;margin:13px 0}.photo-lifecycle[data-photo-lifecycle="absent"]{background:#f5f1e9}.photo-lifecycle strong{display:block;font-size:14px}.photo-lifecycle .row{margin-top:10px}.entry-actions{margin-top:16px;padding-top:14px;border-top:1px solid var(--line)}.entry-actions .btn{margin-bottom:8px}.bucket-typical{background:var(--teal-soft);color:var(--teal)}
.page-title{padding:21px 3px 14px}.page-title h1{font-size:38px}.journal-title{padding-top:18px}.calendar-card,.card{background:var(--surface);border:1px solid var(--line);border-radius:22px;padding:16px}.calendar{display:grid;grid-template-columns:repeat(7,1fr);gap:7px}.day,.cal-head{aspect-ratio:1;display:grid;place-items:center;font-size:12px;border-radius:12px}.cal-head{aspect-ratio:auto;color:var(--muted);font-weight:700;padding-bottom:6px}.day{background:#f2eee6}.day.blank{background:transparent}.day.has-log{background:var(--teal);color:white;font-weight:800;box-shadow:inset 0 0 0 3px var(--teal-soft)}.journal-settings{margin-top:18px}.settings-row{width:100%;min-height:54px;border:1px solid var(--line);background:rgba(255,253,250,.6);border-radius:18px;display:flex;align-items:center;justify-content:space-between;padding:0 16px;font-weight:700;cursor:pointer}.settings-row b{font-size:23px;color:var(--muted)}
.bottom-nav{position:fixed;left:50%;bottom:calc(10px + env(safe-area-inset-bottom));transform:translateX(-50%);width:min(calc(100% - 24px),640px);height:68px;padding:5px;display:grid;grid-template-columns:repeat(3,1fr);background:rgba(255,253,249,.88);border:1px solid rgba(70,54,43,.12);border-radius:23px;box-shadow:0 16px 38px rgba(50,37,29,.18);backdrop-filter:blur(20px);z-index:20}.nav-btn{border:0;background:transparent;border-radius:18px;display:flex;flex-direction:column;align-items:center;justify-content:center;gap:1px;color:var(--muted);font-size:11px;font-weight:700;cursor:pointer}.nav-btn b{font-size:19px;line-height:1}.nav-btn.active{background:var(--ink);color:#fff;box-shadow:0 6px 14px rgba(33,29,26,.15)}
.btn{min-height:50px;border:0;border-radius:16px;padding:0 17px;font-weight:800;cursor:pointer;display:inline-flex;align-items:center;justify-content:center;gap:8px}.btn:active{transform:scale(.98)}.btn:disabled{opacity:.45;cursor:not-allowed}.btn-primary{background:var(--ink);color:#fff}.btn-secondary{background:var(--teal-soft);color:var(--teal)}.btn-ghost{background:transparent;border:1px solid var(--line)}.btn-danger{background:var(--red);color:#fff}.btn-wide{width:100%}.row{display:flex;gap:10px;align-items:center;flex-wrap:wrap}.fine{font-size:12px;color:var(--muted);line-height:1.45}.empty{text-align:center;padding:22px 10px;color:var(--muted)}.empty img{width:68px}.empty h3{color:var(--ink);margin-top:8px}.empty p{margin-bottom:0}
@@ -59,3 +59,19 @@ main{display:block}.eyebrow{font-size:11px;font-weight:800;letter-spacing:.1em;t
.chat-citation{min-height:32px;padding:5px 10px;border:1px solid #bcdfd4;border-radius:999px;background:var(--surface);color:var(--teal);font-weight:800;font-size:11px;cursor:pointer}
.chat-citation:active{background:#e9f5f0}
.chat-unverified{margin:9px 0 0;padding-top:8px;border-top:1px solid rgba(55,40,31,.09);font-size:11px;line-height:1.4;font-weight:700;color:#8a5a12}
/* Issue #24 — an opt-in receipt and its tombstone must read as separate facts, not one run-on line. */
.contribution-list{display:flex;flex-direction:column;margin-top:10px}
.training-consent h3{font-size:13px;margin:16px 0 2px}
.contribution-row{display:flex;align-items:center;justify-content:space-between;gap:12px;padding:11px 0;border-top:1px solid var(--line)}
.contribution-row:first-child{border-top:0}
.contribution-row>div{display:flex;flex-direction:column;gap:3px;min-width:0}
.contribution-row strong{font-size:13px;line-height:1.3}
.contribution-row span{font-size:11px;line-height:1.4;color:var(--muted);overflow-wrap:anywhere}
.contribution-row .btn{min-height:40px;padding:0 14px;font-size:12px;flex:0 0 auto}
.contribution-row.withdrawn strong{color:var(--muted)}
/* Issue #35 — a backup says which version it came from, and a refused file says so in place. */
.portability-notice{margin:0 0 12px;padding:12px 14px;border-radius:15px;font-size:12.5px;line-height:1.45;font-weight:700;background:var(--teal-soft);color:#155c4f;border:1px solid #bcdfd4}
.portability-notice.error{background:var(--red-soft);border-color:#eec1ba;color:#75261f}
.entry-provenance{margin:10px 0 0;padding-top:9px;border-top:1px solid rgba(55,40,31,.09)}
+306
View File
@@ -0,0 +1,306 @@
#!/usr/bin/env python3
"""Tests for the contributor-held-out evaluation harness.
Issue #29 asks for the authoritative comparison and release-gate harness: macro-F1, per-class
precision/recall, a confusion matrix, calibration error, coverage, and precision-at-coverage, over
splits that are contributor-held-out and reproducible, comparing candidate models.
Every fixture in this file is synthetic: fictional pseudonymous subject keys and invented
prediction scores. No real photo, label, or patient record is read or written.
"""
from __future__ import annotations
import importlib.util
import json
import subprocess
import sys
import tempfile
from pathlib import Path
import unittest
ROOT = Path(__file__).resolve().parents[1]
HARNESS = ROOT / "scripts" / "calibration_report.py"
def load_harness():
spec = importlib.util.spec_from_file_location("calibration_report", HARNESS)
assert spec and spec.loader, "scripts/calibration_report.py must exist"
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
harness = load_harness()
def record(subject_key: str, truth: int, predicted: int | None, confidence: float, *, split: str | None = None, model: str = "frozen-encoder") -> dict:
return {
"model": model,
"subjectKey": subject_key,
"split": split or harness.expected_split(subject_key),
"truth": {"bristolType": truth},
"prediction": None if predicted is None else {"bristolType": predicted, "confidence": confidence},
}
class ComparisonTests(unittest.TestCase):
def models(self):
return {
"specialist-candidate": [
record("cd00000000000001", 4, 4, 1.0),
record("ce00000000000002", 3, 3, 1.0),
record("e600000000000001", 4, 4, 1.0),
record("e700000000000002", 6, 6, 1.0),
],
"vlm-bootstrap": [
record("d000000000000004", 4, 4, 1.0),
record("e600000000000003", 4, 2, 1.0),
record("e700000000000004", 6, 6, 1.0),
],
}
def test_measures_each_candidate_on_test_at_a_threshold_chosen_on_validation(self):
verdict = harness.compare(self.models())
self.assertEqual([row["model"] for row in verdict["models"]], ["specialist-candidate", "vlm-bootstrap"])
specialist = verdict["models"][0]
self.assertEqual(specialist["selectionSplit"], "validation")
self.assertEqual(specialist["threshold"], 0.0)
self.assertEqual(specialist["evaluatedSplit"], "test")
self.assertAlmostEqual(specialist["macroF1"], 1.0, places=6)
self.assertTrue(specialist["gate"]["passed"])
self.assertEqual(specialist["gate"]["failed"], [])
self.assertEqual(verdict["recommended"], "specialist-candidate")
self.assertEqual(verdict["verdict"], "GO")
def test_a_candidate_that_is_only_half_right_fails_the_macro_f1_precision_and_calibration_gates(self):
vlm = next(row for row in harness.compare(self.models())["models"] if row["model"] == "vlm-bootstrap")
self.assertAlmostEqual(vlm["precisionAtCoverage"], 0.5, places=6)
self.assertFalse(vlm["gate"]["passed"])
self.assertEqual(vlm["gate"]["failed"], ["macroF1", "precisionAtCoverage", "ece"])
def test_blocks_the_release_when_no_candidate_clears_every_gate(self):
models = {"vlm-bootstrap": self.models()["vlm-bootstrap"]}
verdict = harness.compare(models)
self.assertEqual(verdict["verdict"], "BLOCKED")
self.assertIsNone(verdict["recommended"])
self.assertEqual(verdict["gates"], {"macroF1": 0.80, "precisionAtCoverage": 0.90, "ece": 0.05})
def test_gate_limits_are_inclusive_so_a_candidate_exactly_on_the_line_passes(self):
records = [
record("cd00000000000001", 4, 4, 0.9),
record("e600000000000001", 4, 4, 0.9),
record("e700000000000002", 3, 2, 0.9),
]
verdict = harness.compare({"frozen-encoder": records}, gates={"macroF1": 0.0, "precisionAtCoverage": 0.5, "ece": 1.0})
row = verdict["models"][0]
self.assertAlmostEqual(row["precisionAtCoverage"], 0.5, places=6)
self.assertTrue(row["gate"]["passed"], row["gate"])
def test_refuses_a_candidate_with_no_held_out_predictions(self):
with self.assertRaisesRegex(harness.EvaluationError, "no test-split predictions"):
harness.compare({"frozen-encoder": [record("cd00000000000001", 4, 4, 1.0)]})
class CliTests(unittest.TestCase):
def write_predictions(self, directory: Path, name: str, rows: list[dict]) -> Path:
path = directory / name
path.write_text("\n".join(json.dumps(row) for row in rows) + "\n", encoding="utf-8")
return path
def run_cli(self, *args: str) -> subprocess.CompletedProcess:
return subprocess.run(
[sys.executable, str(HARNESS), *args], cwd=ROOT, capture_output=True, text=True
)
def test_reports_a_go_with_the_candidate_table_and_writes_the_json_report(self):
with tempfile.TemporaryDirectory() as raw:
directory = Path(raw)
predictions = self.write_predictions(directory, "predictions.jsonl", [
{**record("cd00000000000001", 4, 4, 1.0), "model": "specialist-candidate"},
{**record("e600000000000001", 4, 4, 1.0), "model": "specialist-candidate"},
{**record("e700000000000002", 6, 6, 1.0), "model": "specialist-candidate"},
])
output = directory / "report.json"
run = self.run_cli("--predictions", str(predictions), "--json", str(output), "--min-coverage", "0.5")
self.assertEqual(run.returncode, 0, run.stderr)
self.assertIn("verdict: GO", run.stdout)
self.assertIn("specialist-candidate", run.stdout)
self.assertIn("macroF1", run.stdout)
report = json.loads(output.read_text(encoding="utf-8"))
self.assertEqual(report["verdict"], "GO")
self.assertEqual(report["recommended"], "specialist-candidate")
self.assertEqual(report["models"][0]["evaluatedSplit"], "test")
self.assertEqual(report["models"][0]["selectionSplit"], "validation")
def test_exits_three_when_the_release_gate_blocks(self):
with tempfile.TemporaryDirectory() as raw:
directory = Path(raw)
predictions = self.write_predictions(directory, "predictions.jsonl", [
{**record("cd00000000000001", 4, 3, 1.0), "model": "vlm-bootstrap"},
{**record("e600000000000001", 4, 2, 1.0), "model": "vlm-bootstrap"},
])
run = self.run_cli("--predictions", str(predictions))
self.assertEqual(run.returncode, 3, run.stderr)
self.assertIn("verdict: BLOCKED", run.stdout)
def test_refuses_an_out_of_range_label_instead_of_scoring_it(self):
with tempfile.TemporaryDirectory() as raw:
directory = Path(raw)
output = directory / "report.json"
bad_truth = self.write_predictions(directory, "truth.jsonl", [
{**record("cd00000000000001", 9, 9, 1.0), "model": "frozen-encoder"},
])
run = self.run_cli("--predictions", str(bad_truth), "--json", str(output))
self.assertEqual(run.returncode, 2)
self.assertIn("outside Bristol 1-7", run.stderr)
self.assertFalse(output.exists(), "a refused run must not leave a report behind")
bad_prediction = self.write_predictions(directory, "prediction.jsonl", [
{**record("e600000000000001", 4, 9, 1.0), "model": "frozen-encoder"},
])
run = self.run_cli("--predictions", str(bad_prediction))
self.assertEqual(run.returncode, 2)
self.assertIn("predicted bristolType must be 1-7", run.stderr)
def test_refuses_a_corpus_that_leaks_a_contributor_across_splits(self):
with tempfile.TemporaryDirectory() as raw:
directory = Path(raw)
predictions = self.write_predictions(directory, "predictions.jsonl", [
{**record("cd00000000000001", 4, 4, 1.0, split="train"), "model": "frozen-encoder"},
{**record("cd00000000000001", 4, 4, 1.0, split="validation"), "model": "frozen-encoder"},
{**record("e600000000000001", 4, 4, 1.0), "model": "frozen-encoder"},
])
run = self.run_cli("--predictions", str(predictions))
self.assertEqual(run.returncode, 2)
self.assertIn("leaks across", run.stderr)
class ThresholdSelectionTests(unittest.TestCase):
def validation_records(self):
# 0xcd–0xe5 bucket to validation under the ingestion rule. Confident answers are right and
# the least confident one is wrong, which is what a threshold is supposed to exploit.
return [
record("cd00000000000001", 4, 4, 0.90),
record("ce00000000000002", 4, 4, 0.80),
record("cf00000000000003", 3, 3, 0.60),
record("d000000000000004", 7, 4, 0.40),
]
def test_picks_the_most_precise_threshold_that_still_clears_the_coverage_floor(self):
records = self.validation_records()
# At a 0.5 floor, 0.60 and 0.80 are both perfect; the wider cut wins, so 0.60.
floored = harness.select_threshold(records, min_coverage=0.5)
self.assertEqual(floored["threshold"], 0.6)
self.assertEqual(floored["counts"], {"total": 4, "accepted": 3, "abstained": 1})
self.assertAlmostEqual(floored["precisionAtCoverage"], 1.0, places=6)
self.assertAlmostEqual(floored["coverage"], 0.75, places=6)
self.assertEqual(floored["selectionSplit"], "validation")
# A 0.8 floor forbids that cut, so the harness falls back to the widest honest threshold
# instead of silently returning a threshold that violates the floor.
strict = harness.select_threshold(records, min_coverage=0.8)
self.assertEqual(strict["threshold"], 0.0)
self.assertAlmostEqual(strict["precisionAtCoverage"], 0.75, places=6)
def test_refuses_an_unreachable_coverage_floor(self):
with self.assertRaisesRegex(harness.EvaluationError, "coverage floor"):
harness.select_threshold(self.validation_records(), min_coverage=1.5)
def test_refuses_to_choose_a_threshold_on_test_or_train_data(self):
test_only = [record("e700000000000001", 4, 4, 0.9)]
with self.assertRaisesRegex(harness.EvaluationError, "validation split only"):
harness.select_threshold(test_only, min_coverage=0.5)
train_mixed = [record("aa00000000000001", 4, 4, 0.9), record("cd00000000000001", 3, 3, 0.8)]
with self.assertRaisesRegex(harness.EvaluationError, "validation split only"):
harness.select_threshold(train_mixed, min_coverage=0.5)
class ContributorHeldOutTests(unittest.TestCase):
def test_refuses_a_split_that_cannot_be_derived_from_the_subject_key(self):
# 0xaa = 170, below the train/validation edge, so this subject can only ever be train.
records = [record("aa00000000000001", 4, 4, 0.9, split="test")]
with self.assertRaisesRegex(harness.EvaluationError, "not reproducible"):
harness.assert_contributor_held_out(records)
def test_refuses_a_contributor_whose_photos_span_two_splits(self):
records = [
record("aa00000000000001", 4, 4, 0.9, split="train"),
record("aa00000000000001", 3, 3, 0.8, split="validation"),
]
with self.assertRaisesRegex(harness.EvaluationError, "leaks across"):
harness.assert_contributor_held_out(records)
def test_allows_one_contributor_to_supply_several_photos_in_the_same_split(self):
records = [
record("aa00000000000001", 4, 4, 0.9),
record("aa00000000000001", 3, 3, 0.8),
]
harness.assert_contributor_held_out(records) # several photos, one person, one split: fine
class CalibrationTests(unittest.TestCase):
def test_calibration_error_weights_each_confidence_bin_by_its_share_of_accepted_suggestions(self):
records = [
record("aa00000000000001", 4, 4, 0.95),
record("ab00000000000002", 4, 4, 0.90),
record("ac00000000000003", 3, 3, 0.55),
record("ad00000000000004", 7, 7, 0.70),
]
# Bins [0.9,1.0) hold both 0.95 and 0.90: mean confidence 0.925, accuracy 1.0, gap 0.075.
self.assertAlmostEqual(harness.evaluate(records, threshold=0.0)["ece"], 0.225, places=6)
# Raising the threshold to 0.6 drops the 0.55 sample, so it can no longer pull calibration.
self.assertAlmostEqual(harness.evaluate(records, threshold=0.6)["ece"], 0.15, places=6)
class AbstentionTests(unittest.TestCase):
def test_low_confidence_and_explicit_abstentions_shrink_coverage_but_never_count_as_errors(self):
records = [
record("aa00000000000001", 4, 4, 0.90),
record("ab00000000000002", 4, 3, 0.55),
record("ac00000000000003", 3, 3, 0.40),
record("ad00000000000004", 7, None, 0.0),
]
report = harness.evaluate(records, threshold=0.6)
self.assertEqual(report["counts"], {"total": 4, "accepted": 1, "abstained": 3})
self.assertAlmostEqual(report["coverage"], 0.25, places=6)
self.assertAlmostEqual(report["precisionAtCoverage"], 1.0, places=6)
# Only the accepted suggestion is scored; the class that abstained away is named, not hidden.
self.assertEqual(sorted(report["perClass"]), [4])
self.assertAlmostEqual(report["macroF1"], 1.0, places=6)
self.assertEqual(report["excludedClasses"], [3, 7])
self.assertEqual(report["abstentions"], {"low-confidence": 2, "no-suggestion": 1})
def test_a_threshold_above_every_confidence_abstains_on_everything_without_crashing(self):
records = [
record("aa00000000000001", 4, 4, 0.30),
record("ab00000000000002", 2, 2, 0.50),
]
report = harness.evaluate(records, threshold=0.95)
self.assertEqual(report["counts"], {"total": 2, "accepted": 0, "abstained": 2})
self.assertAlmostEqual(report["coverage"], 0.0, places=6)
self.assertAlmostEqual(report["macroF1"], 0.0, places=6)
self.assertEqual(report["perClass"], {})
class MacroF1Tests(unittest.TestCase):
def test_scores_each_bristol_class_and_averages_only_the_classes_present_in_truth(self):
records = [
record("aa00000000000001", 4, 4, 0.90),
record("ab00000000000002", 4, 3, 0.80),
record("ac00000000000003", 3, 3, 0.70),
record("ad00000000000004", 7, 7, 0.60),
]
report = harness.evaluate(records, threshold=0.0)
self.assertEqual(sorted(report["perClass"]), [3, 4, 7])
self.assertAlmostEqual(report["perClass"][4]["precision"], 1.0, places=6)
self.assertAlmostEqual(report["perClass"][4]["recall"], 0.5, places=6)
self.assertAlmostEqual(report["perClass"][4]["f1"], 0.666667, places=6)
self.assertAlmostEqual(report["perClass"][3]["precision"], 0.5, places=6)
self.assertAlmostEqual(report["perClass"][3]["recall"], 1.0, places=6)
self.assertAlmostEqual(report["perClass"][7]["f1"], 1.0, places=6)
self.assertAlmostEqual(report["macroF1"], 0.777778, places=6)
self.assertEqual(report["confusion"], {"4->4": 1, "4->3": 1, "3->3": 1, "7->7": 1})
if __name__ == "__main__":
unittest.main()
+17
View File
@@ -174,3 +174,20 @@ test('the deterministic symptom escalation regression suite runs in the default
assert.match(builder, /"src\/domain\.js"/);
assert.match(builder, /"src\/hermes-agent-service\.js"/);
});
test('the calibration harness is wired into the unit and syntax gates', async () => {
const [packageJsonText, workflow] = await Promise.all([
readFile(packagePath, 'utf8'),
readFile(workflowPath, 'utf8'),
]);
const packageJson = JSON.parse(packageJsonText);
// Issue #29: a release gate nothing runs is a claim, not a gate.
assert.equal(packageJson.scripts['test:calibration'], 'python3 tests/calibration-harness.test.py');
assert.match(packageJson.scripts['check:syntax'], /py_compile[^&]*scripts\/calibration_report\.py/);
assert.match(workflow, /python3 tests\/calibration-harness\.test\.py -v/);
// The release builder has to run it too, or the gate exists only in CI.
const builder = await readFile(new URL('../scripts/build_release.py', import.meta.url), 'utf8');
assert.match(builder, /tests\/calibration-harness\.test\.py/);
assert.match(builder, /"calibration_harness_gates": "passed"/);
});
+273
View File
@@ -0,0 +1,273 @@
#!/usr/bin/env python3
"""Tests for the auditable corpus manifest: provenance, tombstones, and lifecycle rules.
Issue #25 promotes the ingest script into a corpus pipeline with an append-only manifest,
revocation tombstones, and lifecycle transitions that cannot be walked backwards. Every
fixture here is synthetic; no real photo bytes are read or written.
"""
from __future__ import annotations
import hashlib
import importlib.util
import json
import os
import re
import subprocess
import sys
import tempfile
from pathlib import Path
import unittest
from PIL import Image
ROOT = Path(__file__).resolve().parents[1]
INGEST = ROOT / "scripts" / "ingest_training_photo.py"
FIXTURE = ROOT / "tests" / "fixtures" / "synthetic-type4.jpg"
CONTRIBUTION_ID = "6f1d0c9e-2c4a-4f3b-9d1e-8a7b6c5d4e3f"
ISO_UTC = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?\+00:00$")
def load_structures():
spec = importlib.util.spec_from_file_location("corpus_manifest", ROOT / "scripts" / "corpus_manifest.py")
assert spec and spec.loader
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
structures = load_structures()
def active_image_paths(out: Path) -> list[str]:
return [record["relativePath"] for record in structures.active_records(structures.load_manifest(out / "manifest.jsonl"))]
def ingest(args: list[str], env: dict[str, str] | None = None) -> subprocess.CompletedProcess:
return subprocess.run(
[sys.executable, str(INGEST), *args],
cwd=ROOT,
capture_output=True,
text=True,
env={**os.environ, "TIMMY_DATASET_SALT": "test-only-salt", **(env or {})},
)
def ingest_default(out: Path, **overrides: str) -> subprocess.CompletedProcess:
args = [
"--input", str(FIXTURE),
"--output-dir", str(out),
"--subject-id", "subject-123",
"--consent-version", "research-v1",
"--contribution-id", CONTRIBUTION_ID,
"--bristol-type", "4",
"--color", "brown",
"--quality", "good",
"--reviewer", "user-confirmed",
]
for key, value in overrides.items():
flag = "--" + key.replace("_", "-")
if value is None:
index = args.index(flag)
del args[index:index + 2]
continue
args[args.index(flag) + 1] = value
return ingest(args)
def manifest_lines(out: Path) -> list[dict]:
text = (out / "manifest.jsonl").read_text(encoding="utf-8")
return [json.loads(line) for line in text.splitlines() if line.strip()]
class IngestProvenanceTests(unittest.TestCase):
def test_ingest_records_consent_source_hashes_derivation_review_split_and_lifecycle(self):
with tempfile.TemporaryDirectory() as raw:
out = Path(raw)
run = ingest_default(out)
self.assertEqual(run.returncode, 0, run.stderr)
record = json.loads(run.stdout)
raw_bytes = FIXTURE.read_bytes()
self.assertEqual(record["schemaVersion"], 2)
self.assertEqual(record["contributionId"], CONTRIBUTION_ID)
self.assertEqual(record["consentVersion"], "research-v1")
self.assertEqual(record["consent"], {"version": "research-v1", "recordedAt": record["ingestedAt"]})
self.assertRegex(record["consent"]["recordedAt"], ISO_UTC)
# Source is named, and the raw input only ever appears as a digest.
self.assertEqual(record["source"], {
"kind": "cli-ingest",
"inputSha256": hashlib.sha256(raw_bytes).hexdigest(),
})
self.assertEqual(record["hashes"]["rawInput"], hashlib.sha256(raw_bytes).hexdigest())
self.assertEqual(record["hashes"]["derived"], record["imageId"])
self.assertEqual(record["derivation"], {
"format": "jpeg",
"maxDimension": 1200,
"metadataStripped": True,
"pipeline": "timmy-ingest/2",
})
self.assertEqual(record["review"], {"status": "user-confirmed", "reviewedAt": record["ingestedAt"]})
self.assertIn(record["split"], ("train", "validation", "test"))
self.assertEqual(record["lifecycle"], {"state": "active", "updatedAt": record["ingestedAt"]})
self.assertEqual(record["recordType"], "sample")
# Provenance never leaks the caller's path or the source bytes.
self.assertNotIn("inputPath", record)
on_disk = (out / "manifest.jsonl").read_text(encoding="utf-8")
self.assertNotIn(str(FIXTURE), on_disk)
self.assertNotIn(str(out), on_disk)
self.assertEqual(manifest_lines(out), [record])
self.assertTrue((out / record["relativePath"]).is_file())
class RevocationTests(unittest.TestCase):
def test_revocation_appends_a_tombstone_and_removes_the_bytes_from_active_datasets(self):
with tempfile.TemporaryDirectory() as raw:
out = Path(raw)
self.assertEqual(ingest_default(out).returncode, 0)
sample = manifest_lines(out)[0]
derived = out / sample["relativePath"]
self.assertTrue(derived.is_file())
# Some pipelines keep the pre-derivation copy; revocation must clear that too.
raw_dir = out / "raw"
raw_dir.mkdir()
raw_copy = raw_dir / f"{sample['hashes']['rawInput']}.jpg"
raw_copy.write_bytes(FIXTURE.read_bytes())
run = ingest(["--revoke", CONTRIBUTION_ID, "--output-dir", str(out), "--reason", "subject withdrawal"])
self.assertEqual(run.returncode, 0, run.stderr)
tombstone = json.loads(run.stdout)
self.assertEqual(tombstone["schemaVersion"], 2)
self.assertEqual(tombstone["recordType"], "tombstone")
self.assertEqual(tombstone["contributionId"], CONTRIBUTION_ID)
self.assertEqual(tombstone["imageId"], sample["imageId"])
self.assertEqual(tombstone["lifecycle"]["state"], "revoked")
self.assertRegex(tombstone["lifecycle"]["updatedAt"], ISO_UTC)
self.assertEqual(tombstone["lifecycle"]["reason"], "subject withdrawal")
# The tombstone addresses every derived artifact by contribution id, and says what went.
self.assertEqual(tombstone["deletedArtifacts"], [
{"relativePath": sample["relativePath"], "sha256": sample["hashes"]["derived"], "deleted": True},
{"relativePath": f"raw/{sample['hashes']['rawInput']}.jpg", "sha256": sample["hashes"]["rawInput"], "deleted": True},
])
self.assertEqual(tombstone["contributionId"], CONTRIBUTION_ID)
# No byte-bearing copy survives anywhere in the dataset tree.
self.assertFalse(derived.exists())
self.assertFalse(raw_copy.exists())
survivors = []
for path in out.rglob("*"):
if path.is_file() and path.name != "manifest.jsonl":
survivors.append((str(path.relative_to(out)), hashlib.sha256(path.read_bytes()).hexdigest()))
self.assertEqual(survivors, [])
# History is append-only: the sample record is untouched, the tombstone follows it.
history = manifest_lines(out)
self.assertEqual(len(history), 2)
self.assertEqual(history[0], sample)
self.assertEqual(history[1], tombstone)
# A deletion receipt keeps identifiers and hashes, never the label or subject key.
self.assertNotIn("labels", tombstone)
self.assertNotIn("subjectKey", tombstone)
self.assertNotIn("relativePath", tombstone)
class ActiveRecordTests(unittest.TestCase):
def test_only_unrevoked_samples_are_active(self):
with tempfile.TemporaryDirectory() as raw:
out = Path(raw)
self.assertEqual(ingest_default(out).returncode, 0)
self.assertEqual(active_image_paths(out), [f"images/{manifest_lines(out)[0]['imageId']}.jpg"])
self.assertEqual(ingest(["--revoke", CONTRIBUTION_ID, "--output-dir", str(out), "--reason", "gone"]).returncode, 0)
self.assertEqual(active_image_paths(out), [])
class IllegalTransitionTests(unittest.TestCase):
def test_a_second_revocation_is_refused_and_appends_nothing(self):
with tempfile.TemporaryDirectory() as raw:
out = Path(raw)
self.assertEqual(ingest_default(out).returncode, 0)
self.assertEqual(ingest(["--revoke", CONTRIBUTION_ID, "--output-dir", str(out), "--reason", "first"]).returncode, 0)
history = manifest_lines(out)
self.assertEqual(len(history), 2)
again = ingest(["--revoke", CONTRIBUTION_ID, "--output-dir", str(out), "--reason", "again"])
self.assertNotEqual(again.returncode, 0)
self.assertRegex(again.stderr, "already revoked")
self.assertEqual(manifest_lines(out), history)
def test_revoking_an_unknown_contribution_is_refused(self):
with tempfile.TemporaryDirectory() as raw:
out = Path(raw)
self.assertEqual(ingest_default(out).returncode, 0)
history = manifest_lines(out)
unknown = ingest(["--revoke", "11111111-2222-3333-4444-555555555555", "--output-dir", str(out), "--reason", "nope"])
self.assertNotEqual(unknown.returncode, 0)
self.assertRegex(unknown.stderr, "no ingested sample carries contribution")
self.assertEqual(manifest_lines(out), history)
self.assertTrue((out / history[0]["relativePath"]).is_file())
def test_a_contribution_cannot_be_ingested_twice(self):
with tempfile.TemporaryDirectory() as raw:
out = Path(raw)
self.assertEqual(ingest_default(out).returncode, 0)
history = manifest_lines(out)
duplicate = ingest_default(out, color="green")
self.assertNotEqual(duplicate.returncode, 0)
self.assertRegex(duplicate.stderr, "already ingested")
self.assertEqual(manifest_lines(out), history)
def test_a_refused_duplicate_leaves_no_orphan_image_behind(self):
with tempfile.TemporaryDirectory() as raw:
out = Path(raw)
self.assertEqual(ingest_default(out).returncode, 0)
history = manifest_lines(out)
# Same contribution id, different pixels: the derived digest differs, so a guard that
# runs after the write would strand a second image in the active dataset.
variant = out.parent / "variant-type4.jpg"
with Image.open(FIXTURE) as source:
changed = source.convert("RGB")
changed.putpixel((4, 4), (255, 0, 0))
changed.save(variant, "JPEG", quality=95)
duplicate = ingest_default(out, input=str(variant))
self.assertNotEqual(duplicate.returncode, 0)
self.assertRegex(duplicate.stderr, "already ingested")
self.assertEqual(manifest_lines(out), history)
images = sorted(path.name for path in (out / "images").iterdir())
self.assertEqual(images, [f"{history[0]['imageId']}.jpg"])
def test_lifecycle_transitions_cannot_walk_backwards(self):
with tempfile.TemporaryDirectory() as raw:
out = Path(raw)
self.assertEqual(ingest_default(out).returncode, 0)
sample = manifest_lines(out)[0]
at = sample["ingestedAt"]
revoked = structures.transition(sample, "revoked", at, "withdrawal")
self.assertEqual(revoked["lifecycle"]["state"], "revoked")
self.assertEqual(revoked["lifecycle"]["reason"], "withdrawal")
self.assertEqual(sample["lifecycle"]["state"], "active", "transition must not mutate its input")
for args, expected in (
((revoked, "active", at), "cannot return to the active dataset"),
((sample, "revoked", at, " "), "recorded reason"),
((sample, "archived", at), "unknown lifecycle state"),
(({**sample, "lifecycle": {"state": "active", "updatedAt": at}}, "active", at), "already active"),
):
with self.assertRaises(structures.IllegalTransition) as raised:
structures.transition(*args)
self.assertRegex(str(raised.exception), expected)
if __name__ == "__main__":
unittest.main(verbosity=2)
+6 -3
View File
@@ -8,6 +8,7 @@ import {
detectUrgentText,
exportLedger,
hasUrgentLedgerContext,
LEDGER_EXPORT_VERSION,
photoQualityMessage,
sanitizeEntry,
} from '../src/domain.js';
@@ -113,9 +114,10 @@ test('sanitizes a user entry to the MVP data contract', () => {
});
assert.deepEqual(Object.keys(entry).sort(), [
'bristolType', 'color', 'discomfort', 'id', 'note', 'occurredAt',
'photoDataUrl', 'symptoms', 'urgency'
'photoDataUrl', 'provenance', 'symptoms', 'urgency'
].sort());
assert.equal(entry.unexpected, undefined);
assert.equal(entry.provenance, null, 'a manual entry records no AI provenance rather than an invented one');
});
test('photo quality guidance is deterministic and does not claim visual diagnosis', () => {
@@ -125,10 +127,11 @@ test('photo quality guidance is deterministic and does not claim visual diagnosi
assert.doesNotMatch(photoQualityMessage({ width: 1200, height: 900, brightness: 0.5 }), /type [1-7]|disease|diagnos/i);
});
test('export ledger is portable JSON with version and entries', () => {
test('export ledger is portable JSON with the current version and entries', () => {
const text = exportLedger([{ id: 'a', bristolType: 4 }], '2026-08-18T00:00:00.000Z');
const parsed = JSON.parse(text);
assert.equal(parsed.schemaVersion, 1);
assert.equal(parsed.schemaVersion, LEDGER_EXPORT_VERSION);
assert.equal(parsed.schemaVersion, 2);
assert.equal(parsed.exportedAt, '2026-08-18T00:00:00.000Z');
assert.equal(parsed.entries.length, 1);
});
+31
View File
@@ -0,0 +1,31 @@
# Synthetic calibration fixtures
These files prove the evaluation harness in `scripts/calibration_report.py` (issue #29) computes the
release gates correctly. **They are not measurements.** No model produced them, no contributor
supplied them, and no number here says anything about how well Timmy labels a real photo.
Every row is a promise in the same shape the harness accepts:
```json
{"model":"…","subjectKey":"<16 hex>","split":"validation","truth":{"bristolType":4},
"prediction":{"bristolType":4,"confidence":0.97}}
```
`subjectKey` values are fictional but obey the ingestion split rule (the first byte of the key picks
the split), so the harness's contributor-held-out check is exercised for real. `prediction: null`
means the candidate produced no suggestion at all.
| File | What it is for | Harness outcome |
|---|---|---|
| `predictions.jsonl` | Three candidates with invented scores: a bootstrap that calls almost everything Type 4, a frozen encoder that is precise but badly calibrated, and a specialist that looks strong on validation and slips on test. | `BLOCKED` — each candidate fails a different gate. |
| `predictions-go.jsonl` | One engineered candidate that clears every gate, including a low-confidence mistake on validation that the threshold is supposed to cut away. | `GO` — proves the passing branch and threshold selection, not a result. |
| `predictions-leaky.jsonl` | One contributor whose photos sit in both `validation` and `test`. | Refused with exit 2 — a leaked corpus is never scored. |
## Regenerating
The numbers are hand-chosen so the expected outcome is obvious by inspection; there is no generator
to re-run. If you change a row, update the expectation in `tests/calibration-harness.test.py` and the
caption in `scripts/record_release_demo.mjs` together, and read the harness output back before
claiming what it says.
No image, no diagnosis, and no patient data lives in this directory.
+14
View File
@@ -0,0 +1,14 @@
{"model":"gate-plumbing-fixture","subjectKey":"cd1a2b3c4d5e6f01","split":"validation","truth":{"bristolType":1},"prediction":{"bristolType":1,"confidence":0.97}}
{"model":"gate-plumbing-fixture","subjectKey":"ce2b3c4d5e6f7012","split":"validation","truth":{"bristolType":2},"prediction":{"bristolType":2,"confidence":0.97}}
{"model":"gate-plumbing-fixture","subjectKey":"cf3c4d5e6f701223","split":"validation","truth":{"bristolType":3},"prediction":{"bristolType":3,"confidence":0.98}}
{"model":"gate-plumbing-fixture","subjectKey":"d04d5e6f70122334","split":"validation","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.97}}
{"model":"gate-plumbing-fixture","subjectKey":"d15e6f7012233445","split":"validation","truth":{"bristolType":5},"prediction":{"bristolType":5,"confidence":0.98}}
{"model":"gate-plumbing-fixture","subjectKey":"d26f701223344556","split":"validation","truth":{"bristolType":6},"prediction":{"bristolType":3,"confidence":0.6}}
{"model":"gate-plumbing-fixture","subjectKey":"d370122334455667","split":"validation","truth":{"bristolType":7},"prediction":{"bristolType":7,"confidence":0.97}}
{"model":"gate-plumbing-fixture","subjectKey":"e671223344556677","split":"test","truth":{"bristolType":1},"prediction":{"bristolType":1,"confidence":0.97}}
{"model":"gate-plumbing-fixture","subjectKey":"e772334455667788","split":"test","truth":{"bristolType":2},"prediction":{"bristolType":2,"confidence":0.98}}
{"model":"gate-plumbing-fixture","subjectKey":"e883445566778899","split":"test","truth":{"bristolType":3},"prediction":{"bristolType":3,"confidence":0.97}}
{"model":"gate-plumbing-fixture","subjectKey":"e994556677889900","split":"test","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.98}}
{"model":"gate-plumbing-fixture","subjectKey":"eaa5667788990011","split":"test","truth":{"bristolType":5},"prediction":{"bristolType":5,"confidence":0.97}}
{"model":"gate-plumbing-fixture","subjectKey":"ebb6778899001122","split":"test","truth":{"bristolType":6},"prediction":{"bristolType":6,"confidence":0.97}}
{"model":"gate-plumbing-fixture","subjectKey":"ecc7889900112233","split":"test","truth":{"bristolType":7},"prediction":{"bristolType":7,"confidence":0.98}}
+2
View File
@@ -0,0 +1,2 @@
{"model":"frozen-encoder","subjectKey":"cd1a2b3c4d5e6f01","split":"validation","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.7}}
{"model":"frozen-encoder","subjectKey":"cd1a2b3c4d5e6f01","split":"test","truth":{"bristolType":6},"prediction":{"bristolType":6,"confidence":0.7}}
+42
View File
@@ -0,0 +1,42 @@
{"model":"vlm-bootstrap","subjectKey":"cd1a2b3c4d5e6f01","split":"validation","truth":{"bristolType":1},"prediction":{"bristolType":4,"confidence":0.55}}
{"model":"vlm-bootstrap","subjectKey":"ce2b3c4d5e6f7012","split":"validation","truth":{"bristolType":2},"prediction":{"bristolType":4,"confidence":0.52}}
{"model":"vlm-bootstrap","subjectKey":"cf3c4d5e6f701223","split":"validation","truth":{"bristolType":3},"prediction":{"bristolType":4,"confidence":0.61}}
{"model":"vlm-bootstrap","subjectKey":"d04d5e6f70122334","split":"validation","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.72}}
{"model":"vlm-bootstrap","subjectKey":"d15e6f7012233445","split":"validation","truth":{"bristolType":5},"prediction":{"bristolType":4,"confidence":0.64}}
{"model":"vlm-bootstrap","subjectKey":"d26f701223344556","split":"validation","truth":{"bristolType":6},"prediction":{"bristolType":4,"confidence":0.58}}
{"model":"vlm-bootstrap","subjectKey":"d370122334455667","split":"validation","truth":{"bristolType":7},"prediction":{"bristolType":4,"confidence":0.53}}
{"model":"vlm-bootstrap","subjectKey":"e671223344556677","split":"test","truth":{"bristolType":1},"prediction":{"bristolType":4,"confidence":0.57}}
{"model":"vlm-bootstrap","subjectKey":"e772334455667788","split":"test","truth":{"bristolType":2},"prediction":{"bristolType":4,"confidence":0.54}}
{"model":"vlm-bootstrap","subjectKey":"e883445566778899","split":"test","truth":{"bristolType":3},"prediction":{"bristolType":4,"confidence":0.63}}
{"model":"vlm-bootstrap","subjectKey":"e994556677889900","split":"test","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.7}}
{"model":"vlm-bootstrap","subjectKey":"eaa5667788990011","split":"test","truth":{"bristolType":5},"prediction":{"bristolType":4,"confidence":0.62}}
{"model":"vlm-bootstrap","subjectKey":"ebb6778899001122","split":"test","truth":{"bristolType":6},"prediction":{"bristolType":4,"confidence":0.59}}
{"model":"vlm-bootstrap","subjectKey":"ecc7889900112233","split":"test","truth":{"bristolType":7},"prediction":{"bristolType":4,"confidence":0.51}}
{"model":"frozen-encoder","subjectKey":"cd1a2b3c4d5e6f01","split":"validation","truth":{"bristolType":1},"prediction":{"bristolType":1,"confidence":0.62}}
{"model":"frozen-encoder","subjectKey":"ce2b3c4d5e6f7012","split":"validation","truth":{"bristolType":2},"prediction":{"bristolType":2,"confidence":0.58}}
{"model":"frozen-encoder","subjectKey":"cf3c4d5e6f701223","split":"validation","truth":{"bristolType":3},"prediction":{"bristolType":4,"confidence":0.66}}
{"model":"frozen-encoder","subjectKey":"d04d5e6f70122334","split":"validation","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.71}}
{"model":"frozen-encoder","subjectKey":"d15e6f7012233445","split":"validation","truth":{"bristolType":5},"prediction":{"bristolType":5,"confidence":0.57}}
{"model":"frozen-encoder","subjectKey":"d26f701223344556","split":"validation","truth":{"bristolType":6},"prediction":{"bristolType":6,"confidence":0.64}}
{"model":"frozen-encoder","subjectKey":"d370122334455667","split":"validation","truth":{"bristolType":7},"prediction":{"bristolType":6,"confidence":0.55}}
{"model":"frozen-encoder","subjectKey":"e671223344556677","split":"test","truth":{"bristolType":1},"prediction":{"bristolType":1,"confidence":0.6}}
{"model":"frozen-encoder","subjectKey":"e772334455667788","split":"test","truth":{"bristolType":2},"prediction":{"bristolType":2,"confidence":0.59}}
{"model":"frozen-encoder","subjectKey":"e883445566778899","split":"test","truth":{"bristolType":3},"prediction":{"bristolType":3,"confidence":0.63}}
{"model":"frozen-encoder","subjectKey":"e994556677889900","split":"test","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.69}}
{"model":"frozen-encoder","subjectKey":"eaa5667788990011","split":"test","truth":{"bristolType":5},"prediction":{"bristolType":6,"confidence":0.56}}
{"model":"frozen-encoder","subjectKey":"ebb6778899001122","split":"test","truth":{"bristolType":6},"prediction":{"bristolType":6,"confidence":0.61}}
{"model":"frozen-encoder","subjectKey":"ecc7889900112233","split":"test","truth":{"bristolType":7},"prediction":{"bristolType":7,"confidence":0.54}}
{"model":"specialist-candidate","subjectKey":"cd1a2b3c4d5e6f01","split":"validation","truth":{"bristolType":1},"prediction":{"bristolType":1,"confidence":0.88}}
{"model":"specialist-candidate","subjectKey":"ce2b3c4d5e6f7012","split":"validation","truth":{"bristolType":2},"prediction":{"bristolType":2,"confidence":0.86}}
{"model":"specialist-candidate","subjectKey":"cf3c4d5e6f701223","split":"validation","truth":{"bristolType":3},"prediction":{"bristolType":3,"confidence":0.9}}
{"model":"specialist-candidate","subjectKey":"d04d5e6f70122334","split":"validation","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.94}}
{"model":"specialist-candidate","subjectKey":"d15e6f7012233445","split":"validation","truth":{"bristolType":5},"prediction":{"bristolType":5,"confidence":0.87}}
{"model":"specialist-candidate","subjectKey":"d26f701223344556","split":"validation","truth":{"bristolType":6},"prediction":{"bristolType":6,"confidence":0.91}}
{"model":"specialist-candidate","subjectKey":"d370122334455667","split":"validation","truth":{"bristolType":7},"prediction":{"bristolType":7,"confidence":0.85}}
{"model":"specialist-candidate","subjectKey":"e671223344556677","split":"test","truth":{"bristolType":1},"prediction":{"bristolType":1,"confidence":0.86}}
{"model":"specialist-candidate","subjectKey":"e772334455667788","split":"test","truth":{"bristolType":2},"prediction":{"bristolType":2,"confidence":0.84}}
{"model":"specialist-candidate","subjectKey":"e883445566778899","split":"test","truth":{"bristolType":3},"prediction":{"bristolType":3,"confidence":0.89}}
{"model":"specialist-candidate","subjectKey":"e994556677889900","split":"test","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.92}}
{"model":"specialist-candidate","subjectKey":"eaa5667788990011","split":"test","truth":{"bristolType":5},"prediction":{"bristolType":5,"confidence":0.88}}
{"model":"specialist-candidate","subjectKey":"ebb6778899001122","split":"test","truth":{"bristolType":6},"prediction":{"bristolType":3,"confidence":0.83}}
{"model":"specialist-candidate","subjectKey":"ecc7889900112233","split":"test","truth":{"bristolType":7},"prediction":{"bristolType":7,"confidence":0.87}}
+123
View File
@@ -0,0 +1,123 @@
import { chromium } from 'playwright';
import assert from 'node:assert/strict';
import { mkdir } from 'node:fs/promises';
// Issue #36 — a saved photo is reported and removable without touching the confirmed log.
// Every image below is a synthetic 1x1 PNG constant; no real photo is loaded or written.
const SYNTHETIC_PHOTO = 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+ip1sAAAAASUVORK5CYII=';
await mkdir('artifacts', { recursive: true });
const browser = await chromium.launch({ headless: true });
const context = await browser.newContext({ viewport: { width: 390, height: 844 }, deviceScaleFactor: 2, serviceWorkers: 'block' });
const page = await context.newPage();
const errors = [];
page.on('console', message => { if (message.type() === 'error') errors.push(message.text()); });
page.on('pageerror', error => errors.push(error.message));
page.on('dialog', dialog => dialog.accept());
await page.goto('http://127.0.0.1:4173', { waitUntil: 'networkidle' });
await page.evaluate(() => localStorage.clear());
await page.reload({ waitUntil: 'networkidle' });
const sample = (id, daysAgo, photoDataUrl = '', overrides = {}) => ({
id,
occurredAt: new Date(Date.now() - daysAgo * 86400000).toISOString(),
bristolType: 4,
color: 'brown',
urgency: 1,
discomfort: 0,
note: '',
photoDataUrl,
symptoms: {},
...overrides,
});
const ledgerKey = () => page.evaluate(() => `timmy:${document.querySelector('meta[name="timmy-base-path"]')?.content || '/'}:ledger-v1`);
async function seedLedger(list) {
const key = await ledgerKey();
await page.evaluate(({ storeKey, entries }) => localStorage.setItem(storeKey, JSON.stringify(entries)), { storeKey: key, entries: list });
await page.reload({ waitUntil: 'networkidle' });
}
async function storedEntries() {
const key = await ledgerKey();
return page.evaluate(storeKey => JSON.parse(localStorage.getItem(storeKey) || '[]'), key);
}
// 1. Presence is visible in the journal without a thumbnail.
await seedLedger([sample('photo-log', 2, SYNTHETIC_PHOTO), sample('plain-log', 5, '', { bristolType: 6 })]);
await page.locator('[data-view="calendar"]').last().click();
await page.locator('.entries-card .entry').first().waitFor();
assert.equal(await page.locator('.entries-card .entry').count(), 2, 'both confirmed logs must be listed');
assert.equal(await page.locator('.entries-card [data-photo-indicator="present"]').count(), 1, 'exactly the log holding a photo is marked');
assert.equal(await page.locator('.entries-card img').count(), 0, 'the journal must not render a thumbnail by default');
assert.equal(await page.getByText('Photo saved', { exact: true }).count(), 1);
// Visual acceptance: the label sits inside the 390px viewport and adds no horizontal scroll.
const chipBox = await page.locator('.entries-card [data-photo-indicator="present"]').boundingBox();
assert.ok(chipBox && chipBox.width > 0, 'the photo label must be rendered');
assert.ok(chipBox.x + chipBox.width <= 390, 'the photo label must fit the mobile viewport');
assert.ok(await page.evaluate(() => document.documentElement.scrollWidth) <= 390, 'the journal must not scroll horizontally');
// 2. The entry sheet reports the saved photo without showing it.
await page.locator('[data-entry-open="photo-log"]').click();
await page.locator('[data-photo-lifecycle="present"]').waitFor();
assert.equal(await page.locator('#entry-photo').count(), 0, 'opening a log must not show the photo until asked');
assert.match(await page.locator('[data-photo-lifecycle]').innerText(), /Photo saved with this log/i);
assert.match(await page.locator('[data-photo-lifecycle]').innerText(), /stays in this browser/i);
// Touch targets stay usable: every lifecycle action is at least 44px tall on mobile.
for (const selector of ['#show-entry-photo', '#remove-entry-photo', '#delete-entry']) {
const box = await page.locator(selector).boundingBox();
assert.ok(box && box.height >= 44, `${selector} must be a 44px touch target`);
}
// 3. Showing it is an explicit request.
await page.locator('#show-entry-photo').click();
const shown = page.locator('#entry-photo');
await shown.waitFor();
assert.match(await shown.getAttribute('src'), /^data:image\/png;base64,/, 'the revealed image is the stored bytes');
assert.match(await shown.getAttribute('alt'), /photo saved with this log/i);
// 4. Removing the photo keeps the confirmed log.
await page.locator('#remove-entry-photo').click();
await page.locator('[data-photo-lifecycle="absent"]').waitFor();
await page.getByText(/Photo removed\. Your log is still here\./).waitFor();
let stored = await storedEntries();
assert.equal(stored.length, 2, 'removing a photo must not delete the log');
const kept = stored.find(item => item.id === 'photo-log');
assert.ok(kept, 'the log that held the photo must still exist');
assert.equal(kept.photoDataUrl, '', 'only the image bytes are cleared');
assert.equal(kept.bristolType, 4);
assert.equal(kept.occurredAt, stored.find(item => item.id === 'photo-log').occurredAt);
await page.locator('#entry-sheet-close').click();
await page.locator('.entries-card .entry').first().waitFor();
assert.equal(await page.locator('.entries-card [data-photo-indicator="present"]').count(), 0, 'the indicator must follow the stored bytes');
// The sheet is a dialog: Escape dismisses it.
await page.locator('[data-entry-open="photo-log"]').click();
await page.locator('.sheet-backdrop').waitFor();
await page.keyboard.press('Escape');
await page.locator('.sheet-backdrop').waitFor({ state: 'detached' });
// 5. Deleting a log asks once and takes its photo with it.
await seedLedger([sample('delete-me', 1, SYNTHETIC_PHOTO), sample('keep-me', 3, '')]);
await page.locator('[data-view="calendar"]').last().click();
await page.locator('[data-entry-open="delete-me"]').click();
await page.locator('#delete-entry').click();
await page.locator('#delete-entry-confirm').waitFor();
assert.equal((await storedEntries()).length, 2, 'the first tap only arms the delete');
await page.locator('#delete-entry-confirm').click();
await page.locator('.entries-card .entry').first().waitFor();
stored = await storedEntries();
assert.deepEqual(stored.map(item => item.id), ['keep-me'], 'the confirmed log and its photo are gone together');
// 6. Delete-all removes the ledger and every saved photo.
await seedLedger([sample('wipe-1', 1, SYNTHETIC_PHOTO), sample('wipe-2', 2, SYNTHETIC_PHOTO)]);
await page.locator('[data-view="calendar"]').last().click();
await page.locator('.settings-row[data-view="privacy"]').click();
await page.locator('#delete-all').click();
await page.getByText(/Local ledger and photos deleted/).waitFor();
const key = await ledgerKey();
assert.equal(await page.evaluate(storeKey => localStorage.getItem(storeKey), key), null, 'delete-all must remove the ledger');
assert.equal(await page.evaluate(() => JSON.stringify(localStorage).includes('data:image')), false, 'no photo bytes may survive delete-all');
await page.locator('[data-view="calendar"]').last().click();
await page.getByText(/Quiet bowl, clean slate\./).waitFor();
assert.deepEqual(errors, []);
await browser.close();
console.log('PASS photo presence → reveal on request → remove photo keeps log → delete log → delete-all');
+71
View File
@@ -0,0 +1,71 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { photoCount, photoState, withoutEntry, withoutPhoto } from '../src/photo-lifecycle.js';
const entry = (id, photoDataUrl = '', overrides = {}) => ({
id,
occurredAt: '2026-10-14T07:20:00.000Z',
bristolType: 4,
color: 'brown',
urgency: 1,
discomfort: 0,
note: '',
photoDataUrl,
symptoms: { blood: false },
...overrides,
});
test('photo state is decided by the saved image bytes, not a filename or a flag', () => {
assert.deepEqual(photoState(entry('a', 'data:image/png;base64,AAAA')), {
hasPhoto: true,
label: 'Photo saved',
});
});
test('a log with no usable image reports that no photo is saved', () => {
for (const value of [undefined, '', 'synthetic-type4.jpg', 'data:text/plain;base64,AAAA', 42]) {
assert.deepEqual(photoState(entry('a', value)), { hasPhoto: false, label: '' });
}
});
test('removing a photo clears only the image bytes and keeps the confirmed log intact', () => {
const before = entry('a', 'data:image/png;base64,AAAA', { note: 'after lunch', bristolType: 3 });
const [after] = withoutPhoto([before], 'a');
assert.equal(after.id, before.id);
assert.equal(after.occurredAt, before.occurredAt);
assert.equal(after.bristolType, 3);
assert.equal(after.note, 'after lunch');
assert.deepEqual(after.symptoms, before.symptoms);
assert.equal(after.photoDataUrl, '');
});
test('removing a photo never mutates the ledger it was given and leaves other logs alone', () => {
const withPhoto = entry('a', 'data:image/png;base64,AAAA');
const withoutOne = entry('b', '', { note: 'no photo here' });
const input = [withPhoto, withoutOne];
const result = withoutPhoto(input, 'a');
assert.notEqual(result, input);
assert.equal(input[0].photoDataUrl, 'data:image/png;base64,AAAA');
assert.deepEqual(result[1], withoutOne);
assert.deepEqual(withoutPhoto(input, 'missing'), input);
assert.equal(withoutPhoto(input, 'missing')[0], input[0]);
});
test('deleting a log removes exactly that confirmed entry, photo included', () => {
const input = [entry('a', 'data:image/png;base64,AAAA'), entry('b'), entry('c', '', { bristolType: 6 })];
const result = withoutEntry(input, 'a');
assert.deepEqual(result.map(item => item.id), ['b', 'c']);
assert.equal(input.length, 3);
assert.deepEqual(withoutEntry(input, 'missing'), input);
});
test('photo count reports how many confirmed logs still hold an image', () => {
assert.equal(photoCount([entry('a', 'data:image/png;base64,AAAA'), entry('b'), entry('c', 'data:image/jpeg;base64,BBBB')]), 2);
assert.equal(photoCount([]), 0);
assert.equal(photoCount(undefined), 0);
});
+131
View File
@@ -0,0 +1,131 @@
import { chromium } from 'playwright';
import assert from 'node:assert/strict';
import { readFile } from 'node:fs/promises';
// Issue #35 — a backup is the person's own record, and it has to keep working as the app's AI
// provenance fields evolve. This drives the real app in a browser: a version 1 file is upgraded and
// says so, a file from a newer Timmy is refused without touching the ledger, the export carries the
// contribution receipts, and a log saved from a confirmed AI suggestion comes back saying that.
//
// The image used below is the committed synthetic Type 4 fixture; no real photo is involved.
const browser = await chromium.launch({ headless: true });
const context = await browser.newContext({ viewport: { width: 390, height: 844 }, deviceScaleFactor: 2, serviceWorkers: 'block' });
const page = await context.newPage();
const errors = [];
page.on('console', message => { if (message.type() === 'error') errors.push(message.text()); });
page.on('pageerror', error => errors.push(error.message));
page.on('dialog', dialog => dialog.accept());
const suggestionBody = {
status: 'suggestion', isStool: true, bristolType: 4, color: 'brown', confidence: 0.83,
imageQuality: 'good', observations: 'Smooth, formed appearance.',
warning: 'Visual suggestion only. Confirm it yourself; this is not a diagnosis.',
};
await page.route('**/api/vision-status', route => route.fulfill({
status: 200, contentType: 'application/json',
body: JSON.stringify({ enabled: true, profile: 'selfhost', processor: 'self-hosted', model: 'SmolVLM2-2.2B-Instruct', providerReady: true, modelSeen: true }),
}));
await page.route('**/api/analyze', route => route.fulfill({ status: 200, contentType: 'application/json', body: JSON.stringify(suggestionBody) }));
await page.route('**/api/agent/status', route => route.fulfill({
status: 200, contentType: 'application/json',
body: JSON.stringify({ enabled: true, configured: false, authenticated: false, mode: 'local-fallback' }),
}));
await page.goto('http://127.0.0.1:4173', { waitUntil: 'networkidle' });
await page.evaluate(() => localStorage.clear());
await page.reload({ waitUntil: 'networkidle' });
const basePath = () => page.evaluate(() => document.querySelector('meta[name="timmy-base-path"]')?.content || '/');
const storeKey = async suffix => `timmy:${await basePath()}:${suffix}`;
const readStore = async suffix => {
const raw = await page.evaluate(key => localStorage.getItem(key), await storeKey(suffix));
return raw === null ? null : JSON.parse(raw);
};
// The privacy view sits behind the journal's settings row, exactly as a person reaches it.
const openPrivacy = async () => {
await page.locator('[data-view="calendar"]').first().click();
await page.locator('.settings-row[data-view="privacy"]').first().click();
await page.locator('#export').waitFor();
};
const importBackup = async (name, text) => {
await page.locator('#import').setInputFiles({ name, mimeType: 'application/json', buffer: Buffer.from(text) });
await page.waitForTimeout(350);
};
const legacyBackup = JSON.stringify({
product: 'Timmy the Talking Turd',
schemaVersion: 1,
exportedAt: '2026-08-01T09:00:00.000Z',
entries: [
{ id: 'legacy-1', occurredAt: '2026-08-01T08:00:00.000Z', bristolType: 3, color: 'brown', urgency: 1, discomfort: 0, note: 'synthetic', photoDataUrl: '', symptoms: {} },
{ id: 'legacy-2', occurredAt: '2026-08-02T08:00:00.000Z', bristolType: 6, color: 'brown', urgency: 2, discomfort: 1, note: '', photoDataUrl: '', symptoms: {} },
],
});
// 1. A version 1 backup is imported, upgraded, and says which version it came from.
await openPrivacy();
await importBackup('timmy-backup-v1.json', legacyBackup);
const upgraded = await readStore('ledger-v1');
assert.equal(upgraded.length, 2, 'the version 1 backup must restore both logs');
await page.locator('[data-portability-notice]').waitFor();
const noticeText = (await page.locator('[data-portability-notice]').innerText()).trim();
assert.match(noticeText, /version 1/i, 'the notice must name the version the file came from');
assert.match(noticeText, /upgraded/i, 'the notice must say the backup was upgraded');
assert.equal(await page.locator('[data-portability-notice]').getAttribute('data-portability-version'), '1');
assert.equal(upgraded[0].provenance, null, 'a pre-provenance backup must not acquire a source it never recorded');
assert.equal(upgraded[0].bristolType, 3, 'the confirmed form must survive the upgrade');
// 2. A file from a newer Timmy is refused, and the ledger it would have replaced stays exactly as it was.
const before = JSON.stringify(await readStore('ledger-v1'));
await importBackup('timmy-backup-v3.json', JSON.stringify({ product: 'Timmy the Talking Turd', schemaVersion: 3, entries: [{ id: 'future', bristolType: 7 }] }));
await page.locator('[data-portability-error]').waitFor();
const errorText = (await page.locator('[data-portability-error]').innerText()).trim();
assert.match(errorText, /newer version of Timmy/i);
assert.equal(JSON.stringify(await readStore('ledger-v1')), before, 'a refused import must not change the ledger');
// 3. A log saved from a confirmed AI suggestion records that provenance, and the export states it.
await page.locator('[data-view="home"]').first().click();
await page.locator('[data-scan]').first().click();
await page.getByText(/Self-hosted model ready/i).waitFor();
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/synthetic-type4.jpg');
await page.locator('#ai-consent').check();
await page.locator('#analyze-photo').click();
await page.locator('#use-suggestion').waitFor();
await page.locator('#use-suggestion').click();
await page.locator('[data-provenance="confirmed"]').waitFor();
await page.locator('#next').click();
await page.locator('#next').click();
await page.locator('#save').click();
await page.getByText(/Private log saved|Saved/).waitFor();
const savedLedger = await readStore('ledger-v1');
const assisted = savedLedger.find(entry => entry.id !== 'legacy-1' && entry.id !== 'legacy-2');
assert.ok(assisted, 'the saved log must be in the ledger');
assert.equal(assisted.provenance?.source, 'ai-confirmed', 'a confirmed AI suggestion must be recorded on the saved log');
assert.equal(assisted.provenance?.suggestedBristolType, 4);
assert.equal(assisted.provenance?.model, 'SmolVLM2-2.2B-Instruct');
// 4. Export carries the current version, the entries, and the contribution receipts — and nothing else.
await openPrivacy();
const [download] = await Promise.all([page.waitForEvent('download'), page.locator('#export').click()]);
const exported = JSON.parse(await readFile(await download.path(), 'utf8'));
assert.equal(exported.schemaVersion, 2);
assert.equal(exported.entries.length, 3);
assert.deepEqual(exported.contributions, [], 'nothing was contributed, so no receipt may appear');
assert.equal(exported.entries.find(entry => entry.id === assisted.id).provenance.source, 'ai-confirmed');
// 5. Round trip: the export restores into a cleared browser with provenance intact.
const exportedText = JSON.stringify(exported);
await page.evaluate(() => localStorage.clear());
await page.reload({ waitUntil: 'networkidle' });
await openPrivacy();
await importBackup('timmy-backup-v2.json', exportedText);
const roundTripped = await readStore('ledger-v1');
assert.equal(roundTripped.length, 3);
const restoredAssisted = roundTripped.find(entry => entry.id === assisted.id);
assert.deepEqual(restoredAssisted.provenance, assisted.provenance, 'provenance must survive a round trip');
await page.locator('[data-portability-notice]').waitFor();
assert.match((await page.locator('[data-portability-notice]').innerText()).trim(), /version 2/i);
assert.deepEqual(errors, [], `browser errors: ${errors.join(' | ')}`);
await browser.close();
console.log(JSON.stringify({ ok: true, scenes: ['v1-upgrade', 'future-refused', 'provenance-recorded', 'export-v2', 'round-trip'] }));
+152
View File
@@ -0,0 +1,152 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { LEDGER_EXPORT_VERSION, LEDGER_IMPORT_LIMITS, exportLedger, exportLedgerBackup, importLedger, importLedgerBackup } from '../src/domain.js';
// Issue #35 — a backup is the person's own record. Provenance is part of what they own: a log that
// was saved from a confirmed or corrected AI suggestion has to come back out of the export saying
// exactly that, instead of silently reverting to looking like a manual entry.
const confirmedByUser = {
id: 'demo-corrected',
occurredAt: '2026-08-18T10:00:00.000Z',
bristolType: 2,
color: 'brown',
urgency: 1,
discomfort: 0,
note: 'synthetic',
photoDataUrl: '',
symptoms: {},
provenance: {
source: 'ai-corrected',
model: 'SmolVLM2-2.2B-Instruct',
suggestedBristolType: 4,
suggestedColor: 'brown',
confirmedAt: '2026-08-18T09:59:00.000Z',
},
};
test('an export stamps the current schema version and keeps confirmed AI provenance', () => {
const parsed = JSON.parse(exportLedger([confirmedByUser], '2026-08-18T11:00:00.000Z'));
assert.equal(LEDGER_EXPORT_VERSION, 2);
assert.equal(parsed.schemaVersion, 2);
assert.equal(parsed.entries.length, 1);
assert.deepEqual(parsed.entries[0].provenance, confirmedByUser.provenance);
assert.equal(parsed.entries[0].bristolType, 2, 'the corrected value stays the saved value');
});
// A version 1 file was written before entries recorded provenance. Importing it must restore the
// confirmed values and say plainly that this backup predates provenance, rather than inventing a
// source for logs that never had one.
const legacyBackup = JSON.stringify({
product: 'Timmy the Talking Turd',
schemaVersion: 1,
exportedAt: '2026-08-01T09:00:00.000Z',
entries: [
{
id: 'legacy-entry',
occurredAt: '2026-08-01T08:00:00.000Z',
bristolType: 4,
color: 'brown',
urgency: 2,
discomfort: 1,
note: '',
photoDataUrl: '',
symptoms: { blood: false },
},
],
});
test('a version 1 backup imports its confirmed values and reports the upgrade', () => {
const result = importLedgerBackup(legacyBackup);
assert.equal(result.fromVersion, 1);
assert.equal(result.migrated, true);
assert.equal(result.entries.length, 1);
assert.equal(result.entries[0].id, 'legacy-entry');
assert.equal(result.entries[0].bristolType, 4);
assert.equal(result.entries[0].provenance, null, 'a pre-provenance backup must not be given a source it never recorded');
assert.match(result.notices.join(' '), /version 1/i);
assert.match(result.notices.join(' '), /upgraded/i);
// The import screen already reports what was restored; a notice that restates the count renders as
// a duplicated sentence on screen, so the upgrade notice must only describe the upgrade.
assert.doesNotMatch(result.notices.join(' '), /Restored \d+ logs?/);
assert.equal(importLedger(legacyBackup).length, 1, 'the plain entry import still works');
});
// A training contribution receipt is user-owned data too. A backup that restored the journal but
// dropped the receipts would tell someone their opt-in never happened, so version 2 carries them —
// with the withdrawal tombstone intact and nothing else copied out of the device.
const withdrawnReceipt = {
contributionId: 'contrib-1',
entryId: 'demo-corrected',
consentVersion: 'training-consent-v1',
consentedAt: '2026-08-18T10:05:00.000Z',
labels: { bristolType: 2, color: 'brown' },
state: 'withdrawn',
withdrawnAt: '2026-08-18T10:30:00.000Z',
artifacts: [
{ artifact: 'raw-image', contributionId: 'contrib-1', state: 'withdrawn' },
{ artifact: 'confirmed-label', contributionId: 'contrib-1', state: 'withdrawn' },
],
// A crafted file must not be able to hand a credential back to the app.
agentAccessCode: 'leak-me-please',
};
test('a backup carries contribution receipts and their tombstones without copying secrets', () => {
const text = exportLedgerBackup({
entries: [{ ...confirmedByUser, agentAccessCode: 'leak-me-please' }],
contributions: [withdrawnReceipt],
exportedAt: '2026-08-18T12:00:00.000Z',
});
const parsed = JSON.parse(text);
assert.equal(parsed.schemaVersion, 2);
assert.equal(parsed.contributions.length, 1);
assert.equal(parsed.contributions[0].state, 'withdrawn');
assert.deepEqual(parsed.contributions[0].artifacts, withdrawnReceipt.artifacts);
assert.equal(parsed.entries[0].agentAccessCode, undefined);
assert.doesNotMatch(text, /leak-me-please/);
const restored = importLedgerBackup(text);
assert.equal(restored.migrated, false);
assert.equal(restored.fromVersion, 2);
assert.equal(restored.contributions.length, 1);
assert.equal(restored.contributions[0].contributionId, 'contrib-1');
assert.equal(restored.contributions[0].state, 'withdrawn');
assert.deepEqual(restored.contributions[0].artifacts, withdrawnReceipt.artifacts);
assert.equal(restored.contributions[0].agentAccessCode, undefined);
assert.deepEqual(restored.entries[0].provenance, confirmedByUser.provenance);
});
test('a backup from a newer Timmy is refused with an instruction instead of a guess', () => {
const future = JSON.stringify({ product: 'Timmy the Talking Turd', schemaVersion: 3, entries: [], contributions: [] });
assert.throws(() => importLedgerBackup(future), /newer version of Timmy/i);
});
test('a file that is not a readable Timmy backup is refused', () => {
assert.throws(() => importLedgerBackup('{not json'), /not readable JSON/i);
assert.throws(() => importLedgerBackup(JSON.stringify({ hello: 'world' })), /not a supported Timmy backup/i);
assert.throws(() => importLedgerBackup(JSON.stringify({ schemaVersion: 0, entries: [] })), /not a supported Timmy backup/i);
});
test('an oversized backup or one with too many logs is refused before anything is imported', () => {
const huge = `{"schemaVersion":2,"entries":[]}${' '.repeat(LEDGER_IMPORT_LIMITS.maxTextLength)}`;
assert.throws(() => importLedgerBackup(huge), /too large/i);
const many = { schemaVersion: 2, entries: Array.from({ length: LEDGER_IMPORT_LIMITS.maxEntries + 1 }, () => ({ id: 'x', bristolType: 4 })) };
assert.throws(() => importLedgerBackup(JSON.stringify(many)), /more logs than Timmy will import/i);
});
test('an unreadable contribution receipt is skipped and reported, never silently dropped', () => {
const text = JSON.stringify({
schemaVersion: 2,
entries: [{ id: 'kept', bristolType: 4 }],
contributions: [{ entryId: 'kept', state: 'queued-local' }],
});
const result = importLedgerBackup(text);
assert.equal(result.contributions.length, 0);
assert.match(result.notices.join(' '), /could not be read and was skipped/i);
});
+22
View File
@@ -21,6 +21,15 @@ test('product decision record fixes the AI boundary and release authority', asyn
assert.match(decision, /\.\.\/research\/SELF-HOSTED-STOOL-VISION\.md/);
});
test('the decision record keeps training contribution as its own versioned, withdrawable consent', async () => {
const decision = await readFile(decisionPath, 'utf8');
assert.match(decision, /### Training contribution consent/);
assert.match(decision, /never a training contribution/i);
assert.match(decision, /withdrawal keeps a tombstone naming every derived artifact by contribution id/i);
assert.match(decision, /does not transmit a contribution until the corpus pipeline and privacy copy are reviewed/i);
});
test('Hermes deployment installs a checked-in non-diagnostic workspace policy', async () => {
const [policy, readme] = await Promise.all([readFile(policyPath, 'utf8'), readFile(readmePath, 'utf8')]);
assert.match(policy, /never diagnose/i);
@@ -28,3 +37,16 @@ test('Hermes deployment installs a checked-in non-diagnostic workspace policy',
assert.match(policy, /explicit user intent and confirmation/i);
assert.match(readme, /install -m 600 docs\/TIMMY-AGENT-POLICY\.md/);
});
test('the decision record keeps a backup versioned and refuses files it cannot read', async () => {
const decision = await readFile(decisionPath, 'utf8');
const readme = await readFile(readmePath, 'utf8');
assert.match(decision, /### Backup portability and entry provenance/);
assert.match(decision, /schema version/i);
assert.match(decision, /upgraded on import/i);
assert.match(decision, /newer version is refused/i);
assert.match(decision, /a refused import leaves the ledger unchanged/i);
assert.match(decision, /provenance is stored on the confirmed log/i);
assert.match(readme, /versioned JSON backup/i);
});
+154 -2
View File
@@ -168,9 +168,13 @@ test('release demo states the declared release delta on camera and proves it wit
assert.match(demo, /TIMMY_RELEASE_DEMO_DELTA/);
assert.match(demo, /This release: \$\{releaseDelta\}/);
assert.match(demo, /no new feature is claimed/);
// A caption alone is not evidence; the demo must emit a machine-readable receipt of what rendered.
// A caption string alone is not evidence; each caption must be measured as a painted bar, and the
// receipt must carry that per-caption measurement.
assert.match(demo, /TIMMY_DEMO_CAPTIONS_OUT/);
assert.match(demo, /captionsShown\.push\(text\)/);
assert.match(demo, /captionsVerified: captionEvidence/);
assert.match(demo, /Caption is not an on-screen bar/);
assert.match(demo, /Caption painted nothing into the caption band/);
assert.match(demo, /writeFile\(captionsOut/);
});
@@ -181,13 +185,38 @@ test('release demo shows the byte-bearing controls the threat model governs, wit
assert.match(demo, /Every byte-bearing control the model names/);
assert.match(demo, /indicate\('#export'\)/);
assert.match(demo, /indicate\('#delete-all'\)/);
assert.match(demo, /Delete all local data clears the ledger in this browser; there is no server copy of it to keep/);
assert.match(demo, /Delete all local data clears the ledger and every saved photo in this browser; nothing is kept on a server/);
// Real controls only: the selectors must exist in the app shell, not be demo-only scaffolding.
const app = await readFile(new URL('../app.js', import.meta.url), 'utf8');
assert.match(app, /id="export"/);
assert.match(app, /id="delete-all"/);
});
test('release demo exercises the photo lifecycle on camera without showing an image by default', async () => {
const demo = await readFile(demoPath, 'utf8');
// Presence first: a label in the journal, with no thumbnail rendered.
assert.match(demo, /Photo presence is a label, not a thumbnail/);
assert.match(demo, /\.entries-card \[data-photo-indicator="present"\]/);
assert.match(demo, /exactly one log must be marked as holding a photo/);
assert.match(demo, /the journal must not show a thumbnail by default/);
// Revealing the image is an explicit request, asserted before the caption that claims it.
assert.match(demo, /opening a log must not reveal its photo/);
assert.match(demo, /#show-entry-photo/);
assert.match(demo, /Showing the photo is your explicit request/);
// Removing the photo keeps every confirmed field of the log.
assert.match(demo, /#remove-entry-photo/);
assert.match(demo, /Remove just the photo: the confirmed log, form, and note are all still here/);
assert.match(demo, /removing a photo must keep the confirmed log and clear only its image bytes/);
// Delete-all is actually performed, not just pointed at.
assert.match(demo, /tap\('#delete-all'/);
assert.match(demo, /no photo bytes may survive delete-all|JSON\.stringify\(localStorage\)\.includes\('data:image'\)/);
// Synthetic bytes only: a committed test-card PNG constant, never a real photo. A blank 1x1 pixel
// rendered as an empty white box on camera, which read as a broken image instead of a saved photo.
assert.match(demo, /const SYNTHETIC_PHOTO = 'data:image\/png;base64,/);
assert.match(demo, /synthetic test card/);
});
test('release builder refuses to publish without the on-camera delta receipt', async () => {
const builder = await readFile(builderPath, 'utf8');
@@ -196,10 +225,24 @@ test('release builder refuses to publish without the on-camera delta receipt', a
assert.match(builder, /Release demo produced no on-camera caption receipt/);
assert.match(builder, /Release demo did not state the declared release delta on camera/);
assert.match(builder, /Release demo is missing a required on-camera caption/);
assert.match(builder, /Release demo caption never painted on screen/);
assert.match(builder, /recorded\.get\("captionsVerified"\)/);
assert.match(builder, /"release_delta_visible_on_camera": "passed"/);
assert.match(builder, /"release_delta": release_delta/);
});
test('caption styling survives the demo reloads instead of dying with the first document', async () => {
const demo = await readFile(demoPath, 'utf8');
// A stylesheet belongs to one document; the demo reloads the page, which stripped caption and touch
// styling and left later captions as clipped body text. Both now carry inline styles per node.
assert.doesNotMatch(demo, /page\.addStyleTag/);
assert.doesNotMatch(demo, /addInitScript/);
assert.match(demo, /In a stylesheet belongs to one document|a stylesheet belongs to one document/);
assert.match(demo, /node\.style\.cssText = 'position:fixed;left:16px;right:16px;top:14px;z-index:20000;'/);
assert.match(demo, /ring\.style\.cssText = 'position:fixed;z-index:20001/);
});
test('release demo captions expire and the contact sheet spans the whole recording', async () => {
const demo = await readFile(demoPath, 'utf8');
const builder = await readFile(builderPath, 'utf8');
@@ -265,3 +308,112 @@ test('release demo shows readiness waiting for the loaded pinned model before AI
// The caption is a required release gate, so the scene cannot be dropped silently.
assert.match(builder, /Reachable is not loaded: no AI control, no consent, and no upload until the pinned model is resident/);
});
test('release demo shows the training contribution as a separate opt-in receipt that can be withdrawn', async () => {
const demo = await readFile(demoPath, 'utf8');
const builder = await readFile(builderPath, 'utf8');
const app = await readFile(new URL('../app.js', import.meta.url), 'utf8');
// Issue #24: the recording must show the opt-in section, the consent gate, and the withdrawal.
assert.match(demo, /\[data-training-consent\]/);
assert.match(demo, /\[data-contribute-entry="demo-wipe"\]/);
assert.match(demo, /#training-consent-check/);
assert.match(demo, /#contribute-confirm/);
assert.match(demo, /\[data-contribution-state="queued-local"\]/);
assert.match(demo, /data-withdraw=/);
assert.match(demo, /\[data-withdrawn-artifacts\]/);
assert.match(demo, /training-consent-v1/);
// Every claim is measured in the browser before the caption that makes it.
assert.match(demo, /a journal save must not queue a training contribution/);
assert.match(demo, /the contribute control must stay disabled until the terms are accepted/);
assert.match(demo, /no receipt may exist before explicit consent/);
assert.match(demo, /a receipt must never copy the photo bytes/);
assert.match(demo, /every derived artifact must be addressed by contribution id/);
assert.match(demo, /delete-all must clear the contribution receipts as well/);
// The controls are the app's own, not demo-only scaffolding.
assert.match(app, /data-training-consent/);
assert.match(app, /data-contribution-queued-count/);
assert.match(app, /id="training-consent-check"/);
assert.match(app, /id="contribute-confirm" disabled/);
assert.match(app, /data-withdraw=/);
// The captions and the gate key are required, so a later release cannot drop the scenes silently.
assert.match(builder, /Training contribution is off by default: a saved log is never training data/);
assert.match(builder, /The consent version and the receipt are on screen before the contribute button works/);
assert.match(builder, /Withdrawing keeps a tombstone naming every derived artifact by contribution ID/);
assert.match(builder, /"training_contribution_consent": "passed"/);
assert.match(builder, /"test:training-contribution"/);
});
test('release demo shows a versioned backup, an upgraded older file, and a refused newer one', async () => {
const demo = await readFile(demoPath, 'utf8');
const builder = await readFile(builderPath, 'utf8');
const app = await readFile(new URL('../app.js', import.meta.url), 'utf8');
// Issue #35: the recording must export the real file, import an older version, refuse a newer one,
// and reopen a restored log so its provenance is on screen — not merely described.
assert.match(demo, /waitForEvent\('download'\)/);
assert.match(demo, /exportedBackup\.schemaVersion !== 2/);
assert.match(demo, /the backup must carry the withdrawn receipt as a tombstone/);
assert.match(demo, /\[data-portability-notice\]\[data-portability-version="1"\]/);
assert.match(demo, /\[data-portability-error\]/);
assert.match(demo, /\[data-entry-provenance="ai-corrected"\]/);
// Every claim is measured before the caption that makes it.
assert.match(demo, /an older backup must be reported as upgraded/);
assert.match(demo, /a refused import must not change the ledger it would have replaced/);
assert.match(demo, /a restored log must still show that it began as a corrected AI suggestion/);
// The controls belong to the app, and the provenance line is its own rendering.
assert.match(app, /id="export"/);
assert.match(app, /data-portability-notice/);
assert.match(app, /data-entry-provenance=/);
// Captions and the gate key are required, so a later release cannot drop the scenes silently.
assert.match(builder, /A backup is versioned, and it carries the withdrawn receipt as a tombstone rather than silence/);
assert.match(builder, /An older backup is upgraded on import, and the notice says which version the file came from/);
assert.match(builder, /A file from a newer Timmy is refused with an instruction, and the ledger it would have replaced is untouched/);
assert.match(builder, /A restored log still says it began as an AI suggestion you corrected — provenance is part of the record/);
assert.match(builder, /"portable_backup_restore": "passed"/);
assert.match(builder, /"test:portability"/);
});
test('the builder refuses to gate or record a release against a foreign server', async () => {
const builder = await readFile(builderPath, 'utf8');
// A leftover checkout answering on the acceptance port once satisfied every flow and the demo,
// so the release described a tree that was not the commit under test. The builder must refuse a
// live listener before it starts, then prove the served bytes are this tree's before any gate.
assert.match(builder, /def assert_port_free/);
assert.match(builder, /assert_port_free\("127\.0\.0\.1", 4173\)/);
assert.match(builder, /def assert_served_tree/);
assert.match(builder, /assert_served_tree\(tree, "127\.0\.0\.1", 4173\)/);
const identityCheck = builder.indexOf('assert_served_tree(tree, "127.0.0.1", 4173)');
const firstFlow = builder.indexOf('run(["npm", "run", "test:ui"]');
assert.ok(identityCheck > 0, 'the served-tree check must be present');
assert.ok(firstFlow > identityCheck, 'the served-tree check must run before the first acceptance flow');
});
test('release demo replays the calibration harness against synthetic predictions instead of narrating the gates', async () => {
const demo = await readFile(demoPath, 'utf8');
const builder = await readFile(builderPath, 'utf8');
// Issue #29: the gate math is only proven by running the real harness on camera.
assert.match(demo, /scripts\/calibration_report\.py/);
assert.match(demo, /tests\/fixtures\/calibration\/predictions\.jsonl/);
assert.match(demo, /tests\/fixtures\/calibration\/predictions-go\.jsonl/);
assert.match(demo, /tests\/fixtures\/calibration\/predictions-leaky\.jsonl/);
// All three verdict paths are exercised: blocked, passing, and refused before scoring.
assert.match(demo, /the synthetic candidate fixtures must block the release/);
assert.match(demo, /the engineered fixture must prove the passing branch/);
assert.match(demo, /a leaked corpus must be refused before scoring/);
// The card is drawn from the report the run wrote, and gated for readability like the corpus card.
assert.match(demo, /id = 'calibration-report'/);
assert.match(demo, /Calibration card is not fully readable on screen/);
assert.match(demo, /the card must show the macro-F1 the harness measured/);
assert.match(demo, /the card must show the refusal the leaked corpus earned/);
// The scores are labelled as invented: a synthetic fixture is not a measurement of any model.
assert.match(demo, /invented scores, not measurements of any model/);
assert.match(demo, /synthetic predictions · no model was measured/);
// The scene cannot be dropped silently: its captions are release gates.
assert.match(builder, /The calibration harness is a command-line release gate, not a screen in the app/);
assert.match(builder, /Every score on this card is invented to exercise the arithmetic/);
assert.match(builder, /Each candidate is scored on test at a threshold chosen on validation only/);
assert.match(builder, /One contributor spread across two splits is refused before anything is scored/);
});
+4 -1
View File
@@ -44,11 +44,14 @@ async function dispatchFetch(handler, request) {
}
test('root activation deletes only its obsolete Timmy caches including the legacy v4 cache', async () => {
// The backup/provenance release (issue #35) changes app.js and src/domain.js, so the shell cache
// must advance or an installed client keeps serving the old ledger code indefinitely.
const { listeners, deleted } = loadWorker({
cacheKeys: [
'timmy-shell-v4',
'timmy-shell:/:v4',
'timmy-shell:/:v5',
'timmy-shell:/:v6',
'timmy-shell:/timmy-staging/:v4',
'sibling-shared-origin-cache',
],
@@ -56,7 +59,7 @@ test('root activation deletes only its obsolete Timmy caches including the legac
await dispatchExtendable(listeners.get('activate'));
assert.deepEqual(deleted.sort(), ['timmy-shell-v4', 'timmy-shell:/:v4']);
assert.deepEqual(deleted.sort(), ['timmy-shell-v4', 'timmy-shell:/:v4', 'timmy-shell:/:v5']);
});
test('offline shell lookup uses only the current named cache', async () => {
+120
View File
@@ -0,0 +1,120 @@
import { chromium } from 'playwright';
import assert from 'node:assert/strict';
import { mkdir } from 'node:fs/promises';
// Issue #24 — contributing a confirmed photo to training is a separate opt-in. This runs the real
// app in a browser: a journal save must record nothing, the consent sheet must show the version and
// the receipt before the button works, and a withdrawal must leave a tombstone naming every derived
// artifact. The photo is a synthetic 1x1 PNG constant; no real photo is loaded.
const SYNTHETIC_PHOTO = 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+ip1sAAAAASUVORK5CYII=';
const CONSENT_VERSION = 'training-consent-v1';
await mkdir('artifacts', { recursive: true });
const browser = await chromium.launch({ headless: true });
const context = await browser.newContext({ viewport: { width: 390, height: 844 }, deviceScaleFactor: 2, serviceWorkers: 'block' });
const page = await context.newPage();
const errors = [];
page.on('console', message => { if (message.type() === 'error') errors.push(message.text()); });
page.on('pageerror', error => errors.push(error.message));
page.on('dialog', dialog => dialog.accept());
await page.goto('http://127.0.0.1:4173', { waitUntil: 'networkidle' });
await page.evaluate(() => localStorage.clear());
await page.reload({ waitUntil: 'networkidle' });
const basePath = () => page.evaluate(() => document.querySelector('meta[name="timmy-base-path"]')?.content || '/');
const storeKey = async suffix => `timmy:${await basePath()}:${suffix}`;
const readStore = async suffix => {
const key = await storeKey(suffix);
const raw = await page.evaluate(storeKeys => localStorage.getItem(storeKeys), key);
return raw === null ? null : JSON.parse(raw);
};
const writeStore = async (suffix, value) => {
const key = await storeKey(suffix);
await page.evaluate(({ storeKey: k, payload }) => localStorage.setItem(k, payload), { storeKey: key, payload: JSON.stringify(value) });
};
// 1. Saving a log through the app's own flow contributes nothing.
await page.locator('[data-log]').first().click();
await page.locator('[data-type="4"]').click();
await page.locator('#next').click();
await page.locator('#note').fill('Synthetic test entry');
await page.locator('#next').click();
await page.locator('#save').click();
await page.getByText(/Private log saved/).waitFor();
assert.equal(await readStore('training-contributions-v1'), null, 'a journal save must not record a training contribution');
// 2. Seed one photo-bearing log and one text-only log for the privacy section.
const seeded = [
{
id: 'demo-photo', occurredAt: new Date(Date.now() - 86400000).toISOString(), bristolType: 4,
color: 'brown', urgency: 1, discomfort: 0, note: 'synthetic', photoDataUrl: SYNTHETIC_PHOTO, symptoms: {},
},
{
id: 'demo-text', occurredAt: new Date(Date.now() - 2 * 86400000).toISOString(), bristolType: 3,
color: 'brown', urgency: 2, discomfort: 1, note: 'no photo here', photoDataUrl: '', symptoms: {},
},
];
await writeStore('ledger-v1', seeded);
await page.reload({ waitUntil: 'networkidle' });
await page.locator('[data-view="calendar"]').first().click();
await page.locator('[data-view="privacy"]').first().click();
await page.getByText(/Your poop\. Your phone\./i).waitFor();
const section = page.locator('[data-training-consent]');
assert.equal(await section.count(), 1, 'the privacy page must offer the training contribution section');
assert.match(await section.innerText(), /off by default/i);
assert.match(await section.innerText(), /nothing is uploaded/i);
assert.equal((await page.locator('[data-contribution-consent-version]').innerText()).trim(), CONSENT_VERSION);
assert.equal((await page.locator('[data-contribution-queued-count]').innerText()).trim(), '0');
assert.equal(await page.locator('[data-contribute-entry="demo-photo"]').count(), 1, 'a saved photo may be contributed');
assert.equal(await page.locator('[data-contribute-entry="demo-text"]').count(), 0, 'a log without photo bytes cannot be contributed');
// 3. The receipt and the consent version are on screen before the confirm control works.
await page.locator('[data-contribute-entry="demo-photo"]').click();
const sheet = page.locator('[data-contribution-sheet]');
await sheet.waitFor();
assert.match(await sheet.innerText(), new RegExp(CONSENT_VERSION));
const receiptId = (await page.locator('[data-receipt-id]').innerText()).trim();
assert.ok(receiptId.length > 0, 'the receipt shown before consent must name its own id');
assert.match(await sheet.innerText(), /photo bytes/i);
assert.match(await sheet.innerText(), /note text/i);
assert.equal(await page.locator('#contribute-confirm').isDisabled(), true, 'the contribution button stays disabled until the terms are accepted');
assert.equal(await readStore('training-contributions-v1'), null, 'no receipt may be written before explicit consent');
await page.screenshot({ path: 'artifacts/training-contribution-consent-mobile.png' });
// 4. Consent records exactly one queued-on-device receipt.
await page.locator('#training-consent-check').check();
await page.locator('#contribute-confirm').click();
await page.getByText(/Queued on this device/i).first().waitFor();
const queued = await readStore('training-contributions-v1');
assert.equal(queued.length, 1);
assert.equal(queued[0].entryId, 'demo-photo');
assert.equal(queued[0].consentVersion, CONSENT_VERSION);
assert.equal(queued[0].contributionId, receiptId);
assert.equal(queued[0].state, 'queued-local');
assert.equal(JSON.stringify(queued).includes('data:image'), false, 'the receipt must not copy photo bytes');
assert.equal(JSON.stringify(queued).includes('synthetic'), false, 'the receipt must not copy note text');
await page.locator('[data-contribution-state="queued-local"]').waitFor();
assert.equal((await page.locator('[data-contribution-queued-count]').innerText()).trim(), '1');
await page.screenshot({ path: 'artifacts/training-contribution-queued-mobile.png' });
// 5. Withdrawal leaves a tombstone that addresses every derived artifact by contribution id.
await page.locator(`[data-withdraw="${receiptId}"]`).click();
await page.locator('[data-contribution-state="withdrawn"]').waitFor();
const withdrawn = await readStore('training-contributions-v1');
assert.equal(withdrawn.length, 1, 'the withdrawn receipt is kept as a tombstone, not silently erased');
assert.equal(withdrawn[0].state, 'withdrawn');
assert.ok(withdrawn[0].withdrawnAt, 'the tombstone records when the withdrawal happened');
assert.deepEqual(withdrawn[0].artifacts.map(artifact => artifact.artifact).sort(), ['confirmed-label', 'raw-image']);
for (const artifact of withdrawn[0].artifacts) {
assert.equal(artifact.contributionId, receiptId);
assert.equal(artifact.state, 'withdrawn');
}
assert.match(await page.locator('[data-withdrawn-artifacts]').innerText(), /raw image/i);
assert.equal((await page.locator('[data-contribution-queued-count]').innerText()).trim(), '0');
await page.screenshot({ path: 'artifacts/training-contribution-withdrawn-mobile.png' });
assert.deepEqual(errors, []);
await context.close();
await browser.close();
console.log('PASS training contribution stays opt-in, shows its consent version before commit, and withdrawal names every derived artifact');
+161
View File
@@ -0,0 +1,161 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import {
TRAINING_CONSENT_VERSION,
addContribution,
consentTerms,
contributionEligibility,
contributionForEntry,
defaultContributionState,
derivedArtifacts,
receiptPreview,
withdrawContribution,
} from '../src/training-contribution.js';
// Issue #24 — contributing a photo to Timmy's training corpus is a separate decision from saving
// a log. Nothing here is transmitted: the receipt is what a person could carry to a corpus pipeline
// later, and the withdrawal tombstone is what names the artifacts a deletion has to reach.
const PHOTO = 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+ip1sAAAAASUVORK5CYII=';
const entry = (overrides = {}) => ({
id: 'log-1',
occurredAt: '2026-10-14T07:20:00.000Z',
bristolType: 4,
color: 'brown',
urgency: 1,
discomfort: 0,
note: 'after lunch',
photoDataUrl: PHOTO,
symptoms: { blood: false },
...overrides,
});
const preview = (target = entry(), id = 'contrib-1') => receiptPreview(target, {
contributionId: id,
consentedAt: '2026-10-21T10:00:00.000Z',
});
test('saving a journal entry contributes nothing by default', () => {
assert.deepEqual(defaultContributionState(), { optedIn: false, receipt: null });
});
test('an entry with no saved photo bytes cannot be contributed', () => {
const unavailable = contributionEligibility(entry({ photoDataUrl: '' }));
assert.equal(unavailable.eligible, false);
assert.match(unavailable.reason, /no saved photo/i);
assert.equal(contributionEligibility(entry()).eligible, true);
});
test('the consent terms name the version and what is never shared', () => {
const terms = consentTerms();
assert.equal(terms.version, TRAINING_CONSENT_VERSION);
assert.match(terms.version, /^training-consent-/);
assert.ok(terms.includes.some(item => /photo/i.test(item)));
assert.ok(terms.excludes.some(item => /note/i.test(item)));
assert.ok(terms.excludes.some(item => /symptom/i.test(item)));
// This build is honest about transport: a receipt is queued on the device, not uploaded.
assert.match(terms.transport, /nothing is uploaded/i);
});
test('the receipt shown before consent carries the version, the id, and no free text from the log', () => {
const receipt = preview();
assert.equal(receipt.contributionId, 'contrib-1');
assert.equal(receipt.consentVersion, TRAINING_CONSENT_VERSION);
assert.equal(receipt.consentedAt, '2026-10-21T10:00:00.000Z');
assert.deepEqual(receipt.labels, { bristolType: 4, color: 'brown' });
assert.equal(receipt.state, 'queued-local');
// The confirmed note and symptom flags stay in the journal; they are not part of a contribution.
const serialized = JSON.stringify(receipt);
assert.equal(serialized.includes('after lunch'), false);
assert.equal(serialized.includes('"blood"'), false);
assert.equal(serialized.includes('data:image'), false);
});
test('a receipt cannot be recorded without the current consent version', () => {
assert.throws(
() => addContribution([], { ...preview(), consentVersion: 'training-consent-v0' }),
/consent version/i,
);
});
test('recording a contribution returns a new list and leaves the ledger it was given alone', () => {
const before = [];
const after = addContribution(before, preview());
assert.equal(before.length, 0);
assert.equal(after.length, 1);
assert.equal(after[0].contributionId, 'contrib-1');
assert.equal(contributionForEntry(after, 'log-1')?.contributionId, 'contrib-1');
});
test('the same log cannot be contributed twice while its receipt is still active', () => {
const first = addContribution([], preview());
const second = addContribution(first, preview(entry(), 'contrib-2'));
assert.equal(second.length, 1);
assert.equal(second[0].contributionId, 'contrib-1');
});
test('withdrawal keeps a tombstone that names every derived artifact by contribution id', () => {
const contributions = addContribution([], preview());
const artifacts = derivedArtifacts(contributions[0]);
const { contributions: after, tombstone } = withdrawContribution(
contributions, 'contrib-1', '2026-10-21T11:00:00.000Z',
);
assert.equal(after[0].state, 'withdrawn');
assert.equal(after[0].withdrawnAt, '2026-10-21T11:00:00.000Z');
assert.equal(tombstone.contributionId, 'contrib-1');
assert.deepEqual(
tombstone.artifacts.map(artifact => artifact.artifact).sort(),
artifacts.map(artifact => artifact.artifact).sort(),
);
for (const artifact of tombstone.artifacts) {
assert.equal(artifact.contributionId, 'contrib-1');
assert.equal(artifact.state, 'withdrawn');
}
});
test('the derived artifacts are the image bytes and the confirmed label, addressed by contribution id', () => {
const artifacts = derivedArtifacts(preview());
assert.deepEqual(artifacts.map(artifact => artifact.artifact).sort(), ['confirmed-label', 'raw-image']);
for (const artifact of artifacts) {
assert.equal(artifact.contributionId, 'contrib-1');
assert.equal(artifact.state, 'active');
}
});
test('withdrawing one contribution never touches another receipt', () => {
const list = [
...addContribution([], preview()),
...addContribution([], preview(entry({ id: 'log-2' }), 'contrib-2')),
];
const { contributions: after } = withdrawContribution(list, 'contrib-1', '2026-10-21T11:00:00.000Z');
assert.equal(after.find(item => item.contributionId === 'contrib-1').state, 'withdrawn');
assert.equal(after.find(item => item.contributionId === 'contrib-2').state, 'queued-local');
});
test('withdrawing an unknown contribution id changes nothing and reports no tombstone', () => {
const list = addContribution([], preview());
const { contributions: after, tombstone } = withdrawContribution(list, 'missing', '2026-10-21T11:00:00.000Z');
assert.equal(after, list);
assert.equal(tombstone, null);
});
test('a withdrawn log can be contributed again under a fresh receipt', () => {
const { contributions: withdrawn } = withdrawContribution(
addContribution([], preview()), 'contrib-1', '2026-10-21T11:00:00.000Z',
);
const again = addContribution(withdrawn, preview(entry(), 'contrib-2'));
assert.equal(again.length, 2);
assert.equal(contributionForEntry(again, 'log-1').contributionId, 'contrib-2');
assert.equal(contributionForEntry(again, 'log-1').state, 'queued-local');
});
+3 -1
View File
@@ -14,6 +14,7 @@ test('training ingest strips metadata, derives a pseudonymous split, and writes
'--output-dir', out,
'--subject-id', 'subject-123',
'--consent-version', 'research-v1',
'--contribution-id', '6f1d0c9e-2c4a-4f3b-9d1e-8a7b6c5d4e3f',
'--bristol-type', '4',
'--color', 'brown',
'--quality', 'good',
@@ -45,10 +46,11 @@ test('training ingest rejects records without explicit consent or human review',
'--input', 'tests/fixtures/synthetic-type4.jpg',
'--output-dir', '/tmp/should-not-exist-timmy',
'--subject-id', 'subject-123',
'--contribution-id', '6f1d0c9e-2c4a-4f3b-9d1e-8a7b6c5d4e3f',
'--bristol-type', '4',
'--color', 'brown',
'--quality', 'good',
], { encoding: 'utf8', env: { ...process.env, TIMMY_DATASET_SALT: 'test-only-salt' } });
assert.notEqual(run.status, 0);
assert.match(run.stderr, /consent|reviewer/i);
assert.match(run.stderr, /consent/i);
});