Compare commits

...
Author SHA1 Message Date
Timmy Release 664a909957 fix(demo): keep each operator-report label on one line
Quality gates / quality (push) Waiting to run
The first recorded card split 'ANALYZE REQUESTS' and every 'FAILURE' label
across two lines, detaching each label from its count. Long CLI labels are
now folded into values (one wrapped failures line), the key column holds
only short keys, and the card was geometry-probed at the recording's phone
viewport before rebuilding: 706px of content in a 706px panel, no clipped
key.
2026-11-02 12:59:46 +00:00
Timmy Release ef83ad93bb feat(demo): run the operator health report on camera and fix the run-together row
Quality gates / quality (push) Waiting to run
The release video proved the disclosure sentence but never exercised the
operator surface it describes. This records the real
scripts/model_metrics_report.mjs against the recording's own server, shows
the counters that run produced, and proves a wrong token is refused instead
of answered.

The first run of that CLI printed 'failure ingress_refused0': the longest
failure label exactly filled the 22-character column, so label and count
rendered as one broken token. LABEL_WIDTH is now 30, and a format test
fails if any row loses the gap between its label and its value.
2026-11-02 12:21:14 +00:00
rockachopa 48979bf744 Merge pull request '92: privacy-safe operator health report (#19)
Quality gates / quality (push) Waiting to run
Adds /api/metrics token-gated snapshot, pure operator formatter, CLI report, privacy-screen disclosure, and the demo/gate coverage for it. Closes #19.
2026-11-02 12:06:58 +00:00
Timmy e08b537866 feat(metrics): render the operator health snapshot as a report (#19)
Quality gates / quality (pull_request) Waiting to run
The counters had an endpoint and no way to read them without jq. Adds a pure
formatter plus a CLI that prints the app, queue, and model boundaries.

The formatter is a function of the snapshot and the three frozen vocabularies,
so a caller-supplied failure label cannot reach the terminal: unknown classes
are simply not printed. Refusal paths are explicit — 404 means the route is off
because no token is configured, 403 means the token was rejected — and the token
is never echoed in the report or an error line.

RED: tests/model-metrics-report-format.test.js failed with ERR_MODULE_NOT_FOUND
before src/model-metrics-report.js existed; the refusal-path test failed with a
generic 'answered 404' before the 404 branch was written.

Tests: npm test 298/298, npm run test:metrics-report 8/8, syntax and diff gates
clean, real-Chromium ui acceptance green with the privacy disclosure visible.
2026-11-02 10:56:16 +00:00
timmy 3e2253a052 wip(#19): privacy-safe model metrics in progress — preserved snapshot before nightly release
Local work in progress, not reviewed and not part of any PR. Committed only so the
nightly release can build from a clean main without discarding it.
2026-11-01 12:02:43 +00:00
Timmy Release Engineering 9067ce5306 fix(demo): count the deliberate ingress 400 in the browser error budget (#16)
Quality gates / quality (push) Canceled after 0s
The recording allows no browser console errors except the ones a scene
produces on purpose, and it enumerated only the two fail-closed 503s. The new
server-refusal probe adds exactly one 400, so the budget is now: two 503s, one
400, and nothing else. The first recording stopped here instead of shipping a
demo that had never exercised the boundary; the assertion is what caught it.
2026-10-31 12:21:16 +00:00
Timmy Release Engineering 2019d515c0 feat(demo): show the server refusing hostile bytes, not just the device guard (#16)
Quality gates / quality (push) Canceled after 0s
The #16 demo scene stopped at the client guard, so the boundary the ticket is
actually about — the server deciding from bytes, before any model work — was
never on camera. The recording now lifts its own /api/analyze interceptor for
one probe, posts non-image bytes to the running app, and reads the status and
copy back out of that response. A 200 fails the recording.

Both captions are required release gates, so a later release cannot drop the
scene silently. Rehearsed against the real server: 400 not_an_image, card
measured legible (opaque, in frame, no scroll), interceptor restored before
the next scene.
2026-10-31 12:09:06 +00:00
rockachopa c31d0a510f Merge pull request 'Harden image ingress: sniff the container, refuse polyglots and metadata, and bound the request (#16)' (#82) from timmy/16-image-ingress-hardening into main
Quality gates / quality (push) Canceled after 0s
2026-10-31 12:06:26 +00:00
18 changed files with 1267 additions and 17 deletions

No files matched your search

+1
View File
@@ -7,6 +7,7 @@ A working mobile-first bowel diary with an optional **photo-first AI assist**:
- [Product boundary and release authority decision](docs/PRODUCT-DECISIONS.md)
- [Self-hosted stool-vision decision](research/SELF-HOSTED-STOOL-VISION.md)
- [Production topology, secrets, backup, and rollback runbook](docs/PRODUCTION-RUNBOOK.md)
- [Privacy-safe model health counters (operator view)](docs/MODEL-METRICS.md)
## Build a gated review release
+1 -1
View File
@@ -313,7 +313,7 @@ function withdrawTrainingContribution(contributionId){
}
function privacy(){
shell(`<main><div class="page-title"><span class="eyebrow">Private by design</span><h1>Your poop. Your phone.</h1><p>This prototype has no account, analytics, ad tracker, or server database.</p></div><section class="card privacy-list"><div class="privacy-item"><b>⌂</b><div><h3>Stored locally</h3><p>Saved entries and optional photos live in this browser’s local storage. You can remove one photo without deleting its confirmed log.</p></div></div><div class="privacy-item"><b>⇩</b><div><h3>Portable</h3><p>Export a readable JSON file. Import it in another copy of Timmy.</p></div></div><div class="privacy-item"><b>◎</b><div><h3>AI only when you ask</h3><p>Manual logging never uploads. Analyze a photo sends one compressed copy after consent. Hermes chat may receive up to 20 confirmed text-only entries after you connect; photos and backend credentials never enter chat.</p></div></div></section><section class="section card"><h2>Data controls</h2><div class="row"><button class="btn btn-primary" id="export">Export JSON</button><label class="btn btn-secondary" for="import">Import JSON</label><input class="hidden" type="file" id="import" accept="application/json"></div><p class="fine section">Exports can contain sensitive health notes and photos. Store them somewhere you trust.</p></section>${portabilityNoticeBlock()}${trainingConsentSection()}<section class="section card source-list"><h2>Health sources</h2><p class="fine">The Bristol groupings and urgent-symptom copy are grounded in public clinical guidance.</p><p><a target="_blank" rel="noreferrer" href="https://www.continence.org.au/about-incontinence/bowel-incontinence/bristol-stool-chart/">Continence Health Australia</a></p><p><a target="_blank" rel="noreferrer" href="https://www.nhs.uk/conditions/bleeding-from-the-bottom-rectal-bleeding/">NHS rectal bleeding guidance</a></p><p><a target="_blank" rel="noreferrer" href="https://www.niddk.nih.gov/health-information/digestive-diseases/constipation/symptoms-causes">NIDDK constipation guidance</a></p></section><section class="section card danger-zone"><h2>Delete everything</h2><p class="fine">Permanently removes Timmy’s local ledger and every saved photo from this browser.</p><button class="btn btn-danger" id="delete-all">Delete all local data</button></section></main>`);
shell(`<main><div class="page-title"><span class="eyebrow">Private by design</span><h1>Your poop. Your phone.</h1><p>This prototype has no account, analytics, ad tracker, or server database.</p></div><section class="card privacy-list"><div class="privacy-item"><b>⌂</b><div><h3>Stored locally</h3><p>Saved entries and optional photos live in this browser’s local storage. You can remove one photo without deleting its confirmed log.</p></div></div><div class="privacy-item"><b>⇩</b><div><h3>Portable</h3><p>Export a readable JSON file. Import it in another copy of Timmy.</p></div></div><div class="privacy-item"><b>◎</b><div><h3>AI only when you ask</h3><p>Manual logging never uploads. Analyze a photo sends one compressed copy after consent. Hermes chat may receive up to 20 confirmed text-only entries after you connect; photos and backend credentials never enter chat.</p></div></div><div class="privacy-item" data-health-disclosure><b>♥</b><div><h3>Health counters</h3><p>Timmy’s server keeps in-memory counters of how analysis requests went — allowed, refused, failed, and how long they took — so an outage can be seen without opening anyone’s journal. They never include your notes, photos, image fingerprints, or identity, and they reset when the server restarts.</p></div></div></section><section class="section card"><h2>Data controls</h2><div class="row"><button class="btn btn-primary" id="export">Export JSON</button><label class="btn btn-secondary" for="import">Import JSON</label><input class="hidden" type="file" id="import" accept="application/json"></div><p class="fine section">Exports can contain sensitive health notes and photos. Store them somewhere you trust.</p></section>${portabilityNoticeBlock()}${trainingConsentSection()}<section class="section card source-list"><h2>Health sources</h2><p class="fine">The Bristol groupings and urgent-symptom copy are grounded in public clinical guidance.</p><p><a target="_blank" rel="noreferrer" href="https://www.continence.org.au/about-incontinence/bowel-incontinence/bristol-stool-chart/">Continence Health Australia</a></p><p><a target="_blank" rel="noreferrer" href="https://www.nhs.uk/conditions/bleeding-from-the-bottom-rectal-bleeding/">NHS rectal bleeding guidance</a></p><p><a target="_blank" rel="noreferrer" href="https://www.niddk.nih.gov/health-information/digestive-diseases/constipation/symptoms-causes">NIDDK constipation guidance</a></p></section><section class="section card danger-zone"><h2>Delete everything</h2><p class="fine">Permanently removes Timmy’s local ledger and every saved photo from this browser.</p><button class="btn btn-danger" id="delete-all">Delete all local data</button></section></main>`);
document.querySelector('#export').onclick=exportData;document.querySelector('#import').onchange=importData;document.querySelector('#delete-all').onclick=deleteData;
}
function exportData(){const blob=new Blob([exportLedgerBackup({entries,contributions})],{type:'application/json'}),a=document.createElement('a');a.href=URL.createObjectURL(blob);a.download=`timmy-backup-v${LEDGER_EXPORT_VERSION}.json`;a.click();URL.revokeObjectURL(a.href);portabilityNotice={kind:'ok',version:LEDGER_EXPORT_VERSION,text:`Backup written as version ${LEDGER_EXPORT_VERSION}: ${entries.length} log${entries.length===1?'':'s'} and ${contributions.length} training receipt${contributions.length===1?'':'s'}.`};render();toast('Backup exported');}
Binary file not shown.

After

Width:  |  Height:  |  Size: 486 KiB

+111
View File
@@ -0,0 +1,111 @@
# Privacy-safe model health counters
Issue [#19](https://forge.alexanderwhitestone.com/git/stackchain/timmy-talking-turd/issues/19).
This is the operator surface for "is the worker healthy?", and it is deliberately the smallest one
that answers the question. The journal stays the only place a person's bowel data lives.
## Endpoint
```
GET /api/metrics
Authorization: Bearer $TIMMY_METRICS_TOKEN
```
- `TIMMY_METRICS_TOKEN` unset → **404**. A default deployment has no metrics route at all; nothing is
exposed by forgetting to configure something.
- Token set, header missing or wrong → **403**, with a body that contains no counters.
- Token set and matched → **200** with the snapshot below.
The snapshot is read from process memory. It survives no restart, is written to no log, and is sent
to no third party.
## What is in the snapshot
```json
{
"schema": 1,
"generatedAt": "2026-11-01T10:00:00.000Z",
"app": { "uptimeMs": 0, "analyzeRequests": 0, "refusedBeforeModel": 0 },
"queue": { "inFlight": 0, "limit": 4, "atCapacity": false, "accepted": 0 },
"model": {
"outcomes": { "suggestion": 0, "abstained": 0, "failure": 0 },
"failureClasses": { "ingress_refused": 0, "rate_limited": 0, "disabled": 0,
"provider_unreachable": 0, "provider_rejected": 0,
"schema_invalid": 0, "timeout": 0, "unexpected": 0 },
"abstentionRate": null,
"readiness": { "phase": "checking", "ready": false, "coldLoadMs": null },
"latency": { "count": 0, "meanMs": null, "maxMs": 0,
"bucketsMs": { "1000": 0, "2000": 0, "5000": 0, "15000": 0, "60000": 0 } }
}
}
```
The three blocks are the three boundaries, and each counts only what it owns:
| Block | Owns | Does not own |
|---|---|---|
| `app` | Every `POST /api/analyze` the process saw, and the ones refused before any model work (bad bytes, per-device budget, analysis disabled). | Queue occupancy or model answers. |
| `queue` | Slots handed out and released, and the configured concurrency ceiling. | Whether the request succeeded. |
| `model` | Answers (`suggestion` / `abstained`), classified failures, readiness phase, and model-work latency. | Anything decided before a slot was taken. |
`abstentionRate` is `abstained / (suggestion + abstained)`; it is `null` until the model has answered
at least once. A failed request is **not** an abstention — an outage must not look like a cautious
model.
`latency.bucketsMs` is cumulative: `"5000"` counts every sample at or below 5 s. `coldLoadMs` is the
interval from process start to the first observation in which the pinned model was resident; a worker
that flaps back to `offline` keeps the original figure.
`readiness.phase` uses the same vocabulary the browser uses for the AI affordance (`checking`,
`disabled`, `offline`, `loading`, `ready`), and the value is produced by the same
`visionReadiness()` function the app renders, so a metric and a screen cannot disagree.
## What can never appear in it
The only strings this module stores are members of three frozen lists — outcomes, failure classes,
and readiness phases — plus a fixed latency-bucket edge set. Anything else is bucketed, not copied:
- an unrecognised outcome becomes `failure`;
- an unrecognised failure class becomes `unexpected`;
- an unrecognised readiness phase becomes `checking`.
Therefore no note text, no image bytes, no image hash or fingerprint, no subject key, no caller
address, and no session or cookie value can reach a label. `tests/model-metrics.test.js` asserts this
by feeding hostile strings into every label position, and
`tests/metrics-endpoint.acceptance.test.js` asserts it through the shipping server.
## Reading it
```bash
curl -sS -H "Authorization: Bearer $TIMMY_METRICS_TOKEN" http://127.0.0.1:4173/api/metrics | jq .
```
For a quick look that does not need `jq`, the report renders the same snapshot as three labelled blocks:
```bash
TIMMY_METRICS_TOKEN=... node scripts/model_metrics_report.mjs --url http://127.0.0.1:4173
```
Exit codes are the point of the command as much as the numbers: `0` means a snapshot was rendered,
`1` means the server answered non-200 (404 says the route is disabled because no token is configured,
403 says the supplied token was rejected), `2` means the command itself was called wrongly or the
server could not be reached. The token is read from the environment or `--token` and is never echoed
in the report or in an error line.
What the numbers mean to an operator:
| Signal | Reading |
|---|---|
| `readiness.phase: offline` | The worker is not answering. Manual logging is unaffected; do not restart the app for this. |
| `readiness.phase: loading` | The worker answered but the pinned model is not resident. AI is withheld rather than offered and failed. |
| `queue.atCapacity: true` | Every model slot is in use. Watch `latency.maxMs`; a growing figure with no completions is a stuck inference, not load. |
| `failureClasses.provider_unreachable` climbing | Network or worker outage, not caller error. |
| `failureClasses.schema_invalid` climbing | The worker is answering off-contract. Treat as a regression in the pinned model or prompt, not a user problem. |
| `app.refusedBeforeModel` climbing with no model traffic | Abuse or a broken client; the model never saw it. |
## Known gap
`queue.inFlight` is the app's own admission depth for `POST /api/analyze`. The full bounded queue
(concurrency limit enforcement, cancellation, overload fallback) is issue
[#17](https://forge.alexanderwhitestone.com/git/stackchain/timmy-talking-turd/issues/17); this
snapshot reports the slot accounting that exists today and does not claim the rest.
+2 -1
View File
@@ -57,13 +57,14 @@ enforces that distinction and the app withholds the AI control until it passes.
## Secrets
Four secrets exist, and every one of them is injected at runtime:
Five secrets exist, and every one of them is injected at runtime:
| Secret | Source | Scope |
|---|---|---|
| `TIMMY_DATASET_SALT` | mode-600 environment file on the host | Corpus subject keys and split assignment. Rotating it re-splits future ingests, so it is not rotated casually. |
| Vision provider key or worker token | mode-600 environment file on the host | Only the app process reads it; the browser never receives it. |
| Agent access code | mode-600 environment file on the host | Unlocks `/api/agent/unlock`; the cookie it mints is `HttpOnly` and same-origin. |
| `TIMMY_METRICS_TOKEN` | mode-600 environment file on the host | Optional. Bearer token for the operator-only `/api/metrics` snapshot. When it is unset the route returns 404, so an unconfigured deployment exposes no counters at all. |
| Gitea access token | local operator shell or secret store | Issue and PR automation only. It is never written into a release tree, an archive, or a log. |
Rules:
+4 -3
View File
@@ -4,7 +4,7 @@
"private": true,
"type": "module",
"scripts": {
"test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/draft.test.js tests/suggestion.test.js tests/followup.test.js tests/vision-readiness.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js tests/threat-model.test.js tests/vision-corpus.test.js tests/vision-boundary.test.js tests/worker-benchmark.test.js tests/worker-ready.test.js tests/worker-container.test.js tests/summary.test.js tests/escalation-regression.test.js tests/grounded-chat.test.js tests/photo-lifecycle.test.js tests/training-contribution.test.js tests/portability.test.js tests/abstention-policy.test.js tests/release-security-gate.test.js tests/image-ingress.test.js tests/rate-limit.test.js tests/http-body.test.js tests/analyze-ingress.acceptance.test.js",
"test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/draft.test.js tests/suggestion.test.js tests/followup.test.js tests/vision-readiness.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js tests/threat-model.test.js tests/vision-corpus.test.js tests/vision-boundary.test.js tests/worker-benchmark.test.js tests/worker-ready.test.js tests/worker-container.test.js tests/summary.test.js tests/escalation-regression.test.js tests/grounded-chat.test.js tests/photo-lifecycle.test.js tests/training-contribution.test.js tests/portability.test.js tests/abstention-policy.test.js tests/release-security-gate.test.js tests/model-metrics.test.js tests/metrics-endpoint.acceptance.test.js tests/image-ingress.test.js tests/rate-limit.test.js tests/http-body.test.js tests/analyze-ingress.acceptance.test.js tests/model-metrics-report-format.test.js tests/model-metrics-report.acceptance.test.js",
"test:ui": "node tests/ui.acceptance.mjs",
"test:photo": "node tests/photo-first.acceptance.mjs",
"test:mobile-capture": "node tests/mobile-capture.acceptance.mjs",
@@ -19,7 +19,7 @@
"test:grounded-chat": "node tests/grounded-chat.acceptance.mjs",
"test:photo-lifecycle": "node tests/photo-lifecycle.acceptance.mjs",
"test:training-contribution": "node tests/training-contribution.acceptance.mjs",
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check src/release-security-gate.js && node --check scripts/security_attack_suite.mjs && node --check tests/security-attack.acceptance.mjs && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/review_corpus.py scripts/dataset_card.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py scripts/operator_backup.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh && node --check src/image-ingress.js && node --check src/rate-limit.js && node --check src/http-body.js && python3 -m py_compile scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py && node --check tests/ingress-capture.acceptance.mjs && node --check tests/analyze-ingress.acceptance.test.js",
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check src/release-security-gate.js && node --check scripts/security_attack_suite.mjs && node --check tests/security-attack.acceptance.mjs && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/review_corpus.py scripts/dataset_card.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py scripts/operator_backup.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh && node --check src/model-metrics.js && node --check src/model-metrics-report.js && node --check scripts/model_metrics_report.mjs && node --check src/image-ingress.js && node --check src/rate-limit.js && node --check src/http-body.js && python3 -m py_compile scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py && node --check tests/ingress-capture.acceptance.mjs && node --check tests/analyze-ingress.acceptance.test.js",
"check:diff": "bash scripts/check_diff.sh",
"start": "node server.mjs",
"test:worker-readiness": "node tests/worker-readiness.acceptance.mjs",
@@ -31,7 +31,8 @@
"test:dataset-card": "python3 tests/dataset-card.test.py",
"test:backup": "python3 tests/backup-restore.test.py",
"test:runbook": "python3 tests/production-runbook.test.py",
"test:security-attack": "node tests/security-attack.acceptance.mjs"
"test:security-attack": "node tests/security-attack.acceptance.mjs",
"test:metrics-report": "node --test tests/model-metrics-report-format.test.js tests/model-metrics-report.acceptance.test.js"
},
"devDependencies": {
"playwright": "^1.62.1"
+11 -1
View File
@@ -51,6 +51,8 @@ DEMO_REQUIRED_CAPTIONS = (
"Every pattern line names the confirmed logs behind it, and following one shows exactly those records",
"Two logs is not a pattern: the summary refuses to describe a trend it cannot support",
"A file that only claims to be a photo is refused on the device — nothing is uploaded",
"The device guard is not the only one: the same bytes, sent straight to the running server",
"Refused at the server too: 400 not_an_image, before any model work",
"The answer states the count and cites the four confirmed logs it counted",
"Following a citation opens exactly the confirmed log behind that answer",
"Ask something the journal cannot answer and Timmy refuses instead of reassuring",
@@ -69,6 +71,10 @@ DEMO_REQUIRED_CAPTIONS = (
"An older backup is upgraded on import, and the notice says which version the file came from",
"A file from a newer Timmy is refused with an instruction, and the ledger it would have replaced is untouched",
"A restored log still says it began as an AI suggestion you corrected — provenance is part of the record",
"Health counters are disclosed where you read privacy facts — they never include notes, photos, image fingerprints, or identity",
"The operator health report is a command line, not a screen in the app — this runs it against this recording’s own server",
"Every counter on this card came from the requests this recording just made, and a wrong token is refused instead of answered",
"The snapshot holds counts, a readiness phase, latency, and enumerated failure classes — no note, photo, image fingerprint, or identity",
"The review tool is a command line, not a screen in the app — this runs it against the committed synthetic sample",
"The queue shows the proposal and hides the model answer; revealing it is refused until a reviewer commits",
"The model said Type 6 and the reviewer said Type 5: the disagreement is recorded, not smoothed over",
@@ -193,7 +199,10 @@ def main() -> int:
run(["python3", "tests/backup-restore.test.py", "-v"], tree)
run(["python3", "tests/production-runbook.test.py", "-v"], tree)
server_env = dict(os.environ)
server_env.update({"HOST": "127.0.0.1", "PORT": "4173", "TIMMY_VISION_ENABLED": "false"})
# TIMMY_METRICS_TOKEN gates the operator snapshot for this recording only. It is a demo value
# against a loopback-only server, not a credential, and it never appears in an artifact.
server_env.update({"HOST": "127.0.0.1", "PORT": "4173", "TIMMY_VISION_ENABLED": "false",
"TIMMY_METRICS_TOKEN": "demo-metrics-token-not-a-credential"})
# A leftover server from another checkout answers on this port and the readiness probe cannot
# tell it apart from ours, so every gate and the recorded demo would silently describe a
# different tree. Refuse to build over a live listener, and verify the served bytes afterwards.
@@ -395,6 +404,7 @@ def main() -> int:
"production_runbook_commands": "passed",
"portable_backup_restore": "passed",
"image_ingress_refusal": "passed",
"privacy_safe_health_metrics": "passed",
"dependency_audit_high": "passed",
"syntax": "passed",
"secret_scan": "passed",
+88
View File
@@ -0,0 +1,88 @@
#!/usr/bin/env node
// Operator report for the privacy-safe model counters (issue #19).
//
// It reads one snapshot from a running Timmy server and prints it. The token is a credential for the
// operator surface, so it is read from the environment or argv and never echoed — not in errors, not
// in the report, not in a stack trace.
import { formatOperatorReport } from '../src/model-metrics-report.js';
const USAGE = `Usage: node scripts/model_metrics_report.mjs [options]
--url <origin> base URL of the Timmy server (default http://127.0.0.1:4173)
--token <token> operator token (default: $TIMMY_METRICS_TOKEN)
--json print the raw snapshot instead of the text report
--timeout-ms <ms> request deadline (default 5000)
--help show this text
`;
function parseArgs(argv) {
const options = {
url: process.env.TIMMY_METRICS_URL || 'http://127.0.0.1:4173',
token: process.env.TIMMY_METRICS_TOKEN || '',
json: false,
timeoutMs: 5_000,
help: false,
};
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
if (arg === '--help' || arg === '-h') {
options.help = true;
} else if (arg === '--json') {
options.json = true;
} else if (arg === '--url') {
options.url = argv[++index] ?? '';
} else if (arg === '--token') {
options.token = argv[++index] ?? '';
} else if (arg === '--timeout-ms') {
options.timeoutMs = Number(argv[++index]);
} else {
throw new Error(`unknown option: ${arg}`);
}
}
return options;
}
async function main(argv) {
const options = parseArgs(argv);
if (options.help) {
process.stdout.write(USAGE);
return 0;
}
if (!options.token) {
throw new Error('operator token required: pass --token or set TIMMY_METRICS_TOKEN');
}
const endpoint = new URL('api/metrics', options.url.endsWith('/') ? options.url : `${options.url}/`);
let response;
try {
response = await fetch(endpoint, {
headers: { authorization: `Bearer ${options.token}`, accept: 'application/json' },
signal: AbortSignal.timeout(Number.isFinite(options.timeoutMs) ? options.timeoutMs : 5_000),
});
} catch (error) {
process.stderr.write(`Could not read ${endpoint.origin}/api/metrics: ${error?.message || 'request failed'}\n`);
return 1;
}
if (!response.ok) {
if (response.status === 404) {
// The route is absent rather than guarded: that is a deployment fact, not a credential problem.
process.stderr.write('Metrics are disabled on this server: the route only exists when TIMMY_METRICS_TOKEN is set.\n');
return 1;
}
if (response.status === 403) {
process.stderr.write('The server rejected the operator token.\n');
return 1;
}
process.stderr.write(`The server answered ${response.status} for /api/metrics.\n`);
return 1;
}
const snapshot = await response.json();
process.stdout.write(options.json ? `${JSON.stringify(snapshot, null, 2)}\n` : formatOperatorReport(snapshot));
return 0;
}
main(process.argv.slice(2)).then(code => {
process.exitCode = code;
}).catch(error => {
process.stderr.write(`${error?.message || 'report failed'}\n`);
process.exitCode = 2;
});
+153 -5
View File
@@ -329,6 +329,56 @@ if (analyzeCalls !== 0) throw new Error('An unreadable file was posted to /api/a
await caption('A file that only claims to be a photo is refused on the device — nothing is uploaded', 2600);
await indicate('[data-capture-error="unreadable"]');
await caption('Manual logging stays one tap away, and the photo picker is one tap back', 2000);
// Issue #16 has two boundaries, and the device guard is only the first. The bytes below are offered
// straight to this running app with the demo's own /api/analyze interceptor lifted, so the status and
// the refusal copy on the card are read back from the real server rather than narrated.
await caption('The device guard is not the only one: the same bytes, sent straight to the running server', 2600);
await page.unroute('**/api/analyze');
let serverRefusal;
try {
serverRefusal = await page.evaluate(async () => {
const base = (document.querySelector('meta[name="timmy-base-path"]')?.content || '/').replace(/\/$/, '');
const bytes = new TextEncoder().encode('<!doctype html><html><body>named .jpg, not an image</body></html>');
let binary = '';
for (const byte of bytes) binary += String.fromCharCode(byte);
const response = await fetch(`${base}/api/analyze`, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ consent: true, imageDataUrl: `data:image/jpeg;base64,${btoa(binary)}` }),
});
return { status: response.status, body: await response.json(), contentType: response.headers.get('content-type') };
});
} finally {
await page.route('**/api/analyze', route => {
analyzeCalls += 1;
const response = analyzeOverride || { status: 200, body: suggestionBody };
return route.fulfill({ status: response.status, contentType: 'application/json', body: JSON.stringify(response.body) });
});
}
// A 200 here would mean the boundary is untested and the demo is theatre, so the scene fails instead.
if (serverRefusal.status !== 400 || !/not a JPEG, PNG, or WebP photo/i.test(serverRefusal.body?.error || '')) {
throw new Error(`The running server must refuse non-image bytes, saw ${serverRefusal.status} ${JSON.stringify(serverRefusal.body)}`);
}
await evidenceCard({
id: 'demo-ingress-card',
title: 'IMAGE INGRESS, REFUSED BY THE SERVER',
subtitle: 'The same non-image bytes, posted to this running app in this recording',
rows: [
['status', `${serverRefusal.status} refused`],
['code', String(serverRefusal.body?.code || 'not_an_image')],
['copy', String(serverRefusal.body?.error || '')],
['—', ''],
['provider', 'never called — no model request for these bytes'],
['budget', 'counted by the per-device limiter before the provider is asked'],
['after', 'the journal, the draft, and manual logging are untouched'],
['—', ''],
['scope', 'this probe is a non-image container. Polyglot tails, embedded metadata, dimensions, body size, and the rate limit are enforced and asserted in the suite'],
],
});
const ingressProof = await evidenceCardProof('demo-ingress-card');
if (!ingressProof.ok) throw new Error(`server refusal card is not legible: ${JSON.stringify(ingressProof)}`);
await caption('Refused at the server too: 400 not_an_image, before any model work', 2600);
await page.evaluate(() => document.querySelector('#demo-ingress-card')?.remove());
await tap('#retake-photo', 450);
await indicate('label[for="gallery-photo"]');
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/synthetic-type4.jpg');
@@ -1475,6 +1525,102 @@ await tap('#entry-sheet-close', 450);
await tap('[data-view="privacy"]', 550);
await page.locator('#delete-all').waitFor();
// Issue #19 — the operator health counters are disclosed where a user reads privacy facts, and the
// recording shows the real sentence the app renders, checked in the browser first.
const healthDisclosure = page.locator('[data-health-disclosure]');
await healthDisclosure.waitFor();
const healthCopy = (await healthDisclosure.innerText()).trim();
if (!/how analysis requests went/i.test(healthCopy) || !/image fingerprints/i.test(healthCopy) || !/reset/i.test(healthCopy)) {
throw new Error(`the privacy screen must disclose the privacy-safe health counters: ${healthCopy}`);
}
await page.evaluate(node => node.scrollIntoView({ block: 'center' }), await healthDisclosure.elementHandle());
await caption('Health counters are disclosed where you read privacy facts — they never include notes, photos, image fingerprints, or identity', 3200);
await indicate('[data-health-disclosure]');
// Issue #19, on camera: the operator report is a command line, not a screen in the app. It runs the
// real script against this recording's own server (started with a demo-only loopback token), so every
// number printed on the card was produced by the requests this recording just made — the ingress
// refusals above are counted, not staged. The same script with a wrong token must be refused rather
// than answered, and the token itself must never reach the report.
const operatorToken = process.env.TIMMY_METRICS_TOKEN || '';
if (!operatorToken) {
throw new Error('the demo server must be started with a demo TIMMY_METRICS_TOKEN so the operator report can be exercised');
}
async function operatorReport(args) {
return execFileAsync(process.execPath, ['scripts/model_metrics_report.mjs', ...args], { cwd: process.cwd() })
.then(result => ({ code: 0, stdout: result.stdout, stderr: result.stderr }))
.catch(error => ({ code: typeof error.code === 'number' ? error.code : 1, stdout: error.stdout || '', stderr: error.stderr || '' }));
}
const reportRun = await operatorReport(['--url', 'http://127.0.0.1:4173', '--token', operatorToken]);
if (reportRun.code !== 0) throw new Error(`the operator report must read the snapshot: ${reportRun.stderr.trim()}`);
const refusedRun = await operatorReport(['--url', 'http://127.0.0.1:4173', '--token', 'demo-token-that-is-wrong']);
if (refusedRun.code !== 1 || !/rejected the operator token/i.test(refusedRun.stderr)) {
throw new Error('a wrong operator token must be refused with a stated reason instead of answered');
}
if (reportRun.stdout.includes(operatorToken) || refusedRun.stderr.includes('demo-token-that-is-wrong')) {
throw new Error('the operator token must never appear in the report or in an error line');
}
const parsedRows = [];
let sections = 0;
for (const line of reportRun.stdout.split('\n')) {
const match = /^ {2}(\S.*?)\s{2,}(\S.*)$/.exec(line);
if (match) { parsedRows.push([match[1].trim(), match[2].trim()]); continue; }
const heading = line.trim();
if (heading === 'APP' || heading === 'QUEUE' || heading === 'MODEL') {
parsedRows.push(['—', '']);
parsedRows.push(['section', heading]);
sections += 1;
}
}
if (sections !== 3) throw new Error(`the operator report must separate the app, queue, and model boundaries, not ${sections} of them`);
const rowValue = pattern => {
const found = parsedRows.find(([key]) => pattern.test(key));
if (!found) throw new Error(`the operator report is missing the ${pattern} row: ${reportRun.stdout}`);
return found[1];
};
const failureRows = parsedRows.filter(([key]) => /^failure /i.test(key));
if (failureRows.length !== 8) throw new Error(`the report must print all eight enumerated failure classes, not ${failureRows.length}`);
const failureList = failureRows.map(([key, value]) => `${key.replace(/^failure /i, '')} ${value}`).join(' · ');
const analyzeCount = Number(rowValue(/^analyze requests$/i));
if (!Number.isFinite(analyzeCount) || analyzeCount < 1) {
throw new Error(`the report must count this recording's own requests, not a zero: ${reportRun.stdout}`);
}
const metricsCard = await evidenceCard({
id: 'operator-health-report',
title: 'OPERATOR HEALTH REPORT',
// Long CLI labels are folded into values on purpose: a key column narrow enough to fit the phone
// splits `analyze requests` across two lines and detaches it from its count.
subtitle: 'command line, not app UI — loopback :4173, demo token',
rows: [
['section', 'APP'],
['uptime', rowValue(/^uptime$/i)],
['requests', `${analyzeCount} analyzed · ${rowValue(/^refused before model$/i)} refused before the model`],
['section', 'QUEUE'],
['in flight', rowValue(/^in flight$/i)],
['accepted', rowValue(/^accepted$/i)],
['at capacity', rowValue(/^at capacity$/i)],
['section', 'MODEL'],
['readiness', rowValue(/^readiness$/i)],
['outcomes', rowValue(/^outcomes$/i)],
['abstention', rowValue(/^abstention$/i)],
['latency', rowValue(/^latency$/i)],
['failures', failureList],
['—', ''],
['wrong token', 'refused · exit 1 · no counters'],
],
font: '600 9.5px/1.45 ui-monospace, SFMono-Regular, Menlo, monospace',
keyWidth: 72,
gap: '1px',
rowGap: '6px',
});
await sleep(420);
const metricsCardProof = await evidenceCardProof('operator-health-report');
if (!metricsCardProof.ok) throw new Error(`Operator report card is not fully readable on screen: ${JSON.stringify(metricsCardProof)}`);
await caption('The operator health report is a command line, not a screen in the app — this runs it against this recording’s own server', 3800);
await caption('Every counter on this card came from the requests this recording just made, and a wrong token is refused instead of answered', 3600);
await caption('The snapshot holds counts, a readiness phase, latency, and enumerated failure classes — no note, photo, image fingerprint, or identity', 4000);
await page.evaluate(() => document.querySelector('#operator-health-report')?.remove());
await indicate('#delete-all');
await tap('#delete-all', 650);
await page.getByText(/Local ledger and photos deleted/).waitFor();
@@ -1596,12 +1742,14 @@ await page.evaluate(() => {
});
await sleep(2200);
// The refusal pass deliberately produces two fail-closed 503s (an out-of-range reply, and one blank field
// beside an out-of-range one), which the browser logs as failed resources. Requiring exactly those two
// keeps the scene honest: if the boundary stops refusing, this fails instead of quietly recording a happy
// path, and a dropped scene is caught rather than silently shrinking the evidence.
// Two deliberate refusals show up in the browser console as failed resources, and both are counted
// rather than tolerated: the ingress probe above posts non-image bytes to the real server (one 400),
// and the refusal pass produces two fail-closed 503s (an out-of-range reply, and one blank field
// beside an out-of-range one). Enumerating them keeps the scene honest: if the ingress boundary stops
// refusing, or a dropped scene shrinks the evidence, this fails instead of recording a happy path.
const expectedFailClosed = errors.filter(message => /Failed to load resource: the server responded with a status of 503 \(Service Unavailable\)/.test(message));
if (expectedFailClosed.length !== 2 || errors.length !== expectedFailClosed.length) {
const expectedIngressRefusal = errors.filter(message => /Failed to load resource: the server responded with a status of 400 \(Bad Request\)/.test(message));
if (expectedFailClosed.length !== 2 || expectedIngressRefusal.length !== 1 || errors.length !== 3) {
throw new Error(`Browser errors: ${errors.join(' | ') || 'unexpected error count'}`);
}
// Machine-readable receipt of exactly what was rendered on camera, so the release gate can prove
+62 -5
View File
@@ -1,6 +1,6 @@
import http from 'node:http';
import { isIP } from 'node:net';
import { createHash, randomBytes } from 'node:crypto';
import { createHash, randomBytes, timingSafeEqual } from 'node:crypto';
import { readFile, stat } from 'node:fs/promises';
import { extname, join, normalize } from 'node:path';
import { fileURLToPath } from 'node:url';
@@ -10,6 +10,8 @@ import { AgentGatewayError, createHermesAgentService, resolveHermesAgentConfig }
import { readJsonBody } from './src/http-body.js';
import { inspectImageBytes, IngressError } from './src/image-ingress.js';
import { createRateLimiter } from './src/rate-limit.js';
import { createModelMetrics } from './src/model-metrics.js';
import { visionReadiness } from './src/vision-readiness.js';
const root=fileURLToPath(new URL('.',import.meta.url));
const port=Number(process.env.PORT||4173);
@@ -42,6 +44,36 @@ function escapeHtmlAttribute(value){return String(value).replace(/[&<>"]/g,chara
// that lives only in this process's memory, so no address and no image byte is ever logged.
const bodyTimeoutMs=Number(process.env.TIMMY_BODY_TIMEOUT_MS||15_000);
const analyzeLimiter=createRateLimiter({limit:Number(process.env.TIMMY_ANALYZE_RATE_LIMIT||12),windowMs:Number(process.env.TIMMY_ANALYZE_RATE_WINDOW_MS||60_000)});
// Issue #19: operator health counters. The token is what makes this an operator surface rather than
// a public one — with no token configured the route does not exist at all.
const metricsToken=String(process.env.TIMMY_METRICS_TOKEN||'');
const modelMetrics=createModelMetrics({limit:Number(process.env.TIMMY_MODEL_CONCURRENCY||4)});
function classifyFailure(error){
const code=String(error?.code||'');
if(code==='invalid_json'||code==='invalid_fields')return 'schema_invalid';
const message=String(error?.message||'');
if(/timed out|timeout|abort/i.test(message))return 'timeout';
// A payload the semantic validator rejects never reached the model, so it is an ingress refusal
// rather than a model fault.
if(/consent|JPEG|PNG|WebP|empty|too large/i.test(message))return 'ingress_refused';
if(/invalid response/i.test(message))return 'provider_rejected';
return 'provider_unreachable';
}
// Both the browser's readiness probe and the operator snapshot record the same observation, so the
// cold-load figure reflects the moment the model actually became resident, not the first time an
// operator happened to look.
function observeReadiness(provider){
const readiness=visionReadiness({enabled:visionConfig.enabled,providerReady:provider.ready,modelSeen:provider.modelSeen});
modelMetrics.recordReadiness({phase:readiness.phase,ready:readiness.ready});
return readiness;
}
// A token comparison that does not leak its prefix through response timing. The digests are equal
// length by construction, so timingSafeEqual can be used even for a wrong-length guess.
function operatorTokenMatches(supplied){
if(!metricsToken||!supplied)return false;
const digest=value=>createHash('sha256').update(String(value)).digest();
return timingSafeEqual(digest(supplied),digest(metricsToken));
}
const bucketSalt=randomBytes(16).toString('hex');
function clientBucket(req){return createHash('sha256').update(`${bucketSalt}:${req.socket.remoteAddress||'unknown'}`).digest('hex').slice(0,24);}
function readJson(req,maxBytes=6*1024*1024){return readJsonBody(req,{maxBytes,timeoutMs:bodyTimeoutMs});}
@@ -72,21 +104,34 @@ http.createServer(async(req,res)=>{
if(appPath==='/api/healthz'&&req.method==='GET')return sendJson(res,200,{ok:true,release,commit,visionEnabled:visionConfig.enabled,agentEnabled:agentConfig.enabled});
if(appPath==='/api/vision-status'&&req.method==='GET'){
const provider=await probeVisionProvider(visionConfig);
observeReadiness(provider);
const privacy=visionConfig.profile==='selfhost'?'The compressed image is processed by Timmy’s self-hosted model and is not forwarded to a third-party model provider.':'A compressed copy is sent to the configured AI provider only when you explicitly request analysis.';
return sendJson(res,200,{...visionConfig.publicStatus(),providerReady:provider.ready,modelSeen:provider.modelSeen,privacy});
}
// Operator-only health snapshot (issue #19). With TIMMY_METRICS_TOKEN unset the route is absent,
// so a default deployment exposes no counters at all.
if(appPath==='/api/metrics'&&req.method==='GET'){
if(!metricsToken)return sendJson(res,404,{error:'Not found'});
if(!operatorTokenMatches(String(req.headers.authorization||'').replace(/^Bearer\s+/i,'')))return sendJson(res,403,{error:'Operator token required.'});
observeReadiness(await probeVisionProvider(visionConfig));
return sendJson(res,200,modelMetrics.snapshot());
}
if(appPath==='/api/analyze'&&req.method==='POST'){
// Refusals first, in cost order: the request budget, then the body, then the image bytes. None of
// these depend on the model, so hostile bytes are refused as hostile even when analysis is off —
// otherwise the disabled switch hides the ingress boundary behind a 503 and the refusal is untested.
const startedAtMs=Date.now();
const recordFailure=failureClass=>modelMetrics.recordAnalyze({outcome:'failure',failureClass,durationMs:Date.now()-startedAtMs});
const budget=analyzeLimiter.take(clientBucket(req));
if(!budget.allowed){
recordFailure('rate_limited');
res.setHeader('retry-after',String(Math.max(1,Math.ceil(budget.retryAfterMs/1000))));
return sendJson(res,429,{error:'Too many photos from this device. Wait a moment, then try again — your journal and manual logging still work.'});
}
let payload;
try{payload=await readJson(req)}
catch(error){
recordFailure('ingress_refused');
const sent=sendIngressError(res,error);
// The refusal is written first, then the half-read socket is dropped so a stalled body cannot
// hold a connection or its partial buffer open.
@@ -94,11 +139,23 @@ http.createServer(async(req,res)=>{
return sent;
}
try{ingressGuard(payload)}
catch(error){return sendIngressError(res,error)}
catch(error){recordFailure('ingress_refused');return sendIngressError(res,error)}
// Bytes are known to be a real photo at this point, so the disabled switch is the honest answer.
if(!visionConfig.enabled)return sendJson(res,503,{error:'AI analysis is disabled. Continue manually.'});
try{return sendJson(res,200,await analyzePhoto({payload,config:visionConfig}))}
catch(error){const failure=visionFailureResponse(error);return sendJson(res,failure.status,failure.body)}
if(!visionConfig.enabled){recordFailure('disabled');return sendJson(res,503,{error:'AI analysis is disabled. Continue manually.'})}
// The slot is what the queue boundary reports, and it is released on every exit path so a failed
// analysis cannot quietly consume concurrency.
const slot=modelMetrics.enter();
try{
const result=await analyzePhoto({payload,config:visionConfig});
modelMetrics.recordAnalyze({outcome:result?.status==='suggestion'?'suggestion':'abstained',durationMs:Date.now()-startedAtMs});
return sendJson(res,200,result);
}
catch(error){
const failure=visionFailureResponse(error);
recordFailure(classifyFailure(error));
return sendJson(res,failure.status,failure.body);
}
finally{modelMetrics.leave(slot)}
}
if(appPath==='/api/agent/status'&&req.method==='GET')return sendJson(res,200,agentService.status(cookie(req,'timmy_agent')));
if(appPath==='/api/agent/unlock'&&req.method==='POST'){
+77
View File
@@ -0,0 +1,77 @@
// Renders the operator snapshot from `src/model-metrics.js` as a plain-text report.
//
// This module is a pure function of the snapshot, on purpose: the report is built from numbers and
// from the three frozen vocabularies, so there is no path by which request content could reach the
// terminal even if the counters were compromised. Anything outside the enumerated vocabularies is
// not printed — an unknown failure class is visible as a missing row, never as a new label.
import { FAILURE_CLASSES, OUTCOMES } from './model-metrics.js';
// Wide enough for the longest row label (`failure provider_unreachable`) plus a visible gap: a label
// that fills the column renders run-together with its count (`failure ingress_refused0`), which reads
// as one broken token instead of a metric.
const LABEL_WIDTH = 30;
function row(label, value) {
return ` ${String(label).padEnd(LABEL_WIDTH)}${value}`;
}
function duration(value) {
if (!Number.isFinite(value) || value < 0) return 'n/a';
if (value < 1_000) return `${Math.round(value)}ms`;
const seconds = value / 1_000;
if (seconds < 60) return `${seconds.toFixed(1)}s`;
const minutes = Math.floor(seconds / 60);
if (minutes < 60) return `${minutes}m ${Math.round(seconds % 60)}s`;
return `${Math.floor(minutes / 60)}h ${minutes % 60}m`;
}
function count(value) {
return Number.isFinite(value) ? String(value) : '0';
}
function readinessLine(model) {
const readiness = model?.readiness ?? {};
const phase = typeof readiness.phase === 'string' && readiness.phase ? readiness.phase : 'checking';
return readiness.coldLoadMs === null || readiness.coldLoadMs === undefined
? phase
: `${phase} · cold load ${duration(readiness.coldLoadMs)}`;
}
function abstentionLine(model) {
const rate = model?.abstentionRate;
const answered = (model?.outcomes?.suggestion ?? 0) + (model?.outcomes?.abstained ?? 0);
if (rate === null || rate === undefined || answered === 0) return 'no answers yet';
return `${(rate * 100).toFixed(1)}% of ${answered} answer${answered === 1 ? '' : 's'}`;
}
export function formatOperatorReport(snapshot) {
if (!snapshot || typeof snapshot !== 'object') throw new Error('a metrics snapshot is required');
const app = snapshot.app ?? {};
const queue = snapshot.queue ?? {};
const model = snapshot.model ?? {};
const latency = model.latency ?? {};
const outcomes = OUTCOMES.map(name => `${name} ${count(model.outcomes?.[name])}`).join(' · ');
const lines = [
`Timmy model health · ${snapshot.generatedAt ?? 'unknown time'} · schema ${snapshot.schema ?? '?'}`,
'',
'APP',
row('uptime', duration(app.uptimeMs)),
row('analyze requests', count(app.analyzeRequests)),
row('refused before model', count(app.refusedBeforeModel)),
'',
'QUEUE',
row('in flight', `${count(queue.inFlight)} / ${count(queue.limit)}`),
row('accepted', count(queue.accepted)),
row('at capacity', queue.atCapacity === true ? 'yes' : 'no'),
'',
'MODEL',
row('readiness', readinessLine(model)),
row('outcomes', outcomes),
row('abstention', abstentionLine(model)),
row('latency', `mean ${duration(latency.meanMs)} · max ${duration(latency.maxMs)} · samples ${count(latency.count)}`),
...FAILURE_CLASSES.map(name => row(`failure ${name}`, count(model.failureClasses?.[name]))),
];
return `${lines.join('\n')}\n`;
}
+139
View File
@@ -0,0 +1,139 @@
// Privacy-safe model health metrics for issue #19.
//
// The point of this module is the boundary it does not cross: a metric sample is assembled from
// enumerated outcomes and numbers, never from a request. No note text, no photo hash, no image
// bytes, and no caller identity can reach a label, because the only strings this module stores are
// members of the frozen lists below. Anything else is coerced to a bucket instead of copied.
export const OUTCOMES = Object.freeze(['suggestion', 'abstained', 'failure']);
// Failure classes are the operator's vocabulary for "what went wrong". They are chosen here, not
// observed from an error, so an upstream message can never become a metric label.
export const FAILURE_CLASSES = Object.freeze([
'ingress_refused',
'rate_limited',
'disabled',
'provider_unreachable',
'provider_rejected',
'schema_invalid',
'timeout',
'unexpected',
]);
// Readiness is the same vocabulary the browser already uses for the AI affordance, so a metric and
// the screen can never disagree about why AI is withheld.
export const READINESS_PHASES = Object.freeze(['checking', 'disabled', 'offline', 'loading', 'ready']);
// Failure classes that are decided before a request can occupy a queue slot or wait on the model.
const REFUSED_BEFORE_MODEL = Object.freeze(['ingress_refused', 'rate_limited', 'disabled']);
export function createModelMetrics({ now = () => Date.now(), limit = 4 } = {}) {
const outcomes = { suggestion: 0, abstained: 0, failure: 0 };
const failureClasses = Object.fromEntries(FAILURE_CLASSES.map(name => [name, 0]));
// The queue boundary is tracked by opaque tickets, not by request identity: the only thing that
// survives a request here is the count of slots in use.
const inUse = new Set();
let accepted = 0;
const startedAt = now();
let readiness = { phase: 'checking', ready: false, coldLoadMs: null };
let analyzeRequests = 0;
let refusedBeforeModel = 0;
// Latency is a histogram, never a list of individual timings: a bucket count says how the worker
// behaves without carrying a per-request record of when someone photographed their stool.
const bucketEdgesMs = Object.freeze([1_000, 2_000, 5_000, 15_000, 60_000]);
const bucketsMs = Object.fromEntries(bucketEdgesMs.map(edge => [String(edge), 0]));
let latencyCount = 0;
let latencySumMs = 0;
let latencyMaxMs = 0;
function abstentionRate() {
const answered = outcomes.suggestion + outcomes.abstained;
return answered === 0 ? null : outcomes.abstained / answered;
}
function recordLatency(durationMs) {
if (!Number.isFinite(durationMs) || durationMs < 0) return;
latencyCount += 1;
latencySumMs += durationMs;
latencyMaxMs = Math.max(latencyMaxMs, durationMs);
// Edges are cumulative on purpose: `bucketsMs['5000']` answers "how many samples were at or
// below five seconds", so an operator can read a percentile without this module storing the
// individual timings that produced it.
for (const edge of bucketEdgesMs) {
if (durationMs <= edge) bucketsMs[String(edge)] += 1;
}
// Anything slower than the last edge stays counted in the total but out of the histogram, so a
// hanging request cannot invent a bucket.
}
return {
enter() {
const ticket = Symbol('model-slot');
inUse.add(ticket);
accepted += 1;
return ticket;
},
leave(ticket) {
// Deleting a ticket that is already gone is a no-op, so a double release cannot free a slot
// that a different request still holds.
inUse.delete(ticket);
},
recordReadiness({ phase, ready }) {
const next = READINESS_PHASES.includes(phase) ? phase : 'checking';
// A worker can flap between ready and offline. Only the first time the model became resident
// is a cold load; every later observation is just the current phase.
const coldLoadMs = readiness.coldLoadMs === null && next === 'ready' && ready === true
? Math.max(0, now() - startedAt)
: readiness.coldLoadMs;
readiness = { phase: next, ready: ready === true, coldLoadMs };
},
recordAnalyze({ outcome, failureClass, durationMs }) {
// The label is looked up, never stored: an unrecognised outcome becomes `failure` so the
// counters stay a closed set no matter what a caller sends.
const bucket = OUTCOMES.includes(outcome) ? outcome : 'failure';
outcomes[bucket] += 1;
analyzeRequests += 1;
if (bucket === 'failure') {
failureClasses[FAILURE_CLASSES.includes(failureClass) ? failureClass : 'unexpected'] += 1;
// A refusal decided before any model work (bad bytes, the per-device budget, or the disabled
// switch) is an app fact: it never occupied a queue slot and never waited on the model.
if (REFUSED_BEFORE_MODEL.includes(failureClass)) {
refusedBeforeModel += 1;
} else {
recordLatency(durationMs);
}
} else {
recordLatency(durationMs);
}
},
snapshot() {
return {
schema: 1,
generatedAt: new Date(now()).toISOString(),
app: {
uptimeMs: Math.max(0, now() - startedAt),
analyzeRequests,
refusedBeforeModel,
},
queue: {
inFlight: inUse.size,
limit,
atCapacity: inUse.size >= limit,
accepted,
},
model: {
outcomes: { ...outcomes },
failureClasses: { ...failureClasses },
abstentionRate: abstentionRate(),
readiness: { ...readiness },
latency: {
count: latencyCount,
meanMs: latencyCount === 0 ? null : Math.round(latencySumMs / latencyCount),
maxMs: latencyMaxMs,
bucketsMs: { ...bucketsMs },
},
},
};
},
};
}
+181
View File
@@ -0,0 +1,181 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import http from 'node:http';
import net from 'node:net';
import { spawn } from 'node:child_process';
import { once } from 'node:events';
import { fileURLToPath } from 'node:url';
// Issue #19 at the real boundary: the server that ships, behind a provider stub this test owns, so
// "the metric reflects a real request" and "the metric never contains the request" are both
// measurable instead of asserted from a mock.
const repoRoot = fileURLToPath(new URL('..', import.meta.url));
const OPERATOR_TOKEN = 'test-operator-token-9f2a';
function cleanJpeg({ width = 1200, height = 900 } = {}) {
const sof = Buffer.alloc(17);
sof.writeUInt16BE(0xffc0, 0);
sof.writeUInt16BE(15, 2);
sof[4] = 8;
sof.writeUInt16BE(height, 5);
sof.writeUInt16BE(width, 7);
sof[9] = 3;
for (let i = 0; i < 3; i += 1) {
sof[10 + i * 3] = i + 1;
sof[11 + i * 3] = 0x11;
}
return Buffer.concat([Buffer.from([0xff, 0xd8]), sof, Buffer.from([0xff, 0xd9])]);
}
const dataUrl = (mime, bytes) => `data:${mime};base64,${bytes.toString('base64')}`;
async function freePort() {
const server = net.createServer();
server.listen(0, '127.0.0.1');
await once(server, 'listening');
const { port } = server.address();
server.close();
return port;
}
async function startProvider() {
const state = { calls: 0 };
const server = http.createServer(async (req, res) => {
if (req.url === '/v1/models') {
res.writeHead(200, { 'content-type': 'application/json' });
res.end(JSON.stringify({ data: [{ id: 'metrics-test-model' }] }));
return;
}
state.calls += 1;
for await (const _ of req) { /* drain */ }
res.writeHead(200, { 'content-type': 'application/json' });
res.end(JSON.stringify({
choices: [{
message: {
content: JSON.stringify({
isStool: true, bristolType: 4, color: 'brown', confidence: 0.8,
imageQuality: 'good', observations: 'Synthetic test card.',
}),
},
}],
}));
});
server.listen(0, '127.0.0.1');
await once(server, 'listening');
return {
state,
port: server.address().port,
close: () => new Promise(resolve => {
server.closeAllConnections?.();
server.close(resolve);
}),
};
}
async function withServer(env, run) {
const provider = await startProvider();
const port = await freePort();
const child = spawn(process.execPath, ['server.mjs'], {
cwd: repoRoot,
env: {
...process.env,
HOST: '127.0.0.1',
PORT: String(port),
TIMMY_VISION_ENABLED: 'true',
TIMMY_VISION_PROFILE: 'selfhost',
TIMMY_VISION_BASE_URL: `http://127.0.0.1:${provider.port}/v1`,
TIMMY_VISION_MODEL: 'metrics-test-model',
...env,
},
stdio: ['ignore', 'pipe', 'pipe'],
});
const logs = [];
child.stdout.on('data', chunk => logs.push(chunk.toString()));
child.stderr.on('data', chunk => logs.push(chunk.toString()));
const base = `http://127.0.0.1:${port}`;
try {
const deadline = Date.now() + 15_000;
for (;;) {
try {
const response = await fetch(`${base}/api/healthz`);
if (response.ok) break;
} catch { /* not up yet */ }
if (Date.now() > deadline) throw new Error(`server did not start: ${logs.join('')}`);
await new Promise(resolve => setTimeout(resolve, 100));
}
await run({ base, provider, logs });
} finally {
child.kill('SIGKILL');
await provider.close();
}
}
const postAnalyze = (base, payload) => fetch(`${base}/api/analyze`, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify(payload),
});
const getMetrics = (base, token = OPERATOR_TOKEN) => fetch(`${base}/api/metrics`, {
headers: token ? { authorization: `Bearer ${token}` } : {},
});
test('the metrics snapshot reports the request the server actually analysed', async () => {
await withServer({ TIMMY_METRICS_TOKEN: OPERATOR_TOKEN }, async ({ base, provider }) => {
const analyzed = await postAnalyze(base, { consent: true, imageDataUrl: dataUrl('image/jpeg', cleanJpeg()) });
assert.equal(analyzed.status, 200);
assert.equal(provider.state.calls, 1);
const response = await getMetrics(base);
const snapshot = await response.json();
assert.equal(response.status, 200);
assert.equal(snapshot.app.analyzeRequests, 1);
assert.equal(snapshot.queue.accepted, 1);
assert.equal(snapshot.queue.inFlight, 0, 'a finished request must release its slot');
assert.equal(snapshot.model.outcomes.suggestion, 1);
assert.equal(snapshot.model.latency.count, 1);
assert.equal(snapshot.model.abstentionRate, 0);
assert.equal(snapshot.model.readiness.ready, true, 'readiness is probed from the configured worker');
});
});
test('the metrics route is operator-only, and absent when no token is configured', async () => {
await withServer({ TIMMY_METRICS_TOKEN: OPERATOR_TOKEN }, async ({ base }) => {
const wrong = await getMetrics(base, 'not-the-operator-token');
assert.equal(wrong.status, 403);
assert.doesNotMatch(await wrong.text(), /analyzeRequests/, 'a rejected caller learns nothing about traffic');
const anonymous = await getMetrics(base, '');
assert.equal(anonymous.status, 403);
});
await withServer({}, async ({ base }) => {
const response = await getMetrics(base, '');
assert.equal(response.status, 404, 'an unconfigured deployment exposes no metrics route at all');
});
});
test('the snapshot proves the privacy boundary from the outside', async () => {
await withServer({ TIMMY_METRICS_TOKEN: OPERATOR_TOKEN }, async ({ base }) => {
// A real synthetic photo goes through the real path, so any leak would have real bytes to leak.
const bytes = cleanJpeg();
const analyzed = await postAnalyze(base, {
consent: true,
note: 'fictional note about a very private symptom',
imageDataUrl: dataUrl('image/jpeg', bytes),
});
assert.equal(analyzed.status, 200);
const snapshot = await getMetrics(base).then(response => response.json());
const serialized = JSON.stringify(snapshot);
assert.doesNotMatch(serialized, /very private symptom/i, 'note text must never be a metric');
const base64Run = bytes.toString('base64').slice(0, 24);
assert.ok(base64Run.length === 24, 'sanity: the synthetic image has bytes to leak');
assert.doesNotMatch(serialized, new RegExp(base64Run.replace(/[+/=]/g, '\\$&')), 'image bytes must never be a metric');
assert.doesNotMatch(serialized, /data:image/, 'no data URL may appear in the snapshot');
assert.doesNotMatch(serialized, /127\.0\.0\.1|::1/, 'no caller address may appear in the snapshot');
assert.doesNotMatch(serialized, /image|note|photo/i, 'no request-derived key may appear in the snapshot');
});
});
+97
View File
@@ -0,0 +1,97 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { formatOperatorReport } from '../src/model-metrics-report.js';
import { createModelMetrics } from '../src/model-metrics.js';
function snapshot(overrides = {}) {
const metrics = createModelMetrics({ now: () => Date.parse('2026-11-02T10:00:00.000Z') });
metrics.recordReadiness({ phase: 'ready', ready: true });
return { ...metrics.snapshot(), ...overrides };
}
test('the report names the app, queue, and model boundaries in that order', () => {
const report = formatOperatorReport(snapshot());
assert.match(report, /^APP$/m);
assert.match(report, /^QUEUE$/m);
assert.match(report, /^MODEL$/m);
assert.ok(report.indexOf('APP') < report.indexOf('QUEUE'), 'app boundary comes before queue');
assert.ok(report.indexOf('QUEUE') < report.indexOf('MODEL'), 'queue boundary comes before model');
});
test('the report prints the readiness phase the worker actually reported', () => {
const report = formatOperatorReport(snapshot({
model: { ...snapshot().model, readiness: { phase: 'loading', ready: false, coldLoadMs: null } },
}));
assert.match(report, /readiness\s+loading/);
// A loading worker is not ready, and the report must not imply AI is available.
assert.doesNotMatch(report, /readiness\s+ready/);
});
test('an abstention rate with no answers is absence, never zero percent', () => {
const report = formatOperatorReport(snapshot());
assert.match(report, /abstention\s+no answers yet/i);
assert.doesNotMatch(report, /abstention\s+0(\.0)?%/);
});
test('an abstention rate is rendered from answers only, so failures cannot dilute it', () => {
const metrics = createModelMetrics({ now: () => Date.parse('2026-11-02T10:00:00.000Z') });
metrics.recordReadiness({ phase: 'ready', ready: true });
metrics.recordAnalyze({ outcome: 'suggestion', durationMs: 900 });
metrics.recordAnalyze({ outcome: 'abstained', durationMs: 1200 });
metrics.recordAnalyze({ outcome: 'failure', failureClass: 'timeout', durationMs: 30_000 });
const report = formatOperatorReport(metrics.snapshot());
assert.match(report, /abstention\s+50\.0%/);
assert.match(report, /failure\s+timeout\s+1|timeout 1/i);
});
test('the report renders only the enumerated failure vocabulary, never a caller-supplied label', () => {
const base = snapshot();
const report = formatOperatorReport({
...base,
model: {
...base.model,
failureClasses: {
...base.model.failureClasses,
'note: watery stool at 06:40': 3,
'user@example.com': 1,
},
},
});
assert.doesNotMatch(report, /watery stool/);
assert.doesNotMatch(report, /user@example\.com/);
// The closed set is still fully accounted for, so the extra keys are visibly absent rather than
// silently folded into a bucket the operator would read as something else.
assert.match(report, /unexpected\s+0/);
});
test('the report shows queue occupancy against the configured ceiling', () => {
const metrics = createModelMetrics({ now: () => Date.parse('2026-11-02T10:00:00.000Z'), limit: 3 });
metrics.enter();
metrics.enter();
const report = formatOperatorReport(metrics.snapshot());
assert.match(report, /in flight\s+2 \/ 3/);
assert.match(report, /at capacity\s+no/);
});
test('every row keeps a visible gap between its label and its count', () => {
const metrics = createModelMetrics({ now: () => Date.parse('2026-11-02T10:00:00.000Z') });
// The longest label is the one that used to run together with its value: a column that exactly fits
// the text reads as `failure ingress_refused0`, one broken token instead of a metric and its count.
metrics.recordAnalyze({ outcome: 'failure', failureClass: 'ingress_refused', durationMs: 5 });
const report = formatOperatorReport(metrics.snapshot());
for (const line of report.split('\n')) {
if (!/^ {2}\S/.test(line)) continue;
assert.match(line, /^ {2}\S.*?\s{2,}\S/, `row has no gap between label and value: ${JSON.stringify(line)}`);
}
assert.match(report, /failure ingress_refused\s+1/);
assert.match(report, /failure provider_unreachable\s+0/);
});
@@ -0,0 +1,140 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import http from 'node:http';
import net from 'node:net';
import { spawn, execFile } from 'node:child_process';
import { once } from 'node:events';
import { promisify } from 'node:util';
import { fileURLToPath } from 'node:url';
const execFileAsync = promisify(execFile);
const repoRoot = fileURLToPath(new URL('..', import.meta.url));
const OPERATOR_TOKEN = 'report-operator-token-4b1c';
async function freePort() {
const server = net.createServer();
server.listen(0, '127.0.0.1');
await once(server, 'listening');
const { port } = server.address();
server.close();
return port;
}
async function startProvider() {
const server = http.createServer(async (req, res) => {
if (req.url === '/v1/models') {
res.writeHead(200, { 'content-type': 'application/json' });
res.end(JSON.stringify({ data: [{ id: 'report-model' }] }));
return;
}
for await (const _ of req) { /* drain */ }
res.writeHead(200, { 'content-type': 'application/json' });
res.end(JSON.stringify({
choices: [{ message: { content: JSON.stringify({
isStool: true, bristolType: 4, color: 'brown', confidence: 0.8,
imageQuality: 'good', observations: 'Synthetic test card.',
}) } }],
}));
});
server.listen(0, '127.0.0.1');
await once(server, 'listening');
return { port: server.address().port, close: () => new Promise(resolve => { server.closeAllConnections?.(); server.close(resolve); }) };
}
test('the operator report renders the app, queue, and model boundaries from a live snapshot', async () => {
const provider = await startProvider();
const port = await freePort();
const child = spawn(process.execPath, ['server.mjs'], {
cwd: repoRoot,
env: {
...process.env,
HOST: '127.0.0.1',
PORT: String(port),
TIMMY_VISION_ENABLED: 'true',
TIMMY_VISION_PROFILE: 'selfhost',
TIMMY_VISION_BASE_URL: `http://127.0.0.1:${provider.port}/v1`,
TIMMY_VISION_MODEL: 'report-model',
TIMMY_METRICS_TOKEN: OPERATOR_TOKEN,
},
stdio: ['ignore', 'pipe', 'pipe'],
});
const logs = [];
child.stdout.on('data', c => logs.push(c.toString()));
child.stderr.on('data', c => logs.push(c.toString()));
const base = `http://127.0.0.1:${port}`;
try {
const deadline = Date.now() + 15_000;
for (;;) {
try {
const response = await fetch(`${base}/api/healthz`);
if (response.ok) break;
} catch { /* not up yet */ }
if (Date.now() > deadline) throw new Error(`server did not start: ${logs.join('')}`);
await new Promise(resolve => setTimeout(resolve, 100));
}
const { stdout } = await execFileAsync(process.execPath, ['scripts/model_metrics_report.mjs', '--url', base, '--token', OPERATOR_TOKEN], { cwd: repoRoot });
assert.match(stdout, /APP/);
assert.match(stdout, /QUEUE/);
assert.match(stdout, /MODEL/);
assert.match(stdout, /readiness\s+ready/, 'the report must name the readiness phase it observed');
assert.match(stdout, /abstention/i);
assert.match(stdout, /failure/i);
assert.doesNotMatch(stdout, /OPERATOR_TOKEN|report-operator-token/, 'the report must never print the token');
} finally {
child.kill('SIGKILL');
await provider.close();
}
});
async function startServer(env) {
const port = await freePort();
const child = spawn(process.execPath, ['server.mjs'], {
cwd: repoRoot,
env: { ...process.env, HOST: '127.0.0.1', PORT: String(port), ...env },
stdio: ['ignore', 'pipe', 'pipe'],
});
const base = `http://127.0.0.1:${port}`;
const deadline = Date.now() + 15_000;
for (;;) {
try {
const response = await fetch(`${base}/api/healthz`);
if (response.ok) break;
} catch { /* not up yet */ }
if (Date.now() > deadline) throw new Error('server did not start');
await new Promise(resolve => setTimeout(resolve, 100));
}
return { base, stop: () => child.kill('SIGKILL') };
}
test('the report refuses a missing or wrong operator token without echoing it', async () => {
const tokenless = await startServer({ TIMMY_METRICS_ENABLED: undefined, TIMMY_METRICS_TOKEN: '' });
try {
await assert.rejects(
execFileAsync(process.execPath, ['scripts/model_metrics_report.mjs', '--url', tokenless.base, '--token', 'x'], { cwd: repoRoot }),
error => {
assert.notEqual(error.code, 0);
assert.match(error.stderr, /disabled/i, 'a server with no metrics route must say so, not "unauthorized"');
return true;
},
);
} finally {
tokenless.stop();
}
const guarded = await startServer({ TIMMY_METRICS_TOKEN: OPERATOR_TOKEN });
try {
await assert.rejects(
execFileAsync(process.execPath, ['scripts/model_metrics_report.mjs', '--url', guarded.base, '--token', 'not-the-token'], { cwd: repoRoot }),
error => {
assert.notEqual(error.code, 0);
assert.match(error.stderr, /rejected/i);
assert.doesNotMatch(`${error.stderr}${error.stdout}`, /not-the-token/, 'the rejected token must never be echoed');
return true;
},
);
} finally {
guarded.stop();
}
});
+151
View File
@@ -0,0 +1,151 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { createModelMetrics } from '../src/model-metrics.js';
// Issue #19: the worker has to be operable without the metrics becoming a second copy of the
// journal. These tests are the boundary: what a metric sample may say, and what it may never say.
test('abstention rate is computed from the model answers that came back at all', async () => {
const metrics = createModelMetrics({ now: () => 1_000 });
metrics.recordAnalyze({ outcome: 'suggestion', durationMs: 900 });
metrics.recordAnalyze({ outcome: 'suggestion', durationMs: 1_100 });
metrics.recordAnalyze({ outcome: 'abstained', durationMs: 700 });
metrics.recordAnalyze({ outcome: 'abstained', durationMs: 800 });
const snapshot = metrics.snapshot();
assert.equal(snapshot.model.outcomes.suggestion, 2);
assert.equal(snapshot.model.outcomes.abstained, 2);
assert.equal(snapshot.model.abstentionRate, 0.5);
});
test('a label that is not an enumerated outcome is counted, never copied into the snapshot', async () => {
const metrics = createModelMetrics({ now: () => 1_000 });
// The worst case for a metrics boundary: a caller-supplied string arrives where a label belongs.
metrics.recordAnalyze({ outcome: 'Type 4, urgent, 2026-11-01, milkY4B2cQ9w==', durationMs: 900 });
const snapshot = metrics.snapshot();
const serialized = JSON.stringify(snapshot);
assert.deepEqual(Object.keys(snapshot.model.outcomes).sort(), ['abstained', 'failure', 'suggestion']);
assert.doesNotMatch(serialized, /urgent/i);
assert.doesNotMatch(serialized, /milkY4B2cQ9w/);
assert.equal(snapshot.model.outcomes.failure, 1, 'an unrecognised outcome is one failed answer');
});
test('a failed request is classified and kept out of the abstention denominator', async () => {
const metrics = createModelMetrics({ now: () => 1_000 });
metrics.recordAnalyze({ outcome: 'suggestion', durationMs: 900 });
metrics.recordAnalyze({ outcome: 'failure', failureClass: 'provider_unreachable', durationMs: 2_000 });
metrics.recordAnalyze({ outcome: 'failure', failureClass: 'provider_unreachable', durationMs: 2_100 });
metrics.recordAnalyze({ outcome: 'abstained', durationMs: 800 });
const snapshot = metrics.snapshot();
assert.equal(snapshot.model.failureClasses.provider_unreachable, 2);
assert.equal(snapshot.model.abstentionRate, 0.5, 'a failed answer is not an abstention');
});
test('a failure class outside the enumerated set collapses instead of being echoed', async () => {
const metrics = createModelMetrics({ now: () => 1_000 });
metrics.recordAnalyze({ outcome: 'failure', failureClass: 'note:blood in stool, image 9f2a', durationMs: 2_000 });
const snapshot = metrics.snapshot();
assert.equal(snapshot.model.failureClasses.unexpected, 1);
assert.doesNotMatch(JSON.stringify(snapshot), /blood in stool/);
});
test('analysis latency is reported as warm work, with counts and a bounded histogram', async () => {
const metrics = createModelMetrics({ now: () => 1_000 });
metrics.recordAnalyze({ outcome: 'suggestion', durationMs: 900 });
metrics.recordAnalyze({ outcome: 'suggestion', durationMs: 1_500 });
metrics.recordAnalyze({ outcome: 'abstained', durationMs: 6_000 });
// A refused request never waited on the model, so it must not be sampled as latency.
metrics.recordAnalyze({ outcome: 'failure', failureClass: 'ingress_refused', durationMs: 5 });
const latency = metrics.snapshot().model.latency;
assert.equal(latency.count, 3, 'latency counts model work only');
assert.equal(latency.maxMs, 6_000);
assert.equal(latency.meanMs, 2_800);
// Fixed bucket edges keep the snapshot a histogram of numbers, not a list of timings.
assert.deepEqual(latency.bucketsMs, { '1000': 1, '2000': 2, '5000': 2, '15000': 3, '60000': 3 });
});
test('queue depth is the in-flight model work, and a released slot is never counted twice', async () => {
const metrics = createModelMetrics({ now: () => 1_000, limit: 2 });
const first = metrics.enter();
const second = metrics.enter();
assert.equal(metrics.snapshot().queue.inFlight, 2);
assert.equal(metrics.snapshot().queue.atCapacity, true);
metrics.leave(first);
metrics.leave(first); // a double release must not free a slot that is still in use
const queue = metrics.snapshot().queue;
assert.equal(queue.inFlight, 1, 'the second request still holds its slot');
assert.equal(queue.atCapacity, false);
assert.equal(queue.accepted, 2);
metrics.leave(second);
assert.equal(metrics.snapshot().queue.inFlight, 0);
assert.equal(metrics.snapshot().queue.accepted, 2, 'a completed request is still an accepted request');
});
test('readiness is reported as a phase, and the cold model load is measured once', async () => {
let clock = 10_000;
const metrics = createModelMetrics({ now: () => clock });
metrics.recordReadiness({ phase: 'loading', ready: false });
clock = 74_000; // 64s of weights loading
metrics.recordReadiness({ phase: 'ready', ready: true });
clock = 90_000;
metrics.recordReadiness({ phase: 'offline', ready: false });
const readiness = metrics.snapshot().model.readiness;
assert.equal(readiness.phase, 'offline');
assert.equal(readiness.ready, false);
assert.equal(readiness.coldLoadMs, 64_000, 'the cold load is the first time the model became resident');
});
test('a readiness phase outside the enumerated set is reported as checking', async () => {
const metrics = createModelMetrics({ now: () => 1_000 });
metrics.recordReadiness({ phase: 'user says: bloody stool', ready: true });
const snapshot = metrics.snapshot();
assert.equal(snapshot.model.readiness.phase, 'checking');
assert.doesNotMatch(JSON.stringify(snapshot), /bloody stool/);
});
test('the snapshot keeps the app, queue, and model boundaries separate', async () => {
let clock = 1_000;
const metrics = createModelMetrics({ now: () => clock, limit: 2 });
// One request is refused at the app boundary and never takes a model slot; the other is admitted,
// answered, and released. Each of the three boundaries should account for exactly that.
const slot = metrics.enter();
metrics.recordAnalyze({ outcome: 'suggestion', durationMs: 800 });
metrics.leave(slot);
metrics.recordAnalyze({ outcome: 'failure', failureClass: 'rate_limited', durationMs: 0 });
clock = 41_000;
const snapshot = metrics.snapshot();
assert.deepEqual(Object.keys(snapshot).sort(), ['app', 'generatedAt', 'model', 'queue', 'schema']);
assert.equal(snapshot.app.uptimeMs, 40_000);
assert.equal(snapshot.app.analyzeRequests, 2, 'the app boundary counts every request it saw');
assert.equal(snapshot.app.refusedBeforeModel, 1, 'a refusal that never reached the queue is an app fact');
assert.equal(snapshot.queue.accepted, 1, 'the queue boundary counts what it admitted, not what it answered');
assert.equal(snapshot.queue.inFlight, 0, 'a released slot is not in flight');
assert.equal(snapshot.model.outcomes.suggestion, 1, 'the model boundary counts answers only');
});
+36 -1
View File
@@ -311,8 +311,11 @@ test('release demo shows a filled-in out-of-range field failing closed even besi
assert.match(demo, /tests\/fixtures\/vision-corpus\/blank-plus-out-of-range\.png/);
assert.match(demo, /A filled-in field that is out of range still fails closed, even when the other field came back blank/);
assert.match(demo, /no longer told the photo was unreadable when the model returned an impossible value/);
// The extra fail-closed scene is a real 503, so the demo's error budget must count it.
// The extra fail-closed scene is a real 503, so the demo's error budget must count it, and the
// deliberate ingress probe is the only 400 the recording is allowed to produce.
assert.match(demo, /expectedFailClosed\.length !== 2/);
assert.match(demo, /expectedIngressRefusal\.length !== 1/);
assert.match(demo, /errors\.length !== 3/);
// The caption is a required gate, so a later release cannot silently drop the scene.
assert.match(builder, /A filled-in field that is out of range still fails closed/);
});
@@ -558,3 +561,35 @@ test('release demo runs the RC attack suite on camera and prints the verdict it
assert.match(demo, /'providerTextLeaked' in observed/);
assert.doesNotMatch(demo, /http \$\{observed\.status\} · worker \$\{observed\.providerCalls\}`/);
});
test('release demo shows the privacy-safe health counters where the app discloses them', async () => {
const demo = await readFile(demoPath, 'utf8');
const builder = await readFile(builderPath, 'utf8');
// Issue #19 added an operator surface. The recording has to show the user-facing half of it — the
// disclosure on the privacy screen — and the claim has to be checked in the browser first.
assert.match(demo, /\[data-health-disclosure\]/);
assert.match(demo, /the privacy screen must disclose the privacy-safe health counters/);
assert.match(demo, /Health counters are disclosed where you read privacy facts — they never include notes, photos, image fingerprints, or identity/);
// The caption is a release gate, so a later release cannot drop the scene silently.
assert.match(builder, /Health counters are disclosed where you read privacy facts/);
assert.match(builder, /"privacy_safe_health_metrics": "passed"/);
});
test('release demo runs the operator health report against its own server and shows a refused token', async () => {
const demo = await readFile(demoPath, 'utf8');
const builder = await readFile(builderPath, 'utf8');
// The disclosure alone does not demonstrate the feature. The recording must run the real CLI, show
// counters this recording produced, and prove the wrong-token path is refused rather than answered.
assert.match(demo, /scripts\/model_metrics_report\.mjs/);
assert.match(demo, /the operator report must read the snapshot/);
assert.match(demo, /a wrong operator token must be refused with a stated reason instead of answered/);
assert.match(demo, /the report must count this recording's own requests, not a zero/);
assert.match(demo, /const metricsCardProof = await evidenceCardProof\('operator-health-report'\)/);
// The bearer must never be printed on camera or into an error line.
assert.match(demo, /the operator token must never appear in the report or in an error line/);
// The demo server has to carry a token, or the report cannot be exercised at all.
assert.match(builder, /"TIMMY_METRICS_TOKEN": "demo-metrics-token-not-a-credential"/);
assert.match(builder, /The operator health report is a command line, not a screen in the app/);
});
+13
View File
@@ -43,6 +43,19 @@ await page.waitForTimeout(500);
assert.equal(await page.evaluate(() => window.scrollY), 0, 'reply should not push the page header/navigation out of frame');
await page.screenshot({ path: 'artifacts/timmy-chat-mobile.png', fullPage: false });
// Issue #19: the operator counters have to be disclosed where the user already looks for privacy
// facts, and the disclosure has to name what is not collected. A settings screen that stays silent
// about a new counter is the failure this asserts against.
await page.locator('[data-view="calendar"]').first().click();
await page.locator('[data-view="privacy"]').first().click();
const disclosure = page.locator('[data-health-disclosure]');
assert.equal(await disclosure.isVisible(), true, 'the privacy screen must disclose the health counters');
const disclosureText = await disclosure.innerText();
assert.match(disclosureText, /how analysis requests went/i);
assert.match(disclosureText, /never (contain|hold|include) your notes, photos, image fingerprints, or identity/i);
assert.match(disclosureText, /reset/i);
await page.screenshot({ path: 'artifacts/privacy-health-disclosure.png', fullPage: true });
assert.deepEqual(errors, []);
await browser.close();
console.log('PASS mobile home → log → red-flag → save → Timmy boundary');