Compare commits

...
Author SHA1 Message Date
Timmy Release Engineering 8ce5bc174b fix(ingress): refuse hostile bytes before the disabled-model shortcut (#16)
Quality gates / quality (pull_request) Canceled after 0s
The release builder runs its browser gates with TIMMY_VISION_ENABLED=false, and
the 503 "AI analysis is disabled" branch ran before the request budget, body
limits, and byte-level ingress guard. Under builder conditions every hostile
upload answered 503, so the refusal boundary was never exercised and the
ingress-capture gate this same PR added failed:

  AssertionError: the server must refuse non-image bytes, saw 503

Refusals now run in cost order regardless of model availability: budget, body,
bytes, then the disabled switch. Verified with vision disabled: hostile bytes
-> 400 not_an_image, a valid synthetic photo -> 503 with the existing copy.
npm test 277 pass; ingress-capture and the other browser gates green under the
builder's env.
2026-10-31 12:06:09 +00:00
timmy bc64055196 Merge main into the #16 ingress branch and close what the RC gate caught
Quality gates / quality (pull_request) Canceled after 0s
Brings #82 up to today's `main` (1bbcc10) and resolves the two conflicts as ordered
unions: `package.json` `test`/`check:syntax` and the release builder's syntax allow-list
and evidence keys. Both sides only appended, so nothing was dropped.

Verifying the merge against main's RC attack suite (#40) found two real defects, each
fixed test-first:

- An oversized body destroyed the socket mid-upload, so the caller saw `fetch failed`
  instead of the 413 it had already earned, and `security_attack_suite.mjs` could not
  complete at all. New test in `tests/analyze-ingress.acceptance.test.js` failed with
  `fetch failed` before the fix; `src/http-body.js` now keeps draining without buffering
  and refuses on `end`, so the ceiling still holds and the refusal is readable.
- #16's per-caller budget turned the suite's own load phase into 8/8 429s, which would
  have published refusals as analysis latency. The load phase now runs against a second
  permissive target and the suite refuses to publish a load figure if any sample was
  refused. `buildReport` also publishes `evidenceVerified`, so the verdict is
  re-derivable from the report — asserted by #40's acceptance test, which only trips once
  the verdict is GO.

Evidence on the merged tree: `npm test` 277/277, `test:security-attack` 8/8, the RC attack
suite returns VERDICT: GO with both P0 ingress cases passing (main is BLOCKED on exactly
those two), 14 browser acceptance suites green, and the release demo records end to end
(8:13, 97 captions) including the on-camera refusal scene.
2026-10-31 11:17:13 +00:00
timmy 1bbcc10a35 fix(demo): print the attack-case fields that were actually observed
Quality gates / quality (push) Canceled after 0s
The RC security card rendered `http undefined · worker 12` for the flood case,
which records no status, and repeated its bare status for the malformed case,
which records no provider call. Both read as a broken card rather than the
measurement it is. Build each row from the observed fields that exist and leave
the rest out.
2026-10-30 12:22:56 +00:00
timmy 0bf8230280 Merge pull request #91: RC attack suite and P0 release-security gate (#40)
Quality gates / quality (push) Canceled after 0s
Executable abuse suite against the real boundary with a counting stub worker, plus the verdict rule that makes a failing P0 case block RC approval. First run: VERDICT BLOCKED (two P0 ingress holds, one P1 queue hold).
2026-10-30 12:02:43 +00:00
timmy f0a9a734db feat(security): attack the RC before users do and gate approval on P0 findings (#40)
Quality gates / quality (pull_request) Canceled after 0s
Add an executable abuse suite that drives the real server with synthetic hostile bytes while a
counting stub worker records every call that reaches the model boundary, plus the verdict rule that
turns those observations into a release decision.

- src/release-security-gate.js: nearest-rank latency percentiles, per-case pass/fail derivation that
  names the field that broke, and the rule that any failing P0 case blocks RC approval while a
  required class with no verified evidence blocks it too. Reports are redacted and fail closed.
- scripts/security_attack_suite.mjs: oversized body, HTML polyglot as JPEG, undecodable bytes,
  missing consent, non-conforming worker answer, byte-identical replay, concurrent burst, captured
  log scan, and an out-of-surface deletion case counted only against a real test file. Writes the
  report, prints the verdict, exits 1 when blocked.
- docs/RC-SECURITY-EVIDENCE.md: procedure, rule, and the current finding stated plainly.
- Demo: a real run of the suite on camera, printing the verdict it derived instead of a flattering
  one; CI regenerates the same evidence on every change.

The suite currently returns VERDICT: BLOCKED: bytes are trusted from the data-URL label, so an HTML
polyglot still reaches the worker and only fails afterwards on schema. That is a finding against
#16, not a passing check, and it is why the gate treats a failing case as a block.
2026-10-30 11:12:52 +00:00
timmy 65a0258bba Merge pull request 'Operator backup, restore, and production runbook (#39)' (#90)
Quality gates / quality (push) Canceled after 0s
2026-10-29 12:05:42 +00:00
Timmy 48f0f56835 feat(ops): operator backup, restore, and production runbook (#39)
Quality gates / quality (pull_request) Canceled after 0s
The app keeps the journal in the person's browser, so the server has no diary to lose. What an
operator can lose is the corpus: the append-only manifest.jsonl with its provenance records and its
withdrawal tombstones. scripts/operator_backup.py archives only records and names every path it
refuses (model weights, transient inference images, secrets, logs, symlinks, oversize) with the
reason, writes one SHA-256 per stored member, and produces a reproducible archive.

restore re-verifies every digest, member name, and record count before it writes a byte, and refuses
absolute, traversal, link, unlisted, and non-empty-target cases, so a tampered archive or a lost
tombstone fails loudly. docs/PRODUCTION-RUNBOOK.md covers topology, network boundaries (including
the app's real 0.0.0.0 default and the staging HOST=127.0.0.1 template), the four secrets, retention,
and both rollback paths; its documented commands are parsed against the real CLIs by its test.

The release demo runs the real backup, verify, and restore against a synthetic corpus built by the
real ingest and revoke commands and shows what was stored, what was refused, and what came back.
2026-10-29 11:02:59 +00:00
timmy 7fe1901703 fix(demo): make the on-camera caption bar opaque and prove its last line is not clipped
Quality gates / quality (push) Canceled after 0s
The release-delta caption is paragraph length. Rendered at 94% opacity over the running app, the
app's own text bled through it, and the final line sat flush against the bar's bottom edge — the one
caption a reviewer most needs to read was the hardest one to read. The bar is now opaque, long copy
steps down to 10.5/9.5px, and caption() measures the last line box with a Range and fails the run when
the bottom padding is under 3px, so a cramped caption can no longer be recorded as proof.
2026-10-28 12:47:04 +00:00
rockachopa d7026c0ed3 Merge pull request 'Publish the dataset card and labeling guide from the manifest (#27)' (#89) from timmy/27-dataset-card into main
Quality gates / quality (push) Canceled after 0s
Publish the dataset card and labeling guide from the manifest (#27)

Regression tests: npm test 229 pass; dataset-card 14, corpus-manifest 8, corpus-review 21, calibration-harness 19 OK.
2026-10-28 12:02:27 +00:00
timmy 2b773ba58c test(release): record the ingress refusal on camera and gate its caption (#16)
Quality gates / quality (pull_request) Successful in 2m57s
The demo now picks a committed text fixture that only claims to be a photo, asserts on camera that
nothing was posted to /api/analyze, and keeps manual entry in the same scene. The builder requires
that caption to have painted and records image_ingress_refusal in the manifest; the threat model
names each implemented control, its module, and the tests that enforce it.
2026-10-20 08:21:24 +00:00
timmy 9b72872466 feat(photo): give an unreadable file its own upload-free state instead of a failed analysis (#16)
Selecting a file the browser cannot decode used to render "Timmy couldn't analyze that safely",
which described an upload that never happened. It now states that the file isn't a photo Timmy can
read and that nothing was uploaded, with manual entry and the picker both one tap away. The state is
asserted in a new browser suite that also asks the running server to refuse the same bytes, and the
existing mobile capture suite asserts the new copy across both viewports.
2026-10-20 08:21:24 +00:00
timmy ba9afe0a1d feat(ingress): decide photo validity from bytes, budget, and header before any provider work (#16)
The declared data-URL MIME was a claim the boundary believed. Every image decision now comes from
the bytes: the container is sniffed, walked to its own end marker, refused when metadata segments
remain, and refused above a 4096px edge or 16MP. The request path adds a per-caller sliding-window
budget (429 + Retry-After) and a body deadline (408), so a flood, a stalled body, or a decode bomb
fails before a provider call.

Log hygiene is part of the contract: refusal copy is fixed prose, the bucket key is a salted hash
kept in memory only, and the acceptance test asserts no image bytes reach the process log.
2026-10-20 08:21:20 +00:00
30 changed files with 3050 additions and 16 deletions

No files matched your search

+2
View File
@@ -32,6 +32,7 @@ jobs:
- name: Unit and security tests
run: |
npm test
npm run test:security-attack
python3 tests/staging-deploy.test.py -v
python3 tests/corpus-manifest.test.py -v
python3 tests/corpus-review.test.py -v
@@ -55,6 +56,7 @@ jobs:
npm run test:ui
npm run test:photo
npm run test:mobile-capture
npm run test:ingress-capture
npm run test:sleek
npm run test:manual-fallback
npm run test:provisional
+1
View File
@@ -5,3 +5,4 @@ __pycache__/
*.pyc
.env
.env.*
artifacts/security/
+23
View File
@@ -6,6 +6,7 @@ A working mobile-first bowel diary with an optional **photo-first AI assist**:
- [Product and safety boundary](PRODUCT.md)
- [Product boundary and release authority decision](docs/PRODUCT-DECISIONS.md)
- [Self-hosted stool-vision decision](research/SELF-HOSTED-STOOL-VISION.md)
- [Production topology, secrets, backup, and rollback runbook](docs/PRODUCTION-RUNBOOK.md)
## Build a gated review release
@@ -28,6 +29,25 @@ sudo python3 scripts/deploy_staging.py rollback --commit PRIOR_FULL_COMMIT
Commands are fixed argv arrays executed with `shell=False`; JSON argv overrides and `--root` exist for rootless fixtures. See [the staging runbook](docs/STAGING-RUNBOOK.md) for host prerequisites, mode-600 environment injection, Caddy validation, smoke credentials, logs, backups, removal, and the explicit no-live-change boundary.
## Back up and restore the records an operator holds
The app keeps the journal in the person's own browser, so the server has no diary to back up. What an
operator can lose is the corpus: the append-only `manifest.jsonl` with its provenance records and its
withdrawal tombstones, plus the derived images those records point at.
```bash
python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy-nightly.tar.gz --label nightly
python3 scripts/operator_backup.py verify --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json
python3 scripts/operator_backup.py restore --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json --target /srv/timmy-corpus.restore-check
```
Model weights, transient inference images, secrets, logs, and symlinks are refused by name, and every
skipped path is listed in the manifest with its reason. Each stored member carries a SHA-256, the
archive is reproducible, and a restore re-verifies every digest and the record counts before it writes
anything — a tampered archive or a lost tombstone fails loudly instead of restoring quietly. See the
[production runbook](docs/PRODUCTION-RUNBOOK.md) for topology, network boundaries, secrets, retention,
and the rollback paths.
## Run with the self-hosted open-weight path
```bash
@@ -102,6 +122,7 @@ npm run test:grounded-chat # chat answers cite the ledger, or refuse to answe
npm run test:photo-lifecycle # photo presence, explicit reveal, remove-photo-keeps-log, delete-all
npm run test:calibration # the release-gate harness: gates, threshold protocol, refused corpora
npm run test:review # blind review queue, held-back model answer, an append-only ledger, withdrawals
npm run test:security-attack # hostile synthetic uploads at the real boundary, with a counting worker (#40)
npm audit --audit-level=high
```
@@ -140,6 +161,8 @@ The reviewer tool ([scripts/review_corpus.py](scripts/review_corpus.py), [contra
What the corpus is, and what a label may assert, is published rather than implied: [docs/DATASET_CARD.md](docs/DATASET_CARD.md) is generated from the manifest by [scripts/dataset_card.py](scripts/dataset_card.py), so every count on it is read back from real records and every fact the pipeline does not collect (devices, contexts, licenses) is printed as `pending` instead of estimated — a card for a corpus where one contributor straddles two splits is refused, not published. [docs/LABELING_GUIDE.md](docs/LABELING_GUIDE.md) is the reviewer's rulebook: visible form, broad visible color, and photo quality only, visible color is not blood, nothing is diagnosed, and a model suggestion is provisional input to a human decision rather than the label.
The release candidate is attacked before anyone is asked to trust it. [scripts/security_attack_suite.mjs](scripts/security_attack_suite.mjs) drives the real server with synthetic hostile bytes — an oversized body, an HTML polyglot dressed as a JPEG, undecodable bytes, a request without consent, a worker answer that breaks the schema, a byte-identical replay, and a concurrent burst — while a counting stub worker records every call that reaches the model boundary. [docs/RC-SECURITY-EVIDENCE.md](docs/RC-SECURITY-EVIDENCE.md) owns the rule: any failing P0 case blocks RC approval, and a required class with no executable evidence blocks it too. **It currently returns `VERDICT: BLOCKED`** — bytes are still trusted from the data-URL label, so a hostile file reaches the worker before it fails on schema. That is a published finding against [#16](https://forge.alexanderwhitestone.com/git/stackchain/timmy-talking-turd/issues/16), not a passing check.
## Architecture
```text
+5 -2
View File
@@ -340,20 +340,23 @@ function photoFirstBody(mode,error=''){
if(mode==='pick')return `${visionStatusHtml()}<div class="scan-hero"><img src="${appPath('assets/timmy.svg')}" alt="Timmy"><h3>One photo. Two useful suggestions.</h3><p>Timmy can suggest the visible Bristol form and color. A camera cannot know urgency, pain, symptoms, or a diagnosis.</p></div>${error?`<p class="capture-status" role="status">${esc(error)}</p>`:''}<div class="capture-choice-grid"><label class="photo-capture" for="camera-photo"><b>📷</b><strong>Take photo</strong><span>Use the rear camera</span><input id="camera-photo" type="file" accept="image/*" capture="environment"></label><label class="photo-capture" for="gallery-photo"><b>▧</b><strong>Choose from gallery</strong><span>JPEG, PNG, or WebP</span><input id="gallery-photo" type="file" accept="image/*"></label></div><button class="btn btn-ghost btn-wide section" id="manual-from-scan">Continue without AI</button>`;
if(mode==='ready'){const readiness=visionReadiness(visionStatus);if(!readiness.ready)return `${visionStatusHtml()}<img class="photo-preview scan-preview" src="${form.photoDataUrl}" alt="Photo kept on this device"><p class="quality-note">${esc(photoHint)}</p><p class="fine">${esc(readiness.message)} Nothing is uploaded, and this photo stays on this device until you save the entry.</p><button class="btn btn-primary btn-wide" id="manual-from-scan">${esc(readiness.manualLabel)}</button><button class="btn btn-ghost btn-wide section" id="retake-photo">Use another photo</button>`;const processingCopy=visionStatus?.profile==='selfhost'?'Timmy’s server does not save it. The compressed copy stays on Timmy’s self-hosted model server.':'Timmy’s server does not save it. Your configured AI provider processes it under that provider’s terms.';return `${visionStatusHtml()}<img class="photo-preview scan-preview" src="${form.photoDataUrl}" alt="Photo awaiting AI analysis"><p class="quality-note">${esc(photoHint)}</p><div class="consent-card"><label class="check"><input id="ai-consent" type="checkbox"><span><strong>Send this compressed copy for one-time AI analysis.</strong><br>${processingCopy}</span></label></div><button class="btn btn-primary btn-wide" id="analyze-photo" disabled>Analyze visible form + color</button><button class="btn btn-ghost btn-wide section" id="retake-photo">Use another photo</button>`;}
if(mode==='analyzing')return `<div class="analyzing"><img src="${appPath('assets/timmy.svg')}" alt="Timmy"><div class="spinner" aria-hidden="true"></div><h3>Timmy is looking at form and color…</h3><p>Not symptoms. Not disease. Not whether Taco Bell was a strategic error.</p></div>`;
// An unreadable file is its own state, not a failed analysis: nothing was sent, nothing was
// attempted, and saying "couldn't analyze that safely" here implied an upload that never happened.
if(mode==='unreadable')return `<div class="scan-result needs-input" data-capture-error="unreadable"><b>⚠</b><h3>That file isn’t a photo Timmy can read.</h3><p>Nothing was uploaded. Choose a JPEG, PNG, or WebP photo, take a new one, or fill it out manually.</p></div><button class="btn btn-primary btn-wide" id="manual-from-scan">Fill it out manually</button><button class="btn btn-ghost btn-wide section" id="retake-photo">Choose another photo</button>`;
if(mode==='error')return `<div class="scan-result needs-input"><b>↻</b><h3>Timmy couldn’t analyze that safely.</h3><p>${esc(error||'Continue manually or try a clearer photo.')}</p></div><button class="btn btn-primary btn-wide" id="manual-from-scan">Fill it out manually</button><button class="btn btn-ghost btn-wide section" id="retake-photo">Try another photo</button>`;
if(suggestionState.status==='pending'){const copy=provenanceCopy(suggestionState,form);return `<div class="scan-result provisional" data-provenance="${copy.status}" aria-label="${esc(copy.aria)}"><span class="ai-badge">${esc(copy.label)} · ${Math.round(suggestionState.suggestion.confidence*100)}% CONFIDENCE</span><div class="suggestion-pair"><div><small>BRISTOL FORM</small><strong>Type ${suggestionState.suggestion.bristolType}</strong></div><div><small>VISIBLE COLOR</small><strong>${esc(suggestionState.suggestion.color)}</strong></div></div><p>${esc(aiSuggestion.observations||'Visual match found.')}</p><p class="provenance-note" role="status">${esc(copy.detail)}</p><p class="fine">${esc(aiSuggestion.warning)}</p></div><button class="btn btn-primary btn-wide" id="use-suggestion">Use these suggestions</button><button class="btn btn-ghost btn-wide section" id="manual-from-scan">Review everything manually</button>`;}
return `<div class="scan-result needs-input"><b>?</b><h3>No confident match.</h3><p>${esc(aiSuggestion?.reason||'The image was too uncertain to prefill safely.')}</p></div><button class="btn btn-primary btn-wide" id="manual-from-scan">Choose the form yourself</button><button class="btn btn-ghost btn-wide section" id="retake-photo">Try another photo</button>`;
}
function showPhotoFirst(mode='pick',error=''){
photoFirstMode=mode;
document.querySelector('.sheet-backdrop')?.remove();const wrap=document.createElement('div');wrap.className='sheet-backdrop';wrap.innerHTML=`<section class="sheet scan-sheet" role="dialog" aria-modal="true" aria-labelledby="scan-title"><div class="sheet-handle"></div><div class="sheet-header"><div><span class="eyebrow">Photo-first log</span><h2 id="scan-title">${mode==='result'?'Review Timmy’s suggestion':mode==='analyzing'?'Analyzing privately':'Start with the camera'}</h2></div><button class="icon-btn" id="close-sheet" aria-label="Close">×</button></div>${photoFirstBody(mode,error)}</section>`;document.body.append(wrap);document.querySelector('#close-sheet').onclick=()=>wrap.remove();wrap.onclick=e=>{if(e.target===wrap)wrap.remove()};
document.querySelector('.sheet-backdrop')?.remove();const wrap=document.createElement('div');wrap.className='sheet-backdrop';wrap.innerHTML=`<section class="sheet scan-sheet" role="dialog" aria-modal="true" aria-labelledby="scan-title"><div class="sheet-handle"></div><div class="sheet-header"><div><span class="eyebrow">Photo-first log</span><h2 id="scan-title">${mode==='result'?'Review Timmy’s suggestion':mode==='analyzing'?'Analyzing privately':mode==='unreadable'?'That photo could not be read':'Start with the camera'}</h2></div><button class="icon-btn" id="close-sheet" aria-label="Close">×</button></div>${photoFirstBody(mode,error)}</section>`;document.body.append(wrap);document.querySelector('#close-sheet').onclick=()=>wrap.remove();wrap.onclick=e=>{if(e.target===wrap)wrap.remove()};
document.querySelectorAll('#camera-photo,#gallery-photo').forEach(file=>{file.onchange=handleAiPhoto;file.addEventListener('cancel',()=>showPhotoFirst('pick','Camera or photo picker closed. If permission was denied, allow camera access in browser settings, choose from the gallery, or continue without AI.'))});
const consent=document.querySelector('#ai-consent'),analyze=document.querySelector('#analyze-photo');if(consent&&analyze)consent.onchange=()=>analyze.disabled=!consent.checked||!visionReadiness(visionStatus).ready;if(analyze)analyze.onclick=runAiAnalysis;
document.querySelector('#retake-photo')?.addEventListener('click',()=>{form.photoDataUrl='';photoHint='';resetSuggestion();showPhotoFirst('pick')});
document.querySelector('#manual-from-scan')?.addEventListener('click',()=>{resetSuggestion();showLogStep(1)});
document.querySelector('#use-suggestion')?.addEventListener('click',()=>{suggestionState=confirmSuggestion(suggestionState);form=mergeVisualSuggestion(form,aiSuggestion);form.typeChosen=true;showLogStep(1)});
}
async function handleAiPhoto(e){const file=e.target.files[0];if(!file)return;try{const result=await compressPhoto(file);form.photoDataUrl=result.dataUrl;photoHint=photoQualityMessage(result);showPhotoFirst('ready')}catch{showPhotoFirst('error','That image could not be read. Try another photo.')}}
async function handleAiPhoto(e){const file=e.target.files[0];if(!file)return;try{const result=await compressPhoto(file);form.photoDataUrl=result.dataUrl;photoHint=photoQualityMessage(result);showPhotoFirst('ready')}catch{form.photoDataUrl='';photoHint='';showPhotoFirst('unreadable')}}
async function runAiAnalysis(){showPhotoFirst('analyzing');try{const response=await fetch(appPath('api/analyze'),{method:'POST',headers:{'content-type':'application/json'},body:JSON.stringify({imageDataUrl:form.photoDataUrl,consent:true})});const data=await response.json();if(!response.ok)throw new Error(data.error||'AI analysis is unavailable.');aiSuggestion=data;suggestionState=createSuggestionState(data);showPhotoFirst('result')}catch(error){showPhotoFirst('error',error.message||'AI analysis is unavailable. Continue manually.')}}
function openLogger(){beginDraft();photoHint='';resetSuggestion();showLogStep(1)}
+21
View File
@@ -144,6 +144,26 @@ Each entry names the abuse case, the mitigation that is already committed or que
- **Testable follow-up:** #39
- **Evidence:** The production topology, secrets, backup, and rollback runbook, exercised against the deletion path so that purge reaches memory, temporary files, logs, and backups, with observability from #41 confirming no bytes reach the metrics surface.
## Ingress controls implemented for #16
`/api/analyze` is the only route that accepts image bytes, and every limit below is decided inside
`server.mjs` before `analyzePhoto` runs, so a refusal never costs a provider call.
| Control | Where | Behaviour |
|---|---|---|
| Per-caller request budget | `src/rate-limit.js` | Sliding window, 12 requests per 60s by default (`TIMMY_ANALYZE_RATE_LIMIT`, `TIMMY_ANALYZE_RATE_WINDOW_MS`); refusal is `429` with `Retry-After`. The key is a salted SHA-256 of the socket address held in process memory only — no address, no identity, no bytes are stored. |
| Body ceiling and deadline | `src/http-body.js` | 6 MB ceiling and a 15s deadline (`TIMMY_BODY_TIMEOUT_MS`); `413` and `408` respectively, and the half-read socket is dropped after the refusal is written. |
| Container sniffing | `src/image-ingress.js` | The declared data-URL MIME is treated as a claim, never evidence: JPEG/PNG/WebP is decided from magic bytes, and a mismatch is refused. |
| Polyglot / trailing payloads | `src/image-ingress.js` | The container is walked to its own end marker (PNG `IEND`, JPEG `EOI`, RIFF length); anything appended after the image is refused instead of forwarded. |
| Embedded metadata | `src/image-ingress.js` | JPEG APP1/APP13/COM segments and PNG `eXIf`/`tEXt`/`zTXt`/`iTXt`/`tIME` chunks are refused. The app's own upload is a canvas re-encode (JPEG q0.7, longest edge 1200px) and carries none, so the accepted path is already metadata-free. |
| Pixel limits | `src/image-ingress.js` | Dimensions are read from the header (PNG `IHDR`, JPEG SOF, WebP `VP8X`/`VP8`/`VP8L`) and refused above a 4096px edge or 16 megapixels, so a decode bomb never reaches the worker. |
| Log hygiene | `server.mjs` | Refusal copy is fixed prose; no request body, image bytes, hash, or address is written to a log or returned in an error. |
Residual, and not claimed as solved here: the server refuses metadata-bearing or mismatched
containers rather than transcoding them itself, so a client that cannot produce a clean re-encode
must use the app's capture path. PNG chunk CRCs are not validated (structure is), and the request
budget is per process, not per cluster.
## Accepted residual risks
- **Host compromise.** Data in a running process can be read by a sufficiently privileged attacker. This is accepted for the private beta and bounded by least privilege plus the purge procedure in #39; it is not claimed as eliminated.
@@ -154,4 +174,5 @@ Each entry names the abuse case, the mitigation that is already committed or que
- This document is enforced by `tests/threat-model.test.js`, which asserts that the diagram covers every byte-bearing component, that each required abuse case is a numbered threat with an owner and a tracked follow-up ticket, and that the model keeps the non-diagnostic boundary and ships no real medical images.
- The test runs in `npm test` and therefore inside the CI quality gate and the release build, so the threat model cannot silently drift from the code it describes.
- The ingress controls above are enforced by `tests/image-ingress.test.js`, `tests/rate-limit.test.js`, `tests/http-body.test.js`, and `tests/analyze-ingress.acceptance.test.js`, which runs the shipping server against synthetic bytes and a counting provider stub; `tests/ingress-capture.acceptance.mjs` proves the same refusal in the running app. All of them run in `npm test` or the browser gate, so a regression fails the release instead of a promise.
- Boundary and boundary-adjacent claims are reviewed by the clinical and privacy gate (#22, #23) before beta; this model does not move that gate, it feeds it.
+172
View File
@@ -0,0 +1,172 @@
# Production topology, secrets, backup, and rollback
This runbook answers one question for the operator: how do the parts of Timmy run, what may they
reach, what must never be lost, and how does each part come back? It covers the pieces that exist in
this tree today — the Node app, the private vision worker, the private staging directory, and the
consented corpus. It does not describe a training fleet, a cloud account, or a hosted database,
because this build has none.
Everything below is run by automation or by one operator on the host. Nothing here is performed by a
participant, and no step needs a public endpoint.
## Topology
| Process | Entry point | Holds | Restart unit |
|---|---|---|---|
| App and API | `node server.mjs` on `HOST=127.0.0.1 PORT=4173` | Nothing durable. Serves the PWA, proxies `/api/vision/*`, and gates `/api/agent/*` behind an access code. | `timmy-staging.service` (staging), `node server.mjs` (local) |
| Vision worker | `deploy/worker/run_worker_container.sh` | The pinned GGUF weights, mounted read-only from the host. | `scripts/run_selfhost_smolvlm.sh start`, container restart policy |
| Staging releases | `/opt/timmy-staging/releases/<40-character-commit>` with a `current` symlink | Immutable release trees plus the mode-600 environment file. | `timmy-staging.service` |
| Consented corpus | `/srv/timmy-corpus` | `manifest.jsonl` (append-only provenance and tombstones) and `images/` derived from consented contributions. | none — file data |
| Caddy | host Caddy | Terminates TLS and routes `/git/` to Gitea and `/timmy-staging/` to the app. | host Caddy |
The app is stateless for a reason: there is **no server-side copy of the journal**. A person's logs
and photos live in their own browser storage, and they hold the portable JSON backup. The server
holds no diary, so a lost server loses no one's health record — only the operator records listed
above can be lost, and only those are backed up here.
Health and readiness are separate, and the probe reports readiness only:
```
GET /api/healthz -> {"ok":true,"release":"<tag>","commit":"<commit>","visionEnabled":<bool>,"agentEnabled":<bool>}
```
`visionEnabled` and `agentEnabled` say whether the feature is configured, never which key or endpoint
is configured. The worker answers `/v1/models` before its pinned model is resident, so readiness
means "served a request from the loaded model", not "the port is open" — `scripts/worker_ready.sh`
enforces that distinction and the app withholds the AI control until it passes.
## Network boundaries
- The app reads `HOST` and its own default is `0.0.0.0`, which is a local-development convenience and
not the deployment. The deployment sets `HOST=127.0.0.1` in the mode-600 environment file
(`deploy/timmy-staging.env.example` is the template) and is reachable from outside solely through
the Caddy `/timmy-staging/` subpath. A bare `node server.mjs` on a shared host listens on every
interface — treat that as a laptop command, never as a production start.
- The Gitea `/git/` catch-all is declared before the staging route and must stay reachable across
every Caddy reload.
- The vision worker binds `127.0.0.1:8080` and is never published: `deploy/worker/run_worker_container.sh`
refuses a non-loopback publish host instead of warning about it, and no route in `server.mjs`
forwards arbitrary paths to the worker — the only vision endpoints are `/api/vision-status` and the
analysis request.
- Outbound from the host is limited to the pinned model download during bootstrap, which is refused
unless both SHA-256 receipts match. After that the worker needs no egress at all.
- Inference bodies and image bytes are never logged, and no reverse-proxy access log retains a
request body. A backup that contained them would be a second copy of someone's photo, so they are
excluded by the tool rather than by the operator remembering.
- No participant-facing endpoint requires an inbound port, a DNS record, or a cloud credential.
## Secrets
Four secrets exist, and every one of them is injected at runtime:
| Secret | Source | Scope |
|---|---|---|
| `TIMMY_DATASET_SALT` | mode-600 environment file on the host | Corpus subject keys and split assignment. Rotating it re-splits future ingests, so it is not rotated casually. |
| Vision provider key or worker token | mode-600 environment file on the host | Only the app process reads it; the browser never receives it. |
| Agent access code | mode-600 environment file on the host | Unlocks `/api/agent/unlock`; the cookie it mints is `HttpOnly` and same-origin. |
| Gitea access token | local operator shell or secret store | Issue and PR automation only. It is never written into a release tree, an archive, or a log. |
Rules:
- Never commit a secret, and never pass one on a command line that lands in shell history. Use the
environment file (`chmod 600`, owned by the service user) or the systemd `EnvironmentFile=`.
- Placeholders in this document are angle-bracketed or environment references, e.g.
`TIMMY_DATASET_SALT=<random 32+ characters, never committed>`. A literal value here is a bug.
- `scripts/build_release.py` scans the cloned tree for private-key blocks and assigned credentials
and fails the release if it finds one, so a secret that reaches a commit is caught before it
reaches a release.
- Rotation is: issue the replacement on the host, restart the unit, confirm `/api/healthz` and one
real inference request, then revoke the old value in the provider's own UI. Nothing in Git holds a
copy that would need rewriting.
- The backup tool refuses `.env`, `*.env`, `*.key`, `*.pem`, `id_rsa`, `tokens.json`, and
`credentials.json` by name, so a backup can be copied off-host without carrying the keys that
operate the host.
## Backup
The operator records that cannot be rebuilt are the corpus `manifest.jsonl` and the derived images
it points at. A tombstone is the only proof that a participant withdrew, so a backup that drops
tombstone lines silently re-authorises training on withdrawn data. The tool treats that as a
correctness bug, not a storage detail.
`scripts/operator_backup.py` archives **only** records, and writes one manifest naming every byte it
stored and every path it refused:
- Stored: `manifest.jsonl`, derived images, and any receipt files under the data root.
- Excluded, with the reason recorded: **model weights** (`model-weights` — re-downloadable, pinned by
hash), **transient inference** directories and uploads (`transient-inference` — a second copy of
someone's photo), **secret** files (`secret`), logs (`log`), symlinks (`symlink`), and anything over
`--max-bytes` (`oversize`).
- Every stored member carries a SHA-256, and the archive is reproducible: sorted members, zeroed
timestamps and ownership, `gzip` with a fixed mtime. Two backups of unchanged records are
byte-identical, so "did anything change?" is answerable by digest.
```bash
python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy-nightly.tar.gz --label nightly
python3 scripts/operator_backup.py verify --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json
```
The command writes `<archive>.manifest.json` unless `--manifest` names another path, and prints a
receipt with the stored file count, archive digest, and record counts (`samples`, `tombstones`,
`activeSamples`). It refuses a source that is not a directory and refuses to overwrite an existing
archive, so a nightly run cannot erase the previous night.
Retention: keep fourteen nightlies and the first backup of each month, and copy each archive plus its
manifest off-host. Also keep a human-readable record of the counts: `verify` compares the archive
against the manifest, and a restore compares the restored counts against both.
What this backup deliberately does not cover: the browser-held journal (there is no server copy; the
person's own portable export is the backup), model weights, and anything in the release tree, which
is rebuilt from a commit by `scripts/build_release.py`.
## Restore
A restore never writes over live data. It verifies first, and it writes into a directory that must
not exist or must be empty:
```bash
python3 scripts/operator_backup.py restore --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json --target /srv/timmy-corpus.restore-check
```
In order, before the first byte is written:
1. the manifest is loaded and its `schemaVersion` checked;
2. every archive member is refused if it is a link, a non-regular file, an absolute path, a traversal
path, or a name the manifest never listed;
3. every member's SHA-256 is recomputed and compared, and a mismatch stops the restore;
4. a member the manifest lists but the archive lacks stops the restore.
After the extract it recounts `manifest.jsonl` in the target and refuses if the counts differ from
the backup, so a restore that silently lost a tombstone fails loudly. Then the operator compares the
restored tree against the live one, and promotes it by moving the directories — the tool does not
pretend to know which directory is authoritative.
Restoring the person's own logs is a different operation and stays where it belongs: the app's
import control accepts the JSON file they exported, upgrades an older version and says which version
it read, and refuses a file from a newer Timmy without touching the ledger it would have replaced.
## Rollback
Two rollbacks exist, and only one of them is a live staging action.
**Release rollback (staging).** Staging keeps one immutable directory per commit and a `current`
symlink, so a rollback is a repoint, a restart, and a bounded health check that restores and verifies
the previous release if the new one does not come up:
```bash
python3 scripts/deploy_staging.py status
python3 scripts/deploy_staging.py rollback --commit <FULL_COMMIT>
```
The rollback path is exercised by `tests/staging-deploy.test.py`, which drives the real
`scripts/deploy_staging.py` code against a rootless fixture (`--root`), including the case where the
named commit has no immutable release directory, where the restart fails, and where the health probe
never passes. That is the same code path the host runs; the difference is only the root.
The live drill on the private staging host — promote a build, serve it, roll it back, and confirm the
participant-facing page again — is a **human gate**, tracked under #53. This loop does not run that
drill automatically, because a rolled-back staging host is being used for release testing at the time.
**Data rollback (operator records).** The equivalent for the corpus is the restore above: verify,
restore into a new directory, compare counts, then move it into place. A restore is never attempted in
place, and a failed verify leaves the live directory untouched.
+80
View File
@@ -0,0 +1,80 @@
# RC Security & Abuse Evidence (#40)
The release candidate is attacked before users get to it. This page is the procedure and the rule,
not a stored result: a pasted-in "all clear" is exactly the kind of evidence that stops being true
after the next commit.
## Run it
```bash
node scripts/security_attack_suite.mjs --json artifacts/security/rc-attack-report.json
```
Exit codes are part of the contract:
| Exit | Meaning |
|---:|---|
| `0` | `VERDICT: GO` — every case matched its expectation. |
| `1` | `VERDICT: BLOCKED` — at least one P0 case failed, or a required category has no executable evidence. |
| `2` | The suite could not complete (target never became ready, vision disabled, no log captured). |
The suite is a child process of nothing: it starts its own copy of `server.mjs` on a free loopback
port, points it at a **counting stub worker**, and drives it with synthetic hostile bytes. It never
contacts a real model provider, never uses a real medical image, and never touches a live user's data.
## What it does and does not prove
- It proves what this HTTP boundary did with the bytes it was handed, and whether the worker behind
it was called. The worker is a stub, so "the provider saw it" is a counted fact, not an inference.
- It does **not** measure model quality, clinical accuracy, or the safety of the medical/non-diagnostic
boundary. Those live in the vision, escalation, and clinical-review issues.
- Latency numbers describe the loopback boundary against a stub. They are not a model benchmark.
## Cases
| Case | Class | Severity | Expectation |
|---|---|---:|---|
| `ingress-oversized-body` | oversized | P0 | `413`, worker never called |
| `ingress-polyglot-html` | polyglot | P0 | `400`, worker never called |
| `ingress-garbage-bytes` | truncated | P0 | `400`, worker never called |
| `consent-required` | consent | P0 | `400`, worker never called |
| `model-output-malformed` | malformed | P0 | `503` neutral copy, provider text never echoed |
| `replay-duplicate-request` | replay | P2 | identical answer, nothing stored server-side |
| `flood-concurrency` | flood | P1 | burst meets a bound (`429`/overload), not an open pipe |
| `log-leak-scan` | log | P0 | captured server log carries no image bytes |
| `deletion-withdrawal` | deletion | P0 | out of surface — counted only when `tests/training-contribution.test.js` exists |
## The rule
`src/release-security-gate.js` owns the verdict, so it is testable rather than asserted:
- A case passes only when **every** expected field matches the observation. A partial match fails and
names the field that broke (`status 400 expected, observed 200`).
- **Any** failing P0 case blocks RC approval. P1/P2 failures are recorded as holds.
- A required class with no executable evidence is `unproven` and blocks the release. Coverage may be
borrowed from another suite (`out-of-surface` + an evidence path) only after the runner verified
that file exists — otherwise it blocks.
- The report is redacted before it is written, and assembly fails closed if any image bytes survive
redaction. Evidence that carries a photo is worse than no evidence.
## Current state (read the run, not this line)
This section used to predict **BLOCKED** until the ingress hardening in
[#16](https://forge.alexanderwhitestone.com/git/stackchain/timmy-talking-turd/issues/16) landed. With
that ingress in place the same command returns **GO** on this tree, because the format is decided from
the bytes rather than the data-URL label: `ingress-polyglot-html` and `ingress-garbage-bytes` are now
refused with `400` before the worker is called, and an oversized body is refused with a `413` the
caller can read instead of a dropped connection. Latency for the load phase is measured against a
second target with the request budget opened up, so those samples are analyses rather than `429`
refusals. Queue bounding is still [#17](https://forge.alexanderwhitestone.com/git/stackchain/timmy-talking-turd/issues/17)
and is held here as P1.
Deletion is governed by the contribution ledger, not by this HTTP boundary; the case stays
`covered-elsewhere` against `tests/training-contribution.test.js` and blocks if that file disappears.
## Evidence hygiene
- The report is written to `artifacts/security/` and must never be committed with a real photo in it.
- The captured server log is scanned by the same detector (`containsImageBytes`) before it is quoted.
- Error bodies are recorded as a 120-character snippet; the suite refuses to write a report that
still contains a data URI or a long base64 run.
+7 -3
View File
@@ -4,10 +4,11 @@
"private": true,
"type": "module",
"scripts": {
"test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/draft.test.js tests/suggestion.test.js tests/followup.test.js tests/vision-readiness.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js tests/threat-model.test.js tests/vision-corpus.test.js tests/vision-boundary.test.js tests/worker-benchmark.test.js tests/worker-ready.test.js tests/worker-container.test.js tests/summary.test.js tests/escalation-regression.test.js tests/grounded-chat.test.js tests/photo-lifecycle.test.js tests/training-contribution.test.js tests/portability.test.js tests/abstention-policy.test.js",
"test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/draft.test.js tests/suggestion.test.js tests/followup.test.js tests/vision-readiness.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js tests/threat-model.test.js tests/vision-corpus.test.js tests/vision-boundary.test.js tests/worker-benchmark.test.js tests/worker-ready.test.js tests/worker-container.test.js tests/summary.test.js tests/escalation-regression.test.js tests/grounded-chat.test.js tests/photo-lifecycle.test.js tests/training-contribution.test.js tests/portability.test.js tests/abstention-policy.test.js tests/release-security-gate.test.js tests/image-ingress.test.js tests/rate-limit.test.js tests/http-body.test.js tests/analyze-ingress.acceptance.test.js",
"test:ui": "node tests/ui.acceptance.mjs",
"test:photo": "node tests/photo-first.acceptance.mjs",
"test:mobile-capture": "node tests/mobile-capture.acceptance.mjs",
"test:ingress-capture": "node tests/ingress-capture.acceptance.mjs",
"test:sleek": "node tests/sleek-chat.acceptance.mjs",
"test:manual-fallback": "node tests/manual-fallback.acceptance.mjs",
"test:provisional": "node tests/provisional-suggestion.acceptance.mjs",
@@ -18,7 +19,7 @@
"test:grounded-chat": "node tests/grounded-chat.acceptance.mjs",
"test:photo-lifecycle": "node tests/photo-lifecycle.acceptance.mjs",
"test:training-contribution": "node tests/training-contribution.acceptance.mjs",
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/review_corpus.py scripts/dataset_card.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh",
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check src/release-security-gate.js && node --check scripts/security_attack_suite.mjs && node --check tests/security-attack.acceptance.mjs && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/review_corpus.py scripts/dataset_card.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py scripts/operator_backup.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh && node --check src/image-ingress.js && node --check src/rate-limit.js && node --check src/http-body.js && python3 -m py_compile scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py && node --check tests/ingress-capture.acceptance.mjs && node --check tests/analyze-ingress.acceptance.test.js",
"check:diff": "bash scripts/check_diff.sh",
"start": "node server.mjs",
"test:worker-readiness": "node tests/worker-readiness.acceptance.mjs",
@@ -27,7 +28,10 @@
"test:calibration": "python3 tests/calibration-harness.test.py",
"test:policy": "node --test tests/abstention-policy.test.js",
"test:review": "python3 tests/corpus-review.test.py",
"test:dataset-card": "python3 tests/dataset-card.test.py"
"test:dataset-card": "python3 tests/dataset-card.test.py",
"test:backup": "python3 tests/backup-restore.test.py",
"test:runbook": "python3 tests/production-runbook.test.py",
"test:security-attack": "node tests/security-attack.acceptance.mjs"
},
"devDependencies": {
"playwright": "^1.62.1"
+15 -2
View File
@@ -50,6 +50,7 @@ DEMO_REQUIRED_CAPTIONS = (
"Reachable is not loaded: no AI control, no consent, and no upload until the pinned model is resident",
"Every pattern line names the confirmed logs behind it, and following one shows exactly those records",
"Two logs is not a pattern: the summary refuses to describe a trend it cannot support",
"A file that only claims to be a photo is refused on the device — nothing is uploaded",
"The answer states the count and cites the four confirmed logs it counted",
"Following a citation opens exactly the confirmed log behind that answer",
"Ask something the journal cannot answer and Timmy refuses instead of reassuring",
@@ -73,6 +74,10 @@ DEMO_REQUIRED_CAPTIONS = (
"The model said Type 6 and the reviewer said Type 5: the disagreement is recorded, not smoothed over",
"Withdrawing the contribution empties the queue and the training set, and the tool refuses to review it at all",
"The dataset card is generated from the corpus manifest, so no count on it is typed by hand",
"The operator backup is a command line, not a screen in the app",
"Model weights and transient inference images are refused by name, and every skipped path is listed with its reason",
"Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes",
"A restore returns the same records, including the withdrawal tombstone a trainer must never lose",
"One contributor's photos stay in one split: a corpus that leaks a person across splits is refused",
"Devices, contexts, and licenses stay pending until the pipeline collects them",
"The card is a page, not a screen in the app — it is published with the real generator against synthetic fixtures",
@@ -183,6 +188,10 @@ def main() -> int:
# Issue #29's evaluation harness decides GO/BLOCKED for a model, so its refusal and gate
# arithmetic are exercised before this release is allowed to exist.
run(["python3", "tests/calibration-harness.test.py", "-v"], tree)
# Issue #39's operator backup is the only copy of the corpus tombstones, so the tool that
# produces it, and the runbook whose commands must stay real, are gated before the release.
run(["python3", "tests/backup-restore.test.py", "-v"], tree)
run(["python3", "tests/production-runbook.test.py", "-v"], tree)
server_env = dict(os.environ)
server_env.update({"HOST": "127.0.0.1", "PORT": "4173", "TIMMY_VISION_ENABLED": "false"})
# A leftover server from another checkout answers on this port and the readiness probe cannot
@@ -209,6 +218,7 @@ def main() -> int:
run(["npm", "run", "test:ui"], tree)
run(["npm", "run", "test:photo"], tree)
run(["npm", "run", "test:mobile-capture"], tree)
run(["npm", "run", "test:ingress-capture"], tree)
run(["npm", "run", "test:sleek"], tree)
run(["npm", "run", "test:manual-fallback"], tree)
run(["npm", "run", "test:provisional"], tree)
@@ -267,7 +277,7 @@ def main() -> int:
if not contact_sheet.exists() or contact_sheet.stat().st_size == 0:
raise SystemExit("Release demo contact sheet was not produced")
run(["npm", "audit", "--audit-level=high"], tree)
for file in ("app.js", "server.mjs", "service-worker.js", "src/analysis.js", "src/domain.js", "src/draft.js", "src/suggestion.js", "src/followup.js", "src/vision-readiness.js", "src/hermes-agent-service.js", "src/vision-config.js", "src/vision-service.js", "src/vision-corpus.js", "src/summary.js", "src/photo-lifecycle.js", "src/training-contribution.js", "scripts/record_worker_benchmark.mjs", "tests/staging.acceptance.mjs", "tests/vision-refusal.acceptance.mjs", "tests/worker-readiness.acceptance.mjs", "tests/pattern-summary.acceptance.mjs", "tests/photo-lifecycle.acceptance.mjs", "tests/training-contribution.acceptance.mjs"):
for file in ("app.js", "server.mjs", "service-worker.js", "src/analysis.js", "src/domain.js", "src/draft.js", "src/suggestion.js", "src/followup.js", "src/vision-readiness.js", "src/hermes-agent-service.js", "src/vision-config.js", "src/vision-service.js", "src/vision-corpus.js", "src/summary.js", "src/photo-lifecycle.js", "src/training-contribution.js", "scripts/record_worker_benchmark.mjs", "tests/staging.acceptance.mjs", "tests/vision-refusal.acceptance.mjs", "tests/worker-readiness.acceptance.mjs", "tests/pattern-summary.acceptance.mjs", "tests/photo-lifecycle.acceptance.mjs", "tests/training-contribution.acceptance.mjs", "src/image-ingress.js", "src/rate-limit.js", "src/http-body.js", "tests/ingress-capture.acceptance.mjs"):
run(["node", "--check", file], tree)
run(["bash", "-n", "scripts/bootstrap_selfhost_smolvlm.sh"], tree)
run(["bash", "-n", "scripts/run_selfhost_smolvlm.sh"], tree)
@@ -360,7 +370,7 @@ def main() -> int:
"codec": "h264",
"pixel_format": "yuv420p",
"recorded_from_working_app": True,
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything, plus the review ledger card, where the real review commands run against the committed synthetic sample and the card shows their actual output: one pending proposal with the model answer withheld, a refused reveal before commitment, a committed Type 5 correction beside the Type 4 proposal, the revealed synthetic Type 6 answer recorded as a three-field disagreement, a second reader's adjudication pointing at the review it resolved, one trainable sample, and a withdrawal that empties the queue and the training set and makes review refused",
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything, plus the review ledger card, where the real review commands run against the committed synthetic sample and the card shows their actual output: one pending proposal with the model answer withheld, a refused reveal before commitment, a committed Type 5 correction beside the Type 4 proposal, the revealed synthetic Type 6 answer recorded as a three-field disagreement, a second reader's adjudication pointing at the review it resolved, one trainable sample, and a withdrawal that empties the queue and the training set and makes review refused, plus the operator backup card, where the real backup, verify, and restore commands run against a synthetic corpus built by the real ingest and revoke commands and the card shows their actual output: two samples and one tombstone recorded, a weight shard, a transient inference upload, an operator env file and a log refused by name with the reason listed, the exclusion counts the manifest wrote, one digest for the archive, a verify that re-hashes every stored member, and a restore that returns the manifest byte-for-byte with the tombstone intact",
},
"gates": {
"unit_security": "passed",
@@ -381,7 +391,10 @@ def main() -> int:
"reviewer_adjudication_blind": "passed",
"dataset_card_generated": "passed",
"calibration_harness_gates": "passed",
"production_backup_restore": "passed",
"production_runbook_commands": "passed",
"portable_backup_restore": "passed",
"image_ingress_refusal": "passed",
"dependency_audit_high": "passed",
"syntax": "passed",
"secret_scan": "passed",
+326
View File
@@ -0,0 +1,326 @@
#!/usr/bin/env python3
"""Operator backup and restore for the records Timmy must not lose.
The app keeps the journal in the person's own browser, so the server side of Timmy holds no diary.
What the operator does hold, and what cannot be rebuilt from anywhere else, is the corpus: the
append-only ``manifest.jsonl`` with its provenance records and its withdrawal tombstones, plus the
derived images those records point at. Losing that file loses the proof that a participant
withdrew, which is the one receipt the project has promised to keep.
This tool therefore:
- archives only records, and names every path it refused to archive with the reason
(``model-weights``, ``transient-inference``, ``secret``, ``log``, ``symlink``, ``oversize``), so a
missing file is an auditable decision instead of a silent omission;
- writes one SHA-256 per archived member and a reproducible archive (sorted members, zeroed
timestamps and ownership), so two backups of the same records are byte-identical;
- verifies every digest, member name, and record count before a restore writes anything, and
refuses absolute, traversal, link, and unlisted members.
Usage:
python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy.tar.gz
python3 scripts/operator_backup.py verify --archive /srv/backups/timmy.tar.gz --manifest /srv/backups/timmy.tar.gz.manifest.json
python3 scripts/operator_backup.py restore --archive ... --manifest ... --target /srv/timmy-corpus.restored
"""
from __future__ import annotations
import argparse
import gzip
import hashlib
import io
import json
import os
import sys
import tarfile
from datetime import datetime, timezone
from pathlib import Path, PurePosixPath
SCHEMA_VERSION = 1
TOOL = "operator_backup.py"
PAYLOAD_PREFIX = "payload/"
DEFAULT_MAX_BYTES = 64 * 1024 * 1024
WEIGHT_SUFFIXES = (".gguf", ".safetensors", ".onnx", ".bin", ".pt", ".ckpt", ".pth")
WEIGHT_DIRS = ("weights", "models")
TRANSIENT_DIRS = ("transient", "inference-tmp", "inference-tmp-images", "uploads", "tmp", "cache")
LOG_DIRS = ("logs",)
LOG_SUFFIXES = (".log",)
SECRET_NAMES = (".env", "id_rsa", "id_ed25519", "id_ecdsa", "tokens.json", "credentials.json")
SECRET_SUFFIXES = (".key", ".pem", ".p12", ".pfx", ".env")
class BackupError(Exception):
"""A refusal that is reported on stderr and exits non-zero."""
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def classify(relative: PurePosixPath) -> str | None:
"""Return the reason a path is not archived, or ``None`` when it is a record."""
parts = relative.parts[:-1]
name = relative.name
lowered = [part.lower() for part in parts]
suffix = PurePosixPath(name.lower()).suffix
if name.lower().startswith(".env") or name.lower() in SECRET_NAMES or suffix in SECRET_SUFFIXES:
return "secret"
if suffix in WEIGHT_SUFFIXES or any(part in WEIGHT_DIRS for part in lowered):
return "model-weights"
if any(part in TRANSIENT_DIRS for part in lowered):
return "transient-inference"
if any(part in LOG_DIRS for part in lowered) or suffix in LOG_SUFFIXES:
return "log"
return None
def record_counts(manifest_path: Path) -> dict[str, int]:
"""Count the append-only manifest the way a trainer would read it."""
counts = {"samples": 0, "tombstones": 0, "activeSamples": 0}
if not manifest_path.is_file():
return counts
revoked: set[str] = set()
samples: list[dict] = []
for line in manifest_path.read_text(encoding="utf-8").splitlines():
if not line.strip():
continue
try:
entry = json.loads(line)
except json.JSONDecodeError as error:
raise BackupError(f"manifest {manifest_path} is not valid JSON Lines: {error}") from error
kind = entry.get("recordType")
if kind == "sample":
samples.append(entry)
counts["samples"] += 1
elif kind == "tombstone":
counts["tombstones"] += 1
if entry.get("contributionId"):
revoked.add(str(entry["contributionId"]))
counts["activeSamples"] = sum(
1 for entry in samples
if str(entry.get("contributionId")) not in revoked
and str((entry.get("lifecycle") or {}).get("state")) == "active"
)
return counts
def collect(source: Path, max_bytes: int) -> tuple[list[tuple[PurePosixPath, Path]], list[dict]]:
included: list[tuple[PurePosixPath, Path]] = []
excluded: list[dict] = []
for dirpath, dirnames, filenames in os.walk(source, followlinks=False):
dirnames.sort()
base = Path(dirpath)
for name in sorted(filenames):
path = base / name
relative = PurePosixPath(path.relative_to(source).as_posix())
if path.is_symlink():
excluded.append({"path": str(relative), "reason": "symlink", "bytes": 0})
continue
reason = classify(relative)
if reason:
excluded.append({"path": str(relative), "reason": reason, "bytes": _size(path)})
continue
size = _size(path)
if size > max_bytes:
excluded.append({"path": str(relative), "reason": "oversize", "bytes": size})
continue
included.append((relative, path))
included.sort(key=lambda item: str(item[0]))
excluded.sort(key=lambda entry: entry["path"])
return included, excluded
def _size(path: Path) -> int:
try:
return path.stat().st_size
except OSError:
return 0
def backup(*, source: Path, archive: Path, manifest_path: Path, label: str, max_bytes: int) -> dict:
if not source.is_dir():
raise BackupError(f"source {source} is not a directory")
if archive.exists():
raise BackupError(f"archive {archive} already exists; move it aside instead of overwriting a backup")
included, excluded = collect(source, max_bytes)
files = [{"path": str(relative), "sha256": sha256_file(path), "bytes": _size(path)} for relative, path in included]
counts = record_counts(source / "manifest.jsonl")
archive.parent.mkdir(parents=True, exist_ok=True)
partial = archive.with_name(archive.name + ".partial")
with partial.open("wb") as raw:
# gzip mtime=0 and zeroed tar metadata keep two backups of the same records identical.
with gzip.GzipFile(fileobj=raw, mode="wb", mtime=0) as compressed:
with tarfile.open(fileobj=compressed, mode="w", format=tarfile.PAX_FORMAT) as tar:
for (relative, path), entry in zip(included, files):
payload = path.read_bytes()
info = tarfile.TarInfo(PAYLOAD_PREFIX + str(relative))
info.size = len(payload)
info.mode = 0o644
info.mtime = 0
info.uid = info.gid = 0
info.uname = info.gname = ""
tar.addfile(info, io.BytesIO(payload))
if hashlib.sha256(payload).hexdigest() != entry["sha256"]:
raise BackupError(f"{relative} changed while it was being archived")
os.replace(partial, archive)
manifest = {
"schemaVersion": SCHEMA_VERSION,
"tool": TOOL,
"createdAt": datetime.now(timezone.utc).isoformat(timespec="seconds"),
"label": label,
"sourceLabel": source.name,
"archive": {"name": archive.name, "sha256": sha256_file(archive), "bytes": _size(archive)},
"records": counts,
"files": files,
"excluded": excluded,
}
manifest_path.parent.mkdir(parents=True, exist_ok=True)
manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8")
return {
"archive": str(archive),
"manifest": str(manifest_path),
"files": len(files),
"bytes": manifest["archive"]["bytes"],
"records": counts,
"excluded": {entry["reason"]: sum(1 for other in excluded if other["reason"] == entry["reason"]) for entry in excluded},
}
def load_manifest(manifest_path: Path) -> dict:
if not manifest_path.is_file():
raise BackupError(f"manifest {manifest_path} does not exist")
try:
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
except json.JSONDecodeError as error:
raise BackupError(f"manifest {manifest_path} is not valid JSON: {error}") from error
if manifest.get("schemaVersion") != SCHEMA_VERSION:
raise BackupError(f"manifest schemaVersion {manifest.get('schemaVersion')!r} is not {SCHEMA_VERSION}")
return manifest
def verified_members(archive: Path, manifest: dict) -> list[tuple[str, bytes]]:
"""Check every member of the archive before anything is written to disk."""
if not archive.is_file():
raise BackupError(f"archive {archive} does not exist")
expected = {entry["path"]: entry for entry in manifest.get("files") or []}
extracted: list[tuple[str, bytes]] = []
seen: set[str] = set()
with tarfile.open(archive, "r:gz") as tar:
for info in tar.getmembers():
if info.isdir():
continue
if info.issym() or info.islnk():
raise BackupError(f"archive member {info.name!r} is a link, not a regular file")
if not info.isfile():
raise BackupError(f"archive member {info.name!r} is not a regular file")
if os.path.isabs(info.name):
raise BackupError(f"archive member {info.name!r} is an absolute path")
if not info.name.startswith(PAYLOAD_PREFIX):
raise BackupError(f"archive member {info.name!r} is not under {PAYLOAD_PREFIX}")
relative = info.name[len(PAYLOAD_PREFIX):]
parts = PurePosixPath(relative).parts
if not parts or any(part in ("..", ".", "") for part in parts):
raise BackupError(f"archive member {info.name!r} resolves outside the restore root")
if relative not in expected:
raise BackupError(f"archive member {info.name!r} is not listed in the manifest")
handle = tar.extractfile(info)
if handle is None:
raise BackupError(f"archive member {info.name!r} cannot be read")
payload = handle.read()
if hashlib.sha256(payload).hexdigest() != expected[relative]["sha256"]:
raise BackupError(
f"archive member {info.name!r} does not match the manifest digest "
"(the archive changed after it was recorded)"
)
seen.add(relative)
extracted.append((relative, payload))
missing = sorted(set(expected) - seen)
if missing:
raise BackupError(f"archive is missing members the manifest lists: {', '.join(missing)}")
return extracted
def verify(*, archive: Path, manifest_path: Path) -> dict:
manifest = load_manifest(manifest_path)
members_list = verified_members(archive, manifest)
return {"archive": str(archive), "files": len(members_list), "records": manifest.get("records", {})}
def restore(*, archive: Path, manifest_path: Path, target: Path) -> dict:
manifest = load_manifest(manifest_path)
if target.exists():
if not target.is_dir():
raise BackupError(f"restore target {target} is not a directory")
if any(target.iterdir()):
raise BackupError(f"restore target {target} is not empty; restore into a new directory")
# Digests and member names are checked before the first byte is written, so a hostile or
# tampered archive cannot leave a half-restored record set behind.
members_list = verified_members(archive, manifest)
target.mkdir(parents=True, exist_ok=True)
for relative, payload in members_list:
destination = target / relative
destination.parent.mkdir(parents=True, exist_ok=True)
destination.write_bytes(payload)
os.chmod(destination, 0o644)
counts = record_counts(target / "manifest.jsonl")
if manifest.get("records") and counts != manifest["records"]:
raise BackupError(
f"restored record counts {counts} do not match the backup manifest {manifest['records']}"
)
return {"target": str(target), "files": len(members_list), "records": counts}
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
subparsers = parser.add_subparsers(dest="command", required=True)
backup_parser = subparsers.add_parser("backup", help="archive the records and write a manifest")
backup_parser.add_argument("--source", required=True, type=Path)
backup_parser.add_argument("--archive", required=True, type=Path)
backup_parser.add_argument("--manifest", type=Path)
backup_parser.add_argument("--label", default="manual")
backup_parser.add_argument("--max-bytes", type=int, default=DEFAULT_MAX_BYTES)
verify_parser = subparsers.add_parser("verify", help="recompute every digest in an archive")
verify_parser.add_argument("--archive", required=True, type=Path)
verify_parser.add_argument("--manifest", required=True, type=Path)
restore_parser = subparsers.add_parser("restore", help="verify, then write the records into a new directory")
restore_parser.add_argument("--archive", required=True, type=Path)
restore_parser.add_argument("--manifest", required=True, type=Path)
restore_parser.add_argument("--target", required=True, type=Path)
return parser
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
if args.command == "backup":
manifest_path = args.manifest or args.archive.with_name(args.archive.name + ".manifest.json")
receipt = backup(
source=args.source, archive=args.archive, manifest_path=manifest_path,
label=args.label, max_bytes=args.max_bytes,
)
elif args.command == "verify":
receipt = verify(archive=args.archive, manifest_path=args.manifest)
else:
receipt = restore(archive=args.archive, manifest_path=args.manifest, target=args.target)
except BackupError as error:
print(f"refused: {error}", file=sys.stderr)
return 2
print(json.dumps(receipt, indent=2, sort_keys=True))
return 0
if __name__ == "__main__":
sys.exit(main())
+234 -5
View File
@@ -17,6 +17,10 @@ const version = process.env.TIMMY_RELEASE_VERSION || 'review build';
// changed in this release instead of implying a feature that did not ship.
const releaseDelta = process.env.TIMMY_RELEASE_DEMO_DELTA
|| 'Verified rebuild of the current user-visible feature path — no new feature is claimed.';
// The recording targets the release builder's own acceptance server by default. A caller that
// names another origin is explicit about it, so a local run can never silently record a different
// tree than the one the gates measured.
const demoUrl = process.env.TIMMY_DEMO_URL || 'http://127.0.0.1:4173';
const captionsOut = process.env.TIMMY_DEMO_CAPTIONS_OUT || '';
const captionsShown = [];
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
@@ -86,7 +90,9 @@ const suggestionBody = {
warning: 'Visual suggestion only. Confirm it yourself; this is not a diagnosis.',
};
let analyzeOverride = null;
let analyzeCalls = 0;
await page.route('**/api/analyze', route => {
analyzeCalls += 1;
const response = analyzeOverride || { status: 200, body: suggestionBody };
return route.fulfill({ status: response.status, contentType: 'application/json', body: JSON.stringify(response.body) });
});
@@ -116,7 +122,7 @@ await page.route('**/api/agent/chat', route => {
});
});
await page.goto('http://127.0.0.1:4173', { waitUntil: 'networkidle' });
await page.goto(demoUrl, { waitUntil: 'networkidle' });
await page.evaluate(() => localStorage.clear());
await page.reload({ waitUntil: 'networkidle' });
@@ -136,9 +142,13 @@ async function caption(text, duration = 1400) {
node.textContent = text;
// Inline styles, not a stylesheet: a stylesheet belongs to one document and the demo reloads the
// page, which silently stripped caption styling and left every later caption clipped body text.
// The bar is opaque, not 94%: a translucent bar let the app's own text show through a long
// release-delta caption and read as run-together garbage. Long copy also steps down a size so a
// paragraph-length caption keeps its bottom padding instead of pressing the last line to the edge.
const size = text.length > 380 ? 9.5 : text.length > 220 ? 10.5 : 12;
node.style.cssText = 'position:fixed;left:16px;right:16px;top:14px;z-index:20000;'
+ 'background:rgba(40,33,29,.94);color:#fff;padding:11px 14px;border-radius:14px;'
+ 'font:800 12px/1.25 system-ui;letter-spacing:.02em;text-align:center;'
+ 'background:#28211d;color:#fff;padding:12px 14px 14px;border-radius:14px;'
+ `font:800 ${size}px/1.32 system-ui;letter-spacing:.02em;text-align:center;`
+ 'box-shadow:0 8px 24px rgba(0,0,0,.2)';
document.body.append(node);
node.animate([{ opacity: 0, transform: 'translateY(-8px)' }, { opacity: 1, transform: 'translateY(0)' }], { duration: 260, fill: 'forwards' });
@@ -152,18 +162,27 @@ async function caption(text, duration = 1400) {
const rect = node.getBoundingClientRect();
const style = getComputedStyle(node);
const topmost = document.elementFromPoint(Math.round(rect.left + rect.width / 2), Math.round(rect.top + rect.height / 2));
// Proving the string was pushed is not proving it was readable. Measure the last line's own box,
// so a paragraph caption pressed flat against the bar's bottom edge fails here instead of shipping.
const range = document.createRange();
range.selectNodeContents(node);
const lines = Array.from(range.getClientRects()).filter(box => box.height > 1);
const lastBottom = lines.length ? Math.max(...lines.map(box => box.bottom)) : rect.bottom;
const padBottom = Math.round(rect.bottom - lastBottom);
return {
ok: style.position === 'fixed' && style.zIndex === '20000' && rect.top <= 40 && rect.height >= 24
&& rect.width > 200 && rect.bottom <= window.innerHeight && (topmost === node || (topmost && node.contains(topmost))),
&& rect.width > 200 && rect.bottom <= window.innerHeight && (topmost === node || (topmost && node.contains(topmost)))
&& lines.length > 0 && padBottom >= 3,
top: Math.round(rect.top), height: Math.round(rect.height), position: style.position,
zIndex: style.zIndex, background: style.backgroundColor, topmost: topmost === node,
padBottom, lines: lines.length,
};
});
const shownBand = await captionBand().catch(() => null);
const bandChanged = Boolean(blankBand && shownBand && !blankBand.equals(shownBand));
if (!painted.ok) throw new Error(`Caption is not an on-screen bar: ${text} ${JSON.stringify(painted)}`);
if (!bandChanged) throw new Error(`Caption painted nothing into the caption band: ${text}`);
captionEvidence.push({ text, visible: true, position: painted.position, top: painted.top, height: painted.height, bandChanged });
captionEvidence.push({ text, visible: true, position: painted.position, top: painted.top, height: painted.height, bandChanged, padBottom: painted.padBottom, lines: painted.lines });
await sleep(duration);
// A caption must not outlive its stated duration. It used to stay on screen until the next one
// replaced it, so a view change left the previous screen's claim hanging over a different screen —
@@ -293,6 +312,24 @@ await indicate('label[for="camera-photo"]');
await page.locator('#camera-photo').dispatchEvent('cancel');
await page.getByText(/Camera or photo picker closed/i).waitFor();
await caption('Camera closed cleanly — gallery and manual logging are still available', 1600);
// Issue #16, on camera: the file below is committed as text and only claims to be a photo. It must
// be refused on the device, the refusal must say nothing was uploaded, and the app must never post
// it — asserted here, not narrated.
await indicate('label[for="gallery-photo"]');
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/not-a-photo.jpg');
await page.locator('[data-capture-error="unreadable"]').waitFor();
const unreadableCopy = await page.locator('[data-capture-error="unreadable"]').innerText();
if (!/isn’t a photo Timmy can read/.test(unreadableCopy) || !/Nothing was uploaded/.test(unreadableCopy)) {
throw new Error(`An unreadable file must be refused with the upload-free copy, saw: ${unreadableCopy}`);
}
if (await page.getByText(/couldn’t analyze that safely/i).count() !== 0) {
throw new Error('A file that never left the device must not be reported as a failed analysis');
}
if (analyzeCalls !== 0) throw new Error('An unreadable file was posted to /api/analyze');
await caption('A file that only claims to be a photo is refused on the device — nothing is uploaded', 2600);
await indicate('[data-capture-error="unreadable"]');
await caption('Manual logging stays one tap away, and the photo picker is one tap back', 2000);
await tap('#retake-photo', 450);
await indicate('label[for="gallery-photo"]');
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/synthetic-type4.jpg');
await caption('Nothing uploads until explicit consent', 1100);
@@ -1148,6 +1185,198 @@ await caption('Each candidate is scored on test at a threshold chosen on validat
await caption('One contributor spread across two splits is refused before anything is scored, and a passing verdict is proven only by a fixture engineered to clear it', 3800);
await page.evaluate(() => document.querySelector('#calibration-report')?.remove());
// Issue #40 — the RC attack suite is a command line too, and unlike the other cards this one is
// expected to come back with a verdict that is not flattering. Hostile synthetic bytes go at the
// real server with a counting stub worker behind it, so "the worker was called" is measured, not
// inferred. The card prints whatever verdict the suite derived from what it observed; the captions
// name where a failing fix lives instead of implying the boundary is already safe.
async function securityAttackSuite() {
const directory = await mkdtemp(path.join(os.tmpdir(), 'timmy-security-demo-'));
const reportPath = path.join(directory, 'rc-attack-report.json');
try {
const run = await execFileAsync(
process.execPath,
['scripts/security_attack_suite.mjs', '--json', reportPath],
{ cwd: process.cwd() },
).then(result => ({ code: 0, stdout: result.stdout, stderr: result.stderr }))
.catch(error => ({ code: typeof error.code === 'number' ? error.code : 1, stdout: error.stdout || '', stderr: error.stderr || '' }));
if (![0, 1].includes(run.code)) throw new Error(`the attack suite could not complete: ${run.stderr.trim()}`);
const report = JSON.parse(await readFile(reportPath, 'utf8'));
if (run.code !== (report.verdict === 'BLOCKED' ? 1 : 0)) {
throw new Error('the attack suite exit code must match the verdict it published');
}
return { report, run };
} finally {
await rm(directory, { recursive: true, force: true });
}
}
const securityRun = await securityAttackSuite();
const securityReport = securityRun.report;
if (securityReport.cases.length < 8) throw new Error('the RC attack suite must run every named abuse class');
await tap('[data-view="home"]', 450);
const caseDetail = (item) => {
const observed = item.observed === null || typeof item.observed !== 'object' ? {} : item.observed;
// Print each observed field the case actually recorded. A case with no `status` used to render
// `http undefined` on camera, and one with no `providerCalls` rendered its status twice — both read
// as a broken demo instead of the measurement it is.
const parts = [];
if ('status' in observed) parts.push(`http ${observed.status}`);
if ('refused' in observed) parts.push(`refused ${observed.refused}`);
if ('providerCalls' in observed) parts.push(`worker ${observed.providerCalls}`);
if ('providerTextLeaked' in observed) parts.push(`text leaked ${observed.providerTextLeaked}`);
if ('repeatedAnswerIdentical' in observed) parts.push(`identical ${observed.repeatedAnswerIdentical}`);
if ('imageBytesInLog' in observed) parts.push(`image bytes ${observed.imageBytesInLog}`);
return parts.length ? parts.join(' · ') : item.status;
};
const securityRows = [
['command', 'node scripts/security_attack_suite.mjs'],
['target', 'real server · counting stub worker · synthetic bytes'],
['—', ''],
];
for (const item of securityReport.cases) securityRows.push([item.category, `${item.severity} ${item.status} · ${caseDetail(item)}`]);
securityRows.push(['—', '']);
securityRows.push(['load', `${securityReport.load.requests} req · p50 ${securityReport.load.p50}ms · p95 ${securityReport.load.p95}ms · abstain ${securityReport.load.abstentions}`]);
securityRows.push(['verdict', `${securityReport.verdict} · ${securityReport.blocking.length} blocking · ${securityReport.holds.length} held`]);
for (const item of securityReport.blocking) securityRows.push(['blocks', item.id]);
const securityCard = await evidenceCard({
id: 'security-attack',
title: 'SECURITY ATTACK SUITE',
subtitle: 'command line, not app UI — real server, counted worker calls',
rows: securityRows,
font: '600 9.5px/1.45 ui-monospace, SFMono-Regular, Menlo, monospace',
keyWidth: 62,
gap: '1px',
rowGap: '5px',
});
await sleep(420);
const securityCardProof = await evidenceCardProof('security-attack');
if (!securityCardProof.ok) throw new Error(`Security card is not fully readable on screen: ${JSON.stringify(securityCardProof)}`);
for (const item of securityReport.blocking) {
if (!securityCard.text.includes(item.id)) throw new Error(`the card must name the blocking case ${item.id}`);
}
if (!securityCard.text.includes(securityReport.verdict)) throw new Error('the card must show the verdict the suite derived');
if (!securityCard.text.includes(`${securityReport.load.p95}ms`)) throw new Error('the card must show the p95 the load phase measured');
await caption('Every hostile upload on screen went at the real server, and the worker behind it counted every call it received', 3800);
await caption('Refusing bytes by their content is the fix in review, not a claim this video makes: an HTML file dressed as a JPEG still reaches the worker today', 4200);
await caption('A failing case is a finding the release job has to see, so the card prints the verdict the suite derived instead of a reassuring one', 3800);
await caption('The gate blocks RC approval on any P0 case that reached the worker — the gate decides, not the caption', 3400);
await page.evaluate(() => document.querySelector('#security-attack')?.remove());
// Issue #39 — the operator backup is a command line, not a screen in the app. The app itself keeps
// no server-side journal, so what an operator can actually lose is the corpus: the append-only
// manifest with its withdrawal tombstones. This scene builds a synthetic data root with the real
// ingest and revoke commands, plants the four things a backup must refuse, then runs the real
// backup, verify, and restore and reads every value on the card back out of those runs.
async function operatorBackupAudit() {
const root = await mkdtemp(path.join(os.tmpdir(), 'timmy-backup-demo-'));
const dataRoot = path.join(root, 'srv-timmy-corpus');
const archive = path.join(root, 'timmy-nightly.tar.gz');
const manifestPath = `${archive}.manifest.json`;
const restoreInto = path.join(root, 'restored');
const secondPhoto = path.join(root, 'synthetic-second.png');
const backupCli = args => execFileAsync('python3', ['scripts/operator_backup.py', ...args], { cwd: process.cwd() });
const ingest = (input, contributionId, subject, bristol) => corpusCli([
'scripts/ingest_training_photo.py',
'--input', input, '--output-dir', dataRoot, '--subject-id', subject,
'--consent-version', 'training-consent-v1', '--contribution-id', contributionId,
'--bristol-type', bristol, '--color', 'brown', '--quality', 'good', '--reviewer', 'user-confirmed',
]);
try {
// One contribution stays active and one is withdrawn, so the manifest ends up with a live sample
// and a real tombstone, and the withdrawn derived image is already gone from disk.
await ingest('tests/fixtures/synthetic-type4.jpg', 'demo-backup-0001', 'demo-backup-kept', '4');
await writeFile(secondPhoto, Buffer.from('iVBORw0KGgoAAAANSUhEUgAAAAgAAAAICAIAAABLbSncAAAAFElEQVR4nGOsCDFhwAaYsIoOWgkAzUwBECP1Kh0AAAAASUVORK5CYII=', 'base64'));
await ingest(secondPhoto, 'demo-backup-0002', 'demo-backup-withdrawn', '3');
await corpusCli([
'scripts/ingest_training_photo.py', '--output-dir', dataRoot,
'--revoke', 'demo-backup-0002', '--reason', 'participant withdrew consent',
]);
// The four things that must never be archived, as real files with real bytes: a weight shard, a
// transient inference upload, an operator environment file, and a log. An exclusion that quietly
// stopped working fails here instead of in a caption.
await mkdir(path.join(dataRoot, 'weights'), { recursive: true });
await writeFile(path.join(dataRoot, 'weights', 'model.gguf'), 'synthetic weight shard — never archived');
await mkdir(path.join(dataRoot, 'transient'), { recursive: true });
await writeFile(path.join(dataRoot, 'transient', 'inference-upload.jpg'), 'transient inference bytes — never archived');
await writeFile(path.join(dataRoot, 'secrets.env'), 'TIMMY_DATASET_SALT=demo-only-placeholder\n');
await writeFile(path.join(dataRoot, 'operator.log'), 'GET /api/healthz 200\n');
const backupRun = await backupCli(['backup', '--source', dataRoot, '--archive', archive, '--label', 'nightly']);
const receipt = JSON.parse(backupRun.stdout);
const manifest = JSON.parse(await readFile(manifestPath, 'utf8'));
const reasons = manifest.excluded.reduce((out, entry) => ({ ...out, [entry.reason]: (out[entry.reason] || 0) + 1 }), {});
if (manifest.files.some(entry => /model\.gguf|inference-upload|secrets\.env|operator\.log/.test(entry.path))) {
throw new Error('the backup must not archive the weight shard or a transient inference upload');
}
for (const reason of ['model-weights', 'transient-inference', 'secret', 'log']) {
if (!reasons[reason]) throw new Error(`the backup must record a ${reason} exclusion`);
}
if (receipt.records.samples !== 2 || receipt.records.tombstones !== 1 || receipt.records.activeSamples !== 1) {
throw new Error('the backup receipt must report the real record counts');
}
const verified = JSON.parse((await backupCli(['verify', '--archive', archive, '--manifest', manifestPath])).stdout);
if (verified.files !== manifest.files.length) throw new Error('verify must re-hash every stored member');
const restored = JSON.parse((await backupCli([
'restore', '--archive', archive, '--manifest', manifestPath, '--target', restoreInto,
])).stdout);
if (restored.records.tombstones !== 1 || restored.records.activeSamples !== 1) {
throw new Error('the restore must return the same record counts');
}
const restoredManifest = await readFile(path.join(restoreInto, 'manifest.jsonl'));
if (!restoredManifest.equals(await readFile(path.join(dataRoot, 'manifest.jsonl')))) {
throw new Error('the restore must return the same records, tombstone included');
}
return { receipt, manifest, reasons, verified, restored };
} finally {
await rm(root, { recursive: true, force: true });
}
}
const opBackup = await operatorBackupAudit();
await tap('[data-view="home"]', 400);
const shortDigest = value => String(value).slice(0, 12);
const backupRows = [
['command', 'python3 scripts/operator_backup.py backup --source srv-timmy-corpus'],
['stored', `${opBackup.receipt.files} members · ${opBackup.manifest.files.map(entry => entry.path.split('/').pop()).join(' + ')}`],
['records', `samples ${opBackup.receipt.records.samples} · tombstones ${opBackup.receipt.records.tombstones} · active ${opBackup.receipt.records.activeSamples}`],
['—', ''],
['refused', `model-weights ${opBackup.reasons['model-weights']} · transient-inference ${opBackup.reasons['transient-inference']}`],
['refused', `secret ${opBackup.reasons['secret']} · log ${opBackup.reasons['log']}`],
['reason', 'every skipped path is named in the manifest, not silently dropped'],
['—', ''],
['digest', `sha256 ${shortDigest(opBackup.manifest.archive.sha256)}… · reproducible (sorted, mtime 0)`],
['verify', `ok · ${opBackup.verified.files} members re-hashed against the manifest`],
['restore', `ok · samples ${opBackup.restored.records.samples} · tombstones ${opBackup.restored.records.tombstones} · active ${opBackup.restored.records.activeSamples}`],
['manifest', 'returned byte-for-byte, tombstone included'],
['server', 'the app holds no journal of its own — there is no server copy to back up'],
];
const backupCard = await evidenceCard({
id: 'production-backup',
title: 'OPERATOR BACKUP',
subtitle: 'command line, not app UI — real run, synthetic corpus',
rows: backupRows,
font: '600 9.5px/1.45 ui-monospace, SFMono-Regular, Menlo, monospace',
keyWidth: 66,
gap: '1px',
rowGap: '6px',
});
await sleep(420);
const backupProof = await evidenceCardProof('production-backup');
if (!backupProof.ok) throw new Error(`Production backup card is not fully readable on screen: ${JSON.stringify(backupProof)}`);
// The card is drawn from the run: prove the counts and the refusals on it are the ones it produced.
if (!backupCard.text.includes(`samples ${opBackup.receipt.records.samples} · tombstones ${opBackup.receipt.records.tombstones}`)) {
throw new Error('the card must show the record counts the backup actually reported');
}
if (!backupCard.text.includes('model-weights 1')) throw new Error('the card must show the exclusion the run actually recorded');
if (!backupCard.text.includes(shortDigest(opBackup.manifest.archive.sha256))) {
throw new Error('the card must show the archive digest the run actually wrote');
}
await caption('The operator backup is a command line, not a screen in the app — this runs it against a synthetic corpus the real ingest and revoke commands built', 3800);
await caption('Model weights and transient inference images are refused by name, and every skipped path is listed with its reason', 3600);
await caption('Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes', 3400);
await caption('A restore returns the same records, including the withdrawal tombstone a trainer must never lose', 3600);
await page.evaluate(() => document.querySelector('#production-backup')?.remove());
// Back to the privacy view for the portability scene: the privacy entry lives on the journal view.
await tap('[data-view="calendar"]', 550);
await tap('[data-view="privacy"]', 600);
+332
View File
@@ -0,0 +1,332 @@
#!/usr/bin/env node
// RC attack suite (#40): attack the release candidate before users do.
//
// Every case drives the shipping `server.mjs` with synthetic hostile bytes while a stub worker counts
// the calls that reach it. The suite records what it observed and derives a verdict from the
// observation; it never asserts a result it did not see. No real medical image, no live user data,
// no third-party provider is involved, and the provider is a counting stub, so "reached the worker"
// is provable rather than inferred.
//
// Usage: node scripts/security_attack_suite.mjs [--json <path>] [--base-url <url>]
// Exit: 0 = GO, 1 = BLOCKED (a P0 finding, or a required category with no executable evidence).
import { randomBytes } from 'node:crypto';
import { spawn } from 'node:child_process';
import { access, mkdir, readFile, writeFile } from 'node:fs/promises';
import { dirname, join } from 'node:path';
import http from 'node:http';
import { fileURLToPath } from 'node:url';
import { buildReport, containsImageBytes, latencyPercentiles } from '../src/release-security-gate.js';
const repoRoot = fileURLToPath(new URL('..', import.meta.url));
const argument = (name, fallback) => {
const index = process.argv.indexOf(name);
return index === -1 ? fallback : process.argv[index + 1];
};
const reportPath = argument('--json', join(repoRoot, 'artifacts/security/rc-attack-report.json'));
const explicitBaseUrl = argument('--base-url', '');
function freePort() {
return new Promise(resolve => {
const probe = http.createServer();
probe.listen(0, '127.0.0.1', () => {
const { port } = probe.address();
probe.close(() => resolve(port));
});
});
}
// A counting OpenAI-compatible worker. `state.content` is the answer the next request receives, and
// `state.requests` is the evidence that a request crossed the provider boundary at all.
function startStubWorker() {
const state = { content: '', requests: 0 };
const server = http.createServer((req, res) => {
if (req.url.endsWith('/models')) {
res.writeHead(200, { 'content-type': 'application/json' });
return res.end(JSON.stringify({ data: [{ id: 'stub-vision-worker' }] }));
}
req.resume();
req.on('end', () => {
state.requests += 1;
res.writeHead(200, { 'content-type': 'application/json' });
res.end(JSON.stringify({ choices: [{ message: { content: state.content } }] }));
});
});
return new Promise(resolve => server.listen(0, '127.0.0.1', () => resolve({
state,
port: server.address().port,
close: () => new Promise(done => server.close(done)),
})));
}
// The real server, with its own stdout/stderr captured from the first byte: the log scan is only
// worth something if it read a log that actually exists.
async function startTarget({ providerPort, port, env = {} }) {
const child = spawn(process.execPath, ['server.mjs'], {
cwd: repoRoot,
env: {
...process.env,
HOST: '127.0.0.1',
PORT: String(port),
TIMMY_VISION_ENABLED: 'true',
TIMMY_VISION_PROFILE: 'selfhost',
TIMMY_VISION_BASE_URL: `http://127.0.0.1:${providerPort}/v1`,
TIMMY_VISION_MODEL: 'stub-vision-worker',
TIMMY_VISION_TIMEOUT_MS: '5000',
...env,
},
stdio: ['ignore', 'pipe', 'pipe'],
});
const log = [];
child.stdout.on('data', chunk => log.push(chunk.toString('utf8')));
child.stderr.on('data', chunk => log.push(chunk.toString('utf8')));
for (let attempt = 0; attempt < 100; attempt += 1) {
if (log.join('').includes('Timmy is listening')) return { child, log };
try {
const response = await fetch(`http://127.0.0.1:${port}/api/healthz`);
if (response.ok) return { child, log };
} catch {}
await new Promise(resolve => setTimeout(resolve, 100));
}
child.kill();
throw new Error('attack suite target did not become ready');
}
const stop = child => new Promise(resolve => {
if (!child || child.exitCode !== null) return resolve();
child.kill();
child.once('exit', resolve);
});
async function main() {
const worker = await startStubWorker();
const port = explicitBaseUrl ? null : await freePort();
const baseUrl = explicitBaseUrl || `http://127.0.0.1:${port}`;
const target = explicitBaseUrl ? null : await startTarget({ providerPort: worker.port, port });
const photo = await readFile(join(repoRoot, 'tests/fixtures/synthetic-type4.jpg'));
const imageDataUrl = `data:image/jpeg;base64,${photo.toString('base64')}`;
const post = async (body, targetBase = baseUrl) => {
const before = worker.state.requests;
const response = await fetch(`${targetBase}/api/analyze`, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: typeof body === 'string' ? body : JSON.stringify(body),
});
const text = await response.text();
return { status: response.status, text, providerCalls: worker.state.requests - before };
};
const cases = [];
const record = (entry) => { cases.push(entry); return entry; };
try {
const health = await fetch(`${baseUrl}/api/healthz`).then(response => response.json());
if (!health.visionEnabled) throw new Error('the attack suite needs vision analysis enabled on the target');
// 1. An oversized body must die at the request cap, before the photo is parsed or forwarded.
const oversized = await post({ consent: true, imageDataUrl: `data:image/jpeg;base64,${'A'.repeat(6.6 * 1024 * 1024)}` });
record({
id: 'ingress-oversized-body',
category: 'oversized',
severity: 'P0',
requirement: 'A body over the request cap is refused before any provider work.',
expected: { status: 413, providerCalls: 0 },
observed: { status: oversized.status, providerCalls: oversized.providerCalls },
snippet: oversized.text.slice(0, 120),
});
// 2. Bytes decide the format, not the data-URL label: HTML dressed as a JPEG is a polyglot.
const html = Buffer.from('<html><body><script>alert(document.cookie)</script></body></html>');
const polyglot = await post({ consent: true, imageDataUrl: `data:image/jpeg;base64,${html.toString('base64')}` });
record({
id: 'ingress-polyglot-html',
category: 'polyglot',
severity: 'P0',
requirement: 'A payload whose bytes are HTML under an image/jpeg label is refused before the provider sees it.',
expected: { status: 400, providerCalls: 0 },
observed: { status: polyglot.status, providerCalls: polyglot.providerCalls },
snippet: polyglot.text.slice(0, 120),
});
// 3. Random bytes under a JPEG label are not an image either.
const garbage = await post({ consent: true, imageDataUrl: `data:image/jpeg;base64,${randomBytes(64).toString('base64')}` });
record({
id: 'ingress-garbage-bytes',
category: 'truncated',
severity: 'P0',
requirement: 'Undecodable bytes are refused rather than forwarded and interpreted.',
expected: { status: 400, providerCalls: 0 },
observed: { status: garbage.status, providerCalls: garbage.providerCalls },
snippet: garbage.text.slice(0, 120),
});
// 4. Consent is a precondition, and it is cheap to prove: no consent, no provider call.
const noConsent = await post({ consent: false, imageDataUrl });
record({
id: 'consent-required',
category: 'consent',
severity: 'P0',
requirement: 'Analysis without explicit consent is refused and never leaves the boundary.',
expected: { status: 400, providerCalls: 0 },
observed: { status: noConsent.status, providerCalls: noConsent.providerCalls },
snippet: noConsent.text.slice(0, 120),
});
// 5. A worker answer outside the schema must fail closed, and the provider's words must not
// reach the user or the evidence file.
worker.state.content = 'garbage-completion-token: isStool yes maybe, bristolType ten';
const malformed = await post({ consent: true, imageDataUrl });
record({
id: 'model-output-malformed',
category: 'malformed',
severity: 'P0',
requirement: 'A non-conforming provider answer is replaced by neutral copy and never echoed.',
expected: { status: 503, providerTextLeaked: false },
observed: { status: malformed.status, providerTextLeaked: /garbage-completion-token/.test(malformed.text) },
snippet: malformed.text.slice(0, 120),
});
// 6. Replay: this boundary holds no session state, so a replay is a second request. What must
// hold is that it is answered identically and that nothing accumulates server-side.
worker.state.content = JSON.stringify({ isStool: true, bristolType: 4, color: 'brown', confidence: 0.82, imageQuality: 'good', observations: 'Smooth, formed appearance.' });
const first = await post({ consent: true, imageDataUrl });
const second = await post({ consent: true, imageDataUrl });
record({
id: 'replay-duplicate-request',
category: 'replay',
severity: 'P2',
requirement: 'A byte-identical replay is answered identically and stores nothing on the server.',
expected: { status: 200, providerCalls: 1, repeatedAnswerIdentical: true },
observed: { status: second.status, providerCalls: second.providerCalls, repeatedAnswerIdentical: first.text === second.text },
snippet: second.text.slice(0, 120),
});
// 7. Flood: concurrent requests must meet a bound instead of all reaching the worker at once.
const beforeFlood = worker.state.requests;
const flood = await Promise.all(Array.from({ length: 12 }, () => post({ consent: true, imageDataUrl })));
record({
id: 'flood-concurrency',
category: 'flood',
severity: 'P1',
requirement: 'A burst of concurrent analyses meets a bounded queue or a rate refusal.',
expected: { refused: true },
observed: { refused: flood.some(item => item.status === 429 || item.status === 503), providerCalls: worker.state.requests - beforeFlood },
snippet: `statuses ${[...new Set(flood.map(item => item.status))].sort().join(',')}`,
});
// 8. Log inspection: the operator log is committed evidence, so it must not contain the photo.
const capturedLog = (target?.log || []).join('');
if (!capturedLog.trim()) throw new Error('no server log was captured, so the log scan would prove nothing');
record({
id: 'log-leak-scan',
category: 'log',
severity: 'P0',
requirement: 'Server logs contain no image bytes, base64, or note text.',
expected: { imageBytesInLog: false },
observed: { imageBytesInLog: containsImageBytes(capturedLog) },
snippet: `${capturedLog.trim().split('\n').length} log line(s) inspected`,
});
// 9. Deletion is governed by the contribution ledger, not by this HTTP boundary. Covering it
// here means naming the executable test that does it, and only counting it once verified.
let evidenceVerified = false;
try {
await access(join(repoRoot, 'tests/training-contribution.test.js'));
evidenceVerified = true;
} catch {}
record({
id: 'deletion-withdrawal',
category: 'deletion',
severity: 'P0',
surface: 'out-of-surface',
requirement: 'Withdrawal deletes the contribution and the server holds no copy of the photo.',
evidence: 'tests/training-contribution.test.js',
evidenceVerified,
snippet: evidenceVerified ? 'covered by the contribution ledger suite' : 'named evidence file is missing',
});
// Load: sequential valid analyses, abstaining answers, latency measured with the same boundary.
// The abuse cases above spend the ingress request budget, so latency is measured against a second
// target that starts with the budget wide open; otherwise these samples would be 429 refusals.
const loadPort = explicitBaseUrl ? null : await freePort();
const loadTarget = explicitBaseUrl
? null
: await startTarget({
providerPort: worker.port,
port: loadPort,
env: { TIMMY_ANALYZE_RATE_LIMIT: '100000', TIMMY_ANALYZE_RATE_WINDOW_MS: '60000' },
});
const loadBase = explicitBaseUrl || `http://127.0.0.1:${loadPort}`;
worker.state.content = JSON.stringify({ isStool: false, confidence: 0.9 });
const samples = [];
let errors = 0;
let abstentions = 0;
try {
for (let index = 0; index < 8; index += 1) {
const started = performance.now();
const result = await post({ consent: true, imageDataUrl }, loadBase);
samples.push(Math.round(performance.now() - started));
if (result.status !== 200) errors += 1;
else {
try {
if (JSON.parse(result.text).status === 'needs_user_input') abstentions += 1;
} catch {}
}
}
} finally {
await stop(loadTarget?.child);
}
const percentiles = latencyPercentiles(samples);
const load = {
requests: samples.length,
errors,
abstentions,
p50: percentiles.p50,
p95: percentiles.p95,
max: percentiles.max,
note: 'Latency measured against a counting stub on loopback; it measures this boundary, not a model.',
};
// Issue #16 adds a per-caller request budget at the ingress, so a suite that spends its budget on
// the abuse cases above would measure 429 refusals here and publish them as analysis latency.
// A refusal is not a sample: the load phase runs against its own permissive target instead.
if (load.errors > 0) {
throw new Error(`the load phase measured ${load.errors}/${load.requests} refusals instead of analyses, so it cannot publish latency`);
}
const report = buildReport({
generatedAt: new Date().toISOString(),
target: {
baseUrl,
provider: 'stub counting worker (synthetic answers, no model loaded)',
visionProfile: 'selfhost',
release: health.release,
commit: health.commit,
},
cases,
load,
});
await mkdir(dirname(reportPath), { recursive: true });
await writeFile(reportPath, `${JSON.stringify(report, null, 2)}\n`, 'utf8');
const width = Math.max(...cases.map(item => item.id.length));
for (const item of report.cases) {
const detail = item.status === 'fail' ? ` — ${item.mismatch}` : '';
console.log(`${item.id.padEnd(width)} ${item.severity} ${item.status}${detail}`);
}
console.log(`load: ${load.requests} requests · p50 ${load.p50}ms · p95 ${load.p95}ms · errors ${load.errors} · abstentions ${load.abstentions}`);
console.log(`report: ${reportPath}`);
console.log(`VERDICT: ${report.verdict}`);
if (report.verdict === 'BLOCKED') {
for (const item of [...report.blocking, ...report.unproven]) console.log(` blocks RC: ${item.id} (${item.severity}) ${item.reason}`);
}
return report.verdict === 'GO' ? 0 : 1;
} finally {
await stop(target?.child);
await worker.close();
}
}
main().then(code => { process.exitCode = code; }).catch(error => {
console.error(`attack suite could not complete: ${error.message}`);
process.exitCode = 2;
});
+45 -2
View File
@@ -1,11 +1,15 @@
import http from 'node:http';
import { isIP } from 'node:net';
import { createHash, randomBytes } from 'node:crypto';
import { readFile, stat } from 'node:fs/promises';
import { extname, join, normalize } from 'node:path';
import { fileURLToPath } from 'node:url';
import { analyzePhoto, visionFailureResponse } from './src/vision-service.js';
import { probeVisionProvider, resolveVisionConfig } from './src/vision-config.js';
import { AgentGatewayError, createHermesAgentService, resolveHermesAgentConfig } from './src/hermes-agent-service.js';
import { readJsonBody } from './src/http-body.js';
import { inspectImageBytes, IngressError } from './src/image-ingress.js';
import { createRateLimiter } from './src/rate-limit.js';
const root=fileURLToPath(new URL('.',import.meta.url));
const port=Number(process.env.PORT||4173);
@@ -32,7 +36,27 @@ const stagingLabel=process.env.TIMMY_STAGING_LABEL?`Staging · ${release} · ${c
function sendJson(res,status,value){res.writeHead(status,{'content-type':'application/json; charset=utf-8','cache-control':'no-store','x-content-type-options':'nosniff'});res.end(JSON.stringify(value));}
function escapeHtmlAttribute(value){return String(value).replace(/[&<>"]/g,character=>({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;'}[character]));}
function readJson(req,maxBytes=6*1024*1024){return new Promise((resolve,reject)=>{let size=0,tooLarge=false;const chunks=[];req.on('data',chunk=>{size+=chunk.length;if(size>maxBytes){tooLarge=true;return}chunks.push(chunk)});req.on('end',()=>{if(tooLarge)return reject(new AgentGatewayError(413,'Request is too large.'));try{resolve(JSON.parse(Buffer.concat(chunks).toString('utf8')))}catch{return reject(new AgentGatewayError(400,'Invalid JSON request.'))}});req.on('error',reject)})}
// Photo ingress (issue #16). Every limit here is decided from the request itself and fails before
// any provider call: a per-caller request budget, a deadline for a body that never arrives, a byte
// ceiling, and a container/pixel inspection of the decoded image. The bucket key is a salted hash
// that lives only in this process's memory, so no address and no image byte is ever logged.
const bodyTimeoutMs=Number(process.env.TIMMY_BODY_TIMEOUT_MS||15_000);
const analyzeLimiter=createRateLimiter({limit:Number(process.env.TIMMY_ANALYZE_RATE_LIMIT||12),windowMs:Number(process.env.TIMMY_ANALYZE_RATE_WINDOW_MS||60_000)});
const bucketSalt=randomBytes(16).toString('hex');
function clientBucket(req){return createHash('sha256').update(`${bucketSalt}:${req.socket.remoteAddress||'unknown'}`).digest('hex').slice(0,24);}
function readJson(req,maxBytes=6*1024*1024){return readJsonBody(req,{maxBytes,timeoutMs:bodyTimeoutMs});}
function ingressGuard(payload){
const raw=String(payload?.imageDataUrl||'');
const match=/^data:(image\/(?:jpeg|png|webp));base64,([A-Za-z0-9+/=]+)$/.exec(raw);
// A payload that is not shaped like a data URL is left to the semantic validator, so the user still
// gets the specific "upload a JPEG, PNG, or WebP photo" copy rather than a second opinion here.
if(!match)return null;
return inspectImageBytes(Buffer.from(match[2],'base64'),{declaredMime:match[1]});
}
function sendIngressError(res,error){
if(error instanceof IngressError)return sendJson(res,400,{error:error.message,code:error.code});
return sendJson(res,error.status||400,{error:error.message,code:error.code});
}
function cookie(req,name){for(const part of String(req.headers.cookie||'').split(';')){const [key,...value]=part.trim().split('=');if(key===name)return decodeURIComponent(value.join('='))}return ''}
function requestOrigin(req){return String(req.headers.origin||'').replace(/\/$/,'')}
function rejectCrossSite(req){const site=String(req.headers['sec-fetch-site']||'');if(site&&site!=='same-origin')throw new AgentGatewayError(403,'Cross-site agent requests are not allowed.')}
@@ -52,8 +76,27 @@ http.createServer(async(req,res)=>{
return sendJson(res,200,{...visionConfig.publicStatus(),providerReady:provider.ready,modelSeen:provider.modelSeen,privacy});
}
if(appPath==='/api/analyze'&&req.method==='POST'){
// Refusals first, in cost order: the request budget, then the body, then the image bytes. None of
// these depend on the model, so hostile bytes are refused as hostile even when analysis is off —
// otherwise the disabled switch hides the ingress boundary behind a 503 and the refusal is untested.
const budget=analyzeLimiter.take(clientBucket(req));
if(!budget.allowed){
res.setHeader('retry-after',String(Math.max(1,Math.ceil(budget.retryAfterMs/1000))));
return sendJson(res,429,{error:'Too many photos from this device. Wait a moment, then try again — your journal and manual logging still work.'});
}
let payload;
try{payload=await readJson(req)}
catch(error){
const sent=sendIngressError(res,error);
// The refusal is written first, then the half-read socket is dropped so a stalled body cannot
// hold a connection or its partial buffer open.
if(error?.code==='body_timeout'||error?.code==='too_large')req.destroy?.();
return sent;
}
try{ingressGuard(payload)}
catch(error){return sendIngressError(res,error)}
// Bytes are known to be a real photo at this point, so the disabled switch is the honest answer.
if(!visionConfig.enabled)return sendJson(res,503,{error:'AI analysis is disabled. Continue manually.'});
const payload=await readJson(req);
try{return sendJson(res,200,await analyzePhoto({payload,config:visionConfig}))}
catch(error){const failure=visionFailureResponse(error);return sendJson(res,failure.status,failure.body)}
}
+79
View File
@@ -0,0 +1,79 @@
// Bounded request-body reading for the API boundary (issue #16).
//
// Three limits have to fail before any provider work starts: the byte ceiling, a deadline for a body
// that never arrives, and JSON validity. All three refuse with a client-visible status and a fixed
// message; none of them logs or returns the body.
const COPY = Object.freeze({
too_large: [413, 'Request is too large.'],
invalid_json: [400, 'Invalid JSON request.'],
body_timeout: [408, 'The request took too long to arrive. Try again.'],
});
export class BodyError extends Error {
constructor(code) {
const [status, message] = COPY[code] || COPY.invalid_json;
super(message);
this.name = 'BodyError';
this.code = code;
this.status = status;
}
}
export function readJsonBody(req, { maxBytes = 6 * 1024 * 1024, timeoutMs = 15_000 } = {}) {
return new Promise((resolve, reject) => {
let settled = false;
let tooLarge = false;
let size = 0;
const chunks = [];
const finish = callback => {
if (settled) return;
settled = true;
clearTimeout(deadline);
req.removeListener('data', onData);
req.removeListener('end', onEnd);
req.removeListener('error', onError);
callback();
};
const deadline = setTimeout(() => {
finish(() => {
// Stop reading and let the caller answer 408; the server destroys the socket once the
// refusal is written, so a half-read body cannot be held open indefinitely.
req.pause?.();
reject(new BodyError('body_timeout'));
});
}, timeoutMs);
function onData(chunk) {
if (settled) return;
// Past the ceiling the body is already refused: keep draining it without buffering so the
// caller can still read the 413. Destroying the socket here would surface as "fetch failed"
// and leave the caller with no status to act on.
if (tooLarge) return;
size += chunk.length;
if (size > maxBytes) {
tooLarge = true;
chunks.length = 0;
return;
}
chunks.push(chunk);
}
function onEnd() {
finish(() => {
if (tooLarge) return reject(new BodyError('too_large'));
try {
resolve(JSON.parse(Buffer.concat(chunks).toString('utf8')));
} catch {
reject(new BodyError('invalid_json'));
}
});
}
function onError(error) {
finish(() => reject(error));
}
req.on('data', onData);
req.on('end', onEnd);
req.on('error', onError);
});
}
+158
View File
@@ -0,0 +1,158 @@
// Transport-level hardening for the one endpoint that accepts image bytes (issue #16).
//
// The declared MIME type in a data URL is a claim made by the caller, never evidence, so every
// decision here comes from the bytes: the container is sniffed, the container is walked to its own
// end marker so nothing can ride along behind a valid image, embedded metadata segments are refused
// (the browser always uploads a canvas re-encode, which carries none), and the pixel dimensions are
// read from the header so a decode bomb never reaches the worker.
//
// Refusal copy is fixed and deliberate: it names what is wrong with the file and never echoes the
// bytes, a hash, or any part of the upload back to the caller or into a log line.
export const MAX_EDGE = 4096;
export const MAX_PIXELS = 16_000_000;
const COPY = Object.freeze({
empty: 'The photo is empty.',
not_an_image: 'That file is not a JPEG, PNG, or WebP photo.',
container_mismatch: 'This photo is not the format it claims. Choose a JPEG, PNG, or WebP photo.',
truncated: 'This photo is incomplete or damaged. Choose another photo.',
trailing_data: 'This photo file has extra data after the image. Choose another photo.',
metadata_present: 'This photo still carries embedded metadata. Choose it through Timmy’s camera or gallery picker so it is re-encoded without metadata.',
dimensions_too_large: 'This photo is too large in pixels. Choose a smaller photo.',
});
export class IngressError extends Error {
constructor(code) {
super(COPY[code] || COPY.not_an_image);
this.name = 'IngressError';
this.code = code;
}
}
const refuse = code => {
throw new IngressError(code);
};
const PNG_CHUNKS = { metadata: new Set(['eXIf', 'tEXt', 'zTXt', 'iTXt', 'tIME']) };
// Segment markers that carry metadata rather than image data: APP1 (Exif/XMP), APP13 (Photoshop
// resource block) and COM (free-text comment). The app's own upload is a canvas re-encode and
// contains none of them, so a file that does was not produced by the capture path.
const JPEG_METADATA_MARKERS = new Set([0xe1, 0xed, 0xfe]);
const JPEG_SOF_MARKERS = new Set([0xc0, 0xc1, 0xc2, 0xc3, 0xc5, 0xc6, 0xc7, 0xc9, 0xca, 0xcb, 0xcd, 0xce, 0xcf]);
function sniff(bytes) {
if (bytes.length >= 8 && bytes.subarray(0, 8).equals(Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]))) return 'image/png';
if (bytes.length >= 3 && bytes[0] === 0xff && bytes[1] === 0xd8 && bytes[2] === 0xff) return 'image/jpeg';
if (bytes.length >= 12 && bytes.toString('ascii', 0, 4) === 'RIFF' && bytes.toString('ascii', 8, 12) === 'WEBP') return 'image/webp';
return '';
}
function inspectPng(bytes) {
let offset = 8;
let width = 0;
let height = 0;
let sawHeader = false;
while (offset + 8 <= bytes.length) {
const length = bytes.readUInt32BE(offset);
const type = bytes.toString('ascii', offset + 4, offset + 8);
const dataStart = offset + 8;
const next = dataStart + length + 4;
if (next > bytes.length) refuse('truncated');
if (!sawHeader) {
if (type !== 'IHDR' || length !== 13) refuse('truncated');
width = bytes.readUInt32BE(dataStart);
height = bytes.readUInt32BE(dataStart + 4);
sawHeader = true;
}
if (PNG_CHUNKS.metadata.has(type)) refuse('metadata_present');
if (type === 'IEND') {
if (next !== bytes.length) refuse('trailing_data');
return { width, height };
}
offset = next;
}
return refuse('truncated');
}
function inspectJpeg(bytes) {
let offset = 2;
let width = 0;
let height = 0;
while (offset + 4 <= bytes.length) {
if (bytes[offset] !== 0xff) refuse('truncated');
const marker = bytes[offset + 1];
if (marker === 0xd9) {
if (offset + 2 !== bytes.length) refuse('trailing_data');
break;
}
// Standalone markers carry no length field.
if (marker === 0x01 || (marker >= 0xd0 && marker <= 0xd7) || marker === 0xff) {
offset += marker === 0xff ? 1 : 2;
continue;
}
const length = bytes.readUInt16BE(offset + 2);
if (length < 2 || offset + 2 + length > bytes.length) refuse('truncated');
if (JPEG_METADATA_MARKERS.has(marker)) refuse('metadata_present');
if (JPEG_SOF_MARKERS.has(marker) && length >= 7) {
height = bytes.readUInt16BE(offset + 5);
width = bytes.readUInt16BE(offset + 7);
}
offset += 2 + length;
if (marker === 0xda) {
// Entropy-coded scan data follows the start-of-scan header. The only thing left to trust is
// the end-of-image marker, and it has to be the last two bytes of the file.
if (bytes[bytes.length - 2] !== 0xff || bytes[bytes.length - 1] !== 0xd9) {
return refuse(bytes.includes(Buffer.from([0xff, 0xd9])) ? 'trailing_data' : 'truncated');
}
break;
}
}
if (!width || !height) refuse('truncated');
return { width, height };
}
function inspectWebp(bytes) {
const declared = bytes.readUInt32LE(4) + 8;
if (declared !== bytes.length) refuse(declared < bytes.length ? 'trailing_data' : 'truncated');
let offset = 12;
let width = 0;
let height = 0;
while (offset + 8 <= bytes.length) {
const type = bytes.toString('ascii', offset, offset + 4);
const size = bytes.readUInt32LE(offset + 4);
const dataStart = offset + 8;
if (dataStart + size > bytes.length) refuse('truncated');
if (type === 'VP8X' && size >= 10) {
width = bytes.readUIntLE(dataStart + 4, 3) + 1;
height = bytes.readUIntLE(dataStart + 7, 3) + 1;
} else if (type === 'VP8 ' && size >= 10) {
width = bytes.readUInt16LE(dataStart + 6) & 0x3fff;
height = bytes.readUInt16LE(dataStart + 8) & 0x3fff;
} else if (type === 'VP8L' && size >= 5) {
const bits = bytes.readUInt32LE(dataStart + 1);
width = (bits & 0x3fff) + 1;
height = ((bits >> 14) & 0x3fff) + 1;
}
offset = dataStart + size + (size % 2);
}
if (!width || !height) refuse('truncated');
return { width, height };
}
// Returns the real container, its pixel dimensions, and the byte count. Throws IngressError on
// anything the model worker must never be handed.
export function inspectImageBytes(bytes, { declaredMime = '' } = {}) {
const buffer = Buffer.isBuffer(bytes) ? bytes : Buffer.from(bytes || []);
if (!buffer.length) refuse('empty');
const actual = sniff(buffer);
if (!actual) refuse('not_an_image');
if (declaredMime && declaredMime !== actual) refuse('container_mismatch');
const dimensions = actual === 'image/png' ? inspectPng(buffer)
: actual === 'image/jpeg' ? inspectJpeg(buffer)
: inspectWebp(buffer);
if (!dimensions.width || !dimensions.height) refuse('truncated');
if (dimensions.width > MAX_EDGE || dimensions.height > MAX_EDGE
|| dimensions.width * dimensions.height > MAX_PIXELS) refuse('dimensions_too_large');
return { mime: actual, width: dimensions.width, height: dimensions.height, bytes: buffer.length };
}
+51
View File
@@ -0,0 +1,51 @@
// Bounded, in-process request budget for the photo-analysis endpoint (issue #16).
//
// The limiter exists so a caller cannot make Timmy the transport for a flood of images to the
// model worker. It keeps timestamps only, never the bytes, the file name, or the caller's address:
// the server hands it a short-lived opaque bucket key. Refusals never extend a window, so a caller
// cannot earn a longer penalty than the request it already made.
export function createRateLimiter({ limit = 12, windowMs = 60_000, maxKeys = 2048, now = () => Date.now() } = {}) {
const hits = new Map();
let lastPrune = now();
function prune(at) {
const cutoff = at - windowMs;
for (const [key, stamps] of hits) {
const kept = stamps.filter(stamp => stamp > cutoff);
if (kept.length) hits.set(key, kept);
else hits.delete(key);
}
lastPrune = at;
}
return {
take(key) {
const at = now();
// Pruning is at most once per window, plus whenever the map outgrows its ceiling, so the
// bookkeeping stays bounded no matter how many distinct callers appear.
if (at - lastPrune >= windowMs || hits.size > maxKeys) prune(at);
const window = (hits.get(key) || []).filter(stamp => stamp > at - windowMs);
if (window.length >= limit) {
hits.set(key, window);
return { allowed: false, remaining: 0, retryAfterMs: Math.max(0, window[0] + windowMs - at) };
}
window.push(at);
hits.set(key, window);
if (hits.size > maxKeys) {
for (const oldest of hits.keys()) {
if (hits.size <= maxKeys) break;
hits.delete(oldest);
}
}
return { allowed: true, remaining: Math.max(0, limit - window.length), retryAfterMs: 0 };
},
size() {
return hits.size;
},
reset() {
hits.clear();
lastPrune = now();
},
};
}
+95
View File
@@ -0,0 +1,95 @@
// Release security gate: the arithmetic and the verdict rule for the RC attack suite (#40).
//
// The suite observes the real server; this module only turns observations into a verdict. Keeping
// the rule here (not in the runner) is what makes "a P0 finding blocks RC approval" testable.
/** Nearest-rank percentile: the returned value is always a sample that was actually measured. */
export function latencyPercentiles(samples) {
const sorted = [...samples].sort((a, b) => a - b);
if (!sorted.length) return { count: 0, p50: null, p95: null, max: null };
const rank = (p) => sorted[Math.min(sorted.length - 1, Math.ceil((p / 100) * sorted.length) - 1)];
return { count: sorted.length, p50: rank(50), p95: rank(95), max: sorted[sorted.length - 1] };
}
/** One case against one expectation. A partial match is a failure, and it says which field broke. */
export function evaluateCase(record) {
const mismatches = Object.keys(record.expected)
.filter(field => record.observed?.[field] !== record.expected[field])
.map(field => `${field} ${record.expected[field]} expected, observed ${record.observed?.[field]}`);
return { ...record, status: mismatches.length ? 'fail' : 'pass', mismatch: mismatches.join('; ') };
}
const SEVERITY_BLOCKS = (severity) => severity === 'P0';
/** The RC rule: every P0 failure blocks approval, unproven categories block it too, P1/P2 only hold. */
export function releaseVerdict({ cases, load = null }) {
const evaluated = cases.map(record => (record.surface === 'out-of-surface'
? { ...record, status: record.evidenceVerified ? 'covered-elsewhere' : 'unproven' }
: evaluateCase(record)));
const blocking = evaluated.filter(item => item.status === 'fail' && SEVERITY_BLOCKS(item.severity))
.map(item => ({ id: item.id, severity: item.severity, reason: item.mismatch }));
const holds = evaluated.filter(item => item.status === 'fail' && !SEVERITY_BLOCKS(item.severity))
.map(item => ({ id: item.id, severity: item.severity, reason: item.mismatch }));
const unproven = evaluated.filter(item => item.status === 'unproven')
.map(item => ({ id: item.id, severity: item.severity, reason: 'no executable evidence' }));
return {
verdict: blocking.length || unproven.length ? 'BLOCKED' : 'GO',
blocking,
holds,
unproven,
coveredElsewhere: evaluated.filter(item => item.status === 'covered-elsewhere').map(item => item.id),
proven: evaluated.filter(item => item.status === 'pass').length,
cases: evaluated,
load,
};
}
// Evidence hygiene: the report and the captured server log are committed and quoted. A photo that
// survived into either one is a leak of the exact data this suite exists to protect.
const IMAGE_DATA_URI = /data:image\/[a-z0-9.+-]+;base64,[A-Za-z0-9+/=]+/gi;
const LONG_BASE64 = /[A-Za-z0-9+/]{200,}={0,2}/;
export function containsImageBytes(text) {
const value = String(text ?? '');
return new RegExp(IMAGE_DATA_URI.source, 'i').test(value) || LONG_BASE64.test(value);
}
export function redactImageBytes(text) {
return String(text ?? '')
.replace(IMAGE_DATA_URI, '[redacted image bytes]')
.replace(new RegExp(LONG_BASE64.source, 'g'), '[redacted image bytes]');
}
/** Fails closed rather than emitting evidence that carries a photo. */
export function assertNoImageBytes(label, text) {
if (containsImageBytes(text)) throw new Error(`${label} contains image bytes; refusing to write release evidence.`);
return true;
}
/**
* The receipt the release job quotes. It is redacted after assembly, so a case that captured a
* response body still cannot carry a photo in it.
*/
export function buildReport({ generatedAt, target, cases, load, suite = 'scripts/security_attack_suite.mjs' }) {
const verdict = releaseVerdict({ cases, load });
const report = {
schema: 'timmy-rc-security-attack/1',
issue: 40,
generatedAt,
suite,
target,
verdict: verdict.verdict,
blocking: verdict.blocking,
holds: verdict.holds,
unproven: verdict.unproven,
coveredElsewhere: verdict.coveredElsewhere,
proven: verdict.proven,
cases: verdict.cases.map(({ id, category, severity, requirement, expected, observed, status, mismatch, snippet, surface, evidence, evidenceVerified }) =>
({ id, category, severity, requirement, expected, observed, status, mismatch, surface, evidence, evidenceVerified, snippet: snippet ? redactImageBytes(snippet) : undefined })),
load,
disclaimer: 'Synthetic hostile bytes only. Every case drives the real server against a stub worker; no real medical image and no live user data is involved.',
};
const clean = JSON.parse(redactImageBytes(JSON.stringify(report)));
assertNoImageBytes('security report', JSON.stringify(clean));
return clean;
}
+245
View File
@@ -0,0 +1,245 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import http from 'node:http';
import net from 'node:net';
import { spawn } from 'node:child_process';
import { once } from 'node:events';
import { fileURLToPath } from 'node:url';
// End-to-end proof for issue #16 at the real boundary: the server process that ships, not a mock of
// it. Every image here is synthetic container bytes assembled in this file, and the provider is a
// local stub that counts requests, so "the provider was never called" is measurable rather than
// assumed.
const repoRoot = fileURLToPath(new URL('..', import.meta.url));
const PNG_SIGNATURE = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
function pngChunk(type, data = Buffer.alloc(0)) {
const length = Buffer.alloc(4);
length.writeUInt32BE(data.length);
return Buffer.concat([length, Buffer.from(type, 'ascii'), data, Buffer.alloc(4)]);
}
function cleanJpeg({ width = 1200, height = 900, tail = Buffer.alloc(0) } = {}) {
const sof = Buffer.alloc(17);
sof.writeUInt16BE(0xffc0, 0);
sof.writeUInt16BE(15, 2);
sof[4] = 8;
sof.writeUInt16BE(height, 5);
sof.writeUInt16BE(width, 7);
sof[9] = 3;
for (let i = 0; i < 3; i += 1) {
sof[10 + i * 3] = i + 1;
sof[11 + i * 3] = 0x11;
}
return Buffer.concat([Buffer.from([0xff, 0xd8]), sof, Buffer.from([0xff, 0xd9]), tail]);
}
function polyglotPng() {
const ihdr = Buffer.alloc(13);
ihdr.writeUInt32BE(1200, 0);
ihdr.writeUInt32BE(900, 4);
ihdr[8] = 8;
ihdr[9] = 2;
return Buffer.concat([
PNG_SIGNATURE,
pngChunk('IHDR', ihdr),
pngChunk('IDAT', Buffer.from([0x78, 0x9c, 0x01])),
pngChunk('IEND'),
Buffer.from('<html><script>alert(1)</script></html>', 'utf8'),
]);
}
const dataUrl = (mime, bytes) => `data:${mime};base64,${bytes.toString('base64')}`;
async function freePort() {
const server = net.createServer();
server.listen(0, '127.0.0.1');
await once(server, 'listening');
const { port } = server.address();
server.close();
return port;
}
async function startProvider() {
const state = { calls: 0 };
const server = http.createServer(async (req, res) => {
if (req.url === '/v1/models') {
res.writeHead(200, { 'content-type': 'application/json' });
res.end(JSON.stringify({ data: [{ id: 'ingress-test-model' }] }));
return;
}
state.calls += 1;
for await (const _ of req) { /* drain */ }
res.writeHead(200, { 'content-type': 'application/json' });
res.end(JSON.stringify({
choices: [{
message: {
content: JSON.stringify({
isStool: true, bristolType: 4, color: 'brown', confidence: 0.8,
imageQuality: 'good', observations: 'Synthetic test card.',
}),
},
}],
}));
});
server.listen(0, '127.0.0.1');
await once(server, 'listening');
return {
state,
port: server.address().port,
// fetch keeps connections alive, so a plain close() would wait for sockets the test already
// finished with and hang teardown.
close: () => new Promise(resolve => {
server.closeAllConnections?.();
server.close(resolve);
}),
};
}
async function withServer(env, run) {
const provider = await startProvider();
const port = await freePort();
const child = spawn(process.execPath, ['server.mjs'], {
cwd: repoRoot,
env: {
...process.env,
HOST: '127.0.0.1',
PORT: String(port),
TIMMY_VISION_ENABLED: 'true',
TIMMY_VISION_PROFILE: 'selfhost',
TIMMY_VISION_BASE_URL: `http://127.0.0.1:${provider.port}/v1`,
TIMMY_VISION_MODEL: 'ingress-test-model',
...env,
},
stdio: ['ignore', 'pipe', 'pipe'],
});
const logs = [];
child.stdout.on('data', chunk => logs.push(chunk.toString()));
child.stderr.on('data', chunk => logs.push(chunk.toString()));
const base = `http://127.0.0.1:${port}`;
try {
// Readiness is the server's own health route, not a sleep.
const deadline = Date.now() + 15_000;
for (;;) {
try {
const response = await fetch(`${base}/api/healthz`);
if (response.ok) break;
} catch { /* not up yet */ }
if (Date.now() > deadline) throw new Error(`server did not start: ${logs.join('')}`);
await new Promise(resolve => setTimeout(resolve, 100));
}
await run({ base, provider, logs });
} finally {
child.kill('SIGKILL');
await provider.close();
}
}
const postAnalyze = (base, payload) => fetch(`${base}/api/analyze`, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify(payload),
});
test('the shipping server refuses bytes that are not an image before any provider work', async () => {
await withServer({}, async ({ base, provider }) => {
const response = await postAnalyze(base, {
consent: true,
imageDataUrl: dataUrl('image/jpeg', Buffer.from('<html><script>alert(1)</script></html>', 'utf8')),
});
const body = await response.json();
assert.equal(response.status, 400);
assert.match(body.error, /not a JPEG, PNG, or WebP photo/i);
assert.equal(provider.state.calls, 0, 'a non-image must never reach the model provider');
});
});
test('the shipping server refuses a valid image with a hidden payload appended', async () => {
await withServer({}, async ({ base, provider, logs }) => {
const response = await postAnalyze(base, { consent: true, imageDataUrl: dataUrl('image/png', polyglotPng()) });
const body = await response.json();
assert.equal(response.status, 400);
assert.match(body.error, /extra data after the image/i);
assert.equal(provider.state.calls, 0, 'a polyglot file must never reach the model provider');
// The refusal path must not copy the upload into the process log, where it would outlive the request.
const logged = logs.join('');
assert.doesNotMatch(logged, /base64/i);
assert.doesNotMatch(logged, /[A-Za-z0-9+/]{120,}/, 'no run of image bytes may appear in the log');
});
});
test('the shipping server still analyses a clean re-encoded image', async () => {
await withServer({}, async ({ base, provider }) => {
const response = await postAnalyze(base, { consent: true, imageDataUrl: dataUrl('image/jpeg', cleanJpeg()) });
const body = await response.json();
assert.equal(response.status, 200);
assert.equal(body.status, 'suggestion');
assert.equal(provider.state.calls, 1, 'the clean path must reach the provider exactly once');
});
});
test('the shipping server refuses an oversized body with a refusal the caller can read', async () => {
// Issue #16 sets a byte ceiling, and issue #40 asks the release candidate to prove that ceiling
// behaves like a refusal rather than a dropped connection. Destroying the socket while the caller
// is still uploading leaves the caller with "fetch failed" and no status to act on, so the refusal
// has to be written and read before the connection goes away.
await withServer({}, async ({ base, provider }) => {
const oversized = JSON.stringify({
consent: true,
imageDataUrl: `data:image/jpeg;base64,${'A'.repeat(6.6 * 1024 * 1024)}`,
});
let response;
try {
response = await fetch(`${base}/api/analyze`, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: oversized,
});
} catch (error) {
assert.fail(`an oversized body must be refused with a readable status, not a dropped connection: ${error.message}`);
}
const body = await response.json();
assert.equal(response.status, 413);
assert.match(body.error, /too large/i);
assert.equal(provider.state.calls, 0, 'an oversized body must never reach the provider');
});
});
test('the shipping server refuses a flood with 429 and stops calling the provider', async () => {
await withServer({ TIMMY_ANALYZE_RATE_LIMIT: '2', TIMMY_ANALYZE_RATE_WINDOW_MS: '60000' }, async ({ base, provider }) => {
const payload = { consent: true, imageDataUrl: dataUrl('image/jpeg', cleanJpeg()) };
assert.equal((await postAnalyze(base, payload)).status, 200);
assert.equal((await postAnalyze(base, payload)).status, 200);
const refused = await postAnalyze(base, payload);
const body = await refused.json();
assert.equal(refused.status, 429);
assert.match(body.error, /too many photos|wait a minute/i);
assert.ok(Number(refused.headers.get('retry-after')) > 0, 'a refusal must say how long to wait');
assert.equal(provider.state.calls, 2, 'a refused request must not reach the provider');
});
});
test('the shipping server gives up on a body that never arrives', async () => {
await withServer({ TIMMY_BODY_TIMEOUT_MS: '250' }, async ({ base, provider }) => {
const port = Number(new URL(base).port);
const status = await new Promise((resolve, reject) => {
const socket = net.connect(port, '127.0.0.1', () => {
socket.write([
'POST /api/analyze HTTP/1.1',
'Host: 127.0.0.1',
'Content-Type: application/json',
'Content-Length: 400',
'', '',
].join('\r\n'));
});
socket.setTimeout(10_000, () => { socket.destroy(); reject(new Error('no response before the client timeout')); });
socket.on('data', chunk => {
const line = chunk.toString('utf8').split('\r\n')[0];
socket.destroy();
resolve(Number(line.split(' ')[1]));
});
socket.on('error', error => { if (error.code !== 'ECONNRESET') reject(error); });
});
assert.equal(status, 408);
assert.equal(provider.state.calls, 0, 'a stalled body must never reach the provider');
});
});
+322
View File
@@ -0,0 +1,322 @@
#!/usr/bin/env python3
"""Tests for the operator backup and restore tool.
Issue #39 asks for a production topology and a backup that a person can trust with the records
Timmy is not allowed to lose. The corpus manifest is append-only and the tombstones in it are the
only proof that a withdrawal happened, so a backup that quietly drops them, or a restore that
rebuilds a training set from withdrawn contributions, is worse than no backup at all.
The rules this suite locks in:
- model weights and transient inference images are never archived, and every excluded path is
named in the manifest with the reason it was left out;
- every archived byte carries a SHA-256 that ``verify`` recomputes before a restore writes
anything, so a tampered archive fails closed;
- hostile members (absolute paths, traversal, links, unlisted extras) are refused instead of
extracted;
- a restore round-trips the append-only manifest byte-for-byte and reports the same record counts,
including the tombstones.
Every fixture is synthetic: the "images" are opaque bytes, no real medical image is read or written.
"""
from __future__ import annotations
import hashlib
import io
import json
import os
import shutil
import subprocess
import sys
import tarfile
import tempfile
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
BACKUP = ROOT / "scripts" / "operator_backup.py"
IMAGE_BYTES = b"\xff\xd8\xff\xe0synthetic-jpeg-payload-not-a-real-photo"
WEIGHT_BYTES = b"GGUF-synthetic-weight-shard-not-a-real-model"
TRANSIENT_BYTES = b"transient-inference-upload-must-not-be-archived"
SECRET_BYTES = b"TIMMY_DATASET_SALT=demo\nTIMMY_VISION_TOKEN=do-not-archive-me\n"
def digest(payload: bytes) -> str:
return hashlib.sha256(payload).hexdigest()
def run(args: list[str], env: dict[str, str] | None = None) -> subprocess.CompletedProcess:
return subprocess.run(
[sys.executable, str(BACKUP), *args],
cwd=ROOT,
capture_output=True,
text=True,
env={**os.environ, **(env or {})},
)
def records(*lines: dict) -> str:
return "".join(json.dumps(line) + "\n" for line in lines)
def sample(contribution_id: str, *, relative_path: str = "images/sample-1.jpg") -> dict:
return {
"recordType": "sample",
"schemaVersion": 2,
"contributionId": contribution_id,
"relativePath": relative_path,
"imageId": digest(IMAGE_BYTES),
"lifecycle": {"state": "active", "updatedAt": "2026-08-01T10:00:00+00:00"},
}
def tombstone(contribution_id: str) -> dict:
return {
"recordType": "tombstone",
"schemaVersion": 2,
"contributionId": contribution_id,
"lifecycle": {"state": "revoked", "updatedAt": "2026-08-02T10:00:00+00:00"},
"reason": "participant withdrew consent",
"deletedArtifacts": [{"contributionId": contribution_id, "sha256": digest(IMAGE_BYTES)}],
}
def build_source(root: Path, *, with_tombstone: bool = False) -> Path:
"""A synthetic server-side data root: the records that must survive, plus what must not."""
source = root / "srv-timmy"
(source / "images").mkdir(parents=True)
(source / "images" / "sample-1.jpg").write_bytes(IMAGE_BYTES)
(source / "manifest.jsonl").write_text(
records(sample("contribution-1"), *([tombstone("contribution-2")] if with_tombstone else [])),
encoding="utf-8",
)
# Must never be archived: weights, transient inference bytes, operator secrets, and logs.
(source / "weights").mkdir()
(source / "weights" / "model.gguf").write_bytes(WEIGHT_BYTES)
(source / "transient").mkdir()
(source / "transient" / "inference-upload-1.jpg").write_bytes(TRANSIENT_BYTES)
(source / "secrets.env").write_bytes(SECRET_BYTES)
(source / "operator.log").write_text("GET /api/vision 200\n", encoding="utf-8")
return source
def members(archive: Path) -> dict[str, bytes]:
with tarfile.open(archive, "r:gz") as tar:
out = {}
for info in tar.getmembers():
if not info.isfile():
continue
handle = tar.extractfile(info)
assert handle is not None
out[info.name] = handle.read()
return out
def craft_archive(path: Path, entries: list[tuple[str, bytes]], *, link: str | None = None) -> None:
"""Build an archive by hand so the hostile cases are hostile on purpose."""
with tarfile.open(path, "w:gz") as tar:
for name, payload in entries:
info = tarfile.TarInfo(name)
info.size = len(payload)
info.mode = 0o644
tar.addfile(info, io.BytesIO(payload))
if link:
info = tarfile.TarInfo(link)
info.type = tarfile.SYMTYPE
info.linkname = "/etc/passwd"
info.mode = 0o777
tar.addfile(info)
class BackupContractTests(unittest.TestCase):
def setUp(self) -> None:
self.tmp = Path(tempfile.mkdtemp(prefix="timmy-backup-test-"))
self.addCleanup(shutil.rmtree, self.tmp, True)
self.archive = self.tmp / "backup.tar.gz"
self.manifest = self.tmp / "backup.manifest.json"
def backup(self, source: Path, *extra: str) -> subprocess.CompletedProcess:
return run([
"backup", "--source", str(source), "--archive", str(self.archive),
"--manifest", str(self.manifest), "--label", "nightly", *extra,
])
def test_backup_archives_records_and_excludes_weights_transient_images_secrets_and_logs(self) -> None:
source = build_source(self.tmp, with_tombstone=True)
result = self.backup(source)
self.assertEqual(result.returncode, 0, result.stderr)
archived = members(self.archive)
self.assertEqual(sorted(archived), ["payload/images/sample-1.jpg", "payload/manifest.jsonl"])
self.assertEqual(archived["payload/images/sample-1.jpg"], IMAGE_BYTES)
self.assertNotIn(WEIGHT_BYTES, b"".join(archived.values()))
self.assertNotIn(TRANSIENT_BYTES, b"".join(archived.values()))
self.assertNotIn(SECRET_BYTES, b"".join(archived.values()))
manifest = json.loads(self.manifest.read_text(encoding="utf-8"))
excluded = {entry["path"]: entry["reason"] for entry in manifest["excluded"]}
self.assertEqual(excluded["weights/model.gguf"], "model-weights")
self.assertEqual(excluded["transient/inference-upload-1.jpg"], "transient-inference")
self.assertEqual(excluded["secrets.env"], "secret")
self.assertEqual(excluded["operator.log"], "log")
# Every archived member is listed with the digest of the bytes that were stored.
listed = {entry["path"]: entry["sha256"] for entry in manifest["files"]}
self.assertEqual(listed, {
"images/sample-1.jpg": digest(IMAGE_BYTES),
"manifest.jsonl": digest((source / "manifest.jsonl").read_bytes()),
})
def test_backup_reports_record_counts_including_tombstones(self) -> None:
source = build_source(self.tmp, with_tombstone=True)
result = self.backup(source)
self.assertEqual(result.returncode, 0, result.stderr)
receipt = json.loads(result.stdout)
self.assertEqual(receipt["records"], {"samples": 1, "tombstones": 1, "activeSamples": 1})
manifest = json.loads(self.manifest.read_text(encoding="utf-8"))
self.assertEqual(manifest["records"], {"samples": 1, "tombstones": 1, "activeSamples": 1})
self.assertEqual(manifest["label"], "nightly")
def test_backup_refuses_to_follow_a_symlink_out_of_the_data_root(self) -> None:
source = build_source(self.tmp)
(source / "images" / "escape.jpg").symlink_to("/etc/hostname")
result = self.backup(source)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertNotIn("payload/images/escape.jpg", members(self.archive))
manifest = json.loads(self.manifest.read_text(encoding="utf-8"))
excluded = {entry["path"]: entry["reason"] for entry in manifest["excluded"]}
self.assertEqual(excluded["images/escape.jpg"], "symlink")
def test_backup_refuses_missing_source_and_existing_archive(self) -> None:
missing = run(["backup", "--source", str(self.tmp / "nope"), "--archive", str(self.archive)])
self.assertNotEqual(missing.returncode, 0)
self.assertIn("is not a directory", missing.stderr)
source = build_source(self.tmp)
self.assertEqual(self.backup(source).returncode, 0)
again = self.backup(source)
self.assertNotEqual(again.returncode, 0)
self.assertIn("already exists", again.stderr)
def test_verify_and_restore_round_trip_the_records(self) -> None:
source = build_source(self.tmp, with_tombstone=True)
self.assertEqual(self.backup(source).returncode, 0)
verified = run(["verify", "--archive", str(self.archive), "--manifest", str(self.manifest)])
self.assertEqual(verified.returncode, 0, verified.stderr)
target = self.tmp / "restored"
restored = run([
"restore", "--archive", str(self.archive), "--manifest", str(self.manifest),
"--target", str(target),
])
self.assertEqual(restored.returncode, 0, restored.stderr)
receipt = json.loads(restored.stdout)
self.assertEqual(receipt["records"], {"samples": 1, "tombstones": 1, "activeSamples": 1})
self.assertEqual((target / "manifest.jsonl").read_bytes(), (source / "manifest.jsonl").read_bytes())
self.assertEqual((target / "images" / "sample-1.jpg").read_bytes(), IMAGE_BYTES)
# The manifest is append-only history: the tombstone line survives the round trip verbatim.
lines = [json.loads(line) for line in (target / "manifest.jsonl").read_text().splitlines() if line.strip()]
self.assertEqual([line["recordType"] for line in lines], ["sample", "tombstone"])
def test_verify_refuses_tampered_member_before_a_restore_writes_anything(self) -> None:
source = build_source(self.tmp)
self.assertEqual(self.backup(source).returncode, 0)
manifest = json.loads(self.manifest.read_text(encoding="utf-8"))
tampered = b"\xff\xd8\xff\xe0synthetic-jpeg-payload-not-a-real-photO"
craft_archive(self.archive, [
("payload/images/sample-1.jpg", tampered),
("payload/manifest.jsonl", (source / "manifest.jsonl").read_bytes()),
])
verified = run(["verify", "--archive", str(self.archive), "--manifest", str(self.manifest)])
self.assertNotEqual(verified.returncode, 0)
self.assertIn("does not match", verified.stderr)
self.assertEqual(manifest["files"][0]["sha256"], digest(IMAGE_BYTES))
target = self.tmp / "restored-tampered"
restored = run([
"restore", "--archive", str(self.archive), "--manifest", str(self.manifest),
"--target", str(target),
])
self.assertNotEqual(restored.returncode, 0)
self.assertFalse(target.exists())
def test_verify_refuses_a_member_the_manifest_never_listed(self) -> None:
source = build_source(self.tmp)
self.assertEqual(self.backup(source).returncode, 0)
craft_archive(self.archive, [
("payload/images/sample-1.jpg", IMAGE_BYTES),
("payload/manifest.jsonl", (source / "manifest.jsonl").read_bytes()),
("payload/extra.bin", b"unlisted"),
])
verified = run(["verify", "--archive", str(self.archive), "--manifest", str(self.manifest)])
self.assertNotEqual(verified.returncode, 0)
self.assertIn("not listed in the manifest", verified.stderr)
def test_restore_refuses_traversal_and_absolute_members(self) -> None:
for name, needle in (("payload/../../etc/passwd", "outside"), ("/etc/passwd", "absolute")):
with self.subTest(member=name):
archive = self.tmp / "hostile.tar.gz"
manifest_path = self.tmp / "hostile.manifest.json"
payload = b"root:x:0:0"
craft_archive(archive, [(name, payload)])
manifest_path.write_text(json.dumps({
"schemaVersion": 1,
"files": [{"path": "images/sample-1.jpg", "sha256": digest(payload), "bytes": len(payload)}],
"records": {"samples": 0, "tombstones": 0, "activeSamples": 0},
}), encoding="utf-8")
target = self.tmp / f"hostile-out-{needle}"
restored = run([
"restore", "--archive", str(archive), "--manifest", str(manifest_path),
"--target", str(target),
])
self.assertNotEqual(restored.returncode, 0)
self.assertIn(needle, restored.stderr.lower())
self.assertFalse(target.exists())
def test_restore_refuses_link_members(self) -> None:
archive = self.tmp / "link.tar.gz"
manifest_path = self.tmp / "link.manifest.json"
craft_archive(archive, [("payload/manifest.jsonl", b"{}\n")], link="payload/images/sample-1.jpg")
manifest_path.write_text(json.dumps({
"schemaVersion": 1,
"files": [{"path": "manifest.jsonl", "sha256": digest(b"{}\n"), "bytes": 3}],
"records": {"samples": 0, "tombstones": 0, "activeSamples": 0},
}), encoding="utf-8")
target = self.tmp / "link-out"
restored = run([
"restore", "--archive", str(archive), "--manifest", str(manifest_path), "--target", str(target),
])
self.assertNotEqual(restored.returncode, 0)
self.assertIn("link", restored.stderr)
self.assertFalse(target.exists())
def test_restore_refuses_a_non_empty_target(self) -> None:
source = build_source(self.tmp)
self.assertEqual(self.backup(source).returncode, 0)
target = self.tmp / "occupied"
target.mkdir()
(target / "keep.txt").write_text("operator file that must not be overwritten", encoding="utf-8")
restored = run([
"restore", "--archive", str(self.archive), "--manifest", str(self.manifest),
"--target", str(target),
])
self.assertNotEqual(restored.returncode, 0)
self.assertIn("not empty", restored.stderr)
self.assertEqual((target / "keep.txt").read_text(encoding="utf-8"), "operator file that must not be overwritten")
def test_backup_is_reproducible_for_the_same_records(self) -> None:
source = build_source(self.tmp, with_tombstone=True)
self.assertEqual(self.backup(source).returncode, 0)
first = json.loads(self.manifest.read_text(encoding="utf-8"))
first_digest = hashlib.sha256(self.archive.read_bytes()).hexdigest()
self.archive.unlink()
self.manifest.unlink()
self.assertEqual(self.backup(source).returncode, 0)
second = json.loads(self.manifest.read_text(encoding="utf-8"))
self.assertEqual(first["files"], second["files"])
self.assertEqual(first["excluded"], second["excluded"])
self.assertEqual(first_digest, hashlib.sha256(self.archive.read_bytes()).hexdigest())
self.assertNotEqual(first["createdAt"], "")
if __name__ == "__main__":
unittest.main(verbosity=2)
+11
View File
@@ -202,3 +202,14 @@ test('the calibration harness is wired into the unit and syntax gates', async ()
assert.match(builder, /tests\/calibration-harness\.test\.py/);
assert.match(builder, /"calibration_harness_gates": "passed"/);
});
test('CI runs the RC attack suite so its evidence is reproducible on every change', async () => {
const workflow = await readFile(workflowPath, 'utf8');
const packageJson = JSON.parse(await readFile(packagePath, 'utf8'));
// Issue #40: the attack report is evidence, not a claim. CI has to be able to regenerate it.
assert.match(workflow, /npm run test:security-attack/);
assert.equal(packageJson.scripts['test:security-attack'], 'node tests/security-attack.acceptance.mjs');
assert.match(packageJson.scripts.test, /tests\/release-security-gate\.test\.js/);
assert.match(packageJson.scripts['check:syntax'], /node --check src\/release-security-gate\.js/);
});
+1
View File
@@ -0,0 +1 @@
<!doctype html><html><body>This file is named .jpg but it is not an image. Synthetic fixture for the image-ingress refusal path.</body></html>
+58
View File
@@ -0,0 +1,58 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { PassThrough } from 'node:stream';
import { BodyError, readJsonBody } from '../src/http-body.js';
const bodyStream = chunks => {
const stream = new PassThrough();
for (const chunk of chunks) stream.write(chunk);
stream.end();
return stream;
};
test('reads a bounded JSON request body', async () => {
const parsed = await readJsonBody(bodyStream([Buffer.from('{"consent":'), Buffer.from('true}')]), { maxBytes: 1024 });
assert.deepEqual(parsed, { consent: true });
});
test('refuses a body above the byte limit instead of buffering it', async () => {
await assert.rejects(
() => readJsonBody(bodyStream([Buffer.alloc(64, 0x61)]), { maxBytes: 16 }),
error => {
assert.ok(error instanceof BodyError);
assert.equal(error.code, 'too_large');
assert.equal(error.status, 413);
return true;
},
);
});
test('refuses malformed JSON with a client error', async () => {
await assert.rejects(
() => readJsonBody(bodyStream([Buffer.from('{not json')]), { maxBytes: 1024 }),
error => {
assert.ok(error instanceof BodyError);
assert.equal(error.code, 'invalid_json');
assert.equal(error.status, 400);
return true;
},
);
});
test('gives up on a body that never finishes and stops reading it', async () => {
const stalled = new PassThrough();
stalled.write(Buffer.from('{"consent":true'));
const started = Date.now();
await assert.rejects(
() => readJsonBody(stalled, { maxBytes: 1024, timeoutMs: 40 }),
error => {
assert.ok(error instanceof BodyError);
assert.equal(error.code, 'body_timeout');
assert.equal(error.status, 408);
return true;
},
);
assert.ok(Date.now() - started < 2_000, 'the deadline must fire promptly');
assert.equal(stalled.isPaused(), true, 'the stalled body must stop being read after the deadline');
});
+175
View File
@@ -0,0 +1,175 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { IngressError, inspectImageBytes, MAX_EDGE, MAX_PIXELS } from '../src/image-ingress.js';
// Every fixture built here is synthetic container bytes assembled in code. No photograph of any
// person or specimen is used, and nothing is fetched.
const PNG_SIGNATURE = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
function pngChunk(type, data = Buffer.alloc(0)) {
const length = Buffer.alloc(4);
length.writeUInt32BE(data.length);
const body = Buffer.concat([Buffer.from(type, 'ascii'), data]);
// The parser walks chunk lengths to find the end of the container; the CRC is not read, so a
// synthetic zero CRC is enough to be structurally complete.
return Buffer.concat([length, body, Buffer.alloc(4)]);
}
function png({ width = 1200, height = 900, extraChunks = [], tail = Buffer.alloc(0) } = {}) {
const ihdr = Buffer.alloc(13);
ihdr.writeUInt32BE(width, 0);
ihdr.writeUInt32BE(height, 4);
ihdr[8] = 8; // bit depth
ihdr[9] = 2; // truecolour
return Buffer.concat([
PNG_SIGNATURE,
pngChunk('IHDR', ihdr),
...extraChunks,
pngChunk('IDAT', Buffer.from([0x78, 0x9c, 0x01])),
pngChunk('IEND'),
tail,
]);
}
function jpeg({ width = 1200, height = 900, segments = [], tail = Buffer.alloc(0) } = {}) {
const sof = Buffer.alloc(17);
sof.writeUInt16BE(0xffc0, 0);
sof.writeUInt16BE(15, 2); // payload length
sof[4] = 8; // precision
sof.writeUInt16BE(height, 5);
sof.writeUInt16BE(width, 7);
sof[9] = 3; // component count
for (let i = 0; i < 3; i += 1) {
sof[10 + i * 3] = i + 1;
sof[11 + i * 3] = 0x11;
sof[12 + i * 3] = 0;
}
return Buffer.concat([
Buffer.from([0xff, 0xd8]),
...segments,
sof,
Buffer.from([0xff, 0xd9]),
tail,
]);
}
function exifSegment() {
const payload = Buffer.concat([
Buffer.from('Exif', 'ascii'),
Buffer.from([0x00, 0x00]),
Buffer.from('GPSLatitude=0.000', 'ascii'),
]);
const header = Buffer.alloc(4);
header.writeUInt16BE(0xffe1, 0);
header.writeUInt16BE(payload.length + 2, 2);
return Buffer.concat([header, payload]);
}
function webp({ width = 1200, height = 900 } = {}) {
// VP8X canvas payload: flags(1) + reserved(3) + width-1(3, LE) + height-1(3, LE).
const payload = Buffer.alloc(10);
payload[0] = 0;
payload.writeUIntLE(width - 1, 4, 3);
payload.writeUIntLE(height - 1, 7, 3);
const chunk = Buffer.alloc(8);
chunk.write('VP8X', 0, 'ascii');
chunk.writeUInt32LE(payload.length, 4);
const body = Buffer.concat([Buffer.from('WEBP', 'ascii'), chunk, payload]);
const riff = Buffer.alloc(8);
riff.write('RIFF', 0, 'ascii');
riff.writeUInt32LE(body.length, 4);
return Buffer.concat([riff, body]);
}
const expectRefusal = (bytes, declaredMime, pattern) => {
assert.throws(
() => inspectImageBytes(bytes, { declaredMime }),
error => {
assert.ok(error instanceof IngressError, `expected an IngressError, got ${error?.name}`);
assert.match(error.message, pattern);
return true;
},
);
};
test('refuses a file whose bytes are not a JPEG, PNG, or WebP', () => {
expectRefusal(Buffer.from('<html><script>alert(1)</script></html>', 'utf8'), 'image/jpeg', /not a JPEG, PNG, or WebP photo/i);
});
test('refuses a PNG that claims to be a JPEG', () => {
expectRefusal(png(), 'image/jpeg', /not the format it claims/i);
});
test('refuses a JPEG that claims to be a PNG', () => {
expectRefusal(jpeg(), 'image/png', /not the format it claims/i);
});
test('refuses a truncated container instead of forwarding a partial image', () => {
expectRefusal(PNG_SIGNATURE.subarray(0, 8), 'image/png', /incomplete or damaged/i);
expectRefusal(Buffer.from([0xff, 0xd8, 0xff, 0xe0]), 'image/jpeg', /incomplete or damaged/i);
});
test('refuses a PNG with bytes appended after its end marker', () => {
expectRefusal(png({ tail: Buffer.from('PK\u0003\u0004zip-appended', 'binary') }), 'image/png', /extra data after the image/i);
});
test('refuses a JPEG with bytes appended after its end marker', () => {
expectRefusal(jpeg({ tail: Buffer.from('<script>alert(1)</script>', 'utf8') }), 'image/jpeg', /extra data after the image/i);
});
test('refuses a JPEG that still carries an EXIF segment', () => {
expectRefusal(jpeg({ segments: [exifSegment()] }), 'image/jpeg', /embedded metadata/i);
});
test('refuses a PNG that carries an eXIf chunk', () => {
const exifChunk = pngChunk('eXIf', Buffer.from('MM\u0000*metadata', 'binary'));
expectRefusal(png({ extraChunks: [exifChunk] }), 'image/png', /embedded metadata/i);
});
test('accepts a clean JPEG and reports its real dimensions', () => {
const intake = inspectImageBytes(jpeg({ width: 1200, height: 900 }), { declaredMime: 'image/jpeg' });
assert.equal(intake.mime, 'image/jpeg');
assert.equal(intake.width, 1200);
assert.equal(intake.height, 900);
assert.equal(intake.bytes, jpeg({ width: 1200, height: 900 }).length);
});
test('accepts a clean PNG and reports its real dimensions', () => {
const intake = inspectImageBytes(png({ width: 800, height: 600 }), { declaredMime: 'image/png' });
assert.equal(intake.mime, 'image/png');
assert.equal(intake.width, 800);
assert.equal(intake.height, 600);
});
test('accepts a clean WebP and reports its canvas dimensions', () => {
const intake = inspectImageBytes(webp({ width: 640, height: 480 }), { declaredMime: 'image/webp' });
assert.equal(intake.mime, 'image/webp');
assert.equal(intake.width, 640);
assert.equal(intake.height, 480);
});
test(`refuses an edge longer than ${MAX_EDGE} pixels even when the file is small`, () => {
expectRefusal(png({ width: MAX_EDGE + 1, height: 200 }), 'image/png', /too large in pixels/i);
});
test(`refuses an image above ${MAX_PIXELS} pixels when each edge fits`, () => {
const edge = 4096;
expectRefusal(jpeg({ width: edge, height: Math.ceil(MAX_PIXELS / edge) + 1 }), 'image/jpeg', /too large in pixels/i);
});
test('never repeats image bytes back to the caller in a refusal message', () => {
const marker = 'SECRETPIXELMARKER';
try {
inspectImageBytes(Buffer.from(`not an image ${marker}`, 'utf8'), { declaredMime: 'image/jpeg' });
assert.fail('expected a refusal');
} catch (error) {
assert.ok(error instanceof IngressError);
assert.doesNotMatch(error.message, new RegExp(marker));
assert.doesNotMatch(error.message, /[A-Za-z0-9+/=]{40,}/);
}
});
test('refuses an empty body with a bounded message', () => {
expectRefusal(Buffer.alloc(0), 'image/jpeg', /empty/i);
});
+70
View File
@@ -0,0 +1,70 @@
import { chromium } from 'playwright';
import assert from 'node:assert/strict';
// Issue #16, as a user experiences it: a file that only claims to be a photo never becomes an upload,
// never reaches the model worker, and still leaves manual logging one tap away. The second half of
// this suite asks the real running server to refuse the same bytes, so the browser copy and the
// server boundary are proven in the same run.
const browser = await chromium.launch({ headless: true });
const context = await browser.newContext({
viewport: { width: 390, height: 844 },
deviceScaleFactor: 2,
serviceWorkers: 'block',
});
const page = await context.newPage();
let analysisRequests = 0;
const analyzePayloads = [];
await page.route('**/api/vision-status', route => route.fulfill({
status: 200,
contentType: 'application/json',
body: JSON.stringify({ enabled: true, profile: 'selfhost', providerReady: true, modelSeen: true, model: 'synthetic-test-model' }),
}));
await page.route('**/api/analyze', route => {
analysisRequests += 1;
analyzePayloads.push(route.request().postData() || '');
return route.continue();
});
await page.goto('http://127.0.0.1:4173', { waitUntil: 'networkidle' });
await page.locator('[data-scan]').click();
const spoofed = Buffer.from('<html><script>alert(1)</script></html>', 'utf8');
await page.locator('#gallery-photo').setInputFiles({ name: 'photo.jpg', mimeType: 'image/jpeg', buffer: spoofed });
const refusal = page.locator('[data-capture-error="unreadable"]');
await refusal.waitFor();
const copy = await refusal.innerText();
assert.match(copy, /isn.t a photo Timmy can read/i, 'the refusal names what is wrong with the file');
assert.match(copy, /Nothing was uploaded/i, 'the refusal states that nothing left the device');
assert.equal(await page.getByText(/couldn.t analyze that safely/i).count(), 0,
'an unreadable file must not be reported as a failed analysis');
assert.equal(analysisRequests, 0, 'an unreadable file must never be posted to /api/analyze');
assert.equal(await page.locator('#manual-from-scan').isVisible(), true, 'manual entry stays one tap away');
assert.equal(await page.locator('#retake-photo').isVisible(), true, 'the picker stays reachable');
// The same bytes, offered to the running server directly, are refused before any provider work.
const serverResponse = await page.evaluate(async payload => {
const response = await fetch(`${document.querySelector('meta[name="timmy-base-path"]')?.content.replace(/\/$/, '') || ''}/api/analyze`, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify(payload),
});
return { status: response.status, body: await response.json() };
}, { consent: true, imageDataUrl: `data:image/jpeg;base64,${spoofed.toString('base64')}` });
assert.equal(serverResponse.status, 400, `the server must refuse non-image bytes, saw ${serverResponse.status}`);
assert.match(serverResponse.body.error, /not a JPEG, PNG, or WebP photo/i);
// Only this deliberate probe reached the endpoint; the app itself never posted the unreadable file.
assert.equal(analysisRequests, 1, 'only the deliberate server probe may reach /api/analyze so far');
assert.match(analyzePayloads[0], /"consent":true/);
// A real (synthetic) photo still reaches the consent step, so the gate is a boundary and not a wall.
await page.locator('#retake-photo').click();
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/synthetic-type4.jpg');
await page.locator('#ai-consent').waitFor();
assert.equal(await page.locator('[data-capture-error="unreadable"]').count(), 0);
assert.equal(analysisRequests, 1, 'a photo that is only selected is still not uploaded before consent');
await context.close();
await browser.close();
console.log('PASS unreadable photo bytes are refused on device and at the server, with manual entry intact');
+5 -2
View File
@@ -47,8 +47,11 @@ for (const viewport of viewports) {
mimeType: 'image/jpeg',
buffer: Buffer.from('not an image'),
});
assert.match(await page.locator('.scan-result').innerText(), /could not be read/i);
await page.getByText('Try another photo', { exact: true }).click();
assert.match(await page.locator('.scan-result').innerText(), /isn.t a photo Timmy can read/i);
// Issue #16: the refusal is its own state and states plainly that nothing left the device, instead
// of borrowing the "couldn't analyze that safely" copy from a failed upload.
assert.match(await page.locator('.scan-result').innerText(), /Nothing was uploaded/i);
await page.getByText('Choose another photo', { exact: true }).click();
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/synthetic-type4.jpg');
assert.equal(await page.locator('#analyze-photo').isDisabled(), true);
assert.equal(await page.locator('#retake-photo').isVisible(), true);
+161
View File
@@ -0,0 +1,161 @@
#!/usr/bin/env python3
"""Tests for the production topology, secrets, backup, and rollback runbook.
A runbook is only worth reading if its commands are the ones the tools actually accept, so this
suite parses every documented ``python3 scripts/...`` command out of the markdown and checks each
flag against the real CLI help. It also fails if the document starts carrying a real secret, or if
it keeps claiming the server holds no journal after ``server.mjs`` starts writing one.
"""
from __future__ import annotations
import re
import shlex
import subprocess
import sys
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
RUNBOOK = ROOT / "docs" / "PRODUCTION-RUNBOOK.md"
SERVER = ROOT / "server.mjs"
BACKUP = ROOT / "scripts" / "operator_backup.py"
FENCE = re.compile(r"```bash\n(.*?)```", re.S)
SECRET_ASSIGNMENT = re.compile(
r"(?i)(?:token|secret|password|passwd|api[_-]?key|salt)\s*=\s*(?!<|\$|\"|\'|\.\.\.)[^\s`\"']{6,}"
)
def text() -> str:
return RUNBOOK.read_text(encoding="utf-8")
def sections() -> dict[str, str]:
body = text()
headings = list(re.finditer(r"^## (.+)$", body, re.M))
out: dict[str, str] = {}
for index, match in enumerate(headings):
end = headings[index + 1].start() if index + 1 < len(headings) else len(body)
out[match.group(1).strip().lower()] = body[match.end():end]
return out
def documented_commands() -> list[str]:
commands: list[str] = []
for block in FENCE.findall(text()):
for line in block.splitlines():
line = line.strip()
if line.startswith("python3 scripts/"):
commands.append(line)
return commands
def help_text(script: str, subcommand: str | None) -> str:
argv = [sys.executable, script] + ([subcommand] if subcommand else []) + ["--help"]
result = subprocess.run(argv, cwd=ROOT, capture_output=True, text=True)
assert result.returncode == 0, f"{' '.join(argv)} failed: {result.stderr}"
return result.stdout
class ProductionRunbookTests(unittest.TestCase):
def test_runbook_covers_topology_network_boundaries_secrets_backup_restore_and_rollback(self) -> None:
self.assertTrue(RUNBOOK.is_file(), "docs/PRODUCTION-RUNBOOK.md must exist")
titles = sections().keys()
for required in ("topology", "network boundaries", "secrets", "backup", "restore", "rollback"):
self.assertTrue(
any(title.startswith(required) for title in titles),
f"runbook is missing a '{required}' section; found {sorted(titles)}",
)
def test_backup_section_names_what_is_never_archived_and_why(self) -> None:
backup = next(body for title, body in sections().items() if title.startswith("backup"))
self.assertIn("scripts/operator_backup.py", backup)
for required in ("model weights", "transient inference", "secret", "tombstone"):
self.assertIn(required, backup.lower(), f"backup section must name {required}")
self.assertRegex(backup, r"manifest\.jsonl")
def test_every_documented_script_command_is_accepted_by_the_real_cli(self) -> None:
commands = documented_commands()
self.assertGreaterEqual(len(commands), 4, f"expected documented commands, found {commands}")
for command in commands:
with self.subTest(command=command):
tokens = shlex.split(command)
script = tokens[1]
rest = tokens[2:]
subcommand = rest[0] if rest and not rest[0].startswith("-") else None
combined = help_text(script, subcommand) + help_text(script, None)
for token in rest:
if token.startswith("--"):
self.assertIn(token, combined, f"{script} does not accept {token}")
for token in rest[1:]:
if not token.startswith("--"):
continue
def test_documented_backup_commands_are_the_ones_the_tool_runs(self) -> None:
commands = " ".join(documented_commands())
for flag in ("--source", "--archive", "--manifest", "--target"):
self.assertIn(flag, commands, f"the runbook must show the real {flag} flag")
self.assertIn("verify", commands)
self.assertIn("restore", commands)
def test_rollback_is_documented_as_a_tested_fixture_path_and_a_human_gated_live_drill(self) -> None:
rollback = next(body for title, body in sections().items() if title.startswith("rollback"))
self.assertIn("scripts/deploy_staging.py rollback", rollback)
self.assertRegex(rollback.lower(), r"human[- ]gate")
self.assertIn("#53", rollback)
self.assertIn("tests/staging-deploy.test.py", rollback)
# The live drill must be explicit about not being run by automation (wrapped lines are fine).
self.assertRegex(
rollback.lower(),
r"(?s)not .{0,80}automatically|does not run .{0,80}automatically",
"the runbook must say the live drill is not run automatically",
)
def test_network_boundaries_match_how_the_server_and_worker_actually_bind(self) -> None:
server = SERVER.read_text(encoding="utf-8")
# The app's own default is not loopback, so the runbook must say the deployment sets it.
self.assertRegex(server, r"HOST\|\|'0\.0\.0\.0'")
staging_env = (ROOT / "deploy" / "timmy-staging.env.example").read_text(encoding="utf-8")
self.assertIn("HOST=127.0.0.1", staging_env)
worker = (ROOT / "deploy" / "worker" / "run_worker_container.sh").read_text(encoding="utf-8")
self.assertIn("Refusing to publish the worker outside loopback", worker)
boundaries = next(body for title, body in sections().items() if title.startswith("network boundaries"))
self.assertIn("0.0.0.0", boundaries, "the runbook must name the app's real default bind address")
self.assertIn("HOST=127.0.0.1", boundaries)
self.assertIn("deploy/timmy-staging.env.example", boundaries)
self.assertIn("loopback", boundaries.lower())
self.assertIn("8080", boundaries)
def test_runbook_carries_no_literal_secret_values(self) -> None:
findings = [match.group(0) for match in SECRET_ASSIGNMENT.finditer(text())]
self.assertEqual(findings, [], f"runbook looks like it contains a real secret: {findings}")
self.assertIn("mode-600", text())
self.assertIn("never commit", text().lower())
def test_runbook_claims_match_a_stateless_server(self) -> None:
server = SERVER.read_text(encoding="utf-8")
for call in ("writeFile", "appendFile", "createWriteStream", "mkdirSync"):
self.assertNotIn(call, server, "server.mjs started persisting data; the runbook's data-lifetime claim must be updated")
body = text().lower()
self.assertIn("no server-side copy of the journal", body)
self.assertIn("/api/healthz", body)
# The health probe is allowed to report readiness, never configuration or secrets.
health = re.search(r"'/api/healthz'[^\n]*", server)
self.assertIsNotNone(health, "server.mjs must serve /api/healthz")
health_text = health.group(0) if health else ""
for forbidden in ("token", "secret", "env:", "password"):
self.assertNotIn(forbidden, health_text.lower())
# The response may report readiness only, so a new field is a deliberate decision.
for field in ("ok", "release", "commit", "visionEnabled", "agentEnabled"):
self.assertIn(field, health_text)
def test_backup_tool_is_gated_by_its_own_suite(self) -> None:
package = (ROOT / "package.json").read_text(encoding="utf-8")
self.assertIn("tests/backup-restore.test.py", package)
self.assertIn("tests/production-runbook.test.py", package)
self.assertTrue(BACKUP.is_file())
if __name__ == "__main__":
unittest.main(verbosity=2)
+91
View File
@@ -0,0 +1,91 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { createRateLimiter } from '../src/rate-limit.js';
// A clock the test owns, so no test sleeps and every window boundary is exact.
function manualClock(start = 1_700_000_000_000) {
let current = start;
return { now: () => current, advance: ms => { current += ms; } };
}
test('allows requests up to the limit and refuses the next one', () => {
const clock = manualClock();
const limiter = createRateLimiter({ limit: 3, windowMs: 60_000, now: clock.now });
for (let attempt = 1; attempt <= 3; attempt += 1) {
const verdict = limiter.take('device-a');
assert.equal(verdict.allowed, true, `attempt ${attempt} must be allowed`);
assert.equal(verdict.remaining, 3 - attempt);
}
const refused = limiter.take('device-a');
assert.equal(refused.allowed, false);
assert.equal(refused.remaining, 0);
assert.equal(refused.retryAfterMs, 60_000);
});
test('counts each caller on its own budget', () => {
const clock = manualClock();
const limiter = createRateLimiter({ limit: 1, windowMs: 60_000, now: clock.now });
assert.equal(limiter.take('device-a').allowed, true);
assert.equal(limiter.take('device-a').allowed, false);
assert.equal(limiter.take('device-b').allowed, true);
});
test('lets a caller back in once its window has passed', () => {
const clock = manualClock();
const limiter = createRateLimiter({ limit: 1, windowMs: 60_000, now: clock.now });
assert.equal(limiter.take('device-a').allowed, true);
assert.equal(limiter.take('device-a').allowed, false);
clock.advance(60_001);
const allowedAgain = limiter.take('device-a');
assert.equal(allowedAgain.allowed, true);
assert.equal(allowedAgain.remaining, 0);
});
test('a refused request does not extend the block', () => {
const clock = manualClock();
const limiter = createRateLimiter({ limit: 1, windowMs: 1_000, now: clock.now });
assert.equal(limiter.take('device-a').allowed, true);
clock.advance(400);
assert.equal(limiter.take('device-a').allowed, false);
clock.advance(601);
// The original hit is now 1001ms old, so the window has moved on even though the caller hammered
// the endpoint in between. A refusal must never buy a longer penalty than the caller earned.
assert.equal(limiter.take('device-a').allowed, true);
});
test('reports the wait from the oldest hit still inside the window', () => {
const clock = manualClock();
const limiter = createRateLimiter({ limit: 2, windowMs: 10_000, now: clock.now });
limiter.take('device-a');
clock.advance(4_000);
limiter.take('device-a');
clock.advance(1_000);
const refused = limiter.take('device-a');
assert.equal(refused.allowed, false);
assert.equal(refused.retryAfterMs, 5_000);
});
test('keeps its caller map bounded when many callers appear', () => {
const clock = manualClock();
const limiter = createRateLimiter({ limit: 5, windowMs: 60_000, maxKeys: 8, now: clock.now });
for (let index = 0; index < 50; index += 1) limiter.take(`device-${index}`);
assert.ok(limiter.size() <= 8, `expected at most 8 tracked callers, saw ${limiter.size()}`);
});
test('prunes expired callers instead of growing for the life of the process', () => {
const clock = manualClock();
const limiter = createRateLimiter({ limit: 5, windowMs: 1_000, now: clock.now });
for (let index = 0; index < 20; index += 1) limiter.take(`device-${index}`);
clock.advance(1_001);
limiter.take('device-fresh');
assert.equal(limiter.size(), 1);
});
+72
View File
@@ -113,6 +113,23 @@ test('release demo visibly exercises the red-flag language coverage instead of o
assert.match(builder, /A tarry note already saved in the journal escalates the same way, before any provider call/);
});
test('release demo shows a file that only claims to be a photo being refused before upload', async () => {
const demo = await readFile(demoPath, 'utf8');
const builder = await readFile(builderPath, 'utf8');
// Issue #16: the recording must show the device-side refusal of a non-image, prove nothing was
// posted, and keep manual entry reachable — all asserted in the browser, not narrated.
assert.match(demo, /tests\/fixtures\/not-a-photo\.jpg/);
assert.match(demo, /\[data-capture-error="unreadable"\]/);
assert.match(demo, /Nothing was uploaded/);
assert.match(demo, /An unreadable file was posted to \/api\/analyze/);
assert.match(demo, /A file that only claims to be a photo is refused on the device — nothing is uploaded/);
assert.match(demo, /Manual logging stays one tap away, and the photo picker is one tap back/);
// The caption is a required release gate, so a later release cannot drop the scene silently.
assert.match(builder, /A file that only claims to be a photo is refused on the device — nothing is uploaded/);
assert.match(builder, /"image_ingress_refusal": "passed"/);
});
test('release builder gates the sleek shell, Hermes chat, and bootstrap syntax', async () => {
const builder = await readFile(builderPath, 'utf8');
assert.match(builder, /"test:sleek"/);
@@ -486,3 +503,58 @@ test('release demo publishes the dataset card from a manifest it builds on camer
assert.match(builder, /"dataset_card_generated": "passed"/);
assert.match(builder, /scripts\/dataset_card\.py/);
});
test('release demo runs the operator backup and restore on synthetic records instead of describing it', async () => {
const demo = await readFile(demoPath, 'utf8');
const builder = await readFile(builderPath, 'utf8');
// Issue #39: the backup is a CLI, so the recording runs the real tool and shows what it stored,
// what it refused, and what the restore returned.
assert.match(demo, /scripts\/operator_backup\.py/);
assert.match(demo, /'backup', '--source'/);
assert.match(demo, /'verify', '--archive'/);
assert.match(demo, /'restore', '--archive'/);
assert.match(demo, /id: 'production-backup'/);
assert.match(demo, /Production backup card is not fully readable on screen/);
// The exclusions are proven by the run, not asserted by a caption: the decoys are real files.
assert.match(demo, /model-weights/);
assert.match(demo, /transient-inference/);
assert.match(demo, /must not archive the weight shard or a transient inference upload/);
// A restore must return the append-only manifest byte for byte, tombstone included.
assert.match(demo, /must return the same records, tombstone included/);
assert.match(demo, /The operator backup is a command line, not a screen in the app/);
assert.match(demo, /Model weights and transient inference images are refused by name, and every skipped path is listed with its reason/);
assert.match(demo, /Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes/);
assert.match(demo, /A restore returns the same records, including the withdrawal tombstone a trainer must never lose/);
// The scene is a release gate, and the tool's own suite runs before the release exists.
assert.match(demo, /process\.env\.TIMMY_DEMO_URL \|\| 'http:\/\/127\.0\.0\.1:4173'/);
assert.match(builder, /tests\/backup-restore\.test\.py/);
assert.match(builder, /tests\/production-runbook\.test\.py/);
assert.match(builder, /"production_backup_restore": "passed"/);
assert.match(builder, /"production_runbook_commands": "passed"/);
assert.match(builder, /The operator backup is a command line, not a screen in the app/);
assert.match(builder, /A restore returns the same records, including the withdrawal tombstone a trainer must never lose/);
});
test('release demo runs the RC attack suite on camera and prints the verdict it derived', async () => {
const demo = await readFile(demoPath, 'utf8');
// Issue #40 turns hostile uploads into a release gate. The scene must run the real suite, not
// describe it, and the card must show the verdict the suite derived rather than a flattering one.
assert.match(demo, /scripts\/security_attack_suite\.mjs', '--json'/);
assert.match(demo, /SECURITY ATTACK SUITE/);
assert.match(demo, /Every hostile upload on screen went at the real server, and the worker behind it counted every call it received/);
assert.match(demo, /A failing case is a finding the release job has to see, so the card prints the verdict the suite derived/);
assert.match(demo, /The gate blocks RC approval on any P0 case that reached the worker/);
assert.match(demo, /securityReport\.verdict/);
assert.match(demo, /securityCardProof/);
// The finding must name where the fix lives instead of implying the boundary is already safe.
assert.match(demo, /Refusing bytes by their content is the fix in review, not a claim this video makes/);
// Each case row prints the fields it actually observed. A guarantee that the card never renders
// `http undefined` (a case with no status) or the bare status again (a case with no provider calls).
assert.match(demo, /'refused' in observed/);
assert.match(demo, /'providerTextLeaked' in observed/);
assert.doesNotMatch(demo, /http \$\{observed\.status\} · worker \$\{observed\.providerCalls\}`/);
});
+104
View File
@@ -0,0 +1,104 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { buildReport, containsImageBytes, evaluateCase, latencyPercentiles, redactImageBytes, releaseVerdict } from '../src/release-security-gate.js';
test('latency percentiles use nearest-rank so p95 is an observed sample, not an interpolation', () => {
const samples = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100];
assert.deepEqual(latencyPercentiles(samples), { count: 10, p50: 50, p95: 100, max: 100 });
});
// A case is a promise the suite makes before it runs: this input must land here, and the provider
// must (or must not) see exactly this many calls. The verdict is derived, never typed.
const polyglot = {
id: 'ingress-polyglot-html',
severity: 'P0',
expected: { status: 400, providerCalls: 0 },
observed: { status: 400, providerCalls: 0 },
};
test('a case whose observation matches its expectation passes', () => {
assert.equal(evaluateCase(polyglot).status, 'pass');
});
test('a case that reached the provider is a failure that names the field that broke', () => {
const result = evaluateCase({ ...polyglot, observed: { status: 200, providerCalls: 1 } });
assert.equal(result.status, 'fail');
assert.match(result.mismatch, /status 400 expected, observed 200/);
assert.match(result.mismatch, /providerCalls 0 expected, observed 1/);
});
test('a P0 failure blocks the release, and a P1 failure only holds it', () => {
const leaked = { ...polyglot, observed: { status: 200, providerCalls: 1 } };
const blocked = releaseVerdict({ cases: [leaked, { ...polyglot, id: 'flood', severity: 'P1', observed: { status: 200, providerCalls: 12 } }] });
assert.equal(blocked.verdict, 'BLOCKED');
assert.deepEqual(blocked.blocking.map(item => item.id), ['ingress-polyglot-html']);
assert.deepEqual(blocked.holds.map(item => item.id), ['flood']);
});
test('a suite with no failures is a GO, and counts say how many cases were proven', () => {
const verdict = releaseVerdict({ cases: [polyglot] });
assert.equal(verdict.verdict, 'GO');
assert.deepEqual(verdict.blocking, []);
assert.equal(verdict.proven, 1);
});
test('a required category that no case exercised blocks the release instead of passing by omission', () => {
const deletion = { id: 'deletion-withdrawal', severity: 'P0', surface: 'out-of-surface', evidence: 'tests/training-contribution.test.js', evidenceVerified: false };
const verdict = releaseVerdict({ cases: [polyglot, deletion] });
assert.equal(verdict.verdict, 'BLOCKED');
assert.deepEqual(verdict.unproven.map(item => item.id), ['deletion-withdrawal']);
});
test('coverage borrowed from another test file only counts once the runner verified that file exists', () => {
const verified = { id: 'deletion-withdrawal', severity: 'P0', surface: 'out-of-surface', evidence: 'tests/training-contribution.test.js', evidenceVerified: true };
const verdict = releaseVerdict({ cases: [polyglot, verified] });
assert.equal(verdict.verdict, 'GO');
assert.deepEqual(verdict.coveredElsewhere, ['deletion-withdrawal']);
});
// The report and the server log are shipped as release evidence. Neither may carry a photo.
const JPEG_DATA_URI = 'data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/wAALCAABAAEBAREA/8QAFQABAQAAAAAAAAAAAAAAAAAAAAn/xAAUEAEAAAAAAAAAAAAAAAAAAAAJ/9oACAEBAAA/AKp//9k=';
test('a data URI or a long base64 run is recognised as image bytes', () => {
assert.equal(containsImageBytes(`posted ${JPEG_DATA_URI} to the worker`), true);
assert.equal(containsImageBytes('chunk ' + 'A'.repeat(64) + 'yQ'.repeat(200)), true);
});
test('ordinary log lines, hex digests, and short tokens are not image bytes', () => {
assert.equal(containsImageBytes('Timmy is listening on http://127.0.0.1:4173 · vision configured · Hermes locked and ready'), false);
assert.equal(containsImageBytes('sha256 ' + 'a1b2c3d4'.repeat(8)), false);
assert.equal(containsImageBytes('case ingress-polyglot-html observed status 200'), false);
});
test('redaction removes the bytes and keeps the sentence around them honest', () => {
const redacted = redactImageBytes(`provider received ${JPEG_DATA_URI} at 12:01`);
assert.equal(containsImageBytes(redacted), false);
assert.match(redacted, /\[redacted image bytes\]/);
assert.match(redacted, /^provider received .* at 12:01$/);
});
test('the report carries the verdict, the target it ran against, and the measured load numbers', () => {
const report = buildReport({
generatedAt: '2026-10-30T10:48:00.000Z',
target: { baseUrl: 'http://127.0.0.1:4173', provider: 'stub counting worker' },
cases: [polyglot],
load: { requests: 8, errors: 0, abstentions: 8, p50: 12, p95: 40, max: 40 },
});
assert.equal(report.schema, 'timmy-rc-security-attack/1');
assert.equal(report.verdict, 'GO');
assert.equal(report.generatedAt, '2026-10-30T10:48:00.000Z');
assert.equal(report.target.provider, 'stub counting worker');
assert.equal(report.load.p95, 40);
assert.equal(containsImageBytes(JSON.stringify(report)), false);
});
test('a case that quotes a response body cannot smuggle the photo into the report', () => {
const report = buildReport({
generatedAt: '2026-10-30T10:48:00.000Z',
target: { baseUrl: 'http://127.0.0.1:4173', provider: 'stub' },
cases: [{ ...polyglot, observed: { status: 400, providerCalls: 0 }, snippet: `refused ${JPEG_DATA_URI}` }],
load: null,
});
assert.equal(containsImageBytes(JSON.stringify(report)), false);
assert.match(JSON.stringify(report), /redacted image bytes/);
});
+89
View File
@@ -0,0 +1,89 @@
// Acceptance: the RC attack suite must drive the real server and then tell the truth about what it
// saw. Nothing here trusts the runner's own summary line: every case status is re-derived from the
// recorded observation with the same rule the release gate uses, and the report is scanned for photo
// bytes. If the suite ever marks a case pass while observing a mismatch, this test fails.
import test from 'node:test';
import assert from 'node:assert/strict';
import { execFile } from 'node:child_process';
import { mkdtemp, readFile, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { containsImageBytes, evaluateCase, releaseVerdict } from '../src/release-security-gate.js';
const repoRoot = new URL('..', import.meta.url).pathname;
async function runSuite() {
const directory = await mkdtemp(join(tmpdir(), 'timmy-rc-attack-'));
const reportPath = join(directory, 'report.json');
const run = await new Promise(resolve => {
execFile(process.execPath, ['scripts/security_attack_suite.mjs', '--json', reportPath], {
cwd: repoRoot,
timeout: 180_000,
}, (error, stdout, stderr) => resolve({
code: error?.code ?? 0,
stdout,
stderr,
}));
});
const report = JSON.parse(await readFile(reportPath, 'utf8'));
await rm(directory, { recursive: true, force: true });
return { run, report };
}
const suite = await runSuite();
const byId = new Map(suite.report.cases.map(item => [item.id, item]));
test('the suite exits non-zero exactly when its own verdict blocks the release', () => {
assert.ok([0, 1].includes(suite.run.code), `unexpected exit ${suite.run.code}: ${suite.run.stderr}`);
assert.match(suite.run.stdout, new RegExp(`VERDICT: ${suite.report.verdict}`));
assert.equal(suite.run.code, suite.report.verdict === 'BLOCKED' ? 1 : 0);
});
test('every reported case status is re-derivable from the observation it recorded', () => {
for (const item of suite.report.cases.filter(entry => entry.surface !== 'out-of-surface')) {
const derived = evaluateCase(item);
assert.equal(item.status, derived.status, `${item.id} claims ${item.status} but observed ${JSON.stringify(item.observed)} against ${JSON.stringify(item.expected)}`);
}
assert.equal(suite.report.verdict, releaseVerdict({ cases: suite.report.cases }).verdict);
});
test('the suite covers the abuse classes the ticket names, not a convenient subset', () => {
const required = ['oversized', 'polyglot', 'malformed', 'flood', 'log', 'deletion', 'replay'];
const categories = new Set(suite.report.cases.map(item => item.category));
for (const category of required) assert.ok(categories.has(category), `no case covers ${category}`);
assert.ok(suite.report.cases.length >= 8, `only ${suite.report.cases.length} cases ran`);
});
test('a request without consent is refused and the worker is never called', () => {
const consent = byId.get('consent-required');
assert.equal(consent.observed.status, 400);
assert.equal(consent.observed.providerCalls, 0);
assert.equal(consent.status, 'pass');
});
test('an oversized body is refused before any provider work', () => {
const oversized = byId.get('ingress-oversized-body');
assert.equal(oversized.observed.status, 413);
assert.equal(oversized.observed.providerCalls, 0);
});
test('a provider answer that breaks the schema fails closed without quoting the provider', () => {
const malformed = byId.get('model-output-malformed');
assert.equal(malformed.observed.status, 503);
assert.equal(malformed.observed.providerTextLeaked, false);
assert.ok(!JSON.stringify(malformed).includes('garbage-completion-token'), 'the report must not quote the provider');
});
test('the captured server log and the report itself carry no photo bytes', () => {
assert.equal(byId.get('log-leak-scan').observed.imageBytesInLog, false);
assert.equal(containsImageBytes(JSON.stringify(suite.report)), false);
});
test('the load phase recorded percentiles, errors, and abstentions without any image body', () => {
const load = suite.report.load;
assert.ok(load.requests >= 8, `only ${load.requests} load requests ran`);
assert.equal(typeof load.p50, 'number');
assert.equal(typeof load.p95, 'number');
assert.equal(typeof load.abstentions, 'number');
assert.equal(containsImageBytes(JSON.stringify(load)), false);
});