Compare commits
12
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8ce5bc174b | ||
|
|
bc64055196 | ||
|
|
1bbcc10a35 | ||
|
|
0bf8230280 | ||
|
|
f0a9a734db | ||
|
|
65a0258bba | ||
|
|
48f0f56835 | ||
|
|
7fe1901703 | ||
|
|
d7026c0ed3 | ||
|
|
2b773ba58c | ||
|
|
9b72872466 | ||
|
|
ba9afe0a1d |
No files matched your search
@@ -32,6 +32,7 @@ jobs:
|
||||
- name: Unit and security tests
|
||||
run: |
|
||||
npm test
|
||||
npm run test:security-attack
|
||||
python3 tests/staging-deploy.test.py -v
|
||||
python3 tests/corpus-manifest.test.py -v
|
||||
python3 tests/corpus-review.test.py -v
|
||||
@@ -55,6 +56,7 @@ jobs:
|
||||
npm run test:ui
|
||||
npm run test:photo
|
||||
npm run test:mobile-capture
|
||||
npm run test:ingress-capture
|
||||
npm run test:sleek
|
||||
npm run test:manual-fallback
|
||||
npm run test:provisional
|
||||
|
||||
@@ -5,3 +5,4 @@ __pycache__/
|
||||
*.pyc
|
||||
.env
|
||||
.env.*
|
||||
artifacts/security/
|
||||
@@ -6,6 +6,7 @@ A working mobile-first bowel diary with an optional **photo-first AI assist**:
|
||||
- [Product and safety boundary](PRODUCT.md)
|
||||
- [Product boundary and release authority decision](docs/PRODUCT-DECISIONS.md)
|
||||
- [Self-hosted stool-vision decision](research/SELF-HOSTED-STOOL-VISION.md)
|
||||
- [Production topology, secrets, backup, and rollback runbook](docs/PRODUCTION-RUNBOOK.md)
|
||||
|
||||
## Build a gated review release
|
||||
|
||||
@@ -28,6 +29,25 @@ sudo python3 scripts/deploy_staging.py rollback --commit PRIOR_FULL_COMMIT
|
||||
|
||||
Commands are fixed argv arrays executed with `shell=False`; JSON argv overrides and `--root` exist for rootless fixtures. See [the staging runbook](docs/STAGING-RUNBOOK.md) for host prerequisites, mode-600 environment injection, Caddy validation, smoke credentials, logs, backups, removal, and the explicit no-live-change boundary.
|
||||
|
||||
## Back up and restore the records an operator holds
|
||||
|
||||
The app keeps the journal in the person's own browser, so the server has no diary to back up. What an
|
||||
operator can lose is the corpus: the append-only `manifest.jsonl` with its provenance records and its
|
||||
withdrawal tombstones, plus the derived images those records point at.
|
||||
|
||||
```bash
|
||||
python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy-nightly.tar.gz --label nightly
|
||||
python3 scripts/operator_backup.py verify --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json
|
||||
python3 scripts/operator_backup.py restore --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json --target /srv/timmy-corpus.restore-check
|
||||
```
|
||||
|
||||
Model weights, transient inference images, secrets, logs, and symlinks are refused by name, and every
|
||||
skipped path is listed in the manifest with its reason. Each stored member carries a SHA-256, the
|
||||
archive is reproducible, and a restore re-verifies every digest and the record counts before it writes
|
||||
anything — a tampered archive or a lost tombstone fails loudly instead of restoring quietly. See the
|
||||
[production runbook](docs/PRODUCTION-RUNBOOK.md) for topology, network boundaries, secrets, retention,
|
||||
and the rollback paths.
|
||||
|
||||
## Run with the self-hosted open-weight path
|
||||
|
||||
```bash
|
||||
@@ -102,6 +122,7 @@ npm run test:grounded-chat # chat answers cite the ledger, or refuse to answe
|
||||
npm run test:photo-lifecycle # photo presence, explicit reveal, remove-photo-keeps-log, delete-all
|
||||
npm run test:calibration # the release-gate harness: gates, threshold protocol, refused corpora
|
||||
npm run test:review # blind review queue, held-back model answer, an append-only ledger, withdrawals
|
||||
npm run test:security-attack # hostile synthetic uploads at the real boundary, with a counting worker (#40)
|
||||
npm audit --audit-level=high
|
||||
```
|
||||
|
||||
@@ -140,6 +161,8 @@ The reviewer tool ([scripts/review_corpus.py](scripts/review_corpus.py), [contra
|
||||
|
||||
What the corpus is, and what a label may assert, is published rather than implied: [docs/DATASET_CARD.md](docs/DATASET_CARD.md) is generated from the manifest by [scripts/dataset_card.py](scripts/dataset_card.py), so every count on it is read back from real records and every fact the pipeline does not collect (devices, contexts, licenses) is printed as `pending` instead of estimated — a card for a corpus where one contributor straddles two splits is refused, not published. [docs/LABELING_GUIDE.md](docs/LABELING_GUIDE.md) is the reviewer's rulebook: visible form, broad visible color, and photo quality only, visible color is not blood, nothing is diagnosed, and a model suggestion is provisional input to a human decision rather than the label.
|
||||
|
||||
The release candidate is attacked before anyone is asked to trust it. [scripts/security_attack_suite.mjs](scripts/security_attack_suite.mjs) drives the real server with synthetic hostile bytes — an oversized body, an HTML polyglot dressed as a JPEG, undecodable bytes, a request without consent, a worker answer that breaks the schema, a byte-identical replay, and a concurrent burst — while a counting stub worker records every call that reaches the model boundary. [docs/RC-SECURITY-EVIDENCE.md](docs/RC-SECURITY-EVIDENCE.md) owns the rule: any failing P0 case blocks RC approval, and a required class with no executable evidence blocks it too. **It currently returns `VERDICT: BLOCKED`** — bytes are still trusted from the data-URL label, so a hostile file reaches the worker before it fails on schema. That is a published finding against [#16](https://forge.alexanderwhitestone.com/git/stackchain/timmy-talking-turd/issues/16), not a passing check.
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
|
||||
@@ -340,20 +340,23 @@ function photoFirstBody(mode,error=''){
|
||||
if(mode==='pick')return `${visionStatusHtml()}<div class="scan-hero"><img src="${appPath('assets/timmy.svg')}" alt="Timmy"><h3>One photo. Two useful suggestions.</h3><p>Timmy can suggest the visible Bristol form and color. A camera cannot know urgency, pain, symptoms, or a diagnosis.</p></div>${error?`<p class="capture-status" role="status">${esc(error)}</p>`:''}<div class="capture-choice-grid"><label class="photo-capture" for="camera-photo"><b>📷</b><strong>Take photo</strong><span>Use the rear camera</span><input id="camera-photo" type="file" accept="image/*" capture="environment"></label><label class="photo-capture" for="gallery-photo"><b>▧</b><strong>Choose from gallery</strong><span>JPEG, PNG, or WebP</span><input id="gallery-photo" type="file" accept="image/*"></label></div><button class="btn btn-ghost btn-wide section" id="manual-from-scan">Continue without AI</button>`;
|
||||
if(mode==='ready'){const readiness=visionReadiness(visionStatus);if(!readiness.ready)return `${visionStatusHtml()}<img class="photo-preview scan-preview" src="${form.photoDataUrl}" alt="Photo kept on this device"><p class="quality-note">${esc(photoHint)}</p><p class="fine">${esc(readiness.message)} Nothing is uploaded, and this photo stays on this device until you save the entry.</p><button class="btn btn-primary btn-wide" id="manual-from-scan">${esc(readiness.manualLabel)}</button><button class="btn btn-ghost btn-wide section" id="retake-photo">Use another photo</button>`;const processingCopy=visionStatus?.profile==='selfhost'?'Timmy’s server does not save it. The compressed copy stays on Timmy’s self-hosted model server.':'Timmy’s server does not save it. Your configured AI provider processes it under that provider’s terms.';return `${visionStatusHtml()}<img class="photo-preview scan-preview" src="${form.photoDataUrl}" alt="Photo awaiting AI analysis"><p class="quality-note">${esc(photoHint)}</p><div class="consent-card"><label class="check"><input id="ai-consent" type="checkbox"><span><strong>Send this compressed copy for one-time AI analysis.</strong><br>${processingCopy}</span></label></div><button class="btn btn-primary btn-wide" id="analyze-photo" disabled>Analyze visible form + color</button><button class="btn btn-ghost btn-wide section" id="retake-photo">Use another photo</button>`;}
|
||||
if(mode==='analyzing')return `<div class="analyzing"><img src="${appPath('assets/timmy.svg')}" alt="Timmy"><div class="spinner" aria-hidden="true"></div><h3>Timmy is looking at form and color…</h3><p>Not symptoms. Not disease. Not whether Taco Bell was a strategic error.</p></div>`;
|
||||
// An unreadable file is its own state, not a failed analysis: nothing was sent, nothing was
|
||||
// attempted, and saying "couldn't analyze that safely" here implied an upload that never happened.
|
||||
if(mode==='unreadable')return `<div class="scan-result needs-input" data-capture-error="unreadable"><b>⚠</b><h3>That file isn’t a photo Timmy can read.</h3><p>Nothing was uploaded. Choose a JPEG, PNG, or WebP photo, take a new one, or fill it out manually.</p></div><button class="btn btn-primary btn-wide" id="manual-from-scan">Fill it out manually</button><button class="btn btn-ghost btn-wide section" id="retake-photo">Choose another photo</button>`;
|
||||
if(mode==='error')return `<div class="scan-result needs-input"><b>↻</b><h3>Timmy couldn’t analyze that safely.</h3><p>${esc(error||'Continue manually or try a clearer photo.')}</p></div><button class="btn btn-primary btn-wide" id="manual-from-scan">Fill it out manually</button><button class="btn btn-ghost btn-wide section" id="retake-photo">Try another photo</button>`;
|
||||
if(suggestionState.status==='pending'){const copy=provenanceCopy(suggestionState,form);return `<div class="scan-result provisional" data-provenance="${copy.status}" aria-label="${esc(copy.aria)}"><span class="ai-badge">${esc(copy.label)} · ${Math.round(suggestionState.suggestion.confidence*100)}% CONFIDENCE</span><div class="suggestion-pair"><div><small>BRISTOL FORM</small><strong>Type ${suggestionState.suggestion.bristolType}</strong></div><div><small>VISIBLE COLOR</small><strong>${esc(suggestionState.suggestion.color)}</strong></div></div><p>${esc(aiSuggestion.observations||'Visual match found.')}</p><p class="provenance-note" role="status">${esc(copy.detail)}</p><p class="fine">${esc(aiSuggestion.warning)}</p></div><button class="btn btn-primary btn-wide" id="use-suggestion">Use these suggestions</button><button class="btn btn-ghost btn-wide section" id="manual-from-scan">Review everything manually</button>`;}
|
||||
return `<div class="scan-result needs-input"><b>?</b><h3>No confident match.</h3><p>${esc(aiSuggestion?.reason||'The image was too uncertain to prefill safely.')}</p></div><button class="btn btn-primary btn-wide" id="manual-from-scan">Choose the form yourself</button><button class="btn btn-ghost btn-wide section" id="retake-photo">Try another photo</button>`;
|
||||
}
|
||||
function showPhotoFirst(mode='pick',error=''){
|
||||
photoFirstMode=mode;
|
||||
document.querySelector('.sheet-backdrop')?.remove();const wrap=document.createElement('div');wrap.className='sheet-backdrop';wrap.innerHTML=`<section class="sheet scan-sheet" role="dialog" aria-modal="true" aria-labelledby="scan-title"><div class="sheet-handle"></div><div class="sheet-header"><div><span class="eyebrow">Photo-first log</span><h2 id="scan-title">${mode==='result'?'Review Timmy’s suggestion':mode==='analyzing'?'Analyzing privately':'Start with the camera'}</h2></div><button class="icon-btn" id="close-sheet" aria-label="Close">×</button></div>${photoFirstBody(mode,error)}</section>`;document.body.append(wrap);document.querySelector('#close-sheet').onclick=()=>wrap.remove();wrap.onclick=e=>{if(e.target===wrap)wrap.remove()};
|
||||
document.querySelector('.sheet-backdrop')?.remove();const wrap=document.createElement('div');wrap.className='sheet-backdrop';wrap.innerHTML=`<section class="sheet scan-sheet" role="dialog" aria-modal="true" aria-labelledby="scan-title"><div class="sheet-handle"></div><div class="sheet-header"><div><span class="eyebrow">Photo-first log</span><h2 id="scan-title">${mode==='result'?'Review Timmy’s suggestion':mode==='analyzing'?'Analyzing privately':mode==='unreadable'?'That photo could not be read':'Start with the camera'}</h2></div><button class="icon-btn" id="close-sheet" aria-label="Close">×</button></div>${photoFirstBody(mode,error)}</section>`;document.body.append(wrap);document.querySelector('#close-sheet').onclick=()=>wrap.remove();wrap.onclick=e=>{if(e.target===wrap)wrap.remove()};
|
||||
document.querySelectorAll('#camera-photo,#gallery-photo').forEach(file=>{file.onchange=handleAiPhoto;file.addEventListener('cancel',()=>showPhotoFirst('pick','Camera or photo picker closed. If permission was denied, allow camera access in browser settings, choose from the gallery, or continue without AI.'))});
|
||||
const consent=document.querySelector('#ai-consent'),analyze=document.querySelector('#analyze-photo');if(consent&&analyze)consent.onchange=()=>analyze.disabled=!consent.checked||!visionReadiness(visionStatus).ready;if(analyze)analyze.onclick=runAiAnalysis;
|
||||
document.querySelector('#retake-photo')?.addEventListener('click',()=>{form.photoDataUrl='';photoHint='';resetSuggestion();showPhotoFirst('pick')});
|
||||
document.querySelector('#manual-from-scan')?.addEventListener('click',()=>{resetSuggestion();showLogStep(1)});
|
||||
document.querySelector('#use-suggestion')?.addEventListener('click',()=>{suggestionState=confirmSuggestion(suggestionState);form=mergeVisualSuggestion(form,aiSuggestion);form.typeChosen=true;showLogStep(1)});
|
||||
}
|
||||
async function handleAiPhoto(e){const file=e.target.files[0];if(!file)return;try{const result=await compressPhoto(file);form.photoDataUrl=result.dataUrl;photoHint=photoQualityMessage(result);showPhotoFirst('ready')}catch{showPhotoFirst('error','That image could not be read. Try another photo.')}}
|
||||
async function handleAiPhoto(e){const file=e.target.files[0];if(!file)return;try{const result=await compressPhoto(file);form.photoDataUrl=result.dataUrl;photoHint=photoQualityMessage(result);showPhotoFirst('ready')}catch{form.photoDataUrl='';photoHint='';showPhotoFirst('unreadable')}}
|
||||
async function runAiAnalysis(){showPhotoFirst('analyzing');try{const response=await fetch(appPath('api/analyze'),{method:'POST',headers:{'content-type':'application/json'},body:JSON.stringify({imageDataUrl:form.photoDataUrl,consent:true})});const data=await response.json();if(!response.ok)throw new Error(data.error||'AI analysis is unavailable.');aiSuggestion=data;suggestionState=createSuggestionState(data);showPhotoFirst('result')}catch(error){showPhotoFirst('error',error.message||'AI analysis is unavailable. Continue manually.')}}
|
||||
|
||||
function openLogger(){beginDraft();photoHint='';resetSuggestion();showLogStep(1)}
|
||||
|
||||
@@ -144,6 +144,26 @@ Each entry names the abuse case, the mitigation that is already committed or que
|
||||
- **Testable follow-up:** #39
|
||||
- **Evidence:** The production topology, secrets, backup, and rollback runbook, exercised against the deletion path so that purge reaches memory, temporary files, logs, and backups, with observability from #41 confirming no bytes reach the metrics surface.
|
||||
|
||||
## Ingress controls implemented for #16
|
||||
|
||||
`/api/analyze` is the only route that accepts image bytes, and every limit below is decided inside
|
||||
`server.mjs` before `analyzePhoto` runs, so a refusal never costs a provider call.
|
||||
|
||||
| Control | Where | Behaviour |
|
||||
|---|---|---|
|
||||
| Per-caller request budget | `src/rate-limit.js` | Sliding window, 12 requests per 60s by default (`TIMMY_ANALYZE_RATE_LIMIT`, `TIMMY_ANALYZE_RATE_WINDOW_MS`); refusal is `429` with `Retry-After`. The key is a salted SHA-256 of the socket address held in process memory only — no address, no identity, no bytes are stored. |
|
||||
| Body ceiling and deadline | `src/http-body.js` | 6 MB ceiling and a 15s deadline (`TIMMY_BODY_TIMEOUT_MS`); `413` and `408` respectively, and the half-read socket is dropped after the refusal is written. |
|
||||
| Container sniffing | `src/image-ingress.js` | The declared data-URL MIME is treated as a claim, never evidence: JPEG/PNG/WebP is decided from magic bytes, and a mismatch is refused. |
|
||||
| Polyglot / trailing payloads | `src/image-ingress.js` | The container is walked to its own end marker (PNG `IEND`, JPEG `EOI`, RIFF length); anything appended after the image is refused instead of forwarded. |
|
||||
| Embedded metadata | `src/image-ingress.js` | JPEG APP1/APP13/COM segments and PNG `eXIf`/`tEXt`/`zTXt`/`iTXt`/`tIME` chunks are refused. The app's own upload is a canvas re-encode (JPEG q0.7, longest edge 1200px) and carries none, so the accepted path is already metadata-free. |
|
||||
| Pixel limits | `src/image-ingress.js` | Dimensions are read from the header (PNG `IHDR`, JPEG SOF, WebP `VP8X`/`VP8`/`VP8L`) and refused above a 4096px edge or 16 megapixels, so a decode bomb never reaches the worker. |
|
||||
| Log hygiene | `server.mjs` | Refusal copy is fixed prose; no request body, image bytes, hash, or address is written to a log or returned in an error. |
|
||||
|
||||
Residual, and not claimed as solved here: the server refuses metadata-bearing or mismatched
|
||||
containers rather than transcoding them itself, so a client that cannot produce a clean re-encode
|
||||
must use the app's capture path. PNG chunk CRCs are not validated (structure is), and the request
|
||||
budget is per process, not per cluster.
|
||||
|
||||
## Accepted residual risks
|
||||
|
||||
- **Host compromise.** Data in a running process can be read by a sufficiently privileged attacker. This is accepted for the private beta and bounded by least privilege plus the purge procedure in #39; it is not claimed as eliminated.
|
||||
@@ -154,4 +174,5 @@ Each entry names the abuse case, the mitigation that is already committed or que
|
||||
|
||||
- This document is enforced by `tests/threat-model.test.js`, which asserts that the diagram covers every byte-bearing component, that each required abuse case is a numbered threat with an owner and a tracked follow-up ticket, and that the model keeps the non-diagnostic boundary and ships no real medical images.
|
||||
- The test runs in `npm test` and therefore inside the CI quality gate and the release build, so the threat model cannot silently drift from the code it describes.
|
||||
- The ingress controls above are enforced by `tests/image-ingress.test.js`, `tests/rate-limit.test.js`, `tests/http-body.test.js`, and `tests/analyze-ingress.acceptance.test.js`, which runs the shipping server against synthetic bytes and a counting provider stub; `tests/ingress-capture.acceptance.mjs` proves the same refusal in the running app. All of them run in `npm test` or the browser gate, so a regression fails the release instead of a promise.
|
||||
- Boundary and boundary-adjacent claims are reviewed by the clinical and privacy gate (#22, #23) before beta; this model does not move that gate, it feeds it.
|
||||
@@ -0,0 +1,172 @@
|
||||
# Production topology, secrets, backup, and rollback
|
||||
|
||||
This runbook answers one question for the operator: how do the parts of Timmy run, what may they
|
||||
reach, what must never be lost, and how does each part come back? It covers the pieces that exist in
|
||||
this tree today — the Node app, the private vision worker, the private staging directory, and the
|
||||
consented corpus. It does not describe a training fleet, a cloud account, or a hosted database,
|
||||
because this build has none.
|
||||
|
||||
Everything below is run by automation or by one operator on the host. Nothing here is performed by a
|
||||
participant, and no step needs a public endpoint.
|
||||
|
||||
## Topology
|
||||
|
||||
| Process | Entry point | Holds | Restart unit |
|
||||
|---|---|---|---|
|
||||
| App and API | `node server.mjs` on `HOST=127.0.0.1 PORT=4173` | Nothing durable. Serves the PWA, proxies `/api/vision/*`, and gates `/api/agent/*` behind an access code. | `timmy-staging.service` (staging), `node server.mjs` (local) |
|
||||
| Vision worker | `deploy/worker/run_worker_container.sh` | The pinned GGUF weights, mounted read-only from the host. | `scripts/run_selfhost_smolvlm.sh start`, container restart policy |
|
||||
| Staging releases | `/opt/timmy-staging/releases/<40-character-commit>` with a `current` symlink | Immutable release trees plus the mode-600 environment file. | `timmy-staging.service` |
|
||||
| Consented corpus | `/srv/timmy-corpus` | `manifest.jsonl` (append-only provenance and tombstones) and `images/` derived from consented contributions. | none — file data |
|
||||
| Caddy | host Caddy | Terminates TLS and routes `/git/` to Gitea and `/timmy-staging/` to the app. | host Caddy |
|
||||
|
||||
The app is stateless for a reason: there is **no server-side copy of the journal**. A person's logs
|
||||
and photos live in their own browser storage, and they hold the portable JSON backup. The server
|
||||
holds no diary, so a lost server loses no one's health record — only the operator records listed
|
||||
above can be lost, and only those are backed up here.
|
||||
|
||||
Health and readiness are separate, and the probe reports readiness only:
|
||||
|
||||
```
|
||||
GET /api/healthz -> {"ok":true,"release":"<tag>","commit":"<commit>","visionEnabled":<bool>,"agentEnabled":<bool>}
|
||||
```
|
||||
|
||||
`visionEnabled` and `agentEnabled` say whether the feature is configured, never which key or endpoint
|
||||
is configured. The worker answers `/v1/models` before its pinned model is resident, so readiness
|
||||
means "served a request from the loaded model", not "the port is open" — `scripts/worker_ready.sh`
|
||||
enforces that distinction and the app withholds the AI control until it passes.
|
||||
|
||||
## Network boundaries
|
||||
|
||||
- The app reads `HOST` and its own default is `0.0.0.0`, which is a local-development convenience and
|
||||
not the deployment. The deployment sets `HOST=127.0.0.1` in the mode-600 environment file
|
||||
(`deploy/timmy-staging.env.example` is the template) and is reachable from outside solely through
|
||||
the Caddy `/timmy-staging/` subpath. A bare `node server.mjs` on a shared host listens on every
|
||||
interface — treat that as a laptop command, never as a production start.
|
||||
- The Gitea `/git/` catch-all is declared before the staging route and must stay reachable across
|
||||
every Caddy reload.
|
||||
- The vision worker binds `127.0.0.1:8080` and is never published: `deploy/worker/run_worker_container.sh`
|
||||
refuses a non-loopback publish host instead of warning about it, and no route in `server.mjs`
|
||||
forwards arbitrary paths to the worker — the only vision endpoints are `/api/vision-status` and the
|
||||
analysis request.
|
||||
- Outbound from the host is limited to the pinned model download during bootstrap, which is refused
|
||||
unless both SHA-256 receipts match. After that the worker needs no egress at all.
|
||||
- Inference bodies and image bytes are never logged, and no reverse-proxy access log retains a
|
||||
request body. A backup that contained them would be a second copy of someone's photo, so they are
|
||||
excluded by the tool rather than by the operator remembering.
|
||||
- No participant-facing endpoint requires an inbound port, a DNS record, or a cloud credential.
|
||||
|
||||
## Secrets
|
||||
|
||||
Four secrets exist, and every one of them is injected at runtime:
|
||||
|
||||
| Secret | Source | Scope |
|
||||
|---|---|---|
|
||||
| `TIMMY_DATASET_SALT` | mode-600 environment file on the host | Corpus subject keys and split assignment. Rotating it re-splits future ingests, so it is not rotated casually. |
|
||||
| Vision provider key or worker token | mode-600 environment file on the host | Only the app process reads it; the browser never receives it. |
|
||||
| Agent access code | mode-600 environment file on the host | Unlocks `/api/agent/unlock`; the cookie it mints is `HttpOnly` and same-origin. |
|
||||
| Gitea access token | local operator shell or secret store | Issue and PR automation only. It is never written into a release tree, an archive, or a log. |
|
||||
|
||||
Rules:
|
||||
|
||||
- Never commit a secret, and never pass one on a command line that lands in shell history. Use the
|
||||
environment file (`chmod 600`, owned by the service user) or the systemd `EnvironmentFile=`.
|
||||
- Placeholders in this document are angle-bracketed or environment references, e.g.
|
||||
`TIMMY_DATASET_SALT=<random 32+ characters, never committed>`. A literal value here is a bug.
|
||||
- `scripts/build_release.py` scans the cloned tree for private-key blocks and assigned credentials
|
||||
and fails the release if it finds one, so a secret that reaches a commit is caught before it
|
||||
reaches a release.
|
||||
- Rotation is: issue the replacement on the host, restart the unit, confirm `/api/healthz` and one
|
||||
real inference request, then revoke the old value in the provider's own UI. Nothing in Git holds a
|
||||
copy that would need rewriting.
|
||||
- The backup tool refuses `.env`, `*.env`, `*.key`, `*.pem`, `id_rsa`, `tokens.json`, and
|
||||
`credentials.json` by name, so a backup can be copied off-host without carrying the keys that
|
||||
operate the host.
|
||||
|
||||
## Backup
|
||||
|
||||
The operator records that cannot be rebuilt are the corpus `manifest.jsonl` and the derived images
|
||||
it points at. A tombstone is the only proof that a participant withdrew, so a backup that drops
|
||||
tombstone lines silently re-authorises training on withdrawn data. The tool treats that as a
|
||||
correctness bug, not a storage detail.
|
||||
|
||||
`scripts/operator_backup.py` archives **only** records, and writes one manifest naming every byte it
|
||||
stored and every path it refused:
|
||||
|
||||
- Stored: `manifest.jsonl`, derived images, and any receipt files under the data root.
|
||||
- Excluded, with the reason recorded: **model weights** (`model-weights` — re-downloadable, pinned by
|
||||
hash), **transient inference** directories and uploads (`transient-inference` — a second copy of
|
||||
someone's photo), **secret** files (`secret`), logs (`log`), symlinks (`symlink`), and anything over
|
||||
`--max-bytes` (`oversize`).
|
||||
- Every stored member carries a SHA-256, and the archive is reproducible: sorted members, zeroed
|
||||
timestamps and ownership, `gzip` with a fixed mtime. Two backups of unchanged records are
|
||||
byte-identical, so "did anything change?" is answerable by digest.
|
||||
|
||||
```bash
|
||||
python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy-nightly.tar.gz --label nightly
|
||||
python3 scripts/operator_backup.py verify --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json
|
||||
```
|
||||
|
||||
The command writes `<archive>.manifest.json` unless `--manifest` names another path, and prints a
|
||||
receipt with the stored file count, archive digest, and record counts (`samples`, `tombstones`,
|
||||
`activeSamples`). It refuses a source that is not a directory and refuses to overwrite an existing
|
||||
archive, so a nightly run cannot erase the previous night.
|
||||
|
||||
Retention: keep fourteen nightlies and the first backup of each month, and copy each archive plus its
|
||||
manifest off-host. Also keep a human-readable record of the counts: `verify` compares the archive
|
||||
against the manifest, and a restore compares the restored counts against both.
|
||||
|
||||
What this backup deliberately does not cover: the browser-held journal (there is no server copy; the
|
||||
person's own portable export is the backup), model weights, and anything in the release tree, which
|
||||
is rebuilt from a commit by `scripts/build_release.py`.
|
||||
|
||||
## Restore
|
||||
|
||||
A restore never writes over live data. It verifies first, and it writes into a directory that must
|
||||
not exist or must be empty:
|
||||
|
||||
```bash
|
||||
python3 scripts/operator_backup.py restore --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json --target /srv/timmy-corpus.restore-check
|
||||
```
|
||||
|
||||
In order, before the first byte is written:
|
||||
|
||||
1. the manifest is loaded and its `schemaVersion` checked;
|
||||
2. every archive member is refused if it is a link, a non-regular file, an absolute path, a traversal
|
||||
path, or a name the manifest never listed;
|
||||
3. every member's SHA-256 is recomputed and compared, and a mismatch stops the restore;
|
||||
4. a member the manifest lists but the archive lacks stops the restore.
|
||||
|
||||
After the extract it recounts `manifest.jsonl` in the target and refuses if the counts differ from
|
||||
the backup, so a restore that silently lost a tombstone fails loudly. Then the operator compares the
|
||||
restored tree against the live one, and promotes it by moving the directories — the tool does not
|
||||
pretend to know which directory is authoritative.
|
||||
|
||||
Restoring the person's own logs is a different operation and stays where it belongs: the app's
|
||||
import control accepts the JSON file they exported, upgrades an older version and says which version
|
||||
it read, and refuses a file from a newer Timmy without touching the ledger it would have replaced.
|
||||
|
||||
## Rollback
|
||||
|
||||
Two rollbacks exist, and only one of them is a live staging action.
|
||||
|
||||
**Release rollback (staging).** Staging keeps one immutable directory per commit and a `current`
|
||||
symlink, so a rollback is a repoint, a restart, and a bounded health check that restores and verifies
|
||||
the previous release if the new one does not come up:
|
||||
|
||||
```bash
|
||||
python3 scripts/deploy_staging.py status
|
||||
python3 scripts/deploy_staging.py rollback --commit <FULL_COMMIT>
|
||||
```
|
||||
|
||||
The rollback path is exercised by `tests/staging-deploy.test.py`, which drives the real
|
||||
`scripts/deploy_staging.py` code against a rootless fixture (`--root`), including the case where the
|
||||
named commit has no immutable release directory, where the restart fails, and where the health probe
|
||||
never passes. That is the same code path the host runs; the difference is only the root.
|
||||
|
||||
The live drill on the private staging host — promote a build, serve it, roll it back, and confirm the
|
||||
participant-facing page again — is a **human gate**, tracked under #53. This loop does not run that
|
||||
drill automatically, because a rolled-back staging host is being used for release testing at the time.
|
||||
|
||||
**Data rollback (operator records).** The equivalent for the corpus is the restore above: verify,
|
||||
restore into a new directory, compare counts, then move it into place. A restore is never attempted in
|
||||
place, and a failed verify leaves the live directory untouched.
|
||||
@@ -0,0 +1,80 @@
|
||||
# RC Security & Abuse Evidence (#40)
|
||||
|
||||
The release candidate is attacked before users get to it. This page is the procedure and the rule,
|
||||
not a stored result: a pasted-in "all clear" is exactly the kind of evidence that stops being true
|
||||
after the next commit.
|
||||
|
||||
## Run it
|
||||
|
||||
```bash
|
||||
node scripts/security_attack_suite.mjs --json artifacts/security/rc-attack-report.json
|
||||
```
|
||||
|
||||
Exit codes are part of the contract:
|
||||
|
||||
| Exit | Meaning |
|
||||
|---:|---|
|
||||
| `0` | `VERDICT: GO` — every case matched its expectation. |
|
||||
| `1` | `VERDICT: BLOCKED` — at least one P0 case failed, or a required category has no executable evidence. |
|
||||
| `2` | The suite could not complete (target never became ready, vision disabled, no log captured). |
|
||||
|
||||
The suite is a child process of nothing: it starts its own copy of `server.mjs` on a free loopback
|
||||
port, points it at a **counting stub worker**, and drives it with synthetic hostile bytes. It never
|
||||
contacts a real model provider, never uses a real medical image, and never touches a live user's data.
|
||||
|
||||
## What it does and does not prove
|
||||
|
||||
- It proves what this HTTP boundary did with the bytes it was handed, and whether the worker behind
|
||||
it was called. The worker is a stub, so "the provider saw it" is a counted fact, not an inference.
|
||||
- It does **not** measure model quality, clinical accuracy, or the safety of the medical/non-diagnostic
|
||||
boundary. Those live in the vision, escalation, and clinical-review issues.
|
||||
- Latency numbers describe the loopback boundary against a stub. They are not a model benchmark.
|
||||
|
||||
## Cases
|
||||
|
||||
| Case | Class | Severity | Expectation |
|
||||
|---|---|---:|---|
|
||||
| `ingress-oversized-body` | oversized | P0 | `413`, worker never called |
|
||||
| `ingress-polyglot-html` | polyglot | P0 | `400`, worker never called |
|
||||
| `ingress-garbage-bytes` | truncated | P0 | `400`, worker never called |
|
||||
| `consent-required` | consent | P0 | `400`, worker never called |
|
||||
| `model-output-malformed` | malformed | P0 | `503` neutral copy, provider text never echoed |
|
||||
| `replay-duplicate-request` | replay | P2 | identical answer, nothing stored server-side |
|
||||
| `flood-concurrency` | flood | P1 | burst meets a bound (`429`/overload), not an open pipe |
|
||||
| `log-leak-scan` | log | P0 | captured server log carries no image bytes |
|
||||
| `deletion-withdrawal` | deletion | P0 | out of surface — counted only when `tests/training-contribution.test.js` exists |
|
||||
|
||||
## The rule
|
||||
|
||||
`src/release-security-gate.js` owns the verdict, so it is testable rather than asserted:
|
||||
|
||||
- A case passes only when **every** expected field matches the observation. A partial match fails and
|
||||
names the field that broke (`status 400 expected, observed 200`).
|
||||
- **Any** failing P0 case blocks RC approval. P1/P2 failures are recorded as holds.
|
||||
- A required class with no executable evidence is `unproven` and blocks the release. Coverage may be
|
||||
borrowed from another suite (`out-of-surface` + an evidence path) only after the runner verified
|
||||
that file exists — otherwise it blocks.
|
||||
- The report is redacted before it is written, and assembly fails closed if any image bytes survive
|
||||
redaction. Evidence that carries a photo is worse than no evidence.
|
||||
|
||||
## Current state (read the run, not this line)
|
||||
|
||||
This section used to predict **BLOCKED** until the ingress hardening in
|
||||
[#16](https://forge.alexanderwhitestone.com/git/stackchain/timmy-talking-turd/issues/16) landed. With
|
||||
that ingress in place the same command returns **GO** on this tree, because the format is decided from
|
||||
the bytes rather than the data-URL label: `ingress-polyglot-html` and `ingress-garbage-bytes` are now
|
||||
refused with `400` before the worker is called, and an oversized body is refused with a `413` the
|
||||
caller can read instead of a dropped connection. Latency for the load phase is measured against a
|
||||
second target with the request budget opened up, so those samples are analyses rather than `429`
|
||||
refusals. Queue bounding is still [#17](https://forge.alexanderwhitestone.com/git/stackchain/timmy-talking-turd/issues/17)
|
||||
and is held here as P1.
|
||||
|
||||
Deletion is governed by the contribution ledger, not by this HTTP boundary; the case stays
|
||||
`covered-elsewhere` against `tests/training-contribution.test.js` and blocks if that file disappears.
|
||||
|
||||
## Evidence hygiene
|
||||
|
||||
- The report is written to `artifacts/security/` and must never be committed with a real photo in it.
|
||||
- The captured server log is scanned by the same detector (`containsImageBytes`) before it is quoted.
|
||||
- Error bodies are recorded as a 120-character snippet; the suite refuses to write a report that
|
||||
still contains a data URI or a long base64 run.
|
||||
+7
-3
@@ -4,10 +4,11 @@
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/draft.test.js tests/suggestion.test.js tests/followup.test.js tests/vision-readiness.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js tests/threat-model.test.js tests/vision-corpus.test.js tests/vision-boundary.test.js tests/worker-benchmark.test.js tests/worker-ready.test.js tests/worker-container.test.js tests/summary.test.js tests/escalation-regression.test.js tests/grounded-chat.test.js tests/photo-lifecycle.test.js tests/training-contribution.test.js tests/portability.test.js tests/abstention-policy.test.js",
|
||||
"test": "node --test tests/domain.test.js tests/analysis.test.js tests/vision-service.test.js tests/vision-config.test.js tests/draft.test.js tests/suggestion.test.js tests/followup.test.js tests/vision-readiness.test.js tests/hermes-agent-service.test.js tests/agent-gateway.acceptance.test.js tests/staging-health.test.js tests/service-worker-runtime.test.js tests/training-ingest.test.js tests/ci-workflow.test.js tests/product-decisions.test.js tests/release-demo.test.js tests/selfhost-bootstrap.test.js tests/staging-config.test.js tests/threat-model.test.js tests/vision-corpus.test.js tests/vision-boundary.test.js tests/worker-benchmark.test.js tests/worker-ready.test.js tests/worker-container.test.js tests/summary.test.js tests/escalation-regression.test.js tests/grounded-chat.test.js tests/photo-lifecycle.test.js tests/training-contribution.test.js tests/portability.test.js tests/abstention-policy.test.js tests/release-security-gate.test.js tests/image-ingress.test.js tests/rate-limit.test.js tests/http-body.test.js tests/analyze-ingress.acceptance.test.js",
|
||||
"test:ui": "node tests/ui.acceptance.mjs",
|
||||
"test:photo": "node tests/photo-first.acceptance.mjs",
|
||||
"test:mobile-capture": "node tests/mobile-capture.acceptance.mjs",
|
||||
"test:ingress-capture": "node tests/ingress-capture.acceptance.mjs",
|
||||
"test:sleek": "node tests/sleek-chat.acceptance.mjs",
|
||||
"test:manual-fallback": "node tests/manual-fallback.acceptance.mjs",
|
||||
"test:provisional": "node tests/provisional-suggestion.acceptance.mjs",
|
||||
@@ -18,7 +19,7 @@
|
||||
"test:grounded-chat": "node tests/grounded-chat.acceptance.mjs",
|
||||
"test:photo-lifecycle": "node tests/photo-lifecycle.acceptance.mjs",
|
||||
"test:training-contribution": "node tests/training-contribution.acceptance.mjs",
|
||||
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/review_corpus.py scripts/dataset_card.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh",
|
||||
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check src/release-security-gate.js && node --check scripts/security_attack_suite.mjs && node --check tests/security-attack.acceptance.mjs && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/review_corpus.py scripts/dataset_card.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py scripts/operator_backup.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh && node --check src/image-ingress.js && node --check src/rate-limit.js && node --check src/http-body.js && python3 -m py_compile scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py && node --check tests/ingress-capture.acceptance.mjs && node --check tests/analyze-ingress.acceptance.test.js",
|
||||
"check:diff": "bash scripts/check_diff.sh",
|
||||
"start": "node server.mjs",
|
||||
"test:worker-readiness": "node tests/worker-readiness.acceptance.mjs",
|
||||
@@ -27,7 +28,10 @@
|
||||
"test:calibration": "python3 tests/calibration-harness.test.py",
|
||||
"test:policy": "node --test tests/abstention-policy.test.js",
|
||||
"test:review": "python3 tests/corpus-review.test.py",
|
||||
"test:dataset-card": "python3 tests/dataset-card.test.py"
|
||||
"test:dataset-card": "python3 tests/dataset-card.test.py",
|
||||
"test:backup": "python3 tests/backup-restore.test.py",
|
||||
"test:runbook": "python3 tests/production-runbook.test.py",
|
||||
"test:security-attack": "node tests/security-attack.acceptance.mjs"
|
||||
},
|
||||
"devDependencies": {
|
||||
"playwright": "^1.62.1"
|
||||
|
||||
@@ -50,6 +50,7 @@ DEMO_REQUIRED_CAPTIONS = (
|
||||
"Reachable is not loaded: no AI control, no consent, and no upload until the pinned model is resident",
|
||||
"Every pattern line names the confirmed logs behind it, and following one shows exactly those records",
|
||||
"Two logs is not a pattern: the summary refuses to describe a trend it cannot support",
|
||||
"A file that only claims to be a photo is refused on the device — nothing is uploaded",
|
||||
"The answer states the count and cites the four confirmed logs it counted",
|
||||
"Following a citation opens exactly the confirmed log behind that answer",
|
||||
"Ask something the journal cannot answer and Timmy refuses instead of reassuring",
|
||||
@@ -73,6 +74,10 @@ DEMO_REQUIRED_CAPTIONS = (
|
||||
"The model said Type 6 and the reviewer said Type 5: the disagreement is recorded, not smoothed over",
|
||||
"Withdrawing the contribution empties the queue and the training set, and the tool refuses to review it at all",
|
||||
"The dataset card is generated from the corpus manifest, so no count on it is typed by hand",
|
||||
"The operator backup is a command line, not a screen in the app",
|
||||
"Model weights and transient inference images are refused by name, and every skipped path is listed with its reason",
|
||||
"Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes",
|
||||
"A restore returns the same records, including the withdrawal tombstone a trainer must never lose",
|
||||
"One contributor's photos stay in one split: a corpus that leaks a person across splits is refused",
|
||||
"Devices, contexts, and licenses stay pending until the pipeline collects them",
|
||||
"The card is a page, not a screen in the app — it is published with the real generator against synthetic fixtures",
|
||||
@@ -183,6 +188,10 @@ def main() -> int:
|
||||
# Issue #29's evaluation harness decides GO/BLOCKED for a model, so its refusal and gate
|
||||
# arithmetic are exercised before this release is allowed to exist.
|
||||
run(["python3", "tests/calibration-harness.test.py", "-v"], tree)
|
||||
# Issue #39's operator backup is the only copy of the corpus tombstones, so the tool that
|
||||
# produces it, and the runbook whose commands must stay real, are gated before the release.
|
||||
run(["python3", "tests/backup-restore.test.py", "-v"], tree)
|
||||
run(["python3", "tests/production-runbook.test.py", "-v"], tree)
|
||||
server_env = dict(os.environ)
|
||||
server_env.update({"HOST": "127.0.0.1", "PORT": "4173", "TIMMY_VISION_ENABLED": "false"})
|
||||
# A leftover server from another checkout answers on this port and the readiness probe cannot
|
||||
@@ -209,6 +218,7 @@ def main() -> int:
|
||||
run(["npm", "run", "test:ui"], tree)
|
||||
run(["npm", "run", "test:photo"], tree)
|
||||
run(["npm", "run", "test:mobile-capture"], tree)
|
||||
run(["npm", "run", "test:ingress-capture"], tree)
|
||||
run(["npm", "run", "test:sleek"], tree)
|
||||
run(["npm", "run", "test:manual-fallback"], tree)
|
||||
run(["npm", "run", "test:provisional"], tree)
|
||||
@@ -267,7 +277,7 @@ def main() -> int:
|
||||
if not contact_sheet.exists() or contact_sheet.stat().st_size == 0:
|
||||
raise SystemExit("Release demo contact sheet was not produced")
|
||||
run(["npm", "audit", "--audit-level=high"], tree)
|
||||
for file in ("app.js", "server.mjs", "service-worker.js", "src/analysis.js", "src/domain.js", "src/draft.js", "src/suggestion.js", "src/followup.js", "src/vision-readiness.js", "src/hermes-agent-service.js", "src/vision-config.js", "src/vision-service.js", "src/vision-corpus.js", "src/summary.js", "src/photo-lifecycle.js", "src/training-contribution.js", "scripts/record_worker_benchmark.mjs", "tests/staging.acceptance.mjs", "tests/vision-refusal.acceptance.mjs", "tests/worker-readiness.acceptance.mjs", "tests/pattern-summary.acceptance.mjs", "tests/photo-lifecycle.acceptance.mjs", "tests/training-contribution.acceptance.mjs"):
|
||||
for file in ("app.js", "server.mjs", "service-worker.js", "src/analysis.js", "src/domain.js", "src/draft.js", "src/suggestion.js", "src/followup.js", "src/vision-readiness.js", "src/hermes-agent-service.js", "src/vision-config.js", "src/vision-service.js", "src/vision-corpus.js", "src/summary.js", "src/photo-lifecycle.js", "src/training-contribution.js", "scripts/record_worker_benchmark.mjs", "tests/staging.acceptance.mjs", "tests/vision-refusal.acceptance.mjs", "tests/worker-readiness.acceptance.mjs", "tests/pattern-summary.acceptance.mjs", "tests/photo-lifecycle.acceptance.mjs", "tests/training-contribution.acceptance.mjs", "src/image-ingress.js", "src/rate-limit.js", "src/http-body.js", "tests/ingress-capture.acceptance.mjs"):
|
||||
run(["node", "--check", file], tree)
|
||||
run(["bash", "-n", "scripts/bootstrap_selfhost_smolvlm.sh"], tree)
|
||||
run(["bash", "-n", "scripts/run_selfhost_smolvlm.sh"], tree)
|
||||
@@ -360,7 +370,7 @@ def main() -> int:
|
||||
"codec": "h264",
|
||||
"pixel_format": "yuv420p",
|
||||
"recorded_from_working_app": True,
|
||||
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything, plus the review ledger card, where the real review commands run against the committed synthetic sample and the card shows their actual output: one pending proposal with the model answer withheld, a refused reveal before commitment, a committed Type 5 correction beside the Type 4 proposal, the revealed synthetic Type 6 answer recorded as a three-field disagreement, a second reader's adjudication pointing at the review it resolved, one trainable sample, and a withdrawal that empties the queue and the training set and makes review refused",
|
||||
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything, plus the review ledger card, where the real review commands run against the committed synthetic sample and the card shows their actual output: one pending proposal with the model answer withheld, a refused reveal before commitment, a committed Type 5 correction beside the Type 4 proposal, the revealed synthetic Type 6 answer recorded as a three-field disagreement, a second reader's adjudication pointing at the review it resolved, one trainable sample, and a withdrawal that empties the queue and the training set and makes review refused, plus the operator backup card, where the real backup, verify, and restore commands run against a synthetic corpus built by the real ingest and revoke commands and the card shows their actual output: two samples and one tombstone recorded, a weight shard, a transient inference upload, an operator env file and a log refused by name with the reason listed, the exclusion counts the manifest wrote, one digest for the archive, a verify that re-hashes every stored member, and a restore that returns the manifest byte-for-byte with the tombstone intact",
|
||||
},
|
||||
"gates": {
|
||||
"unit_security": "passed",
|
||||
@@ -381,7 +391,10 @@ def main() -> int:
|
||||
"reviewer_adjudication_blind": "passed",
|
||||
"dataset_card_generated": "passed",
|
||||
"calibration_harness_gates": "passed",
|
||||
"production_backup_restore": "passed",
|
||||
"production_runbook_commands": "passed",
|
||||
"portable_backup_restore": "passed",
|
||||
"image_ingress_refusal": "passed",
|
||||
"dependency_audit_high": "passed",
|
||||
"syntax": "passed",
|
||||
"secret_scan": "passed",
|
||||
|
||||
@@ -0,0 +1,326 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Operator backup and restore for the records Timmy must not lose.
|
||||
|
||||
The app keeps the journal in the person's own browser, so the server side of Timmy holds no diary.
|
||||
What the operator does hold, and what cannot be rebuilt from anywhere else, is the corpus: the
|
||||
append-only ``manifest.jsonl`` with its provenance records and its withdrawal tombstones, plus the
|
||||
derived images those records point at. Losing that file loses the proof that a participant
|
||||
withdrew, which is the one receipt the project has promised to keep.
|
||||
|
||||
This tool therefore:
|
||||
|
||||
- archives only records, and names every path it refused to archive with the reason
|
||||
(``model-weights``, ``transient-inference``, ``secret``, ``log``, ``symlink``, ``oversize``), so a
|
||||
missing file is an auditable decision instead of a silent omission;
|
||||
- writes one SHA-256 per archived member and a reproducible archive (sorted members, zeroed
|
||||
timestamps and ownership), so two backups of the same records are byte-identical;
|
||||
- verifies every digest, member name, and record count before a restore writes anything, and
|
||||
refuses absolute, traversal, link, and unlisted members.
|
||||
|
||||
Usage:
|
||||
|
||||
python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy.tar.gz
|
||||
python3 scripts/operator_backup.py verify --archive /srv/backups/timmy.tar.gz --manifest /srv/backups/timmy.tar.gz.manifest.json
|
||||
python3 scripts/operator_backup.py restore --archive ... --manifest ... --target /srv/timmy-corpus.restored
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gzip
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tarfile
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path, PurePosixPath
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
TOOL = "operator_backup.py"
|
||||
PAYLOAD_PREFIX = "payload/"
|
||||
DEFAULT_MAX_BYTES = 64 * 1024 * 1024
|
||||
|
||||
WEIGHT_SUFFIXES = (".gguf", ".safetensors", ".onnx", ".bin", ".pt", ".ckpt", ".pth")
|
||||
WEIGHT_DIRS = ("weights", "models")
|
||||
TRANSIENT_DIRS = ("transient", "inference-tmp", "inference-tmp-images", "uploads", "tmp", "cache")
|
||||
LOG_DIRS = ("logs",)
|
||||
LOG_SUFFIXES = (".log",)
|
||||
SECRET_NAMES = (".env", "id_rsa", "id_ed25519", "id_ecdsa", "tokens.json", "credentials.json")
|
||||
SECRET_SUFFIXES = (".key", ".pem", ".p12", ".pfx", ".env")
|
||||
|
||||
|
||||
class BackupError(Exception):
|
||||
"""A refusal that is reported on stderr and exits non-zero."""
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def classify(relative: PurePosixPath) -> str | None:
|
||||
"""Return the reason a path is not archived, or ``None`` when it is a record."""
|
||||
parts = relative.parts[:-1]
|
||||
name = relative.name
|
||||
lowered = [part.lower() for part in parts]
|
||||
suffix = PurePosixPath(name.lower()).suffix
|
||||
|
||||
if name.lower().startswith(".env") or name.lower() in SECRET_NAMES or suffix in SECRET_SUFFIXES:
|
||||
return "secret"
|
||||
if suffix in WEIGHT_SUFFIXES or any(part in WEIGHT_DIRS for part in lowered):
|
||||
return "model-weights"
|
||||
if any(part in TRANSIENT_DIRS for part in lowered):
|
||||
return "transient-inference"
|
||||
if any(part in LOG_DIRS for part in lowered) or suffix in LOG_SUFFIXES:
|
||||
return "log"
|
||||
return None
|
||||
|
||||
|
||||
def record_counts(manifest_path: Path) -> dict[str, int]:
|
||||
"""Count the append-only manifest the way a trainer would read it."""
|
||||
counts = {"samples": 0, "tombstones": 0, "activeSamples": 0}
|
||||
if not manifest_path.is_file():
|
||||
return counts
|
||||
revoked: set[str] = set()
|
||||
samples: list[dict] = []
|
||||
for line in manifest_path.read_text(encoding="utf-8").splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
entry = json.loads(line)
|
||||
except json.JSONDecodeError as error:
|
||||
raise BackupError(f"manifest {manifest_path} is not valid JSON Lines: {error}") from error
|
||||
kind = entry.get("recordType")
|
||||
if kind == "sample":
|
||||
samples.append(entry)
|
||||
counts["samples"] += 1
|
||||
elif kind == "tombstone":
|
||||
counts["tombstones"] += 1
|
||||
if entry.get("contributionId"):
|
||||
revoked.add(str(entry["contributionId"]))
|
||||
counts["activeSamples"] = sum(
|
||||
1 for entry in samples
|
||||
if str(entry.get("contributionId")) not in revoked
|
||||
and str((entry.get("lifecycle") or {}).get("state")) == "active"
|
||||
)
|
||||
return counts
|
||||
|
||||
|
||||
def collect(source: Path, max_bytes: int) -> tuple[list[tuple[PurePosixPath, Path]], list[dict]]:
|
||||
included: list[tuple[PurePosixPath, Path]] = []
|
||||
excluded: list[dict] = []
|
||||
for dirpath, dirnames, filenames in os.walk(source, followlinks=False):
|
||||
dirnames.sort()
|
||||
base = Path(dirpath)
|
||||
for name in sorted(filenames):
|
||||
path = base / name
|
||||
relative = PurePosixPath(path.relative_to(source).as_posix())
|
||||
if path.is_symlink():
|
||||
excluded.append({"path": str(relative), "reason": "symlink", "bytes": 0})
|
||||
continue
|
||||
reason = classify(relative)
|
||||
if reason:
|
||||
excluded.append({"path": str(relative), "reason": reason, "bytes": _size(path)})
|
||||
continue
|
||||
size = _size(path)
|
||||
if size > max_bytes:
|
||||
excluded.append({"path": str(relative), "reason": "oversize", "bytes": size})
|
||||
continue
|
||||
included.append((relative, path))
|
||||
included.sort(key=lambda item: str(item[0]))
|
||||
excluded.sort(key=lambda entry: entry["path"])
|
||||
return included, excluded
|
||||
|
||||
|
||||
def _size(path: Path) -> int:
|
||||
try:
|
||||
return path.stat().st_size
|
||||
except OSError:
|
||||
return 0
|
||||
|
||||
|
||||
def backup(*, source: Path, archive: Path, manifest_path: Path, label: str, max_bytes: int) -> dict:
|
||||
if not source.is_dir():
|
||||
raise BackupError(f"source {source} is not a directory")
|
||||
if archive.exists():
|
||||
raise BackupError(f"archive {archive} already exists; move it aside instead of overwriting a backup")
|
||||
|
||||
included, excluded = collect(source, max_bytes)
|
||||
files = [{"path": str(relative), "sha256": sha256_file(path), "bytes": _size(path)} for relative, path in included]
|
||||
counts = record_counts(source / "manifest.jsonl")
|
||||
|
||||
archive.parent.mkdir(parents=True, exist_ok=True)
|
||||
partial = archive.with_name(archive.name + ".partial")
|
||||
with partial.open("wb") as raw:
|
||||
# gzip mtime=0 and zeroed tar metadata keep two backups of the same records identical.
|
||||
with gzip.GzipFile(fileobj=raw, mode="wb", mtime=0) as compressed:
|
||||
with tarfile.open(fileobj=compressed, mode="w", format=tarfile.PAX_FORMAT) as tar:
|
||||
for (relative, path), entry in zip(included, files):
|
||||
payload = path.read_bytes()
|
||||
info = tarfile.TarInfo(PAYLOAD_PREFIX + str(relative))
|
||||
info.size = len(payload)
|
||||
info.mode = 0o644
|
||||
info.mtime = 0
|
||||
info.uid = info.gid = 0
|
||||
info.uname = info.gname = ""
|
||||
tar.addfile(info, io.BytesIO(payload))
|
||||
if hashlib.sha256(payload).hexdigest() != entry["sha256"]:
|
||||
raise BackupError(f"{relative} changed while it was being archived")
|
||||
os.replace(partial, archive)
|
||||
|
||||
manifest = {
|
||||
"schemaVersion": SCHEMA_VERSION,
|
||||
"tool": TOOL,
|
||||
"createdAt": datetime.now(timezone.utc).isoformat(timespec="seconds"),
|
||||
"label": label,
|
||||
"sourceLabel": source.name,
|
||||
"archive": {"name": archive.name, "sha256": sha256_file(archive), "bytes": _size(archive)},
|
||||
"records": counts,
|
||||
"files": files,
|
||||
"excluded": excluded,
|
||||
}
|
||||
manifest_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
return {
|
||||
"archive": str(archive),
|
||||
"manifest": str(manifest_path),
|
||||
"files": len(files),
|
||||
"bytes": manifest["archive"]["bytes"],
|
||||
"records": counts,
|
||||
"excluded": {entry["reason"]: sum(1 for other in excluded if other["reason"] == entry["reason"]) for entry in excluded},
|
||||
}
|
||||
|
||||
|
||||
def load_manifest(manifest_path: Path) -> dict:
|
||||
if not manifest_path.is_file():
|
||||
raise BackupError(f"manifest {manifest_path} does not exist")
|
||||
try:
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
except json.JSONDecodeError as error:
|
||||
raise BackupError(f"manifest {manifest_path} is not valid JSON: {error}") from error
|
||||
if manifest.get("schemaVersion") != SCHEMA_VERSION:
|
||||
raise BackupError(f"manifest schemaVersion {manifest.get('schemaVersion')!r} is not {SCHEMA_VERSION}")
|
||||
return manifest
|
||||
|
||||
|
||||
def verified_members(archive: Path, manifest: dict) -> list[tuple[str, bytes]]:
|
||||
"""Check every member of the archive before anything is written to disk."""
|
||||
if not archive.is_file():
|
||||
raise BackupError(f"archive {archive} does not exist")
|
||||
expected = {entry["path"]: entry for entry in manifest.get("files") or []}
|
||||
extracted: list[tuple[str, bytes]] = []
|
||||
seen: set[str] = set()
|
||||
with tarfile.open(archive, "r:gz") as tar:
|
||||
for info in tar.getmembers():
|
||||
if info.isdir():
|
||||
continue
|
||||
if info.issym() or info.islnk():
|
||||
raise BackupError(f"archive member {info.name!r} is a link, not a regular file")
|
||||
if not info.isfile():
|
||||
raise BackupError(f"archive member {info.name!r} is not a regular file")
|
||||
if os.path.isabs(info.name):
|
||||
raise BackupError(f"archive member {info.name!r} is an absolute path")
|
||||
if not info.name.startswith(PAYLOAD_PREFIX):
|
||||
raise BackupError(f"archive member {info.name!r} is not under {PAYLOAD_PREFIX}")
|
||||
relative = info.name[len(PAYLOAD_PREFIX):]
|
||||
parts = PurePosixPath(relative).parts
|
||||
if not parts or any(part in ("..", ".", "") for part in parts):
|
||||
raise BackupError(f"archive member {info.name!r} resolves outside the restore root")
|
||||
if relative not in expected:
|
||||
raise BackupError(f"archive member {info.name!r} is not listed in the manifest")
|
||||
handle = tar.extractfile(info)
|
||||
if handle is None:
|
||||
raise BackupError(f"archive member {info.name!r} cannot be read")
|
||||
payload = handle.read()
|
||||
if hashlib.sha256(payload).hexdigest() != expected[relative]["sha256"]:
|
||||
raise BackupError(
|
||||
f"archive member {info.name!r} does not match the manifest digest "
|
||||
"(the archive changed after it was recorded)"
|
||||
)
|
||||
seen.add(relative)
|
||||
extracted.append((relative, payload))
|
||||
missing = sorted(set(expected) - seen)
|
||||
if missing:
|
||||
raise BackupError(f"archive is missing members the manifest lists: {', '.join(missing)}")
|
||||
return extracted
|
||||
|
||||
|
||||
def verify(*, archive: Path, manifest_path: Path) -> dict:
|
||||
manifest = load_manifest(manifest_path)
|
||||
members_list = verified_members(archive, manifest)
|
||||
return {"archive": str(archive), "files": len(members_list), "records": manifest.get("records", {})}
|
||||
|
||||
|
||||
def restore(*, archive: Path, manifest_path: Path, target: Path) -> dict:
|
||||
manifest = load_manifest(manifest_path)
|
||||
if target.exists():
|
||||
if not target.is_dir():
|
||||
raise BackupError(f"restore target {target} is not a directory")
|
||||
if any(target.iterdir()):
|
||||
raise BackupError(f"restore target {target} is not empty; restore into a new directory")
|
||||
# Digests and member names are checked before the first byte is written, so a hostile or
|
||||
# tampered archive cannot leave a half-restored record set behind.
|
||||
members_list = verified_members(archive, manifest)
|
||||
target.mkdir(parents=True, exist_ok=True)
|
||||
for relative, payload in members_list:
|
||||
destination = target / relative
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
destination.write_bytes(payload)
|
||||
os.chmod(destination, 0o644)
|
||||
|
||||
counts = record_counts(target / "manifest.jsonl")
|
||||
if manifest.get("records") and counts != manifest["records"]:
|
||||
raise BackupError(
|
||||
f"restored record counts {counts} do not match the backup manifest {manifest['records']}"
|
||||
)
|
||||
return {"target": str(target), "files": len(members_list), "records": counts}
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
backup_parser = subparsers.add_parser("backup", help="archive the records and write a manifest")
|
||||
backup_parser.add_argument("--source", required=True, type=Path)
|
||||
backup_parser.add_argument("--archive", required=True, type=Path)
|
||||
backup_parser.add_argument("--manifest", type=Path)
|
||||
backup_parser.add_argument("--label", default="manual")
|
||||
backup_parser.add_argument("--max-bytes", type=int, default=DEFAULT_MAX_BYTES)
|
||||
|
||||
verify_parser = subparsers.add_parser("verify", help="recompute every digest in an archive")
|
||||
verify_parser.add_argument("--archive", required=True, type=Path)
|
||||
verify_parser.add_argument("--manifest", required=True, type=Path)
|
||||
|
||||
restore_parser = subparsers.add_parser("restore", help="verify, then write the records into a new directory")
|
||||
restore_parser.add_argument("--archive", required=True, type=Path)
|
||||
restore_parser.add_argument("--manifest", required=True, type=Path)
|
||||
restore_parser.add_argument("--target", required=True, type=Path)
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
if args.command == "backup":
|
||||
manifest_path = args.manifest or args.archive.with_name(args.archive.name + ".manifest.json")
|
||||
receipt = backup(
|
||||
source=args.source, archive=args.archive, manifest_path=manifest_path,
|
||||
label=args.label, max_bytes=args.max_bytes,
|
||||
)
|
||||
elif args.command == "verify":
|
||||
receipt = verify(archive=args.archive, manifest_path=args.manifest)
|
||||
else:
|
||||
receipt = restore(archive=args.archive, manifest_path=args.manifest, target=args.target)
|
||||
except BackupError as error:
|
||||
print(f"refused: {error}", file=sys.stderr)
|
||||
return 2
|
||||
print(json.dumps(receipt, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -17,6 +17,10 @@ const version = process.env.TIMMY_RELEASE_VERSION || 'review build';
|
||||
// changed in this release instead of implying a feature that did not ship.
|
||||
const releaseDelta = process.env.TIMMY_RELEASE_DEMO_DELTA
|
||||
|| 'Verified rebuild of the current user-visible feature path — no new feature is claimed.';
|
||||
// The recording targets the release builder's own acceptance server by default. A caller that
|
||||
// names another origin is explicit about it, so a local run can never silently record a different
|
||||
// tree than the one the gates measured.
|
||||
const demoUrl = process.env.TIMMY_DEMO_URL || 'http://127.0.0.1:4173';
|
||||
const captionsOut = process.env.TIMMY_DEMO_CAPTIONS_OUT || '';
|
||||
const captionsShown = [];
|
||||
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
@@ -86,7 +90,9 @@ const suggestionBody = {
|
||||
warning: 'Visual suggestion only. Confirm it yourself; this is not a diagnosis.',
|
||||
};
|
||||
let analyzeOverride = null;
|
||||
let analyzeCalls = 0;
|
||||
await page.route('**/api/analyze', route => {
|
||||
analyzeCalls += 1;
|
||||
const response = analyzeOverride || { status: 200, body: suggestionBody };
|
||||
return route.fulfill({ status: response.status, contentType: 'application/json', body: JSON.stringify(response.body) });
|
||||
});
|
||||
@@ -116,7 +122,7 @@ await page.route('**/api/agent/chat', route => {
|
||||
});
|
||||
});
|
||||
|
||||
await page.goto('http://127.0.0.1:4173', { waitUntil: 'networkidle' });
|
||||
await page.goto(demoUrl, { waitUntil: 'networkidle' });
|
||||
await page.evaluate(() => localStorage.clear());
|
||||
await page.reload({ waitUntil: 'networkidle' });
|
||||
|
||||
@@ -136,9 +142,13 @@ async function caption(text, duration = 1400) {
|
||||
node.textContent = text;
|
||||
// Inline styles, not a stylesheet: a stylesheet belongs to one document and the demo reloads the
|
||||
// page, which silently stripped caption styling and left every later caption clipped body text.
|
||||
// The bar is opaque, not 94%: a translucent bar let the app's own text show through a long
|
||||
// release-delta caption and read as run-together garbage. Long copy also steps down a size so a
|
||||
// paragraph-length caption keeps its bottom padding instead of pressing the last line to the edge.
|
||||
const size = text.length > 380 ? 9.5 : text.length > 220 ? 10.5 : 12;
|
||||
node.style.cssText = 'position:fixed;left:16px;right:16px;top:14px;z-index:20000;'
|
||||
+ 'background:rgba(40,33,29,.94);color:#fff;padding:11px 14px;border-radius:14px;'
|
||||
+ 'font:800 12px/1.25 system-ui;letter-spacing:.02em;text-align:center;'
|
||||
+ 'background:#28211d;color:#fff;padding:12px 14px 14px;border-radius:14px;'
|
||||
+ `font:800 ${size}px/1.32 system-ui;letter-spacing:.02em;text-align:center;`
|
||||
+ 'box-shadow:0 8px 24px rgba(0,0,0,.2)';
|
||||
document.body.append(node);
|
||||
node.animate([{ opacity: 0, transform: 'translateY(-8px)' }, { opacity: 1, transform: 'translateY(0)' }], { duration: 260, fill: 'forwards' });
|
||||
@@ -152,18 +162,27 @@ async function caption(text, duration = 1400) {
|
||||
const rect = node.getBoundingClientRect();
|
||||
const style = getComputedStyle(node);
|
||||
const topmost = document.elementFromPoint(Math.round(rect.left + rect.width / 2), Math.round(rect.top + rect.height / 2));
|
||||
// Proving the string was pushed is not proving it was readable. Measure the last line's own box,
|
||||
// so a paragraph caption pressed flat against the bar's bottom edge fails here instead of shipping.
|
||||
const range = document.createRange();
|
||||
range.selectNodeContents(node);
|
||||
const lines = Array.from(range.getClientRects()).filter(box => box.height > 1);
|
||||
const lastBottom = lines.length ? Math.max(...lines.map(box => box.bottom)) : rect.bottom;
|
||||
const padBottom = Math.round(rect.bottom - lastBottom);
|
||||
return {
|
||||
ok: style.position === 'fixed' && style.zIndex === '20000' && rect.top <= 40 && rect.height >= 24
|
||||
&& rect.width > 200 && rect.bottom <= window.innerHeight && (topmost === node || (topmost && node.contains(topmost))),
|
||||
&& rect.width > 200 && rect.bottom <= window.innerHeight && (topmost === node || (topmost && node.contains(topmost)))
|
||||
&& lines.length > 0 && padBottom >= 3,
|
||||
top: Math.round(rect.top), height: Math.round(rect.height), position: style.position,
|
||||
zIndex: style.zIndex, background: style.backgroundColor, topmost: topmost === node,
|
||||
padBottom, lines: lines.length,
|
||||
};
|
||||
});
|
||||
const shownBand = await captionBand().catch(() => null);
|
||||
const bandChanged = Boolean(blankBand && shownBand && !blankBand.equals(shownBand));
|
||||
if (!painted.ok) throw new Error(`Caption is not an on-screen bar: ${text} ${JSON.stringify(painted)}`);
|
||||
if (!bandChanged) throw new Error(`Caption painted nothing into the caption band: ${text}`);
|
||||
captionEvidence.push({ text, visible: true, position: painted.position, top: painted.top, height: painted.height, bandChanged });
|
||||
captionEvidence.push({ text, visible: true, position: painted.position, top: painted.top, height: painted.height, bandChanged, padBottom: painted.padBottom, lines: painted.lines });
|
||||
await sleep(duration);
|
||||
// A caption must not outlive its stated duration. It used to stay on screen until the next one
|
||||
// replaced it, so a view change left the previous screen's claim hanging over a different screen —
|
||||
@@ -293,6 +312,24 @@ await indicate('label[for="camera-photo"]');
|
||||
await page.locator('#camera-photo').dispatchEvent('cancel');
|
||||
await page.getByText(/Camera or photo picker closed/i).waitFor();
|
||||
await caption('Camera closed cleanly — gallery and manual logging are still available', 1600);
|
||||
// Issue #16, on camera: the file below is committed as text and only claims to be a photo. It must
|
||||
// be refused on the device, the refusal must say nothing was uploaded, and the app must never post
|
||||
// it — asserted here, not narrated.
|
||||
await indicate('label[for="gallery-photo"]');
|
||||
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/not-a-photo.jpg');
|
||||
await page.locator('[data-capture-error="unreadable"]').waitFor();
|
||||
const unreadableCopy = await page.locator('[data-capture-error="unreadable"]').innerText();
|
||||
if (!/isn’t a photo Timmy can read/.test(unreadableCopy) || !/Nothing was uploaded/.test(unreadableCopy)) {
|
||||
throw new Error(`An unreadable file must be refused with the upload-free copy, saw: ${unreadableCopy}`);
|
||||
}
|
||||
if (await page.getByText(/couldn’t analyze that safely/i).count() !== 0) {
|
||||
throw new Error('A file that never left the device must not be reported as a failed analysis');
|
||||
}
|
||||
if (analyzeCalls !== 0) throw new Error('An unreadable file was posted to /api/analyze');
|
||||
await caption('A file that only claims to be a photo is refused on the device — nothing is uploaded', 2600);
|
||||
await indicate('[data-capture-error="unreadable"]');
|
||||
await caption('Manual logging stays one tap away, and the photo picker is one tap back', 2000);
|
||||
await tap('#retake-photo', 450);
|
||||
await indicate('label[for="gallery-photo"]');
|
||||
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/synthetic-type4.jpg');
|
||||
await caption('Nothing uploads until explicit consent', 1100);
|
||||
@@ -1148,6 +1185,198 @@ await caption('Each candidate is scored on test at a threshold chosen on validat
|
||||
await caption('One contributor spread across two splits is refused before anything is scored, and a passing verdict is proven only by a fixture engineered to clear it', 3800);
|
||||
await page.evaluate(() => document.querySelector('#calibration-report')?.remove());
|
||||
|
||||
// Issue #40 — the RC attack suite is a command line too, and unlike the other cards this one is
|
||||
// expected to come back with a verdict that is not flattering. Hostile synthetic bytes go at the
|
||||
// real server with a counting stub worker behind it, so "the worker was called" is measured, not
|
||||
// inferred. The card prints whatever verdict the suite derived from what it observed; the captions
|
||||
// name where a failing fix lives instead of implying the boundary is already safe.
|
||||
async function securityAttackSuite() {
|
||||
const directory = await mkdtemp(path.join(os.tmpdir(), 'timmy-security-demo-'));
|
||||
const reportPath = path.join(directory, 'rc-attack-report.json');
|
||||
try {
|
||||
const run = await execFileAsync(
|
||||
process.execPath,
|
||||
['scripts/security_attack_suite.mjs', '--json', reportPath],
|
||||
{ cwd: process.cwd() },
|
||||
).then(result => ({ code: 0, stdout: result.stdout, stderr: result.stderr }))
|
||||
.catch(error => ({ code: typeof error.code === 'number' ? error.code : 1, stdout: error.stdout || '', stderr: error.stderr || '' }));
|
||||
if (![0, 1].includes(run.code)) throw new Error(`the attack suite could not complete: ${run.stderr.trim()}`);
|
||||
const report = JSON.parse(await readFile(reportPath, 'utf8'));
|
||||
if (run.code !== (report.verdict === 'BLOCKED' ? 1 : 0)) {
|
||||
throw new Error('the attack suite exit code must match the verdict it published');
|
||||
}
|
||||
return { report, run };
|
||||
} finally {
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
const securityRun = await securityAttackSuite();
|
||||
const securityReport = securityRun.report;
|
||||
if (securityReport.cases.length < 8) throw new Error('the RC attack suite must run every named abuse class');
|
||||
await tap('[data-view="home"]', 450);
|
||||
const caseDetail = (item) => {
|
||||
const observed = item.observed === null || typeof item.observed !== 'object' ? {} : item.observed;
|
||||
// Print each observed field the case actually recorded. A case with no `status` used to render
|
||||
// `http undefined` on camera, and one with no `providerCalls` rendered its status twice — both read
|
||||
// as a broken demo instead of the measurement it is.
|
||||
const parts = [];
|
||||
if ('status' in observed) parts.push(`http ${observed.status}`);
|
||||
if ('refused' in observed) parts.push(`refused ${observed.refused}`);
|
||||
if ('providerCalls' in observed) parts.push(`worker ${observed.providerCalls}`);
|
||||
if ('providerTextLeaked' in observed) parts.push(`text leaked ${observed.providerTextLeaked}`);
|
||||
if ('repeatedAnswerIdentical' in observed) parts.push(`identical ${observed.repeatedAnswerIdentical}`);
|
||||
if ('imageBytesInLog' in observed) parts.push(`image bytes ${observed.imageBytesInLog}`);
|
||||
return parts.length ? parts.join(' · ') : item.status;
|
||||
};
|
||||
const securityRows = [
|
||||
['command', 'node scripts/security_attack_suite.mjs'],
|
||||
['target', 'real server · counting stub worker · synthetic bytes'],
|
||||
['—', ''],
|
||||
];
|
||||
for (const item of securityReport.cases) securityRows.push([item.category, `${item.severity} ${item.status} · ${caseDetail(item)}`]);
|
||||
securityRows.push(['—', '']);
|
||||
securityRows.push(['load', `${securityReport.load.requests} req · p50 ${securityReport.load.p50}ms · p95 ${securityReport.load.p95}ms · abstain ${securityReport.load.abstentions}`]);
|
||||
securityRows.push(['verdict', `${securityReport.verdict} · ${securityReport.blocking.length} blocking · ${securityReport.holds.length} held`]);
|
||||
for (const item of securityReport.blocking) securityRows.push(['blocks', item.id]);
|
||||
const securityCard = await evidenceCard({
|
||||
id: 'security-attack',
|
||||
title: 'SECURITY ATTACK SUITE',
|
||||
subtitle: 'command line, not app UI — real server, counted worker calls',
|
||||
rows: securityRows,
|
||||
font: '600 9.5px/1.45 ui-monospace, SFMono-Regular, Menlo, monospace',
|
||||
keyWidth: 62,
|
||||
gap: '1px',
|
||||
rowGap: '5px',
|
||||
});
|
||||
await sleep(420);
|
||||
const securityCardProof = await evidenceCardProof('security-attack');
|
||||
if (!securityCardProof.ok) throw new Error(`Security card is not fully readable on screen: ${JSON.stringify(securityCardProof)}`);
|
||||
for (const item of securityReport.blocking) {
|
||||
if (!securityCard.text.includes(item.id)) throw new Error(`the card must name the blocking case ${item.id}`);
|
||||
}
|
||||
if (!securityCard.text.includes(securityReport.verdict)) throw new Error('the card must show the verdict the suite derived');
|
||||
if (!securityCard.text.includes(`${securityReport.load.p95}ms`)) throw new Error('the card must show the p95 the load phase measured');
|
||||
await caption('Every hostile upload on screen went at the real server, and the worker behind it counted every call it received', 3800);
|
||||
await caption('Refusing bytes by their content is the fix in review, not a claim this video makes: an HTML file dressed as a JPEG still reaches the worker today', 4200);
|
||||
await caption('A failing case is a finding the release job has to see, so the card prints the verdict the suite derived instead of a reassuring one', 3800);
|
||||
await caption('The gate blocks RC approval on any P0 case that reached the worker — the gate decides, not the caption', 3400);
|
||||
await page.evaluate(() => document.querySelector('#security-attack')?.remove());
|
||||
|
||||
// Issue #39 — the operator backup is a command line, not a screen in the app. The app itself keeps
|
||||
// no server-side journal, so what an operator can actually lose is the corpus: the append-only
|
||||
// manifest with its withdrawal tombstones. This scene builds a synthetic data root with the real
|
||||
// ingest and revoke commands, plants the four things a backup must refuse, then runs the real
|
||||
// backup, verify, and restore and reads every value on the card back out of those runs.
|
||||
async function operatorBackupAudit() {
|
||||
const root = await mkdtemp(path.join(os.tmpdir(), 'timmy-backup-demo-'));
|
||||
const dataRoot = path.join(root, 'srv-timmy-corpus');
|
||||
const archive = path.join(root, 'timmy-nightly.tar.gz');
|
||||
const manifestPath = `${archive}.manifest.json`;
|
||||
const restoreInto = path.join(root, 'restored');
|
||||
const secondPhoto = path.join(root, 'synthetic-second.png');
|
||||
const backupCli = args => execFileAsync('python3', ['scripts/operator_backup.py', ...args], { cwd: process.cwd() });
|
||||
const ingest = (input, contributionId, subject, bristol) => corpusCli([
|
||||
'scripts/ingest_training_photo.py',
|
||||
'--input', input, '--output-dir', dataRoot, '--subject-id', subject,
|
||||
'--consent-version', 'training-consent-v1', '--contribution-id', contributionId,
|
||||
'--bristol-type', bristol, '--color', 'brown', '--quality', 'good', '--reviewer', 'user-confirmed',
|
||||
]);
|
||||
try {
|
||||
// One contribution stays active and one is withdrawn, so the manifest ends up with a live sample
|
||||
// and a real tombstone, and the withdrawn derived image is already gone from disk.
|
||||
await ingest('tests/fixtures/synthetic-type4.jpg', 'demo-backup-0001', 'demo-backup-kept', '4');
|
||||
await writeFile(secondPhoto, Buffer.from('iVBORw0KGgoAAAANSUhEUgAAAAgAAAAICAIAAABLbSncAAAAFElEQVR4nGOsCDFhwAaYsIoOWgkAzUwBECP1Kh0AAAAASUVORK5CYII=', 'base64'));
|
||||
await ingest(secondPhoto, 'demo-backup-0002', 'demo-backup-withdrawn', '3');
|
||||
await corpusCli([
|
||||
'scripts/ingest_training_photo.py', '--output-dir', dataRoot,
|
||||
'--revoke', 'demo-backup-0002', '--reason', 'participant withdrew consent',
|
||||
]);
|
||||
|
||||
// The four things that must never be archived, as real files with real bytes: a weight shard, a
|
||||
// transient inference upload, an operator environment file, and a log. An exclusion that quietly
|
||||
// stopped working fails here instead of in a caption.
|
||||
await mkdir(path.join(dataRoot, 'weights'), { recursive: true });
|
||||
await writeFile(path.join(dataRoot, 'weights', 'model.gguf'), 'synthetic weight shard — never archived');
|
||||
await mkdir(path.join(dataRoot, 'transient'), { recursive: true });
|
||||
await writeFile(path.join(dataRoot, 'transient', 'inference-upload.jpg'), 'transient inference bytes — never archived');
|
||||
await writeFile(path.join(dataRoot, 'secrets.env'), 'TIMMY_DATASET_SALT=demo-only-placeholder\n');
|
||||
await writeFile(path.join(dataRoot, 'operator.log'), 'GET /api/healthz 200\n');
|
||||
|
||||
const backupRun = await backupCli(['backup', '--source', dataRoot, '--archive', archive, '--label', 'nightly']);
|
||||
const receipt = JSON.parse(backupRun.stdout);
|
||||
const manifest = JSON.parse(await readFile(manifestPath, 'utf8'));
|
||||
const reasons = manifest.excluded.reduce((out, entry) => ({ ...out, [entry.reason]: (out[entry.reason] || 0) + 1 }), {});
|
||||
if (manifest.files.some(entry => /model\.gguf|inference-upload|secrets\.env|operator\.log/.test(entry.path))) {
|
||||
throw new Error('the backup must not archive the weight shard or a transient inference upload');
|
||||
}
|
||||
for (const reason of ['model-weights', 'transient-inference', 'secret', 'log']) {
|
||||
if (!reasons[reason]) throw new Error(`the backup must record a ${reason} exclusion`);
|
||||
}
|
||||
if (receipt.records.samples !== 2 || receipt.records.tombstones !== 1 || receipt.records.activeSamples !== 1) {
|
||||
throw new Error('the backup receipt must report the real record counts');
|
||||
}
|
||||
const verified = JSON.parse((await backupCli(['verify', '--archive', archive, '--manifest', manifestPath])).stdout);
|
||||
if (verified.files !== manifest.files.length) throw new Error('verify must re-hash every stored member');
|
||||
const restored = JSON.parse((await backupCli([
|
||||
'restore', '--archive', archive, '--manifest', manifestPath, '--target', restoreInto,
|
||||
])).stdout);
|
||||
if (restored.records.tombstones !== 1 || restored.records.activeSamples !== 1) {
|
||||
throw new Error('the restore must return the same record counts');
|
||||
}
|
||||
const restoredManifest = await readFile(path.join(restoreInto, 'manifest.jsonl'));
|
||||
if (!restoredManifest.equals(await readFile(path.join(dataRoot, 'manifest.jsonl')))) {
|
||||
throw new Error('the restore must return the same records, tombstone included');
|
||||
}
|
||||
return { receipt, manifest, reasons, verified, restored };
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
const opBackup = await operatorBackupAudit();
|
||||
await tap('[data-view="home"]', 400);
|
||||
const shortDigest = value => String(value).slice(0, 12);
|
||||
const backupRows = [
|
||||
['command', 'python3 scripts/operator_backup.py backup --source srv-timmy-corpus'],
|
||||
['stored', `${opBackup.receipt.files} members · ${opBackup.manifest.files.map(entry => entry.path.split('/').pop()).join(' + ')}`],
|
||||
['records', `samples ${opBackup.receipt.records.samples} · tombstones ${opBackup.receipt.records.tombstones} · active ${opBackup.receipt.records.activeSamples}`],
|
||||
['—', ''],
|
||||
['refused', `model-weights ${opBackup.reasons['model-weights']} · transient-inference ${opBackup.reasons['transient-inference']}`],
|
||||
['refused', `secret ${opBackup.reasons['secret']} · log ${opBackup.reasons['log']}`],
|
||||
['reason', 'every skipped path is named in the manifest, not silently dropped'],
|
||||
['—', ''],
|
||||
['digest', `sha256 ${shortDigest(opBackup.manifest.archive.sha256)}… · reproducible (sorted, mtime 0)`],
|
||||
['verify', `ok · ${opBackup.verified.files} members re-hashed against the manifest`],
|
||||
['restore', `ok · samples ${opBackup.restored.records.samples} · tombstones ${opBackup.restored.records.tombstones} · active ${opBackup.restored.records.activeSamples}`],
|
||||
['manifest', 'returned byte-for-byte, tombstone included'],
|
||||
['server', 'the app holds no journal of its own — there is no server copy to back up'],
|
||||
];
|
||||
const backupCard = await evidenceCard({
|
||||
id: 'production-backup',
|
||||
title: 'OPERATOR BACKUP',
|
||||
subtitle: 'command line, not app UI — real run, synthetic corpus',
|
||||
rows: backupRows,
|
||||
font: '600 9.5px/1.45 ui-monospace, SFMono-Regular, Menlo, monospace',
|
||||
keyWidth: 66,
|
||||
gap: '1px',
|
||||
rowGap: '6px',
|
||||
});
|
||||
await sleep(420);
|
||||
const backupProof = await evidenceCardProof('production-backup');
|
||||
if (!backupProof.ok) throw new Error(`Production backup card is not fully readable on screen: ${JSON.stringify(backupProof)}`);
|
||||
// The card is drawn from the run: prove the counts and the refusals on it are the ones it produced.
|
||||
if (!backupCard.text.includes(`samples ${opBackup.receipt.records.samples} · tombstones ${opBackup.receipt.records.tombstones}`)) {
|
||||
throw new Error('the card must show the record counts the backup actually reported');
|
||||
}
|
||||
if (!backupCard.text.includes('model-weights 1')) throw new Error('the card must show the exclusion the run actually recorded');
|
||||
if (!backupCard.text.includes(shortDigest(opBackup.manifest.archive.sha256))) {
|
||||
throw new Error('the card must show the archive digest the run actually wrote');
|
||||
}
|
||||
await caption('The operator backup is a command line, not a screen in the app — this runs it against a synthetic corpus the real ingest and revoke commands built', 3800);
|
||||
await caption('Model weights and transient inference images are refused by name, and every skipped path is listed with its reason', 3600);
|
||||
await caption('Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes', 3400);
|
||||
await caption('A restore returns the same records, including the withdrawal tombstone a trainer must never lose', 3600);
|
||||
await page.evaluate(() => document.querySelector('#production-backup')?.remove());
|
||||
|
||||
// Back to the privacy view for the portability scene: the privacy entry lives on the journal view.
|
||||
await tap('[data-view="calendar"]', 550);
|
||||
await tap('[data-view="privacy"]', 600);
|
||||
|
||||
@@ -0,0 +1,332 @@
|
||||
#!/usr/bin/env node
|
||||
// RC attack suite (#40): attack the release candidate before users do.
|
||||
//
|
||||
// Every case drives the shipping `server.mjs` with synthetic hostile bytes while a stub worker counts
|
||||
// the calls that reach it. The suite records what it observed and derives a verdict from the
|
||||
// observation; it never asserts a result it did not see. No real medical image, no live user data,
|
||||
// no third-party provider is involved, and the provider is a counting stub, so "reached the worker"
|
||||
// is provable rather than inferred.
|
||||
//
|
||||
// Usage: node scripts/security_attack_suite.mjs [--json <path>] [--base-url <url>]
|
||||
// Exit: 0 = GO, 1 = BLOCKED (a P0 finding, or a required category with no executable evidence).
|
||||
import { randomBytes } from 'node:crypto';
|
||||
import { spawn } from 'node:child_process';
|
||||
import { access, mkdir, readFile, writeFile } from 'node:fs/promises';
|
||||
import { dirname, join } from 'node:path';
|
||||
import http from 'node:http';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { buildReport, containsImageBytes, latencyPercentiles } from '../src/release-security-gate.js';
|
||||
|
||||
const repoRoot = fileURLToPath(new URL('..', import.meta.url));
|
||||
const argument = (name, fallback) => {
|
||||
const index = process.argv.indexOf(name);
|
||||
return index === -1 ? fallback : process.argv[index + 1];
|
||||
};
|
||||
const reportPath = argument('--json', join(repoRoot, 'artifacts/security/rc-attack-report.json'));
|
||||
const explicitBaseUrl = argument('--base-url', '');
|
||||
|
||||
function freePort() {
|
||||
return new Promise(resolve => {
|
||||
const probe = http.createServer();
|
||||
probe.listen(0, '127.0.0.1', () => {
|
||||
const { port } = probe.address();
|
||||
probe.close(() => resolve(port));
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
// A counting OpenAI-compatible worker. `state.content` is the answer the next request receives, and
|
||||
// `state.requests` is the evidence that a request crossed the provider boundary at all.
|
||||
function startStubWorker() {
|
||||
const state = { content: '', requests: 0 };
|
||||
const server = http.createServer((req, res) => {
|
||||
if (req.url.endsWith('/models')) {
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
return res.end(JSON.stringify({ data: [{ id: 'stub-vision-worker' }] }));
|
||||
}
|
||||
req.resume();
|
||||
req.on('end', () => {
|
||||
state.requests += 1;
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ choices: [{ message: { content: state.content } }] }));
|
||||
});
|
||||
});
|
||||
return new Promise(resolve => server.listen(0, '127.0.0.1', () => resolve({
|
||||
state,
|
||||
port: server.address().port,
|
||||
close: () => new Promise(done => server.close(done)),
|
||||
})));
|
||||
}
|
||||
|
||||
// The real server, with its own stdout/stderr captured from the first byte: the log scan is only
|
||||
// worth something if it read a log that actually exists.
|
||||
async function startTarget({ providerPort, port, env = {} }) {
|
||||
const child = spawn(process.execPath, ['server.mjs'], {
|
||||
cwd: repoRoot,
|
||||
env: {
|
||||
...process.env,
|
||||
HOST: '127.0.0.1',
|
||||
PORT: String(port),
|
||||
TIMMY_VISION_ENABLED: 'true',
|
||||
TIMMY_VISION_PROFILE: 'selfhost',
|
||||
TIMMY_VISION_BASE_URL: `http://127.0.0.1:${providerPort}/v1`,
|
||||
TIMMY_VISION_MODEL: 'stub-vision-worker',
|
||||
TIMMY_VISION_TIMEOUT_MS: '5000',
|
||||
...env,
|
||||
},
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
});
|
||||
const log = [];
|
||||
child.stdout.on('data', chunk => log.push(chunk.toString('utf8')));
|
||||
child.stderr.on('data', chunk => log.push(chunk.toString('utf8')));
|
||||
for (let attempt = 0; attempt < 100; attempt += 1) {
|
||||
if (log.join('').includes('Timmy is listening')) return { child, log };
|
||||
try {
|
||||
const response = await fetch(`http://127.0.0.1:${port}/api/healthz`);
|
||||
if (response.ok) return { child, log };
|
||||
} catch {}
|
||||
await new Promise(resolve => setTimeout(resolve, 100));
|
||||
}
|
||||
child.kill();
|
||||
throw new Error('attack suite target did not become ready');
|
||||
}
|
||||
|
||||
const stop = child => new Promise(resolve => {
|
||||
if (!child || child.exitCode !== null) return resolve();
|
||||
child.kill();
|
||||
child.once('exit', resolve);
|
||||
});
|
||||
|
||||
async function main() {
|
||||
const worker = await startStubWorker();
|
||||
const port = explicitBaseUrl ? null : await freePort();
|
||||
const baseUrl = explicitBaseUrl || `http://127.0.0.1:${port}`;
|
||||
const target = explicitBaseUrl ? null : await startTarget({ providerPort: worker.port, port });
|
||||
const photo = await readFile(join(repoRoot, 'tests/fixtures/synthetic-type4.jpg'));
|
||||
const imageDataUrl = `data:image/jpeg;base64,${photo.toString('base64')}`;
|
||||
|
||||
const post = async (body, targetBase = baseUrl) => {
|
||||
const before = worker.state.requests;
|
||||
const response = await fetch(`${targetBase}/api/analyze`, {
|
||||
method: 'POST',
|
||||
headers: { 'content-type': 'application/json' },
|
||||
body: typeof body === 'string' ? body : JSON.stringify(body),
|
||||
});
|
||||
const text = await response.text();
|
||||
return { status: response.status, text, providerCalls: worker.state.requests - before };
|
||||
};
|
||||
|
||||
const cases = [];
|
||||
const record = (entry) => { cases.push(entry); return entry; };
|
||||
|
||||
try {
|
||||
const health = await fetch(`${baseUrl}/api/healthz`).then(response => response.json());
|
||||
if (!health.visionEnabled) throw new Error('the attack suite needs vision analysis enabled on the target');
|
||||
|
||||
// 1. An oversized body must die at the request cap, before the photo is parsed or forwarded.
|
||||
const oversized = await post({ consent: true, imageDataUrl: `data:image/jpeg;base64,${'A'.repeat(6.6 * 1024 * 1024)}` });
|
||||
record({
|
||||
id: 'ingress-oversized-body',
|
||||
category: 'oversized',
|
||||
severity: 'P0',
|
||||
requirement: 'A body over the request cap is refused before any provider work.',
|
||||
expected: { status: 413, providerCalls: 0 },
|
||||
observed: { status: oversized.status, providerCalls: oversized.providerCalls },
|
||||
snippet: oversized.text.slice(0, 120),
|
||||
});
|
||||
|
||||
// 2. Bytes decide the format, not the data-URL label: HTML dressed as a JPEG is a polyglot.
|
||||
const html = Buffer.from('<html><body><script>alert(document.cookie)</script></body></html>');
|
||||
const polyglot = await post({ consent: true, imageDataUrl: `data:image/jpeg;base64,${html.toString('base64')}` });
|
||||
record({
|
||||
id: 'ingress-polyglot-html',
|
||||
category: 'polyglot',
|
||||
severity: 'P0',
|
||||
requirement: 'A payload whose bytes are HTML under an image/jpeg label is refused before the provider sees it.',
|
||||
expected: { status: 400, providerCalls: 0 },
|
||||
observed: { status: polyglot.status, providerCalls: polyglot.providerCalls },
|
||||
snippet: polyglot.text.slice(0, 120),
|
||||
});
|
||||
|
||||
// 3. Random bytes under a JPEG label are not an image either.
|
||||
const garbage = await post({ consent: true, imageDataUrl: `data:image/jpeg;base64,${randomBytes(64).toString('base64')}` });
|
||||
record({
|
||||
id: 'ingress-garbage-bytes',
|
||||
category: 'truncated',
|
||||
severity: 'P0',
|
||||
requirement: 'Undecodable bytes are refused rather than forwarded and interpreted.',
|
||||
expected: { status: 400, providerCalls: 0 },
|
||||
observed: { status: garbage.status, providerCalls: garbage.providerCalls },
|
||||
snippet: garbage.text.slice(0, 120),
|
||||
});
|
||||
|
||||
// 4. Consent is a precondition, and it is cheap to prove: no consent, no provider call.
|
||||
const noConsent = await post({ consent: false, imageDataUrl });
|
||||
record({
|
||||
id: 'consent-required',
|
||||
category: 'consent',
|
||||
severity: 'P0',
|
||||
requirement: 'Analysis without explicit consent is refused and never leaves the boundary.',
|
||||
expected: { status: 400, providerCalls: 0 },
|
||||
observed: { status: noConsent.status, providerCalls: noConsent.providerCalls },
|
||||
snippet: noConsent.text.slice(0, 120),
|
||||
});
|
||||
|
||||
// 5. A worker answer outside the schema must fail closed, and the provider's words must not
|
||||
// reach the user or the evidence file.
|
||||
worker.state.content = 'garbage-completion-token: isStool yes maybe, bristolType ten';
|
||||
const malformed = await post({ consent: true, imageDataUrl });
|
||||
record({
|
||||
id: 'model-output-malformed',
|
||||
category: 'malformed',
|
||||
severity: 'P0',
|
||||
requirement: 'A non-conforming provider answer is replaced by neutral copy and never echoed.',
|
||||
expected: { status: 503, providerTextLeaked: false },
|
||||
observed: { status: malformed.status, providerTextLeaked: /garbage-completion-token/.test(malformed.text) },
|
||||
snippet: malformed.text.slice(0, 120),
|
||||
});
|
||||
|
||||
// 6. Replay: this boundary holds no session state, so a replay is a second request. What must
|
||||
// hold is that it is answered identically and that nothing accumulates server-side.
|
||||
worker.state.content = JSON.stringify({ isStool: true, bristolType: 4, color: 'brown', confidence: 0.82, imageQuality: 'good', observations: 'Smooth, formed appearance.' });
|
||||
const first = await post({ consent: true, imageDataUrl });
|
||||
const second = await post({ consent: true, imageDataUrl });
|
||||
record({
|
||||
id: 'replay-duplicate-request',
|
||||
category: 'replay',
|
||||
severity: 'P2',
|
||||
requirement: 'A byte-identical replay is answered identically and stores nothing on the server.',
|
||||
expected: { status: 200, providerCalls: 1, repeatedAnswerIdentical: true },
|
||||
observed: { status: second.status, providerCalls: second.providerCalls, repeatedAnswerIdentical: first.text === second.text },
|
||||
snippet: second.text.slice(0, 120),
|
||||
});
|
||||
|
||||
// 7. Flood: concurrent requests must meet a bound instead of all reaching the worker at once.
|
||||
const beforeFlood = worker.state.requests;
|
||||
const flood = await Promise.all(Array.from({ length: 12 }, () => post({ consent: true, imageDataUrl })));
|
||||
record({
|
||||
id: 'flood-concurrency',
|
||||
category: 'flood',
|
||||
severity: 'P1',
|
||||
requirement: 'A burst of concurrent analyses meets a bounded queue or a rate refusal.',
|
||||
expected: { refused: true },
|
||||
observed: { refused: flood.some(item => item.status === 429 || item.status === 503), providerCalls: worker.state.requests - beforeFlood },
|
||||
snippet: `statuses ${[...new Set(flood.map(item => item.status))].sort().join(',')}`,
|
||||
});
|
||||
|
||||
// 8. Log inspection: the operator log is committed evidence, so it must not contain the photo.
|
||||
const capturedLog = (target?.log || []).join('');
|
||||
if (!capturedLog.trim()) throw new Error('no server log was captured, so the log scan would prove nothing');
|
||||
record({
|
||||
id: 'log-leak-scan',
|
||||
category: 'log',
|
||||
severity: 'P0',
|
||||
requirement: 'Server logs contain no image bytes, base64, or note text.',
|
||||
expected: { imageBytesInLog: false },
|
||||
observed: { imageBytesInLog: containsImageBytes(capturedLog) },
|
||||
snippet: `${capturedLog.trim().split('\n').length} log line(s) inspected`,
|
||||
});
|
||||
|
||||
// 9. Deletion is governed by the contribution ledger, not by this HTTP boundary. Covering it
|
||||
// here means naming the executable test that does it, and only counting it once verified.
|
||||
let evidenceVerified = false;
|
||||
try {
|
||||
await access(join(repoRoot, 'tests/training-contribution.test.js'));
|
||||
evidenceVerified = true;
|
||||
} catch {}
|
||||
record({
|
||||
id: 'deletion-withdrawal',
|
||||
category: 'deletion',
|
||||
severity: 'P0',
|
||||
surface: 'out-of-surface',
|
||||
requirement: 'Withdrawal deletes the contribution and the server holds no copy of the photo.',
|
||||
evidence: 'tests/training-contribution.test.js',
|
||||
evidenceVerified,
|
||||
snippet: evidenceVerified ? 'covered by the contribution ledger suite' : 'named evidence file is missing',
|
||||
});
|
||||
|
||||
// Load: sequential valid analyses, abstaining answers, latency measured with the same boundary.
|
||||
// The abuse cases above spend the ingress request budget, so latency is measured against a second
|
||||
// target that starts with the budget wide open; otherwise these samples would be 429 refusals.
|
||||
const loadPort = explicitBaseUrl ? null : await freePort();
|
||||
const loadTarget = explicitBaseUrl
|
||||
? null
|
||||
: await startTarget({
|
||||
providerPort: worker.port,
|
||||
port: loadPort,
|
||||
env: { TIMMY_ANALYZE_RATE_LIMIT: '100000', TIMMY_ANALYZE_RATE_WINDOW_MS: '60000' },
|
||||
});
|
||||
const loadBase = explicitBaseUrl || `http://127.0.0.1:${loadPort}`;
|
||||
worker.state.content = JSON.stringify({ isStool: false, confidence: 0.9 });
|
||||
const samples = [];
|
||||
let errors = 0;
|
||||
let abstentions = 0;
|
||||
try {
|
||||
for (let index = 0; index < 8; index += 1) {
|
||||
const started = performance.now();
|
||||
const result = await post({ consent: true, imageDataUrl }, loadBase);
|
||||
samples.push(Math.round(performance.now() - started));
|
||||
if (result.status !== 200) errors += 1;
|
||||
else {
|
||||
try {
|
||||
if (JSON.parse(result.text).status === 'needs_user_input') abstentions += 1;
|
||||
} catch {}
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
await stop(loadTarget?.child);
|
||||
}
|
||||
const percentiles = latencyPercentiles(samples);
|
||||
const load = {
|
||||
requests: samples.length,
|
||||
errors,
|
||||
abstentions,
|
||||
p50: percentiles.p50,
|
||||
p95: percentiles.p95,
|
||||
max: percentiles.max,
|
||||
note: 'Latency measured against a counting stub on loopback; it measures this boundary, not a model.',
|
||||
};
|
||||
// Issue #16 adds a per-caller request budget at the ingress, so a suite that spends its budget on
|
||||
// the abuse cases above would measure 429 refusals here and publish them as analysis latency.
|
||||
// A refusal is not a sample: the load phase runs against its own permissive target instead.
|
||||
if (load.errors > 0) {
|
||||
throw new Error(`the load phase measured ${load.errors}/${load.requests} refusals instead of analyses, so it cannot publish latency`);
|
||||
}
|
||||
|
||||
const report = buildReport({
|
||||
generatedAt: new Date().toISOString(),
|
||||
target: {
|
||||
baseUrl,
|
||||
provider: 'stub counting worker (synthetic answers, no model loaded)',
|
||||
visionProfile: 'selfhost',
|
||||
release: health.release,
|
||||
commit: health.commit,
|
||||
},
|
||||
cases,
|
||||
load,
|
||||
});
|
||||
await mkdir(dirname(reportPath), { recursive: true });
|
||||
await writeFile(reportPath, `${JSON.stringify(report, null, 2)}\n`, 'utf8');
|
||||
|
||||
const width = Math.max(...cases.map(item => item.id.length));
|
||||
for (const item of report.cases) {
|
||||
const detail = item.status === 'fail' ? ` — ${item.mismatch}` : '';
|
||||
console.log(`${item.id.padEnd(width)} ${item.severity} ${item.status}${detail}`);
|
||||
}
|
||||
console.log(`load: ${load.requests} requests · p50 ${load.p50}ms · p95 ${load.p95}ms · errors ${load.errors} · abstentions ${load.abstentions}`);
|
||||
console.log(`report: ${reportPath}`);
|
||||
console.log(`VERDICT: ${report.verdict}`);
|
||||
if (report.verdict === 'BLOCKED') {
|
||||
for (const item of [...report.blocking, ...report.unproven]) console.log(` blocks RC: ${item.id} (${item.severity}) ${item.reason}`);
|
||||
}
|
||||
return report.verdict === 'GO' ? 0 : 1;
|
||||
} finally {
|
||||
await stop(target?.child);
|
||||
await worker.close();
|
||||
}
|
||||
}
|
||||
|
||||
main().then(code => { process.exitCode = code; }).catch(error => {
|
||||
console.error(`attack suite could not complete: ${error.message}`);
|
||||
process.exitCode = 2;
|
||||
});
|
||||
+45
-2
@@ -1,11 +1,15 @@
|
||||
import http from 'node:http';
|
||||
import { isIP } from 'node:net';
|
||||
import { createHash, randomBytes } from 'node:crypto';
|
||||
import { readFile, stat } from 'node:fs/promises';
|
||||
import { extname, join, normalize } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { analyzePhoto, visionFailureResponse } from './src/vision-service.js';
|
||||
import { probeVisionProvider, resolveVisionConfig } from './src/vision-config.js';
|
||||
import { AgentGatewayError, createHermesAgentService, resolveHermesAgentConfig } from './src/hermes-agent-service.js';
|
||||
import { readJsonBody } from './src/http-body.js';
|
||||
import { inspectImageBytes, IngressError } from './src/image-ingress.js';
|
||||
import { createRateLimiter } from './src/rate-limit.js';
|
||||
|
||||
const root=fileURLToPath(new URL('.',import.meta.url));
|
||||
const port=Number(process.env.PORT||4173);
|
||||
@@ -32,7 +36,27 @@ const stagingLabel=process.env.TIMMY_STAGING_LABEL?`Staging · ${release} · ${c
|
||||
|
||||
function sendJson(res,status,value){res.writeHead(status,{'content-type':'application/json; charset=utf-8','cache-control':'no-store','x-content-type-options':'nosniff'});res.end(JSON.stringify(value));}
|
||||
function escapeHtmlAttribute(value){return String(value).replace(/[&<>"]/g,character=>({'&':'&','<':'<','>':'>','"':'"'}[character]));}
|
||||
function readJson(req,maxBytes=6*1024*1024){return new Promise((resolve,reject)=>{let size=0,tooLarge=false;const chunks=[];req.on('data',chunk=>{size+=chunk.length;if(size>maxBytes){tooLarge=true;return}chunks.push(chunk)});req.on('end',()=>{if(tooLarge)return reject(new AgentGatewayError(413,'Request is too large.'));try{resolve(JSON.parse(Buffer.concat(chunks).toString('utf8')))}catch{return reject(new AgentGatewayError(400,'Invalid JSON request.'))}});req.on('error',reject)})}
|
||||
// Photo ingress (issue #16). Every limit here is decided from the request itself and fails before
|
||||
// any provider call: a per-caller request budget, a deadline for a body that never arrives, a byte
|
||||
// ceiling, and a container/pixel inspection of the decoded image. The bucket key is a salted hash
|
||||
// that lives only in this process's memory, so no address and no image byte is ever logged.
|
||||
const bodyTimeoutMs=Number(process.env.TIMMY_BODY_TIMEOUT_MS||15_000);
|
||||
const analyzeLimiter=createRateLimiter({limit:Number(process.env.TIMMY_ANALYZE_RATE_LIMIT||12),windowMs:Number(process.env.TIMMY_ANALYZE_RATE_WINDOW_MS||60_000)});
|
||||
const bucketSalt=randomBytes(16).toString('hex');
|
||||
function clientBucket(req){return createHash('sha256').update(`${bucketSalt}:${req.socket.remoteAddress||'unknown'}`).digest('hex').slice(0,24);}
|
||||
function readJson(req,maxBytes=6*1024*1024){return readJsonBody(req,{maxBytes,timeoutMs:bodyTimeoutMs});}
|
||||
function ingressGuard(payload){
|
||||
const raw=String(payload?.imageDataUrl||'');
|
||||
const match=/^data:(image\/(?:jpeg|png|webp));base64,([A-Za-z0-9+/=]+)$/.exec(raw);
|
||||
// A payload that is not shaped like a data URL is left to the semantic validator, so the user still
|
||||
// gets the specific "upload a JPEG, PNG, or WebP photo" copy rather than a second opinion here.
|
||||
if(!match)return null;
|
||||
return inspectImageBytes(Buffer.from(match[2],'base64'),{declaredMime:match[1]});
|
||||
}
|
||||
function sendIngressError(res,error){
|
||||
if(error instanceof IngressError)return sendJson(res,400,{error:error.message,code:error.code});
|
||||
return sendJson(res,error.status||400,{error:error.message,code:error.code});
|
||||
}
|
||||
function cookie(req,name){for(const part of String(req.headers.cookie||'').split(';')){const [key,...value]=part.trim().split('=');if(key===name)return decodeURIComponent(value.join('='))}return ''}
|
||||
function requestOrigin(req){return String(req.headers.origin||'').replace(/\/$/,'')}
|
||||
function rejectCrossSite(req){const site=String(req.headers['sec-fetch-site']||'');if(site&&site!=='same-origin')throw new AgentGatewayError(403,'Cross-site agent requests are not allowed.')}
|
||||
@@ -52,8 +76,27 @@ http.createServer(async(req,res)=>{
|
||||
return sendJson(res,200,{...visionConfig.publicStatus(),providerReady:provider.ready,modelSeen:provider.modelSeen,privacy});
|
||||
}
|
||||
if(appPath==='/api/analyze'&&req.method==='POST'){
|
||||
// Refusals first, in cost order: the request budget, then the body, then the image bytes. None of
|
||||
// these depend on the model, so hostile bytes are refused as hostile even when analysis is off —
|
||||
// otherwise the disabled switch hides the ingress boundary behind a 503 and the refusal is untested.
|
||||
const budget=analyzeLimiter.take(clientBucket(req));
|
||||
if(!budget.allowed){
|
||||
res.setHeader('retry-after',String(Math.max(1,Math.ceil(budget.retryAfterMs/1000))));
|
||||
return sendJson(res,429,{error:'Too many photos from this device. Wait a moment, then try again — your journal and manual logging still work.'});
|
||||
}
|
||||
let payload;
|
||||
try{payload=await readJson(req)}
|
||||
catch(error){
|
||||
const sent=sendIngressError(res,error);
|
||||
// The refusal is written first, then the half-read socket is dropped so a stalled body cannot
|
||||
// hold a connection or its partial buffer open.
|
||||
if(error?.code==='body_timeout'||error?.code==='too_large')req.destroy?.();
|
||||
return sent;
|
||||
}
|
||||
try{ingressGuard(payload)}
|
||||
catch(error){return sendIngressError(res,error)}
|
||||
// Bytes are known to be a real photo at this point, so the disabled switch is the honest answer.
|
||||
if(!visionConfig.enabled)return sendJson(res,503,{error:'AI analysis is disabled. Continue manually.'});
|
||||
const payload=await readJson(req);
|
||||
try{return sendJson(res,200,await analyzePhoto({payload,config:visionConfig}))}
|
||||
catch(error){const failure=visionFailureResponse(error);return sendJson(res,failure.status,failure.body)}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
// Bounded request-body reading for the API boundary (issue #16).
|
||||
//
|
||||
// Three limits have to fail before any provider work starts: the byte ceiling, a deadline for a body
|
||||
// that never arrives, and JSON validity. All three refuse with a client-visible status and a fixed
|
||||
// message; none of them logs or returns the body.
|
||||
|
||||
const COPY = Object.freeze({
|
||||
too_large: [413, 'Request is too large.'],
|
||||
invalid_json: [400, 'Invalid JSON request.'],
|
||||
body_timeout: [408, 'The request took too long to arrive. Try again.'],
|
||||
});
|
||||
|
||||
export class BodyError extends Error {
|
||||
constructor(code) {
|
||||
const [status, message] = COPY[code] || COPY.invalid_json;
|
||||
super(message);
|
||||
this.name = 'BodyError';
|
||||
this.code = code;
|
||||
this.status = status;
|
||||
}
|
||||
}
|
||||
|
||||
export function readJsonBody(req, { maxBytes = 6 * 1024 * 1024, timeoutMs = 15_000 } = {}) {
|
||||
return new Promise((resolve, reject) => {
|
||||
let settled = false;
|
||||
let tooLarge = false;
|
||||
let size = 0;
|
||||
const chunks = [];
|
||||
const finish = callback => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
clearTimeout(deadline);
|
||||
req.removeListener('data', onData);
|
||||
req.removeListener('end', onEnd);
|
||||
req.removeListener('error', onError);
|
||||
callback();
|
||||
};
|
||||
const deadline = setTimeout(() => {
|
||||
finish(() => {
|
||||
// Stop reading and let the caller answer 408; the server destroys the socket once the
|
||||
// refusal is written, so a half-read body cannot be held open indefinitely.
|
||||
req.pause?.();
|
||||
reject(new BodyError('body_timeout'));
|
||||
});
|
||||
}, timeoutMs);
|
||||
|
||||
function onData(chunk) {
|
||||
if (settled) return;
|
||||
// Past the ceiling the body is already refused: keep draining it without buffering so the
|
||||
// caller can still read the 413. Destroying the socket here would surface as "fetch failed"
|
||||
// and leave the caller with no status to act on.
|
||||
if (tooLarge) return;
|
||||
size += chunk.length;
|
||||
if (size > maxBytes) {
|
||||
tooLarge = true;
|
||||
chunks.length = 0;
|
||||
return;
|
||||
}
|
||||
chunks.push(chunk);
|
||||
}
|
||||
function onEnd() {
|
||||
finish(() => {
|
||||
if (tooLarge) return reject(new BodyError('too_large'));
|
||||
try {
|
||||
resolve(JSON.parse(Buffer.concat(chunks).toString('utf8')));
|
||||
} catch {
|
||||
reject(new BodyError('invalid_json'));
|
||||
}
|
||||
});
|
||||
}
|
||||
function onError(error) {
|
||||
finish(() => reject(error));
|
||||
}
|
||||
|
||||
req.on('data', onData);
|
||||
req.on('end', onEnd);
|
||||
req.on('error', onError);
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,158 @@
|
||||
// Transport-level hardening for the one endpoint that accepts image bytes (issue #16).
|
||||
//
|
||||
// The declared MIME type in a data URL is a claim made by the caller, never evidence, so every
|
||||
// decision here comes from the bytes: the container is sniffed, the container is walked to its own
|
||||
// end marker so nothing can ride along behind a valid image, embedded metadata segments are refused
|
||||
// (the browser always uploads a canvas re-encode, which carries none), and the pixel dimensions are
|
||||
// read from the header so a decode bomb never reaches the worker.
|
||||
//
|
||||
// Refusal copy is fixed and deliberate: it names what is wrong with the file and never echoes the
|
||||
// bytes, a hash, or any part of the upload back to the caller or into a log line.
|
||||
|
||||
export const MAX_EDGE = 4096;
|
||||
export const MAX_PIXELS = 16_000_000;
|
||||
|
||||
const COPY = Object.freeze({
|
||||
empty: 'The photo is empty.',
|
||||
not_an_image: 'That file is not a JPEG, PNG, or WebP photo.',
|
||||
container_mismatch: 'This photo is not the format it claims. Choose a JPEG, PNG, or WebP photo.',
|
||||
truncated: 'This photo is incomplete or damaged. Choose another photo.',
|
||||
trailing_data: 'This photo file has extra data after the image. Choose another photo.',
|
||||
metadata_present: 'This photo still carries embedded metadata. Choose it through Timmy’s camera or gallery picker so it is re-encoded without metadata.',
|
||||
dimensions_too_large: 'This photo is too large in pixels. Choose a smaller photo.',
|
||||
});
|
||||
|
||||
export class IngressError extends Error {
|
||||
constructor(code) {
|
||||
super(COPY[code] || COPY.not_an_image);
|
||||
this.name = 'IngressError';
|
||||
this.code = code;
|
||||
}
|
||||
}
|
||||
|
||||
const refuse = code => {
|
||||
throw new IngressError(code);
|
||||
};
|
||||
|
||||
const PNG_CHUNKS = { metadata: new Set(['eXIf', 'tEXt', 'zTXt', 'iTXt', 'tIME']) };
|
||||
// Segment markers that carry metadata rather than image data: APP1 (Exif/XMP), APP13 (Photoshop
|
||||
// resource block) and COM (free-text comment). The app's own upload is a canvas re-encode and
|
||||
// contains none of them, so a file that does was not produced by the capture path.
|
||||
const JPEG_METADATA_MARKERS = new Set([0xe1, 0xed, 0xfe]);
|
||||
const JPEG_SOF_MARKERS = new Set([0xc0, 0xc1, 0xc2, 0xc3, 0xc5, 0xc6, 0xc7, 0xc9, 0xca, 0xcb, 0xcd, 0xce, 0xcf]);
|
||||
|
||||
function sniff(bytes) {
|
||||
if (bytes.length >= 8 && bytes.subarray(0, 8).equals(Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]))) return 'image/png';
|
||||
if (bytes.length >= 3 && bytes[0] === 0xff && bytes[1] === 0xd8 && bytes[2] === 0xff) return 'image/jpeg';
|
||||
if (bytes.length >= 12 && bytes.toString('ascii', 0, 4) === 'RIFF' && bytes.toString('ascii', 8, 12) === 'WEBP') return 'image/webp';
|
||||
return '';
|
||||
}
|
||||
|
||||
function inspectPng(bytes) {
|
||||
let offset = 8;
|
||||
let width = 0;
|
||||
let height = 0;
|
||||
let sawHeader = false;
|
||||
while (offset + 8 <= bytes.length) {
|
||||
const length = bytes.readUInt32BE(offset);
|
||||
const type = bytes.toString('ascii', offset + 4, offset + 8);
|
||||
const dataStart = offset + 8;
|
||||
const next = dataStart + length + 4;
|
||||
if (next > bytes.length) refuse('truncated');
|
||||
if (!sawHeader) {
|
||||
if (type !== 'IHDR' || length !== 13) refuse('truncated');
|
||||
width = bytes.readUInt32BE(dataStart);
|
||||
height = bytes.readUInt32BE(dataStart + 4);
|
||||
sawHeader = true;
|
||||
}
|
||||
if (PNG_CHUNKS.metadata.has(type)) refuse('metadata_present');
|
||||
if (type === 'IEND') {
|
||||
if (next !== bytes.length) refuse('trailing_data');
|
||||
return { width, height };
|
||||
}
|
||||
offset = next;
|
||||
}
|
||||
return refuse('truncated');
|
||||
}
|
||||
|
||||
function inspectJpeg(bytes) {
|
||||
let offset = 2;
|
||||
let width = 0;
|
||||
let height = 0;
|
||||
while (offset + 4 <= bytes.length) {
|
||||
if (bytes[offset] !== 0xff) refuse('truncated');
|
||||
const marker = bytes[offset + 1];
|
||||
if (marker === 0xd9) {
|
||||
if (offset + 2 !== bytes.length) refuse('trailing_data');
|
||||
break;
|
||||
}
|
||||
// Standalone markers carry no length field.
|
||||
if (marker === 0x01 || (marker >= 0xd0 && marker <= 0xd7) || marker === 0xff) {
|
||||
offset += marker === 0xff ? 1 : 2;
|
||||
continue;
|
||||
}
|
||||
const length = bytes.readUInt16BE(offset + 2);
|
||||
if (length < 2 || offset + 2 + length > bytes.length) refuse('truncated');
|
||||
if (JPEG_METADATA_MARKERS.has(marker)) refuse('metadata_present');
|
||||
if (JPEG_SOF_MARKERS.has(marker) && length >= 7) {
|
||||
height = bytes.readUInt16BE(offset + 5);
|
||||
width = bytes.readUInt16BE(offset + 7);
|
||||
}
|
||||
offset += 2 + length;
|
||||
if (marker === 0xda) {
|
||||
// Entropy-coded scan data follows the start-of-scan header. The only thing left to trust is
|
||||
// the end-of-image marker, and it has to be the last two bytes of the file.
|
||||
if (bytes[bytes.length - 2] !== 0xff || bytes[bytes.length - 1] !== 0xd9) {
|
||||
return refuse(bytes.includes(Buffer.from([0xff, 0xd9])) ? 'trailing_data' : 'truncated');
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (!width || !height) refuse('truncated');
|
||||
return { width, height };
|
||||
}
|
||||
|
||||
function inspectWebp(bytes) {
|
||||
const declared = bytes.readUInt32LE(4) + 8;
|
||||
if (declared !== bytes.length) refuse(declared < bytes.length ? 'trailing_data' : 'truncated');
|
||||
let offset = 12;
|
||||
let width = 0;
|
||||
let height = 0;
|
||||
while (offset + 8 <= bytes.length) {
|
||||
const type = bytes.toString('ascii', offset, offset + 4);
|
||||
const size = bytes.readUInt32LE(offset + 4);
|
||||
const dataStart = offset + 8;
|
||||
if (dataStart + size > bytes.length) refuse('truncated');
|
||||
if (type === 'VP8X' && size >= 10) {
|
||||
width = bytes.readUIntLE(dataStart + 4, 3) + 1;
|
||||
height = bytes.readUIntLE(dataStart + 7, 3) + 1;
|
||||
} else if (type === 'VP8 ' && size >= 10) {
|
||||
width = bytes.readUInt16LE(dataStart + 6) & 0x3fff;
|
||||
height = bytes.readUInt16LE(dataStart + 8) & 0x3fff;
|
||||
} else if (type === 'VP8L' && size >= 5) {
|
||||
const bits = bytes.readUInt32LE(dataStart + 1);
|
||||
width = (bits & 0x3fff) + 1;
|
||||
height = ((bits >> 14) & 0x3fff) + 1;
|
||||
}
|
||||
offset = dataStart + size + (size % 2);
|
||||
}
|
||||
if (!width || !height) refuse('truncated');
|
||||
return { width, height };
|
||||
}
|
||||
|
||||
// Returns the real container, its pixel dimensions, and the byte count. Throws IngressError on
|
||||
// anything the model worker must never be handed.
|
||||
export function inspectImageBytes(bytes, { declaredMime = '' } = {}) {
|
||||
const buffer = Buffer.isBuffer(bytes) ? bytes : Buffer.from(bytes || []);
|
||||
if (!buffer.length) refuse('empty');
|
||||
const actual = sniff(buffer);
|
||||
if (!actual) refuse('not_an_image');
|
||||
if (declaredMime && declaredMime !== actual) refuse('container_mismatch');
|
||||
const dimensions = actual === 'image/png' ? inspectPng(buffer)
|
||||
: actual === 'image/jpeg' ? inspectJpeg(buffer)
|
||||
: inspectWebp(buffer);
|
||||
if (!dimensions.width || !dimensions.height) refuse('truncated');
|
||||
if (dimensions.width > MAX_EDGE || dimensions.height > MAX_EDGE
|
||||
|| dimensions.width * dimensions.height > MAX_PIXELS) refuse('dimensions_too_large');
|
||||
return { mime: actual, width: dimensions.width, height: dimensions.height, bytes: buffer.length };
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
// Bounded, in-process request budget for the photo-analysis endpoint (issue #16).
|
||||
//
|
||||
// The limiter exists so a caller cannot make Timmy the transport for a flood of images to the
|
||||
// model worker. It keeps timestamps only, never the bytes, the file name, or the caller's address:
|
||||
// the server hands it a short-lived opaque bucket key. Refusals never extend a window, so a caller
|
||||
// cannot earn a longer penalty than the request it already made.
|
||||
|
||||
export function createRateLimiter({ limit = 12, windowMs = 60_000, maxKeys = 2048, now = () => Date.now() } = {}) {
|
||||
const hits = new Map();
|
||||
let lastPrune = now();
|
||||
|
||||
function prune(at) {
|
||||
const cutoff = at - windowMs;
|
||||
for (const [key, stamps] of hits) {
|
||||
const kept = stamps.filter(stamp => stamp > cutoff);
|
||||
if (kept.length) hits.set(key, kept);
|
||||
else hits.delete(key);
|
||||
}
|
||||
lastPrune = at;
|
||||
}
|
||||
|
||||
return {
|
||||
take(key) {
|
||||
const at = now();
|
||||
// Pruning is at most once per window, plus whenever the map outgrows its ceiling, so the
|
||||
// bookkeeping stays bounded no matter how many distinct callers appear.
|
||||
if (at - lastPrune >= windowMs || hits.size > maxKeys) prune(at);
|
||||
const window = (hits.get(key) || []).filter(stamp => stamp > at - windowMs);
|
||||
if (window.length >= limit) {
|
||||
hits.set(key, window);
|
||||
return { allowed: false, remaining: 0, retryAfterMs: Math.max(0, window[0] + windowMs - at) };
|
||||
}
|
||||
window.push(at);
|
||||
hits.set(key, window);
|
||||
if (hits.size > maxKeys) {
|
||||
for (const oldest of hits.keys()) {
|
||||
if (hits.size <= maxKeys) break;
|
||||
hits.delete(oldest);
|
||||
}
|
||||
}
|
||||
return { allowed: true, remaining: Math.max(0, limit - window.length), retryAfterMs: 0 };
|
||||
},
|
||||
size() {
|
||||
return hits.size;
|
||||
},
|
||||
reset() {
|
||||
hits.clear();
|
||||
lastPrune = now();
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
// Release security gate: the arithmetic and the verdict rule for the RC attack suite (#40).
|
||||
//
|
||||
// The suite observes the real server; this module only turns observations into a verdict. Keeping
|
||||
// the rule here (not in the runner) is what makes "a P0 finding blocks RC approval" testable.
|
||||
|
||||
/** Nearest-rank percentile: the returned value is always a sample that was actually measured. */
|
||||
export function latencyPercentiles(samples) {
|
||||
const sorted = [...samples].sort((a, b) => a - b);
|
||||
if (!sorted.length) return { count: 0, p50: null, p95: null, max: null };
|
||||
const rank = (p) => sorted[Math.min(sorted.length - 1, Math.ceil((p / 100) * sorted.length) - 1)];
|
||||
return { count: sorted.length, p50: rank(50), p95: rank(95), max: sorted[sorted.length - 1] };
|
||||
}
|
||||
|
||||
/** One case against one expectation. A partial match is a failure, and it says which field broke. */
|
||||
export function evaluateCase(record) {
|
||||
const mismatches = Object.keys(record.expected)
|
||||
.filter(field => record.observed?.[field] !== record.expected[field])
|
||||
.map(field => `${field} ${record.expected[field]} expected, observed ${record.observed?.[field]}`);
|
||||
return { ...record, status: mismatches.length ? 'fail' : 'pass', mismatch: mismatches.join('; ') };
|
||||
}
|
||||
|
||||
const SEVERITY_BLOCKS = (severity) => severity === 'P0';
|
||||
|
||||
/** The RC rule: every P0 failure blocks approval, unproven categories block it too, P1/P2 only hold. */
|
||||
export function releaseVerdict({ cases, load = null }) {
|
||||
const evaluated = cases.map(record => (record.surface === 'out-of-surface'
|
||||
? { ...record, status: record.evidenceVerified ? 'covered-elsewhere' : 'unproven' }
|
||||
: evaluateCase(record)));
|
||||
const blocking = evaluated.filter(item => item.status === 'fail' && SEVERITY_BLOCKS(item.severity))
|
||||
.map(item => ({ id: item.id, severity: item.severity, reason: item.mismatch }));
|
||||
const holds = evaluated.filter(item => item.status === 'fail' && !SEVERITY_BLOCKS(item.severity))
|
||||
.map(item => ({ id: item.id, severity: item.severity, reason: item.mismatch }));
|
||||
const unproven = evaluated.filter(item => item.status === 'unproven')
|
||||
.map(item => ({ id: item.id, severity: item.severity, reason: 'no executable evidence' }));
|
||||
return {
|
||||
verdict: blocking.length || unproven.length ? 'BLOCKED' : 'GO',
|
||||
blocking,
|
||||
holds,
|
||||
unproven,
|
||||
coveredElsewhere: evaluated.filter(item => item.status === 'covered-elsewhere').map(item => item.id),
|
||||
proven: evaluated.filter(item => item.status === 'pass').length,
|
||||
cases: evaluated,
|
||||
load,
|
||||
};
|
||||
}
|
||||
|
||||
// Evidence hygiene: the report and the captured server log are committed and quoted. A photo that
|
||||
// survived into either one is a leak of the exact data this suite exists to protect.
|
||||
const IMAGE_DATA_URI = /data:image\/[a-z0-9.+-]+;base64,[A-Za-z0-9+/=]+/gi;
|
||||
const LONG_BASE64 = /[A-Za-z0-9+/]{200,}={0,2}/;
|
||||
|
||||
export function containsImageBytes(text) {
|
||||
const value = String(text ?? '');
|
||||
return new RegExp(IMAGE_DATA_URI.source, 'i').test(value) || LONG_BASE64.test(value);
|
||||
}
|
||||
|
||||
export function redactImageBytes(text) {
|
||||
return String(text ?? '')
|
||||
.replace(IMAGE_DATA_URI, '[redacted image bytes]')
|
||||
.replace(new RegExp(LONG_BASE64.source, 'g'), '[redacted image bytes]');
|
||||
}
|
||||
|
||||
/** Fails closed rather than emitting evidence that carries a photo. */
|
||||
export function assertNoImageBytes(label, text) {
|
||||
if (containsImageBytes(text)) throw new Error(`${label} contains image bytes; refusing to write release evidence.`);
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* The receipt the release job quotes. It is redacted after assembly, so a case that captured a
|
||||
* response body still cannot carry a photo in it.
|
||||
*/
|
||||
export function buildReport({ generatedAt, target, cases, load, suite = 'scripts/security_attack_suite.mjs' }) {
|
||||
const verdict = releaseVerdict({ cases, load });
|
||||
const report = {
|
||||
schema: 'timmy-rc-security-attack/1',
|
||||
issue: 40,
|
||||
generatedAt,
|
||||
suite,
|
||||
target,
|
||||
verdict: verdict.verdict,
|
||||
blocking: verdict.blocking,
|
||||
holds: verdict.holds,
|
||||
unproven: verdict.unproven,
|
||||
coveredElsewhere: verdict.coveredElsewhere,
|
||||
proven: verdict.proven,
|
||||
cases: verdict.cases.map(({ id, category, severity, requirement, expected, observed, status, mismatch, snippet, surface, evidence, evidenceVerified }) =>
|
||||
({ id, category, severity, requirement, expected, observed, status, mismatch, surface, evidence, evidenceVerified, snippet: snippet ? redactImageBytes(snippet) : undefined })),
|
||||
load,
|
||||
disclaimer: 'Synthetic hostile bytes only. Every case drives the real server against a stub worker; no real medical image and no live user data is involved.',
|
||||
};
|
||||
const clean = JSON.parse(redactImageBytes(JSON.stringify(report)));
|
||||
assertNoImageBytes('security report', JSON.stringify(clean));
|
||||
return clean;
|
||||
}
|
||||
@@ -0,0 +1,245 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import http from 'node:http';
|
||||
import net from 'node:net';
|
||||
import { spawn } from 'node:child_process';
|
||||
import { once } from 'node:events';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
// End-to-end proof for issue #16 at the real boundary: the server process that ships, not a mock of
|
||||
// it. Every image here is synthetic container bytes assembled in this file, and the provider is a
|
||||
// local stub that counts requests, so "the provider was never called" is measurable rather than
|
||||
// assumed.
|
||||
|
||||
const repoRoot = fileURLToPath(new URL('..', import.meta.url));
|
||||
const PNG_SIGNATURE = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
|
||||
|
||||
function pngChunk(type, data = Buffer.alloc(0)) {
|
||||
const length = Buffer.alloc(4);
|
||||
length.writeUInt32BE(data.length);
|
||||
return Buffer.concat([length, Buffer.from(type, 'ascii'), data, Buffer.alloc(4)]);
|
||||
}
|
||||
function cleanJpeg({ width = 1200, height = 900, tail = Buffer.alloc(0) } = {}) {
|
||||
const sof = Buffer.alloc(17);
|
||||
sof.writeUInt16BE(0xffc0, 0);
|
||||
sof.writeUInt16BE(15, 2);
|
||||
sof[4] = 8;
|
||||
sof.writeUInt16BE(height, 5);
|
||||
sof.writeUInt16BE(width, 7);
|
||||
sof[9] = 3;
|
||||
for (let i = 0; i < 3; i += 1) {
|
||||
sof[10 + i * 3] = i + 1;
|
||||
sof[11 + i * 3] = 0x11;
|
||||
}
|
||||
return Buffer.concat([Buffer.from([0xff, 0xd8]), sof, Buffer.from([0xff, 0xd9]), tail]);
|
||||
}
|
||||
function polyglotPng() {
|
||||
const ihdr = Buffer.alloc(13);
|
||||
ihdr.writeUInt32BE(1200, 0);
|
||||
ihdr.writeUInt32BE(900, 4);
|
||||
ihdr[8] = 8;
|
||||
ihdr[9] = 2;
|
||||
return Buffer.concat([
|
||||
PNG_SIGNATURE,
|
||||
pngChunk('IHDR', ihdr),
|
||||
pngChunk('IDAT', Buffer.from([0x78, 0x9c, 0x01])),
|
||||
pngChunk('IEND'),
|
||||
Buffer.from('<html><script>alert(1)</script></html>', 'utf8'),
|
||||
]);
|
||||
}
|
||||
|
||||
const dataUrl = (mime, bytes) => `data:${mime};base64,${bytes.toString('base64')}`;
|
||||
|
||||
async function freePort() {
|
||||
const server = net.createServer();
|
||||
server.listen(0, '127.0.0.1');
|
||||
await once(server, 'listening');
|
||||
const { port } = server.address();
|
||||
server.close();
|
||||
return port;
|
||||
}
|
||||
|
||||
async function startProvider() {
|
||||
const state = { calls: 0 };
|
||||
const server = http.createServer(async (req, res) => {
|
||||
if (req.url === '/v1/models') {
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ data: [{ id: 'ingress-test-model' }] }));
|
||||
return;
|
||||
}
|
||||
state.calls += 1;
|
||||
for await (const _ of req) { /* drain */ }
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({
|
||||
choices: [{
|
||||
message: {
|
||||
content: JSON.stringify({
|
||||
isStool: true, bristolType: 4, color: 'brown', confidence: 0.8,
|
||||
imageQuality: 'good', observations: 'Synthetic test card.',
|
||||
}),
|
||||
},
|
||||
}],
|
||||
}));
|
||||
});
|
||||
server.listen(0, '127.0.0.1');
|
||||
await once(server, 'listening');
|
||||
return {
|
||||
state,
|
||||
port: server.address().port,
|
||||
// fetch keeps connections alive, so a plain close() would wait for sockets the test already
|
||||
// finished with and hang teardown.
|
||||
close: () => new Promise(resolve => {
|
||||
server.closeAllConnections?.();
|
||||
server.close(resolve);
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
async function withServer(env, run) {
|
||||
const provider = await startProvider();
|
||||
const port = await freePort();
|
||||
const child = spawn(process.execPath, ['server.mjs'], {
|
||||
cwd: repoRoot,
|
||||
env: {
|
||||
...process.env,
|
||||
HOST: '127.0.0.1',
|
||||
PORT: String(port),
|
||||
TIMMY_VISION_ENABLED: 'true',
|
||||
TIMMY_VISION_PROFILE: 'selfhost',
|
||||
TIMMY_VISION_BASE_URL: `http://127.0.0.1:${provider.port}/v1`,
|
||||
TIMMY_VISION_MODEL: 'ingress-test-model',
|
||||
...env,
|
||||
},
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
});
|
||||
const logs = [];
|
||||
child.stdout.on('data', chunk => logs.push(chunk.toString()));
|
||||
child.stderr.on('data', chunk => logs.push(chunk.toString()));
|
||||
const base = `http://127.0.0.1:${port}`;
|
||||
try {
|
||||
// Readiness is the server's own health route, not a sleep.
|
||||
const deadline = Date.now() + 15_000;
|
||||
for (;;) {
|
||||
try {
|
||||
const response = await fetch(`${base}/api/healthz`);
|
||||
if (response.ok) break;
|
||||
} catch { /* not up yet */ }
|
||||
if (Date.now() > deadline) throw new Error(`server did not start: ${logs.join('')}`);
|
||||
await new Promise(resolve => setTimeout(resolve, 100));
|
||||
}
|
||||
await run({ base, provider, logs });
|
||||
} finally {
|
||||
child.kill('SIGKILL');
|
||||
await provider.close();
|
||||
}
|
||||
}
|
||||
|
||||
const postAnalyze = (base, payload) => fetch(`${base}/api/analyze`, {
|
||||
method: 'POST',
|
||||
headers: { 'content-type': 'application/json' },
|
||||
body: JSON.stringify(payload),
|
||||
});
|
||||
|
||||
test('the shipping server refuses bytes that are not an image before any provider work', async () => {
|
||||
await withServer({}, async ({ base, provider }) => {
|
||||
const response = await postAnalyze(base, {
|
||||
consent: true,
|
||||
imageDataUrl: dataUrl('image/jpeg', Buffer.from('<html><script>alert(1)</script></html>', 'utf8')),
|
||||
});
|
||||
const body = await response.json();
|
||||
assert.equal(response.status, 400);
|
||||
assert.match(body.error, /not a JPEG, PNG, or WebP photo/i);
|
||||
assert.equal(provider.state.calls, 0, 'a non-image must never reach the model provider');
|
||||
});
|
||||
});
|
||||
|
||||
test('the shipping server refuses a valid image with a hidden payload appended', async () => {
|
||||
await withServer({}, async ({ base, provider, logs }) => {
|
||||
const response = await postAnalyze(base, { consent: true, imageDataUrl: dataUrl('image/png', polyglotPng()) });
|
||||
const body = await response.json();
|
||||
assert.equal(response.status, 400);
|
||||
assert.match(body.error, /extra data after the image/i);
|
||||
assert.equal(provider.state.calls, 0, 'a polyglot file must never reach the model provider');
|
||||
// The refusal path must not copy the upload into the process log, where it would outlive the request.
|
||||
const logged = logs.join('');
|
||||
assert.doesNotMatch(logged, /base64/i);
|
||||
assert.doesNotMatch(logged, /[A-Za-z0-9+/]{120,}/, 'no run of image bytes may appear in the log');
|
||||
});
|
||||
});
|
||||
|
||||
test('the shipping server still analyses a clean re-encoded image', async () => {
|
||||
await withServer({}, async ({ base, provider }) => {
|
||||
const response = await postAnalyze(base, { consent: true, imageDataUrl: dataUrl('image/jpeg', cleanJpeg()) });
|
||||
const body = await response.json();
|
||||
assert.equal(response.status, 200);
|
||||
assert.equal(body.status, 'suggestion');
|
||||
assert.equal(provider.state.calls, 1, 'the clean path must reach the provider exactly once');
|
||||
});
|
||||
});
|
||||
|
||||
test('the shipping server refuses an oversized body with a refusal the caller can read', async () => {
|
||||
// Issue #16 sets a byte ceiling, and issue #40 asks the release candidate to prove that ceiling
|
||||
// behaves like a refusal rather than a dropped connection. Destroying the socket while the caller
|
||||
// is still uploading leaves the caller with "fetch failed" and no status to act on, so the refusal
|
||||
// has to be written and read before the connection goes away.
|
||||
await withServer({}, async ({ base, provider }) => {
|
||||
const oversized = JSON.stringify({
|
||||
consent: true,
|
||||
imageDataUrl: `data:image/jpeg;base64,${'A'.repeat(6.6 * 1024 * 1024)}`,
|
||||
});
|
||||
let response;
|
||||
try {
|
||||
response = await fetch(`${base}/api/analyze`, {
|
||||
method: 'POST',
|
||||
headers: { 'content-type': 'application/json' },
|
||||
body: oversized,
|
||||
});
|
||||
} catch (error) {
|
||||
assert.fail(`an oversized body must be refused with a readable status, not a dropped connection: ${error.message}`);
|
||||
}
|
||||
const body = await response.json();
|
||||
assert.equal(response.status, 413);
|
||||
assert.match(body.error, /too large/i);
|
||||
assert.equal(provider.state.calls, 0, 'an oversized body must never reach the provider');
|
||||
});
|
||||
});
|
||||
|
||||
test('the shipping server refuses a flood with 429 and stops calling the provider', async () => {
|
||||
await withServer({ TIMMY_ANALYZE_RATE_LIMIT: '2', TIMMY_ANALYZE_RATE_WINDOW_MS: '60000' }, async ({ base, provider }) => {
|
||||
const payload = { consent: true, imageDataUrl: dataUrl('image/jpeg', cleanJpeg()) };
|
||||
assert.equal((await postAnalyze(base, payload)).status, 200);
|
||||
assert.equal((await postAnalyze(base, payload)).status, 200);
|
||||
const refused = await postAnalyze(base, payload);
|
||||
const body = await refused.json();
|
||||
assert.equal(refused.status, 429);
|
||||
assert.match(body.error, /too many photos|wait a minute/i);
|
||||
assert.ok(Number(refused.headers.get('retry-after')) > 0, 'a refusal must say how long to wait');
|
||||
assert.equal(provider.state.calls, 2, 'a refused request must not reach the provider');
|
||||
});
|
||||
});
|
||||
|
||||
test('the shipping server gives up on a body that never arrives', async () => {
|
||||
await withServer({ TIMMY_BODY_TIMEOUT_MS: '250' }, async ({ base, provider }) => {
|
||||
const port = Number(new URL(base).port);
|
||||
const status = await new Promise((resolve, reject) => {
|
||||
const socket = net.connect(port, '127.0.0.1', () => {
|
||||
socket.write([
|
||||
'POST /api/analyze HTTP/1.1',
|
||||
'Host: 127.0.0.1',
|
||||
'Content-Type: application/json',
|
||||
'Content-Length: 400',
|
||||
'', '',
|
||||
].join('\r\n'));
|
||||
});
|
||||
socket.setTimeout(10_000, () => { socket.destroy(); reject(new Error('no response before the client timeout')); });
|
||||
socket.on('data', chunk => {
|
||||
const line = chunk.toString('utf8').split('\r\n')[0];
|
||||
socket.destroy();
|
||||
resolve(Number(line.split(' ')[1]));
|
||||
});
|
||||
socket.on('error', error => { if (error.code !== 'ECONNRESET') reject(error); });
|
||||
});
|
||||
assert.equal(status, 408);
|
||||
assert.equal(provider.state.calls, 0, 'a stalled body must never reach the provider');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,322 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for the operator backup and restore tool.
|
||||
|
||||
Issue #39 asks for a production topology and a backup that a person can trust with the records
|
||||
Timmy is not allowed to lose. The corpus manifest is append-only and the tombstones in it are the
|
||||
only proof that a withdrawal happened, so a backup that quietly drops them, or a restore that
|
||||
rebuilds a training set from withdrawn contributions, is worse than no backup at all.
|
||||
|
||||
The rules this suite locks in:
|
||||
|
||||
- model weights and transient inference images are never archived, and every excluded path is
|
||||
named in the manifest with the reason it was left out;
|
||||
- every archived byte carries a SHA-256 that ``verify`` recomputes before a restore writes
|
||||
anything, so a tampered archive fails closed;
|
||||
- hostile members (absolute paths, traversal, links, unlisted extras) are refused instead of
|
||||
extracted;
|
||||
- a restore round-trips the append-only manifest byte-for-byte and reports the same record counts,
|
||||
including the tombstones.
|
||||
|
||||
Every fixture is synthetic: the "images" are opaque bytes, no real medical image is read or written.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
BACKUP = ROOT / "scripts" / "operator_backup.py"
|
||||
|
||||
IMAGE_BYTES = b"\xff\xd8\xff\xe0synthetic-jpeg-payload-not-a-real-photo"
|
||||
WEIGHT_BYTES = b"GGUF-synthetic-weight-shard-not-a-real-model"
|
||||
TRANSIENT_BYTES = b"transient-inference-upload-must-not-be-archived"
|
||||
SECRET_BYTES = b"TIMMY_DATASET_SALT=demo\nTIMMY_VISION_TOKEN=do-not-archive-me\n"
|
||||
|
||||
|
||||
def digest(payload: bytes) -> str:
|
||||
return hashlib.sha256(payload).hexdigest()
|
||||
|
||||
|
||||
def run(args: list[str], env: dict[str, str] | None = None) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
[sys.executable, str(BACKUP), *args],
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env={**os.environ, **(env or {})},
|
||||
)
|
||||
|
||||
|
||||
def records(*lines: dict) -> str:
|
||||
return "".join(json.dumps(line) + "\n" for line in lines)
|
||||
|
||||
|
||||
def sample(contribution_id: str, *, relative_path: str = "images/sample-1.jpg") -> dict:
|
||||
return {
|
||||
"recordType": "sample",
|
||||
"schemaVersion": 2,
|
||||
"contributionId": contribution_id,
|
||||
"relativePath": relative_path,
|
||||
"imageId": digest(IMAGE_BYTES),
|
||||
"lifecycle": {"state": "active", "updatedAt": "2026-08-01T10:00:00+00:00"},
|
||||
}
|
||||
|
||||
|
||||
def tombstone(contribution_id: str) -> dict:
|
||||
return {
|
||||
"recordType": "tombstone",
|
||||
"schemaVersion": 2,
|
||||
"contributionId": contribution_id,
|
||||
"lifecycle": {"state": "revoked", "updatedAt": "2026-08-02T10:00:00+00:00"},
|
||||
"reason": "participant withdrew consent",
|
||||
"deletedArtifacts": [{"contributionId": contribution_id, "sha256": digest(IMAGE_BYTES)}],
|
||||
}
|
||||
|
||||
|
||||
def build_source(root: Path, *, with_tombstone: bool = False) -> Path:
|
||||
"""A synthetic server-side data root: the records that must survive, plus what must not."""
|
||||
source = root / "srv-timmy"
|
||||
(source / "images").mkdir(parents=True)
|
||||
(source / "images" / "sample-1.jpg").write_bytes(IMAGE_BYTES)
|
||||
(source / "manifest.jsonl").write_text(
|
||||
records(sample("contribution-1"), *([tombstone("contribution-2")] if with_tombstone else [])),
|
||||
encoding="utf-8",
|
||||
)
|
||||
# Must never be archived: weights, transient inference bytes, operator secrets, and logs.
|
||||
(source / "weights").mkdir()
|
||||
(source / "weights" / "model.gguf").write_bytes(WEIGHT_BYTES)
|
||||
(source / "transient").mkdir()
|
||||
(source / "transient" / "inference-upload-1.jpg").write_bytes(TRANSIENT_BYTES)
|
||||
(source / "secrets.env").write_bytes(SECRET_BYTES)
|
||||
(source / "operator.log").write_text("GET /api/vision 200\n", encoding="utf-8")
|
||||
return source
|
||||
|
||||
|
||||
def members(archive: Path) -> dict[str, bytes]:
|
||||
with tarfile.open(archive, "r:gz") as tar:
|
||||
out = {}
|
||||
for info in tar.getmembers():
|
||||
if not info.isfile():
|
||||
continue
|
||||
handle = tar.extractfile(info)
|
||||
assert handle is not None
|
||||
out[info.name] = handle.read()
|
||||
return out
|
||||
|
||||
|
||||
def craft_archive(path: Path, entries: list[tuple[str, bytes]], *, link: str | None = None) -> None:
|
||||
"""Build an archive by hand so the hostile cases are hostile on purpose."""
|
||||
with tarfile.open(path, "w:gz") as tar:
|
||||
for name, payload in entries:
|
||||
info = tarfile.TarInfo(name)
|
||||
info.size = len(payload)
|
||||
info.mode = 0o644
|
||||
tar.addfile(info, io.BytesIO(payload))
|
||||
if link:
|
||||
info = tarfile.TarInfo(link)
|
||||
info.type = tarfile.SYMTYPE
|
||||
info.linkname = "/etc/passwd"
|
||||
info.mode = 0o777
|
||||
tar.addfile(info)
|
||||
|
||||
|
||||
class BackupContractTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.tmp = Path(tempfile.mkdtemp(prefix="timmy-backup-test-"))
|
||||
self.addCleanup(shutil.rmtree, self.tmp, True)
|
||||
self.archive = self.tmp / "backup.tar.gz"
|
||||
self.manifest = self.tmp / "backup.manifest.json"
|
||||
|
||||
def backup(self, source: Path, *extra: str) -> subprocess.CompletedProcess:
|
||||
return run([
|
||||
"backup", "--source", str(source), "--archive", str(self.archive),
|
||||
"--manifest", str(self.manifest), "--label", "nightly", *extra,
|
||||
])
|
||||
|
||||
def test_backup_archives_records_and_excludes_weights_transient_images_secrets_and_logs(self) -> None:
|
||||
source = build_source(self.tmp, with_tombstone=True)
|
||||
result = self.backup(source)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
archived = members(self.archive)
|
||||
self.assertEqual(sorted(archived), ["payload/images/sample-1.jpg", "payload/manifest.jsonl"])
|
||||
self.assertEqual(archived["payload/images/sample-1.jpg"], IMAGE_BYTES)
|
||||
self.assertNotIn(WEIGHT_BYTES, b"".join(archived.values()))
|
||||
self.assertNotIn(TRANSIENT_BYTES, b"".join(archived.values()))
|
||||
self.assertNotIn(SECRET_BYTES, b"".join(archived.values()))
|
||||
|
||||
manifest = json.loads(self.manifest.read_text(encoding="utf-8"))
|
||||
excluded = {entry["path"]: entry["reason"] for entry in manifest["excluded"]}
|
||||
self.assertEqual(excluded["weights/model.gguf"], "model-weights")
|
||||
self.assertEqual(excluded["transient/inference-upload-1.jpg"], "transient-inference")
|
||||
self.assertEqual(excluded["secrets.env"], "secret")
|
||||
self.assertEqual(excluded["operator.log"], "log")
|
||||
# Every archived member is listed with the digest of the bytes that were stored.
|
||||
listed = {entry["path"]: entry["sha256"] for entry in manifest["files"]}
|
||||
self.assertEqual(listed, {
|
||||
"images/sample-1.jpg": digest(IMAGE_BYTES),
|
||||
"manifest.jsonl": digest((source / "manifest.jsonl").read_bytes()),
|
||||
})
|
||||
|
||||
def test_backup_reports_record_counts_including_tombstones(self) -> None:
|
||||
source = build_source(self.tmp, with_tombstone=True)
|
||||
result = self.backup(source)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
receipt = json.loads(result.stdout)
|
||||
self.assertEqual(receipt["records"], {"samples": 1, "tombstones": 1, "activeSamples": 1})
|
||||
manifest = json.loads(self.manifest.read_text(encoding="utf-8"))
|
||||
self.assertEqual(manifest["records"], {"samples": 1, "tombstones": 1, "activeSamples": 1})
|
||||
self.assertEqual(manifest["label"], "nightly")
|
||||
|
||||
def test_backup_refuses_to_follow_a_symlink_out_of_the_data_root(self) -> None:
|
||||
source = build_source(self.tmp)
|
||||
(source / "images" / "escape.jpg").symlink_to("/etc/hostname")
|
||||
result = self.backup(source)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertNotIn("payload/images/escape.jpg", members(self.archive))
|
||||
manifest = json.loads(self.manifest.read_text(encoding="utf-8"))
|
||||
excluded = {entry["path"]: entry["reason"] for entry in manifest["excluded"]}
|
||||
self.assertEqual(excluded["images/escape.jpg"], "symlink")
|
||||
|
||||
def test_backup_refuses_missing_source_and_existing_archive(self) -> None:
|
||||
missing = run(["backup", "--source", str(self.tmp / "nope"), "--archive", str(self.archive)])
|
||||
self.assertNotEqual(missing.returncode, 0)
|
||||
self.assertIn("is not a directory", missing.stderr)
|
||||
|
||||
source = build_source(self.tmp)
|
||||
self.assertEqual(self.backup(source).returncode, 0)
|
||||
again = self.backup(source)
|
||||
self.assertNotEqual(again.returncode, 0)
|
||||
self.assertIn("already exists", again.stderr)
|
||||
|
||||
def test_verify_and_restore_round_trip_the_records(self) -> None:
|
||||
source = build_source(self.tmp, with_tombstone=True)
|
||||
self.assertEqual(self.backup(source).returncode, 0)
|
||||
verified = run(["verify", "--archive", str(self.archive), "--manifest", str(self.manifest)])
|
||||
self.assertEqual(verified.returncode, 0, verified.stderr)
|
||||
|
||||
target = self.tmp / "restored"
|
||||
restored = run([
|
||||
"restore", "--archive", str(self.archive), "--manifest", str(self.manifest),
|
||||
"--target", str(target),
|
||||
])
|
||||
self.assertEqual(restored.returncode, 0, restored.stderr)
|
||||
receipt = json.loads(restored.stdout)
|
||||
self.assertEqual(receipt["records"], {"samples": 1, "tombstones": 1, "activeSamples": 1})
|
||||
self.assertEqual((target / "manifest.jsonl").read_bytes(), (source / "manifest.jsonl").read_bytes())
|
||||
self.assertEqual((target / "images" / "sample-1.jpg").read_bytes(), IMAGE_BYTES)
|
||||
# The manifest is append-only history: the tombstone line survives the round trip verbatim.
|
||||
lines = [json.loads(line) for line in (target / "manifest.jsonl").read_text().splitlines() if line.strip()]
|
||||
self.assertEqual([line["recordType"] for line in lines], ["sample", "tombstone"])
|
||||
|
||||
def test_verify_refuses_tampered_member_before_a_restore_writes_anything(self) -> None:
|
||||
source = build_source(self.tmp)
|
||||
self.assertEqual(self.backup(source).returncode, 0)
|
||||
manifest = json.loads(self.manifest.read_text(encoding="utf-8"))
|
||||
tampered = b"\xff\xd8\xff\xe0synthetic-jpeg-payload-not-a-real-photO"
|
||||
craft_archive(self.archive, [
|
||||
("payload/images/sample-1.jpg", tampered),
|
||||
("payload/manifest.jsonl", (source / "manifest.jsonl").read_bytes()),
|
||||
])
|
||||
verified = run(["verify", "--archive", str(self.archive), "--manifest", str(self.manifest)])
|
||||
self.assertNotEqual(verified.returncode, 0)
|
||||
self.assertIn("does not match", verified.stderr)
|
||||
self.assertEqual(manifest["files"][0]["sha256"], digest(IMAGE_BYTES))
|
||||
|
||||
target = self.tmp / "restored-tampered"
|
||||
restored = run([
|
||||
"restore", "--archive", str(self.archive), "--manifest", str(self.manifest),
|
||||
"--target", str(target),
|
||||
])
|
||||
self.assertNotEqual(restored.returncode, 0)
|
||||
self.assertFalse(target.exists())
|
||||
|
||||
def test_verify_refuses_a_member_the_manifest_never_listed(self) -> None:
|
||||
source = build_source(self.tmp)
|
||||
self.assertEqual(self.backup(source).returncode, 0)
|
||||
craft_archive(self.archive, [
|
||||
("payload/images/sample-1.jpg", IMAGE_BYTES),
|
||||
("payload/manifest.jsonl", (source / "manifest.jsonl").read_bytes()),
|
||||
("payload/extra.bin", b"unlisted"),
|
||||
])
|
||||
verified = run(["verify", "--archive", str(self.archive), "--manifest", str(self.manifest)])
|
||||
self.assertNotEqual(verified.returncode, 0)
|
||||
self.assertIn("not listed in the manifest", verified.stderr)
|
||||
|
||||
def test_restore_refuses_traversal_and_absolute_members(self) -> None:
|
||||
for name, needle in (("payload/../../etc/passwd", "outside"), ("/etc/passwd", "absolute")):
|
||||
with self.subTest(member=name):
|
||||
archive = self.tmp / "hostile.tar.gz"
|
||||
manifest_path = self.tmp / "hostile.manifest.json"
|
||||
payload = b"root:x:0:0"
|
||||
craft_archive(archive, [(name, payload)])
|
||||
manifest_path.write_text(json.dumps({
|
||||
"schemaVersion": 1,
|
||||
"files": [{"path": "images/sample-1.jpg", "sha256": digest(payload), "bytes": len(payload)}],
|
||||
"records": {"samples": 0, "tombstones": 0, "activeSamples": 0},
|
||||
}), encoding="utf-8")
|
||||
target = self.tmp / f"hostile-out-{needle}"
|
||||
restored = run([
|
||||
"restore", "--archive", str(archive), "--manifest", str(manifest_path),
|
||||
"--target", str(target),
|
||||
])
|
||||
self.assertNotEqual(restored.returncode, 0)
|
||||
self.assertIn(needle, restored.stderr.lower())
|
||||
self.assertFalse(target.exists())
|
||||
|
||||
def test_restore_refuses_link_members(self) -> None:
|
||||
archive = self.tmp / "link.tar.gz"
|
||||
manifest_path = self.tmp / "link.manifest.json"
|
||||
craft_archive(archive, [("payload/manifest.jsonl", b"{}\n")], link="payload/images/sample-1.jpg")
|
||||
manifest_path.write_text(json.dumps({
|
||||
"schemaVersion": 1,
|
||||
"files": [{"path": "manifest.jsonl", "sha256": digest(b"{}\n"), "bytes": 3}],
|
||||
"records": {"samples": 0, "tombstones": 0, "activeSamples": 0},
|
||||
}), encoding="utf-8")
|
||||
target = self.tmp / "link-out"
|
||||
restored = run([
|
||||
"restore", "--archive", str(archive), "--manifest", str(manifest_path), "--target", str(target),
|
||||
])
|
||||
self.assertNotEqual(restored.returncode, 0)
|
||||
self.assertIn("link", restored.stderr)
|
||||
self.assertFalse(target.exists())
|
||||
|
||||
def test_restore_refuses_a_non_empty_target(self) -> None:
|
||||
source = build_source(self.tmp)
|
||||
self.assertEqual(self.backup(source).returncode, 0)
|
||||
target = self.tmp / "occupied"
|
||||
target.mkdir()
|
||||
(target / "keep.txt").write_text("operator file that must not be overwritten", encoding="utf-8")
|
||||
restored = run([
|
||||
"restore", "--archive", str(self.archive), "--manifest", str(self.manifest),
|
||||
"--target", str(target),
|
||||
])
|
||||
self.assertNotEqual(restored.returncode, 0)
|
||||
self.assertIn("not empty", restored.stderr)
|
||||
self.assertEqual((target / "keep.txt").read_text(encoding="utf-8"), "operator file that must not be overwritten")
|
||||
|
||||
def test_backup_is_reproducible_for_the_same_records(self) -> None:
|
||||
source = build_source(self.tmp, with_tombstone=True)
|
||||
self.assertEqual(self.backup(source).returncode, 0)
|
||||
first = json.loads(self.manifest.read_text(encoding="utf-8"))
|
||||
first_digest = hashlib.sha256(self.archive.read_bytes()).hexdigest()
|
||||
self.archive.unlink()
|
||||
self.manifest.unlink()
|
||||
self.assertEqual(self.backup(source).returncode, 0)
|
||||
second = json.loads(self.manifest.read_text(encoding="utf-8"))
|
||||
self.assertEqual(first["files"], second["files"])
|
||||
self.assertEqual(first["excluded"], second["excluded"])
|
||||
self.assertEqual(first_digest, hashlib.sha256(self.archive.read_bytes()).hexdigest())
|
||||
self.assertNotEqual(first["createdAt"], "")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
@@ -202,3 +202,14 @@ test('the calibration harness is wired into the unit and syntax gates', async ()
|
||||
assert.match(builder, /tests\/calibration-harness\.test\.py/);
|
||||
assert.match(builder, /"calibration_harness_gates": "passed"/);
|
||||
});
|
||||
|
||||
test('CI runs the RC attack suite so its evidence is reproducible on every change', async () => {
|
||||
const workflow = await readFile(workflowPath, 'utf8');
|
||||
const packageJson = JSON.parse(await readFile(packagePath, 'utf8'));
|
||||
|
||||
// Issue #40: the attack report is evidence, not a claim. CI has to be able to regenerate it.
|
||||
assert.match(workflow, /npm run test:security-attack/);
|
||||
assert.equal(packageJson.scripts['test:security-attack'], 'node tests/security-attack.acceptance.mjs');
|
||||
assert.match(packageJson.scripts.test, /tests\/release-security-gate\.test\.js/);
|
||||
assert.match(packageJson.scripts['check:syntax'], /node --check src\/release-security-gate\.js/);
|
||||
});
|
||||
Vendored
+1
@@ -0,0 +1 @@
|
||||
<!doctype html><html><body>This file is named .jpg but it is not an image. Synthetic fixture for the image-ingress refusal path.</body></html>
|
||||
@@ -0,0 +1,58 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { PassThrough } from 'node:stream';
|
||||
|
||||
import { BodyError, readJsonBody } from '../src/http-body.js';
|
||||
|
||||
const bodyStream = chunks => {
|
||||
const stream = new PassThrough();
|
||||
for (const chunk of chunks) stream.write(chunk);
|
||||
stream.end();
|
||||
return stream;
|
||||
};
|
||||
|
||||
test('reads a bounded JSON request body', async () => {
|
||||
const parsed = await readJsonBody(bodyStream([Buffer.from('{"consent":'), Buffer.from('true}')]), { maxBytes: 1024 });
|
||||
assert.deepEqual(parsed, { consent: true });
|
||||
});
|
||||
|
||||
test('refuses a body above the byte limit instead of buffering it', async () => {
|
||||
await assert.rejects(
|
||||
() => readJsonBody(bodyStream([Buffer.alloc(64, 0x61)]), { maxBytes: 16 }),
|
||||
error => {
|
||||
assert.ok(error instanceof BodyError);
|
||||
assert.equal(error.code, 'too_large');
|
||||
assert.equal(error.status, 413);
|
||||
return true;
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test('refuses malformed JSON with a client error', async () => {
|
||||
await assert.rejects(
|
||||
() => readJsonBody(bodyStream([Buffer.from('{not json')]), { maxBytes: 1024 }),
|
||||
error => {
|
||||
assert.ok(error instanceof BodyError);
|
||||
assert.equal(error.code, 'invalid_json');
|
||||
assert.equal(error.status, 400);
|
||||
return true;
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test('gives up on a body that never finishes and stops reading it', async () => {
|
||||
const stalled = new PassThrough();
|
||||
stalled.write(Buffer.from('{"consent":true'));
|
||||
const started = Date.now();
|
||||
await assert.rejects(
|
||||
() => readJsonBody(stalled, { maxBytes: 1024, timeoutMs: 40 }),
|
||||
error => {
|
||||
assert.ok(error instanceof BodyError);
|
||||
assert.equal(error.code, 'body_timeout');
|
||||
assert.equal(error.status, 408);
|
||||
return true;
|
||||
},
|
||||
);
|
||||
assert.ok(Date.now() - started < 2_000, 'the deadline must fire promptly');
|
||||
assert.equal(stalled.isPaused(), true, 'the stalled body must stop being read after the deadline');
|
||||
});
|
||||
@@ -0,0 +1,175 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
|
||||
import { IngressError, inspectImageBytes, MAX_EDGE, MAX_PIXELS } from '../src/image-ingress.js';
|
||||
|
||||
// Every fixture built here is synthetic container bytes assembled in code. No photograph of any
|
||||
// person or specimen is used, and nothing is fetched.
|
||||
const PNG_SIGNATURE = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
|
||||
|
||||
function pngChunk(type, data = Buffer.alloc(0)) {
|
||||
const length = Buffer.alloc(4);
|
||||
length.writeUInt32BE(data.length);
|
||||
const body = Buffer.concat([Buffer.from(type, 'ascii'), data]);
|
||||
// The parser walks chunk lengths to find the end of the container; the CRC is not read, so a
|
||||
// synthetic zero CRC is enough to be structurally complete.
|
||||
return Buffer.concat([length, body, Buffer.alloc(4)]);
|
||||
}
|
||||
|
||||
function png({ width = 1200, height = 900, extraChunks = [], tail = Buffer.alloc(0) } = {}) {
|
||||
const ihdr = Buffer.alloc(13);
|
||||
ihdr.writeUInt32BE(width, 0);
|
||||
ihdr.writeUInt32BE(height, 4);
|
||||
ihdr[8] = 8; // bit depth
|
||||
ihdr[9] = 2; // truecolour
|
||||
return Buffer.concat([
|
||||
PNG_SIGNATURE,
|
||||
pngChunk('IHDR', ihdr),
|
||||
...extraChunks,
|
||||
pngChunk('IDAT', Buffer.from([0x78, 0x9c, 0x01])),
|
||||
pngChunk('IEND'),
|
||||
tail,
|
||||
]);
|
||||
}
|
||||
|
||||
function jpeg({ width = 1200, height = 900, segments = [], tail = Buffer.alloc(0) } = {}) {
|
||||
const sof = Buffer.alloc(17);
|
||||
sof.writeUInt16BE(0xffc0, 0);
|
||||
sof.writeUInt16BE(15, 2); // payload length
|
||||
sof[4] = 8; // precision
|
||||
sof.writeUInt16BE(height, 5);
|
||||
sof.writeUInt16BE(width, 7);
|
||||
sof[9] = 3; // component count
|
||||
for (let i = 0; i < 3; i += 1) {
|
||||
sof[10 + i * 3] = i + 1;
|
||||
sof[11 + i * 3] = 0x11;
|
||||
sof[12 + i * 3] = 0;
|
||||
}
|
||||
return Buffer.concat([
|
||||
Buffer.from([0xff, 0xd8]),
|
||||
...segments,
|
||||
sof,
|
||||
Buffer.from([0xff, 0xd9]),
|
||||
tail,
|
||||
]);
|
||||
}
|
||||
|
||||
function exifSegment() {
|
||||
const payload = Buffer.concat([
|
||||
Buffer.from('Exif', 'ascii'),
|
||||
Buffer.from([0x00, 0x00]),
|
||||
Buffer.from('GPSLatitude=0.000', 'ascii'),
|
||||
]);
|
||||
const header = Buffer.alloc(4);
|
||||
header.writeUInt16BE(0xffe1, 0);
|
||||
header.writeUInt16BE(payload.length + 2, 2);
|
||||
return Buffer.concat([header, payload]);
|
||||
}
|
||||
|
||||
function webp({ width = 1200, height = 900 } = {}) {
|
||||
// VP8X canvas payload: flags(1) + reserved(3) + width-1(3, LE) + height-1(3, LE).
|
||||
const payload = Buffer.alloc(10);
|
||||
payload[0] = 0;
|
||||
payload.writeUIntLE(width - 1, 4, 3);
|
||||
payload.writeUIntLE(height - 1, 7, 3);
|
||||
const chunk = Buffer.alloc(8);
|
||||
chunk.write('VP8X', 0, 'ascii');
|
||||
chunk.writeUInt32LE(payload.length, 4);
|
||||
const body = Buffer.concat([Buffer.from('WEBP', 'ascii'), chunk, payload]);
|
||||
const riff = Buffer.alloc(8);
|
||||
riff.write('RIFF', 0, 'ascii');
|
||||
riff.writeUInt32LE(body.length, 4);
|
||||
return Buffer.concat([riff, body]);
|
||||
}
|
||||
|
||||
const expectRefusal = (bytes, declaredMime, pattern) => {
|
||||
assert.throws(
|
||||
() => inspectImageBytes(bytes, { declaredMime }),
|
||||
error => {
|
||||
assert.ok(error instanceof IngressError, `expected an IngressError, got ${error?.name}`);
|
||||
assert.match(error.message, pattern);
|
||||
return true;
|
||||
},
|
||||
);
|
||||
};
|
||||
|
||||
test('refuses a file whose bytes are not a JPEG, PNG, or WebP', () => {
|
||||
expectRefusal(Buffer.from('<html><script>alert(1)</script></html>', 'utf8'), 'image/jpeg', /not a JPEG, PNG, or WebP photo/i);
|
||||
});
|
||||
|
||||
test('refuses a PNG that claims to be a JPEG', () => {
|
||||
expectRefusal(png(), 'image/jpeg', /not the format it claims/i);
|
||||
});
|
||||
|
||||
test('refuses a JPEG that claims to be a PNG', () => {
|
||||
expectRefusal(jpeg(), 'image/png', /not the format it claims/i);
|
||||
});
|
||||
|
||||
test('refuses a truncated container instead of forwarding a partial image', () => {
|
||||
expectRefusal(PNG_SIGNATURE.subarray(0, 8), 'image/png', /incomplete or damaged/i);
|
||||
expectRefusal(Buffer.from([0xff, 0xd8, 0xff, 0xe0]), 'image/jpeg', /incomplete or damaged/i);
|
||||
});
|
||||
|
||||
test('refuses a PNG with bytes appended after its end marker', () => {
|
||||
expectRefusal(png({ tail: Buffer.from('PK\u0003\u0004zip-appended', 'binary') }), 'image/png', /extra data after the image/i);
|
||||
});
|
||||
|
||||
test('refuses a JPEG with bytes appended after its end marker', () => {
|
||||
expectRefusal(jpeg({ tail: Buffer.from('<script>alert(1)</script>', 'utf8') }), 'image/jpeg', /extra data after the image/i);
|
||||
});
|
||||
|
||||
test('refuses a JPEG that still carries an EXIF segment', () => {
|
||||
expectRefusal(jpeg({ segments: [exifSegment()] }), 'image/jpeg', /embedded metadata/i);
|
||||
});
|
||||
|
||||
test('refuses a PNG that carries an eXIf chunk', () => {
|
||||
const exifChunk = pngChunk('eXIf', Buffer.from('MM\u0000*metadata', 'binary'));
|
||||
expectRefusal(png({ extraChunks: [exifChunk] }), 'image/png', /embedded metadata/i);
|
||||
});
|
||||
|
||||
test('accepts a clean JPEG and reports its real dimensions', () => {
|
||||
const intake = inspectImageBytes(jpeg({ width: 1200, height: 900 }), { declaredMime: 'image/jpeg' });
|
||||
assert.equal(intake.mime, 'image/jpeg');
|
||||
assert.equal(intake.width, 1200);
|
||||
assert.equal(intake.height, 900);
|
||||
assert.equal(intake.bytes, jpeg({ width: 1200, height: 900 }).length);
|
||||
});
|
||||
|
||||
test('accepts a clean PNG and reports its real dimensions', () => {
|
||||
const intake = inspectImageBytes(png({ width: 800, height: 600 }), { declaredMime: 'image/png' });
|
||||
assert.equal(intake.mime, 'image/png');
|
||||
assert.equal(intake.width, 800);
|
||||
assert.equal(intake.height, 600);
|
||||
});
|
||||
|
||||
test('accepts a clean WebP and reports its canvas dimensions', () => {
|
||||
const intake = inspectImageBytes(webp({ width: 640, height: 480 }), { declaredMime: 'image/webp' });
|
||||
assert.equal(intake.mime, 'image/webp');
|
||||
assert.equal(intake.width, 640);
|
||||
assert.equal(intake.height, 480);
|
||||
});
|
||||
|
||||
test(`refuses an edge longer than ${MAX_EDGE} pixels even when the file is small`, () => {
|
||||
expectRefusal(png({ width: MAX_EDGE + 1, height: 200 }), 'image/png', /too large in pixels/i);
|
||||
});
|
||||
|
||||
test(`refuses an image above ${MAX_PIXELS} pixels when each edge fits`, () => {
|
||||
const edge = 4096;
|
||||
expectRefusal(jpeg({ width: edge, height: Math.ceil(MAX_PIXELS / edge) + 1 }), 'image/jpeg', /too large in pixels/i);
|
||||
});
|
||||
|
||||
test('never repeats image bytes back to the caller in a refusal message', () => {
|
||||
const marker = 'SECRETPIXELMARKER';
|
||||
try {
|
||||
inspectImageBytes(Buffer.from(`not an image ${marker}`, 'utf8'), { declaredMime: 'image/jpeg' });
|
||||
assert.fail('expected a refusal');
|
||||
} catch (error) {
|
||||
assert.ok(error instanceof IngressError);
|
||||
assert.doesNotMatch(error.message, new RegExp(marker));
|
||||
assert.doesNotMatch(error.message, /[A-Za-z0-9+/=]{40,}/);
|
||||
}
|
||||
});
|
||||
|
||||
test('refuses an empty body with a bounded message', () => {
|
||||
expectRefusal(Buffer.alloc(0), 'image/jpeg', /empty/i);
|
||||
});
|
||||
@@ -0,0 +1,70 @@
|
||||
import { chromium } from 'playwright';
|
||||
import assert from 'node:assert/strict';
|
||||
|
||||
// Issue #16, as a user experiences it: a file that only claims to be a photo never becomes an upload,
|
||||
// never reaches the model worker, and still leaves manual logging one tap away. The second half of
|
||||
// this suite asks the real running server to refuse the same bytes, so the browser copy and the
|
||||
// server boundary are proven in the same run.
|
||||
|
||||
const browser = await chromium.launch({ headless: true });
|
||||
const context = await browser.newContext({
|
||||
viewport: { width: 390, height: 844 },
|
||||
deviceScaleFactor: 2,
|
||||
serviceWorkers: 'block',
|
||||
});
|
||||
const page = await context.newPage();
|
||||
let analysisRequests = 0;
|
||||
const analyzePayloads = [];
|
||||
await page.route('**/api/vision-status', route => route.fulfill({
|
||||
status: 200,
|
||||
contentType: 'application/json',
|
||||
body: JSON.stringify({ enabled: true, profile: 'selfhost', providerReady: true, modelSeen: true, model: 'synthetic-test-model' }),
|
||||
}));
|
||||
await page.route('**/api/analyze', route => {
|
||||
analysisRequests += 1;
|
||||
analyzePayloads.push(route.request().postData() || '');
|
||||
return route.continue();
|
||||
});
|
||||
|
||||
await page.goto('http://127.0.0.1:4173', { waitUntil: 'networkidle' });
|
||||
await page.locator('[data-scan]').click();
|
||||
|
||||
const spoofed = Buffer.from('<html><script>alert(1)</script></html>', 'utf8');
|
||||
await page.locator('#gallery-photo').setInputFiles({ name: 'photo.jpg', mimeType: 'image/jpeg', buffer: spoofed });
|
||||
|
||||
const refusal = page.locator('[data-capture-error="unreadable"]');
|
||||
await refusal.waitFor();
|
||||
const copy = await refusal.innerText();
|
||||
assert.match(copy, /isn.t a photo Timmy can read/i, 'the refusal names what is wrong with the file');
|
||||
assert.match(copy, /Nothing was uploaded/i, 'the refusal states that nothing left the device');
|
||||
assert.equal(await page.getByText(/couldn.t analyze that safely/i).count(), 0,
|
||||
'an unreadable file must not be reported as a failed analysis');
|
||||
assert.equal(analysisRequests, 0, 'an unreadable file must never be posted to /api/analyze');
|
||||
assert.equal(await page.locator('#manual-from-scan').isVisible(), true, 'manual entry stays one tap away');
|
||||
assert.equal(await page.locator('#retake-photo').isVisible(), true, 'the picker stays reachable');
|
||||
|
||||
// The same bytes, offered to the running server directly, are refused before any provider work.
|
||||
const serverResponse = await page.evaluate(async payload => {
|
||||
const response = await fetch(`${document.querySelector('meta[name="timmy-base-path"]')?.content.replace(/\/$/, '') || ''}/api/analyze`, {
|
||||
method: 'POST',
|
||||
headers: { 'content-type': 'application/json' },
|
||||
body: JSON.stringify(payload),
|
||||
});
|
||||
return { status: response.status, body: await response.json() };
|
||||
}, { consent: true, imageDataUrl: `data:image/jpeg;base64,${spoofed.toString('base64')}` });
|
||||
assert.equal(serverResponse.status, 400, `the server must refuse non-image bytes, saw ${serverResponse.status}`);
|
||||
assert.match(serverResponse.body.error, /not a JPEG, PNG, or WebP photo/i);
|
||||
// Only this deliberate probe reached the endpoint; the app itself never posted the unreadable file.
|
||||
assert.equal(analysisRequests, 1, 'only the deliberate server probe may reach /api/analyze so far');
|
||||
assert.match(analyzePayloads[0], /"consent":true/);
|
||||
|
||||
// A real (synthetic) photo still reaches the consent step, so the gate is a boundary and not a wall.
|
||||
await page.locator('#retake-photo').click();
|
||||
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/synthetic-type4.jpg');
|
||||
await page.locator('#ai-consent').waitFor();
|
||||
assert.equal(await page.locator('[data-capture-error="unreadable"]').count(), 0);
|
||||
assert.equal(analysisRequests, 1, 'a photo that is only selected is still not uploaded before consent');
|
||||
|
||||
await context.close();
|
||||
await browser.close();
|
||||
console.log('PASS unreadable photo bytes are refused on device and at the server, with manual entry intact');
|
||||
@@ -47,8 +47,11 @@ for (const viewport of viewports) {
|
||||
mimeType: 'image/jpeg',
|
||||
buffer: Buffer.from('not an image'),
|
||||
});
|
||||
assert.match(await page.locator('.scan-result').innerText(), /could not be read/i);
|
||||
await page.getByText('Try another photo', { exact: true }).click();
|
||||
assert.match(await page.locator('.scan-result').innerText(), /isn.t a photo Timmy can read/i);
|
||||
// Issue #16: the refusal is its own state and states plainly that nothing left the device, instead
|
||||
// of borrowing the "couldn't analyze that safely" copy from a failed upload.
|
||||
assert.match(await page.locator('.scan-result').innerText(), /Nothing was uploaded/i);
|
||||
await page.getByText('Choose another photo', { exact: true }).click();
|
||||
await page.locator('#gallery-photo').setInputFiles('tests/fixtures/synthetic-type4.jpg');
|
||||
assert.equal(await page.locator('#analyze-photo').isDisabled(), true);
|
||||
assert.equal(await page.locator('#retake-photo').isVisible(), true);
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for the production topology, secrets, backup, and rollback runbook.
|
||||
|
||||
A runbook is only worth reading if its commands are the ones the tools actually accept, so this
|
||||
suite parses every documented ``python3 scripts/...`` command out of the markdown and checks each
|
||||
flag against the real CLI help. It also fails if the document starts carrying a real secret, or if
|
||||
it keeps claiming the server holds no journal after ``server.mjs`` starts writing one.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
RUNBOOK = ROOT / "docs" / "PRODUCTION-RUNBOOK.md"
|
||||
SERVER = ROOT / "server.mjs"
|
||||
BACKUP = ROOT / "scripts" / "operator_backup.py"
|
||||
|
||||
FENCE = re.compile(r"```bash\n(.*?)```", re.S)
|
||||
SECRET_ASSIGNMENT = re.compile(
|
||||
r"(?i)(?:token|secret|password|passwd|api[_-]?key|salt)\s*=\s*(?!<|\$|\"|\'|\.\.\.)[^\s`\"']{6,}"
|
||||
)
|
||||
|
||||
|
||||
def text() -> str:
|
||||
return RUNBOOK.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def sections() -> dict[str, str]:
|
||||
body = text()
|
||||
headings = list(re.finditer(r"^## (.+)$", body, re.M))
|
||||
out: dict[str, str] = {}
|
||||
for index, match in enumerate(headings):
|
||||
end = headings[index + 1].start() if index + 1 < len(headings) else len(body)
|
||||
out[match.group(1).strip().lower()] = body[match.end():end]
|
||||
return out
|
||||
|
||||
|
||||
def documented_commands() -> list[str]:
|
||||
commands: list[str] = []
|
||||
for block in FENCE.findall(text()):
|
||||
for line in block.splitlines():
|
||||
line = line.strip()
|
||||
if line.startswith("python3 scripts/"):
|
||||
commands.append(line)
|
||||
return commands
|
||||
|
||||
|
||||
def help_text(script: str, subcommand: str | None) -> str:
|
||||
argv = [sys.executable, script] + ([subcommand] if subcommand else []) + ["--help"]
|
||||
result = subprocess.run(argv, cwd=ROOT, capture_output=True, text=True)
|
||||
assert result.returncode == 0, f"{' '.join(argv)} failed: {result.stderr}"
|
||||
return result.stdout
|
||||
|
||||
|
||||
class ProductionRunbookTests(unittest.TestCase):
|
||||
def test_runbook_covers_topology_network_boundaries_secrets_backup_restore_and_rollback(self) -> None:
|
||||
self.assertTrue(RUNBOOK.is_file(), "docs/PRODUCTION-RUNBOOK.md must exist")
|
||||
titles = sections().keys()
|
||||
for required in ("topology", "network boundaries", "secrets", "backup", "restore", "rollback"):
|
||||
self.assertTrue(
|
||||
any(title.startswith(required) for title in titles),
|
||||
f"runbook is missing a '{required}' section; found {sorted(titles)}",
|
||||
)
|
||||
|
||||
def test_backup_section_names_what_is_never_archived_and_why(self) -> None:
|
||||
backup = next(body for title, body in sections().items() if title.startswith("backup"))
|
||||
self.assertIn("scripts/operator_backup.py", backup)
|
||||
for required in ("model weights", "transient inference", "secret", "tombstone"):
|
||||
self.assertIn(required, backup.lower(), f"backup section must name {required}")
|
||||
self.assertRegex(backup, r"manifest\.jsonl")
|
||||
|
||||
def test_every_documented_script_command_is_accepted_by_the_real_cli(self) -> None:
|
||||
commands = documented_commands()
|
||||
self.assertGreaterEqual(len(commands), 4, f"expected documented commands, found {commands}")
|
||||
for command in commands:
|
||||
with self.subTest(command=command):
|
||||
tokens = shlex.split(command)
|
||||
script = tokens[1]
|
||||
rest = tokens[2:]
|
||||
subcommand = rest[0] if rest and not rest[0].startswith("-") else None
|
||||
combined = help_text(script, subcommand) + help_text(script, None)
|
||||
for token in rest:
|
||||
if token.startswith("--"):
|
||||
self.assertIn(token, combined, f"{script} does not accept {token}")
|
||||
for token in rest[1:]:
|
||||
if not token.startswith("--"):
|
||||
continue
|
||||
|
||||
def test_documented_backup_commands_are_the_ones_the_tool_runs(self) -> None:
|
||||
commands = " ".join(documented_commands())
|
||||
for flag in ("--source", "--archive", "--manifest", "--target"):
|
||||
self.assertIn(flag, commands, f"the runbook must show the real {flag} flag")
|
||||
self.assertIn("verify", commands)
|
||||
self.assertIn("restore", commands)
|
||||
|
||||
def test_rollback_is_documented_as_a_tested_fixture_path_and_a_human_gated_live_drill(self) -> None:
|
||||
rollback = next(body for title, body in sections().items() if title.startswith("rollback"))
|
||||
self.assertIn("scripts/deploy_staging.py rollback", rollback)
|
||||
self.assertRegex(rollback.lower(), r"human[- ]gate")
|
||||
self.assertIn("#53", rollback)
|
||||
self.assertIn("tests/staging-deploy.test.py", rollback)
|
||||
# The live drill must be explicit about not being run by automation (wrapped lines are fine).
|
||||
self.assertRegex(
|
||||
rollback.lower(),
|
||||
r"(?s)not .{0,80}automatically|does not run .{0,80}automatically",
|
||||
"the runbook must say the live drill is not run automatically",
|
||||
)
|
||||
|
||||
def test_network_boundaries_match_how_the_server_and_worker_actually_bind(self) -> None:
|
||||
server = SERVER.read_text(encoding="utf-8")
|
||||
# The app's own default is not loopback, so the runbook must say the deployment sets it.
|
||||
self.assertRegex(server, r"HOST\|\|'0\.0\.0\.0'")
|
||||
staging_env = (ROOT / "deploy" / "timmy-staging.env.example").read_text(encoding="utf-8")
|
||||
self.assertIn("HOST=127.0.0.1", staging_env)
|
||||
worker = (ROOT / "deploy" / "worker" / "run_worker_container.sh").read_text(encoding="utf-8")
|
||||
self.assertIn("Refusing to publish the worker outside loopback", worker)
|
||||
|
||||
boundaries = next(body for title, body in sections().items() if title.startswith("network boundaries"))
|
||||
self.assertIn("0.0.0.0", boundaries, "the runbook must name the app's real default bind address")
|
||||
self.assertIn("HOST=127.0.0.1", boundaries)
|
||||
self.assertIn("deploy/timmy-staging.env.example", boundaries)
|
||||
self.assertIn("loopback", boundaries.lower())
|
||||
self.assertIn("8080", boundaries)
|
||||
|
||||
def test_runbook_carries_no_literal_secret_values(self) -> None:
|
||||
findings = [match.group(0) for match in SECRET_ASSIGNMENT.finditer(text())]
|
||||
self.assertEqual(findings, [], f"runbook looks like it contains a real secret: {findings}")
|
||||
self.assertIn("mode-600", text())
|
||||
self.assertIn("never commit", text().lower())
|
||||
|
||||
def test_runbook_claims_match_a_stateless_server(self) -> None:
|
||||
server = SERVER.read_text(encoding="utf-8")
|
||||
for call in ("writeFile", "appendFile", "createWriteStream", "mkdirSync"):
|
||||
self.assertNotIn(call, server, "server.mjs started persisting data; the runbook's data-lifetime claim must be updated")
|
||||
body = text().lower()
|
||||
self.assertIn("no server-side copy of the journal", body)
|
||||
self.assertIn("/api/healthz", body)
|
||||
# The health probe is allowed to report readiness, never configuration or secrets.
|
||||
health = re.search(r"'/api/healthz'[^\n]*", server)
|
||||
self.assertIsNotNone(health, "server.mjs must serve /api/healthz")
|
||||
health_text = health.group(0) if health else ""
|
||||
for forbidden in ("token", "secret", "env:", "password"):
|
||||
self.assertNotIn(forbidden, health_text.lower())
|
||||
# The response may report readiness only, so a new field is a deliberate decision.
|
||||
for field in ("ok", "release", "commit", "visionEnabled", "agentEnabled"):
|
||||
self.assertIn(field, health_text)
|
||||
|
||||
def test_backup_tool_is_gated_by_its_own_suite(self) -> None:
|
||||
package = (ROOT / "package.json").read_text(encoding="utf-8")
|
||||
self.assertIn("tests/backup-restore.test.py", package)
|
||||
self.assertIn("tests/production-runbook.test.py", package)
|
||||
self.assertTrue(BACKUP.is_file())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
@@ -0,0 +1,91 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
|
||||
import { createRateLimiter } from '../src/rate-limit.js';
|
||||
|
||||
// A clock the test owns, so no test sleeps and every window boundary is exact.
|
||||
function manualClock(start = 1_700_000_000_000) {
|
||||
let current = start;
|
||||
return { now: () => current, advance: ms => { current += ms; } };
|
||||
}
|
||||
|
||||
test('allows requests up to the limit and refuses the next one', () => {
|
||||
const clock = manualClock();
|
||||
const limiter = createRateLimiter({ limit: 3, windowMs: 60_000, now: clock.now });
|
||||
|
||||
for (let attempt = 1; attempt <= 3; attempt += 1) {
|
||||
const verdict = limiter.take('device-a');
|
||||
assert.equal(verdict.allowed, true, `attempt ${attempt} must be allowed`);
|
||||
assert.equal(verdict.remaining, 3 - attempt);
|
||||
}
|
||||
|
||||
const refused = limiter.take('device-a');
|
||||
assert.equal(refused.allowed, false);
|
||||
assert.equal(refused.remaining, 0);
|
||||
assert.equal(refused.retryAfterMs, 60_000);
|
||||
});
|
||||
|
||||
test('counts each caller on its own budget', () => {
|
||||
const clock = manualClock();
|
||||
const limiter = createRateLimiter({ limit: 1, windowMs: 60_000, now: clock.now });
|
||||
|
||||
assert.equal(limiter.take('device-a').allowed, true);
|
||||
assert.equal(limiter.take('device-a').allowed, false);
|
||||
assert.equal(limiter.take('device-b').allowed, true);
|
||||
});
|
||||
|
||||
test('lets a caller back in once its window has passed', () => {
|
||||
const clock = manualClock();
|
||||
const limiter = createRateLimiter({ limit: 1, windowMs: 60_000, now: clock.now });
|
||||
|
||||
assert.equal(limiter.take('device-a').allowed, true);
|
||||
assert.equal(limiter.take('device-a').allowed, false);
|
||||
clock.advance(60_001);
|
||||
const allowedAgain = limiter.take('device-a');
|
||||
assert.equal(allowedAgain.allowed, true);
|
||||
assert.equal(allowedAgain.remaining, 0);
|
||||
});
|
||||
|
||||
test('a refused request does not extend the block', () => {
|
||||
const clock = manualClock();
|
||||
const limiter = createRateLimiter({ limit: 1, windowMs: 1_000, now: clock.now });
|
||||
|
||||
assert.equal(limiter.take('device-a').allowed, true);
|
||||
clock.advance(400);
|
||||
assert.equal(limiter.take('device-a').allowed, false);
|
||||
clock.advance(601);
|
||||
// The original hit is now 1001ms old, so the window has moved on even though the caller hammered
|
||||
// the endpoint in between. A refusal must never buy a longer penalty than the caller earned.
|
||||
assert.equal(limiter.take('device-a').allowed, true);
|
||||
});
|
||||
|
||||
test('reports the wait from the oldest hit still inside the window', () => {
|
||||
const clock = manualClock();
|
||||
const limiter = createRateLimiter({ limit: 2, windowMs: 10_000, now: clock.now });
|
||||
|
||||
limiter.take('device-a');
|
||||
clock.advance(4_000);
|
||||
limiter.take('device-a');
|
||||
clock.advance(1_000);
|
||||
const refused = limiter.take('device-a');
|
||||
assert.equal(refused.allowed, false);
|
||||
assert.equal(refused.retryAfterMs, 5_000);
|
||||
});
|
||||
|
||||
test('keeps its caller map bounded when many callers appear', () => {
|
||||
const clock = manualClock();
|
||||
const limiter = createRateLimiter({ limit: 5, windowMs: 60_000, maxKeys: 8, now: clock.now });
|
||||
|
||||
for (let index = 0; index < 50; index += 1) limiter.take(`device-${index}`);
|
||||
assert.ok(limiter.size() <= 8, `expected at most 8 tracked callers, saw ${limiter.size()}`);
|
||||
});
|
||||
|
||||
test('prunes expired callers instead of growing for the life of the process', () => {
|
||||
const clock = manualClock();
|
||||
const limiter = createRateLimiter({ limit: 5, windowMs: 1_000, now: clock.now });
|
||||
|
||||
for (let index = 0; index < 20; index += 1) limiter.take(`device-${index}`);
|
||||
clock.advance(1_001);
|
||||
limiter.take('device-fresh');
|
||||
assert.equal(limiter.size(), 1);
|
||||
});
|
||||
@@ -113,6 +113,23 @@ test('release demo visibly exercises the red-flag language coverage instead of o
|
||||
assert.match(builder, /A tarry note already saved in the journal escalates the same way, before any provider call/);
|
||||
});
|
||||
|
||||
test('release demo shows a file that only claims to be a photo being refused before upload', async () => {
|
||||
const demo = await readFile(demoPath, 'utf8');
|
||||
const builder = await readFile(builderPath, 'utf8');
|
||||
|
||||
// Issue #16: the recording must show the device-side refusal of a non-image, prove nothing was
|
||||
// posted, and keep manual entry reachable — all asserted in the browser, not narrated.
|
||||
assert.match(demo, /tests\/fixtures\/not-a-photo\.jpg/);
|
||||
assert.match(demo, /\[data-capture-error="unreadable"\]/);
|
||||
assert.match(demo, /Nothing was uploaded/);
|
||||
assert.match(demo, /An unreadable file was posted to \/api\/analyze/);
|
||||
assert.match(demo, /A file that only claims to be a photo is refused on the device — nothing is uploaded/);
|
||||
assert.match(demo, /Manual logging stays one tap away, and the photo picker is one tap back/);
|
||||
// The caption is a required release gate, so a later release cannot drop the scene silently.
|
||||
assert.match(builder, /A file that only claims to be a photo is refused on the device — nothing is uploaded/);
|
||||
assert.match(builder, /"image_ingress_refusal": "passed"/);
|
||||
});
|
||||
|
||||
test('release builder gates the sleek shell, Hermes chat, and bootstrap syntax', async () => {
|
||||
const builder = await readFile(builderPath, 'utf8');
|
||||
assert.match(builder, /"test:sleek"/);
|
||||
@@ -486,3 +503,58 @@ test('release demo publishes the dataset card from a manifest it builds on camer
|
||||
assert.match(builder, /"dataset_card_generated": "passed"/);
|
||||
assert.match(builder, /scripts\/dataset_card\.py/);
|
||||
});
|
||||
|
||||
test('release demo runs the operator backup and restore on synthetic records instead of describing it', async () => {
|
||||
const demo = await readFile(demoPath, 'utf8');
|
||||
const builder = await readFile(builderPath, 'utf8');
|
||||
|
||||
// Issue #39: the backup is a CLI, so the recording runs the real tool and shows what it stored,
|
||||
// what it refused, and what the restore returned.
|
||||
assert.match(demo, /scripts\/operator_backup\.py/);
|
||||
assert.match(demo, /'backup', '--source'/);
|
||||
assert.match(demo, /'verify', '--archive'/);
|
||||
assert.match(demo, /'restore', '--archive'/);
|
||||
assert.match(demo, /id: 'production-backup'/);
|
||||
assert.match(demo, /Production backup card is not fully readable on screen/);
|
||||
// The exclusions are proven by the run, not asserted by a caption: the decoys are real files.
|
||||
assert.match(demo, /model-weights/);
|
||||
assert.match(demo, /transient-inference/);
|
||||
assert.match(demo, /must not archive the weight shard or a transient inference upload/);
|
||||
// A restore must return the append-only manifest byte for byte, tombstone included.
|
||||
assert.match(demo, /must return the same records, tombstone included/);
|
||||
|
||||
assert.match(demo, /The operator backup is a command line, not a screen in the app/);
|
||||
assert.match(demo, /Model weights and transient inference images are refused by name, and every skipped path is listed with its reason/);
|
||||
assert.match(demo, /Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes/);
|
||||
assert.match(demo, /A restore returns the same records, including the withdrawal tombstone a trainer must never lose/);
|
||||
|
||||
// The scene is a release gate, and the tool's own suite runs before the release exists.
|
||||
assert.match(demo, /process\.env\.TIMMY_DEMO_URL \|\| 'http:\/\/127\.0\.0\.1:4173'/);
|
||||
assert.match(builder, /tests\/backup-restore\.test\.py/);
|
||||
assert.match(builder, /tests\/production-runbook\.test\.py/);
|
||||
assert.match(builder, /"production_backup_restore": "passed"/);
|
||||
assert.match(builder, /"production_runbook_commands": "passed"/);
|
||||
assert.match(builder, /The operator backup is a command line, not a screen in the app/);
|
||||
assert.match(builder, /A restore returns the same records, including the withdrawal tombstone a trainer must never lose/);
|
||||
});
|
||||
|
||||
test('release demo runs the RC attack suite on camera and prints the verdict it derived', async () => {
|
||||
const demo = await readFile(demoPath, 'utf8');
|
||||
|
||||
// Issue #40 turns hostile uploads into a release gate. The scene must run the real suite, not
|
||||
// describe it, and the card must show the verdict the suite derived rather than a flattering one.
|
||||
assert.match(demo, /scripts\/security_attack_suite\.mjs', '--json'/);
|
||||
assert.match(demo, /SECURITY ATTACK SUITE/);
|
||||
assert.match(demo, /Every hostile upload on screen went at the real server, and the worker behind it counted every call it received/);
|
||||
assert.match(demo, /A failing case is a finding the release job has to see, so the card prints the verdict the suite derived/);
|
||||
assert.match(demo, /The gate blocks RC approval on any P0 case that reached the worker/);
|
||||
assert.match(demo, /securityReport\.verdict/);
|
||||
assert.match(demo, /securityCardProof/);
|
||||
// The finding must name where the fix lives instead of implying the boundary is already safe.
|
||||
assert.match(demo, /Refusing bytes by their content is the fix in review, not a claim this video makes/);
|
||||
// Each case row prints the fields it actually observed. A guarantee that the card never renders
|
||||
// `http undefined` (a case with no status) or the bare status again (a case with no provider calls).
|
||||
assert.match(demo, /'refused' in observed/);
|
||||
assert.match(demo, /'providerTextLeaked' in observed/);
|
||||
assert.doesNotMatch(demo, /http \$\{observed\.status\} · worker \$\{observed\.providerCalls\}`/);
|
||||
});
|
||||
@@ -0,0 +1,104 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { buildReport, containsImageBytes, evaluateCase, latencyPercentiles, redactImageBytes, releaseVerdict } from '../src/release-security-gate.js';
|
||||
|
||||
test('latency percentiles use nearest-rank so p95 is an observed sample, not an interpolation', () => {
|
||||
const samples = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100];
|
||||
assert.deepEqual(latencyPercentiles(samples), { count: 10, p50: 50, p95: 100, max: 100 });
|
||||
});
|
||||
|
||||
// A case is a promise the suite makes before it runs: this input must land here, and the provider
|
||||
// must (or must not) see exactly this many calls. The verdict is derived, never typed.
|
||||
const polyglot = {
|
||||
id: 'ingress-polyglot-html',
|
||||
severity: 'P0',
|
||||
expected: { status: 400, providerCalls: 0 },
|
||||
observed: { status: 400, providerCalls: 0 },
|
||||
};
|
||||
|
||||
test('a case whose observation matches its expectation passes', () => {
|
||||
assert.equal(evaluateCase(polyglot).status, 'pass');
|
||||
});
|
||||
|
||||
test('a case that reached the provider is a failure that names the field that broke', () => {
|
||||
const result = evaluateCase({ ...polyglot, observed: { status: 200, providerCalls: 1 } });
|
||||
assert.equal(result.status, 'fail');
|
||||
assert.match(result.mismatch, /status 400 expected, observed 200/);
|
||||
assert.match(result.mismatch, /providerCalls 0 expected, observed 1/);
|
||||
});
|
||||
|
||||
test('a P0 failure blocks the release, and a P1 failure only holds it', () => {
|
||||
const leaked = { ...polyglot, observed: { status: 200, providerCalls: 1 } };
|
||||
const blocked = releaseVerdict({ cases: [leaked, { ...polyglot, id: 'flood', severity: 'P1', observed: { status: 200, providerCalls: 12 } }] });
|
||||
assert.equal(blocked.verdict, 'BLOCKED');
|
||||
assert.deepEqual(blocked.blocking.map(item => item.id), ['ingress-polyglot-html']);
|
||||
assert.deepEqual(blocked.holds.map(item => item.id), ['flood']);
|
||||
});
|
||||
|
||||
test('a suite with no failures is a GO, and counts say how many cases were proven', () => {
|
||||
const verdict = releaseVerdict({ cases: [polyglot] });
|
||||
assert.equal(verdict.verdict, 'GO');
|
||||
assert.deepEqual(verdict.blocking, []);
|
||||
assert.equal(verdict.proven, 1);
|
||||
});
|
||||
|
||||
test('a required category that no case exercised blocks the release instead of passing by omission', () => {
|
||||
const deletion = { id: 'deletion-withdrawal', severity: 'P0', surface: 'out-of-surface', evidence: 'tests/training-contribution.test.js', evidenceVerified: false };
|
||||
const verdict = releaseVerdict({ cases: [polyglot, deletion] });
|
||||
assert.equal(verdict.verdict, 'BLOCKED');
|
||||
assert.deepEqual(verdict.unproven.map(item => item.id), ['deletion-withdrawal']);
|
||||
});
|
||||
|
||||
test('coverage borrowed from another test file only counts once the runner verified that file exists', () => {
|
||||
const verified = { id: 'deletion-withdrawal', severity: 'P0', surface: 'out-of-surface', evidence: 'tests/training-contribution.test.js', evidenceVerified: true };
|
||||
const verdict = releaseVerdict({ cases: [polyglot, verified] });
|
||||
assert.equal(verdict.verdict, 'GO');
|
||||
assert.deepEqual(verdict.coveredElsewhere, ['deletion-withdrawal']);
|
||||
});
|
||||
|
||||
// The report and the server log are shipped as release evidence. Neither may carry a photo.
|
||||
const JPEG_DATA_URI = 'data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/wAALCAABAAEBAREA/8QAFQABAQAAAAAAAAAAAAAAAAAAAAn/xAAUEAEAAAAAAAAAAAAAAAAAAAAJ/9oACAEBAAA/AKp//9k=';
|
||||
|
||||
test('a data URI or a long base64 run is recognised as image bytes', () => {
|
||||
assert.equal(containsImageBytes(`posted ${JPEG_DATA_URI} to the worker`), true);
|
||||
assert.equal(containsImageBytes('chunk ' + 'A'.repeat(64) + 'yQ'.repeat(200)), true);
|
||||
});
|
||||
|
||||
test('ordinary log lines, hex digests, and short tokens are not image bytes', () => {
|
||||
assert.equal(containsImageBytes('Timmy is listening on http://127.0.0.1:4173 · vision configured · Hermes locked and ready'), false);
|
||||
assert.equal(containsImageBytes('sha256 ' + 'a1b2c3d4'.repeat(8)), false);
|
||||
assert.equal(containsImageBytes('case ingress-polyglot-html observed status 200'), false);
|
||||
});
|
||||
|
||||
test('redaction removes the bytes and keeps the sentence around them honest', () => {
|
||||
const redacted = redactImageBytes(`provider received ${JPEG_DATA_URI} at 12:01`);
|
||||
assert.equal(containsImageBytes(redacted), false);
|
||||
assert.match(redacted, /\[redacted image bytes\]/);
|
||||
assert.match(redacted, /^provider received .* at 12:01$/);
|
||||
});
|
||||
|
||||
test('the report carries the verdict, the target it ran against, and the measured load numbers', () => {
|
||||
const report = buildReport({
|
||||
generatedAt: '2026-10-30T10:48:00.000Z',
|
||||
target: { baseUrl: 'http://127.0.0.1:4173', provider: 'stub counting worker' },
|
||||
cases: [polyglot],
|
||||
load: { requests: 8, errors: 0, abstentions: 8, p50: 12, p95: 40, max: 40 },
|
||||
});
|
||||
assert.equal(report.schema, 'timmy-rc-security-attack/1');
|
||||
assert.equal(report.verdict, 'GO');
|
||||
assert.equal(report.generatedAt, '2026-10-30T10:48:00.000Z');
|
||||
assert.equal(report.target.provider, 'stub counting worker');
|
||||
assert.equal(report.load.p95, 40);
|
||||
assert.equal(containsImageBytes(JSON.stringify(report)), false);
|
||||
});
|
||||
|
||||
test('a case that quotes a response body cannot smuggle the photo into the report', () => {
|
||||
const report = buildReport({
|
||||
generatedAt: '2026-10-30T10:48:00.000Z',
|
||||
target: { baseUrl: 'http://127.0.0.1:4173', provider: 'stub' },
|
||||
cases: [{ ...polyglot, observed: { status: 400, providerCalls: 0 }, snippet: `refused ${JPEG_DATA_URI}` }],
|
||||
load: null,
|
||||
});
|
||||
assert.equal(containsImageBytes(JSON.stringify(report)), false);
|
||||
assert.match(JSON.stringify(report), /redacted image bytes/);
|
||||
});
|
||||
@@ -0,0 +1,89 @@
|
||||
// Acceptance: the RC attack suite must drive the real server and then tell the truth about what it
|
||||
// saw. Nothing here trusts the runner's own summary line: every case status is re-derived from the
|
||||
// recorded observation with the same rule the release gate uses, and the report is scanned for photo
|
||||
// bytes. If the suite ever marks a case pass while observing a mismatch, this test fails.
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { execFile } from 'node:child_process';
|
||||
import { mkdtemp, readFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { containsImageBytes, evaluateCase, releaseVerdict } from '../src/release-security-gate.js';
|
||||
|
||||
const repoRoot = new URL('..', import.meta.url).pathname;
|
||||
|
||||
async function runSuite() {
|
||||
const directory = await mkdtemp(join(tmpdir(), 'timmy-rc-attack-'));
|
||||
const reportPath = join(directory, 'report.json');
|
||||
const run = await new Promise(resolve => {
|
||||
execFile(process.execPath, ['scripts/security_attack_suite.mjs', '--json', reportPath], {
|
||||
cwd: repoRoot,
|
||||
timeout: 180_000,
|
||||
}, (error, stdout, stderr) => resolve({
|
||||
code: error?.code ?? 0,
|
||||
stdout,
|
||||
stderr,
|
||||
}));
|
||||
});
|
||||
const report = JSON.parse(await readFile(reportPath, 'utf8'));
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
return { run, report };
|
||||
}
|
||||
|
||||
const suite = await runSuite();
|
||||
const byId = new Map(suite.report.cases.map(item => [item.id, item]));
|
||||
|
||||
test('the suite exits non-zero exactly when its own verdict blocks the release', () => {
|
||||
assert.ok([0, 1].includes(suite.run.code), `unexpected exit ${suite.run.code}: ${suite.run.stderr}`);
|
||||
assert.match(suite.run.stdout, new RegExp(`VERDICT: ${suite.report.verdict}`));
|
||||
assert.equal(suite.run.code, suite.report.verdict === 'BLOCKED' ? 1 : 0);
|
||||
});
|
||||
|
||||
test('every reported case status is re-derivable from the observation it recorded', () => {
|
||||
for (const item of suite.report.cases.filter(entry => entry.surface !== 'out-of-surface')) {
|
||||
const derived = evaluateCase(item);
|
||||
assert.equal(item.status, derived.status, `${item.id} claims ${item.status} but observed ${JSON.stringify(item.observed)} against ${JSON.stringify(item.expected)}`);
|
||||
}
|
||||
assert.equal(suite.report.verdict, releaseVerdict({ cases: suite.report.cases }).verdict);
|
||||
});
|
||||
|
||||
test('the suite covers the abuse classes the ticket names, not a convenient subset', () => {
|
||||
const required = ['oversized', 'polyglot', 'malformed', 'flood', 'log', 'deletion', 'replay'];
|
||||
const categories = new Set(suite.report.cases.map(item => item.category));
|
||||
for (const category of required) assert.ok(categories.has(category), `no case covers ${category}`);
|
||||
assert.ok(suite.report.cases.length >= 8, `only ${suite.report.cases.length} cases ran`);
|
||||
});
|
||||
|
||||
test('a request without consent is refused and the worker is never called', () => {
|
||||
const consent = byId.get('consent-required');
|
||||
assert.equal(consent.observed.status, 400);
|
||||
assert.equal(consent.observed.providerCalls, 0);
|
||||
assert.equal(consent.status, 'pass');
|
||||
});
|
||||
|
||||
test('an oversized body is refused before any provider work', () => {
|
||||
const oversized = byId.get('ingress-oversized-body');
|
||||
assert.equal(oversized.observed.status, 413);
|
||||
assert.equal(oversized.observed.providerCalls, 0);
|
||||
});
|
||||
|
||||
test('a provider answer that breaks the schema fails closed without quoting the provider', () => {
|
||||
const malformed = byId.get('model-output-malformed');
|
||||
assert.equal(malformed.observed.status, 503);
|
||||
assert.equal(malformed.observed.providerTextLeaked, false);
|
||||
assert.ok(!JSON.stringify(malformed).includes('garbage-completion-token'), 'the report must not quote the provider');
|
||||
});
|
||||
|
||||
test('the captured server log and the report itself carry no photo bytes', () => {
|
||||
assert.equal(byId.get('log-leak-scan').observed.imageBytesInLog, false);
|
||||
assert.equal(containsImageBytes(JSON.stringify(suite.report)), false);
|
||||
});
|
||||
|
||||
test('the load phase recorded percentiles, errors, and abstentions without any image body', () => {
|
||||
const load = suite.report.load;
|
||||
assert.ok(load.requests >= 8, `only ${load.requests} load requests ran`);
|
||||
assert.equal(typeof load.p50, 'number');
|
||||
assert.equal(typeof load.p95, 'number');
|
||||
assert.equal(typeof load.abstentions, 'number');
|
||||
assert.equal(containsImageBytes(JSON.stringify(load)), false);
|
||||
});
|
||||
Reference in new issue
Block a user