From 48f0f568354c52a42e5923bbdf0a7e2309a04cee Mon Sep 17 00:00:00 2001 From: Timmy Date: Thu, 29 Oct 2026 11:02:59 +0000 Subject: [PATCH] feat(ops): operator backup, restore, and production runbook (#39) The app keeps the journal in the person's browser, so the server has no diary to lose. What an operator can lose is the corpus: the append-only manifest.jsonl with its provenance records and its withdrawal tombstones. scripts/operator_backup.py archives only records and names every path it refuses (model weights, transient inference images, secrets, logs, symlinks, oversize) with the reason, writes one SHA-256 per stored member, and produces a reproducible archive. restore re-verifies every digest, member name, and record count before it writes a byte, and refuses absolute, traversal, link, unlisted, and non-empty-target cases, so a tampered archive or a lost tombstone fails loudly. docs/PRODUCTION-RUNBOOK.md covers topology, network boundaries (including the app's real 0.0.0.0 default and the staging HOST=127.0.0.1 template), the four secrets, retention, and both rollback paths; its documented commands are parsed against the real CLIs by its test. The release demo runs the real backup, verify, and restore against a synthetic corpus built by the real ingest and revoke commands and shows what was stored, what was refused, and what came back. --- README.md | 20 ++ docs/PRODUCTION-RUNBOOK.md | 172 ++++++++++++++++ package.json | 6 +- scripts/build_release.py | 12 +- scripts/operator_backup.py | 326 +++++++++++++++++++++++++++++++ scripts/record_release_demo.mjs | 121 +++++++++++- tests/backup-restore.test.py | 322 ++++++++++++++++++++++++++++++ tests/production-runbook.test.py | 161 +++++++++++++++ tests/release-demo.test.js | 34 ++++ 9 files changed, 1170 insertions(+), 4 deletions(-) create mode 100644 docs/PRODUCTION-RUNBOOK.md create mode 100644 scripts/operator_backup.py create mode 100644 tests/backup-restore.test.py create mode 100644 tests/production-runbook.test.py diff --git a/README.md b/README.md index fa82a34..1abae5f 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,7 @@ A working mobile-first bowel diary with an optional **photo-first AI assist**: - [Product and safety boundary](PRODUCT.md) - [Product boundary and release authority decision](docs/PRODUCT-DECISIONS.md) - [Self-hosted stool-vision decision](research/SELF-HOSTED-STOOL-VISION.md) +- [Production topology, secrets, backup, and rollback runbook](docs/PRODUCTION-RUNBOOK.md) ## Build a gated review release @@ -28,6 +29,25 @@ sudo python3 scripts/deploy_staging.py rollback --commit PRIOR_FULL_COMMIT Commands are fixed argv arrays executed with `shell=False`; JSON argv overrides and `--root` exist for rootless fixtures. See [the staging runbook](docs/STAGING-RUNBOOK.md) for host prerequisites, mode-600 environment injection, Caddy validation, smoke credentials, logs, backups, removal, and the explicit no-live-change boundary. +## Back up and restore the records an operator holds + +The app keeps the journal in the person's own browser, so the server has no diary to back up. What an +operator can lose is the corpus: the append-only `manifest.jsonl` with its provenance records and its +withdrawal tombstones, plus the derived images those records point at. + +```bash +python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy-nightly.tar.gz --label nightly +python3 scripts/operator_backup.py verify --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json +python3 scripts/operator_backup.py restore --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json --target /srv/timmy-corpus.restore-check +``` + +Model weights, transient inference images, secrets, logs, and symlinks are refused by name, and every +skipped path is listed in the manifest with its reason. Each stored member carries a SHA-256, the +archive is reproducible, and a restore re-verifies every digest and the record counts before it writes +anything — a tampered archive or a lost tombstone fails loudly instead of restoring quietly. See the +[production runbook](docs/PRODUCTION-RUNBOOK.md) for topology, network boundaries, secrets, retention, +and the rollback paths. + ## Run with the self-hosted open-weight path ```bash diff --git a/docs/PRODUCTION-RUNBOOK.md b/docs/PRODUCTION-RUNBOOK.md new file mode 100644 index 0000000..0cffe27 --- /dev/null +++ b/docs/PRODUCTION-RUNBOOK.md @@ -0,0 +1,172 @@ +# Production topology, secrets, backup, and rollback + +This runbook answers one question for the operator: how do the parts of Timmy run, what may they +reach, what must never be lost, and how does each part come back? It covers the pieces that exist in +this tree today — the Node app, the private vision worker, the private staging directory, and the +consented corpus. It does not describe a training fleet, a cloud account, or a hosted database, +because this build has none. + +Everything below is run by automation or by one operator on the host. Nothing here is performed by a +participant, and no step needs a public endpoint. + +## Topology + +| Process | Entry point | Holds | Restart unit | +|---|---|---|---| +| App and API | `node server.mjs` on `HOST=127.0.0.1 PORT=4173` | Nothing durable. Serves the PWA, proxies `/api/vision/*`, and gates `/api/agent/*` behind an access code. | `timmy-staging.service` (staging), `node server.mjs` (local) | +| Vision worker | `deploy/worker/run_worker_container.sh` | The pinned GGUF weights, mounted read-only from the host. | `scripts/run_selfhost_smolvlm.sh start`, container restart policy | +| Staging releases | `/opt/timmy-staging/releases/<40-character-commit>` with a `current` symlink | Immutable release trees plus the mode-600 environment file. | `timmy-staging.service` | +| Consented corpus | `/srv/timmy-corpus` | `manifest.jsonl` (append-only provenance and tombstones) and `images/` derived from consented contributions. | none — file data | +| Caddy | host Caddy | Terminates TLS and routes `/git/` to Gitea and `/timmy-staging/` to the app. | host Caddy | + +The app is stateless for a reason: there is **no server-side copy of the journal**. A person's logs +and photos live in their own browser storage, and they hold the portable JSON backup. The server +holds no diary, so a lost server loses no one's health record — only the operator records listed +above can be lost, and only those are backed up here. + +Health and readiness are separate, and the probe reports readiness only: + +``` +GET /api/healthz -> {"ok":true,"release":"","commit":"","visionEnabled":,"agentEnabled":} +``` + +`visionEnabled` and `agentEnabled` say whether the feature is configured, never which key or endpoint +is configured. The worker answers `/v1/models` before its pinned model is resident, so readiness +means "served a request from the loaded model", not "the port is open" — `scripts/worker_ready.sh` +enforces that distinction and the app withholds the AI control until it passes. + +## Network boundaries + +- The app reads `HOST` and its own default is `0.0.0.0`, which is a local-development convenience and + not the deployment. The deployment sets `HOST=127.0.0.1` in the mode-600 environment file + (`deploy/timmy-staging.env.example` is the template) and is reachable from outside solely through + the Caddy `/timmy-staging/` subpath. A bare `node server.mjs` on a shared host listens on every + interface — treat that as a laptop command, never as a production start. +- The Gitea `/git/` catch-all is declared before the staging route and must stay reachable across + every Caddy reload. +- The vision worker binds `127.0.0.1:8080` and is never published: `deploy/worker/run_worker_container.sh` + refuses a non-loopback publish host instead of warning about it, and no route in `server.mjs` + forwards arbitrary paths to the worker — the only vision endpoints are `/api/vision-status` and the + analysis request. +- Outbound from the host is limited to the pinned model download during bootstrap, which is refused + unless both SHA-256 receipts match. After that the worker needs no egress at all. +- Inference bodies and image bytes are never logged, and no reverse-proxy access log retains a + request body. A backup that contained them would be a second copy of someone's photo, so they are + excluded by the tool rather than by the operator remembering. +- No participant-facing endpoint requires an inbound port, a DNS record, or a cloud credential. + +## Secrets + +Four secrets exist, and every one of them is injected at runtime: + +| Secret | Source | Scope | +|---|---|---| +| `TIMMY_DATASET_SALT` | mode-600 environment file on the host | Corpus subject keys and split assignment. Rotating it re-splits future ingests, so it is not rotated casually. | +| Vision provider key or worker token | mode-600 environment file on the host | Only the app process reads it; the browser never receives it. | +| Agent access code | mode-600 environment file on the host | Unlocks `/api/agent/unlock`; the cookie it mints is `HttpOnly` and same-origin. | +| Gitea access token | local operator shell or secret store | Issue and PR automation only. It is never written into a release tree, an archive, or a log. | + +Rules: + +- Never commit a secret, and never pass one on a command line that lands in shell history. Use the + environment file (`chmod 600`, owned by the service user) or the systemd `EnvironmentFile=`. +- Placeholders in this document are angle-bracketed or environment references, e.g. + `TIMMY_DATASET_SALT=`. A literal value here is a bug. +- `scripts/build_release.py` scans the cloned tree for private-key blocks and assigned credentials + and fails the release if it finds one, so a secret that reaches a commit is caught before it + reaches a release. +- Rotation is: issue the replacement on the host, restart the unit, confirm `/api/healthz` and one + real inference request, then revoke the old value in the provider's own UI. Nothing in Git holds a + copy that would need rewriting. +- The backup tool refuses `.env`, `*.env`, `*.key`, `*.pem`, `id_rsa`, `tokens.json`, and + `credentials.json` by name, so a backup can be copied off-host without carrying the keys that + operate the host. + +## Backup + +The operator records that cannot be rebuilt are the corpus `manifest.jsonl` and the derived images +it points at. A tombstone is the only proof that a participant withdrew, so a backup that drops +tombstone lines silently re-authorises training on withdrawn data. The tool treats that as a +correctness bug, not a storage detail. + +`scripts/operator_backup.py` archives **only** records, and writes one manifest naming every byte it +stored and every path it refused: + +- Stored: `manifest.jsonl`, derived images, and any receipt files under the data root. +- Excluded, with the reason recorded: **model weights** (`model-weights` — re-downloadable, pinned by + hash), **transient inference** directories and uploads (`transient-inference` — a second copy of + someone's photo), **secret** files (`secret`), logs (`log`), symlinks (`symlink`), and anything over + `--max-bytes` (`oversize`). +- Every stored member carries a SHA-256, and the archive is reproducible: sorted members, zeroed + timestamps and ownership, `gzip` with a fixed mtime. Two backups of unchanged records are + byte-identical, so "did anything change?" is answerable by digest. + +```bash +python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy-nightly.tar.gz --label nightly +python3 scripts/operator_backup.py verify --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json +``` + +The command writes `.manifest.json` unless `--manifest` names another path, and prints a +receipt with the stored file count, archive digest, and record counts (`samples`, `tombstones`, +`activeSamples`). It refuses a source that is not a directory and refuses to overwrite an existing +archive, so a nightly run cannot erase the previous night. + +Retention: keep fourteen nightlies and the first backup of each month, and copy each archive plus its +manifest off-host. Also keep a human-readable record of the counts: `verify` compares the archive +against the manifest, and a restore compares the restored counts against both. + +What this backup deliberately does not cover: the browser-held journal (there is no server copy; the +person's own portable export is the backup), model weights, and anything in the release tree, which +is rebuilt from a commit by `scripts/build_release.py`. + +## Restore + +A restore never writes over live data. It verifies first, and it writes into a directory that must +not exist or must be empty: + +```bash +python3 scripts/operator_backup.py restore --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json --target /srv/timmy-corpus.restore-check +``` + +In order, before the first byte is written: + +1. the manifest is loaded and its `schemaVersion` checked; +2. every archive member is refused if it is a link, a non-regular file, an absolute path, a traversal + path, or a name the manifest never listed; +3. every member's SHA-256 is recomputed and compared, and a mismatch stops the restore; +4. a member the manifest lists but the archive lacks stops the restore. + +After the extract it recounts `manifest.jsonl` in the target and refuses if the counts differ from +the backup, so a restore that silently lost a tombstone fails loudly. Then the operator compares the +restored tree against the live one, and promotes it by moving the directories — the tool does not +pretend to know which directory is authoritative. + +Restoring the person's own logs is a different operation and stays where it belongs: the app's +import control accepts the JSON file they exported, upgrades an older version and says which version +it read, and refuses a file from a newer Timmy without touching the ledger it would have replaced. + +## Rollback + +Two rollbacks exist, and only one of them is a live staging action. + +**Release rollback (staging).** Staging keeps one immutable directory per commit and a `current` +symlink, so a rollback is a repoint, a restart, and a bounded health check that restores and verifies +the previous release if the new one does not come up: + +```bash +python3 scripts/deploy_staging.py status +python3 scripts/deploy_staging.py rollback --commit +``` + +The rollback path is exercised by `tests/staging-deploy.test.py`, which drives the real +`scripts/deploy_staging.py` code against a rootless fixture (`--root`), including the case where the +named commit has no immutable release directory, where the restart fails, and where the health probe +never passes. That is the same code path the host runs; the difference is only the root. + +The live drill on the private staging host — promote a build, serve it, roll it back, and confirm the +participant-facing page again — is a **human gate**, tracked under #53. This loop does not run that +drill automatically, because a rolled-back staging host is being used for release testing at the time. + +**Data rollback (operator records).** The equivalent for the corpus is the restore above: verify, +restore into a new directory, compare counts, then move it into place. A restore is never attempted in +place, and a failed verify leaves the live directory untouched. diff --git a/package.json b/package.json index 9c589fe..644461b 100644 --- a/package.json +++ b/package.json @@ -18,7 +18,7 @@ "test:grounded-chat": "node tests/grounded-chat.acceptance.mjs", "test:photo-lifecycle": "node tests/photo-lifecycle.acceptance.mjs", "test:training-contribution": "node tests/training-contribution.acceptance.mjs", - "check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/review_corpus.py scripts/dataset_card.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh", + "check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/review_corpus.py scripts/dataset_card.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py scripts/operator_backup.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh", "check:diff": "bash scripts/check_diff.sh", "start": "node server.mjs", "test:worker-readiness": "node tests/worker-readiness.acceptance.mjs", @@ -27,7 +27,9 @@ "test:calibration": "python3 tests/calibration-harness.test.py", "test:policy": "node --test tests/abstention-policy.test.js", "test:review": "python3 tests/corpus-review.test.py", - "test:dataset-card": "python3 tests/dataset-card.test.py" + "test:dataset-card": "python3 tests/dataset-card.test.py", + "test:backup": "python3 tests/backup-restore.test.py", + "test:runbook": "python3 tests/production-runbook.test.py" }, "devDependencies": { "playwright": "^1.62.1" diff --git a/scripts/build_release.py b/scripts/build_release.py index e9d323e..8245531 100755 --- a/scripts/build_release.py +++ b/scripts/build_release.py @@ -73,6 +73,10 @@ DEMO_REQUIRED_CAPTIONS = ( "The model said Type 6 and the reviewer said Type 5: the disagreement is recorded, not smoothed over", "Withdrawing the contribution empties the queue and the training set, and the tool refuses to review it at all", "The dataset card is generated from the corpus manifest, so no count on it is typed by hand", + "The operator backup is a command line, not a screen in the app", + "Model weights and transient inference images are refused by name, and every skipped path is listed with its reason", + "Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes", + "A restore returns the same records, including the withdrawal tombstone a trainer must never lose", "One contributor's photos stay in one split: a corpus that leaks a person across splits is refused", "Devices, contexts, and licenses stay pending until the pipeline collects them", "The card is a page, not a screen in the app — it is published with the real generator against synthetic fixtures", @@ -183,6 +187,10 @@ def main() -> int: # Issue #29's evaluation harness decides GO/BLOCKED for a model, so its refusal and gate # arithmetic are exercised before this release is allowed to exist. run(["python3", "tests/calibration-harness.test.py", "-v"], tree) + # Issue #39's operator backup is the only copy of the corpus tombstones, so the tool that + # produces it, and the runbook whose commands must stay real, are gated before the release. + run(["python3", "tests/backup-restore.test.py", "-v"], tree) + run(["python3", "tests/production-runbook.test.py", "-v"], tree) server_env = dict(os.environ) server_env.update({"HOST": "127.0.0.1", "PORT": "4173", "TIMMY_VISION_ENABLED": "false"}) # A leftover server from another checkout answers on this port and the readiness probe cannot @@ -360,7 +368,7 @@ def main() -> int: "codec": "h264", "pixel_format": "yuv420p", "recorded_from_working_app": True, - "fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything, plus the review ledger card, where the real review commands run against the committed synthetic sample and the card shows their actual output: one pending proposal with the model answer withheld, a refused reveal before commitment, a committed Type 5 correction beside the Type 4 proposal, the revealed synthetic Type 6 answer recorded as a three-field disagreement, a second reader's adjudication pointing at the review it resolved, one trainable sample, and a withdrawal that empties the queue and the training set and makes review refused", + "fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything, plus the review ledger card, where the real review commands run against the committed synthetic sample and the card shows their actual output: one pending proposal with the model answer withheld, a refused reveal before commitment, a committed Type 5 correction beside the Type 4 proposal, the revealed synthetic Type 6 answer recorded as a three-field disagreement, a second reader's adjudication pointing at the review it resolved, one trainable sample, and a withdrawal that empties the queue and the training set and makes review refused, plus the operator backup card, where the real backup, verify, and restore commands run against a synthetic corpus built by the real ingest and revoke commands and the card shows their actual output: two samples and one tombstone recorded, a weight shard, a transient inference upload, an operator env file and a log refused by name with the reason listed, the exclusion counts the manifest wrote, one digest for the archive, a verify that re-hashes every stored member, and a restore that returns the manifest byte-for-byte with the tombstone intact", }, "gates": { "unit_security": "passed", @@ -381,6 +389,8 @@ def main() -> int: "reviewer_adjudication_blind": "passed", "dataset_card_generated": "passed", "calibration_harness_gates": "passed", + "production_backup_restore": "passed", + "production_runbook_commands": "passed", "portable_backup_restore": "passed", "dependency_audit_high": "passed", "syntax": "passed", diff --git a/scripts/operator_backup.py b/scripts/operator_backup.py new file mode 100644 index 0000000..e13bfc5 --- /dev/null +++ b/scripts/operator_backup.py @@ -0,0 +1,326 @@ +#!/usr/bin/env python3 +"""Operator backup and restore for the records Timmy must not lose. + +The app keeps the journal in the person's own browser, so the server side of Timmy holds no diary. +What the operator does hold, and what cannot be rebuilt from anywhere else, is the corpus: the +append-only ``manifest.jsonl`` with its provenance records and its withdrawal tombstones, plus the +derived images those records point at. Losing that file loses the proof that a participant +withdrew, which is the one receipt the project has promised to keep. + +This tool therefore: + +- archives only records, and names every path it refused to archive with the reason + (``model-weights``, ``transient-inference``, ``secret``, ``log``, ``symlink``, ``oversize``), so a + missing file is an auditable decision instead of a silent omission; +- writes one SHA-256 per archived member and a reproducible archive (sorted members, zeroed + timestamps and ownership), so two backups of the same records are byte-identical; +- verifies every digest, member name, and record count before a restore writes anything, and + refuses absolute, traversal, link, and unlisted members. + +Usage: + + python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy.tar.gz + python3 scripts/operator_backup.py verify --archive /srv/backups/timmy.tar.gz --manifest /srv/backups/timmy.tar.gz.manifest.json + python3 scripts/operator_backup.py restore --archive ... --manifest ... --target /srv/timmy-corpus.restored +""" +from __future__ import annotations + +import argparse +import gzip +import hashlib +import io +import json +import os +import sys +import tarfile +from datetime import datetime, timezone +from pathlib import Path, PurePosixPath + +SCHEMA_VERSION = 1 +TOOL = "operator_backup.py" +PAYLOAD_PREFIX = "payload/" +DEFAULT_MAX_BYTES = 64 * 1024 * 1024 + +WEIGHT_SUFFIXES = (".gguf", ".safetensors", ".onnx", ".bin", ".pt", ".ckpt", ".pth") +WEIGHT_DIRS = ("weights", "models") +TRANSIENT_DIRS = ("transient", "inference-tmp", "inference-tmp-images", "uploads", "tmp", "cache") +LOG_DIRS = ("logs",) +LOG_SUFFIXES = (".log",) +SECRET_NAMES = (".env", "id_rsa", "id_ed25519", "id_ecdsa", "tokens.json", "credentials.json") +SECRET_SUFFIXES = (".key", ".pem", ".p12", ".pfx", ".env") + + +class BackupError(Exception): + """A refusal that is reported on stderr and exits non-zero.""" + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def classify(relative: PurePosixPath) -> str | None: + """Return the reason a path is not archived, or ``None`` when it is a record.""" + parts = relative.parts[:-1] + name = relative.name + lowered = [part.lower() for part in parts] + suffix = PurePosixPath(name.lower()).suffix + + if name.lower().startswith(".env") or name.lower() in SECRET_NAMES or suffix in SECRET_SUFFIXES: + return "secret" + if suffix in WEIGHT_SUFFIXES or any(part in WEIGHT_DIRS for part in lowered): + return "model-weights" + if any(part in TRANSIENT_DIRS for part in lowered): + return "transient-inference" + if any(part in LOG_DIRS for part in lowered) or suffix in LOG_SUFFIXES: + return "log" + return None + + +def record_counts(manifest_path: Path) -> dict[str, int]: + """Count the append-only manifest the way a trainer would read it.""" + counts = {"samples": 0, "tombstones": 0, "activeSamples": 0} + if not manifest_path.is_file(): + return counts + revoked: set[str] = set() + samples: list[dict] = [] + for line in manifest_path.read_text(encoding="utf-8").splitlines(): + if not line.strip(): + continue + try: + entry = json.loads(line) + except json.JSONDecodeError as error: + raise BackupError(f"manifest {manifest_path} is not valid JSON Lines: {error}") from error + kind = entry.get("recordType") + if kind == "sample": + samples.append(entry) + counts["samples"] += 1 + elif kind == "tombstone": + counts["tombstones"] += 1 + if entry.get("contributionId"): + revoked.add(str(entry["contributionId"])) + counts["activeSamples"] = sum( + 1 for entry in samples + if str(entry.get("contributionId")) not in revoked + and str((entry.get("lifecycle") or {}).get("state")) == "active" + ) + return counts + + +def collect(source: Path, max_bytes: int) -> tuple[list[tuple[PurePosixPath, Path]], list[dict]]: + included: list[tuple[PurePosixPath, Path]] = [] + excluded: list[dict] = [] + for dirpath, dirnames, filenames in os.walk(source, followlinks=False): + dirnames.sort() + base = Path(dirpath) + for name in sorted(filenames): + path = base / name + relative = PurePosixPath(path.relative_to(source).as_posix()) + if path.is_symlink(): + excluded.append({"path": str(relative), "reason": "symlink", "bytes": 0}) + continue + reason = classify(relative) + if reason: + excluded.append({"path": str(relative), "reason": reason, "bytes": _size(path)}) + continue + size = _size(path) + if size > max_bytes: + excluded.append({"path": str(relative), "reason": "oversize", "bytes": size}) + continue + included.append((relative, path)) + included.sort(key=lambda item: str(item[0])) + excluded.sort(key=lambda entry: entry["path"]) + return included, excluded + + +def _size(path: Path) -> int: + try: + return path.stat().st_size + except OSError: + return 0 + + +def backup(*, source: Path, archive: Path, manifest_path: Path, label: str, max_bytes: int) -> dict: + if not source.is_dir(): + raise BackupError(f"source {source} is not a directory") + if archive.exists(): + raise BackupError(f"archive {archive} already exists; move it aside instead of overwriting a backup") + + included, excluded = collect(source, max_bytes) + files = [{"path": str(relative), "sha256": sha256_file(path), "bytes": _size(path)} for relative, path in included] + counts = record_counts(source / "manifest.jsonl") + + archive.parent.mkdir(parents=True, exist_ok=True) + partial = archive.with_name(archive.name + ".partial") + with partial.open("wb") as raw: + # gzip mtime=0 and zeroed tar metadata keep two backups of the same records identical. + with gzip.GzipFile(fileobj=raw, mode="wb", mtime=0) as compressed: + with tarfile.open(fileobj=compressed, mode="w", format=tarfile.PAX_FORMAT) as tar: + for (relative, path), entry in zip(included, files): + payload = path.read_bytes() + info = tarfile.TarInfo(PAYLOAD_PREFIX + str(relative)) + info.size = len(payload) + info.mode = 0o644 + info.mtime = 0 + info.uid = info.gid = 0 + info.uname = info.gname = "" + tar.addfile(info, io.BytesIO(payload)) + if hashlib.sha256(payload).hexdigest() != entry["sha256"]: + raise BackupError(f"{relative} changed while it was being archived") + os.replace(partial, archive) + + manifest = { + "schemaVersion": SCHEMA_VERSION, + "tool": TOOL, + "createdAt": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "label": label, + "sourceLabel": source.name, + "archive": {"name": archive.name, "sha256": sha256_file(archive), "bytes": _size(archive)}, + "records": counts, + "files": files, + "excluded": excluded, + } + manifest_path.parent.mkdir(parents=True, exist_ok=True) + manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8") + return { + "archive": str(archive), + "manifest": str(manifest_path), + "files": len(files), + "bytes": manifest["archive"]["bytes"], + "records": counts, + "excluded": {entry["reason"]: sum(1 for other in excluded if other["reason"] == entry["reason"]) for entry in excluded}, + } + + +def load_manifest(manifest_path: Path) -> dict: + if not manifest_path.is_file(): + raise BackupError(f"manifest {manifest_path} does not exist") + try: + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + except json.JSONDecodeError as error: + raise BackupError(f"manifest {manifest_path} is not valid JSON: {error}") from error + if manifest.get("schemaVersion") != SCHEMA_VERSION: + raise BackupError(f"manifest schemaVersion {manifest.get('schemaVersion')!r} is not {SCHEMA_VERSION}") + return manifest + + +def verified_members(archive: Path, manifest: dict) -> list[tuple[str, bytes]]: + """Check every member of the archive before anything is written to disk.""" + if not archive.is_file(): + raise BackupError(f"archive {archive} does not exist") + expected = {entry["path"]: entry for entry in manifest.get("files") or []} + extracted: list[tuple[str, bytes]] = [] + seen: set[str] = set() + with tarfile.open(archive, "r:gz") as tar: + for info in tar.getmembers(): + if info.isdir(): + continue + if info.issym() or info.islnk(): + raise BackupError(f"archive member {info.name!r} is a link, not a regular file") + if not info.isfile(): + raise BackupError(f"archive member {info.name!r} is not a regular file") + if os.path.isabs(info.name): + raise BackupError(f"archive member {info.name!r} is an absolute path") + if not info.name.startswith(PAYLOAD_PREFIX): + raise BackupError(f"archive member {info.name!r} is not under {PAYLOAD_PREFIX}") + relative = info.name[len(PAYLOAD_PREFIX):] + parts = PurePosixPath(relative).parts + if not parts or any(part in ("..", ".", "") for part in parts): + raise BackupError(f"archive member {info.name!r} resolves outside the restore root") + if relative not in expected: + raise BackupError(f"archive member {info.name!r} is not listed in the manifest") + handle = tar.extractfile(info) + if handle is None: + raise BackupError(f"archive member {info.name!r} cannot be read") + payload = handle.read() + if hashlib.sha256(payload).hexdigest() != expected[relative]["sha256"]: + raise BackupError( + f"archive member {info.name!r} does not match the manifest digest " + "(the archive changed after it was recorded)" + ) + seen.add(relative) + extracted.append((relative, payload)) + missing = sorted(set(expected) - seen) + if missing: + raise BackupError(f"archive is missing members the manifest lists: {', '.join(missing)}") + return extracted + + +def verify(*, archive: Path, manifest_path: Path) -> dict: + manifest = load_manifest(manifest_path) + members_list = verified_members(archive, manifest) + return {"archive": str(archive), "files": len(members_list), "records": manifest.get("records", {})} + + +def restore(*, archive: Path, manifest_path: Path, target: Path) -> dict: + manifest = load_manifest(manifest_path) + if target.exists(): + if not target.is_dir(): + raise BackupError(f"restore target {target} is not a directory") + if any(target.iterdir()): + raise BackupError(f"restore target {target} is not empty; restore into a new directory") + # Digests and member names are checked before the first byte is written, so a hostile or + # tampered archive cannot leave a half-restored record set behind. + members_list = verified_members(archive, manifest) + target.mkdir(parents=True, exist_ok=True) + for relative, payload in members_list: + destination = target / relative + destination.parent.mkdir(parents=True, exist_ok=True) + destination.write_bytes(payload) + os.chmod(destination, 0o644) + + counts = record_counts(target / "manifest.jsonl") + if manifest.get("records") and counts != manifest["records"]: + raise BackupError( + f"restored record counts {counts} do not match the backup manifest {manifest['records']}" + ) + return {"target": str(target), "files": len(members_list), "records": counts} + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + subparsers = parser.add_subparsers(dest="command", required=True) + + backup_parser = subparsers.add_parser("backup", help="archive the records and write a manifest") + backup_parser.add_argument("--source", required=True, type=Path) + backup_parser.add_argument("--archive", required=True, type=Path) + backup_parser.add_argument("--manifest", type=Path) + backup_parser.add_argument("--label", default="manual") + backup_parser.add_argument("--max-bytes", type=int, default=DEFAULT_MAX_BYTES) + + verify_parser = subparsers.add_parser("verify", help="recompute every digest in an archive") + verify_parser.add_argument("--archive", required=True, type=Path) + verify_parser.add_argument("--manifest", required=True, type=Path) + + restore_parser = subparsers.add_parser("restore", help="verify, then write the records into a new directory") + restore_parser.add_argument("--archive", required=True, type=Path) + restore_parser.add_argument("--manifest", required=True, type=Path) + restore_parser.add_argument("--target", required=True, type=Path) + return parser + + +def main(argv: list[str] | None = None) -> int: + args = build_parser().parse_args(argv) + try: + if args.command == "backup": + manifest_path = args.manifest or args.archive.with_name(args.archive.name + ".manifest.json") + receipt = backup( + source=args.source, archive=args.archive, manifest_path=manifest_path, + label=args.label, max_bytes=args.max_bytes, + ) + elif args.command == "verify": + receipt = verify(archive=args.archive, manifest_path=args.manifest) + else: + receipt = restore(archive=args.archive, manifest_path=args.manifest, target=args.target) + except BackupError as error: + print(f"refused: {error}", file=sys.stderr) + return 2 + print(json.dumps(receipt, indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/record_release_demo.mjs b/scripts/record_release_demo.mjs index f4087ff..b965f8e 100644 --- a/scripts/record_release_demo.mjs +++ b/scripts/record_release_demo.mjs @@ -17,6 +17,10 @@ const version = process.env.TIMMY_RELEASE_VERSION || 'review build'; // changed in this release instead of implying a feature that did not ship. const releaseDelta = process.env.TIMMY_RELEASE_DEMO_DELTA || 'Verified rebuild of the current user-visible feature path — no new feature is claimed.'; +// The recording targets the release builder's own acceptance server by default. A caller that +// names another origin is explicit about it, so a local run can never silently record a different +// tree than the one the gates measured. +const demoUrl = process.env.TIMMY_DEMO_URL || 'http://127.0.0.1:4173'; const captionsOut = process.env.TIMMY_DEMO_CAPTIONS_OUT || ''; const captionsShown = []; const sleep = ms => new Promise(resolve => setTimeout(resolve, ms)); @@ -116,7 +120,7 @@ await page.route('**/api/agent/chat', route => { }); }); -await page.goto('http://127.0.0.1:4173', { waitUntil: 'networkidle' }); +await page.goto(demoUrl, { waitUntil: 'networkidle' }); await page.evaluate(() => localStorage.clear()); await page.reload({ waitUntil: 'networkidle' }); @@ -1161,6 +1165,121 @@ await caption('Each candidate is scored on test at a threshold chosen on validat await caption('One contributor spread across two splits is refused before anything is scored, and a passing verdict is proven only by a fixture engineered to clear it', 3800); await page.evaluate(() => document.querySelector('#calibration-report')?.remove()); +// Issue #39 — the operator backup is a command line, not a screen in the app. The app itself keeps +// no server-side journal, so what an operator can actually lose is the corpus: the append-only +// manifest with its withdrawal tombstones. This scene builds a synthetic data root with the real +// ingest and revoke commands, plants the four things a backup must refuse, then runs the real +// backup, verify, and restore and reads every value on the card back out of those runs. +async function operatorBackupAudit() { + const root = await mkdtemp(path.join(os.tmpdir(), 'timmy-backup-demo-')); + const dataRoot = path.join(root, 'srv-timmy-corpus'); + const archive = path.join(root, 'timmy-nightly.tar.gz'); + const manifestPath = `${archive}.manifest.json`; + const restoreInto = path.join(root, 'restored'); + const secondPhoto = path.join(root, 'synthetic-second.png'); + const backupCli = args => execFileAsync('python3', ['scripts/operator_backup.py', ...args], { cwd: process.cwd() }); + const ingest = (input, contributionId, subject, bristol) => corpusCli([ + 'scripts/ingest_training_photo.py', + '--input', input, '--output-dir', dataRoot, '--subject-id', subject, + '--consent-version', 'training-consent-v1', '--contribution-id', contributionId, + '--bristol-type', bristol, '--color', 'brown', '--quality', 'good', '--reviewer', 'user-confirmed', + ]); + try { + // One contribution stays active and one is withdrawn, so the manifest ends up with a live sample + // and a real tombstone, and the withdrawn derived image is already gone from disk. + await ingest('tests/fixtures/synthetic-type4.jpg', 'demo-backup-0001', 'demo-backup-kept', '4'); + await writeFile(secondPhoto, Buffer.from('iVBORw0KGgoAAAANSUhEUgAAAAgAAAAICAIAAABLbSncAAAAFElEQVR4nGOsCDFhwAaYsIoOWgkAzUwBECP1Kh0AAAAASUVORK5CYII=', 'base64')); + await ingest(secondPhoto, 'demo-backup-0002', 'demo-backup-withdrawn', '3'); + await corpusCli([ + 'scripts/ingest_training_photo.py', '--output-dir', dataRoot, + '--revoke', 'demo-backup-0002', '--reason', 'participant withdrew consent', + ]); + + // The four things that must never be archived, as real files with real bytes: a weight shard, a + // transient inference upload, an operator environment file, and a log. An exclusion that quietly + // stopped working fails here instead of in a caption. + await mkdir(path.join(dataRoot, 'weights'), { recursive: true }); + await writeFile(path.join(dataRoot, 'weights', 'model.gguf'), 'synthetic weight shard — never archived'); + await mkdir(path.join(dataRoot, 'transient'), { recursive: true }); + await writeFile(path.join(dataRoot, 'transient', 'inference-upload.jpg'), 'transient inference bytes — never archived'); + await writeFile(path.join(dataRoot, 'secrets.env'), 'TIMMY_DATASET_SALT=demo-only-placeholder\n'); + await writeFile(path.join(dataRoot, 'operator.log'), 'GET /api/healthz 200\n'); + + const backupRun = await backupCli(['backup', '--source', dataRoot, '--archive', archive, '--label', 'nightly']); + const receipt = JSON.parse(backupRun.stdout); + const manifest = JSON.parse(await readFile(manifestPath, 'utf8')); + const reasons = manifest.excluded.reduce((out, entry) => ({ ...out, [entry.reason]: (out[entry.reason] || 0) + 1 }), {}); + if (manifest.files.some(entry => /model\.gguf|inference-upload|secrets\.env|operator\.log/.test(entry.path))) { + throw new Error('the backup must not archive the weight shard or a transient inference upload'); + } + for (const reason of ['model-weights', 'transient-inference', 'secret', 'log']) { + if (!reasons[reason]) throw new Error(`the backup must record a ${reason} exclusion`); + } + if (receipt.records.samples !== 2 || receipt.records.tombstones !== 1 || receipt.records.activeSamples !== 1) { + throw new Error('the backup receipt must report the real record counts'); + } + const verified = JSON.parse((await backupCli(['verify', '--archive', archive, '--manifest', manifestPath])).stdout); + if (verified.files !== manifest.files.length) throw new Error('verify must re-hash every stored member'); + const restored = JSON.parse((await backupCli([ + 'restore', '--archive', archive, '--manifest', manifestPath, '--target', restoreInto, + ])).stdout); + if (restored.records.tombstones !== 1 || restored.records.activeSamples !== 1) { + throw new Error('the restore must return the same record counts'); + } + const restoredManifest = await readFile(path.join(restoreInto, 'manifest.jsonl')); + if (!restoredManifest.equals(await readFile(path.join(dataRoot, 'manifest.jsonl')))) { + throw new Error('the restore must return the same records, tombstone included'); + } + return { receipt, manifest, reasons, verified, restored }; + } finally { + await rm(root, { recursive: true, force: true }); + } +} +const opBackup = await operatorBackupAudit(); +await tap('[data-view="home"]', 400); +const shortDigest = value => String(value).slice(0, 12); +const backupRows = [ + ['command', 'python3 scripts/operator_backup.py backup --source srv-timmy-corpus'], + ['stored', `${opBackup.receipt.files} members · ${opBackup.manifest.files.map(entry => entry.path.split('/').pop()).join(' + ')}`], + ['records', `samples ${opBackup.receipt.records.samples} · tombstones ${opBackup.receipt.records.tombstones} · active ${opBackup.receipt.records.activeSamples}`], + ['—', ''], + ['refused', `model-weights ${opBackup.reasons['model-weights']} · transient-inference ${opBackup.reasons['transient-inference']}`], + ['refused', `secret ${opBackup.reasons['secret']} · log ${opBackup.reasons['log']}`], + ['reason', 'every skipped path is named in the manifest, not silently dropped'], + ['—', ''], + ['digest', `sha256 ${shortDigest(opBackup.manifest.archive.sha256)}… · reproducible (sorted, mtime 0)`], + ['verify', `ok · ${opBackup.verified.files} members re-hashed against the manifest`], + ['restore', `ok · samples ${opBackup.restored.records.samples} · tombstones ${opBackup.restored.records.tombstones} · active ${opBackup.restored.records.activeSamples}`], + ['manifest', 'returned byte-for-byte, tombstone included'], + ['server', 'the app holds no journal of its own — there is no server copy to back up'], +]; +const backupCard = await evidenceCard({ + id: 'production-backup', + title: 'OPERATOR BACKUP', + subtitle: 'command line, not app UI — real run, synthetic corpus', + rows: backupRows, + font: '600 9.5px/1.45 ui-monospace, SFMono-Regular, Menlo, monospace', + keyWidth: 66, + gap: '1px', + rowGap: '6px', +}); +await sleep(420); +const backupProof = await evidenceCardProof('production-backup'); +if (!backupProof.ok) throw new Error(`Production backup card is not fully readable on screen: ${JSON.stringify(backupProof)}`); +// The card is drawn from the run: prove the counts and the refusals on it are the ones it produced. +if (!backupCard.text.includes(`samples ${opBackup.receipt.records.samples} · tombstones ${opBackup.receipt.records.tombstones}`)) { + throw new Error('the card must show the record counts the backup actually reported'); +} +if (!backupCard.text.includes('model-weights 1')) throw new Error('the card must show the exclusion the run actually recorded'); +if (!backupCard.text.includes(shortDigest(opBackup.manifest.archive.sha256))) { + throw new Error('the card must show the archive digest the run actually wrote'); +} +await caption('The operator backup is a command line, not a screen in the app — this runs it against a synthetic corpus the real ingest and revoke commands built', 3800); +await caption('Model weights and transient inference images are refused by name, and every skipped path is listed with its reason', 3600); +await caption('Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes', 3400); +await caption('A restore returns the same records, including the withdrawal tombstone a trainer must never lose', 3600); +await page.evaluate(() => document.querySelector('#production-backup')?.remove()); + // Back to the privacy view for the portability scene: the privacy entry lives on the journal view. await tap('[data-view="calendar"]', 550); await tap('[data-view="privacy"]', 600); diff --git a/tests/backup-restore.test.py b/tests/backup-restore.test.py new file mode 100644 index 0000000..c0ee60c --- /dev/null +++ b/tests/backup-restore.test.py @@ -0,0 +1,322 @@ +#!/usr/bin/env python3 +"""Tests for the operator backup and restore tool. + +Issue #39 asks for a production topology and a backup that a person can trust with the records +Timmy is not allowed to lose. The corpus manifest is append-only and the tombstones in it are the +only proof that a withdrawal happened, so a backup that quietly drops them, or a restore that +rebuilds a training set from withdrawn contributions, is worse than no backup at all. + +The rules this suite locks in: + +- model weights and transient inference images are never archived, and every excluded path is + named in the manifest with the reason it was left out; +- every archived byte carries a SHA-256 that ``verify`` recomputes before a restore writes + anything, so a tampered archive fails closed; +- hostile members (absolute paths, traversal, links, unlisted extras) are refused instead of + extracted; +- a restore round-trips the append-only manifest byte-for-byte and reports the same record counts, + including the tombstones. + +Every fixture is synthetic: the "images" are opaque bytes, no real medical image is read or written. +""" +from __future__ import annotations + +import hashlib +import io +import json +import os +import shutil +import subprocess +import sys +import tarfile +import tempfile +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +BACKUP = ROOT / "scripts" / "operator_backup.py" + +IMAGE_BYTES = b"\xff\xd8\xff\xe0synthetic-jpeg-payload-not-a-real-photo" +WEIGHT_BYTES = b"GGUF-synthetic-weight-shard-not-a-real-model" +TRANSIENT_BYTES = b"transient-inference-upload-must-not-be-archived" +SECRET_BYTES = b"TIMMY_DATASET_SALT=demo\nTIMMY_VISION_TOKEN=do-not-archive-me\n" + + +def digest(payload: bytes) -> str: + return hashlib.sha256(payload).hexdigest() + + +def run(args: list[str], env: dict[str, str] | None = None) -> subprocess.CompletedProcess: + return subprocess.run( + [sys.executable, str(BACKUP), *args], + cwd=ROOT, + capture_output=True, + text=True, + env={**os.environ, **(env or {})}, + ) + + +def records(*lines: dict) -> str: + return "".join(json.dumps(line) + "\n" for line in lines) + + +def sample(contribution_id: str, *, relative_path: str = "images/sample-1.jpg") -> dict: + return { + "recordType": "sample", + "schemaVersion": 2, + "contributionId": contribution_id, + "relativePath": relative_path, + "imageId": digest(IMAGE_BYTES), + "lifecycle": {"state": "active", "updatedAt": "2026-08-01T10:00:00+00:00"}, + } + + +def tombstone(contribution_id: str) -> dict: + return { + "recordType": "tombstone", + "schemaVersion": 2, + "contributionId": contribution_id, + "lifecycle": {"state": "revoked", "updatedAt": "2026-08-02T10:00:00+00:00"}, + "reason": "participant withdrew consent", + "deletedArtifacts": [{"contributionId": contribution_id, "sha256": digest(IMAGE_BYTES)}], + } + + +def build_source(root: Path, *, with_tombstone: bool = False) -> Path: + """A synthetic server-side data root: the records that must survive, plus what must not.""" + source = root / "srv-timmy" + (source / "images").mkdir(parents=True) + (source / "images" / "sample-1.jpg").write_bytes(IMAGE_BYTES) + (source / "manifest.jsonl").write_text( + records(sample("contribution-1"), *([tombstone("contribution-2")] if with_tombstone else [])), + encoding="utf-8", + ) + # Must never be archived: weights, transient inference bytes, operator secrets, and logs. + (source / "weights").mkdir() + (source / "weights" / "model.gguf").write_bytes(WEIGHT_BYTES) + (source / "transient").mkdir() + (source / "transient" / "inference-upload-1.jpg").write_bytes(TRANSIENT_BYTES) + (source / "secrets.env").write_bytes(SECRET_BYTES) + (source / "operator.log").write_text("GET /api/vision 200\n", encoding="utf-8") + return source + + +def members(archive: Path) -> dict[str, bytes]: + with tarfile.open(archive, "r:gz") as tar: + out = {} + for info in tar.getmembers(): + if not info.isfile(): + continue + handle = tar.extractfile(info) + assert handle is not None + out[info.name] = handle.read() + return out + + +def craft_archive(path: Path, entries: list[tuple[str, bytes]], *, link: str | None = None) -> None: + """Build an archive by hand so the hostile cases are hostile on purpose.""" + with tarfile.open(path, "w:gz") as tar: + for name, payload in entries: + info = tarfile.TarInfo(name) + info.size = len(payload) + info.mode = 0o644 + tar.addfile(info, io.BytesIO(payload)) + if link: + info = tarfile.TarInfo(link) + info.type = tarfile.SYMTYPE + info.linkname = "/etc/passwd" + info.mode = 0o777 + tar.addfile(info) + + +class BackupContractTests(unittest.TestCase): + def setUp(self) -> None: + self.tmp = Path(tempfile.mkdtemp(prefix="timmy-backup-test-")) + self.addCleanup(shutil.rmtree, self.tmp, True) + self.archive = self.tmp / "backup.tar.gz" + self.manifest = self.tmp / "backup.manifest.json" + + def backup(self, source: Path, *extra: str) -> subprocess.CompletedProcess: + return run([ + "backup", "--source", str(source), "--archive", str(self.archive), + "--manifest", str(self.manifest), "--label", "nightly", *extra, + ]) + + def test_backup_archives_records_and_excludes_weights_transient_images_secrets_and_logs(self) -> None: + source = build_source(self.tmp, with_tombstone=True) + result = self.backup(source) + self.assertEqual(result.returncode, 0, result.stderr) + archived = members(self.archive) + self.assertEqual(sorted(archived), ["payload/images/sample-1.jpg", "payload/manifest.jsonl"]) + self.assertEqual(archived["payload/images/sample-1.jpg"], IMAGE_BYTES) + self.assertNotIn(WEIGHT_BYTES, b"".join(archived.values())) + self.assertNotIn(TRANSIENT_BYTES, b"".join(archived.values())) + self.assertNotIn(SECRET_BYTES, b"".join(archived.values())) + + manifest = json.loads(self.manifest.read_text(encoding="utf-8")) + excluded = {entry["path"]: entry["reason"] for entry in manifest["excluded"]} + self.assertEqual(excluded["weights/model.gguf"], "model-weights") + self.assertEqual(excluded["transient/inference-upload-1.jpg"], "transient-inference") + self.assertEqual(excluded["secrets.env"], "secret") + self.assertEqual(excluded["operator.log"], "log") + # Every archived member is listed with the digest of the bytes that were stored. + listed = {entry["path"]: entry["sha256"] for entry in manifest["files"]} + self.assertEqual(listed, { + "images/sample-1.jpg": digest(IMAGE_BYTES), + "manifest.jsonl": digest((source / "manifest.jsonl").read_bytes()), + }) + + def test_backup_reports_record_counts_including_tombstones(self) -> None: + source = build_source(self.tmp, with_tombstone=True) + result = self.backup(source) + self.assertEqual(result.returncode, 0, result.stderr) + receipt = json.loads(result.stdout) + self.assertEqual(receipt["records"], {"samples": 1, "tombstones": 1, "activeSamples": 1}) + manifest = json.loads(self.manifest.read_text(encoding="utf-8")) + self.assertEqual(manifest["records"], {"samples": 1, "tombstones": 1, "activeSamples": 1}) + self.assertEqual(manifest["label"], "nightly") + + def test_backup_refuses_to_follow_a_symlink_out_of_the_data_root(self) -> None: + source = build_source(self.tmp) + (source / "images" / "escape.jpg").symlink_to("/etc/hostname") + result = self.backup(source) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertNotIn("payload/images/escape.jpg", members(self.archive)) + manifest = json.loads(self.manifest.read_text(encoding="utf-8")) + excluded = {entry["path"]: entry["reason"] for entry in manifest["excluded"]} + self.assertEqual(excluded["images/escape.jpg"], "symlink") + + def test_backup_refuses_missing_source_and_existing_archive(self) -> None: + missing = run(["backup", "--source", str(self.tmp / "nope"), "--archive", str(self.archive)]) + self.assertNotEqual(missing.returncode, 0) + self.assertIn("is not a directory", missing.stderr) + + source = build_source(self.tmp) + self.assertEqual(self.backup(source).returncode, 0) + again = self.backup(source) + self.assertNotEqual(again.returncode, 0) + self.assertIn("already exists", again.stderr) + + def test_verify_and_restore_round_trip_the_records(self) -> None: + source = build_source(self.tmp, with_tombstone=True) + self.assertEqual(self.backup(source).returncode, 0) + verified = run(["verify", "--archive", str(self.archive), "--manifest", str(self.manifest)]) + self.assertEqual(verified.returncode, 0, verified.stderr) + + target = self.tmp / "restored" + restored = run([ + "restore", "--archive", str(self.archive), "--manifest", str(self.manifest), + "--target", str(target), + ]) + self.assertEqual(restored.returncode, 0, restored.stderr) + receipt = json.loads(restored.stdout) + self.assertEqual(receipt["records"], {"samples": 1, "tombstones": 1, "activeSamples": 1}) + self.assertEqual((target / "manifest.jsonl").read_bytes(), (source / "manifest.jsonl").read_bytes()) + self.assertEqual((target / "images" / "sample-1.jpg").read_bytes(), IMAGE_BYTES) + # The manifest is append-only history: the tombstone line survives the round trip verbatim. + lines = [json.loads(line) for line in (target / "manifest.jsonl").read_text().splitlines() if line.strip()] + self.assertEqual([line["recordType"] for line in lines], ["sample", "tombstone"]) + + def test_verify_refuses_tampered_member_before_a_restore_writes_anything(self) -> None: + source = build_source(self.tmp) + self.assertEqual(self.backup(source).returncode, 0) + manifest = json.loads(self.manifest.read_text(encoding="utf-8")) + tampered = b"\xff\xd8\xff\xe0synthetic-jpeg-payload-not-a-real-photO" + craft_archive(self.archive, [ + ("payload/images/sample-1.jpg", tampered), + ("payload/manifest.jsonl", (source / "manifest.jsonl").read_bytes()), + ]) + verified = run(["verify", "--archive", str(self.archive), "--manifest", str(self.manifest)]) + self.assertNotEqual(verified.returncode, 0) + self.assertIn("does not match", verified.stderr) + self.assertEqual(manifest["files"][0]["sha256"], digest(IMAGE_BYTES)) + + target = self.tmp / "restored-tampered" + restored = run([ + "restore", "--archive", str(self.archive), "--manifest", str(self.manifest), + "--target", str(target), + ]) + self.assertNotEqual(restored.returncode, 0) + self.assertFalse(target.exists()) + + def test_verify_refuses_a_member_the_manifest_never_listed(self) -> None: + source = build_source(self.tmp) + self.assertEqual(self.backup(source).returncode, 0) + craft_archive(self.archive, [ + ("payload/images/sample-1.jpg", IMAGE_BYTES), + ("payload/manifest.jsonl", (source / "manifest.jsonl").read_bytes()), + ("payload/extra.bin", b"unlisted"), + ]) + verified = run(["verify", "--archive", str(self.archive), "--manifest", str(self.manifest)]) + self.assertNotEqual(verified.returncode, 0) + self.assertIn("not listed in the manifest", verified.stderr) + + def test_restore_refuses_traversal_and_absolute_members(self) -> None: + for name, needle in (("payload/../../etc/passwd", "outside"), ("/etc/passwd", "absolute")): + with self.subTest(member=name): + archive = self.tmp / "hostile.tar.gz" + manifest_path = self.tmp / "hostile.manifest.json" + payload = b"root:x:0:0" + craft_archive(archive, [(name, payload)]) + manifest_path.write_text(json.dumps({ + "schemaVersion": 1, + "files": [{"path": "images/sample-1.jpg", "sha256": digest(payload), "bytes": len(payload)}], + "records": {"samples": 0, "tombstones": 0, "activeSamples": 0}, + }), encoding="utf-8") + target = self.tmp / f"hostile-out-{needle}" + restored = run([ + "restore", "--archive", str(archive), "--manifest", str(manifest_path), + "--target", str(target), + ]) + self.assertNotEqual(restored.returncode, 0) + self.assertIn(needle, restored.stderr.lower()) + self.assertFalse(target.exists()) + + def test_restore_refuses_link_members(self) -> None: + archive = self.tmp / "link.tar.gz" + manifest_path = self.tmp / "link.manifest.json" + craft_archive(archive, [("payload/manifest.jsonl", b"{}\n")], link="payload/images/sample-1.jpg") + manifest_path.write_text(json.dumps({ + "schemaVersion": 1, + "files": [{"path": "manifest.jsonl", "sha256": digest(b"{}\n"), "bytes": 3}], + "records": {"samples": 0, "tombstones": 0, "activeSamples": 0}, + }), encoding="utf-8") + target = self.tmp / "link-out" + restored = run([ + "restore", "--archive", str(archive), "--manifest", str(manifest_path), "--target", str(target), + ]) + self.assertNotEqual(restored.returncode, 0) + self.assertIn("link", restored.stderr) + self.assertFalse(target.exists()) + + def test_restore_refuses_a_non_empty_target(self) -> None: + source = build_source(self.tmp) + self.assertEqual(self.backup(source).returncode, 0) + target = self.tmp / "occupied" + target.mkdir() + (target / "keep.txt").write_text("operator file that must not be overwritten", encoding="utf-8") + restored = run([ + "restore", "--archive", str(self.archive), "--manifest", str(self.manifest), + "--target", str(target), + ]) + self.assertNotEqual(restored.returncode, 0) + self.assertIn("not empty", restored.stderr) + self.assertEqual((target / "keep.txt").read_text(encoding="utf-8"), "operator file that must not be overwritten") + + def test_backup_is_reproducible_for_the_same_records(self) -> None: + source = build_source(self.tmp, with_tombstone=True) + self.assertEqual(self.backup(source).returncode, 0) + first = json.loads(self.manifest.read_text(encoding="utf-8")) + first_digest = hashlib.sha256(self.archive.read_bytes()).hexdigest() + self.archive.unlink() + self.manifest.unlink() + self.assertEqual(self.backup(source).returncode, 0) + second = json.loads(self.manifest.read_text(encoding="utf-8")) + self.assertEqual(first["files"], second["files"]) + self.assertEqual(first["excluded"], second["excluded"]) + self.assertEqual(first_digest, hashlib.sha256(self.archive.read_bytes()).hexdigest()) + self.assertNotEqual(first["createdAt"], "") + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/tests/production-runbook.test.py b/tests/production-runbook.test.py new file mode 100644 index 0000000..c16073b --- /dev/null +++ b/tests/production-runbook.test.py @@ -0,0 +1,161 @@ +#!/usr/bin/env python3 +"""Tests for the production topology, secrets, backup, and rollback runbook. + +A runbook is only worth reading if its commands are the ones the tools actually accept, so this +suite parses every documented ``python3 scripts/...`` command out of the markdown and checks each +flag against the real CLI help. It also fails if the document starts carrying a real secret, or if +it keeps claiming the server holds no journal after ``server.mjs`` starts writing one. +""" +from __future__ import annotations + +import re +import shlex +import subprocess +import sys +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +RUNBOOK = ROOT / "docs" / "PRODUCTION-RUNBOOK.md" +SERVER = ROOT / "server.mjs" +BACKUP = ROOT / "scripts" / "operator_backup.py" + +FENCE = re.compile(r"```bash\n(.*?)```", re.S) +SECRET_ASSIGNMENT = re.compile( + r"(?i)(?:token|secret|password|passwd|api[_-]?key|salt)\s*=\s*(?!<|\$|\"|\'|\.\.\.)[^\s`\"']{6,}" +) + + +def text() -> str: + return RUNBOOK.read_text(encoding="utf-8") + + +def sections() -> dict[str, str]: + body = text() + headings = list(re.finditer(r"^## (.+)$", body, re.M)) + out: dict[str, str] = {} + for index, match in enumerate(headings): + end = headings[index + 1].start() if index + 1 < len(headings) else len(body) + out[match.group(1).strip().lower()] = body[match.end():end] + return out + + +def documented_commands() -> list[str]: + commands: list[str] = [] + for block in FENCE.findall(text()): + for line in block.splitlines(): + line = line.strip() + if line.startswith("python3 scripts/"): + commands.append(line) + return commands + + +def help_text(script: str, subcommand: str | None) -> str: + argv = [sys.executable, script] + ([subcommand] if subcommand else []) + ["--help"] + result = subprocess.run(argv, cwd=ROOT, capture_output=True, text=True) + assert result.returncode == 0, f"{' '.join(argv)} failed: {result.stderr}" + return result.stdout + + +class ProductionRunbookTests(unittest.TestCase): + def test_runbook_covers_topology_network_boundaries_secrets_backup_restore_and_rollback(self) -> None: + self.assertTrue(RUNBOOK.is_file(), "docs/PRODUCTION-RUNBOOK.md must exist") + titles = sections().keys() + for required in ("topology", "network boundaries", "secrets", "backup", "restore", "rollback"): + self.assertTrue( + any(title.startswith(required) for title in titles), + f"runbook is missing a '{required}' section; found {sorted(titles)}", + ) + + def test_backup_section_names_what_is_never_archived_and_why(self) -> None: + backup = next(body for title, body in sections().items() if title.startswith("backup")) + self.assertIn("scripts/operator_backup.py", backup) + for required in ("model weights", "transient inference", "secret", "tombstone"): + self.assertIn(required, backup.lower(), f"backup section must name {required}") + self.assertRegex(backup, r"manifest\.jsonl") + + def test_every_documented_script_command_is_accepted_by_the_real_cli(self) -> None: + commands = documented_commands() + self.assertGreaterEqual(len(commands), 4, f"expected documented commands, found {commands}") + for command in commands: + with self.subTest(command=command): + tokens = shlex.split(command) + script = tokens[1] + rest = tokens[2:] + subcommand = rest[0] if rest and not rest[0].startswith("-") else None + combined = help_text(script, subcommand) + help_text(script, None) + for token in rest: + if token.startswith("--"): + self.assertIn(token, combined, f"{script} does not accept {token}") + for token in rest[1:]: + if not token.startswith("--"): + continue + + def test_documented_backup_commands_are_the_ones_the_tool_runs(self) -> None: + commands = " ".join(documented_commands()) + for flag in ("--source", "--archive", "--manifest", "--target"): + self.assertIn(flag, commands, f"the runbook must show the real {flag} flag") + self.assertIn("verify", commands) + self.assertIn("restore", commands) + + def test_rollback_is_documented_as_a_tested_fixture_path_and_a_human_gated_live_drill(self) -> None: + rollback = next(body for title, body in sections().items() if title.startswith("rollback")) + self.assertIn("scripts/deploy_staging.py rollback", rollback) + self.assertRegex(rollback.lower(), r"human[- ]gate") + self.assertIn("#53", rollback) + self.assertIn("tests/staging-deploy.test.py", rollback) + # The live drill must be explicit about not being run by automation (wrapped lines are fine). + self.assertRegex( + rollback.lower(), + r"(?s)not .{0,80}automatically|does not run .{0,80}automatically", + "the runbook must say the live drill is not run automatically", + ) + + def test_network_boundaries_match_how_the_server_and_worker_actually_bind(self) -> None: + server = SERVER.read_text(encoding="utf-8") + # The app's own default is not loopback, so the runbook must say the deployment sets it. + self.assertRegex(server, r"HOST\|\|'0\.0\.0\.0'") + staging_env = (ROOT / "deploy" / "timmy-staging.env.example").read_text(encoding="utf-8") + self.assertIn("HOST=127.0.0.1", staging_env) + worker = (ROOT / "deploy" / "worker" / "run_worker_container.sh").read_text(encoding="utf-8") + self.assertIn("Refusing to publish the worker outside loopback", worker) + + boundaries = next(body for title, body in sections().items() if title.startswith("network boundaries")) + self.assertIn("0.0.0.0", boundaries, "the runbook must name the app's real default bind address") + self.assertIn("HOST=127.0.0.1", boundaries) + self.assertIn("deploy/timmy-staging.env.example", boundaries) + self.assertIn("loopback", boundaries.lower()) + self.assertIn("8080", boundaries) + + def test_runbook_carries_no_literal_secret_values(self) -> None: + findings = [match.group(0) for match in SECRET_ASSIGNMENT.finditer(text())] + self.assertEqual(findings, [], f"runbook looks like it contains a real secret: {findings}") + self.assertIn("mode-600", text()) + self.assertIn("never commit", text().lower()) + + def test_runbook_claims_match_a_stateless_server(self) -> None: + server = SERVER.read_text(encoding="utf-8") + for call in ("writeFile", "appendFile", "createWriteStream", "mkdirSync"): + self.assertNotIn(call, server, "server.mjs started persisting data; the runbook's data-lifetime claim must be updated") + body = text().lower() + self.assertIn("no server-side copy of the journal", body) + self.assertIn("/api/healthz", body) + # The health probe is allowed to report readiness, never configuration or secrets. + health = re.search(r"'/api/healthz'[^\n]*", server) + self.assertIsNotNone(health, "server.mjs must serve /api/healthz") + health_text = health.group(0) if health else "" + for forbidden in ("token", "secret", "env:", "password"): + self.assertNotIn(forbidden, health_text.lower()) + # The response may report readiness only, so a new field is a deliberate decision. + for field in ("ok", "release", "commit", "visionEnabled", "agentEnabled"): + self.assertIn(field, health_text) + + def test_backup_tool_is_gated_by_its_own_suite(self) -> None: + package = (ROOT / "package.json").read_text(encoding="utf-8") + self.assertIn("tests/backup-restore.test.py", package) + self.assertIn("tests/production-runbook.test.py", package) + self.assertTrue(BACKUP.is_file()) + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/tests/release-demo.test.js b/tests/release-demo.test.js index be8d2d3..e742339 100644 --- a/tests/release-demo.test.js +++ b/tests/release-demo.test.js @@ -486,3 +486,37 @@ test('release demo publishes the dataset card from a manifest it builds on camer assert.match(builder, /"dataset_card_generated": "passed"/); assert.match(builder, /scripts\/dataset_card\.py/); }); + +test('release demo runs the operator backup and restore on synthetic records instead of describing it', async () => { + const demo = await readFile(demoPath, 'utf8'); + const builder = await readFile(builderPath, 'utf8'); + + // Issue #39: the backup is a CLI, so the recording runs the real tool and shows what it stored, + // what it refused, and what the restore returned. + assert.match(demo, /scripts\/operator_backup\.py/); + assert.match(demo, /'backup', '--source'/); + assert.match(demo, /'verify', '--archive'/); + assert.match(demo, /'restore', '--archive'/); + assert.match(demo, /id: 'production-backup'/); + assert.match(demo, /Production backup card is not fully readable on screen/); + // The exclusions are proven by the run, not asserted by a caption: the decoys are real files. + assert.match(demo, /model-weights/); + assert.match(demo, /transient-inference/); + assert.match(demo, /must not archive the weight shard or a transient inference upload/); + // A restore must return the append-only manifest byte for byte, tombstone included. + assert.match(demo, /must return the same records, tombstone included/); + + assert.match(demo, /The operator backup is a command line, not a screen in the app/); + assert.match(demo, /Model weights and transient inference images are refused by name, and every skipped path is listed with its reason/); + assert.match(demo, /Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes/); + assert.match(demo, /A restore returns the same records, including the withdrawal tombstone a trainer must never lose/); + + // The scene is a release gate, and the tool's own suite runs before the release exists. + assert.match(demo, /process\.env\.TIMMY_DEMO_URL \|\| 'http:\/\/127\.0\.0\.1:4173'/); + assert.match(builder, /tests\/backup-restore\.test\.py/); + assert.match(builder, /tests\/production-runbook\.test\.py/); + assert.match(builder, /"production_backup_restore": "passed"/); + assert.match(builder, /"production_runbook_commands": "passed"/); + assert.match(builder, /The operator backup is a command line, not a screen in the app/); + assert.match(builder, /A restore returns the same records, including the withdrawal tombstone a trainer must never lose/); +});