2026-10-29 12:05:43 +00:00
9 changed files with 1170 additions and 4 deletions
@@ -6,6 +6,7 @@ A working mobile-first bowel diary with an optional **photo-first AI assist**:
 - [Product and safety boundary](PRODUCT.md)
 - [Product boundary and release authority decision](docs/PRODUCT-DECISIONS.md)
 - [Self-hosted stool-vision decision](research/SELF-HOSTED-STOOL-VISION.md)
 - [Production topology, secrets, backup, and rollback runbook](docs/PRODUCTION-RUNBOOK.md)
 ## Build a gated review release
@@ -28,6 +29,25 @@ sudo python3 scripts/deploy_staging.py rollback --commit PRIOR_FULL_COMMIT
 Commands are fixed argv arrays executed with `shell=False`; JSON argv overrides and `--root` exist for rootless fixtures. See [the staging runbook](docs/STAGING-RUNBOOK.md) for host prerequisites, mode-600 environment injection, Caddy validation, smoke credentials, logs, backups, removal, and the explicit no-live-change boundary.
 ## Back up and restore the records an operator holds
 The app keeps the journal in the person's own browser, so the server has no diary to back up. What an
 operator can lose is the corpus: the append-only `manifest.jsonl` with its provenance records and its
 withdrawal tombstones, plus the derived images those records point at.
 ```bash
 python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy-nightly.tar.gz --label nightly
 python3 scripts/operator_backup.py verify --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json
 python3 scripts/operator_backup.py restore --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json --target /srv/timmy-corpus.restore-check
 ```
 Model weights, transient inference images, secrets, logs, and symlinks are refused by name, and every
 skipped path is listed in the manifest with its reason. Each stored member carries a SHA-256, the
 archive is reproducible, and a restore re-verifies every digest and the record counts before it writes
 anything — a tampered archive or a lost tombstone fails loudly instead of restoring quietly. See the
 [production runbook](docs/PRODUCTION-RUNBOOK.md) for topology, network boundaries, secrets, retention,
 and the rollback paths.
 ## Run with the self-hosted open-weight path
 ```bash
@@ -0,0 +1,172 @@
 # Production topology, secrets, backup, and rollback
 This runbook answers one question for the operator: how do the parts of Timmy run, what may they
 reach, what must never be lost, and how does each part come back? It covers the pieces that exist in
 this tree today — the Node app, the private vision worker, the private staging directory, and the
 consented corpus. It does not describe a training fleet, a cloud account, or a hosted database,
 because this build has none.
 Everything below is run by automation or by one operator on the host. Nothing here is performed by a
 participant, and no step needs a public endpoint.
 ## Topology
 | Process | Entry point | Holds | Restart unit |
 |---|---|---|---|
 | App and API | `node server.mjs` on `HOST=127.0.0.1 PORT=4173` | Nothing durable. Serves the PWA, proxies `/api/vision/*`, and gates `/api/agent/*` behind an access code. | `timmy-staging.service` (staging), `node server.mjs` (local) |
 | Vision worker | `deploy/worker/run_worker_container.sh` | The pinned GGUF weights, mounted read-only from the host. | `scripts/run_selfhost_smolvlm.sh start`, container restart policy |
 | Staging releases | `/opt/timmy-staging/releases/<40-character-commit>` with a `current` symlink | Immutable release trees plus the mode-600 environment file. | `timmy-staging.service` |
 | Consented corpus | `/srv/timmy-corpus` | `manifest.jsonl` (append-only provenance and tombstones) and `images/` derived from consented contributions. | none — file data |
 | Caddy | host Caddy | Terminates TLS and routes `/git/` to Gitea and `/timmy-staging/` to the app. | host Caddy |
 The app is stateless for a reason: there is **no server-side copy of the journal**. A person's logs
 and photos live in their own browser storage, and they hold the portable JSON backup. The server
 holds no diary, so a lost server loses no one's health record — only the operator records listed
 above can be lost, and only those are backed up here.
 Health and readiness are separate, and the probe reports readiness only:
 ```
 GET /api/healthz  ->  {"ok":true,"release":"<tag>","commit":"<commit>","visionEnabled":<bool>,"agentEnabled":<bool>}
 ```
 `visionEnabled` and `agentEnabled` say whether the feature is configured, never which key or endpoint
 is configured. The worker answers `/v1/models` before its pinned model is resident, so readiness
 means "served a request from the loaded model", not "the port is open" — `scripts/worker_ready.sh`
 enforces that distinction and the app withholds the AI control until it passes.
 ## Network boundaries
 - The app reads `HOST` and its own default is `0.0.0.0`, which is a local-development convenience and
  not the deployment. The deployment sets `HOST=127.0.0.1` in the mode-600 environment file
  (`deploy/timmy-staging.env.example` is the template) and is reachable from outside solely through
  the Caddy `/timmy-staging/` subpath. A bare `node server.mjs` on a shared host listens on every
  interface — treat that as a laptop command, never as a production start.
 - The Gitea `/git/` catch-all is declared before the staging route and must stay reachable across
  every Caddy reload.
 - The vision worker binds `127.0.0.1:8080` and is never published: `deploy/worker/run_worker_container.sh`
  refuses a non-loopback publish host instead of warning about it, and no route in `server.mjs`
  forwards arbitrary paths to the worker — the only vision endpoints are `/api/vision-status` and the
  analysis request.
 - Outbound from the host is limited to the pinned model download during bootstrap, which is refused
  unless both SHA-256 receipts match. After that the worker needs no egress at all.
 - Inference bodies and image bytes are never logged, and no reverse-proxy access log retains a
  request body. A backup that contained them would be a second copy of someone's photo, so they are
  excluded by the tool rather than by the operator remembering.
 - No participant-facing endpoint requires an inbound port, a DNS record, or a cloud credential.
 ## Secrets
 Four secrets exist, and every one of them is injected at runtime:
 | Secret | Source | Scope |
 |---|---|---|
 | `TIMMY_DATASET_SALT` | mode-600 environment file on the host | Corpus subject keys and split assignment. Rotating it re-splits future ingests, so it is not rotated casually. |
 | Vision provider key or worker token | mode-600 environment file on the host | Only the app process reads it; the browser never receives it. |
 | Agent access code | mode-600 environment file on the host | Unlocks `/api/agent/unlock`; the cookie it mints is `HttpOnly` and same-origin. |
 | Gitea access token | local operator shell or secret store | Issue and PR automation only. It is never written into a release tree, an archive, or a log. |
 Rules:
 - Never commit a secret, and never pass one on a command line that lands in shell history. Use the
  environment file (`chmod 600`, owned by the service user) or the systemd `EnvironmentFile=`.
 - Placeholders in this document are angle-bracketed or environment references, e.g.
  `TIMMY_DATASET_SALT=<random 32+ characters, never committed>`. A literal value here is a bug.
 - `scripts/build_release.py` scans the cloned tree for private-key blocks and assigned credentials
  and fails the release if it finds one, so a secret that reaches a commit is caught before it
  reaches a release.
 - Rotation is: issue the replacement on the host, restart the unit, confirm `/api/healthz` and one
  real inference request, then revoke the old value in the provider's own UI. Nothing in Git holds a
  copy that would need rewriting.
 - The backup tool refuses `.env`, `*.env`, `*.key`, `*.pem`, `id_rsa`, `tokens.json`, and
  `credentials.json` by name, so a backup can be copied off-host without carrying the keys that
  operate the host.
 ## Backup
 The operator records that cannot be rebuilt are the corpus `manifest.jsonl` and the derived images
 it points at. A tombstone is the only proof that a participant withdrew, so a backup that drops
 tombstone lines silently re-authorises training on withdrawn data. The tool treats that as a
 correctness bug, not a storage detail.
 `scripts/operator_backup.py` archives **only** records, and writes one manifest naming every byte it
 stored and every path it refused:
 - Stored: `manifest.jsonl`, derived images, and any receipt files under the data root.
 - Excluded, with the reason recorded: **model weights** (`model-weights` — re-downloadable, pinned by
  hash), **transient inference** directories and uploads (`transient-inference` — a second copy of
  someone's photo), **secret** files (`secret`), logs (`log`), symlinks (`symlink`), and anything over
  `--max-bytes` (`oversize`).
 - Every stored member carries a SHA-256, and the archive is reproducible: sorted members, zeroed
  timestamps and ownership, `gzip` with a fixed mtime. Two backups of unchanged records are
  byte-identical, so "did anything change?" is answerable by digest.
 ```bash
 python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy-nightly.tar.gz --label nightly
 python3 scripts/operator_backup.py verify --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json
 ```
 The command writes `<archive>.manifest.json` unless `--manifest` names another path, and prints a
 receipt with the stored file count, archive digest, and record counts (`samples`, `tombstones`,
 `activeSamples`). It refuses a source that is not a directory and refuses to overwrite an existing
 archive, so a nightly run cannot erase the previous night.
 Retention: keep fourteen nightlies and the first backup of each month, and copy each archive plus its
 manifest off-host. Also keep a human-readable record of the counts: `verify` compares the archive
 against the manifest, and a restore compares the restored counts against both.
 What this backup deliberately does not cover: the browser-held journal (there is no server copy; the
 person's own portable export is the backup), model weights, and anything in the release tree, which
 is rebuilt from a commit by `scripts/build_release.py`.
 ## Restore
 A restore never writes over live data. It verifies first, and it writes into a directory that must
 not exist or must be empty:
 ```bash
 python3 scripts/operator_backup.py restore --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json --target /srv/timmy-corpus.restore-check
 ```
 In order, before the first byte is written:
 1. the manifest is loaded and its `schemaVersion` checked;
 2. every archive member is refused if it is a link, a non-regular file, an absolute path, a traversal
   path, or a name the manifest never listed;
 3. every member's SHA-256 is recomputed and compared, and a mismatch stops the restore;
 4. a member the manifest lists but the archive lacks stops the restore.
 After the extract it recounts `manifest.jsonl` in the target and refuses if the counts differ from
 the backup, so a restore that silently lost a tombstone fails loudly. Then the operator compares the
 restored tree against the live one, and promotes it by moving the directories — the tool does not
 pretend to know which directory is authoritative.
 Restoring the person's own logs is a different operation and stays where it belongs: the app's
 import control accepts the JSON file they exported, upgrades an older version and says which version
 it read, and refuses a file from a newer Timmy without touching the ledger it would have replaced.
 ## Rollback
 Two rollbacks exist, and only one of them is a live staging action.
 **Release rollback (staging).** Staging keeps one immutable directory per commit and a `current`
 symlink, so a rollback is a repoint, a restart, and a bounded health check that restores and verifies
 the previous release if the new one does not come up:
 ```bash
 python3 scripts/deploy_staging.py status
 python3 scripts/deploy_staging.py rollback --commit <FULL_COMMIT>
 ```
 The rollback path is exercised by `tests/staging-deploy.test.py`, which drives the real
 `scripts/deploy_staging.py` code against a rootless fixture (`--root`), including the case where the
 named commit has no immutable release directory, where the restart fails, and where the health probe
 never passes. That is the same code path the host runs; the difference is only the root.
 The live drill on the private staging host — promote a build, serve it, roll it back, and confirm the
 participant-facing page again — is a **human gate**, tracked under #53. This loop does not run that
 drill automatically, because a rolled-back staging host is being used for release testing at the time.
 **Data rollback (operator records).** The equivalent for the corpus is the restore above: verify,
 restore into a new directory, compare counts, then move it into place. A restore is never attempted in
 place, and a failed verify leaves the live directory untouched.
@@ -18,7 +18,7 @@
    "test:grounded-chat": "node tests/grounded-chat.acceptance.mjs",
    "test:photo-lifecycle": "node tests/photo-lifecycle.acceptance.mjs",
    "test:training-contribution": "node tests/training-contribution.acceptance.mjs",
-    "check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/review_corpus.py scripts/dataset_card.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh",
+    "check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/review_corpus.py scripts/dataset_card.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py scripts/operator_backup.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh",
    "check:diff": "bash scripts/check_diff.sh",
    "start": "node server.mjs",
    "test:worker-readiness": "node tests/worker-readiness.acceptance.mjs",
@@ -27,7 +27,9 @@
    "test:calibration": "python3 tests/calibration-harness.test.py",
    "test:policy": "node --test tests/abstention-policy.test.js",
    "test:review": "python3 tests/corpus-review.test.py",
-    "test:dataset-card": "python3 tests/dataset-card.test.py"
+    "test:dataset-card": "python3 tests/dataset-card.test.py",
    "test:backup": "python3 tests/backup-restore.test.py",
    "test:runbook": "python3 tests/production-runbook.test.py"
  },
  "devDependencies": {
    "playwright": "^1.62.1"
@@ -73,6 +73,10 @@ DEMO_REQUIRED_CAPTIONS = (
    "The model said Type 6 and the reviewer said Type 5: the disagreement is recorded, not smoothed over",
    "Withdrawing the contribution empties the queue and the training set, and the tool refuses to review it at all",
    "The dataset card is generated from the corpus manifest, so no count on it is typed by hand",
    "The operator backup is a command line, not a screen in the app",
    "Model weights and transient inference images are refused by name, and every skipped path is listed with its reason",
    "Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes",
    "A restore returns the same records, including the withdrawal tombstone a trainer must never lose",
    "One contributor's photos stay in one split: a corpus that leaks a person across splits is refused",
    "Devices, contexts, and licenses stay pending until the pipeline collects them",
    "The card is a page, not a screen in the app — it is published with the real generator against synthetic fixtures",
@@ -183,6 +187,10 @@ def main() -> int:
        # Issue #29's evaluation harness decides GO/BLOCKED for a model, so its refusal and gate
        # arithmetic are exercised before this release is allowed to exist.
        run(["python3", "tests/calibration-harness.test.py", "-v"], tree)
        # Issue #39's operator backup is the only copy of the corpus tombstones, so the tool that
        # produces it, and the runbook whose commands must stay real, are gated before the release.
        run(["python3", "tests/backup-restore.test.py", "-v"], tree)
        run(["python3", "tests/production-runbook.test.py", "-v"], tree)
        server_env = dict(os.environ)
        server_env.update({"HOST": "127.0.0.1", "PORT": "4173", "TIMMY_VISION_ENABLED": "false"})
        # A leftover server from another checkout answers on this port and the readiness probe cannot
@@ -360,7 +368,7 @@ def main() -> int:
            "codec": "h264",
            "pixel_format": "yuv420p",
            "recorded_from_working_app": True,
-            "fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything, plus the review ledger card, where the real review commands run against the committed synthetic sample and the card shows their actual output: one pending proposal with the model answer withheld, a refused reveal before commitment, a committed Type 5 correction beside the Type 4 proposal, the revealed synthetic Type 6 answer recorded as a three-field disagreement, a second reader's adjudication pointing at the review it resolved, one trainable sample, and a withdrawal that empties the queue and the training set and makes review refused",
+            "fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything, plus the review ledger card, where the real review commands run against the committed synthetic sample and the card shows their actual output: one pending proposal with the model answer withheld, a refused reveal before commitment, a committed Type 5 correction beside the Type 4 proposal, the revealed synthetic Type 6 answer recorded as a three-field disagreement, a second reader's adjudication pointing at the review it resolved, one trainable sample, and a withdrawal that empties the queue and the training set and makes review refused, plus the operator backup card, where the real backup, verify, and restore commands run against a synthetic corpus built by the real ingest and revoke commands and the card shows their actual output: two samples and one tombstone recorded, a weight shard, a transient inference upload, an operator env file and a log refused by name with the reason listed, the exclusion counts the manifest wrote, one digest for the archive, a verify that re-hashes every stored member, and a restore that returns the manifest byte-for-byte with the tombstone intact",
        },
        "gates": {
            "unit_security": "passed",
@@ -381,6 +389,8 @@ def main() -> int:
            "reviewer_adjudication_blind": "passed",
            "dataset_card_generated": "passed",
            "calibration_harness_gates": "passed",
            "production_backup_restore": "passed",
            "production_runbook_commands": "passed",
            "portable_backup_restore": "passed",
            "dependency_audit_high": "passed",
            "syntax": "passed",
@@ -0,0 +1,326 @@
 #!/usr/bin/env python3
 """Operator backup and restore for the records Timmy must not lose.
 The app keeps the journal in the person's own browser, so the server side of Timmy holds no diary.
 What the operator does hold, and what cannot be rebuilt from anywhere else, is the corpus: the
 append-only ``manifest.jsonl`` with its provenance records and its withdrawal tombstones, plus the
 derived images those records point at. Losing that file loses the proof that a participant
 withdrew, which is the one receipt the project has promised to keep.
 This tool therefore:
 - archives only records, and names every path it refused to archive with the reason
  (``model-weights``, ``transient-inference``, ``secret``, ``log``, ``symlink``, ``oversize``), so a
  missing file is an auditable decision instead of a silent omission;
 - writes one SHA-256 per archived member and a reproducible archive (sorted members, zeroed
  timestamps and ownership), so two backups of the same records are byte-identical;
 - verifies every digest, member name, and record count before a restore writes anything, and
  refuses absolute, traversal, link, and unlisted members.
 Usage:
    python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy.tar.gz
    python3 scripts/operator_backup.py verify --archive /srv/backups/timmy.tar.gz --manifest /srv/backups/timmy.tar.gz.manifest.json
    python3 scripts/operator_backup.py restore --archive ... --manifest ... --target /srv/timmy-corpus.restored
 """
 from __future__ import annotations
 import argparse
 import gzip
 import hashlib
 import io
 import json
 import os
 import sys
 import tarfile
 from datetime import datetime, timezone
 from pathlib import Path, PurePosixPath
 SCHEMA_VERSION = 1
 TOOL = "operator_backup.py"
 PAYLOAD_PREFIX = "payload/"
 DEFAULT_MAX_BYTES = 64 * 1024 * 1024
 WEIGHT_SUFFIXES = (".gguf", ".safetensors", ".onnx", ".bin", ".pt", ".ckpt", ".pth")
 WEIGHT_DIRS = ("weights", "models")
 TRANSIENT_DIRS = ("transient", "inference-tmp", "inference-tmp-images", "uploads", "tmp", "cache")
 LOG_DIRS = ("logs",)
 LOG_SUFFIXES = (".log",)
 SECRET_NAMES = (".env", "id_rsa", "id_ed25519", "id_ecdsa", "tokens.json", "credentials.json")
 SECRET_SUFFIXES = (".key", ".pem", ".p12", ".pfx", ".env")
 class BackupError(Exception):
    """A refusal that is reported on stderr and exits non-zero."""
 def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()
 def classify(relative: PurePosixPath) -> str | None:
    """Return the reason a path is not archived, or ``None`` when it is a record."""
    parts = relative.parts[:-1]
    name = relative.name
    lowered = [part.lower() for part in parts]
    suffix = PurePosixPath(name.lower()).suffix
    if name.lower().startswith(".env") or name.lower() in SECRET_NAMES or suffix in SECRET_SUFFIXES:
        return "secret"
    if suffix in WEIGHT_SUFFIXES or any(part in WEIGHT_DIRS for part in lowered):
        return "model-weights"
    if any(part in TRANSIENT_DIRS for part in lowered):
        return "transient-inference"
    if any(part in LOG_DIRS for part in lowered) or suffix in LOG_SUFFIXES:
        return "log"
    return None
 def record_counts(manifest_path: Path) -> dict[str, int]:
    """Count the append-only manifest the way a trainer would read it."""
    counts = {"samples": 0, "tombstones": 0, "activeSamples": 0}
    if not manifest_path.is_file():
        return counts
    revoked: set[str] = set()
    samples: list[dict] = []
    for line in manifest_path.read_text(encoding="utf-8").splitlines():
        if not line.strip():
            continue
        try:
            entry = json.loads(line)
        except json.JSONDecodeError as error:
            raise BackupError(f"manifest {manifest_path} is not valid JSON Lines: {error}") from error
        kind = entry.get("recordType")
        if kind == "sample":
            samples.append(entry)
            counts["samples"] += 1
        elif kind == "tombstone":
            counts["tombstones"] += 1
            if entry.get("contributionId"):
                revoked.add(str(entry["contributionId"]))
    counts["activeSamples"] = sum(
        1 for entry in samples
        if str(entry.get("contributionId")) not in revoked
        and str((entry.get("lifecycle") or {}).get("state")) == "active"
    )
    return counts
 def collect(source: Path, max_bytes: int) -> tuple[list[tuple[PurePosixPath, Path]], list[dict]]:
    included: list[tuple[PurePosixPath, Path]] = []
    excluded: list[dict] = []
    for dirpath, dirnames, filenames in os.walk(source, followlinks=False):
        dirnames.sort()
        base = Path(dirpath)
        for name in sorted(filenames):
            path = base / name
            relative = PurePosixPath(path.relative_to(source).as_posix())
            if path.is_symlink():
                excluded.append({"path": str(relative), "reason": "symlink", "bytes": 0})
                continue
            reason = classify(relative)
            if reason:
                excluded.append({"path": str(relative), "reason": reason, "bytes": _size(path)})
                continue
            size = _size(path)
            if size > max_bytes:
                excluded.append({"path": str(relative), "reason": "oversize", "bytes": size})
                continue
            included.append((relative, path))
    included.sort(key=lambda item: str(item[0]))
    excluded.sort(key=lambda entry: entry["path"])
    return included, excluded
 def _size(path: Path) -> int:
    try:
        return path.stat().st_size
    except OSError:
        return 0
 def backup(*, source: Path, archive: Path, manifest_path: Path, label: str, max_bytes: int) -> dict:
    if not source.is_dir():
        raise BackupError(f"source {source} is not a directory")
    if archive.exists():
        raise BackupError(f"archive {archive} already exists; move it aside instead of overwriting a backup")
    included, excluded = collect(source, max_bytes)
    files = [{"path": str(relative), "sha256": sha256_file(path), "bytes": _size(path)} for relative, path in included]
    counts = record_counts(source / "manifest.jsonl")
    archive.parent.mkdir(parents=True, exist_ok=True)
    partial = archive.with_name(archive.name + ".partial")
    with partial.open("wb") as raw:
        # gzip mtime=0 and zeroed tar metadata keep two backups of the same records identical.
        with gzip.GzipFile(fileobj=raw, mode="wb", mtime=0) as compressed:
            with tarfile.open(fileobj=compressed, mode="w", format=tarfile.PAX_FORMAT) as tar:
                for (relative, path), entry in zip(included, files):
                    payload = path.read_bytes()
                    info = tarfile.TarInfo(PAYLOAD_PREFIX + str(relative))
                    info.size = len(payload)
                    info.mode = 0o644
                    info.mtime = 0
                    info.uid = info.gid = 0
                    info.uname = info.gname = ""
                    tar.addfile(info, io.BytesIO(payload))
                    if hashlib.sha256(payload).hexdigest() != entry["sha256"]:
                        raise BackupError(f"{relative} changed while it was being archived")
    os.replace(partial, archive)
    manifest = {
        "schemaVersion": SCHEMA_VERSION,
        "tool": TOOL,
        "createdAt": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "label": label,
        "sourceLabel": source.name,
        "archive": {"name": archive.name, "sha256": sha256_file(archive), "bytes": _size(archive)},
        "records": counts,
        "files": files,
        "excluded": excluded,
    }
    manifest_path.parent.mkdir(parents=True, exist_ok=True)
    manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8")
    return {
        "archive": str(archive),
        "manifest": str(manifest_path),
        "files": len(files),
        "bytes": manifest["archive"]["bytes"],
        "records": counts,
        "excluded": {entry["reason"]: sum(1 for other in excluded if other["reason"] == entry["reason"]) for entry in excluded},
    }
 def load_manifest(manifest_path: Path) -> dict:
    if not manifest_path.is_file():
        raise BackupError(f"manifest {manifest_path} does not exist")
    try:
        manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
    except json.JSONDecodeError as error:
        raise BackupError(f"manifest {manifest_path} is not valid JSON: {error}") from error
    if manifest.get("schemaVersion") != SCHEMA_VERSION:
        raise BackupError(f"manifest schemaVersion {manifest.get('schemaVersion')!r} is not {SCHEMA_VERSION}")
    return manifest
 def verified_members(archive: Path, manifest: dict) -> list[tuple[str, bytes]]:
    """Check every member of the archive before anything is written to disk."""
    if not archive.is_file():
        raise BackupError(f"archive {archive} does not exist")
    expected = {entry["path"]: entry for entry in manifest.get("files") or []}
    extracted: list[tuple[str, bytes]] = []
    seen: set[str] = set()
    with tarfile.open(archive, "r:gz") as tar:
        for info in tar.getmembers():
            if info.isdir():
                continue
            if info.issym() or info.islnk():
                raise BackupError(f"archive member {info.name!r} is a link, not a regular file")
            if not info.isfile():
                raise BackupError(f"archive member {info.name!r} is not a regular file")
            if os.path.isabs(info.name):
                raise BackupError(f"archive member {info.name!r} is an absolute path")
            if not info.name.startswith(PAYLOAD_PREFIX):
                raise BackupError(f"archive member {info.name!r} is not under {PAYLOAD_PREFIX}")
            relative = info.name[len(PAYLOAD_PREFIX):]
            parts = PurePosixPath(relative).parts
            if not parts or any(part in ("..", ".", "") for part in parts):
                raise BackupError(f"archive member {info.name!r} resolves outside the restore root")
            if relative not in expected:
                raise BackupError(f"archive member {info.name!r} is not listed in the manifest")
            handle = tar.extractfile(info)
            if handle is None:
                raise BackupError(f"archive member {info.name!r} cannot be read")
            payload = handle.read()
            if hashlib.sha256(payload).hexdigest() != expected[relative]["sha256"]:
                raise BackupError(
                    f"archive member {info.name!r} does not match the manifest digest "
                    "(the archive changed after it was recorded)"
                )
            seen.add(relative)
            extracted.append((relative, payload))
    missing = sorted(set(expected) - seen)
    if missing:
        raise BackupError(f"archive is missing members the manifest lists: {', '.join(missing)}")
    return extracted
 def verify(*, archive: Path, manifest_path: Path) -> dict:
    manifest = load_manifest(manifest_path)
    members_list = verified_members(archive, manifest)
    return {"archive": str(archive), "files": len(members_list), "records": manifest.get("records", {})}
 def restore(*, archive: Path, manifest_path: Path, target: Path) -> dict:
    manifest = load_manifest(manifest_path)
    if target.exists():
        if not target.is_dir():
            raise BackupError(f"restore target {target} is not a directory")
        if any(target.iterdir()):
            raise BackupError(f"restore target {target} is not empty; restore into a new directory")
    # Digests and member names are checked before the first byte is written, so a hostile or
    # tampered archive cannot leave a half-restored record set behind.
    members_list = verified_members(archive, manifest)
    target.mkdir(parents=True, exist_ok=True)
    for relative, payload in members_list:
        destination = target / relative
        destination.parent.mkdir(parents=True, exist_ok=True)
        destination.write_bytes(payload)
        os.chmod(destination, 0o644)
    counts = record_counts(target / "manifest.jsonl")
    if manifest.get("records") and counts != manifest["records"]:
        raise BackupError(
            f"restored record counts {counts} do not match the backup manifest {manifest['records']}"
        )
    return {"target": str(target), "files": len(members_list), "records": counts}
 def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    subparsers = parser.add_subparsers(dest="command", required=True)
    backup_parser = subparsers.add_parser("backup", help="archive the records and write a manifest")
    backup_parser.add_argument("--source", required=True, type=Path)
    backup_parser.add_argument("--archive", required=True, type=Path)
    backup_parser.add_argument("--manifest", type=Path)
    backup_parser.add_argument("--label", default="manual")
    backup_parser.add_argument("--max-bytes", type=int, default=DEFAULT_MAX_BYTES)
    verify_parser = subparsers.add_parser("verify", help="recompute every digest in an archive")
    verify_parser.add_argument("--archive", required=True, type=Path)
    verify_parser.add_argument("--manifest", required=True, type=Path)
    restore_parser = subparsers.add_parser("restore", help="verify, then write the records into a new directory")
    restore_parser.add_argument("--archive", required=True, type=Path)
    restore_parser.add_argument("--manifest", required=True, type=Path)
    restore_parser.add_argument("--target", required=True, type=Path)
    return parser
 def main(argv: list[str] | None = None) -> int:
    args = build_parser().parse_args(argv)
    try:
        if args.command == "backup":
            manifest_path = args.manifest or args.archive.with_name(args.archive.name + ".manifest.json")
            receipt = backup(
                source=args.source, archive=args.archive, manifest_path=manifest_path,
                label=args.label, max_bytes=args.max_bytes,
            )
        elif args.command == "verify":
            receipt = verify(archive=args.archive, manifest_path=args.manifest)
        else:
            receipt = restore(archive=args.archive, manifest_path=args.manifest, target=args.target)
    except BackupError as error:
        print(f"refused: {error}", file=sys.stderr)
        return 2
    print(json.dumps(receipt, indent=2, sort_keys=True))
    return 0
 if __name__ == "__main__":
    sys.exit(main())
@@ -17,6 +17,10 @@ const version = process.env.TIMMY_RELEASE_VERSION || 'review build';
 // changed in this release instead of implying a feature that did not ship.
 const releaseDelta = process.env.TIMMY_RELEASE_DEMO_DELTA
  || 'Verified rebuild of the current user-visible feature path — no new feature is claimed.';
 // The recording targets the release builder's own acceptance server by default. A caller that
 // names another origin is explicit about it, so a local run can never silently record a different
 // tree than the one the gates measured.
 const demoUrl = process.env.TIMMY_DEMO_URL || 'http://127.0.0.1:4173';
 const captionsOut = process.env.TIMMY_DEMO_CAPTIONS_OUT || '';
 const captionsShown = [];
 const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
@@ -116,7 +120,7 @@ await page.route('**/api/agent/chat', route => {
  });
 });
-await page.goto('http://127.0.0.1:4173', { waitUntil: 'networkidle' });
+await page.goto(demoUrl, { waitUntil: 'networkidle' });
 await page.evaluate(() => localStorage.clear());
 await page.reload({ waitUntil: 'networkidle' });
@@ -1161,6 +1165,121 @@ await caption('Each candidate is scored on test at a threshold chosen on validat
 await caption('One contributor spread across two splits is refused before anything is scored, and a passing verdict is proven only by a fixture engineered to clear it', 3800);
 await page.evaluate(() => document.querySelector('#calibration-report')?.remove());
 // Issue #39 — the operator backup is a command line, not a screen in the app. The app itself keeps
 // no server-side journal, so what an operator can actually lose is the corpus: the append-only
 // manifest with its withdrawal tombstones. This scene builds a synthetic data root with the real
 // ingest and revoke commands, plants the four things a backup must refuse, then runs the real
 // backup, verify, and restore and reads every value on the card back out of those runs.
 async function operatorBackupAudit() {
  const root = await mkdtemp(path.join(os.tmpdir(), 'timmy-backup-demo-'));
  const dataRoot = path.join(root, 'srv-timmy-corpus');
  const archive = path.join(root, 'timmy-nightly.tar.gz');
  const manifestPath = `${archive}.manifest.json`;
  const restoreInto = path.join(root, 'restored');
  const secondPhoto = path.join(root, 'synthetic-second.png');
  const backupCli = args => execFileAsync('python3', ['scripts/operator_backup.py', ...args], { cwd: process.cwd() });
  const ingest = (input, contributionId, subject, bristol) => corpusCli([
    'scripts/ingest_training_photo.py',
    '--input', input, '--output-dir', dataRoot, '--subject-id', subject,
    '--consent-version', 'training-consent-v1', '--contribution-id', contributionId,
    '--bristol-type', bristol, '--color', 'brown', '--quality', 'good', '--reviewer', 'user-confirmed',
  ]);
  try {
    // One contribution stays active and one is withdrawn, so the manifest ends up with a live sample
    // and a real tombstone, and the withdrawn derived image is already gone from disk.
    await ingest('tests/fixtures/synthetic-type4.jpg', 'demo-backup-0001', 'demo-backup-kept', '4');
    await writeFile(secondPhoto, Buffer.from('iVBORw0KGgoAAAANSUhEUgAAAAgAAAAICAIAAABLbSncAAAAFElEQVR4nGOsCDFhwAaYsIoOWgkAzUwBECP1Kh0AAAAASUVORK5CYII=', 'base64'));
    await ingest(secondPhoto, 'demo-backup-0002', 'demo-backup-withdrawn', '3');
    await corpusCli([
      'scripts/ingest_training_photo.py', '--output-dir', dataRoot,
      '--revoke', 'demo-backup-0002', '--reason', 'participant withdrew consent',
    ]);
    // The four things that must never be archived, as real files with real bytes: a weight shard, a
    // transient inference upload, an operator environment file, and a log. An exclusion that quietly
    // stopped working fails here instead of in a caption.
    await mkdir(path.join(dataRoot, 'weights'), { recursive: true });
    await writeFile(path.join(dataRoot, 'weights', 'model.gguf'), 'synthetic weight shard — never archived');
    await mkdir(path.join(dataRoot, 'transient'), { recursive: true });
    await writeFile(path.join(dataRoot, 'transient', 'inference-upload.jpg'), 'transient inference bytes — never archived');
    await writeFile(path.join(dataRoot, 'secrets.env'), 'TIMMY_DATASET_SALT=demo-only-placeholder\n');
    await writeFile(path.join(dataRoot, 'operator.log'), 'GET /api/healthz 200\n');
    const backupRun = await backupCli(['backup', '--source', dataRoot, '--archive', archive, '--label', 'nightly']);
    const receipt = JSON.parse(backupRun.stdout);
    const manifest = JSON.parse(await readFile(manifestPath, 'utf8'));
    const reasons = manifest.excluded.reduce((out, entry) => ({ ...out, [entry.reason]: (out[entry.reason] || 0) + 1 }), {});
    if (manifest.files.some(entry => /model\.gguf|inference-upload|secrets\.env|operator\.log/.test(entry.path))) {
      throw new Error('the backup must not archive the weight shard or a transient inference upload');
    }
    for (const reason of ['model-weights', 'transient-inference', 'secret', 'log']) {
      if (!reasons[reason]) throw new Error(`the backup must record a ${reason} exclusion`);
    }
    if (receipt.records.samples !== 2 || receipt.records.tombstones !== 1 || receipt.records.activeSamples !== 1) {
      throw new Error('the backup receipt must report the real record counts');
    }
    const verified = JSON.parse((await backupCli(['verify', '--archive', archive, '--manifest', manifestPath])).stdout);
    if (verified.files !== manifest.files.length) throw new Error('verify must re-hash every stored member');
    const restored = JSON.parse((await backupCli([
      'restore', '--archive', archive, '--manifest', manifestPath, '--target', restoreInto,
    ])).stdout);
    if (restored.records.tombstones !== 1 || restored.records.activeSamples !== 1) {
      throw new Error('the restore must return the same record counts');
    }
    const restoredManifest = await readFile(path.join(restoreInto, 'manifest.jsonl'));
    if (!restoredManifest.equals(await readFile(path.join(dataRoot, 'manifest.jsonl')))) {
      throw new Error('the restore must return the same records, tombstone included');
    }
    return { receipt, manifest, reasons, verified, restored };
  } finally {
    await rm(root, { recursive: true, force: true });
  }
 }
 const opBackup = await operatorBackupAudit();
 await tap('[data-view="home"]', 400);
 const shortDigest = value => String(value).slice(0, 12);
 const backupRows = [
  ['command', 'python3 scripts/operator_backup.py backup --source srv-timmy-corpus'],
  ['stored', `${opBackup.receipt.files} members · ${opBackup.manifest.files.map(entry => entry.path.split('/').pop()).join(' + ')}`],
  ['records', `samples ${opBackup.receipt.records.samples} · tombstones ${opBackup.receipt.records.tombstones} · active ${opBackup.receipt.records.activeSamples}`],
  ['—', ''],
  ['refused', `model-weights ${opBackup.reasons['model-weights']} · transient-inference ${opBackup.reasons['transient-inference']}`],
  ['refused', `secret ${opBackup.reasons['secret']} · log ${opBackup.reasons['log']}`],
  ['reason', 'every skipped path is named in the manifest, not silently dropped'],
  ['—', ''],
  ['digest', `sha256 ${shortDigest(opBackup.manifest.archive.sha256)}… · reproducible (sorted, mtime 0)`],
  ['verify', `ok · ${opBackup.verified.files} members re-hashed against the manifest`],
  ['restore', `ok · samples ${opBackup.restored.records.samples} · tombstones ${opBackup.restored.records.tombstones} · active ${opBackup.restored.records.activeSamples}`],
  ['manifest', 'returned byte-for-byte, tombstone included'],
  ['server', 'the app holds no journal of its own — there is no server copy to back up'],
 ];
 const backupCard = await evidenceCard({
  id: 'production-backup',
  title: 'OPERATOR BACKUP',
  subtitle: 'command line, not app UI — real run, synthetic corpus',
  rows: backupRows,
  font: '600 9.5px/1.45 ui-monospace, SFMono-Regular, Menlo, monospace',
  keyWidth: 66,
  gap: '1px',
  rowGap: '6px',
 });
 await sleep(420);
 const backupProof = await evidenceCardProof('production-backup');
 if (!backupProof.ok) throw new Error(`Production backup card is not fully readable on screen: ${JSON.stringify(backupProof)}`);
 // The card is drawn from the run: prove the counts and the refusals on it are the ones it produced.
 if (!backupCard.text.includes(`samples ${opBackup.receipt.records.samples} · tombstones ${opBackup.receipt.records.tombstones}`)) {
  throw new Error('the card must show the record counts the backup actually reported');
 }
 if (!backupCard.text.includes('model-weights 1')) throw new Error('the card must show the exclusion the run actually recorded');
 if (!backupCard.text.includes(shortDigest(opBackup.manifest.archive.sha256))) {
  throw new Error('the card must show the archive digest the run actually wrote');
 }
 await caption('The operator backup is a command line, not a screen in the app — this runs it against a synthetic corpus the real ingest and revoke commands built', 3800);
 await caption('Model weights and transient inference images are refused by name, and every skipped path is listed with its reason', 3600);
 await caption('Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes', 3400);
 await caption('A restore returns the same records, including the withdrawal tombstone a trainer must never lose', 3600);
 await page.evaluate(() => document.querySelector('#production-backup')?.remove());
 // Back to the privacy view for the portability scene: the privacy entry lives on the journal view.
 await tap('[data-view="calendar"]', 550);
 await tap('[data-view="privacy"]', 600);
@@ -0,0 +1,322 @@
 #!/usr/bin/env python3
 """Tests for the operator backup and restore tool.
 Issue #39 asks for a production topology and a backup that a person can trust with the records
 Timmy is not allowed to lose. The corpus manifest is append-only and the tombstones in it are the
 only proof that a withdrawal happened, so a backup that quietly drops them, or a restore that
 rebuilds a training set from withdrawn contributions, is worse than no backup at all.
 The rules this suite locks in:
 - model weights and transient inference images are never archived, and every excluded path is
  named in the manifest with the reason it was left out;
 - every archived byte carries a SHA-256 that ``verify`` recomputes before a restore writes
  anything, so a tampered archive fails closed;
 - hostile members (absolute paths, traversal, links, unlisted extras) are refused instead of
  extracted;
 - a restore round-trips the append-only manifest byte-for-byte and reports the same record counts,
  including the tombstones.
 Every fixture is synthetic: the "images" are opaque bytes, no real medical image is read or written.
 """
 from __future__ import annotations
 import hashlib
 import io
 import json
 import os
 import shutil
 import subprocess
 import sys
 import tarfile
 import tempfile
 import unittest
 from pathlib import Path
 ROOT = Path(__file__).resolve().parents[1]
 BACKUP = ROOT / "scripts" / "operator_backup.py"
 IMAGE_BYTES = b"\xff\xd8\xff\xe0synthetic-jpeg-payload-not-a-real-photo"
 WEIGHT_BYTES = b"GGUF-synthetic-weight-shard-not-a-real-model"
 TRANSIENT_BYTES = b"transient-inference-upload-must-not-be-archived"
 SECRET_BYTES = b"TIMMY_DATASET_SALT=demo\nTIMMY_VISION_TOKEN=do-not-archive-me\n"
 def digest(payload: bytes) -> str:
    return hashlib.sha256(payload).hexdigest()
 def run(args: list[str], env: dict[str, str] | None = None) -> subprocess.CompletedProcess:
    return subprocess.run(
        [sys.executable, str(BACKUP), *args],
        cwd=ROOT,
        capture_output=True,
        text=True,
        env={**os.environ, **(env or {})},
    )
 def records(*lines: dict) -> str:
    return "".join(json.dumps(line) + "\n" for line in lines)
 def sample(contribution_id: str, *, relative_path: str = "images/sample-1.jpg") -> dict:
    return {
        "recordType": "sample",
        "schemaVersion": 2,
        "contributionId": contribution_id,
        "relativePath": relative_path,
        "imageId": digest(IMAGE_BYTES),
        "lifecycle": {"state": "active", "updatedAt": "2026-08-01T10:00:00+00:00"},
    }
 def tombstone(contribution_id: str) -> dict:
    return {
        "recordType": "tombstone",
        "schemaVersion": 2,
        "contributionId": contribution_id,
        "lifecycle": {"state": "revoked", "updatedAt": "2026-08-02T10:00:00+00:00"},
        "reason": "participant withdrew consent",
        "deletedArtifacts": [{"contributionId": contribution_id, "sha256": digest(IMAGE_BYTES)}],
    }
 def build_source(root: Path, *, with_tombstone: bool = False) -> Path:
    """A synthetic server-side data root: the records that must survive, plus what must not."""
    source = root / "srv-timmy"
    (source / "images").mkdir(parents=True)
    (source / "images" / "sample-1.jpg").write_bytes(IMAGE_BYTES)
    (source / "manifest.jsonl").write_text(
        records(sample("contribution-1"), *([tombstone("contribution-2")] if with_tombstone else [])),
        encoding="utf-8",
    )
    # Must never be archived: weights, transient inference bytes, operator secrets, and logs.
    (source / "weights").mkdir()
    (source / "weights" / "model.gguf").write_bytes(WEIGHT_BYTES)
    (source / "transient").mkdir()
    (source / "transient" / "inference-upload-1.jpg").write_bytes(TRANSIENT_BYTES)
    (source / "secrets.env").write_bytes(SECRET_BYTES)
    (source / "operator.log").write_text("GET /api/vision 200\n", encoding="utf-8")
    return source
 def members(archive: Path) -> dict[str, bytes]:
    with tarfile.open(archive, "r:gz") as tar:
        out = {}
        for info in tar.getmembers():
            if not info.isfile():
                continue
            handle = tar.extractfile(info)
            assert handle is not None
            out[info.name] = handle.read()
        return out
 def craft_archive(path: Path, entries: list[tuple[str, bytes]], *, link: str | None = None) -> None:
    """Build an archive by hand so the hostile cases are hostile on purpose."""
    with tarfile.open(path, "w:gz") as tar:
        for name, payload in entries:
            info = tarfile.TarInfo(name)
            info.size = len(payload)
            info.mode = 0o644
            tar.addfile(info, io.BytesIO(payload))
        if link:
            info = tarfile.TarInfo(link)
            info.type = tarfile.SYMTYPE
            info.linkname = "/etc/passwd"
            info.mode = 0o777
            tar.addfile(info)
 class BackupContractTests(unittest.TestCase):
    def setUp(self) -> None:
        self.tmp = Path(tempfile.mkdtemp(prefix="timmy-backup-test-"))
        self.addCleanup(shutil.rmtree, self.tmp, True)
        self.archive = self.tmp / "backup.tar.gz"
        self.manifest = self.tmp / "backup.manifest.json"
    def backup(self, source: Path, *extra: str) -> subprocess.CompletedProcess:
        return run([
            "backup", "--source", str(source), "--archive", str(self.archive),
            "--manifest", str(self.manifest), "--label", "nightly", *extra,
        ])
    def test_backup_archives_records_and_excludes_weights_transient_images_secrets_and_logs(self) -> None:
        source = build_source(self.tmp, with_tombstone=True)
        result = self.backup(source)
        self.assertEqual(result.returncode, 0, result.stderr)
        archived = members(self.archive)
        self.assertEqual(sorted(archived), ["payload/images/sample-1.jpg", "payload/manifest.jsonl"])
        self.assertEqual(archived["payload/images/sample-1.jpg"], IMAGE_BYTES)
        self.assertNotIn(WEIGHT_BYTES, b"".join(archived.values()))
        self.assertNotIn(TRANSIENT_BYTES, b"".join(archived.values()))
        self.assertNotIn(SECRET_BYTES, b"".join(archived.values()))
        manifest = json.loads(self.manifest.read_text(encoding="utf-8"))
        excluded = {entry["path"]: entry["reason"] for entry in manifest["excluded"]}
        self.assertEqual(excluded["weights/model.gguf"], "model-weights")
        self.assertEqual(excluded["transient/inference-upload-1.jpg"], "transient-inference")
        self.assertEqual(excluded["secrets.env"], "secret")
        self.assertEqual(excluded["operator.log"], "log")
        # Every archived member is listed with the digest of the bytes that were stored.
        listed = {entry["path"]: entry["sha256"] for entry in manifest["files"]}
        self.assertEqual(listed, {
            "images/sample-1.jpg": digest(IMAGE_BYTES),
            "manifest.jsonl": digest((source / "manifest.jsonl").read_bytes()),
        })
    def test_backup_reports_record_counts_including_tombstones(self) -> None:
        source = build_source(self.tmp, with_tombstone=True)
        result = self.backup(source)
        self.assertEqual(result.returncode, 0, result.stderr)
        receipt = json.loads(result.stdout)
        self.assertEqual(receipt["records"], {"samples": 1, "tombstones": 1, "activeSamples": 1})
        manifest = json.loads(self.manifest.read_text(encoding="utf-8"))
        self.assertEqual(manifest["records"], {"samples": 1, "tombstones": 1, "activeSamples": 1})
        self.assertEqual(manifest["label"], "nightly")
    def test_backup_refuses_to_follow_a_symlink_out_of_the_data_root(self) -> None:
        source = build_source(self.tmp)
        (source / "images" / "escape.jpg").symlink_to("/etc/hostname")
        result = self.backup(source)
        self.assertEqual(result.returncode, 0, result.stderr)
        self.assertNotIn("payload/images/escape.jpg", members(self.archive))
        manifest = json.loads(self.manifest.read_text(encoding="utf-8"))
        excluded = {entry["path"]: entry["reason"] for entry in manifest["excluded"]}
        self.assertEqual(excluded["images/escape.jpg"], "symlink")
    def test_backup_refuses_missing_source_and_existing_archive(self) -> None:
        missing = run(["backup", "--source", str(self.tmp / "nope"), "--archive", str(self.archive)])
        self.assertNotEqual(missing.returncode, 0)
        self.assertIn("is not a directory", missing.stderr)
        source = build_source(self.tmp)
        self.assertEqual(self.backup(source).returncode, 0)
        again = self.backup(source)
        self.assertNotEqual(again.returncode, 0)
        self.assertIn("already exists", again.stderr)
    def test_verify_and_restore_round_trip_the_records(self) -> None:
        source = build_source(self.tmp, with_tombstone=True)
        self.assertEqual(self.backup(source).returncode, 0)
        verified = run(["verify", "--archive", str(self.archive), "--manifest", str(self.manifest)])
        self.assertEqual(verified.returncode, 0, verified.stderr)
        target = self.tmp / "restored"
        restored = run([
            "restore", "--archive", str(self.archive), "--manifest", str(self.manifest),
            "--target", str(target),
        ])
        self.assertEqual(restored.returncode, 0, restored.stderr)
        receipt = json.loads(restored.stdout)
        self.assertEqual(receipt["records"], {"samples": 1, "tombstones": 1, "activeSamples": 1})
        self.assertEqual((target / "manifest.jsonl").read_bytes(), (source / "manifest.jsonl").read_bytes())
        self.assertEqual((target / "images" / "sample-1.jpg").read_bytes(), IMAGE_BYTES)
        # The manifest is append-only history: the tombstone line survives the round trip verbatim.
        lines = [json.loads(line) for line in (target / "manifest.jsonl").read_text().splitlines() if line.strip()]
        self.assertEqual([line["recordType"] for line in lines], ["sample", "tombstone"])
    def test_verify_refuses_tampered_member_before_a_restore_writes_anything(self) -> None:
        source = build_source(self.tmp)
        self.assertEqual(self.backup(source).returncode, 0)
        manifest = json.loads(self.manifest.read_text(encoding="utf-8"))
        tampered = b"\xff\xd8\xff\xe0synthetic-jpeg-payload-not-a-real-photO"
        craft_archive(self.archive, [
            ("payload/images/sample-1.jpg", tampered),
            ("payload/manifest.jsonl", (source / "manifest.jsonl").read_bytes()),
        ])
        verified = run(["verify", "--archive", str(self.archive), "--manifest", str(self.manifest)])
        self.assertNotEqual(verified.returncode, 0)
        self.assertIn("does not match", verified.stderr)
        self.assertEqual(manifest["files"][0]["sha256"], digest(IMAGE_BYTES))
        target = self.tmp / "restored-tampered"
        restored = run([
            "restore", "--archive", str(self.archive), "--manifest", str(self.manifest),
            "--target", str(target),
        ])
        self.assertNotEqual(restored.returncode, 0)
        self.assertFalse(target.exists())
    def test_verify_refuses_a_member_the_manifest_never_listed(self) -> None:
        source = build_source(self.tmp)
        self.assertEqual(self.backup(source).returncode, 0)
        craft_archive(self.archive, [
            ("payload/images/sample-1.jpg", IMAGE_BYTES),
            ("payload/manifest.jsonl", (source / "manifest.jsonl").read_bytes()),
            ("payload/extra.bin", b"unlisted"),
        ])
        verified = run(["verify", "--archive", str(self.archive), "--manifest", str(self.manifest)])
        self.assertNotEqual(verified.returncode, 0)
        self.assertIn("not listed in the manifest", verified.stderr)
    def test_restore_refuses_traversal_and_absolute_members(self) -> None:
        for name, needle in (("payload/../../etc/passwd", "outside"), ("/etc/passwd", "absolute")):
            with self.subTest(member=name):
                archive = self.tmp / "hostile.tar.gz"
                manifest_path = self.tmp / "hostile.manifest.json"
                payload = b"root:x:0:0"
                craft_archive(archive, [(name, payload)])
                manifest_path.write_text(json.dumps({
                    "schemaVersion": 1,
                    "files": [{"path": "images/sample-1.jpg", "sha256": digest(payload), "bytes": len(payload)}],
                    "records": {"samples": 0, "tombstones": 0, "activeSamples": 0},
                }), encoding="utf-8")
                target = self.tmp / f"hostile-out-{needle}"
                restored = run([
                    "restore", "--archive", str(archive), "--manifest", str(manifest_path),
                    "--target", str(target),
                ])
                self.assertNotEqual(restored.returncode, 0)
                self.assertIn(needle, restored.stderr.lower())
                self.assertFalse(target.exists())
    def test_restore_refuses_link_members(self) -> None:
        archive = self.tmp / "link.tar.gz"
        manifest_path = self.tmp / "link.manifest.json"
        craft_archive(archive, [("payload/manifest.jsonl", b"{}\n")], link="payload/images/sample-1.jpg")
        manifest_path.write_text(json.dumps({
            "schemaVersion": 1,
            "files": [{"path": "manifest.jsonl", "sha256": digest(b"{}\n"), "bytes": 3}],
            "records": {"samples": 0, "tombstones": 0, "activeSamples": 0},
        }), encoding="utf-8")
        target = self.tmp / "link-out"
        restored = run([
            "restore", "--archive", str(archive), "--manifest", str(manifest_path), "--target", str(target),
        ])
        self.assertNotEqual(restored.returncode, 0)
        self.assertIn("link", restored.stderr)
        self.assertFalse(target.exists())
    def test_restore_refuses_a_non_empty_target(self) -> None:
        source = build_source(self.tmp)
        self.assertEqual(self.backup(source).returncode, 0)
        target = self.tmp / "occupied"
        target.mkdir()
        (target / "keep.txt").write_text("operator file that must not be overwritten", encoding="utf-8")
        restored = run([
            "restore", "--archive", str(self.archive), "--manifest", str(self.manifest),
            "--target", str(target),
        ])
        self.assertNotEqual(restored.returncode, 0)
        self.assertIn("not empty", restored.stderr)
        self.assertEqual((target / "keep.txt").read_text(encoding="utf-8"), "operator file that must not be overwritten")
    def test_backup_is_reproducible_for_the_same_records(self) -> None:
        source = build_source(self.tmp, with_tombstone=True)
        self.assertEqual(self.backup(source).returncode, 0)
        first = json.loads(self.manifest.read_text(encoding="utf-8"))
        first_digest = hashlib.sha256(self.archive.read_bytes()).hexdigest()
        self.archive.unlink()
        self.manifest.unlink()
        self.assertEqual(self.backup(source).returncode, 0)
        second = json.loads(self.manifest.read_text(encoding="utf-8"))
        self.assertEqual(first["files"], second["files"])
        self.assertEqual(first["excluded"], second["excluded"])
        self.assertEqual(first_digest, hashlib.sha256(self.archive.read_bytes()).hexdigest())
        self.assertNotEqual(first["createdAt"], "")
 if __name__ == "__main__":
    unittest.main(verbosity=2)
@@ -0,0 +1,161 @@
 #!/usr/bin/env python3
 """Tests for the production topology, secrets, backup, and rollback runbook.
 A runbook is only worth reading if its commands are the ones the tools actually accept, so this
 suite parses every documented ``python3 scripts/...`` command out of the markdown and checks each
 flag against the real CLI help. It also fails if the document starts carrying a real secret, or if
 it keeps claiming the server holds no journal after ``server.mjs`` starts writing one.
 """
 from __future__ import annotations
 import re
 import shlex
 import subprocess
 import sys
 import unittest
 from pathlib import Path
 ROOT = Path(__file__).resolve().parents[1]
 RUNBOOK = ROOT / "docs" / "PRODUCTION-RUNBOOK.md"
 SERVER = ROOT / "server.mjs"
 BACKUP = ROOT / "scripts" / "operator_backup.py"
 FENCE = re.compile(r"```bash\n(.*?)```", re.S)
 SECRET_ASSIGNMENT = re.compile(
    r"(?i)(?:token|secret|password|passwd|api[_-]?key|salt)\s*=\s*(?!<|\$|\"|\'|\.\.\.)[^\s`\"']{6,}"
 )
 def text() -> str:
    return RUNBOOK.read_text(encoding="utf-8")
 def sections() -> dict[str, str]:
    body = text()
    headings = list(re.finditer(r"^## (.+)$", body, re.M))
    out: dict[str, str] = {}
    for index, match in enumerate(headings):
        end = headings[index + 1].start() if index + 1 < len(headings) else len(body)
        out[match.group(1).strip().lower()] = body[match.end():end]
    return out
 def documented_commands() -> list[str]:
    commands: list[str] = []
    for block in FENCE.findall(text()):
        for line in block.splitlines():
            line = line.strip()
            if line.startswith("python3 scripts/"):
                commands.append(line)
    return commands
 def help_text(script: str, subcommand: str | None) -> str:
    argv = [sys.executable, script] + ([subcommand] if subcommand else []) + ["--help"]
    result = subprocess.run(argv, cwd=ROOT, capture_output=True, text=True)
    assert result.returncode == 0, f"{' '.join(argv)} failed: {result.stderr}"
    return result.stdout
 class ProductionRunbookTests(unittest.TestCase):
    def test_runbook_covers_topology_network_boundaries_secrets_backup_restore_and_rollback(self) -> None:
        self.assertTrue(RUNBOOK.is_file(), "docs/PRODUCTION-RUNBOOK.md must exist")
        titles = sections().keys()
        for required in ("topology", "network boundaries", "secrets", "backup", "restore", "rollback"):
            self.assertTrue(
                any(title.startswith(required) for title in titles),
                f"runbook is missing a '{required}' section; found {sorted(titles)}",
            )
    def test_backup_section_names_what_is_never_archived_and_why(self) -> None:
        backup = next(body for title, body in sections().items() if title.startswith("backup"))
        self.assertIn("scripts/operator_backup.py", backup)
        for required in ("model weights", "transient inference", "secret", "tombstone"):
            self.assertIn(required, backup.lower(), f"backup section must name {required}")
        self.assertRegex(backup, r"manifest\.jsonl")
    def test_every_documented_script_command_is_accepted_by_the_real_cli(self) -> None:
        commands = documented_commands()
        self.assertGreaterEqual(len(commands), 4, f"expected documented commands, found {commands}")
        for command in commands:
            with self.subTest(command=command):
                tokens = shlex.split(command)
                script = tokens[1]
                rest = tokens[2:]
                subcommand = rest[0] if rest and not rest[0].startswith("-") else None
                combined = help_text(script, subcommand) + help_text(script, None)
                for token in rest:
                    if token.startswith("--"):
                        self.assertIn(token, combined, f"{script} does not accept {token}")
                for token in rest[1:]:
                    if not token.startswith("--"):
                        continue
    def test_documented_backup_commands_are_the_ones_the_tool_runs(self) -> None:
        commands = " ".join(documented_commands())
        for flag in ("--source", "--archive", "--manifest", "--target"):
            self.assertIn(flag, commands, f"the runbook must show the real {flag} flag")
        self.assertIn("verify", commands)
        self.assertIn("restore", commands)
    def test_rollback_is_documented_as_a_tested_fixture_path_and_a_human_gated_live_drill(self) -> None:
        rollback = next(body for title, body in sections().items() if title.startswith("rollback"))
        self.assertIn("scripts/deploy_staging.py rollback", rollback)
        self.assertRegex(rollback.lower(), r"human[- ]gate")
        self.assertIn("#53", rollback)
        self.assertIn("tests/staging-deploy.test.py", rollback)
        # The live drill must be explicit about not being run by automation (wrapped lines are fine).
        self.assertRegex(
            rollback.lower(),
            r"(?s)not .{0,80}automatically|does not run .{0,80}automatically",
            "the runbook must say the live drill is not run automatically",
        )
    def test_network_boundaries_match_how_the_server_and_worker_actually_bind(self) -> None:
        server = SERVER.read_text(encoding="utf-8")
        # The app's own default is not loopback, so the runbook must say the deployment sets it.
        self.assertRegex(server, r"HOST\|\|'0\.0\.0\.0'")
        staging_env = (ROOT / "deploy" / "timmy-staging.env.example").read_text(encoding="utf-8")
        self.assertIn("HOST=127.0.0.1", staging_env)
        worker = (ROOT / "deploy" / "worker" / "run_worker_container.sh").read_text(encoding="utf-8")
        self.assertIn("Refusing to publish the worker outside loopback", worker)
        boundaries = next(body for title, body in sections().items() if title.startswith("network boundaries"))
        self.assertIn("0.0.0.0", boundaries, "the runbook must name the app's real default bind address")
        self.assertIn("HOST=127.0.0.1", boundaries)
        self.assertIn("deploy/timmy-staging.env.example", boundaries)
        self.assertIn("loopback", boundaries.lower())
        self.assertIn("8080", boundaries)
    def test_runbook_carries_no_literal_secret_values(self) -> None:
        findings = [match.group(0) for match in SECRET_ASSIGNMENT.finditer(text())]
        self.assertEqual(findings, [], f"runbook looks like it contains a real secret: {findings}")
        self.assertIn("mode-600", text())
        self.assertIn("never commit", text().lower())
    def test_runbook_claims_match_a_stateless_server(self) -> None:
        server = SERVER.read_text(encoding="utf-8")
        for call in ("writeFile", "appendFile", "createWriteStream", "mkdirSync"):
            self.assertNotIn(call, server, "server.mjs started persisting data; the runbook's data-lifetime claim must be updated")
        body = text().lower()
        self.assertIn("no server-side copy of the journal", body)
        self.assertIn("/api/healthz", body)
        # The health probe is allowed to report readiness, never configuration or secrets.
        health = re.search(r"'/api/healthz'[^\n]*", server)
        self.assertIsNotNone(health, "server.mjs must serve /api/healthz")
        health_text = health.group(0) if health else ""
        for forbidden in ("token", "secret", "env:", "password"):
            self.assertNotIn(forbidden, health_text.lower())
        # The response may report readiness only, so a new field is a deliberate decision.
        for field in ("ok", "release", "commit", "visionEnabled", "agentEnabled"):
            self.assertIn(field, health_text)
    def test_backup_tool_is_gated_by_its_own_suite(self) -> None:
        package = (ROOT / "package.json").read_text(encoding="utf-8")
        self.assertIn("tests/backup-restore.test.py", package)
        self.assertIn("tests/production-runbook.test.py", package)
        self.assertTrue(BACKUP.is_file())
 if __name__ == "__main__":
    unittest.main(verbosity=2)
@@ -486,3 +486,37 @@ test('release demo publishes the dataset card from a manifest it builds on camer
  assert.match(builder, /"dataset_card_generated": "passed"/);
  assert.match(builder, /scripts\/dataset_card\.py/);
 });
 test('release demo runs the operator backup and restore on synthetic records instead of describing it', async () => {
  const demo = await readFile(demoPath, 'utf8');
  const builder = await readFile(builderPath, 'utf8');
  // Issue #39: the backup is a CLI, so the recording runs the real tool and shows what it stored,
  // what it refused, and what the restore returned.
  assert.match(demo, /scripts\/operator_backup\.py/);
  assert.match(demo, /'backup', '--source'/);
  assert.match(demo, /'verify', '--archive'/);
  assert.match(demo, /'restore', '--archive'/);
  assert.match(demo, /id: 'production-backup'/);
  assert.match(demo, /Production backup card is not fully readable on screen/);
  // The exclusions are proven by the run, not asserted by a caption: the decoys are real files.
  assert.match(demo, /model-weights/);
  assert.match(demo, /transient-inference/);
  assert.match(demo, /must not archive the weight shard or a transient inference upload/);
  // A restore must return the append-only manifest byte for byte, tombstone included.
  assert.match(demo, /must return the same records, tombstone included/);
  assert.match(demo, /The operator backup is a command line, not a screen in the app/);
  assert.match(demo, /Model weights and transient inference images are refused by name, and every skipped path is listed with its reason/);
  assert.match(demo, /Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes/);
  assert.match(demo, /A restore returns the same records, including the withdrawal tombstone a trainer must never lose/);
  // The scene is a release gate, and the tool's own suite runs before the release exists.
  assert.match(demo, /process\.env\.TIMMY_DEMO_URL \|\| 'http:\/\/127\.0\.0\.1:4173'/);
  assert.match(builder, /tests\/backup-restore\.test\.py/);
  assert.match(builder, /tests\/production-runbook\.test\.py/);
  assert.match(builder, /"production_backup_restore": "passed"/);
  assert.match(builder, /"production_runbook_commands": "passed"/);
  assert.match(builder, /The operator backup is a command line, not a screen in the app/);
  assert.match(builder, /A restore returns the same records, including the withdrawal tombstone a trainer must never lose/);
 });