Operator backup, restore, and production runbook (#39) #90
No files matched your search
@@ -6,6 +6,7 @@ A working mobile-first bowel diary with an optional **photo-first AI assist**:
|
||||
- [Product and safety boundary](PRODUCT.md)
|
||||
- [Product boundary and release authority decision](docs/PRODUCT-DECISIONS.md)
|
||||
- [Self-hosted stool-vision decision](research/SELF-HOSTED-STOOL-VISION.md)
|
||||
- [Production topology, secrets, backup, and rollback runbook](docs/PRODUCTION-RUNBOOK.md)
|
||||
|
||||
## Build a gated review release
|
||||
|
||||
@@ -28,6 +29,25 @@ sudo python3 scripts/deploy_staging.py rollback --commit PRIOR_FULL_COMMIT
|
||||
|
||||
Commands are fixed argv arrays executed with `shell=False`; JSON argv overrides and `--root` exist for rootless fixtures. See [the staging runbook](docs/STAGING-RUNBOOK.md) for host prerequisites, mode-600 environment injection, Caddy validation, smoke credentials, logs, backups, removal, and the explicit no-live-change boundary.
|
||||
|
||||
## Back up and restore the records an operator holds
|
||||
|
||||
The app keeps the journal in the person's own browser, so the server has no diary to back up. What an
|
||||
operator can lose is the corpus: the append-only `manifest.jsonl` with its provenance records and its
|
||||
withdrawal tombstones, plus the derived images those records point at.
|
||||
|
||||
```bash
|
||||
python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy-nightly.tar.gz --label nightly
|
||||
python3 scripts/operator_backup.py verify --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json
|
||||
python3 scripts/operator_backup.py restore --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json --target /srv/timmy-corpus.restore-check
|
||||
```
|
||||
|
||||
Model weights, transient inference images, secrets, logs, and symlinks are refused by name, and every
|
||||
skipped path is listed in the manifest with its reason. Each stored member carries a SHA-256, the
|
||||
archive is reproducible, and a restore re-verifies every digest and the record counts before it writes
|
||||
anything — a tampered archive or a lost tombstone fails loudly instead of restoring quietly. See the
|
||||
[production runbook](docs/PRODUCTION-RUNBOOK.md) for topology, network boundaries, secrets, retention,
|
||||
and the rollback paths.
|
||||
|
||||
## Run with the self-hosted open-weight path
|
||||
|
||||
```bash
|
||||
|
||||
@@ -0,0 +1,172 @@
|
||||
# Production topology, secrets, backup, and rollback
|
||||
|
||||
This runbook answers one question for the operator: how do the parts of Timmy run, what may they
|
||||
reach, what must never be lost, and how does each part come back? It covers the pieces that exist in
|
||||
this tree today — the Node app, the private vision worker, the private staging directory, and the
|
||||
consented corpus. It does not describe a training fleet, a cloud account, or a hosted database,
|
||||
because this build has none.
|
||||
|
||||
Everything below is run by automation or by one operator on the host. Nothing here is performed by a
|
||||
participant, and no step needs a public endpoint.
|
||||
|
||||
## Topology
|
||||
|
||||
| Process | Entry point | Holds | Restart unit |
|
||||
|---|---|---|---|
|
||||
| App and API | `node server.mjs` on `HOST=127.0.0.1 PORT=4173` | Nothing durable. Serves the PWA, proxies `/api/vision/*`, and gates `/api/agent/*` behind an access code. | `timmy-staging.service` (staging), `node server.mjs` (local) |
|
||||
| Vision worker | `deploy/worker/run_worker_container.sh` | The pinned GGUF weights, mounted read-only from the host. | `scripts/run_selfhost_smolvlm.sh start`, container restart policy |
|
||||
| Staging releases | `/opt/timmy-staging/releases/<40-character-commit>` with a `current` symlink | Immutable release trees plus the mode-600 environment file. | `timmy-staging.service` |
|
||||
| Consented corpus | `/srv/timmy-corpus` | `manifest.jsonl` (append-only provenance and tombstones) and `images/` derived from consented contributions. | none — file data |
|
||||
| Caddy | host Caddy | Terminates TLS and routes `/git/` to Gitea and `/timmy-staging/` to the app. | host Caddy |
|
||||
|
||||
The app is stateless for a reason: there is **no server-side copy of the journal**. A person's logs
|
||||
and photos live in their own browser storage, and they hold the portable JSON backup. The server
|
||||
holds no diary, so a lost server loses no one's health record — only the operator records listed
|
||||
above can be lost, and only those are backed up here.
|
||||
|
||||
Health and readiness are separate, and the probe reports readiness only:
|
||||
|
||||
```
|
||||
GET /api/healthz -> {"ok":true,"release":"<tag>","commit":"<commit>","visionEnabled":<bool>,"agentEnabled":<bool>}
|
||||
```
|
||||
|
||||
`visionEnabled` and `agentEnabled` say whether the feature is configured, never which key or endpoint
|
||||
is configured. The worker answers `/v1/models` before its pinned model is resident, so readiness
|
||||
means "served a request from the loaded model", not "the port is open" — `scripts/worker_ready.sh`
|
||||
enforces that distinction and the app withholds the AI control until it passes.
|
||||
|
||||
## Network boundaries
|
||||
|
||||
- The app reads `HOST` and its own default is `0.0.0.0`, which is a local-development convenience and
|
||||
not the deployment. The deployment sets `HOST=127.0.0.1` in the mode-600 environment file
|
||||
(`deploy/timmy-staging.env.example` is the template) and is reachable from outside solely through
|
||||
the Caddy `/timmy-staging/` subpath. A bare `node server.mjs` on a shared host listens on every
|
||||
interface — treat that as a laptop command, never as a production start.
|
||||
- The Gitea `/git/` catch-all is declared before the staging route and must stay reachable across
|
||||
every Caddy reload.
|
||||
- The vision worker binds `127.0.0.1:8080` and is never published: `deploy/worker/run_worker_container.sh`
|
||||
refuses a non-loopback publish host instead of warning about it, and no route in `server.mjs`
|
||||
forwards arbitrary paths to the worker — the only vision endpoints are `/api/vision-status` and the
|
||||
analysis request.
|
||||
- Outbound from the host is limited to the pinned model download during bootstrap, which is refused
|
||||
unless both SHA-256 receipts match. After that the worker needs no egress at all.
|
||||
- Inference bodies and image bytes are never logged, and no reverse-proxy access log retains a
|
||||
request body. A backup that contained them would be a second copy of someone's photo, so they are
|
||||
excluded by the tool rather than by the operator remembering.
|
||||
- No participant-facing endpoint requires an inbound port, a DNS record, or a cloud credential.
|
||||
|
||||
## Secrets
|
||||
|
||||
Four secrets exist, and every one of them is injected at runtime:
|
||||
|
||||
| Secret | Source | Scope |
|
||||
|---|---|---|
|
||||
| `TIMMY_DATASET_SALT` | mode-600 environment file on the host | Corpus subject keys and split assignment. Rotating it re-splits future ingests, so it is not rotated casually. |
|
||||
| Vision provider key or worker token | mode-600 environment file on the host | Only the app process reads it; the browser never receives it. |
|
||||
| Agent access code | mode-600 environment file on the host | Unlocks `/api/agent/unlock`; the cookie it mints is `HttpOnly` and same-origin. |
|
||||
| Gitea access token | local operator shell or secret store | Issue and PR automation only. It is never written into a release tree, an archive, or a log. |
|
||||
|
||||
Rules:
|
||||
|
||||
- Never commit a secret, and never pass one on a command line that lands in shell history. Use the
|
||||
environment file (`chmod 600`, owned by the service user) or the systemd `EnvironmentFile=`.
|
||||
- Placeholders in this document are angle-bracketed or environment references, e.g.
|
||||
`TIMMY_DATASET_SALT=<random 32+ characters, never committed>`. A literal value here is a bug.
|
||||
- `scripts/build_release.py` scans the cloned tree for private-key blocks and assigned credentials
|
||||
and fails the release if it finds one, so a secret that reaches a commit is caught before it
|
||||
reaches a release.
|
||||
- Rotation is: issue the replacement on the host, restart the unit, confirm `/api/healthz` and one
|
||||
real inference request, then revoke the old value in the provider's own UI. Nothing in Git holds a
|
||||
copy that would need rewriting.
|
||||
- The backup tool refuses `.env`, `*.env`, `*.key`, `*.pem`, `id_rsa`, `tokens.json`, and
|
||||
`credentials.json` by name, so a backup can be copied off-host without carrying the keys that
|
||||
operate the host.
|
||||
|
||||
## Backup
|
||||
|
||||
The operator records that cannot be rebuilt are the corpus `manifest.jsonl` and the derived images
|
||||
it points at. A tombstone is the only proof that a participant withdrew, so a backup that drops
|
||||
tombstone lines silently re-authorises training on withdrawn data. The tool treats that as a
|
||||
correctness bug, not a storage detail.
|
||||
|
||||
`scripts/operator_backup.py` archives **only** records, and writes one manifest naming every byte it
|
||||
stored and every path it refused:
|
||||
|
||||
- Stored: `manifest.jsonl`, derived images, and any receipt files under the data root.
|
||||
- Excluded, with the reason recorded: **model weights** (`model-weights` — re-downloadable, pinned by
|
||||
hash), **transient inference** directories and uploads (`transient-inference` — a second copy of
|
||||
someone's photo), **secret** files (`secret`), logs (`log`), symlinks (`symlink`), and anything over
|
||||
`--max-bytes` (`oversize`).
|
||||
- Every stored member carries a SHA-256, and the archive is reproducible: sorted members, zeroed
|
||||
timestamps and ownership, `gzip` with a fixed mtime. Two backups of unchanged records are
|
||||
byte-identical, so "did anything change?" is answerable by digest.
|
||||
|
||||
```bash
|
||||
python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy-nightly.tar.gz --label nightly
|
||||
python3 scripts/operator_backup.py verify --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json
|
||||
```
|
||||
|
||||
The command writes `<archive>.manifest.json` unless `--manifest` names another path, and prints a
|
||||
receipt with the stored file count, archive digest, and record counts (`samples`, `tombstones`,
|
||||
`activeSamples`). It refuses a source that is not a directory and refuses to overwrite an existing
|
||||
archive, so a nightly run cannot erase the previous night.
|
||||
|
||||
Retention: keep fourteen nightlies and the first backup of each month, and copy each archive plus its
|
||||
manifest off-host. Also keep a human-readable record of the counts: `verify` compares the archive
|
||||
against the manifest, and a restore compares the restored counts against both.
|
||||
|
||||
What this backup deliberately does not cover: the browser-held journal (there is no server copy; the
|
||||
person's own portable export is the backup), model weights, and anything in the release tree, which
|
||||
is rebuilt from a commit by `scripts/build_release.py`.
|
||||
|
||||
## Restore
|
||||
|
||||
A restore never writes over live data. It verifies first, and it writes into a directory that must
|
||||
not exist or must be empty:
|
||||
|
||||
```bash
|
||||
python3 scripts/operator_backup.py restore --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json --target /srv/timmy-corpus.restore-check
|
||||
```
|
||||
|
||||
In order, before the first byte is written:
|
||||
|
||||
1. the manifest is loaded and its `schemaVersion` checked;
|
||||
2. every archive member is refused if it is a link, a non-regular file, an absolute path, a traversal
|
||||
path, or a name the manifest never listed;
|
||||
3. every member's SHA-256 is recomputed and compared, and a mismatch stops the restore;
|
||||
4. a member the manifest lists but the archive lacks stops the restore.
|
||||
|
||||
After the extract it recounts `manifest.jsonl` in the target and refuses if the counts differ from
|
||||
the backup, so a restore that silently lost a tombstone fails loudly. Then the operator compares the
|
||||
restored tree against the live one, and promotes it by moving the directories — the tool does not
|
||||
pretend to know which directory is authoritative.
|
||||
|
||||
Restoring the person's own logs is a different operation and stays where it belongs: the app's
|
||||
import control accepts the JSON file they exported, upgrades an older version and says which version
|
||||
it read, and refuses a file from a newer Timmy without touching the ledger it would have replaced.
|
||||
|
||||
## Rollback
|
||||
|
||||
Two rollbacks exist, and only one of them is a live staging action.
|
||||
|
||||
**Release rollback (staging).** Staging keeps one immutable directory per commit and a `current`
|
||||
symlink, so a rollback is a repoint, a restart, and a bounded health check that restores and verifies
|
||||
the previous release if the new one does not come up:
|
||||
|
||||
```bash
|
||||
python3 scripts/deploy_staging.py status
|
||||
python3 scripts/deploy_staging.py rollback --commit <FULL_COMMIT>
|
||||
```
|
||||
|
||||
The rollback path is exercised by `tests/staging-deploy.test.py`, which drives the real
|
||||
`scripts/deploy_staging.py` code against a rootless fixture (`--root`), including the case where the
|
||||
named commit has no immutable release directory, where the restart fails, and where the health probe
|
||||
never passes. That is the same code path the host runs; the difference is only the root.
|
||||
|
||||
The live drill on the private staging host — promote a build, serve it, roll it back, and confirm the
|
||||
participant-facing page again — is a **human gate**, tracked under #53. This loop does not run that
|
||||
drill automatically, because a rolled-back staging host is being used for release testing at the time.
|
||||
|
||||
**Data rollback (operator records).** The equivalent for the corpus is the restore above: verify,
|
||||
restore into a new directory, compare counts, then move it into place. A restore is never attempted in
|
||||
place, and a failed verify leaves the live directory untouched.
|
||||
+4
-2
@@ -18,7 +18,7 @@
|
||||
"test:grounded-chat": "node tests/grounded-chat.acceptance.mjs",
|
||||
"test:photo-lifecycle": "node tests/photo-lifecycle.acceptance.mjs",
|
||||
"test:training-contribution": "node tests/training-contribution.acceptance.mjs",
|
||||
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/review_corpus.py scripts/dataset_card.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh",
|
||||
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/review_corpus.py scripts/dataset_card.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py scripts/operator_backup.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh",
|
||||
"check:diff": "bash scripts/check_diff.sh",
|
||||
"start": "node server.mjs",
|
||||
"test:worker-readiness": "node tests/worker-readiness.acceptance.mjs",
|
||||
@@ -27,7 +27,9 @@
|
||||
"test:calibration": "python3 tests/calibration-harness.test.py",
|
||||
"test:policy": "node --test tests/abstention-policy.test.js",
|
||||
"test:review": "python3 tests/corpus-review.test.py",
|
||||
"test:dataset-card": "python3 tests/dataset-card.test.py"
|
||||
"test:dataset-card": "python3 tests/dataset-card.test.py",
|
||||
"test:backup": "python3 tests/backup-restore.test.py",
|
||||
"test:runbook": "python3 tests/production-runbook.test.py"
|
||||
},
|
||||
"devDependencies": {
|
||||
"playwright": "^1.62.1"
|
||||
|
||||
@@ -73,6 +73,10 @@ DEMO_REQUIRED_CAPTIONS = (
|
||||
"The model said Type 6 and the reviewer said Type 5: the disagreement is recorded, not smoothed over",
|
||||
"Withdrawing the contribution empties the queue and the training set, and the tool refuses to review it at all",
|
||||
"The dataset card is generated from the corpus manifest, so no count on it is typed by hand",
|
||||
"The operator backup is a command line, not a screen in the app",
|
||||
"Model weights and transient inference images are refused by name, and every skipped path is listed with its reason",
|
||||
"Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes",
|
||||
"A restore returns the same records, including the withdrawal tombstone a trainer must never lose",
|
||||
"One contributor's photos stay in one split: a corpus that leaks a person across splits is refused",
|
||||
"Devices, contexts, and licenses stay pending until the pipeline collects them",
|
||||
"The card is a page, not a screen in the app — it is published with the real generator against synthetic fixtures",
|
||||
@@ -183,6 +187,10 @@ def main() -> int:
|
||||
# Issue #29's evaluation harness decides GO/BLOCKED for a model, so its refusal and gate
|
||||
# arithmetic are exercised before this release is allowed to exist.
|
||||
run(["python3", "tests/calibration-harness.test.py", "-v"], tree)
|
||||
# Issue #39's operator backup is the only copy of the corpus tombstones, so the tool that
|
||||
# produces it, and the runbook whose commands must stay real, are gated before the release.
|
||||
run(["python3", "tests/backup-restore.test.py", "-v"], tree)
|
||||
run(["python3", "tests/production-runbook.test.py", "-v"], tree)
|
||||
server_env = dict(os.environ)
|
||||
server_env.update({"HOST": "127.0.0.1", "PORT": "4173", "TIMMY_VISION_ENABLED": "false"})
|
||||
# A leftover server from another checkout answers on this port and the readiness probe cannot
|
||||
@@ -360,7 +368,7 @@ def main() -> int:
|
||||
"codec": "h264",
|
||||
"pixel_format": "yuv420p",
|
||||
"recorded_from_working_app": True,
|
||||
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything, plus the review ledger card, where the real review commands run against the committed synthetic sample and the card shows their actual output: one pending proposal with the model answer withheld, a refused reveal before commitment, a committed Type 5 correction beside the Type 4 proposal, the revealed synthetic Type 6 answer recorded as a three-field disagreement, a second reader's adjudication pointing at the review it resolved, one trainable sample, and a withdrawal that empties the queue and the training set and makes review refused",
|
||||
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything, plus the review ledger card, where the real review commands run against the committed synthetic sample and the card shows their actual output: one pending proposal with the model answer withheld, a refused reveal before commitment, a committed Type 5 correction beside the Type 4 proposal, the revealed synthetic Type 6 answer recorded as a three-field disagreement, a second reader's adjudication pointing at the review it resolved, one trainable sample, and a withdrawal that empties the queue and the training set and makes review refused, plus the operator backup card, where the real backup, verify, and restore commands run against a synthetic corpus built by the real ingest and revoke commands and the card shows their actual output: two samples and one tombstone recorded, a weight shard, a transient inference upload, an operator env file and a log refused by name with the reason listed, the exclusion counts the manifest wrote, one digest for the archive, a verify that re-hashes every stored member, and a restore that returns the manifest byte-for-byte with the tombstone intact",
|
||||
},
|
||||
"gates": {
|
||||
"unit_security": "passed",
|
||||
@@ -381,6 +389,8 @@ def main() -> int:
|
||||
"reviewer_adjudication_blind": "passed",
|
||||
"dataset_card_generated": "passed",
|
||||
"calibration_harness_gates": "passed",
|
||||
"production_backup_restore": "passed",
|
||||
"production_runbook_commands": "passed",
|
||||
"portable_backup_restore": "passed",
|
||||
"dependency_audit_high": "passed",
|
||||
"syntax": "passed",
|
||||
|
||||
@@ -0,0 +1,326 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Operator backup and restore for the records Timmy must not lose.
|
||||
|
||||
The app keeps the journal in the person's own browser, so the server side of Timmy holds no diary.
|
||||
What the operator does hold, and what cannot be rebuilt from anywhere else, is the corpus: the
|
||||
append-only ``manifest.jsonl`` with its provenance records and its withdrawal tombstones, plus the
|
||||
derived images those records point at. Losing that file loses the proof that a participant
|
||||
withdrew, which is the one receipt the project has promised to keep.
|
||||
|
||||
This tool therefore:
|
||||
|
||||
- archives only records, and names every path it refused to archive with the reason
|
||||
(``model-weights``, ``transient-inference``, ``secret``, ``log``, ``symlink``, ``oversize``), so a
|
||||
missing file is an auditable decision instead of a silent omission;
|
||||
- writes one SHA-256 per archived member and a reproducible archive (sorted members, zeroed
|
||||
timestamps and ownership), so two backups of the same records are byte-identical;
|
||||
- verifies every digest, member name, and record count before a restore writes anything, and
|
||||
refuses absolute, traversal, link, and unlisted members.
|
||||
|
||||
Usage:
|
||||
|
||||
python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy.tar.gz
|
||||
python3 scripts/operator_backup.py verify --archive /srv/backups/timmy.tar.gz --manifest /srv/backups/timmy.tar.gz.manifest.json
|
||||
python3 scripts/operator_backup.py restore --archive ... --manifest ... --target /srv/timmy-corpus.restored
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gzip
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tarfile
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path, PurePosixPath
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
TOOL = "operator_backup.py"
|
||||
PAYLOAD_PREFIX = "payload/"
|
||||
DEFAULT_MAX_BYTES = 64 * 1024 * 1024
|
||||
|
||||
WEIGHT_SUFFIXES = (".gguf", ".safetensors", ".onnx", ".bin", ".pt", ".ckpt", ".pth")
|
||||
WEIGHT_DIRS = ("weights", "models")
|
||||
TRANSIENT_DIRS = ("transient", "inference-tmp", "inference-tmp-images", "uploads", "tmp", "cache")
|
||||
LOG_DIRS = ("logs",)
|
||||
LOG_SUFFIXES = (".log",)
|
||||
SECRET_NAMES = (".env", "id_rsa", "id_ed25519", "id_ecdsa", "tokens.json", "credentials.json")
|
||||
SECRET_SUFFIXES = (".key", ".pem", ".p12", ".pfx", ".env")
|
||||
|
||||
|
||||
class BackupError(Exception):
|
||||
"""A refusal that is reported on stderr and exits non-zero."""
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def classify(relative: PurePosixPath) -> str | None:
|
||||
"""Return the reason a path is not archived, or ``None`` when it is a record."""
|
||||
parts = relative.parts[:-1]
|
||||
name = relative.name
|
||||
lowered = [part.lower() for part in parts]
|
||||
suffix = PurePosixPath(name.lower()).suffix
|
||||
|
||||
if name.lower().startswith(".env") or name.lower() in SECRET_NAMES or suffix in SECRET_SUFFIXES:
|
||||
return "secret"
|
||||
if suffix in WEIGHT_SUFFIXES or any(part in WEIGHT_DIRS for part in lowered):
|
||||
return "model-weights"
|
||||
if any(part in TRANSIENT_DIRS for part in lowered):
|
||||
return "transient-inference"
|
||||
if any(part in LOG_DIRS for part in lowered) or suffix in LOG_SUFFIXES:
|
||||
return "log"
|
||||
return None
|
||||
|
||||
|
||||
def record_counts(manifest_path: Path) -> dict[str, int]:
|
||||
"""Count the append-only manifest the way a trainer would read it."""
|
||||
counts = {"samples": 0, "tombstones": 0, "activeSamples": 0}
|
||||
if not manifest_path.is_file():
|
||||
return counts
|
||||
revoked: set[str] = set()
|
||||
samples: list[dict] = []
|
||||
for line in manifest_path.read_text(encoding="utf-8").splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
entry = json.loads(line)
|
||||
except json.JSONDecodeError as error:
|
||||
raise BackupError(f"manifest {manifest_path} is not valid JSON Lines: {error}") from error
|
||||
kind = entry.get("recordType")
|
||||
if kind == "sample":
|
||||
samples.append(entry)
|
||||
counts["samples"] += 1
|
||||
elif kind == "tombstone":
|
||||
counts["tombstones"] += 1
|
||||
if entry.get("contributionId"):
|
||||
revoked.add(str(entry["contributionId"]))
|
||||
counts["activeSamples"] = sum(
|
||||
1 for entry in samples
|
||||
if str(entry.get("contributionId")) not in revoked
|
||||
and str((entry.get("lifecycle") or {}).get("state")) == "active"
|
||||
)
|
||||
return counts
|
||||
|
||||
|
||||
def collect(source: Path, max_bytes: int) -> tuple[list[tuple[PurePosixPath, Path]], list[dict]]:
|
||||
included: list[tuple[PurePosixPath, Path]] = []
|
||||
excluded: list[dict] = []
|
||||
for dirpath, dirnames, filenames in os.walk(source, followlinks=False):
|
||||
dirnames.sort()
|
||||
base = Path(dirpath)
|
||||
for name in sorted(filenames):
|
||||
path = base / name
|
||||
relative = PurePosixPath(path.relative_to(source).as_posix())
|
||||
if path.is_symlink():
|
||||
excluded.append({"path": str(relative), "reason": "symlink", "bytes": 0})
|
||||
continue
|
||||
reason = classify(relative)
|
||||
if reason:
|
||||
excluded.append({"path": str(relative), "reason": reason, "bytes": _size(path)})
|
||||
continue
|
||||
size = _size(path)
|
||||
if size > max_bytes:
|
||||
excluded.append({"path": str(relative), "reason": "oversize", "bytes": size})
|
||||
continue
|
||||
included.append((relative, path))
|
||||
included.sort(key=lambda item: str(item[0]))
|
||||
excluded.sort(key=lambda entry: entry["path"])
|
||||
return included, excluded
|
||||
|
||||
|
||||
def _size(path: Path) -> int:
|
||||
try:
|
||||
return path.stat().st_size
|
||||
except OSError:
|
||||
return 0
|
||||
|
||||
|
||||
def backup(*, source: Path, archive: Path, manifest_path: Path, label: str, max_bytes: int) -> dict:
|
||||
if not source.is_dir():
|
||||
raise BackupError(f"source {source} is not a directory")
|
||||
if archive.exists():
|
||||
raise BackupError(f"archive {archive} already exists; move it aside instead of overwriting a backup")
|
||||
|
||||
included, excluded = collect(source, max_bytes)
|
||||
files = [{"path": str(relative), "sha256": sha256_file(path), "bytes": _size(path)} for relative, path in included]
|
||||
counts = record_counts(source / "manifest.jsonl")
|
||||
|
||||
archive.parent.mkdir(parents=True, exist_ok=True)
|
||||
partial = archive.with_name(archive.name + ".partial")
|
||||
with partial.open("wb") as raw:
|
||||
# gzip mtime=0 and zeroed tar metadata keep two backups of the same records identical.
|
||||
with gzip.GzipFile(fileobj=raw, mode="wb", mtime=0) as compressed:
|
||||
with tarfile.open(fileobj=compressed, mode="w", format=tarfile.PAX_FORMAT) as tar:
|
||||
for (relative, path), entry in zip(included, files):
|
||||
payload = path.read_bytes()
|
||||
info = tarfile.TarInfo(PAYLOAD_PREFIX + str(relative))
|
||||
info.size = len(payload)
|
||||
info.mode = 0o644
|
||||
info.mtime = 0
|
||||
info.uid = info.gid = 0
|
||||
info.uname = info.gname = ""
|
||||
tar.addfile(info, io.BytesIO(payload))
|
||||
if hashlib.sha256(payload).hexdigest() != entry["sha256"]:
|
||||
raise BackupError(f"{relative} changed while it was being archived")
|
||||
os.replace(partial, archive)
|
||||
|
||||
manifest = {
|
||||
"schemaVersion": SCHEMA_VERSION,
|
||||
"tool": TOOL,
|
||||
"createdAt": datetime.now(timezone.utc).isoformat(timespec="seconds"),
|
||||
"label": label,
|
||||
"sourceLabel": source.name,
|
||||
"archive": {"name": archive.name, "sha256": sha256_file(archive), "bytes": _size(archive)},
|
||||
"records": counts,
|
||||
"files": files,
|
||||
"excluded": excluded,
|
||||
}
|
||||
manifest_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
return {
|
||||
"archive": str(archive),
|
||||
"manifest": str(manifest_path),
|
||||
"files": len(files),
|
||||
"bytes": manifest["archive"]["bytes"],
|
||||
"records": counts,
|
||||
"excluded": {entry["reason"]: sum(1 for other in excluded if other["reason"] == entry["reason"]) for entry in excluded},
|
||||
}
|
||||
|
||||
|
||||
def load_manifest(manifest_path: Path) -> dict:
|
||||
if not manifest_path.is_file():
|
||||
raise BackupError(f"manifest {manifest_path} does not exist")
|
||||
try:
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
except json.JSONDecodeError as error:
|
||||
raise BackupError(f"manifest {manifest_path} is not valid JSON: {error}") from error
|
||||
if manifest.get("schemaVersion") != SCHEMA_VERSION:
|
||||
raise BackupError(f"manifest schemaVersion {manifest.get('schemaVersion')!r} is not {SCHEMA_VERSION}")
|
||||
return manifest
|
||||
|
||||
|
||||
def verified_members(archive: Path, manifest: dict) -> list[tuple[str, bytes]]:
|
||||
"""Check every member of the archive before anything is written to disk."""
|
||||
if not archive.is_file():
|
||||
raise BackupError(f"archive {archive} does not exist")
|
||||
expected = {entry["path"]: entry for entry in manifest.get("files") or []}
|
||||
extracted: list[tuple[str, bytes]] = []
|
||||
seen: set[str] = set()
|
||||
with tarfile.open(archive, "r:gz") as tar:
|
||||
for info in tar.getmembers():
|
||||
if info.isdir():
|
||||
continue
|
||||
if info.issym() or info.islnk():
|
||||
raise BackupError(f"archive member {info.name!r} is a link, not a regular file")
|
||||
if not info.isfile():
|
||||
raise BackupError(f"archive member {info.name!r} is not a regular file")
|
||||
if os.path.isabs(info.name):
|
||||
raise BackupError(f"archive member {info.name!r} is an absolute path")
|
||||
if not info.name.startswith(PAYLOAD_PREFIX):
|
||||
raise BackupError(f"archive member {info.name!r} is not under {PAYLOAD_PREFIX}")
|
||||
relative = info.name[len(PAYLOAD_PREFIX):]
|
||||
parts = PurePosixPath(relative).parts
|
||||
if not parts or any(part in ("..", ".", "") for part in parts):
|
||||
raise BackupError(f"archive member {info.name!r} resolves outside the restore root")
|
||||
if relative not in expected:
|
||||
raise BackupError(f"archive member {info.name!r} is not listed in the manifest")
|
||||
handle = tar.extractfile(info)
|
||||
if handle is None:
|
||||
raise BackupError(f"archive member {info.name!r} cannot be read")
|
||||
payload = handle.read()
|
||||
if hashlib.sha256(payload).hexdigest() != expected[relative]["sha256"]:
|
||||
raise BackupError(
|
||||
f"archive member {info.name!r} does not match the manifest digest "
|
||||
"(the archive changed after it was recorded)"
|
||||
)
|
||||
seen.add(relative)
|
||||
extracted.append((relative, payload))
|
||||
missing = sorted(set(expected) - seen)
|
||||
if missing:
|
||||
raise BackupError(f"archive is missing members the manifest lists: {', '.join(missing)}")
|
||||
return extracted
|
||||
|
||||
|
||||
def verify(*, archive: Path, manifest_path: Path) -> dict:
|
||||
manifest = load_manifest(manifest_path)
|
||||
members_list = verified_members(archive, manifest)
|
||||
return {"archive": str(archive), "files": len(members_list), "records": manifest.get("records", {})}
|
||||
|
||||
|
||||
def restore(*, archive: Path, manifest_path: Path, target: Path) -> dict:
|
||||
manifest = load_manifest(manifest_path)
|
||||
if target.exists():
|
||||
if not target.is_dir():
|
||||
raise BackupError(f"restore target {target} is not a directory")
|
||||
if any(target.iterdir()):
|
||||
raise BackupError(f"restore target {target} is not empty; restore into a new directory")
|
||||
# Digests and member names are checked before the first byte is written, so a hostile or
|
||||
# tampered archive cannot leave a half-restored record set behind.
|
||||
members_list = verified_members(archive, manifest)
|
||||
target.mkdir(parents=True, exist_ok=True)
|
||||
for relative, payload in members_list:
|
||||
destination = target / relative
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
destination.write_bytes(payload)
|
||||
os.chmod(destination, 0o644)
|
||||
|
||||
counts = record_counts(target / "manifest.jsonl")
|
||||
if manifest.get("records") and counts != manifest["records"]:
|
||||
raise BackupError(
|
||||
f"restored record counts {counts} do not match the backup manifest {manifest['records']}"
|
||||
)
|
||||
return {"target": str(target), "files": len(members_list), "records": counts}
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
backup_parser = subparsers.add_parser("backup", help="archive the records and write a manifest")
|
||||
backup_parser.add_argument("--source", required=True, type=Path)
|
||||
backup_parser.add_argument("--archive", required=True, type=Path)
|
||||
backup_parser.add_argument("--manifest", type=Path)
|
||||
backup_parser.add_argument("--label", default="manual")
|
||||
backup_parser.add_argument("--max-bytes", type=int, default=DEFAULT_MAX_BYTES)
|
||||
|
||||
verify_parser = subparsers.add_parser("verify", help="recompute every digest in an archive")
|
||||
verify_parser.add_argument("--archive", required=True, type=Path)
|
||||
verify_parser.add_argument("--manifest", required=True, type=Path)
|
||||
|
||||
restore_parser = subparsers.add_parser("restore", help="verify, then write the records into a new directory")
|
||||
restore_parser.add_argument("--archive", required=True, type=Path)
|
||||
restore_parser.add_argument("--manifest", required=True, type=Path)
|
||||
restore_parser.add_argument("--target", required=True, type=Path)
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
if args.command == "backup":
|
||||
manifest_path = args.manifest or args.archive.with_name(args.archive.name + ".manifest.json")
|
||||
receipt = backup(
|
||||
source=args.source, archive=args.archive, manifest_path=manifest_path,
|
||||
label=args.label, max_bytes=args.max_bytes,
|
||||
)
|
||||
elif args.command == "verify":
|
||||
receipt = verify(archive=args.archive, manifest_path=args.manifest)
|
||||
else:
|
||||
receipt = restore(archive=args.archive, manifest_path=args.manifest, target=args.target)
|
||||
except BackupError as error:
|
||||
print(f"refused: {error}", file=sys.stderr)
|
||||
return 2
|
||||
print(json.dumps(receipt, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -17,6 +17,10 @@ const version = process.env.TIMMY_RELEASE_VERSION || 'review build';
|
||||
// changed in this release instead of implying a feature that did not ship.
|
||||
const releaseDelta = process.env.TIMMY_RELEASE_DEMO_DELTA
|
||||
|| 'Verified rebuild of the current user-visible feature path — no new feature is claimed.';
|
||||
// The recording targets the release builder's own acceptance server by default. A caller that
|
||||
// names another origin is explicit about it, so a local run can never silently record a different
|
||||
// tree than the one the gates measured.
|
||||
const demoUrl = process.env.TIMMY_DEMO_URL || 'http://127.0.0.1:4173';
|
||||
const captionsOut = process.env.TIMMY_DEMO_CAPTIONS_OUT || '';
|
||||
const captionsShown = [];
|
||||
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
@@ -116,7 +120,7 @@ await page.route('**/api/agent/chat', route => {
|
||||
});
|
||||
});
|
||||
|
||||
await page.goto('http://127.0.0.1:4173', { waitUntil: 'networkidle' });
|
||||
await page.goto(demoUrl, { waitUntil: 'networkidle' });
|
||||
await page.evaluate(() => localStorage.clear());
|
||||
await page.reload({ waitUntil: 'networkidle' });
|
||||
|
||||
@@ -1161,6 +1165,121 @@ await caption('Each candidate is scored on test at a threshold chosen on validat
|
||||
await caption('One contributor spread across two splits is refused before anything is scored, and a passing verdict is proven only by a fixture engineered to clear it', 3800);
|
||||
await page.evaluate(() => document.querySelector('#calibration-report')?.remove());
|
||||
|
||||
// Issue #39 — the operator backup is a command line, not a screen in the app. The app itself keeps
|
||||
// no server-side journal, so what an operator can actually lose is the corpus: the append-only
|
||||
// manifest with its withdrawal tombstones. This scene builds a synthetic data root with the real
|
||||
// ingest and revoke commands, plants the four things a backup must refuse, then runs the real
|
||||
// backup, verify, and restore and reads every value on the card back out of those runs.
|
||||
async function operatorBackupAudit() {
|
||||
const root = await mkdtemp(path.join(os.tmpdir(), 'timmy-backup-demo-'));
|
||||
const dataRoot = path.join(root, 'srv-timmy-corpus');
|
||||
const archive = path.join(root, 'timmy-nightly.tar.gz');
|
||||
const manifestPath = `${archive}.manifest.json`;
|
||||
const restoreInto = path.join(root, 'restored');
|
||||
const secondPhoto = path.join(root, 'synthetic-second.png');
|
||||
const backupCli = args => execFileAsync('python3', ['scripts/operator_backup.py', ...args], { cwd: process.cwd() });
|
||||
const ingest = (input, contributionId, subject, bristol) => corpusCli([
|
||||
'scripts/ingest_training_photo.py',
|
||||
'--input', input, '--output-dir', dataRoot, '--subject-id', subject,
|
||||
'--consent-version', 'training-consent-v1', '--contribution-id', contributionId,
|
||||
'--bristol-type', bristol, '--color', 'brown', '--quality', 'good', '--reviewer', 'user-confirmed',
|
||||
]);
|
||||
try {
|
||||
// One contribution stays active and one is withdrawn, so the manifest ends up with a live sample
|
||||
// and a real tombstone, and the withdrawn derived image is already gone from disk.
|
||||
await ingest('tests/fixtures/synthetic-type4.jpg', 'demo-backup-0001', 'demo-backup-kept', '4');
|
||||
await writeFile(secondPhoto, Buffer.from('iVBORw0KGgoAAAANSUhEUgAAAAgAAAAICAIAAABLbSncAAAAFElEQVR4nGOsCDFhwAaYsIoOWgkAzUwBECP1Kh0AAAAASUVORK5CYII=', 'base64'));
|
||||
await ingest(secondPhoto, 'demo-backup-0002', 'demo-backup-withdrawn', '3');
|
||||
await corpusCli([
|
||||
'scripts/ingest_training_photo.py', '--output-dir', dataRoot,
|
||||
'--revoke', 'demo-backup-0002', '--reason', 'participant withdrew consent',
|
||||
]);
|
||||
|
||||
// The four things that must never be archived, as real files with real bytes: a weight shard, a
|
||||
// transient inference upload, an operator environment file, and a log. An exclusion that quietly
|
||||
// stopped working fails here instead of in a caption.
|
||||
await mkdir(path.join(dataRoot, 'weights'), { recursive: true });
|
||||
await writeFile(path.join(dataRoot, 'weights', 'model.gguf'), 'synthetic weight shard — never archived');
|
||||
await mkdir(path.join(dataRoot, 'transient'), { recursive: true });
|
||||
await writeFile(path.join(dataRoot, 'transient', 'inference-upload.jpg'), 'transient inference bytes — never archived');
|
||||
await writeFile(path.join(dataRoot, 'secrets.env'), 'TIMMY_DATASET_SALT=demo-only-placeholder\n');
|
||||
await writeFile(path.join(dataRoot, 'operator.log'), 'GET /api/healthz 200\n');
|
||||
|
||||
const backupRun = await backupCli(['backup', '--source', dataRoot, '--archive', archive, '--label', 'nightly']);
|
||||
const receipt = JSON.parse(backupRun.stdout);
|
||||
const manifest = JSON.parse(await readFile(manifestPath, 'utf8'));
|
||||
const reasons = manifest.excluded.reduce((out, entry) => ({ ...out, [entry.reason]: (out[entry.reason] || 0) + 1 }), {});
|
||||
if (manifest.files.some(entry => /model\.gguf|inference-upload|secrets\.env|operator\.log/.test(entry.path))) {
|
||||
throw new Error('the backup must not archive the weight shard or a transient inference upload');
|
||||
}
|
||||
for (const reason of ['model-weights', 'transient-inference', 'secret', 'log']) {
|
||||
if (!reasons[reason]) throw new Error(`the backup must record a ${reason} exclusion`);
|
||||
}
|
||||
if (receipt.records.samples !== 2 || receipt.records.tombstones !== 1 || receipt.records.activeSamples !== 1) {
|
||||
throw new Error('the backup receipt must report the real record counts');
|
||||
}
|
||||
const verified = JSON.parse((await backupCli(['verify', '--archive', archive, '--manifest', manifestPath])).stdout);
|
||||
if (verified.files !== manifest.files.length) throw new Error('verify must re-hash every stored member');
|
||||
const restored = JSON.parse((await backupCli([
|
||||
'restore', '--archive', archive, '--manifest', manifestPath, '--target', restoreInto,
|
||||
])).stdout);
|
||||
if (restored.records.tombstones !== 1 || restored.records.activeSamples !== 1) {
|
||||
throw new Error('the restore must return the same record counts');
|
||||
}
|
||||
const restoredManifest = await readFile(path.join(restoreInto, 'manifest.jsonl'));
|
||||
if (!restoredManifest.equals(await readFile(path.join(dataRoot, 'manifest.jsonl')))) {
|
||||
throw new Error('the restore must return the same records, tombstone included');
|
||||
}
|
||||
return { receipt, manifest, reasons, verified, restored };
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
const opBackup = await operatorBackupAudit();
|
||||
await tap('[data-view="home"]', 400);
|
||||
const shortDigest = value => String(value).slice(0, 12);
|
||||
const backupRows = [
|
||||
['command', 'python3 scripts/operator_backup.py backup --source srv-timmy-corpus'],
|
||||
['stored', `${opBackup.receipt.files} members · ${opBackup.manifest.files.map(entry => entry.path.split('/').pop()).join(' + ')}`],
|
||||
['records', `samples ${opBackup.receipt.records.samples} · tombstones ${opBackup.receipt.records.tombstones} · active ${opBackup.receipt.records.activeSamples}`],
|
||||
['—', ''],
|
||||
['refused', `model-weights ${opBackup.reasons['model-weights']} · transient-inference ${opBackup.reasons['transient-inference']}`],
|
||||
['refused', `secret ${opBackup.reasons['secret']} · log ${opBackup.reasons['log']}`],
|
||||
['reason', 'every skipped path is named in the manifest, not silently dropped'],
|
||||
['—', ''],
|
||||
['digest', `sha256 ${shortDigest(opBackup.manifest.archive.sha256)}… · reproducible (sorted, mtime 0)`],
|
||||
['verify', `ok · ${opBackup.verified.files} members re-hashed against the manifest`],
|
||||
['restore', `ok · samples ${opBackup.restored.records.samples} · tombstones ${opBackup.restored.records.tombstones} · active ${opBackup.restored.records.activeSamples}`],
|
||||
['manifest', 'returned byte-for-byte, tombstone included'],
|
||||
['server', 'the app holds no journal of its own — there is no server copy to back up'],
|
||||
];
|
||||
const backupCard = await evidenceCard({
|
||||
id: 'production-backup',
|
||||
title: 'OPERATOR BACKUP',
|
||||
subtitle: 'command line, not app UI — real run, synthetic corpus',
|
||||
rows: backupRows,
|
||||
font: '600 9.5px/1.45 ui-monospace, SFMono-Regular, Menlo, monospace',
|
||||
keyWidth: 66,
|
||||
gap: '1px',
|
||||
rowGap: '6px',
|
||||
});
|
||||
await sleep(420);
|
||||
const backupProof = await evidenceCardProof('production-backup');
|
||||
if (!backupProof.ok) throw new Error(`Production backup card is not fully readable on screen: ${JSON.stringify(backupProof)}`);
|
||||
// The card is drawn from the run: prove the counts and the refusals on it are the ones it produced.
|
||||
if (!backupCard.text.includes(`samples ${opBackup.receipt.records.samples} · tombstones ${opBackup.receipt.records.tombstones}`)) {
|
||||
throw new Error('the card must show the record counts the backup actually reported');
|
||||
}
|
||||
if (!backupCard.text.includes('model-weights 1')) throw new Error('the card must show the exclusion the run actually recorded');
|
||||
if (!backupCard.text.includes(shortDigest(opBackup.manifest.archive.sha256))) {
|
||||
throw new Error('the card must show the archive digest the run actually wrote');
|
||||
}
|
||||
await caption('The operator backup is a command line, not a screen in the app — this runs it against a synthetic corpus the real ingest and revoke commands built', 3800);
|
||||
await caption('Model weights and transient inference images are refused by name, and every skipped path is listed with its reason', 3600);
|
||||
await caption('Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes', 3400);
|
||||
await caption('A restore returns the same records, including the withdrawal tombstone a trainer must never lose', 3600);
|
||||
await page.evaluate(() => document.querySelector('#production-backup')?.remove());
|
||||
|
||||
// Back to the privacy view for the portability scene: the privacy entry lives on the journal view.
|
||||
await tap('[data-view="calendar"]', 550);
|
||||
await tap('[data-view="privacy"]', 600);
|
||||
|
||||
@@ -0,0 +1,322 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for the operator backup and restore tool.
|
||||
|
||||
Issue #39 asks for a production topology and a backup that a person can trust with the records
|
||||
Timmy is not allowed to lose. The corpus manifest is append-only and the tombstones in it are the
|
||||
only proof that a withdrawal happened, so a backup that quietly drops them, or a restore that
|
||||
rebuilds a training set from withdrawn contributions, is worse than no backup at all.
|
||||
|
||||
The rules this suite locks in:
|
||||
|
||||
- model weights and transient inference images are never archived, and every excluded path is
|
||||
named in the manifest with the reason it was left out;
|
||||
- every archived byte carries a SHA-256 that ``verify`` recomputes before a restore writes
|
||||
anything, so a tampered archive fails closed;
|
||||
- hostile members (absolute paths, traversal, links, unlisted extras) are refused instead of
|
||||
extracted;
|
||||
- a restore round-trips the append-only manifest byte-for-byte and reports the same record counts,
|
||||
including the tombstones.
|
||||
|
||||
Every fixture is synthetic: the "images" are opaque bytes, no real medical image is read or written.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
BACKUP = ROOT / "scripts" / "operator_backup.py"
|
||||
|
||||
IMAGE_BYTES = b"\xff\xd8\xff\xe0synthetic-jpeg-payload-not-a-real-photo"
|
||||
WEIGHT_BYTES = b"GGUF-synthetic-weight-shard-not-a-real-model"
|
||||
TRANSIENT_BYTES = b"transient-inference-upload-must-not-be-archived"
|
||||
SECRET_BYTES = b"TIMMY_DATASET_SALT=demo\nTIMMY_VISION_TOKEN=do-not-archive-me\n"
|
||||
|
||||
|
||||
def digest(payload: bytes) -> str:
|
||||
return hashlib.sha256(payload).hexdigest()
|
||||
|
||||
|
||||
def run(args: list[str], env: dict[str, str] | None = None) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
[sys.executable, str(BACKUP), *args],
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env={**os.environ, **(env or {})},
|
||||
)
|
||||
|
||||
|
||||
def records(*lines: dict) -> str:
|
||||
return "".join(json.dumps(line) + "\n" for line in lines)
|
||||
|
||||
|
||||
def sample(contribution_id: str, *, relative_path: str = "images/sample-1.jpg") -> dict:
|
||||
return {
|
||||
"recordType": "sample",
|
||||
"schemaVersion": 2,
|
||||
"contributionId": contribution_id,
|
||||
"relativePath": relative_path,
|
||||
"imageId": digest(IMAGE_BYTES),
|
||||
"lifecycle": {"state": "active", "updatedAt": "2026-08-01T10:00:00+00:00"},
|
||||
}
|
||||
|
||||
|
||||
def tombstone(contribution_id: str) -> dict:
|
||||
return {
|
||||
"recordType": "tombstone",
|
||||
"schemaVersion": 2,
|
||||
"contributionId": contribution_id,
|
||||
"lifecycle": {"state": "revoked", "updatedAt": "2026-08-02T10:00:00+00:00"},
|
||||
"reason": "participant withdrew consent",
|
||||
"deletedArtifacts": [{"contributionId": contribution_id, "sha256": digest(IMAGE_BYTES)}],
|
||||
}
|
||||
|
||||
|
||||
def build_source(root: Path, *, with_tombstone: bool = False) -> Path:
|
||||
"""A synthetic server-side data root: the records that must survive, plus what must not."""
|
||||
source = root / "srv-timmy"
|
||||
(source / "images").mkdir(parents=True)
|
||||
(source / "images" / "sample-1.jpg").write_bytes(IMAGE_BYTES)
|
||||
(source / "manifest.jsonl").write_text(
|
||||
records(sample("contribution-1"), *([tombstone("contribution-2")] if with_tombstone else [])),
|
||||
encoding="utf-8",
|
||||
)
|
||||
# Must never be archived: weights, transient inference bytes, operator secrets, and logs.
|
||||
(source / "weights").mkdir()
|
||||
(source / "weights" / "model.gguf").write_bytes(WEIGHT_BYTES)
|
||||
(source / "transient").mkdir()
|
||||
(source / "transient" / "inference-upload-1.jpg").write_bytes(TRANSIENT_BYTES)
|
||||
(source / "secrets.env").write_bytes(SECRET_BYTES)
|
||||
(source / "operator.log").write_text("GET /api/vision 200\n", encoding="utf-8")
|
||||
return source
|
||||
|
||||
|
||||
def members(archive: Path) -> dict[str, bytes]:
|
||||
with tarfile.open(archive, "r:gz") as tar:
|
||||
out = {}
|
||||
for info in tar.getmembers():
|
||||
if not info.isfile():
|
||||
continue
|
||||
handle = tar.extractfile(info)
|
||||
assert handle is not None
|
||||
out[info.name] = handle.read()
|
||||
return out
|
||||
|
||||
|
||||
def craft_archive(path: Path, entries: list[tuple[str, bytes]], *, link: str | None = None) -> None:
|
||||
"""Build an archive by hand so the hostile cases are hostile on purpose."""
|
||||
with tarfile.open(path, "w:gz") as tar:
|
||||
for name, payload in entries:
|
||||
info = tarfile.TarInfo(name)
|
||||
info.size = len(payload)
|
||||
info.mode = 0o644
|
||||
tar.addfile(info, io.BytesIO(payload))
|
||||
if link:
|
||||
info = tarfile.TarInfo(link)
|
||||
info.type = tarfile.SYMTYPE
|
||||
info.linkname = "/etc/passwd"
|
||||
info.mode = 0o777
|
||||
tar.addfile(info)
|
||||
|
||||
|
||||
class BackupContractTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.tmp = Path(tempfile.mkdtemp(prefix="timmy-backup-test-"))
|
||||
self.addCleanup(shutil.rmtree, self.tmp, True)
|
||||
self.archive = self.tmp / "backup.tar.gz"
|
||||
self.manifest = self.tmp / "backup.manifest.json"
|
||||
|
||||
def backup(self, source: Path, *extra: str) -> subprocess.CompletedProcess:
|
||||
return run([
|
||||
"backup", "--source", str(source), "--archive", str(self.archive),
|
||||
"--manifest", str(self.manifest), "--label", "nightly", *extra,
|
||||
])
|
||||
|
||||
def test_backup_archives_records_and_excludes_weights_transient_images_secrets_and_logs(self) -> None:
|
||||
source = build_source(self.tmp, with_tombstone=True)
|
||||
result = self.backup(source)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
archived = members(self.archive)
|
||||
self.assertEqual(sorted(archived), ["payload/images/sample-1.jpg", "payload/manifest.jsonl"])
|
||||
self.assertEqual(archived["payload/images/sample-1.jpg"], IMAGE_BYTES)
|
||||
self.assertNotIn(WEIGHT_BYTES, b"".join(archived.values()))
|
||||
self.assertNotIn(TRANSIENT_BYTES, b"".join(archived.values()))
|
||||
self.assertNotIn(SECRET_BYTES, b"".join(archived.values()))
|
||||
|
||||
manifest = json.loads(self.manifest.read_text(encoding="utf-8"))
|
||||
excluded = {entry["path"]: entry["reason"] for entry in manifest["excluded"]}
|
||||
self.assertEqual(excluded["weights/model.gguf"], "model-weights")
|
||||
self.assertEqual(excluded["transient/inference-upload-1.jpg"], "transient-inference")
|
||||
self.assertEqual(excluded["secrets.env"], "secret")
|
||||
self.assertEqual(excluded["operator.log"], "log")
|
||||
# Every archived member is listed with the digest of the bytes that were stored.
|
||||
listed = {entry["path"]: entry["sha256"] for entry in manifest["files"]}
|
||||
self.assertEqual(listed, {
|
||||
"images/sample-1.jpg": digest(IMAGE_BYTES),
|
||||
"manifest.jsonl": digest((source / "manifest.jsonl").read_bytes()),
|
||||
})
|
||||
|
||||
def test_backup_reports_record_counts_including_tombstones(self) -> None:
|
||||
source = build_source(self.tmp, with_tombstone=True)
|
||||
result = self.backup(source)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
receipt = json.loads(result.stdout)
|
||||
self.assertEqual(receipt["records"], {"samples": 1, "tombstones": 1, "activeSamples": 1})
|
||||
manifest = json.loads(self.manifest.read_text(encoding="utf-8"))
|
||||
self.assertEqual(manifest["records"], {"samples": 1, "tombstones": 1, "activeSamples": 1})
|
||||
self.assertEqual(manifest["label"], "nightly")
|
||||
|
||||
def test_backup_refuses_to_follow_a_symlink_out_of_the_data_root(self) -> None:
|
||||
source = build_source(self.tmp)
|
||||
(source / "images" / "escape.jpg").symlink_to("/etc/hostname")
|
||||
result = self.backup(source)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertNotIn("payload/images/escape.jpg", members(self.archive))
|
||||
manifest = json.loads(self.manifest.read_text(encoding="utf-8"))
|
||||
excluded = {entry["path"]: entry["reason"] for entry in manifest["excluded"]}
|
||||
self.assertEqual(excluded["images/escape.jpg"], "symlink")
|
||||
|
||||
def test_backup_refuses_missing_source_and_existing_archive(self) -> None:
|
||||
missing = run(["backup", "--source", str(self.tmp / "nope"), "--archive", str(self.archive)])
|
||||
self.assertNotEqual(missing.returncode, 0)
|
||||
self.assertIn("is not a directory", missing.stderr)
|
||||
|
||||
source = build_source(self.tmp)
|
||||
self.assertEqual(self.backup(source).returncode, 0)
|
||||
again = self.backup(source)
|
||||
self.assertNotEqual(again.returncode, 0)
|
||||
self.assertIn("already exists", again.stderr)
|
||||
|
||||
def test_verify_and_restore_round_trip_the_records(self) -> None:
|
||||
source = build_source(self.tmp, with_tombstone=True)
|
||||
self.assertEqual(self.backup(source).returncode, 0)
|
||||
verified = run(["verify", "--archive", str(self.archive), "--manifest", str(self.manifest)])
|
||||
self.assertEqual(verified.returncode, 0, verified.stderr)
|
||||
|
||||
target = self.tmp / "restored"
|
||||
restored = run([
|
||||
"restore", "--archive", str(self.archive), "--manifest", str(self.manifest),
|
||||
"--target", str(target),
|
||||
])
|
||||
self.assertEqual(restored.returncode, 0, restored.stderr)
|
||||
receipt = json.loads(restored.stdout)
|
||||
self.assertEqual(receipt["records"], {"samples": 1, "tombstones": 1, "activeSamples": 1})
|
||||
self.assertEqual((target / "manifest.jsonl").read_bytes(), (source / "manifest.jsonl").read_bytes())
|
||||
self.assertEqual((target / "images" / "sample-1.jpg").read_bytes(), IMAGE_BYTES)
|
||||
# The manifest is append-only history: the tombstone line survives the round trip verbatim.
|
||||
lines = [json.loads(line) for line in (target / "manifest.jsonl").read_text().splitlines() if line.strip()]
|
||||
self.assertEqual([line["recordType"] for line in lines], ["sample", "tombstone"])
|
||||
|
||||
def test_verify_refuses_tampered_member_before_a_restore_writes_anything(self) -> None:
|
||||
source = build_source(self.tmp)
|
||||
self.assertEqual(self.backup(source).returncode, 0)
|
||||
manifest = json.loads(self.manifest.read_text(encoding="utf-8"))
|
||||
tampered = b"\xff\xd8\xff\xe0synthetic-jpeg-payload-not-a-real-photO"
|
||||
craft_archive(self.archive, [
|
||||
("payload/images/sample-1.jpg", tampered),
|
||||
("payload/manifest.jsonl", (source / "manifest.jsonl").read_bytes()),
|
||||
])
|
||||
verified = run(["verify", "--archive", str(self.archive), "--manifest", str(self.manifest)])
|
||||
self.assertNotEqual(verified.returncode, 0)
|
||||
self.assertIn("does not match", verified.stderr)
|
||||
self.assertEqual(manifest["files"][0]["sha256"], digest(IMAGE_BYTES))
|
||||
|
||||
target = self.tmp / "restored-tampered"
|
||||
restored = run([
|
||||
"restore", "--archive", str(self.archive), "--manifest", str(self.manifest),
|
||||
"--target", str(target),
|
||||
])
|
||||
self.assertNotEqual(restored.returncode, 0)
|
||||
self.assertFalse(target.exists())
|
||||
|
||||
def test_verify_refuses_a_member_the_manifest_never_listed(self) -> None:
|
||||
source = build_source(self.tmp)
|
||||
self.assertEqual(self.backup(source).returncode, 0)
|
||||
craft_archive(self.archive, [
|
||||
("payload/images/sample-1.jpg", IMAGE_BYTES),
|
||||
("payload/manifest.jsonl", (source / "manifest.jsonl").read_bytes()),
|
||||
("payload/extra.bin", b"unlisted"),
|
||||
])
|
||||
verified = run(["verify", "--archive", str(self.archive), "--manifest", str(self.manifest)])
|
||||
self.assertNotEqual(verified.returncode, 0)
|
||||
self.assertIn("not listed in the manifest", verified.stderr)
|
||||
|
||||
def test_restore_refuses_traversal_and_absolute_members(self) -> None:
|
||||
for name, needle in (("payload/../../etc/passwd", "outside"), ("/etc/passwd", "absolute")):
|
||||
with self.subTest(member=name):
|
||||
archive = self.tmp / "hostile.tar.gz"
|
||||
manifest_path = self.tmp / "hostile.manifest.json"
|
||||
payload = b"root:x:0:0"
|
||||
craft_archive(archive, [(name, payload)])
|
||||
manifest_path.write_text(json.dumps({
|
||||
"schemaVersion": 1,
|
||||
"files": [{"path": "images/sample-1.jpg", "sha256": digest(payload), "bytes": len(payload)}],
|
||||
"records": {"samples": 0, "tombstones": 0, "activeSamples": 0},
|
||||
}), encoding="utf-8")
|
||||
target = self.tmp / f"hostile-out-{needle}"
|
||||
restored = run([
|
||||
"restore", "--archive", str(archive), "--manifest", str(manifest_path),
|
||||
"--target", str(target),
|
||||
])
|
||||
self.assertNotEqual(restored.returncode, 0)
|
||||
self.assertIn(needle, restored.stderr.lower())
|
||||
self.assertFalse(target.exists())
|
||||
|
||||
def test_restore_refuses_link_members(self) -> None:
|
||||
archive = self.tmp / "link.tar.gz"
|
||||
manifest_path = self.tmp / "link.manifest.json"
|
||||
craft_archive(archive, [("payload/manifest.jsonl", b"{}\n")], link="payload/images/sample-1.jpg")
|
||||
manifest_path.write_text(json.dumps({
|
||||
"schemaVersion": 1,
|
||||
"files": [{"path": "manifest.jsonl", "sha256": digest(b"{}\n"), "bytes": 3}],
|
||||
"records": {"samples": 0, "tombstones": 0, "activeSamples": 0},
|
||||
}), encoding="utf-8")
|
||||
target = self.tmp / "link-out"
|
||||
restored = run([
|
||||
"restore", "--archive", str(archive), "--manifest", str(manifest_path), "--target", str(target),
|
||||
])
|
||||
self.assertNotEqual(restored.returncode, 0)
|
||||
self.assertIn("link", restored.stderr)
|
||||
self.assertFalse(target.exists())
|
||||
|
||||
def test_restore_refuses_a_non_empty_target(self) -> None:
|
||||
source = build_source(self.tmp)
|
||||
self.assertEqual(self.backup(source).returncode, 0)
|
||||
target = self.tmp / "occupied"
|
||||
target.mkdir()
|
||||
(target / "keep.txt").write_text("operator file that must not be overwritten", encoding="utf-8")
|
||||
restored = run([
|
||||
"restore", "--archive", str(self.archive), "--manifest", str(self.manifest),
|
||||
"--target", str(target),
|
||||
])
|
||||
self.assertNotEqual(restored.returncode, 0)
|
||||
self.assertIn("not empty", restored.stderr)
|
||||
self.assertEqual((target / "keep.txt").read_text(encoding="utf-8"), "operator file that must not be overwritten")
|
||||
|
||||
def test_backup_is_reproducible_for_the_same_records(self) -> None:
|
||||
source = build_source(self.tmp, with_tombstone=True)
|
||||
self.assertEqual(self.backup(source).returncode, 0)
|
||||
first = json.loads(self.manifest.read_text(encoding="utf-8"))
|
||||
first_digest = hashlib.sha256(self.archive.read_bytes()).hexdigest()
|
||||
self.archive.unlink()
|
||||
self.manifest.unlink()
|
||||
self.assertEqual(self.backup(source).returncode, 0)
|
||||
second = json.loads(self.manifest.read_text(encoding="utf-8"))
|
||||
self.assertEqual(first["files"], second["files"])
|
||||
self.assertEqual(first["excluded"], second["excluded"])
|
||||
self.assertEqual(first_digest, hashlib.sha256(self.archive.read_bytes()).hexdigest())
|
||||
self.assertNotEqual(first["createdAt"], "")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
@@ -0,0 +1,161 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for the production topology, secrets, backup, and rollback runbook.
|
||||
|
||||
A runbook is only worth reading if its commands are the ones the tools actually accept, so this
|
||||
suite parses every documented ``python3 scripts/...`` command out of the markdown and checks each
|
||||
flag against the real CLI help. It also fails if the document starts carrying a real secret, or if
|
||||
it keeps claiming the server holds no journal after ``server.mjs`` starts writing one.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
RUNBOOK = ROOT / "docs" / "PRODUCTION-RUNBOOK.md"
|
||||
SERVER = ROOT / "server.mjs"
|
||||
BACKUP = ROOT / "scripts" / "operator_backup.py"
|
||||
|
||||
FENCE = re.compile(r"```bash\n(.*?)```", re.S)
|
||||
SECRET_ASSIGNMENT = re.compile(
|
||||
r"(?i)(?:token|secret|password|passwd|api[_-]?key|salt)\s*=\s*(?!<|\$|\"|\'|\.\.\.)[^\s`\"']{6,}"
|
||||
)
|
||||
|
||||
|
||||
def text() -> str:
|
||||
return RUNBOOK.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def sections() -> dict[str, str]:
|
||||
body = text()
|
||||
headings = list(re.finditer(r"^## (.+)$", body, re.M))
|
||||
out: dict[str, str] = {}
|
||||
for index, match in enumerate(headings):
|
||||
end = headings[index + 1].start() if index + 1 < len(headings) else len(body)
|
||||
out[match.group(1).strip().lower()] = body[match.end():end]
|
||||
return out
|
||||
|
||||
|
||||
def documented_commands() -> list[str]:
|
||||
commands: list[str] = []
|
||||
for block in FENCE.findall(text()):
|
||||
for line in block.splitlines():
|
||||
line = line.strip()
|
||||
if line.startswith("python3 scripts/"):
|
||||
commands.append(line)
|
||||
return commands
|
||||
|
||||
|
||||
def help_text(script: str, subcommand: str | None) -> str:
|
||||
argv = [sys.executable, script] + ([subcommand] if subcommand else []) + ["--help"]
|
||||
result = subprocess.run(argv, cwd=ROOT, capture_output=True, text=True)
|
||||
assert result.returncode == 0, f"{' '.join(argv)} failed: {result.stderr}"
|
||||
return result.stdout
|
||||
|
||||
|
||||
class ProductionRunbookTests(unittest.TestCase):
|
||||
def test_runbook_covers_topology_network_boundaries_secrets_backup_restore_and_rollback(self) -> None:
|
||||
self.assertTrue(RUNBOOK.is_file(), "docs/PRODUCTION-RUNBOOK.md must exist")
|
||||
titles = sections().keys()
|
||||
for required in ("topology", "network boundaries", "secrets", "backup", "restore", "rollback"):
|
||||
self.assertTrue(
|
||||
any(title.startswith(required) for title in titles),
|
||||
f"runbook is missing a '{required}' section; found {sorted(titles)}",
|
||||
)
|
||||
|
||||
def test_backup_section_names_what_is_never_archived_and_why(self) -> None:
|
||||
backup = next(body for title, body in sections().items() if title.startswith("backup"))
|
||||
self.assertIn("scripts/operator_backup.py", backup)
|
||||
for required in ("model weights", "transient inference", "secret", "tombstone"):
|
||||
self.assertIn(required, backup.lower(), f"backup section must name {required}")
|
||||
self.assertRegex(backup, r"manifest\.jsonl")
|
||||
|
||||
def test_every_documented_script_command_is_accepted_by_the_real_cli(self) -> None:
|
||||
commands = documented_commands()
|
||||
self.assertGreaterEqual(len(commands), 4, f"expected documented commands, found {commands}")
|
||||
for command in commands:
|
||||
with self.subTest(command=command):
|
||||
tokens = shlex.split(command)
|
||||
script = tokens[1]
|
||||
rest = tokens[2:]
|
||||
subcommand = rest[0] if rest and not rest[0].startswith("-") else None
|
||||
combined = help_text(script, subcommand) + help_text(script, None)
|
||||
for token in rest:
|
||||
if token.startswith("--"):
|
||||
self.assertIn(token, combined, f"{script} does not accept {token}")
|
||||
for token in rest[1:]:
|
||||
if not token.startswith("--"):
|
||||
continue
|
||||
|
||||
def test_documented_backup_commands_are_the_ones_the_tool_runs(self) -> None:
|
||||
commands = " ".join(documented_commands())
|
||||
for flag in ("--source", "--archive", "--manifest", "--target"):
|
||||
self.assertIn(flag, commands, f"the runbook must show the real {flag} flag")
|
||||
self.assertIn("verify", commands)
|
||||
self.assertIn("restore", commands)
|
||||
|
||||
def test_rollback_is_documented_as_a_tested_fixture_path_and_a_human_gated_live_drill(self) -> None:
|
||||
rollback = next(body for title, body in sections().items() if title.startswith("rollback"))
|
||||
self.assertIn("scripts/deploy_staging.py rollback", rollback)
|
||||
self.assertRegex(rollback.lower(), r"human[- ]gate")
|
||||
self.assertIn("#53", rollback)
|
||||
self.assertIn("tests/staging-deploy.test.py", rollback)
|
||||
# The live drill must be explicit about not being run by automation (wrapped lines are fine).
|
||||
self.assertRegex(
|
||||
rollback.lower(),
|
||||
r"(?s)not .{0,80}automatically|does not run .{0,80}automatically",
|
||||
"the runbook must say the live drill is not run automatically",
|
||||
)
|
||||
|
||||
def test_network_boundaries_match_how_the_server_and_worker_actually_bind(self) -> None:
|
||||
server = SERVER.read_text(encoding="utf-8")
|
||||
# The app's own default is not loopback, so the runbook must say the deployment sets it.
|
||||
self.assertRegex(server, r"HOST\|\|'0\.0\.0\.0'")
|
||||
staging_env = (ROOT / "deploy" / "timmy-staging.env.example").read_text(encoding="utf-8")
|
||||
self.assertIn("HOST=127.0.0.1", staging_env)
|
||||
worker = (ROOT / "deploy" / "worker" / "run_worker_container.sh").read_text(encoding="utf-8")
|
||||
self.assertIn("Refusing to publish the worker outside loopback", worker)
|
||||
|
||||
boundaries = next(body for title, body in sections().items() if title.startswith("network boundaries"))
|
||||
self.assertIn("0.0.0.0", boundaries, "the runbook must name the app's real default bind address")
|
||||
self.assertIn("HOST=127.0.0.1", boundaries)
|
||||
self.assertIn("deploy/timmy-staging.env.example", boundaries)
|
||||
self.assertIn("loopback", boundaries.lower())
|
||||
self.assertIn("8080", boundaries)
|
||||
|
||||
def test_runbook_carries_no_literal_secret_values(self) -> None:
|
||||
findings = [match.group(0) for match in SECRET_ASSIGNMENT.finditer(text())]
|
||||
self.assertEqual(findings, [], f"runbook looks like it contains a real secret: {findings}")
|
||||
self.assertIn("mode-600", text())
|
||||
self.assertIn("never commit", text().lower())
|
||||
|
||||
def test_runbook_claims_match_a_stateless_server(self) -> None:
|
||||
server = SERVER.read_text(encoding="utf-8")
|
||||
for call in ("writeFile", "appendFile", "createWriteStream", "mkdirSync"):
|
||||
self.assertNotIn(call, server, "server.mjs started persisting data; the runbook's data-lifetime claim must be updated")
|
||||
body = text().lower()
|
||||
self.assertIn("no server-side copy of the journal", body)
|
||||
self.assertIn("/api/healthz", body)
|
||||
# The health probe is allowed to report readiness, never configuration or secrets.
|
||||
health = re.search(r"'/api/healthz'[^\n]*", server)
|
||||
self.assertIsNotNone(health, "server.mjs must serve /api/healthz")
|
||||
health_text = health.group(0) if health else ""
|
||||
for forbidden in ("token", "secret", "env:", "password"):
|
||||
self.assertNotIn(forbidden, health_text.lower())
|
||||
# The response may report readiness only, so a new field is a deliberate decision.
|
||||
for field in ("ok", "release", "commit", "visionEnabled", "agentEnabled"):
|
||||
self.assertIn(field, health_text)
|
||||
|
||||
def test_backup_tool_is_gated_by_its_own_suite(self) -> None:
|
||||
package = (ROOT / "package.json").read_text(encoding="utf-8")
|
||||
self.assertIn("tests/backup-restore.test.py", package)
|
||||
self.assertIn("tests/production-runbook.test.py", package)
|
||||
self.assertTrue(BACKUP.is_file())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
@@ -486,3 +486,37 @@ test('release demo publishes the dataset card from a manifest it builds on camer
|
||||
assert.match(builder, /"dataset_card_generated": "passed"/);
|
||||
assert.match(builder, /scripts\/dataset_card\.py/);
|
||||
});
|
||||
|
||||
test('release demo runs the operator backup and restore on synthetic records instead of describing it', async () => {
|
||||
const demo = await readFile(demoPath, 'utf8');
|
||||
const builder = await readFile(builderPath, 'utf8');
|
||||
|
||||
// Issue #39: the backup is a CLI, so the recording runs the real tool and shows what it stored,
|
||||
// what it refused, and what the restore returned.
|
||||
assert.match(demo, /scripts\/operator_backup\.py/);
|
||||
assert.match(demo, /'backup', '--source'/);
|
||||
assert.match(demo, /'verify', '--archive'/);
|
||||
assert.match(demo, /'restore', '--archive'/);
|
||||
assert.match(demo, /id: 'production-backup'/);
|
||||
assert.match(demo, /Production backup card is not fully readable on screen/);
|
||||
// The exclusions are proven by the run, not asserted by a caption: the decoys are real files.
|
||||
assert.match(demo, /model-weights/);
|
||||
assert.match(demo, /transient-inference/);
|
||||
assert.match(demo, /must not archive the weight shard or a transient inference upload/);
|
||||
// A restore must return the append-only manifest byte for byte, tombstone included.
|
||||
assert.match(demo, /must return the same records, tombstone included/);
|
||||
|
||||
assert.match(demo, /The operator backup is a command line, not a screen in the app/);
|
||||
assert.match(demo, /Model weights and transient inference images are refused by name, and every skipped path is listed with its reason/);
|
||||
assert.match(demo, /Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes/);
|
||||
assert.match(demo, /A restore returns the same records, including the withdrawal tombstone a trainer must never lose/);
|
||||
|
||||
// The scene is a release gate, and the tool's own suite runs before the release exists.
|
||||
assert.match(demo, /process\.env\.TIMMY_DEMO_URL \|\| 'http:\/\/127\.0\.0\.1:4173'/);
|
||||
assert.match(builder, /tests\/backup-restore\.test\.py/);
|
||||
assert.match(builder, /tests\/production-runbook\.test\.py/);
|
||||
assert.match(builder, /"production_backup_restore": "passed"/);
|
||||
assert.match(builder, /"production_runbook_commands": "passed"/);
|
||||
assert.match(builder, /The operator backup is a command line, not a screen in the app/);
|
||||
assert.match(builder, /A restore returns the same records, including the withdrawal tombstone a trainer must never lose/);
|
||||
});
|
||||
Reference in new issue
Block a user