feat(ops): operator backup, restore, and production runbook (#39)
Quality gates / quality (pull_request) Canceled after 0s

The app keeps the journal in the person's browser, so the server has no diary to lose. What an
operator can lose is the corpus: the append-only manifest.jsonl with its provenance records and its
withdrawal tombstones. scripts/operator_backup.py archives only records and names every path it
refuses (model weights, transient inference images, secrets, logs, symlinks, oversize) with the
reason, writes one SHA-256 per stored member, and produces a reproducible archive.

restore re-verifies every digest, member name, and record count before it writes a byte, and refuses
absolute, traversal, link, unlisted, and non-empty-target cases, so a tampered archive or a lost
tombstone fails loudly. docs/PRODUCTION-RUNBOOK.md covers topology, network boundaries (including
the app's real 0.0.0.0 default and the staging HOST=127.0.0.1 template), the four secrets, retention,
and both rollback paths; its documented commands are parsed against the real CLIs by its test.

The release demo runs the real backup, verify, and restore against a synthetic corpus built by the
real ingest and revoke commands and shows what was stored, what was refused, and what came back.
This commit is contained in:
Timmy committed 2026-10-29 11:02:59 +00:00
1 parent 7fe1901703
commit 48f0f56835
9 files changed
+1170 -4

No files matched your search

+11 -1
View File
@@ -73,6 +73,10 @@ DEMO_REQUIRED_CAPTIONS = (
"The model said Type 6 and the reviewer said Type 5: the disagreement is recorded, not smoothed over",
"Withdrawing the contribution empties the queue and the training set, and the tool refuses to review it at all",
"The dataset card is generated from the corpus manifest, so no count on it is typed by hand",
"The operator backup is a command line, not a screen in the app",
"Model weights and transient inference images are refused by name, and every skipped path is listed with its reason",
"Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes",
"A restore returns the same records, including the withdrawal tombstone a trainer must never lose",
"One contributor's photos stay in one split: a corpus that leaks a person across splits is refused",
"Devices, contexts, and licenses stay pending until the pipeline collects them",
"The card is a page, not a screen in the app — it is published with the real generator against synthetic fixtures",
@@ -183,6 +187,10 @@ def main() -> int:
# Issue #29's evaluation harness decides GO/BLOCKED for a model, so its refusal and gate
# arithmetic are exercised before this release is allowed to exist.
run(["python3", "tests/calibration-harness.test.py", "-v"], tree)
# Issue #39's operator backup is the only copy of the corpus tombstones, so the tool that
# produces it, and the runbook whose commands must stay real, are gated before the release.
run(["python3", "tests/backup-restore.test.py", "-v"], tree)
run(["python3", "tests/production-runbook.test.py", "-v"], tree)
server_env = dict(os.environ)
server_env.update({"HOST": "127.0.0.1", "PORT": "4173", "TIMMY_VISION_ENABLED": "false"})
# A leftover server from another checkout answers on this port and the readiness probe cannot
@@ -360,7 +368,7 @@ def main() -> int:
"codec": "h264",
"pixel_format": "yuv420p",
"recorded_from_working_app": True,
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything, plus the review ledger card, where the real review commands run against the committed synthetic sample and the card shows their actual output: one pending proposal with the model answer withheld, a refused reveal before commitment, a committed Type 5 correction beside the Type 4 proposal, the revealed synthetic Type 6 answer recorded as a three-field disagreement, a second reader's adjudication pointing at the review it resolved, one trainable sample, and a withdrawal that empties the queue and the training set and makes review refused",
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything, plus the review ledger card, where the real review commands run against the committed synthetic sample and the card shows their actual output: one pending proposal with the model answer withheld, a refused reveal before commitment, a committed Type 5 correction beside the Type 4 proposal, the revealed synthetic Type 6 answer recorded as a three-field disagreement, a second reader's adjudication pointing at the review it resolved, one trainable sample, and a withdrawal that empties the queue and the training set and makes review refused, plus the operator backup card, where the real backup, verify, and restore commands run against a synthetic corpus built by the real ingest and revoke commands and the card shows their actual output: two samples and one tombstone recorded, a weight shard, a transient inference upload, an operator env file and a log refused by name with the reason listed, the exclusion counts the manifest wrote, one digest for the archive, a verify that re-hashes every stored member, and a restore that returns the manifest byte-for-byte with the tombstone intact",
},
"gates": {
"unit_security": "passed",
@@ -381,6 +389,8 @@ def main() -> int:
"reviewer_adjudication_blind": "passed",
"dataset_card_generated": "passed",
"calibration_harness_gates": "passed",
"production_backup_restore": "passed",
"production_runbook_commands": "passed",
"portable_backup_restore": "passed",
"dependency_audit_high": "passed",
"syntax": "passed",
+326
View File
@@ -0,0 +1,326 @@
#!/usr/bin/env python3
"""Operator backup and restore for the records Timmy must not lose.
The app keeps the journal in the person's own browser, so the server side of Timmy holds no diary.
What the operator does hold, and what cannot be rebuilt from anywhere else, is the corpus: the
append-only ``manifest.jsonl`` with its provenance records and its withdrawal tombstones, plus the
derived images those records point at. Losing that file loses the proof that a participant
withdrew, which is the one receipt the project has promised to keep.
This tool therefore:
- archives only records, and names every path it refused to archive with the reason
(``model-weights``, ``transient-inference``, ``secret``, ``log``, ``symlink``, ``oversize``), so a
missing file is an auditable decision instead of a silent omission;
- writes one SHA-256 per archived member and a reproducible archive (sorted members, zeroed
timestamps and ownership), so two backups of the same records are byte-identical;
- verifies every digest, member name, and record count before a restore writes anything, and
refuses absolute, traversal, link, and unlisted members.
Usage:
python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy.tar.gz
python3 scripts/operator_backup.py verify --archive /srv/backups/timmy.tar.gz --manifest /srv/backups/timmy.tar.gz.manifest.json
python3 scripts/operator_backup.py restore --archive ... --manifest ... --target /srv/timmy-corpus.restored
"""
from __future__ import annotations
import argparse
import gzip
import hashlib
import io
import json
import os
import sys
import tarfile
from datetime import datetime, timezone
from pathlib import Path, PurePosixPath
SCHEMA_VERSION = 1
TOOL = "operator_backup.py"
PAYLOAD_PREFIX = "payload/"
DEFAULT_MAX_BYTES = 64 * 1024 * 1024
WEIGHT_SUFFIXES = (".gguf", ".safetensors", ".onnx", ".bin", ".pt", ".ckpt", ".pth")
WEIGHT_DIRS = ("weights", "models")
TRANSIENT_DIRS = ("transient", "inference-tmp", "inference-tmp-images", "uploads", "tmp", "cache")
LOG_DIRS = ("logs",)
LOG_SUFFIXES = (".log",)
SECRET_NAMES = (".env", "id_rsa", "id_ed25519", "id_ecdsa", "tokens.json", "credentials.json")
SECRET_SUFFIXES = (".key", ".pem", ".p12", ".pfx", ".env")
class BackupError(Exception):
"""A refusal that is reported on stderr and exits non-zero."""
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def classify(relative: PurePosixPath) -> str | None:
"""Return the reason a path is not archived, or ``None`` when it is a record."""
parts = relative.parts[:-1]
name = relative.name
lowered = [part.lower() for part in parts]
suffix = PurePosixPath(name.lower()).suffix
if name.lower().startswith(".env") or name.lower() in SECRET_NAMES or suffix in SECRET_SUFFIXES:
return "secret"
if suffix in WEIGHT_SUFFIXES or any(part in WEIGHT_DIRS for part in lowered):
return "model-weights"
if any(part in TRANSIENT_DIRS for part in lowered):
return "transient-inference"
if any(part in LOG_DIRS for part in lowered) or suffix in LOG_SUFFIXES:
return "log"
return None
def record_counts(manifest_path: Path) -> dict[str, int]:
"""Count the append-only manifest the way a trainer would read it."""
counts = {"samples": 0, "tombstones": 0, "activeSamples": 0}
if not manifest_path.is_file():
return counts
revoked: set[str] = set()
samples: list[dict] = []
for line in manifest_path.read_text(encoding="utf-8").splitlines():
if not line.strip():
continue
try:
entry = json.loads(line)
except json.JSONDecodeError as error:
raise BackupError(f"manifest {manifest_path} is not valid JSON Lines: {error}") from error
kind = entry.get("recordType")
if kind == "sample":
samples.append(entry)
counts["samples"] += 1
elif kind == "tombstone":
counts["tombstones"] += 1
if entry.get("contributionId"):
revoked.add(str(entry["contributionId"]))
counts["activeSamples"] = sum(
1 for entry in samples
if str(entry.get("contributionId")) not in revoked
and str((entry.get("lifecycle") or {}).get("state")) == "active"
)
return counts
def collect(source: Path, max_bytes: int) -> tuple[list[tuple[PurePosixPath, Path]], list[dict]]:
included: list[tuple[PurePosixPath, Path]] = []
excluded: list[dict] = []
for dirpath, dirnames, filenames in os.walk(source, followlinks=False):
dirnames.sort()
base = Path(dirpath)
for name in sorted(filenames):
path = base / name
relative = PurePosixPath(path.relative_to(source).as_posix())
if path.is_symlink():
excluded.append({"path": str(relative), "reason": "symlink", "bytes": 0})
continue
reason = classify(relative)
if reason:
excluded.append({"path": str(relative), "reason": reason, "bytes": _size(path)})
continue
size = _size(path)
if size > max_bytes:
excluded.append({"path": str(relative), "reason": "oversize", "bytes": size})
continue
included.append((relative, path))
included.sort(key=lambda item: str(item[0]))
excluded.sort(key=lambda entry: entry["path"])
return included, excluded
def _size(path: Path) -> int:
try:
return path.stat().st_size
except OSError:
return 0
def backup(*, source: Path, archive: Path, manifest_path: Path, label: str, max_bytes: int) -> dict:
if not source.is_dir():
raise BackupError(f"source {source} is not a directory")
if archive.exists():
raise BackupError(f"archive {archive} already exists; move it aside instead of overwriting a backup")
included, excluded = collect(source, max_bytes)
files = [{"path": str(relative), "sha256": sha256_file(path), "bytes": _size(path)} for relative, path in included]
counts = record_counts(source / "manifest.jsonl")
archive.parent.mkdir(parents=True, exist_ok=True)
partial = archive.with_name(archive.name + ".partial")
with partial.open("wb") as raw:
# gzip mtime=0 and zeroed tar metadata keep two backups of the same records identical.
with gzip.GzipFile(fileobj=raw, mode="wb", mtime=0) as compressed:
with tarfile.open(fileobj=compressed, mode="w", format=tarfile.PAX_FORMAT) as tar:
for (relative, path), entry in zip(included, files):
payload = path.read_bytes()
info = tarfile.TarInfo(PAYLOAD_PREFIX + str(relative))
info.size = len(payload)
info.mode = 0o644
info.mtime = 0
info.uid = info.gid = 0
info.uname = info.gname = ""
tar.addfile(info, io.BytesIO(payload))
if hashlib.sha256(payload).hexdigest() != entry["sha256"]:
raise BackupError(f"{relative} changed while it was being archived")
os.replace(partial, archive)
manifest = {
"schemaVersion": SCHEMA_VERSION,
"tool": TOOL,
"createdAt": datetime.now(timezone.utc).isoformat(timespec="seconds"),
"label": label,
"sourceLabel": source.name,
"archive": {"name": archive.name, "sha256": sha256_file(archive), "bytes": _size(archive)},
"records": counts,
"files": files,
"excluded": excluded,
}
manifest_path.parent.mkdir(parents=True, exist_ok=True)
manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8")
return {
"archive": str(archive),
"manifest": str(manifest_path),
"files": len(files),
"bytes": manifest["archive"]["bytes"],
"records": counts,
"excluded": {entry["reason"]: sum(1 for other in excluded if other["reason"] == entry["reason"]) for entry in excluded},
}
def load_manifest(manifest_path: Path) -> dict:
if not manifest_path.is_file():
raise BackupError(f"manifest {manifest_path} does not exist")
try:
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
except json.JSONDecodeError as error:
raise BackupError(f"manifest {manifest_path} is not valid JSON: {error}") from error
if manifest.get("schemaVersion") != SCHEMA_VERSION:
raise BackupError(f"manifest schemaVersion {manifest.get('schemaVersion')!r} is not {SCHEMA_VERSION}")
return manifest
def verified_members(archive: Path, manifest: dict) -> list[tuple[str, bytes]]:
"""Check every member of the archive before anything is written to disk."""
if not archive.is_file():
raise BackupError(f"archive {archive} does not exist")
expected = {entry["path"]: entry for entry in manifest.get("files") or []}
extracted: list[tuple[str, bytes]] = []
seen: set[str] = set()
with tarfile.open(archive, "r:gz") as tar:
for info in tar.getmembers():
if info.isdir():
continue
if info.issym() or info.islnk():
raise BackupError(f"archive member {info.name!r} is a link, not a regular file")
if not info.isfile():
raise BackupError(f"archive member {info.name!r} is not a regular file")
if os.path.isabs(info.name):
raise BackupError(f"archive member {info.name!r} is an absolute path")
if not info.name.startswith(PAYLOAD_PREFIX):
raise BackupError(f"archive member {info.name!r} is not under {PAYLOAD_PREFIX}")
relative = info.name[len(PAYLOAD_PREFIX):]
parts = PurePosixPath(relative).parts
if not parts or any(part in ("..", ".", "") for part in parts):
raise BackupError(f"archive member {info.name!r} resolves outside the restore root")
if relative not in expected:
raise BackupError(f"archive member {info.name!r} is not listed in the manifest")
handle = tar.extractfile(info)
if handle is None:
raise BackupError(f"archive member {info.name!r} cannot be read")
payload = handle.read()
if hashlib.sha256(payload).hexdigest() != expected[relative]["sha256"]:
raise BackupError(
f"archive member {info.name!r} does not match the manifest digest "
"(the archive changed after it was recorded)"
)
seen.add(relative)
extracted.append((relative, payload))
missing = sorted(set(expected) - seen)
if missing:
raise BackupError(f"archive is missing members the manifest lists: {', '.join(missing)}")
return extracted
def verify(*, archive: Path, manifest_path: Path) -> dict:
manifest = load_manifest(manifest_path)
members_list = verified_members(archive, manifest)
return {"archive": str(archive), "files": len(members_list), "records": manifest.get("records", {})}
def restore(*, archive: Path, manifest_path: Path, target: Path) -> dict:
manifest = load_manifest(manifest_path)
if target.exists():
if not target.is_dir():
raise BackupError(f"restore target {target} is not a directory")
if any(target.iterdir()):
raise BackupError(f"restore target {target} is not empty; restore into a new directory")
# Digests and member names are checked before the first byte is written, so a hostile or
# tampered archive cannot leave a half-restored record set behind.
members_list = verified_members(archive, manifest)
target.mkdir(parents=True, exist_ok=True)
for relative, payload in members_list:
destination = target / relative
destination.parent.mkdir(parents=True, exist_ok=True)
destination.write_bytes(payload)
os.chmod(destination, 0o644)
counts = record_counts(target / "manifest.jsonl")
if manifest.get("records") and counts != manifest["records"]:
raise BackupError(
f"restored record counts {counts} do not match the backup manifest {manifest['records']}"
)
return {"target": str(target), "files": len(members_list), "records": counts}
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
subparsers = parser.add_subparsers(dest="command", required=True)
backup_parser = subparsers.add_parser("backup", help="archive the records and write a manifest")
backup_parser.add_argument("--source", required=True, type=Path)
backup_parser.add_argument("--archive", required=True, type=Path)
backup_parser.add_argument("--manifest", type=Path)
backup_parser.add_argument("--label", default="manual")
backup_parser.add_argument("--max-bytes", type=int, default=DEFAULT_MAX_BYTES)
verify_parser = subparsers.add_parser("verify", help="recompute every digest in an archive")
verify_parser.add_argument("--archive", required=True, type=Path)
verify_parser.add_argument("--manifest", required=True, type=Path)
restore_parser = subparsers.add_parser("restore", help="verify, then write the records into a new directory")
restore_parser.add_argument("--archive", required=True, type=Path)
restore_parser.add_argument("--manifest", required=True, type=Path)
restore_parser.add_argument("--target", required=True, type=Path)
return parser
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
if args.command == "backup":
manifest_path = args.manifest or args.archive.with_name(args.archive.name + ".manifest.json")
receipt = backup(
source=args.source, archive=args.archive, manifest_path=manifest_path,
label=args.label, max_bytes=args.max_bytes,
)
elif args.command == "verify":
receipt = verify(archive=args.archive, manifest_path=args.manifest)
else:
receipt = restore(archive=args.archive, manifest_path=args.manifest, target=args.target)
except BackupError as error:
print(f"refused: {error}", file=sys.stderr)
return 2
print(json.dumps(receipt, indent=2, sort_keys=True))
return 0
if __name__ == "__main__":
sys.exit(main())
+120 -1
View File
@@ -17,6 +17,10 @@ const version = process.env.TIMMY_RELEASE_VERSION || 'review build';
// changed in this release instead of implying a feature that did not ship.
const releaseDelta = process.env.TIMMY_RELEASE_DEMO_DELTA
|| 'Verified rebuild of the current user-visible feature path — no new feature is claimed.';
// The recording targets the release builder's own acceptance server by default. A caller that
// names another origin is explicit about it, so a local run can never silently record a different
// tree than the one the gates measured.
const demoUrl = process.env.TIMMY_DEMO_URL || 'http://127.0.0.1:4173';
const captionsOut = process.env.TIMMY_DEMO_CAPTIONS_OUT || '';
const captionsShown = [];
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
@@ -116,7 +120,7 @@ await page.route('**/api/agent/chat', route => {
});
});
await page.goto('http://127.0.0.1:4173', { waitUntil: 'networkidle' });
await page.goto(demoUrl, { waitUntil: 'networkidle' });
await page.evaluate(() => localStorage.clear());
await page.reload({ waitUntil: 'networkidle' });
@@ -1161,6 +1165,121 @@ await caption('Each candidate is scored on test at a threshold chosen on validat
await caption('One contributor spread across two splits is refused before anything is scored, and a passing verdict is proven only by a fixture engineered to clear it', 3800);
await page.evaluate(() => document.querySelector('#calibration-report')?.remove());
// Issue #39 — the operator backup is a command line, not a screen in the app. The app itself keeps
// no server-side journal, so what an operator can actually lose is the corpus: the append-only
// manifest with its withdrawal tombstones. This scene builds a synthetic data root with the real
// ingest and revoke commands, plants the four things a backup must refuse, then runs the real
// backup, verify, and restore and reads every value on the card back out of those runs.
async function operatorBackupAudit() {
const root = await mkdtemp(path.join(os.tmpdir(), 'timmy-backup-demo-'));
const dataRoot = path.join(root, 'srv-timmy-corpus');
const archive = path.join(root, 'timmy-nightly.tar.gz');
const manifestPath = `${archive}.manifest.json`;
const restoreInto = path.join(root, 'restored');
const secondPhoto = path.join(root, 'synthetic-second.png');
const backupCli = args => execFileAsync('python3', ['scripts/operator_backup.py', ...args], { cwd: process.cwd() });
const ingest = (input, contributionId, subject, bristol) => corpusCli([
'scripts/ingest_training_photo.py',
'--input', input, '--output-dir', dataRoot, '--subject-id', subject,
'--consent-version', 'training-consent-v1', '--contribution-id', contributionId,
'--bristol-type', bristol, '--color', 'brown', '--quality', 'good', '--reviewer', 'user-confirmed',
]);
try {
// One contribution stays active and one is withdrawn, so the manifest ends up with a live sample
// and a real tombstone, and the withdrawn derived image is already gone from disk.
await ingest('tests/fixtures/synthetic-type4.jpg', 'demo-backup-0001', 'demo-backup-kept', '4');
await writeFile(secondPhoto, Buffer.from('iVBORw0KGgoAAAANSUhEUgAAAAgAAAAICAIAAABLbSncAAAAFElEQVR4nGOsCDFhwAaYsIoOWgkAzUwBECP1Kh0AAAAASUVORK5CYII=', 'base64'));
await ingest(secondPhoto, 'demo-backup-0002', 'demo-backup-withdrawn', '3');
await corpusCli([
'scripts/ingest_training_photo.py', '--output-dir', dataRoot,
'--revoke', 'demo-backup-0002', '--reason', 'participant withdrew consent',
]);
// The four things that must never be archived, as real files with real bytes: a weight shard, a
// transient inference upload, an operator environment file, and a log. An exclusion that quietly
// stopped working fails here instead of in a caption.
await mkdir(path.join(dataRoot, 'weights'), { recursive: true });
await writeFile(path.join(dataRoot, 'weights', 'model.gguf'), 'synthetic weight shard — never archived');
await mkdir(path.join(dataRoot, 'transient'), { recursive: true });
await writeFile(path.join(dataRoot, 'transient', 'inference-upload.jpg'), 'transient inference bytes — never archived');
await writeFile(path.join(dataRoot, 'secrets.env'), 'TIMMY_DATASET_SALT=demo-only-placeholder\n');
await writeFile(path.join(dataRoot, 'operator.log'), 'GET /api/healthz 200\n');
const backupRun = await backupCli(['backup', '--source', dataRoot, '--archive', archive, '--label', 'nightly']);
const receipt = JSON.parse(backupRun.stdout);
const manifest = JSON.parse(await readFile(manifestPath, 'utf8'));
const reasons = manifest.excluded.reduce((out, entry) => ({ ...out, [entry.reason]: (out[entry.reason] || 0) + 1 }), {});
if (manifest.files.some(entry => /model\.gguf|inference-upload|secrets\.env|operator\.log/.test(entry.path))) {
throw new Error('the backup must not archive the weight shard or a transient inference upload');
}
for (const reason of ['model-weights', 'transient-inference', 'secret', 'log']) {
if (!reasons[reason]) throw new Error(`the backup must record a ${reason} exclusion`);
}
if (receipt.records.samples !== 2 || receipt.records.tombstones !== 1 || receipt.records.activeSamples !== 1) {
throw new Error('the backup receipt must report the real record counts');
}
const verified = JSON.parse((await backupCli(['verify', '--archive', archive, '--manifest', manifestPath])).stdout);
if (verified.files !== manifest.files.length) throw new Error('verify must re-hash every stored member');
const restored = JSON.parse((await backupCli([
'restore', '--archive', archive, '--manifest', manifestPath, '--target', restoreInto,
])).stdout);
if (restored.records.tombstones !== 1 || restored.records.activeSamples !== 1) {
throw new Error('the restore must return the same record counts');
}
const restoredManifest = await readFile(path.join(restoreInto, 'manifest.jsonl'));
if (!restoredManifest.equals(await readFile(path.join(dataRoot, 'manifest.jsonl')))) {
throw new Error('the restore must return the same records, tombstone included');
}
return { receipt, manifest, reasons, verified, restored };
} finally {
await rm(root, { recursive: true, force: true });
}
}
const opBackup = await operatorBackupAudit();
await tap('[data-view="home"]', 400);
const shortDigest = value => String(value).slice(0, 12);
const backupRows = [
['command', 'python3 scripts/operator_backup.py backup --source srv-timmy-corpus'],
['stored', `${opBackup.receipt.files} members · ${opBackup.manifest.files.map(entry => entry.path.split('/').pop()).join(' + ')}`],
['records', `samples ${opBackup.receipt.records.samples} · tombstones ${opBackup.receipt.records.tombstones} · active ${opBackup.receipt.records.activeSamples}`],
['—', ''],
['refused', `model-weights ${opBackup.reasons['model-weights']} · transient-inference ${opBackup.reasons['transient-inference']}`],
['refused', `secret ${opBackup.reasons['secret']} · log ${opBackup.reasons['log']}`],
['reason', 'every skipped path is named in the manifest, not silently dropped'],
['—', ''],
['digest', `sha256 ${shortDigest(opBackup.manifest.archive.sha256)}… · reproducible (sorted, mtime 0)`],
['verify', `ok · ${opBackup.verified.files} members re-hashed against the manifest`],
['restore', `ok · samples ${opBackup.restored.records.samples} · tombstones ${opBackup.restored.records.tombstones} · active ${opBackup.restored.records.activeSamples}`],
['manifest', 'returned byte-for-byte, tombstone included'],
['server', 'the app holds no journal of its own — there is no server copy to back up'],
];
const backupCard = await evidenceCard({
id: 'production-backup',
title: 'OPERATOR BACKUP',
subtitle: 'command line, not app UI — real run, synthetic corpus',
rows: backupRows,
font: '600 9.5px/1.45 ui-monospace, SFMono-Regular, Menlo, monospace',
keyWidth: 66,
gap: '1px',
rowGap: '6px',
});
await sleep(420);
const backupProof = await evidenceCardProof('production-backup');
if (!backupProof.ok) throw new Error(`Production backup card is not fully readable on screen: ${JSON.stringify(backupProof)}`);
// The card is drawn from the run: prove the counts and the refusals on it are the ones it produced.
if (!backupCard.text.includes(`samples ${opBackup.receipt.records.samples} · tombstones ${opBackup.receipt.records.tombstones}`)) {
throw new Error('the card must show the record counts the backup actually reported');
}
if (!backupCard.text.includes('model-weights 1')) throw new Error('the card must show the exclusion the run actually recorded');
if (!backupCard.text.includes(shortDigest(opBackup.manifest.archive.sha256))) {
throw new Error('the card must show the archive digest the run actually wrote');
}
await caption('The operator backup is a command line, not a screen in the app — this runs it against a synthetic corpus the real ingest and revoke commands built', 3800);
await caption('Model weights and transient inference images are refused by name, and every skipped path is listed with its reason', 3600);
await caption('Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes', 3400);
await caption('A restore returns the same records, including the withdrawal tombstone a trainer must never lose', 3600);
await page.evaluate(() => document.querySelector('#production-backup')?.remove());
// Back to the privacy view for the portability scene: the privacy entry lives on the journal view.
await tap('[data-view="calendar"]', 550);
await tap('[data-view="privacy"]', 600);