feat(ops): operator backup, restore, and production runbook (#39)
Quality gates / quality (pull_request) Canceled after 0s
Quality gates / quality (pull_request) Canceled after 0s
The app keeps the journal in the person's browser, so the server has no diary to lose. What an operator can lose is the corpus: the append-only manifest.jsonl with its provenance records and its withdrawal tombstones. scripts/operator_backup.py archives only records and names every path it refuses (model weights, transient inference images, secrets, logs, symlinks, oversize) with the reason, writes one SHA-256 per stored member, and produces a reproducible archive. restore re-verifies every digest, member name, and record count before it writes a byte, and refuses absolute, traversal, link, unlisted, and non-empty-target cases, so a tampered archive or a lost tombstone fails loudly. docs/PRODUCTION-RUNBOOK.md covers topology, network boundaries (including the app's real 0.0.0.0 default and the staging HOST=127.0.0.1 template), the four secrets, retention, and both rollback paths; its documented commands are parsed against the real CLIs by its test. The release demo runs the real backup, verify, and restore against a synthetic corpus built by the real ingest and revoke commands and shows what was stored, what was refused, and what came back.
This commit is contained in:
1 parent
7fe1901703
commit
48f0f56835
9 files changed
+1170
-4
No files matched your search
@@ -73,6 +73,10 @@ DEMO_REQUIRED_CAPTIONS = (
|
||||
"The model said Type 6 and the reviewer said Type 5: the disagreement is recorded, not smoothed over",
|
||||
"Withdrawing the contribution empties the queue and the training set, and the tool refuses to review it at all",
|
||||
"The dataset card is generated from the corpus manifest, so no count on it is typed by hand",
|
||||
"The operator backup is a command line, not a screen in the app",
|
||||
"Model weights and transient inference images are refused by name, and every skipped path is listed with its reason",
|
||||
"Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes",
|
||||
"A restore returns the same records, including the withdrawal tombstone a trainer must never lose",
|
||||
"One contributor's photos stay in one split: a corpus that leaks a person across splits is refused",
|
||||
"Devices, contexts, and licenses stay pending until the pipeline collects them",
|
||||
"The card is a page, not a screen in the app — it is published with the real generator against synthetic fixtures",
|
||||
@@ -183,6 +187,10 @@ def main() -> int:
|
||||
# Issue #29's evaluation harness decides GO/BLOCKED for a model, so its refusal and gate
|
||||
# arithmetic are exercised before this release is allowed to exist.
|
||||
run(["python3", "tests/calibration-harness.test.py", "-v"], tree)
|
||||
# Issue #39's operator backup is the only copy of the corpus tombstones, so the tool that
|
||||
# produces it, and the runbook whose commands must stay real, are gated before the release.
|
||||
run(["python3", "tests/backup-restore.test.py", "-v"], tree)
|
||||
run(["python3", "tests/production-runbook.test.py", "-v"], tree)
|
||||
server_env = dict(os.environ)
|
||||
server_env.update({"HOST": "127.0.0.1", "PORT": "4173", "TIMMY_VISION_ENABLED": "false"})
|
||||
# A leftover server from another checkout answers on this port and the readiness probe cannot
|
||||
@@ -360,7 +368,7 @@ def main() -> int:
|
||||
"codec": "h264",
|
||||
"pixel_format": "yuv420p",
|
||||
"recorded_from_working_app": True,
|
||||
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything, plus the review ledger card, where the real review commands run against the committed synthetic sample and the card shows their actual output: one pending proposal with the model answer withheld, a refused reveal before commitment, a committed Type 5 correction beside the Type 4 proposal, the revealed synthetic Type 6 answer recorded as a three-field disagreement, a second reader's adjudication pointing at the review it resolved, one trainable sample, and a withdrawal that empties the queue and the training set and makes review refused",
|
||||
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything, plus the review ledger card, where the real review commands run against the committed synthetic sample and the card shows their actual output: one pending proposal with the model answer withheld, a refused reveal before commitment, a committed Type 5 correction beside the Type 4 proposal, the revealed synthetic Type 6 answer recorded as a three-field disagreement, a second reader's adjudication pointing at the review it resolved, one trainable sample, and a withdrawal that empties the queue and the training set and makes review refused, plus the operator backup card, where the real backup, verify, and restore commands run against a synthetic corpus built by the real ingest and revoke commands and the card shows their actual output: two samples and one tombstone recorded, a weight shard, a transient inference upload, an operator env file and a log refused by name with the reason listed, the exclusion counts the manifest wrote, one digest for the archive, a verify that re-hashes every stored member, and a restore that returns the manifest byte-for-byte with the tombstone intact",
|
||||
},
|
||||
"gates": {
|
||||
"unit_security": "passed",
|
||||
@@ -381,6 +389,8 @@ def main() -> int:
|
||||
"reviewer_adjudication_blind": "passed",
|
||||
"dataset_card_generated": "passed",
|
||||
"calibration_harness_gates": "passed",
|
||||
"production_backup_restore": "passed",
|
||||
"production_runbook_commands": "passed",
|
||||
"portable_backup_restore": "passed",
|
||||
"dependency_audit_high": "passed",
|
||||
"syntax": "passed",
|
||||
|
||||
@@ -0,0 +1,326 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Operator backup and restore for the records Timmy must not lose.
|
||||
|
||||
The app keeps the journal in the person's own browser, so the server side of Timmy holds no diary.
|
||||
What the operator does hold, and what cannot be rebuilt from anywhere else, is the corpus: the
|
||||
append-only ``manifest.jsonl`` with its provenance records and its withdrawal tombstones, plus the
|
||||
derived images those records point at. Losing that file loses the proof that a participant
|
||||
withdrew, which is the one receipt the project has promised to keep.
|
||||
|
||||
This tool therefore:
|
||||
|
||||
- archives only records, and names every path it refused to archive with the reason
|
||||
(``model-weights``, ``transient-inference``, ``secret``, ``log``, ``symlink``, ``oversize``), so a
|
||||
missing file is an auditable decision instead of a silent omission;
|
||||
- writes one SHA-256 per archived member and a reproducible archive (sorted members, zeroed
|
||||
timestamps and ownership), so two backups of the same records are byte-identical;
|
||||
- verifies every digest, member name, and record count before a restore writes anything, and
|
||||
refuses absolute, traversal, link, and unlisted members.
|
||||
|
||||
Usage:
|
||||
|
||||
python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy.tar.gz
|
||||
python3 scripts/operator_backup.py verify --archive /srv/backups/timmy.tar.gz --manifest /srv/backups/timmy.tar.gz.manifest.json
|
||||
python3 scripts/operator_backup.py restore --archive ... --manifest ... --target /srv/timmy-corpus.restored
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gzip
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tarfile
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path, PurePosixPath
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
TOOL = "operator_backup.py"
|
||||
PAYLOAD_PREFIX = "payload/"
|
||||
DEFAULT_MAX_BYTES = 64 * 1024 * 1024
|
||||
|
||||
WEIGHT_SUFFIXES = (".gguf", ".safetensors", ".onnx", ".bin", ".pt", ".ckpt", ".pth")
|
||||
WEIGHT_DIRS = ("weights", "models")
|
||||
TRANSIENT_DIRS = ("transient", "inference-tmp", "inference-tmp-images", "uploads", "tmp", "cache")
|
||||
LOG_DIRS = ("logs",)
|
||||
LOG_SUFFIXES = (".log",)
|
||||
SECRET_NAMES = (".env", "id_rsa", "id_ed25519", "id_ecdsa", "tokens.json", "credentials.json")
|
||||
SECRET_SUFFIXES = (".key", ".pem", ".p12", ".pfx", ".env")
|
||||
|
||||
|
||||
class BackupError(Exception):
|
||||
"""A refusal that is reported on stderr and exits non-zero."""
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def classify(relative: PurePosixPath) -> str | None:
|
||||
"""Return the reason a path is not archived, or ``None`` when it is a record."""
|
||||
parts = relative.parts[:-1]
|
||||
name = relative.name
|
||||
lowered = [part.lower() for part in parts]
|
||||
suffix = PurePosixPath(name.lower()).suffix
|
||||
|
||||
if name.lower().startswith(".env") or name.lower() in SECRET_NAMES or suffix in SECRET_SUFFIXES:
|
||||
return "secret"
|
||||
if suffix in WEIGHT_SUFFIXES or any(part in WEIGHT_DIRS for part in lowered):
|
||||
return "model-weights"
|
||||
if any(part in TRANSIENT_DIRS for part in lowered):
|
||||
return "transient-inference"
|
||||
if any(part in LOG_DIRS for part in lowered) or suffix in LOG_SUFFIXES:
|
||||
return "log"
|
||||
return None
|
||||
|
||||
|
||||
def record_counts(manifest_path: Path) -> dict[str, int]:
|
||||
"""Count the append-only manifest the way a trainer would read it."""
|
||||
counts = {"samples": 0, "tombstones": 0, "activeSamples": 0}
|
||||
if not manifest_path.is_file():
|
||||
return counts
|
||||
revoked: set[str] = set()
|
||||
samples: list[dict] = []
|
||||
for line in manifest_path.read_text(encoding="utf-8").splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
entry = json.loads(line)
|
||||
except json.JSONDecodeError as error:
|
||||
raise BackupError(f"manifest {manifest_path} is not valid JSON Lines: {error}") from error
|
||||
kind = entry.get("recordType")
|
||||
if kind == "sample":
|
||||
samples.append(entry)
|
||||
counts["samples"] += 1
|
||||
elif kind == "tombstone":
|
||||
counts["tombstones"] += 1
|
||||
if entry.get("contributionId"):
|
||||
revoked.add(str(entry["contributionId"]))
|
||||
counts["activeSamples"] = sum(
|
||||
1 for entry in samples
|
||||
if str(entry.get("contributionId")) not in revoked
|
||||
and str((entry.get("lifecycle") or {}).get("state")) == "active"
|
||||
)
|
||||
return counts
|
||||
|
||||
|
||||
def collect(source: Path, max_bytes: int) -> tuple[list[tuple[PurePosixPath, Path]], list[dict]]:
|
||||
included: list[tuple[PurePosixPath, Path]] = []
|
||||
excluded: list[dict] = []
|
||||
for dirpath, dirnames, filenames in os.walk(source, followlinks=False):
|
||||
dirnames.sort()
|
||||
base = Path(dirpath)
|
||||
for name in sorted(filenames):
|
||||
path = base / name
|
||||
relative = PurePosixPath(path.relative_to(source).as_posix())
|
||||
if path.is_symlink():
|
||||
excluded.append({"path": str(relative), "reason": "symlink", "bytes": 0})
|
||||
continue
|
||||
reason = classify(relative)
|
||||
if reason:
|
||||
excluded.append({"path": str(relative), "reason": reason, "bytes": _size(path)})
|
||||
continue
|
||||
size = _size(path)
|
||||
if size > max_bytes:
|
||||
excluded.append({"path": str(relative), "reason": "oversize", "bytes": size})
|
||||
continue
|
||||
included.append((relative, path))
|
||||
included.sort(key=lambda item: str(item[0]))
|
||||
excluded.sort(key=lambda entry: entry["path"])
|
||||
return included, excluded
|
||||
|
||||
|
||||
def _size(path: Path) -> int:
|
||||
try:
|
||||
return path.stat().st_size
|
||||
except OSError:
|
||||
return 0
|
||||
|
||||
|
||||
def backup(*, source: Path, archive: Path, manifest_path: Path, label: str, max_bytes: int) -> dict:
|
||||
if not source.is_dir():
|
||||
raise BackupError(f"source {source} is not a directory")
|
||||
if archive.exists():
|
||||
raise BackupError(f"archive {archive} already exists; move it aside instead of overwriting a backup")
|
||||
|
||||
included, excluded = collect(source, max_bytes)
|
||||
files = [{"path": str(relative), "sha256": sha256_file(path), "bytes": _size(path)} for relative, path in included]
|
||||
counts = record_counts(source / "manifest.jsonl")
|
||||
|
||||
archive.parent.mkdir(parents=True, exist_ok=True)
|
||||
partial = archive.with_name(archive.name + ".partial")
|
||||
with partial.open("wb") as raw:
|
||||
# gzip mtime=0 and zeroed tar metadata keep two backups of the same records identical.
|
||||
with gzip.GzipFile(fileobj=raw, mode="wb", mtime=0) as compressed:
|
||||
with tarfile.open(fileobj=compressed, mode="w", format=tarfile.PAX_FORMAT) as tar:
|
||||
for (relative, path), entry in zip(included, files):
|
||||
payload = path.read_bytes()
|
||||
info = tarfile.TarInfo(PAYLOAD_PREFIX + str(relative))
|
||||
info.size = len(payload)
|
||||
info.mode = 0o644
|
||||
info.mtime = 0
|
||||
info.uid = info.gid = 0
|
||||
info.uname = info.gname = ""
|
||||
tar.addfile(info, io.BytesIO(payload))
|
||||
if hashlib.sha256(payload).hexdigest() != entry["sha256"]:
|
||||
raise BackupError(f"{relative} changed while it was being archived")
|
||||
os.replace(partial, archive)
|
||||
|
||||
manifest = {
|
||||
"schemaVersion": SCHEMA_VERSION,
|
||||
"tool": TOOL,
|
||||
"createdAt": datetime.now(timezone.utc).isoformat(timespec="seconds"),
|
||||
"label": label,
|
||||
"sourceLabel": source.name,
|
||||
"archive": {"name": archive.name, "sha256": sha256_file(archive), "bytes": _size(archive)},
|
||||
"records": counts,
|
||||
"files": files,
|
||||
"excluded": excluded,
|
||||
}
|
||||
manifest_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
return {
|
||||
"archive": str(archive),
|
||||
"manifest": str(manifest_path),
|
||||
"files": len(files),
|
||||
"bytes": manifest["archive"]["bytes"],
|
||||
"records": counts,
|
||||
"excluded": {entry["reason"]: sum(1 for other in excluded if other["reason"] == entry["reason"]) for entry in excluded},
|
||||
}
|
||||
|
||||
|
||||
def load_manifest(manifest_path: Path) -> dict:
|
||||
if not manifest_path.is_file():
|
||||
raise BackupError(f"manifest {manifest_path} does not exist")
|
||||
try:
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
except json.JSONDecodeError as error:
|
||||
raise BackupError(f"manifest {manifest_path} is not valid JSON: {error}") from error
|
||||
if manifest.get("schemaVersion") != SCHEMA_VERSION:
|
||||
raise BackupError(f"manifest schemaVersion {manifest.get('schemaVersion')!r} is not {SCHEMA_VERSION}")
|
||||
return manifest
|
||||
|
||||
|
||||
def verified_members(archive: Path, manifest: dict) -> list[tuple[str, bytes]]:
|
||||
"""Check every member of the archive before anything is written to disk."""
|
||||
if not archive.is_file():
|
||||
raise BackupError(f"archive {archive} does not exist")
|
||||
expected = {entry["path"]: entry for entry in manifest.get("files") or []}
|
||||
extracted: list[tuple[str, bytes]] = []
|
||||
seen: set[str] = set()
|
||||
with tarfile.open(archive, "r:gz") as tar:
|
||||
for info in tar.getmembers():
|
||||
if info.isdir():
|
||||
continue
|
||||
if info.issym() or info.islnk():
|
||||
raise BackupError(f"archive member {info.name!r} is a link, not a regular file")
|
||||
if not info.isfile():
|
||||
raise BackupError(f"archive member {info.name!r} is not a regular file")
|
||||
if os.path.isabs(info.name):
|
||||
raise BackupError(f"archive member {info.name!r} is an absolute path")
|
||||
if not info.name.startswith(PAYLOAD_PREFIX):
|
||||
raise BackupError(f"archive member {info.name!r} is not under {PAYLOAD_PREFIX}")
|
||||
relative = info.name[len(PAYLOAD_PREFIX):]
|
||||
parts = PurePosixPath(relative).parts
|
||||
if not parts or any(part in ("..", ".", "") for part in parts):
|
||||
raise BackupError(f"archive member {info.name!r} resolves outside the restore root")
|
||||
if relative not in expected:
|
||||
raise BackupError(f"archive member {info.name!r} is not listed in the manifest")
|
||||
handle = tar.extractfile(info)
|
||||
if handle is None:
|
||||
raise BackupError(f"archive member {info.name!r} cannot be read")
|
||||
payload = handle.read()
|
||||
if hashlib.sha256(payload).hexdigest() != expected[relative]["sha256"]:
|
||||
raise BackupError(
|
||||
f"archive member {info.name!r} does not match the manifest digest "
|
||||
"(the archive changed after it was recorded)"
|
||||
)
|
||||
seen.add(relative)
|
||||
extracted.append((relative, payload))
|
||||
missing = sorted(set(expected) - seen)
|
||||
if missing:
|
||||
raise BackupError(f"archive is missing members the manifest lists: {', '.join(missing)}")
|
||||
return extracted
|
||||
|
||||
|
||||
def verify(*, archive: Path, manifest_path: Path) -> dict:
|
||||
manifest = load_manifest(manifest_path)
|
||||
members_list = verified_members(archive, manifest)
|
||||
return {"archive": str(archive), "files": len(members_list), "records": manifest.get("records", {})}
|
||||
|
||||
|
||||
def restore(*, archive: Path, manifest_path: Path, target: Path) -> dict:
|
||||
manifest = load_manifest(manifest_path)
|
||||
if target.exists():
|
||||
if not target.is_dir():
|
||||
raise BackupError(f"restore target {target} is not a directory")
|
||||
if any(target.iterdir()):
|
||||
raise BackupError(f"restore target {target} is not empty; restore into a new directory")
|
||||
# Digests and member names are checked before the first byte is written, so a hostile or
|
||||
# tampered archive cannot leave a half-restored record set behind.
|
||||
members_list = verified_members(archive, manifest)
|
||||
target.mkdir(parents=True, exist_ok=True)
|
||||
for relative, payload in members_list:
|
||||
destination = target / relative
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
destination.write_bytes(payload)
|
||||
os.chmod(destination, 0o644)
|
||||
|
||||
counts = record_counts(target / "manifest.jsonl")
|
||||
if manifest.get("records") and counts != manifest["records"]:
|
||||
raise BackupError(
|
||||
f"restored record counts {counts} do not match the backup manifest {manifest['records']}"
|
||||
)
|
||||
return {"target": str(target), "files": len(members_list), "records": counts}
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
backup_parser = subparsers.add_parser("backup", help="archive the records and write a manifest")
|
||||
backup_parser.add_argument("--source", required=True, type=Path)
|
||||
backup_parser.add_argument("--archive", required=True, type=Path)
|
||||
backup_parser.add_argument("--manifest", type=Path)
|
||||
backup_parser.add_argument("--label", default="manual")
|
||||
backup_parser.add_argument("--max-bytes", type=int, default=DEFAULT_MAX_BYTES)
|
||||
|
||||
verify_parser = subparsers.add_parser("verify", help="recompute every digest in an archive")
|
||||
verify_parser.add_argument("--archive", required=True, type=Path)
|
||||
verify_parser.add_argument("--manifest", required=True, type=Path)
|
||||
|
||||
restore_parser = subparsers.add_parser("restore", help="verify, then write the records into a new directory")
|
||||
restore_parser.add_argument("--archive", required=True, type=Path)
|
||||
restore_parser.add_argument("--manifest", required=True, type=Path)
|
||||
restore_parser.add_argument("--target", required=True, type=Path)
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
if args.command == "backup":
|
||||
manifest_path = args.manifest or args.archive.with_name(args.archive.name + ".manifest.json")
|
||||
receipt = backup(
|
||||
source=args.source, archive=args.archive, manifest_path=manifest_path,
|
||||
label=args.label, max_bytes=args.max_bytes,
|
||||
)
|
||||
elif args.command == "verify":
|
||||
receipt = verify(archive=args.archive, manifest_path=args.manifest)
|
||||
else:
|
||||
receipt = restore(archive=args.archive, manifest_path=args.manifest, target=args.target)
|
||||
except BackupError as error:
|
||||
print(f"refused: {error}", file=sys.stderr)
|
||||
return 2
|
||||
print(json.dumps(receipt, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -17,6 +17,10 @@ const version = process.env.TIMMY_RELEASE_VERSION || 'review build';
|
||||
// changed in this release instead of implying a feature that did not ship.
|
||||
const releaseDelta = process.env.TIMMY_RELEASE_DEMO_DELTA
|
||||
|| 'Verified rebuild of the current user-visible feature path — no new feature is claimed.';
|
||||
// The recording targets the release builder's own acceptance server by default. A caller that
|
||||
// names another origin is explicit about it, so a local run can never silently record a different
|
||||
// tree than the one the gates measured.
|
||||
const demoUrl = process.env.TIMMY_DEMO_URL || 'http://127.0.0.1:4173';
|
||||
const captionsOut = process.env.TIMMY_DEMO_CAPTIONS_OUT || '';
|
||||
const captionsShown = [];
|
||||
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
@@ -116,7 +120,7 @@ await page.route('**/api/agent/chat', route => {
|
||||
});
|
||||
});
|
||||
|
||||
await page.goto('http://127.0.0.1:4173', { waitUntil: 'networkidle' });
|
||||
await page.goto(demoUrl, { waitUntil: 'networkidle' });
|
||||
await page.evaluate(() => localStorage.clear());
|
||||
await page.reload({ waitUntil: 'networkidle' });
|
||||
|
||||
@@ -1161,6 +1165,121 @@ await caption('Each candidate is scored on test at a threshold chosen on validat
|
||||
await caption('One contributor spread across two splits is refused before anything is scored, and a passing verdict is proven only by a fixture engineered to clear it', 3800);
|
||||
await page.evaluate(() => document.querySelector('#calibration-report')?.remove());
|
||||
|
||||
// Issue #39 — the operator backup is a command line, not a screen in the app. The app itself keeps
|
||||
// no server-side journal, so what an operator can actually lose is the corpus: the append-only
|
||||
// manifest with its withdrawal tombstones. This scene builds a synthetic data root with the real
|
||||
// ingest and revoke commands, plants the four things a backup must refuse, then runs the real
|
||||
// backup, verify, and restore and reads every value on the card back out of those runs.
|
||||
async function operatorBackupAudit() {
|
||||
const root = await mkdtemp(path.join(os.tmpdir(), 'timmy-backup-demo-'));
|
||||
const dataRoot = path.join(root, 'srv-timmy-corpus');
|
||||
const archive = path.join(root, 'timmy-nightly.tar.gz');
|
||||
const manifestPath = `${archive}.manifest.json`;
|
||||
const restoreInto = path.join(root, 'restored');
|
||||
const secondPhoto = path.join(root, 'synthetic-second.png');
|
||||
const backupCli = args => execFileAsync('python3', ['scripts/operator_backup.py', ...args], { cwd: process.cwd() });
|
||||
const ingest = (input, contributionId, subject, bristol) => corpusCli([
|
||||
'scripts/ingest_training_photo.py',
|
||||
'--input', input, '--output-dir', dataRoot, '--subject-id', subject,
|
||||
'--consent-version', 'training-consent-v1', '--contribution-id', contributionId,
|
||||
'--bristol-type', bristol, '--color', 'brown', '--quality', 'good', '--reviewer', 'user-confirmed',
|
||||
]);
|
||||
try {
|
||||
// One contribution stays active and one is withdrawn, so the manifest ends up with a live sample
|
||||
// and a real tombstone, and the withdrawn derived image is already gone from disk.
|
||||
await ingest('tests/fixtures/synthetic-type4.jpg', 'demo-backup-0001', 'demo-backup-kept', '4');
|
||||
await writeFile(secondPhoto, Buffer.from('iVBORw0KGgoAAAANSUhEUgAAAAgAAAAICAIAAABLbSncAAAAFElEQVR4nGOsCDFhwAaYsIoOWgkAzUwBECP1Kh0AAAAASUVORK5CYII=', 'base64'));
|
||||
await ingest(secondPhoto, 'demo-backup-0002', 'demo-backup-withdrawn', '3');
|
||||
await corpusCli([
|
||||
'scripts/ingest_training_photo.py', '--output-dir', dataRoot,
|
||||
'--revoke', 'demo-backup-0002', '--reason', 'participant withdrew consent',
|
||||
]);
|
||||
|
||||
// The four things that must never be archived, as real files with real bytes: a weight shard, a
|
||||
// transient inference upload, an operator environment file, and a log. An exclusion that quietly
|
||||
// stopped working fails here instead of in a caption.
|
||||
await mkdir(path.join(dataRoot, 'weights'), { recursive: true });
|
||||
await writeFile(path.join(dataRoot, 'weights', 'model.gguf'), 'synthetic weight shard — never archived');
|
||||
await mkdir(path.join(dataRoot, 'transient'), { recursive: true });
|
||||
await writeFile(path.join(dataRoot, 'transient', 'inference-upload.jpg'), 'transient inference bytes — never archived');
|
||||
await writeFile(path.join(dataRoot, 'secrets.env'), 'TIMMY_DATASET_SALT=demo-only-placeholder\n');
|
||||
await writeFile(path.join(dataRoot, 'operator.log'), 'GET /api/healthz 200\n');
|
||||
|
||||
const backupRun = await backupCli(['backup', '--source', dataRoot, '--archive', archive, '--label', 'nightly']);
|
||||
const receipt = JSON.parse(backupRun.stdout);
|
||||
const manifest = JSON.parse(await readFile(manifestPath, 'utf8'));
|
||||
const reasons = manifest.excluded.reduce((out, entry) => ({ ...out, [entry.reason]: (out[entry.reason] || 0) + 1 }), {});
|
||||
if (manifest.files.some(entry => /model\.gguf|inference-upload|secrets\.env|operator\.log/.test(entry.path))) {
|
||||
throw new Error('the backup must not archive the weight shard or a transient inference upload');
|
||||
}
|
||||
for (const reason of ['model-weights', 'transient-inference', 'secret', 'log']) {
|
||||
if (!reasons[reason]) throw new Error(`the backup must record a ${reason} exclusion`);
|
||||
}
|
||||
if (receipt.records.samples !== 2 || receipt.records.tombstones !== 1 || receipt.records.activeSamples !== 1) {
|
||||
throw new Error('the backup receipt must report the real record counts');
|
||||
}
|
||||
const verified = JSON.parse((await backupCli(['verify', '--archive', archive, '--manifest', manifestPath])).stdout);
|
||||
if (verified.files !== manifest.files.length) throw new Error('verify must re-hash every stored member');
|
||||
const restored = JSON.parse((await backupCli([
|
||||
'restore', '--archive', archive, '--manifest', manifestPath, '--target', restoreInto,
|
||||
])).stdout);
|
||||
if (restored.records.tombstones !== 1 || restored.records.activeSamples !== 1) {
|
||||
throw new Error('the restore must return the same record counts');
|
||||
}
|
||||
const restoredManifest = await readFile(path.join(restoreInto, 'manifest.jsonl'));
|
||||
if (!restoredManifest.equals(await readFile(path.join(dataRoot, 'manifest.jsonl')))) {
|
||||
throw new Error('the restore must return the same records, tombstone included');
|
||||
}
|
||||
return { receipt, manifest, reasons, verified, restored };
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
const opBackup = await operatorBackupAudit();
|
||||
await tap('[data-view="home"]', 400);
|
||||
const shortDigest = value => String(value).slice(0, 12);
|
||||
const backupRows = [
|
||||
['command', 'python3 scripts/operator_backup.py backup --source srv-timmy-corpus'],
|
||||
['stored', `${opBackup.receipt.files} members · ${opBackup.manifest.files.map(entry => entry.path.split('/').pop()).join(' + ')}`],
|
||||
['records', `samples ${opBackup.receipt.records.samples} · tombstones ${opBackup.receipt.records.tombstones} · active ${opBackup.receipt.records.activeSamples}`],
|
||||
['—', ''],
|
||||
['refused', `model-weights ${opBackup.reasons['model-weights']} · transient-inference ${opBackup.reasons['transient-inference']}`],
|
||||
['refused', `secret ${opBackup.reasons['secret']} · log ${opBackup.reasons['log']}`],
|
||||
['reason', 'every skipped path is named in the manifest, not silently dropped'],
|
||||
['—', ''],
|
||||
['digest', `sha256 ${shortDigest(opBackup.manifest.archive.sha256)}… · reproducible (sorted, mtime 0)`],
|
||||
['verify', `ok · ${opBackup.verified.files} members re-hashed against the manifest`],
|
||||
['restore', `ok · samples ${opBackup.restored.records.samples} · tombstones ${opBackup.restored.records.tombstones} · active ${opBackup.restored.records.activeSamples}`],
|
||||
['manifest', 'returned byte-for-byte, tombstone included'],
|
||||
['server', 'the app holds no journal of its own — there is no server copy to back up'],
|
||||
];
|
||||
const backupCard = await evidenceCard({
|
||||
id: 'production-backup',
|
||||
title: 'OPERATOR BACKUP',
|
||||
subtitle: 'command line, not app UI — real run, synthetic corpus',
|
||||
rows: backupRows,
|
||||
font: '600 9.5px/1.45 ui-monospace, SFMono-Regular, Menlo, monospace',
|
||||
keyWidth: 66,
|
||||
gap: '1px',
|
||||
rowGap: '6px',
|
||||
});
|
||||
await sleep(420);
|
||||
const backupProof = await evidenceCardProof('production-backup');
|
||||
if (!backupProof.ok) throw new Error(`Production backup card is not fully readable on screen: ${JSON.stringify(backupProof)}`);
|
||||
// The card is drawn from the run: prove the counts and the refusals on it are the ones it produced.
|
||||
if (!backupCard.text.includes(`samples ${opBackup.receipt.records.samples} · tombstones ${opBackup.receipt.records.tombstones}`)) {
|
||||
throw new Error('the card must show the record counts the backup actually reported');
|
||||
}
|
||||
if (!backupCard.text.includes('model-weights 1')) throw new Error('the card must show the exclusion the run actually recorded');
|
||||
if (!backupCard.text.includes(shortDigest(opBackup.manifest.archive.sha256))) {
|
||||
throw new Error('the card must show the archive digest the run actually wrote');
|
||||
}
|
||||
await caption('The operator backup is a command line, not a screen in the app — this runs it against a synthetic corpus the real ingest and revoke commands built', 3800);
|
||||
await caption('Model weights and transient inference images are refused by name, and every skipped path is listed with its reason', 3600);
|
||||
await caption('Verify re-hashes every stored member against the manifest, and the restore only starts once that check passes', 3400);
|
||||
await caption('A restore returns the same records, including the withdrawal tombstone a trainer must never lose', 3600);
|
||||
await page.evaluate(() => document.querySelector('#production-backup')?.remove());
|
||||
|
||||
// Back to the privacy view for the portability scene: the privacy entry lives on the journal view.
|
||||
await tap('[data-view="calendar"]', 550);
|
||||
await tap('[data-view="privacy"]', 600);
|
||||
|
||||
Reference in new issue
Block a user