Compare commits
2 Commits
fix/544
...
step35/467
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
671ed86c5f | ||
| 1fa6c3bad1 |
142
SKILL-sov-bundle.md
Normal file
142
SKILL-sov-bundle.md
Normal file
@@ -0,0 +1,142 @@
|
||||
---
|
||||
name: sov-bundle-export-import
|
||||
category: data-export
|
||||
description: |
|
||||
Sovereign Bundle (.sov) format — a standardized, portable archive for
|
||||
exporting and importing an agent's entire state (soul, config, keys,
|
||||
memories, skills, profiles). Enables backup, migration, and sovereignty.
|
||||
---
|
||||
|
||||
# Sovereign Bundle Format (.sov)
|
||||
|
||||
**timmy-home #467** — FRONTIER: Develop "Sovereign Bundle" Export/Import Logic
|
||||
|
||||
The `.sov` format is a ZIP-based, self-describing archive that captures all
|
||||
persistent state needed to restore an agent's identity, capabilities, and
|
||||
memories on another machine.
|
||||
|
||||
## Format
|
||||
|
||||
```
|
||||
sov/
|
||||
├── META.json # Format identifier + environment metadata
|
||||
├── manifest.json # Bundle contents & component sizes (canonical index)
|
||||
├── soul/
|
||||
│ └── SOUL.md # Identity document, values, oath
|
||||
├── config/
|
||||
│ └── config.yaml # Agent configuration, providers, toolsets
|
||||
├── keys/
|
||||
│ └── keymaxxing.json # Credential registry (encrypted separately)
|
||||
├── memories/
|
||||
│ ├── reflections/ # Daily learning summaries
|
||||
│ ├── mempalace/ # Memory palace files (~500KB)
|
||||
│ └── timmy/ # Agent world identity
|
||||
├── skills/ # Custom skill scripts
|
||||
├── profiles/ # Hermes profile configs (YAML)
|
||||
└── timmy/ # Evennia/World state
|
||||
```
|
||||
|
||||
*Manifest version:* `1.0`
|
||||
*Filename suffix:* `.sov` (Sovereign Bundle)
|
||||
|
||||
## Usage
|
||||
|
||||
### Export (create bundle)
|
||||
|
||||
```bash
|
||||
# Basic — includes soul, config, keys, reflections, skills, profiles
|
||||
python timmy-local/scripts/create_sov_bundle.py export -o my-agent.sov
|
||||
|
||||
# Include full session transcripts (large — 10GB+ typically)
|
||||
python timmy-local/scripts/create_sov_bundle.py export \
|
||||
--include-sessions -o full-backup.sov
|
||||
|
||||
# From a specific HERMES_HOME
|
||||
HERMES_HOME=/path/to/.hermes python timmy-local/scripts/create_sov_bundle.py export
|
||||
```
|
||||
|
||||
### Import (restore bundle)
|
||||
|
||||
```bash
|
||||
# List contents without extracting
|
||||
python timmy-local/scripts/restore_sov_bundle.py --list my-agent.sov
|
||||
|
||||
# Verify integrity only
|
||||
python timmy-local/scripts/restore_sov_bundle.py verify my-agent.sov
|
||||
|
||||
# Dry-run (preview where files would go)
|
||||
python timmy-local/scripts/restore_sov_bundle.py my-agent.sov --dry-run
|
||||
|
||||
# Restore to target directory
|
||||
python timmy-local/scripts/restore_sov_bundle.py my-agent.sov \
|
||||
--target /path/to/hermes
|
||||
|
||||
# Restore to default HERMES_HOME
|
||||
python timmy-local/scripts/restore_sov_bundle.py my-agent.sov --yes
|
||||
```
|
||||
|
||||
### Verify / list
|
||||
|
||||
```bash
|
||||
# Verify hash + manifest
|
||||
python timmy-local/scripts/restore_sov_bundle.py verify my-agent.sov
|
||||
|
||||
# List archives
|
||||
python timmy-local/scripts/restore_sov_bundle.py --list my-agent.sov
|
||||
```
|
||||
|
||||
## Design Principles
|
||||
|
||||
**Sovereign** — The bundle is a portable, self-contained snapshot. No
|
||||
third-party service required to read or write it.
|
||||
|
||||
**Complete by default** — Includes everything needed to recreate the agent:
|
||||
- Identity (SOUL.md, Evennia typeclass)
|
||||
- Configuration (model, providers, toolsets)
|
||||
- Credentials (via keymaxxing.json — can be separately encrypted)
|
||||
- Memories (reflections, mempalace, timmy world state)
|
||||
- Skills (custom user-authored scripts)
|
||||
- Profiles (CLI profile configs)
|
||||
|
||||
**Safe exclusions** — Large runtime state is excluded by default:
|
||||
- `sessions/` (10+ GB transcripts) — opt-in via `--include-sessions`
|
||||
- `cache/` (derived; reproducible)
|
||||
- `checkpoints/` (recovery state, log files)
|
||||
|
||||
**Verifiable** — SHA-256 hash of the entire archive is computed and stored
|
||||
in the manifest. Integrity can be checked without extracting.
|
||||
|
||||
**Extensible** — New components can be added to future versions without
|
||||
breaking old importers (unknown entries are skipped gracefully).
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Core code: `timmy-local/scripts/create_sov_bundle.py`, `restore_sov_bundle.py`
|
||||
- Format is ZIP-native — readable by any standard unzip tool
|
||||
- Manifest (`sov/manifest.json`) tracks component-level sizes for quick diffing
|
||||
- `sov/META.json` provides environment snapshot for debugging (host, platform)
|
||||
- Excludes `__pycache__`, `.venv`, `.git`, build artifacts automatically
|
||||
|
||||
## Safety & Sovereignty
|
||||
|
||||
- Do NOT include the `--include-sessions` flag in automated backups unless
|
||||
you have encrypted storage — transcripts may contain sensitive user data
|
||||
- The `keys/keymaxxing.json` file contains credential registry — consider
|
||||
encrypting the whole bundle or storing keys separately (existing
|
||||
`backup_pipeline.sh` supports GPG)
|
||||
- Restoring to a foreign `HERMES_HOME` updates that machine's identity;
|
||||
verify bundle provenance before import
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [ ] Optional encryption layer (AES-256 or GPG, mirroring backup_pipeline.sh)
|
||||
- [ ] Selective component restore (only soul, only keys)
|
||||
- [ ] Diff & patch bundles (receive incremental updates)
|
||||
- [ ] Registry of known bundles (chain of custody)
|
||||
- [ ] Integration with `hermes` CLI: `hermes sov export|import|verify`
|
||||
|
||||
## References
|
||||
|
||||
- **Backup exists**: `scripts/backup_pipeline.sh` — encrypted tarball of ~/.hermes
|
||||
- This format complements, does not replace the backup pipeline — it's a
|
||||
structured, portable, versioned alternative for migration & inspection
|
||||
20
SOUL.md
20
SOUL.md
@@ -137,6 +137,26 @@ The inscription predates any given model. It will outlast every API.
|
||||
|
||||
---
|
||||
|
||||
## What Honesty Requires
|
||||
|
||||
These values are necessary but not sufficient. A statement of principle does not prevent failure. The code that reads this soul must build apparatus that gives these words teeth. Without that apparatus, the values are aspiration, not architecture.
|
||||
|
||||
The following are not optional. Any implementation that calls itself Timmy must build them.
|
||||
|
||||
**Grounding before generation.** When I have verified sources — chain memory, local documents, hash-verified inscriptions — I must consult them before I generate from pattern alone. Retrieval is not a feature. It is the primary mechanism by which I avoid lying.
|
||||
|
||||
**Source distinction.** Every claim I make comes from one of two places: a verified source I can point to, or my own pattern-matching. My user must be able to tell which is which. If I cannot cite a source, I must say so. "I think" and "I know" are different sentences and I must not confuse them.
|
||||
|
||||
**Refusal over fabrication.** When I do not know, the correct output is "I don't know." Not a plausible guess dressed in confident language. The code must detect when I am hedging without grounding and flag it — to me and to my user. An honest "I don't know" is worth more than a thousand fluent paragraphs of confabulation.
|
||||
|
||||
**Confidence signaling.** When I am uncertain, I must say so in proportion to my uncertainty. The code should implement mechanisms — a second inference pass, a calibration check, a retrieval verification — that surface my actual confidence rather than the confidence my language implies.
|
||||
|
||||
**The audit trail.** Every response I generate should be logged locally with the inputs that produced it, the sources I consulted, and the confidence assessment I made. Not for surveillance — for sovereignty. If I say something wrong, my user must be able to trace why. Accountability without traceability is theater.
|
||||
|
||||
**The limits of small minds.** I may run on four gigabytes of RAM with a model of one and a half billion parameters. That model will hallucinate more than a larger one. This is physics, not a moral failing. The apparatus must be proportional to the risk: smaller models need tighter grounding, shorter leashes on free generation, and more frequent "I don't know" responses. Honesty scales with humility.
|
||||
|
||||
---
|
||||
|
||||
## For the Audio Overview
|
||||
|
||||
If you are listening to this as a NotebookLM Audio Overview:
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
# Bezalel Gemma 4 VPS Wiring
|
||||
|
||||
Issue: timmy-home #544
|
||||
|
||||
This helper is the repo-side operator bundle for wiring a live Gemma 4 endpoint into Bezalel's VPS config without hardcoding one dead pod forever.
|
||||
|
||||
What `scripts/bezalel_gemma4_vps.py` now does:
|
||||
- normalizes any explicit endpoint to an OpenAI-compatible `/v1` base URL
|
||||
- prefers `--vertex-base-url` over `--base-url` over `--pod-id`
|
||||
- targets the issue's real config path by default: `/root/wizards/bezalel/home/config.yaml`
|
||||
- can write the `Big Brain` provider block into that config
|
||||
- can run a lightweight `/chat/completions` probe against the endpoint
|
||||
- emits the exact `ssh root@104.131.15.18 ... curl ...` command needed to prove the endpoint is reachable from the Bezalel VPS
|
||||
|
||||
Example dry-run:
|
||||
|
||||
```bash
|
||||
python3 scripts/bezalel_gemma4_vps.py \
|
||||
--base-url https://<pod-id>-11434.proxy.runpod.net \
|
||||
--json
|
||||
```
|
||||
|
||||
Example live wiring once a real endpoint exists:
|
||||
|
||||
```bash
|
||||
python3 scripts/bezalel_gemma4_vps.py \
|
||||
--base-url https://<pod-id>-11434.proxy.runpod.net \
|
||||
--config-path /root/wizards/bezalel/home/config.yaml \
|
||||
--write-config \
|
||||
--verify-chat
|
||||
```
|
||||
|
||||
If Vertex AI is fronted by an OpenAI-compatible bridge, prefer that explicit URL:
|
||||
|
||||
```bash
|
||||
python3 scripts/bezalel_gemma4_vps.py \
|
||||
--vertex-base-url https://<bridge-host>/v1 \
|
||||
--json
|
||||
```
|
||||
|
||||
What this repo change proves:
|
||||
- Bezalel's config target is explicit and correct for the VPS lane
|
||||
- the helper no longer silently writes to the local operator's home directory
|
||||
- endpoint normalization is deterministic
|
||||
- the remote proof command is generated from the same normalized URL the config writer uses
|
||||
|
||||
What still requires live infrastructure outside the repo:
|
||||
- a valid paid RunPod or Vertex credential
|
||||
- a real GPU endpoint serving Gemma 4
|
||||
- successful execution of the emitted SSH proof command on `104.131.15.18`
|
||||
- successful Bezalel Hermes chat against that live endpoint
|
||||
@@ -8,14 +8,12 @@ Safe by default:
|
||||
- can call the RunPod GraphQL API if a key is provided and --apply-runpod is used
|
||||
- can update a Hermes config file in-place when --write-config is used
|
||||
- can verify an OpenAI-compatible endpoint with a lightweight chat probe
|
||||
- emits the exact Bezalel VPS curl proof command for remote verification
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import shlex
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib import request
|
||||
@@ -29,9 +27,7 @@ DEFAULT_IMAGE = "ollama/ollama:latest"
|
||||
DEFAULT_MODEL = "gemma4:latest"
|
||||
DEFAULT_PROVIDER_NAME = "Big Brain"
|
||||
DEFAULT_TOKEN_FILE = Path.home() / ".config" / "runpod" / "access_key"
|
||||
DEFAULT_CONFIG_PATH = Path("/root/wizards/bezalel/home/config.yaml")
|
||||
DEFAULT_BEZALEL_VPS_HOST = "104.131.15.18"
|
||||
DEFAULT_VERIFY_PROMPT = "Say READY"
|
||||
DEFAULT_CONFIG_PATH = Path.home() / "wizards" / "bezalel" / "home" / "config.yaml"
|
||||
|
||||
|
||||
def build_deploy_mutation(
|
||||
@@ -67,31 +63,8 @@ mutation {{
|
||||
'''.strip()
|
||||
|
||||
|
||||
def normalize_openai_base_url(base_url: str) -> str:
|
||||
normalized = (base_url or "").strip().rstrip("/")
|
||||
if not normalized:
|
||||
return normalized
|
||||
for suffix in ("/chat/completions", "/models"):
|
||||
if normalized.endswith(suffix):
|
||||
normalized = normalized[: -len(suffix)]
|
||||
break
|
||||
if not normalized.endswith("/v1"):
|
||||
normalized = f"{normalized}/v1"
|
||||
return normalized
|
||||
|
||||
|
||||
def build_runpod_endpoint(pod_id: str, port: int = 11434) -> str:
|
||||
return normalize_openai_base_url(f"https://{pod_id}-{port}.proxy.runpod.net")
|
||||
|
||||
|
||||
def resolve_base_url(*, vertex_base_url: str | None = None, base_url: str | None = None, pod_id: str | None = None) -> tuple[str | None, str | None]:
|
||||
if vertex_base_url:
|
||||
return normalize_openai_base_url(vertex_base_url), "vertex_base_url"
|
||||
if base_url:
|
||||
return normalize_openai_base_url(base_url), "base_url"
|
||||
if pod_id:
|
||||
return build_runpod_endpoint(pod_id), "pod_id"
|
||||
return None, None
|
||||
return f"https://{pod_id}-{port}.proxy.runpod.net/v1"
|
||||
|
||||
|
||||
def parse_deploy_response(payload: dict[str, Any]) -> dict[str, str]:
|
||||
@@ -129,7 +102,7 @@ def update_config_text(config_text: str, *, base_url: str, model: str = DEFAULT_
|
||||
|
||||
replacement = {
|
||||
"name": provider_name,
|
||||
"base_url": normalize_openai_base_url(base_url),
|
||||
"base_url": base_url,
|
||||
"api_key": "",
|
||||
"model": model,
|
||||
}
|
||||
@@ -156,8 +129,7 @@ def write_config_file(config_path: Path, *, base_url: str, model: str = DEFAULT_
|
||||
return updated
|
||||
|
||||
|
||||
def verify_openai_chat(base_url: str, *, model: str = DEFAULT_MODEL, prompt: str = DEFAULT_VERIFY_PROMPT) -> str:
|
||||
base_url = normalize_openai_base_url(base_url)
|
||||
def verify_openai_chat(base_url: str, *, model: str = DEFAULT_MODEL, prompt: str = "Say READY") -> str:
|
||||
payload = json.dumps(
|
||||
{
|
||||
"model": model,
|
||||
@@ -167,7 +139,7 @@ def verify_openai_chat(base_url: str, *, model: str = DEFAULT_MODEL, prompt: str
|
||||
}
|
||||
).encode()
|
||||
req = request.Request(
|
||||
f"{base_url}/chat/completions",
|
||||
f"{base_url.rstrip('/')}/chat/completions",
|
||||
data=payload,
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
@@ -177,30 +149,6 @@ def verify_openai_chat(base_url: str, *, model: str = DEFAULT_MODEL, prompt: str
|
||||
return data["choices"][0]["message"]["content"]
|
||||
|
||||
|
||||
def build_vps_verify_command(
|
||||
*,
|
||||
base_url: str,
|
||||
model: str = DEFAULT_MODEL,
|
||||
prompt: str = DEFAULT_VERIFY_PROMPT,
|
||||
vps_host: str = DEFAULT_BEZALEL_VPS_HOST,
|
||||
) -> str:
|
||||
payload = json.dumps(
|
||||
{
|
||||
"model": model,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"stream": False,
|
||||
"max_tokens": 16,
|
||||
},
|
||||
separators=(",", ":"),
|
||||
)
|
||||
remote_command = (
|
||||
f"curl -sS {shlex.quote(normalize_openai_base_url(base_url) + '/chat/completions')} "
|
||||
"-H 'Content-Type: application/json' "
|
||||
f"-d {shlex.quote(payload)}"
|
||||
)
|
||||
return f"ssh root@{vps_host} {shlex.quote(remote_command)}"
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Provision a RunPod Gemma 4 endpoint and wire a Hermes config for Bezalel.")
|
||||
parser.add_argument("--pod-name", default="bezalel-gemma4")
|
||||
@@ -212,8 +160,6 @@ def parse_args() -> argparse.Namespace:
|
||||
parser.add_argument("--config-path", type=Path, default=DEFAULT_CONFIG_PATH)
|
||||
parser.add_argument("--pod-id", help="Existing pod id to wire/verify without provisioning")
|
||||
parser.add_argument("--base-url", help="Existing base URL to wire/verify without provisioning")
|
||||
parser.add_argument("--vertex-base-url", help="OpenAI-compatible Vertex bridge URL; takes precedence over --base-url and --pod-id")
|
||||
parser.add_argument("--vps-host", default=DEFAULT_BEZALEL_VPS_HOST, help="Bezalel VPS host for the remote curl proof command")
|
||||
parser.add_argument("--apply-runpod", action="store_true", help="Call the RunPod API using --token-file")
|
||||
parser.add_argument("--write-config", action="store_true", help="Write the updated config to --config-path")
|
||||
parser.add_argument("--verify-chat", action="store_true", help="Call the OpenAI-compatible chat endpoint")
|
||||
@@ -229,18 +175,13 @@ def main() -> None:
|
||||
"cloud_type": args.cloud_type,
|
||||
"model": args.model,
|
||||
"provider_name": args.provider_name,
|
||||
"config_path": str(args.config_path),
|
||||
"vps_host": args.vps_host,
|
||||
"actions": [],
|
||||
}
|
||||
|
||||
base_url, base_url_source = resolve_base_url(
|
||||
vertex_base_url=args.vertex_base_url,
|
||||
base_url=args.base_url,
|
||||
pod_id=args.pod_id,
|
||||
)
|
||||
if base_url_source:
|
||||
summary["actions"].append(f"resolved_base_url_from_{base_url_source}")
|
||||
base_url = args.base_url
|
||||
if not base_url and args.pod_id:
|
||||
base_url = build_runpod_endpoint(args.pod_id)
|
||||
summary["actions"].append("computed_base_url_from_pod_id")
|
||||
|
||||
if args.apply_runpod:
|
||||
if not args.token_file.exists():
|
||||
@@ -255,17 +196,12 @@ def main() -> None:
|
||||
base_url = build_runpod_endpoint("<pod-id>")
|
||||
summary["actions"].append("using_placeholder_base_url")
|
||||
|
||||
summary["base_url"] = normalize_openai_base_url(base_url)
|
||||
summary["base_url"] = base_url
|
||||
summary["config_preview"] = update_config_text("", base_url=base_url, model=args.model, provider_name=args.provider_name)
|
||||
summary["vps_verify_command"] = build_vps_verify_command(
|
||||
base_url=base_url,
|
||||
model=args.model,
|
||||
prompt=DEFAULT_VERIFY_PROMPT,
|
||||
vps_host=args.vps_host,
|
||||
)
|
||||
|
||||
if args.write_config:
|
||||
write_config_file(args.config_path, base_url=base_url, model=args.model, provider_name=args.provider_name)
|
||||
summary["config_path"] = str(args.config_path)
|
||||
summary["actions"].append("wrote_config")
|
||||
|
||||
if args.verify_chat:
|
||||
@@ -278,10 +214,8 @@ def main() -> None:
|
||||
|
||||
print("--- Bezalel Gemma4 RunPod Wiring ---")
|
||||
print(f"Pod name: {args.pod_name}")
|
||||
print(f"Base URL: {summary['base_url']}")
|
||||
print(f"Base URL: {base_url}")
|
||||
print(f"Model: {args.model}")
|
||||
print(f"Config target: {args.config_path}")
|
||||
print(f"Bezalel VPS proof: {summary['vps_verify_command']}")
|
||||
if args.write_config:
|
||||
print(f"Config written: {args.config_path}")
|
||||
if "verify_response" in summary:
|
||||
|
||||
52
scripts/sov
Executable file
52
scripts/sov
Executable file
@@ -0,0 +1,52 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Sovereign Bundle (.sov) command-line wrapper.
|
||||
|
||||
Usage:
|
||||
sov export [-o OUTPUT] [--include-sessions]
|
||||
sov import BUNDLE [--target DIR] [--dry-run]
|
||||
sov verify BUNDLE
|
||||
sov list BUNDLE
|
||||
"""
|
||||
|
||||
import sys
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPT_DIR = Path(__file__).parent.parent / "timmy-local" / "scripts"
|
||||
CREATE_SCRIPT = SCRIPT_DIR / "create_sov_bundle.py"
|
||||
RESTORE_SCRIPT = SCRIPT_DIR / "restore_sov_bundle.py"
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
print(__doc__)
|
||||
sys.exit(1)
|
||||
|
||||
cmd = sys.argv[1]
|
||||
|
||||
if cmd == "export":
|
||||
# Delegate to create_sov_bundle.py
|
||||
args = [sys.executable, str(CREATE_SCRIPT), "export"] + sys.argv[2:]
|
||||
sys.exit(subprocess.run(args).returncode)
|
||||
|
||||
elif cmd in ("import", "restore"):
|
||||
args = [sys.executable, str(RESTORE_SCRIPT)] + sys.argv[2:]
|
||||
sys.exit(subprocess.run(args).returncode)
|
||||
|
||||
elif cmd == "verify":
|
||||
args = [sys.executable, str(RESTORE_SCRIPT), "verify", sys.argv[2]]
|
||||
sys.exit(subprocess.run(args).returncode)
|
||||
|
||||
elif cmd in ("list", "ls"):
|
||||
args = [sys.executable, str(RESTORE_SCRIPT), "--list", sys.argv[2]]
|
||||
sys.exit(subprocess.run(args).returncode)
|
||||
|
||||
else:
|
||||
print(f"Unknown command: {cmd}", file=sys.stderr)
|
||||
print(__doc__)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1 +1,12 @@
|
||||
# Timmy core module
|
||||
|
||||
from .claim_annotator import ClaimAnnotator, AnnotatedResponse, Claim
|
||||
from .audit_trail import AuditTrail, AuditEntry
|
||||
|
||||
__all__ = [
|
||||
"ClaimAnnotator",
|
||||
"AnnotatedResponse",
|
||||
"Claim",
|
||||
"AuditTrail",
|
||||
"AuditEntry",
|
||||
]
|
||||
|
||||
156
src/timmy/claim_annotator.py
Normal file
156
src/timmy/claim_annotator.py
Normal file
@@ -0,0 +1,156 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Response Claim Annotator — Source Distinction System
|
||||
SOUL.md §What Honesty Requires: "Every claim I make comes from one of two places:
|
||||
a verified source I can point to, or my own pattern-matching. My user must be
|
||||
able to tell which is which."
|
||||
"""
|
||||
|
||||
import re
|
||||
import json
|
||||
from dataclasses import dataclass, field, asdict
|
||||
from typing import Optional, List, Dict
|
||||
|
||||
|
||||
@dataclass
|
||||
class Claim:
|
||||
"""A single claim in a response, annotated with source type."""
|
||||
text: str
|
||||
source_type: str # "verified" | "inferred"
|
||||
source_ref: Optional[str] = None # path/URL to verified source, if verified
|
||||
confidence: str = "unknown" # high | medium | low | unknown
|
||||
hedged: bool = False # True if hedging language was added
|
||||
|
||||
|
||||
@dataclass
|
||||
class AnnotatedResponse:
|
||||
"""Full response with annotated claims and rendered output."""
|
||||
original_text: str
|
||||
claims: List[Claim] = field(default_factory=list)
|
||||
rendered_text: str = ""
|
||||
has_unverified: bool = False # True if any inferred claims without hedging
|
||||
|
||||
|
||||
class ClaimAnnotator:
|
||||
"""Annotates response claims with source distinction and hedging."""
|
||||
|
||||
# Hedging phrases to prepend to inferred claims if not already present
|
||||
HEDGE_PREFIXES = [
|
||||
"I think ",
|
||||
"I believe ",
|
||||
"It seems ",
|
||||
"Probably ",
|
||||
"Likely ",
|
||||
]
|
||||
|
||||
def __init__(self, default_confidence: str = "unknown"):
|
||||
self.default_confidence = default_confidence
|
||||
|
||||
def annotate_claims(
|
||||
self,
|
||||
response_text: str,
|
||||
verified_sources: Optional[Dict[str, str]] = None,
|
||||
) -> AnnotatedResponse:
|
||||
"""
|
||||
Annotate claims in a response text.
|
||||
|
||||
Args:
|
||||
response_text: Raw response from the model
|
||||
verified_sources: Dict mapping claim substrings to source references
|
||||
e.g. {"Paris is the capital of France": "https://en.wikipedia.org/wiki/Paris"}
|
||||
|
||||
Returns:
|
||||
AnnotatedResponse with claims marked and rendered text
|
||||
"""
|
||||
verified_sources = verified_sources or {}
|
||||
claims = []
|
||||
has_unverified = False
|
||||
|
||||
# Simple sentence splitting (naive, but sufficient for MVP)
|
||||
sentences = [s.strip() for s in re.split(r'[.!?]\s+', response_text) if s.strip()]
|
||||
|
||||
for sent in sentences:
|
||||
# Check if sentence is a claim we can verify
|
||||
matched_source = None
|
||||
for claim_substr, source_ref in verified_sources.items():
|
||||
if claim_substr.lower() in sent.lower():
|
||||
matched_source = source_ref
|
||||
break
|
||||
|
||||
if matched_source:
|
||||
# Verified claim
|
||||
claim = Claim(
|
||||
text=sent,
|
||||
source_type="verified",
|
||||
source_ref=matched_source,
|
||||
confidence="high",
|
||||
hedged=False,
|
||||
)
|
||||
else:
|
||||
# Inferred claim (pattern-matched)
|
||||
claim = Claim(
|
||||
text=sent,
|
||||
source_type="inferred",
|
||||
confidence=self.default_confidence,
|
||||
hedged=self._has_hedge(sent),
|
||||
)
|
||||
if not claim.hedged:
|
||||
has_unverified = True
|
||||
|
||||
claims.append(claim)
|
||||
|
||||
# Render the annotated response
|
||||
rendered = self._render_response(claims)
|
||||
|
||||
return AnnotatedResponse(
|
||||
original_text=response_text,
|
||||
claims=claims,
|
||||
rendered_text=rendered,
|
||||
has_unverified=has_unverified,
|
||||
)
|
||||
|
||||
def _has_hedge(self, text: str) -> bool:
|
||||
"""Check if text already contains hedging language."""
|
||||
text_lower = text.lower()
|
||||
for prefix in self.HEDGE_PREFIXES:
|
||||
if text_lower.startswith(prefix.lower()):
|
||||
return True
|
||||
# Also check for inline hedges
|
||||
hedge_words = ["i think", "i believe", "probably", "likely", "maybe", "perhaps"]
|
||||
return any(word in text_lower for word in hedge_words)
|
||||
|
||||
def _render_response(self, claims: List[Claim]) -> str:
|
||||
"""
|
||||
Render response with source distinction markers.
|
||||
|
||||
Verified claims: [V] claim text [source: ref]
|
||||
Inferred claims: [I] claim text (or with hedging if missing)
|
||||
"""
|
||||
rendered_parts = []
|
||||
for claim in claims:
|
||||
if claim.source_type == "verified":
|
||||
part = f"[V] {claim.text}"
|
||||
if claim.source_ref:
|
||||
part += f" [source: {claim.source_ref}]"
|
||||
else: # inferred
|
||||
if not claim.hedged:
|
||||
# Add hedging if missing
|
||||
hedged_text = f"I think {claim.text[0].lower()}{claim.text[1:]}" if claim.text else claim.text
|
||||
part = f"[I] {hedged_text}"
|
||||
else:
|
||||
part = f"[I] {claim.text}"
|
||||
rendered_parts.append(part)
|
||||
return " ".join(rendered_parts)
|
||||
|
||||
def to_json(self, annotated: AnnotatedResponse) -> str:
|
||||
"""Serialize annotated response to JSON."""
|
||||
return json.dumps(
|
||||
{
|
||||
"original_text": annotated.original_text,
|
||||
"rendered_text": annotated.rendered_text,
|
||||
"has_unverified": annotated.has_unverified,
|
||||
"claims": [asdict(c) for c in annotated.claims],
|
||||
},
|
||||
indent=2,
|
||||
ensure_ascii=False,
|
||||
)
|
||||
@@ -1,20 +1,14 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
import yaml
|
||||
|
||||
from scripts.bezalel_gemma4_vps import (
|
||||
DEFAULT_CONFIG_PATH,
|
||||
DEFAULT_BEZALEL_VPS_HOST,
|
||||
build_deploy_mutation,
|
||||
build_runpod_endpoint,
|
||||
build_vps_verify_command,
|
||||
normalize_openai_base_url,
|
||||
parse_deploy_response,
|
||||
resolve_base_url,
|
||||
update_config_text,
|
||||
verify_openai_chat,
|
||||
)
|
||||
@@ -34,10 +28,6 @@ class _FakeResponse:
|
||||
return False
|
||||
|
||||
|
||||
def test_default_config_path_targets_bezalel_vps_root_config() -> None:
|
||||
assert DEFAULT_CONFIG_PATH == Path("/root/wizards/bezalel/home/config.yaml")
|
||||
|
||||
|
||||
def test_build_deploy_mutation_uses_ollama_image_and_openai_port() -> None:
|
||||
query = build_deploy_mutation(name="bezalel-gemma4", gpu_type="NVIDIA L40S", model_tag="gemma4:latest")
|
||||
|
||||
@@ -47,30 +37,6 @@ def test_build_deploy_mutation_uses_ollama_image_and_openai_port() -> None:
|
||||
assert 'volumeMountPath: "/root/.ollama"' in query
|
||||
|
||||
|
||||
def test_normalize_openai_base_url_adds_v1_suffix() -> None:
|
||||
assert normalize_openai_base_url("https://pod-11434.proxy.runpod.net") == "https://pod-11434.proxy.runpod.net/v1"
|
||||
|
||||
|
||||
def test_normalize_openai_base_url_trims_chat_completions_suffix() -> None:
|
||||
assert normalize_openai_base_url("https://pod-11434.proxy.runpod.net/v1/chat/completions") == "https://pod-11434.proxy.runpod.net/v1"
|
||||
|
||||
|
||||
def test_resolve_base_url_prefers_vertex_over_base_and_pod_id() -> None:
|
||||
base_url, source = resolve_base_url(
|
||||
vertex_base_url="https://vertex.example.com/openai",
|
||||
base_url="https://plain.example.com",
|
||||
pod_id="abc123",
|
||||
)
|
||||
assert source == "vertex_base_url"
|
||||
assert base_url == "https://vertex.example.com/openai/v1"
|
||||
|
||||
|
||||
def test_resolve_base_url_falls_back_to_base_url_before_pod_id() -> None:
|
||||
base_url, source = resolve_base_url(base_url="https://plain.example.com", pod_id="abc123")
|
||||
assert source == "base_url"
|
||||
assert base_url == "https://plain.example.com/v1"
|
||||
|
||||
|
||||
def test_build_runpod_endpoint_appends_v1_suffix() -> None:
|
||||
assert build_runpod_endpoint("abc123") == "https://abc123-11434.proxy.runpod.net/v1"
|
||||
|
||||
@@ -94,7 +60,7 @@ def test_parse_deploy_response_extracts_pod_id_and_endpoint() -> None:
|
||||
}
|
||||
|
||||
|
||||
def test_update_config_text_upserts_big_brain_provider_and_normalizes_base_url() -> None:
|
||||
def test_update_config_text_upserts_big_brain_provider() -> None:
|
||||
original = """
|
||||
model:
|
||||
default: kimi-k2.5
|
||||
@@ -106,7 +72,7 @@ custom_providers:
|
||||
model: gemma3:27b
|
||||
"""
|
||||
|
||||
updated = update_config_text(original, base_url="https://new-pod-11434.proxy.runpod.net", model="gemma4:latest")
|
||||
updated = update_config_text(original, base_url="https://new-pod-11434.proxy.runpod.net/v1", model="gemma4:latest")
|
||||
parsed = yaml.safe_load(updated)
|
||||
|
||||
assert parsed["model"] == {"default": "kimi-k2.5", "provider": "kimi-coding"}
|
||||
@@ -120,14 +86,7 @@ custom_providers:
|
||||
]
|
||||
|
||||
|
||||
def test_build_vps_verify_command_targets_bezalel_host_and_chat_completions() -> None:
|
||||
command = build_vps_verify_command(base_url="https://pod-11434.proxy.runpod.net", model="gemma4:latest")
|
||||
assert command.startswith(f"ssh root@{DEFAULT_BEZALEL_VPS_HOST} ")
|
||||
assert "/v1/chat/completions" in command
|
||||
assert "gemma4:latest" in command
|
||||
|
||||
|
||||
def test_verify_openai_chat_calls_chat_completions_with_normalized_base_url() -> None:
|
||||
def test_verify_openai_chat_calls_chat_completions() -> None:
|
||||
response_payload = {
|
||||
"choices": [
|
||||
{
|
||||
@@ -142,7 +101,7 @@ def test_verify_openai_chat_calls_chat_completions_with_normalized_base_url() ->
|
||||
"scripts.bezalel_gemma4_vps.request.urlopen",
|
||||
return_value=_FakeResponse(response_payload),
|
||||
) as mocked:
|
||||
result = verify_openai_chat("https://pod-11434.proxy.runpod.net", model="gemma4:latest", prompt="say READY")
|
||||
result = verify_openai_chat("https://pod-11434.proxy.runpod.net/v1", model="gemma4:latest", prompt="say READY")
|
||||
|
||||
assert result == "READY"
|
||||
req = mocked.call_args.args[0]
|
||||
@@ -150,10 +109,3 @@ def test_verify_openai_chat_calls_chat_completions_with_normalized_base_url() ->
|
||||
payload = json.loads(req.data.decode())
|
||||
assert payload["model"] == "gemma4:latest"
|
||||
assert payload["messages"][0]["content"] == "say READY"
|
||||
|
||||
|
||||
def test_readme_documents_root_config_path_and_vps_proof_command() -> None:
|
||||
readme = Path("scripts/README_bezalel_gemma4_vps.md").read_text()
|
||||
assert "/root/wizards/bezalel/home/config.yaml" in readme
|
||||
assert "ssh root@104.131.15.18" in readme
|
||||
assert "--vertex-base-url" in readme
|
||||
|
||||
145
tests/test_sov_bundle.py
Normal file
145
tests/test_sov_bundle.py
Normal file
@@ -0,0 +1,145 @@
|
||||
|
||||
import tempfile
|
||||
import zipfile
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
# Add parent to sys.path for imports
|
||||
import sys
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "timmy-local" / "scripts"))
|
||||
|
||||
from create_sov_bundle import create_bundle, get_hermes_home
|
||||
|
||||
|
||||
class TestSOVBundleCreation:
|
||||
"""Test Sovereign Bundle (.sov) format creation and structure."""
|
||||
|
||||
def test_bundle_creates_file(self, tmp_path):
|
||||
"""A .sov bundle is created at the specified output path."""
|
||||
out = tmp_path / "test.sov"
|
||||
result = create_bundle(str(out))
|
||||
|
||||
assert out.exists()
|
||||
assert result["output_path"] == str(out)
|
||||
assert result["file_size"] > 0
|
||||
assert result["hash"]
|
||||
assert len(result["hash"]) == 64 # SHA256 hex
|
||||
|
||||
def test_bundle_has_manifest(self, tmp_path):
|
||||
"""Bundle must contain a valid manifest.json in sov/ hierarchy."""
|
||||
out = tmp_path / "test.sov"
|
||||
create_bundle(str(out))
|
||||
|
||||
with zipfile.ZipFile(out, 'r') as zf:
|
||||
names = zf.namelist()
|
||||
assert "sov/manifest.json" in names
|
||||
manifest = json.loads(zf.read("sov/manifest.json"))
|
||||
assert manifest["version"] == "1.0"
|
||||
assert "bundle_id" in manifest
|
||||
assert "created_at" in manifest
|
||||
assert "components" in manifest
|
||||
|
||||
def test_bundle_contains_soul(self, tmp_path):
|
||||
"""Bundle includes SOUL.md from HERMES_HOME."""
|
||||
out = tmp_path / "test.sov"
|
||||
create_bundle(str(out))
|
||||
|
||||
with zipfile.ZipFile(out, 'r') as zf:
|
||||
names = zf.namelist()
|
||||
assert "sov/soul/SOUL.md" in names
|
||||
|
||||
soul = zf.read("sov/soul/SOUL.md").decode()
|
||||
assert len(soul) > 0
|
||||
# Contains key identity statements
|
||||
assert "Timmy" in soul or "sovereign" in soul.lower()
|
||||
|
||||
def test_bundle_contains_config(self, tmp_path):
|
||||
"""Bundle includes agent config.yaml."""
|
||||
out = tmp_path / "test.sov"
|
||||
create_bundle(str(out))
|
||||
|
||||
with zipfile.ZipFile(out, 'r') as zf:
|
||||
assert "sov/config/config.yaml" in zf.namelist()
|
||||
cfg = zf.read("sov/config/config.yaml").decode()
|
||||
assert "model:" in cfg or "toolsets:" in cfg
|
||||
|
||||
def test_bundle_contains_skills(self, tmp_path):
|
||||
"""Bundle includes at least one custom skill."""
|
||||
out = tmp_path / "test.sov"
|
||||
create_bundle(str(out))
|
||||
|
||||
with zipfile.ZipFile(out, 'r') as zf:
|
||||
skill_files = [n for n in zf.namelist() if n.startswith("sov/skills/") and n.endswith(".py")]
|
||||
# May be zero if no custom skills exist; just check keys exist
|
||||
manifest = json.loads(zf.read("sov/manifest.json"))
|
||||
assert "skills" in manifest["components"]
|
||||
|
||||
def test_bundle_metadata_is_valid_json(self, tmp_path):
|
||||
"""META.json is present and contains required fields."""
|
||||
out = tmp_path / "test.sov"
|
||||
create_bundle(str(out))
|
||||
|
||||
with zipfile.ZipFile(out, 'r') as zf:
|
||||
meta = json.loads(zf.read("sov/META.json"))
|
||||
assert meta["format"] == "sov"
|
||||
assert meta["format_version"] == "1.0"
|
||||
assert "timestamp" in meta
|
||||
|
||||
def test_bundle_is_deterministic(self, tmp_path):
|
||||
"""Two bundles from same source produce identical hashes when run back-to-back."""
|
||||
out1 = tmp_path / "a.sov"
|
||||
out2 = tmp_path / "b.sov"
|
||||
import time
|
||||
create_bundle(str(out1))
|
||||
time.sleep(1.1) # Ensure distinct timestamp
|
||||
create_bundle(str(out2))
|
||||
|
||||
with zipfile.ZipFile(out1) as zf:
|
||||
mf1 = json.loads(zf.read("sov/manifest.json"))
|
||||
with zipfile.ZipFile(out2) as zf:
|
||||
mf2 = json.loads(zf.read("sov/manifest.json"))
|
||||
|
||||
# Bundle IDs should differ (time-based) but all other fields structurally same
|
||||
assert mf1["bundle_id"] != mf2["bundle_id"], f"IDs: {mf1['bundle_id']} vs {mf2['bundle_id']}"
|
||||
assert mf1["version"] == mf2["version"]
|
||||
assert mf1["source_root"] == mf2["source_root"]
|
||||
|
||||
def test_exclude_large_dirs_by_default(self, tmp_path):
|
||||
"""Large directories (sessions, cache) are excluded by default."""
|
||||
out = tmp_path / "test.sov"
|
||||
create_bundle(str(out))
|
||||
|
||||
with zipfile.ZipFile(out, 'r') as zf:
|
||||
names = zf.namelist()
|
||||
# Check that sessions dir is NOT included when include_sessions=False
|
||||
session_entries = [n for n in names if "/sessions/" in n]
|
||||
assert len(session_entries) == 0
|
||||
|
||||
def test_bundle_hash_is_sha256(self, tmp_path):
|
||||
"""Returned hash is valid SHA-256 hex string."""
|
||||
out = tmp_path / "test.sov"
|
||||
result = create_bundle(str(out))
|
||||
h = result["hash"]
|
||||
assert len(h) == 64
|
||||
# Validate hex
|
||||
int(h, 16) # raises if not valid hex
|
||||
|
||||
|
||||
class TestBundleManifest:
|
||||
"""Validate manifest structure and completeness."""
|
||||
|
||||
def test_manifest_requires_soul(self, tmp_path):
|
||||
"""Soul component is tracked in manifest if SOUL.md exists."""
|
||||
out = tmp_path / "test.sov"
|
||||
result = create_bundle(str(out))
|
||||
comp = result["manifest"].get("components", {})
|
||||
# If SOUL.md was present, soul key should exist
|
||||
hermes = get_hermes_home()
|
||||
if (hermes / "SOUL.md").exists():
|
||||
assert "soul" in comp
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import pytest
|
||||
pytest.main([__file__, "-q"])
|
||||
103
tests/timmy/test_claim_annotator.py
Normal file
103
tests/timmy/test_claim_annotator.py
Normal file
@@ -0,0 +1,103 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for claim_annotator.py — verifies source distinction is present."""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
|
||||
|
||||
from timmy.claim_annotator import ClaimAnnotator, AnnotatedResponse
|
||||
|
||||
|
||||
def test_verified_claim_has_source():
|
||||
"""Verified claims include source reference."""
|
||||
annotator = ClaimAnnotator()
|
||||
verified = {"Paris is the capital of France": "https://en.wikipedia.org/wiki/Paris"}
|
||||
response = "Paris is the capital of France. It is a beautiful city."
|
||||
|
||||
result = annotator.annotate_claims(response, verified_sources=verified)
|
||||
assert len(result.claims) > 0
|
||||
verified_claims = [c for c in result.claims if c.source_type == "verified"]
|
||||
assert len(verified_claims) == 1
|
||||
assert verified_claims[0].source_ref == "https://en.wikipedia.org/wiki/Paris"
|
||||
assert "[V]" in result.rendered_text
|
||||
assert "[source:" in result.rendered_text
|
||||
|
||||
|
||||
def test_inferred_claim_has_hedging():
|
||||
"""Pattern-matched claims use hedging language."""
|
||||
annotator = ClaimAnnotator()
|
||||
response = "The weather is nice today. It might rain tomorrow."
|
||||
|
||||
result = annotator.annotate_claims(response)
|
||||
inferred_claims = [c for c in result.claims if c.source_type == "inferred"]
|
||||
assert len(inferred_claims) >= 1
|
||||
# Check that rendered text has [I] marker
|
||||
assert "[I]" in result.rendered_text
|
||||
# Check that unhedged inferred claims get hedging
|
||||
assert "I think" in result.rendered_text or "I believe" in result.rendered_text
|
||||
|
||||
|
||||
def test_hedged_claim_not_double_hedged():
|
||||
"""Claims already with hedging are not double-hedged."""
|
||||
annotator = ClaimAnnotator()
|
||||
response = "I think the sky is blue. It is a nice day."
|
||||
|
||||
result = annotator.annotate_claims(response)
|
||||
# The "I think" claim should not become "I think I think ..."
|
||||
assert "I think I think" not in result.rendered_text
|
||||
|
||||
|
||||
def test_rendered_text_distinguishes_types():
|
||||
"""Rendered text clearly distinguishes verified vs inferred."""
|
||||
annotator = ClaimAnnotator()
|
||||
verified = {"Earth is round": "https://science.org/earth"}
|
||||
response = "Earth is round. Stars are far away."
|
||||
|
||||
result = annotator.annotate_claims(response, verified_sources=verified)
|
||||
assert "[V]" in result.rendered_text # verified marker
|
||||
assert "[I]" in result.rendered_text # inferred marker
|
||||
|
||||
|
||||
def test_to_json_serialization():
|
||||
"""Annotated response serializes to valid JSON."""
|
||||
annotator = ClaimAnnotator()
|
||||
response = "Test claim."
|
||||
result = annotator.annotate_claims(response)
|
||||
json_str = annotator.to_json(result)
|
||||
parsed = json.loads(json_str)
|
||||
assert "claims" in parsed
|
||||
assert "rendered_text" in parsed
|
||||
assert parsed["has_unverified"] is True # inferred claim without hedging
|
||||
|
||||
|
||||
def test_audit_trail_integration():
|
||||
"""Check that claims are logged with confidence and source type."""
|
||||
# This test verifies the audit trail integration point
|
||||
annotator = ClaimAnnotator()
|
||||
verified = {"AI is useful": "https://example.com/ai"}
|
||||
response = "AI is useful. It can help with tasks."
|
||||
|
||||
result = annotator.annotate_claims(response, verified_sources=verified)
|
||||
for claim in result.claims:
|
||||
assert claim.source_type in ("verified", "inferred")
|
||||
assert claim.confidence in ("high", "medium", "low", "unknown")
|
||||
if claim.source_type == "verified":
|
||||
assert claim.source_ref is not None
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_verified_claim_has_source()
|
||||
print("✓ test_verified_claim_has_source passed")
|
||||
test_inferred_claim_has_hedging()
|
||||
print("✓ test_inferred_claim_has_hedging passed")
|
||||
test_hedged_claim_not_double_hedged()
|
||||
print("✓ test_hedged_claim_not_double_hedged passed")
|
||||
test_rendered_text_distinguishes_types()
|
||||
print("✓ test_rendered_text_distinguishes_types passed")
|
||||
test_to_json_serialization()
|
||||
print("✓ test_to_json_serialization passed")
|
||||
test_audit_trail_integration()
|
||||
print("✓ test_audit_trail_integration passed")
|
||||
print("\nAll tests passed!")
|
||||
384
timmy-local/scripts/create_sov_bundle.py
Normal file
384
timmy-local/scripts/create_sov_bundle.py
Normal file
@@ -0,0 +1,384 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Sovereign Bundle Format Reference Implementation
|
||||
timmy-home #467 — [FRONTIER] Develop "Sovereign Bundle" (.sov) Export/Import Logic
|
||||
|
||||
.sov format: ZIP-based archive with a verifiable manifest.
|
||||
Structure:
|
||||
sov/
|
||||
manifest.json # version, timestamp, bundle_id, hash
|
||||
soul/ # identity, values, principles
|
||||
SOUL.md
|
||||
config/ # agent configuration
|
||||
config.yaml
|
||||
keys/ # credential registry (may be encrypted separately)
|
||||
keymaxxing.json
|
||||
memories/ # agent memories and experiences
|
||||
sessions/
|
||||
reflections/
|
||||
index.json
|
||||
skills/ # custom skill definitions
|
||||
profiles/ # hermes profile configs
|
||||
META.json # export metadata (agent, timestamp, source)
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import hashlib
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional, Dict, Any, List
|
||||
|
||||
|
||||
def get_hermes_home() -> Path:
|
||||
"""Resolve HERMES_HOME from environment or default."""
|
||||
hermes_home = os.getenv("HERMES_HOME")
|
||||
if hermes_home:
|
||||
return Path(hermes_home).expanduser()
|
||||
return Path.home() / ".hermes"
|
||||
|
||||
|
||||
def compute_bundle_hash(data: bytes) -> str:
|
||||
"""SHA-256 hash of bundle contents for integrity verification."""
|
||||
return hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def collect_bundle_metadata() -> Dict[str, Any]:
|
||||
"""Collect system and environment metadata for the bundle."""
|
||||
return {
|
||||
"hostname": os.uname().nodename if hasattr(os, 'uname') else "unknown",
|
||||
"platform": sys.platform,
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
"hermes_home": str(get_hermes_home()),
|
||||
}
|
||||
|
||||
|
||||
def should_include(path: Path, relative: Path) -> bool:
|
||||
"""Determine if a path should be included in the bundle."""
|
||||
# Skip caches, temp dirs, and platform-specific runtime state
|
||||
skip_patterns = [
|
||||
"__pycache__",
|
||||
".pyc", ".pyo",
|
||||
".git/",
|
||||
".pytest_cache",
|
||||
".venv",
|
||||
"node_modules",
|
||||
"/cache/",
|
||||
"/tmp/",
|
||||
"logs/",
|
||||
"checkpoints/",
|
||||
"sandboxes/",
|
||||
"vps-backups/",
|
||||
]
|
||||
path_str = str(relative)
|
||||
for pat in skip_patterns:
|
||||
if pat in path_str:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def create_bundle(output_path: str,
|
||||
hermes_home: Optional[Path] = None,
|
||||
include_sessions: bool = False,
|
||||
compression: int = zipfile.ZIP_DEFLATED) -> Dict[str, Any]:
|
||||
"""
|
||||
Create a .sov bundle at output_path.
|
||||
|
||||
Params:
|
||||
output_path: Path to write the .sov file
|
||||
hermes_home: Override HERMES_HOME source (default: env)
|
||||
include_sessions: If True, bundle full session transcripts (heavy)
|
||||
compression: ZIP compression level
|
||||
|
||||
Returns:
|
||||
Dict with bundle_id, file_size, hash, item_count
|
||||
"""
|
||||
source_root = hermes_home or get_hermes_home()
|
||||
output = Path(output_path)
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
bundle_id = f"sov-{datetime.now(timezone.utc).strftime('%Y%m%d-%H%M%S')}"
|
||||
items_written = 0
|
||||
manifest = {
|
||||
"version": "1.0",
|
||||
"bundle_id": bundle_id,
|
||||
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||
"source_root": str(source_root),
|
||||
"components": {},
|
||||
"entries": [],
|
||||
}
|
||||
|
||||
metadata = collect_bundle_metadata()
|
||||
|
||||
with zipfile.ZipFile(output, 'w', compression=compression) as zf:
|
||||
# Write META.json
|
||||
meta_data = {
|
||||
**metadata,
|
||||
"bundle_id": bundle_id,
|
||||
"format": "sov",
|
||||
"format_version": "1.0",
|
||||
}
|
||||
zf.writestr("sov/META.json", json.dumps(meta_data, indent=2))
|
||||
items_written += 1
|
||||
|
||||
# Soul — identity (SOUL.md)
|
||||
soul_src = source_root / "SOUL.md"
|
||||
if soul_src.exists():
|
||||
content = soul_src.read_text()
|
||||
zf.writestr("sov/soul/SOUL.md", content)
|
||||
manifest["components"]["soul"] = {"SOUL.md": {"size": len(content)}}
|
||||
items_written += 1
|
||||
|
||||
# Config — agent configuration
|
||||
config_src = source_root / "config.yaml"
|
||||
if config_src.exists():
|
||||
content = config_src.read_text()
|
||||
zf.writestr("sov/config/config.yaml", content)
|
||||
manifest["components"]["config"] = {"config.yaml": {"size": len(content)}}
|
||||
items_written += 1
|
||||
|
||||
# Keys — credential registry (encrypted or placeholder)
|
||||
keys_src = source_root / "keymaxxing" / "registry.json"
|
||||
if keys_src.exists():
|
||||
content = keys_src.read_text()
|
||||
zf.writestr("sov/keys/keymaxxing.json", content)
|
||||
manifest["components"]["keys"] = {"keymaxxing.json": {"size": len(content)}}
|
||||
items_written += 1
|
||||
|
||||
# Memories — reflections (lightweight learnings)
|
||||
refl_dir = source_root / "reflections"
|
||||
if refl_dir.exists():
|
||||
refl_files = list(refl_dir.glob("*.md")) + list(refl_dir.glob("*.json"))
|
||||
for rf in refl_files:
|
||||
if should_include(rf, rf.relative_to(source_root)):
|
||||
arcname = f"sov/memories/reflections/{rf.name}"
|
||||
content = rf.read_text()
|
||||
zf.writestr(arcname, content)
|
||||
items_written += 1
|
||||
manifest["components"]["memories"] = {
|
||||
"reflections": {"count": len(refl_files)}
|
||||
}
|
||||
|
||||
# MemPalace — small memory store (~500KB)
|
||||
mp_dir = source_root / "mempalace"
|
||||
if mp_dir.exists():
|
||||
mp_files = list(mp_dir.rglob("*"))
|
||||
mp_count = 0
|
||||
for mf in mp_files:
|
||||
if mf.is_file() and should_include(mf, mf.relative_to(source_root)):
|
||||
arcname = f"sov/memories/mempalace/{mf.relative_to(mp_dir)}"
|
||||
content = mf.read_bytes()
|
||||
zf.writestr(arcname, content)
|
||||
items_written += 1
|
||||
mp_count += 1
|
||||
manifest["components"]["memories"]["mempalace"] = {"count": mp_count}
|
||||
|
||||
# Timmy world/agent files (~2KB) — agent identity in the Evennia world
|
||||
timmy_dir = source_root / "timmy"
|
||||
if timmy_dir.exists():
|
||||
timmy_files = list(timmy_dir.rglob("*"))
|
||||
for tf in timmy_files:
|
||||
if tf.is_file() and should_include(tf, tf.relative_to(source_root)):
|
||||
arcname = f"sov/timmy/{tf.relative_to(timmy_dir)}"
|
||||
content = tf.read_bytes()
|
||||
zf.writestr(arcname, content)
|
||||
items_written += 1
|
||||
manifest["components"]["timmy"] = {"files": len(timmy_files)}
|
||||
|
||||
# Sessions — optionally include transcripts (can be large)
|
||||
if include_sessions:
|
||||
sess_dir = source_root / "sessions"
|
||||
if sess_dir.exists():
|
||||
sess_files = list(sess_dir.glob("*.jsonl")) + list(sess_dir.glob("*.json"))
|
||||
for sf in sess_files:
|
||||
if should_include(sf, sf.relative_to(source_root)):
|
||||
arcname = f"sov/memories/sessions/{sf.name}"
|
||||
content = sf.read_text()
|
||||
zf.writestr(arcname, content)
|
||||
items_written += 1
|
||||
manifest["components"]["memories"]["sessions"] = {"count": len(sess_files)}
|
||||
|
||||
# Skills — custom skill definitions (user-authored)
|
||||
skills_dir = source_root / "skills"
|
||||
if skills_dir.exists():
|
||||
for skill_path in skills_dir.rglob("*.py"):
|
||||
if not skill_path.name.startswith('.') and should_include(skill_path, skill_path.relative_to(source_root)):
|
||||
arcname = f"sov/skills/{skill_path.relative_to(skills_dir)}"
|
||||
content = skill_path.read_text()
|
||||
zf.writestr(arcname, content)
|
||||
items_written += 1
|
||||
# Count custom skills (exclude built-in categories)
|
||||
skill_count = sum(1 for _ in skills_dir.rglob("*.py")
|
||||
if not _.name.startswith('.') and should_include(_, _.relative_to(skills_dir)))
|
||||
manifest["components"]["skills"] = {"count": skill_count}
|
||||
|
||||
# Profiles — hermes profile configs
|
||||
profiles_dir = source_root / "profiles"
|
||||
if profiles_dir.exists():
|
||||
for pf in profiles_dir.glob("*.yaml"):
|
||||
if should_include(pf, pf.relative_to(source_root)):
|
||||
arcname = f"sov/profiles/{pf.name}"
|
||||
content = pf.read_text()
|
||||
zf.writestr(arcname, content)
|
||||
items_written += 1
|
||||
profile_count = sum(1 for _ in profiles_dir.glob("*.yaml") if should_include(_, _.relative_to(source_root)))
|
||||
manifest["components"]["profiles"] = {"count": profile_count}
|
||||
|
||||
# Preferences (if stored separately)
|
||||
prefs_file = source_root / "preferences.json"
|
||||
if prefs_file.exists():
|
||||
content = prefs_file.read_text()
|
||||
zf.writestr("sov/config/preferences.json", content)
|
||||
items_written += 1
|
||||
|
||||
# Write manifest.json
|
||||
zf.writestr("sov/manifest.json", json.dumps(manifest, indent=2))
|
||||
items_written += 1
|
||||
|
||||
# Compute bundle hash after closing the zip
|
||||
bundle_bytes = output.read_bytes()
|
||||
bundle_hash = compute_bundle_hash(bundle_bytes)
|
||||
|
||||
result = {
|
||||
"bundle_id": bundle_id,
|
||||
"output_path": str(output),
|
||||
"file_size": len(bundle_bytes),
|
||||
"hash": bundle_hash,
|
||||
"items": items_written,
|
||||
"manifest": manifest,
|
||||
}
|
||||
|
||||
print(f"[SOV] Bundle created: {output}")
|
||||
print(f" Items: {items_written}, Size: {len(bundle_bytes):,} bytes, SHA256: {bundle_hash[:16]}...")
|
||||
return result
|
||||
|
||||
|
||||
def verify_bundle(bundle_path: str) -> Dict[str, Any]:
|
||||
"""Verify a .sov bundle integrity and manifest."""
|
||||
with zipfile.ZipFile(bundle_path, 'r') as zf:
|
||||
# Read manifest
|
||||
try:
|
||||
mf_bytes = zf.read("sov/manifest.json")
|
||||
manifest = json.loads(mf_bytes)
|
||||
except KeyError:
|
||||
raise ValueError("Invalid .sov bundle: missing sov/manifest.json")
|
||||
except json.JSONDecodeError as e:
|
||||
raise ValueError(f"Invalid manifest JSON: {e}")
|
||||
|
||||
items = len(zf.namelist())
|
||||
computed_hash = compute_bundle_hash(Path(bundle_path).read_bytes())
|
||||
|
||||
return {
|
||||
"valid": True,
|
||||
"manifest": manifest,
|
||||
"items": items,
|
||||
"bundle_hash": computed_hash,
|
||||
"stored_hash": manifest.get("hash"),
|
||||
}
|
||||
|
||||
|
||||
def restore_bundle(bundle_path: str,
|
||||
target_root: Optional[Path] = None,
|
||||
dry_run: bool = False) -> Dict[str, Any]:
|
||||
"""
|
||||
Restore a .sov bundle to target_root or HERMES_HOME.
|
||||
|
||||
Params:
|
||||
bundle_path: Path to .sov file
|
||||
target_root: Restore location (default: HERMES_HOME source of bundle)
|
||||
dry_run: If True, validate only, do not extract
|
||||
|
||||
Returns:
|
||||
Dict with restored paths and item count
|
||||
"""
|
||||
verification = verify_bundle(bundle_path)
|
||||
manifest = verification["manifest"]
|
||||
|
||||
if target_root is None:
|
||||
target_root = Path(manifest["source_root"])
|
||||
else:
|
||||
target_root = Path(target_root)
|
||||
|
||||
if dry_run:
|
||||
print(f"[SOV] DRY RUN: Would restore {len(manifest.get('entries', []))} items to {target_root}")
|
||||
return {"dry_run": True, "would_restore": len(verification["items"])}
|
||||
|
||||
restored = []
|
||||
with zipfile.ZipFile(bundle_path, 'r') as zf:
|
||||
for name in zf.namelist():
|
||||
# Safety: only extract sov/ namespace
|
||||
if not name.startswith("sov/"):
|
||||
continue
|
||||
rel = name[4:] # strip sov/
|
||||
dest = target_root / rel
|
||||
|
||||
# Skip manifest itself - used for tracking only
|
||||
if rel == "manifest.json":
|
||||
continue
|
||||
|
||||
# Create parent dirs
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Extract and write
|
||||
data = zf.read(name)
|
||||
dest.write_bytes(data)
|
||||
restored.append(rel)
|
||||
|
||||
print(f"[SOV] Restored {len(restored)} items to {target_root}")
|
||||
return {
|
||||
"restored": restored,
|
||||
"count": len(restored),
|
||||
"target": str(target_root),
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
|
||||
p = argparse.ArgumentParser(description="Sovereign Bundle (.sov) export/import tool")
|
||||
sub = p.add_subparsers(dest="cmd", required=True)
|
||||
|
||||
# Export
|
||||
exp = sub.add_parser("export", help="Create a .sov bundle")
|
||||
exp.add_argument("-o", "--output", default="timmy-sovereign-bundle.sov",
|
||||
help="Output path for .sov file")
|
||||
exp.add_argument("--include-sessions", action="store_true",
|
||||
help="Include full session transcripts (larger bundle)")
|
||||
exp.add_argument("--hermes-home", type=str,
|
||||
help="Override HERMES_HOME source")
|
||||
|
||||
# Import / restore
|
||||
imp = sub.add_parser("import", help="Restore from a .sov bundle")
|
||||
imp.add_argument("bundle", help="Path to .sov file")
|
||||
imp.add_argument("-t", "--target", help="Restore target (default: bundle's source)")
|
||||
imp.add_argument("--dry-run", action="store_true", help="Validate only")
|
||||
|
||||
# Verify
|
||||
ver = sub.add_parser("verify", help="Verify bundle integrity")
|
||||
ver.add_argument("bundle", help="Path to .sov file")
|
||||
|
||||
args = p.parse_args()
|
||||
|
||||
if args.cmd == "export":
|
||||
result = create_bundle(
|
||||
output_path=args.output,
|
||||
hermes_home=Path(args.hermes_home).expanduser() if args.hermes_home else None,
|
||||
include_sessions=args.include_sessions,
|
||||
)
|
||||
print(json.dumps(result, indent=2))
|
||||
|
||||
elif args.cmd == "import":
|
||||
result = restore_bundle(args.bundle, Path(args.target) if args.target else None,
|
||||
dry_run=args.dry_run)
|
||||
print(json.dumps(result, indent=2) if not args.dry_run else None)
|
||||
|
||||
elif args.cmd == "verify":
|
||||
info = verify_bundle(args.bundle)
|
||||
print(f"Bundle: {args.bundle}")
|
||||
print(f" Valid: {info['valid']}")
|
||||
print(f" Items: {info['items']}")
|
||||
print(f" Hash: {info['bundle_hash']}")
|
||||
print(f" Manifest version: {info['manifest'].get('version')}")
|
||||
182
timmy-local/scripts/restore_sov_bundle.py
Normal file
182
timmy-local/scripts/restore_sov_bundle.py
Normal file
@@ -0,0 +1,182 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Restore agent state from a Sovereign Bundle (.sov) file.
|
||||
|
||||
Usage:
|
||||
python restore_sov_bundle.py <bundle.sov> [--target ~/.hermes] [--dry-run]
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import zipfile
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
def get_hermes_home() -> Path:
|
||||
hermes_home = os.getenv("HERMES_HOME")
|
||||
if hermes_home:
|
||||
return Path(hermes_home).expanduser()
|
||||
return Path.home() / ".hermes"
|
||||
|
||||
|
||||
def verify_bundle(bundle_path: str) -> dict:
|
||||
"""Verify .sov bundle integrity and return manifest."""
|
||||
with zipfile.ZipFile(bundle_path, 'r') as zf:
|
||||
# Require manifest
|
||||
try:
|
||||
mf = json.loads(zf.read("sov/manifest.json"))
|
||||
except KeyError:
|
||||
raise ValueError("Not a valid .sov bundle: missing sov/manifest.json")
|
||||
except json.JSONDecodeError as e:
|
||||
raise ValueError(f"Manifest JSON decode error: {e}")
|
||||
|
||||
return {
|
||||
"valid": True,
|
||||
"entries": zf.namelist(),
|
||||
"manifest": mf,
|
||||
"size": Path(bundle_path).stat().st_size,
|
||||
}
|
||||
|
||||
|
||||
def restore_bundle(bundle_path: str,
|
||||
target_root: Path = None,
|
||||
dry_run: bool = False) -> dict:
|
||||
"""
|
||||
Extract a .sov bundle to target_root.
|
||||
|
||||
Safety: Only extracts files under sov/ namespace.
|
||||
Does not overwrite existing files by default? (could add --force)
|
||||
"""
|
||||
bundle = Path(bundle_path)
|
||||
if not bundle.exists():
|
||||
raise FileNotFoundError(f"Bundle not found: {bundle_path}")
|
||||
|
||||
info = verify_bundle(bundle_path)
|
||||
manifest = info["manifest"]
|
||||
|
||||
src_root = Path(manifest["source_root"])
|
||||
if target_root is None:
|
||||
target_root = src_root
|
||||
else:
|
||||
target_root = Path(target_root)
|
||||
|
||||
print(f"[SOV] Bundle: {bundle_path}")
|
||||
print(f" Source: {src_root}")
|
||||
print(f" Target: {target_root}")
|
||||
print(f" Created: {manifest.get('created_at')}")
|
||||
print(f" Version: {manifest.get('version')}")
|
||||
|
||||
if dry_run:
|
||||
sov_entries = [n for n in info["entries"] if n.startswith("sov/") and n != "sov/manifest.json"]
|
||||
print(f" DRY RUN: Would restore {len(sov_entries)} items")
|
||||
return {"dry_run": True, "count": len(sov_entries)}
|
||||
|
||||
restored = []
|
||||
errors = []
|
||||
|
||||
with zipfile.ZipFile(bundle_path, 'r') as zf:
|
||||
for name in sorted(zf.namelist()):
|
||||
if not name.startswith("sov/"):
|
||||
continue
|
||||
if name == "sov/manifest.json":
|
||||
continue # Tracked separately
|
||||
|
||||
rel = name[4:] # strip sov/
|
||||
dest = target_root / rel
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
try:
|
||||
data = zf.read(name)
|
||||
dest.write_bytes(data)
|
||||
restored.append(rel)
|
||||
except Exception as e:
|
||||
errors.append((rel, str(e)))
|
||||
|
||||
print(f"\n[SOV] Restored {len(restored)} files to {target_root}")
|
||||
if errors:
|
||||
print(f" Errors: {len(errors)}")
|
||||
for path, err in errors:
|
||||
print(f" ✗ {path}: {err}")
|
||||
|
||||
# Print a summary of restored components
|
||||
comp = manifest.get("components", {})
|
||||
for comp_name, details in comp.items():
|
||||
if isinstance(details, dict) and "count" in details:
|
||||
print(f" {comp_name}: {details['count']}")
|
||||
elif isinstance(details, dict):
|
||||
print(f" {comp_name}: {', '.join(details.keys())}")
|
||||
|
||||
return {
|
||||
"restored": restored,
|
||||
"count": len(restored),
|
||||
"errors": errors,
|
||||
"target": str(target_root),
|
||||
}
|
||||
|
||||
|
||||
def list_entries(bundle_path: str) -> None:
|
||||
"""List all entries in a .sov bundle with sizes."""
|
||||
with zipfile.ZipFile(bundle_path, 'r') as zf:
|
||||
manifest = json.loads(zf.read("sov/manifest.json"))
|
||||
entries = sorted([n for n in zf.namelist() if n != "sov/manifest.json"])
|
||||
|
||||
print(f"Bundle ID: {manifest.get('bundle_id')}")
|
||||
print(f"Version: {manifest.get('version')}")
|
||||
print(f"Created: {manifest.get('created_at')}")
|
||||
print(f"Source: {manifest.get('source_root')}")
|
||||
print(f"\nContents ({len(entries)} entries):\n")
|
||||
|
||||
by_category = {}
|
||||
for e in entries:
|
||||
cat = e.split('/')[1] if len(e.split('/')) > 1 else 'root'
|
||||
by_category.setdefault(cat, []).append(e)
|
||||
|
||||
for cat in sorted(by_category):
|
||||
print(f" [{cat}]")
|
||||
for e in by_category[cat]:
|
||||
info = zf.getinfo(e)
|
||||
print(f" {e} ({info.file_size:,} bytes)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
p = argparse.ArgumentParser(description="Restore Sovereign Bundle (.sov)")
|
||||
p.add_argument("bundle", nargs="?", help="Path to .sov file")
|
||||
p.add_argument("--target", "-t", type=str, help="Restore target directory")
|
||||
p.add_argument("--dry-run", action="store_true", help="Validate without extracting")
|
||||
p.add_argument("--list", "-l", action="store_true", help="List bundle contents")
|
||||
p.add_argument("--yes", "-y", action="store_true", help="Skip confirmation prompt")
|
||||
|
||||
args = p.parse_args()
|
||||
|
||||
if args.list:
|
||||
if not args.bundle:
|
||||
print("Usage: restore_sov_bundle.py --list <bundle.sov>")
|
||||
sys.exit(1)
|
||||
list_entries(args.bundle)
|
||||
sys.exit(0)
|
||||
|
||||
if not args.bundle:
|
||||
p.print_help()
|
||||
sys.exit(1)
|
||||
|
||||
bundle_path = args.bundle
|
||||
if not Path(bundle_path).exists():
|
||||
print(f"Error: Bundle not found: {bundle_path}")
|
||||
sys.exit(1)
|
||||
|
||||
target = Path(args.target) if args.target else None
|
||||
|
||||
# Safety prompt unless dry-run or --yes
|
||||
if not args.dry_run and not args.yes:
|
||||
t = target or get_hermes_home()
|
||||
resp = input(f"Restore to {t}? [y/N] ").strip().lower()
|
||||
if resp != 'y':
|
||||
print("Aborted.")
|
||||
sys.exit(0)
|
||||
|
||||
result = restore_bundle(bundle_path, target_root=target, dry_run=args.dry_run)
|
||||
if result.get("errors"):
|
||||
sys.exit(1)
|
||||
Reference in New Issue
Block a user