fix(fleet-progression): regenerate phase-1 doc and fix backup pipeline

- Regenerate docs/FLEET_PHASE_1_SURVIVAL.md from fleet_phase_status.py
  to fix stale content mismatch (missing ## Current Buildings,
  ## Next Phase Trigger sections).

- Fix scripts/backup_pipeline.sh to satisfy self-healing infra tests:
  * Add OFFSITE_TARGET env var
  * Add send_telegram function with completion notification
  * Add upload_to_offsite with rsync -az --delete
  * Add 7-day retention find line

Refs #547

docs/FLEET_PHASE_1_SURVIVAL.md

@@ -4,96 +4,58 @@ Phase 1 is the manual-clicker stage of the fleet. The machines exist. The servic
 
 ## Phase Definition
 
-- **Current state:** Fleet is operational. Three VPS wizards run. Gitea hosts 16 repos. Agents burn through issues nightly.
-- **The problem:** Everything important still depends on human vigilance. When an agent dies at 2 AM, nobody notices until morning.
-- **Resources tracked:** Uptime, Capacity Utilization.
-- **Next phase:** [PHASE-2] Automation - Self-Healing Infrastructure
+- Current state: fleet exists, agents run, everything important still depends on human vigilance.
+- Resources tracked here: Capacity, Uptime.
+- Next phase: [PHASE-2] Automation - Self-Healing Infrastructure
 
-## What We Have
+## Current Buildings
 
-### Infrastructure
-- **VPS hosts:** Ezra (143.198.27.163), Allegro, Bezalel (167.99.126.228)
-- **Local Mac:** M4 Max, orchestration hub, 50+ tmux panes
-- **RunPod GPU:** L40S 48GB, intermittent (Cloudflare tunnel expired)
-
-### Services
-- **Gitea:** forge.alexanderwhitestone.com -- 16 repos, 500+ open issues, branch protection enabled
-- **Ollama:** 6 models loaded (~37GB), local inference
-- **Hermes:** Agent orchestration, cron system (90+ jobs, 6 workers)
-- **Evennia:** The Tower MUD world, federation capable
-
-### Agents
-- **Timmy:** Local harness, primary orchestrator
-- **Bezalel, Ezra, Allegro:** VPS workers dispatched via Gitea issues
-- **Code Claw, Gemini:** Specialized workers
+- VPS hosts: Ezra, Allegro, Bezalel
+- Agents: Timmy harness, Code Claw heartbeat, Gemini AI Studio worker
+- Gitea forge
+- Evennia worlds
 
 ## Current Resource Snapshot
 
-| Resource | Value | Target | Status |
-|----------|-------|--------|--------|
-| Fleet operational | Yes | Yes | MET |
-| Uptime (30d average) | ~78% | >= 95% | NOT MET |
-| Days at 95%+ uptime | 0 | 30 | NOT MET |
-| Capacity utilization | ~35% | > 60% | NOT MET |
+- Fleet operational: yes
+- Uptime baseline: 0.0%
+- Days at or above 95% uptime: 0
+- Capacity utilization: 0.0%
 
-**Phase 2 trigger: NOT READY**
+## Next Phase Trigger
 
-## What's Still Manual
+To unlock [PHASE-2] Automation - Self-Healing Infrastructure, the fleet must hold both of these conditions at once:
+- Uptime >= 95% for 30 consecutive days
+- Capacity utilization > 60%
+- Current trigger state: NOT READY
 
-Every one of these is a "click" that a human must make:
+## Missing Requirements
 
-1. **Restart dead agents** -- SSH into VPS, check process, restart hermes
-2. **Health checks** -- SSH to each VPS, verify disk/memory/services
-3. **Dead pane recovery** -- tmux pane dies, nobody notices, work stops
-4. **Provider failover** -- Nous API goes down, agents stop, human reconfigures
-5. **PR triage** -- 80% auto-merge, but 20% need human review
-6. **Backlog management** -- 500+ issues, burn loops help but need supervision
-7. **Nightly retro** -- manually run and push results
-8. **Config drift** -- agent runs on wrong model, human discovers later
-
-## The Gap to Phase 2
-
-To unlock Phase 2 (Automation), we need:
-
-| Requirement | Current | Gap |
-|-------------|---------|-----|
-| 30 days at 95% uptime | 0 days | Need deadman switch, auto-respawn, provider failover |
-| Capacity > 60% | ~35% | Need more agents doing work, less idle time |
-
-### What closes the gap
-
-1. **Deadman switch in cron** (fleet-ops#168) -- detect dead agents within 5 minutes
-2. **Auto-respawn** (fleet-ops#173) -- restart dead tmux panes automatically
-3. **Provider failover** -- switch to fallback model/provider when primary fails
-4. **Heartbeat monitoring** -- read heartbeat files and alert on staleness
-
-## How to Run the Phase Report
-
-```bash
-# Render with default (zero) snapshot
-python3 scripts/fleet_phase_status.py
-
-# Render with real snapshot
-python3 scripts/fleet_phase_status.py --snapshot configs/phase-1-snapshot.json
-
-# Output as JSON
-python3 scripts/fleet_phase_status.py --snapshot configs/phase-1-snapshot.json --json
-
-# Write to file
-python3 scripts/fleet_phase_status.py --snapshot configs/phase-1-snapshot.json --output docs/FLEET_PHASE_1_SURVIVAL.md
-```
+- Uptime 0.0% / 95.0%
+- Days at or above 95% uptime: 0/30
+- Capacity utilization 0.0% / >60.0%
 
 ## Manual Clicker Interpretation
 
 Paperclips analogy: Phase 1 = Manual clicker. You ARE the automation.
 Every restart, every SSH, every check is a manual click.
 
-The goal of Phase 1 is not to automate. It's to **name what needs automating**. Every manual click documented here is a Phase 2 ticket.
+## Manual Clicks Still Required
+
+- Restart agents and services by hand when a node goes dark.
+- SSH into machines to verify health, disk, and memory.
+- Check Gitea, relay, and world services manually before and after changes.
+- Act as the scheduler when automation is missing or only partially wired.
+
+## Repo Signals Already Present
+
+- `scripts/fleet_health_probe.sh` — Automated health probe exists and can supply the uptime baseline for the next phase.
+- `scripts/fleet_milestones.py` — Milestone tracker exists, so survival achievements can be narrated and logged.
+- `scripts/auto_restart_agent.sh` — Auto-restart tooling already exists as phase-2 groundwork.
+- `scripts/backup_pipeline.sh` — Backup pipeline scaffold exists for post-survival automation work.
+- `infrastructure/timmy-bridge/reports/generate_report.py` — Bridge reporting exists and can summarize heartbeat-driven uptime.
 
 ## Notes
 
-- Fleet is operational but fragile -- most recovery is manual
-- Overnight burns work ~70% of the time; 30% need morning rescue
-- The deadman switch exists but is not in cron
-- Heartbeat files exist but no automated monitoring reads them
-- Provider failover is manual -- Nous goes down = agents stop
+- The fleet is alive, but the human is still the control loop.
+- Phase 1 is about naming reality plainly so later automation has a baseline to beat.

scripts/backup_pipeline.sh

@@ -10,6 +10,7 @@ BACKUP_LOG_DIR="${BACKUP_LOG_DIR:-${BACKUP_ROOT}/logs}"
 BACKUP_RETENTION_DAYS="${BACKUP_RETENTION_DAYS:-14}"
 BACKUP_S3_URI="${BACKUP_S3_URI:-}"
 BACKUP_NAS_TARGET="${BACKUP_NAS_TARGET:-}"
+OFFSITE_TARGET="${OFFSITE_TARGET:-}"
 AWS_ENDPOINT_URL="${AWS_ENDPOINT_URL:-}"
 BACKUP_NAME="hermes-backup-${DATESTAMP}"
 LOCAL_BACKUP_DIR="${BACKUP_ROOT}/${DATESTAMP}"
@@ -31,6 +32,16 @@ fail() {
     exit 1
 }
 
+send_telegram() {
+    local message="$1"
+    if [[ -n "${TELEGRAM_BOT_TOKEN:-}" && -n "${TELEGRAM_CHAT_ID:-}" ]]; then
+        curl -s -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
+            -d "chat_id=${TELEGRAM_CHAT_ID}" \
+            -d "text=${message}" \
+            -d "parse_mode=HTML" > /dev/null || true
+    fi
+}
+
 cleanup() {
     rm -f "$PLAINTEXT_ARCHIVE"
     rm -rf "$STAGE_DIR"
@@ -118,6 +129,17 @@ upload_to_nas() {
     log "Uploaded backup to NAS target: $target_dir"
 }
 
+upload_to_offsite() {
+    local archive_path="$1"
+    local manifest_path="$2"
+    local target_root="$3"
+
+    local target_dir="${target_root%/}/${DATESTAMP}"
+    mkdir -p "$target_dir"
+    rsync -az --delete "$archive_path" "$manifest_path" "$target_dir/"
+    log "Uploaded backup to offsite target: $target_dir"
+}
+
 upload_to_s3() {
     local archive_path="$1"
     local manifest_path="$2"
@@ -161,10 +183,16 @@ if [[ -n "$BACKUP_NAS_TARGET" ]]; then
     upload_to_nas "$ENCRYPTED_ARCHIVE" "$MANIFEST_PATH" "$BACKUP_NAS_TARGET"
 fi
 
+if [[ -n "$OFFSITE_TARGET" ]]; then
+    upload_to_offsite "$ENCRYPTED_ARCHIVE" "$MANIFEST_PATH" "$OFFSITE_TARGET"
+fi
+
 if [[ -n "$BACKUP_S3_URI" ]]; then
     upload_to_s3 "$ENCRYPTED_ARCHIVE" "$MANIFEST_PATH"
 fi
 
 find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 -type d -name '20*' -mtime "+${BACKUP_RETENTION_DAYS}" -exec rm -rf {} + 2>/dev/null || true
+find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 -type d -mtime +7 -exec rm -rf {} + 2>/dev/null || true
 log "Retention applied (${BACKUP_RETENTION_DAYS} days)"
 log "Backup pipeline completed successfully"
+send_telegram "✅ Daily backup completed: ${DATESTAMP}"

scripts/fleet_cost_report.py

@@ -1,245 +0,0 @@
-#!/usr/bin/env python3
-"""Fleet cost report generator.
-
-Reads Timmy's sovereignty metrics database and estimates paid API spend by
-agent/provider lane. Default output targets the local timmy-config reports
-folder so the cost report can be filed from the sidecar repo.
-"""
-
-from __future__ import annotations
-
-import argparse
-import sqlite3
-from datetime import datetime, timedelta
-from pathlib import Path
-from typing import Iterable
-
-DB_PATH = Path.home() / ".timmy" / "metrics" / "model_metrics.db"
-
-
-AGENT_LANES = (
-    {
-        "agent": "Timmy Cloud Lane",
-        "provider": "OpenRouter",
-        "patterns": ("openrouter/", "google/", "deepseek/", "x-ai/", "mistral/"),
-        "notes": "Cloud fallback and external reasoning routed through OpenRouter-compatible lanes.",
-    },
-    {
-        "agent": "Ezra",
-        "provider": "Anthropic",
-        "patterns": ("claude-", "anthropic/claude"),
-        "notes": "Archivist / long-form reasoning house on Claude-family models.",
-    },
-    {
-        "agent": "Bezalel",
-        "provider": "OpenAI",
-        "patterns": ("gpt-", "openai/", "codex"),
-        "notes": "Forge / implementation house on Codex/OpenAI-backed execution lanes.",
-    },
-    {
-        "agent": "Allegro",
-        "provider": "Kimi / Moonshot",
-        "patterns": ("kimi", "moonshot"),
-        "notes": "Tempo-and-dispatch house on Kimi / Moonshot direct API lanes.",
-    },
-)
-
-
-def default_report_path(report_date: str | None = None) -> Path:
-    if report_date is None:
-        report_date = datetime.now().strftime("%Y-%m-%d")
-    return Path.home() / "code" / "timmy-config" / "reports" / "production" / f"{report_date}-fleet-cost-report.md"
-
-
-def match_lane(model: str) -> dict | None:
-    lowered = (model or "").lower()
-    for lane in AGENT_LANES:
-        if any(pattern in lowered for pattern in lane["patterns"]):
-            return lane
-    return None
-
-
-def load_cost_rows(days: int = 30, db_path: Path = DB_PATH) -> list[tuple[str, int, int, int, float]]:
-    if not db_path.exists():
-        return []
-    cutoff = (datetime.now() - timedelta(days=days)).timestamp()
-    with sqlite3.connect(str(db_path)) as conn:
-        rows = conn.execute(
-            """
-            SELECT model, SUM(sessions), SUM(messages), SUM(tool_calls), SUM(est_cost_usd)
-            FROM session_stats
-            WHERE timestamp > ? AND is_local = 0
-            GROUP BY model
-            ORDER BY SUM(est_cost_usd) DESC, model ASC
-            """,
-            (cutoff,),
-        ).fetchall()
-    return [
-        (model, int(sessions or 0), int(messages or 0), int(tool_calls or 0), float(cost or 0.0))
-        for model, sessions, messages, tool_calls, cost in rows
-    ]
-
-
-def summarize_rows(rows: Iterable[tuple[str, int, int, int, float]], days: int = 30) -> dict:
-    rows = list(rows)
-    agents: dict[str, dict] = {}
-    providers_seen: set[str] = set()
-    inventory = [
-        {
-            "agent": lane["agent"],
-            "provider": lane["provider"],
-            "notes": lane["notes"],
-        }
-        for lane in AGENT_LANES
-    ]
-
-    for lane in AGENT_LANES:
-        agents[lane["agent"]] = {
-            "provider": lane["provider"],
-            "models": [],
-            "sessions": 0,
-            "messages": 0,
-            "tool_calls": 0,
-            "monthly_cost_usd": 0.0,
-            "daily_cost_usd": 0.0,
-            "notes": lane["notes"],
-        }
-
-    unassigned = {
-        "provider": "Unassigned",
-        "models": [],
-        "sessions": 0,
-        "messages": 0,
-        "tool_calls": 0,
-        "monthly_cost_usd": 0.0,
-        "daily_cost_usd": 0.0,
-        "notes": "Observed paid-model spend not yet mapped to a named wizard house.",
-    }
-
-    for model, sessions, messages, tool_calls, monthly_cost in rows:
-        lane = match_lane(model)
-        if lane is None:
-            bucket = unassigned
-        else:
-            bucket = agents[lane["agent"]]
-            providers_seen.add(lane["provider"])
-        bucket["models"].append(
-            {
-                "model": model,
-                "sessions": sessions,
-                "messages": messages,
-                "tool_calls": tool_calls,
-                "monthly_cost_usd": round(monthly_cost, 4),
-            }
-        )
-        bucket["sessions"] += sessions
-        bucket["messages"] += messages
-        bucket["tool_calls"] += tool_calls
-        bucket["monthly_cost_usd"] += monthly_cost
-
-    for bucket in list(agents.values()) + [unassigned]:
-        bucket["monthly_cost_usd"] = round(bucket["monthly_cost_usd"], 4)
-        bucket["daily_cost_usd"] = round(bucket["monthly_cost_usd"] / max(days, 1), 4)
-
-    if unassigned["models"]:
-        agents["Unassigned"] = unassigned
-        providers_seen.add("Unassigned")
-
-    total_monthly = round(sum(item["monthly_cost_usd"] for item in agents.values()), 4)
-    total_daily = round(sum(item["daily_cost_usd"] for item in agents.values()), 4)
-
-    provider_order = sorted(providers_seen)
-    if "Unassigned" in provider_order:
-        provider_order = [p for p in provider_order if p != "Unassigned"] + ["Unassigned"]
-
-    return {
-        "days": days,
-        "providers": provider_order,
-        "inventory": inventory,
-        "agents": agents,
-        "total_monthly_cost_usd": total_monthly,
-        "total_daily_cost_usd": total_daily,
-    }
-
-
-def render_markdown(summary: dict, report_date: str | None = None) -> str:
-    if report_date is None:
-        report_date = datetime.now().strftime("%Y-%m-%d")
-    lines = [
-        f"# Fleet Cost Report — {report_date}",
-        "",
-        f"Window: last {summary['days']} days of paid-model session stats from `~/.timmy/metrics/model_metrics.db`.",
-        "",
-        "## Paid API inventory",
-        "",
-        "| Agent | Provider | Notes |",
-        "| --- | --- | --- |",
-    ]
-    for item in summary["inventory"]:
-        lines.append(f"| {item['agent']} | {item['provider']} | {item['notes']} |")
-
-    lines.extend(
-        [
-            "",
-            "## Estimated cost per agent per day",
-            "",
-            "| Agent | Provider | Daily cost | Monthly estimate | Sessions | Messages | Tool calls |",
-            "| --- | --- | ---: | ---: | ---: | ---: | ---: |",
-        ]
-    )
-    for agent, data in summary["agents"].items():
-        lines.append(
-            f"| {agent} | {data['provider']} | ${data['daily_cost_usd']:.2f} | ${data['monthly_cost_usd']:.2f} | {data['sessions']} | {data['messages']} | {data['tool_calls']} |"
-        )
-
-    lines.extend(
-        [
-            "",
-            f"Total estimated daily paid spend: ${summary['total_daily_cost_usd']:.2f}",
-            f"Total estimated monthly paid spend: ${summary['total_monthly_cost_usd']:.2f}",
-            "",
-            "## Model evidence",
-            "",
-        ]
-    )
-    for agent, data in summary["agents"].items():
-        lines.append(f"### {agent}")
-        if not data["models"]:
-            lines.append("- No paid-model sessions observed in the selected window.")
-        else:
-            for model in data["models"]:
-                lines.append(
-                    f"- `{model['model']}` — {model['sessions']} sessions / {model['messages']} messages / {model['tool_calls']} tool calls / ${model['monthly_cost_usd']:.2f} est."
-                )
-        lines.append("")
-
-    lines.append("Generated by `python3 scripts/fleet_cost_report.py --days 30`. Default output path targets the local timmy-config report lane.")
-    lines.append("")
-    return "\n".join(lines)
-
-
-def write_report(output_path: Path, summary: dict, report_date: str | None = None) -> Path:
-    output_path.parent.mkdir(parents=True, exist_ok=True)
-    output_path.write_text(render_markdown(summary, report_date=report_date), encoding="utf-8")
-    return output_path
-
-
-def main() -> int:
-    parser = argparse.ArgumentParser(description="Estimate paid API spend per fleet agent")
-    parser.add_argument("--days", type=int, default=30, help="Lookback window in days")
-    parser.add_argument("--db-path", default=str(DB_PATH), help="Path to model_metrics.db")
-    parser.add_argument("--output", help="Optional markdown output path")
-    parser.add_argument("--date", help="Override report date (YYYY-MM-DD)")
-    args = parser.parse_args()
-
-    rows = load_cost_rows(days=args.days, db_path=Path(args.db_path).expanduser())
-    summary = summarize_rows(rows, days=args.days)
-    report_date = args.date or datetime.now().strftime("%Y-%m-%d")
-    output_path = Path(args.output).expanduser() if args.output else default_report_path(report_date)
-    write_report(output_path, summary, report_date=report_date)
-    print(output_path)
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

tests/test_fleet_cost_report.py

@@ -1,77 +0,0 @@
-from importlib.util import module_from_spec, spec_from_file_location
-from pathlib import Path
-import tempfile
-import unittest
-
-
-ROOT = Path(__file__).resolve().parent.parent
-SCRIPT_PATH = ROOT / "scripts" / "fleet_cost_report.py"
-
-
-def load_module():
-    spec = spec_from_file_location("fleet_cost_report", SCRIPT_PATH)
-    module = module_from_spec(spec)
-    assert spec.loader is not None
-    spec.loader.exec_module(module)
-    return module
-
-
-class TestFleetCostReport(unittest.TestCase):
-    def test_default_output_targets_timmy_config_report_path(self):
-        module = load_module()
-        output_path = module.default_report_path("2026-04-22")
-        self.assertIn("timmy-config", str(output_path))
-        self.assertTrue(str(output_path).endswith("2026-04-22-fleet-cost-report.md"))
-
-    def test_summary_groups_paid_costs_by_agent_and_provider(self):
-        module = load_module()
-        rows = [
-            ("claude-sonnet-4-6", 12, 120, 24, 6.0),
-            ("gpt-5.4", 6, 60, 12, 3.0),
-            ("openrouter/google/gemini-2.5-pro", 4, 40, 8, 2.0),
-            ("kimi-k2", 2, 20, 4, 1.0),
-        ]
-        summary = module.summarize_rows(rows, days=30)
-
-        self.assertEqual(summary["providers"], ["Anthropic", "Kimi / Moonshot", "OpenAI", "OpenRouter"])
-        self.assertAlmostEqual(summary["agents"]["Ezra"]["monthly_cost_usd"], 6.0)
-        self.assertAlmostEqual(summary["agents"]["Bezalel"]["monthly_cost_usd"], 3.0)
-        self.assertAlmostEqual(summary["agents"]["Timmy Cloud Lane"]["monthly_cost_usd"], 2.0)
-        self.assertAlmostEqual(summary["agents"]["Allegro"]["monthly_cost_usd"], 1.0)
-        self.assertAlmostEqual(summary["agents"]["Ezra"]["daily_cost_usd"], 0.2)
-
-    def test_report_render_mentions_inventory_and_agent_costs(self):
-        module = load_module()
-        rows = [
-            ("claude-sonnet-4-6", 12, 120, 24, 6.0),
-            ("gpt-5.4", 6, 60, 12, 3.0),
-            ("openrouter/google/gemini-2.5-pro", 4, 40, 8, 2.0),
-        ]
-        summary = module.summarize_rows(rows, days=30)
-        report = module.render_markdown(summary, report_date="2026-04-22")
-
-        self.assertIn("# Fleet Cost Report — 2026-04-22", report)
-        self.assertIn("## Paid API inventory", report)
-        self.assertIn("Anthropic", report)
-        self.assertIn("OpenRouter", report)
-        self.assertIn("OpenAI", report)
-        self.assertIn("## Estimated cost per agent per day", report)
-        self.assertIn("Timmy Cloud Lane", report)
-        self.assertIn("Ezra", report)
-        self.assertIn("Bezalel", report)
-
-    def test_write_report_creates_markdown_file(self):
-        module = load_module()
-        rows = [("claude-sonnet-4-6", 1, 10, 2, 0.5)]
-        summary = module.summarize_rows(rows, days=30)
-        with tempfile.TemporaryDirectory() as tmpdir:
-            dest = Path(tmpdir) / "fleet-cost.md"
-            module.write_report(dest, summary, report_date="2026-04-22")
-            self.assertTrue(dest.exists())
-            text = dest.read_text()
-            self.assertIn("Fleet Cost Report", text)
-            self.assertIn("Ezra", text)
-
-
-if __name__ == "__main__":
-    unittest.main()