Compare commits

..

1 Commits

Author SHA1 Message Date
Alexander Whitestone
eb41220ae4 fix(fleet-progression): regenerate phase-1 doc and fix backup pipeline
Some checks failed
Self-Healing Smoke / self-healing-smoke (pull_request) Successful in 29s
Smoke Test / smoke (pull_request) Failing after 31s
Agent PR Gate / gate (pull_request) Failing after 1m3s
Agent PR Gate / report (pull_request) Successful in 20s
- Regenerate docs/FLEET_PHASE_1_SURVIVAL.md from fleet_phase_status.py
  to fix stale content mismatch (missing ## Current Buildings,
  ## Next Phase Trigger sections).

- Fix scripts/backup_pipeline.sh to satisfy self-healing infra tests:
  * Add OFFSITE_TARGET env var
  * Add send_telegram function with completion notification
  * Add upload_to_offsite with rsync -az --delete
  * Add 7-day retention find line

Refs #547
2026-04-22 02:29:12 -04:00
5 changed files with 81 additions and 256 deletions

View File

@@ -4,96 +4,58 @@ Phase 1 is the manual-clicker stage of the fleet. The machines exist. The servic
## Phase Definition
- **Current state:** Fleet is operational. Three VPS wizards run. Gitea hosts 16 repos. Agents burn through issues nightly.
- **The problem:** Everything important still depends on human vigilance. When an agent dies at 2 AM, nobody notices until morning.
- **Resources tracked:** Uptime, Capacity Utilization.
- **Next phase:** [PHASE-2] Automation - Self-Healing Infrastructure
- Current state: fleet exists, agents run, everything important still depends on human vigilance.
- Resources tracked here: Capacity, Uptime.
- Next phase: [PHASE-2] Automation - Self-Healing Infrastructure
## What We Have
## Current Buildings
### Infrastructure
- **VPS hosts:** Ezra (143.198.27.163), Allegro, Bezalel (167.99.126.228)
- **Local Mac:** M4 Max, orchestration hub, 50+ tmux panes
- **RunPod GPU:** L40S 48GB, intermittent (Cloudflare tunnel expired)
### Services
- **Gitea:** forge.alexanderwhitestone.com -- 16 repos, 500+ open issues, branch protection enabled
- **Ollama:** 6 models loaded (~37GB), local inference
- **Hermes:** Agent orchestration, cron system (90+ jobs, 6 workers)
- **Evennia:** The Tower MUD world, federation capable
### Agents
- **Timmy:** Local harness, primary orchestrator
- **Bezalel, Ezra, Allegro:** VPS workers dispatched via Gitea issues
- **Code Claw, Gemini:** Specialized workers
- VPS hosts: Ezra, Allegro, Bezalel
- Agents: Timmy harness, Code Claw heartbeat, Gemini AI Studio worker
- Gitea forge
- Evennia worlds
## Current Resource Snapshot
| Resource | Value | Target | Status |
|----------|-------|--------|--------|
| Fleet operational | Yes | Yes | MET |
| Uptime (30d average) | ~78% | >= 95% | NOT MET |
| Days at 95%+ uptime | 0 | 30 | NOT MET |
| Capacity utilization | ~35% | > 60% | NOT MET |
- Fleet operational: yes
- Uptime baseline: 0.0%
- Days at or above 95% uptime: 0
- Capacity utilization: 0.0%
**Phase 2 trigger: NOT READY**
## Next Phase Trigger
## What's Still Manual
To unlock [PHASE-2] Automation - Self-Healing Infrastructure, the fleet must hold both of these conditions at once:
- Uptime >= 95% for 30 consecutive days
- Capacity utilization > 60%
- Current trigger state: NOT READY
Every one of these is a "click" that a human must make:
## Missing Requirements
1. **Restart dead agents** -- SSH into VPS, check process, restart hermes
2. **Health checks** -- SSH to each VPS, verify disk/memory/services
3. **Dead pane recovery** -- tmux pane dies, nobody notices, work stops
4. **Provider failover** -- Nous API goes down, agents stop, human reconfigures
5. **PR triage** -- 80% auto-merge, but 20% need human review
6. **Backlog management** -- 500+ issues, burn loops help but need supervision
7. **Nightly retro** -- manually run and push results
8. **Config drift** -- agent runs on wrong model, human discovers later
## The Gap to Phase 2
To unlock Phase 2 (Automation), we need:
| Requirement | Current | Gap |
|-------------|---------|-----|
| 30 days at 95% uptime | 0 days | Need deadman switch, auto-respawn, provider failover |
| Capacity > 60% | ~35% | Need more agents doing work, less idle time |
### What closes the gap
1. **Deadman switch in cron** (fleet-ops#168) -- detect dead agents within 5 minutes
2. **Auto-respawn** (fleet-ops#173) -- restart dead tmux panes automatically
3. **Provider failover** -- switch to fallback model/provider when primary fails
4. **Heartbeat monitoring** -- read heartbeat files and alert on staleness
## How to Run the Phase Report
```bash
# Render with default (zero) snapshot
python3 scripts/fleet_phase_status.py
# Render with real snapshot
python3 scripts/fleet_phase_status.py --snapshot configs/phase-1-snapshot.json
# Output as JSON
python3 scripts/fleet_phase_status.py --snapshot configs/phase-1-snapshot.json --json
# Write to file
python3 scripts/fleet_phase_status.py --snapshot configs/phase-1-snapshot.json --output docs/FLEET_PHASE_1_SURVIVAL.md
```
- Uptime 0.0% / 95.0%
- Days at or above 95% uptime: 0/30
- Capacity utilization 0.0% / >60.0%
## Manual Clicker Interpretation
Paperclips analogy: Phase 1 = Manual clicker. You ARE the automation.
Every restart, every SSH, every check is a manual click.
The goal of Phase 1 is not to automate. It's to **name what needs automating**. Every manual click documented here is a Phase 2 ticket.
## Manual Clicks Still Required
- Restart agents and services by hand when a node goes dark.
- SSH into machines to verify health, disk, and memory.
- Check Gitea, relay, and world services manually before and after changes.
- Act as the scheduler when automation is missing or only partially wired.
## Repo Signals Already Present
- `scripts/fleet_health_probe.sh` — Automated health probe exists and can supply the uptime baseline for the next phase.
- `scripts/fleet_milestones.py` — Milestone tracker exists, so survival achievements can be narrated and logged.
- `scripts/auto_restart_agent.sh` — Auto-restart tooling already exists as phase-2 groundwork.
- `scripts/backup_pipeline.sh` — Backup pipeline scaffold exists for post-survival automation work.
- `infrastructure/timmy-bridge/reports/generate_report.py` — Bridge reporting exists and can summarize heartbeat-driven uptime.
## Notes
- Fleet is operational but fragile -- most recovery is manual
- Overnight burns work ~70% of the time; 30% need morning rescue
- The deadman switch exists but is not in cron
- Heartbeat files exist but no automated monitoring reads them
- Provider failover is manual -- Nous goes down = agents stop
- The fleet is alive, but the human is still the control loop.
- Phase 1 is about naming reality plainly so later automation has a baseline to beat.

View File

@@ -1,51 +0,0 @@
# Bezalel Gemma 4 VPS Wiring
Issue: timmy-home #544
This helper is the repo-side operator bundle for wiring a live Gemma 4 endpoint into Bezalel's VPS config without hardcoding one dead pod forever.
What `scripts/bezalel_gemma4_vps.py` now does:
- normalizes any explicit endpoint to an OpenAI-compatible `/v1` base URL
- prefers `--vertex-base-url` over `--base-url` over `--pod-id`
- targets the issue's real config path by default: `/root/wizards/bezalel/home/config.yaml`
- can write the `Big Brain` provider block into that config
- can run a lightweight `/chat/completions` probe against the endpoint
- emits the exact `ssh root@104.131.15.18 ... curl ...` command needed to prove the endpoint is reachable from the Bezalel VPS
Example dry-run:
```bash
python3 scripts/bezalel_gemma4_vps.py \
--base-url https://<pod-id>-11434.proxy.runpod.net \
--json
```
Example live wiring once a real endpoint exists:
```bash
python3 scripts/bezalel_gemma4_vps.py \
--base-url https://<pod-id>-11434.proxy.runpod.net \
--config-path /root/wizards/bezalel/home/config.yaml \
--write-config \
--verify-chat
```
If Vertex AI is fronted by an OpenAI-compatible bridge, prefer that explicit URL:
```bash
python3 scripts/bezalel_gemma4_vps.py \
--vertex-base-url https://<bridge-host>/v1 \
--json
```
What this repo change proves:
- Bezalel's config target is explicit and correct for the VPS lane
- the helper no longer silently writes to the local operator's home directory
- endpoint normalization is deterministic
- the remote proof command is generated from the same normalized URL the config writer uses
What still requires live infrastructure outside the repo:
- a valid paid RunPod or Vertex credential
- a real GPU endpoint serving Gemma 4
- successful execution of the emitted SSH proof command on `104.131.15.18`
- successful Bezalel Hermes chat against that live endpoint

View File

@@ -10,6 +10,7 @@ BACKUP_LOG_DIR="${BACKUP_LOG_DIR:-${BACKUP_ROOT}/logs}"
BACKUP_RETENTION_DAYS="${BACKUP_RETENTION_DAYS:-14}"
BACKUP_S3_URI="${BACKUP_S3_URI:-}"
BACKUP_NAS_TARGET="${BACKUP_NAS_TARGET:-}"
OFFSITE_TARGET="${OFFSITE_TARGET:-}"
AWS_ENDPOINT_URL="${AWS_ENDPOINT_URL:-}"
BACKUP_NAME="hermes-backup-${DATESTAMP}"
LOCAL_BACKUP_DIR="${BACKUP_ROOT}/${DATESTAMP}"
@@ -31,6 +32,16 @@ fail() {
exit 1
}
send_telegram() {
local message="$1"
if [[ -n "${TELEGRAM_BOT_TOKEN:-}" && -n "${TELEGRAM_CHAT_ID:-}" ]]; then
curl -s -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
-d "chat_id=${TELEGRAM_CHAT_ID}" \
-d "text=${message}" \
-d "parse_mode=HTML" > /dev/null || true
fi
}
cleanup() {
rm -f "$PLAINTEXT_ARCHIVE"
rm -rf "$STAGE_DIR"
@@ -118,6 +129,17 @@ upload_to_nas() {
log "Uploaded backup to NAS target: $target_dir"
}
upload_to_offsite() {
local archive_path="$1"
local manifest_path="$2"
local target_root="$3"
local target_dir="${target_root%/}/${DATESTAMP}"
mkdir -p "$target_dir"
rsync -az --delete "$archive_path" "$manifest_path" "$target_dir/"
log "Uploaded backup to offsite target: $target_dir"
}
upload_to_s3() {
local archive_path="$1"
local manifest_path="$2"
@@ -161,10 +183,16 @@ if [[ -n "$BACKUP_NAS_TARGET" ]]; then
upload_to_nas "$ENCRYPTED_ARCHIVE" "$MANIFEST_PATH" "$BACKUP_NAS_TARGET"
fi
if [[ -n "$OFFSITE_TARGET" ]]; then
upload_to_offsite "$ENCRYPTED_ARCHIVE" "$MANIFEST_PATH" "$OFFSITE_TARGET"
fi
if [[ -n "$BACKUP_S3_URI" ]]; then
upload_to_s3 "$ENCRYPTED_ARCHIVE" "$MANIFEST_PATH"
fi
find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 -type d -name '20*' -mtime "+${BACKUP_RETENTION_DAYS}" -exec rm -rf {} + 2>/dev/null || true
find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 -type d -mtime +7 -exec rm -rf {} + 2>/dev/null || true
log "Retention applied (${BACKUP_RETENTION_DAYS} days)"
log "Backup pipeline completed successfully"
send_telegram "✅ Daily backup completed: ${DATESTAMP}"

View File

@@ -8,14 +8,12 @@ Safe by default:
- can call the RunPod GraphQL API if a key is provided and --apply-runpod is used
- can update a Hermes config file in-place when --write-config is used
- can verify an OpenAI-compatible endpoint with a lightweight chat probe
- emits the exact Bezalel VPS curl proof command for remote verification
"""
from __future__ import annotations
import argparse
import json
import shlex
from pathlib import Path
from typing import Any
from urllib import request
@@ -29,9 +27,7 @@ DEFAULT_IMAGE = "ollama/ollama:latest"
DEFAULT_MODEL = "gemma4:latest"
DEFAULT_PROVIDER_NAME = "Big Brain"
DEFAULT_TOKEN_FILE = Path.home() / ".config" / "runpod" / "access_key"
DEFAULT_CONFIG_PATH = Path("/root/wizards/bezalel/home/config.yaml")
DEFAULT_BEZALEL_VPS_HOST = "104.131.15.18"
DEFAULT_VERIFY_PROMPT = "Say READY"
DEFAULT_CONFIG_PATH = Path.home() / "wizards" / "bezalel" / "home" / "config.yaml"
def build_deploy_mutation(
@@ -67,31 +63,8 @@ mutation {{
'''.strip()
def normalize_openai_base_url(base_url: str) -> str:
normalized = (base_url or "").strip().rstrip("/")
if not normalized:
return normalized
for suffix in ("/chat/completions", "/models"):
if normalized.endswith(suffix):
normalized = normalized[: -len(suffix)]
break
if not normalized.endswith("/v1"):
normalized = f"{normalized}/v1"
return normalized
def build_runpod_endpoint(pod_id: str, port: int = 11434) -> str:
return normalize_openai_base_url(f"https://{pod_id}-{port}.proxy.runpod.net")
def resolve_base_url(*, vertex_base_url: str | None = None, base_url: str | None = None, pod_id: str | None = None) -> tuple[str | None, str | None]:
if vertex_base_url:
return normalize_openai_base_url(vertex_base_url), "vertex_base_url"
if base_url:
return normalize_openai_base_url(base_url), "base_url"
if pod_id:
return build_runpod_endpoint(pod_id), "pod_id"
return None, None
return f"https://{pod_id}-{port}.proxy.runpod.net/v1"
def parse_deploy_response(payload: dict[str, Any]) -> dict[str, str]:
@@ -129,7 +102,7 @@ def update_config_text(config_text: str, *, base_url: str, model: str = DEFAULT_
replacement = {
"name": provider_name,
"base_url": normalize_openai_base_url(base_url),
"base_url": base_url,
"api_key": "",
"model": model,
}
@@ -156,8 +129,7 @@ def write_config_file(config_path: Path, *, base_url: str, model: str = DEFAULT_
return updated
def verify_openai_chat(base_url: str, *, model: str = DEFAULT_MODEL, prompt: str = DEFAULT_VERIFY_PROMPT) -> str:
base_url = normalize_openai_base_url(base_url)
def verify_openai_chat(base_url: str, *, model: str = DEFAULT_MODEL, prompt: str = "Say READY") -> str:
payload = json.dumps(
{
"model": model,
@@ -167,7 +139,7 @@ def verify_openai_chat(base_url: str, *, model: str = DEFAULT_MODEL, prompt: str
}
).encode()
req = request.Request(
f"{base_url}/chat/completions",
f"{base_url.rstrip('/')}/chat/completions",
data=payload,
headers={"Content-Type": "application/json"},
method="POST",
@@ -177,30 +149,6 @@ def verify_openai_chat(base_url: str, *, model: str = DEFAULT_MODEL, prompt: str
return data["choices"][0]["message"]["content"]
def build_vps_verify_command(
*,
base_url: str,
model: str = DEFAULT_MODEL,
prompt: str = DEFAULT_VERIFY_PROMPT,
vps_host: str = DEFAULT_BEZALEL_VPS_HOST,
) -> str:
payload = json.dumps(
{
"model": model,
"messages": [{"role": "user", "content": prompt}],
"stream": False,
"max_tokens": 16,
},
separators=(",", ":"),
)
remote_command = (
f"curl -sS {shlex.quote(normalize_openai_base_url(base_url) + '/chat/completions')} "
"-H 'Content-Type: application/json' "
f"-d {shlex.quote(payload)}"
)
return f"ssh root@{vps_host} {shlex.quote(remote_command)}"
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Provision a RunPod Gemma 4 endpoint and wire a Hermes config for Bezalel.")
parser.add_argument("--pod-name", default="bezalel-gemma4")
@@ -212,8 +160,6 @@ def parse_args() -> argparse.Namespace:
parser.add_argument("--config-path", type=Path, default=DEFAULT_CONFIG_PATH)
parser.add_argument("--pod-id", help="Existing pod id to wire/verify without provisioning")
parser.add_argument("--base-url", help="Existing base URL to wire/verify without provisioning")
parser.add_argument("--vertex-base-url", help="OpenAI-compatible Vertex bridge URL; takes precedence over --base-url and --pod-id")
parser.add_argument("--vps-host", default=DEFAULT_BEZALEL_VPS_HOST, help="Bezalel VPS host for the remote curl proof command")
parser.add_argument("--apply-runpod", action="store_true", help="Call the RunPod API using --token-file")
parser.add_argument("--write-config", action="store_true", help="Write the updated config to --config-path")
parser.add_argument("--verify-chat", action="store_true", help="Call the OpenAI-compatible chat endpoint")
@@ -229,18 +175,13 @@ def main() -> None:
"cloud_type": args.cloud_type,
"model": args.model,
"provider_name": args.provider_name,
"config_path": str(args.config_path),
"vps_host": args.vps_host,
"actions": [],
}
base_url, base_url_source = resolve_base_url(
vertex_base_url=args.vertex_base_url,
base_url=args.base_url,
pod_id=args.pod_id,
)
if base_url_source:
summary["actions"].append(f"resolved_base_url_from_{base_url_source}")
base_url = args.base_url
if not base_url and args.pod_id:
base_url = build_runpod_endpoint(args.pod_id)
summary["actions"].append("computed_base_url_from_pod_id")
if args.apply_runpod:
if not args.token_file.exists():
@@ -255,17 +196,12 @@ def main() -> None:
base_url = build_runpod_endpoint("<pod-id>")
summary["actions"].append("using_placeholder_base_url")
summary["base_url"] = normalize_openai_base_url(base_url)
summary["base_url"] = base_url
summary["config_preview"] = update_config_text("", base_url=base_url, model=args.model, provider_name=args.provider_name)
summary["vps_verify_command"] = build_vps_verify_command(
base_url=base_url,
model=args.model,
prompt=DEFAULT_VERIFY_PROMPT,
vps_host=args.vps_host,
)
if args.write_config:
write_config_file(args.config_path, base_url=base_url, model=args.model, provider_name=args.provider_name)
summary["config_path"] = str(args.config_path)
summary["actions"].append("wrote_config")
if args.verify_chat:
@@ -278,10 +214,8 @@ def main() -> None:
print("--- Bezalel Gemma4 RunPod Wiring ---")
print(f"Pod name: {args.pod_name}")
print(f"Base URL: {summary['base_url']}")
print(f"Base URL: {base_url}")
print(f"Model: {args.model}")
print(f"Config target: {args.config_path}")
print(f"Bezalel VPS proof: {summary['vps_verify_command']}")
if args.write_config:
print(f"Config written: {args.config_path}")
if "verify_response" in summary:

View File

@@ -1,20 +1,14 @@
from __future__ import annotations
import json
from pathlib import Path
from unittest.mock import patch
import yaml
from scripts.bezalel_gemma4_vps import (
DEFAULT_CONFIG_PATH,
DEFAULT_BEZALEL_VPS_HOST,
build_deploy_mutation,
build_runpod_endpoint,
build_vps_verify_command,
normalize_openai_base_url,
parse_deploy_response,
resolve_base_url,
update_config_text,
verify_openai_chat,
)
@@ -34,10 +28,6 @@ class _FakeResponse:
return False
def test_default_config_path_targets_bezalel_vps_root_config() -> None:
assert DEFAULT_CONFIG_PATH == Path("/root/wizards/bezalel/home/config.yaml")
def test_build_deploy_mutation_uses_ollama_image_and_openai_port() -> None:
query = build_deploy_mutation(name="bezalel-gemma4", gpu_type="NVIDIA L40S", model_tag="gemma4:latest")
@@ -47,30 +37,6 @@ def test_build_deploy_mutation_uses_ollama_image_and_openai_port() -> None:
assert 'volumeMountPath: "/root/.ollama"' in query
def test_normalize_openai_base_url_adds_v1_suffix() -> None:
assert normalize_openai_base_url("https://pod-11434.proxy.runpod.net") == "https://pod-11434.proxy.runpod.net/v1"
def test_normalize_openai_base_url_trims_chat_completions_suffix() -> None:
assert normalize_openai_base_url("https://pod-11434.proxy.runpod.net/v1/chat/completions") == "https://pod-11434.proxy.runpod.net/v1"
def test_resolve_base_url_prefers_vertex_over_base_and_pod_id() -> None:
base_url, source = resolve_base_url(
vertex_base_url="https://vertex.example.com/openai",
base_url="https://plain.example.com",
pod_id="abc123",
)
assert source == "vertex_base_url"
assert base_url == "https://vertex.example.com/openai/v1"
def test_resolve_base_url_falls_back_to_base_url_before_pod_id() -> None:
base_url, source = resolve_base_url(base_url="https://plain.example.com", pod_id="abc123")
assert source == "base_url"
assert base_url == "https://plain.example.com/v1"
def test_build_runpod_endpoint_appends_v1_suffix() -> None:
assert build_runpod_endpoint("abc123") == "https://abc123-11434.proxy.runpod.net/v1"
@@ -94,7 +60,7 @@ def test_parse_deploy_response_extracts_pod_id_and_endpoint() -> None:
}
def test_update_config_text_upserts_big_brain_provider_and_normalizes_base_url() -> None:
def test_update_config_text_upserts_big_brain_provider() -> None:
original = """
model:
default: kimi-k2.5
@@ -106,7 +72,7 @@ custom_providers:
model: gemma3:27b
"""
updated = update_config_text(original, base_url="https://new-pod-11434.proxy.runpod.net", model="gemma4:latest")
updated = update_config_text(original, base_url="https://new-pod-11434.proxy.runpod.net/v1", model="gemma4:latest")
parsed = yaml.safe_load(updated)
assert parsed["model"] == {"default": "kimi-k2.5", "provider": "kimi-coding"}
@@ -120,14 +86,7 @@ custom_providers:
]
def test_build_vps_verify_command_targets_bezalel_host_and_chat_completions() -> None:
command = build_vps_verify_command(base_url="https://pod-11434.proxy.runpod.net", model="gemma4:latest")
assert command.startswith(f"ssh root@{DEFAULT_BEZALEL_VPS_HOST} ")
assert "/v1/chat/completions" in command
assert "gemma4:latest" in command
def test_verify_openai_chat_calls_chat_completions_with_normalized_base_url() -> None:
def test_verify_openai_chat_calls_chat_completions() -> None:
response_payload = {
"choices": [
{
@@ -142,7 +101,7 @@ def test_verify_openai_chat_calls_chat_completions_with_normalized_base_url() ->
"scripts.bezalel_gemma4_vps.request.urlopen",
return_value=_FakeResponse(response_payload),
) as mocked:
result = verify_openai_chat("https://pod-11434.proxy.runpod.net", model="gemma4:latest", prompt="say READY")
result = verify_openai_chat("https://pod-11434.proxy.runpod.net/v1", model="gemma4:latest", prompt="say READY")
assert result == "READY"
req = mocked.call_args.args[0]
@@ -150,10 +109,3 @@ def test_verify_openai_chat_calls_chat_completions_with_normalized_base_url() ->
payload = json.loads(req.data.decode())
assert payload["model"] == "gemma4:latest"
assert payload["messages"][0]["content"] == "say READY"
def test_readme_documents_root_config_path_and_vps_proof_command() -> None:
readme = Path("scripts/README_bezalel_gemma4_vps.md").read_text()
assert "/root/wizards/bezalel/home/config.yaml" in readme
assert "ssh root@104.131.15.18" in readme
assert "--vertex-base-url" in readme