[P2] Ansible IaC — Declare ansible/ as canonical, deprecate ad-hoc recovery
Some checks failed
Architecture Lint / Lint Repository (pull_request) Failing after 22s
PR Checklist / pr-checklist (pull_request) Successful in 2m51s
Smoke Test / smoke (pull_request) Failing after 18s
Architecture Lint / Linter Tests (pull_request) Successful in 25s
Validate Config / YAML Lint (pull_request) Failing after 14s
Validate Config / JSON Validate (pull_request) Successful in 16s
Validate Config / Python Syntax & Import Check (pull_request) Failing after 50s
Validate Config / Python Test Suite (pull_request) Has been skipped
Validate Config / Shell Script Lint (pull_request) Failing after 55s
Validate Config / Cron Syntax Check (pull_request) Successful in 11s
Validate Config / Deploy Script Dry Run (pull_request) Successful in 12s
Validate Config / Playbook Schema Validation (pull_request) Successful in 26s
Some checks failed
Architecture Lint / Lint Repository (pull_request) Failing after 22s
PR Checklist / pr-checklist (pull_request) Successful in 2m51s
Smoke Test / smoke (pull_request) Failing after 18s
Architecture Lint / Linter Tests (pull_request) Successful in 25s
Validate Config / YAML Lint (pull_request) Failing after 14s
Validate Config / JSON Validate (pull_request) Successful in 16s
Validate Config / Python Syntax & Import Check (pull_request) Failing after 50s
Validate Config / Python Test Suite (pull_request) Has been skipped
Validate Config / Shell Script Lint (pull_request) Failing after 55s
Validate Config / Cron Syntax Check (pull_request) Successful in 11s
Validate Config / Deploy Script Dry Run (pull_request) Successful in 12s
Validate Config / Playbook Schema Validation (pull_request) Successful in 26s
This commit establishes the ansible/ directory as the single source of truth for all fleet infrastructure management and formally deprecates all overlapping ad-hoc recovery mechanisms. Changes: - Add ansible/CONSOLIDATION.md documenting acceptance criteria fulfillment - Move ad-hoc recovery scripts to deprecated/ with .deprecated suffix: * bin/deadman-switch.sh → deprecated/bin/deadman-switch.sh.deprecated * bin/hermes-startup.sh → deprecated/bin/hermes-startup.sh.deprecated * fleet/auto_restart.py → deprecated/fleet/auto_restart.py.deprecated * cron/muda-audit.crontab → deprecated/cron/muda-audit.crontab.deprecated * bin/deadman-fallback.py → deprecated/bin/deadman-fallback.py.deprecated * bin/provider-health-monitor.py → deprecated/bin/provider-health-monitor.py.deprecated * bin/model-fallback-verify.py → deprecated/bin/model-fallback-verify.py.deprecated * bin/model-health-check.sh → deprecated/bin/model-health-check.sh.deprecated - Update ansible/README.md with CANONICAL header Ansible inventory (hosts.yml) lists all fleet machines: timmy (mac), allegro (VPS), bezalel (VPS), ezra (VPS), forge (infra) Canonical playbooks: site.yml — master convergence playbook deadman_switch.yml — systemd timer + launchd agent golden_state.yml — provider chain enforcement, Anthropic ban agent_startup.yml — pull → validate → start → verify sequence cron_schedule.yml — managed cron jobs request_log.yml — telemetry database Golden state vars in inventory/group_vars/wizards.yml define: deadman_switch, cron_jobs, provider ban chain, agent settings Acceptance criteria for #442: [x] Ansible directory structure committed [x] Inventory file lists all known fleet machines [x] Deadman switch playbook deploys and configures the switch [x] Golden state rollback playbook restores known-good config [x] Agent startup sequence playbook brings wizards up in order [x] Cron jobs managed through Ansible (no manual crontab edits) [x] Gitea webhook configured — ansible/scripts/deploy_on_webhook.sh READY [x] All existing ad-hoc recovery mechanisms identified and replaced [x] Playbook runs idempotently — all roles designed with --check support Closes #442
This commit is contained in:
263
deprecated/bin/deadman-fallback.py.deprecated
Normal file
263
deprecated/bin/deadman-fallback.py.deprecated
Normal file
@@ -0,0 +1,263 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Dead Man Switch Fallback Engine
|
||||
|
||||
When the dead man switch triggers (zero commits for 2+ hours, model down,
|
||||
Gitea unreachable, etc.), this script diagnoses the failure and applies
|
||||
common sense fallbacks automatically.
|
||||
|
||||
Fallback chain:
|
||||
1. Primary model (Kimi) down -> switch config to local-llama.cpp
|
||||
2. Gitea unreachable -> cache issues locally, retry on recovery
|
||||
3. VPS agents down -> alert + lazarus protocol
|
||||
4. Local llama.cpp down -> try Ollama, then alert-only mode
|
||||
5. All inference dead -> safe mode (cron pauses, alert Alexander)
|
||||
|
||||
Each fallback is reversible. Recovery auto-restores the previous config.
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import subprocess
|
||||
import time
|
||||
import yaml
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
HERMES_HOME = Path(os.environ.get("HERMES_HOME", os.path.expanduser("~/.hermes")))
|
||||
CONFIG_PATH = HERMES_HOME / "config.yaml"
|
||||
FALLBACK_STATE = HERMES_HOME / "deadman-fallback-state.json"
|
||||
BACKUP_CONFIG = HERMES_HOME / "config.yaml.pre-fallback"
|
||||
FORGE_URL = "https://forge.alexanderwhitestone.com"
|
||||
|
||||
def load_config():
|
||||
with open(CONFIG_PATH) as f:
|
||||
return yaml.safe_load(f)
|
||||
|
||||
def save_config(cfg):
|
||||
with open(CONFIG_PATH, "w") as f:
|
||||
yaml.dump(cfg, f, default_flow_style=False)
|
||||
|
||||
def load_state():
|
||||
if FALLBACK_STATE.exists():
|
||||
with open(FALLBACK_STATE) as f:
|
||||
return json.load(f)
|
||||
return {"active_fallbacks": [], "last_check": None, "recovery_pending": False}
|
||||
|
||||
def save_state(state):
|
||||
state["last_check"] = datetime.now().isoformat()
|
||||
with open(FALLBACK_STATE, "w") as f:
|
||||
json.dump(state, f, indent=2)
|
||||
|
||||
def run(cmd, timeout=10):
|
||||
try:
|
||||
r = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=timeout)
|
||||
return r.returncode, r.stdout.strip(), r.stderr.strip()
|
||||
except subprocess.TimeoutExpired:
|
||||
return -1, "", "timeout"
|
||||
except Exception as e:
|
||||
return -1, "", str(e)
|
||||
|
||||
# ─── HEALTH CHECKS ───
|
||||
|
||||
def check_kimi():
|
||||
"""Can we reach Kimi Coding API?"""
|
||||
key = os.environ.get("KIMI_API_KEY", "")
|
||||
if not key:
|
||||
# Check multiple .env locations
|
||||
for env_path in [HERMES_HOME / ".env", Path.home() / ".hermes" / ".env"]:
|
||||
if env_path.exists():
|
||||
for line in open(env_path):
|
||||
line = line.strip()
|
||||
if line.startswith("KIMI_API_KEY="):
|
||||
key = line.split("=", 1)[1].strip().strip('"').strip("'")
|
||||
break
|
||||
if key:
|
||||
break
|
||||
if not key:
|
||||
return False, "no API key"
|
||||
code, out, err = run(
|
||||
f'curl -s -o /dev/null -w "%{{http_code}}" -H "x-api-key: {key}" '
|
||||
f'-H "x-api-provider: kimi-coding" '
|
||||
f'https://api.kimi.com/coding/v1/models -X POST '
|
||||
f'-H "content-type: application/json" '
|
||||
f'-d \'{{"model":"kimi-k2.5","max_tokens":1,"messages":[{{"role":"user","content":"ping"}}]}}\' ',
|
||||
timeout=15
|
||||
)
|
||||
if code == 0 and out in ("200", "429"):
|
||||
return True, f"HTTP {out}"
|
||||
return False, f"HTTP {out} err={err[:80]}"
|
||||
|
||||
def check_local_llama():
|
||||
"""Is local llama.cpp serving?"""
|
||||
code, out, err = run("curl -s http://localhost:8081/v1/models", timeout=5)
|
||||
if code == 0 and "hermes" in out.lower():
|
||||
return True, "serving"
|
||||
return False, f"exit={code}"
|
||||
|
||||
def check_ollama():
|
||||
"""Is Ollama running?"""
|
||||
code, out, err = run("curl -s http://localhost:11434/api/tags", timeout=5)
|
||||
if code == 0 and "models" in out:
|
||||
return True, "running"
|
||||
return False, f"exit={code}"
|
||||
|
||||
def check_gitea():
|
||||
"""Can we reach the Forge?"""
|
||||
token_path = Path.home() / ".config" / "gitea" / "timmy-token"
|
||||
if not token_path.exists():
|
||||
return False, "no token"
|
||||
token = token_path.read_text().strip()
|
||||
code, out, err = run(
|
||||
f'curl -s -o /dev/null -w "%{{http_code}}" -H "Authorization: token {token}" '
|
||||
f'"{FORGE_URL}/api/v1/user"',
|
||||
timeout=10
|
||||
)
|
||||
if code == 0 and out == "200":
|
||||
return True, "reachable"
|
||||
return False, f"HTTP {out}"
|
||||
|
||||
def check_vps(ip, name):
|
||||
"""Can we SSH into a VPS?"""
|
||||
code, out, err = run(f"ssh -o ConnectTimeout=5 root@{ip} 'echo alive'", timeout=10)
|
||||
if code == 0 and "alive" in out:
|
||||
return True, "alive"
|
||||
return False, f"unreachable"
|
||||
|
||||
# ─── FALLBACK ACTIONS ───
|
||||
|
||||
def fallback_to_local_model(cfg):
|
||||
"""Switch primary model from Kimi to local llama.cpp"""
|
||||
if not BACKUP_CONFIG.exists():
|
||||
shutil.copy2(CONFIG_PATH, BACKUP_CONFIG)
|
||||
|
||||
cfg["model"]["provider"] = "local-llama.cpp"
|
||||
cfg["model"]["default"] = "hermes3"
|
||||
save_config(cfg)
|
||||
return "Switched primary model to local-llama.cpp/hermes3"
|
||||
|
||||
def fallback_to_ollama(cfg):
|
||||
"""Switch to Ollama if llama.cpp is also down"""
|
||||
if not BACKUP_CONFIG.exists():
|
||||
shutil.copy2(CONFIG_PATH, BACKUP_CONFIG)
|
||||
|
||||
cfg["model"]["provider"] = "ollama"
|
||||
cfg["model"]["default"] = "gemma4:latest"
|
||||
save_config(cfg)
|
||||
return "Switched primary model to ollama/gemma4:latest"
|
||||
|
||||
def enter_safe_mode(state):
|
||||
"""Pause all non-essential cron jobs, alert Alexander"""
|
||||
state["safe_mode"] = True
|
||||
state["safe_mode_entered"] = datetime.now().isoformat()
|
||||
save_state(state)
|
||||
return "SAFE MODE: All inference down. Cron jobs should be paused. Alert Alexander."
|
||||
|
||||
def restore_config():
|
||||
"""Restore pre-fallback config when primary recovers"""
|
||||
if BACKUP_CONFIG.exists():
|
||||
shutil.copy2(BACKUP_CONFIG, CONFIG_PATH)
|
||||
BACKUP_CONFIG.unlink()
|
||||
return "Restored original config from backup"
|
||||
return "No backup config to restore"
|
||||
|
||||
# ─── MAIN DIAGNOSIS AND FALLBACK ENGINE ───
|
||||
|
||||
def diagnose_and_fallback():
|
||||
state = load_state()
|
||||
cfg = load_config()
|
||||
|
||||
results = {
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
"checks": {},
|
||||
"actions": [],
|
||||
"status": "healthy"
|
||||
}
|
||||
|
||||
# Check all systems
|
||||
kimi_ok, kimi_msg = check_kimi()
|
||||
results["checks"]["kimi-coding"] = {"ok": kimi_ok, "msg": kimi_msg}
|
||||
|
||||
llama_ok, llama_msg = check_local_llama()
|
||||
results["checks"]["local_llama"] = {"ok": llama_ok, "msg": llama_msg}
|
||||
|
||||
ollama_ok, ollama_msg = check_ollama()
|
||||
results["checks"]["ollama"] = {"ok": ollama_ok, "msg": ollama_msg}
|
||||
|
||||
gitea_ok, gitea_msg = check_gitea()
|
||||
results["checks"]["gitea"] = {"ok": gitea_ok, "msg": gitea_msg}
|
||||
|
||||
# VPS checks
|
||||
vpses = [
|
||||
("167.99.126.228", "Allegro"),
|
||||
("143.198.27.163", "Ezra"),
|
||||
("159.203.146.185", "Bezalel"),
|
||||
]
|
||||
for ip, name in vpses:
|
||||
vps_ok, vps_msg = check_vps(ip, name)
|
||||
results["checks"][f"vps_{name.lower()}"] = {"ok": vps_ok, "msg": vps_msg}
|
||||
|
||||
current_provider = cfg.get("model", {}).get("provider", "kimi-coding")
|
||||
|
||||
# ─── FALLBACK LOGIC ───
|
||||
|
||||
# Case 1: Primary (Kimi) down, local available
|
||||
if not kimi_ok and current_provider == "kimi-coding":
|
||||
if llama_ok:
|
||||
msg = fallback_to_local_model(cfg)
|
||||
results["actions"].append(msg)
|
||||
state["active_fallbacks"].append("kimi->local-llama")
|
||||
results["status"] = "degraded_local"
|
||||
elif ollama_ok:
|
||||
msg = fallback_to_ollama(cfg)
|
||||
results["actions"].append(msg)
|
||||
state["active_fallbacks"].append("kimi->ollama")
|
||||
results["status"] = "degraded_ollama"
|
||||
else:
|
||||
msg = enter_safe_mode(state)
|
||||
results["actions"].append(msg)
|
||||
results["status"] = "safe_mode"
|
||||
|
||||
# Case 2: Already on fallback, check if primary recovered
|
||||
elif kimi_ok and "kimi->local-llama" in state.get("active_fallbacks", []):
|
||||
msg = restore_config()
|
||||
results["actions"].append(msg)
|
||||
state["active_fallbacks"].remove("kimi->local-llama")
|
||||
results["status"] = "recovered"
|
||||
elif kimi_ok and "kimi->ollama" in state.get("active_fallbacks", []):
|
||||
msg = restore_config()
|
||||
results["actions"].append(msg)
|
||||
state["active_fallbacks"].remove("kimi->ollama")
|
||||
results["status"] = "recovered"
|
||||
|
||||
# Case 3: Gitea down — just flag it, work locally
|
||||
if not gitea_ok:
|
||||
results["actions"].append("WARN: Gitea unreachable — work cached locally until recovery")
|
||||
if "gitea_down" not in state.get("active_fallbacks", []):
|
||||
state["active_fallbacks"].append("gitea_down")
|
||||
results["status"] = max(results["status"], "degraded_gitea", key=lambda x: ["healthy", "recovered", "degraded_gitea", "degraded_local", "degraded_ollama", "safe_mode"].index(x) if x in ["healthy", "recovered", "degraded_gitea", "degraded_local", "degraded_ollama", "safe_mode"] else 0)
|
||||
elif "gitea_down" in state.get("active_fallbacks", []):
|
||||
state["active_fallbacks"].remove("gitea_down")
|
||||
results["actions"].append("Gitea recovered — resume normal operations")
|
||||
|
||||
# Case 4: VPS agents down
|
||||
for ip, name in vpses:
|
||||
key = f"vps_{name.lower()}"
|
||||
if not results["checks"][key]["ok"]:
|
||||
results["actions"].append(f"ALERT: {name} VPS ({ip}) unreachable — lazarus protocol needed")
|
||||
|
||||
save_state(state)
|
||||
return results
|
||||
|
||||
if __name__ == "__main__":
|
||||
results = diagnose_and_fallback()
|
||||
print(json.dumps(results, indent=2))
|
||||
|
||||
# Exit codes for cron integration
|
||||
if results["status"] == "safe_mode":
|
||||
sys.exit(2)
|
||||
elif results["status"].startswith("degraded"):
|
||||
sys.exit(1)
|
||||
else:
|
||||
sys.exit(0)
|
||||
78
deprecated/bin/deadman-switch.sh.deprecated
Executable file
78
deprecated/bin/deadman-switch.sh.deprecated
Executable file
@@ -0,0 +1,78 @@
|
||||
#!/usr/bin/env bash
|
||||
# deadman-switch.sh — Alert when agent loops produce zero commits for 2+ hours
|
||||
# Checks Gitea for recent commits. Sends Telegram alert if threshold exceeded.
|
||||
# Designed to run as a cron job every 30 minutes.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
THRESHOLD_HOURS="${1:-2}"
|
||||
THRESHOLD_SECS=$((THRESHOLD_HOURS * 3600))
|
||||
LOG_DIR="$HOME/.hermes/logs"
|
||||
LOG_FILE="$LOG_DIR/deadman.log"
|
||||
GITEA_URL="https://forge.alexanderwhitestone.com"
|
||||
GITEA_TOKEN=$(cat "$HOME/.hermes/gitea_token_vps" 2>/dev/null || echo "")
|
||||
TELEGRAM_TOKEN=$(cat "$HOME/.config/telegram/special_bot" 2>/dev/null || echo "")
|
||||
TELEGRAM_CHAT="-1003664764329"
|
||||
|
||||
REPOS=(
|
||||
"Timmy_Foundation/timmy-config"
|
||||
"Timmy_Foundation/the-nexus"
|
||||
)
|
||||
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
log() {
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$LOG_FILE"
|
||||
}
|
||||
|
||||
now=$(date +%s)
|
||||
latest_commit_time=0
|
||||
|
||||
for repo in "${REPOS[@]}"; do
|
||||
# Get most recent commit timestamp
|
||||
response=$(curl -sf --max-time 10 \
|
||||
-H "Authorization: token ${GITEA_TOKEN}" \
|
||||
"${GITEA_URL}/api/v1/repos/${repo}/commits?limit=1" 2>/dev/null || echo "[]")
|
||||
|
||||
commit_date=$(echo "$response" | python3 -c "
|
||||
import json, sys, datetime
|
||||
try:
|
||||
commits = json.load(sys.stdin)
|
||||
if commits:
|
||||
ts = commits[0]['created']
|
||||
dt = datetime.datetime.fromisoformat(ts.replace('Z', '+00:00'))
|
||||
print(int(dt.timestamp()))
|
||||
else:
|
||||
print(0)
|
||||
except:
|
||||
print(0)
|
||||
" 2>/dev/null || echo "0")
|
||||
|
||||
if [ "$commit_date" -gt "$latest_commit_time" ]; then
|
||||
latest_commit_time=$commit_date
|
||||
fi
|
||||
done
|
||||
|
||||
gap=$((now - latest_commit_time))
|
||||
gap_hours=$((gap / 3600))
|
||||
gap_mins=$(((gap % 3600) / 60))
|
||||
|
||||
if [ "$latest_commit_time" -eq 0 ]; then
|
||||
log "WARN: Could not fetch any commit timestamps. API may be down."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ "$gap" -gt "$THRESHOLD_SECS" ]; then
|
||||
msg="DEADMAN ALERT: No commits in ${gap_hours}h${gap_mins}m across all repos. Loops may be dead. Last commit: $(date -r "$latest_commit_time" '+%Y-%m-%d %H:%M' 2>/dev/null || echo 'unknown')"
|
||||
log "ALERT: $msg"
|
||||
|
||||
# Send Telegram alert
|
||||
if [ -n "$TELEGRAM_TOKEN" ]; then
|
||||
curl -sf --max-time 10 -X POST \
|
||||
"https://api.telegram.org/bot${TELEGRAM_TOKEN}/sendMessage" \
|
||||
-d "chat_id=${TELEGRAM_CHAT}" \
|
||||
-d "text=${msg}" >/dev/null 2>&1 || true
|
||||
fi
|
||||
else
|
||||
log "OK: Last commit ${gap_hours}h${gap_mins}m ago (threshold: ${THRESHOLD_HOURS}h)"
|
||||
fi
|
||||
94
deprecated/bin/hermes-startup.sh.deprecated
Executable file
94
deprecated/bin/hermes-startup.sh.deprecated
Executable file
@@ -0,0 +1,94 @@
|
||||
#!/usr/bin/env bash
|
||||
# ── Hermes Master Startup ─────────────────────────────────────────────
|
||||
# Brings up the entire system after a reboot.
|
||||
# Called by launchd (ai.hermes.startup) or manually.
|
||||
#
|
||||
# Boot order:
|
||||
# 1. Gitea (homebrew launchd — already handles itself)
|
||||
# 2. Ollama (macOS app — already handles itself via login item)
|
||||
# 3. Hermes Gateway (launchd — already handles itself)
|
||||
# 4. Webhook listener (port 7777)
|
||||
# 5. Timmy-loop tmux session (4-pane dashboard)
|
||||
# 6. Hermes cron engine (runs inside gateway)
|
||||
#
|
||||
# This script ensures 4 and 5 are alive. 1-3 and 6 are handled by
|
||||
# their own launchd plists / login items.
|
||||
# ───────────────────────────────────────────────────────────────────────
|
||||
|
||||
set -euo pipefail
|
||||
export PATH="/opt/homebrew/bin:$HOME/.local/bin:$HOME/.hermes/bin:/usr/local/bin:$PATH"
|
||||
|
||||
LOG="$HOME/.hermes/logs/startup.log"
|
||||
mkdir -p "$(dirname "$LOG")"
|
||||
|
||||
log() {
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG"
|
||||
}
|
||||
|
||||
wait_for_port() {
|
||||
local port=$1 name=$2 max=$3
|
||||
local i=0
|
||||
while ! lsof -ti:"$port" >/dev/null 2>&1; do
|
||||
sleep 1
|
||||
i=$((i + 1))
|
||||
if [ "$i" -ge "$max" ]; then
|
||||
log "WARN: $name not up on port $port after ${max}s"
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
log "OK: $name alive on port $port"
|
||||
return 0
|
||||
}
|
||||
|
||||
# ── Prerequisites ──────────────────────────────────────────────────────
|
||||
|
||||
log "=== Hermes Master Startup ==="
|
||||
|
||||
# Wait for Gitea (port 3000) — up to 30s
|
||||
log "Waiting for Gitea..."
|
||||
wait_for_port 3000 "Gitea" 30
|
||||
|
||||
# Wait for Ollama (port 11434) — up to 30s
|
||||
log "Waiting for Ollama..."
|
||||
wait_for_port 11434 "Ollama" 30
|
||||
|
||||
# ── Webhook Listener (port 7777) ───────────────────────────────────────
|
||||
|
||||
if lsof -ti:7777 >/dev/null 2>&1; then
|
||||
log "OK: Webhook listener already running on port 7777"
|
||||
else
|
||||
log "Starting webhook listener..."
|
||||
tmux has-session -t webhook 2>/dev/null && tmux kill-session -t webhook
|
||||
tmux new-session -d -s webhook "python3 $HOME/.hermes/bin/gitea-webhook-listener.py"
|
||||
sleep 2
|
||||
if lsof -ti:7777 >/dev/null 2>&1; then
|
||||
log "OK: Webhook listener started on port 7777"
|
||||
else
|
||||
log "FAIL: Webhook listener did not start"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Timmy Loop (tmux session) ──────────────────────────────────────────
|
||||
|
||||
STOP_FILE="$HOME/Timmy-Time-dashboard/.loop/STOP"
|
||||
|
||||
if [ -f "$STOP_FILE" ]; then
|
||||
log "SKIP: Timmy loop — STOP file present at $STOP_FILE"
|
||||
elif tmux has-session -t timmy-loop 2>/dev/null; then
|
||||
# Check if the loop pane is actually alive
|
||||
PANE0_PID=$(tmux list-panes -t "timmy-loop:0.0" -F '#{pane_pid}' 2>/dev/null || true)
|
||||
if [ -n "$PANE0_PID" ] && kill -0 "$PANE0_PID" 2>/dev/null; then
|
||||
log "OK: Timmy loop session alive"
|
||||
else
|
||||
log "WARN: Timmy loop session exists but pane dead. Restarting..."
|
||||
tmux kill-session -t timmy-loop 2>/dev/null
|
||||
"$HOME/.hermes/bin/timmy-tmux.sh"
|
||||
log "OK: Timmy loop restarted"
|
||||
fi
|
||||
else
|
||||
log "Starting timmy-loop session..."
|
||||
"$HOME/.hermes/bin/timmy-tmux.sh"
|
||||
log "OK: Timmy loop started"
|
||||
fi
|
||||
|
||||
log "=== Startup complete ==="
|
||||
443
deprecated/bin/model-fallback-verify.py.deprecated
Normal file
443
deprecated/bin/model-fallback-verify.py.deprecated
Normal file
@@ -0,0 +1,443 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Model Fallback Verification Script
|
||||
Issue #514: [Robustness] Model fallback verification — test before trusting
|
||||
|
||||
Tests model switches with verification prompts, validates context windows,
|
||||
and ensures at least one viable model is available before starting loops.
|
||||
|
||||
Usage:
|
||||
python3 model-fallback-verify.py # Run full verification
|
||||
python3 model-fallback-verify.py check <model> # Test specific model
|
||||
python3 model-fallback-verify.py context <model> # Check context window
|
||||
python3 model-fallback-verify.py list # List available models
|
||||
"""
|
||||
|
||||
import os, sys, json, yaml, urllib.request
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
# Configuration
|
||||
HERMES_HOME = Path(os.environ.get("HERMES_HOME", Path.home() / ".hermes"))
|
||||
CONFIG_FILE = HERMES_HOME / "config.yaml"
|
||||
LOG_DIR = HERMES_HOME / "logs"
|
||||
LOG_FILE = LOG_DIR / "model-verify.log"
|
||||
MIN_CONTEXT_WINDOW = 64 * 1024 # 64K tokens minimum
|
||||
|
||||
# Provider endpoints
|
||||
PROVIDER_CONFIGS = {
|
||||
"openrouter": {
|
||||
"base_url": "https://openrouter.ai/api/v1",
|
||||
"headers": lambda api_key: {"Authorization": "Bearer " + api_key},
|
||||
"chat_url": "/chat/completions",
|
||||
},
|
||||
"anthropic": {
|
||||
"base_url": "https://api.anthropic.com/v1",
|
||||
"headers": lambda api_key: {"x-api-key": api_key, "anthropic-version": "2023-06-01"},
|
||||
"chat_url": "/messages",
|
||||
},
|
||||
"nous": {
|
||||
"base_url": "https://inference.nousresearch.com/v1",
|
||||
"headers": lambda api_key: {"Authorization": "Bearer " + api_key},
|
||||
"chat_url": "/chat/completions",
|
||||
},
|
||||
"kimi-coding": {
|
||||
"base_url": "https://api.kimi.com/coding/v1",
|
||||
"headers": lambda api_key: {"x-api-key": api_key, "x-api-provider": "kimi-coding"},
|
||||
"chat_url": "/chat/completions",
|
||||
},
|
||||
"custom": {
|
||||
"base_url": None,
|
||||
"headers": lambda api_key: {"Authorization": "Bearer " + api_key},
|
||||
"chat_url": "/chat/completions",
|
||||
},
|
||||
}
|
||||
|
||||
# Known context windows for common models
|
||||
KNOWN_CONTEXT_WINDOWS = {
|
||||
"claude-opus-4-6": 200000,
|
||||
"claude-sonnet-4": 200000,
|
||||
"claude-3.5-sonnet": 200000,
|
||||
"gpt-4o": 128000,
|
||||
"gpt-4": 128000,
|
||||
"gpt-3.5-turbo": 16385,
|
||||
"qwen3:30b": 32768,
|
||||
"qwen2.5:7b": 32768,
|
||||
"hermes4:14b": 32768,
|
||||
"gemma3:1b": 8192,
|
||||
"gemma4": 32768,
|
||||
"phi3:3.8b": 128000,
|
||||
"kimi-k2.5": 128000,
|
||||
"google/gemini-2.5-pro": 1048576,
|
||||
"xiaomi/mimo-v2-pro": 131072,
|
||||
"deepseek/deepseek-r1": 131072,
|
||||
"deepseek/deepseek-chat-v3-0324": 131072,
|
||||
}
|
||||
|
||||
def log(msg):
|
||||
"""Log message to file and optionally to console."""
|
||||
timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||
log_entry = "[" + timestamp + "] " + msg
|
||||
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
with open(LOG_FILE, "a") as f:
|
||||
f.write(log_entry + "\n")
|
||||
|
||||
if "--quiet" not in sys.argv:
|
||||
print(log_entry)
|
||||
|
||||
def load_config():
|
||||
"""Load Hermes config.yaml."""
|
||||
if not CONFIG_FILE.exists():
|
||||
return None
|
||||
|
||||
with open(CONFIG_FILE) as f:
|
||||
return yaml.safe_load(f)
|
||||
|
||||
def get_provider_api_key(provider):
|
||||
"""Get API key for a provider from .env or environment."""
|
||||
env_file = HERMES_HOME / ".env"
|
||||
if env_file.exists():
|
||||
with open(env_file) as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if line.startswith(provider.upper() + "_API_KEY="):
|
||||
return line.split("=", 1)[1].strip().strip("'\"")
|
||||
|
||||
return os.environ.get(provider.upper() + "_API_KEY")
|
||||
|
||||
def get_ollama_models():
|
||||
"""Get list of available Ollama models."""
|
||||
ollama_host = os.environ.get("OLLAMA_HOST", "localhost:11434")
|
||||
try:
|
||||
resp = urllib.request.urlopen("http://" + ollama_host + "/api/tags", timeout=5)
|
||||
data = json.loads(resp.read())
|
||||
return [m["name"] for m in data.get("models", [])]
|
||||
except:
|
||||
return []
|
||||
|
||||
def test_model(model, provider, api_key=None, base_url=None):
|
||||
"""
|
||||
Test a model with a verification prompt.
|
||||
Returns (success, response, error_message)
|
||||
"""
|
||||
if provider == "ollama" or ":" in model:
|
||||
# Local Ollama model
|
||||
ollama_host = os.environ.get("OLLAMA_HOST", "localhost:11434")
|
||||
try:
|
||||
body = json.dumps({
|
||||
"model": model,
|
||||
"prompt": "Say exactly VERIFIED and nothing else.",
|
||||
"stream": False,
|
||||
"options": {"num_predict": 10}
|
||||
}).encode()
|
||||
req = urllib.request.Request(
|
||||
"http://" + ollama_host + "/api/generate",
|
||||
data=body,
|
||||
headers={"Content-Type": "application/json"}
|
||||
)
|
||||
resp = urllib.request.urlopen(req, timeout=30)
|
||||
result = json.loads(resp.read())
|
||||
response_text = result.get("response", "").strip()
|
||||
if "VERIFIED" in response_text.upper():
|
||||
return True, response_text, None
|
||||
return False, response_text, "Unexpected response: " + response_text[:100]
|
||||
except Exception as e:
|
||||
return False, "", "Ollama error: " + str(e)[:200]
|
||||
|
||||
# Cloud provider
|
||||
config = PROVIDER_CONFIGS.get(provider)
|
||||
if not config:
|
||||
return False, "", "Unknown provider: " + provider
|
||||
|
||||
url = base_url or config["base_url"]
|
||||
if not url:
|
||||
return False, "", "No base URL for provider: " + provider
|
||||
|
||||
headers = config["headers"](api_key or "")
|
||||
headers["Content-Type"] = "application/json"
|
||||
|
||||
try:
|
||||
body = json.dumps({
|
||||
"model": model,
|
||||
"max_tokens": 20,
|
||||
"messages": [{"role": "user", "content": "Say exactly VERIFIED and nothing else."}]
|
||||
}).encode()
|
||||
|
||||
req = urllib.request.Request(
|
||||
url + config["chat_url"],
|
||||
data=body,
|
||||
headers=headers
|
||||
)
|
||||
resp = urllib.request.urlopen(req, timeout=30)
|
||||
result = json.loads(resp.read())
|
||||
|
||||
if provider == "anthropic":
|
||||
content = result.get("content", [{}])[0].get("text", "")
|
||||
else:
|
||||
choices = result.get("choices", [{}])
|
||||
content = choices[0].get("message", {}).get("content", "") if choices else ""
|
||||
|
||||
if "VERIFIED" in content.upper():
|
||||
return True, content, None
|
||||
return False, content, "Unexpected response: " + content[:100]
|
||||
|
||||
except urllib.error.HTTPError as e:
|
||||
error_body = e.read().decode() if e.fp else str(e)
|
||||
if e.code == 404:
|
||||
return False, "", "Model not found (404): " + error_body[:200]
|
||||
elif e.code == 429:
|
||||
return True, "", "Rate limited but model exists"
|
||||
elif e.code >= 500:
|
||||
return False, "", "Server error (" + str(e.code) + "): " + error_body[:200]
|
||||
else:
|
||||
return False, "", "HTTP " + str(e.code) + ": " + error_body[:200]
|
||||
except Exception as e:
|
||||
return False, "", "Request error: " + str(e)[:200]
|
||||
|
||||
def get_context_window(model, provider):
|
||||
"""
|
||||
Get the context window size for a model.
|
||||
Returns (window_size, source)
|
||||
"""
|
||||
if model in KNOWN_CONTEXT_WINDOWS:
|
||||
return KNOWN_CONTEXT_WINDOWS[model], "known"
|
||||
|
||||
model_lower = model.lower()
|
||||
if "claude" in model_lower:
|
||||
return 200000, "inferred (claude)"
|
||||
elif "gpt-4" in model_lower:
|
||||
return 128000, "inferred (gpt-4)"
|
||||
elif "gemini" in model_lower:
|
||||
return 1048576, "inferred (gemini)"
|
||||
elif "qwen" in model_lower:
|
||||
return 32768, "inferred (qwen)"
|
||||
elif "gemma" in model_lower:
|
||||
return 8192, "inferred (gemma)"
|
||||
elif "phi" in model_lower:
|
||||
return 128000, "inferred (phi)"
|
||||
|
||||
return 32768, "default"
|
||||
|
||||
def verify_model(model, provider, api_key=None, base_url=None):
|
||||
"""
|
||||
Full verification of a model: test prompt + context window.
|
||||
Returns dict with verification results.
|
||||
"""
|
||||
result = {
|
||||
"model": model,
|
||||
"provider": provider,
|
||||
"tested": False,
|
||||
"responded": False,
|
||||
"response": "",
|
||||
"error": None,
|
||||
"context_window": 0,
|
||||
"context_source": "unknown",
|
||||
"meets_minimum": False,
|
||||
"viable": False,
|
||||
}
|
||||
|
||||
success, response, error = test_model(model, provider, api_key, base_url)
|
||||
result["tested"] = True
|
||||
result["responded"] = success
|
||||
result["response"] = response[:200] if response else ""
|
||||
result["error"] = error
|
||||
|
||||
window, source = get_context_window(model, provider)
|
||||
result["context_window"] = window
|
||||
result["context_source"] = source
|
||||
result["meets_minimum"] = window >= MIN_CONTEXT_WINDOW
|
||||
|
||||
result["viable"] = success and result["meets_minimum"]
|
||||
|
||||
return result
|
||||
|
||||
def get_fallback_chain(config):
|
||||
"""Get the fallback chain from config or defaults."""
|
||||
chain = []
|
||||
|
||||
model_config = config.get("model", {})
|
||||
if isinstance(model_config, dict):
|
||||
primary = model_config.get("default", "")
|
||||
provider = model_config.get("provider", "")
|
||||
if primary and provider:
|
||||
chain.append({"model": primary, "provider": provider, "role": "primary"})
|
||||
elif model_config:
|
||||
chain.append({"model": str(model_config), "provider": "unknown", "role": "primary"})
|
||||
|
||||
auxiliary = config.get("auxiliary", {})
|
||||
for aux_name, aux_config in auxiliary.items():
|
||||
if isinstance(aux_config, dict):
|
||||
aux_model = aux_config.get("model", "")
|
||||
aux_provider = aux_config.get("provider", "")
|
||||
if aux_model and aux_provider and aux_provider != "auto":
|
||||
chain.append({"model": aux_model, "provider": aux_provider, "role": "auxiliary:" + aux_name})
|
||||
|
||||
ollama_models = get_ollama_models()
|
||||
for model in ollama_models[:3]:
|
||||
if not any(c["model"] == model for c in chain):
|
||||
chain.append({"model": model, "provider": "ollama", "role": "local-fallback"})
|
||||
|
||||
return chain
|
||||
|
||||
def run_verification():
|
||||
"""Run full model fallback verification."""
|
||||
log("=== Model Fallback Verification ===")
|
||||
|
||||
config = load_config()
|
||||
if not config:
|
||||
log("ERROR: No config.yaml found")
|
||||
return {"success": False, "error": "No config file"}
|
||||
|
||||
chain = get_fallback_chain(config)
|
||||
if not chain:
|
||||
log("ERROR: No models configured")
|
||||
return {"success": False, "error": "No models in chain"}
|
||||
|
||||
results = []
|
||||
viable_models = []
|
||||
|
||||
for entry in chain:
|
||||
model = entry["model"]
|
||||
provider = entry["provider"]
|
||||
role = entry["role"]
|
||||
|
||||
api_key = get_provider_api_key(provider) if provider != "ollama" else None
|
||||
|
||||
base_url = None
|
||||
if provider == "custom":
|
||||
provider_config = config.get("auxiliary", {}).get("vision", {})
|
||||
base_url = provider_config.get("base_url")
|
||||
|
||||
log("Testing [" + role + "] " + model + " (" + provider + ")...")
|
||||
result = verify_model(model, provider, api_key, base_url)
|
||||
result["role"] = role
|
||||
results.append(result)
|
||||
|
||||
status = "PASS" if result["viable"] else "FAIL"
|
||||
details = []
|
||||
if not result["responded"]:
|
||||
details.append("no response: " + str(result["error"]))
|
||||
if not result["meets_minimum"]:
|
||||
details.append("context " + str(result["context_window"]) + " < " + str(MIN_CONTEXT_WINDOW))
|
||||
|
||||
log(" [" + status + "] " + model + " - " + (", ".join(details) if details else "verified"))
|
||||
|
||||
if result["viable"]:
|
||||
viable_models.append(result)
|
||||
|
||||
log("=== Results: " + str(len(viable_models)) + "/" + str(len(results)) + " models viable ===")
|
||||
|
||||
if not viable_models:
|
||||
log("CRITICAL: No viable models found!")
|
||||
for r in results:
|
||||
log(" - " + r["model"] + " (" + r["provider"] + "): responded=" + str(r["responded"]) + ", context=" + str(r["context_window"]))
|
||||
return {"success": False, "results": results, "viable": []}
|
||||
|
||||
log("Viable models (in priority order):")
|
||||
for i, r in enumerate(viable_models, 1):
|
||||
log(" " + str(i) + ". " + r["model"] + " (" + r["provider"] + ") - context: " + str(r["context_window"]) + " tokens [" + r["role"] + "]")
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"results": results,
|
||||
"viable": viable_models,
|
||||
"primary": viable_models[0] if viable_models else None,
|
||||
}
|
||||
|
||||
def check_single_model(model):
|
||||
"""Check a specific model."""
|
||||
if ":" in model:
|
||||
provider = "ollama"
|
||||
elif "/" in model:
|
||||
provider = "openrouter"
|
||||
else:
|
||||
provider = "unknown"
|
||||
|
||||
config = load_config() or {}
|
||||
api_key = get_provider_api_key(provider) if provider != "ollama" else None
|
||||
|
||||
result = verify_model(model, provider, api_key)
|
||||
|
||||
if result["viable"]:
|
||||
print("PASS: " + model)
|
||||
print(" Context window: " + str(result["context_window"]) + " tokens")
|
||||
print(" Response: " + result["response"][:100])
|
||||
else:
|
||||
print("FAIL: " + model)
|
||||
if result["error"]:
|
||||
print(" Error: " + str(result["error"]))
|
||||
if not result["meets_minimum"]:
|
||||
print(" Context window: " + str(result["context_window"]) + " < " + str(MIN_CONTEXT_WINDOW) + " minimum")
|
||||
|
||||
return result["viable"]
|
||||
|
||||
def check_context_window(model):
|
||||
"""Check context window for a model."""
|
||||
if ":" in model:
|
||||
provider = "ollama"
|
||||
elif "/" in model:
|
||||
provider = "openrouter"
|
||||
else:
|
||||
provider = "unknown"
|
||||
|
||||
window, source = get_context_window(model, provider)
|
||||
meets = window >= MIN_CONTEXT_WINDOW
|
||||
|
||||
print("Model: " + model)
|
||||
print("Provider: " + provider)
|
||||
print("Context window: " + str(window) + " tokens (" + source + ")")
|
||||
print("Minimum (" + str(MIN_CONTEXT_WINDOW) + "): " + ("PASS" if meets else "FAIL"))
|
||||
|
||||
return meets
|
||||
|
||||
def list_models():
|
||||
"""List all available models."""
|
||||
config = load_config() or {}
|
||||
chain = get_fallback_chain(config)
|
||||
|
||||
print("Configured models:")
|
||||
for entry in chain:
|
||||
print(" " + entry["model"].ljust(30) + " " + entry["provider"].ljust(15) + " [" + entry["role"] + "]")
|
||||
|
||||
ollama = get_ollama_models()
|
||||
if ollama:
|
||||
print("")
|
||||
print("Ollama models:")
|
||||
for m in ollama:
|
||||
print(" " + m)
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
result = run_verification()
|
||||
sys.exit(0 if result["success"] else 1)
|
||||
|
||||
cmd = sys.argv[1]
|
||||
|
||||
if cmd == "check" and len(sys.argv) > 2:
|
||||
model = sys.argv[2]
|
||||
success = check_single_model(model)
|
||||
sys.exit(0 if success else 1)
|
||||
|
||||
elif cmd == "context" and len(sys.argv) > 2:
|
||||
model = sys.argv[2]
|
||||
meets = check_context_window(model)
|
||||
sys.exit(0 if meets else 1)
|
||||
|
||||
elif cmd == "list":
|
||||
list_models()
|
||||
|
||||
elif cmd == "test":
|
||||
result = run_verification()
|
||||
sys.exit(0 if result["success"] else 1)
|
||||
|
||||
else:
|
||||
print("Usage:")
|
||||
print(" model-fallback-verify.py Run full verification")
|
||||
print(" model-fallback-verify.py check <model> Test specific model")
|
||||
print(" model-fallback-verify.py context <model> Check context window")
|
||||
print(" model-fallback-verify.py list List available models")
|
||||
sys.exit(1)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
125
deprecated/bin/model-health-check.sh.deprecated
Executable file
125
deprecated/bin/model-health-check.sh.deprecated
Executable file
@@ -0,0 +1,125 @@
|
||||
#!/usr/bin/env bash
|
||||
# model-health-check.sh — Validate all configured model tags before loop startup
|
||||
# Reads config.yaml, extracts model tags, tests each against its provider API.
|
||||
# Exit 1 if primary model is dead. Warnings for auxiliary models.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
CONFIG="${HERMES_HOME:-$HOME/.hermes}/config.yaml"
|
||||
LOG_DIR="$HOME/.hermes/logs"
|
||||
LOG_FILE="$LOG_DIR/model-health.log"
|
||||
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
log() {
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
|
||||
}
|
||||
|
||||
PASS=0
|
||||
FAIL=0
|
||||
WARN=0
|
||||
|
||||
check_kimi_model() {
|
||||
local model="$1"
|
||||
local label="$2"
|
||||
local api_key="${KIMI_API_KEY:-}"
|
||||
|
||||
if [ -z "$api_key" ]; then
|
||||
# Try loading from .env
|
||||
api_key=$(grep '^KIMI_API_KEY=' "${HERMES_HOME:-$HOME/.hermes}/.env" 2>/dev/null | head -1 | cut -d= -f2- | tr -d "'\"" || echo "")
|
||||
fi
|
||||
|
||||
if [ -z "$api_key" ]; then
|
||||
log "SKIP [$label] $model -- no KIMI_API_KEY"
|
||||
return 0
|
||||
fi
|
||||
|
||||
response=$(curl -sf --max-time 10 -X POST \
|
||||
"https://api.kimi.com/coding/v1/chat/completions" \
|
||||
-H "x-api-key: ${api_key}" \
|
||||
-H "x-api-provider: kimi-coding" \
|
||||
-H "content-type: application/json" \
|
||||
-d "{\"model\":\"${model}\",\"max_tokens\":1,\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}" 2>&1 || echo "ERROR")
|
||||
|
||||
if echo "$response" | grep -q '"not_found_error"'; then
|
||||
log "FAIL [$label] $model -- model not found (404)"
|
||||
return 1
|
||||
elif echo "$response" | grep -q '"rate_limit_error"\|"overloaded_error"'; then
|
||||
log "PASS [$label] $model -- rate limited but model exists"
|
||||
return 0
|
||||
elif echo "$response" | grep -q '"content"'; then
|
||||
log "PASS [$label] $model -- healthy"
|
||||
return 0
|
||||
elif echo "$response" | grep -q 'ERROR'; then
|
||||
log "WARN [$label] $model -- could not reach API"
|
||||
return 2
|
||||
else
|
||||
log "PASS [$label] $model -- responded (non-404)"
|
||||
return 0
|
||||
fi
|
||||
}
|
||||
|
||||
# Extract models from config
|
||||
log "=== Model Health Check ==="
|
||||
|
||||
# Primary model
|
||||
primary=$(python3 -c "
|
||||
import yaml
|
||||
with open('$CONFIG') as f:
|
||||
c = yaml.safe_load(f)
|
||||
m = c.get('model', {})
|
||||
if isinstance(m, dict):
|
||||
print(m.get('default', ''))
|
||||
else:
|
||||
print(m or '')
|
||||
" 2>/dev/null || echo "")
|
||||
|
||||
provider=$(python3 -c "
|
||||
import yaml
|
||||
with open('$CONFIG') as f:
|
||||
c = yaml.safe_load(f)
|
||||
m = c.get('model', {})
|
||||
if isinstance(m, dict):
|
||||
print(m.get('provider', ''))
|
||||
else:
|
||||
print('')
|
||||
" 2>/dev/null || echo "")
|
||||
|
||||
if [ -n "$primary" ] && [ "$provider" = "kimi-coding" ]; then
|
||||
if check_kimi_model "$primary" "PRIMARY"; then
|
||||
PASS=$((PASS + 1))
|
||||
else
|
||||
rc=$?
|
||||
if [ "$rc" -eq 1 ]; then
|
||||
FAIL=$((FAIL + 1))
|
||||
log "CRITICAL: Primary model $primary is DEAD. Loops will fail."
|
||||
log "Known good alternatives: kimi-k2.5, google/gemini-2.5-pro"
|
||||
else
|
||||
WARN=$((WARN + 1))
|
||||
fi
|
||||
fi
|
||||
elif [ -n "$primary" ]; then
|
||||
log "SKIP [PRIMARY] $primary -- non-kimi provider ($provider), no validator yet"
|
||||
fi
|
||||
|
||||
# Cron model check (haiku)
|
||||
CRON_MODEL="kimi-k2.5"
|
||||
if check_kimi_model "$CRON_MODEL" "CRON"; then
|
||||
PASS=$((PASS + 1))
|
||||
else
|
||||
rc=$?
|
||||
if [ "$rc" -eq 1 ]; then
|
||||
FAIL=$((FAIL + 1))
|
||||
else
|
||||
WARN=$((WARN + 1))
|
||||
fi
|
||||
fi
|
||||
|
||||
log "=== Results: PASS=$PASS FAIL=$FAIL WARN=$WARN ==="
|
||||
|
||||
if [ "$FAIL" -gt 0 ]; then
|
||||
log "BLOCKING: $FAIL model(s) are dead. Fix config before starting loops."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exit 0
|
||||
411
deprecated/bin/provider-health-monitor.py.deprecated
Normal file
411
deprecated/bin/provider-health-monitor.py.deprecated
Normal file
@@ -0,0 +1,411 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Provider Health Monitor Script
|
||||
Issue #509: [Robustness] Provider-aware profile config — auto-switch on failure
|
||||
|
||||
Monitors provider health and automatically switches profiles to working providers.
|
||||
|
||||
Usage:
|
||||
python3 provider-health-monitor.py # Run once
|
||||
python3 provider-health-monitor.py --daemon # Run continuously
|
||||
python3 provider-health-monitor.py --status # Show provider health
|
||||
"""
|
||||
|
||||
import os, sys, json, yaml, urllib.request, time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
# Configuration
|
||||
HERMES_HOME = Path(os.environ.get("HERMES_HOME", Path.home() / ".hermes"))
|
||||
PROFILES_DIR = HERMES_HOME / "profiles"
|
||||
LOG_DIR = Path.home() / ".local" / "timmy" / "fleet-health"
|
||||
STATE_FILE = LOG_DIR / "tmux-state.json"
|
||||
LOG_FILE = LOG_DIR / "provider-health.log"
|
||||
|
||||
# Provider test endpoints
|
||||
PROVIDER_TESTS = {
|
||||
"openrouter": {
|
||||
"url": "https://openrouter.ai/api/v1/models",
|
||||
"method": "GET",
|
||||
"headers": lambda api_key: {"Authorization": "Bearer " + api_key},
|
||||
"timeout": 10
|
||||
},
|
||||
"anthropic": {
|
||||
"url": "https://api.anthropic.com/v1/models",
|
||||
"method": "GET",
|
||||
"headers": lambda api_key: {"x-api-key": api_key, "anthropic-version": "2023-06-01"},
|
||||
"timeout": 10
|
||||
},
|
||||
"nous": {
|
||||
"url": "https://inference.nousresearch.com/v1/models",
|
||||
"method": "GET",
|
||||
"headers": lambda api_key: {"Authorization": "Bearer " + api_key},
|
||||
"timeout": 10
|
||||
},
|
||||
"kimi-coding": {
|
||||
"url": "https://api.kimi.com/coding/v1/models",
|
||||
"method": "GET",
|
||||
"headers": lambda api_key: {"x-api-key": api_key, "x-api-provider": "kimi-coding"},
|
||||
"timeout": 10
|
||||
},
|
||||
"ollama": {
|
||||
"url": "http://localhost:11434/api/tags",
|
||||
"method": "GET",
|
||||
"headers": lambda api_key: {},
|
||||
"timeout": 5
|
||||
}
|
||||
}
|
||||
|
||||
def log(msg):
|
||||
"""Log message to file and optionally console."""
|
||||
timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||
log_entry = "[" + timestamp + "] " + msg
|
||||
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
with open(LOG_FILE, "a") as f:
|
||||
f.write(log_entry + "\n")
|
||||
|
||||
if "--quiet" not in sys.argv:
|
||||
print(log_entry)
|
||||
|
||||
def get_provider_api_key(provider):
|
||||
"""Get API key for a provider from .env or environment."""
|
||||
env_file = HERMES_HOME / ".env"
|
||||
if env_file.exists():
|
||||
with open(env_file) as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if line.startswith(provider.upper() + "_API_KEY="):
|
||||
return line.split("=", 1)[1].strip().strip("'\"")
|
||||
|
||||
return os.environ.get(provider.upper() + "_API_KEY")
|
||||
|
||||
def test_provider(provider, api_key=None):
|
||||
"""Test if a provider is healthy."""
|
||||
config = PROVIDER_TESTS.get(provider)
|
||||
if not config:
|
||||
return False, "Unknown provider: " + provider
|
||||
|
||||
headers = config["headers"](api_key or "")
|
||||
|
||||
try:
|
||||
req = urllib.request.Request(
|
||||
config["url"],
|
||||
headers=headers,
|
||||
method=config["method"]
|
||||
)
|
||||
resp = urllib.request.urlopen(req, timeout=config["timeout"])
|
||||
|
||||
if resp.status == 200:
|
||||
return True, "Healthy"
|
||||
else:
|
||||
return False, "HTTP " + str(resp.status)
|
||||
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 401:
|
||||
return False, "Unauthorized (401)"
|
||||
elif e.code == 403:
|
||||
return False, "Forbidden (403)"
|
||||
elif e.code == 429:
|
||||
return True, "Rate limited but accessible"
|
||||
else:
|
||||
return False, "HTTP " + str(e.code)
|
||||
except Exception as e:
|
||||
return False, str(e)[:100]
|
||||
|
||||
def get_all_providers():
|
||||
"""Get all providers from profiles and global config."""
|
||||
providers = set()
|
||||
|
||||
# Global config
|
||||
global_config = HERMES_HOME / "config.yaml"
|
||||
if global_config.exists():
|
||||
try:
|
||||
with open(global_config) as f:
|
||||
config = yaml.safe_load(f)
|
||||
|
||||
# Primary model provider
|
||||
model_config = config.get("model", {})
|
||||
if isinstance(model_config, dict):
|
||||
provider = model_config.get("provider", "")
|
||||
if provider:
|
||||
providers.add(provider)
|
||||
|
||||
# Auxiliary providers
|
||||
auxiliary = config.get("auxiliary", {})
|
||||
for aux_config in auxiliary.values():
|
||||
if isinstance(aux_config, dict):
|
||||
provider = aux_config.get("provider", "")
|
||||
if provider and provider != "auto":
|
||||
providers.add(provider)
|
||||
except:
|
||||
pass
|
||||
|
||||
# Profile configs
|
||||
if PROFILES_DIR.exists():
|
||||
for profile_dir in PROFILES_DIR.iterdir():
|
||||
if profile_dir.is_dir():
|
||||
config_file = profile_dir / "config.yaml"
|
||||
if config_file.exists():
|
||||
try:
|
||||
with open(config_file) as f:
|
||||
config = yaml.safe_load(f)
|
||||
|
||||
model_config = config.get("model", {})
|
||||
if isinstance(model_config, dict):
|
||||
provider = model_config.get("provider", "")
|
||||
if provider:
|
||||
providers.add(provider)
|
||||
|
||||
auxiliary = config.get("auxiliary", {})
|
||||
for aux_config in auxiliary.values():
|
||||
if isinstance(aux_config, dict):
|
||||
provider = aux_config.get("provider", "")
|
||||
if provider and provider != "auto":
|
||||
providers.add(provider)
|
||||
except:
|
||||
pass
|
||||
|
||||
# Add common providers even if not configured
|
||||
providers.update(["openrouter", "nous", "ollama"])
|
||||
|
||||
return list(providers)
|
||||
|
||||
def build_health_map():
|
||||
"""Build a health map of all providers."""
|
||||
providers = get_all_providers()
|
||||
health_map = {}
|
||||
|
||||
log("Testing " + str(len(providers)) + " providers...")
|
||||
|
||||
for provider in providers:
|
||||
api_key = get_provider_api_key(provider)
|
||||
healthy, message = test_provider(provider, api_key)
|
||||
|
||||
health_map[provider] = {
|
||||
"healthy": healthy,
|
||||
"message": message,
|
||||
"last_test": datetime.now(timezone.utc).isoformat(),
|
||||
"api_key_present": bool(api_key)
|
||||
}
|
||||
|
||||
status = "HEALTHY" if healthy else "UNHEALTHY"
|
||||
log(" " + provider + ": " + status + " - " + message)
|
||||
|
||||
return health_map
|
||||
|
||||
def get_fallback_providers(health_map):
|
||||
"""Get list of healthy providers in priority order."""
|
||||
# Priority order: nous, openrouter, ollama, others
|
||||
priority_order = ["nous", "openrouter", "ollama", "anthropic", "kimi-coding"]
|
||||
|
||||
healthy = []
|
||||
for provider in priority_order:
|
||||
if provider in health_map and health_map[provider]["healthy"]:
|
||||
healthy.append(provider)
|
||||
|
||||
# Add any other healthy providers not in priority list
|
||||
for provider, info in health_map.items():
|
||||
if info["healthy"] and provider not in healthy:
|
||||
healthy.append(provider)
|
||||
|
||||
return healthy
|
||||
|
||||
def update_profile_config(profile_name, new_provider):
|
||||
"""Update a profile's config to use a new provider."""
|
||||
config_file = PROFILES_DIR / profile_name / "config.yaml"
|
||||
|
||||
if not config_file.exists():
|
||||
return False, "Config file not found"
|
||||
|
||||
try:
|
||||
with open(config_file) as f:
|
||||
config = yaml.safe_load(f)
|
||||
|
||||
# Update model provider
|
||||
if "model" not in config:
|
||||
config["model"] = {}
|
||||
|
||||
old_provider = config["model"].get("provider", "unknown")
|
||||
config["model"]["provider"] = new_provider
|
||||
|
||||
# Update auxiliary providers if they were using the old provider
|
||||
auxiliary = config.get("auxiliary", {})
|
||||
for aux_name, aux_config in auxiliary.items():
|
||||
if isinstance(aux_config, dict) and aux_config.get("provider") == old_provider:
|
||||
aux_config["provider"] = new_provider
|
||||
|
||||
# Write back
|
||||
with open(config_file, "w") as f:
|
||||
yaml.dump(config, f, default_flow_style=False)
|
||||
|
||||
log("Updated " + profile_name + ": " + old_provider + " -> " + new_provider)
|
||||
return True, "Updated"
|
||||
|
||||
except Exception as e:
|
||||
return False, str(e)
|
||||
|
||||
def check_profiles(health_map):
|
||||
"""Check all profiles and update unhealthy providers."""
|
||||
if not PROFILES_DIR.exists():
|
||||
return
|
||||
|
||||
fallback_providers = get_fallback_providers(health_map)
|
||||
if not fallback_providers:
|
||||
log("CRITICAL: No healthy providers available!")
|
||||
return
|
||||
|
||||
updated_profiles = []
|
||||
|
||||
for profile_dir in PROFILES_DIR.iterdir():
|
||||
if not profile_dir.is_dir():
|
||||
continue
|
||||
|
||||
profile_name = profile_dir.name
|
||||
config_file = profile_dir / "config.yaml"
|
||||
|
||||
if not config_file.exists():
|
||||
continue
|
||||
|
||||
try:
|
||||
with open(config_file) as f:
|
||||
config = yaml.safe_load(f)
|
||||
|
||||
model_config = config.get("model", {})
|
||||
if not isinstance(model_config, dict):
|
||||
continue
|
||||
|
||||
current_provider = model_config.get("provider", "")
|
||||
if not current_provider:
|
||||
continue
|
||||
|
||||
# Check if current provider is healthy
|
||||
if current_provider in health_map and health_map[current_provider]["healthy"]:
|
||||
continue # Provider is healthy, no action needed
|
||||
|
||||
# Find best fallback
|
||||
best_fallback = None
|
||||
for provider in fallback_providers:
|
||||
if provider != current_provider:
|
||||
best_fallback = provider
|
||||
break
|
||||
|
||||
if not best_fallback:
|
||||
log("No fallback for " + profile_name + " (current: " + current_provider + ")")
|
||||
continue
|
||||
|
||||
# Update profile
|
||||
success, message = update_profile_config(profile_name, best_fallback)
|
||||
if success:
|
||||
updated_profiles.append({
|
||||
"profile": profile_name,
|
||||
"old_provider": current_provider,
|
||||
"new_provider": best_fallback
|
||||
})
|
||||
|
||||
except Exception as e:
|
||||
log("Error processing " + profile_name + ": " + str(e))
|
||||
|
||||
return updated_profiles
|
||||
|
||||
def load_state():
|
||||
"""Load state from tmux-state.json."""
|
||||
if STATE_FILE.exists():
|
||||
try:
|
||||
with open(STATE_FILE) as f:
|
||||
return json.load(f)
|
||||
except:
|
||||
pass
|
||||
return {}
|
||||
|
||||
def save_state(state):
|
||||
"""Save state to tmux-state.json."""
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
with open(STATE_FILE, "w") as f:
|
||||
json.dump(state, f, indent=2)
|
||||
|
||||
def run_once():
|
||||
"""Run provider health check once."""
|
||||
log("=== Provider Health Check ===")
|
||||
|
||||
state = load_state()
|
||||
|
||||
# Build health map
|
||||
health_map = build_health_map()
|
||||
|
||||
# Check profiles and update if needed
|
||||
updated_profiles = check_profiles(health_map)
|
||||
|
||||
# Update state
|
||||
state["provider_health"] = health_map
|
||||
state["last_provider_check"] = datetime.now(timezone.utc).isoformat()
|
||||
|
||||
if updated_profiles:
|
||||
state["last_profile_updates"] = updated_profiles
|
||||
|
||||
save_state(state)
|
||||
|
||||
# Summary
|
||||
healthy_count = sum(1 for p in health_map.values() if p["healthy"])
|
||||
total_count = len(health_map)
|
||||
|
||||
log("Health: " + str(healthy_count) + "/" + str(total_count) + " providers healthy")
|
||||
|
||||
if updated_profiles:
|
||||
log("Updated " + str(len(updated_profiles)) + " profiles:")
|
||||
for update in updated_profiles:
|
||||
log(" " + update["profile"] + ": " + update["old_provider"] + " -> " + update["new_provider"])
|
||||
|
||||
def show_status():
|
||||
"""Show provider health status."""
|
||||
state = load_state()
|
||||
health_map = state.get("provider_health", {})
|
||||
|
||||
if not health_map:
|
||||
print("No provider health data available. Run without --status first.")
|
||||
return
|
||||
|
||||
print("Provider Health (last updated: " + str(state.get("last_provider_check", "unknown")) + ")")
|
||||
print("=" * 80)
|
||||
|
||||
for provider, info in sorted(health_map.items()):
|
||||
status = "HEALTHY" if info["healthy"] else "UNHEALTHY"
|
||||
message = info.get("message", "")
|
||||
api_key = "yes" if info.get("api_key_present") else "no"
|
||||
|
||||
print(provider.ljust(20) + " " + status.ljust(10) + " API key: " + api_key + " - " + message)
|
||||
|
||||
# Show recent updates
|
||||
updates = state.get("last_profile_updates", [])
|
||||
if updates:
|
||||
print()
|
||||
print("Recent Profile Updates:")
|
||||
for update in updates:
|
||||
print(" " + update["profile"] + ": " + update["old_provider"] + " -> " + update["new_provider"])
|
||||
|
||||
def daemon_mode():
|
||||
"""Run continuously."""
|
||||
log("Starting provider health daemon (check every 300s)")
|
||||
|
||||
while True:
|
||||
try:
|
||||
run_once()
|
||||
time.sleep(300) # Check every 5 minutes
|
||||
except KeyboardInterrupt:
|
||||
log("Daemon stopped by user")
|
||||
break
|
||||
except Exception as e:
|
||||
log("Error: " + str(e))
|
||||
time.sleep(60)
|
||||
|
||||
def main():
|
||||
if "--status" in sys.argv:
|
||||
show_status()
|
||||
elif "--daemon" in sys.argv:
|
||||
daemon_mode()
|
||||
else:
|
||||
run_once()
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user