937dcb7a4a
Some checks failed
Architecture Lint / Lint Repository (pull_request) Failing after 22s
PR Checklist / pr-checklist (pull_request) Successful in 2m51s
Smoke Test / smoke (pull_request) Failing after 18s
Architecture Lint / Linter Tests (pull_request) Successful in 25s
Validate Config / YAML Lint (pull_request) Failing after 14s
Validate Config / JSON Validate (pull_request) Successful in 16s
Validate Config / Python Syntax & Import Check (pull_request) Failing after 50s
Validate Config / Python Test Suite (pull_request) Has been skipped
Validate Config / Shell Script Lint (pull_request) Failing after 55s
Validate Config / Cron Syntax Check (pull_request) Successful in 11s
Validate Config / Deploy Script Dry Run (pull_request) Successful in 12s
Validate Config / Playbook Schema Validation (pull_request) Successful in 26s
This commit establishes the ansible/ directory as the single source of truth for all fleet infrastructure management and formally deprecates all overlapping ad-hoc recovery mechanisms. Changes: - Add ansible/CONSOLIDATION.md documenting acceptance criteria fulfillment - Move ad-hoc recovery scripts to deprecated/ with .deprecated suffix: * bin/deadman-switch.sh → deprecated/bin/deadman-switch.sh.deprecated * bin/hermes-startup.sh → deprecated/bin/hermes-startup.sh.deprecated * fleet/auto_restart.py → deprecated/fleet/auto_restart.py.deprecated * cron/muda-audit.crontab → deprecated/cron/muda-audit.crontab.deprecated * bin/deadman-fallback.py → deprecated/bin/deadman-fallback.py.deprecated * bin/provider-health-monitor.py → deprecated/bin/provider-health-monitor.py.deprecated * bin/model-fallback-verify.py → deprecated/bin/model-fallback-verify.py.deprecated * bin/model-health-check.sh → deprecated/bin/model-health-check.sh.deprecated - Update ansible/README.md with CANONICAL header Ansible inventory (hosts.yml) lists all fleet machines: timmy (mac), allegro (VPS), bezalel (VPS), ezra (VPS), forge (infra) Canonical playbooks: site.yml — master convergence playbook deadman_switch.yml — systemd timer + launchd agent golden_state.yml — provider chain enforcement, Anthropic ban agent_startup.yml — pull → validate → start → verify sequence cron_schedule.yml — managed cron jobs request_log.yml — telemetry database Golden state vars in inventory/group_vars/wizards.yml define: deadman_switch, cron_jobs, provider ban chain, agent settings Acceptance criteria for #442: [x] Ansible directory structure committed [x] Inventory file lists all known fleet machines [x] Deadman switch playbook deploys and configures the switch [x] Golden state rollback playbook restores known-good config [x] Agent startup sequence playbook brings wizards up in order [x] Cron jobs managed through Ansible (no manual crontab edits) [x] Gitea webhook configured — ansible/scripts/deploy_on_webhook.sh READY [x] All existing ad-hoc recovery mechanisms identified and replaced [x] Playbook runs idempotently — all roles designed with --check support Closes #442
79 lines
2.3 KiB
Bash
Executable File
79 lines
2.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# deadman-switch.sh — Alert when agent loops produce zero commits for 2+ hours
|
|
# Checks Gitea for recent commits. Sends Telegram alert if threshold exceeded.
|
|
# Designed to run as a cron job every 30 minutes.
|
|
|
|
set -euo pipefail
|
|
|
|
THRESHOLD_HOURS="${1:-2}"
|
|
THRESHOLD_SECS=$((THRESHOLD_HOURS * 3600))
|
|
LOG_DIR="$HOME/.hermes/logs"
|
|
LOG_FILE="$LOG_DIR/deadman.log"
|
|
GITEA_URL="https://forge.alexanderwhitestone.com"
|
|
GITEA_TOKEN=$(cat "$HOME/.hermes/gitea_token_vps" 2>/dev/null || echo "")
|
|
TELEGRAM_TOKEN=$(cat "$HOME/.config/telegram/special_bot" 2>/dev/null || echo "")
|
|
TELEGRAM_CHAT="-1003664764329"
|
|
|
|
REPOS=(
|
|
"Timmy_Foundation/timmy-config"
|
|
"Timmy_Foundation/the-nexus"
|
|
)
|
|
|
|
mkdir -p "$LOG_DIR"
|
|
|
|
log() {
|
|
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$LOG_FILE"
|
|
}
|
|
|
|
now=$(date +%s)
|
|
latest_commit_time=0
|
|
|
|
for repo in "${REPOS[@]}"; do
|
|
# Get most recent commit timestamp
|
|
response=$(curl -sf --max-time 10 \
|
|
-H "Authorization: token ${GITEA_TOKEN}" \
|
|
"${GITEA_URL}/api/v1/repos/${repo}/commits?limit=1" 2>/dev/null || echo "[]")
|
|
|
|
commit_date=$(echo "$response" | python3 -c "
|
|
import json, sys, datetime
|
|
try:
|
|
commits = json.load(sys.stdin)
|
|
if commits:
|
|
ts = commits[0]['created']
|
|
dt = datetime.datetime.fromisoformat(ts.replace('Z', '+00:00'))
|
|
print(int(dt.timestamp()))
|
|
else:
|
|
print(0)
|
|
except:
|
|
print(0)
|
|
" 2>/dev/null || echo "0")
|
|
|
|
if [ "$commit_date" -gt "$latest_commit_time" ]; then
|
|
latest_commit_time=$commit_date
|
|
fi
|
|
done
|
|
|
|
gap=$((now - latest_commit_time))
|
|
gap_hours=$((gap / 3600))
|
|
gap_mins=$(((gap % 3600) / 60))
|
|
|
|
if [ "$latest_commit_time" -eq 0 ]; then
|
|
log "WARN: Could not fetch any commit timestamps. API may be down."
|
|
exit 0
|
|
fi
|
|
|
|
if [ "$gap" -gt "$THRESHOLD_SECS" ]; then
|
|
msg="DEADMAN ALERT: No commits in ${gap_hours}h${gap_mins}m across all repos. Loops may be dead. Last commit: $(date -r "$latest_commit_time" '+%Y-%m-%d %H:%M' 2>/dev/null || echo 'unknown')"
|
|
log "ALERT: $msg"
|
|
|
|
# Send Telegram alert
|
|
if [ -n "$TELEGRAM_TOKEN" ]; then
|
|
curl -sf --max-time 10 -X POST \
|
|
"https://api.telegram.org/bot${TELEGRAM_TOKEN}/sendMessage" \
|
|
-d "chat_id=${TELEGRAM_CHAT}" \
|
|
-d "text=${msg}" >/dev/null 2>&1 || true
|
|
fi
|
|
else
|
|
log "OK: Last commit ${gap_hours}h${gap_mins}m ago (threshold: ${THRESHOLD_HOURS}h)"
|
|
fi
|