fix(rca): VPS Gitea heartbeat for Ezra/Bezalel @mentions (#579)

Root cause: Ezra and Bezalel do not respond to Gitea @mention tagging.

Two compounding causes:
1. Mac-local gitea-event-watcher.py enqueues work for ezra/bezalel but
   the dispatch queue lives on the Mac — VPS agents have no reader.
2. No VPS-native polling existed to detect mentions on those boxes.

Fix: standalone VPS polling heartbeat that runs on each VPS box:
- Polls Gitea API every 5 min for @mentions and assignments
- Tracks seen events in per-agent state file (deduplication)
- Dispatches to local `hermes chat` with full issue context
- Supports cron or systemd deployment

Files:
- scripts/vps-gitea-heartbeat.py — polling script (Python, ~250 lines)
- scripts/vps-gitea-heartbeat-install.sh — install script (cron/systemd)
- skills/autonomous-ai-agents/vps-gitea-heartbeat/SKILL.md — docs

Closes #579

scripts/vps-gitea-heartbeat-install.sh

@@ -0,0 +1,102 @@
+#!/bin/bash
+# ══════════════════════════════════════════════
+# Install VPS Gitea Heartbeat on Ezra or Bezalel
+# Sets up systemd service or crontab for polling
+# ══════════════════════════════════════════════
+
+set -euo pipefail
+
+AGENT="${1:-}"
+METHOD="${2:-cron}"  # "cron" or "systemd"
+
+if [ -z "$AGENT" ]; then
+    echo "Usage: $0 <agent-name> [cron|systemd]"
+    echo "  agent-name: ezra or bezalel"
+    echo "  method: cron (default) or systemd"
+    exit 1
+fi
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+HEARTBEAT_SCRIPT="$SCRIPT_DIR/vps-gitea-heartbeat.py"
+
+echo "════════════════════════════════════════"
+echo "  VPS Gitea Heartbeat Installer"
+echo "  Agent: $AGENT"
+echo "  Method: $METHOD"
+echo "════════════════════════════════════════"
+
+# Verify script exists
+if [ ! -f "$HEARTBEAT_SCRIPT" ]; then
+    echo "ERROR: $HEARTBEAT_SCRIPT not found"
+    exit 1
+fi
+
+# Verify hermes is available
+if ! command -v hermes &>/dev/null; then
+    echo "WARNING: hermes not found on PATH. Heartbeat will fail to dispatch."
+fi
+
+# Verify Gitea token exists
+TOKEN_FILE="$HOME/.hermes/gitea_token_vps"
+if [ ! -f "$TOKEN_FILE" ]; then
+    echo "WARNING: $TOKEN_FILE not found."
+    echo "  Create it with: echo 'YOUR_TOKEN' > $TOKEN_FILE"
+fi
+
+# Create log directory
+mkdir -p "$HOME/.hermes/logs/gitea-heartbeat"
+
+if [ "$METHOD" = "systemd" ]; then
+    # ── Systemd service ────────────────────────────────
+    SERVICE_FILE="/etc/systemd/system/gitea-heartbeat-${AGENT}.service"
+
+    echo "Creating systemd service: $SERVICE_FILE"
+
+    cat > "$SERVICE_FILE" <<SERVICE
+[Unit]
+Description=Gitea Heartbeat for @${AGENT}
+After=network.target
+
+[Service]
+Type=simple
+User=$(whoami)
+WorkingDirectory=$HOME
+ExecStart=$(which python3) $HEARTBEAT_SCRIPT --agent $AGENT --daemon --interval 300
+Restart=always
+RestartSec=30
+Environment=HOME=$HOME
+Environment=PATH=$PATH
+
+[Install]
+WantedBy=multi-user.target
+SERVICE
+
+    sudo systemctl daemon-reload
+    sudo systemctl enable "gitea-heartbeat-${AGENT}"
+    sudo systemctl start "gitea-heartbeat-${AGENT}"
+    echo "Service installed and started."
+    echo "Check status: sudo systemctl status gitea-heartbeat-${AGENT}"
+    echo "View logs:    journalctl -u gitea-heartbeat-${AGENT} -f"
+
+else
+    # ── Crontab ────────────────────────────────────────
+    CRON_LINE="*/5 * * * * $(which python3) $HEARTBEAT_SCRIPT --agent $AGENT --once >> $HOME/.hermes/logs/gitea-heartbeat/cron.log 2>&1"
+
+    echo "Adding crontab entry..."
+    # Check if already exists
+    if crontab -l 2>/dev/null | grep -q "gitea-heartbeat.*--agent $AGENT"; then
+        echo "Crontab entry already exists. Updating..."
+        crontab -l 2>/dev/null | grep -v "gitea-heartbeat.*--agent $AGENT" | crontab -
+    fi
+
+    (crontab -l 2>/dev/null; echo "$CRON_LINE") | crontab -
+    echo "Crontab installed."
+    echo "Entry: $CRON_LINE"
+    echo ""
+    echo "View crontab: crontab -l"
+    echo "View logs:    tail -f ~/.hermes/logs/gitea-heartbeat/cron.log"
+fi
+
+echo ""
+echo "Test: python3 $HEARTBEAT_SCRIPT --agent $AGENT --once"
+echo "════════════════════════════════════════"

scripts/vps-gitea-heartbeat.py

@@ -0,0 +1,262 @@
+#!/usr/bin/env python3
+"""
+VPS Gitea Heartbeat — Poll Gitea for @mentions and dispatch locally.
+
+Runs on Ezra or Bezalel VPS boxes. Polls Gitea every 5 minutes for:
+- Issue comments mentioning this agent (@ezra, @bezalel, etc.)
+- Issues assigned to this agent
+- New comments on issues this agent is working on
+
+Dispatches to local `hermes chat` with the issue context.
+
+Usage:
+    python3 vps-gitea-heartbeat.py --agent ezra
+    python3 vps-gitea-heartbeat.py --agent bezalel --once  # single poll
+    python3 vps-gitea-heartbeat.py --agent ezra --daemon    # continuous polling
+"""
+
+import argparse
+import hashlib
+import json
+import os
+import subprocess
+import sys
+import time
+import urllib.request
+from datetime import datetime, timezone
+from pathlib import Path
+
+# ── Config ──────────────────────────────────────────────
+GITEA = os.environ.get("GITEA_URL", "https://forge.alexanderwhitestone.com")
+TOKEN_FILE = os.environ.get("GITEA_TOKEN_FILE", os.path.expanduser("~/.hermes/gitea_token_vps"))
+STATE_FILE_TPL = os.environ.get("STATE_FILE", "~/.hermes/gitea-heartbeat-{agent}.json")
+LOG_DIR = os.environ.get("LOG_DIR", os.path.expanduser("~/.hermes/logs/gitea-heartbeat"))
+POLL_INTERVAL = int(os.environ.get("POLL_INTERVAL", "300"))  # 5 min
+DISPATCH_TIMEOUT = int(os.environ.get("DISPATCH_TIMEOUT", "600"))  # 10 min for hermes chat
+
+KNOWN_AGENTS = {"timmy", "ezra", "bezalel", "allegro", "claude", "gemini", "grok", "kimi", "fenrir", "manus", "perplexity", "rockachopa"}
+
+REPOS_TO_WATCH = [
+    "Timmy_Foundation/hermes-agent",
+    "Timmy_Foundation/the-nexus",
+    "Timmy_Foundation/timmy-config",
+    "Timmy_Foundation/timmy-home",
+    "Timmy_Foundation/the-beacon",
+]
+
+
+# ── Utilities ───────────────────────────────────────────
+
+def log(msg: str, level: str = "INFO"):
+    ts = datetime.now(timezone.utc).strftime("%H:%M:%S")
+    line = f"[{ts}] [{level}] {msg}"
+    print(line)
+    try:
+        os.makedirs(LOG_DIR, exist_ok=True)
+        log_file = os.path.join(LOG_DIR, f"{datetime.now(timezone.utc).strftime('%Y-%m-%d')}.log")
+        with open(log_file, "a") as f:
+            f.write(line + "\n")
+    except Exception:
+        pass
+
+
+def load_token() -> str:
+    try:
+        return Path(TOKEN_FILE).read_text().strip()
+    except Exception:
+        return os.environ.get("GITEA_TOKEN", "")
+
+
+def load_state(agent: str) -> dict:
+    path = os.path.expanduser(STATE_FILE_TPL.format(agent=agent))
+    try:
+        return json.loads(Path(path).read_text())
+    except Exception:
+        return {"seen_comments": {}, "seen_issues": {}, "last_run": None}
+
+
+def save_state(agent: str, state: dict):
+    path = os.path.expanduser(STATE_FILE_TPL.format(agent=agent))
+    os.makedirs(os.path.dirname(path), exist_ok=True)
+    state["last_run"] = datetime.now(timezone.utc).isoformat()
+    with open(path, "w") as f:
+        json.dump(state, f, indent=2)
+
+
+def gitea_api(path: str, token: str) -> dict | list | None:
+    url = f"{GITEA}/api/v1{path}"
+    req = urllib.request.Request(url, headers={"Authorization": f"token {token}"})
+    try:
+        with urllib.request.urlopen(req, timeout=20) as resp:
+            return json.loads(resp.read())
+    except Exception as e:
+        log(f"API error: {path} — {e}", "WARN")
+        return None
+
+
+# ── Core Logic ──────────────────────────────────────────
+
+def hash_key(*parts) -> str:
+    return hashlib.sha256("|".join(str(p) for p in parts).encode()).hexdigest()[:16]
+
+
+def check_repo(repo: str, agent: str, token: str, state: dict) -> list:
+    """Check a single repo for actionable events. Returns list of tasks to dispatch."""
+    tasks = []
+    owner, repo_name = repo.split("/")
+
+    # 1. Check open issues assigned to this agent
+    issues = gitea_api(f"/repos/{repo}/issues?state=open&limit=30&sort=recentupdate", token)
+    if not isinstance(issues, list):
+        return tasks
+
+    for issue in issues:
+        issue_num = issue.get("number", 0)
+        issue_key = f"{repo}#{issue_num}"
+        assignee = ((issue.get("assignee") or {}).get("login") or "").lower()
+
+        # Fetch comments
+        comments = gitea_api(f"/repos/{repo}/issues/{issue_num}/comments?limit=10&sort=created", token)
+        if not isinstance(comments, list):
+            comments = []
+
+        # Check for new @mention in comments
+        for c in comments:
+            ckey = f"{issue_key}/comment-{c['id']}"
+            if ckey in state.get("seen_comments", {}):
+                continue
+
+            commenter = ((c.get("user") or {}).get("login") or "").lower()
+            body = (c.get("body", "") or "").lower()
+
+            # Skip self-mentions and other agent mentions
+            if commenter == agent or commenter in KNOWN_AGENTS:
+                state.setdefault("seen_comments", {})[ckey] = True
+                continue
+
+            if f"@{agent}" in body:
+                log(f"MENTION @{agent} in {issue_key} comment {c['id']} by {commenter}")
+                tasks.append({
+                    "type": "mention",
+                    "repo": repo,
+                    "issue": issue_num,
+                    "title": issue.get("title", ""),
+                    "comment_by": commenter,
+                    "comment_body": (c.get("body", "") or "")[:500],
+                    "work_id": f"{issue_key}/mention-{c['id']}",
+                })
+                state.setdefault("seen_comments", {})[ckey] = True
+
+            # Mark as seen regardless
+            state.setdefault("seen_comments", {})[ckey] = True
+
+        # Check for assignment
+        if assignee == agent:
+            hk = hash_key(issue_key, issue.get("updated_at", ""))
+            if state.get("seen_issues", {}).get(issue_key) != hk:
+                state.setdefault("seen_issues", {})[issue_key] = hk
+                # Only dispatch if there are new comments
+                new_comments = [c for c in comments if f"{issue_key}/comment-{c['id']}" not in state.get("seen_comments", {})]
+                if new_comments or not state.get("seen_issues", {}).get(issue_key):
+                    log(f"ASSIGNED {issue_key} to @{agent}")
+                    tasks.append({
+                        "type": "assigned",
+                        "repo": repo,
+                        "issue": issue_num,
+                        "title": issue.get("title", ""),
+                        "work_id": f"{issue_key}/assign",
+                    })
+
+    return tasks
+
+
+def dispatch_task(task: dict, agent: str):
+    """Dispatch a task to local hermes chat."""
+    prompt = f"""You are {agent}. A Gitea event requires your attention.
+
+Type: {task['type']}
+Repo: {task['repo']}
+Issue: #{task['issue']} — {task['title']}
+
+"""
+    if task.get("comment_by"):
+        prompt += f"Comment by @{task['comment_by']}:\n{task.get('comment_body', '')}\n\n"
+
+    prompt += f"""Steps:
+1. Read the full issue: https://forge.alexanderwhitestone.com/{task['repo']}/issues/{task['issue']}
+2. Understand what is being asked
+3. Clone the repo if needed, make changes
+4. Commit, push, and create a PR
+5. Comment on the issue acknowledging the work
+
+Be terse. One issue. Commit early."""
+
+    log(f"Dispatching to hermes chat: {task['work_id']}")
+    try:
+        result = subprocess.run(
+            ["hermes", "chat", "--quiet", "-q", prompt],
+            capture_output=True, text=True, timeout=DISPATCH_TIMEOUT
+        )
+        if result.returncode == 0:
+            log(f"Dispatch complete: {task['work_id']}")
+        else:
+            log(f"Dispatch failed (exit {result.returncode}): {result.stderr[:200]}", "WARN")
+    except subprocess.TimeoutExpired:
+        log(f"Dispatch timed out ({DISPATCH_TIMEOUT}s): {task['work_id']}", "WARN")
+    except Exception as e:
+        log(f"Dispatch error: {e}", "ERROR")
+
+
+def run_once(agent: str, token: str):
+    """Single poll cycle."""
+    log(f"Polling Gitea for @{agent} events...")
+    state = load_state(agent)
+    all_tasks = []
+
+    for repo in REPOS_TO_WATCH:
+        tasks = check_repo(repo, agent, token, state)
+        all_tasks.extend(tasks)
+
+    if all_tasks:
+        log(f"Found {len(all_tasks)} task(s) for @{agent}")
+        for task in all_tasks:
+            dispatch_task(task, agent)
+    else:
+        log(f"No new events for @{agent}")
+
+    save_state(agent, state)
+    return len(all_tasks)
+
+
+# ── CLI ─────────────────────────────────────────────────
+
+def main():
+    parser = argparse.ArgumentParser(description="VPS Gitea Heartbeat — Poll for @mentions")
+    parser.add_argument("--agent", required=True, help="Agent name (e.g., ezra, bezalel)")
+    parser.add_argument("--once", action="store_true", help="Single poll, then exit")
+    parser.add_argument("--daemon", action="store_true", help="Continuous polling loop")
+    parser.add_argument("--interval", type=int, default=POLL_INTERVAL, help="Poll interval in seconds")
+
+    args = parser.parse_args()
+    agent = args.agent.lower()
+
+    token = load_token()
+    if not token:
+        log("No Gitea token found. Set GITEA_TOKEN env var or create ~/.hermes/gitea_token_vps", "ERROR")
+        sys.exit(1)
+
+    if args.daemon:
+        log(f"Starting daemon for @{agent} (interval: {args.interval}s)")
+        while True:
+            try:
+                run_once(agent, token)
+            except Exception as e:
+                log(f"Poll error: {e}", "ERROR")
+            time.sleep(args.interval)
+    else:
+        count = run_once(agent, token)
+        sys.exit(0 if count == 0 else 1)
+
+
+if __name__ == "__main__":
+    sys.exit(main())

skills/autonomous-ai-agents/vps-gitea-heartbeat/SKILL.md

@@ -0,0 +1,140 @@
+---
+name: vps-gitea-heartbeat
+description: "RCA fix: VPS agents (Ezra, Bezalel) not responding to Gitea @mentions. Polls Gitea for mentions and dispatches to local hermes chat."
+version: 1.0.0
+author: Timmy Time
+license: MIT
+metadata:
+  hermes:
+    tags: [gitea, heartbeat, vps, dispatch, ezra, bezalel, rca]
+    related_skills: [gitea-workflow-automation, sprint-backlog-burner]
+---
+
+# VPS Gitea Heartbeat
+
+## Problem
+
+Tagging @ezra or @bezalel in a Gitea issue comment produced no response.
+
+## Root Causes (two compounding)
+
+1. **Ezra/Bezalel in AGENT_USERS with `vps: True`** — the Mac-local `gitea-event-watcher.py` detects mentions and enqueues work, but the dispatch queue is on the Mac. VPS agents have no process reading it.
+
+2. **No VPS-native polling** — Ezra (143.198.27.163) and Bezalel (159.203.146.185) run `hermes gateway` on separate VPS boxes. No process on those boxes polls Gitea for mentions.
+
+## Solution
+
+A standalone polling heartbeat that runs on each VPS box. Every 5 minutes, it:
+1. Polls Gitea API for issues assigned to this agent
+2. Scans recent comments for @mentions of this agent
+3. Dispatches actionable events to local `hermes chat`
+4. Tracks seen events to avoid duplicates
+
+## Files
+
+```
+scripts/vps-gitea-heartbeat.py          # Polling script (Python)
+scripts/vps-gitea-heartbeat-install.sh  # Install script (cron or systemd)
+```
+
+## Installation
+
+### On Ezra VPS (143.198.27.163):
+```bash
+scp scripts/vps-gitea-heartbeat.py root@ezra:~/.hermes/bin/
+ssh root@ezra 'bash -s' < scripts/vps-gitea-heartbeat-install.sh ezra cron
+```
+
+### On Bezalel VPS (159.203.146.185):
+```bash
+scp scripts/vps-gitea-heartbeat.py root@bezalel:~/.hermes/bin/
+ssh root@bezalel 'bash -s' < scripts/vps-gitea-heartbeat-install.sh bezalel cron
+```
+
+### Manual test:
+```bash
+python3 vps-gitea-heartbeat.py --agent ezra --once
+```
+
+## Usage
+
+```bash
+# Single poll (for testing or crontab)
+python3 vps-gitea-heartbeat.py --agent ezra --once
+
+# Daemon mode (continuous polling every 5 min)
+python3 vps-gitea-heartbeat.py --agent bezalel --daemon --interval 300
+```
+
+## How It Works
+
+```
+┌─────────────────────┐
+│  Gitea API          │
+│  (forge.alexander…) │
+└─────────┬───────────┘
+          │ poll every 5 min
+          ▼
+┌─────────────────────┐
+│  vps-gitea-heartbeat│  (runs on Ezra/Bezalel VPS)
+│  - check mentions   │
+│  - check assignments│
+│  - track seen state │
+└─────────┬───────────┘
+          │ dispatch
+          ▼
+┌─────────────────────┐
+│  hermes chat        │  (local on VPS)
+│  - implement issue  │
+│  - commit, push, PR │
+└─────────────────────┘
+```
+
+## State Tracking
+
+Each agent has its own state file: `~/.hermes/gitea-heartbeat-{agent}.json`
+
+```json
+{
+  "seen_comments": {
+    "Timmy_Foundation/timmy-home#123/comment-456": true
+  },
+  "seen_issues": {
+    "Timmy_Foundation/timmy-home#123": "hash_of_updated_at"
+  },
+  "last_run": "2026-04-13T20:00:00Z"
+}
+```
+
+Events are deduplicated — each comment/issue is processed only once.
+
+## Configuration (env vars)
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `GITEA_URL` | `https://forge.alexanderwhitestone.com` | Gitea instance URL |
+| `GITEA_TOKEN_FILE` | `~/.hermes/gitea_token_vps` | Token file path |
+| `GITEA_TOKEN` | (none) | Token env var fallback |
+| `POLL_INTERVAL` | `300` | Seconds between polls |
+| `DISPATCH_TIMEOUT` | `600` | Max seconds for hermes chat |
+| `LOG_DIR` | `~/.hermes/logs/gitea-heartbeat` | Log directory |
+
+## Repos Watched
+
+- Timmy_Foundation/hermes-agent
+- Timmy_Foundation/the-nexus
+- Timmy_Foundation/timmy-config
+- Timmy_Foundation/timmy-home
+- Timmy_Foundation/the-beacon
+
+## Pitfalls
+
+1. **Token must be on the VPS** — Copy `~/.hermes/gitea_token_vps` to each VPS box.
+
+2. **hermes must be on PATH** — The heartbeat dispatches via `hermes chat`. If hermes isn't installed on the VPS, dispatch fails.
+
+3. **5-minute latency** — Crontab polls every 5 minutes. For faster response, use `--daemon` mode.
+
+4. **Duplicate prevention** — Each comment is seen only once. If dispatch fails, the comment is still marked as seen. To retry, delete the state file.
+
+5. **Agent name must match Gitea username** — The script checks for `@{agent}` in comment bodies, case-insensitive.