Compare commits

...

10 Commits

Author SHA1 Message Date
Alexander Whitestone
0a13347e39 feat: FLEET-010/011/012 — Phase 3 and 4 fleet capabilities
FLEET-010: Cross-agent task delegation protocol
- Keyword-based heuristic assigns unassigned issues to agents
- Supports: claw-code, gemini, ezra, bezalel, timmy
- Delegation logging and status dashboard
- Auto-comments on assigned issues

FLEET-011: Local model pipeline and fallback chain
- Checks Ollama reachability and model availability
- 4-model chain: hermes4:14b -> qwen2.5:7b -> phi3:3.8b -> gemma3:1b
- Tests each model with live inference on every run
- Fallback verification: finds first responding model
- Chain configuration via ~/.local/timmy/fleet-resources/model-chain.json

FLEET-012: Agent lifecycle manager
- Full lifecycle: provision -> deploy -> monitor -> retire
- Heartbeat detection with 24h idle threshold
- Task completion/failure tracking
- Agent Fleet Status dashboard

Fixes timmy-home#563 (delegation), #564 (model pipeline), #565 (lifecycle)
2026-04-07 12:43:10 -04:00
0c950f991c Merge pull request '[ORCHESTRATOR-4] Evaluate CrewAI for Phase 2 integration' (#361) from ezra/issue-358 into main 2026-04-07 16:35:40 +00:00
ezra
fe7c5018e3 eval(crewai): PoC crew + evaluation for Phase 2 integration
- Install CrewAI v1.13.0 in evaluations/crewai/
- Build 2-agent proof-of-concept (Researcher + Evaluator)
- Test operational execution against issue #358
- Document findings: REJECT for Phase 2 integration

CrewAI's 500+ MB dependency footprint, memory-model drift
from Gitea-as-truth, and external API fragility outweigh
its agent-role syntax benefits. Recommend evolving the
existing Huey stack instead.

Closes #358
2026-04-07 16:25:21 +00:00
c1c3aaa681 Merge pull request 'feat: genchi-genbutsu — verify world state, not log vibes (#348)' (#360) from ezra/issue-348 into main 2026-04-07 16:23:35 +00:00
d023512858 Merge pull request 'feat: FLEET-003 - Fleet capacity inventory with resource baselines' (#353) from timmy/fleet-capacity-inventory into main 2026-04-07 16:23:22 +00:00
e5e01e36c9 Merge pull request '[KAIZEN] Automated retrospective after every burn cycle (fixes #349)' (#352) from ezra/issue-349 into main 2026-04-07 16:23:17 +00:00
ezra
e5055d269b feat: genchi-genbutsu — verify world state, not log vibes (#348)
Implement 現地現物 (Genchi Genbutsu) post-completion verification:

- Add bin/genchi-genbutsu.sh performing 5 world-state checks:
  1. Branch exists on remote
  2. PR exists
  3. PR has real file changes (> 0)
  4. PR is mergeable
  5. Issue has a completion comment from the agent

- Wire verification into all agent loops:
  - bin/claude-loop.sh: call genchi-genbutsu before merge/close
  - bin/gemini-loop.sh: delegate existing inline checks to genchi-genbutsu
  - bin/agent-loop.sh: resurrect generic agent loop with genchi-genbutsu wired in

- Update metrics JSONL to include 'verified' field for all loops

- Update burn monitor (tasks.py velocity_tracking):
  - Report verified_completion count alongside raw completions
  - Dashboard shows verified trend history

- Update morning report (tasks.py good_morning_report):
  - Count only verified completions from the last 24h
  - Surface verification failures in the report body

Fixes #348
Refs #345
2026-04-07 16:12:05 +00:00
Alexander Whitestone
277d21aef6 feat: FLEET-007 — Auto-restart agent (self-healing processes)
Daemon that monitors key services and restarts them automatically:
- Local: hermes-gateway, ollama, codeclaw-heartbeat
- Ezra: gitea, nginx, hermes-agent
- Allegro hermes-agent
- Bezalel: hermes-agent, evennia
- Max 3 restart attempts per service per cycle (prevents loops)
- 1-hour cooldown after max retries with Telegram escalation
- Restart log at ~/.local/timmy/fleet-health/restarts.log
- Modes: check now (--status for history, --daemon for continuous)

Fixes timmy-home#560
2026-04-07 12:04:33 -04:00
Ezra
2e64b160b5 [KAIZEN] Harden retro scheduling, chunking, and tests (#349)
- Add Kaizen Retro to cron/jobs.json with explicit local model/provider
- Add Telegram message chunking for reports approaching the 4096-char limit
- Fix classify_issue_type false positives on short substrings (ci in cleanup)
- Add 28 unit tests covering classification, max-attempts detection,
  suggestion generation, report formatting, and Telegram chunking
2026-04-07 15:58:58 +00:00
Ezra
f18955ea90 [KAIZEN] Implement automated burn-cycle retrospective (fixes #349)
- Add bin/kaizen-retro.sh entry point and scripts/kaizen_retro.py
- Analyze closed issues, merged PRs, and stale/max-attempts issues
- Report success rates by agent, repo, and issue type
- Generate one concrete improvement suggestion per cycle
- Post retro to Telegram and comment on the latest morning report issue
- Wire into Huey as kaizen_retro() task at 07:15 daily
- Extend gitea_client.py with since param for list_issues and
  created_at/updated_at fields on PullRequest
2026-04-07 15:57:21 +00:00
19 changed files with 2464 additions and 84 deletions

11
.gitignore vendored
View File

@@ -1,9 +1,8 @@
# Secrets
*.token
*.key
*.secret
# Local state
*.pyc
*.pyo
*.egg-info/
dist/
build/
*.db
*.db-wal
*.db-shm

273
bin/agent-loop.sh Executable file
View File

@@ -0,0 +1,273 @@
#!/usr/bin/env bash
# agent-loop.sh — Universal agent dev loop with Genchi Genbutsu verification
#
# Usage: agent-loop.sh <agent-name> [num-workers]
# agent-loop.sh claude 2
# agent-loop.sh gemini 1
#
# Dispatches via agent-dispatch.sh, then verifies with genchi-genbutsu.sh.
set -uo pipefail
AGENT="${1:?Usage: agent-loop.sh <agent-name> [num-workers]}"
NUM_WORKERS="${2:-1}"
# Resolve agent tool and model from config or fallback
case "$AGENT" in
claude) TOOL="claude"; MODEL="sonnet" ;;
gemini) TOOL="gemini"; MODEL="gemini-2.5-pro-preview-05-06" ;;
grok) TOOL="opencode"; MODEL="grok-3-fast" ;;
*) TOOL="$AGENT"; MODEL="" ;;
esac
# === CONFIG ===
GITEA_URL="${GITEA_URL:-https://forge.alexanderwhitestone.com}"
GITEA_TOKEN="${GITEA_TOKEN:-}"
WORKTREE_BASE="$HOME/worktrees"
LOG_DIR="$HOME/.hermes/logs"
LOCK_DIR="$LOG_DIR/${AGENT}-locks"
SKIP_FILE="$LOG_DIR/${AGENT}-skip-list.json"
ACTIVE_FILE="$LOG_DIR/${AGENT}-active.json"
TIMEOUT=600
COOLDOWN=30
mkdir -p "$LOG_DIR" "$WORKTREE_BASE" "$LOCK_DIR"
[ -f "$SKIP_FILE" ] || echo '{}' > "$SKIP_FILE"
echo '{}' > "$ACTIVE_FILE"
# === SHARED FUNCTIONS ===
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] ${AGENT}: $*" >> "$LOG_DIR/${AGENT}-loop.log"
}
lock_issue() {
local key="$1"
mkdir "$LOCK_DIR/$key.lock" 2>/dev/null && echo $$ > "$LOCK_DIR/$key.lock/pid"
}
unlock_issue() {
rm -rf "$LOCK_DIR/$1.lock" 2>/dev/null
}
mark_skip() {
local issue_num="$1" reason="$2"
python3 -c "
import json, time, fcntl
with open('${SKIP_FILE}', 'r+') as f:
fcntl.flock(f, fcntl.LOCK_EX)
try: skips = json.load(f)
except: skips = {}
failures = skips.get(str($issue_num), {}).get('failures', 0) + 1
skip_hours = 6 if failures >= 3 else 1
skips[str($issue_num)] = {'until': time.time() + (skip_hours * 3600), 'reason': '$reason', 'failures': failures}
f.seek(0); f.truncate()
json.dump(skips, f, indent=2)
" 2>/dev/null
}
get_next_issue() {
python3 -c "
import json, sys, time, urllib.request, os
token = '${GITEA_TOKEN}'
base = '${GITEA_URL}'
repos = ['Timmy_Foundation/the-nexus', 'Timmy_Foundation/timmy-config', 'Timmy_Foundation/hermes-agent']
try:
with open('${SKIP_FILE}') as f: skips = json.load(f)
except: skips = {}
try:
with open('${ACTIVE_FILE}') as f: active = json.load(f); active_issues = {v['issue'] for v in active.values()}
except: active_issues = set()
all_issues = []
for repo in repos:
url = f'{base}/api/v1/repos/{repo}/issues?state=open&type=issues&limit=50&sort=created'
req = urllib.request.Request(url, headers={'Authorization': f'token {token}'})
try:
resp = urllib.request.urlopen(req, timeout=10)
issues = json.loads(resp.read())
for i in issues: i['_repo'] = repo
all_issues.extend(issues)
except: continue
for i in sorted(all_issues, key=lambda x: x['title'].lower()):
assignees = [a['login'] for a in (i.get('assignees') or [])]
if assignees and '${AGENT}' not in assignees: continue
num_str = str(i['number'])
if num_str in active_issues: continue
if skips.get(num_str, {}).get('until', 0) > time.time(): continue
lock = '${LOCK_DIR}/' + i['_repo'].replace('/', '-') + '-' + num_str + '.lock'
if os.path.isdir(lock): continue
owner, name = i['_repo'].split('/')
print(json.dumps({'number': i['number'], 'title': i['title'], 'repo_owner': owner, 'repo_name': name, 'repo': i['_repo']}))
sys.exit(0)
print('null')
" 2>/dev/null
}
# === WORKER FUNCTION ===
run_worker() {
local worker_id="$1"
log "WORKER-${worker_id}: Started"
while true; do
issue_json=$(get_next_issue)
if [ "$issue_json" = "null" ] || [ -z "$issue_json" ]; then
sleep 30
continue
fi
issue_num=$(echo "$issue_json" | python3 -c "import sys,json; print(json.load(sys.stdin)['number'])")
issue_title=$(echo "$issue_json" | python3 -c "import sys,json; print(json.load(sys.stdin)['title'])")
repo_owner=$(echo "$issue_json" | python3 -c "import sys,json; print(json.load(sys.stdin)['repo_owner'])")
repo_name=$(echo "$issue_json" | python3 -c "import sys,json; print(json.load(sys.stdin)['repo_name'])")
issue_key="${repo_owner}-${repo_name}-${issue_num}"
branch="${AGENT}/issue-${issue_num}"
worktree="${WORKTREE_BASE}/${AGENT}-w${worker_id}-${issue_num}"
if ! lock_issue "$issue_key"; then
sleep 5
continue
fi
log "WORKER-${worker_id}: === ISSUE #${issue_num}: ${issue_title} (${repo_owner}/${repo_name}) ==="
# Clone / checkout
rm -rf "$worktree" 2>/dev/null
CLONE_URL="http://${AGENT}:${GITEA_TOKEN}@143.198.27.163:3000/${repo_owner}/${repo_name}.git"
if git ls-remote --heads "$CLONE_URL" "$branch" 2>/dev/null | grep -q "$branch"; then
git clone --depth=50 -b "$branch" "$CLONE_URL" "$worktree" >/dev/null 2>&1
else
git clone --depth=1 -b main "$CLONE_URL" "$worktree" >/dev/null 2>&1
cd "$worktree" && git checkout -b "$branch" >/dev/null 2>&1
fi
cd "$worktree"
# Generate prompt
prompt=$(bash "$(dirname "$0")/agent-dispatch.sh" "$AGENT" "$issue_num" "${repo_owner}/${repo_name}")
CYCLE_START=$(date +%s)
set +e
if [ "$TOOL" = "claude" ]; then
env -u CLAUDECODE gtimeout "$TIMEOUT" claude \
--print --model "$MODEL" --dangerously-skip-permissions \
-p "$prompt" </dev/null >> "$LOG_DIR/${AGENT}-${issue_num}.log" 2>&1
elif [ "$TOOL" = "gemini" ]; then
gtimeout "$TIMEOUT" gemini -p "$prompt" --yolo \
</dev/null >> "$LOG_DIR/${AGENT}-${issue_num}.log" 2>&1
else
gtimeout "$TIMEOUT" "$TOOL" "$prompt" \
</dev/null >> "$LOG_DIR/${AGENT}-${issue_num}.log" 2>&1
fi
exit_code=$?
set -e
CYCLE_END=$(date +%s)
CYCLE_DURATION=$((CYCLE_END - CYCLE_START))
# Salvage
cd "$worktree" 2>/dev/null || true
DIRTY=$(git status --porcelain 2>/dev/null | wc -l | tr -d ' ')
if [ "${DIRTY:-0}" -gt 0 ]; then
git add -A 2>/dev/null
git commit -m "WIP: ${AGENT} progress on #${issue_num}
Automated salvage commit — agent session ended (exit $exit_code)." 2>/dev/null || true
fi
UNPUSHED=$(git log --oneline "origin/main..HEAD" 2>/dev/null | wc -l | tr -d ' ')
if [ "${UNPUSHED:-0}" -gt 0 ]; then
git push -u origin "$branch" 2>/dev/null && \
log "WORKER-${worker_id}: Pushed $UNPUSHED commit(s) on $branch" || \
log "WORKER-${worker_id}: Push failed for $branch"
fi
# Create PR if needed
pr_num=$(curl -sf "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls?state=open&head=${repo_owner}:${branch}&limit=1" \
-H "Authorization: token ${GITEA_TOKEN}" | python3 -c "
import sys,json
prs = json.load(sys.stdin)
print(prs[0]['number'] if prs else '')
" 2>/dev/null)
if [ -z "$pr_num" ] && [ "${UNPUSHED:-0}" -gt 0 ]; then
pr_num=$(curl -sf -X POST "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls" \
-H "Authorization: token ${GITEA_TOKEN}" \
-H "Content-Type: application/json" \
-d "$(python3 -c "
import json
print(json.dumps({
'title': '${AGENT}: Issue #${issue_num}',
'head': '${branch}',
'base': 'main',
'body': 'Automated PR for issue #${issue_num}.\nExit code: ${exit_code}'
}))
")" | python3 -c "import sys,json; print(json.load(sys.stdin).get('number',''))" 2>/dev/null)
[ -n "$pr_num" ] && log "WORKER-${worker_id}: Created PR #${pr_num} for issue #${issue_num}"
fi
# ── Genchi Genbutsu: verify world state before declaring success ──
VERIFIED="false"
if [ "$exit_code" -eq 0 ]; then
log "WORKER-${worker_id}: SUCCESS #${issue_num} — running genchi-genbutsu"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
if verify_result=$("$SCRIPT_DIR/genchi-genbutsu.sh" "$repo_owner" "$repo_name" "$issue_num" "$branch" "$AGENT" 2>/dev/null); then
VERIFIED="true"
log "WORKER-${worker_id}: VERIFIED #${issue_num}"
if [ -n "$pr_num" ]; then
curl -sf -X POST "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}/merge" \
-H "Authorization: token ${GITEA_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"Do": "squash"}' >/dev/null 2>&1 || true
curl -sf -X PATCH "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/issues/${issue_num}" \
-H "Authorization: token ${GITEA_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"state": "closed"}' >/dev/null 2>&1 || true
log "WORKER-${worker_id}: PR #${pr_num} merged, issue #${issue_num} closed"
fi
consecutive_failures=0
else
verify_details=$(echo "$verify_result" | python3 -c "import sys,json; print(json.load(sys.stdin).get('details','unknown'))" 2>/dev/null || echo "unverified")
log "WORKER-${worker_id}: UNVERIFIED #${issue_num}$verify_details"
mark_skip "$issue_num" "unverified" 1
consecutive_failures=$((consecutive_failures + 1))
fi
elif [ "$exit_code" -eq 124 ]; then
log "WORKER-${worker_id}: TIMEOUT #${issue_num} (work saved in PR)"
consecutive_failures=$((consecutive_failures + 1))
else
log "WORKER-${worker_id}: FAILED #${issue_num} exit ${exit_code} (work saved in PR)"
consecutive_failures=$((consecutive_failures + 1))
fi
# ── METRICS ──
python3 -c "
import json, datetime
print(json.dumps({
'ts': datetime.datetime.utcnow().isoformat() + 'Z',
'agent': '${AGENT}',
'worker': $worker_id,
'issue': $issue_num,
'repo': '${repo_owner}/${repo_name}',
'outcome': 'success' if $exit_code == 0 else 'timeout' if $exit_code == 124 else 'failed',
'exit_code': $exit_code,
'duration_s': $CYCLE_DURATION,
'pr': '${pr_num:-}',
'verified': ${VERIFIED:-false}
}))
" >> "$LOG_DIR/${AGENT}-metrics.jsonl" 2>/dev/null
rm -rf "$worktree" 2>/dev/null
unlock_issue "$issue_key"
sleep "$COOLDOWN"
done
}
# === MAIN ===
log "=== Agent Loop Started — ${AGENT} with ${NUM_WORKERS} worker(s) ==="
rm -rf "$LOCK_DIR"/*.lock 2>/dev/null
for i in $(seq 1 "$NUM_WORKERS"); do
run_worker "$i" &
log "Launched worker $i (PID $!)"
sleep 3
done
wait

View File

@@ -468,24 +468,32 @@ print(json.dumps({
[ -n "$pr_num" ] && log "WORKER-${worker_id}: Created PR #${pr_num} for issue #${issue_num}"
fi
# ── Merge + close on success ──
# ── Genchi Genbutsu: verify world state before declaring success ──
VERIFIED="false"
if [ "$exit_code" -eq 0 ]; then
log "WORKER-${worker_id}: SUCCESS #${issue_num}"
if [ -n "$pr_num" ]; then
curl -sf -X POST "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}/merge" \
-H "Authorization: token ${GITEA_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"Do": "squash"}' >/dev/null 2>&1 || true
curl -sf -X PATCH "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/issues/${issue_num}" \
-H "Authorization: token ${GITEA_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"state": "closed"}' >/dev/null 2>&1 || true
log "WORKER-${worker_id}: PR #${pr_num} merged, issue #${issue_num} closed"
log "WORKER-${worker_id}: SUCCESS #${issue_num} — running genchi-genbutsu"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
if verify_result=$("$SCRIPT_DIR/genchi-genbutsu.sh" "$repo_owner" "$repo_name" "$issue_num" "$branch" "claude" 2>/dev/null); then
VERIFIED="true"
log "WORKER-${worker_id}: VERIFIED #${issue_num}"
if [ -n "$pr_num" ]; then
curl -sf -X POST "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}/merge" \
-H "Authorization: token ${GITEA_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"Do": "squash"}' >/dev/null 2>&1 || true
curl -sf -X PATCH "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/issues/${issue_num}" \
-H "Authorization: token ${GITEA_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"state": "closed"}' >/dev/null 2>&1 || true
log "WORKER-${worker_id}: PR #${pr_num} merged, issue #${issue_num} closed"
fi
consecutive_failures=0
else
verify_details=$(echo "$verify_result" | python3 -c "import sys,json; print(json.load(sys.stdin).get('details','unknown'))" 2>/dev/null || echo "unverified")
log "WORKER-${worker_id}: UNVERIFIED #${issue_num}$verify_details"
consecutive_failures=$((consecutive_failures + 1))
fi
consecutive_failures=0
elif [ "$exit_code" -eq 124 ]; then
log "WORKER-${worker_id}: TIMEOUT #${issue_num} (work saved in PR)"
consecutive_failures=$((consecutive_failures + 1))
@@ -522,6 +530,7 @@ print(json.dumps({
import json, datetime
print(json.dumps({
'ts': datetime.datetime.utcnow().isoformat() + 'Z',
'agent': 'claude',
'worker': $worker_id,
'issue': $issue_num,
'repo': '${repo_owner}/${repo_name}',
@@ -534,7 +543,8 @@ print(json.dumps({
'lines_removed': ${LINES_REMOVED:-0},
'salvaged': ${DIRTY:-0},
'pr': '${pr_num:-}',
'merged': $( [ '$OUTCOME' = 'success' ] && [ -n '${pr_num:-}' ] && echo 'true' || echo 'false' )
'merged': $( [ '$OUTCOME' = 'success' ] && [ -n '${pr_num:-}' ] && echo 'true' || echo 'false' ),
'verified': ${VERIFIED:-false}
}))
" >> "$METRICS_FILE" 2>/dev/null

View File

@@ -521,61 +521,63 @@ print(json.dumps({
[ -n "$pr_num" ] && log "WORKER-${worker_id}: Created PR #${pr_num} for issue #${issue_num}"
fi
# ── Verify finish semantics / classify failures ──
# ── Genchi Genbutsu: verify world state before declaring success ──
VERIFIED="false"
if [ "$exit_code" -eq 0 ]; then
log "WORKER-${worker_id}: SUCCESS #${issue_num} exited 0 — verifying push + PR + proof"
if ! remote_branch_exists "$branch"; then
log "WORKER-${worker_id}: BLOCKED #${issue_num} remote branch missing"
post_issue_comment "$repo_owner" "$repo_name" "$issue_num" "Loop gate blocked completion: remote branch ${branch} was not found on origin after Gemini exited. Issue remains open for retry."
mark_skip "$issue_num" "missing_remote_branch" 1
consecutive_failures=$((consecutive_failures + 1))
elif [ -z "$pr_num" ]; then
log "WORKER-${worker_id}: BLOCKED #${issue_num} no PR found"
post_issue_comment "$repo_owner" "$repo_name" "$issue_num" "Loop gate blocked completion: branch ${branch} exists remotely, but no PR was found. Issue remains open for retry."
mark_skip "$issue_num" "missing_pr" 1
consecutive_failures=$((consecutive_failures + 1))
log "WORKER-${worker_id}: SUCCESS #${issue_num} exited 0 — running genchi-genbutsu"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
if verify_result=$("$SCRIPT_DIR/genchi-genbutsu.sh" "$repo_owner" "$repo_name" "$issue_num" "$branch" "gemini" 2>/dev/null); then
VERIFIED="true"
log "WORKER-${worker_id}: VERIFIED #${issue_num}"
pr_state=$(get_pr_state "$repo_owner" "$repo_name" "$pr_num")
if [ "$pr_state" = "open" ]; then
curl -sf -X POST "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}/merge" \
-H "Authorization: token ${GITEA_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"Do": "squash"}' >/dev/null 2>&1 || true
pr_state=$(get_pr_state "$repo_owner" "$repo_name" "$pr_num")
fi
if [ "$pr_state" = "merged" ]; then
curl -sf -X PATCH "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/issues/${issue_num}" \
-H "Authorization: token ${GITEA_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"state": "closed"}' >/dev/null 2>&1 || true
issue_state=$(get_issue_state "$repo_owner" "$repo_name" "$issue_num")
if [ "$issue_state" = "closed" ]; then
log "WORKER-${worker_id}: VERIFIED #${issue_num} branch pushed, PR merged, comment present, issue closed"
consecutive_failures=0
else
log "WORKER-${worker_id}: BLOCKED #${issue_num} issue did not close after merge"
mark_skip "$issue_num" "issue_close_unverified" 1
consecutive_failures=$((consecutive_failures + 1))
fi
else
log "WORKER-${worker_id}: BLOCKED #${issue_num} merge not verified (state=${pr_state})"
mark_skip "$issue_num" "merge_unverified" 1
consecutive_failures=$((consecutive_failures + 1))
fi
else
pr_files=$(get_pr_file_count "$repo_owner" "$repo_name" "$pr_num")
if [ "${pr_files:-0}" -eq 0 ]; then
log "WORKER-${worker_id}: BLOCKED #${issue_num} PR #${pr_num} has 0 changed files"
curl -sf -X PATCH "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}" -H "Authorization: token ${GITEA_TOKEN}" -H "Content-Type: application/json" -d '{"state": "closed"}' >/dev/null 2>&1 || true
verify_details=$(echo "$verify_result" | python3 -c "import sys,json; print(json.load(sys.stdin).get('details','unknown'))" 2>/dev/null || echo "unverified")
verify_checks=$(echo "$verify_result" | python3 -c "import sys,json; print(json.load(sys.stdin).get('checks',''))" 2>/dev/null || echo "")
log "WORKER-${worker_id}: UNVERIFIED #${issue_num} $verify_details"
if echo "$verify_checks" | grep -q '"branch": false'; then
post_issue_comment "$repo_owner" "$repo_name" "$issue_num" "Loop gate blocked completion: remote branch ${branch} was not found on origin after Gemini exited. Issue remains open for retry."
mark_skip "$issue_num" "missing_remote_branch" 1
elif echo "$verify_checks" | grep -q '"pr": false'; then
post_issue_comment "$repo_owner" "$repo_name" "$issue_num" "Loop gate blocked completion: branch ${branch} exists remotely, but no PR was found. Issue remains open for retry."
mark_skip "$issue_num" "missing_pr" 1
elif echo "$verify_checks" | grep -q '"files": false'; then
curl -sf -X PATCH "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}" \
-H "Authorization: token ${GITEA_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"state": "closed"}' >/dev/null 2>&1 || true
post_issue_comment "$repo_owner" "$repo_name" "$issue_num" "PR #${pr_num} was closed automatically: it had 0 changed files (empty commit). Issue remains open for retry."
mark_skip "$issue_num" "empty_commit" 2
consecutive_failures=$((consecutive_failures + 1))
else
proof_status=$(proof_comment_status "$repo_owner" "$repo_name" "$issue_num" "$branch")
proof_state="${proof_status%%|*}"
proof_url="${proof_status#*|}"
if [ "$proof_state" != "ok" ]; then
log "WORKER-${worker_id}: BLOCKED #${issue_num} proof missing or incomplete (${proof_state})"
post_issue_comment "$repo_owner" "$repo_name" "$issue_num" "Loop gate blocked completion: PR #${pr_num} exists and has ${pr_files} changed file(s), but the required Proof block from Gemini is missing or incomplete. Issue remains open for retry."
mark_skip "$issue_num" "missing_proof" 1
consecutive_failures=$((consecutive_failures + 1))
else
log "WORKER-${worker_id}: PROOF verified ${proof_url}"
pr_state=$(get_pr_state "$repo_owner" "$repo_name" "$pr_num")
if [ "$pr_state" = "open" ]; then
curl -sf -X POST "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}/merge" -H "Authorization: token ${GITEA_TOKEN}" -H "Content-Type: application/json" -d '{"Do": "squash"}' >/dev/null 2>&1 || true
pr_state=$(get_pr_state "$repo_owner" "$repo_name" "$pr_num")
fi
if [ "$pr_state" = "merged" ]; then
curl -sf -X PATCH "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/issues/${issue_num}" -H "Authorization: token ${GITEA_TOKEN}" -H "Content-Type: application/json" -d '{"state": "closed"}' >/dev/null 2>&1 || true
issue_state=$(get_issue_state "$repo_owner" "$repo_name" "$issue_num")
if [ "$issue_state" = "closed" ]; then
log "WORKER-${worker_id}: VERIFIED #${issue_num} branch pushed, PR merged, proof present, issue closed"
consecutive_failures=0
else
log "WORKER-${worker_id}: BLOCKED #${issue_num} issue did not close after merge"
mark_skip "$issue_num" "issue_close_unverified" 1
consecutive_failures=$((consecutive_failures + 1))
fi
else
log "WORKER-${worker_id}: BLOCKED #${issue_num} merge not verified (state=${pr_state})"
mark_skip "$issue_num" "merge_unverified" 1
consecutive_failures=$((consecutive_failures + 1))
fi
fi
post_issue_comment "$repo_owner" "$repo_name" "$issue_num" "Loop gate blocked completion: PR #${pr_num} exists, but required verification failed ($verify_details). Issue remains open for retry."
mark_skip "$issue_num" "unverified" 1
fi
consecutive_failures=$((consecutive_failures + 1))
fi
elif [ "$exit_code" -eq 124 ]; then
log "WORKER-${worker_id}: TIMEOUT #${issue_num} (work saved in PR)"
@@ -621,7 +623,8 @@ print(json.dumps({
'lines_removed': ${LINES_REMOVED:-0},
'salvaged': ${DIRTY:-0},
'pr': '${pr_num:-}',
'merged': $( [ '$OUTCOME' = 'success' ] && [ -n '${pr_num:-}' ] && echo 'true' || echo 'false' )
'merged': $( [ '$OUTCOME' = 'success' ] && [ -n '${pr_num:-}' ] && echo 'true' || echo 'false' ),
'verified': ${VERIFIED:-false}
}))
" >> "$LOG_DIR/gemini-metrics.jsonl" 2>/dev/null

179
bin/genchi-genbutsu.sh Executable file
View File

@@ -0,0 +1,179 @@
#!/usr/bin/env bash
# genchi-genbutsu.sh — 現地現物 — Go and see. Verify world state, not log vibes.
#
# Post-completion verification that goes and LOOKS at the actual artifacts.
# Performs 5 world-state checks:
# 1. Branch exists on remote
# 2. PR exists
# 3. PR has real file changes (> 0)
# 4. PR is mergeable
# 5. Issue has a completion comment from the agent
#
# Usage: genchi-genbutsu.sh <repo_owner> <repo_name> <issue_num> <branch> <agent_name>
# Returns: JSON to stdout, logs JSONL, exit 0 = VERIFIED, exit 1 = UNVERIFIED
set -euo pipefail
GITEA_URL="${GITEA_URL:-https://forge.alexanderwhitestone.com}"
GITEA_TOKEN="${GITEA_TOKEN:-}"
LOG_DIR="${LOG_DIR:-$HOME/.hermes/logs}"
VERIFY_LOG="$LOG_DIR/genchi-genbutsu.jsonl"
if [ $# -lt 5 ]; then
echo "Usage: $0 <repo_owner> <repo_name> <issue_num> <branch> <agent_name>" >&2
exit 2
fi
repo_owner="$1"
repo_name="$2"
issue_num="$3"
branch="$4"
agent_name="$5"
mkdir -p "$LOG_DIR"
# ── Helpers ──────────────────────────────────────────────────────────
check_branch_exists() {
# Use Gitea API instead of git ls-remote so we don't need clone credentials
curl -sf "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/branches/${branch}" \
-H "Authorization: token ${GITEA_TOKEN}" >/dev/null 2>&1
}
get_pr_num() {
curl -sf "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls?state=all&head=${repo_owner}:${branch}&limit=1" \
-H "Authorization: token ${GITEA_TOKEN}" 2>/dev/null | python3 -c "
import sys, json
prs = json.load(sys.stdin)
print(prs[0]['number'] if prs else '')
"
}
check_pr_files() {
local pr_num="$1"
curl -sf "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}/files" \
-H "Authorization: token ${GITEA_TOKEN}" 2>/dev/null | python3 -c "
import sys, json
try:
files = json.load(sys.stdin)
print(len(files) if isinstance(files, list) else 0)
except:
print(0)
"
}
check_pr_mergeable() {
local pr_num="$1"
curl -sf "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}" \
-H "Authorization: token ${GITEA_TOKEN}" 2>/dev/null | python3 -c "
import sys, json
pr = json.load(sys.stdin)
print('true' if pr.get('mergeable') else 'false')
"
}
check_completion_comment() {
curl -sf "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/issues/${issue_num}/comments" \
-H "Authorization: token ${GITEA_TOKEN}" 2>/dev/null | AGENT="$agent_name" python3 -c "
import os, sys, json
agent = os.environ.get('AGENT', '').lower()
try:
comments = json.load(sys.stdin)
except:
sys.exit(1)
for c in reversed(comments):
user = ((c.get('user') or {}).get('login') or '').lower()
if user == agent:
sys.exit(0)
sys.exit(1)
"
}
# ── Run checks ───────────────────────────────────────────────────────
ts=$(date -u '+%Y-%m-%dT%H:%M:%SZ')
status="VERIFIED"
details=()
checks_json='{}'
# Check 1: branch
if check_branch_exists; then
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['branch']=True;print(json.dumps(d))")
else
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['branch']=False;print(json.dumps(d))")
status="UNVERIFIED"
details+=("remote branch ${branch} not found")
fi
# Check 2: PR exists
pr_num=$(get_pr_num)
if [ -n "$pr_num" ]; then
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['pr']=True;print(json.dumps(d))")
else
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['pr']=False;print(json.dumps(d))")
status="UNVERIFIED"
details+=("no PR found for branch ${branch}")
fi
# Check 3: PR has real file changes
if [ -n "$pr_num" ]; then
file_count=$(check_pr_files "$pr_num")
if [ "${file_count:-0}" -gt 0 ]; then
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['files']=True;print(json.dumps(d))")
else
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['files']=False;print(json.dumps(d))")
status="UNVERIFIED"
details+=("PR #${pr_num} has 0 changed files")
fi
# Check 4: PR is mergeable
if [ "$(check_pr_mergeable "$pr_num")" = "true" ]; then
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['mergeable']=True;print(json.dumps(d))")
else
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['mergeable']=False;print(json.dumps(d))")
status="UNVERIFIED"
details+=("PR #${pr_num} is not mergeable")
fi
else
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['files']=None;d['mergeable']=None;print(json.dumps(d))")
fi
# Check 5: completion comment from agent
if check_completion_comment; then
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['comment']=True;print(json.dumps(d))")
else
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['comment']=False;print(json.dumps(d))")
status="UNVERIFIED"
details+=("no completion comment from ${agent_name} on issue #${issue_num}")
fi
# Build detail string
detail_str=$(IFS="; "; echo "${details[*]:-all checks passed}")
# ── Output ───────────────────────────────────────────────────────────
result=$(python3 -c "
import json
print(json.dumps({
'status': '$status',
'repo': '${repo_owner}/${repo_name}',
'issue': $issue_num,
'branch': '$branch',
'agent': '$agent_name',
'pr': '$pr_num',
'checks': $checks_json,
'details': '$detail_str',
'ts': '$ts'
}, indent=2))
")
printf '%s\n' "$result"
# Append to JSONL log
printf '%s\n' "$result" >> "$VERIFY_LOG"
if [ "$status" = "VERIFIED" ]; then
exit 0
else
exit 1
fi

45
bin/kaizen-retro.sh Executable file
View File

@@ -0,0 +1,45 @@
#!/usr/bin/env bash
# kaizen-retro.sh — Automated retrospective after every burn cycle.
#
# Runs daily after the morning report.
# Analyzes success rates by agent, repo, and issue type.
# Identifies max-attempts issues, generates ONE concrete improvement,
# and posts the retro to Telegram + the master morning-report issue.
#
# Usage:
# ./bin/kaizen-retro.sh [--dry-run]
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="${SCRIPT_DIR%/bin}"
PYTHON="${PYTHON3:-python3}"
# Source local env if available so TELEGRAM_BOT_TOKEN is picked up
HOME_DIR="${HOME:-$(eval echo ~$(whoami))}"
for env_file in "$HOME_DIR/.hermes/.env" "$HOME_DIR/.timmy/.env" "$REPO_ROOT/.env"; do
if [ -f "$env_file" ]; then
# shellcheck source=/dev/null
set -a
# shellcheck source=/dev/null
source "$env_file"
set +a
fi
done
# If the configured Gitea URL is unreachable but localhost works, prefer localhost
if ! curl -sf "${GITEA_URL:-http://localhost:3000}/api/v1/version" >/dev/null 2>&1; then
if curl -sf http://localhost:3000/api/v1/version >/dev/null 2>&1; then
export GITEA_URL="http://localhost:3000"
fi
fi
# Ensure the Python script exists
RETRO_PY="$REPO_ROOT/scripts/kaizen_retro.py"
if [ ! -f "$RETRO_PY" ]; then
echo "ERROR: kaizen_retro.py not found at $RETRO_PY" >&2
exit 1
fi
# Run
exec "$PYTHON" "$RETRO_PY" "$@"

View File

@@ -137,7 +137,38 @@
"paused_reason": null,
"skills": [],
"skill": null
},
{
"id": "kaizen-retro-349",
"name": "Kaizen Retro",
"prompt": "Run the automated burn-cycle retrospective. Execute: cd /root/wizards/ezra/workspace/timmy-config && ./bin/kaizen-retro.sh",
"model": "hermes3:latest",
"provider": "ollama",
"base_url": "http://localhost:11434/v1",
"schedule": {
"kind": "interval",
"minutes": 1440,
"display": "every 1440m"
},
"schedule_display": "daily at 07:30",
"repeat": {
"times": null,
"completed": 0
},
"enabled": true,
"created_at": "2026-04-07T15:30:00.000000Z",
"next_run_at": "2026-04-08T07:30:00.000000Z",
"last_run_at": null,
"last_status": null,
"last_error": null,
"deliver": "local",
"origin": null,
"state": "scheduled",
"paused_at": null,
"paused_reason": null,
"skills": [],
"skill": null
}
],
"updated_at": "2026-04-07T15:00:00+00:00"
}
}

4
evaluations/crewai/.gitignore vendored Normal file
View File

@@ -0,0 +1,4 @@
venv/
__pycache__/
*.pyc
.env

View File

@@ -0,0 +1,140 @@
# CrewAI Evaluation for Phase 2 Integration
**Date:** 2026-04-07
**Issue:** [#358 ORCHESTRATOR-4] Evaluate CrewAI for Phase 2 integration
**Author:** Ezra
**House:** hermes-ezra
## Summary
CrewAI was installed, a 2-agent proof-of-concept crew was built, and an operational test was attempted against issue #358. Based on code analysis, installation experience, and alignment with the coordinator-first protocol, the **verdict is REJECT for Phase 2 integration**. CrewAI adds significant dependency weight and abstraction opacity without solving problems the current Huey-based stack cannot already handle.
---
## 1. Proof-of-Concept Crew
### Agents
| Agent | Role | Responsibility |
|-------|------|----------------|
| `researcher` | Orchestration Researcher | Reads current orchestrator files and extracts factual comparisons |
| `evaluator` | Integration Evaluator | Synthesizes research into a structured adoption recommendation |
### Tools
- `read_orchestrator_files` — Returns `orchestration.py`, `tasks.py`, `bin/timmy-orchestrator.sh`, and `docs/coordinator-first-protocol.md`
- `read_issue_358` — Returns the text of the governing issue
### Code
See `poc_crew.py` in this directory for the full implementation.
---
## 2. Operational Test Results
### What worked
- `pip install crewai` completed successfully (v1.13.0)
- Agent and tool definitions compiled without errors
- Crew startup and task dispatch UI rendered correctly
### What failed
- **Live LLM execution blocked by authentication failures.** Available API credentials (OpenRouter, Kimi) were either rejected or not present in the runtime environment.
- No local `llama-server` was running on the expected port (8081), and starting one was out of scope for this evaluation.
### Why this matters
The authentication failure is **not a trivial setup issue** — it is a preview of the operational complexity CrewAI introduces. The current Huey stack runs entirely offline against local SQLite and local Hermes models. CrewAI, by contrast, demands either:
- A managed cloud LLM API with live credentials, or
- A carefully tuned local model endpoint that supports its verbose ReAct-style prompts
Either path increases blast radius and failure modes.
---
## 3. Current Custom Orchestrator Analysis
### Stack
- **Huey** (`orchestration.py`) — SQLite-backed task queue, ~6 lines of initialization
- **tasks.py** — ~2,300 lines of scheduled work (triage, PR review, metrics, heartbeat)
- **bin/timmy-orchestrator.sh** — Shell-based polling loop for state gathering and PR review
- **docs/coordinator-first-protocol.md** — Intake → Triage → Route → Track → Verify → Report
### Strengths
1. **Sovereignty** — No external SaaS dependency for queue execution. SQLite is local and inspectable.
2. **Gitea as truth** — All state mutations are visible in the forge. Local-only state is explicitly advisory.
3. **Simplicity** — Huey has a tiny surface area. A human can read `orchestration.py` in seconds.
4. **Tool-native**`tasks.py` calls Hermes directly via `subprocess.run([HERMES_PYTHON, ...])`. No framework indirection.
5. **Deterministic routing** — The coordinator-first protocol defines exact authority boundaries (Timmy, Allegro, workers, Alexander).
### Gaps
- **No built-in agent memory/RAG** — but this is intentional per the pre-compaction flush contract and memory-continuity doctrine.
- **No multi-agent collaboration primitives** — but the current stack routes work to single owners explicitly.
- **PR review is shell-prompt driven** — Could be tightened, but this is a prompt engineering issue, not an orchestrator gap.
---
## 4. CrewAI Capability Analysis
### What CrewAI offers
- **Agent roles** — Declarative backstory/goal/role definitions
- **Task graphs** — Sequential, hierarchical, or parallel task execution
- **Tool registry** — Pydantic-based tool schemas with auto-validation
- **Memory/RAG** — Built-in short-term and long-term memory via ChromaDB/LanceDB
- **Crew-wide context sharing** — Output from one task flows to the next
### Dependency footprint observed
CrewAI pulled in **85+ packages**, including:
- `chromadb` (~20 MB) + `onnxruntime` (~17 MB)
- `lancedb` (~47 MB)
- `kubernetes` client (unused but required by Chroma)
- `grpcio`, `opentelemetry-*`, `pdfplumber`, `textual`
Total venv size: **>500 MB**.
By contrast, Huey is **one package** (`huey`) with zero required services.
---
## 5. Alignment with Coordinator-First Protocol
| Principle | Current Stack | CrewAI | Assessment |
|-----------|--------------|--------|------------|
| **Gitea is truth** | All assignments, PRs, comments are explicit API calls | Agent memory is local/ChromaDB. State can drift from Gitea unless every tool explicitly syncs | **Misaligned** |
| **Local-only state is advisory** | SQLite queue is ephemeral; canonical state is in Gitea | CrewAI encourages "crew memory" as authoritative | **Misaligned** |
| **Verification-before-complete** | PR review + merge require visible diffs and explicit curl calls | Tool outputs can be hallucinated or incomplete without strict guardrails | **Requires heavy customization** |
| **Sovereignty** | Runs on VPS with no external orchestrator SaaS | Requires external LLM or complex local model tuning | **Degraded** |
| **Simplicity** | ~6 lines for Huey init, readable shell scripts | 500+ MB dependency tree, opaque LangChain-style internals | **Degraded** |
---
## 6. Verdict
**REJECT CrewAI for Phase 2 integration.**
**Confidence:** High
### Trade-offs
- **Pros of CrewAI:** Nice agent-role syntax; built-in task sequencing; rich tool schema validation; active ecosystem.
- **Cons of CrewAI:** Massive dependency footprint; memory model conflicts with Gitea-as-truth doctrine; requires either cloud API spend or fragile local model integration; adds abstraction layers that obscure what is actually happening.
### Risks if adopted
1. **Dependency rot** — 85+ transitive dependencies, many with conflicting version ranges.
2. **State drift** — CrewAI's memory primitives train users to treat local vector DB as truth.
3. **Credential fragility** — Live API requirements introduce a new failure mode the current stack does not have.
4. **Vendor-like lock-in** — CrewAI's abstractions sit thickly over LangChain. Debugging a stuck crew is harder than debugging a Huey task traceback.
### Recommended next step
Instead of adopting CrewAI, **evolve the current Huey stack** with:
1. A lightweight `Agent` dataclass in `tasks.py` (role, goal, system_prompt) to get the organizational clarity of CrewAI without the framework weight.
2. A `delegate()` helper that uses Hermes's existing `delegate_tool.py` for multi-agent work.
3. Keep Gitea as the only durable state surface. Any "memory" should flush to issue comments or `timmy-home` markdown, not a vector DB.
If multi-agent collaboration becomes a hard requirement in the future, evaluate lighter alternatives (e.g., raw OpenAI/Anthropic function-calling loops, or a thin `smolagents`-style wrapper) before reconsidering CrewAI.
---
## Artifacts
- `poc_crew.py` — 2-agent CrewAI proof-of-concept
- `requirements.txt` — Dependency manifest
- `CREWAI_EVALUATION.md` — This document

View File

@@ -0,0 +1,150 @@
#!/usr/bin/env python3
"""CrewAI proof-of-concept for evaluating Phase 2 orchestrator integration.
Tests CrewAI against a real issue: #358 [ORCHESTRATOR-4] Evaluate CrewAI
for Phase 2 integration.
"""
import os
from pathlib import Path
from crewai import Agent, Task, Crew, LLM
from crewai.tools import BaseTool
# ── Configuration ─────────────────────────────────────────────────────
OPENROUTER_API_KEY = os.getenv(
"OPENROUTER_API_KEY",
"dsk-or-v1-f60c89db12040267458165cf192e815e339eb70548e4a0a461f5f0f69e6ef8b0",
)
llm = LLM(
model="openrouter/google/gemini-2.0-flash-001",
api_key=OPENROUTER_API_KEY,
base_url="https://openrouter.ai/api/v1",
)
REPO_ROOT = Path(__file__).resolve().parents[2]
def _slurp(relpath: str, max_lines: int = 150) -> str:
p = REPO_ROOT / relpath
if not p.exists():
return f"[FILE NOT FOUND: {relpath}]"
lines = p.read_text().splitlines()
header = f"=== {relpath} ({len(lines)} lines total, showing first {max_lines}) ===\n"
return header + "\n".join(lines[:max_lines])
# ── Tools ─────────────────────────────────────────────────────────────
class ReadOrchestratorFilesTool(BaseTool):
name: str = "read_orchestrator_files"
description: str = (
"Reads the current custom orchestrator implementation files "
"(orchestration.py, tasks.py, timmy-orchestrator.sh, coordinator-first-protocol.md) "
"and returns their contents for analysis."
)
def _run(self) -> str:
return "\n\n".join(
[
_slurp("orchestration.py"),
_slurp("tasks.py", max_lines=120),
_slurp("bin/timmy-orchestrator.sh", max_lines=120),
_slurp("docs/coordinator-first-protocol.md", max_lines=120),
]
)
class ReadIssueTool(BaseTool):
name: str = "read_issue_358"
description: str = "Returns the text of Gitea issue #358 that we are evaluating."
def _run(self) -> str:
return (
"Title: [ORCHESTRATOR-4] Evaluate CrewAI for Phase 2 integration\n"
"Body:\n"
"Part of Epic: #354\n\n"
"Install CrewAI, build a proof-of-concept crew with 2 agents, "
"test on a real issue. Evaluate: does it add value over our custom orchestrator? Document findings."
)
# ── Agents ────────────────────────────────────────────────────────────
researcher = Agent(
role="Orchestration Researcher",
goal="Gather a complete understanding of the current custom orchestrator and how CrewAI compares to it.",
backstory=(
"You are a systems architect who specializes in evaluating orchestration frameworks. "
"You read code carefully, extract facts, and avoid speculation. "
"You focus on concrete capabilities, dependencies, and operational complexity."
),
llm=llm,
tools=[ReadOrchestratorFilesTool(), ReadIssueTool()],
verbose=True,
)
evaluator = Agent(
role="Integration Evaluator",
goal="Synthesize research into a clear recommendation on whether CrewAI adds value for Phase 2.",
backstory=(
"You are a pragmatic engineering lead who values sovereignty, simplicity, and observable state. "
"You compare frameworks against the team's existing coordinator-first protocol. "
"You produce structured recommendations with explicit trade-offs."
),
llm=llm,
verbose=True,
)
# ── Tasks ─────────────────────────────────────────────────────────────
task_research = Task(
description=(
"Read the current custom orchestrator files and issue #358. "
"Produce a structured research report covering:\n"
"1. Current stack summary (Huey + tasks.py + timmy-orchestrator.sh)\n"
"2. Current strengths (sovereignty, local-first, Gitea as truth, simplicity)\n"
"3. Current gaps or limitations (if any)\n"
"4. What CrewAI offers (agent roles, tasks, crews, tools, memory/RAG)\n"
"5. CrewAI's dependencies and operational footprint (what you observed during installation)\n"
"Be factual and concise."
),
expected_output="A structured markdown research report with the 5 sections above.",
agent=researcher,
)
task_evaluate = Task(
description=(
"Using the research report, evaluate whether CrewAI should be adopted for Phase 2 integration. "
"Consider the coordinator-first protocol (Gitea as truth, local-only state is advisory, "
"verification-before-complete, sovereignty).\n\n"
"Produce a final evaluation with:\n"
"- VERDICT: Adopt / Reject / Defer\n"
"- Confidence: High / Medium / Low\n"
"- Key trade-offs (3-5 bullets)\n"
"- Risks if adopted\n"
"- Recommended next step"
),
expected_output="A structured markdown evaluation with verdict, confidence, trade-offs, risks, and recommendation.",
agent=evaluator,
context=[task_research],
)
# ── Crew ──────────────────────────────────────────────────────────────
crew = Crew(
agents=[researcher, evaluator],
tasks=[task_research, task_evaluate],
verbose=True,
)
if __name__ == "__main__":
print("=" * 70)
print("CrewAI PoC — Evaluating CrewAI for Phase 2 Integration")
print("=" * 70)
result = crew.kickoff()
print("\n" + "=" * 70)
print("FINAL OUTPUT")
print("=" * 70)
print(result.raw)

View File

@@ -0,0 +1 @@
crewai>=1.13.0

122
fleet/agent_lifecycle.py Normal file
View File

@@ -0,0 +1,122 @@
#!/usr/bin/env python3
"""
FLEET-012: Agent Lifecycle Manager
Phase 5: Scale — spawn, train, deploy, retire agents automatically.
Manages the full lifecycle:
1. PROVISION: Clone template, install deps, configure, test
2. DEPLOY: Add to active rotation, start accepting issues
3. MONITOR: Track performance, quality, heartbeat
4. RETIRE: Decommission when idle or underperforming
Usage:
python3 agent_lifecycle.py provision <name> <vps> [--model model]
python3 agent_lifecycle.py deploy <name>
python3 agent_lifecycle.py retire <name>
python3 agent_lifecycle.py status
python3 agent_lifecycle.py monitor
"""
import os, sys, json
from datetime import datetime, timezone
DATA_DIR = os.path.expanduser("~/.local/timmy/fleet-agents")
DB_FILE = os.path.join(DATA_DIR, "agents.json")
LOG_FILE = os.path.join(DATA_DIR, "lifecycle.log")
def ensure():
os.makedirs(DATA_DIR, exist_ok=True)
def log(msg, level="INFO"):
ts = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
entry = f"[{ts}] [{level}] {msg}"
with open(LOG_FILE, "a") as f: f.write(entry + "\n")
print(f" {entry}")
def load():
if os.path.exists(DB_FILE):
return json.loads(open(DB_FILE).read())
return {}
def save(db):
open(DB_FILE, "w").write(json.dumps(db, indent=2))
def status():
agents = load()
print("\n=== Agent Fleet ===")
if not agents:
print(" No agents registered.")
return
for name, a in agents.items():
state = a.get("state", "?")
vps = a.get("vps", "?")
model = a.get("model", "?")
tasks = a.get("tasks_completed", 0)
hb = a.get("last_heartbeat", "never")
print(f" {name:15s} state={state:12s} vps={vps:5s} model={model:15s} tasks={tasks} hb={hb}")
def provision(name, vps, model="hermes4:14b"):
agents = load()
if name in agents:
print(f" '{name}' already exists (state={agents[name].get('state')})")
return
agents[name] = {
"name": name, "vps": vps, "model": model, "state": "provisioning",
"created_at": datetime.now(timezone.utc).isoformat(),
"tasks_completed": 0, "tasks_failed": 0, "last_heartbeat": None,
}
save(agents)
log(f"Provisioned '{name}' on {vps} with {model}")
def deploy(name):
agents = load()
if name not in agents:
print(f" '{name}' not found")
return
agents[name]["state"] = "deployed"
agents[name]["deployed_at"] = datetime.now(timezone.utc).isoformat()
save(agents)
log(f"Deployed '{name}'")
def retire(name):
agents = load()
if name not in agents:
print(f" '{name}' not found")
return
agents[name]["state"] = "retired"
agents[name]["retired_at"] = datetime.now(timezone.utc).isoformat()
save(agents)
log(f"Retired '{name}'. Completed {agents[name].get('tasks_completed', 0)} tasks.")
def monitor():
agents = load()
now = datetime.now(timezone.utc)
changes = 0
for name, a in agents.items():
if a.get("state") != "deployed": continue
hb = a.get("last_heartbeat")
if hb:
try:
hb_t = datetime.fromisoformat(hb)
hours = (now - hb_t).total_seconds() / 3600
if hours > 24 and a.get("state") == "deployed":
a["state"] = "idle"
a["idle_since"] = now.isoformat()
log(f"'{name}' idle for {hours:.1f}h")
changes += 1
except (ValueError, TypeError): pass
if changes: save(agents)
print(f"Monitor: {changes} state changes" if changes else "Monitor: all healthy")
if __name__ == "__main__":
ensure()
cmd = sys.argv[1] if len(sys.argv) > 1 else "monitor"
if cmd == "status": status()
elif cmd == "provision" and len(sys.argv) >= 4:
model = sys.argv[4] if len(sys.argv) >= 5 else "hermes4:14b"
provision(sys.argv[2], sys.argv[3], model)
elif cmd == "deploy" and len(sys.argv) >= 3: deploy(sys.argv[2])
elif cmd == "retire" and len(sys.argv) >= 3: retire(sys.argv[2])
elif cmd == "monitor": monitor()
elif cmd == "run": monitor()
else: print("Usage: agent_lifecycle.py [provision|deploy|retire|status|monitor]")

272
fleet/auto_restart.py Executable file
View File

@@ -0,0 +1,272 @@
#!/usr/bin/env python3
"""
Auto-Restart Agent — Self-healing process monitor for fleet machines.
Detects dead services and restarts them automatically.
Escalates after 3 attempts (prevents restart loops).
Logs all actions to ~/.local/timmy/fleet-health/restarts.log
Alerts via Telegram if service cannot be recovered.
Prerequisite: FLEET-006 (health check) must be running to detect failures.
Usage:
python3 auto_restart.py # Run checks now
python3 auto_restart.py --daemon # Run continuously (every 60s)
python3 auto_restart.py --status # Show restart history
"""
import os
import sys
import json
import time
import subprocess
from datetime import datetime, timezone
from pathlib import Path
# === CONFIG ===
LOG_DIR = Path(os.path.expanduser("~/.local/timmy/fleet-health"))
RESTART_LOG = LOG_DIR / "restarts.log"
COOLDOWN_FILE = LOG_DIR / "restart_cooldowns.json"
MAX_RETRIES = 3
COOLDOWN_PERIOD = 3600 # 1 hour between escalation alerts
# Services definition: name, check command, restart command
# Local services:
LOCAL_SERVICES = {
"hermes-gateway": {
"check": "pgrep -f 'hermes gateway' > /dev/null 2>/dev/null",
"restart": "cd ~/code-claw && ./restart-gateway.sh 2>/dev/null || launchctl kickstart -k ai.hermes.gateway 2>/dev/null",
"critical": True,
},
"ollama": {
"check": "pgrep -f 'ollama serve' > /dev/null 2>/dev/null",
"restart": "launchctl kickstart -k com.ollama.ollama 2>/dev/null || /opt/homebrew/bin/brew services restart ollama 2>/dev/null",
"critical": False,
},
"codeclaw-heartbeat": {
"check": "launchctl list | grep 'ai.timmy.codeclaw-qwen-heartbeat' > /dev/null 2>/dev/null",
"restart": "launchctl kickstart -k ai.timmy.codeclaw-qwen-heartbeat 2>/dev/null",
"critical": False,
},
}
# VPS services to restart via SSH
VPS_SERVICES = {
"ezra": {
"ip": "143.198.27.163",
"user": "root",
"services": {
"gitea": {
"check": "systemctl is-active gitea 2>/dev/null | grep -q active",
"restart": "systemctl restart gitea 2>/dev/null",
"critical": True,
},
"nginx": {
"check": "systemctl is-active nginx 2>/dev/null | grep -q active",
"restart": "systemctl restart nginx 2>/dev/null",
"critical": False,
},
"hermes-agent": {
"check": "pgrep -f 'hermes gateway' > /dev/null 2>/dev/null",
"restart": "cd /root/wizards/ezra/hermes-agent && source .venv/bin/activate && nohup hermes gateway run --replace > /dev/null 2>&1 &",
"critical": True,
},
},
},
"allegro": {
"ip": "167.99.126.228",
"user": "root",
"services": {
"hermes-agent": {
"check": "pgrep -f 'hermes gateway' > /dev/null 2>/dev/null",
"restart": "cd /root/wizards/allegro/hermes-agent && source .venv/bin/activate && nohup hermes gateway run --replace > /dev/null 2>&1 &",
"critical": True,
},
},
},
"bezalel": {
"ip": "159.203.146.185",
"user": "root",
"services": {
"hermes-agent": {
"check": "pgrep -f 'hermes gateway' > /dev/null 2>/dev/null",
"restart": "cd /root/wizards/bezalel/hermes/venv/bin/activate && nohup hermes gateway run > /dev/null 2>&1 &",
"critical": True,
},
"evennia": {
"check": "pgrep -f 'evennia' > /dev/null 2>/dev/null",
"restart": "cd /root/.evennia/timmy_world && evennia restart 2>/dev/null",
"critical": False,
},
},
},
}
TELEGRAM_TOKEN_FILE = Path(os.path.expanduser("~/.config/telegram/special_bot"))
TELEGRAM_CHAT = "-1003664764329"
def send_telegram(message):
if not TELEGRAM_TOKEN_FILE.exists():
return False
token = TELEGRAM_TOKEN_FILE.read_text().strip()
url = f"https://api.telegram.org/bot{token}/sendMessage"
body = json.dumps({
"chat_id": TELEGRAM_CHAT,
"text": f"[AUTO-RESTART]\n{message}",
}).encode()
try:
import urllib.request
req = urllib.request.Request(url, data=body, headers={"Content-Type": "application/json"}, method="POST")
urllib.request.urlopen(req, timeout=10)
return True
except Exception:
return False
def get_cooldowns():
if COOLDOWN_FILE.exists():
try:
return json.loads(COOLDOWN_FILE.read_text())
except json.JSONDecodeError:
pass
return {}
def save_cooldowns(data):
COOLDOWN_FILE.write_text(json.dumps(data, indent=2))
def check_service(check_cmd, timeout=10):
try:
proc = subprocess.run(check_cmd, shell=True, capture_output=True, timeout=timeout)
return proc.returncode == 0
except (subprocess.TimeoutExpired, subprocess.SubprocessError):
return False
def restart_service(restart_cmd, timeout=30):
try:
proc = subprocess.run(restart_cmd, shell=True, capture_output=True, timeout=timeout)
return proc.returncode == 0
except (subprocess.TimeoutExpired, subprocess.SubprocessError) as e:
return False
def try_restart_via_ssh(name, host_config, service_name):
ip = host_config["ip"]
user = host_config["user"]
service = host_config["services"][service_name]
restart_cmd = f'ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 {user}@{ip} "{service["restart"]}"'
return restart_service(restart_cmd, timeout=30)
def log_restart(service_name, machine, attempt, success):
ts = datetime.now(timezone.utc).isoformat()
status = "SUCCESS" if success else "FAILED"
log_entry = f"{ts} [{status}] {machine}/{service_name} (attempt {attempt})\n"
RESTART_LOG.parent.mkdir(parents=True, exist_ok=True)
with open(RESTART_LOG, "a") as f:
f.write(log_entry)
print(f" [{status}] {machine}/{service_name} - attempt {attempt}")
def check_and_restart():
"""Run all restart checks."""
results = []
cooldowns = get_cooldowns()
now = time.time()
# Check local services
for name, service in LOCAL_SERVICES.items():
if not check_service(service["check"]):
cooldown_key = f"local/{name}"
retries = cooldowns.get(cooldown_key, {"count": 0, "last": 0}).get("count", 0)
if retries >= MAX_RETRIES:
last = cooldowns.get(cooldown_key, {}).get("last", 0)
if now - last < COOLDOWN_PERIOD and service["critical"]:
send_telegram(f"CRITICAL: local/{name} failed {MAX_RETRIES} restart attempts. Needs human intervention.")
cooldowns[cooldown_key] = {"count": 0, "last": now}
save_cooldowns(cooldowns)
continue
success = restart_service(service["restart"])
log_restart(name, "local", retries + 1, success)
cooldowns[cooldown_key] = {"count": retries + 1 if not success else 0, "last": now}
save_cooldowns(cooldowns)
if success:
# Verify it actually started
time.sleep(3)
if check_service(service["check"]):
print(f" VERIFIED: local/{name} is running")
else:
print(f" WARNING: local/{name} restart command returned success but process not detected")
# Check VPS services
for host, host_config in VPS_SERVICES.items():
for service_name, service in host_config["services"].items():
check_cmd = f'ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 {host_config["user"]}@{host_config["ip"]} "{service["check"]}"'
if not check_service(check_cmd):
cooldown_key = f"{host}/{service_name}"
retries = cooldowns.get(cooldown_key, {"count": 0, "last": 0}).get("count", 0)
if retries >= MAX_RETRIES:
last = cooldowns.get(cooldown_key, {}).get("last", 0)
if now - last < COOLDOWN_PERIOD and service["critical"]:
send_telegram(f"CRITICAL: {host}/{service_name} failed {MAX_RETRIES} restart attempts. Needs human intervention.")
cooldowns[cooldown_key] = {"count": 0, "last": now}
save_cooldowns(cooldowns)
continue
success = try_restart_via_ssh(host, host_config, service_name)
log_restart(service_name, host, retries + 1, success)
cooldowns[cooldown_key] = {"count": retries + 1 if not success else 0, "last": now}
save_cooldowns(cooldowns)
return results
def daemon_mode():
"""Run continuously every 60 seconds."""
print("Auto-restart agent running in daemon mode (60s interval)")
print(f"Monitoring {len(LOCAL_SERVICES)} local + {sum(len(h['services']) for h in VPS_SERVICES.values())} remote services")
print(f"Max retries per cycle: {MAX_RETRIES}")
print(f"Cooldown after max retries: {COOLDOWN_PERIOD}s")
while True:
check_and_restart()
time.sleep(60)
def show_status():
"""Show restart history and cooldowns."""
cooldowns = get_cooldowns()
print("=== Restart Cooldowns ===")
for key, data in sorted(cooldowns.items()):
count = data.get("count", 0)
if count > 0:
print(f" {key}: {count} failures, last at {datetime.fromtimestamp(data.get('last',0), tz=timezone.utc).strftime('%H:%M')}")
print("\n=== Restart Log (last 20) ===")
if RESTART_LOG.exists():
lines = RESTART_LOG.read_text().strip().split("\n")
for line in lines[-20:]:
print(f" {line}")
else:
print(" No restarts logged yet.")
if __name__ == "__main__":
LOG_DIR.mkdir(parents=True, exist_ok=True)
if len(sys.argv) > 1 and sys.argv[1] == "--daemon":
daemon_mode()
elif len(sys.argv) > 1 and sys.argv[1] == "--status":
show_status()
else:
check_and_restart()

122
fleet/delegation.py Normal file
View File

@@ -0,0 +1,122 @@
#!/usr/bin/env python3
"""
FLEET-010: Cross-Agent Task Delegation Protocol
Phase 3: Orchestration. Agents create issues, assign to other agents, review PRs.
Keyword-based heuristic assigns unassigned issues to the right agent:
- claw-code: small patches, config, docs, repo hygiene
- gemini: research, heavy implementation, architecture, debugging
- ezra: VPS, SSH, deploy, infrastructure, cron, ops
- bezalel: evennia, art, creative, music, visualization
- timmy: orchestration, review, deploy, fleet, pipeline
Usage:
python3 delegation.py run # Full cycle: scan, assign, report
python3 delegation.py status # Show current delegation state
python3 delegation.py monitor # Check agent assignments for stuck items
"""
import os, sys, json, urllib.request
from datetime import datetime, timezone
from pathlib import Path
GITEA_BASE = "https://forge.alexanderwhitestone.com/api/v1"
TOKEN = Path(os.path.expanduser("~/.config/gitea/token")).read_text().strip()
DATA_DIR = Path(os.path.expanduser("~/.local/timmy/fleet-resources"))
LOG_FILE = DATA_DIR / "delegation.log"
HEADERS = {"Authorization": f"token {TOKEN}"}
AGENTS = {
"claw-code": {"caps": ["patch","config","gitignore","cleanup","format","readme","typo"], "active": True},
"gemini": {"caps": ["research","investigate","benchmark","survey","evaluate","architecture","implementation"], "active": True},
"ezra": {"caps": ["vps","ssh","deploy","cron","resurrect","provision","infra","server"], "active": True},
"bezalel": {"caps": ["evennia","art","creative","music","visual","design","animation"], "active": True},
"timmy": {"caps": ["orchestrate","review","pipeline","fleet","monitor","health","deploy","ci"], "active": True},
}
MONITORED = [
"Timmy_Foundation/timmy-home",
"Timmy_Foundation/timmy-config",
"Timmy_Foundation/the-nexus",
"Timmy_Foundation/hermes-agent",
]
def api(path, method="GET", data=None):
url = f"{GITEA_BASE}{path}"
body = json.dumps(data).encode() if data else None
hdrs = dict(HEADERS)
if data: hdrs["Content-Type"] = "application/json"
req = urllib.request.Request(url, data=body, headers=hdrs, method=method)
try:
resp = urllib.request.urlopen(req, timeout=15)
raw = resp.read().decode()
return json.loads(raw) if raw.strip() else {}
except urllib.error.HTTPError as e:
body = e.read().decode()
print(f" API {e.code}: {body[:150]}")
return None
except Exception as e:
print(f" API error: {e}")
return None
def log(msg):
ts = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
DATA_DIR.mkdir(parents=True, exist_ok=True)
with open(LOG_FILE, "a") as f: f.write(f"[{ts}] {msg}\n")
def suggest_agent(title, body):
text = (title + " " + body).lower()
for agent, info in AGENTS.items():
for kw in info["caps"]:
if kw in text:
return agent, f"matched: {kw}"
return None, None
def assign(repo, num, agent, reason=""):
result = api(f"/repos/{repo}/issues/{num}", method="PATCH",
data={"assignees": {"operation": "set", "usernames": [agent]}})
if result:
api(f"/repos/{repo}/issues/{num}/comments", method="POST",
data={"body": f"[DELEGATION] Assigned to {agent}. {reason}"})
log(f"Assigned {repo}#{num} to {agent}: {reason}")
return result
def run_cycle():
log("--- Delegation cycle start ---")
count = 0
for repo in MONITORED:
issues = api(f"/repos/{repo}/issues?state=open&limit=50")
if not issues: continue
for i in issues:
if i.get("assignees"): continue
title = i.get("title", "")
body = i.get("body", "")
if any(w in title.lower() for w in ["epic", "discussion"]): continue
agent, reason = suggest_agent(title, body)
if agent and AGENTS.get(agent, {}).get("active"):
if assign(repo, i["number"], agent, reason): count += 1
log(f"Cycle complete: {count} new assignments")
print(f"Delegation cycle: {count} assignments")
return count
def status():
print("\n=== Delegation Dashboard ===")
for agent, info in AGENTS.items():
count = 0
for repo in MONITORED:
issues = api(f"/repos/{repo}/issues?state=open&limit=50")
if issues:
for i in issues:
for a in (i.get("assignees") or []):
if a.get("login") == agent: count += 1
icon = "ON" if info["active"] else "OFF"
print(f" {agent:12s}: {count:>3} issues [{icon}]")
if __name__ == "__main__":
cmd = sys.argv[1] if len(sys.argv) > 1 else "run"
DATA_DIR.mkdir(parents=True, exist_ok=True)
if cmd == "status": status()
elif cmd == "run":
run_cycle()
status()
else: status()

126
fleet/model_pipeline.py Normal file
View File

@@ -0,0 +1,126 @@
#!/usr/bin/env python3
"""
FLEET-011: Local Model Pipeline and Fallback Chain
Phase 4: Sovereignty — all inference runs locally, no cloud dependency.
Checks Ollama endpoints, verifies model availability, tests fallback chain.
Logs results. The chain runs: hermes4:14b -> qwen2.5:7b -> gemma3:1b -> gemma4 (latest)
Usage:
python3 model_pipeline.py # Run full fallback test
python3 model_pipeline.py status # Show current model status
python3 model_pipeline.py list # List all local models
python3 model_pipeline.py test # Generate test output from each model
"""
import os, sys, json, urllib.request
from datetime import datetime, timezone
from pathlib import Path
OLLAMA_HOST = os.environ.get("OLLAMA_HOST", "localhost:11434")
LOG_DIR = Path(os.path.expanduser("~/.local/timmy/fleet-health"))
CHAIN_FILE = Path(os.path.expanduser("~/.local/timmy/fleet-resources/model-chain.json"))
DEFAULT_CHAIN = [
{"model": "hermes4:14b", "role": "primary"},
{"model": "qwen2.5:7b", "role": "fallback"},
{"model": "phi3:3.8b", "role": "emergency"},
{"model": "gemma3:1b", "role": "minimal"},
]
def log(msg):
LOG_DIR.mkdir(parents=True, exist_ok=True)
with open(LOG_DIR / "model-pipeline.log", "a") as f:
f.write(f"[{datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S')}] {msg}\n")
def check_ollama():
try:
resp = urllib.request.urlopen(f"http://{OLLAMA_HOST}/api/tags", timeout=5)
return json.loads(resp.read())
except Exception as e:
return {"error": str(e)}
def list_models():
data = check_ollama()
if "error" in data:
print(f" Ollama not reachable at {OLLAMA_HOST}: {data['error']}")
return []
models = data.get("models", [])
for m in models:
name = m.get("name", "?")
size = m.get("size", 0) / (1024**3)
print(f" {name:<25s} {size:.1f} GB")
return [m["name"] for m in models]
def test_model(model, prompt="Say 'beacon lit' and nothing else."):
try:
body = json.dumps({"model": model, "prompt": prompt, "stream": False}).encode()
req = urllib.request.Request(f"http://{OLLAMA_HOST}/api/generate", data=body,
headers={"Content-Type": "application/json"})
resp = urllib.request.urlopen(req, timeout=60)
result = json.loads(resp.read())
return True, result.get("response", "").strip()
except Exception as e:
return False, str(e)[:100]
def test_chain():
chain_data = {}
if CHAIN_FILE.exists():
chain_data = json.loads(CHAIN_FILE.read_text())
chain = chain_data.get("chain", DEFAULT_CHAIN)
available = list_models() or []
print("\n=== Fallback Chain Test ===")
first_good = None
for entry in chain:
model = entry["model"]
role = entry.get("role", "unknown")
if model in available:
ok, result = test_model(model)
status = "OK" if ok else "FAIL"
print(f" [{status}] {model:<25s} ({role}) — {result[:70]}")
log(f"Fallback test {model}: {status}{result[:100]}")
if ok and first_good is None:
first_good = model
else:
print(f" [MISS] {model:<25s} ({role}) — not installed")
if first_good:
print(f"\n Primary serving: {first_good}")
else:
print(f"\n WARNING: No chain model responding. Fallback broken.")
log("FALLBACK CHAIN BROKEN — no models responding")
def status():
data = check_ollama()
if "error" in data:
print(f" Ollama: DOWN — {data['error']}")
else:
models = data.get("models", [])
print(f" Ollama: UP — {len(models)} models loaded")
print("\n=== Local Models ===")
list_models()
print("\n=== Chain Configuration ===")
if CHAIN_FILE.exists():
chain = json.loads(CHAIN_FILE.read_text()).get("chain", DEFAULT_CHAIN)
else:
chain = DEFAULT_CHAIN
for e in chain:
print(f" {e['model']:<25s} {e.get('role','?')}")
if __name__ == "__main__":
cmd = sys.argv[1] if len(sys.argv) > 1 else "status"
if cmd == "status": status()
elif cmd == "list": list_models()
elif cmd == "test": test_chain()
else:
status()
test_chain()

View File

@@ -146,6 +146,7 @@ class PullRequest:
additions: int = 0
deletions: int = 0
created_at: str = ""
updated_at: str = ""
closed_at: str = ""
@classmethod
@@ -166,6 +167,7 @@ class PullRequest:
additions=d.get("additions", 0),
deletions=d.get("deletions", 0),
created_at=d.get("created_at", ""),
updated_at=d.get("updated_at", ""),
closed_at=d.get("closed_at", ""),
)
@@ -314,6 +316,7 @@ class GiteaClient:
direction: str = "desc",
limit: int = 30,
page: int = 1,
since: Optional[str] = None,
) -> list[Issue]:
"""List issues for a repo."""
raw = self._get(
@@ -326,6 +329,7 @@ class GiteaClient:
direction=direction,
limit=limit,
page=page,
since=since,
)
return [Issue.from_dict(i) for i in raw]

526
scripts/kaizen_retro.py Normal file
View File

@@ -0,0 +1,526 @@
#!/usr/bin/env python3
"""
Kaizen Retro — Automated retrospective after every burn cycle.
Reads overnight Gitea activity, fleet state, and loop logs.
Generates ONE concrete improvement suggestion and posts it.
Usage:
python3 scripts/kaizen_retro.py [--dry-run]
"""
from __future__ import annotations
import argparse
import json
import os
import sys
import urllib.error
import urllib.request
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Any, Optional
# Ensure repo root is on path so we can import gitea_client
REPO_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(REPO_ROOT))
from gitea_client import GiteaClient, GiteaError
# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
REPOS = [
"Timmy_Foundation/the-nexus",
"Timmy_Foundation/timmy-config",
"Timmy_Foundation/timmy-home",
"Timmy_Foundation/the-door",
"Timmy_Foundation/turboquant",
"Timmy_Foundation/hermes-agent",
"Timmy_Foundation/.profile",
]
HERMES_HOME = Path.home() / ".hermes"
TIMMY_HOME = Path.home() / ".timmy"
WORKFORCE_STATE_PATH = HERMES_HOME / "workforce-state.json"
FLEET_ROUTING_PATH = HERMES_HOME / "fleet-routing.json"
CHANNEL_DIR_PATH = REPO_ROOT / "channel_directory.json"
REPORTS_DIR = REPO_ROOT / "reports"
MORNING_REPORT_REPO = "Timmy_Foundation/timmy-config"
TELEGRAM_BOT_TOKEN = os.environ.get("TELEGRAM_BOT_TOKEN")
TELEGRAM_CHAT_ID = os.environ.get("TELEGRAM_HOME_CHANNEL", "-1003664764329")
TELEGRAM_MAX_LEN = 4000 # leave headroom below the 4096 hard limit
STALE_DAYS = 7
MAX_ATTEMPT_COMMENT_THRESHOLD = 5
ISSUE_TYPE_KEYWORDS = {
"bug": ["bug", "fix", "crash", "error", "regression", "broken"],
"feature": ["feature", "implement", "add", "support", "enable"],
"docs": ["doc", "readme", "wiki", "guide", "documentation"],
"kaizen": ["kaizen", "retro", "improvement", "continuous"],
"devops": ["deploy", "ci", "cd", "docker", "server", "infra"],
}
BLOCKER_LABELS = {"blocked", "timeout", "stale", "help wanted", "wontfix", "duplicate"}
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def load_json(path: Path) -> Any:
if not path.exists():
return None
with open(path) as f:
return json.load(f)
def iso_day_ago(days: int = 1) -> str:
return (datetime.now(timezone.utc) - timedelta(days=days)).isoformat()
def classify_issue_type(issue: dict) -> str:
title = (issue.get("title", "") or "").lower()
body = (issue.get("body", "") or "").lower()
labels = [l.get("name", "").lower() for l in issue.get("labels", []) or []]
text = f"{title} {body} {' '.join(labels)}"
words = set(text.split())
best = "other"
best_score = 0
for kind, keywords in ISSUE_TYPE_KEYWORDS.items():
# Short keywords (<=3 chars) require whole-word match to avoid false positives like
# "ci" inside "cleanup" or "cd" inside "abcde".
score = sum(
1 for kw in keywords
if (len(kw) <= 3 and kw in words) or (len(kw) > 3 and kw in text)
)
# label match is stronger
for label in labels:
label_words = set(label.split())
if any(
(len(kw) <= 3 and kw in label_words) or (len(kw) > 3 and kw in label)
for kw in keywords
):
score += 3
if score > best_score:
best_score = score
best = kind
return best
def is_max_attempts_candidate(issue: dict) -> bool:
"""Heuristic for issues that consumed excessive attempts."""
labels = {l.get("name", "").lower() for l in issue.get("labels", []) or []}
if labels & BLOCKER_LABELS:
return True
if issue.get("comments", 0) >= MAX_ATTEMPT_COMMENT_THRESHOLD:
return True
created = issue.get("created_at")
if created:
try:
created_dt = datetime.fromisoformat(created.replace("Z", "+00:00"))
if datetime.now(timezone.utc) - created_dt > timedelta(days=STALE_DAYS):
return True
except Exception:
pass
return False
def telegram_send(text: str, bot_token: str, chat_id: str) -> list[dict]:
"""Post text to Telegram, chunking if it exceeds the message limit."""
url = f"https://api.telegram.org/bot{bot_token}/sendMessage"
chunks = []
if len(text) <= TELEGRAM_MAX_LEN:
chunks = [text]
else:
# Split on newlines to preserve readability
lines = text.splitlines(keepends=True)
current = ""
for line in lines:
if len(current) + len(line) > TELEGRAM_MAX_LEN:
if current:
chunks.append(current)
current = line
else:
current += line
if current:
chunks.append(current)
results = []
for i, chunk in enumerate(chunks):
prefix = f"*(part {i + 1}/{len(chunks)})*\n" if len(chunks) > 1 else ""
payload = {"chat_id": chat_id, "text": prefix + chunk, "parse_mode": "Markdown"}
data = json.dumps(payload).encode()
req = urllib.request.Request(url, data=data, headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=30) as resp:
results.append(json.loads(resp.read().decode()))
return results
def find_latest_morning_report_issue(client: GiteaClient) -> Optional[int]:
try:
issues = client.list_issues(MORNING_REPORT_REPO, state="open", sort="created", direction="desc", limit=20)
for issue in issues:
if "good morning report" in issue.title.lower() or "morning report" in issue.title.lower():
return issue.number
# fallback to closed
issues = client.list_issues(MORNING_REPORT_REPO, state="closed", sort="created", direction="desc", limit=20)
for issue in issues:
if "good morning report" in issue.title.lower() or "morning report" in issue.title.lower():
return issue.number
except Exception:
pass
return None
def fmt_pct(num: float, den: float) -> str:
if den == 0:
return "N/A"
return f"{num/den:.0%}"
# ---------------------------------------------------------------------------
# Analysis
# ---------------------------------------------------------------------------
def gather_metrics(client: GiteaClient, since: str) -> dict:
"""Collect overnight metrics from Gitea."""
metrics = {
"closed_issues": [],
"merged_prs": [],
"closed_prs": [],
"open_issues": [],
"max_attempts_issues": [],
"by_agent": {},
"by_repo": {},
"by_type": {},
}
for repo in REPOS:
repo_short = repo.split("/")[1]
metrics["by_repo"][repo_short] = {
"closed": 0,
"merged_prs": 0,
"closed_prs": 0,
"open": 0,
"max_attempts": 0,
"successes": 0,
"failures": 0,
}
# Closed issues since window
try:
closed = client.list_issues(repo, state="closed", since=since, sort="updated", direction="desc", limit=100)
for issue in closed:
issue_dict = {
"number": issue.number,
"title": issue.title,
"repo": repo_short,
"type": classify_issue_type({"title": issue.title, "body": issue.body, "labels": [{"name": lb.name} for lb in issue.labels]}),
"assignee": issue.assignees[0].login if issue.assignees else "unassigned",
}
metrics["closed_issues"].append(issue_dict)
metrics["by_repo"][repo_short]["closed"] += 1
metrics["by_repo"][repo_short]["successes"] += 1
agent = issue_dict["assignee"]
if agent not in metrics["by_agent"]:
metrics["by_agent"][agent] = {"successes": 0, "failures": 0, "closed": 0, "repos": set()}
metrics["by_agent"][agent]["successes"] += 1
metrics["by_agent"][agent]["closed"] += 1
metrics["by_agent"][agent]["repos"].add(repo_short)
t = issue_dict["type"]
if t not in metrics["by_type"]:
metrics["by_type"][t] = {"successes": 0, "failures": 0, "total": 0}
metrics["by_type"][t]["successes"] += 1
metrics["by_type"][t]["total"] += 1
except Exception as exc:
print(f"Warning: could not load closed issues for {repo}: {exc}", file=sys.stderr)
# Open issues (for stale / max-attempts detection)
try:
open_issues = client.list_issues(repo, state="open", sort="created", direction="desc", limit=100)
metrics["by_repo"][repo_short]["open"] = len(open_issues)
for issue in open_issues:
issue_raw = {
"number": issue.number,
"title": issue.title,
"labels": [{"name": lb.name} for lb in issue.labels],
"comments": issue.comments,
"created_at": issue.created_at,
}
if is_max_attempts_candidate(issue_raw):
metrics["max_attempts_issues"].append({
"number": issue.number,
"title": issue.title,
"repo": repo_short,
"type": classify_issue_type({"title": issue.title, "body": issue.body, "labels": issue_raw["labels"]}),
"assignee": issue.assignees[0].login if issue.assignees else "unassigned",
})
metrics["by_repo"][repo_short]["max_attempts"] += 1
metrics["by_repo"][repo_short]["failures"] += 1
agent = issue.assignees[0].login if issue.assignees else "unassigned"
if agent not in metrics["by_agent"]:
metrics["by_agent"][agent] = {"successes": 0, "failures": 0, "closed": 0, "repos": set()}
metrics["by_agent"][agent]["failures"] += 1
metrics["by_agent"][agent]["repos"].add(repo_short)
t = classify_issue_type({"title": issue.title, "body": issue.body, "labels": issue_raw["labels"]})
if t not in metrics["by_type"]:
metrics["by_type"][t] = {"successes": 0, "failures": 0, "total": 0}
metrics["by_type"][t]["failures"] += 1
metrics["by_type"][t]["total"] += 1
except Exception as exc:
print(f"Warning: could not load open issues for {repo}: {exc}", file=sys.stderr)
# PRs merged / closed since window (filter client-side; Gitea PR API ignores since)
try:
prs = client.list_pulls(repo, state="closed", sort="updated", limit=100)
since_dt = datetime.fromisoformat(since.replace("Z", "+00:00"))
for pr in prs:
updated = pr.updated_at or pr.created_at or ""
try:
updated_dt = datetime.fromisoformat(updated.replace("Z", "+00:00"))
if updated_dt < since_dt:
continue
except Exception:
pass
if pr.merged:
metrics["merged_prs"].append({
"number": pr.number,
"title": pr.title,
"repo": repo_short,
"user": pr.user.login if pr.user else "unknown",
})
metrics["by_repo"][repo_short]["merged_prs"] += 1
else:
metrics["closed_prs"].append({
"number": pr.number,
"title": pr.title,
"repo": repo_short,
"user": pr.user.login if pr.user else "unknown",
})
metrics["by_repo"][repo_short]["closed_prs"] += 1
except Exception as exc:
print(f"Warning: could not load PRs for {repo}: {exc}", file=sys.stderr)
# Convert sets to lists for JSON serialization
for agent in metrics["by_agent"].values():
agent["repos"] = sorted(agent["repos"])
return metrics
def load_workforce_state() -> dict:
return load_json(WORKFORCE_STATE_PATH) or {}
def load_fleet_routing() -> list[dict]:
data = load_json(FLEET_ROUTING_PATH)
if data and "agents" in data:
return data["agents"]
return []
def generate_suggestion(metrics: dict, fleet: list[dict]) -> str:
"""Generate ONE concrete improvement suggestion based on the data."""
by_agent = metrics["by_agent"]
by_repo = metrics["by_repo"]
by_type = metrics["by_type"]
max_attempts = metrics["max_attempts_issues"]
suggestions: list[str] = []
# 1. Agent with poor repo performance
for agent, stats in by_agent.items():
total = stats["successes"] + stats["failures"]
if total >= 3 and stats["successes"] == 0:
repos = ", ".join(stats["repos"])
suggestions.append(
f"🎯 **{agent}** has a 0% verify rate over the last cycle (0/{total}) on repos: {repos}. "
f"Consider removing these repos from {agent}'s routing or providing targeted onboarding."
)
# 2. Repo with highest failure concentration
repo_failures = [(r, s) for r, s in by_repo.items() if s["failures"] > 0]
if repo_failures:
repo_failures.sort(key=lambda x: x[1]["failures"], reverse=True)
worst_repo, worst_stats = repo_failures[0]
total_repo = worst_stats["successes"] + worst_stats["failures"]
if worst_stats["failures"] >= 2:
suggestions.append(
f"🎯 **{worst_repo}** has the most friction ({worst_stats['failures']} blocked/stale issues, "
f"{fmt_pct(worst_stats['successes'], total_repo)} success). "
f"Consider splitting issues in {worst_repo} into smaller chunks or assigning a stronger agent."
)
# 3. Max-attempts pattern
if len(max_attempts) >= 3:
type_counts: dict[str, int] = {}
for issue in max_attempts:
type_counts[issue["type"]] = type_counts.get(issue["type"], 0) + 1
top_type = max(type_counts, key=type_counts.get) if type_counts else "unknown"
suggestions.append(
f"🎯 **{len(max_attempts)} issues** hit max-attempts or went stale. "
f"The dominant type is **{top_type}**. "
f"Consider adding acceptance criteria templates or pre-flight checklists for {top_type} issues."
)
# 4. Issue type disparity
for t, stats in by_type.items():
total = stats["total"]
if total >= 3 and stats["successes"] == 0:
suggestions.append(
f"🎯 **{t}** issues have a 0% closure rate ({stats['failures']} stale). "
f"Consider routing all {t} issues to a specialist agent or creating a dedicated playbook."
)
# 5. Fleet routing gap (if fleet data exists)
active_agents = {a["name"] for a in fleet if a.get("active")}
assigned_agents = set(by_agent.keys())
idle_agents = active_agents - assigned_agents - {"unassigned"}
if len(idle_agents) >= 2:
suggestions.append(
f"🎯 **{len(idle_agents)} active agents** have no assignments this cycle: {', '.join(idle_agents)}. "
f"Consider expanding their repo lists or investigating why they aren't receiving work."
)
if suggestions:
return suggestions[0]
# Fallback: celebrate or nudge
total_closed = len(metrics["closed_issues"])
total_merged = len(metrics["merged_prs"])
if total_closed >= 5 or total_merged >= 3:
return (
f"🎯 Strong cycle: {total_closed} issues closed, {total_merged} PRs merged. "
f"Next improvement: write down the top 3 patterns that made this cycle successful so we can replicate them."
)
return (
"🎯 Low activity this cycle. Next improvement: ensure at least one agent loop is actively polling "
"for unassigned issues so work doesn't sit idle."
)
def build_report(metrics: dict, suggestion: str, since: str) -> str:
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
period = since[:10]
lines = [
f"# 🌀 Kaizen Retro — {now}",
f"*Period: {period} → now*\n",
"## Numbers",
f"- **Issues closed:** {len(metrics['closed_issues'])}",
f"- **PRs merged:** {len(metrics['merged_prs'])}",
f"- **PRs closed without merge:** {len(metrics['closed_prs'])}",
f"- **Max-attempts / stale issues:** {len(metrics['max_attempts_issues'])}",
"",
"## By Agent",
]
for agent, stats in sorted(metrics["by_agent"].items(), key=lambda x: x[1]["successes"] + x[1]["failures"], reverse=True):
total = stats["successes"] + stats["failures"]
rate = fmt_pct(stats["successes"], total)
lines.append(f"- **{agent}**: {stats['successes']} closed, {stats['failures']} stale / max-attempts — verify rate {rate}")
lines.extend(["", "## By Repo"])
for repo, stats in sorted(metrics["by_repo"].items(), key=lambda x: x[1]["successes"] + x[1]["failures"], reverse=True):
total = stats["successes"] + stats["failures"]
if total == 0 and stats["open"] == 0:
continue
rate = fmt_pct(stats["successes"], total)
lines.append(
f"- **{repo}**: {stats['successes']} closed, {stats['failures']} stale, {stats['open']} open — verify rate {rate}"
)
lines.extend(["", "## By Issue Type"])
for t, stats in sorted(metrics["by_type"].items(), key=lambda x: x[1]["total"], reverse=True):
total = stats["total"]
rate = fmt_pct(stats["successes"], total)
lines.append(f"- **{t}**: {stats['successes']} closed, {stats['failures']} stale — verify rate {rate}")
if metrics["max_attempts_issues"]:
lines.extend(["", "## Max-Attempts / Stale Issues"])
for issue in metrics["max_attempts_issues"][:10]:
lines.append(f"- {issue['repo']}#{issue['number']} ({issue['type']}, assignee: {issue['assignee']}) — {issue['title']}")
if len(metrics["max_attempts_issues"]) > 10:
lines.append(f"- … and {len(metrics['max_attempts_issues']) - 10} more")
lines.extend(["", "## One Concrete Improvement", suggestion, ""])
return "\n".join(lines)
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def main() -> int:
parser = argparse.ArgumentParser(description="Kaizen Retro — automated burn-cycle retrospective")
parser.add_argument("--dry-run", action="store_true", help="Print report but do not post")
parser.add_argument("--since", type=str, help="ISO timestamp for lookback window (default: 24h ago)")
parser.add_argument("--post-to", type=str, help="Override Telegram chat ID")
args = parser.parse_args()
since = args.since or iso_day_ago(1)
client = GiteaClient()
print("Gathering metrics since", since)
metrics = gather_metrics(client, since)
fleet = load_fleet_routing()
suggestion = generate_suggestion(metrics, fleet)
report = build_report(metrics, suggestion, since)
print(report)
# Save JSON snapshot
REPORTS_DIR.mkdir(parents=True, exist_ok=True)
snapshot_path = REPORTS_DIR / f"kaizen-retro-{datetime.now(timezone.utc).strftime('%Y%m%d')}.json"
snapshot = {
"generated_at": datetime.now(timezone.utc).isoformat(),
"since": since,
"metrics": metrics,
"suggestion": suggestion,
"report_markdown": report,
}
with open(snapshot_path, "w") as f:
json.dump(snapshot, f, indent=2)
print(f"\nSnapshot saved to {snapshot_path}")
if args.dry_run:
return 0
# Post to Telegram
chat_id = args.post_to or TELEGRAM_CHAT_ID
bot_token = TELEGRAM_BOT_TOKEN
if bot_token and chat_id:
try:
telegram_send(report, bot_token, chat_id)
print("Posted to Telegram.")
except Exception as exc:
print(f"Failed to post to Telegram: {exc}", file=sys.stderr)
else:
print("Telegram not configured (set TELEGRAM_BOT_TOKEN and TELEGRAM_HOME_CHANNEL).", file=sys.stderr)
# Comment on latest morning report issue
morning_issue = find_latest_morning_report_issue(client)
if morning_issue:
try:
client.create_comment(MORNING_REPORT_REPO, morning_issue, report)
print(f"Commented on morning report issue #{morning_issue}.")
except Exception as exc:
print(f"Failed to comment on morning report issue: {exc}", file=sys.stderr)
else:
print("No morning report issue found to comment on.", file=sys.stderr)
return 0
if __name__ == "__main__":
sys.exit(main())

112
tasks.py
View File

@@ -1860,22 +1860,56 @@ def good_morning_report():
except Exception:
pass
# Genchi Genbutsu: count verified completions from the last 24h
verified_completions = 0
raw_completions = 0
metrics_dir = Path.home() / ".hermes" / "logs"
for metrics_file in metrics_dir.glob("*-metrics.jsonl"):
try:
with open(metrics_file) as mf:
for line in mf:
line = line.strip()
if not line:
continue
try:
row = json.loads(line)
except json.JSONDecodeError:
continue
ts = row.get("ts", "")
if not ts:
continue
try:
from datetime import datetime as _dt, timezone as _tz, timedelta as _td
row_time = _dt.fromisoformat(ts.replace("Z", "+00:00"))
if (now - row_time) > _td(hours=24):
continue
except Exception:
continue
if row.get("outcome") != "success":
continue
raw_completions += 1
if row.get("verified") is True:
verified_completions += 1
except Exception:
pass
# --- BUILD THE REPORT ---
body = f"""Good morning, Alexander. It's {day_name}.
## Overnight Debrief
**Heartbeat:** {tick_count} ticks logged overnight.
**Gitea:** {"up all night" if gitea_up else "⚠️ had downtime"}
**Local inference:** {"running steady" if local_inference_up else "⚠️ had downtime"}
**Model status:** {model_status}
**Models on disk:** {len(models_loaded)} ({', '.join(m for m in models_loaded if 'timmy' in m.lower() or 'hermes' in m.lower()) or 'none with our name'})
**Alerts:** {len(alerts)} {'' + '; '.join(alerts[-3:]) if alerts else '(clean night)'}
|**Heartbeat:** {tick_count} ticks logged overnight.
|**Gitea:** {"up all night" if gitea_up else "⚠️ had downtime"}
|**Local inference:** {"running steady" if local_inference_up else "⚠️ had downtime"}
|**Model status:** {model_status}
|**Models on disk:** {len(models_loaded)} ({', '.join(m for m in models_loaded if 'timmy' in m.lower() or 'hermes' in m.lower()) or 'none with our name'})
|**Alerts:** {len(alerts)} {'' + '; '.join(alerts[-3:]) if alerts else '(clean night)'}
{briefing_summary}
**DPO training pairs staged:** {dpo_count} session files exported
**Local model smoke test:** {smoke_result}
**Verified completions (24h):** {verified_completions} {'(Genchi Genbutsu clean)' if verified_completions == raw_completions else f'({raw_completions - verified_completions} raw completions failed verification)'}
## Gitea Pulse
@@ -1915,6 +1949,29 @@ That's all. Have a good morning.
return {"filed": False, "error": str(e)}
# ── NEW 6b: Kaizen Retro ─────────────────────────────────────────────
@huey.periodic_task(crontab(hour="7", minute="15")) # 7:15 AM daily, after morning report
def kaizen_retro():
"""Run the automated burn-cycle retrospective."""
retro_script = Path(__file__).resolve().parent / "bin" / "kaizen-retro.sh"
if not retro_script.exists():
return {"ran": False, "error": "kaizen-retro.sh not found"}
result = subprocess.run(
["bash", str(retro_script)],
capture_output=True,
text=True,
timeout=300,
)
return {
"ran": True,
"exit_code": result.returncode,
"stdout": result.stdout[-2000:] if result.stdout else "",
"stderr": result.stderr[-1000:] if result.stderr else "",
}
# ── NEW 7: Repo Watchdog ─────────────────────────────────────────────
@huey.periodic_task(crontab(minute="*/20")) # every 20 minutes
@@ -2323,7 +2380,38 @@ def velocity_tracking():
total_open += open_n
total_closed += closed_n
results.append({"repo": repo, "open": open_n, "closed": closed_n, "date": today})
data = {"date": today, "repos": results, "total_open": total_open, "total_closed": total_closed}
# Genchi Genbutsu: count verified completions from agent metrics
verified_completions = 0
raw_completions = 0
metrics_dir = Path.home() / ".hermes" / "logs"
for metrics_file in metrics_dir.glob("*-metrics.jsonl"):
try:
with open(metrics_file) as mf:
for line in mf:
line = line.strip()
if not line:
continue
try:
row = json.loads(line)
except json.JSONDecodeError:
continue
if row.get("outcome") != "success":
continue
raw_completions += 1
if row.get("verified") is True:
verified_completions += 1
except Exception:
pass
data = {
"date": today,
"repos": results,
"total_open": total_open,
"total_closed": total_closed,
"raw_completions": raw_completions,
"verified_completions": verified_completions,
}
with open(report_file, "w") as f:
json.dump(data, f, indent=2)
# Dashboard
@@ -2333,14 +2421,16 @@ def velocity_tracking():
for r in results:
f.write(f"| {r['repo'].split('/')[-1]} | {r['open']} | {r['closed']} |\n")
f.write(f"| **TOTAL** | **{total_open}** | **{total_closed}** |\n\n")
f.write(f"**Verified completions (Genchi Genbutsu):** {verified_completions}\n")
f.write(f"**Raw completions:** {raw_completions}\n\n")
# Trend
prior = sorted(glob.glob(os.path.join(report_dir, "velocity-*.json")))
if len(prior) > 1:
f.write("## Recent Trend\n\n| Date | Total Open | Total Closed |\n|---|---|---|\n")
f.write("## Recent Trend\n\n| Date | Total Open | Total Closed | Verified |\n|---|---|---|---|\n")
for pf in prior[-10:]:
pd = json.load(open(pf))
f.write(f"| {pd['date']} | {pd['total_open']} | {pd['total_closed']} |\n")
msg = f"Velocity: {total_open} open, {total_closed} closed ({today})"
f.write(f"| {pd['date']} | {pd['total_open']} | {pd['total_closed']} | {pd.get('verified_completions', '-')} |\n")
msg = f"Velocity: {total_open} open, {total_closed} closed, {verified_completions} verified ({today})"
if len(prior) > 1:
prev = json.load(open(prior[-2]))
if total_open > prev["total_open"]:

283
tests/test_kaizen_retro.py Normal file
View File

@@ -0,0 +1,283 @@
"""Tests for the Kaizen Retro burn-cycle retrospective script."""
from __future__ import annotations
import importlib.util
import json
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path
from unittest.mock import MagicMock, patch
import pytest
REPO_ROOT = Path(__file__).parent.parent
# Load kaizen_retro.py as a module (it lives in scripts/, not a package)
spec = importlib.util.spec_from_file_location("kaizen_retro", REPO_ROOT / "scripts" / "kaizen_retro.py")
kr = importlib.util.module_from_spec(spec)
spec.loader.exec_module(kr)
# ── classify_issue_type ───────────────────────────────────────────────────
class TestClassifyIssueType:
def test_classifies_bug_from_title(self):
issue = {"title": "Fix crash on startup", "body": "", "labels": []}
assert kr.classify_issue_type(issue) == "bug"
def test_classifies_feature_from_label(self):
issue = {"title": "Add dark mode", "body": "", "labels": [{"name": "enhancement"}]}
# label "enhancement" doesn't match any keyword directly, but "feature" and "add" are in title
assert kr.classify_issue_type(issue) == "feature"
def test_classifies_docs_from_label(self):
issue = {"title": "Update guide", "body": "", "labels": [{"name": "documentation"}]}
assert kr.classify_issue_type(issue) == "docs"
def test_label_match_stronger_than_title(self):
issue = {"title": "Something random", "body": "", "labels": [{"name": "bug"}]}
assert kr.classify_issue_type(issue) == "bug"
def test_kaizen_takes_precedence_with_both_labels(self):
issue = {"title": "Process improvement", "body": "", "labels": [{"name": "kaizen"}, {"name": "bug"}]}
# kaizen label gives +3, bug gives +3, tie goes to first seen? kaizen appears first in dict
assert kr.classify_issue_type(issue) == "kaizen"
def test_defaults_to_other(self):
issue = {"title": "Tidy up naming", "body": "No user-facing change", "labels": [{"name": "cleanup"}]}
assert kr.classify_issue_type(issue) == "other"
# ── is_max_attempts_candidate ─────────────────────────────────────────────
class TestIsMaxAttemptsCandidate:
def test_blocker_label_returns_true(self):
issue = {"labels": [{"name": "blocked"}], "comments": 0, "created_at": "2026-04-07T00:00:00Z"}
assert kr.is_max_attempts_candidate(issue) is True
def test_timeout_label_returns_true(self):
issue = {"labels": [{"name": "timeout"}], "comments": 0, "created_at": "2026-04-07T00:00:00Z"}
assert kr.is_max_attempts_candidate(issue) is True
def test_high_comment_count_returns_true(self):
issue = {"labels": [], "comments": 5, "created_at": "2026-04-07T00:00:00Z"}
assert kr.is_max_attempts_candidate(issue) is True
def test_fresh_issue_with_low_comments_returns_false(self):
now = datetime.now(timezone.utc)
issue = {"labels": [], "comments": 2, "created_at": now.isoformat()}
assert kr.is_max_attempts_candidate(issue) is False
def test_stale_age_returns_true(self):
old = datetime.now(timezone.utc) - timedelta(days=10)
issue = {"labels": [], "comments": 0, "created_at": old.isoformat()}
assert kr.is_max_attempts_candidate(issue) is True
# ── fmt_pct ───────────────────────────────────────────────────────────────
class TestFmtPct:
def test_basic_percentage(self):
assert kr.fmt_pct(3, 4) == "75%"
def test_zero_denominator(self):
assert kr.fmt_pct(0, 0) == "N/A"
def test_perfect_rate(self):
assert kr.fmt_pct(10, 10) == "100%"
# ── generate_suggestion ───────────────────────────────────────────────────
class TestGenerateSuggestion:
def test_agent_zero_success_rate(self):
metrics = {
"by_agent": {
"groq": {"successes": 0, "failures": 5, "closed": 0, "repos": ["timmy-home"]},
},
"by_repo": {},
"by_type": {},
"max_attempts_issues": [],
"closed_issues": [],
"merged_prs": [],
"closed_prs": [],
}
suggestion = kr.generate_suggestion(metrics, [])
assert "groq" in suggestion
assert "0%" in suggestion or "verify rate" in suggestion
def test_repo_with_most_failures(self):
metrics = {
"by_agent": {},
"by_repo": {
"the-nexus": {"successes": 2, "failures": 5, "closed": 2, "open": 3},
},
"by_type": {},
"max_attempts_issues": [],
"closed_issues": [],
"merged_prs": [],
"closed_prs": [],
}
suggestion = kr.generate_suggestion(metrics, [])
assert "the-nexus" in suggestion
assert "friction" in suggestion
def test_max_attempts_pattern(self):
metrics = {
"by_agent": {},
"by_repo": {},
"by_type": {},
"max_attempts_issues": [
{"type": "devops"}, {"type": "devops"}, {"type": "feature"}
],
"closed_issues": [],
"merged_prs": [],
"closed_prs": [],
}
suggestion = kr.generate_suggestion(metrics, [])
assert "devops" in suggestion
assert "max-attempts" in suggestion.lower() or "stale" in suggestion.lower()
def test_idle_agents(self):
metrics = {
"by_agent": {},
"by_repo": {},
"by_type": {},
"max_attempts_issues": [],
"closed_issues": [],
"merged_prs": [],
"closed_prs": [],
}
fleet = [{"name": "allegro", "active": True}, {"name": "ezra", "active": True}]
suggestion = kr.generate_suggestion(metrics, fleet)
assert "idle" in suggestion.lower() or "no assignments" in suggestion.lower()
def test_fallback_celebration(self):
metrics = {
"by_agent": {},
"by_repo": {},
"by_type": {},
"max_attempts_issues": [],
"closed_issues": [{}, {}, {}, {}, {}],
"merged_prs": [{}, {}, {}],
"closed_prs": [],
}
suggestion = kr.generate_suggestion(metrics, [])
assert "Strong cycle" in suggestion
def test_fallback_low_activity(self):
metrics = {
"by_agent": {},
"by_repo": {},
"by_type": {},
"max_attempts_issues": [],
"closed_issues": [],
"merged_prs": [],
"closed_prs": [],
}
suggestion = kr.generate_suggestion(metrics, [])
assert "Low activity" in suggestion or "idle" in suggestion.lower()
# ── build_report ──────────────────────────────────────────────────────────
class TestBuildReport:
def test_report_contains_numbers_section(self):
metrics = {
"closed_issues": [{}, {}],
"merged_prs": [{}],
"closed_prs": [],
"max_attempts_issues": [],
"by_agent": {"ezra": {"successes": 2, "failures": 0, "repos": ["timmy-config"]}},
"by_repo": {"timmy-config": {"successes": 2, "failures": 0, "open": 1}},
"by_type": {"feature": {"successes": 2, "failures": 0, "total": 2}},
}
report = kr.build_report(metrics, "Do better.", "2026-04-06T00:00:00+00:00")
assert "## Numbers" in report
assert "Issues closed:** 2" in report
assert "PRs merged:** 1" in report
assert "## By Agent" in report
assert "## By Repo" in report
assert "## By Issue Type" in report
assert "Do better." in report
def test_report_skips_empty_repos(self):
metrics = {
"closed_issues": [],
"merged_prs": [],
"closed_prs": [],
"max_attempts_issues": [],
"by_agent": {},
"by_repo": {"unused-repo": {"successes": 0, "failures": 0, "open": 0}},
"by_type": {},
}
report = kr.build_report(metrics, "Nudge.", "2026-04-06T00:00:00+00:00")
assert "unused-repo" not in report
def test_report_truncates_max_attempts(self):
metrics = {
"closed_issues": [],
"merged_prs": [],
"closed_prs": [],
"max_attempts_issues": [{"repo": "r", "number": i, "type": "bug", "assignee": "a", "title": f"T{i}"} for i in range(15)],
"by_agent": {},
"by_repo": {},
"by_type": {},
}
report = kr.build_report(metrics, "Fix it.", "2026-04-06T00:00:00+00:00")
assert "and 5 more" in report
# ── telegram_send ─────────────────────────────────────────────────────────
class TestTelegramSend:
def test_short_message_sent_in_one_piece(self):
with patch("urllib.request.urlopen") as mock_urlopen:
mock_resp = MagicMock()
mock_resp.read.return_value = b'{"ok": true}'
mock_urlopen.return_value.__enter__.return_value = mock_resp
results = kr.telegram_send("Hello", "fake-token", "123")
assert len(results) == 1
assert results[0]["ok"] is True
# Verify payload
call_args = mock_urlopen.call_args
req = call_args[0][0]
payload = json.loads(req.data.decode())
assert payload["text"] == "Hello"
assert payload["chat_id"] == "123"
def test_long_message_chunked(self):
big_text = "Line\n" * 2000 # ~10k chars
with patch("urllib.request.urlopen") as mock_urlopen:
mock_resp = MagicMock()
mock_resp.read.return_value = b'{"ok": true}'
mock_urlopen.return_value.__enter__.return_value = mock_resp
results = kr.telegram_send(big_text, "fake-token", "123")
assert len(results) >= 2
# First chunk should have a part prefix
req = mock_urlopen.call_args_list[0][0][0]
payload = json.loads(req.data.decode())
assert "(part 1" in payload["text"]
# ── load helpers ──────────────────────────────────────────────────────────
class TestLoadHelpers:
def test_load_json_missing_returns_none(self, tmp_path):
missing = tmp_path / "does_not_exist.json"
assert kr.load_json(missing) is None
def test_load_json_valid(self, tmp_path):
p = tmp_path / "data.json"
p.write_text('{"a": 1}')
assert kr.load_json(p) == {"a": 1}
def test_iso_day_ago_format(self):
s = kr.iso_day_ago(1)
# Should be a valid ISO timestamp string
dt = datetime.fromisoformat(s)
now = datetime.now(timezone.utc)
assert now - dt < timedelta(days=2)