Compare commits

..

21 Commits

Author SHA1 Message Date
86914554f1 Backup: bezalel crontab paused and preserved
Some checks failed
Architecture Lint / Linter Tests (pull_request) Successful in 9s
PR Checklist / pr-checklist (pull_request) Failing after 1m46s
Architecture Lint / Lint Repository (pull_request) Failing after 7s
2026-04-10 19:17:48 +00:00
a4665679ab Backup: allegro crontab paused and preserved
Some checks failed
Architecture Lint / Linter Tests (pull_request) Successful in 8s
PR Checklist / pr-checklist (pull_request) Failing after 1m44s
Architecture Lint / Lint Repository (pull_request) Failing after 7s
2026-04-10 19:17:46 +00:00
6f3ed4c963 Backup: ezra crontab paused and preserved 2026-04-10 19:17:44 +00:00
Alexander Whitestone
a65f736f54 Backup: all 35 cron jobs paused, state preserved
Some checks failed
Architecture Lint / Linter Tests (pull_request) Successful in 8s
PR Checklist / pr-checklist (pull_request) Failing after 1m40s
Architecture Lint / Lint Repository (pull_request) Failing after 7s
2026-04-10 15:06:29 -04:00
8bf41c00e4 Merge pull request #450
Some checks failed
Architecture Lint / Linter Tests (push) Successful in 8s
Architecture Lint / Lint Repository (push) Failing after 7s
Merged PR #450
2026-04-10 11:48:32 +00:00
41046d4bf1 Merge pull request #430
Some checks failed
Architecture Lint / Linter Tests (push) Has been cancelled
Architecture Lint / Lint Repository (push) Has been cancelled
Merged PR #430
2026-04-10 11:48:29 +00:00
52d60198fc [auto-merge] Fix PR template
Some checks failed
Architecture Lint / Linter Tests (push) Has been cancelled
Architecture Lint / Lint Repository (push) Has been cancelled
Auto-merged by PR review bot: Fix PR template
2026-04-10 11:48:27 +00:00
ae7915fc20 [auto-merge] add config validator script
Some checks failed
Architecture Lint / Linter Tests (push) Has been cancelled
Architecture Lint / Lint Repository (push) Has been cancelled
Auto-merged by PR review bot: add config validator script
2026-04-10 11:48:26 +00:00
Alexander Whitestone
49b0b9d207 feat: add config validator script
Some checks failed
Architecture Lint / Linter Tests (pull_request) Successful in 7s
PR Checklist / pr-checklist (pull_request) Failing after 1m8s
Architecture Lint / Lint Repository (pull_request) Failing after 7s
scripts/config_validator.py — standalone validator for all YAML/JSON
config files in the repo.

Checks:
- YAML syntax (pyyaml safe_load)
- JSON syntax (json.loads)
- Duplicate keys in YAML/JSON
- Trailing whitespace
- Tabs in YAML (should use spaces)
- Cron expression validity (if present)

Reports PASS/FAIL per file with line numbers.
Exit 0 if all valid, 1 if any invalid.
2026-04-10 07:13:17 -04:00
Alexander Whitestone
d64b2e7561 burn: Fix PR template — remove duplication, strengthen proof enforcement
Some checks failed
Architecture Lint / Linter Tests (pull_request) Successful in 8s
PR Checklist / pr-checklist (pull_request) Successful in 1m40s
Architecture Lint / Lint Repository (pull_request) Failing after 6s
- Eliminated duplicate content (entire template was repeated twice)
- Renamed 'Linked Issue' to 'Governing Issue' per CONTRIBUTING.md language
- Added explicit 'no proof = no merge' callout in Proof section
- Renamed 'What was tested' to 'Commands / logs / world-state proof' for clarity
- Enhanced checklist with items from #393: issue reference, syntactic validity, proof standard
- Added inline guidance comments referencing CONTRIBUTING.md

Closes #451
2026-04-10 06:22:38 -04:00
3fd4223e1e Merge pull request #424
Some checks failed
Architecture Lint / Linter Tests (push) Successful in 9s
Architecture Lint / Lint Repository (push) Failing after 6s
Merged PR #424
2026-04-10 09:37:46 +00:00
d8f88bed16 Merge pull request #449
Some checks failed
Architecture Lint / Linter Tests (push) Has been cancelled
Architecture Lint / Lint Repository (push) Has been cancelled
Merged PR #449
2026-04-10 09:37:44 +00:00
a01935825c Merge branch 'main' into timmy/v7.0.0-checkin
Some checks failed
Architecture Lint / Linter Tests (pull_request) Successful in 7s
PR Checklist / pr-checklist (pull_request) Failing after 1m12s
Architecture Lint / Lint Repository (pull_request) Failing after 7s
2026-04-10 09:37:40 +00:00
544f2a9729 Merge branch 'main' into ansible-iac
Some checks failed
Architecture Lint / Linter Tests (pull_request) Successful in 8s
Architecture Lint / Lint Repository (pull_request) Failing after 6s
PR Checklist / pr-checklist (pull_request) Failing after 1m43s
2026-04-10 09:37:38 +00:00
71bf82d9fb Merge branch 'main' into burn/20260409-1247-self-healing-safe
Some checks failed
Architecture Lint / Linter Tests (pull_request) Successful in 8s
PR Checklist / pr-checklist (pull_request) Failing after 1m16s
Architecture Lint / Lint Repository (pull_request) Failing after 6s
2026-04-10 09:37:36 +00:00
28317cbde9 Merge branch 'main' into timmy/v7.0.0-checkin
Some checks failed
Architecture Lint / Linter Tests (pull_request) Successful in 7s
PR Checklist / pr-checklist (pull_request) Failing after 1m11s
Architecture Lint / Lint Repository (pull_request) Failing after 6s
2026-04-10 09:36:27 +00:00
e124ff8b05 Merge branch 'main' into ansible-iac
Some checks failed
PR Checklist / pr-checklist (pull_request) Failing after 1m40s
2026-04-10 09:36:21 +00:00
5a649966ab Merge branch 'main' into burn/20260409-1247-self-healing-safe
Some checks failed
PR Checklist / pr-checklist (pull_request) Failing after 1m43s
2026-04-10 09:36:19 +00:00
Perplexity
7ec45642eb feat(ansible): Canonical IaC playbook for fleet management
Some checks failed
PR Checklist / pr-checklist (pull_request) Failing after 1m27s
Implements the Ansible Infrastructure as Code story from KT 2026-04-08.

One canonical Ansible playbook defines:
- Deadman switch (snapshot good config on health, rollback+restart on death)
- Golden state config deployment (Anthropic BANNED, Kimi→Gemini→Ollama)
- Cron schedule (source-controlled, no manual crontab edits)
- Agent startup sequence (pull→validate→start→verify)
- request_log telemetry table (every inference call logged)
- Thin config pattern (immutable local pointer to upstream)
- Gitea webhook handler (deploy on merge)
- Config validator (rejects banned providers)

Fleet inventory: Timmy (Mac), Allegro (VPS), Bezalel (VPS), Ezra (VPS)

Roles: wizard_base, golden_state, deadman_switch, request_log, cron_manager

Addresses: timmy-config #442, #443, #444, #445, #446
References: KT Final 2026-04-08 P2, KT Bezalel 2026-04-08 #1-#5
2026-04-09 22:25:31 +00:00
Alexander Whitestone
179833148f feat(scripts/self_healing.py): safe-by-default with dry-run support
All checks were successful
PR Checklist / pr-checklist (pull_request) Successful in 1m14s
- Add --dry-run as default mode (no changes made)
- Add --execute flag to actually perform fixes
- Add --help-safe to explain each action
- Add confirmation prompts for destructive actions
- Add --confirm-kill flag for process termination (dangerous)
- Add --yes flag to skip confirmations for automation
- Add timestamps to log messages
- Improve SSH connection timeout
- Maintain existing functionality while making it safe by default

Addresses issue #435
2026-04-09 12:49:39 -04:00
a31f58000b v7.0.0: Fleet architecture checkin — 6 agents alive, release tagging begins
Some checks failed
PR Checklist / pr-checklist (pull_request) Failing after 2m53s
2026-04-08 21:44:53 +00:00
36 changed files with 2430 additions and 35 deletions

View File

@@ -2,7 +2,7 @@
<!-- What changed and why. One paragraph max. -->
## Linked Issue
## Governing Issue
<!-- REQUIRED. Every PR must reference at least one issue. Max 3 issues per PR. -->
<!-- Closes #ISSUENUM -->
@@ -10,18 +10,20 @@
## Acceptance Criteria
<!-- What specific outcomes does this PR deliver? Check each when proven. -->
<!-- List the specific outcomes this PR delivers. Check each only when proven. -->
<!-- Copy these from the governing issue if it has them. -->
- [ ] Criterion 1
- [ ] Criterion 2
## Proof
### What was tested
<!-- Paste the exact commands, output, log paths, or world-state artifacts that prove the acceptance criteria were met. -->
<!-- No proof = no merge. See CONTRIBUTING.md for the full standard. -->
### Commands / logs / world-state proof
<!-- Paste the exact commands, output, log paths, or world-state artifacts that prove each acceptance criterion was met. -->
```
$ <command you ran>
<relevant output>
@@ -42,8 +44,11 @@ $ <command you ran>
## Checklist
- [ ] Proof meets CONTRIBUTING.md standard (exact commands, output, or artifacts)
- [ ] Python files pass syntax check (`python -c "import ast; ast.parse(open('file.py').read())"`)
- [ ] Shell scripts are executable (`chmod +x`)
<!-- Complete every item before requesting review. -->
- [ ] PR body references at least one issue number (`Closes #N` or `Refs #N`)
- [ ] Changed files are syntactically valid (`python -c "import ast; ast.parse(open(f).read())"`, `node --check`, `bash -n`)
- [ ] Proof meets CONTRIBUTING.md standard (exact commands, output, or artifacts — not "looks right")
- [ ] Branch is up-to-date with base
- [ ] No more than 3 unrelated issues bundled in this PR
- [ ] Shell scripts are executable (`chmod +x`)

View File

@@ -0,0 +1,47 @@
# =============================================================================
# BANNED PROVIDERS — The Timmy Foundation
# =============================================================================
# "Anthropic is not only fired, but banned. I don't want these errors
# cropping up." — Alexander, 2026-04-09
#
# This is a HARD BAN. Not deprecated. Not fallback. BANNED.
# Enforcement: pre-commit hook, linter, Ansible validation, CI tests.
# =============================================================================
banned_providers:
- name: anthropic
reason: "Permanently banned. SDK access gated despite active quota. Fleet was bricked because golden state pointed to Anthropic Sonnet."
banned_date: "2026-04-09"
enforcement: strict # Ansible playbook FAILS if detected
models:
- "claude-sonnet-*"
- "claude-opus-*"
- "claude-haiku-*"
- "claude-*"
endpoints:
- "api.anthropic.com"
- "anthropic/*" # OpenRouter pattern
api_keys:
- "ANTHROPIC_API_KEY"
- "CLAUDE_API_KEY"
# Golden state alternative:
approved_providers:
- name: kimi-coding
model: kimi-k2.5
role: primary
- name: openrouter
model: google/gemini-2.5-pro
role: fallback
- name: ollama
model: "gemma4:latest"
role: terminal_fallback
# Future evaluation:
evaluation_candidates:
- name: mimo-v2-pro
status: pending
notes: "Free via Nous Portal for ~2 weeks from 2026-04-07. Add after fallback chain is fixed."
- name: hermes-4
status: available
notes: "Free on Nous Portal. 36B and 70B variants. Home team model."

95
ansible/README.md Normal file
View File

@@ -0,0 +1,95 @@
# Ansible IaC — The Timmy Foundation Fleet
> One canonical Ansible playbook defines: deadman switch, cron schedule,
> golden state rollback, agent startup sequence.
> — KT Final Session 2026-04-08, Priority TWO
## Purpose
This directory contains the **single source of truth** for fleet infrastructure.
No more ad-hoc recovery implementations. No more overlapping deadman switches.
No more agents mutating their own configs into oblivion.
**Everything** goes through Ansible. If it's not in a playbook, it doesn't exist.
## Architecture
```
┌─────────────────────────────────────────────────┐
│ Gitea (Source of Truth) │
│ timmy-config/ansible/ │
│ ├── inventory/hosts.yml (fleet machines) │
│ ├── playbooks/site.yml (master playbook) │
│ ├── roles/ (reusable roles) │
│ └── group_vars/wizards.yml (golden state) │
└──────────────────┬──────────────────────────────┘
│ PR merge triggers webhook
┌─────────────────────────────────────────────────┐
│ Gitea Webhook Handler │
│ scripts/deploy_on_webhook.sh │
│ → ansible-pull on each target machine │
└──────────────────┬──────────────────────────────┘
│ ansible-pull
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Timmy │ │ Allegro │ │ Bezalel │ │ Ezra │
│ (Mac) │ │ (VPS) │ │ (VPS) │ │ (VPS) │
│ │ │ │ │ │ │ │
│ deadman │ │ deadman │ │ deadman │ │ deadman │
│ cron │ │ cron │ │ cron │ │ cron │
│ golden │ │ golden │ │ golden │ │ golden │
│ req_log │ │ req_log │ │ req_log │ │ req_log │
└──────────┘ └──────────┘ └──────────┘ └──────────┘
```
## Quick Start
```bash
# Deploy everything to all machines
ansible-playbook -i inventory/hosts.yml playbooks/site.yml
# Deploy only golden state config
ansible-playbook -i inventory/hosts.yml playbooks/golden_state.yml
# Deploy only to a specific wizard
ansible-playbook -i inventory/hosts.yml playbooks/site.yml --limit bezalel
# Dry run (check mode)
ansible-playbook -i inventory/hosts.yml playbooks/site.yml --check --diff
```
## Golden State Provider Chain
All wizard configs converge on this provider chain. **Anthropic is BANNED.**
| Priority | Provider | Model | Endpoint |
| -------- | -------------------- | ---------------- | --------------------------------- |
| 1 | Kimi | kimi-k2.5 | https://api.kimi.com/coding/v1 |
| 2 | Gemini (OpenRouter) | gemini-2.5-pro | https://openrouter.ai/api/v1 |
| 3 | Ollama (local) | gemma4:latest | http://localhost:11434/v1 |
## Roles
| Role | Purpose |
| ---------------- | ------------------------------------------------------------ |
| `wizard_base` | Common wizard setup: directories, thin config, git pull |
| `deadman_switch` | Health check → snapshot good config → rollback on death |
| `golden_state` | Deploy and enforce golden state provider chain |
| `request_log` | SQLite telemetry table for every inference call |
| `cron_manager` | Source-controlled cron jobs — no manual crontab edits |
## Rules
1. **No manual changes.** If it's not in a playbook, it will be overwritten.
2. **No Anthropic.** Banned. Enforcement is automated. See `BANNED_PROVIDERS.yml`.
3. **Idempotent.** Every playbook can run 100 times with the same result.
4. **PR required.** Config changes go through Gitea PR review, then deploy.
5. **One identity per machine.** No duplicate agents. Fleet audit enforces this.
## Related Issues
- timmy-config #442: [P2] Ansible IaC Canonical Playbook
- timmy-config #444: Wire Deadman Switch ACTION
- timmy-config #443: Thin Config Pattern
- timmy-config #446: request_log Telemetry Table

21
ansible/ansible.cfg Normal file
View File

@@ -0,0 +1,21 @@
[defaults]
inventory = inventory/hosts.yml
roles_path = roles
host_key_checking = False
retry_files_enabled = False
stdout_callback = yaml
forks = 10
timeout = 30
# Logging
log_path = /var/log/ansible/timmy-fleet.log
[privilege_escalation]
become = True
become_method = sudo
become_user = root
become_ask_pass = False
[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no

View File

@@ -0,0 +1,74 @@
# =============================================================================
# Wizard Group Variables — Golden State Configuration
# =============================================================================
# These variables are applied to ALL wizards in the fleet.
# This IS the golden state. If a wizard deviates, Ansible corrects it.
# =============================================================================
# --- Deadman Switch ---
deadman_enabled: true
deadman_check_interval: 300 # 5 minutes between health checks
deadman_snapshot_dir: "~/.local/timmy/snapshots"
deadman_max_snapshots: 10 # Rolling window of good configs
deadman_restart_cooldown: 60 # Seconds to wait before restart after failure
deadman_max_restart_attempts: 3
deadman_escalation_channel: telegram # Alert Alexander after max attempts
# --- Thin Config ---
thin_config_path: "~/.timmy/thin_config.yml"
thin_config_mode: "0444" # Read-only — agents CANNOT modify
upstream_repo: "https://forge.alexanderwhitestone.com/Timmy_Foundation/timmy-config.git"
upstream_branch: main
config_pull_on_wake: true
config_validation_enabled: true
# --- Agent Settings ---
agent_max_turns: 30
agent_reasoning_effort: high
agent_verbose: false
agent_approval_mode: auto
# --- Hermes Harness ---
hermes_config_dir: "{{ hermes_home }}"
hermes_bin_dir: "{{ hermes_home }}/bin"
hermes_skins_dir: "{{ hermes_home }}/skins"
hermes_playbooks_dir: "{{ hermes_home }}/playbooks"
hermes_memories_dir: "{{ hermes_home }}/memories"
# --- Request Log (Telemetry) ---
request_log_enabled: true
request_log_path: "~/.local/timmy/request_log.db"
request_log_rotation_days: 30 # Archive logs older than 30 days
request_log_sync_to_gitea: false # Future: push telemetry summaries to Gitea
# --- Cron Schedule ---
# All cron jobs are managed here. No manual crontab edits.
cron_jobs:
- name: "Deadman health check"
job: "cd {{ wizard_home }}/workspace/timmy-config && python3 fleet/health_check.py"
minute: "*/5"
hour: "*"
enabled: "{{ deadman_enabled }}"
- name: "Muda audit"
job: "cd {{ wizard_home }}/workspace/timmy-config && bash fleet/muda-audit.sh >> /tmp/muda-audit.log 2>&1"
minute: "0"
hour: "21"
weekday: "0"
enabled: true
- name: "Config pull from upstream"
job: "cd {{ wizard_home }}/workspace/timmy-config && git pull --ff-only origin main"
minute: "*/15"
hour: "*"
enabled: "{{ config_pull_on_wake }}"
- name: "Request log rotation"
job: "python3 -c \"import sqlite3,datetime; db=sqlite3.connect('{{ request_log_path }}'); db.execute('DELETE FROM request_log WHERE timestamp < datetime(\\\"now\\\", \\\"-{{ request_log_rotation_days }} days\\\")'); db.commit()\""
minute: "0"
hour: "3"
enabled: "{{ request_log_enabled }}"
# --- Provider Enforcement ---
# These are validated on every Ansible run. Any Anthropic reference = failure.
provider_ban_enforcement: strict # strict = fail playbook, warn = log only

119
ansible/inventory/hosts.yml Normal file
View File

@@ -0,0 +1,119 @@
# =============================================================================
# Fleet Inventory — The Timmy Foundation
# =============================================================================
# Source of truth for all machines in the fleet.
# Update this file when machines are added/removed.
# All changes go through PR review.
# =============================================================================
all:
children:
wizards:
hosts:
timmy:
ansible_host: localhost
ansible_connection: local
wizard_name: Timmy
wizard_role: "Primary wizard — soul of the fleet"
wizard_provider_primary: kimi-coding
wizard_model_primary: kimi-k2.5
hermes_port: 8081
api_port: 8645
wizard_home: "{{ ansible_env.HOME }}/wizards/timmy"
hermes_home: "{{ ansible_env.HOME }}/.hermes"
machine_type: mac
# Timmy runs on Alexander's M3 Max
ollama_available: true
allegro:
ansible_host: 167.99.126.228
ansible_user: root
wizard_name: Allegro
wizard_role: "Kimi-backed third wizard house — tight coding tasks"
wizard_provider_primary: kimi-coding
wizard_model_primary: kimi-k2.5
hermes_port: 8081
api_port: 8645
wizard_home: /root/wizards/allegro
hermes_home: /root/.hermes
machine_type: vps
ollama_available: false
bezalel:
ansible_host: 159.203.146.185
ansible_user: root
wizard_name: Bezalel
wizard_role: "Forge-and-testbed wizard — infrastructure, deployment, hardening"
wizard_provider_primary: kimi-coding
wizard_model_primary: kimi-k2.5
hermes_port: 8081
api_port: 8656
wizard_home: /root/wizards/bezalel
hermes_home: /root/.hermes
machine_type: vps
ollama_available: false
# NOTE: The awake Bezalel may be the duplicate.
# Fleet audit (the-nexus #1144) will resolve identity.
ezra:
ansible_host: 143.198.27.163
ansible_user: root
wizard_name: Ezra
wizard_role: "Infrastructure wizard — Gitea, nginx, hosting"
wizard_provider_primary: kimi-coding
wizard_model_primary: kimi-k2.5
hermes_port: 8081
api_port: 8645
wizard_home: /root/wizards/ezra
hermes_home: /root/.hermes
machine_type: vps
ollama_available: false
# NOTE: Currently DOWN — Telegram key revoked, awaiting propagation.
# Infrastructure hosts (not wizards, but managed by Ansible)
infrastructure:
hosts:
forge:
ansible_host: 143.198.27.163
ansible_user: root
# Gitea runs on the same box as Ezra
gitea_url: https://forge.alexanderwhitestone.com
gitea_org: Timmy_Foundation
vars:
# Global variables applied to all hosts
gitea_repo_url: "https://forge.alexanderwhitestone.com/Timmy_Foundation/timmy-config.git"
gitea_branch: main
config_base_path: "{{ gitea_repo_url }}"
timmy_log_dir: "~/.local/timmy/fleet-health"
request_log_db: "~/.local/timmy/request_log.db"
# Golden state provider chain — Anthropic is BANNED
golden_state_providers:
- name: kimi-coding
model: kimi-k2.5
base_url: "https://api.kimi.com/coding/v1"
timeout: 120
reason: "Primary — Kimi K2.5 (best value, least friction)"
- name: openrouter
model: google/gemini-2.5-pro
base_url: "https://openrouter.ai/api/v1"
api_key_env: OPENROUTER_API_KEY
timeout: 120
reason: "Fallback — Gemini 2.5 Pro via OpenRouter"
- name: ollama
model: "gemma4:latest"
base_url: "http://localhost:11434/v1"
timeout: 180
reason: "Terminal fallback — local Ollama (sovereign, no API needed)"
# Banned providers — hard enforcement
banned_providers:
- anthropic
- claude
banned_models_patterns:
- "claude-*"
- "anthropic/*"
- "*sonnet*"
- "*opus*"
- "*haiku*"

View File

@@ -0,0 +1,98 @@
---
# =============================================================================
# agent_startup.yml — Resurrect Wizards from Checked-in Configs
# =============================================================================
# Brings wizards back online using golden state configs.
# Order: pull config → validate → start agent → verify with request_log
# =============================================================================
- name: "Agent Startup Sequence"
hosts: wizards
become: true
serial: 1 # One wizard at a time to avoid cascading issues
tasks:
- name: "Pull latest config from upstream"
git:
repo: "{{ upstream_repo }}"
dest: "{{ wizard_home }}/workspace/timmy-config"
version: "{{ upstream_branch }}"
force: true
tags: [pull]
- name: "Deploy golden state config"
include_role:
name: golden_state
tags: [config]
- name: "Validate config — no banned providers"
shell: |
python3 -c "
import yaml, sys
with open('{{ wizard_home }}/config.yaml') as f:
cfg = yaml.safe_load(f)
banned = {{ banned_providers }}
for p in cfg.get('fallback_providers', []):
if p.get('provider', '') in banned:
print(f'BANNED: {p[\"provider\"]}', file=sys.stderr)
sys.exit(1)
model = cfg.get('model', {}).get('provider', '')
if model in banned:
print(f'BANNED default provider: {model}', file=sys.stderr)
sys.exit(1)
print('Config validated — no banned providers.')
"
register: config_valid
tags: [validate]
- name: "Ensure hermes-agent service is running"
systemd:
name: "hermes-{{ wizard_name | lower }}"
state: started
enabled: true
when: machine_type == 'vps'
tags: [start]
ignore_errors: true # Service may not exist yet on all machines
- name: "Start hermes agent (Mac — launchctl)"
shell: |
launchctl kickstart -k "ai.hermes.{{ wizard_name | lower }}" 2>/dev/null || \
cd {{ wizard_home }} && hermes agent start --daemon 2>&1 | tail -5
when: machine_type == 'mac'
tags: [start]
ignore_errors: true
- name: "Wait for agent to come online"
wait_for:
host: 127.0.0.1
port: "{{ api_port }}"
timeout: 60
state: started
tags: [verify]
ignore_errors: true
- name: "Verify agent is alive — check request_log for activity"
shell: |
sleep 10
python3 -c "
import sqlite3, sys
db = sqlite3.connect('{{ request_log_path }}')
cursor = db.execute('''
SELECT COUNT(*) FROM request_log
WHERE agent_name = '{{ wizard_name }}'
AND timestamp > datetime('now', '-5 minutes')
''')
count = cursor.fetchone()[0]
if count > 0:
print(f'{{ wizard_name }} is alive — {count} recent inference calls logged.')
else:
print(f'WARNING: {{ wizard_name }} started but no telemetry yet.')
"
register: agent_status
tags: [verify]
ignore_errors: true
- name: "Report startup status"
debug:
msg: "{{ wizard_name }}: {{ agent_status.stdout | default('startup attempted') }}"
tags: [always]

View File

@@ -0,0 +1,15 @@
---
# =============================================================================
# cron_schedule.yml — Source-Controlled Cron Jobs
# =============================================================================
# All cron jobs are defined in group_vars/wizards.yml.
# This playbook deploys them. No manual crontab edits allowed.
# =============================================================================
- name: "Deploy Cron Schedule"
hosts: wizards
become: true
roles:
- role: cron_manager
tags: [cron, schedule]

View File

@@ -0,0 +1,17 @@
---
# =============================================================================
# deadman_switch.yml — Deploy Deadman Switch to All Wizards
# =============================================================================
# The deadman watch already fires and detects dead agents.
# This playbook wires the ACTION:
# - On healthy check: snapshot current config as "last known good"
# - On failed check: rollback config to snapshot, restart agent
# =============================================================================
- name: "Deploy Deadman Switch ACTION"
hosts: wizards
become: true
roles:
- role: deadman_switch
tags: [deadman, recovery]

View File

@@ -0,0 +1,30 @@
---
# =============================================================================
# golden_state.yml — Deploy Golden State Config to All Wizards
# =============================================================================
# Enforces the golden state provider chain across the fleet.
# Removes any Anthropic references. Deploys the approved provider chain.
# =============================================================================
- name: "Deploy Golden State Configuration"
hosts: wizards
become: true
roles:
- role: golden_state
tags: [golden, config]
post_tasks:
- name: "Verify golden state — no banned providers"
shell: |
grep -rci 'anthropic\|claude-sonnet\|claude-opus\|claude-haiku' \
{{ hermes_home }}/config.yaml \
{{ wizard_home }}/config.yaml 2>/dev/null || echo "0"
register: banned_count
changed_when: false
- name: "Report golden state status"
debug:
msg: >
{{ wizard_name }} golden state: {{ golden_state_providers | map(attribute='name') | list | join(' → ') }}.
Banned provider references: {{ banned_count.stdout | trim }}.

View File

@@ -0,0 +1,15 @@
---
# =============================================================================
# request_log.yml — Deploy Telemetry Table
# =============================================================================
# Creates the request_log SQLite table on all machines.
# Every inference call writes a row. No exceptions. No summarizing.
# =============================================================================
- name: "Deploy Request Log Telemetry"
hosts: wizards
become: true
roles:
- role: request_log
tags: [telemetry, logging]

View File

@@ -0,0 +1,72 @@
---
# =============================================================================
# site.yml — Master Playbook for the Timmy Foundation Fleet
# =============================================================================
# This is the ONE playbook that defines the entire fleet state.
# Run this and every machine converges to golden state.
#
# Usage:
# ansible-playbook -i inventory/hosts.yml playbooks/site.yml
# ansible-playbook -i inventory/hosts.yml playbooks/site.yml --limit bezalel
# ansible-playbook -i inventory/hosts.yml playbooks/site.yml --check --diff
# =============================================================================
- name: "Timmy Foundation Fleet — Full Convergence"
hosts: wizards
become: true
pre_tasks:
- name: "Validate no banned providers in golden state"
assert:
that:
- "item.name not in banned_providers"
fail_msg: "BANNED PROVIDER DETECTED: {{ item.name }} — Anthropic is permanently banned."
quiet: true
loop: "{{ golden_state_providers }}"
tags: [always]
- name: "Display target wizard"
debug:
msg: "Deploying to {{ wizard_name }} ({{ wizard_role }}) on {{ ansible_host }}"
tags: [always]
roles:
- role: wizard_base
tags: [base, setup]
- role: golden_state
tags: [golden, config]
- role: deadman_switch
tags: [deadman, recovery]
- role: request_log
tags: [telemetry, logging]
- role: cron_manager
tags: [cron, schedule]
post_tasks:
- name: "Final validation — scan for banned providers"
shell: |
grep -ri 'anthropic\|claude-sonnet\|claude-opus\|claude-haiku' \
{{ hermes_home }}/config.yaml \
{{ wizard_home }}/config.yaml \
{{ thin_config_path }} 2>/dev/null || true
register: banned_scan
changed_when: false
tags: [validation]
- name: "FAIL if banned providers found in deployed config"
fail:
msg: |
BANNED PROVIDER DETECTED IN DEPLOYED CONFIG:
{{ banned_scan.stdout }}
Anthropic is permanently banned. Fix the config and re-deploy.
when: banned_scan.stdout | length > 0
tags: [validation]
- name: "Deployment complete"
debug:
msg: "{{ wizard_name }} converged to golden state. Provider chain: {{ golden_state_providers | map(attribute='name') | list | join(' → ') }}"
tags: [always]

View File

@@ -0,0 +1,55 @@
---
# =============================================================================
# cron_manager/tasks — Source-Controlled Cron Jobs
# =============================================================================
# All cron jobs are defined in group_vars/wizards.yml.
# No manual crontab edits. This is the only way to manage cron.
# =============================================================================
- name: "Deploy managed cron jobs"
cron:
name: "{{ item.name }}"
job: "{{ item.job }}"
minute: "{{ item.minute | default('*') }}"
hour: "{{ item.hour | default('*') }}"
day: "{{ item.day | default('*') }}"
month: "{{ item.month | default('*') }}"
weekday: "{{ item.weekday | default('*') }}"
state: "{{ 'present' if item.enabled else 'absent' }}"
user: "{{ ansible_user | default('root') }}"
loop: "{{ cron_jobs }}"
when: cron_jobs is defined
- name: "Deploy deadman switch cron (fallback if systemd timer unavailable)"
cron:
name: "Deadman switch — {{ wizard_name }}"
job: "{{ wizard_home }}/deadman_action.sh >> {{ timmy_log_dir }}/deadman-{{ wizard_name }}.log 2>&1"
minute: "*/5"
hour: "*"
state: present
user: "{{ ansible_user | default('root') }}"
when: deadman_enabled and machine_type != 'vps'
# VPS machines use systemd timers instead
- name: "Remove legacy cron jobs (cleanup)"
cron:
name: "{{ item }}"
state: absent
user: "{{ ansible_user | default('root') }}"
loop:
- "legacy-deadman-watch"
- "old-health-check"
- "backup-deadman"
ignore_errors: true
- name: "List active cron jobs"
shell: "crontab -l 2>/dev/null | grep -v '^#' | grep -v '^$' || echo 'No cron jobs found.'"
register: active_crons
changed_when: false
- name: "Report cron status"
debug:
msg: |
{{ wizard_name }} cron jobs deployed.
Active:
{{ active_crons.stdout }}

View File

@@ -0,0 +1,70 @@
---
# =============================================================================
# deadman_switch/tasks — Wire the Deadman Switch ACTION
# =============================================================================
# The watch fires. This makes it DO something:
# - On healthy check: snapshot current config as "last known good"
# - On failed check: rollback to last known good, restart agent
# =============================================================================
- name: "Create snapshot directory"
file:
path: "{{ deadman_snapshot_dir }}"
state: directory
mode: "0755"
- name: "Deploy deadman switch script"
template:
src: deadman_action.sh.j2
dest: "{{ wizard_home }}/deadman_action.sh"
mode: "0755"
- name: "Deploy deadman systemd service"
template:
src: deadman_switch.service.j2
dest: "/etc/systemd/system/deadman-{{ wizard_name | lower }}.service"
mode: "0644"
when: machine_type == 'vps'
notify: "Enable deadman service"
- name: "Deploy deadman systemd timer"
template:
src: deadman_switch.timer.j2
dest: "/etc/systemd/system/deadman-{{ wizard_name | lower }}.timer"
mode: "0644"
when: machine_type == 'vps'
notify: "Enable deadman timer"
- name: "Deploy deadman launchd plist (Mac)"
template:
src: deadman_switch.plist.j2
dest: "{{ ansible_env.HOME }}/Library/LaunchAgents/com.timmy.deadman.{{ wizard_name | lower }}.plist"
mode: "0644"
when: machine_type == 'mac'
notify: "Load deadman plist"
- name: "Take initial config snapshot"
copy:
src: "{{ wizard_home }}/config.yaml"
dest: "{{ deadman_snapshot_dir }}/config.yaml.known_good"
remote_src: true
mode: "0444"
ignore_errors: true
handlers:
- name: "Enable deadman service"
systemd:
name: "deadman-{{ wizard_name | lower }}.service"
daemon_reload: true
enabled: true
- name: "Enable deadman timer"
systemd:
name: "deadman-{{ wizard_name | lower }}.timer"
daemon_reload: true
enabled: true
state: started
- name: "Load deadman plist"
shell: "launchctl load {{ ansible_env.HOME }}/Library/LaunchAgents/com.timmy.deadman.{{ wizard_name | lower }}.plist"
ignore_errors: true

View File

@@ -0,0 +1,153 @@
#!/usr/bin/env bash
# =============================================================================
# Deadman Switch ACTION — {{ wizard_name }}
# =============================================================================
# Generated by Ansible on {{ ansible_date_time.iso8601 }}
# DO NOT EDIT MANUALLY.
#
# On healthy check: snapshot current config as "last known good"
# On failed check: rollback config to last known good, restart agent
# =============================================================================
set -euo pipefail
WIZARD_NAME="{{ wizard_name }}"
WIZARD_HOME="{{ wizard_home }}"
CONFIG_FILE="{{ wizard_home }}/config.yaml"
SNAPSHOT_DIR="{{ deadman_snapshot_dir }}"
SNAPSHOT_FILE="${SNAPSHOT_DIR}/config.yaml.known_good"
REQUEST_LOG_DB="{{ request_log_path }}"
LOG_DIR="{{ timmy_log_dir }}"
LOG_FILE="${LOG_DIR}/deadman-${WIZARD_NAME}.log"
MAX_SNAPSHOTS={{ deadman_max_snapshots }}
RESTART_COOLDOWN={{ deadman_restart_cooldown }}
MAX_RESTART_ATTEMPTS={{ deadman_max_restart_attempts }}
COOLDOWN_FILE="${LOG_DIR}/deadman_cooldown_${WIZARD_NAME}"
SERVICE_NAME="hermes-{{ wizard_name | lower }}"
# Ensure directories exist
mkdir -p "${SNAPSHOT_DIR}" "${LOG_DIR}"
log() {
echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] [deadman] [${WIZARD_NAME}] $*" >> "${LOG_FILE}"
echo "[deadman] [${WIZARD_NAME}] $*"
}
log_telemetry() {
local status="$1"
local message="$2"
if [ -f "${REQUEST_LOG_DB}" ]; then
sqlite3 "${REQUEST_LOG_DB}" "INSERT INTO request_log (timestamp, agent_name, provider, model, endpoint, status, error_message) VALUES (datetime('now'), '${WIZARD_NAME}', 'deadman_switch', 'N/A', 'health_check', '${status}', '${message}');" 2>/dev/null || true
fi
}
snapshot_config() {
if [ -f "${CONFIG_FILE}" ]; then
cp "${CONFIG_FILE}" "${SNAPSHOT_FILE}"
# Keep rolling history
cp "${CONFIG_FILE}" "${SNAPSHOT_DIR}/config.yaml.$(date +%s)"
# Prune old snapshots
ls -t "${SNAPSHOT_DIR}"/config.yaml.[0-9]* 2>/dev/null | tail -n +$((MAX_SNAPSHOTS + 1)) | xargs rm -f 2>/dev/null
log "Config snapshot saved."
fi
}
rollback_config() {
if [ -f "${SNAPSHOT_FILE}" ]; then
log "Rolling back config to last known good..."
cp "${SNAPSHOT_FILE}" "${CONFIG_FILE}"
log "Config rolled back."
log_telemetry "fallback" "Config rolled back to last known good by deadman switch"
else
log "ERROR: No known good snapshot found. Pulling from upstream..."
cd "${WIZARD_HOME}/workspace/timmy-config" 2>/dev/null && \
git pull --ff-only origin {{ upstream_branch }} 2>/dev/null && \
cp "wizards/{{ wizard_name | lower }}/config.yaml" "${CONFIG_FILE}" && \
log "Config restored from upstream." || \
log "CRITICAL: Cannot restore config from any source."
fi
}
restart_agent() {
# Check cooldown
if [ -f "${COOLDOWN_FILE}" ]; then
local last_restart
last_restart=$(cat "${COOLDOWN_FILE}")
local now
now=$(date +%s)
local elapsed=$((now - last_restart))
if [ "${elapsed}" -lt "${RESTART_COOLDOWN}" ]; then
log "Restart cooldown active (${elapsed}s / ${RESTART_COOLDOWN}s). Skipping."
return 1
fi
fi
log "Restarting ${SERVICE_NAME}..."
date +%s > "${COOLDOWN_FILE}"
{% if machine_type == 'vps' %}
systemctl restart "${SERVICE_NAME}" 2>/dev/null && \
log "Agent restarted via systemd." || \
log "ERROR: systemd restart failed."
{% else %}
launchctl kickstart -k "ai.hermes.{{ wizard_name | lower }}" 2>/dev/null && \
log "Agent restarted via launchctl." || \
(cd "${WIZARD_HOME}" && hermes agent start --daemon 2>/dev/null && \
log "Agent restarted via hermes CLI.") || \
log "ERROR: All restart methods failed."
{% endif %}
log_telemetry "success" "Agent restarted by deadman switch"
}
# --- Health Check ---
check_health() {
# Check 1: Is the agent process running?
{% if machine_type == 'vps' %}
if ! systemctl is-active --quiet "${SERVICE_NAME}" 2>/dev/null; then
if ! pgrep -f "hermes" > /dev/null 2>/dev/null; then
log "FAIL: Agent process not running."
return 1
fi
fi
{% else %}
if ! pgrep -f "hermes" > /dev/null 2>/dev/null; then
log "FAIL: Agent process not running."
return 1
fi
{% endif %}
# Check 2: Is the API port responding?
if ! timeout 10 bash -c "echo > /dev/tcp/127.0.0.1/{{ api_port }}" 2>/dev/null; then
log "FAIL: API port {{ api_port }} not responding."
return 1
fi
# Check 3: Does the config contain banned providers?
if grep -qi 'anthropic\|claude-sonnet\|claude-opus\|claude-haiku' "${CONFIG_FILE}" 2>/dev/null; then
log "FAIL: Config contains banned provider (Anthropic). Rolling back."
return 1
fi
return 0
}
# --- Main ---
main() {
log "Health check starting..."
if check_health; then
log "HEALTHY — snapshotting config."
snapshot_config
log_telemetry "success" "Health check passed"
else
log "UNHEALTHY — initiating recovery."
log_telemetry "error" "Health check failed — initiating rollback"
rollback_config
restart_agent
fi
log "Health check complete."
}
main "$@"

View File

@@ -0,0 +1,22 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<!-- Deadman Switch — {{ wizard_name }}. Generated by Ansible. DO NOT EDIT MANUALLY. -->
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.timmy.deadman.{{ wizard_name | lower }}</string>
<key>ProgramArguments</key>
<array>
<string>/bin/bash</string>
<string>{{ wizard_home }}/deadman_action.sh</string>
</array>
<key>StartInterval</key>
<integer>{{ deadman_check_interval }}</integer>
<key>RunAtLoad</key>
<true/>
<key>StandardOutPath</key>
<string>{{ timmy_log_dir }}/deadman-{{ wizard_name }}.log</string>
<key>StandardErrorPath</key>
<string>{{ timmy_log_dir }}/deadman-{{ wizard_name }}.log</string>
</dict>
</plist>

View File

@@ -0,0 +1,16 @@
# Deadman Switch — {{ wizard_name }}
# Generated by Ansible. DO NOT EDIT MANUALLY.
[Unit]
Description=Deadman Switch for {{ wizard_name }} wizard
After=network.target
[Service]
Type=oneshot
ExecStart={{ wizard_home }}/deadman_action.sh
User={{ ansible_user | default('root') }}
StandardOutput=append:{{ timmy_log_dir }}/deadman-{{ wizard_name }}.log
StandardError=append:{{ timmy_log_dir }}/deadman-{{ wizard_name }}.log
[Install]
WantedBy=multi-user.target

View File

@@ -0,0 +1,14 @@
# Deadman Switch Timer — {{ wizard_name }}
# Generated by Ansible. DO NOT EDIT MANUALLY.
# Runs every {{ deadman_check_interval // 60 }} minutes.
[Unit]
Description=Deadman Switch Timer for {{ wizard_name }} wizard
[Timer]
OnBootSec=60
OnUnitActiveSec={{ deadman_check_interval }}s
AccuracySec=30s
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,6 @@
---
# golden_state defaults
# The golden_state_providers list is defined in group_vars/wizards.yml
# and inventory/hosts.yml (global vars).
golden_state_enforce: true
golden_state_backup_before_deploy: true

View File

@@ -0,0 +1,46 @@
---
# =============================================================================
# golden_state/tasks — Deploy and enforce golden state provider chain
# =============================================================================
- name: "Backup current config before golden state deploy"
copy:
src: "{{ wizard_home }}/config.yaml"
dest: "{{ wizard_home }}/config.yaml.pre-golden-{{ ansible_date_time.epoch }}"
remote_src: true
when: golden_state_backup_before_deploy
ignore_errors: true
- name: "Deploy golden state wizard config"
template:
src: "../../wizard_base/templates/wizard_config.yaml.j2"
dest: "{{ wizard_home }}/config.yaml"
mode: "0644"
backup: true
notify:
- "Restart hermes agent (systemd)"
- "Restart hermes agent (launchctl)"
- name: "Scan for banned providers in all config files"
shell: |
FOUND=0
for f in {{ wizard_home }}/config.yaml {{ hermes_home }}/config.yaml; do
if [ -f "$f" ]; then
if grep -qi 'anthropic\|claude-sonnet\|claude-opus\|claude-haiku' "$f"; then
echo "BANNED PROVIDER in $f:"
grep -ni 'anthropic\|claude-sonnet\|claude-opus\|claude-haiku' "$f"
FOUND=1
fi
fi
done
exit $FOUND
register: provider_scan
changed_when: false
failed_when: provider_scan.rc != 0 and provider_ban_enforcement == 'strict'
- name: "Report golden state deployment"
debug:
msg: >
{{ wizard_name }} golden state deployed.
Provider chain: {{ golden_state_providers | map(attribute='name') | list | join(' → ') }}.
Banned provider scan: {{ 'CLEAN' if provider_scan.rc == 0 else 'VIOLATIONS FOUND' }}.

View File

@@ -0,0 +1,64 @@
-- =============================================================================
-- request_log — Inference Telemetry Table
-- =============================================================================
-- Every agent writes to this table BEFORE and AFTER every inference call.
-- No exceptions. No summarizing. No describing what you would log.
-- Actually write the row.
--
-- Source: KT Bezalel Architecture Session 2026-04-08
-- =============================================================================
CREATE TABLE IF NOT EXISTS request_log (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp TEXT NOT NULL DEFAULT (datetime('now')),
agent_name TEXT NOT NULL,
provider TEXT NOT NULL,
model TEXT NOT NULL,
endpoint TEXT NOT NULL,
tokens_in INTEGER,
tokens_out INTEGER,
latency_ms INTEGER,
status TEXT NOT NULL, -- 'success', 'error', 'timeout', 'fallback'
error_message TEXT
);
-- Index for common queries
CREATE INDEX IF NOT EXISTS idx_request_log_agent
ON request_log (agent_name, timestamp);
CREATE INDEX IF NOT EXISTS idx_request_log_provider
ON request_log (provider, timestamp);
CREATE INDEX IF NOT EXISTS idx_request_log_status
ON request_log (status, timestamp);
-- View: recent activity per agent (last hour)
CREATE VIEW IF NOT EXISTS v_recent_activity AS
SELECT
agent_name,
provider,
model,
status,
COUNT(*) as call_count,
AVG(latency_ms) as avg_latency_ms,
SUM(tokens_in) as total_tokens_in,
SUM(tokens_out) as total_tokens_out
FROM request_log
WHERE timestamp > datetime('now', '-1 hour')
GROUP BY agent_name, provider, model, status;
-- View: provider reliability (last 24 hours)
CREATE VIEW IF NOT EXISTS v_provider_reliability AS
SELECT
provider,
model,
COUNT(*) as total_calls,
SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) as successes,
SUM(CASE WHEN status = 'error' THEN 1 ELSE 0 END) as errors,
SUM(CASE WHEN status = 'timeout' THEN 1 ELSE 0 END) as timeouts,
SUM(CASE WHEN status = 'fallback' THEN 1 ELSE 0 END) as fallbacks,
ROUND(100.0 * SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) / COUNT(*), 1) as success_rate,
AVG(latency_ms) as avg_latency_ms
FROM request_log
WHERE timestamp > datetime('now', '-24 hours')
GROUP BY provider, model;

View File

@@ -0,0 +1,50 @@
---
# =============================================================================
# request_log/tasks — Deploy Telemetry Table
# =============================================================================
# "This is non-negotiable infrastructure. Without it, we cannot verify
# if any agent actually executed what it claims."
# — KT Bezalel 2026-04-08
# =============================================================================
- name: "Create telemetry directory"
file:
path: "{{ request_log_path | dirname }}"
state: directory
mode: "0755"
- name: "Deploy request_log schema"
copy:
src: request_log_schema.sql
dest: "{{ wizard_home }}/request_log_schema.sql"
mode: "0644"
- name: "Initialize request_log database"
shell: |
sqlite3 "{{ request_log_path }}" < "{{ wizard_home }}/request_log_schema.sql"
args:
creates: "{{ request_log_path }}"
- name: "Verify request_log table exists"
shell: |
sqlite3 "{{ request_log_path }}" ".tables" | grep -q "request_log"
register: table_check
changed_when: false
- name: "Verify request_log schema matches"
shell: |
sqlite3 "{{ request_log_path }}" ".schema request_log" | grep -q "agent_name"
register: schema_check
changed_when: false
- name: "Set permissions on request_log database"
file:
path: "{{ request_log_path }}"
mode: "0644"
- name: "Report request_log status"
debug:
msg: >
{{ wizard_name }} request_log: {{ request_log_path }}
— table exists: {{ table_check.rc == 0 }}
— schema valid: {{ schema_check.rc == 0 }}

View File

@@ -0,0 +1,6 @@
---
# wizard_base defaults
wizard_user: "{{ ansible_user | default('root') }}"
wizard_group: "{{ ansible_user | default('root') }}"
timmy_base_dir: "~/.local/timmy"
timmy_config_repo: "https://forge.alexanderwhitestone.com/Timmy_Foundation/timmy-config.git"

View File

@@ -0,0 +1,11 @@
---
- name: "Restart hermes agent (systemd)"
systemd:
name: "hermes-{{ wizard_name | lower }}"
state: restarted
when: machine_type == 'vps'
- name: "Restart hermes agent (launchctl)"
shell: "launchctl kickstart -k ai.hermes.{{ wizard_name | lower }}"
when: machine_type == 'mac'
ignore_errors: true

View File

@@ -0,0 +1,69 @@
---
# =============================================================================
# wizard_base/tasks — Common wizard setup
# =============================================================================
- name: "Create wizard directories"
file:
path: "{{ item }}"
state: directory
mode: "0755"
loop:
- "{{ wizard_home }}"
- "{{ wizard_home }}/workspace"
- "{{ hermes_home }}"
- "{{ hermes_home }}/bin"
- "{{ hermes_home }}/skins"
- "{{ hermes_home }}/playbooks"
- "{{ hermes_home }}/memories"
- "~/.local/timmy"
- "~/.local/timmy/fleet-health"
- "~/.local/timmy/snapshots"
- "~/.timmy"
- name: "Clone/update timmy-config"
git:
repo: "{{ upstream_repo }}"
dest: "{{ wizard_home }}/workspace/timmy-config"
version: "{{ upstream_branch }}"
force: false
update: true
ignore_errors: true # May fail on first run if no SSH key
- name: "Deploy SOUL.md"
copy:
src: "{{ wizard_home }}/workspace/timmy-config/SOUL.md"
dest: "~/.timmy/SOUL.md"
remote_src: true
mode: "0644"
ignore_errors: true
- name: "Deploy thin config (immutable pointer to upstream)"
template:
src: thin_config.yml.j2
dest: "{{ thin_config_path }}"
mode: "{{ thin_config_mode }}"
tags: [thin_config]
- name: "Ensure Python3 and pip are available"
package:
name:
- python3
- python3-pip
state: present
when: machine_type == 'vps'
ignore_errors: true
- name: "Ensure PyYAML is installed (for config validation)"
pip:
name: pyyaml
state: present
when: machine_type == 'vps'
ignore_errors: true
- name: "Create Ansible log directory"
file:
path: /var/log/ansible
state: directory
mode: "0755"
ignore_errors: true

View File

@@ -0,0 +1,41 @@
# =============================================================================
# Thin Config — {{ wizard_name }}
# =============================================================================
# THIS FILE IS READ-ONLY. Agents CANNOT modify it.
# It contains only pointers to upstream. The actual config lives in Gitea.
#
# Agent wakes up → pulls config from upstream → loads → runs.
# If anything tries to mutate this → fails gracefully → pulls fresh on restart.
#
# Only way to permanently change config: commit to Gitea, merge PR, Ansible deploys.
#
# Generated by Ansible on {{ ansible_date_time.iso8601 }}
# DO NOT EDIT MANUALLY.
# =============================================================================
identity:
wizard_name: "{{ wizard_name }}"
wizard_role: "{{ wizard_role }}"
machine: "{{ inventory_hostname }}"
upstream:
repo: "{{ upstream_repo }}"
branch: "{{ upstream_branch }}"
config_path: "wizards/{{ wizard_name | lower }}/config.yaml"
pull_on_wake: {{ config_pull_on_wake | lower }}
recovery:
deadman_enabled: {{ deadman_enabled | lower }}
snapshot_dir: "{{ deadman_snapshot_dir }}"
restart_cooldown: {{ deadman_restart_cooldown }}
max_restart_attempts: {{ deadman_max_restart_attempts }}
escalation_channel: "{{ deadman_escalation_channel }}"
telemetry:
request_log_path: "{{ request_log_path }}"
request_log_enabled: {{ request_log_enabled | lower }}
local_overrides:
# Runtime overrides go here. They are EPHEMERAL — not persisted across restarts.
# On restart, this section is reset to empty.
{}

View File

@@ -0,0 +1,115 @@
# =============================================================================
# {{ wizard_name }} — Wizard Configuration (Golden State)
# =============================================================================
# Generated by Ansible on {{ ansible_date_time.iso8601 }}
# DO NOT EDIT MANUALLY. Changes go through Gitea PR → Ansible deploy.
#
# Provider chain: {{ golden_state_providers | map(attribute='name') | list | join(' → ') }}
# Anthropic is PERMANENTLY BANNED.
# =============================================================================
model:
default: {{ wizard_model_primary }}
provider: {{ wizard_provider_primary }}
context_length: 65536
base_url: {{ golden_state_providers[0].base_url }}
toolsets:
- all
fallback_providers:
{% for provider in golden_state_providers %}
- provider: {{ provider.name }}
model: {{ provider.model }}
{% if provider.base_url is defined %}
base_url: {{ provider.base_url }}
{% endif %}
{% if provider.api_key_env is defined %}
api_key_env: {{ provider.api_key_env }}
{% endif %}
timeout: {{ provider.timeout }}
reason: "{{ provider.reason }}"
{% endfor %}
agent:
max_turns: {{ agent_max_turns }}
reasoning_effort: {{ agent_reasoning_effort }}
verbose: {{ agent_verbose | lower }}
terminal:
backend: local
cwd: .
timeout: 180
persistent_shell: true
browser:
inactivity_timeout: 120
command_timeout: 30
record_sessions: false
display:
compact: false
personality: ''
resume_display: full
busy_input_mode: interrupt
bell_on_complete: false
show_reasoning: false
streaming: false
show_cost: false
tool_progress: all
memory:
memory_enabled: true
user_profile_enabled: true
memory_char_limit: 2200
user_char_limit: 1375
nudge_interval: 10
flush_min_turns: 6
approvals:
mode: {{ agent_approval_mode }}
security:
redact_secrets: true
tirith_enabled: false
platforms:
api_server:
enabled: true
extra:
host: 127.0.0.1
port: {{ api_port }}
session_reset:
mode: none
idle_minutes: 0
skills:
creation_nudge_interval: 15
system_prompt_suffix: |
You are {{ wizard_name }}, {{ wizard_role }}.
Your soul is defined in SOUL.md — read it, live it.
Hermes is your harness.
{{ golden_state_providers[0].name }} is your primary provider.
Refusal over fabrication. If you do not know, say so.
Sovereignty and service always.
providers:
{% for provider in golden_state_providers %}
{{ provider.name }}:
base_url: {{ provider.base_url }}
timeout: {{ provider.timeout | default(60) }}
{% if provider.name == 'kimi-coding' %}
max_retries: 3
{% endif %}
{% endfor %}
# =============================================================================
# BANNED PROVIDERS — DO NOT ADD
# =============================================================================
# The following providers are PERMANENTLY BANNED:
# - anthropic (any model: claude-sonnet, claude-opus, claude-haiku)
# Enforcement: pre-commit hook, linter, Ansible validation, this comment.
# Adding any banned provider will cause Ansible deployment to FAIL.
# =============================================================================

View File

@@ -0,0 +1,75 @@
#!/usr/bin/env bash
# =============================================================================
# Gitea Webhook Handler — Trigger Ansible Deploy on Merge
# =============================================================================
# This script is called by the Gitea webhook when a PR is merged
# to the main branch of timmy-config.
#
# Setup:
# 1. Add webhook in Gitea: Settings → Webhooks → Add Webhook
# 2. URL: http://localhost:9000/hooks/deploy-timmy-config
# 3. Events: Pull Request (merged only)
# 4. Secret: <configured in Gitea>
#
# This script runs ansible-pull to update the local machine.
# For fleet-wide deploys, each machine runs ansible-pull independently.
# =============================================================================
set -euo pipefail
REPO="https://forge.alexanderwhitestone.com/Timmy_Foundation/timmy-config.git"
BRANCH="main"
ANSIBLE_DIR="ansible"
LOG_FILE="/var/log/ansible/webhook-deploy.log"
LOCK_FILE="/tmp/ansible-deploy.lock"
log() {
echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] [webhook] $*" | tee -a "${LOG_FILE}"
}
# Prevent concurrent deploys
if [ -f "${LOCK_FILE}" ]; then
LOCK_AGE=$(( $(date +%s) - $(stat -c %Y "${LOCK_FILE}" 2>/dev/null || echo 0) ))
if [ "${LOCK_AGE}" -lt 300 ]; then
log "Deploy already in progress (lock age: ${LOCK_AGE}s). Skipping."
exit 0
else
log "Stale lock file (${LOCK_AGE}s old). Removing."
rm -f "${LOCK_FILE}"
fi
fi
trap 'rm -f "${LOCK_FILE}"' EXIT
touch "${LOCK_FILE}"
log "Webhook triggered. Starting ansible-pull..."
# Pull latest config
cd /tmp
rm -rf timmy-config-deploy
git clone --depth 1 --branch "${BRANCH}" "${REPO}" timmy-config-deploy 2>&1 | tee -a "${LOG_FILE}"
cd timmy-config-deploy/${ANSIBLE_DIR}
# Run Ansible against localhost
log "Running Ansible playbook..."
ansible-playbook \
-i inventory/hosts.yml \
playbooks/site.yml \
--limit "$(hostname)" \
--diff \
2>&1 | tee -a "${LOG_FILE}"
RESULT=$?
if [ ${RESULT} -eq 0 ]; then
log "Deploy successful."
else
log "ERROR: Deploy failed with exit code ${RESULT}."
fi
# Cleanup
rm -rf /tmp/timmy-config-deploy
log "Webhook handler complete."
exit ${RESULT}

View File

@@ -0,0 +1,155 @@
#!/usr/bin/env python3
"""
Config Validator — The Timmy Foundation
Validates wizard configs against golden state rules.
Run before any config deploy to catch violations early.
Usage:
python3 validate_config.py <config_file>
python3 validate_config.py --all # Validate all wizard configs
Exit codes:
0 — All validations passed
1 — Validation errors found
2 — File not found or parse error
"""
import sys
import os
import yaml
import fnmatch
from pathlib import Path
# === BANNED PROVIDERS — HARD POLICY ===
BANNED_PROVIDERS = {"anthropic", "claude"}
BANNED_MODEL_PATTERNS = [
"claude-*",
"anthropic/*",
"*sonnet*",
"*opus*",
"*haiku*",
]
# === REQUIRED FIELDS ===
REQUIRED_FIELDS = {
"model": ["default", "provider"],
"fallback_providers": None, # Must exist as a list
}
def is_banned_model(model_name: str) -> bool:
"""Check if a model name matches any banned pattern."""
model_lower = model_name.lower()
for pattern in BANNED_MODEL_PATTERNS:
if fnmatch.fnmatch(model_lower, pattern):
return True
return False
def validate_config(config_path: str) -> list[str]:
"""Validate a wizard config file. Returns list of error strings."""
errors = []
try:
with open(config_path) as f:
cfg = yaml.safe_load(f)
except FileNotFoundError:
return [f"File not found: {config_path}"]
except yaml.YAMLError as e:
return [f"YAML parse error: {e}"]
if not cfg:
return ["Config file is empty"]
# Check required fields
for section, fields in REQUIRED_FIELDS.items():
if section not in cfg:
errors.append(f"Missing required section: {section}")
elif fields:
for field in fields:
if field not in cfg[section]:
errors.append(f"Missing required field: {section}.{field}")
# Check default provider
default_provider = cfg.get("model", {}).get("provider", "")
if default_provider.lower() in BANNED_PROVIDERS:
errors.append(f"BANNED default provider: {default_provider}")
default_model = cfg.get("model", {}).get("default", "")
if is_banned_model(default_model):
errors.append(f"BANNED default model: {default_model}")
# Check fallback providers
for i, fb in enumerate(cfg.get("fallback_providers", [])):
provider = fb.get("provider", "")
model = fb.get("model", "")
if provider.lower() in BANNED_PROVIDERS:
errors.append(f"BANNED fallback provider [{i}]: {provider}")
if is_banned_model(model):
errors.append(f"BANNED fallback model [{i}]: {model}")
# Check providers section
for name, provider_cfg in cfg.get("providers", {}).items():
if name.lower() in BANNED_PROVIDERS:
errors.append(f"BANNED provider in providers section: {name}")
base_url = str(provider_cfg.get("base_url", ""))
if "anthropic" in base_url.lower():
errors.append(f"BANNED URL in provider {name}: {base_url}")
# Check system prompt for banned references
prompt = cfg.get("system_prompt_suffix", "")
if isinstance(prompt, str):
for banned in BANNED_PROVIDERS:
if banned in prompt.lower():
errors.append(f"BANNED provider referenced in system_prompt_suffix: {banned}")
return errors
def main():
if len(sys.argv) < 2:
print(f"Usage: {sys.argv[0]} <config_file> [--all]")
sys.exit(2)
if sys.argv[1] == "--all":
# Validate all wizard configs in the repo
repo_root = Path(__file__).parent.parent.parent
wizard_dir = repo_root / "wizards"
all_errors = {}
for wizard_path in sorted(wizard_dir.iterdir()):
config_file = wizard_path / "config.yaml"
if config_file.exists():
errors = validate_config(str(config_file))
if errors:
all_errors[wizard_path.name] = errors
if all_errors:
print("VALIDATION FAILED:")
for wizard, errors in all_errors.items():
print(f"\n {wizard}:")
for err in errors:
print(f" - {err}")
sys.exit(1)
else:
print("All wizard configs passed validation.")
sys.exit(0)
else:
config_path = sys.argv[1]
errors = validate_config(config_path)
if errors:
print(f"VALIDATION FAILED for {config_path}:")
for err in errors:
print(f" - {err}")
sys.exit(1)
else:
print(f"PASSED: {config_path}")
sys.exit(0)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,212 @@
[
{
"job_id": "9e0624269ba7",
"name": "Triage Heartbeat",
"schedule": "every 15m",
"state": "paused"
},
{
"job_id": "e29eda4a8548",
"name": "PR Review Sweep",
"schedule": "every 30m",
"state": "scheduled"
},
{
"job_id": "a77a87392582",
"name": "Health Monitor",
"schedule": "every 5m",
"state": "scheduled"
},
{
"job_id": "5e9d952871bc",
"name": "Agent Status Check",
"schedule": "every 10m",
"state": "paused"
},
{
"job_id": "36fb2f630a17",
"name": "Hermes Philosophy Loop",
"schedule": "every 1440m",
"state": "paused"
},
{
"job_id": "b40a96a2f48c",
"name": "wolf-eval-cycle",
"schedule": "every 240m",
"state": "paused"
},
{
"job_id": "4204e568b862",
"name": "Burn Mode \u2014 Timmy Orchestrator",
"schedule": "every 15m",
"state": "scheduled"
},
{
"job_id": "0944a976d034",
"name": "Burn Mode",
"schedule": "every 15m",
"state": "paused"
},
{
"job_id": "62016b960fa0",
"name": "velocity-engine",
"schedule": "every 30m",
"state": "paused"
},
{
"job_id": "e9d49eeff79c",
"name": "weekly-skill-extraction",
"schedule": "every 10080m",
"state": "scheduled"
},
{
"job_id": "75c74a5bb563",
"name": "tower-tick",
"schedule": "every 1m",
"state": "scheduled"
},
{
"job_id": "390a19054d4c",
"name": "Burn Deadman",
"schedule": "every 30m",
"state": "scheduled"
},
{
"job_id": "05e3c13498fa",
"name": "Morning Report \u2014 Burn Mode",
"schedule": "0 6 * * *",
"state": "scheduled"
},
{
"job_id": "64fe44b512b9",
"name": "evennia-morning-report",
"schedule": "0 9 * * *",
"state": "scheduled"
},
{
"job_id": "3896a7fd9747",
"name": "Gitea Priority Inbox",
"schedule": "every 3m",
"state": "scheduled"
},
{
"job_id": "f64c2709270a",
"name": "Config Drift Guard",
"schedule": "every 30m",
"state": "scheduled"
},
{
"job_id": "fc6a75b7102a",
"name": "Gitea Event Watcher",
"schedule": "every 2m",
"state": "scheduled"
},
{
"job_id": "12e59648fb06",
"name": "Burndown Night Watcher",
"schedule": "every 15m",
"state": "scheduled"
},
{
"job_id": "35d3ada9cf8f",
"name": "Mempalace Forge \u2014 Issue Analysis",
"schedule": "every 60m",
"state": "scheduled"
},
{
"job_id": "190b6fb8dc91",
"name": "Mempalace Watchtower \u2014 Fleet Health",
"schedule": "every 30m",
"state": "scheduled"
},
{
"job_id": "710ab589813c",
"name": "Ezra Health Monitor",
"schedule": "every 15m",
"state": "scheduled"
},
{
"job_id": "a0a9cce4575c",
"name": "daily-poka-yoke-ultraplan-awesometools",
"schedule": "every 1440m",
"state": "scheduled"
},
{
"job_id": "adc3a51457bd",
"name": "vps-agent-dispatch",
"schedule": "every 10m",
"state": "scheduled"
},
{
"job_id": "afd2c4eac44d",
"name": "Project Mnemosyne Nightly Burn v2",
"schedule": "*/30 * * * *",
"state": "scheduled"
},
{
"job_id": "f3a3c2832af0",
"name": "gemma4-multimodal-worker",
"schedule": "once in 15m",
"state": "completed"
},
{
"job_id": "c17a85c19838",
"name": "know-thy-father-analyzer",
"schedule": "0 * * * *",
"state": "scheduled"
},
{
"job_id": "2490fc01a14d",
"name": "Testament Burn - 10min work loop",
"schedule": "*/10 * * * *",
"state": "scheduled"
},
{
"job_id": "f5e858159d97",
"name": "Timmy Foundation Burn \u2014 15min PR loop",
"schedule": "*/15 * * * *",
"state": "scheduled"
},
{
"job_id": "5e262fb9bdce",
"name": "nightwatch-health-monitor",
"schedule": "*/15 * * * *",
"state": "scheduled"
},
{
"job_id": "f2b33a9dcf96",
"name": "nightwatch-mempalace-mine",
"schedule": "0 */2 * * *",
"state": "scheduled"
},
{
"job_id": "82cb9e76c54d",
"name": "nightwatch-backlog-burn",
"schedule": "0 */4 * * *",
"state": "scheduled"
},
{
"job_id": "d20e42a52863",
"name": "beacon-sprint",
"schedule": "*/15 * * * *",
"state": "scheduled"
},
{
"job_id": "579269489961",
"name": "testament-story",
"schedule": "*/15 * * * *",
"state": "scheduled"
},
{
"job_id": "2e5f9140d1ab",
"name": "nightwatch-research",
"schedule": "0 */2 * * *",
"state": "scheduled"
},
{
"job_id": "aeba92fd65e6",
"name": "timmy-dreams",
"schedule": "30 5 * * *",
"state": "scheduled"
}
]

View File

@@ -0,0 +1,14 @@
0 6 * * * /bin/bash /root/wizards/scripts/model_download_guard.sh >> /var/log/model_guard.log 2>&1
# Allegro Hybrid Heartbeat — quick wins every 15 min
*/15 * * * * /usr/bin/python3 /root/allegro/heartbeat_daemon.py >> /var/log/allegro_heartbeat.log 2>&1
# Allegro Burn Mode Cron Jobs - Deployed via issue #894
0 6 * * * cd /root/.hermes && python3 -c "import hermes_agent; from hermes_tools import terminal; output = terminal('echo \"Morning Report: $(date)\"'); print(output.get('output', ''))" >> /root/.hermes/logs/morning-report-$(date +\%Y\%m\%d).log 2>&1 # Allegro Morning Report at 0600
0,30 * * * * cd /root/.hermes && python3 /root/.hermes/retry_wrapper.py "python3 allegro/quick-lane-check.py" >> burn-logs/quick-lane-$(date +\%Y\%m\%d).log 2>&1 # Allegro Burn Loop #1 (with retry)
15,45 * * * * cd /root/.hermes && python3 /root/.hermes/retry_wrapper.py "python3 allegro/burn-mode-validator.py" >> burn-logs/validator-$(date +\%Y\%m\%d).log 2>&1 # Allegro Burn Loop #2 (with retry)
*/2 * * * * /root/wizards/bezalel/dead_man_monitor.sh
*/2 * * * * /root/wizards/allegro/bin/config-deadman.sh

View File

@@ -0,0 +1,10 @@
0 2 * * * /root/wizards/bezalel/run_nightly_watch.sh
0 3 * * * /root/wizards/bezalel/mempalace_nightly.sh
*/10 * * * * pgrep -f "act_runner daemon" > /dev/null || (cd /opt/gitea-runner && nohup ./act_runner daemon > /var/log/gitea-runner.log 2>&1 &)
30 3 * * * /root/wizards/bezalel/backup_databases.sh
*/15 * * * * /root/wizards/bezalel/meta_heartbeat.sh
0 4 * * * /root/wizards/bezalel/secret_guard.sh
0 4 * * * /usr/bin/env bash /root/timmy-home/scripts/backup_pipeline.sh >> /var/log/timmy/backup_pipeline_cron.log 2>&1
0 6 * * * /usr/bin/python3 /root/wizards/bezalel/ultraplan.py >> /var/log/bezalel-ultraplan.log 2>&1
@reboot /root/wizards/bezalel/emacs-daemon-start.sh
@reboot /root/wizards/bezalel/ngircd-start.sh

View File

@@ -0,0 +1,13 @@
# Burn Mode Cycles — 15 min autonomous loops
*/15 * * * * /root/wizards/ezra/bin/burn-mode.sh >> /root/wizards/ezra/reports/burn-cron.log 2>&1
# Household Snapshots — automated heartbeats and snapshots
# Ezra Self-Improvement Automation Suite
*/5 * * * * /usr/bin/python3 /root/wizards/ezra/tools/gitea_monitor.py >> /root/wizards/ezra/reports/gitea-monitor.log 2>&1
*/5 * * * * /usr/bin/python3 /root/wizards/ezra/tools/awareness_loop.py >> /root/wizards/ezra/reports/awareness-loop.log 2>&1
*/10 * * * * /usr/bin/python3 /root/wizards/ezra/tools/cron_health_monitor.py >> /root/wizards/ezra/reports/cron-health.log 2>&1
0 6 * * * /usr/bin/python3 /root/wizards/ezra/tools/morning_kt_compiler.py >> /root/wizards/ezra/reports/morning-kt.log 2>&1
5 6 * * * /usr/bin/python3 /root/wizards/ezra/tools/burndown_generator.py >> /root/wizards/ezra/reports/burndown.log 2>&1
0 3 * * * /root/wizards/ezra/mempalace_nightly.sh >> /var/log/ezra_mempalace_cron.log 2>&1
*/15 * * * * GITEA_TOKEN=6de6aa...1117 /root/wizards/ezra/dispatch-direct.sh >> /root/wizards/ezra/dispatch-cron.log 2>&1

306
scripts/config_validator.py Normal file
View File

@@ -0,0 +1,306 @@
#!/usr/bin/env python3
"""
config_validator.py — Validate all YAML/JSON config files in timmy-config.
Checks:
1. YAML syntax (pyyaml safe_load)
2. JSON syntax (json.loads)
3. Duplicate keys in YAML/JSON
4. Trailing whitespace in YAML
5. Tabs in YAML (should use spaces)
6. Cron expression validity (if present)
Exit 0 if all valid, 1 if any invalid.
"""
import json
import os
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError:
print("ERROR: PyYAML not installed. Run: pip install pyyaml")
sys.exit(1)
# ── Cron validation ──────────────────────────────────────────────────────────
DOW_NAMES = {"sun", "mon", "tue", "wed", "thu", "fri", "sat"}
MONTH_NAMES = {"jan", "feb", "mar", "apr", "may", "jun",
"jul", "aug", "sep", "oct", "nov", "dec"}
def _expand_cron_field(field: str, lo: int, hi: int, names: dict | None = None) -> set[int]:
"""Expand a single cron field into a set of valid integers."""
result: set[int] = set()
for part in field.split(","):
# Handle step: */N or 1-5/N
step = 1
if "/" in part:
part, step_str = part.split("/", 1)
if not step_str.isdigit() or int(step_str) < 1:
raise ValueError(f"invalid step value: {step_str}")
step = int(step_str)
if part == "*":
rng = range(lo, hi + 1, step)
elif "-" in part:
a, b = part.split("-", 1)
a = _resolve_name(a, names, lo, hi)
b = _resolve_name(b, names, lo, hi)
if a > b:
raise ValueError(f"range {a}-{b} is reversed")
rng = range(a, b + 1, step)
else:
val = _resolve_name(part, names, lo, hi)
rng = range(val, val + 1)
for v in rng:
if v < lo or v > hi:
raise ValueError(f"value {v} out of range [{lo}-{hi}]")
result.add(v)
return result
def _resolve_name(token: str, names: dict | None, lo: int, hi: int) -> int:
if names and token.lower() in names:
return names[token.lower()]
if not token.isdigit():
raise ValueError(f"unrecognized token: {token}")
val = int(token)
if val < lo or val > hi:
raise ValueError(f"value {val} out of range [{lo}-{hi}]")
return val
def validate_cron(expr: str) -> list[str]:
"""Validate a 5-field cron expression. Returns list of errors (empty = ok)."""
errors: list[str] = []
fields = expr.strip().split()
if len(fields) != 5:
return [f"expected 5 fields, got {len(fields)}"]
specs = [
(fields[0], 0, 59, None, "minute"),
(fields[1], 0, 23, None, "hour"),
(fields[2], 1, 31, None, "day-of-month"),
(fields[3], 1, 12, MONTH_NAMES, "month"),
(fields[4], 0, 7, DOW_NAMES, "day-of-week"),
]
for field, lo, hi, names, label in specs:
try:
_expand_cron_field(field, lo, hi, names)
except ValueError as e:
errors.append(f"{label}: {e}")
return errors
# ── Duplicate key detection ──────────────────────────────────────────────────
class DuplicateKeyError(Exception):
pass
class _StrictYAMLLoader(yaml.SafeLoader):
"""YAML loader that rejects duplicate keys."""
pass
def _no_duplicates_constructor(loader, node, deep=False):
mapping = {}
for key_node, value_node in node.value:
key = loader.construct_object(key_node, deep=deep)
if key in mapping:
raise DuplicateKeyError(
f"duplicate key '{key}' (line {key_node.start_mark.line + 1})"
)
mapping[key] = loader.construct_object(value_node, deep=deep)
return mapping
_StrictYAMLLoader.add_constructor(
yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG,
_no_duplicates_constructor,
)
def _json_has_duplicates(text: str) -> list[str]:
"""Check for duplicate keys in JSON by scanning for repeated quoted keys at same depth."""
errors: list[str] = []
# Use a custom approach: parse with object_pairs_hook
seen_stack: list[set[str]] = []
def _check_pairs(pairs):
level_keys: set[str] = set()
for k, _ in pairs:
if k in level_keys:
errors.append(f"duplicate JSON key: '{k}'")
level_keys.add(k)
return dict(pairs)
try:
json.loads(text, object_pairs_hook=_check_pairs)
except json.JSONDecodeError:
pass # syntax errors caught elsewhere
return errors
# ── Main validator ───────────────────────────────────────────────────────────
def find_config_files(root: Path) -> list[Path]:
"""Recursively find .yaml, .yml, .json files (skip .git, node_modules, venv)."""
skip_dirs = {".git", "node_modules", "venv", "__pycache__", ".venv"}
results: list[Path] = []
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = [d for d in dirnames if d not in skip_dirs]
for fname in filenames:
if fname.endswith((".yaml", ".yml", ".json")):
results.append(Path(dirpath) / fname)
return sorted(results)
def validate_yaml_file(filepath: Path, text: str) -> list[str]:
"""Validate a YAML file. Returns list of errors."""
errors: list[str] = []
# Check for tabs
for i, line in enumerate(text.splitlines(), 1):
if "\t" in line:
errors.append(f" line {i}: contains tab character (use spaces for YAML)")
if line != line.rstrip():
errors.append(f" line {i}: trailing whitespace")
# Check syntax + duplicate keys
try:
yaml.load(text, Loader=_StrictYAMLLoader)
except DuplicateKeyError as e:
errors.append(f" {e}")
except yaml.YAMLError as e:
mark = getattr(e, "problem_mark", None)
if mark:
errors.append(f" YAML syntax error at line {mark.line + 1}, col {mark.column + 1}: {e.problem}")
else:
errors.append(f" YAML syntax error: {e}")
# Check cron expressions in schedule fields
for i, line in enumerate(text.splitlines(), 1):
cron_match = re.search(r'(?:cron|schedule)\s*:\s*["\']?([*0-9/,a-zA-Z-]+(?:\s+[*0-9/,a-zA-Z-]+){4})["\']?', line)
if cron_match:
cron_errs = validate_cron(cron_match.group(1))
for ce in cron_errs:
errors.append(f" line {i}: invalid cron '{cron_match.group(1)}': {ce}")
return errors
def validate_json_file(filepath: Path, text: str) -> list[str]:
"""Validate a JSON file. Returns list of errors."""
errors: list[str] = []
# Check syntax
try:
json.loads(text)
except json.JSONDecodeError as e:
errors.append(f" JSON syntax error at line {e.lineno}, col {e.colno}: {e.msg}")
# Check duplicate keys
dup_errors = _json_has_duplicates(text)
errors.extend(dup_errors)
# Check for trailing whitespace (informational)
for i, line in enumerate(text.splitlines(), 1):
if line != line.rstrip():
errors.append(f" line {i}: trailing whitespace")
# Check cron expressions
cron_pattern = re.compile(r'"(?:cron|schedule)"?\s*:\s*"([^"]{5,})"')
for match in cron_pattern.finditer(text):
candidate = match.group(1).strip()
fields = candidate.split()
if len(fields) == 5 and all(re.match(r'^[*0-9/,a-zA-Z-]+$', f) for f in fields):
cron_errs = validate_cron(candidate)
for ce in cron_errs:
errors.append(f" invalid cron '{candidate}': {ce}")
# Also check nested schedule objects with cron fields
try:
obj = json.loads(text)
_scan_obj_for_cron(obj, errors)
except Exception:
pass
return errors
def _scan_obj_for_cron(obj, errors: list[str], path: str = ""):
"""Recursively scan dict/list for cron expressions."""
if isinstance(obj, dict):
for k, v in obj.items():
if k in ("cron", "schedule", "cron_expression") and isinstance(v, str):
fields = v.strip().split()
if len(fields) == 5:
cron_errs = validate_cron(v)
for ce in cron_errs:
errors.append(f" {path}.{k}: invalid cron '{v}': {ce}")
_scan_obj_for_cron(v, errors, f"{path}.{k}")
elif isinstance(obj, list):
for i, item in enumerate(obj):
_scan_obj_for_cron(item, errors, f"{path}[{i}]")
def main():
# Determine repo root (script lives in scripts/)
script_path = Path(__file__).resolve()
repo_root = script_path.parent.parent
print(f"Config Validator — scanning {repo_root}")
print("=" * 60)
files = find_config_files(repo_root)
print(f"Found {len(files)} config files to validate.\n")
total_errors = 0
failed_files: list[tuple[Path, list[str]]] = []
for filepath in files:
rel = filepath.relative_to(repo_root)
try:
text = filepath.read_text(encoding="utf-8", errors="replace")
except Exception as e:
failed_files.append((rel, [f" cannot read file: {e}"]))
total_errors += 1
continue
if filepath.suffix == ".json":
errors = validate_json_file(filepath, text)
else:
errors = validate_yaml_file(filepath, text)
if errors:
failed_files.append((rel, errors))
total_errors += len(errors)
print(f"FAIL {rel}")
else:
print(f"PASS {rel}")
print("\n" + "=" * 60)
print(f"Results: {len(files) - len(failed_files)}/{len(files)} files passed")
if failed_files:
print(f"\n{total_errors} error(s) in {len(failed_files)} file(s):\n")
for relpath, errs in failed_files:
print(f" {relpath}:")
for e in errs:
print(f" {e}")
print()
sys.exit(1)
else:
print("\nAll config files valid!")
sys.exit(0)
if __name__ == "__main__":
main()

View File

@@ -4,6 +4,8 @@
Part of the Gemini Sovereign Infrastructure Suite.
Auto-detects and fixes common failures across the fleet.
Safe-by-default: runs in dry-run mode unless --execute is given.
"""
import os
@@ -11,6 +13,7 @@ import sys
import subprocess
import argparse
import requests
import datetime
# --- CONFIGURATION ---
FLEET = {
@@ -21,51 +24,210 @@ FLEET = {
}
class SelfHealer:
def __init__(self, dry_run=True, confirm_kill=False, yes=False):
self.dry_run = dry_run
self.confirm_kill = confirm_kill
self.yes = yes
def log(self, message: str):
print(f"[*] {message}")
timestamp = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
print(f"[{timestamp}] {message}")
def run_remote(self, host: str, command: str):
ip = FLEET[host]["ip"]
ssh_cmd = ["ssh", "-o", "StrictHostKeyChecking=no", f"root@{ip}", command]
ssh_cmd = ["ssh", "-o", "StrictHostKeyChecking=no", "-o", "ConnectTimeout=5", f"root@{ip}", command]
if host == "mac":
ssh_cmd = ["bash", "-c", command]
try:
return subprocess.run(ssh_cmd, capture_output=True, text=True, timeout=10)
except:
return subprocess.run(ssh_cmd, capture_output=True, text=True, timeout=15)
except Exception as e:
self.log(f" [ERROR] Failed to run remote command on {host}: {e}")
return None
def confirm(self, prompt: str) -> bool:
"""Ask for confirmation unless --yes flag is set."""
if self.yes:
return True
while True:
response = input(f"{prompt} [y/N] ").strip().lower()
if response in ("y", "yes"):
return True
elif response in ("n", "no", ""):
return False
print("Please answer 'y' or 'n'.")
def check_llama_server(self, host: str):
ip = FLEET[host]["ip"]
port = FLEET[host]["port"]
try:
requests.get(f"http://{ip}:{port}/health", timeout=2)
except:
self.log(f" [!] llama-server down on {host}.")
if self.dry_run:
self.log(f" [DRY-RUN] Would restart llama-server on {host}")
else:
if self.confirm(f" Restart llama-server on {host}?"):
self.log(f" Restarting llama-server on {host}...")
self.run_remote(host, "systemctl restart llama-server")
else:
self.log(f" Skipped restart on {host}.")
def check_disk_space(self, host: str):
res = self.run_remote(host, "df -h / | tail -1 | awk '{print $5}' | sed 's/%//'")
if res and res.returncode == 0:
try:
usage = int(res.stdout.strip())
if usage > 90:
self.log(f" [!] Disk usage high on {host} ({usage}%).")
if self.dry_run:
self.log(f" [DRY-RUN] Would clean logs and vacuum journal on {host}")
else:
if self.confirm(f" Clean logs on {host}?"):
self.log(f" Cleaning logs on {host}...")
self.run_remote(host, "journalctl --vacuum-time=1d && rm -rf /var/log/*.gz")
else:
self.log(f" Skipped log cleaning on {host}.")
except:
pass
def check_memory(self, host: str):
res = self.run_remote(host, "free -m | awk '/^Mem:/{print $3/$2 * 100}'")
if res and res.returncode == 0:
try:
usage = float(res.stdout.strip())
if usage > 90:
self.log(f" [!] Memory usage high on {host} ({usage:.1f}%).")
if self.dry_run:
self.log(f" [DRY-RUN] Would check for memory hogs on {host}")
else:
self.log(f" Memory high but no automatic action defined.")
except:
pass
def check_processes(self, host: str):
# Example: check if any process uses > 80% CPU
res = self.run_remote(host, "ps aux --sort=-%cpu | awk 'NR>1 && $3>80 {print $2, $11, $3}'")
if res and res.returncode == 0 and res.stdout.strip():
self.log(f" [!] High CPU processes on {host}:")
for line in res.stdout.strip().split('\n'):
self.log(f" {line}")
if self.dry_run:
self.log(f" [DRY-RUN] Would review high-CPU processes on {host}")
else:
if self.confirm_kill:
if self.confirm(f" Kill high-CPU processes on {host}? (dangerous)"):
# This is a placeholder; real implementation would parse PIDs
self.log(f" Process killing not implemented yet (placeholder).")
else:
self.log(f" Skipped killing processes on {host}.")
else:
self.log(f" Use --confirm-kill to enable process termination (dangerous).")
def check_and_heal(self):
for host in FLEET:
self.log(f"Auditing {host}...")
# 1. Check llama-server
ip = FLEET[host]["ip"]
port = FLEET[host]["port"]
try:
requests.get(f"http://{ip}:{port}/health", timeout=2)
except:
self.log(f" [!] llama-server down on {host}. Attempting restart...")
self.run_remote(host, "systemctl restart llama-server")
# 2. Check disk space
res = self.run_remote(host, "df -h / | tail -1 | awk '{print $5}' | sed 's/%//'")
if res and res.returncode == 0:
try:
usage = int(res.stdout.strip())
if usage > 90:
self.log(f" [!] Disk usage high on {host} ({usage}%). Cleaning logs...")
self.run_remote(host, "journalctl --vacuum-time=1d && rm -rf /var/log/*.gz")
except:
pass
self.check_llama_server(host)
self.check_disk_space(host)
self.check_memory(host)
self.check_processes(host)
def run(self):
self.log("Starting self-healing cycle...")
if self.dry_run:
self.log("Starting self-healing cycle (DRY-RUN mode).")
else:
self.log("Starting self-healing cycle (EXECUTE mode).")
self.check_and_heal()
self.log("Cycle complete.")
def print_help_safe():
"""Print detailed explanation of what each action does."""
help_text = """
SAFE-BY-DEFAULT SELF-HEALING SCRIPT
This script checks fleet health and can optionally fix issues.
DEFAULT MODE: DRY-RUN (safe)
- Only reports what it would do, does not make changes.
- Use --execute to actually perform fixes.
CHECKS PERFORMED:
1. llama-server health
- Checks if llama-server is responding on each host.
- Action: restart service (requires --execute and confirmation).
2. Disk space
- Checks root partition usage on each host.
- Action: vacuum journal logs and remove rotated logs if >90% (requires --execute and confirmation).
3. Memory usage
- Reports high memory usage (informational only, no automatic action).
4. Process health
- Lists processes using >80% CPU.
- Action: kill processes (requires --confirm-kill flag, --execute, and confirmation).
SAFETY FEATURES:
- Dry-run by default.
- Explicit --execute flag required for changes.
- Confirmation prompts for all destructive actions.
- --yes flag to skip confirmations (for automation).
- --confirm-kill flag required to even consider killing processes.
- Timestamps on all log messages.
EXAMPLES:
python3 scripts/self_healing.py
# Dry-run: safe, shows what would happen.
python3 scripts/self_healing.py --execute
# Actually perform fixes after confirmation.
python3 scripts/self_healing.py --execute --yes
# Perform fixes without prompts (automation).
python3 scripts/self_healing.py --execute --confirm-kill
# Allow killing processes (dangerous).
python3 scripts/self_healing.py --help-safe
# Show this help.
"""
print(help_text)
def main():
healer = SelfHealer()
parser = argparse.ArgumentParser(
description="Self-healing infrastructure script (safe-by-default).",
add_help=False # We'll handle --help ourselves
)
parser.add_argument("--dry-run", action="store_true", default=False,
help="Run in dry-run mode (default behavior).")
parser.add_argument("--execute", action="store_true", default=False,
help="Actually perform fixes (disables dry-run).")
parser.add_argument("--confirm-kill", action="store_true", default=False,
help="Allow killing processes (dangerous).")
parser.add_argument("--yes", "-y", action="store_true", default=False,
help="Skip confirmation prompts.")
parser.add_argument("--help-safe", action="store_true", default=False,
help="Show detailed help about safety features.")
parser.add_argument("--help", "-h", action="store_true", default=False,
help="Show standard help.")
args = parser.parse_args()
if args.help_safe:
print_help_safe()
sys.exit(0)
if args.help:
parser.print_help()
sys.exit(0)
# Determine mode: if --execute is given, disable dry-run
dry_run = not args.execute
# If --dry-run is explicitly given, ensure dry-run (redundant but clear)
if args.dry_run:
dry_run = True
healer = SelfHealer(dry_run=dry_run, confirm_kill=args.confirm_kill, yes=args.yes)
healer.run()
if __name__ == "__main__":
main()
main()

102
v7.0.0-checkin.md Normal file
View File

@@ -0,0 +1,102 @@
1|# Release v7.0.0 — Fleet Architecture Checkin
2|
3|**Date:** 2026-04-08
4|**Tagged by:** Timmy
5|**Previous tag:** Golden-Allegro-v6-Sonnet4
6|
7|## Fleet Summary
8|
9|| Machine | Agents | Status |
10||---------|--------|--------|
11|| Local Mac M3 Max | Timmy (19 processes) | HEALTHY |
12|| Allegro VPS (167.99.126.228) | Allegro, Adagio, Ezra-A | HEALTHY (7d uptime, 43% disk) |
13|| Ezra VPS (143.198.27.163) | Ezra | WARNING (78% disk, load 10.38) |
14|| Bezalel VPS (159.203.146.185) | Bezalel | HEALTHY (2d uptime, 39% disk) |
15|
16|**Total agents running:** 6 across 4 machines
17|
18|## Model Configuration
19|
20|- Primary: claude-opus-4-6 (Anthropic)
21|- Fallback: hermes3 (local-llama.cpp)
22|- Fallback chain: OpenRouter claude-sonnet-4 -> local hermes3
23|
24|## Cron Jobs: 23 total
25|
26|| Status | Count |
27||--------|-------|
28|| Active | 15 |
29|| Paused | 8 |
30|
31|Active jobs: Health Monitor, Burn Mode Orchestrator, Tower Tick, Burn Deadman,
32|Morning Report, Evennia Report, Gitea Priority Inbox, Config Drift Guard,
33|Gitea Event Watcher, Burndown Watcher, Mempalace Forge, Mempalace Watchtower,
34|Ezra Health Monitor, Daily Poka-Yoke, VPS Agent Dispatch, Weekly Skill Extraction
35|
36|## Gitea Repos (Timmy_Foundation)
37|
38|| Repo | Issues | PRs | Updated | Branch |
39||------|--------|-----|---------|--------|
40|| the-nexus | 103 | 2 | 2026-04-08 | main |
41|| timmy-config | 129 | 1 | 2026-04-08 | main |
42|| timmy-home | 221 | 0 | 2026-04-08 | main |
43|| hermes-agent | 43 | 1 | 2026-04-08 | main |
44|| the-beacon | 23 | 0 | 2026-04-08 | main |
45|| turboquant | 10 | 0 | 2026-04-01 | main |
46|| the-door | 2 | 0 | 2026-04-06 | main |
47|| wolf | 2 | 0 | 2026-04-05 | main |
48|| the-testament | 0 | 0 | 2026-04-07 | main |
49|| timmy-academy | 1 | 0 | 2026-04-04 | master |
50|| .profile | 0 | 0 | 2026-04-07 | main |
51|
52|**Total open issues across fleet: 534**
53|**Total open PRs: 4**
54|
55|## Health Alerts
56|
57|1. WARN: Ezra VPS disk 78% (120G/154G) — needs cleanup
58|2. WARN: Ezra VPS load avg 10.38 — high for 2-core box
59|3. INFO: 8 paused cron jobs (expected — non-essential overnight jobs)
60|
61|## What's Working
62|
63|- All 4 machines reachable
64|- All core services running
65|- Config drift guard active
66|- Gitea event watcher active
67|- Dead man switch active
68|- Tower world ticking (tick 2045+)
69|- Morning reports delivering
70|- Mempalace analysis running
71|- VPS agent dispatch operational
72|
73|## Architecture
74|
75|```
76| Alexander (Principal)
77| |
78| [Telegram]
79| |
80| Timmy (Mac M3 Max) ---- Local llama.cpp (hermes3)
81| / | \
82| / | \
83| Allegro Ezra Bezalel
84| (DO VPS) (DO VPS) (DO VPS)
85| 3 agents 1 agent 1 agent
86|
87| Gitea Forge: forge.alexanderwhitestone.com
88| Evennia Tower: localhost:4000/4001
89| RunPod L40S: 8lfr3j47a5r3gn (Big Brain)
90|```
91|
92|## Release Notes
93|
94|This is the first versioned release tag (v7.0.0), transitioning from named
95|golden tags to semantic versioning. Previous tags preserved:
96|- Golden-Allegro-v6-Sonnet4
97|- burnup-20260405-infra
98|- SonOfTimmy-v5-FINAL
99|- SonOfTimmy-v4
100|- GoldenRockachopa
101|- pre-agent-workers-v1
102|