timmy-config/ansible/CONSOLIDATION.md

Ansible IaC — Canonical Consolidation

Issue: timmy-config#442 — [P2] Ansible IaC — Canonical Playbook for Fleet Management
Status: Canonical structure in place — ad-hoc mechanisms deprecated
Date: 2026-04-27

---

Canonical Structure Committed

The ansible/ directory is now the single source of truth for all fleet infrastructure:

• Inventory: ansible/inventory/hosts.yml
• Golden state: ansible/inventory/group_vars/wizards.yml
• Master playbook: ansible/playbooks/site.yml
• Sub-playbooks: deadman_switch.yml, golden_state.yml, agent_startup.yml, cron_schedule.yml, request_log.yml
• Roles: wizard_base, deadman_switch, golden_state, cron_manager, request_log
• Templates: systemd units, launchd plists, config templates
• Webhook deploy script: ansible/scripts/deploy_on_webhook.sh

All changes go through PR review. No direct edits on machines.

---

Acceptance Criteria — Status

Criterion	Status	Evidence
Ansible directory structure committed to timmy-config	✅ DONE	ansible/ fully populated at repo HEAD
Inventory file lists all known fleet machines	✅ DONE	inventory/hosts.yml — timmy, allegro, bezalel, ezra, forge
Deadman switch playbook deploys and configures the switch	✅ DONE	playbooks/deadman_switch.yml + roles/deadman_switch/ — systemd timer + launchd plist
Golden state rollback playbook restores known-good config	✅ DONE	playbooks/golden_state.yml + roles/golden_state/ — provider chain enforcement, Anthropic ban
Agent startup sequence playbook brings wizards up in order	✅ DONE	playbooks/agent_startup.yml — pull → validate → start → verify; serial:1 for safety
Cron jobs managed through Ansible (no manual crontab edits)	✅ DONE	roles/cron_manager/ + group_vars cron_jobs; old ad-hoc crontabs deprecated
Gitea webhook configured to trigger ansible-pull on merge	✅ READY	ansible/scripts/deploy_on_webhook.sh exists; webhook URL: http://localhost:9000/hooks/deploy-timmy-config (manual registration in Gitea Settings required on each target machine's webhook receiver)
All existing ad-hoc recovery mechanisms identified and replaced	✅ DONE	See "Deprecated Ad-Hoc Mechanisms" below — all superseded by Ansible roles
Playbook runs idempotently (can re-run without side effects)	✅ DESIGNED	All roles use creates, backup: true, changed_when checks. --check --diff supported; safe to re-run

---

Deprecated Ad-Hoc Mechanisms

The following ad-hoc recovery/cron/startup mechanisms have been replaced by the canonical Ansible deployment:

Ad-hoc Mechanism	Replaced By	Ansible Role / Playbook
bin/deadman-switch.sh (standalone deadman watch)	Deployed systemd timer + launchd agent with snapshot rollback	roles/deadman_switch/
bin/hermes-startup.sh (master startup sequence)	Agent startup playbook deploying golden state + service activation	playbooks/agent_startup.yml + roles/wizard_base/
fleet/auto_restart.py (process monitor + auto-restart)	Deadman switch with config snapshots + systemd restart handlers	roles/deadman_switch/
cron/muda-audit.crontab (muda audit cron on ezra)	Managed cron job via cron_manager role from group_vars/wizards.yml:cron_jobs	roles/cron_manager/
Legacy crontab entries on VPS machines	Full Ansible-managed cron set; old entries absent on next run	inventory/group_vars/wizards.yml

Preservation: Original ad-hoc scripts moved to deprecated/ with .deprecated suffix for audit trail. Do NOT re-enable.

---

Idempotency Guarantees

• wizard_base: directory creation is idempotent; git clone uses force: false, update: true; thin_config uses template idempotently
• golden_state: config template with backup: true; consenters scan doesn't trigger on clean state
• deadman_switch: timer/service plist deployment only if changed; initial snapshot uses ignore_errors: true if no config yet
• cron_manager: Ansible cron module ensures exact state (present/absent); --check safe
• request_log: database initialization guarded by creates

Verify with:

ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/site.yml --check --diff

---

Webhook Deployment

The webhook receiver must be running on each target machine to auto-deploy on Gitea merge:

• Handler script: ansible/scripts/deploy_on_webhook.sh
• Endpoint: http://localhost:9000/hooks/deploy-timmy-config
• Gitea configuration: Add webhook to timmy-config (Settings → Webhooks) with events: Pull Request (merged only)
• Systemd service (example unit) to run the webhook listener:

  [Unit]
  Description=Timmy Config Webhook Deploy
  After=network.target
  
  [Service]
  Type=simple
  ExecStart=/bin/bash /path/to/timmy-config/ansible/scripts/deploy_on_webhook.sh
  User=root
  Restart=on-failure
  
  [Install]
  WantedBy=multi-user.target
  

Manual registration per machine — not automated in playbooks yet (future enhancement).

---

Golden State Provider Chain

Anthropic permanently banned. Approved provider priority:

1. Kimi (kimi-k2.5) — primary
2. Gemini via OpenRouter (google/gemini-2.5-pro) — fallback
3. Ollama local (gemma4:latest) — terminal fallback

Enforced in group_vars/wizards.yml and validated in golden_state.yml and site.yml.

---

Smoke Test Checklist

Before closing #442, verify:

• Run ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/site.yml on a test machine or dry-run with --check
• Confirm systemd timers (systemctl list-timers) and/or launchd plists loaded
• Confirm cron_jobs present in crontab -l
• Confirm request_log DB created at ~/.local/timmy/request_log.db
• Verify config.yaml matches template and contains no Anthropic references
• Register webhook in Gitea UI and test with merge-to-main (staging fork or test repo first)

---

This consolidation establishes the Ansible directory as the canonical fleet management system for the Timmy Foundation.