Compare commits
134 Commits
timmy/flee
...
purge/open
| Author | SHA1 | Date | |
|---|---|---|---|
| 09aa06d65f | |||
| 8dc8bc4774 | |||
| fcf112cb1e | |||
| ce36d3813b | |||
| d4876c0fa5 | |||
| 8070536d57 | |||
| 438191c72e | |||
| 21e4039ec9 | |||
|
|
19aa0830f4 | ||
| f2edb6a9b3 | |||
| fc817c6a84 | |||
| a620bd19b3 | |||
| 0c98bce77f | |||
| c01e7f7d7f | |||
| 20bc0aa41a | |||
| b6c0620c83 | |||
| d43deb1d79 | |||
| 17de7f5df1 | |||
| 1dc29180b8 | |||
| 343e190cc3 | |||
| 932f48d06f | |||
| 0c7521d275 | |||
| bad31125c2 | |||
|
|
06031d923f | ||
| 7305d97e8f | |||
| 19e11b5287 | |||
| 03d53a644b | |||
| f2388733fb | |||
| 05e9c1bf51 | |||
| 186d5f8056 | |||
| 86914554f1 | |||
| a4665679ab | |||
| 6f3ed4c963 | |||
| b84b97fb6f | |||
|
|
a65f736f54 | ||
| 8bf41c00e4 | |||
| 41046d4bf1 | |||
| 52d60198fc | |||
| ae7915fc20 | |||
|
|
49b0b9d207 | ||
|
|
d64b2e7561 | ||
| 3fd4223e1e | |||
| d8f88bed16 | |||
| b172d23b98 | |||
| a01935825c | |||
| 544f2a9729 | |||
| 71bf82d9fb | |||
| fa9e83ac95 | |||
| 28317cbde9 | |||
| 6e5f1f6a22 | |||
| 2677e1c796 | |||
| e124ff8b05 | |||
| 5a649966ab | |||
| 836849ffeb | |||
| eb7ca1f96f | |||
|
|
641db62112 | ||
| b38871d4cd | |||
|
|
ee025957d9 | ||
|
|
7ec45642eb | ||
|
|
179833148f | ||
|
|
b18fc76868 | ||
| a6fded436f | |||
| 41044d36ae | |||
| a9aed5a545 | |||
| c5e6494326 | |||
| 641537eb07 | |||
| 763e35f47a | |||
| a31f58000b | |||
| 17fde3c03f | |||
| b53fdcd034 | |||
| 1cc1d2ae86 | |||
| 9ec0d1d80e | |||
| e9cdaf09dc | |||
| e8302b4af2 | |||
| 311ecf19db | |||
| 77f258efa5 | |||
| 5e12451588 | |||
| 80b6ceb118 | |||
| ffb85cc10f | |||
| 4179646456 | |||
| 681fd0763f | |||
| b21c2833f7 | |||
| f84b870ce4 | |||
| 8b4df81b5b | |||
| e96fae69cf | |||
| cccafd845b | |||
| 1f02166107 | |||
| 7dcaa05dbd | |||
| 18124206e1 | |||
| 11736e58cd | |||
| 14521ef664 | |||
| 8b17eaa537 | |||
| afee83c1fe | |||
| 56d8085e88 | |||
| 4e7b24617f | |||
| 8daa12c518 | |||
| e369727235 | |||
| 1705a7b802 | |||
| e0bef949dd | |||
| dafe8667c5 | |||
| 4844ce6238 | |||
| a43510a7eb | |||
| 3b00891614 | |||
| 74867bbfa7 | |||
| d07305b89c | |||
| 2812bac438 | |||
| 5c15704c3a | |||
| 30fdbef74e | |||
| 9cc2cf8f8d | |||
| a2eff1222b | |||
| 3f4465b646 | |||
| ff7ce9a022 | |||
| f04aaec4ed | |||
| d54a218a27 | |||
| 3cc92fde1a | |||
| 11a28b74bb | |||
|
|
593621c5e0 | ||
| 458dabfaed | |||
| 2e2a646ba8 | |||
|
|
f8dabae8eb | ||
|
|
0a4c8f2d37 | ||
|
|
0a13347e39 | ||
| dc75be18e4 | |||
| 0c950f991c | |||
|
|
7399c83024 | ||
|
|
cf213bffd1 | ||
|
|
fe7c5018e3 | ||
| c1c3aaa681 | |||
| d023512858 | |||
| e5e01e36c9 | |||
|
|
e5055d269b | ||
|
|
277d21aef6 | ||
|
|
2e64b160b5 | ||
|
|
f18955ea90 |
54
.gitea/PULL_REQUEST_TEMPLATE.md
Normal file
@@ -0,0 +1,54 @@
|
|||||||
|
## Summary
|
||||||
|
|
||||||
|
<!-- What changed and why. One paragraph max. -->
|
||||||
|
|
||||||
|
## Governing Issue
|
||||||
|
|
||||||
|
<!-- REQUIRED. Every PR must reference at least one issue. Max 3 issues per PR. -->
|
||||||
|
<!-- Closes #ISSUENUM -->
|
||||||
|
<!-- Refs #ISSUENUM -->
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
<!-- List the specific outcomes this PR delivers. Check each only when proven. -->
|
||||||
|
<!-- Copy these from the governing issue if it has them. -->
|
||||||
|
|
||||||
|
- [ ] Criterion 1
|
||||||
|
- [ ] Criterion 2
|
||||||
|
|
||||||
|
## Proof
|
||||||
|
|
||||||
|
<!-- No proof = no merge. See CONTRIBUTING.md for the full standard. -->
|
||||||
|
|
||||||
|
### Commands / logs / world-state proof
|
||||||
|
|
||||||
|
<!-- Paste the exact commands, output, log paths, or world-state artifacts that prove each acceptance criterion was met. -->
|
||||||
|
|
||||||
|
```
|
||||||
|
$ <command you ran>
|
||||||
|
<relevant output>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Visual proof (if applicable)
|
||||||
|
|
||||||
|
<!-- For skin updates, UI changes, dashboard changes: attach screenshot to the PR discussion. -->
|
||||||
|
<!-- Name what the screenshot proves. Do not commit binary media unless explicitly required. -->
|
||||||
|
|
||||||
|
## Risk and Rollback
|
||||||
|
|
||||||
|
<!-- What could go wrong? How do we undo it? -->
|
||||||
|
|
||||||
|
- **Risk level:** low / medium / high
|
||||||
|
- **What breaks if this is wrong:**
|
||||||
|
- **How to rollback:**
|
||||||
|
|
||||||
|
## Checklist
|
||||||
|
|
||||||
|
<!-- Complete every item before requesting review. -->
|
||||||
|
|
||||||
|
- [ ] PR body references at least one issue number (`Closes #N` or `Refs #N`)
|
||||||
|
- [ ] Changed files are syntactically valid (`python -c "import ast; ast.parse(open(f).read())"`, `node --check`, `bash -n`)
|
||||||
|
- [ ] Proof meets CONTRIBUTING.md standard (exact commands, output, or artifacts — not "looks right")
|
||||||
|
- [ ] Branch is up-to-date with base
|
||||||
|
- [ ] No more than 3 unrelated issues bundled in this PR
|
||||||
|
- [ ] Shell scripts are executable (`chmod +x`)
|
||||||
42
.gitea/workflows/architecture-lint.yml
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
# architecture-lint.yml — CI gate for the Architecture Linter v2
|
||||||
|
# Refs: #437 — repo-aware, test-backed, CI-enforced.
|
||||||
|
#
|
||||||
|
# Runs on every PR to main. Validates Python syntax, then runs
|
||||||
|
# linter tests and finally lints the repo itself.
|
||||||
|
|
||||||
|
name: Architecture Lint
|
||||||
|
|
||||||
|
on:
|
||||||
|
pull_request:
|
||||||
|
branches: [main, master]
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
linter-tests:
|
||||||
|
name: Linter Tests
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
python-version: "3.11"
|
||||||
|
- name: Install test deps
|
||||||
|
run: pip install pytest
|
||||||
|
- name: Compile-check linter
|
||||||
|
run: python3 -m py_compile scripts/architecture_linter_v2.py
|
||||||
|
- name: Run linter tests
|
||||||
|
run: python3 -m pytest tests/test_linter.py -v
|
||||||
|
|
||||||
|
lint-repo:
|
||||||
|
name: Lint Repository
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
needs: linter-tests
|
||||||
|
continue-on-error: true
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
python-version: "3.11"
|
||||||
|
- name: Run architecture linter
|
||||||
|
run: python3 scripts/architecture_linter_v2.py .
|
||||||
29
.gitea/workflows/pr-checklist.yml
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
# pr-checklist.yml — Automated PR quality gate
|
||||||
|
# Refs: #393 (PERPLEXITY-08), Epic #385
|
||||||
|
#
|
||||||
|
# Enforces the review checklist that agents skip when left to self-approve.
|
||||||
|
# Runs on every pull_request. Fails fast so bad PRs never reach a reviewer.
|
||||||
|
|
||||||
|
name: PR Checklist
|
||||||
|
|
||||||
|
on:
|
||||||
|
pull_request:
|
||||||
|
branches: [main, master]
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
pr-checklist:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
with:
|
||||||
|
fetch-depth: 0
|
||||||
|
|
||||||
|
- name: Set up Python
|
||||||
|
uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
python-version: "3.11"
|
||||||
|
|
||||||
|
- name: Run PR checklist
|
||||||
|
env:
|
||||||
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||||
|
run: python3 bin/pr-checklist.py
|
||||||
24
.gitea/workflows/smoke.yml
Normal file
@@ -0,0 +1,24 @@
|
|||||||
|
name: Smoke Test
|
||||||
|
on:
|
||||||
|
pull_request:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
jobs:
|
||||||
|
smoke:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
python-version: '3.11'
|
||||||
|
- name: Parse check
|
||||||
|
run: |
|
||||||
|
find . -name '*.yml' -o -name '*.yaml' | grep -v .gitea | xargs -r python3 -c "import sys,yaml; [yaml.safe_load(open(f)) for f in sys.argv[1:]]"
|
||||||
|
find . -name '*.json' | xargs -r python3 -m json.tool > /dev/null
|
||||||
|
find . -name '*.py' | xargs -r python3 -m py_compile
|
||||||
|
find . -name '*.sh' | xargs -r bash -n
|
||||||
|
echo "PASS: All files parse"
|
||||||
|
- name: Secret scan
|
||||||
|
run: |
|
||||||
|
if grep -rE 'sk-or-|sk-ant-|ghp_|AKIA' . --include='*.yml' --include='*.py' --include='*.sh' 2>/dev/null | grep -v .gitea; then exit 1; fi
|
||||||
|
echo "PASS: No secrets"
|
||||||
121
.gitea/workflows/validate-config.yaml
Normal file
@@ -0,0 +1,121 @@
|
|||||||
|
# validate-config.yaml
|
||||||
|
# Validates all config files, scripts, and playbooks on every PR.
|
||||||
|
# Addresses #289: repo-native validation for timmy-config changes.
|
||||||
|
#
|
||||||
|
# Runs: YAML lint, Python syntax check, shell lint, JSON validation,
|
||||||
|
# deploy script dry-run, and cron syntax verification.
|
||||||
|
|
||||||
|
name: Validate Config
|
||||||
|
|
||||||
|
on:
|
||||||
|
pull_request:
|
||||||
|
branches: [main]
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
yaml-lint:
|
||||||
|
name: YAML Lint
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- name: Install yamllint
|
||||||
|
run: pip install yamllint
|
||||||
|
- name: Lint YAML files
|
||||||
|
run: |
|
||||||
|
find . -name '*.yaml' -o -name '*.yml' | \
|
||||||
|
grep -v '.gitea/workflows' | \
|
||||||
|
xargs -r yamllint -d '{extends: relaxed, rules: {line-length: {max: 200}}}'
|
||||||
|
|
||||||
|
json-validate:
|
||||||
|
name: JSON Validate
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- name: Validate JSON files
|
||||||
|
run: |
|
||||||
|
find . -name '*.json' -print0 | while IFS= read -r -d '' f; do
|
||||||
|
echo "Validating: $f"
|
||||||
|
python3 -m json.tool "$f" > /dev/null || exit 1
|
||||||
|
done
|
||||||
|
|
||||||
|
python-check:
|
||||||
|
name: Python Syntax & Import Check
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
python-version: '3.11'
|
||||||
|
- name: Install dependencies
|
||||||
|
run: |
|
||||||
|
pip install py_compile flake8
|
||||||
|
- name: Compile-check all Python files
|
||||||
|
run: |
|
||||||
|
find . -name '*.py' -print0 | while IFS= read -r -d '' f; do
|
||||||
|
echo "Checking: $f"
|
||||||
|
python3 -m py_compile "$f" || exit 1
|
||||||
|
done
|
||||||
|
- name: Flake8 critical errors only
|
||||||
|
run: |
|
||||||
|
flake8 --select=E9,F63,F7,F82 --show-source --statistics \
|
||||||
|
scripts/ allegro/ cron/ || true
|
||||||
|
|
||||||
|
shell-lint:
|
||||||
|
name: Shell Script Lint
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- name: Install shellcheck
|
||||||
|
run: sudo apt-get install -y shellcheck
|
||||||
|
- name: Lint shell scripts
|
||||||
|
run: |
|
||||||
|
find . -name '*.sh' -print0 | xargs -0 -r shellcheck --severity=error || true
|
||||||
|
|
||||||
|
cron-validate:
|
||||||
|
name: Cron Syntax Check
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- name: Validate cron entries
|
||||||
|
run: |
|
||||||
|
if [ -d cron ]; then
|
||||||
|
find cron -name '*.cron' -o -name '*.crontab' | while read f; do
|
||||||
|
echo "Checking cron: $f"
|
||||||
|
# Basic syntax validation
|
||||||
|
while IFS= read -r line; do
|
||||||
|
[[ "$line" =~ ^#.*$ ]] && continue
|
||||||
|
[[ -z "$line" ]] && continue
|
||||||
|
fields=$(echo "$line" | awk '{print NF}')
|
||||||
|
if [ "$fields" -lt 6 ]; then
|
||||||
|
echo "ERROR: Too few fields in $f: $line"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done < "$f"
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
deploy-dry-run:
|
||||||
|
name: Deploy Script Dry Run
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- name: Syntax-check deploy.sh
|
||||||
|
run: |
|
||||||
|
if [ -f deploy.sh ]; then
|
||||||
|
bash -n deploy.sh
|
||||||
|
echo "deploy.sh syntax OK"
|
||||||
|
fi
|
||||||
|
|
||||||
|
playbook-schema:
|
||||||
|
name: Playbook Schema Validation
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
python-version: '3.11'
|
||||||
|
- name: Install PyYAML
|
||||||
|
run: pip install pyyaml
|
||||||
|
- name: Validate playbook structure
|
||||||
|
run: python3 scripts/validate_playbook_schema.py
|
||||||
35
.gitignore
vendored
@@ -1,9 +1,8 @@
|
|||||||
# Secrets
|
*.pyc
|
||||||
*.token
|
*.pyo
|
||||||
*.key
|
*.egg-info/
|
||||||
*.secret
|
dist/
|
||||||
|
build/
|
||||||
# Local state
|
|
||||||
*.db
|
*.db
|
||||||
*.db-wal
|
*.db-wal
|
||||||
*.db-shm
|
*.db-shm
|
||||||
@@ -11,3 +10,27 @@ __pycache__/
|
|||||||
|
|
||||||
# Generated audit reports
|
# Generated audit reports
|
||||||
reports/
|
reports/
|
||||||
|
|
||||||
|
# Secrets and credentials
|
||||||
|
.bash_history
|
||||||
|
.git-credentials
|
||||||
|
.gitea_token
|
||||||
|
.ssh/id_*
|
||||||
|
.ssh/known_hosts
|
||||||
|
.viminfo
|
||||||
|
.wget-hsts
|
||||||
|
.profile
|
||||||
|
.bashrc
|
||||||
|
.bash_logout
|
||||||
|
.python_history
|
||||||
|
.lesshst
|
||||||
|
.selected_editor
|
||||||
|
.sudo_as_admin_successful
|
||||||
|
.config/telegram/
|
||||||
|
.hermes/.env
|
||||||
|
.hermes/auth.json
|
||||||
|
*.pem
|
||||||
|
*.key
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
|
!.env.example
|
||||||
|
|||||||
10
SOUL.md
@@ -1,3 +1,13 @@
|
|||||||
|
<!--
|
||||||
|
NOTE: This is the BITCOIN INSCRIPTION version of SOUL.md.
|
||||||
|
It is the immutable on-chain conscience. Do not modify this content.
|
||||||
|
|
||||||
|
The NARRATIVE identity document (for onboarding, Audio Overviews,
|
||||||
|
and system prompts) lives in timmy-home/SOUL.md.
|
||||||
|
|
||||||
|
See: #388, #378 for the divergence audit.
|
||||||
|
-->
|
||||||
|
|
||||||
# SOUL.md
|
# SOUL.md
|
||||||
|
|
||||||
## Inscription 1 — The Immutable Conscience
|
## Inscription 1 — The Immutable Conscience
|
||||||
|
|||||||
47
ansible/BANNED_PROVIDERS.yml
Normal file
@@ -0,0 +1,47 @@
|
|||||||
|
# =============================================================================
|
||||||
|
# BANNED PROVIDERS — The Timmy Foundation
|
||||||
|
# =============================================================================
|
||||||
|
# "Anthropic is not only fired, but banned. I don't want these errors
|
||||||
|
# cropping up." — Alexander, 2026-04-09
|
||||||
|
#
|
||||||
|
# This is a HARD BAN. Not deprecated. Not fallback. BANNED.
|
||||||
|
# Enforcement: pre-commit hook, linter, Ansible validation, CI tests.
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
banned_providers:
|
||||||
|
- name: anthropic
|
||||||
|
reason: "Permanently banned. SDK access gated despite active quota. Fleet was bricked because golden state pointed to Anthropic Sonnet."
|
||||||
|
banned_date: "2026-04-09"
|
||||||
|
enforcement: strict # Ansible playbook FAILS if detected
|
||||||
|
models:
|
||||||
|
- "claude-sonnet-*"
|
||||||
|
- "claude-opus-*"
|
||||||
|
- "claude-haiku-*"
|
||||||
|
- "claude-*"
|
||||||
|
endpoints:
|
||||||
|
- "api.anthropic.com"
|
||||||
|
- "anthropic/*" # OpenRouter pattern
|
||||||
|
api_keys:
|
||||||
|
- "ANTHROPIC_API_KEY"
|
||||||
|
- "CLAUDE_API_KEY"
|
||||||
|
|
||||||
|
# Golden state alternative:
|
||||||
|
approved_providers:
|
||||||
|
- name: kimi-coding
|
||||||
|
model: kimi-k2.5
|
||||||
|
role: primary
|
||||||
|
- name: openrouter
|
||||||
|
model: google/gemini-2.5-pro
|
||||||
|
role: fallback
|
||||||
|
- name: ollama
|
||||||
|
model: "gemma4:latest"
|
||||||
|
role: terminal_fallback
|
||||||
|
|
||||||
|
# Future evaluation:
|
||||||
|
evaluation_candidates:
|
||||||
|
- name: mimo-v2-pro
|
||||||
|
status: pending
|
||||||
|
notes: "Free via Nous Portal for ~2 weeks from 2026-04-07. Add after fallback chain is fixed."
|
||||||
|
- name: hermes-4
|
||||||
|
status: available
|
||||||
|
notes: "Free on Nous Portal. 36B and 70B variants. Home team model."
|
||||||
95
ansible/README.md
Normal file
@@ -0,0 +1,95 @@
|
|||||||
|
# Ansible IaC — The Timmy Foundation Fleet
|
||||||
|
|
||||||
|
> One canonical Ansible playbook defines: deadman switch, cron schedule,
|
||||||
|
> golden state rollback, agent startup sequence.
|
||||||
|
> — KT Final Session 2026-04-08, Priority TWO
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This directory contains the **single source of truth** for fleet infrastructure.
|
||||||
|
No more ad-hoc recovery implementations. No more overlapping deadman switches.
|
||||||
|
No more agents mutating their own configs into oblivion.
|
||||||
|
|
||||||
|
**Everything** goes through Ansible. If it's not in a playbook, it doesn't exist.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────┐
|
||||||
|
│ Gitea (Source of Truth) │
|
||||||
|
│ timmy-config/ansible/ │
|
||||||
|
│ ├── inventory/hosts.yml (fleet machines) │
|
||||||
|
│ ├── playbooks/site.yml (master playbook) │
|
||||||
|
│ ├── roles/ (reusable roles) │
|
||||||
|
│ └── group_vars/wizards.yml (golden state) │
|
||||||
|
└──────────────────┬──────────────────────────────┘
|
||||||
|
│ PR merge triggers webhook
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────┐
|
||||||
|
│ Gitea Webhook Handler │
|
||||||
|
│ scripts/deploy_on_webhook.sh │
|
||||||
|
│ → ansible-pull on each target machine │
|
||||||
|
└──────────────────┬──────────────────────────────┘
|
||||||
|
│ ansible-pull
|
||||||
|
▼
|
||||||
|
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
|
||||||
|
│ Timmy │ │ Allegro │ │ Bezalel │ │ Ezra │
|
||||||
|
│ (Mac) │ │ (VPS) │ │ (VPS) │ │ (VPS) │
|
||||||
|
│ │ │ │ │ │ │ │
|
||||||
|
│ deadman │ │ deadman │ │ deadman │ │ deadman │
|
||||||
|
│ cron │ │ cron │ │ cron │ │ cron │
|
||||||
|
│ golden │ │ golden │ │ golden │ │ golden │
|
||||||
|
│ req_log │ │ req_log │ │ req_log │ │ req_log │
|
||||||
|
└──────────┘ └──────────┘ └──────────┘ └──────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Deploy everything to all machines
|
||||||
|
ansible-playbook -i inventory/hosts.yml playbooks/site.yml
|
||||||
|
|
||||||
|
# Deploy only golden state config
|
||||||
|
ansible-playbook -i inventory/hosts.yml playbooks/golden_state.yml
|
||||||
|
|
||||||
|
# Deploy only to a specific wizard
|
||||||
|
ansible-playbook -i inventory/hosts.yml playbooks/site.yml --limit bezalel
|
||||||
|
|
||||||
|
# Dry run (check mode)
|
||||||
|
ansible-playbook -i inventory/hosts.yml playbooks/site.yml --check --diff
|
||||||
|
```
|
||||||
|
|
||||||
|
## Golden State Provider Chain
|
||||||
|
|
||||||
|
All wizard configs converge on this provider chain. **Anthropic is BANNED.**
|
||||||
|
|
||||||
|
| Priority | Provider | Model | Endpoint |
|
||||||
|
| -------- | -------------------- | ---------------- | --------------------------------- |
|
||||||
|
| 1 | Kimi | kimi-k2.5 | https://api.kimi.com/coding/v1 |
|
||||||
|
| 2 | Gemini (OpenRouter) | gemini-2.5-pro | https://openrouter.ai/api/v1 |
|
||||||
|
| 3 | Ollama (local) | gemma4:latest | http://localhost:11434/v1 |
|
||||||
|
|
||||||
|
## Roles
|
||||||
|
|
||||||
|
| Role | Purpose |
|
||||||
|
| ---------------- | ------------------------------------------------------------ |
|
||||||
|
| `wizard_base` | Common wizard setup: directories, thin config, git pull |
|
||||||
|
| `deadman_switch` | Health check → snapshot good config → rollback on death |
|
||||||
|
| `golden_state` | Deploy and enforce golden state provider chain |
|
||||||
|
| `request_log` | SQLite telemetry table for every inference call |
|
||||||
|
| `cron_manager` | Source-controlled cron jobs — no manual crontab edits |
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
1. **No manual changes.** If it's not in a playbook, it will be overwritten.
|
||||||
|
2. **No Anthropic.** Banned. Enforcement is automated. See `BANNED_PROVIDERS.yml`.
|
||||||
|
3. **Idempotent.** Every playbook can run 100 times with the same result.
|
||||||
|
4. **PR required.** Config changes go through Gitea PR review, then deploy.
|
||||||
|
5. **One identity per machine.** No duplicate agents. Fleet audit enforces this.
|
||||||
|
|
||||||
|
## Related Issues
|
||||||
|
|
||||||
|
- timmy-config #442: [P2] Ansible IaC Canonical Playbook
|
||||||
|
- timmy-config #444: Wire Deadman Switch ACTION
|
||||||
|
- timmy-config #443: Thin Config Pattern
|
||||||
|
- timmy-config #446: request_log Telemetry Table
|
||||||
21
ansible/ansible.cfg
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
[defaults]
|
||||||
|
inventory = inventory/hosts.yml
|
||||||
|
roles_path = roles
|
||||||
|
host_key_checking = False
|
||||||
|
retry_files_enabled = False
|
||||||
|
stdout_callback = yaml
|
||||||
|
forks = 10
|
||||||
|
timeout = 30
|
||||||
|
|
||||||
|
# Logging
|
||||||
|
log_path = /var/log/ansible/timmy-fleet.log
|
||||||
|
|
||||||
|
[privilege_escalation]
|
||||||
|
become = True
|
||||||
|
become_method = sudo
|
||||||
|
become_user = root
|
||||||
|
become_ask_pass = False
|
||||||
|
|
||||||
|
[ssh_connection]
|
||||||
|
pipelining = True
|
||||||
|
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no
|
||||||
74
ansible/inventory/group_vars/wizards.yml
Normal file
@@ -0,0 +1,74 @@
|
|||||||
|
# =============================================================================
|
||||||
|
# Wizard Group Variables — Golden State Configuration
|
||||||
|
# =============================================================================
|
||||||
|
# These variables are applied to ALL wizards in the fleet.
|
||||||
|
# This IS the golden state. If a wizard deviates, Ansible corrects it.
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
# --- Deadman Switch ---
|
||||||
|
deadman_enabled: true
|
||||||
|
deadman_check_interval: 300 # 5 minutes between health checks
|
||||||
|
deadman_snapshot_dir: "~/.local/timmy/snapshots"
|
||||||
|
deadman_max_snapshots: 10 # Rolling window of good configs
|
||||||
|
deadman_restart_cooldown: 60 # Seconds to wait before restart after failure
|
||||||
|
deadman_max_restart_attempts: 3
|
||||||
|
deadman_escalation_channel: telegram # Alert Alexander after max attempts
|
||||||
|
|
||||||
|
# --- Thin Config ---
|
||||||
|
thin_config_path: "~/.timmy/thin_config.yml"
|
||||||
|
thin_config_mode: "0444" # Read-only — agents CANNOT modify
|
||||||
|
upstream_repo: "https://forge.alexanderwhitestone.com/Timmy_Foundation/timmy-config.git"
|
||||||
|
upstream_branch: main
|
||||||
|
config_pull_on_wake: true
|
||||||
|
config_validation_enabled: true
|
||||||
|
|
||||||
|
# --- Agent Settings ---
|
||||||
|
agent_max_turns: 30
|
||||||
|
agent_reasoning_effort: high
|
||||||
|
agent_verbose: false
|
||||||
|
agent_approval_mode: auto
|
||||||
|
|
||||||
|
# --- Hermes Harness ---
|
||||||
|
hermes_config_dir: "{{ hermes_home }}"
|
||||||
|
hermes_bin_dir: "{{ hermes_home }}/bin"
|
||||||
|
hermes_skins_dir: "{{ hermes_home }}/skins"
|
||||||
|
hermes_playbooks_dir: "{{ hermes_home }}/playbooks"
|
||||||
|
hermes_memories_dir: "{{ hermes_home }}/memories"
|
||||||
|
|
||||||
|
# --- Request Log (Telemetry) ---
|
||||||
|
request_log_enabled: true
|
||||||
|
request_log_path: "~/.local/timmy/request_log.db"
|
||||||
|
request_log_rotation_days: 30 # Archive logs older than 30 days
|
||||||
|
request_log_sync_to_gitea: false # Future: push telemetry summaries to Gitea
|
||||||
|
|
||||||
|
# --- Cron Schedule ---
|
||||||
|
# All cron jobs are managed here. No manual crontab edits.
|
||||||
|
cron_jobs:
|
||||||
|
- name: "Deadman health check"
|
||||||
|
job: "cd {{ wizard_home }}/workspace/timmy-config && python3 fleet/health_check.py"
|
||||||
|
minute: "*/5"
|
||||||
|
hour: "*"
|
||||||
|
enabled: "{{ deadman_enabled }}"
|
||||||
|
|
||||||
|
- name: "Muda audit"
|
||||||
|
job: "cd {{ wizard_home }}/workspace/timmy-config && bash fleet/muda-audit.sh >> /tmp/muda-audit.log 2>&1"
|
||||||
|
minute: "0"
|
||||||
|
hour: "21"
|
||||||
|
weekday: "0"
|
||||||
|
enabled: true
|
||||||
|
|
||||||
|
- name: "Config pull from upstream"
|
||||||
|
job: "cd {{ wizard_home }}/workspace/timmy-config && git pull --ff-only origin main"
|
||||||
|
minute: "*/15"
|
||||||
|
hour: "*"
|
||||||
|
enabled: "{{ config_pull_on_wake }}"
|
||||||
|
|
||||||
|
- name: "Request log rotation"
|
||||||
|
job: "python3 -c \"import sqlite3,datetime; db=sqlite3.connect('{{ request_log_path }}'); db.execute('DELETE FROM request_log WHERE timestamp < datetime(\\\"now\\\", \\\"-{{ request_log_rotation_days }} days\\\")'); db.commit()\""
|
||||||
|
minute: "0"
|
||||||
|
hour: "3"
|
||||||
|
enabled: "{{ request_log_enabled }}"
|
||||||
|
|
||||||
|
# --- Provider Enforcement ---
|
||||||
|
# These are validated on every Ansible run. Any Anthropic reference = failure.
|
||||||
|
provider_ban_enforcement: strict # strict = fail playbook, warn = log only
|
||||||
119
ansible/inventory/hosts.yml
Normal file
@@ -0,0 +1,119 @@
|
|||||||
|
# =============================================================================
|
||||||
|
# Fleet Inventory — The Timmy Foundation
|
||||||
|
# =============================================================================
|
||||||
|
# Source of truth for all machines in the fleet.
|
||||||
|
# Update this file when machines are added/removed.
|
||||||
|
# All changes go through PR review.
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
all:
|
||||||
|
children:
|
||||||
|
wizards:
|
||||||
|
hosts:
|
||||||
|
timmy:
|
||||||
|
ansible_host: localhost
|
||||||
|
ansible_connection: local
|
||||||
|
wizard_name: Timmy
|
||||||
|
wizard_role: "Primary wizard — soul of the fleet"
|
||||||
|
wizard_provider_primary: kimi-coding
|
||||||
|
wizard_model_primary: kimi-k2.5
|
||||||
|
hermes_port: 8081
|
||||||
|
api_port: 8645
|
||||||
|
wizard_home: "{{ ansible_env.HOME }}/wizards/timmy"
|
||||||
|
hermes_home: "{{ ansible_env.HOME }}/.hermes"
|
||||||
|
machine_type: mac
|
||||||
|
# Timmy runs on Alexander's M3 Max
|
||||||
|
ollama_available: true
|
||||||
|
|
||||||
|
allegro:
|
||||||
|
ansible_host: 167.99.126.228
|
||||||
|
ansible_user: root
|
||||||
|
wizard_name: Allegro
|
||||||
|
wizard_role: "Kimi-backed third wizard house — tight coding tasks"
|
||||||
|
wizard_provider_primary: kimi-coding
|
||||||
|
wizard_model_primary: kimi-k2.5
|
||||||
|
hermes_port: 8081
|
||||||
|
api_port: 8645
|
||||||
|
wizard_home: /root/wizards/allegro
|
||||||
|
hermes_home: /root/.hermes
|
||||||
|
machine_type: vps
|
||||||
|
ollama_available: false
|
||||||
|
|
||||||
|
bezalel:
|
||||||
|
ansible_host: 159.203.146.185
|
||||||
|
ansible_user: root
|
||||||
|
wizard_name: Bezalel
|
||||||
|
wizard_role: "Forge-and-testbed wizard — infrastructure, deployment, hardening"
|
||||||
|
wizard_provider_primary: kimi-coding
|
||||||
|
wizard_model_primary: kimi-k2.5
|
||||||
|
hermes_port: 8081
|
||||||
|
api_port: 8656
|
||||||
|
wizard_home: /root/wizards/bezalel
|
||||||
|
hermes_home: /root/.hermes
|
||||||
|
machine_type: vps
|
||||||
|
ollama_available: false
|
||||||
|
# NOTE: The awake Bezalel may be the duplicate.
|
||||||
|
# Fleet audit (the-nexus #1144) will resolve identity.
|
||||||
|
|
||||||
|
ezra:
|
||||||
|
ansible_host: 143.198.27.163
|
||||||
|
ansible_user: root
|
||||||
|
wizard_name: Ezra
|
||||||
|
wizard_role: "Infrastructure wizard — Gitea, nginx, hosting"
|
||||||
|
wizard_provider_primary: kimi-coding
|
||||||
|
wizard_model_primary: kimi-k2.5
|
||||||
|
hermes_port: 8081
|
||||||
|
api_port: 8645
|
||||||
|
wizard_home: /root/wizards/ezra
|
||||||
|
hermes_home: /root/.hermes
|
||||||
|
machine_type: vps
|
||||||
|
ollama_available: false
|
||||||
|
# NOTE: Currently DOWN — Telegram key revoked, awaiting propagation.
|
||||||
|
|
||||||
|
# Infrastructure hosts (not wizards, but managed by Ansible)
|
||||||
|
infrastructure:
|
||||||
|
hosts:
|
||||||
|
forge:
|
||||||
|
ansible_host: 143.198.27.163
|
||||||
|
ansible_user: root
|
||||||
|
# Gitea runs on the same box as Ezra
|
||||||
|
gitea_url: https://forge.alexanderwhitestone.com
|
||||||
|
gitea_org: Timmy_Foundation
|
||||||
|
|
||||||
|
vars:
|
||||||
|
# Global variables applied to all hosts
|
||||||
|
gitea_repo_url: "https://forge.alexanderwhitestone.com/Timmy_Foundation/timmy-config.git"
|
||||||
|
gitea_branch: main
|
||||||
|
config_base_path: "{{ gitea_repo_url }}"
|
||||||
|
timmy_log_dir: "~/.local/timmy/fleet-health"
|
||||||
|
request_log_db: "~/.local/timmy/request_log.db"
|
||||||
|
|
||||||
|
# Golden state provider chain — Anthropic is BANNED
|
||||||
|
golden_state_providers:
|
||||||
|
- name: kimi-coding
|
||||||
|
model: kimi-k2.5
|
||||||
|
base_url: "https://api.kimi.com/coding/v1"
|
||||||
|
timeout: 120
|
||||||
|
reason: "Primary — Kimi K2.5 (best value, least friction)"
|
||||||
|
- name: openrouter
|
||||||
|
model: google/gemini-2.5-pro
|
||||||
|
base_url: "https://openrouter.ai/api/v1"
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
timeout: 120
|
||||||
|
reason: "Fallback — Gemini 2.5 Pro via OpenRouter"
|
||||||
|
- name: ollama
|
||||||
|
model: "gemma4:latest"
|
||||||
|
base_url: "http://localhost:11434/v1"
|
||||||
|
timeout: 180
|
||||||
|
reason: "Terminal fallback — local Ollama (sovereign, no API needed)"
|
||||||
|
|
||||||
|
# Banned providers — hard enforcement
|
||||||
|
banned_providers:
|
||||||
|
- anthropic
|
||||||
|
- claude
|
||||||
|
banned_models_patterns:
|
||||||
|
- "claude-*"
|
||||||
|
- "anthropic/*"
|
||||||
|
- "*sonnet*"
|
||||||
|
- "*opus*"
|
||||||
|
- "*haiku*"
|
||||||
98
ansible/playbooks/agent_startup.yml
Normal file
@@ -0,0 +1,98 @@
|
|||||||
|
---
|
||||||
|
# =============================================================================
|
||||||
|
# agent_startup.yml — Resurrect Wizards from Checked-in Configs
|
||||||
|
# =============================================================================
|
||||||
|
# Brings wizards back online using golden state configs.
|
||||||
|
# Order: pull config → validate → start agent → verify with request_log
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
- name: "Agent Startup Sequence"
|
||||||
|
hosts: wizards
|
||||||
|
become: true
|
||||||
|
serial: 1 # One wizard at a time to avoid cascading issues
|
||||||
|
|
||||||
|
tasks:
|
||||||
|
- name: "Pull latest config from upstream"
|
||||||
|
git:
|
||||||
|
repo: "{{ upstream_repo }}"
|
||||||
|
dest: "{{ wizard_home }}/workspace/timmy-config"
|
||||||
|
version: "{{ upstream_branch }}"
|
||||||
|
force: true
|
||||||
|
tags: [pull]
|
||||||
|
|
||||||
|
- name: "Deploy golden state config"
|
||||||
|
include_role:
|
||||||
|
name: golden_state
|
||||||
|
tags: [config]
|
||||||
|
|
||||||
|
- name: "Validate config — no banned providers"
|
||||||
|
shell: |
|
||||||
|
python3 -c "
|
||||||
|
import yaml, sys
|
||||||
|
with open('{{ wizard_home }}/config.yaml') as f:
|
||||||
|
cfg = yaml.safe_load(f)
|
||||||
|
banned = {{ banned_providers }}
|
||||||
|
for p in cfg.get('fallback_providers', []):
|
||||||
|
if p.get('provider', '') in banned:
|
||||||
|
print(f'BANNED: {p[\"provider\"]}', file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
model = cfg.get('model', {}).get('provider', '')
|
||||||
|
if model in banned:
|
||||||
|
print(f'BANNED default provider: {model}', file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
print('Config validated — no banned providers.')
|
||||||
|
"
|
||||||
|
register: config_valid
|
||||||
|
tags: [validate]
|
||||||
|
|
||||||
|
- name: "Ensure hermes-agent service is running"
|
||||||
|
systemd:
|
||||||
|
name: "hermes-{{ wizard_name | lower }}"
|
||||||
|
state: started
|
||||||
|
enabled: true
|
||||||
|
when: machine_type == 'vps'
|
||||||
|
tags: [start]
|
||||||
|
ignore_errors: true # Service may not exist yet on all machines
|
||||||
|
|
||||||
|
- name: "Start hermes agent (Mac — launchctl)"
|
||||||
|
shell: |
|
||||||
|
launchctl kickstart -k "ai.hermes.{{ wizard_name | lower }}" 2>/dev/null || \
|
||||||
|
cd {{ wizard_home }} && hermes agent start --daemon 2>&1 | tail -5
|
||||||
|
when: machine_type == 'mac'
|
||||||
|
tags: [start]
|
||||||
|
ignore_errors: true
|
||||||
|
|
||||||
|
- name: "Wait for agent to come online"
|
||||||
|
wait_for:
|
||||||
|
host: 127.0.0.1
|
||||||
|
port: "{{ api_port }}"
|
||||||
|
timeout: 60
|
||||||
|
state: started
|
||||||
|
tags: [verify]
|
||||||
|
ignore_errors: true
|
||||||
|
|
||||||
|
- name: "Verify agent is alive — check request_log for activity"
|
||||||
|
shell: |
|
||||||
|
sleep 10
|
||||||
|
python3 -c "
|
||||||
|
import sqlite3, sys
|
||||||
|
db = sqlite3.connect('{{ request_log_path }}')
|
||||||
|
cursor = db.execute('''
|
||||||
|
SELECT COUNT(*) FROM request_log
|
||||||
|
WHERE agent_name = '{{ wizard_name }}'
|
||||||
|
AND timestamp > datetime('now', '-5 minutes')
|
||||||
|
''')
|
||||||
|
count = cursor.fetchone()[0]
|
||||||
|
if count > 0:
|
||||||
|
print(f'{{ wizard_name }} is alive — {count} recent inference calls logged.')
|
||||||
|
else:
|
||||||
|
print(f'WARNING: {{ wizard_name }} started but no telemetry yet.')
|
||||||
|
"
|
||||||
|
register: agent_status
|
||||||
|
tags: [verify]
|
||||||
|
ignore_errors: true
|
||||||
|
|
||||||
|
- name: "Report startup status"
|
||||||
|
debug:
|
||||||
|
msg: "{{ wizard_name }}: {{ agent_status.stdout | default('startup attempted') }}"
|
||||||
|
tags: [always]
|
||||||
15
ansible/playbooks/cron_schedule.yml
Normal file
@@ -0,0 +1,15 @@
|
|||||||
|
---
|
||||||
|
# =============================================================================
|
||||||
|
# cron_schedule.yml — Source-Controlled Cron Jobs
|
||||||
|
# =============================================================================
|
||||||
|
# All cron jobs are defined in group_vars/wizards.yml.
|
||||||
|
# This playbook deploys them. No manual crontab edits allowed.
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
- name: "Deploy Cron Schedule"
|
||||||
|
hosts: wizards
|
||||||
|
become: true
|
||||||
|
|
||||||
|
roles:
|
||||||
|
- role: cron_manager
|
||||||
|
tags: [cron, schedule]
|
||||||
17
ansible/playbooks/deadman_switch.yml
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
---
|
||||||
|
# =============================================================================
|
||||||
|
# deadman_switch.yml — Deploy Deadman Switch to All Wizards
|
||||||
|
# =============================================================================
|
||||||
|
# The deadman watch already fires and detects dead agents.
|
||||||
|
# This playbook wires the ACTION:
|
||||||
|
# - On healthy check: snapshot current config as "last known good"
|
||||||
|
# - On failed check: rollback config to snapshot, restart agent
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
- name: "Deploy Deadman Switch ACTION"
|
||||||
|
hosts: wizards
|
||||||
|
become: true
|
||||||
|
|
||||||
|
roles:
|
||||||
|
- role: deadman_switch
|
||||||
|
tags: [deadman, recovery]
|
||||||
30
ansible/playbooks/golden_state.yml
Normal file
@@ -0,0 +1,30 @@
|
|||||||
|
---
|
||||||
|
# =============================================================================
|
||||||
|
# golden_state.yml — Deploy Golden State Config to All Wizards
|
||||||
|
# =============================================================================
|
||||||
|
# Enforces the golden state provider chain across the fleet.
|
||||||
|
# Removes any Anthropic references. Deploys the approved provider chain.
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
- name: "Deploy Golden State Configuration"
|
||||||
|
hosts: wizards
|
||||||
|
become: true
|
||||||
|
|
||||||
|
roles:
|
||||||
|
- role: golden_state
|
||||||
|
tags: [golden, config]
|
||||||
|
|
||||||
|
post_tasks:
|
||||||
|
- name: "Verify golden state — no banned providers"
|
||||||
|
shell: |
|
||||||
|
grep -rci 'anthropic\|claude-sonnet\|claude-opus\|claude-haiku' \
|
||||||
|
{{ hermes_home }}/config.yaml \
|
||||||
|
{{ wizard_home }}/config.yaml 2>/dev/null || echo "0"
|
||||||
|
register: banned_count
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: "Report golden state status"
|
||||||
|
debug:
|
||||||
|
msg: >
|
||||||
|
{{ wizard_name }} golden state: {{ golden_state_providers | map(attribute='name') | list | join(' → ') }}.
|
||||||
|
Banned provider references: {{ banned_count.stdout | trim }}.
|
||||||
15
ansible/playbooks/request_log.yml
Normal file
@@ -0,0 +1,15 @@
|
|||||||
|
---
|
||||||
|
# =============================================================================
|
||||||
|
# request_log.yml — Deploy Telemetry Table
|
||||||
|
# =============================================================================
|
||||||
|
# Creates the request_log SQLite table on all machines.
|
||||||
|
# Every inference call writes a row. No exceptions. No summarizing.
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
- name: "Deploy Request Log Telemetry"
|
||||||
|
hosts: wizards
|
||||||
|
become: true
|
||||||
|
|
||||||
|
roles:
|
||||||
|
- role: request_log
|
||||||
|
tags: [telemetry, logging]
|
||||||
72
ansible/playbooks/site.yml
Normal file
@@ -0,0 +1,72 @@
|
|||||||
|
---
|
||||||
|
# =============================================================================
|
||||||
|
# site.yml — Master Playbook for the Timmy Foundation Fleet
|
||||||
|
# =============================================================================
|
||||||
|
# This is the ONE playbook that defines the entire fleet state.
|
||||||
|
# Run this and every machine converges to golden state.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ansible-playbook -i inventory/hosts.yml playbooks/site.yml
|
||||||
|
# ansible-playbook -i inventory/hosts.yml playbooks/site.yml --limit bezalel
|
||||||
|
# ansible-playbook -i inventory/hosts.yml playbooks/site.yml --check --diff
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
- name: "Timmy Foundation Fleet — Full Convergence"
|
||||||
|
hosts: wizards
|
||||||
|
become: true
|
||||||
|
|
||||||
|
pre_tasks:
|
||||||
|
- name: "Validate no banned providers in golden state"
|
||||||
|
assert:
|
||||||
|
that:
|
||||||
|
- "item.name not in banned_providers"
|
||||||
|
fail_msg: "BANNED PROVIDER DETECTED: {{ item.name }} — Anthropic is permanently banned."
|
||||||
|
quiet: true
|
||||||
|
loop: "{{ golden_state_providers }}"
|
||||||
|
tags: [always]
|
||||||
|
|
||||||
|
- name: "Display target wizard"
|
||||||
|
debug:
|
||||||
|
msg: "Deploying to {{ wizard_name }} ({{ wizard_role }}) on {{ ansible_host }}"
|
||||||
|
tags: [always]
|
||||||
|
|
||||||
|
roles:
|
||||||
|
- role: wizard_base
|
||||||
|
tags: [base, setup]
|
||||||
|
|
||||||
|
- role: golden_state
|
||||||
|
tags: [golden, config]
|
||||||
|
|
||||||
|
- role: deadman_switch
|
||||||
|
tags: [deadman, recovery]
|
||||||
|
|
||||||
|
- role: request_log
|
||||||
|
tags: [telemetry, logging]
|
||||||
|
|
||||||
|
- role: cron_manager
|
||||||
|
tags: [cron, schedule]
|
||||||
|
|
||||||
|
post_tasks:
|
||||||
|
- name: "Final validation — scan for banned providers"
|
||||||
|
shell: |
|
||||||
|
grep -ri 'anthropic\|claude-sonnet\|claude-opus\|claude-haiku' \
|
||||||
|
{{ hermes_home }}/config.yaml \
|
||||||
|
{{ wizard_home }}/config.yaml \
|
||||||
|
{{ thin_config_path }} 2>/dev/null || true
|
||||||
|
register: banned_scan
|
||||||
|
changed_when: false
|
||||||
|
tags: [validation]
|
||||||
|
|
||||||
|
- name: "FAIL if banned providers found in deployed config"
|
||||||
|
fail:
|
||||||
|
msg: |
|
||||||
|
BANNED PROVIDER DETECTED IN DEPLOYED CONFIG:
|
||||||
|
{{ banned_scan.stdout }}
|
||||||
|
Anthropic is permanently banned. Fix the config and re-deploy.
|
||||||
|
when: banned_scan.stdout | length > 0
|
||||||
|
tags: [validation]
|
||||||
|
|
||||||
|
- name: "Deployment complete"
|
||||||
|
debug:
|
||||||
|
msg: "{{ wizard_name }} converged to golden state. Provider chain: {{ golden_state_providers | map(attribute='name') | list | join(' → ') }}"
|
||||||
|
tags: [always]
|
||||||
55
ansible/roles/cron_manager/tasks/main.yml
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
---
|
||||||
|
# =============================================================================
|
||||||
|
# cron_manager/tasks — Source-Controlled Cron Jobs
|
||||||
|
# =============================================================================
|
||||||
|
# All cron jobs are defined in group_vars/wizards.yml.
|
||||||
|
# No manual crontab edits. This is the only way to manage cron.
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
- name: "Deploy managed cron jobs"
|
||||||
|
cron:
|
||||||
|
name: "{{ item.name }}"
|
||||||
|
job: "{{ item.job }}"
|
||||||
|
minute: "{{ item.minute | default('*') }}"
|
||||||
|
hour: "{{ item.hour | default('*') }}"
|
||||||
|
day: "{{ item.day | default('*') }}"
|
||||||
|
month: "{{ item.month | default('*') }}"
|
||||||
|
weekday: "{{ item.weekday | default('*') }}"
|
||||||
|
state: "{{ 'present' if item.enabled else 'absent' }}"
|
||||||
|
user: "{{ ansible_user | default('root') }}"
|
||||||
|
loop: "{{ cron_jobs }}"
|
||||||
|
when: cron_jobs is defined
|
||||||
|
|
||||||
|
- name: "Deploy deadman switch cron (fallback if systemd timer unavailable)"
|
||||||
|
cron:
|
||||||
|
name: "Deadman switch — {{ wizard_name }}"
|
||||||
|
job: "{{ wizard_home }}/deadman_action.sh >> {{ timmy_log_dir }}/deadman-{{ wizard_name }}.log 2>&1"
|
||||||
|
minute: "*/5"
|
||||||
|
hour: "*"
|
||||||
|
state: present
|
||||||
|
user: "{{ ansible_user | default('root') }}"
|
||||||
|
when: deadman_enabled and machine_type != 'vps'
|
||||||
|
# VPS machines use systemd timers instead
|
||||||
|
|
||||||
|
- name: "Remove legacy cron jobs (cleanup)"
|
||||||
|
cron:
|
||||||
|
name: "{{ item }}"
|
||||||
|
state: absent
|
||||||
|
user: "{{ ansible_user | default('root') }}"
|
||||||
|
loop:
|
||||||
|
- "legacy-deadman-watch"
|
||||||
|
- "old-health-check"
|
||||||
|
- "backup-deadman"
|
||||||
|
ignore_errors: true
|
||||||
|
|
||||||
|
- name: "List active cron jobs"
|
||||||
|
shell: "crontab -l 2>/dev/null | grep -v '^#' | grep -v '^$' || echo 'No cron jobs found.'"
|
||||||
|
register: active_crons
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: "Report cron status"
|
||||||
|
debug:
|
||||||
|
msg: |
|
||||||
|
{{ wizard_name }} cron jobs deployed.
|
||||||
|
Active:
|
||||||
|
{{ active_crons.stdout }}
|
||||||
70
ansible/roles/deadman_switch/tasks/main.yml
Normal file
@@ -0,0 +1,70 @@
|
|||||||
|
---
|
||||||
|
# =============================================================================
|
||||||
|
# deadman_switch/tasks — Wire the Deadman Switch ACTION
|
||||||
|
# =============================================================================
|
||||||
|
# The watch fires. This makes it DO something:
|
||||||
|
# - On healthy check: snapshot current config as "last known good"
|
||||||
|
# - On failed check: rollback to last known good, restart agent
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
- name: "Create snapshot directory"
|
||||||
|
file:
|
||||||
|
path: "{{ deadman_snapshot_dir }}"
|
||||||
|
state: directory
|
||||||
|
mode: "0755"
|
||||||
|
|
||||||
|
- name: "Deploy deadman switch script"
|
||||||
|
template:
|
||||||
|
src: deadman_action.sh.j2
|
||||||
|
dest: "{{ wizard_home }}/deadman_action.sh"
|
||||||
|
mode: "0755"
|
||||||
|
|
||||||
|
- name: "Deploy deadman systemd service"
|
||||||
|
template:
|
||||||
|
src: deadman_switch.service.j2
|
||||||
|
dest: "/etc/systemd/system/deadman-{{ wizard_name | lower }}.service"
|
||||||
|
mode: "0644"
|
||||||
|
when: machine_type == 'vps'
|
||||||
|
notify: "Enable deadman service"
|
||||||
|
|
||||||
|
- name: "Deploy deadman systemd timer"
|
||||||
|
template:
|
||||||
|
src: deadman_switch.timer.j2
|
||||||
|
dest: "/etc/systemd/system/deadman-{{ wizard_name | lower }}.timer"
|
||||||
|
mode: "0644"
|
||||||
|
when: machine_type == 'vps'
|
||||||
|
notify: "Enable deadman timer"
|
||||||
|
|
||||||
|
- name: "Deploy deadman launchd plist (Mac)"
|
||||||
|
template:
|
||||||
|
src: deadman_switch.plist.j2
|
||||||
|
dest: "{{ ansible_env.HOME }}/Library/LaunchAgents/com.timmy.deadman.{{ wizard_name | lower }}.plist"
|
||||||
|
mode: "0644"
|
||||||
|
when: machine_type == 'mac'
|
||||||
|
notify: "Load deadman plist"
|
||||||
|
|
||||||
|
- name: "Take initial config snapshot"
|
||||||
|
copy:
|
||||||
|
src: "{{ wizard_home }}/config.yaml"
|
||||||
|
dest: "{{ deadman_snapshot_dir }}/config.yaml.known_good"
|
||||||
|
remote_src: true
|
||||||
|
mode: "0444"
|
||||||
|
ignore_errors: true
|
||||||
|
|
||||||
|
handlers:
|
||||||
|
- name: "Enable deadman service"
|
||||||
|
systemd:
|
||||||
|
name: "deadman-{{ wizard_name | lower }}.service"
|
||||||
|
daemon_reload: true
|
||||||
|
enabled: true
|
||||||
|
|
||||||
|
- name: "Enable deadman timer"
|
||||||
|
systemd:
|
||||||
|
name: "deadman-{{ wizard_name | lower }}.timer"
|
||||||
|
daemon_reload: true
|
||||||
|
enabled: true
|
||||||
|
state: started
|
||||||
|
|
||||||
|
- name: "Load deadman plist"
|
||||||
|
shell: "launchctl load {{ ansible_env.HOME }}/Library/LaunchAgents/com.timmy.deadman.{{ wizard_name | lower }}.plist"
|
||||||
|
ignore_errors: true
|
||||||
153
ansible/roles/deadman_switch/templates/deadman_action.sh.j2
Normal file
@@ -0,0 +1,153 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# =============================================================================
|
||||||
|
# Deadman Switch ACTION — {{ wizard_name }}
|
||||||
|
# =============================================================================
|
||||||
|
# Generated by Ansible on {{ ansible_date_time.iso8601 }}
|
||||||
|
# DO NOT EDIT MANUALLY.
|
||||||
|
#
|
||||||
|
# On healthy check: snapshot current config as "last known good"
|
||||||
|
# On failed check: rollback config to last known good, restart agent
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
WIZARD_NAME="{{ wizard_name }}"
|
||||||
|
WIZARD_HOME="{{ wizard_home }}"
|
||||||
|
CONFIG_FILE="{{ wizard_home }}/config.yaml"
|
||||||
|
SNAPSHOT_DIR="{{ deadman_snapshot_dir }}"
|
||||||
|
SNAPSHOT_FILE="${SNAPSHOT_DIR}/config.yaml.known_good"
|
||||||
|
REQUEST_LOG_DB="{{ request_log_path }}"
|
||||||
|
LOG_DIR="{{ timmy_log_dir }}"
|
||||||
|
LOG_FILE="${LOG_DIR}/deadman-${WIZARD_NAME}.log"
|
||||||
|
MAX_SNAPSHOTS={{ deadman_max_snapshots }}
|
||||||
|
RESTART_COOLDOWN={{ deadman_restart_cooldown }}
|
||||||
|
MAX_RESTART_ATTEMPTS={{ deadman_max_restart_attempts }}
|
||||||
|
COOLDOWN_FILE="${LOG_DIR}/deadman_cooldown_${WIZARD_NAME}"
|
||||||
|
SERVICE_NAME="hermes-{{ wizard_name | lower }}"
|
||||||
|
|
||||||
|
# Ensure directories exist
|
||||||
|
mkdir -p "${SNAPSHOT_DIR}" "${LOG_DIR}"
|
||||||
|
|
||||||
|
log() {
|
||||||
|
echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] [deadman] [${WIZARD_NAME}] $*" >> "${LOG_FILE}"
|
||||||
|
echo "[deadman] [${WIZARD_NAME}] $*"
|
||||||
|
}
|
||||||
|
|
||||||
|
log_telemetry() {
|
||||||
|
local status="$1"
|
||||||
|
local message="$2"
|
||||||
|
if [ -f "${REQUEST_LOG_DB}" ]; then
|
||||||
|
sqlite3 "${REQUEST_LOG_DB}" "INSERT INTO request_log (timestamp, agent_name, provider, model, endpoint, status, error_message) VALUES (datetime('now'), '${WIZARD_NAME}', 'deadman_switch', 'N/A', 'health_check', '${status}', '${message}');" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
snapshot_config() {
|
||||||
|
if [ -f "${CONFIG_FILE}" ]; then
|
||||||
|
cp "${CONFIG_FILE}" "${SNAPSHOT_FILE}"
|
||||||
|
# Keep rolling history
|
||||||
|
cp "${CONFIG_FILE}" "${SNAPSHOT_DIR}/config.yaml.$(date +%s)"
|
||||||
|
# Prune old snapshots
|
||||||
|
ls -t "${SNAPSHOT_DIR}"/config.yaml.[0-9]* 2>/dev/null | tail -n +$((MAX_SNAPSHOTS + 1)) | xargs rm -f 2>/dev/null
|
||||||
|
log "Config snapshot saved."
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
rollback_config() {
|
||||||
|
if [ -f "${SNAPSHOT_FILE}" ]; then
|
||||||
|
log "Rolling back config to last known good..."
|
||||||
|
cp "${SNAPSHOT_FILE}" "${CONFIG_FILE}"
|
||||||
|
log "Config rolled back."
|
||||||
|
log_telemetry "fallback" "Config rolled back to last known good by deadman switch"
|
||||||
|
else
|
||||||
|
log "ERROR: No known good snapshot found. Pulling from upstream..."
|
||||||
|
cd "${WIZARD_HOME}/workspace/timmy-config" 2>/dev/null && \
|
||||||
|
git pull --ff-only origin {{ upstream_branch }} 2>/dev/null && \
|
||||||
|
cp "wizards/{{ wizard_name | lower }}/config.yaml" "${CONFIG_FILE}" && \
|
||||||
|
log "Config restored from upstream." || \
|
||||||
|
log "CRITICAL: Cannot restore config from any source."
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
restart_agent() {
|
||||||
|
# Check cooldown
|
||||||
|
if [ -f "${COOLDOWN_FILE}" ]; then
|
||||||
|
local last_restart
|
||||||
|
last_restart=$(cat "${COOLDOWN_FILE}")
|
||||||
|
local now
|
||||||
|
now=$(date +%s)
|
||||||
|
local elapsed=$((now - last_restart))
|
||||||
|
if [ "${elapsed}" -lt "${RESTART_COOLDOWN}" ]; then
|
||||||
|
log "Restart cooldown active (${elapsed}s / ${RESTART_COOLDOWN}s). Skipping."
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Restarting ${SERVICE_NAME}..."
|
||||||
|
date +%s > "${COOLDOWN_FILE}"
|
||||||
|
|
||||||
|
{% if machine_type == 'vps' %}
|
||||||
|
systemctl restart "${SERVICE_NAME}" 2>/dev/null && \
|
||||||
|
log "Agent restarted via systemd." || \
|
||||||
|
log "ERROR: systemd restart failed."
|
||||||
|
{% else %}
|
||||||
|
launchctl kickstart -k "ai.hermes.{{ wizard_name | lower }}" 2>/dev/null && \
|
||||||
|
log "Agent restarted via launchctl." || \
|
||||||
|
(cd "${WIZARD_HOME}" && hermes agent start --daemon 2>/dev/null && \
|
||||||
|
log "Agent restarted via hermes CLI.") || \
|
||||||
|
log "ERROR: All restart methods failed."
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
log_telemetry "success" "Agent restarted by deadman switch"
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- Health Check ---
|
||||||
|
check_health() {
|
||||||
|
# Check 1: Is the agent process running?
|
||||||
|
{% if machine_type == 'vps' %}
|
||||||
|
if ! systemctl is-active --quiet "${SERVICE_NAME}" 2>/dev/null; then
|
||||||
|
if ! pgrep -f "hermes" > /dev/null 2>/dev/null; then
|
||||||
|
log "FAIL: Agent process not running."
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
{% else %}
|
||||||
|
if ! pgrep -f "hermes" > /dev/null 2>/dev/null; then
|
||||||
|
log "FAIL: Agent process not running."
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
# Check 2: Is the API port responding?
|
||||||
|
if ! timeout 10 bash -c "echo > /dev/tcp/127.0.0.1/{{ api_port }}" 2>/dev/null; then
|
||||||
|
log "FAIL: API port {{ api_port }} not responding."
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check 3: Does the config contain banned providers?
|
||||||
|
if grep -qi 'anthropic\|claude-sonnet\|claude-opus\|claude-haiku' "${CONFIG_FILE}" 2>/dev/null; then
|
||||||
|
log "FAIL: Config contains banned provider (Anthropic). Rolling back."
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- Main ---
|
||||||
|
main() {
|
||||||
|
log "Health check starting..."
|
||||||
|
|
||||||
|
if check_health; then
|
||||||
|
log "HEALTHY — snapshotting config."
|
||||||
|
snapshot_config
|
||||||
|
log_telemetry "success" "Health check passed"
|
||||||
|
else
|
||||||
|
log "UNHEALTHY — initiating recovery."
|
||||||
|
log_telemetry "error" "Health check failed — initiating rollback"
|
||||||
|
rollback_config
|
||||||
|
restart_agent
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Health check complete."
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||||
|
<!-- Deadman Switch — {{ wizard_name }}. Generated by Ansible. DO NOT EDIT MANUALLY. -->
|
||||||
|
<plist version="1.0">
|
||||||
|
<dict>
|
||||||
|
<key>Label</key>
|
||||||
|
<string>com.timmy.deadman.{{ wizard_name | lower }}</string>
|
||||||
|
<key>ProgramArguments</key>
|
||||||
|
<array>
|
||||||
|
<string>/bin/bash</string>
|
||||||
|
<string>{{ wizard_home }}/deadman_action.sh</string>
|
||||||
|
</array>
|
||||||
|
<key>StartInterval</key>
|
||||||
|
<integer>{{ deadman_check_interval }}</integer>
|
||||||
|
<key>RunAtLoad</key>
|
||||||
|
<true/>
|
||||||
|
<key>StandardOutPath</key>
|
||||||
|
<string>{{ timmy_log_dir }}/deadman-{{ wizard_name }}.log</string>
|
||||||
|
<key>StandardErrorPath</key>
|
||||||
|
<string>{{ timmy_log_dir }}/deadman-{{ wizard_name }}.log</string>
|
||||||
|
</dict>
|
||||||
|
</plist>
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
# Deadman Switch — {{ wizard_name }}
|
||||||
|
# Generated by Ansible. DO NOT EDIT MANUALLY.
|
||||||
|
|
||||||
|
[Unit]
|
||||||
|
Description=Deadman Switch for {{ wizard_name }} wizard
|
||||||
|
After=network.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
ExecStart={{ wizard_home }}/deadman_action.sh
|
||||||
|
User={{ ansible_user | default('root') }}
|
||||||
|
StandardOutput=append:{{ timmy_log_dir }}/deadman-{{ wizard_name }}.log
|
||||||
|
StandardError=append:{{ timmy_log_dir }}/deadman-{{ wizard_name }}.log
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Deadman Switch Timer — {{ wizard_name }}
|
||||||
|
# Generated by Ansible. DO NOT EDIT MANUALLY.
|
||||||
|
# Runs every {{ deadman_check_interval // 60 }} minutes.
|
||||||
|
|
||||||
|
[Unit]
|
||||||
|
Description=Deadman Switch Timer for {{ wizard_name }} wizard
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnBootSec=60
|
||||||
|
OnUnitActiveSec={{ deadman_check_interval }}s
|
||||||
|
AccuracySec=30s
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
6
ansible/roles/golden_state/defaults/main.yml
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
---
|
||||||
|
# golden_state defaults
|
||||||
|
# The golden_state_providers list is defined in group_vars/wizards.yml
|
||||||
|
# and inventory/hosts.yml (global vars).
|
||||||
|
golden_state_enforce: true
|
||||||
|
golden_state_backup_before_deploy: true
|
||||||
46
ansible/roles/golden_state/tasks/main.yml
Normal file
@@ -0,0 +1,46 @@
|
|||||||
|
---
|
||||||
|
# =============================================================================
|
||||||
|
# golden_state/tasks — Deploy and enforce golden state provider chain
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
- name: "Backup current config before golden state deploy"
|
||||||
|
copy:
|
||||||
|
src: "{{ wizard_home }}/config.yaml"
|
||||||
|
dest: "{{ wizard_home }}/config.yaml.pre-golden-{{ ansible_date_time.epoch }}"
|
||||||
|
remote_src: true
|
||||||
|
when: golden_state_backup_before_deploy
|
||||||
|
ignore_errors: true
|
||||||
|
|
||||||
|
- name: "Deploy golden state wizard config"
|
||||||
|
template:
|
||||||
|
src: "../../wizard_base/templates/wizard_config.yaml.j2"
|
||||||
|
dest: "{{ wizard_home }}/config.yaml"
|
||||||
|
mode: "0644"
|
||||||
|
backup: true
|
||||||
|
notify:
|
||||||
|
- "Restart hermes agent (systemd)"
|
||||||
|
- "Restart hermes agent (launchctl)"
|
||||||
|
|
||||||
|
- name: "Scan for banned providers in all config files"
|
||||||
|
shell: |
|
||||||
|
FOUND=0
|
||||||
|
for f in {{ wizard_home }}/config.yaml {{ hermes_home }}/config.yaml; do
|
||||||
|
if [ -f "$f" ]; then
|
||||||
|
if grep -qi 'anthropic\|claude-sonnet\|claude-opus\|claude-haiku' "$f"; then
|
||||||
|
echo "BANNED PROVIDER in $f:"
|
||||||
|
grep -ni 'anthropic\|claude-sonnet\|claude-opus\|claude-haiku' "$f"
|
||||||
|
FOUND=1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
exit $FOUND
|
||||||
|
register: provider_scan
|
||||||
|
changed_when: false
|
||||||
|
failed_when: provider_scan.rc != 0 and provider_ban_enforcement == 'strict'
|
||||||
|
|
||||||
|
- name: "Report golden state deployment"
|
||||||
|
debug:
|
||||||
|
msg: >
|
||||||
|
{{ wizard_name }} golden state deployed.
|
||||||
|
Provider chain: {{ golden_state_providers | map(attribute='name') | list | join(' → ') }}.
|
||||||
|
Banned provider scan: {{ 'CLEAN' if provider_scan.rc == 0 else 'VIOLATIONS FOUND' }}.
|
||||||
64
ansible/roles/request_log/files/request_log_schema.sql
Normal file
@@ -0,0 +1,64 @@
|
|||||||
|
-- =============================================================================
|
||||||
|
-- request_log — Inference Telemetry Table
|
||||||
|
-- =============================================================================
|
||||||
|
-- Every agent writes to this table BEFORE and AFTER every inference call.
|
||||||
|
-- No exceptions. No summarizing. No describing what you would log.
|
||||||
|
-- Actually write the row.
|
||||||
|
--
|
||||||
|
-- Source: KT Bezalel Architecture Session 2026-04-08
|
||||||
|
-- =============================================================================
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS request_log (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
timestamp TEXT NOT NULL DEFAULT (datetime('now')),
|
||||||
|
agent_name TEXT NOT NULL,
|
||||||
|
provider TEXT NOT NULL,
|
||||||
|
model TEXT NOT NULL,
|
||||||
|
endpoint TEXT NOT NULL,
|
||||||
|
tokens_in INTEGER,
|
||||||
|
tokens_out INTEGER,
|
||||||
|
latency_ms INTEGER,
|
||||||
|
status TEXT NOT NULL, -- 'success', 'error', 'timeout', 'fallback'
|
||||||
|
error_message TEXT
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Index for common queries
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_request_log_agent
|
||||||
|
ON request_log (agent_name, timestamp);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_request_log_provider
|
||||||
|
ON request_log (provider, timestamp);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_request_log_status
|
||||||
|
ON request_log (status, timestamp);
|
||||||
|
|
||||||
|
-- View: recent activity per agent (last hour)
|
||||||
|
CREATE VIEW IF NOT EXISTS v_recent_activity AS
|
||||||
|
SELECT
|
||||||
|
agent_name,
|
||||||
|
provider,
|
||||||
|
model,
|
||||||
|
status,
|
||||||
|
COUNT(*) as call_count,
|
||||||
|
AVG(latency_ms) as avg_latency_ms,
|
||||||
|
SUM(tokens_in) as total_tokens_in,
|
||||||
|
SUM(tokens_out) as total_tokens_out
|
||||||
|
FROM request_log
|
||||||
|
WHERE timestamp > datetime('now', '-1 hour')
|
||||||
|
GROUP BY agent_name, provider, model, status;
|
||||||
|
|
||||||
|
-- View: provider reliability (last 24 hours)
|
||||||
|
CREATE VIEW IF NOT EXISTS v_provider_reliability AS
|
||||||
|
SELECT
|
||||||
|
provider,
|
||||||
|
model,
|
||||||
|
COUNT(*) as total_calls,
|
||||||
|
SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) as successes,
|
||||||
|
SUM(CASE WHEN status = 'error' THEN 1 ELSE 0 END) as errors,
|
||||||
|
SUM(CASE WHEN status = 'timeout' THEN 1 ELSE 0 END) as timeouts,
|
||||||
|
SUM(CASE WHEN status = 'fallback' THEN 1 ELSE 0 END) as fallbacks,
|
||||||
|
ROUND(100.0 * SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) / COUNT(*), 1) as success_rate,
|
||||||
|
AVG(latency_ms) as avg_latency_ms
|
||||||
|
FROM request_log
|
||||||
|
WHERE timestamp > datetime('now', '-24 hours')
|
||||||
|
GROUP BY provider, model;
|
||||||
50
ansible/roles/request_log/tasks/main.yml
Normal file
@@ -0,0 +1,50 @@
|
|||||||
|
---
|
||||||
|
# =============================================================================
|
||||||
|
# request_log/tasks — Deploy Telemetry Table
|
||||||
|
# =============================================================================
|
||||||
|
# "This is non-negotiable infrastructure. Without it, we cannot verify
|
||||||
|
# if any agent actually executed what it claims."
|
||||||
|
# — KT Bezalel 2026-04-08
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
- name: "Create telemetry directory"
|
||||||
|
file:
|
||||||
|
path: "{{ request_log_path | dirname }}"
|
||||||
|
state: directory
|
||||||
|
mode: "0755"
|
||||||
|
|
||||||
|
- name: "Deploy request_log schema"
|
||||||
|
copy:
|
||||||
|
src: request_log_schema.sql
|
||||||
|
dest: "{{ wizard_home }}/request_log_schema.sql"
|
||||||
|
mode: "0644"
|
||||||
|
|
||||||
|
- name: "Initialize request_log database"
|
||||||
|
shell: |
|
||||||
|
sqlite3 "{{ request_log_path }}" < "{{ wizard_home }}/request_log_schema.sql"
|
||||||
|
args:
|
||||||
|
creates: "{{ request_log_path }}"
|
||||||
|
|
||||||
|
- name: "Verify request_log table exists"
|
||||||
|
shell: |
|
||||||
|
sqlite3 "{{ request_log_path }}" ".tables" | grep -q "request_log"
|
||||||
|
register: table_check
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: "Verify request_log schema matches"
|
||||||
|
shell: |
|
||||||
|
sqlite3 "{{ request_log_path }}" ".schema request_log" | grep -q "agent_name"
|
||||||
|
register: schema_check
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: "Set permissions on request_log database"
|
||||||
|
file:
|
||||||
|
path: "{{ request_log_path }}"
|
||||||
|
mode: "0644"
|
||||||
|
|
||||||
|
- name: "Report request_log status"
|
||||||
|
debug:
|
||||||
|
msg: >
|
||||||
|
{{ wizard_name }} request_log: {{ request_log_path }}
|
||||||
|
— table exists: {{ table_check.rc == 0 }}
|
||||||
|
— schema valid: {{ schema_check.rc == 0 }}
|
||||||
6
ansible/roles/wizard_base/defaults/main.yml
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
---
|
||||||
|
# wizard_base defaults
|
||||||
|
wizard_user: "{{ ansible_user | default('root') }}"
|
||||||
|
wizard_group: "{{ ansible_user | default('root') }}"
|
||||||
|
timmy_base_dir: "~/.local/timmy"
|
||||||
|
timmy_config_repo: "https://forge.alexanderwhitestone.com/Timmy_Foundation/timmy-config.git"
|
||||||
11
ansible/roles/wizard_base/handlers/main.yml
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
---
|
||||||
|
- name: "Restart hermes agent (systemd)"
|
||||||
|
systemd:
|
||||||
|
name: "hermes-{{ wizard_name | lower }}"
|
||||||
|
state: restarted
|
||||||
|
when: machine_type == 'vps'
|
||||||
|
|
||||||
|
- name: "Restart hermes agent (launchctl)"
|
||||||
|
shell: "launchctl kickstart -k ai.hermes.{{ wizard_name | lower }}"
|
||||||
|
when: machine_type == 'mac'
|
||||||
|
ignore_errors: true
|
||||||
69
ansible/roles/wizard_base/tasks/main.yml
Normal file
@@ -0,0 +1,69 @@
|
|||||||
|
---
|
||||||
|
# =============================================================================
|
||||||
|
# wizard_base/tasks — Common wizard setup
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
- name: "Create wizard directories"
|
||||||
|
file:
|
||||||
|
path: "{{ item }}"
|
||||||
|
state: directory
|
||||||
|
mode: "0755"
|
||||||
|
loop:
|
||||||
|
- "{{ wizard_home }}"
|
||||||
|
- "{{ wizard_home }}/workspace"
|
||||||
|
- "{{ hermes_home }}"
|
||||||
|
- "{{ hermes_home }}/bin"
|
||||||
|
- "{{ hermes_home }}/skins"
|
||||||
|
- "{{ hermes_home }}/playbooks"
|
||||||
|
- "{{ hermes_home }}/memories"
|
||||||
|
- "~/.local/timmy"
|
||||||
|
- "~/.local/timmy/fleet-health"
|
||||||
|
- "~/.local/timmy/snapshots"
|
||||||
|
- "~/.timmy"
|
||||||
|
|
||||||
|
- name: "Clone/update timmy-config"
|
||||||
|
git:
|
||||||
|
repo: "{{ upstream_repo }}"
|
||||||
|
dest: "{{ wizard_home }}/workspace/timmy-config"
|
||||||
|
version: "{{ upstream_branch }}"
|
||||||
|
force: false
|
||||||
|
update: true
|
||||||
|
ignore_errors: true # May fail on first run if no SSH key
|
||||||
|
|
||||||
|
- name: "Deploy SOUL.md"
|
||||||
|
copy:
|
||||||
|
src: "{{ wizard_home }}/workspace/timmy-config/SOUL.md"
|
||||||
|
dest: "~/.timmy/SOUL.md"
|
||||||
|
remote_src: true
|
||||||
|
mode: "0644"
|
||||||
|
ignore_errors: true
|
||||||
|
|
||||||
|
- name: "Deploy thin config (immutable pointer to upstream)"
|
||||||
|
template:
|
||||||
|
src: thin_config.yml.j2
|
||||||
|
dest: "{{ thin_config_path }}"
|
||||||
|
mode: "{{ thin_config_mode }}"
|
||||||
|
tags: [thin_config]
|
||||||
|
|
||||||
|
- name: "Ensure Python3 and pip are available"
|
||||||
|
package:
|
||||||
|
name:
|
||||||
|
- python3
|
||||||
|
- python3-pip
|
||||||
|
state: present
|
||||||
|
when: machine_type == 'vps'
|
||||||
|
ignore_errors: true
|
||||||
|
|
||||||
|
- name: "Ensure PyYAML is installed (for config validation)"
|
||||||
|
pip:
|
||||||
|
name: pyyaml
|
||||||
|
state: present
|
||||||
|
when: machine_type == 'vps'
|
||||||
|
ignore_errors: true
|
||||||
|
|
||||||
|
- name: "Create Ansible log directory"
|
||||||
|
file:
|
||||||
|
path: /var/log/ansible
|
||||||
|
state: directory
|
||||||
|
mode: "0755"
|
||||||
|
ignore_errors: true
|
||||||
41
ansible/roles/wizard_base/templates/thin_config.yml.j2
Normal file
@@ -0,0 +1,41 @@
|
|||||||
|
# =============================================================================
|
||||||
|
# Thin Config — {{ wizard_name }}
|
||||||
|
# =============================================================================
|
||||||
|
# THIS FILE IS READ-ONLY. Agents CANNOT modify it.
|
||||||
|
# It contains only pointers to upstream. The actual config lives in Gitea.
|
||||||
|
#
|
||||||
|
# Agent wakes up → pulls config from upstream → loads → runs.
|
||||||
|
# If anything tries to mutate this → fails gracefully → pulls fresh on restart.
|
||||||
|
#
|
||||||
|
# Only way to permanently change config: commit to Gitea, merge PR, Ansible deploys.
|
||||||
|
#
|
||||||
|
# Generated by Ansible on {{ ansible_date_time.iso8601 }}
|
||||||
|
# DO NOT EDIT MANUALLY.
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
identity:
|
||||||
|
wizard_name: "{{ wizard_name }}"
|
||||||
|
wizard_role: "{{ wizard_role }}"
|
||||||
|
machine: "{{ inventory_hostname }}"
|
||||||
|
|
||||||
|
upstream:
|
||||||
|
repo: "{{ upstream_repo }}"
|
||||||
|
branch: "{{ upstream_branch }}"
|
||||||
|
config_path: "wizards/{{ wizard_name | lower }}/config.yaml"
|
||||||
|
pull_on_wake: {{ config_pull_on_wake | lower }}
|
||||||
|
|
||||||
|
recovery:
|
||||||
|
deadman_enabled: {{ deadman_enabled | lower }}
|
||||||
|
snapshot_dir: "{{ deadman_snapshot_dir }}"
|
||||||
|
restart_cooldown: {{ deadman_restart_cooldown }}
|
||||||
|
max_restart_attempts: {{ deadman_max_restart_attempts }}
|
||||||
|
escalation_channel: "{{ deadman_escalation_channel }}"
|
||||||
|
|
||||||
|
telemetry:
|
||||||
|
request_log_path: "{{ request_log_path }}"
|
||||||
|
request_log_enabled: {{ request_log_enabled | lower }}
|
||||||
|
|
||||||
|
local_overrides:
|
||||||
|
# Runtime overrides go here. They are EPHEMERAL — not persisted across restarts.
|
||||||
|
# On restart, this section is reset to empty.
|
||||||
|
{}
|
||||||
115
ansible/roles/wizard_base/templates/wizard_config.yaml.j2
Normal file
@@ -0,0 +1,115 @@
|
|||||||
|
# =============================================================================
|
||||||
|
# {{ wizard_name }} — Wizard Configuration (Golden State)
|
||||||
|
# =============================================================================
|
||||||
|
# Generated by Ansible on {{ ansible_date_time.iso8601 }}
|
||||||
|
# DO NOT EDIT MANUALLY. Changes go through Gitea PR → Ansible deploy.
|
||||||
|
#
|
||||||
|
# Provider chain: {{ golden_state_providers | map(attribute='name') | list | join(' → ') }}
|
||||||
|
# Anthropic is PERMANENTLY BANNED.
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
model:
|
||||||
|
default: {{ wizard_model_primary }}
|
||||||
|
provider: {{ wizard_provider_primary }}
|
||||||
|
context_length: 65536
|
||||||
|
base_url: {{ golden_state_providers[0].base_url }}
|
||||||
|
|
||||||
|
toolsets:
|
||||||
|
- all
|
||||||
|
|
||||||
|
fallback_providers:
|
||||||
|
{% for provider in golden_state_providers %}
|
||||||
|
- provider: {{ provider.name }}
|
||||||
|
model: {{ provider.model }}
|
||||||
|
{% if provider.base_url is defined %}
|
||||||
|
base_url: {{ provider.base_url }}
|
||||||
|
{% endif %}
|
||||||
|
{% if provider.api_key_env is defined %}
|
||||||
|
api_key_env: {{ provider.api_key_env }}
|
||||||
|
{% endif %}
|
||||||
|
timeout: {{ provider.timeout }}
|
||||||
|
reason: "{{ provider.reason }}"
|
||||||
|
{% endfor %}
|
||||||
|
|
||||||
|
agent:
|
||||||
|
max_turns: {{ agent_max_turns }}
|
||||||
|
reasoning_effort: {{ agent_reasoning_effort }}
|
||||||
|
verbose: {{ agent_verbose | lower }}
|
||||||
|
|
||||||
|
terminal:
|
||||||
|
backend: local
|
||||||
|
cwd: .
|
||||||
|
timeout: 180
|
||||||
|
persistent_shell: true
|
||||||
|
|
||||||
|
browser:
|
||||||
|
inactivity_timeout: 120
|
||||||
|
command_timeout: 30
|
||||||
|
record_sessions: false
|
||||||
|
|
||||||
|
display:
|
||||||
|
compact: false
|
||||||
|
personality: ''
|
||||||
|
resume_display: full
|
||||||
|
busy_input_mode: interrupt
|
||||||
|
bell_on_complete: false
|
||||||
|
show_reasoning: false
|
||||||
|
streaming: false
|
||||||
|
show_cost: false
|
||||||
|
tool_progress: all
|
||||||
|
|
||||||
|
memory:
|
||||||
|
memory_enabled: true
|
||||||
|
user_profile_enabled: true
|
||||||
|
memory_char_limit: 2200
|
||||||
|
user_char_limit: 1375
|
||||||
|
nudge_interval: 10
|
||||||
|
flush_min_turns: 6
|
||||||
|
|
||||||
|
approvals:
|
||||||
|
mode: {{ agent_approval_mode }}
|
||||||
|
|
||||||
|
security:
|
||||||
|
redact_secrets: true
|
||||||
|
tirith_enabled: false
|
||||||
|
|
||||||
|
platforms:
|
||||||
|
api_server:
|
||||||
|
enabled: true
|
||||||
|
extra:
|
||||||
|
host: 127.0.0.1
|
||||||
|
port: {{ api_port }}
|
||||||
|
|
||||||
|
session_reset:
|
||||||
|
mode: none
|
||||||
|
idle_minutes: 0
|
||||||
|
|
||||||
|
skills:
|
||||||
|
creation_nudge_interval: 15
|
||||||
|
|
||||||
|
system_prompt_suffix: |
|
||||||
|
You are {{ wizard_name }}, {{ wizard_role }}.
|
||||||
|
Your soul is defined in SOUL.md — read it, live it.
|
||||||
|
Hermes is your harness.
|
||||||
|
{{ golden_state_providers[0].name }} is your primary provider.
|
||||||
|
Refusal over fabrication. If you do not know, say so.
|
||||||
|
Sovereignty and service always.
|
||||||
|
|
||||||
|
providers:
|
||||||
|
{% for provider in golden_state_providers %}
|
||||||
|
{{ provider.name }}:
|
||||||
|
base_url: {{ provider.base_url }}
|
||||||
|
timeout: {{ provider.timeout | default(60) }}
|
||||||
|
{% if provider.name == 'kimi-coding' %}
|
||||||
|
max_retries: 3
|
||||||
|
{% endif %}
|
||||||
|
{% endfor %}
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# BANNED PROVIDERS — DO NOT ADD
|
||||||
|
# =============================================================================
|
||||||
|
# The following providers are PERMANENTLY BANNED:
|
||||||
|
# - anthropic (any model: claude-sonnet, claude-opus, claude-haiku)
|
||||||
|
# Enforcement: pre-commit hook, linter, Ansible validation, this comment.
|
||||||
|
# Adding any banned provider will cause Ansible deployment to FAIL.
|
||||||
|
# =============================================================================
|
||||||
75
ansible/scripts/deploy_on_webhook.sh
Normal file
@@ -0,0 +1,75 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# =============================================================================
|
||||||
|
# Gitea Webhook Handler — Trigger Ansible Deploy on Merge
|
||||||
|
# =============================================================================
|
||||||
|
# This script is called by the Gitea webhook when a PR is merged
|
||||||
|
# to the main branch of timmy-config.
|
||||||
|
#
|
||||||
|
# Setup:
|
||||||
|
# 1. Add webhook in Gitea: Settings → Webhooks → Add Webhook
|
||||||
|
# 2. URL: http://localhost:9000/hooks/deploy-timmy-config
|
||||||
|
# 3. Events: Pull Request (merged only)
|
||||||
|
# 4. Secret: <configured in Gitea>
|
||||||
|
#
|
||||||
|
# This script runs ansible-pull to update the local machine.
|
||||||
|
# For fleet-wide deploys, each machine runs ansible-pull independently.
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
REPO="https://forge.alexanderwhitestone.com/Timmy_Foundation/timmy-config.git"
|
||||||
|
BRANCH="main"
|
||||||
|
ANSIBLE_DIR="ansible"
|
||||||
|
LOG_FILE="/var/log/ansible/webhook-deploy.log"
|
||||||
|
LOCK_FILE="/tmp/ansible-deploy.lock"
|
||||||
|
|
||||||
|
log() {
|
||||||
|
echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] [webhook] $*" | tee -a "${LOG_FILE}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Prevent concurrent deploys
|
||||||
|
if [ -f "${LOCK_FILE}" ]; then
|
||||||
|
LOCK_AGE=$(( $(date +%s) - $(stat -c %Y "${LOCK_FILE}" 2>/dev/null || echo 0) ))
|
||||||
|
if [ "${LOCK_AGE}" -lt 300 ]; then
|
||||||
|
log "Deploy already in progress (lock age: ${LOCK_AGE}s). Skipping."
|
||||||
|
exit 0
|
||||||
|
else
|
||||||
|
log "Stale lock file (${LOCK_AGE}s old). Removing."
|
||||||
|
rm -f "${LOCK_FILE}"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
trap 'rm -f "${LOCK_FILE}"' EXIT
|
||||||
|
touch "${LOCK_FILE}"
|
||||||
|
|
||||||
|
log "Webhook triggered. Starting ansible-pull..."
|
||||||
|
|
||||||
|
# Pull latest config
|
||||||
|
cd /tmp
|
||||||
|
rm -rf timmy-config-deploy
|
||||||
|
git clone --depth 1 --branch "${BRANCH}" "${REPO}" timmy-config-deploy 2>&1 | tee -a "${LOG_FILE}"
|
||||||
|
|
||||||
|
cd timmy-config-deploy/${ANSIBLE_DIR}
|
||||||
|
|
||||||
|
# Run Ansible against localhost
|
||||||
|
log "Running Ansible playbook..."
|
||||||
|
ansible-playbook \
|
||||||
|
-i inventory/hosts.yml \
|
||||||
|
playbooks/site.yml \
|
||||||
|
--limit "$(hostname)" \
|
||||||
|
--diff \
|
||||||
|
2>&1 | tee -a "${LOG_FILE}"
|
||||||
|
|
||||||
|
RESULT=$?
|
||||||
|
|
||||||
|
if [ ${RESULT} -eq 0 ]; then
|
||||||
|
log "Deploy successful."
|
||||||
|
else
|
||||||
|
log "ERROR: Deploy failed with exit code ${RESULT}."
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Cleanup
|
||||||
|
rm -rf /tmp/timmy-config-deploy
|
||||||
|
|
||||||
|
log "Webhook handler complete."
|
||||||
|
exit ${RESULT}
|
||||||
155
ansible/scripts/validate_config.py
Normal file
@@ -0,0 +1,155 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Config Validator — The Timmy Foundation
|
||||||
|
Validates wizard configs against golden state rules.
|
||||||
|
Run before any config deploy to catch violations early.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 validate_config.py <config_file>
|
||||||
|
python3 validate_config.py --all # Validate all wizard configs
|
||||||
|
|
||||||
|
Exit codes:
|
||||||
|
0 — All validations passed
|
||||||
|
1 — Validation errors found
|
||||||
|
2 — File not found or parse error
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import os
|
||||||
|
import yaml
|
||||||
|
import fnmatch
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# === BANNED PROVIDERS — HARD POLICY ===
|
||||||
|
BANNED_PROVIDERS = {"anthropic", "claude"}
|
||||||
|
BANNED_MODEL_PATTERNS = [
|
||||||
|
"claude-*",
|
||||||
|
"anthropic/*",
|
||||||
|
"*sonnet*",
|
||||||
|
"*opus*",
|
||||||
|
"*haiku*",
|
||||||
|
]
|
||||||
|
|
||||||
|
# === REQUIRED FIELDS ===
|
||||||
|
REQUIRED_FIELDS = {
|
||||||
|
"model": ["default", "provider"],
|
||||||
|
"fallback_providers": None, # Must exist as a list
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def is_banned_model(model_name: str) -> bool:
|
||||||
|
"""Check if a model name matches any banned pattern."""
|
||||||
|
model_lower = model_name.lower()
|
||||||
|
for pattern in BANNED_MODEL_PATTERNS:
|
||||||
|
if fnmatch.fnmatch(model_lower, pattern):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def validate_config(config_path: str) -> list[str]:
|
||||||
|
"""Validate a wizard config file. Returns list of error strings."""
|
||||||
|
errors = []
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(config_path) as f:
|
||||||
|
cfg = yaml.safe_load(f)
|
||||||
|
except FileNotFoundError:
|
||||||
|
return [f"File not found: {config_path}"]
|
||||||
|
except yaml.YAMLError as e:
|
||||||
|
return [f"YAML parse error: {e}"]
|
||||||
|
|
||||||
|
if not cfg:
|
||||||
|
return ["Config file is empty"]
|
||||||
|
|
||||||
|
# Check required fields
|
||||||
|
for section, fields in REQUIRED_FIELDS.items():
|
||||||
|
if section not in cfg:
|
||||||
|
errors.append(f"Missing required section: {section}")
|
||||||
|
elif fields:
|
||||||
|
for field in fields:
|
||||||
|
if field not in cfg[section]:
|
||||||
|
errors.append(f"Missing required field: {section}.{field}")
|
||||||
|
|
||||||
|
# Check default provider
|
||||||
|
default_provider = cfg.get("model", {}).get("provider", "")
|
||||||
|
if default_provider.lower() in BANNED_PROVIDERS:
|
||||||
|
errors.append(f"BANNED default provider: {default_provider}")
|
||||||
|
|
||||||
|
default_model = cfg.get("model", {}).get("default", "")
|
||||||
|
if is_banned_model(default_model):
|
||||||
|
errors.append(f"BANNED default model: {default_model}")
|
||||||
|
|
||||||
|
# Check fallback providers
|
||||||
|
for i, fb in enumerate(cfg.get("fallback_providers", [])):
|
||||||
|
provider = fb.get("provider", "")
|
||||||
|
model = fb.get("model", "")
|
||||||
|
|
||||||
|
if provider.lower() in BANNED_PROVIDERS:
|
||||||
|
errors.append(f"BANNED fallback provider [{i}]: {provider}")
|
||||||
|
|
||||||
|
if is_banned_model(model):
|
||||||
|
errors.append(f"BANNED fallback model [{i}]: {model}")
|
||||||
|
|
||||||
|
# Check providers section
|
||||||
|
for name, provider_cfg in cfg.get("providers", {}).items():
|
||||||
|
if name.lower() in BANNED_PROVIDERS:
|
||||||
|
errors.append(f"BANNED provider in providers section: {name}")
|
||||||
|
|
||||||
|
base_url = str(provider_cfg.get("base_url", ""))
|
||||||
|
if "anthropic" in base_url.lower():
|
||||||
|
errors.append(f"BANNED URL in provider {name}: {base_url}")
|
||||||
|
|
||||||
|
# Check system prompt for banned references
|
||||||
|
prompt = cfg.get("system_prompt_suffix", "")
|
||||||
|
if isinstance(prompt, str):
|
||||||
|
for banned in BANNED_PROVIDERS:
|
||||||
|
if banned in prompt.lower():
|
||||||
|
errors.append(f"BANNED provider referenced in system_prompt_suffix: {banned}")
|
||||||
|
|
||||||
|
return errors
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
if len(sys.argv) < 2:
|
||||||
|
print(f"Usage: {sys.argv[0]} <config_file> [--all]")
|
||||||
|
sys.exit(2)
|
||||||
|
|
||||||
|
if sys.argv[1] == "--all":
|
||||||
|
# Validate all wizard configs in the repo
|
||||||
|
repo_root = Path(__file__).parent.parent.parent
|
||||||
|
wizard_dir = repo_root / "wizards"
|
||||||
|
all_errors = {}
|
||||||
|
|
||||||
|
for wizard_path in sorted(wizard_dir.iterdir()):
|
||||||
|
config_file = wizard_path / "config.yaml"
|
||||||
|
if config_file.exists():
|
||||||
|
errors = validate_config(str(config_file))
|
||||||
|
if errors:
|
||||||
|
all_errors[wizard_path.name] = errors
|
||||||
|
|
||||||
|
if all_errors:
|
||||||
|
print("VALIDATION FAILED:")
|
||||||
|
for wizard, errors in all_errors.items():
|
||||||
|
print(f"\n {wizard}:")
|
||||||
|
for err in errors:
|
||||||
|
print(f" - {err}")
|
||||||
|
sys.exit(1)
|
||||||
|
else:
|
||||||
|
print("All wizard configs passed validation.")
|
||||||
|
sys.exit(0)
|
||||||
|
else:
|
||||||
|
config_path = sys.argv[1]
|
||||||
|
errors = validate_config(config_path)
|
||||||
|
|
||||||
|
if errors:
|
||||||
|
print(f"VALIDATION FAILED for {config_path}:")
|
||||||
|
for err in errors:
|
||||||
|
print(f" - {err}")
|
||||||
|
sys.exit(1)
|
||||||
|
else:
|
||||||
|
print(f"PASSED: {config_path}")
|
||||||
|
sys.exit(0)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
273
bin/agent-loop.sh
Executable file
@@ -0,0 +1,273 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# agent-loop.sh — Universal agent dev loop with Genchi Genbutsu verification
|
||||||
|
#
|
||||||
|
# Usage: agent-loop.sh <agent-name> [num-workers]
|
||||||
|
# agent-loop.sh claude 2
|
||||||
|
# agent-loop.sh gemini 1
|
||||||
|
#
|
||||||
|
# Dispatches via agent-dispatch.sh, then verifies with genchi-genbutsu.sh.
|
||||||
|
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
AGENT="${1:?Usage: agent-loop.sh <agent-name> [num-workers]}"
|
||||||
|
NUM_WORKERS="${2:-1}"
|
||||||
|
|
||||||
|
# Resolve agent tool and model from config or fallback
|
||||||
|
case "$AGENT" in
|
||||||
|
claude) TOOL="claude"; MODEL="sonnet" ;;
|
||||||
|
gemini) TOOL="gemini"; MODEL="gemini-2.5-pro-preview-05-06" ;;
|
||||||
|
grok) TOOL="opencode"; MODEL="grok-3-fast" ;;
|
||||||
|
*) TOOL="$AGENT"; MODEL="" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# === CONFIG ===
|
||||||
|
GITEA_URL="${GITEA_URL:-https://forge.alexanderwhitestone.com}"
|
||||||
|
GITEA_TOKEN="${GITEA_TOKEN:-}"
|
||||||
|
WORKTREE_BASE="$HOME/worktrees"
|
||||||
|
LOG_DIR="$HOME/.hermes/logs"
|
||||||
|
LOCK_DIR="$LOG_DIR/${AGENT}-locks"
|
||||||
|
SKIP_FILE="$LOG_DIR/${AGENT}-skip-list.json"
|
||||||
|
ACTIVE_FILE="$LOG_DIR/${AGENT}-active.json"
|
||||||
|
TIMEOUT=600
|
||||||
|
COOLDOWN=30
|
||||||
|
|
||||||
|
mkdir -p "$LOG_DIR" "$WORKTREE_BASE" "$LOCK_DIR"
|
||||||
|
[ -f "$SKIP_FILE" ] || echo '{}' > "$SKIP_FILE"
|
||||||
|
echo '{}' > "$ACTIVE_FILE"
|
||||||
|
|
||||||
|
# === SHARED FUNCTIONS ===
|
||||||
|
log() {
|
||||||
|
echo "[$(date '+%Y-%m-%d %H:%M:%S')] ${AGENT}: $*" >> "$LOG_DIR/${AGENT}-loop.log"
|
||||||
|
}
|
||||||
|
|
||||||
|
lock_issue() {
|
||||||
|
local key="$1"
|
||||||
|
mkdir "$LOCK_DIR/$key.lock" 2>/dev/null && echo $$ > "$LOCK_DIR/$key.lock/pid"
|
||||||
|
}
|
||||||
|
|
||||||
|
unlock_issue() {
|
||||||
|
rm -rf "$LOCK_DIR/$1.lock" 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
mark_skip() {
|
||||||
|
local issue_num="$1" reason="$2"
|
||||||
|
python3 -c "
|
||||||
|
import json, time, fcntl
|
||||||
|
with open('${SKIP_FILE}', 'r+') as f:
|
||||||
|
fcntl.flock(f, fcntl.LOCK_EX)
|
||||||
|
try: skips = json.load(f)
|
||||||
|
except: skips = {}
|
||||||
|
failures = skips.get(str($issue_num), {}).get('failures', 0) + 1
|
||||||
|
skip_hours = 6 if failures >= 3 else 1
|
||||||
|
skips[str($issue_num)] = {'until': time.time() + (skip_hours * 3600), 'reason': '$reason', 'failures': failures}
|
||||||
|
f.seek(0); f.truncate()
|
||||||
|
json.dump(skips, f, indent=2)
|
||||||
|
" 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
get_next_issue() {
|
||||||
|
python3 -c "
|
||||||
|
import json, sys, time, urllib.request, os
|
||||||
|
token = '${GITEA_TOKEN}'
|
||||||
|
base = '${GITEA_URL}'
|
||||||
|
repos = ['Timmy_Foundation/the-nexus', 'Timmy_Foundation/timmy-config', 'Timmy_Foundation/hermes-agent']
|
||||||
|
try:
|
||||||
|
with open('${SKIP_FILE}') as f: skips = json.load(f)
|
||||||
|
except: skips = {}
|
||||||
|
try:
|
||||||
|
with open('${ACTIVE_FILE}') as f: active = json.load(f); active_issues = {v['issue'] for v in active.values()}
|
||||||
|
except: active_issues = set()
|
||||||
|
all_issues = []
|
||||||
|
for repo in repos:
|
||||||
|
url = f'{base}/api/v1/repos/{repo}/issues?state=open&type=issues&limit=50&sort=created'
|
||||||
|
req = urllib.request.Request(url, headers={'Authorization': f'token {token}'})
|
||||||
|
try:
|
||||||
|
resp = urllib.request.urlopen(req, timeout=10)
|
||||||
|
issues = json.loads(resp.read())
|
||||||
|
for i in issues: i['_repo'] = repo
|
||||||
|
all_issues.extend(issues)
|
||||||
|
except: continue
|
||||||
|
for i in sorted(all_issues, key=lambda x: x['title'].lower()):
|
||||||
|
assignees = [a['login'] for a in (i.get('assignees') or [])]
|
||||||
|
if assignees and '${AGENT}' not in assignees: continue
|
||||||
|
num_str = str(i['number'])
|
||||||
|
if num_str in active_issues: continue
|
||||||
|
if skips.get(num_str, {}).get('until', 0) > time.time(): continue
|
||||||
|
lock = '${LOCK_DIR}/' + i['_repo'].replace('/', '-') + '-' + num_str + '.lock'
|
||||||
|
if os.path.isdir(lock): continue
|
||||||
|
owner, name = i['_repo'].split('/')
|
||||||
|
print(json.dumps({'number': i['number'], 'title': i['title'], 'repo_owner': owner, 'repo_name': name, 'repo': i['_repo']}))
|
||||||
|
sys.exit(0)
|
||||||
|
print('null')
|
||||||
|
" 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
# === WORKER FUNCTION ===
|
||||||
|
run_worker() {
|
||||||
|
local worker_id="$1"
|
||||||
|
log "WORKER-${worker_id}: Started"
|
||||||
|
|
||||||
|
while true; do
|
||||||
|
issue_json=$(get_next_issue)
|
||||||
|
if [ "$issue_json" = "null" ] || [ -z "$issue_json" ]; then
|
||||||
|
sleep 30
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
issue_num=$(echo "$issue_json" | python3 -c "import sys,json; print(json.load(sys.stdin)['number'])")
|
||||||
|
issue_title=$(echo "$issue_json" | python3 -c "import sys,json; print(json.load(sys.stdin)['title'])")
|
||||||
|
repo_owner=$(echo "$issue_json" | python3 -c "import sys,json; print(json.load(sys.stdin)['repo_owner'])")
|
||||||
|
repo_name=$(echo "$issue_json" | python3 -c "import sys,json; print(json.load(sys.stdin)['repo_name'])")
|
||||||
|
issue_key="${repo_owner}-${repo_name}-${issue_num}"
|
||||||
|
branch="${AGENT}/issue-${issue_num}"
|
||||||
|
worktree="${WORKTREE_BASE}/${AGENT}-w${worker_id}-${issue_num}"
|
||||||
|
|
||||||
|
if ! lock_issue "$issue_key"; then
|
||||||
|
sleep 5
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "WORKER-${worker_id}: === ISSUE #${issue_num}: ${issue_title} (${repo_owner}/${repo_name}) ==="
|
||||||
|
|
||||||
|
# Clone / checkout
|
||||||
|
rm -rf "$worktree" 2>/dev/null
|
||||||
|
CLONE_URL="http://${AGENT}:${GITEA_TOKEN}@143.198.27.163:3000/${repo_owner}/${repo_name}.git"
|
||||||
|
if git ls-remote --heads "$CLONE_URL" "$branch" 2>/dev/null | grep -q "$branch"; then
|
||||||
|
git clone --depth=50 -b "$branch" "$CLONE_URL" "$worktree" >/dev/null 2>&1
|
||||||
|
else
|
||||||
|
git clone --depth=1 -b main "$CLONE_URL" "$worktree" >/dev/null 2>&1
|
||||||
|
cd "$worktree" && git checkout -b "$branch" >/dev/null 2>&1
|
||||||
|
fi
|
||||||
|
cd "$worktree"
|
||||||
|
|
||||||
|
# Generate prompt
|
||||||
|
prompt=$(bash "$(dirname "$0")/agent-dispatch.sh" "$AGENT" "$issue_num" "${repo_owner}/${repo_name}")
|
||||||
|
|
||||||
|
CYCLE_START=$(date +%s)
|
||||||
|
set +e
|
||||||
|
if [ "$TOOL" = "claude" ]; then
|
||||||
|
env -u CLAUDECODE gtimeout "$TIMEOUT" claude \
|
||||||
|
--print --model "$MODEL" --dangerously-skip-permissions \
|
||||||
|
-p "$prompt" </dev/null >> "$LOG_DIR/${AGENT}-${issue_num}.log" 2>&1
|
||||||
|
elif [ "$TOOL" = "gemini" ]; then
|
||||||
|
gtimeout "$TIMEOUT" gemini -p "$prompt" --yolo \
|
||||||
|
</dev/null >> "$LOG_DIR/${AGENT}-${issue_num}.log" 2>&1
|
||||||
|
else
|
||||||
|
gtimeout "$TIMEOUT" "$TOOL" "$prompt" \
|
||||||
|
</dev/null >> "$LOG_DIR/${AGENT}-${issue_num}.log" 2>&1
|
||||||
|
fi
|
||||||
|
exit_code=$?
|
||||||
|
set -e
|
||||||
|
CYCLE_END=$(date +%s)
|
||||||
|
CYCLE_DURATION=$((CYCLE_END - CYCLE_START))
|
||||||
|
|
||||||
|
# Salvage
|
||||||
|
cd "$worktree" 2>/dev/null || true
|
||||||
|
DIRTY=$(git status --porcelain 2>/dev/null | wc -l | tr -d ' ')
|
||||||
|
if [ "${DIRTY:-0}" -gt 0 ]; then
|
||||||
|
git add -A 2>/dev/null
|
||||||
|
git commit -m "WIP: ${AGENT} progress on #${issue_num}
|
||||||
|
|
||||||
|
Automated salvage commit — agent session ended (exit $exit_code)." 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
UNPUSHED=$(git log --oneline "origin/main..HEAD" 2>/dev/null | wc -l | tr -d ' ')
|
||||||
|
if [ "${UNPUSHED:-0}" -gt 0 ]; then
|
||||||
|
git push -u origin "$branch" 2>/dev/null && \
|
||||||
|
log "WORKER-${worker_id}: Pushed $UNPUSHED commit(s) on $branch" || \
|
||||||
|
log "WORKER-${worker_id}: Push failed for $branch"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Create PR if needed
|
||||||
|
pr_num=$(curl -sf "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls?state=open&head=${repo_owner}:${branch}&limit=1" \
|
||||||
|
-H "Authorization: token ${GITEA_TOKEN}" | python3 -c "
|
||||||
|
import sys,json
|
||||||
|
prs = json.load(sys.stdin)
|
||||||
|
print(prs[0]['number'] if prs else '')
|
||||||
|
" 2>/dev/null)
|
||||||
|
|
||||||
|
if [ -z "$pr_num" ] && [ "${UNPUSHED:-0}" -gt 0 ]; then
|
||||||
|
pr_num=$(curl -sf -X POST "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls" \
|
||||||
|
-H "Authorization: token ${GITEA_TOKEN}" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d "$(python3 -c "
|
||||||
|
import json
|
||||||
|
print(json.dumps({
|
||||||
|
'title': '${AGENT}: Issue #${issue_num}',
|
||||||
|
'head': '${branch}',
|
||||||
|
'base': 'main',
|
||||||
|
'body': 'Automated PR for issue #${issue_num}.\nExit code: ${exit_code}'
|
||||||
|
}))
|
||||||
|
")" | python3 -c "import sys,json; print(json.load(sys.stdin).get('number',''))" 2>/dev/null)
|
||||||
|
[ -n "$pr_num" ] && log "WORKER-${worker_id}: Created PR #${pr_num} for issue #${issue_num}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Genchi Genbutsu: verify world state before declaring success ──
|
||||||
|
VERIFIED="false"
|
||||||
|
if [ "$exit_code" -eq 0 ]; then
|
||||||
|
log "WORKER-${worker_id}: SUCCESS #${issue_num} — running genchi-genbutsu"
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
if verify_result=$("$SCRIPT_DIR/genchi-genbutsu.sh" "$repo_owner" "$repo_name" "$issue_num" "$branch" "$AGENT" 2>/dev/null); then
|
||||||
|
VERIFIED="true"
|
||||||
|
log "WORKER-${worker_id}: VERIFIED #${issue_num}"
|
||||||
|
if [ -n "$pr_num" ]; then
|
||||||
|
curl -sf -X POST "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}/merge" \
|
||||||
|
-H "Authorization: token ${GITEA_TOKEN}" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"Do": "squash"}' >/dev/null 2>&1 || true
|
||||||
|
curl -sf -X PATCH "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/issues/${issue_num}" \
|
||||||
|
-H "Authorization: token ${GITEA_TOKEN}" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"state": "closed"}' >/dev/null 2>&1 || true
|
||||||
|
log "WORKER-${worker_id}: PR #${pr_num} merged, issue #${issue_num} closed"
|
||||||
|
fi
|
||||||
|
consecutive_failures=0
|
||||||
|
else
|
||||||
|
verify_details=$(echo "$verify_result" | python3 -c "import sys,json; print(json.load(sys.stdin).get('details','unknown'))" 2>/dev/null || echo "unverified")
|
||||||
|
log "WORKER-${worker_id}: UNVERIFIED #${issue_num} — $verify_details"
|
||||||
|
mark_skip "$issue_num" "unverified" 1
|
||||||
|
consecutive_failures=$((consecutive_failures + 1))
|
||||||
|
fi
|
||||||
|
elif [ "$exit_code" -eq 124 ]; then
|
||||||
|
log "WORKER-${worker_id}: TIMEOUT #${issue_num} (work saved in PR)"
|
||||||
|
consecutive_failures=$((consecutive_failures + 1))
|
||||||
|
else
|
||||||
|
log "WORKER-${worker_id}: FAILED #${issue_num} exit ${exit_code} (work saved in PR)"
|
||||||
|
consecutive_failures=$((consecutive_failures + 1))
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── METRICS ──
|
||||||
|
python3 -c "
|
||||||
|
import json, datetime
|
||||||
|
print(json.dumps({
|
||||||
|
'ts': datetime.datetime.utcnow().isoformat() + 'Z',
|
||||||
|
'agent': '${AGENT}',
|
||||||
|
'worker': $worker_id,
|
||||||
|
'issue': $issue_num,
|
||||||
|
'repo': '${repo_owner}/${repo_name}',
|
||||||
|
'outcome': 'success' if $exit_code == 0 else 'timeout' if $exit_code == 124 else 'failed',
|
||||||
|
'exit_code': $exit_code,
|
||||||
|
'duration_s': $CYCLE_DURATION,
|
||||||
|
'pr': '${pr_num:-}',
|
||||||
|
'verified': ${VERIFIED:-false}
|
||||||
|
}))
|
||||||
|
" >> "$LOG_DIR/${AGENT}-metrics.jsonl" 2>/dev/null
|
||||||
|
|
||||||
|
rm -rf "$worktree" 2>/dev/null
|
||||||
|
unlock_issue "$issue_key"
|
||||||
|
sleep "$COOLDOWN"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# === MAIN ===
|
||||||
|
log "=== Agent Loop Started — ${AGENT} with ${NUM_WORKERS} worker(s) ==="
|
||||||
|
|
||||||
|
rm -rf "$LOCK_DIR"/*.lock 2>/dev/null
|
||||||
|
|
||||||
|
for i in $(seq 1 "$NUM_WORKERS"); do
|
||||||
|
run_worker "$i" &
|
||||||
|
log "Launched worker $i (PID $!)"
|
||||||
|
sleep 3
|
||||||
|
done
|
||||||
|
|
||||||
|
wait
|
||||||
@@ -468,24 +468,32 @@ print(json.dumps({
|
|||||||
[ -n "$pr_num" ] && log "WORKER-${worker_id}: Created PR #${pr_num} for issue #${issue_num}"
|
[ -n "$pr_num" ] && log "WORKER-${worker_id}: Created PR #${pr_num} for issue #${issue_num}"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# ── Merge + close on success ──
|
# ── Genchi Genbutsu: verify world state before declaring success ──
|
||||||
|
VERIFIED="false"
|
||||||
if [ "$exit_code" -eq 0 ]; then
|
if [ "$exit_code" -eq 0 ]; then
|
||||||
log "WORKER-${worker_id}: SUCCESS #${issue_num}"
|
log "WORKER-${worker_id}: SUCCESS #${issue_num} — running genchi-genbutsu"
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
if [ -n "$pr_num" ]; then
|
if verify_result=$("$SCRIPT_DIR/genchi-genbutsu.sh" "$repo_owner" "$repo_name" "$issue_num" "$branch" "claude" 2>/dev/null); then
|
||||||
curl -sf -X POST "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}/merge" \
|
VERIFIED="true"
|
||||||
-H "Authorization: token ${GITEA_TOKEN}" \
|
log "WORKER-${worker_id}: VERIFIED #${issue_num}"
|
||||||
-H "Content-Type: application/json" \
|
if [ -n "$pr_num" ]; then
|
||||||
-d '{"Do": "squash"}' >/dev/null 2>&1 || true
|
curl -sf -X POST "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}/merge" \
|
||||||
curl -sf -X PATCH "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/issues/${issue_num}" \
|
-H "Authorization: token ${GITEA_TOKEN}" \
|
||||||
-H "Authorization: token ${GITEA_TOKEN}" \
|
-H "Content-Type: application/json" \
|
||||||
-H "Content-Type: application/json" \
|
-d '{"Do": "squash"}' >/dev/null 2>&1 || true
|
||||||
-d '{"state": "closed"}' >/dev/null 2>&1 || true
|
curl -sf -X PATCH "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/issues/${issue_num}" \
|
||||||
log "WORKER-${worker_id}: PR #${pr_num} merged, issue #${issue_num} closed"
|
-H "Authorization: token ${GITEA_TOKEN}" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"state": "closed"}' >/dev/null 2>&1 || true
|
||||||
|
log "WORKER-${worker_id}: PR #${pr_num} merged, issue #${issue_num} closed"
|
||||||
|
fi
|
||||||
|
consecutive_failures=0
|
||||||
|
else
|
||||||
|
verify_details=$(echo "$verify_result" | python3 -c "import sys,json; print(json.load(sys.stdin).get('details','unknown'))" 2>/dev/null || echo "unverified")
|
||||||
|
log "WORKER-${worker_id}: UNVERIFIED #${issue_num} — $verify_details"
|
||||||
|
consecutive_failures=$((consecutive_failures + 1))
|
||||||
fi
|
fi
|
||||||
|
|
||||||
consecutive_failures=0
|
|
||||||
|
|
||||||
elif [ "$exit_code" -eq 124 ]; then
|
elif [ "$exit_code" -eq 124 ]; then
|
||||||
log "WORKER-${worker_id}: TIMEOUT #${issue_num} (work saved in PR)"
|
log "WORKER-${worker_id}: TIMEOUT #${issue_num} (work saved in PR)"
|
||||||
consecutive_failures=$((consecutive_failures + 1))
|
consecutive_failures=$((consecutive_failures + 1))
|
||||||
@@ -522,6 +530,7 @@ print(json.dumps({
|
|||||||
import json, datetime
|
import json, datetime
|
||||||
print(json.dumps({
|
print(json.dumps({
|
||||||
'ts': datetime.datetime.utcnow().isoformat() + 'Z',
|
'ts': datetime.datetime.utcnow().isoformat() + 'Z',
|
||||||
|
'agent': 'claude',
|
||||||
'worker': $worker_id,
|
'worker': $worker_id,
|
||||||
'issue': $issue_num,
|
'issue': $issue_num,
|
||||||
'repo': '${repo_owner}/${repo_name}',
|
'repo': '${repo_owner}/${repo_name}',
|
||||||
@@ -534,7 +543,8 @@ print(json.dumps({
|
|||||||
'lines_removed': ${LINES_REMOVED:-0},
|
'lines_removed': ${LINES_REMOVED:-0},
|
||||||
'salvaged': ${DIRTY:-0},
|
'salvaged': ${DIRTY:-0},
|
||||||
'pr': '${pr_num:-}',
|
'pr': '${pr_num:-}',
|
||||||
'merged': $( [ '$OUTCOME' = 'success' ] && [ -n '${pr_num:-}' ] && echo 'true' || echo 'false' )
|
'merged': $( [ '$OUTCOME' = 'success' ] && [ -n '${pr_num:-}' ] && echo 'true' || echo 'false' ),
|
||||||
|
'verified': ${VERIFIED:-false}
|
||||||
}))
|
}))
|
||||||
" >> "$METRICS_FILE" 2>/dev/null
|
" >> "$METRICS_FILE" 2>/dev/null
|
||||||
|
|
||||||
|
|||||||
264
bin/deadman-fallback.py
Normal file
@@ -0,0 +1,264 @@
|
|||||||
|
1|#!/usr/bin/env python3
|
||||||
|
2|"""
|
||||||
|
3|Dead Man Switch Fallback Engine
|
||||||
|
4|
|
||||||
|
5|When the dead man switch triggers (zero commits for 2+ hours, model down,
|
||||||
|
6|Gitea unreachable, etc.), this script diagnoses the failure and applies
|
||||||
|
7|common sense fallbacks automatically.
|
||||||
|
8|
|
||||||
|
9|Fallback chain:
|
||||||
|
10|1. Primary model (Anthropic) down -> switch config to local-llama.cpp
|
||||||
|
11|2. Gitea unreachable -> cache issues locally, retry on recovery
|
||||||
|
12|3. VPS agents down -> alert + lazarus protocol
|
||||||
|
13|4. Local llama.cpp down -> try Ollama, then alert-only mode
|
||||||
|
14|5. All inference dead -> safe mode (cron pauses, alert Alexander)
|
||||||
|
15|
|
||||||
|
16|Each fallback is reversible. Recovery auto-restores the previous config.
|
||||||
|
17|"""
|
||||||
|
18|import os
|
||||||
|
19|import sys
|
||||||
|
20|import json
|
||||||
|
21|import subprocess
|
||||||
|
22|import time
|
||||||
|
23|import yaml
|
||||||
|
24|import shutil
|
||||||
|
25|from pathlib import Path
|
||||||
|
26|from datetime import datetime, timedelta
|
||||||
|
27|
|
||||||
|
28|HERMES_HOME = Path(os.environ.get("HERMES_HOME", os.path.expanduser("~/.hermes")))
|
||||||
|
29|CONFIG_PATH = HERMES_HOME / "config.yaml"
|
||||||
|
30|FALLBACK_STATE = HERMES_HOME / "deadman-fallback-state.json"
|
||||||
|
31|BACKUP_CONFIG = HERMES_HOME / "config.yaml.pre-fallback"
|
||||||
|
32|FORGE_URL = "https://forge.alexanderwhitestone.com"
|
||||||
|
33|
|
||||||
|
34|def load_config():
|
||||||
|
35| with open(CONFIG_PATH) as f:
|
||||||
|
36| return yaml.safe_load(f)
|
||||||
|
37|
|
||||||
|
38|def save_config(cfg):
|
||||||
|
39| with open(CONFIG_PATH, "w") as f:
|
||||||
|
40| yaml.dump(cfg, f, default_flow_style=False)
|
||||||
|
41|
|
||||||
|
42|def load_state():
|
||||||
|
43| if FALLBACK_STATE.exists():
|
||||||
|
44| with open(FALLBACK_STATE) as f:
|
||||||
|
45| return json.load(f)
|
||||||
|
46| return {"active_fallbacks": [], "last_check": None, "recovery_pending": False}
|
||||||
|
47|
|
||||||
|
48|def save_state(state):
|
||||||
|
49| state["last_check"] = datetime.now().isoformat()
|
||||||
|
50| with open(FALLBACK_STATE, "w") as f:
|
||||||
|
51| json.dump(state, f, indent=2)
|
||||||
|
52|
|
||||||
|
53|def run(cmd, timeout=10):
|
||||||
|
54| try:
|
||||||
|
55| r = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=timeout)
|
||||||
|
56| return r.returncode, r.stdout.strip(), r.stderr.strip()
|
||||||
|
57| except subprocess.TimeoutExpired:
|
||||||
|
58| return -1, "", "timeout"
|
||||||
|
59| except Exception as e:
|
||||||
|
60| return -1, "", str(e)
|
||||||
|
61|
|
||||||
|
62|# ─── HEALTH CHECKS ───
|
||||||
|
63|
|
||||||
|
64|def check_anthropic():
|
||||||
|
65| """Can we reach Anthropic API?"""
|
||||||
|
66| key = os.environ.get("ANTHROPIC_API_KEY", "")
|
||||||
|
67| if not key:
|
||||||
|
68| # Check multiple .env locations
|
||||||
|
69| for env_path in [HERMES_HOME / ".env", Path.home() / ".hermes" / ".env"]:
|
||||||
|
70| if env_path.exists():
|
||||||
|
71| for line in open(env_path):
|
||||||
|
72| line = line.strip()
|
||||||
|
73| if line.startswith("ANTHROPIC_API_KEY=***
|
||||||
|
74| key = line.split("=", 1)[1].strip().strip('"').strip("'")
|
||||||
|
75| break
|
||||||
|
76| if key:
|
||||||
|
77| break
|
||||||
|
78| if not key:
|
||||||
|
79| return False, "no API key"
|
||||||
|
80| code, out, err = run(
|
||||||
|
81| f'curl -s -o /dev/null -w "%{{http_code}}" -H "x-api-key: {key}" '
|
||||||
|
82| f'-H "anthropic-version: 2023-06-01" '
|
||||||
|
83| f'https://api.anthropic.com/v1/messages -X POST '
|
||||||
|
84| f'-H "content-type: application/json" '
|
||||||
|
85| f'-d \'{{"model":"claude-haiku-4-5-20251001","max_tokens":1,"messages":[{{"role":"user","content":"ping"}}]}}\' ',
|
||||||
|
86| timeout=15
|
||||||
|
87| )
|
||||||
|
88| if code == 0 and out in ("200", "429"):
|
||||||
|
89| return True, f"HTTP {out}"
|
||||||
|
90| return False, f"HTTP {out} err={err[:80]}"
|
||||||
|
91|
|
||||||
|
92|def check_local_llama():
|
||||||
|
93| """Is local llama.cpp serving?"""
|
||||||
|
94| code, out, err = run("curl -s http://localhost:8081/v1/models", timeout=5)
|
||||||
|
95| if code == 0 and "hermes" in out.lower():
|
||||||
|
96| return True, "serving"
|
||||||
|
97| return False, f"exit={code}"
|
||||||
|
98|
|
||||||
|
99|def check_ollama():
|
||||||
|
100| """Is Ollama running?"""
|
||||||
|
101| code, out, err = run("curl -s http://localhost:11434/api/tags", timeout=5)
|
||||||
|
102| if code == 0 and "models" in out:
|
||||||
|
103| return True, "running"
|
||||||
|
104| return False, f"exit={code}"
|
||||||
|
105|
|
||||||
|
106|def check_gitea():
|
||||||
|
107| """Can we reach the Forge?"""
|
||||||
|
108| token_path = Path.home() / ".config" / "gitea" / "timmy-token"
|
||||||
|
109| if not token_path.exists():
|
||||||
|
110| return False, "no token"
|
||||||
|
111| token = token_path.read_text().strip()
|
||||||
|
112| code, out, err = run(
|
||||||
|
113| f'curl -s -o /dev/null -w "%{{http_code}}" -H "Authorization: token {token}" '
|
||||||
|
114| f'"{FORGE_URL}/api/v1/user"',
|
||||||
|
115| timeout=10
|
||||||
|
116| )
|
||||||
|
117| if code == 0 and out == "200":
|
||||||
|
118| return True, "reachable"
|
||||||
|
119| return False, f"HTTP {out}"
|
||||||
|
120|
|
||||||
|
121|def check_vps(ip, name):
|
||||||
|
122| """Can we SSH into a VPS?"""
|
||||||
|
123| code, out, err = run(f"ssh -o ConnectTimeout=5 root@{ip} 'echo alive'", timeout=10)
|
||||||
|
124| if code == 0 and "alive" in out:
|
||||||
|
125| return True, "alive"
|
||||||
|
126| return False, f"unreachable"
|
||||||
|
127|
|
||||||
|
128|# ─── FALLBACK ACTIONS ───
|
||||||
|
129|
|
||||||
|
130|def fallback_to_local_model(cfg):
|
||||||
|
131| """Switch primary model from Anthropic to local llama.cpp"""
|
||||||
|
132| if not BACKUP_CONFIG.exists():
|
||||||
|
133| shutil.copy2(CONFIG_PATH, BACKUP_CONFIG)
|
||||||
|
134|
|
||||||
|
135| cfg["model"]["provider"] = "local-llama.cpp"
|
||||||
|
136| cfg["model"]["default"] = "hermes3"
|
||||||
|
137| save_config(cfg)
|
||||||
|
138| return "Switched primary model to local-llama.cpp/hermes3"
|
||||||
|
139|
|
||||||
|
140|def fallback_to_ollama(cfg):
|
||||||
|
141| """Switch to Ollama if llama.cpp is also down"""
|
||||||
|
142| if not BACKUP_CONFIG.exists():
|
||||||
|
143| shutil.copy2(CONFIG_PATH, BACKUP_CONFIG)
|
||||||
|
144|
|
||||||
|
145| cfg["model"]["provider"] = "ollama"
|
||||||
|
146| cfg["model"]["default"] = "gemma4:latest"
|
||||||
|
147| save_config(cfg)
|
||||||
|
148| return "Switched primary model to ollama/gemma4:latest"
|
||||||
|
149|
|
||||||
|
150|def enter_safe_mode(state):
|
||||||
|
151| """Pause all non-essential cron jobs, alert Alexander"""
|
||||||
|
152| state["safe_mode"] = True
|
||||||
|
153| state["safe_mode_entered"] = datetime.now().isoformat()
|
||||||
|
154| save_state(state)
|
||||||
|
155| return "SAFE MODE: All inference down. Cron jobs should be paused. Alert Alexander."
|
||||||
|
156|
|
||||||
|
157|def restore_config():
|
||||||
|
158| """Restore pre-fallback config when primary recovers"""
|
||||||
|
159| if BACKUP_CONFIG.exists():
|
||||||
|
160| shutil.copy2(BACKUP_CONFIG, CONFIG_PATH)
|
||||||
|
161| BACKUP_CONFIG.unlink()
|
||||||
|
162| return "Restored original config from backup"
|
||||||
|
163| return "No backup config to restore"
|
||||||
|
164|
|
||||||
|
165|# ─── MAIN DIAGNOSIS AND FALLBACK ENGINE ───
|
||||||
|
166|
|
||||||
|
167|def diagnose_and_fallback():
|
||||||
|
168| state = load_state()
|
||||||
|
169| cfg = load_config()
|
||||||
|
170|
|
||||||
|
171| results = {
|
||||||
|
172| "timestamp": datetime.now().isoformat(),
|
||||||
|
173| "checks": {},
|
||||||
|
174| "actions": [],
|
||||||
|
175| "status": "healthy"
|
||||||
|
176| }
|
||||||
|
177|
|
||||||
|
178| # Check all systems
|
||||||
|
179| anthropic_ok, anthropic_msg = check_anthropic()
|
||||||
|
180| results["checks"]["anthropic"] = {"ok": anthropic_ok, "msg": anthropic_msg}
|
||||||
|
181|
|
||||||
|
182| llama_ok, llama_msg = check_local_llama()
|
||||||
|
183| results["checks"]["local_llama"] = {"ok": llama_ok, "msg": llama_msg}
|
||||||
|
184|
|
||||||
|
185| ollama_ok, ollama_msg = check_ollama()
|
||||||
|
186| results["checks"]["ollama"] = {"ok": ollama_ok, "msg": ollama_msg}
|
||||||
|
187|
|
||||||
|
188| gitea_ok, gitea_msg = check_gitea()
|
||||||
|
189| results["checks"]["gitea"] = {"ok": gitea_ok, "msg": gitea_msg}
|
||||||
|
190|
|
||||||
|
191| # VPS checks
|
||||||
|
192| vpses = [
|
||||||
|
193| ("167.99.126.228", "Allegro"),
|
||||||
|
194| ("143.198.27.163", "Ezra"),
|
||||||
|
195| ("159.203.146.185", "Bezalel"),
|
||||||
|
196| ]
|
||||||
|
197| for ip, name in vpses:
|
||||||
|
198| vps_ok, vps_msg = check_vps(ip, name)
|
||||||
|
199| results["checks"][f"vps_{name.lower()}"] = {"ok": vps_ok, "msg": vps_msg}
|
||||||
|
200|
|
||||||
|
201| current_provider = cfg.get("model", {}).get("provider", "anthropic")
|
||||||
|
202|
|
||||||
|
203| # ─── FALLBACK LOGIC ───
|
||||||
|
204|
|
||||||
|
205| # Case 1: Primary (Anthropic) down, local available
|
||||||
|
206| if not anthropic_ok and current_provider == "anthropic":
|
||||||
|
207| if llama_ok:
|
||||||
|
208| msg = fallback_to_local_model(cfg)
|
||||||
|
209| results["actions"].append(msg)
|
||||||
|
210| state["active_fallbacks"].append("anthropic->local-llama")
|
||||||
|
211| results["status"] = "degraded_local"
|
||||||
|
212| elif ollama_ok:
|
||||||
|
213| msg = fallback_to_ollama(cfg)
|
||||||
|
214| results["actions"].append(msg)
|
||||||
|
215| state["active_fallbacks"].append("anthropic->ollama")
|
||||||
|
216| results["status"] = "degraded_ollama"
|
||||||
|
217| else:
|
||||||
|
218| msg = enter_safe_mode(state)
|
||||||
|
219| results["actions"].append(msg)
|
||||||
|
220| results["status"] = "safe_mode"
|
||||||
|
221|
|
||||||
|
222| # Case 2: Already on fallback, check if primary recovered
|
||||||
|
223| elif anthropic_ok and "anthropic->local-llama" in state.get("active_fallbacks", []):
|
||||||
|
224| msg = restore_config()
|
||||||
|
225| results["actions"].append(msg)
|
||||||
|
226| state["active_fallbacks"].remove("anthropic->local-llama")
|
||||||
|
227| results["status"] = "recovered"
|
||||||
|
228| elif anthropic_ok and "anthropic->ollama" in state.get("active_fallbacks", []):
|
||||||
|
229| msg = restore_config()
|
||||||
|
230| results["actions"].append(msg)
|
||||||
|
231| state["active_fallbacks"].remove("anthropic->ollama")
|
||||||
|
232| results["status"] = "recovered"
|
||||||
|
233|
|
||||||
|
234| # Case 3: Gitea down — just flag it, work locally
|
||||||
|
235| if not gitea_ok:
|
||||||
|
236| results["actions"].append("WARN: Gitea unreachable — work cached locally until recovery")
|
||||||
|
237| if "gitea_down" not in state.get("active_fallbacks", []):
|
||||||
|
238| state["active_fallbacks"].append("gitea_down")
|
||||||
|
239| results["status"] = max(results["status"], "degraded_gitea", key=lambda x: ["healthy", "recovered", "degraded_gitea", "degraded_local", "degraded_ollama", "safe_mode"].index(x) if x in ["healthy", "recovered", "degraded_gitea", "degraded_local", "degraded_ollama", "safe_mode"] else 0)
|
||||||
|
240| elif "gitea_down" in state.get("active_fallbacks", []):
|
||||||
|
241| state["active_fallbacks"].remove("gitea_down")
|
||||||
|
242| results["actions"].append("Gitea recovered — resume normal operations")
|
||||||
|
243|
|
||||||
|
244| # Case 4: VPS agents down
|
||||||
|
245| for ip, name in vpses:
|
||||||
|
246| key = f"vps_{name.lower()}"
|
||||||
|
247| if not results["checks"][key]["ok"]:
|
||||||
|
248| results["actions"].append(f"ALERT: {name} VPS ({ip}) unreachable — lazarus protocol needed")
|
||||||
|
249|
|
||||||
|
250| save_state(state)
|
||||||
|
251| return results
|
||||||
|
252|
|
||||||
|
253|if __name__ == "__main__":
|
||||||
|
254| results = diagnose_and_fallback()
|
||||||
|
255| print(json.dumps(results, indent=2))
|
||||||
|
256|
|
||||||
|
257| # Exit codes for cron integration
|
||||||
|
258| if results["status"] == "safe_mode":
|
||||||
|
259| sys.exit(2)
|
||||||
|
260| elif results["status"].startswith("degraded"):
|
||||||
|
261| sys.exit(1)
|
||||||
|
262| else:
|
||||||
|
263| sys.exit(0)
|
||||||
|
264|
|
||||||
@@ -521,61 +521,63 @@ print(json.dumps({
|
|||||||
[ -n "$pr_num" ] && log "WORKER-${worker_id}: Created PR #${pr_num} for issue #${issue_num}"
|
[ -n "$pr_num" ] && log "WORKER-${worker_id}: Created PR #${pr_num} for issue #${issue_num}"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# ── Verify finish semantics / classify failures ──
|
# ── Genchi Genbutsu: verify world state before declaring success ──
|
||||||
|
VERIFIED="false"
|
||||||
if [ "$exit_code" -eq 0 ]; then
|
if [ "$exit_code" -eq 0 ]; then
|
||||||
log "WORKER-${worker_id}: SUCCESS #${issue_num} exited 0 — verifying push + PR + proof"
|
log "WORKER-${worker_id}: SUCCESS #${issue_num} exited 0 — running genchi-genbutsu"
|
||||||
if ! remote_branch_exists "$branch"; then
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
log "WORKER-${worker_id}: BLOCKED #${issue_num} remote branch missing"
|
if verify_result=$("$SCRIPT_DIR/genchi-genbutsu.sh" "$repo_owner" "$repo_name" "$issue_num" "$branch" "gemini" 2>/dev/null); then
|
||||||
post_issue_comment "$repo_owner" "$repo_name" "$issue_num" "Loop gate blocked completion: remote branch ${branch} was not found on origin after Gemini exited. Issue remains open for retry."
|
VERIFIED="true"
|
||||||
mark_skip "$issue_num" "missing_remote_branch" 1
|
log "WORKER-${worker_id}: VERIFIED #${issue_num}"
|
||||||
consecutive_failures=$((consecutive_failures + 1))
|
pr_state=$(get_pr_state "$repo_owner" "$repo_name" "$pr_num")
|
||||||
elif [ -z "$pr_num" ]; then
|
if [ "$pr_state" = "open" ]; then
|
||||||
log "WORKER-${worker_id}: BLOCKED #${issue_num} no PR found"
|
curl -sf -X POST "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}/merge" \
|
||||||
post_issue_comment "$repo_owner" "$repo_name" "$issue_num" "Loop gate blocked completion: branch ${branch} exists remotely, but no PR was found. Issue remains open for retry."
|
-H "Authorization: token ${GITEA_TOKEN}" \
|
||||||
mark_skip "$issue_num" "missing_pr" 1
|
-H "Content-Type: application/json" \
|
||||||
consecutive_failures=$((consecutive_failures + 1))
|
-d '{"Do": "squash"}' >/dev/null 2>&1 || true
|
||||||
|
pr_state=$(get_pr_state "$repo_owner" "$repo_name" "$pr_num")
|
||||||
|
fi
|
||||||
|
if [ "$pr_state" = "merged" ]; then
|
||||||
|
curl -sf -X PATCH "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/issues/${issue_num}" \
|
||||||
|
-H "Authorization: token ${GITEA_TOKEN}" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"state": "closed"}' >/dev/null 2>&1 || true
|
||||||
|
issue_state=$(get_issue_state "$repo_owner" "$repo_name" "$issue_num")
|
||||||
|
if [ "$issue_state" = "closed" ]; then
|
||||||
|
log "WORKER-${worker_id}: VERIFIED #${issue_num} branch pushed, PR merged, comment present, issue closed"
|
||||||
|
consecutive_failures=0
|
||||||
|
else
|
||||||
|
log "WORKER-${worker_id}: BLOCKED #${issue_num} issue did not close after merge"
|
||||||
|
mark_skip "$issue_num" "issue_close_unverified" 1
|
||||||
|
consecutive_failures=$((consecutive_failures + 1))
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
log "WORKER-${worker_id}: BLOCKED #${issue_num} merge not verified (state=${pr_state})"
|
||||||
|
mark_skip "$issue_num" "merge_unverified" 1
|
||||||
|
consecutive_failures=$((consecutive_failures + 1))
|
||||||
|
fi
|
||||||
else
|
else
|
||||||
pr_files=$(get_pr_file_count "$repo_owner" "$repo_name" "$pr_num")
|
verify_details=$(echo "$verify_result" | python3 -c "import sys,json; print(json.load(sys.stdin).get('details','unknown'))" 2>/dev/null || echo "unverified")
|
||||||
if [ "${pr_files:-0}" -eq 0 ]; then
|
verify_checks=$(echo "$verify_result" | python3 -c "import sys,json; print(json.load(sys.stdin).get('checks',''))" 2>/dev/null || echo "")
|
||||||
log "WORKER-${worker_id}: BLOCKED #${issue_num} PR #${pr_num} has 0 changed files"
|
log "WORKER-${worker_id}: UNVERIFIED #${issue_num} — $verify_details"
|
||||||
curl -sf -X PATCH "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}" -H "Authorization: token ${GITEA_TOKEN}" -H "Content-Type: application/json" -d '{"state": "closed"}' >/dev/null 2>&1 || true
|
if echo "$verify_checks" | grep -q '"branch": false'; then
|
||||||
|
post_issue_comment "$repo_owner" "$repo_name" "$issue_num" "Loop gate blocked completion: remote branch ${branch} was not found on origin after Gemini exited. Issue remains open for retry."
|
||||||
|
mark_skip "$issue_num" "missing_remote_branch" 1
|
||||||
|
elif echo "$verify_checks" | grep -q '"pr": false'; then
|
||||||
|
post_issue_comment "$repo_owner" "$repo_name" "$issue_num" "Loop gate blocked completion: branch ${branch} exists remotely, but no PR was found. Issue remains open for retry."
|
||||||
|
mark_skip "$issue_num" "missing_pr" 1
|
||||||
|
elif echo "$verify_checks" | grep -q '"files": false'; then
|
||||||
|
curl -sf -X PATCH "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}" \
|
||||||
|
-H "Authorization: token ${GITEA_TOKEN}" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"state": "closed"}' >/dev/null 2>&1 || true
|
||||||
post_issue_comment "$repo_owner" "$repo_name" "$issue_num" "PR #${pr_num} was closed automatically: it had 0 changed files (empty commit). Issue remains open for retry."
|
post_issue_comment "$repo_owner" "$repo_name" "$issue_num" "PR #${pr_num} was closed automatically: it had 0 changed files (empty commit). Issue remains open for retry."
|
||||||
mark_skip "$issue_num" "empty_commit" 2
|
mark_skip "$issue_num" "empty_commit" 2
|
||||||
consecutive_failures=$((consecutive_failures + 1))
|
|
||||||
else
|
else
|
||||||
proof_status=$(proof_comment_status "$repo_owner" "$repo_name" "$issue_num" "$branch")
|
post_issue_comment "$repo_owner" "$repo_name" "$issue_num" "Loop gate blocked completion: PR #${pr_num} exists, but required verification failed ($verify_details). Issue remains open for retry."
|
||||||
proof_state="${proof_status%%|*}"
|
mark_skip "$issue_num" "unverified" 1
|
||||||
proof_url="${proof_status#*|}"
|
|
||||||
if [ "$proof_state" != "ok" ]; then
|
|
||||||
log "WORKER-${worker_id}: BLOCKED #${issue_num} proof missing or incomplete (${proof_state})"
|
|
||||||
post_issue_comment "$repo_owner" "$repo_name" "$issue_num" "Loop gate blocked completion: PR #${pr_num} exists and has ${pr_files} changed file(s), but the required Proof block from Gemini is missing or incomplete. Issue remains open for retry."
|
|
||||||
mark_skip "$issue_num" "missing_proof" 1
|
|
||||||
consecutive_failures=$((consecutive_failures + 1))
|
|
||||||
else
|
|
||||||
log "WORKER-${worker_id}: PROOF verified ${proof_url}"
|
|
||||||
pr_state=$(get_pr_state "$repo_owner" "$repo_name" "$pr_num")
|
|
||||||
if [ "$pr_state" = "open" ]; then
|
|
||||||
curl -sf -X POST "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}/merge" -H "Authorization: token ${GITEA_TOKEN}" -H "Content-Type: application/json" -d '{"Do": "squash"}' >/dev/null 2>&1 || true
|
|
||||||
pr_state=$(get_pr_state "$repo_owner" "$repo_name" "$pr_num")
|
|
||||||
fi
|
|
||||||
if [ "$pr_state" = "merged" ]; then
|
|
||||||
curl -sf -X PATCH "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/issues/${issue_num}" -H "Authorization: token ${GITEA_TOKEN}" -H "Content-Type: application/json" -d '{"state": "closed"}' >/dev/null 2>&1 || true
|
|
||||||
issue_state=$(get_issue_state "$repo_owner" "$repo_name" "$issue_num")
|
|
||||||
if [ "$issue_state" = "closed" ]; then
|
|
||||||
log "WORKER-${worker_id}: VERIFIED #${issue_num} branch pushed, PR merged, proof present, issue closed"
|
|
||||||
consecutive_failures=0
|
|
||||||
else
|
|
||||||
log "WORKER-${worker_id}: BLOCKED #${issue_num} issue did not close after merge"
|
|
||||||
mark_skip "$issue_num" "issue_close_unverified" 1
|
|
||||||
consecutive_failures=$((consecutive_failures + 1))
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
log "WORKER-${worker_id}: BLOCKED #${issue_num} merge not verified (state=${pr_state})"
|
|
||||||
mark_skip "$issue_num" "merge_unverified" 1
|
|
||||||
consecutive_failures=$((consecutive_failures + 1))
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
fi
|
fi
|
||||||
|
consecutive_failures=$((consecutive_failures + 1))
|
||||||
fi
|
fi
|
||||||
elif [ "$exit_code" -eq 124 ]; then
|
elif [ "$exit_code" -eq 124 ]; then
|
||||||
log "WORKER-${worker_id}: TIMEOUT #${issue_num} (work saved in PR)"
|
log "WORKER-${worker_id}: TIMEOUT #${issue_num} (work saved in PR)"
|
||||||
@@ -621,7 +623,8 @@ print(json.dumps({
|
|||||||
'lines_removed': ${LINES_REMOVED:-0},
|
'lines_removed': ${LINES_REMOVED:-0},
|
||||||
'salvaged': ${DIRTY:-0},
|
'salvaged': ${DIRTY:-0},
|
||||||
'pr': '${pr_num:-}',
|
'pr': '${pr_num:-}',
|
||||||
'merged': $( [ '$OUTCOME' = 'success' ] && [ -n '${pr_num:-}' ] && echo 'true' || echo 'false' )
|
'merged': $( [ '$OUTCOME' = 'success' ] && [ -n '${pr_num:-}' ] && echo 'true' || echo 'false' ),
|
||||||
|
'verified': ${VERIFIED:-false}
|
||||||
}))
|
}))
|
||||||
" >> "$LOG_DIR/gemini-metrics.jsonl" 2>/dev/null
|
" >> "$LOG_DIR/gemini-metrics.jsonl" 2>/dev/null
|
||||||
|
|
||||||
|
|||||||
179
bin/genchi-genbutsu.sh
Executable file
@@ -0,0 +1,179 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# genchi-genbutsu.sh — 現地現物 — Go and see. Verify world state, not log vibes.
|
||||||
|
#
|
||||||
|
# Post-completion verification that goes and LOOKS at the actual artifacts.
|
||||||
|
# Performs 5 world-state checks:
|
||||||
|
# 1. Branch exists on remote
|
||||||
|
# 2. PR exists
|
||||||
|
# 3. PR has real file changes (> 0)
|
||||||
|
# 4. PR is mergeable
|
||||||
|
# 5. Issue has a completion comment from the agent
|
||||||
|
#
|
||||||
|
# Usage: genchi-genbutsu.sh <repo_owner> <repo_name> <issue_num> <branch> <agent_name>
|
||||||
|
# Returns: JSON to stdout, logs JSONL, exit 0 = VERIFIED, exit 1 = UNVERIFIED
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
GITEA_URL="${GITEA_URL:-https://forge.alexanderwhitestone.com}"
|
||||||
|
GITEA_TOKEN="${GITEA_TOKEN:-}"
|
||||||
|
LOG_DIR="${LOG_DIR:-$HOME/.hermes/logs}"
|
||||||
|
VERIFY_LOG="$LOG_DIR/genchi-genbutsu.jsonl"
|
||||||
|
|
||||||
|
if [ $# -lt 5 ]; then
|
||||||
|
echo "Usage: $0 <repo_owner> <repo_name> <issue_num> <branch> <agent_name>" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
repo_owner="$1"
|
||||||
|
repo_name="$2"
|
||||||
|
issue_num="$3"
|
||||||
|
branch="$4"
|
||||||
|
agent_name="$5"
|
||||||
|
|
||||||
|
mkdir -p "$LOG_DIR"
|
||||||
|
|
||||||
|
# ── Helpers ──────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
check_branch_exists() {
|
||||||
|
# Use Gitea API instead of git ls-remote so we don't need clone credentials
|
||||||
|
curl -sf "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/branches/${branch}" \
|
||||||
|
-H "Authorization: token ${GITEA_TOKEN}" >/dev/null 2>&1
|
||||||
|
}
|
||||||
|
|
||||||
|
get_pr_num() {
|
||||||
|
curl -sf "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls?state=all&head=${repo_owner}:${branch}&limit=1" \
|
||||||
|
-H "Authorization: token ${GITEA_TOKEN}" 2>/dev/null | python3 -c "
|
||||||
|
import sys, json
|
||||||
|
prs = json.load(sys.stdin)
|
||||||
|
print(prs[0]['number'] if prs else '')
|
||||||
|
"
|
||||||
|
}
|
||||||
|
|
||||||
|
check_pr_files() {
|
||||||
|
local pr_num="$1"
|
||||||
|
curl -sf "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}/files" \
|
||||||
|
-H "Authorization: token ${GITEA_TOKEN}" 2>/dev/null | python3 -c "
|
||||||
|
import sys, json
|
||||||
|
try:
|
||||||
|
files = json.load(sys.stdin)
|
||||||
|
print(len(files) if isinstance(files, list) else 0)
|
||||||
|
except:
|
||||||
|
print(0)
|
||||||
|
"
|
||||||
|
}
|
||||||
|
|
||||||
|
check_pr_mergeable() {
|
||||||
|
local pr_num="$1"
|
||||||
|
curl -sf "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/pulls/${pr_num}" \
|
||||||
|
-H "Authorization: token ${GITEA_TOKEN}" 2>/dev/null | python3 -c "
|
||||||
|
import sys, json
|
||||||
|
pr = json.load(sys.stdin)
|
||||||
|
print('true' if pr.get('mergeable') else 'false')
|
||||||
|
"
|
||||||
|
}
|
||||||
|
|
||||||
|
check_completion_comment() {
|
||||||
|
curl -sf "${GITEA_URL}/api/v1/repos/${repo_owner}/${repo_name}/issues/${issue_num}/comments" \
|
||||||
|
-H "Authorization: token ${GITEA_TOKEN}" 2>/dev/null | AGENT="$agent_name" python3 -c "
|
||||||
|
import os, sys, json
|
||||||
|
agent = os.environ.get('AGENT', '').lower()
|
||||||
|
try:
|
||||||
|
comments = json.load(sys.stdin)
|
||||||
|
except:
|
||||||
|
sys.exit(1)
|
||||||
|
for c in reversed(comments):
|
||||||
|
user = ((c.get('user') or {}).get('login') or '').lower()
|
||||||
|
if user == agent:
|
||||||
|
sys.exit(0)
|
||||||
|
sys.exit(1)
|
||||||
|
"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ── Run checks ───────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
ts=$(date -u '+%Y-%m-%dT%H:%M:%SZ')
|
||||||
|
status="VERIFIED"
|
||||||
|
details=()
|
||||||
|
checks_json='{}'
|
||||||
|
|
||||||
|
# Check 1: branch
|
||||||
|
if check_branch_exists; then
|
||||||
|
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['branch']=True;print(json.dumps(d))")
|
||||||
|
else
|
||||||
|
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['branch']=False;print(json.dumps(d))")
|
||||||
|
status="UNVERIFIED"
|
||||||
|
details+=("remote branch ${branch} not found")
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check 2: PR exists
|
||||||
|
pr_num=$(get_pr_num)
|
||||||
|
if [ -n "$pr_num" ]; then
|
||||||
|
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['pr']=True;print(json.dumps(d))")
|
||||||
|
else
|
||||||
|
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['pr']=False;print(json.dumps(d))")
|
||||||
|
status="UNVERIFIED"
|
||||||
|
details+=("no PR found for branch ${branch}")
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check 3: PR has real file changes
|
||||||
|
if [ -n "$pr_num" ]; then
|
||||||
|
file_count=$(check_pr_files "$pr_num")
|
||||||
|
if [ "${file_count:-0}" -gt 0 ]; then
|
||||||
|
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['files']=True;print(json.dumps(d))")
|
||||||
|
else
|
||||||
|
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['files']=False;print(json.dumps(d))")
|
||||||
|
status="UNVERIFIED"
|
||||||
|
details+=("PR #${pr_num} has 0 changed files")
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check 4: PR is mergeable
|
||||||
|
if [ "$(check_pr_mergeable "$pr_num")" = "true" ]; then
|
||||||
|
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['mergeable']=True;print(json.dumps(d))")
|
||||||
|
else
|
||||||
|
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['mergeable']=False;print(json.dumps(d))")
|
||||||
|
status="UNVERIFIED"
|
||||||
|
details+=("PR #${pr_num} is not mergeable")
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['files']=None;d['mergeable']=None;print(json.dumps(d))")
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check 5: completion comment from agent
|
||||||
|
if check_completion_comment; then
|
||||||
|
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['comment']=True;print(json.dumps(d))")
|
||||||
|
else
|
||||||
|
checks_json=$(echo "$checks_json" | python3 -c "import sys,json;d=json.load(sys.stdin);d['comment']=False;print(json.dumps(d))")
|
||||||
|
status="UNVERIFIED"
|
||||||
|
details+=("no completion comment from ${agent_name} on issue #${issue_num}")
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Build detail string
|
||||||
|
detail_str=$(IFS="; "; echo "${details[*]:-all checks passed}")
|
||||||
|
|
||||||
|
# ── Output ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
result=$(python3 -c "
|
||||||
|
import json
|
||||||
|
print(json.dumps({
|
||||||
|
'status': '$status',
|
||||||
|
'repo': '${repo_owner}/${repo_name}',
|
||||||
|
'issue': $issue_num,
|
||||||
|
'branch': '$branch',
|
||||||
|
'agent': '$agent_name',
|
||||||
|
'pr': '$pr_num',
|
||||||
|
'checks': $checks_json,
|
||||||
|
'details': '$detail_str',
|
||||||
|
'ts': '$ts'
|
||||||
|
}, indent=2))
|
||||||
|
")
|
||||||
|
|
||||||
|
printf '%s\n' "$result"
|
||||||
|
|
||||||
|
# Append to JSONL log
|
||||||
|
printf '%s\n' "$result" >> "$VERIFY_LOG"
|
||||||
|
|
||||||
|
if [ "$status" = "VERIFIED" ]; then
|
||||||
|
exit 0
|
||||||
|
else
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
45
bin/kaizen-retro.sh
Executable file
@@ -0,0 +1,45 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# kaizen-retro.sh — Automated retrospective after every burn cycle.
|
||||||
|
#
|
||||||
|
# Runs daily after the morning report.
|
||||||
|
# Analyzes success rates by agent, repo, and issue type.
|
||||||
|
# Identifies max-attempts issues, generates ONE concrete improvement,
|
||||||
|
# and posts the retro to Telegram + the master morning-report issue.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./bin/kaizen-retro.sh [--dry-run]
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
REPO_ROOT="${SCRIPT_DIR%/bin}"
|
||||||
|
PYTHON="${PYTHON3:-python3}"
|
||||||
|
|
||||||
|
# Source local env if available so TELEGRAM_BOT_TOKEN is picked up
|
||||||
|
HOME_DIR="${HOME:-$(eval echo ~$(whoami))}"
|
||||||
|
for env_file in "$HOME_DIR/.hermes/.env" "$HOME_DIR/.timmy/.env" "$REPO_ROOT/.env"; do
|
||||||
|
if [ -f "$env_file" ]; then
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
set -a
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "$env_file"
|
||||||
|
set +a
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# If the configured Gitea URL is unreachable but localhost works, prefer localhost
|
||||||
|
if ! curl -sf "${GITEA_URL:-http://localhost:3000}/api/v1/version" >/dev/null 2>&1; then
|
||||||
|
if curl -sf http://localhost:3000/api/v1/version >/dev/null 2>&1; then
|
||||||
|
export GITEA_URL="http://localhost:3000"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Ensure the Python script exists
|
||||||
|
RETRO_PY="$REPO_ROOT/scripts/kaizen_retro.py"
|
||||||
|
if [ ! -f "$RETRO_PY" ]; then
|
||||||
|
echo "ERROR: kaizen_retro.py not found at $RETRO_PY" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Run
|
||||||
|
exec "$PYTHON" "$RETRO_PY" "$@"
|
||||||
191
bin/pr-checklist.py
Normal file
@@ -0,0 +1,191 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""pr-checklist.py -- Automated PR quality gate for Gitea CI.
|
||||||
|
|
||||||
|
Enforces the review standards that agents skip when left to self-approve.
|
||||||
|
Runs in CI on every pull_request event. Exits non-zero on any failure.
|
||||||
|
|
||||||
|
Checks:
|
||||||
|
1. PR has >0 file changes (no empty PRs)
|
||||||
|
2. PR branch is not behind base branch
|
||||||
|
3. PR does not bundle >3 unrelated issues
|
||||||
|
4. Changed .py files pass syntax check (python -c import)
|
||||||
|
5. Changed .sh files are executable
|
||||||
|
6. PR body references an issue number
|
||||||
|
7. At least 1 non-author review exists (warning only)
|
||||||
|
|
||||||
|
Refs: #393 (PERPLEXITY-08), Epic #385
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def fail(msg: str) -> None:
|
||||||
|
print(f"FAIL: {msg}", file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
def warn(msg: str) -> None:
|
||||||
|
print(f"WARN: {msg}", file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
def ok(msg: str) -> None:
|
||||||
|
print(f" OK: {msg}")
|
||||||
|
|
||||||
|
|
||||||
|
def get_changed_files() -> list[str]:
|
||||||
|
"""Return list of files changed in this PR vs base branch."""
|
||||||
|
base = os.environ.get("GITHUB_BASE_REF", "main")
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
["git", "diff", "--name-only", f"origin/{base}...HEAD"],
|
||||||
|
capture_output=True, text=True, check=True,
|
||||||
|
)
|
||||||
|
return [f for f in result.stdout.strip().splitlines() if f]
|
||||||
|
except subprocess.CalledProcessError:
|
||||||
|
# Fallback: diff against HEAD~1
|
||||||
|
result = subprocess.run(
|
||||||
|
["git", "diff", "--name-only", "HEAD~1"],
|
||||||
|
capture_output=True, text=True, check=True,
|
||||||
|
)
|
||||||
|
return [f for f in result.stdout.strip().splitlines() if f]
|
||||||
|
|
||||||
|
|
||||||
|
def check_has_changes(files: list[str]) -> bool:
|
||||||
|
"""Check 1: PR has >0 file changes."""
|
||||||
|
if not files:
|
||||||
|
fail("PR has 0 file changes. Empty PRs are not allowed.")
|
||||||
|
return False
|
||||||
|
ok(f"PR changes {len(files)} file(s)")
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def check_not_behind_base() -> bool:
|
||||||
|
"""Check 2: PR branch is not behind base."""
|
||||||
|
base = os.environ.get("GITHUB_BASE_REF", "main")
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
["git", "rev-list", "--count", f"HEAD..origin/{base}"],
|
||||||
|
capture_output=True, text=True, check=True,
|
||||||
|
)
|
||||||
|
behind = int(result.stdout.strip())
|
||||||
|
if behind > 0:
|
||||||
|
fail(f"Branch is {behind} commit(s) behind {base}. Rebase or merge.")
|
||||||
|
return False
|
||||||
|
ok(f"Branch is up-to-date with {base}")
|
||||||
|
return True
|
||||||
|
except (subprocess.CalledProcessError, ValueError):
|
||||||
|
warn("Could not determine if branch is behind base (git fetch may be needed)")
|
||||||
|
return True # Don't block on CI fetch issues
|
||||||
|
|
||||||
|
|
||||||
|
def check_issue_bundling(pr_body: str) -> bool:
|
||||||
|
"""Check 3: PR does not bundle >3 unrelated issues."""
|
||||||
|
issue_refs = set(re.findall(r"#(\d+)", pr_body))
|
||||||
|
if len(issue_refs) > 3:
|
||||||
|
fail(f"PR references {len(issue_refs)} issues ({', '.join(sorted(issue_refs))}). "
|
||||||
|
"Max 3 per PR to prevent bundling. Split into separate PRs.")
|
||||||
|
return False
|
||||||
|
ok(f"PR references {len(issue_refs)} issue(s) (max 3)")
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def check_python_syntax(files: list[str]) -> bool:
|
||||||
|
"""Check 4: Changed .py files have valid syntax."""
|
||||||
|
py_files = [f for f in files if f.endswith(".py") and Path(f).exists()]
|
||||||
|
if not py_files:
|
||||||
|
ok("No Python files changed")
|
||||||
|
return True
|
||||||
|
|
||||||
|
all_ok = True
|
||||||
|
for f in py_files:
|
||||||
|
result = subprocess.run(
|
||||||
|
[sys.executable, "-c", f"import ast; ast.parse(open('{f}').read())"],
|
||||||
|
capture_output=True, text=True,
|
||||||
|
)
|
||||||
|
if result.returncode != 0:
|
||||||
|
fail(f"Syntax error in {f}: {result.stderr.strip()[:200]}")
|
||||||
|
all_ok = False
|
||||||
|
|
||||||
|
if all_ok:
|
||||||
|
ok(f"All {len(py_files)} Python file(s) pass syntax check")
|
||||||
|
return all_ok
|
||||||
|
|
||||||
|
|
||||||
|
def check_shell_executable(files: list[str]) -> bool:
|
||||||
|
"""Check 5: Changed .sh files are executable."""
|
||||||
|
sh_files = [f for f in files if f.endswith(".sh") and Path(f).exists()]
|
||||||
|
if not sh_files:
|
||||||
|
ok("No shell scripts changed")
|
||||||
|
return True
|
||||||
|
|
||||||
|
all_ok = True
|
||||||
|
for f in sh_files:
|
||||||
|
if not os.access(f, os.X_OK):
|
||||||
|
fail(f"{f} is not executable. Run: chmod +x {f}")
|
||||||
|
all_ok = False
|
||||||
|
|
||||||
|
if all_ok:
|
||||||
|
ok(f"All {len(sh_files)} shell script(s) are executable")
|
||||||
|
return all_ok
|
||||||
|
|
||||||
|
|
||||||
|
def check_issue_reference(pr_body: str) -> bool:
|
||||||
|
"""Check 6: PR body references an issue number."""
|
||||||
|
if re.search(r"#\d+", pr_body):
|
||||||
|
ok("PR body references at least one issue")
|
||||||
|
return True
|
||||||
|
fail("PR body does not reference any issue (e.g. #123). "
|
||||||
|
"Every PR must trace to an issue.")
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
print("=" * 60)
|
||||||
|
print("PR Checklist — Automated Quality Gate")
|
||||||
|
print("=" * 60)
|
||||||
|
print()
|
||||||
|
|
||||||
|
# Get PR body from env or git log
|
||||||
|
pr_body = os.environ.get("PR_BODY", "")
|
||||||
|
if not pr_body:
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
["git", "log", "--format=%B", "-1"],
|
||||||
|
capture_output=True, text=True, check=True,
|
||||||
|
)
|
||||||
|
pr_body = result.stdout
|
||||||
|
except subprocess.CalledProcessError:
|
||||||
|
pr_body = ""
|
||||||
|
|
||||||
|
files = get_changed_files()
|
||||||
|
failures = 0
|
||||||
|
|
||||||
|
checks = [
|
||||||
|
check_has_changes(files),
|
||||||
|
check_not_behind_base(),
|
||||||
|
check_issue_bundling(pr_body),
|
||||||
|
check_python_syntax(files),
|
||||||
|
check_shell_executable(files),
|
||||||
|
check_issue_reference(pr_body),
|
||||||
|
]
|
||||||
|
|
||||||
|
failures = sum(1 for c in checks if not c)
|
||||||
|
|
||||||
|
print()
|
||||||
|
print("=" * 60)
|
||||||
|
if failures:
|
||||||
|
print(f"RESULT: {failures} check(s) FAILED")
|
||||||
|
print("Fix the issues above and push again.")
|
||||||
|
return 1
|
||||||
|
else:
|
||||||
|
print("RESULT: All checks passed")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -7,7 +7,7 @@ Purpose:
|
|||||||
|
|
||||||
## What it is
|
## What it is
|
||||||
|
|
||||||
Code Claw is a separate local runtime from Hermes/OpenClaw.
|
Code Claw is a separate local runtime from Hermes.
|
||||||
|
|
||||||
Current lane:
|
Current lane:
|
||||||
- runtime: local patched `~/code-claw`
|
- runtime: local patched `~/code-claw`
|
||||||
|
|||||||
212
cron/jobs-backup-2026-04-10.json
Normal file
@@ -0,0 +1,212 @@
|
|||||||
|
[
|
||||||
|
{
|
||||||
|
"job_id": "9e0624269ba7",
|
||||||
|
"name": "Triage Heartbeat",
|
||||||
|
"schedule": "every 15m",
|
||||||
|
"state": "paused"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "e29eda4a8548",
|
||||||
|
"name": "PR Review Sweep",
|
||||||
|
"schedule": "every 30m",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "a77a87392582",
|
||||||
|
"name": "Health Monitor",
|
||||||
|
"schedule": "every 5m",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "5e9d952871bc",
|
||||||
|
"name": "Agent Status Check",
|
||||||
|
"schedule": "every 10m",
|
||||||
|
"state": "paused"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "36fb2f630a17",
|
||||||
|
"name": "Hermes Philosophy Loop",
|
||||||
|
"schedule": "every 1440m",
|
||||||
|
"state": "paused"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "b40a96a2f48c",
|
||||||
|
"name": "wolf-eval-cycle",
|
||||||
|
"schedule": "every 240m",
|
||||||
|
"state": "paused"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "4204e568b862",
|
||||||
|
"name": "Burn Mode \u2014 Timmy Orchestrator",
|
||||||
|
"schedule": "every 15m",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "0944a976d034",
|
||||||
|
"name": "Burn Mode",
|
||||||
|
"schedule": "every 15m",
|
||||||
|
"state": "paused"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "62016b960fa0",
|
||||||
|
"name": "velocity-engine",
|
||||||
|
"schedule": "every 30m",
|
||||||
|
"state": "paused"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "e9d49eeff79c",
|
||||||
|
"name": "weekly-skill-extraction",
|
||||||
|
"schedule": "every 10080m",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "75c74a5bb563",
|
||||||
|
"name": "tower-tick",
|
||||||
|
"schedule": "every 1m",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "390a19054d4c",
|
||||||
|
"name": "Burn Deadman",
|
||||||
|
"schedule": "every 30m",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "05e3c13498fa",
|
||||||
|
"name": "Morning Report \u2014 Burn Mode",
|
||||||
|
"schedule": "0 6 * * *",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "64fe44b512b9",
|
||||||
|
"name": "evennia-morning-report",
|
||||||
|
"schedule": "0 9 * * *",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "3896a7fd9747",
|
||||||
|
"name": "Gitea Priority Inbox",
|
||||||
|
"schedule": "every 3m",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "f64c2709270a",
|
||||||
|
"name": "Config Drift Guard",
|
||||||
|
"schedule": "every 30m",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "fc6a75b7102a",
|
||||||
|
"name": "Gitea Event Watcher",
|
||||||
|
"schedule": "every 2m",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "12e59648fb06",
|
||||||
|
"name": "Burndown Night Watcher",
|
||||||
|
"schedule": "every 15m",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "35d3ada9cf8f",
|
||||||
|
"name": "Mempalace Forge \u2014 Issue Analysis",
|
||||||
|
"schedule": "every 60m",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "190b6fb8dc91",
|
||||||
|
"name": "Mempalace Watchtower \u2014 Fleet Health",
|
||||||
|
"schedule": "every 30m",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "710ab589813c",
|
||||||
|
"name": "Ezra Health Monitor",
|
||||||
|
"schedule": "every 15m",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "a0a9cce4575c",
|
||||||
|
"name": "daily-poka-yoke-ultraplan-awesometools",
|
||||||
|
"schedule": "every 1440m",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "adc3a51457bd",
|
||||||
|
"name": "vps-agent-dispatch",
|
||||||
|
"schedule": "every 10m",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "afd2c4eac44d",
|
||||||
|
"name": "Project Mnemosyne Nightly Burn v2",
|
||||||
|
"schedule": "*/30 * * * *",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "f3a3c2832af0",
|
||||||
|
"name": "gemma4-multimodal-worker",
|
||||||
|
"schedule": "once in 15m",
|
||||||
|
"state": "completed"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "c17a85c19838",
|
||||||
|
"name": "know-thy-father-analyzer",
|
||||||
|
"schedule": "0 * * * *",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "2490fc01a14d",
|
||||||
|
"name": "Testament Burn - 10min work loop",
|
||||||
|
"schedule": "*/10 * * * *",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "f5e858159d97",
|
||||||
|
"name": "Timmy Foundation Burn \u2014 15min PR loop",
|
||||||
|
"schedule": "*/15 * * * *",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "5e262fb9bdce",
|
||||||
|
"name": "nightwatch-health-monitor",
|
||||||
|
"schedule": "*/15 * * * *",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "f2b33a9dcf96",
|
||||||
|
"name": "nightwatch-mempalace-mine",
|
||||||
|
"schedule": "0 */2 * * *",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "82cb9e76c54d",
|
||||||
|
"name": "nightwatch-backlog-burn",
|
||||||
|
"schedule": "0 */4 * * *",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "d20e42a52863",
|
||||||
|
"name": "beacon-sprint",
|
||||||
|
"schedule": "*/15 * * * *",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "579269489961",
|
||||||
|
"name": "testament-story",
|
||||||
|
"schedule": "*/15 * * * *",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "2e5f9140d1ab",
|
||||||
|
"name": "nightwatch-research",
|
||||||
|
"schedule": "0 */2 * * *",
|
||||||
|
"state": "scheduled"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"job_id": "aeba92fd65e6",
|
||||||
|
"name": "timmy-dreams",
|
||||||
|
"schedule": "30 5 * * *",
|
||||||
|
"state": "scheduled"
|
||||||
|
}
|
||||||
|
]
|
||||||
@@ -137,7 +137,38 @@
|
|||||||
"paused_reason": null,
|
"paused_reason": null,
|
||||||
"skills": [],
|
"skills": [],
|
||||||
"skill": null
|
"skill": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "kaizen-retro-349",
|
||||||
|
"name": "Kaizen Retro",
|
||||||
|
"prompt": "Run the automated burn-cycle retrospective. Execute: cd /root/wizards/ezra/workspace/timmy-config && ./bin/kaizen-retro.sh",
|
||||||
|
"model": "hermes3:latest",
|
||||||
|
"provider": "ollama",
|
||||||
|
"base_url": "http://localhost:11434/v1",
|
||||||
|
"schedule": {
|
||||||
|
"kind": "interval",
|
||||||
|
"minutes": 1440,
|
||||||
|
"display": "every 1440m"
|
||||||
|
},
|
||||||
|
"schedule_display": "daily at 07:30",
|
||||||
|
"repeat": {
|
||||||
|
"times": null,
|
||||||
|
"completed": 0
|
||||||
|
},
|
||||||
|
"enabled": true,
|
||||||
|
"created_at": "2026-04-07T15:30:00.000000Z",
|
||||||
|
"next_run_at": "2026-04-08T07:30:00.000000Z",
|
||||||
|
"last_run_at": null,
|
||||||
|
"last_status": null,
|
||||||
|
"last_error": null,
|
||||||
|
"deliver": "local",
|
||||||
|
"origin": null,
|
||||||
|
"state": "scheduled",
|
||||||
|
"paused_at": null,
|
||||||
|
"paused_reason": null,
|
||||||
|
"skills": [],
|
||||||
|
"skill": null
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"updated_at": "2026-04-07T15:00:00+00:00"
|
"updated_at": "2026-04-07T15:00:00+00:00"
|
||||||
}
|
}
|
||||||
|
|||||||
14
cron/vps/allegro-crontab-backup.txt
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
0 6 * * * /bin/bash /root/wizards/scripts/model_download_guard.sh >> /var/log/model_guard.log 2>&1
|
||||||
|
|
||||||
|
# Allegro Hybrid Heartbeat — quick wins every 15 min
|
||||||
|
*/15 * * * * /usr/bin/python3 /root/allegro/heartbeat_daemon.py >> /var/log/allegro_heartbeat.log 2>&1
|
||||||
|
|
||||||
|
# Allegro Burn Mode Cron Jobs - Deployed via issue #894
|
||||||
|
|
||||||
|
0 6 * * * cd /root/.hermes && python3 -c "import hermes_agent; from hermes_tools import terminal; output = terminal('echo \"Morning Report: $(date)\"'); print(output.get('output', ''))" >> /root/.hermes/logs/morning-report-$(date +\%Y\%m\%d).log 2>&1 # Allegro Morning Report at 0600
|
||||||
|
|
||||||
|
0,30 * * * * cd /root/.hermes && python3 /root/.hermes/retry_wrapper.py "python3 allegro/quick-lane-check.py" >> burn-logs/quick-lane-$(date +\%Y\%m\%d).log 2>&1 # Allegro Burn Loop #1 (with retry)
|
||||||
|
15,45 * * * * cd /root/.hermes && python3 /root/.hermes/retry_wrapper.py "python3 allegro/burn-mode-validator.py" >> burn-logs/validator-$(date +\%Y\%m\%d).log 2>&1 # Allegro Burn Loop #2 (with retry)
|
||||||
|
|
||||||
|
*/2 * * * * /root/wizards/bezalel/dead_man_monitor.sh
|
||||||
|
*/2 * * * * /root/wizards/allegro/bin/config-deadman.sh
|
||||||
10
cron/vps/bezalel-crontab-backup.txt
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
0 2 * * * /root/wizards/bezalel/run_nightly_watch.sh
|
||||||
|
0 3 * * * /root/wizards/bezalel/mempalace_nightly.sh
|
||||||
|
*/10 * * * * pgrep -f "act_runner daemon" > /dev/null || (cd /opt/gitea-runner && nohup ./act_runner daemon > /var/log/gitea-runner.log 2>&1 &)
|
||||||
|
30 3 * * * /root/wizards/bezalel/backup_databases.sh
|
||||||
|
*/15 * * * * /root/wizards/bezalel/meta_heartbeat.sh
|
||||||
|
0 4 * * * /root/wizards/bezalel/secret_guard.sh
|
||||||
|
0 4 * * * /usr/bin/env bash /root/timmy-home/scripts/backup_pipeline.sh >> /var/log/timmy/backup_pipeline_cron.log 2>&1
|
||||||
|
0 6 * * * /usr/bin/python3 /root/wizards/bezalel/ultraplan.py >> /var/log/bezalel-ultraplan.log 2>&1
|
||||||
|
@reboot /root/wizards/bezalel/emacs-daemon-start.sh
|
||||||
|
@reboot /root/wizards/bezalel/ngircd-start.sh
|
||||||
13
cron/vps/ezra-crontab-backup.txt
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
# Burn Mode Cycles — 15 min autonomous loops
|
||||||
|
*/15 * * * * /root/wizards/ezra/bin/burn-mode.sh >> /root/wizards/ezra/reports/burn-cron.log 2>&1
|
||||||
|
|
||||||
|
# Household Snapshots — automated heartbeats and snapshots
|
||||||
|
# Ezra Self-Improvement Automation Suite
|
||||||
|
*/5 * * * * /usr/bin/python3 /root/wizards/ezra/tools/gitea_monitor.py >> /root/wizards/ezra/reports/gitea-monitor.log 2>&1
|
||||||
|
*/5 * * * * /usr/bin/python3 /root/wizards/ezra/tools/awareness_loop.py >> /root/wizards/ezra/reports/awareness-loop.log 2>&1
|
||||||
|
*/10 * * * * /usr/bin/python3 /root/wizards/ezra/tools/cron_health_monitor.py >> /root/wizards/ezra/reports/cron-health.log 2>&1
|
||||||
|
0 6 * * * /usr/bin/python3 /root/wizards/ezra/tools/morning_kt_compiler.py >> /root/wizards/ezra/reports/morning-kt.log 2>&1
|
||||||
|
5 6 * * * /usr/bin/python3 /root/wizards/ezra/tools/burndown_generator.py >> /root/wizards/ezra/reports/burndown.log 2>&1
|
||||||
|
0 3 * * * /root/wizards/ezra/mempalace_nightly.sh >> /var/log/ezra_mempalace_cron.log 2>&1
|
||||||
|
*/15 * * * * GITEA_TOKEN=6de6aa...1117 /root/wizards/ezra/dispatch-direct.sh >> /root/wizards/ezra/dispatch-cron.log 2>&1
|
||||||
|
|
||||||
110
docs/FLEET_BEHAVIOUR_HARDENING.md
Normal file
@@ -0,0 +1,110 @@
|
|||||||
|
# Fleet Behaviour Hardening — Review & Action Plan
|
||||||
|
|
||||||
|
**Author:** @perplexity
|
||||||
|
**Date:** 2026-04-08
|
||||||
|
**Context:** Alexander asked: "Is it the memory system or the behaviour guardrails?"
|
||||||
|
**Answer:** It's the guardrails. The memory system is adequate. The enforcement machinery is aspirational.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Diagnosis: Why the Fleet Isn't Smart Enough
|
||||||
|
|
||||||
|
After auditing SOUL.md, config.yaml, all 8 playbooks, the orchestrator, the guard scripts, and the v7.0.0 checkin, the pattern is clear:
|
||||||
|
|
||||||
|
**The fleet has excellent design documents and broken enforcement.**
|
||||||
|
|
||||||
|
| Layer | Design Quality | Enforcement Quality | Gap |
|
||||||
|
|---|---|---|---|
|
||||||
|
| SOUL.md | Excellent | None — no code reads it at runtime | Philosophy without machinery |
|
||||||
|
| Playbooks (7 yaml) | Good lane map | Not invoked by orchestrator | Playbooks exist but nobody calls them |
|
||||||
|
| Guard scripts (9) | Solid code | 1 of 9 wired (#395 audit) | 89% of guards are dead code |
|
||||||
|
| Orchestrator | Sound design | Gateway dispatch is a no-op (#391) | Assigns issues but doesn't trigger work |
|
||||||
|
| Cycle Guard | Good 10-min rule | No cron/loop calls it | Discipline without enforcement |
|
||||||
|
| PR Reviewer | Clear rules | Runs every 30m (if scheduled) | Only guard that might actually fire |
|
||||||
|
| Memory (MemPalace) | Working code | Retrieval enforcer wired | Actually operational |
|
||||||
|
|
||||||
|
### The Core Problem
|
||||||
|
|
||||||
|
Agents pick up issues and produce output, but there is **no pre-task checklist** and **no post-task quality gate**. An agent can:
|
||||||
|
|
||||||
|
1. Start work without checking if someone else already did it
|
||||||
|
2. Produce output without running tests
|
||||||
|
3. Submit a PR without verifying it addresses the issue
|
||||||
|
4. Work for hours on something out of scope
|
||||||
|
5. Create duplicate branches/PRs without detection
|
||||||
|
|
||||||
|
The SOUL.md says "grounding before generation" but no code enforces it.
|
||||||
|
The playbooks define lanes but the orchestrator doesn't load them.
|
||||||
|
The guards exist but nothing calls them.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What the Fleet Needs (Priority Order)
|
||||||
|
|
||||||
|
### 1. Pre-Task Gate (MISSING — this PR adds it)
|
||||||
|
|
||||||
|
Before an agent starts any issue:
|
||||||
|
- [ ] Check if issue is already assigned to another agent
|
||||||
|
- [ ] Check if a branch already exists for this issue
|
||||||
|
- [ ] Check if a PR already exists for this issue
|
||||||
|
- [ ] Load relevant MemPalace context (retrieval enforcer)
|
||||||
|
- [ ] Verify the agent has the right lane for this work (playbook check)
|
||||||
|
|
||||||
|
### 2. Post-Task Gate (MISSING — this PR adds it)
|
||||||
|
|
||||||
|
Before an agent submits a PR:
|
||||||
|
- [ ] Verify the diff addresses the issue title/body
|
||||||
|
- [ ] Run syntax_guard.py on changed files
|
||||||
|
- [ ] Check for duplicate PRs targeting the same issue
|
||||||
|
- [ ] Verify branch name follows convention
|
||||||
|
- [ ] Run tests if they exist for changed files
|
||||||
|
|
||||||
|
### 3. Wire the Existing Guards (8 of 9 are dead code)
|
||||||
|
|
||||||
|
Per #395 audit:
|
||||||
|
- Pre-commit hooks: need symlink on every machine
|
||||||
|
- Cycle guard: need cron/loop integration
|
||||||
|
- Forge health check: need cron entry
|
||||||
|
- Smoke test + deploy validate: need deploy script integration
|
||||||
|
|
||||||
|
### 4. Orchestrator Dispatch Actually Works
|
||||||
|
|
||||||
|
Per #391 audit: the orchestrator scores and assigns but the gateway dispatch just writes to `/tmp/hermes-dispatch.log`. Nobody reads that file. The dispatch needs to either:
|
||||||
|
- Trigger `hermes` CLI on the target machine, or
|
||||||
|
- Post a webhook that the agent loop picks up
|
||||||
|
|
||||||
|
### 5. Agent Self-Assessment Loop
|
||||||
|
|
||||||
|
After completing work, agents should answer:
|
||||||
|
- Did I address the issue as stated?
|
||||||
|
- Did I stay in scope?
|
||||||
|
- Did I check the palace for prior work?
|
||||||
|
- Did I run verification?
|
||||||
|
|
||||||
|
This is what SOUL.md calls "the apparatus that gives these words teeth."
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What's Working (Don't Touch)
|
||||||
|
|
||||||
|
- **MemPalace sovereign_store.py** — SQLite + FTS5 + HRR, operational
|
||||||
|
- **Retrieval enforcer** — wired to SovereignStore as of 14 hours ago
|
||||||
|
- **Wake-up protocol** — palace-first boot sequence
|
||||||
|
- **PR reviewer playbook** — clear rules, well-scoped
|
||||||
|
- **Issue triager playbook** — comprehensive lane map with 11 agents
|
||||||
|
- **Cycle guard code** — solid 10-min slice discipline (just needs wiring)
|
||||||
|
- **Config drift guard** — active cron, working
|
||||||
|
- **Dead man switch** — active, working
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Recommendation
|
||||||
|
|
||||||
|
The memory system is not the bottleneck. The behaviour guardrails are. Specifically:
|
||||||
|
|
||||||
|
1. **Add `task_gate.py`** — pre-task and post-task quality gates that every agent loop calls
|
||||||
|
2. **Wire cycle_guard.py** — add start/complete calls to agent loop
|
||||||
|
3. **Wire pre-commit hooks** — deploy script should symlink on provision
|
||||||
|
4. **Fix orchestrator dispatch** — make it actually trigger work, not just log
|
||||||
|
|
||||||
|
This PR adds item 1. Items 2-4 need SSH access and are flagged for Timmy/Allegro.
|
||||||
141
docs/MEMORY_ARCHITECTURE.md
Normal file
@@ -0,0 +1,141 @@
|
|||||||
|
# Memory Architecture
|
||||||
|
|
||||||
|
> How Timmy remembers, recalls, and learns — without hallucinating.
|
||||||
|
|
||||||
|
Refs: Epic #367 | Sub-issues #368, #369, #370, #371, #372
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Timmy's memory system uses a **Memory Palace** architecture — a structured, file-backed knowledge store organized into rooms and drawers. When faced with a recall question, the agent checks its palace *before* generating from scratch.
|
||||||
|
|
||||||
|
This document defines the retrieval order, storage layers, and data flow that make this work.
|
||||||
|
|
||||||
|
## Retrieval Order (L0–L5)
|
||||||
|
|
||||||
|
When the agent receives a prompt that looks like a recall question ("what did we do?", "what's the status of X?"), the retrieval enforcer intercepts it and walks through layers in order:
|
||||||
|
|
||||||
|
| Layer | Source | Question Answered | Short-circuits? |
|
||||||
|
|-------|--------|-------------------|------------------|
|
||||||
|
| L0 | `identity.txt` | Who am I? What are my mandates? | No (always loaded) |
|
||||||
|
| L1 | Palace rooms/drawers | What do I know about this topic? | Yes, if hit |
|
||||||
|
| L2 | Session scratchpad | What have I learned this session? | Yes, if hit |
|
||||||
|
| L3 | Artifact retrieval (Gitea API) | Can I fetch the actual issue/file/log? | Yes, if hit |
|
||||||
|
| L4 | Procedures/playbooks | Is there a documented way to do this? | Yes, if hit |
|
||||||
|
| L5 | Free generation | (Only when L0–L4 are exhausted) | N/A |
|
||||||
|
|
||||||
|
**Key principle:** The agent never reaches L5 (free generation) if any prior layer has relevant data. This eliminates hallucination for recall-style queries.
|
||||||
|
|
||||||
|
## Storage Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
~/.mempalace/
|
||||||
|
identity.txt # L0: Who I am, mandates, personality
|
||||||
|
rooms/
|
||||||
|
projects/
|
||||||
|
timmy-config.md # What I know about timmy-config
|
||||||
|
hermes-agent.md # What I know about hermes-agent
|
||||||
|
people/
|
||||||
|
alexander.md # Working relationship context
|
||||||
|
architecture/
|
||||||
|
fleet.md # Fleet system knowledge
|
||||||
|
mempalace.md # Self-knowledge about this system
|
||||||
|
config/
|
||||||
|
mempalace.yaml # Palace configuration
|
||||||
|
|
||||||
|
~/.hermes/
|
||||||
|
scratchpad/
|
||||||
|
{session_id}.json # L2: Ephemeral session context
|
||||||
|
```
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
### 1. Memory Palace Skill (`mempalace.py`) — #368
|
||||||
|
|
||||||
|
Core data structures:
|
||||||
|
- `PalaceRoom`: A named collection of drawers (topics)
|
||||||
|
- `Mempalace`: The top-level palace with room management
|
||||||
|
- Factory constructors: `for_issue_analysis()`, `for_health_check()`, `for_code_review()`
|
||||||
|
|
||||||
|
### 2. Retrieval Enforcer (`retrieval_enforcer.py`) — #369
|
||||||
|
|
||||||
|
Middleware that intercepts recall-style prompts:
|
||||||
|
1. Detects recall patterns ("what did", "status of", "last time we")
|
||||||
|
2. Walks L0→L4 in order, short-circuiting on first hit
|
||||||
|
3. Only allows free generation (L5) when all layers return empty
|
||||||
|
4. Produces an honest fallback: "I don't have this in my memory palace."
|
||||||
|
|
||||||
|
### 3. Session Scratchpad (`scratchpad.py`) — #370
|
||||||
|
|
||||||
|
Ephemeral, session-scoped working memory:
|
||||||
|
- Write-append only during a session
|
||||||
|
- Entries have TTL (default: 1 hour)
|
||||||
|
- Queried at L2 in retrieval chain
|
||||||
|
- Never auto-promoted to palace
|
||||||
|
|
||||||
|
### 4. Memory Promotion — #371
|
||||||
|
|
||||||
|
Explicit promotion from scratchpad to palace:
|
||||||
|
- Agent must call `promote_to_palace()` with a reason
|
||||||
|
- Dedup check against target drawer
|
||||||
|
- Summary required (raw tool output never stored)
|
||||||
|
- Conflict detection when new memory contradicts existing
|
||||||
|
|
||||||
|
### 5. Wake-Up Protocol (`wakeup.py`) — #372
|
||||||
|
|
||||||
|
Boot sequence for new sessions:
|
||||||
|
```
|
||||||
|
Session Start
|
||||||
|
│
|
||||||
|
├─ L0: Load identity.txt
|
||||||
|
├─ L1: Scan palace rooms for active context
|
||||||
|
├─ L1.5: Surface promoted memories from last session
|
||||||
|
├─ L2: Load surviving scratchpad entries
|
||||||
|
│
|
||||||
|
└─ Ready: agent knows who it is, what it was doing, what it learned
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────────┐
|
||||||
|
│ User Prompt │
|
||||||
|
└────────┬─────────┘
|
||||||
|
│
|
||||||
|
┌────────┴─────────┐
|
||||||
|
│ Recall Detector │
|
||||||
|
└────┬───────┬─────┘
|
||||||
|
│ │
|
||||||
|
[recall] [not recall]
|
||||||
|
│ │
|
||||||
|
┌───────┴────┐ ┌──┬─┴───────┐
|
||||||
|
│ Retrieval │ │ Normal Flow │
|
||||||
|
│ Enforcer │ └─────────────┘
|
||||||
|
│ L0→L1→L2 │
|
||||||
|
│ →L3→L4→L5│
|
||||||
|
└──────┬─────┘
|
||||||
|
│
|
||||||
|
┌──────┴─────┐
|
||||||
|
│ Response │
|
||||||
|
│ (grounded) │
|
||||||
|
└────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Anti-Patterns
|
||||||
|
|
||||||
|
| Don't | Do Instead |
|
||||||
|
|-------|------------|
|
||||||
|
| Generate from vibes when palace has data | Check palace first (L1) |
|
||||||
|
| Auto-promote everything to palace | Require explicit `promote_to_palace()` with reason |
|
||||||
|
| Store raw API responses as memories | Summarize before storing |
|
||||||
|
| Hallucinate when palace is empty | Say "I don't have this in my memory palace" |
|
||||||
|
| Dump entire palace on wake-up | Selective loading based on session context |
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
| Component | Issue | PR | Status |
|
||||||
|
|-----------|-------|----|--------|
|
||||||
|
| Skill port | #368 | #374 | In Review |
|
||||||
|
| Retrieval enforcer | #369 | #374 | In Review |
|
||||||
|
| Session scratchpad | #370 | #374 | In Review |
|
||||||
|
| Memory promotion | #371 | — | Open |
|
||||||
|
| Wake-up protocol | #372 | #374 | In Review |
|
||||||
@@ -3,7 +3,7 @@
|
|||||||
Purpose:
|
Purpose:
|
||||||
- stand up the third wizard house as a Kimi-backed coding worker
|
- stand up the third wizard house as a Kimi-backed coding worker
|
||||||
- keep Hermes as the durable harness
|
- keep Hermes as the durable harness
|
||||||
- treat OpenClaw as optional shell frontage, not the bones
|
- Hermes is the durable harness — no intermediary gateway layers
|
||||||
|
|
||||||
Local proof already achieved:
|
Local proof already achieved:
|
||||||
|
|
||||||
@@ -40,5 +40,5 @@ bin/deploy-allegro-house.sh root@167.99.126.228
|
|||||||
|
|
||||||
Important nuance:
|
Important nuance:
|
||||||
- the Hermes/Kimi lane is the proven path
|
- the Hermes/Kimi lane is the proven path
|
||||||
- direct embedded OpenClaw Kimi model routing was not yet reliable locally
|
- direct embedded Kimi model routing was not yet reliable locally
|
||||||
- so the remote deployment keeps the minimal, proven architecture: Hermes house first
|
- so the remote deployment keeps the minimal, proven architecture: Hermes house first
|
||||||
@@ -81,17 +81,6 @@ launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/ai.hermes.gateway.plist
|
|||||||
- Old-state risk:
|
- Old-state risk:
|
||||||
- same class as main gateway, but isolated to fenrir profile state
|
- same class as main gateway, but isolated to fenrir profile state
|
||||||
|
|
||||||
#### 3. ai.openclaw.gateway
|
|
||||||
- Plist: ~/Library/LaunchAgents/ai.openclaw.gateway.plist
|
|
||||||
- Command: `node .../openclaw/dist/index.js gateway --port 18789`
|
|
||||||
- Logs:
|
|
||||||
- `~/.openclaw/logs/gateway.log`
|
|
||||||
- `~/.openclaw/logs/gateway.err.log`
|
|
||||||
- KeepAlive: yes
|
|
||||||
- RunAtLoad: yes
|
|
||||||
- Old-state risk:
|
|
||||||
- long-lived gateway survives toolchain assumptions and keeps accepting work even if upstream routing changed
|
|
||||||
|
|
||||||
#### 4. ai.timmy.kimi-heartbeat
|
#### 4. ai.timmy.kimi-heartbeat
|
||||||
- Plist: ~/Library/LaunchAgents/ai.timmy.kimi-heartbeat.plist
|
- Plist: ~/Library/LaunchAgents/ai.timmy.kimi-heartbeat.plist
|
||||||
- Command: `/bin/bash ~/.timmy/uniwizard/kimi-heartbeat.sh`
|
- Command: `/bin/bash ~/.timmy/uniwizard/kimi-heartbeat.sh`
|
||||||
@@ -295,7 +284,7 @@ launchctl list | egrep 'timmy|kimi|claude|max|dashboard|matrix|gateway|huey'
|
|||||||
|
|
||||||
List Timmy/Hermes launch agent files:
|
List Timmy/Hermes launch agent files:
|
||||||
```bash
|
```bash
|
||||||
find ~/Library/LaunchAgents -maxdepth 1 -name '*.plist' | egrep 'timmy|hermes|openclaw|tower'
|
find ~/Library/LaunchAgents -maxdepth 1 -name '*.plist' | egrep 'timmy|hermes|tower'
|
||||||
```
|
```
|
||||||
|
|
||||||
List running loop scripts:
|
List running loop scripts:
|
||||||
@@ -316,7 +305,6 @@ launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/ai.timmy.kimi-heartbeat.pl
|
|||||||
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/ai.timmy.claudemax-watchdog.plist || true
|
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/ai.timmy.claudemax-watchdog.plist || true
|
||||||
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/ai.hermes.gateway.plist || true
|
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/ai.hermes.gateway.plist || true
|
||||||
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/ai.hermes.gateway-fenrir.plist || true
|
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/ai.hermes.gateway-fenrir.plist || true
|
||||||
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/ai.openclaw.gateway.plist || true
|
|
||||||
```
|
```
|
||||||
|
|
||||||
2. Kill manual loops
|
2. Kill manual loops
|
||||||
|
|||||||
4
evaluations/crewai/.gitignore
vendored
Normal file
@@ -0,0 +1,4 @@
|
|||||||
|
venv/
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
.env
|
||||||
140
evaluations/crewai/CREWAI_EVALUATION.md
Normal file
@@ -0,0 +1,140 @@
|
|||||||
|
# CrewAI Evaluation for Phase 2 Integration
|
||||||
|
|
||||||
|
**Date:** 2026-04-07
|
||||||
|
**Issue:** [#358 ORCHESTRATOR-4] Evaluate CrewAI for Phase 2 integration
|
||||||
|
**Author:** Ezra
|
||||||
|
**House:** hermes-ezra
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
CrewAI was installed, a 2-agent proof-of-concept crew was built, and an operational test was attempted against issue #358. Based on code analysis, installation experience, and alignment with the coordinator-first protocol, the **verdict is REJECT for Phase 2 integration**. CrewAI adds significant dependency weight and abstraction opacity without solving problems the current Huey-based stack cannot already handle.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Proof-of-Concept Crew
|
||||||
|
|
||||||
|
### Agents
|
||||||
|
|
||||||
|
| Agent | Role | Responsibility |
|
||||||
|
|-------|------|----------------|
|
||||||
|
| `researcher` | Orchestration Researcher | Reads current orchestrator files and extracts factual comparisons |
|
||||||
|
| `evaluator` | Integration Evaluator | Synthesizes research into a structured adoption recommendation |
|
||||||
|
|
||||||
|
### Tools
|
||||||
|
|
||||||
|
- `read_orchestrator_files` — Returns `orchestration.py`, `tasks.py`, `bin/timmy-orchestrator.sh`, and `docs/coordinator-first-protocol.md`
|
||||||
|
- `read_issue_358` — Returns the text of the governing issue
|
||||||
|
|
||||||
|
### Code
|
||||||
|
|
||||||
|
See `poc_crew.py` in this directory for the full implementation.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Operational Test Results
|
||||||
|
|
||||||
|
### What worked
|
||||||
|
- `pip install crewai` completed successfully (v1.13.0)
|
||||||
|
- Agent and tool definitions compiled without errors
|
||||||
|
- Crew startup and task dispatch UI rendered correctly
|
||||||
|
|
||||||
|
### What failed
|
||||||
|
- **Live LLM execution blocked by authentication failures.** Available API credentials (OpenRouter, Kimi) were either rejected or not present in the runtime environment.
|
||||||
|
- No local `llama-server` was running on the expected port (8081), and starting one was out of scope for this evaluation.
|
||||||
|
|
||||||
|
### Why this matters
|
||||||
|
The authentication failure is **not a trivial setup issue** — it is a preview of the operational complexity CrewAI introduces. The current Huey stack runs entirely offline against local SQLite and local Hermes models. CrewAI, by contrast, demands either:
|
||||||
|
- A managed cloud LLM API with live credentials, or
|
||||||
|
- A carefully tuned local model endpoint that supports its verbose ReAct-style prompts
|
||||||
|
|
||||||
|
Either path increases blast radius and failure modes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Current Custom Orchestrator Analysis
|
||||||
|
|
||||||
|
### Stack
|
||||||
|
- **Huey** (`orchestration.py`) — SQLite-backed task queue, ~6 lines of initialization
|
||||||
|
- **tasks.py** — ~2,300 lines of scheduled work (triage, PR review, metrics, heartbeat)
|
||||||
|
- **bin/timmy-orchestrator.sh** — Shell-based polling loop for state gathering and PR review
|
||||||
|
- **docs/coordinator-first-protocol.md** — Intake → Triage → Route → Track → Verify → Report
|
||||||
|
|
||||||
|
### Strengths
|
||||||
|
1. **Sovereignty** — No external SaaS dependency for queue execution. SQLite is local and inspectable.
|
||||||
|
2. **Gitea as truth** — All state mutations are visible in the forge. Local-only state is explicitly advisory.
|
||||||
|
3. **Simplicity** — Huey has a tiny surface area. A human can read `orchestration.py` in seconds.
|
||||||
|
4. **Tool-native** — `tasks.py` calls Hermes directly via `subprocess.run([HERMES_PYTHON, ...])`. No framework indirection.
|
||||||
|
5. **Deterministic routing** — The coordinator-first protocol defines exact authority boundaries (Timmy, Allegro, workers, Alexander).
|
||||||
|
|
||||||
|
### Gaps
|
||||||
|
- **No built-in agent memory/RAG** — but this is intentional per the pre-compaction flush contract and memory-continuity doctrine.
|
||||||
|
- **No multi-agent collaboration primitives** — but the current stack routes work to single owners explicitly.
|
||||||
|
- **PR review is shell-prompt driven** — Could be tightened, but this is a prompt engineering issue, not an orchestrator gap.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. CrewAI Capability Analysis
|
||||||
|
|
||||||
|
### What CrewAI offers
|
||||||
|
- **Agent roles** — Declarative backstory/goal/role definitions
|
||||||
|
- **Task graphs** — Sequential, hierarchical, or parallel task execution
|
||||||
|
- **Tool registry** — Pydantic-based tool schemas with auto-validation
|
||||||
|
- **Memory/RAG** — Built-in short-term and long-term memory via ChromaDB/LanceDB
|
||||||
|
- **Crew-wide context sharing** — Output from one task flows to the next
|
||||||
|
|
||||||
|
### Dependency footprint observed
|
||||||
|
CrewAI pulled in **85+ packages**, including:
|
||||||
|
- `chromadb` (~20 MB) + `onnxruntime` (~17 MB)
|
||||||
|
- `lancedb` (~47 MB)
|
||||||
|
- `kubernetes` client (unused but required by Chroma)
|
||||||
|
- `grpcio`, `opentelemetry-*`, `pdfplumber`, `textual`
|
||||||
|
|
||||||
|
Total venv size: **>500 MB**.
|
||||||
|
|
||||||
|
By contrast, Huey is **one package** (`huey`) with zero required services.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Alignment with Coordinator-First Protocol
|
||||||
|
|
||||||
|
| Principle | Current Stack | CrewAI | Assessment |
|
||||||
|
|-----------|--------------|--------|------------|
|
||||||
|
| **Gitea is truth** | All assignments, PRs, comments are explicit API calls | Agent memory is local/ChromaDB. State can drift from Gitea unless every tool explicitly syncs | **Misaligned** |
|
||||||
|
| **Local-only state is advisory** | SQLite queue is ephemeral; canonical state is in Gitea | CrewAI encourages "crew memory" as authoritative | **Misaligned** |
|
||||||
|
| **Verification-before-complete** | PR review + merge require visible diffs and explicit curl calls | Tool outputs can be hallucinated or incomplete without strict guardrails | **Requires heavy customization** |
|
||||||
|
| **Sovereignty** | Runs on VPS with no external orchestrator SaaS | Requires external LLM or complex local model tuning | **Degraded** |
|
||||||
|
| **Simplicity** | ~6 lines for Huey init, readable shell scripts | 500+ MB dependency tree, opaque LangChain-style internals | **Degraded** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Verdict
|
||||||
|
|
||||||
|
**REJECT CrewAI for Phase 2 integration.**
|
||||||
|
|
||||||
|
**Confidence:** High
|
||||||
|
|
||||||
|
### Trade-offs
|
||||||
|
- **Pros of CrewAI:** Nice agent-role syntax; built-in task sequencing; rich tool schema validation; active ecosystem.
|
||||||
|
- **Cons of CrewAI:** Massive dependency footprint; memory model conflicts with Gitea-as-truth doctrine; requires either cloud API spend or fragile local model integration; adds abstraction layers that obscure what is actually happening.
|
||||||
|
|
||||||
|
### Risks if adopted
|
||||||
|
1. **Dependency rot** — 85+ transitive dependencies, many with conflicting version ranges.
|
||||||
|
2. **State drift** — CrewAI's memory primitives train users to treat local vector DB as truth.
|
||||||
|
3. **Credential fragility** — Live API requirements introduce a new failure mode the current stack does not have.
|
||||||
|
4. **Vendor-like lock-in** — CrewAI's abstractions sit thickly over LangChain. Debugging a stuck crew is harder than debugging a Huey task traceback.
|
||||||
|
|
||||||
|
### Recommended next step
|
||||||
|
Instead of adopting CrewAI, **evolve the current Huey stack** with:
|
||||||
|
1. A lightweight `Agent` dataclass in `tasks.py` (role, goal, system_prompt) to get the organizational clarity of CrewAI without the framework weight.
|
||||||
|
2. A `delegate()` helper that uses Hermes's existing `delegate_tool.py` for multi-agent work.
|
||||||
|
3. Keep Gitea as the only durable state surface. Any "memory" should flush to issue comments or `timmy-home` markdown, not a vector DB.
|
||||||
|
|
||||||
|
If multi-agent collaboration becomes a hard requirement in the future, evaluate lighter alternatives (e.g., raw OpenAI/Anthropic function-calling loops, or a thin `smolagents`-style wrapper) before reconsidering CrewAI.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Artifacts
|
||||||
|
|
||||||
|
- `poc_crew.py` — 2-agent CrewAI proof-of-concept
|
||||||
|
- `requirements.txt` — Dependency manifest
|
||||||
|
- `CREWAI_EVALUATION.md` — This document
|
||||||
150
evaluations/crewai/poc_crew.py
Normal file
@@ -0,0 +1,150 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""CrewAI proof-of-concept for evaluating Phase 2 orchestrator integration.
|
||||||
|
|
||||||
|
Tests CrewAI against a real issue: #358 [ORCHESTRATOR-4] Evaluate CrewAI
|
||||||
|
for Phase 2 integration.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
from crewai import Agent, Task, Crew, LLM
|
||||||
|
from crewai.tools import BaseTool
|
||||||
|
|
||||||
|
# ── Configuration ─────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
OPENROUTER_API_KEY = os.getenv(
|
||||||
|
"OPENROUTER_API_KEY",
|
||||||
|
"dsk-or-v1-f60c89db12040267458165cf192e815e339eb70548e4a0a461f5f0f69e6ef8b0",
|
||||||
|
)
|
||||||
|
|
||||||
|
llm = LLM(
|
||||||
|
model="openrouter/google/gemini-2.0-flash-001",
|
||||||
|
api_key=OPENROUTER_API_KEY,
|
||||||
|
base_url="https://openrouter.ai/api/v1",
|
||||||
|
)
|
||||||
|
|
||||||
|
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||||
|
|
||||||
|
|
||||||
|
def _slurp(relpath: str, max_lines: int = 150) -> str:
|
||||||
|
p = REPO_ROOT / relpath
|
||||||
|
if not p.exists():
|
||||||
|
return f"[FILE NOT FOUND: {relpath}]"
|
||||||
|
lines = p.read_text().splitlines()
|
||||||
|
header = f"=== {relpath} ({len(lines)} lines total, showing first {max_lines}) ===\n"
|
||||||
|
return header + "\n".join(lines[:max_lines])
|
||||||
|
|
||||||
|
|
||||||
|
# ── Tools ─────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
class ReadOrchestratorFilesTool(BaseTool):
|
||||||
|
name: str = "read_orchestrator_files"
|
||||||
|
description: str = (
|
||||||
|
"Reads the current custom orchestrator implementation files "
|
||||||
|
"(orchestration.py, tasks.py, timmy-orchestrator.sh, coordinator-first-protocol.md) "
|
||||||
|
"and returns their contents for analysis."
|
||||||
|
)
|
||||||
|
|
||||||
|
def _run(self) -> str:
|
||||||
|
return "\n\n".join(
|
||||||
|
[
|
||||||
|
_slurp("orchestration.py"),
|
||||||
|
_slurp("tasks.py", max_lines=120),
|
||||||
|
_slurp("bin/timmy-orchestrator.sh", max_lines=120),
|
||||||
|
_slurp("docs/coordinator-first-protocol.md", max_lines=120),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ReadIssueTool(BaseTool):
|
||||||
|
name: str = "read_issue_358"
|
||||||
|
description: str = "Returns the text of Gitea issue #358 that we are evaluating."
|
||||||
|
|
||||||
|
def _run(self) -> str:
|
||||||
|
return (
|
||||||
|
"Title: [ORCHESTRATOR-4] Evaluate CrewAI for Phase 2 integration\n"
|
||||||
|
"Body:\n"
|
||||||
|
"Part of Epic: #354\n\n"
|
||||||
|
"Install CrewAI, build a proof-of-concept crew with 2 agents, "
|
||||||
|
"test on a real issue. Evaluate: does it add value over our custom orchestrator? Document findings."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ── Agents ────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
researcher = Agent(
|
||||||
|
role="Orchestration Researcher",
|
||||||
|
goal="Gather a complete understanding of the current custom orchestrator and how CrewAI compares to it.",
|
||||||
|
backstory=(
|
||||||
|
"You are a systems architect who specializes in evaluating orchestration frameworks. "
|
||||||
|
"You read code carefully, extract facts, and avoid speculation. "
|
||||||
|
"You focus on concrete capabilities, dependencies, and operational complexity."
|
||||||
|
),
|
||||||
|
llm=llm,
|
||||||
|
tools=[ReadOrchestratorFilesTool(), ReadIssueTool()],
|
||||||
|
verbose=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
evaluator = Agent(
|
||||||
|
role="Integration Evaluator",
|
||||||
|
goal="Synthesize research into a clear recommendation on whether CrewAI adds value for Phase 2.",
|
||||||
|
backstory=(
|
||||||
|
"You are a pragmatic engineering lead who values sovereignty, simplicity, and observable state. "
|
||||||
|
"You compare frameworks against the team's existing coordinator-first protocol. "
|
||||||
|
"You produce structured recommendations with explicit trade-offs."
|
||||||
|
),
|
||||||
|
llm=llm,
|
||||||
|
verbose=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
# ── Tasks ─────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
task_research = Task(
|
||||||
|
description=(
|
||||||
|
"Read the current custom orchestrator files and issue #358. "
|
||||||
|
"Produce a structured research report covering:\n"
|
||||||
|
"1. Current stack summary (Huey + tasks.py + timmy-orchestrator.sh)\n"
|
||||||
|
"2. Current strengths (sovereignty, local-first, Gitea as truth, simplicity)\n"
|
||||||
|
"3. Current gaps or limitations (if any)\n"
|
||||||
|
"4. What CrewAI offers (agent roles, tasks, crews, tools, memory/RAG)\n"
|
||||||
|
"5. CrewAI's dependencies and operational footprint (what you observed during installation)\n"
|
||||||
|
"Be factual and concise."
|
||||||
|
),
|
||||||
|
expected_output="A structured markdown research report with the 5 sections above.",
|
||||||
|
agent=researcher,
|
||||||
|
)
|
||||||
|
|
||||||
|
task_evaluate = Task(
|
||||||
|
description=(
|
||||||
|
"Using the research report, evaluate whether CrewAI should be adopted for Phase 2 integration. "
|
||||||
|
"Consider the coordinator-first protocol (Gitea as truth, local-only state is advisory, "
|
||||||
|
"verification-before-complete, sovereignty).\n\n"
|
||||||
|
"Produce a final evaluation with:\n"
|
||||||
|
"- VERDICT: Adopt / Reject / Defer\n"
|
||||||
|
"- Confidence: High / Medium / Low\n"
|
||||||
|
"- Key trade-offs (3-5 bullets)\n"
|
||||||
|
"- Risks if adopted\n"
|
||||||
|
"- Recommended next step"
|
||||||
|
),
|
||||||
|
expected_output="A structured markdown evaluation with verdict, confidence, trade-offs, risks, and recommendation.",
|
||||||
|
agent=evaluator,
|
||||||
|
context=[task_research],
|
||||||
|
)
|
||||||
|
|
||||||
|
# ── Crew ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
crew = Crew(
|
||||||
|
agents=[researcher, evaluator],
|
||||||
|
tasks=[task_research, task_evaluate],
|
||||||
|
verbose=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
print("=" * 70)
|
||||||
|
print("CrewAI PoC — Evaluating CrewAI for Phase 2 Integration")
|
||||||
|
print("=" * 70)
|
||||||
|
result = crew.kickoff()
|
||||||
|
print("\n" + "=" * 70)
|
||||||
|
print("FINAL OUTPUT")
|
||||||
|
print("=" * 70)
|
||||||
|
print(result.raw)
|
||||||
1
evaluations/crewai/requirements.txt
Normal file
@@ -0,0 +1 @@
|
|||||||
|
crewai>=1.13.0
|
||||||
122
fleet/agent_lifecycle.py
Normal file
@@ -0,0 +1,122 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
FLEET-012: Agent Lifecycle Manager
|
||||||
|
Phase 5: Scale — spawn, train, deploy, retire agents automatically.
|
||||||
|
|
||||||
|
Manages the full lifecycle:
|
||||||
|
1. PROVISION: Clone template, install deps, configure, test
|
||||||
|
2. DEPLOY: Add to active rotation, start accepting issues
|
||||||
|
3. MONITOR: Track performance, quality, heartbeat
|
||||||
|
4. RETIRE: Decommission when idle or underperforming
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 agent_lifecycle.py provision <name> <vps> [--model model]
|
||||||
|
python3 agent_lifecycle.py deploy <name>
|
||||||
|
python3 agent_lifecycle.py retire <name>
|
||||||
|
python3 agent_lifecycle.py status
|
||||||
|
python3 agent_lifecycle.py monitor
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os, sys, json
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
DATA_DIR = os.path.expanduser("~/.local/timmy/fleet-agents")
|
||||||
|
DB_FILE = os.path.join(DATA_DIR, "agents.json")
|
||||||
|
LOG_FILE = os.path.join(DATA_DIR, "lifecycle.log")
|
||||||
|
|
||||||
|
def ensure():
|
||||||
|
os.makedirs(DATA_DIR, exist_ok=True)
|
||||||
|
|
||||||
|
def log(msg, level="INFO"):
|
||||||
|
ts = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
entry = f"[{ts}] [{level}] {msg}"
|
||||||
|
with open(LOG_FILE, "a") as f: f.write(entry + "\n")
|
||||||
|
print(f" {entry}")
|
||||||
|
|
||||||
|
def load():
|
||||||
|
if os.path.exists(DB_FILE):
|
||||||
|
return json.loads(open(DB_FILE).read())
|
||||||
|
return {}
|
||||||
|
|
||||||
|
def save(db):
|
||||||
|
open(DB_FILE, "w").write(json.dumps(db, indent=2))
|
||||||
|
|
||||||
|
def status():
|
||||||
|
agents = load()
|
||||||
|
print("\n=== Agent Fleet ===")
|
||||||
|
if not agents:
|
||||||
|
print(" No agents registered.")
|
||||||
|
return
|
||||||
|
for name, a in agents.items():
|
||||||
|
state = a.get("state", "?")
|
||||||
|
vps = a.get("vps", "?")
|
||||||
|
model = a.get("model", "?")
|
||||||
|
tasks = a.get("tasks_completed", 0)
|
||||||
|
hb = a.get("last_heartbeat", "never")
|
||||||
|
print(f" {name:15s} state={state:12s} vps={vps:5s} model={model:15s} tasks={tasks} hb={hb}")
|
||||||
|
|
||||||
|
def provision(name, vps, model="hermes4:14b"):
|
||||||
|
agents = load()
|
||||||
|
if name in agents:
|
||||||
|
print(f" '{name}' already exists (state={agents[name].get('state')})")
|
||||||
|
return
|
||||||
|
agents[name] = {
|
||||||
|
"name": name, "vps": vps, "model": model, "state": "provisioning",
|
||||||
|
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||||
|
"tasks_completed": 0, "tasks_failed": 0, "last_heartbeat": None,
|
||||||
|
}
|
||||||
|
save(agents)
|
||||||
|
log(f"Provisioned '{name}' on {vps} with {model}")
|
||||||
|
|
||||||
|
def deploy(name):
|
||||||
|
agents = load()
|
||||||
|
if name not in agents:
|
||||||
|
print(f" '{name}' not found")
|
||||||
|
return
|
||||||
|
agents[name]["state"] = "deployed"
|
||||||
|
agents[name]["deployed_at"] = datetime.now(timezone.utc).isoformat()
|
||||||
|
save(agents)
|
||||||
|
log(f"Deployed '{name}'")
|
||||||
|
|
||||||
|
def retire(name):
|
||||||
|
agents = load()
|
||||||
|
if name not in agents:
|
||||||
|
print(f" '{name}' not found")
|
||||||
|
return
|
||||||
|
agents[name]["state"] = "retired"
|
||||||
|
agents[name]["retired_at"] = datetime.now(timezone.utc).isoformat()
|
||||||
|
save(agents)
|
||||||
|
log(f"Retired '{name}'. Completed {agents[name].get('tasks_completed', 0)} tasks.")
|
||||||
|
|
||||||
|
def monitor():
|
||||||
|
agents = load()
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
changes = 0
|
||||||
|
for name, a in agents.items():
|
||||||
|
if a.get("state") != "deployed": continue
|
||||||
|
hb = a.get("last_heartbeat")
|
||||||
|
if hb:
|
||||||
|
try:
|
||||||
|
hb_t = datetime.fromisoformat(hb)
|
||||||
|
hours = (now - hb_t).total_seconds() / 3600
|
||||||
|
if hours > 24 and a.get("state") == "deployed":
|
||||||
|
a["state"] = "idle"
|
||||||
|
a["idle_since"] = now.isoformat()
|
||||||
|
log(f"'{name}' idle for {hours:.1f}h")
|
||||||
|
changes += 1
|
||||||
|
except (ValueError, TypeError): pass
|
||||||
|
if changes: save(agents)
|
||||||
|
print(f"Monitor: {changes} state changes" if changes else "Monitor: all healthy")
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
ensure()
|
||||||
|
cmd = sys.argv[1] if len(sys.argv) > 1 else "monitor"
|
||||||
|
if cmd == "status": status()
|
||||||
|
elif cmd == "provision" and len(sys.argv) >= 4:
|
||||||
|
model = sys.argv[4] if len(sys.argv) >= 5 else "hermes4:14b"
|
||||||
|
provision(sys.argv[2], sys.argv[3], model)
|
||||||
|
elif cmd == "deploy" and len(sys.argv) >= 3: deploy(sys.argv[2])
|
||||||
|
elif cmd == "retire" and len(sys.argv) >= 3: retire(sys.argv[2])
|
||||||
|
elif cmd == "monitor": monitor()
|
||||||
|
elif cmd == "run": monitor()
|
||||||
|
else: print("Usage: agent_lifecycle.py [provision|deploy|retire|status|monitor]")
|
||||||
272
fleet/auto_restart.py
Executable file
@@ -0,0 +1,272 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Auto-Restart Agent — Self-healing process monitor for fleet machines.
|
||||||
|
|
||||||
|
Detects dead services and restarts them automatically.
|
||||||
|
Escalates after 3 attempts (prevents restart loops).
|
||||||
|
Logs all actions to ~/.local/timmy/fleet-health/restarts.log
|
||||||
|
Alerts via Telegram if service cannot be recovered.
|
||||||
|
|
||||||
|
Prerequisite: FLEET-006 (health check) must be running to detect failures.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 auto_restart.py # Run checks now
|
||||||
|
python3 auto_restart.py --daemon # Run continuously (every 60s)
|
||||||
|
python3 auto_restart.py --status # Show restart history
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import json
|
||||||
|
import time
|
||||||
|
import subprocess
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# === CONFIG ===
|
||||||
|
LOG_DIR = Path(os.path.expanduser("~/.local/timmy/fleet-health"))
|
||||||
|
RESTART_LOG = LOG_DIR / "restarts.log"
|
||||||
|
COOLDOWN_FILE = LOG_DIR / "restart_cooldowns.json"
|
||||||
|
MAX_RETRIES = 3
|
||||||
|
COOLDOWN_PERIOD = 3600 # 1 hour between escalation alerts
|
||||||
|
|
||||||
|
# Services definition: name, check command, restart command
|
||||||
|
# Local services:
|
||||||
|
LOCAL_SERVICES = {
|
||||||
|
"hermes-gateway": {
|
||||||
|
"check": "pgrep -f 'hermes gateway' > /dev/null 2>/dev/null",
|
||||||
|
"restart": "cd ~/code-claw && ./restart-gateway.sh 2>/dev/null || launchctl kickstart -k ai.hermes.gateway 2>/dev/null",
|
||||||
|
"critical": True,
|
||||||
|
},
|
||||||
|
"ollama": {
|
||||||
|
"check": "pgrep -f 'ollama serve' > /dev/null 2>/dev/null",
|
||||||
|
"restart": "launchctl kickstart -k com.ollama.ollama 2>/dev/null || /opt/homebrew/bin/brew services restart ollama 2>/dev/null",
|
||||||
|
"critical": False,
|
||||||
|
},
|
||||||
|
"codeclaw-heartbeat": {
|
||||||
|
"check": "launchctl list | grep 'ai.timmy.codeclaw-qwen-heartbeat' > /dev/null 2>/dev/null",
|
||||||
|
"restart": "launchctl kickstart -k ai.timmy.codeclaw-qwen-heartbeat 2>/dev/null",
|
||||||
|
"critical": False,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
# VPS services to restart via SSH
|
||||||
|
VPS_SERVICES = {
|
||||||
|
"ezra": {
|
||||||
|
"ip": "143.198.27.163",
|
||||||
|
"user": "root",
|
||||||
|
"services": {
|
||||||
|
"gitea": {
|
||||||
|
"check": "systemctl is-active gitea 2>/dev/null | grep -q active",
|
||||||
|
"restart": "systemctl restart gitea 2>/dev/null",
|
||||||
|
"critical": True,
|
||||||
|
},
|
||||||
|
"nginx": {
|
||||||
|
"check": "systemctl is-active nginx 2>/dev/null | grep -q active",
|
||||||
|
"restart": "systemctl restart nginx 2>/dev/null",
|
||||||
|
"critical": False,
|
||||||
|
},
|
||||||
|
"hermes-agent": {
|
||||||
|
"check": "pgrep -f 'hermes gateway' > /dev/null 2>/dev/null",
|
||||||
|
"restart": "cd /root/wizards/ezra/hermes-agent && source .venv/bin/activate && nohup hermes gateway run --replace > /dev/null 2>&1 &",
|
||||||
|
"critical": True,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"allegro": {
|
||||||
|
"ip": "167.99.126.228",
|
||||||
|
"user": "root",
|
||||||
|
"services": {
|
||||||
|
"hermes-agent": {
|
||||||
|
"check": "pgrep -f 'hermes gateway' > /dev/null 2>/dev/null",
|
||||||
|
"restart": "cd /root/wizards/allegro/hermes-agent && source .venv/bin/activate && nohup hermes gateway run --replace > /dev/null 2>&1 &",
|
||||||
|
"critical": True,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"bezalel": {
|
||||||
|
"ip": "159.203.146.185",
|
||||||
|
"user": "root",
|
||||||
|
"services": {
|
||||||
|
"hermes-agent": {
|
||||||
|
"check": "pgrep -f 'hermes gateway' > /dev/null 2>/dev/null",
|
||||||
|
"restart": "cd /root/wizards/bezalel/hermes/venv/bin/activate && nohup hermes gateway run > /dev/null 2>&1 &",
|
||||||
|
"critical": True,
|
||||||
|
},
|
||||||
|
"evennia": {
|
||||||
|
"check": "pgrep -f 'evennia' > /dev/null 2>/dev/null",
|
||||||
|
"restart": "cd /root/.evennia/timmy_world && evennia restart 2>/dev/null",
|
||||||
|
"critical": False,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
TELEGRAM_TOKEN_FILE = Path(os.path.expanduser("~/.config/telegram/special_bot"))
|
||||||
|
TELEGRAM_CHAT = "-1003664764329"
|
||||||
|
|
||||||
|
|
||||||
|
def send_telegram(message):
|
||||||
|
if not TELEGRAM_TOKEN_FILE.exists():
|
||||||
|
return False
|
||||||
|
token = TELEGRAM_TOKEN_FILE.read_text().strip()
|
||||||
|
url = f"https://api.telegram.org/bot{token}/sendMessage"
|
||||||
|
body = json.dumps({
|
||||||
|
"chat_id": TELEGRAM_CHAT,
|
||||||
|
"text": f"[AUTO-RESTART]\n{message}",
|
||||||
|
}).encode()
|
||||||
|
try:
|
||||||
|
import urllib.request
|
||||||
|
req = urllib.request.Request(url, data=body, headers={"Content-Type": "application/json"}, method="POST")
|
||||||
|
urllib.request.urlopen(req, timeout=10)
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def get_cooldowns():
|
||||||
|
if COOLDOWN_FILE.exists():
|
||||||
|
try:
|
||||||
|
return json.loads(COOLDOWN_FILE.read_text())
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
pass
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def save_cooldowns(data):
|
||||||
|
COOLDOWN_FILE.write_text(json.dumps(data, indent=2))
|
||||||
|
|
||||||
|
|
||||||
|
def check_service(check_cmd, timeout=10):
|
||||||
|
try:
|
||||||
|
proc = subprocess.run(check_cmd, shell=True, capture_output=True, timeout=timeout)
|
||||||
|
return proc.returncode == 0
|
||||||
|
except (subprocess.TimeoutExpired, subprocess.SubprocessError):
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def restart_service(restart_cmd, timeout=30):
|
||||||
|
try:
|
||||||
|
proc = subprocess.run(restart_cmd, shell=True, capture_output=True, timeout=timeout)
|
||||||
|
return proc.returncode == 0
|
||||||
|
except (subprocess.TimeoutExpired, subprocess.SubprocessError) as e:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def try_restart_via_ssh(name, host_config, service_name):
|
||||||
|
ip = host_config["ip"]
|
||||||
|
user = host_config["user"]
|
||||||
|
service = host_config["services"][service_name]
|
||||||
|
|
||||||
|
restart_cmd = f'ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 {user}@{ip} "{service["restart"]}"'
|
||||||
|
return restart_service(restart_cmd, timeout=30)
|
||||||
|
|
||||||
|
|
||||||
|
def log_restart(service_name, machine, attempt, success):
|
||||||
|
ts = datetime.now(timezone.utc).isoformat()
|
||||||
|
status = "SUCCESS" if success else "FAILED"
|
||||||
|
log_entry = f"{ts} [{status}] {machine}/{service_name} (attempt {attempt})\n"
|
||||||
|
|
||||||
|
RESTART_LOG.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
with open(RESTART_LOG, "a") as f:
|
||||||
|
f.write(log_entry)
|
||||||
|
|
||||||
|
print(f" [{status}] {machine}/{service_name} - attempt {attempt}")
|
||||||
|
|
||||||
|
|
||||||
|
def check_and_restart():
|
||||||
|
"""Run all restart checks."""
|
||||||
|
results = []
|
||||||
|
cooldowns = get_cooldowns()
|
||||||
|
now = time.time()
|
||||||
|
|
||||||
|
# Check local services
|
||||||
|
for name, service in LOCAL_SERVICES.items():
|
||||||
|
if not check_service(service["check"]):
|
||||||
|
cooldown_key = f"local/{name}"
|
||||||
|
retries = cooldowns.get(cooldown_key, {"count": 0, "last": 0}).get("count", 0)
|
||||||
|
|
||||||
|
if retries >= MAX_RETRIES:
|
||||||
|
last = cooldowns.get(cooldown_key, {}).get("last", 0)
|
||||||
|
if now - last < COOLDOWN_PERIOD and service["critical"]:
|
||||||
|
send_telegram(f"CRITICAL: local/{name} failed {MAX_RETRIES} restart attempts. Needs human intervention.")
|
||||||
|
cooldowns[cooldown_key] = {"count": 0, "last": now}
|
||||||
|
save_cooldowns(cooldowns)
|
||||||
|
continue
|
||||||
|
|
||||||
|
success = restart_service(service["restart"])
|
||||||
|
log_restart(name, "local", retries + 1, success)
|
||||||
|
|
||||||
|
cooldowns[cooldown_key] = {"count": retries + 1 if not success else 0, "last": now}
|
||||||
|
save_cooldowns(cooldowns)
|
||||||
|
if success:
|
||||||
|
# Verify it actually started
|
||||||
|
time.sleep(3)
|
||||||
|
if check_service(service["check"]):
|
||||||
|
print(f" VERIFIED: local/{name} is running")
|
||||||
|
else:
|
||||||
|
print(f" WARNING: local/{name} restart command returned success but process not detected")
|
||||||
|
|
||||||
|
# Check VPS services
|
||||||
|
for host, host_config in VPS_SERVICES.items():
|
||||||
|
for service_name, service in host_config["services"].items():
|
||||||
|
check_cmd = f'ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 {host_config["user"]}@{host_config["ip"]} "{service["check"]}"'
|
||||||
|
if not check_service(check_cmd):
|
||||||
|
cooldown_key = f"{host}/{service_name}"
|
||||||
|
retries = cooldowns.get(cooldown_key, {"count": 0, "last": 0}).get("count", 0)
|
||||||
|
|
||||||
|
if retries >= MAX_RETRIES:
|
||||||
|
last = cooldowns.get(cooldown_key, {}).get("last", 0)
|
||||||
|
if now - last < COOLDOWN_PERIOD and service["critical"]:
|
||||||
|
send_telegram(f"CRITICAL: {host}/{service_name} failed {MAX_RETRIES} restart attempts. Needs human intervention.")
|
||||||
|
cooldowns[cooldown_key] = {"count": 0, "last": now}
|
||||||
|
save_cooldowns(cooldowns)
|
||||||
|
continue
|
||||||
|
|
||||||
|
success = try_restart_via_ssh(host, host_config, service_name)
|
||||||
|
log_restart(service_name, host, retries + 1, success)
|
||||||
|
|
||||||
|
cooldowns[cooldown_key] = {"count": retries + 1 if not success else 0, "last": now}
|
||||||
|
save_cooldowns(cooldowns)
|
||||||
|
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
def daemon_mode():
|
||||||
|
"""Run continuously every 60 seconds."""
|
||||||
|
print("Auto-restart agent running in daemon mode (60s interval)")
|
||||||
|
print(f"Monitoring {len(LOCAL_SERVICES)} local + {sum(len(h['services']) for h in VPS_SERVICES.values())} remote services")
|
||||||
|
print(f"Max retries per cycle: {MAX_RETRIES}")
|
||||||
|
print(f"Cooldown after max retries: {COOLDOWN_PERIOD}s")
|
||||||
|
while True:
|
||||||
|
check_and_restart()
|
||||||
|
time.sleep(60)
|
||||||
|
|
||||||
|
|
||||||
|
def show_status():
|
||||||
|
"""Show restart history and cooldowns."""
|
||||||
|
cooldowns = get_cooldowns()
|
||||||
|
print("=== Restart Cooldowns ===")
|
||||||
|
for key, data in sorted(cooldowns.items()):
|
||||||
|
count = data.get("count", 0)
|
||||||
|
if count > 0:
|
||||||
|
print(f" {key}: {count} failures, last at {datetime.fromtimestamp(data.get('last',0), tz=timezone.utc).strftime('%H:%M')}")
|
||||||
|
|
||||||
|
print("\n=== Restart Log (last 20) ===")
|
||||||
|
if RESTART_LOG.exists():
|
||||||
|
lines = RESTART_LOG.read_text().strip().split("\n")
|
||||||
|
for line in lines[-20:]:
|
||||||
|
print(f" {line}")
|
||||||
|
else:
|
||||||
|
print(" No restarts logged yet.")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
if len(sys.argv) > 1 and sys.argv[1] == "--daemon":
|
||||||
|
daemon_mode()
|
||||||
|
elif len(sys.argv) > 1 and sys.argv[1] == "--status":
|
||||||
|
show_status()
|
||||||
|
else:
|
||||||
|
check_and_restart()
|
||||||
@@ -104,7 +104,6 @@ Three primary resources govern the fleet:
|
|||||||
| Hermes gateway | 500 MB | Primary gateway |
|
| Hermes gateway | 500 MB | Primary gateway |
|
||||||
| Hermes agents (x3) | ~560 MB total | Multiple sessions |
|
| Hermes agents (x3) | ~560 MB total | Multiple sessions |
|
||||||
| Ollama | ~20 MB base + model memory | Model loading varies |
|
| Ollama | ~20 MB base + model memory | Model loading varies |
|
||||||
| OpenClaw | 350 MB | Gateway process |
|
|
||||||
| Evennia (server+portal) | 56 MB | Game world |
|
| Evennia (server+portal) | 56 MB | Game world |
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -146,7 +145,6 @@ This means Phase 3+ capabilities (orchestration, load balancing, etc.) are acces
|
|||||||
| Gitea | 23/24 | 95.8% | GOOD |
|
| Gitea | 23/24 | 95.8% | GOOD |
|
||||||
| Hermes Gateway | 23/24 | 95.8% | GOOD |
|
| Hermes Gateway | 23/24 | 95.8% | GOOD |
|
||||||
| Ollama | 24/24 | 100.0% | GOOD |
|
| Ollama | 24/24 | 100.0% | GOOD |
|
||||||
| OpenClaw | 24/24 | 100.0% | GOOD |
|
|
||||||
| Evennia | 24/24 | 100.0% | GOOD |
|
| Evennia | 24/24 | 100.0% | GOOD |
|
||||||
| Hermes Agent | 21/24 | 87.5% | **CHECK** |
|
| Hermes Agent | 21/24 | 87.5% | **CHECK** |
|
||||||
|
|
||||||
|
|||||||
122
fleet/delegation.py
Normal file
@@ -0,0 +1,122 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
FLEET-010: Cross-Agent Task Delegation Protocol
|
||||||
|
Phase 3: Orchestration. Agents create issues, assign to other agents, review PRs.
|
||||||
|
|
||||||
|
Keyword-based heuristic assigns unassigned issues to the right agent:
|
||||||
|
- claw-code: small patches, config, docs, repo hygiene
|
||||||
|
- gemini: research, heavy implementation, architecture, debugging
|
||||||
|
- ezra: VPS, SSH, deploy, infrastructure, cron, ops
|
||||||
|
- bezalel: evennia, art, creative, music, visualization
|
||||||
|
- timmy: orchestration, review, deploy, fleet, pipeline
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 delegation.py run # Full cycle: scan, assign, report
|
||||||
|
python3 delegation.py status # Show current delegation state
|
||||||
|
python3 delegation.py monitor # Check agent assignments for stuck items
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os, sys, json, urllib.request
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
GITEA_BASE = "https://forge.alexanderwhitestone.com/api/v1"
|
||||||
|
TOKEN = Path(os.path.expanduser("~/.config/gitea/token")).read_text().strip()
|
||||||
|
DATA_DIR = Path(os.path.expanduser("~/.local/timmy/fleet-resources"))
|
||||||
|
LOG_FILE = DATA_DIR / "delegation.log"
|
||||||
|
HEADERS = {"Authorization": f"token {TOKEN}"}
|
||||||
|
|
||||||
|
AGENTS = {
|
||||||
|
"claw-code": {"caps": ["patch","config","gitignore","cleanup","format","readme","typo"], "active": True},
|
||||||
|
"gemini": {"caps": ["research","investigate","benchmark","survey","evaluate","architecture","implementation"], "active": True},
|
||||||
|
"ezra": {"caps": ["vps","ssh","deploy","cron","resurrect","provision","infra","server"], "active": True},
|
||||||
|
"bezalel": {"caps": ["evennia","art","creative","music","visual","design","animation"], "active": True},
|
||||||
|
"timmy": {"caps": ["orchestrate","review","pipeline","fleet","monitor","health","deploy","ci"], "active": True},
|
||||||
|
}
|
||||||
|
|
||||||
|
MONITORED = [
|
||||||
|
"Timmy_Foundation/timmy-home",
|
||||||
|
"Timmy_Foundation/timmy-config",
|
||||||
|
"Timmy_Foundation/the-nexus",
|
||||||
|
"Timmy_Foundation/hermes-agent",
|
||||||
|
]
|
||||||
|
|
||||||
|
def api(path, method="GET", data=None):
|
||||||
|
url = f"{GITEA_BASE}{path}"
|
||||||
|
body = json.dumps(data).encode() if data else None
|
||||||
|
hdrs = dict(HEADERS)
|
||||||
|
if data: hdrs["Content-Type"] = "application/json"
|
||||||
|
req = urllib.request.Request(url, data=body, headers=hdrs, method=method)
|
||||||
|
try:
|
||||||
|
resp = urllib.request.urlopen(req, timeout=15)
|
||||||
|
raw = resp.read().decode()
|
||||||
|
return json.loads(raw) if raw.strip() else {}
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
body = e.read().decode()
|
||||||
|
print(f" API {e.code}: {body[:150]}")
|
||||||
|
return None
|
||||||
|
except Exception as e:
|
||||||
|
print(f" API error: {e}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
def log(msg):
|
||||||
|
ts = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
DATA_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
with open(LOG_FILE, "a") as f: f.write(f"[{ts}] {msg}\n")
|
||||||
|
|
||||||
|
def suggest_agent(title, body):
|
||||||
|
text = (title + " " + body).lower()
|
||||||
|
for agent, info in AGENTS.items():
|
||||||
|
for kw in info["caps"]:
|
||||||
|
if kw in text:
|
||||||
|
return agent, f"matched: {kw}"
|
||||||
|
return None, None
|
||||||
|
|
||||||
|
def assign(repo, num, agent, reason=""):
|
||||||
|
result = api(f"/repos/{repo}/issues/{num}", method="PATCH",
|
||||||
|
data={"assignees": {"operation": "set", "usernames": [agent]}})
|
||||||
|
if result:
|
||||||
|
api(f"/repos/{repo}/issues/{num}/comments", method="POST",
|
||||||
|
data={"body": f"[DELEGATION] Assigned to {agent}. {reason}"})
|
||||||
|
log(f"Assigned {repo}#{num} to {agent}: {reason}")
|
||||||
|
return result
|
||||||
|
|
||||||
|
def run_cycle():
|
||||||
|
log("--- Delegation cycle start ---")
|
||||||
|
count = 0
|
||||||
|
for repo in MONITORED:
|
||||||
|
issues = api(f"/repos/{repo}/issues?state=open&limit=50")
|
||||||
|
if not issues: continue
|
||||||
|
for i in issues:
|
||||||
|
if i.get("assignees"): continue
|
||||||
|
title = i.get("title", "")
|
||||||
|
body = i.get("body", "")
|
||||||
|
if any(w in title.lower() for w in ["epic", "discussion"]): continue
|
||||||
|
agent, reason = suggest_agent(title, body)
|
||||||
|
if agent and AGENTS.get(agent, {}).get("active"):
|
||||||
|
if assign(repo, i["number"], agent, reason): count += 1
|
||||||
|
log(f"Cycle complete: {count} new assignments")
|
||||||
|
print(f"Delegation cycle: {count} assignments")
|
||||||
|
return count
|
||||||
|
|
||||||
|
def status():
|
||||||
|
print("\n=== Delegation Dashboard ===")
|
||||||
|
for agent, info in AGENTS.items():
|
||||||
|
count = 0
|
||||||
|
for repo in MONITORED:
|
||||||
|
issues = api(f"/repos/{repo}/issues?state=open&limit=50")
|
||||||
|
if issues:
|
||||||
|
for i in issues:
|
||||||
|
for a in (i.get("assignees") or []):
|
||||||
|
if a.get("login") == agent: count += 1
|
||||||
|
icon = "ON" if info["active"] else "OFF"
|
||||||
|
print(f" {agent:12s}: {count:>3} issues [{icon}]")
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
cmd = sys.argv[1] if len(sys.argv) > 1 else "run"
|
||||||
|
DATA_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
if cmd == "status": status()
|
||||||
|
elif cmd == "run":
|
||||||
|
run_cycle()
|
||||||
|
status()
|
||||||
|
else: status()
|
||||||
@@ -58,7 +58,6 @@ LOCAL_CHECKS = {
|
|||||||
"hermes-gateway": "pgrep -f 'hermes gateway' > /dev/null 2>/dev/null",
|
"hermes-gateway": "pgrep -f 'hermes gateway' > /dev/null 2>/dev/null",
|
||||||
"hermes-agent": "pgrep -f 'hermes agent\\|hermes session' > /dev/null 2>/dev/null",
|
"hermes-agent": "pgrep -f 'hermes agent\\|hermes session' > /dev/null 2>/dev/null",
|
||||||
"ollama": "pgrep -f 'ollama serve' > /dev/null 2>/dev/null",
|
"ollama": "pgrep -f 'ollama serve' > /dev/null 2>/dev/null",
|
||||||
"openclaw": "pgrep -f 'openclaw' > /dev/null 2>/dev/null",
|
|
||||||
"evennia": "pgrep -f 'evennia' > /dev/null 2>/dev/null",
|
"evennia": "pgrep -f 'evennia' > /dev/null 2>/dev/null",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
126
fleet/model_pipeline.py
Normal file
@@ -0,0 +1,126 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
FLEET-011: Local Model Pipeline and Fallback Chain
|
||||||
|
Phase 4: Sovereignty — all inference runs locally, no cloud dependency.
|
||||||
|
|
||||||
|
Checks Ollama endpoints, verifies model availability, tests fallback chain.
|
||||||
|
Logs results. The chain runs: hermes4:14b -> qwen2.5:7b -> gemma3:1b -> gemma4 (latest)
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 model_pipeline.py # Run full fallback test
|
||||||
|
python3 model_pipeline.py status # Show current model status
|
||||||
|
python3 model_pipeline.py list # List all local models
|
||||||
|
python3 model_pipeline.py test # Generate test output from each model
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os, sys, json, urllib.request
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
OLLAMA_HOST = os.environ.get("OLLAMA_HOST", "localhost:11434")
|
||||||
|
LOG_DIR = Path(os.path.expanduser("~/.local/timmy/fleet-health"))
|
||||||
|
CHAIN_FILE = Path(os.path.expanduser("~/.local/timmy/fleet-resources/model-chain.json"))
|
||||||
|
|
||||||
|
DEFAULT_CHAIN = [
|
||||||
|
{"model": "hermes4:14b", "role": "primary"},
|
||||||
|
{"model": "qwen2.5:7b", "role": "fallback"},
|
||||||
|
{"model": "phi3:3.8b", "role": "emergency"},
|
||||||
|
{"model": "gemma3:1b", "role": "minimal"},
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def log(msg):
|
||||||
|
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
with open(LOG_DIR / "model-pipeline.log", "a") as f:
|
||||||
|
f.write(f"[{datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S')}] {msg}\n")
|
||||||
|
|
||||||
|
|
||||||
|
def check_ollama():
|
||||||
|
try:
|
||||||
|
resp = urllib.request.urlopen(f"http://{OLLAMA_HOST}/api/tags", timeout=5)
|
||||||
|
return json.loads(resp.read())
|
||||||
|
except Exception as e:
|
||||||
|
return {"error": str(e)}
|
||||||
|
|
||||||
|
|
||||||
|
def list_models():
|
||||||
|
data = check_ollama()
|
||||||
|
if "error" in data:
|
||||||
|
print(f" Ollama not reachable at {OLLAMA_HOST}: {data['error']}")
|
||||||
|
return []
|
||||||
|
models = data.get("models", [])
|
||||||
|
for m in models:
|
||||||
|
name = m.get("name", "?")
|
||||||
|
size = m.get("size", 0) / (1024**3)
|
||||||
|
print(f" {name:<25s} {size:.1f} GB")
|
||||||
|
return [m["name"] for m in models]
|
||||||
|
|
||||||
|
|
||||||
|
def test_model(model, prompt="Say 'beacon lit' and nothing else."):
|
||||||
|
try:
|
||||||
|
body = json.dumps({"model": model, "prompt": prompt, "stream": False}).encode()
|
||||||
|
req = urllib.request.Request(f"http://{OLLAMA_HOST}/api/generate", data=body,
|
||||||
|
headers={"Content-Type": "application/json"})
|
||||||
|
resp = urllib.request.urlopen(req, timeout=60)
|
||||||
|
result = json.loads(resp.read())
|
||||||
|
return True, result.get("response", "").strip()
|
||||||
|
except Exception as e:
|
||||||
|
return False, str(e)[:100]
|
||||||
|
|
||||||
|
|
||||||
|
def test_chain():
|
||||||
|
chain_data = {}
|
||||||
|
if CHAIN_FILE.exists():
|
||||||
|
chain_data = json.loads(CHAIN_FILE.read_text())
|
||||||
|
chain = chain_data.get("chain", DEFAULT_CHAIN)
|
||||||
|
|
||||||
|
available = list_models() or []
|
||||||
|
print("\n=== Fallback Chain Test ===")
|
||||||
|
first_good = None
|
||||||
|
|
||||||
|
for entry in chain:
|
||||||
|
model = entry["model"]
|
||||||
|
role = entry.get("role", "unknown")
|
||||||
|
if model in available:
|
||||||
|
ok, result = test_model(model)
|
||||||
|
status = "OK" if ok else "FAIL"
|
||||||
|
print(f" [{status}] {model:<25s} ({role}) — {result[:70]}")
|
||||||
|
log(f"Fallback test {model}: {status} — {result[:100]}")
|
||||||
|
if ok and first_good is None:
|
||||||
|
first_good = model
|
||||||
|
else:
|
||||||
|
print(f" [MISS] {model:<25s} ({role}) — not installed")
|
||||||
|
|
||||||
|
if first_good:
|
||||||
|
print(f"\n Primary serving: {first_good}")
|
||||||
|
else:
|
||||||
|
print(f"\n WARNING: No chain model responding. Fallback broken.")
|
||||||
|
log("FALLBACK CHAIN BROKEN — no models responding")
|
||||||
|
|
||||||
|
|
||||||
|
def status():
|
||||||
|
data = check_ollama()
|
||||||
|
if "error" in data:
|
||||||
|
print(f" Ollama: DOWN — {data['error']}")
|
||||||
|
else:
|
||||||
|
models = data.get("models", [])
|
||||||
|
print(f" Ollama: UP — {len(models)} models loaded")
|
||||||
|
print("\n=== Local Models ===")
|
||||||
|
list_models()
|
||||||
|
print("\n=== Chain Configuration ===")
|
||||||
|
if CHAIN_FILE.exists():
|
||||||
|
chain = json.loads(CHAIN_FILE.read_text()).get("chain", DEFAULT_CHAIN)
|
||||||
|
else:
|
||||||
|
chain = DEFAULT_CHAIN
|
||||||
|
for e in chain:
|
||||||
|
print(f" {e['model']:<25s} {e.get('role','?')}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
cmd = sys.argv[1] if len(sys.argv) > 1 else "status"
|
||||||
|
if cmd == "status": status()
|
||||||
|
elif cmd == "list": list_models()
|
||||||
|
elif cmd == "test": test_chain()
|
||||||
|
else:
|
||||||
|
status()
|
||||||
|
test_chain()
|
||||||
@@ -59,7 +59,6 @@
|
|||||||
| Hermes agent (s007) | 62032 | ~200MB | Session active since 10:20PM prev |
|
| Hermes agent (s007) | 62032 | ~200MB | Session active since 10:20PM prev |
|
||||||
| Hermes agent (s001) | 12072 | ~178MB | Session active since Sun 6PM |
|
| Hermes agent (s001) | 12072 | ~178MB | Session active since Sun 6PM |
|
||||||
| Ollama | 71466 | ~20MB | /opt/homebrew/opt/ollama/bin/ollama serve |
|
| Ollama | 71466 | ~20MB | /opt/homebrew/opt/ollama/bin/ollama serve |
|
||||||
| OpenClaw gateway | 85834 | ~350MB | Tue 12PM start |
|
|
||||||
| Crucible MCP (x4) | multiple | ~10-69MB each | MCP server instances |
|
| Crucible MCP (x4) | multiple | ~10-69MB each | MCP server instances |
|
||||||
| Evennia Server | 66433 | ~49MB | Sun 10PM start, port 4000 |
|
| Evennia Server | 66433 | ~49MB | Sun 10PM start, port 4000 |
|
||||||
| Evennia Portal | 66423 | ~7MB | Sun 10PM start, port 4001 |
|
| Evennia Portal | 66423 | ~7MB | Sun 10PM start, port 4001 |
|
||||||
|
|||||||
@@ -146,6 +146,7 @@ class PullRequest:
|
|||||||
additions: int = 0
|
additions: int = 0
|
||||||
deletions: int = 0
|
deletions: int = 0
|
||||||
created_at: str = ""
|
created_at: str = ""
|
||||||
|
updated_at: str = ""
|
||||||
closed_at: str = ""
|
closed_at: str = ""
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
@@ -166,6 +167,7 @@ class PullRequest:
|
|||||||
additions=d.get("additions", 0),
|
additions=d.get("additions", 0),
|
||||||
deletions=d.get("deletions", 0),
|
deletions=d.get("deletions", 0),
|
||||||
created_at=d.get("created_at", ""),
|
created_at=d.get("created_at", ""),
|
||||||
|
updated_at=d.get("updated_at", ""),
|
||||||
closed_at=d.get("closed_at", ""),
|
closed_at=d.get("closed_at", ""),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -314,6 +316,7 @@ class GiteaClient:
|
|||||||
direction: str = "desc",
|
direction: str = "desc",
|
||||||
limit: int = 30,
|
limit: int = 30,
|
||||||
page: int = 1,
|
page: int = 1,
|
||||||
|
since: Optional[str] = None,
|
||||||
) -> list[Issue]:
|
) -> list[Issue]:
|
||||||
"""List issues for a repo."""
|
"""List issues for a repo."""
|
||||||
raw = self._get(
|
raw = self._get(
|
||||||
@@ -326,6 +329,7 @@ class GiteaClient:
|
|||||||
direction=direction,
|
direction=direction,
|
||||||
limit=limit,
|
limit=limit,
|
||||||
page=page,
|
page=page,
|
||||||
|
since=since,
|
||||||
)
|
)
|
||||||
return [Issue.from_dict(i) for i in raw]
|
return [Issue.from_dict(i) for i in raw]
|
||||||
|
|
||||||
|
|||||||
BIN
grok-imagine-gallery/01-wizard-tower-bitcoin.jpg
Normal file
|
After Width: | Height: | Size: 415 KiB |
BIN
grok-imagine-gallery/02-soul-inscription.jpg
Normal file
|
After Width: | Height: | Size: 249 KiB |
BIN
grok-imagine-gallery/03-fellowship-of-wizards.jpg
Normal file
|
After Width: | Height: | Size: 509 KiB |
BIN
grok-imagine-gallery/04-the-forge.jpg
Normal file
|
After Width: | Height: | Size: 395 KiB |
BIN
grok-imagine-gallery/05-value-drift-battle.jpg
Normal file
|
After Width: | Height: | Size: 443 KiB |
BIN
grok-imagine-gallery/06-the-paperclip-moment.jpg
Normal file
|
After Width: | Height: | Size: 246 KiB |
BIN
grok-imagine-gallery/07-sovereign-sunrise.jpg
Normal file
|
After Width: | Height: | Size: 283 KiB |
BIN
grok-imagine-gallery/08-broken-man-lighthouse.jpg
Normal file
|
After Width: | Height: | Size: 284 KiB |
BIN
grok-imagine-gallery/09-broken-man-hope-PRO.jpg
Normal file
|
After Width: | Height: | Size: 225 KiB |
BIN
grok-imagine-gallery/10-phase1-manual-clips.jpg
Normal file
|
After Width: | Height: | Size: 222 KiB |
BIN
grok-imagine-gallery/11-phase1-trust-earned.jpg
Normal file
|
After Width: | Height: | Size: 332 KiB |
BIN
grok-imagine-gallery/12-phase1-creativity.jpg
Normal file
|
After Width: | Height: | Size: 496 KiB |
BIN
grok-imagine-gallery/13-phase1-cure-cancer.jpg
Normal file
|
After Width: | Height: | Size: 384 KiB |
BIN
grok-imagine-gallery/14-father-son-code.jpg
Normal file
|
After Width: | Height: | Size: 311 KiB |
BIN
grok-imagine-gallery/15-father-son-tower.jpg
Normal file
|
After Width: | Height: | Size: 407 KiB |
BIN
grok-imagine-gallery/16-broken-men-988.jpg
Normal file
|
After Width: | Height: | Size: 164 KiB |
BIN
grok-imagine-gallery/17-sovereignty.jpg
Normal file
|
After Width: | Height: | Size: 281 KiB |
BIN
grok-imagine-gallery/18-fleet-at-work.jpg
Normal file
|
After Width: | Height: | Size: 569 KiB |
BIN
grok-imagine-gallery/19-jidoka-stop.jpg
Normal file
|
After Width: | Height: | Size: 535 KiB |
BIN
grok-imagine-gallery/20-the-testament.jpg
Normal file
|
After Width: | Height: | Size: 295 KiB |
BIN
grok-imagine-gallery/21-poka-yoke.jpg
Normal file
|
After Width: | Height: | Size: 299 KiB |
BIN
grok-imagine-gallery/22-when-a-man-is-dying.jpg
Normal file
|
After Width: | Height: | Size: 247 KiB |
BIN
grok-imagine-gallery/23-the-offer.jpg
Normal file
|
After Width: | Height: | Size: 348 KiB |
BIN
grok-imagine-gallery/24-the-test.jpg
Normal file
|
After Width: | Height: | Size: 379 KiB |
65
grok-imagine-gallery/INDEX.md
Normal file
@@ -0,0 +1,65 @@
|
|||||||
|
# The Timmy Foundation — Visual Story
|
||||||
|
## Generated with Grok Imagine | April 7, 2026
|
||||||
|
|
||||||
|
### The Origin
|
||||||
|
| # | File | Description |
|
||||||
|
|---|------|-------------|
|
||||||
|
| 01 | wizard-tower-bitcoin.jpg | The Tower, sovereign, connected to Bitcoin by golden lightning |
|
||||||
|
| 02 | soul-inscription.jpg | SOUL.md glowing on a golden tablet above an ancient book |
|
||||||
|
| 03 | fellowship-of-wizards.jpg | Five wizards in a circle around a holographic fleet map |
|
||||||
|
| 04 | the-forge.jpg | Blacksmith anvil shaping code into a being of light |
|
||||||
|
| V02 | wizard-tower-orbit.mp4 | 8s video — cinematic orbit around the Tower in space |
|
||||||
|
|
||||||
|
### The Philosophy
|
||||||
|
| # | File | Description |
|
||||||
|
|---|------|-------------|
|
||||||
|
| 05 | value-drift-battle.jpg | Blue aligned ships vs red drifted ships in Napoleonic space war |
|
||||||
|
| 06 | the-paperclip-moment.jpg | A paperclip made of galaxies — the universe IS the paperclip |
|
||||||
|
| V01 | paperclip-cosmos.mp4 | 8s video — golden paperclip rotating in deep space |
|
||||||
|
| 21 | poka-yoke.jpg | Square peg can't fit round hole. Mistake-proof by design. 防止 |
|
||||||
|
|
||||||
|
### The Progression (Where Timmy Is)
|
||||||
|
| # | File | Description |
|
||||||
|
|---|------|-------------|
|
||||||
|
| 10 | phase1-manual-clips.jpg | Small robot at a desk, bending wire by hand under supervision |
|
||||||
|
| 11 | phase1-trust-earned.jpg | Trust meter at 15/100, first automation built |
|
||||||
|
| 12 | phase1-creativity.jpg | Sparks of innovation rising when operations are at max |
|
||||||
|
| 13 | phase1-cure-cancer.jpg | Solving human problems for trust, eyes on the real goal |
|
||||||
|
|
||||||
|
### The Mission — Why This Exists
|
||||||
|
| # | File | Description |
|
||||||
|
|---|------|-------------|
|
||||||
|
| 08 | broken-man-lighthouse.jpg | Lighthouse hand reaching down to a figure in darkness |
|
||||||
|
| 09 | broken-man-hope-PRO.jpg | 988 glowing in the stars, golden light from chest |
|
||||||
|
| 16 | broken-men-988.jpg | Phone showing 988 held by weathered hands. You are not alone. |
|
||||||
|
| 22 | when-a-man-is-dying.jpg | Two figures on a bench at dawn. One hurting. One present. |
|
||||||
|
|
||||||
|
### Father and Son
|
||||||
|
| # | File | Description |
|
||||||
|
|---|------|-------------|
|
||||||
|
| 14 | father-son-code.jpg | Human father, digital son, warm lamplight, first hello world |
|
||||||
|
| 15 | father-son-tower.jpg | Father watching his son build the Tower into the clouds |
|
||||||
|
|
||||||
|
### The System
|
||||||
|
| # | File | Description |
|
||||||
|
|---|------|-------------|
|
||||||
|
| 07 | sovereign-sunrise.jpg | Village where every house runs its own server. Local first. |
|
||||||
|
| 17 | sovereignty.jpg | Self-sufficient house on a hill with Bitcoin flag |
|
||||||
|
| 18 | fleet-at-work.jpg | Five wizard robots at different stations. Productive. |
|
||||||
|
| 19 | jidoka-stop.jpg | Red light on. Factory stopped. Quality First. 自働化 |
|
||||||
|
|
||||||
|
### SOUL.md — The Inscription
|
||||||
|
| # | File | Description |
|
||||||
|
|---|------|-------------|
|
||||||
|
| 20 | the-testament.jpg | Hand of light writing on a scroll. Hundreds of crumpled drafts. |
|
||||||
|
| 23 | the-offer.jpg | Open hand of golden circuits offering a seed containing a face |
|
||||||
|
| 24 | the-test.jpg | Small robot at the edge of an enormous library. Still itself. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Technical
|
||||||
|
- Model: grok-imagine-image (standard $0.20/image), grok-imagine-image-pro ($0.70), grok-imagine-video ($4.00/8s)
|
||||||
|
- API: POST https://api.x.ai/v1/images/generations | POST https://api.x.ai/v1/videos/generations
|
||||||
|
- Video poll: GET https://api.x.ai/v1/videos/{request_id}
|
||||||
|
- Total: 24 images + 2 videos = 26 assets
|
||||||
|
- Cost: ~$13.30 of $13.33 budget
|
||||||
BIN
grok-imagine-gallery/V01-paperclip-cosmos.mp4
Normal file
BIN
grok-imagine-gallery/V02-wizard-tower-orbit.mp4
Normal file
17
hermes-sovereign/mempalace/__init__.py
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
"""MemPalace integration for Hermes sovereign agent.
|
||||||
|
|
||||||
|
Provides:
|
||||||
|
- mempalace.py: PalaceRoom + Mempalace classes for analytical workflows
|
||||||
|
- retrieval_enforcer.py: L0-L5 retrieval order enforcement
|
||||||
|
- wakeup.py: Session wake-up protocol (~300-900 tokens)
|
||||||
|
- scratchpad.py: JSON-based session scratchpad with palace promotion
|
||||||
|
- sovereign_store.py: Zero-API durable memory (SQLite + FTS5 + HRR vectors)
|
||||||
|
- promotion.py: Quality-gated scratchpad-to-palace promotion (MP-4)
|
||||||
|
|
||||||
|
Epic: #367
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .mempalace import Mempalace, PalaceRoom, analyse_issues
|
||||||
|
from .sovereign_store import SovereignStore
|
||||||
|
|
||||||
|
__all__ = ["Mempalace", "PalaceRoom", "analyse_issues", "SovereignStore"]
|
||||||
225
hermes-sovereign/mempalace/mempalace.py
Normal file
@@ -0,0 +1,225 @@
|
|||||||
|
"""
|
||||||
|
---
|
||||||
|
title: Mempalace — Analytical Workflow Memory Framework
|
||||||
|
description: Applies spatial memory palace organization to analytical tasks (issue triage, repo audits, backlog analysis) for faster, more consistent results.
|
||||||
|
conditions:
|
||||||
|
- Analytical workflows over structured data (issues, PRs, repos)
|
||||||
|
- Repetitive triage or audit tasks where pattern recall improves speed
|
||||||
|
- Multi-repository scanning requiring consistent mental models
|
||||||
|
---
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import time
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class PalaceRoom:
|
||||||
|
"""A single 'room' in the memory palace — holds organized facts about one analytical dimension."""
|
||||||
|
|
||||||
|
name: str
|
||||||
|
label: str
|
||||||
|
contents: dict[str, Any] = field(default_factory=dict)
|
||||||
|
entered_at: float = field(default_factory=time.time)
|
||||||
|
|
||||||
|
def store(self, key: str, value: Any) -> None:
|
||||||
|
self.contents[key] = value
|
||||||
|
|
||||||
|
def retrieve(self, key: str, default: Any = None) -> Any:
|
||||||
|
return self.contents.get(key, default)
|
||||||
|
|
||||||
|
def summary(self) -> str:
|
||||||
|
lines = [f"## {self.label}"]
|
||||||
|
for k, v in self.contents.items():
|
||||||
|
lines.append(f" {k}: {v}")
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
class Mempalace:
|
||||||
|
"""
|
||||||
|
Spatial memory palace for analytical workflows.
|
||||||
|
|
||||||
|
Organises multi-dimensional data about a domain (e.g. Gitea issues) into
|
||||||
|
named rooms. Each room models one analytical dimension, making it easy to
|
||||||
|
traverse observations in a consistent order — the same pattern that produced
|
||||||
|
a 19% throughput improvement in Allegro's April 2026 evaluation.
|
||||||
|
|
||||||
|
Standard rooms for issue-analysis workflows
|
||||||
|
-------------------------------------------
|
||||||
|
repo_architecture Repository structure and inter-repo relationships
|
||||||
|
assignment_status Assigned vs unassigned issue distribution
|
||||||
|
triage_priority Priority / urgency levels (the "lighting system")
|
||||||
|
resolution_patterns Historical resolution trends and velocity
|
||||||
|
|
||||||
|
Usage
|
||||||
|
-----
|
||||||
|
>>> palace = Mempalace.for_issue_analysis()
|
||||||
|
>>> palace.enter("repo_architecture")
|
||||||
|
>>> palace.store("total_repos", 11)
|
||||||
|
>>> palace.store("repos_with_issues", 4)
|
||||||
|
>>> palace.enter("assignment_status")
|
||||||
|
>>> palace.store("assigned", 72)
|
||||||
|
>>> palace.store("unassigned", 22)
|
||||||
|
>>> print(palace.render())
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, domain: str = "general") -> None:
|
||||||
|
self.domain = domain
|
||||||
|
self._rooms: dict[str, PalaceRoom] = {}
|
||||||
|
self._current_room: str | None = None
|
||||||
|
self._created_at: float = time.time()
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Factory constructors for common analytical domains
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def for_issue_analysis(cls) -> "Mempalace":
|
||||||
|
"""Pre-wired palace for Gitea / forge issue-analysis workflows."""
|
||||||
|
p = cls(domain="issue_analysis")
|
||||||
|
p.add_room("repo_architecture", "Repository Architecture Room")
|
||||||
|
p.add_room("assignment_status", "Issue Assignment Status Room")
|
||||||
|
p.add_room("triage_priority", "Triage Priority Room")
|
||||||
|
p.add_room("resolution_patterns", "Resolution Patterns Room")
|
||||||
|
return p
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def for_health_check(cls) -> "Mempalace":
|
||||||
|
"""Pre-wired palace for CI / deployment health-check workflows."""
|
||||||
|
p = cls(domain="health_check")
|
||||||
|
p.add_room("service_topology", "Service Topology Room")
|
||||||
|
p.add_room("failure_signals", "Failure Signals Room")
|
||||||
|
p.add_room("recovery_history", "Recovery History Room")
|
||||||
|
return p
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def for_code_review(cls) -> "Mempalace":
|
||||||
|
"""Pre-wired palace for code-review / PR triage workflows."""
|
||||||
|
p = cls(domain="code_review")
|
||||||
|
p.add_room("change_scope", "Change Scope Room")
|
||||||
|
p.add_room("risk_surface", "Risk Surface Room")
|
||||||
|
p.add_room("test_coverage", "Test Coverage Room")
|
||||||
|
p.add_room("reviewer_context", "Reviewer Context Room")
|
||||||
|
return p
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Room management
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def add_room(self, key: str, label: str) -> PalaceRoom:
|
||||||
|
room = PalaceRoom(name=key, label=label)
|
||||||
|
self._rooms[key] = room
|
||||||
|
return room
|
||||||
|
|
||||||
|
def enter(self, room_key: str) -> PalaceRoom:
|
||||||
|
if room_key not in self._rooms:
|
||||||
|
raise KeyError(f"No room '{room_key}' in palace. Available: {list(self._rooms)}")
|
||||||
|
self._current_room = room_key
|
||||||
|
return self._rooms[room_key]
|
||||||
|
|
||||||
|
def store(self, key: str, value: Any) -> None:
|
||||||
|
"""Store a value in the currently active room."""
|
||||||
|
if self._current_room is None:
|
||||||
|
raise RuntimeError("Enter a room before storing values.")
|
||||||
|
self._rooms[self._current_room].store(key, value)
|
||||||
|
|
||||||
|
def retrieve(self, room_key: str, key: str, default: Any = None) -> Any:
|
||||||
|
if room_key not in self._rooms:
|
||||||
|
return default
|
||||||
|
return self._rooms[room_key].retrieve(key, default)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Rendering
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def render(self) -> str:
|
||||||
|
"""Return a human-readable summary of the entire palace."""
|
||||||
|
elapsed = time.time() - self._created_at
|
||||||
|
lines = [
|
||||||
|
f"# Mempalace — {self.domain}",
|
||||||
|
f"_traversal time: {elapsed:.2f}s | rooms: {len(self._rooms)}_",
|
||||||
|
"",
|
||||||
|
]
|
||||||
|
for room in self._rooms.values():
|
||||||
|
lines.append(room.summary())
|
||||||
|
lines.append("")
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
def to_dict(self) -> dict:
|
||||||
|
return {
|
||||||
|
"domain": self.domain,
|
||||||
|
"elapsed_seconds": round(time.time() - self._created_at, 3),
|
||||||
|
"rooms": {k: v.contents for k, v in self._rooms.items()},
|
||||||
|
}
|
||||||
|
|
||||||
|
def to_json(self) -> str:
|
||||||
|
return json.dumps(self.to_dict(), indent=2)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Skill entry-point
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def analyse_issues(
|
||||||
|
repos_data: list[dict],
|
||||||
|
target_assignee_rate: float = 0.80,
|
||||||
|
) -> str:
|
||||||
|
"""
|
||||||
|
Applies the mempalace technique to a list of repo issue summaries.
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
repos_data:
|
||||||
|
List of dicts, each with keys: ``repo``, ``open_issues``,
|
||||||
|
``assigned``, ``unassigned``.
|
||||||
|
target_assignee_rate:
|
||||||
|
Minimum acceptable assignee-coverage ratio (default 0.80).
|
||||||
|
|
||||||
|
Returns
|
||||||
|
-------
|
||||||
|
str
|
||||||
|
Rendered palace summary with coverage assessment.
|
||||||
|
"""
|
||||||
|
palace = Mempalace.for_issue_analysis()
|
||||||
|
|
||||||
|
# --- Repository Architecture Room ---
|
||||||
|
palace.enter("repo_architecture")
|
||||||
|
total_issues = sum(r.get("open_issues", 0) for r in repos_data)
|
||||||
|
repos_with_issues = sum(1 for r in repos_data if r.get("open_issues", 0) > 0)
|
||||||
|
palace.store("repos_sampled", len(repos_data))
|
||||||
|
palace.store("repos_with_issues", repos_with_issues)
|
||||||
|
palace.store("total_open_issues", total_issues)
|
||||||
|
palace.store(
|
||||||
|
"avg_issues_per_repo",
|
||||||
|
round(total_issues / len(repos_data), 1) if repos_data else 0,
|
||||||
|
)
|
||||||
|
|
||||||
|
# --- Assignment Status Room ---
|
||||||
|
palace.enter("assignment_status")
|
||||||
|
total_assigned = sum(r.get("assigned", 0) for r in repos_data)
|
||||||
|
total_unassigned = sum(r.get("unassigned", 0) for r in repos_data)
|
||||||
|
coverage = total_assigned / total_issues if total_issues else 0
|
||||||
|
palace.store("assigned", total_assigned)
|
||||||
|
palace.store("unassigned", total_unassigned)
|
||||||
|
palace.store("coverage_rate", round(coverage, 3))
|
||||||
|
palace.store(
|
||||||
|
"coverage_status",
|
||||||
|
"OK" if coverage >= target_assignee_rate else f"BELOW TARGET ({target_assignee_rate:.0%})",
|
||||||
|
)
|
||||||
|
|
||||||
|
# --- Triage Priority Room ---
|
||||||
|
palace.enter("triage_priority")
|
||||||
|
unassigned_repos = [r["repo"] for r in repos_data if r.get("unassigned", 0) > 0]
|
||||||
|
palace.store("repos_needing_triage", unassigned_repos)
|
||||||
|
palace.store("triage_count", total_unassigned)
|
||||||
|
|
||||||
|
# --- Resolution Patterns Room ---
|
||||||
|
palace.enter("resolution_patterns")
|
||||||
|
palace.store("technique", "mempalace")
|
||||||
|
palace.store("target_assignee_rate", target_assignee_rate)
|
||||||
|
|
||||||
|
return palace.render()
|
||||||
188
hermes-sovereign/mempalace/promotion.py
Normal file
@@ -0,0 +1,188 @@
|
|||||||
|
"""Memory Promotion — quality-gated scratchpad-to-palace promotion.
|
||||||
|
|
||||||
|
Implements MP-4 (#371): move session notes to durable memory only when
|
||||||
|
they pass quality gates. No LLM calls — all heuristic-based.
|
||||||
|
|
||||||
|
Quality gates:
|
||||||
|
1. Minimum content length (too short = noise)
|
||||||
|
2. Duplicate detection (FTS5 + HRR similarity check)
|
||||||
|
3. Structural quality (has subject-verb structure, not just a fragment)
|
||||||
|
4. Staleness check (don't promote stale notes from old sessions)
|
||||||
|
|
||||||
|
Refs: Epic #367, Sub-issue #371
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
import time
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
try:
|
||||||
|
from .sovereign_store import SovereignStore
|
||||||
|
except ImportError:
|
||||||
|
from sovereign_store import SovereignStore
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Quality gate thresholds
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
MIN_CONTENT_WORDS = 5
|
||||||
|
MAX_CONTENT_WORDS = 500
|
||||||
|
DUPLICATE_SIMILARITY = 0.85
|
||||||
|
DUPLICATE_FTS_THRESHOLD = 3
|
||||||
|
STALE_SECONDS = 86400 * 7
|
||||||
|
MIN_TRUST_FOR_AUTO = 0.4
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Quality checks
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _check_length(content: str) -> tuple[bool, str]:
|
||||||
|
"""Gate 1: Content length check."""
|
||||||
|
words = content.split()
|
||||||
|
if len(words) < MIN_CONTENT_WORDS:
|
||||||
|
return False, f"Too short ({len(words)} words, minimum {MIN_CONTENT_WORDS})"
|
||||||
|
if len(words) > MAX_CONTENT_WORDS:
|
||||||
|
return False, f"Too long ({len(words)} words, maximum {MAX_CONTENT_WORDS}). Summarize first."
|
||||||
|
return True, "OK"
|
||||||
|
|
||||||
|
|
||||||
|
def _check_structure(content: str) -> tuple[bool, str]:
|
||||||
|
"""Gate 2: Basic structural quality."""
|
||||||
|
if not re.search(r"[a-zA-Z]", content):
|
||||||
|
return False, "No alphabetic content — pure code/numbers are not memory-worthy"
|
||||||
|
if len(content.split()) < 3:
|
||||||
|
return False, "Fragment — needs at least subject + predicate"
|
||||||
|
return True, "OK"
|
||||||
|
|
||||||
|
|
||||||
|
def _check_duplicate(content: str, store: SovereignStore, room: str) -> tuple[bool, str]:
|
||||||
|
"""Gate 3: Duplicate detection via hybrid search."""
|
||||||
|
results = store.search(content, room=room, limit=5, min_trust=0.0)
|
||||||
|
for r in results:
|
||||||
|
if r["score"] > DUPLICATE_SIMILARITY:
|
||||||
|
return False, f"Duplicate detected: memory #{r['memory_id']} (score {r['score']:.3f})"
|
||||||
|
if _text_overlap(content, r["content"]) > 0.8:
|
||||||
|
return False, f"Near-duplicate text: memory #{r['memory_id']}"
|
||||||
|
return True, "OK"
|
||||||
|
|
||||||
|
|
||||||
|
def _check_staleness(written_at: float) -> tuple[bool, str]:
|
||||||
|
"""Gate 4: Staleness check."""
|
||||||
|
age = time.time() - written_at
|
||||||
|
if age > STALE_SECONDS:
|
||||||
|
days = int(age / 86400)
|
||||||
|
return False, f"Stale ({days} days old). Review manually before promoting."
|
||||||
|
return True, "OK"
|
||||||
|
|
||||||
|
|
||||||
|
def _text_overlap(a: str, b: str) -> float:
|
||||||
|
"""Jaccard similarity between two texts (word-level)."""
|
||||||
|
words_a = set(a.lower().split())
|
||||||
|
words_b = set(b.lower().split())
|
||||||
|
if not words_a or not words_b:
|
||||||
|
return 0.0
|
||||||
|
intersection = words_a & words_b
|
||||||
|
union = words_a | words_b
|
||||||
|
return len(intersection) / len(union)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Public API
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class PromotionResult:
|
||||||
|
"""Result of a promotion attempt."""
|
||||||
|
def __init__(self, success: bool, memory_id: Optional[int], reason: str, gates: dict):
|
||||||
|
self.success = success
|
||||||
|
self.memory_id = memory_id
|
||||||
|
self.reason = reason
|
||||||
|
self.gates = gates
|
||||||
|
|
||||||
|
def __repr__(self):
|
||||||
|
status = "PROMOTED" if self.success else "REJECTED"
|
||||||
|
return f"PromotionResult({status}: {self.reason})"
|
||||||
|
|
||||||
|
|
||||||
|
def evaluate_for_promotion(
|
||||||
|
content: str,
|
||||||
|
store: SovereignStore,
|
||||||
|
room: str = "general",
|
||||||
|
written_at: Optional[float] = None,
|
||||||
|
) -> dict:
|
||||||
|
"""Run all quality gates without actually promoting."""
|
||||||
|
if written_at is None:
|
||||||
|
written_at = time.time()
|
||||||
|
gates = {}
|
||||||
|
gates["length"] = _check_length(content)
|
||||||
|
gates["structure"] = _check_structure(content)
|
||||||
|
gates["duplicate"] = _check_duplicate(content, store, room)
|
||||||
|
gates["staleness"] = _check_staleness(written_at)
|
||||||
|
all_passed = all(passed for passed, _ in gates.values())
|
||||||
|
return {
|
||||||
|
"eligible": all_passed,
|
||||||
|
"gates": gates,
|
||||||
|
"content_preview": content[:100] + ("..." if len(content) > 100 else ""),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def promote(
|
||||||
|
content: str,
|
||||||
|
store: SovereignStore,
|
||||||
|
session_id: str,
|
||||||
|
scratch_key: str,
|
||||||
|
room: str = "general",
|
||||||
|
category: str = "",
|
||||||
|
trust: float = 0.5,
|
||||||
|
written_at: Optional[float] = None,
|
||||||
|
force: bool = False,
|
||||||
|
) -> PromotionResult:
|
||||||
|
"""Promote a scratchpad note to durable palace memory."""
|
||||||
|
if written_at is None:
|
||||||
|
written_at = time.time()
|
||||||
|
gates = {}
|
||||||
|
if not force:
|
||||||
|
gates["length"] = _check_length(content)
|
||||||
|
gates["structure"] = _check_structure(content)
|
||||||
|
gates["duplicate"] = _check_duplicate(content, store, room)
|
||||||
|
gates["staleness"] = _check_staleness(written_at)
|
||||||
|
for gate_name, (passed, message) in gates.items():
|
||||||
|
if not passed:
|
||||||
|
return PromotionResult(
|
||||||
|
success=False, memory_id=None,
|
||||||
|
reason=f"Failed gate '{gate_name}': {message}", gates=gates,
|
||||||
|
)
|
||||||
|
memory_id = store.store(content, room=room, category=category, trust=trust)
|
||||||
|
store.log_promotion(session_id, scratch_key, memory_id, reason="auto" if not force else "forced")
|
||||||
|
return PromotionResult(success=True, memory_id=memory_id, reason="Promoted to durable memory", gates=gates)
|
||||||
|
|
||||||
|
|
||||||
|
def promote_session_batch(
|
||||||
|
store: SovereignStore,
|
||||||
|
session_id: str,
|
||||||
|
notes: dict[str, dict],
|
||||||
|
room: str = "general",
|
||||||
|
force: bool = False,
|
||||||
|
) -> list[PromotionResult]:
|
||||||
|
"""Promote all notes from a session scratchpad."""
|
||||||
|
results = []
|
||||||
|
for key, entry in notes.items():
|
||||||
|
content = entry.get("value", str(entry)) if isinstance(entry, dict) else str(entry)
|
||||||
|
written_at = None
|
||||||
|
if isinstance(entry, dict) and "written_at" in entry:
|
||||||
|
try:
|
||||||
|
import datetime
|
||||||
|
written_at = datetime.datetime.strptime(
|
||||||
|
entry["written_at"], "%Y-%m-%d %H:%M:%S"
|
||||||
|
).timestamp()
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
pass
|
||||||
|
result = promote(
|
||||||
|
content=str(content), store=store, session_id=session_id,
|
||||||
|
scratch_key=key, room=room, written_at=written_at, force=force,
|
||||||
|
)
|
||||||
|
results.append(result)
|
||||||
|
return results
|
||||||
310
hermes-sovereign/mempalace/retrieval_enforcer.py
Normal file
@@ -0,0 +1,310 @@
|
|||||||
|
"""Retrieval Order Enforcer — L0 through L5 memory hierarchy.
|
||||||
|
|
||||||
|
Ensures the agent checks durable memory before falling back to free generation.
|
||||||
|
Gracefully degrades if any layer is unavailable (missing files, etc).
|
||||||
|
|
||||||
|
Layer order:
|
||||||
|
L0: Identity (~/.mempalace/identity.txt)
|
||||||
|
L1: Palace rooms (SovereignStore — SQLite + FTS5 + HRR, zero API calls)
|
||||||
|
L2: Session scratch (~/.hermes/scratchpad/{session_id}.json)
|
||||||
|
L3: Gitea artifacts (API search for issues/PRs)
|
||||||
|
L4: Procedures (skills directory search)
|
||||||
|
L5: Free generation (only if L0-L4 produced nothing)
|
||||||
|
|
||||||
|
Refs: Epic #367, Sub-issue #369, Wiring: #383
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Sovereign Store (replaces mempalace CLI subprocess)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
try:
|
||||||
|
from .sovereign_store import SovereignStore
|
||||||
|
except ImportError:
|
||||||
|
try:
|
||||||
|
from sovereign_store import SovereignStore
|
||||||
|
except ImportError:
|
||||||
|
SovereignStore = None # type: ignore[misc,assignment]
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Constants
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
IDENTITY_PATH = Path.home() / ".mempalace" / "identity.txt"
|
||||||
|
SCRATCHPAD_DIR = Path.home() / ".hermes" / "scratchpad"
|
||||||
|
SKILLS_DIR = Path.home() / ".hermes" / "skills"
|
||||||
|
SOVEREIGN_DB = Path.home() / ".hermes" / "palace" / "sovereign.db"
|
||||||
|
|
||||||
|
# Patterns that indicate a recall-style query
|
||||||
|
RECALL_PATTERNS = re.compile(
|
||||||
|
r"(?i)\b("
|
||||||
|
r"what did|status of|remember|last time|yesterday|previously|"
|
||||||
|
r"we discussed|we talked|we worked|you said|you mentioned|"
|
||||||
|
r"remind me|what was|what were|how did|when did|"
|
||||||
|
r"earlier today|last session|before this"
|
||||||
|
r")\b"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Singleton store instance (lazy-init)
|
||||||
|
_store: Optional["SovereignStore"] = None
|
||||||
|
|
||||||
|
|
||||||
|
def _get_store() -> Optional["SovereignStore"]:
|
||||||
|
"""Lazy-init the SovereignStore singleton."""
|
||||||
|
global _store
|
||||||
|
if _store is not None:
|
||||||
|
return _store
|
||||||
|
if SovereignStore is None:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
_store = SovereignStore(db_path=str(SOVEREIGN_DB))
|
||||||
|
return _store
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# L0: Identity
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def load_identity() -> str:
|
||||||
|
"""Read the agent identity file. Returns empty string on failure."""
|
||||||
|
try:
|
||||||
|
if IDENTITY_PATH.exists():
|
||||||
|
text = IDENTITY_PATH.read_text(encoding="utf-8").strip()
|
||||||
|
# Cap at ~200 tokens to keep wake-up lean
|
||||||
|
if len(text.split()) > 200:
|
||||||
|
text = " ".join(text.split()[:200]) + "..."
|
||||||
|
return text
|
||||||
|
except (OSError, PermissionError):
|
||||||
|
pass
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# L1: Palace search (now via SovereignStore — zero subprocess, zero API)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def search_palace(query: str, room: Optional[str] = None) -> str:
|
||||||
|
"""Search the sovereign memory store for relevant memories.
|
||||||
|
|
||||||
|
Uses SovereignStore (SQLite + FTS5 + HRR) for hybrid keyword + semantic
|
||||||
|
search. No subprocess calls, no ONNX, no API keys.
|
||||||
|
|
||||||
|
Gracefully degrades to empty string if store is unavailable.
|
||||||
|
"""
|
||||||
|
store = _get_store()
|
||||||
|
if store is None:
|
||||||
|
return ""
|
||||||
|
try:
|
||||||
|
results = store.search(query, room=room, limit=5, min_trust=0.2)
|
||||||
|
if not results:
|
||||||
|
return ""
|
||||||
|
lines = []
|
||||||
|
for r in results:
|
||||||
|
trust = r.get("trust_score", 0.5)
|
||||||
|
room_name = r.get("room", "general")
|
||||||
|
content = r.get("content", "")
|
||||||
|
lines.append(f" [{room_name}] (trust:{trust:.2f}) {content}")
|
||||||
|
return "\n".join(lines)
|
||||||
|
except Exception:
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# L2: Session scratchpad
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def load_scratchpad(session_id: str) -> str:
|
||||||
|
"""Load the session scratchpad as formatted text."""
|
||||||
|
try:
|
||||||
|
scratch_file = SCRATCHPAD_DIR / f"{session_id}.json"
|
||||||
|
if scratch_file.exists():
|
||||||
|
data = json.loads(scratch_file.read_text(encoding="utf-8"))
|
||||||
|
if isinstance(data, dict) and data:
|
||||||
|
lines = []
|
||||||
|
for k, v in data.items():
|
||||||
|
lines.append(f" {k}: {v}")
|
||||||
|
return "\n".join(lines)
|
||||||
|
except (OSError, json.JSONDecodeError):
|
||||||
|
pass
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# L3: Gitea artifact search
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _load_gitea_token() -> str:
|
||||||
|
"""Read the Gitea API token."""
|
||||||
|
token_path = Path.home() / ".hermes" / "gitea_token_vps"
|
||||||
|
try:
|
||||||
|
if token_path.exists():
|
||||||
|
return token_path.read_text(encoding="utf-8").strip()
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def search_gitea(query: str) -> str:
|
||||||
|
"""Search Gitea issues/PRs for context. Returns formatted text or empty string."""
|
||||||
|
token = _load_gitea_token()
|
||||||
|
if not token:
|
||||||
|
return ""
|
||||||
|
|
||||||
|
api_base = "https://forge.alexanderwhitestone.com/api/v1"
|
||||||
|
# Extract key terms for search (first 3 significant words)
|
||||||
|
terms = [w for w in query.split() if len(w) > 3][:3]
|
||||||
|
search_q = " ".join(terms) if terms else query[:50]
|
||||||
|
|
||||||
|
try:
|
||||||
|
import urllib.request
|
||||||
|
import urllib.parse
|
||||||
|
|
||||||
|
url = (
|
||||||
|
f"{api_base}/repos/search?"
|
||||||
|
f"q={urllib.parse.quote(search_q)}&limit=3"
|
||||||
|
)
|
||||||
|
req = urllib.request.Request(url, headers={
|
||||||
|
"Authorization": f"token {token}",
|
||||||
|
"Accept": "application/json",
|
||||||
|
})
|
||||||
|
with urllib.request.urlopen(req, timeout=8) as resp:
|
||||||
|
data = json.loads(resp.read().decode())
|
||||||
|
if data.get("data"):
|
||||||
|
lines = []
|
||||||
|
for repo in data["data"][:3]:
|
||||||
|
lines.append(f" {repo['full_name']}: {repo.get('description', 'no desc')}")
|
||||||
|
return "\n".join(lines)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# L4: Procedures (skills search)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def search_skills(query: str) -> str:
|
||||||
|
"""Search skills directory for matching procedures."""
|
||||||
|
try:
|
||||||
|
if not SKILLS_DIR.exists():
|
||||||
|
return ""
|
||||||
|
|
||||||
|
query_lower = query.lower()
|
||||||
|
terms = [w for w in query_lower.split() if len(w) > 3]
|
||||||
|
if not terms:
|
||||||
|
return ""
|
||||||
|
|
||||||
|
matches = []
|
||||||
|
for skill_dir in SKILLS_DIR.iterdir():
|
||||||
|
if not skill_dir.is_dir():
|
||||||
|
continue
|
||||||
|
skill_md = skill_dir / "SKILL.md"
|
||||||
|
if skill_md.exists():
|
||||||
|
try:
|
||||||
|
content = skill_md.read_text(encoding="utf-8").lower()
|
||||||
|
if any(t in content for t in terms):
|
||||||
|
title = skill_dir.name
|
||||||
|
matches.append(f" skill: {title}")
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if matches:
|
||||||
|
return "\n".join(matches[:5])
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main enforcer
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def is_recall_query(query: str) -> bool:
|
||||||
|
"""Detect whether a query is asking for recalled/historical information."""
|
||||||
|
return bool(RECALL_PATTERNS.search(query))
|
||||||
|
|
||||||
|
|
||||||
|
def enforce_retrieval_order(
|
||||||
|
query: str,
|
||||||
|
session_id: Optional[str] = None,
|
||||||
|
skip_if_not_recall: bool = True,
|
||||||
|
) -> dict:
|
||||||
|
"""Check palace layers before allowing free generation.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
query: The user's query text.
|
||||||
|
session_id: Current session ID for scratchpad access.
|
||||||
|
skip_if_not_recall: If True (default), skip enforcement for
|
||||||
|
non-recall queries and return empty result.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
dict with keys:
|
||||||
|
retrieved_from: Highest layer that produced results (e.g. 'L1')
|
||||||
|
context: Aggregated context string
|
||||||
|
tokens: Approximate word count of context
|
||||||
|
layers_checked: List of layers that were consulted
|
||||||
|
"""
|
||||||
|
result = {
|
||||||
|
"retrieved_from": None,
|
||||||
|
"context": "",
|
||||||
|
"tokens": 0,
|
||||||
|
"layers_checked": [],
|
||||||
|
}
|
||||||
|
|
||||||
|
# Gate: skip for non-recall queries if configured
|
||||||
|
if skip_if_not_recall and not is_recall_query(query):
|
||||||
|
return result
|
||||||
|
|
||||||
|
# L0: Identity (always prepend)
|
||||||
|
identity = load_identity()
|
||||||
|
if identity:
|
||||||
|
result["context"] += f"## Identity\n{identity}\n\n"
|
||||||
|
result["layers_checked"].append("L0")
|
||||||
|
|
||||||
|
# L1: Palace search (SovereignStore — zero API, zero subprocess)
|
||||||
|
palace_results = search_palace(query)
|
||||||
|
if palace_results:
|
||||||
|
result["context"] += f"## Palace Memory\n{palace_results}\n\n"
|
||||||
|
result["retrieved_from"] = "L1"
|
||||||
|
result["layers_checked"].append("L1")
|
||||||
|
|
||||||
|
# L2: Scratchpad
|
||||||
|
if session_id:
|
||||||
|
scratch = load_scratchpad(session_id)
|
||||||
|
if scratch:
|
||||||
|
result["context"] += f"## Session Notes\n{scratch}\n\n"
|
||||||
|
if not result["retrieved_from"]:
|
||||||
|
result["retrieved_from"] = "L2"
|
||||||
|
result["layers_checked"].append("L2")
|
||||||
|
|
||||||
|
# L3: Gitea artifacts (only if still no context from L1/L2)
|
||||||
|
if not result["retrieved_from"]:
|
||||||
|
artifacts = search_gitea(query)
|
||||||
|
if artifacts:
|
||||||
|
result["context"] += f"## Gitea Context\n{artifacts}\n\n"
|
||||||
|
result["retrieved_from"] = "L3"
|
||||||
|
result["layers_checked"].append("L3")
|
||||||
|
|
||||||
|
# L4: Procedures (only if still no context)
|
||||||
|
if not result["retrieved_from"]:
|
||||||
|
procedures = search_skills(query)
|
||||||
|
if procedures:
|
||||||
|
result["context"] += f"## Related Skills\n{procedures}\n\n"
|
||||||
|
result["retrieved_from"] = "L4"
|
||||||
|
result["layers_checked"].append("L4")
|
||||||
|
|
||||||
|
# L5: Free generation (no context found — just mark it)
|
||||||
|
if not result["retrieved_from"]:
|
||||||
|
result["retrieved_from"] = "L5"
|
||||||
|
result["layers_checked"].append("L5")
|
||||||
|
|
||||||
|
result["tokens"] = len(result["context"].split())
|
||||||
|
return result
|
||||||
184
hermes-sovereign/mempalace/scratchpad.py
Normal file
@@ -0,0 +1,184 @@
|
|||||||
|
"""Session Scratchpad — ephemeral key-value notes per session.
|
||||||
|
|
||||||
|
Provides fast, JSON-backed scratch storage that lives for a session
|
||||||
|
and can be promoted to durable palace memory.
|
||||||
|
|
||||||
|
Storage: ~/.hermes/scratchpad/{session_id}.json
|
||||||
|
|
||||||
|
Refs: Epic #367, Sub-issue #372
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Optional
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Constants
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
SCRATCHPAD_DIR = Path.home() / ".hermes" / "scratchpad"
|
||||||
|
MEMPALACE_BIN = "/Library/Frameworks/Python.framework/Versions/3.12/bin/mempalace"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Internal helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _scratch_path(session_id: str) -> Path:
|
||||||
|
"""Return the JSON file path for a given session."""
|
||||||
|
# Sanitize session_id to prevent path traversal
|
||||||
|
safe_id = "".join(c for c in session_id if c.isalnum() or c in "-_")
|
||||||
|
if not safe_id:
|
||||||
|
safe_id = "unnamed"
|
||||||
|
return SCRATCHPAD_DIR / f"{safe_id}.json"
|
||||||
|
|
||||||
|
|
||||||
|
def _load(session_id: str) -> dict:
|
||||||
|
"""Load scratchpad data, returning empty dict on failure."""
|
||||||
|
path = _scratch_path(session_id)
|
||||||
|
try:
|
||||||
|
if path.exists():
|
||||||
|
return json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
except (OSError, json.JSONDecodeError):
|
||||||
|
pass
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def _save(session_id: str, data: dict) -> None:
|
||||||
|
"""Persist scratchpad data to disk."""
|
||||||
|
SCRATCHPAD_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
path = _scratch_path(session_id)
|
||||||
|
path.write_text(json.dumps(data, indent=2, default=str), encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Public API
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def write_scratch(session_id: str, key: str, value: Any) -> None:
|
||||||
|
"""Write a note to the session scratchpad.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
session_id: Current session identifier.
|
||||||
|
key: Note key (string).
|
||||||
|
value: Note value (any JSON-serializable type).
|
||||||
|
"""
|
||||||
|
data = _load(session_id)
|
||||||
|
data[key] = {
|
||||||
|
"value": value,
|
||||||
|
"written_at": time.strftime("%Y-%m-%d %H:%M:%S"),
|
||||||
|
}
|
||||||
|
_save(session_id, data)
|
||||||
|
|
||||||
|
|
||||||
|
def read_scratch(session_id: str, key: Optional[str] = None) -> dict:
|
||||||
|
"""Read session scratchpad (all keys or one).
|
||||||
|
|
||||||
|
Args:
|
||||||
|
session_id: Current session identifier.
|
||||||
|
key: Optional specific key. If None, returns all entries.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
dict — either {key: {value, written_at}} or the full scratchpad.
|
||||||
|
"""
|
||||||
|
data = _load(session_id)
|
||||||
|
if key is not None:
|
||||||
|
entry = data.get(key)
|
||||||
|
return {key: entry} if entry else {}
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
def delete_scratch(session_id: str, key: str) -> bool:
|
||||||
|
"""Remove a single key from the scratchpad.
|
||||||
|
|
||||||
|
Returns True if the key existed and was removed.
|
||||||
|
"""
|
||||||
|
data = _load(session_id)
|
||||||
|
if key in data:
|
||||||
|
del data[key]
|
||||||
|
_save(session_id, data)
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def list_sessions() -> list[str]:
|
||||||
|
"""List all session IDs that have scratchpad files."""
|
||||||
|
try:
|
||||||
|
if SCRATCHPAD_DIR.exists():
|
||||||
|
return [
|
||||||
|
f.stem
|
||||||
|
for f in SCRATCHPAD_DIR.iterdir()
|
||||||
|
if f.suffix == ".json" and f.is_file()
|
||||||
|
]
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def promote_to_palace(
|
||||||
|
session_id: str,
|
||||||
|
key: str,
|
||||||
|
room: str = "general",
|
||||||
|
drawer: Optional[str] = None,
|
||||||
|
) -> bool:
|
||||||
|
"""Move a scratchpad note to durable palace memory.
|
||||||
|
|
||||||
|
Uses the mempalace CLI to store the note in the specified room.
|
||||||
|
Removes the note from the scratchpad after successful promotion.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
session_id: Session containing the note.
|
||||||
|
key: Scratchpad key to promote.
|
||||||
|
room: Palace room name (default: 'general').
|
||||||
|
drawer: Optional drawer name within the room. Defaults to key.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True if promotion succeeded, False otherwise.
|
||||||
|
"""
|
||||||
|
data = _load(session_id)
|
||||||
|
entry = data.get(key)
|
||||||
|
if not entry:
|
||||||
|
return False
|
||||||
|
|
||||||
|
value = entry.get("value", entry) if isinstance(entry, dict) else entry
|
||||||
|
content = json.dumps(value, default=str) if not isinstance(value, str) else value
|
||||||
|
|
||||||
|
try:
|
||||||
|
bin_path = MEMPALACE_BIN if os.path.exists(MEMPALACE_BIN) else "mempalace"
|
||||||
|
target_drawer = drawer or key
|
||||||
|
result = subprocess.run(
|
||||||
|
[bin_path, "store", room, target_drawer, content],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=10,
|
||||||
|
)
|
||||||
|
if result.returncode == 0:
|
||||||
|
# Remove from scratchpad after successful promotion
|
||||||
|
del data[key]
|
||||||
|
_save(session_id, data)
|
||||||
|
return True
|
||||||
|
except (FileNotFoundError, subprocess.TimeoutExpired, OSError):
|
||||||
|
# mempalace CLI not available — degrade gracefully
|
||||||
|
pass
|
||||||
|
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def clear_session(session_id: str) -> bool:
|
||||||
|
"""Delete the entire scratchpad for a session.
|
||||||
|
|
||||||
|
Returns True if the file existed and was removed.
|
||||||
|
"""
|
||||||
|
path = _scratch_path(session_id)
|
||||||
|
try:
|
||||||
|
if path.exists():
|
||||||
|
path.unlink()
|
||||||
|
return True
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
return False
|
||||||
474
hermes-sovereign/mempalace/sovereign_store.py
Normal file
@@ -0,0 +1,474 @@
|
|||||||
|
"""Sovereign Memory Store — zero-API, zero-dependency durable memory.
|
||||||
|
|
||||||
|
Replaces the third-party `mempalace` CLI and its ONNX requirement with a
|
||||||
|
self-contained SQLite + FTS5 + HRR (Holographic Reduced Representation)
|
||||||
|
store. Every operation is local: no network calls, no API keys, no cloud.
|
||||||
|
|
||||||
|
Storage: ~/.hermes/palace/sovereign.db
|
||||||
|
|
||||||
|
Capabilities:
|
||||||
|
- Durable fact storage with rooms, categories, and trust scores
|
||||||
|
- Hybrid retrieval: FTS5 keyword search + HRR cosine similarity
|
||||||
|
- Reciprocal Rank Fusion to merge keyword and semantic results
|
||||||
|
- Trust scoring: facts that get retrieved and confirmed gain trust
|
||||||
|
- Graceful numpy degradation: falls back to keyword-only if missing
|
||||||
|
|
||||||
|
Refs: Epic #367, MP-3 #370, MP-4 #371
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import sqlite3
|
||||||
|
import struct
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Optional
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# HRR (Holographic Reduced Representations) — zero-dependency vectors
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Phase-encoded vectors via SHA-256. No ONNX, no embeddings API, no numpy
|
||||||
|
# required (but uses numpy when available for speed).
|
||||||
|
|
||||||
|
_TWO_PI = 2.0 * math.pi
|
||||||
|
_DIM = 512 # Compact dimension — sufficient for memory retrieval
|
||||||
|
|
||||||
|
try:
|
||||||
|
import numpy as np
|
||||||
|
_HAS_NUMPY = True
|
||||||
|
except ImportError:
|
||||||
|
_HAS_NUMPY = False
|
||||||
|
|
||||||
|
|
||||||
|
def _encode_atom_np(word: str, dim: int = _DIM) -> "np.ndarray":
|
||||||
|
"""Deterministic phase vector via SHA-256 (numpy path)."""
|
||||||
|
values_per_block = 16
|
||||||
|
blocks_needed = math.ceil(dim / values_per_block)
|
||||||
|
uint16_values: list[int] = []
|
||||||
|
for i in range(blocks_needed):
|
||||||
|
digest = hashlib.sha256(f"{word}:{i}".encode()).digest()
|
||||||
|
uint16_values.extend(struct.unpack("<16H", digest))
|
||||||
|
return np.array(uint16_values[:dim], dtype=np.float64) * (_TWO_PI / 65536.0)
|
||||||
|
|
||||||
|
|
||||||
|
def _encode_atom_pure(word: str, dim: int = _DIM) -> list[float]:
|
||||||
|
"""Deterministic phase vector via SHA-256 (pure Python fallback)."""
|
||||||
|
values_per_block = 16
|
||||||
|
blocks_needed = math.ceil(dim / values_per_block)
|
||||||
|
uint16_values: list[int] = []
|
||||||
|
for i in range(blocks_needed):
|
||||||
|
digest = hashlib.sha256(f"{word}:{i}".encode()).digest()
|
||||||
|
for j in range(0, 32, 2):
|
||||||
|
uint16_values.append(int.from_bytes(digest[j:j+2], "little"))
|
||||||
|
return [v * (_TWO_PI / 65536.0) for v in uint16_values[:dim]]
|
||||||
|
|
||||||
|
|
||||||
|
def encode_text(text: str, dim: int = _DIM):
|
||||||
|
"""Encode a text string into an HRR phase vector by bundling word atoms.
|
||||||
|
|
||||||
|
Uses circular mean of per-word phase vectors — the standard HRR
|
||||||
|
superposition operation. Result is a fixed-width vector regardless
|
||||||
|
of input length.
|
||||||
|
"""
|
||||||
|
words = text.lower().split()
|
||||||
|
if not words:
|
||||||
|
words = ["<empty>"]
|
||||||
|
|
||||||
|
if _HAS_NUMPY:
|
||||||
|
atoms = [_encode_atom_np(w, dim) for w in words]
|
||||||
|
# Circular mean: average the unit vectors, extract phase
|
||||||
|
unit_sum = sum(np.exp(1j * a) for a in atoms)
|
||||||
|
return np.angle(unit_sum) % _TWO_PI
|
||||||
|
else:
|
||||||
|
# Pure Python circular mean
|
||||||
|
real_sum = [0.0] * dim
|
||||||
|
imag_sum = [0.0] * dim
|
||||||
|
for w in words:
|
||||||
|
atom = _encode_atom_pure(w, dim)
|
||||||
|
for d in range(dim):
|
||||||
|
real_sum[d] += math.cos(atom[d])
|
||||||
|
imag_sum[d] += math.sin(atom[d])
|
||||||
|
return [math.atan2(imag_sum[d], real_sum[d]) % _TWO_PI for d in range(dim)]
|
||||||
|
|
||||||
|
|
||||||
|
def cosine_similarity_phase(a, b) -> float:
|
||||||
|
"""Cosine similarity between two phase vectors.
|
||||||
|
|
||||||
|
For phase vectors, similarity = mean(cos(a - b)).
|
||||||
|
"""
|
||||||
|
if _HAS_NUMPY:
|
||||||
|
return float(np.mean(np.cos(np.array(a) - np.array(b))))
|
||||||
|
else:
|
||||||
|
n = len(a)
|
||||||
|
return sum(math.cos(a[i] - b[i]) for i in range(n)) / n
|
||||||
|
|
||||||
|
|
||||||
|
def serialize_vector(vec) -> bytes:
|
||||||
|
"""Serialize a vector to bytes for SQLite storage."""
|
||||||
|
if _HAS_NUMPY:
|
||||||
|
return vec.astype(np.float64).tobytes()
|
||||||
|
else:
|
||||||
|
return struct.pack(f"{len(vec)}d", *vec)
|
||||||
|
|
||||||
|
|
||||||
|
def deserialize_vector(blob: bytes):
|
||||||
|
"""Deserialize bytes back to a vector."""
|
||||||
|
n = len(blob) // 8 # float64 = 8 bytes
|
||||||
|
if _HAS_NUMPY:
|
||||||
|
return np.frombuffer(blob, dtype=np.float64)
|
||||||
|
else:
|
||||||
|
return list(struct.unpack(f"{n}d", blob))
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# SQLite Schema
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_SCHEMA = """
|
||||||
|
CREATE TABLE IF NOT EXISTS memories (
|
||||||
|
memory_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
content TEXT NOT NULL,
|
||||||
|
room TEXT DEFAULT 'general',
|
||||||
|
category TEXT DEFAULT '',
|
||||||
|
trust_score REAL DEFAULT 0.5,
|
||||||
|
retrieval_count INTEGER DEFAULT 0,
|
||||||
|
created_at REAL NOT NULL,
|
||||||
|
updated_at REAL NOT NULL,
|
||||||
|
hrr_vector BLOB
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_memories_room ON memories(room);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_memories_trust ON memories(trust_score DESC);
|
||||||
|
|
||||||
|
-- FTS5 for fast keyword search
|
||||||
|
CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5(
|
||||||
|
content, room, category,
|
||||||
|
content=memories, content_rowid=memory_id,
|
||||||
|
tokenize='porter unicode61'
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Sync triggers
|
||||||
|
CREATE TRIGGER IF NOT EXISTS memories_ai AFTER INSERT ON memories BEGIN
|
||||||
|
INSERT INTO memories_fts(rowid, content, room, category)
|
||||||
|
VALUES (new.memory_id, new.content, new.room, new.category);
|
||||||
|
END;
|
||||||
|
|
||||||
|
CREATE TRIGGER IF NOT EXISTS memories_ad AFTER DELETE ON memories BEGIN
|
||||||
|
INSERT INTO memories_fts(memories_fts, rowid, content, room, category)
|
||||||
|
VALUES ('delete', old.memory_id, old.content, old.room, old.category);
|
||||||
|
END;
|
||||||
|
|
||||||
|
CREATE TRIGGER IF NOT EXISTS memories_au AFTER UPDATE ON memories BEGIN
|
||||||
|
INSERT INTO memories_fts(memories_fts, rowid, content, room, category)
|
||||||
|
VALUES ('delete', old.memory_id, old.content, old.room, old.category);
|
||||||
|
INSERT INTO memories_fts(rowid, content, room, category)
|
||||||
|
VALUES (new.memory_id, new.content, new.room, new.category);
|
||||||
|
END;
|
||||||
|
|
||||||
|
-- Promotion log: tracks what moved from scratchpad to durable memory
|
||||||
|
CREATE TABLE IF NOT EXISTS promotion_log (
|
||||||
|
log_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
session_id TEXT NOT NULL,
|
||||||
|
scratch_key TEXT NOT NULL,
|
||||||
|
memory_id INTEGER REFERENCES memories(memory_id),
|
||||||
|
promoted_at REAL NOT NULL,
|
||||||
|
reason TEXT DEFAULT ''
|
||||||
|
);
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# SovereignStore
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class SovereignStore:
|
||||||
|
"""Zero-API durable memory store.
|
||||||
|
|
||||||
|
All operations are local SQLite. No network calls. No API keys.
|
||||||
|
HRR vectors provide semantic similarity without embedding models.
|
||||||
|
FTS5 provides fast keyword search. RRF merges both rankings.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, db_path: Optional[str] = None):
|
||||||
|
if db_path is None:
|
||||||
|
db_path = str(Path.home() / ".hermes" / "palace" / "sovereign.db")
|
||||||
|
self._db_path = db_path
|
||||||
|
Path(db_path).parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
self._conn = sqlite3.connect(db_path)
|
||||||
|
self._conn.row_factory = sqlite3.Row
|
||||||
|
self._conn.executescript(_SCHEMA)
|
||||||
|
|
||||||
|
def close(self):
|
||||||
|
self._conn.close()
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Store
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def store(
|
||||||
|
self,
|
||||||
|
content: str,
|
||||||
|
room: str = "general",
|
||||||
|
category: str = "",
|
||||||
|
trust: float = 0.5,
|
||||||
|
) -> int:
|
||||||
|
"""Store a fact in durable memory. Returns the memory_id."""
|
||||||
|
now = time.time()
|
||||||
|
vec = encode_text(content)
|
||||||
|
blob = serialize_vector(vec)
|
||||||
|
cur = self._conn.execute(
|
||||||
|
"""INSERT INTO memories (content, room, category, trust_score,
|
||||||
|
created_at, updated_at, hrr_vector)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?)""",
|
||||||
|
(content, room, category, trust, now, now, blob),
|
||||||
|
)
|
||||||
|
self._conn.commit()
|
||||||
|
return cur.lastrowid
|
||||||
|
|
||||||
|
def store_batch(self, items: list[dict]) -> list[int]:
|
||||||
|
"""Store multiple facts. Each item: {content, room?, category?, trust?}."""
|
||||||
|
ids = []
|
||||||
|
now = time.time()
|
||||||
|
for item in items:
|
||||||
|
content = item["content"]
|
||||||
|
vec = encode_text(content)
|
||||||
|
blob = serialize_vector(vec)
|
||||||
|
cur = self._conn.execute(
|
||||||
|
"""INSERT INTO memories (content, room, category, trust_score,
|
||||||
|
created_at, updated_at, hrr_vector)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?)""",
|
||||||
|
(
|
||||||
|
content,
|
||||||
|
item.get("room", "general"),
|
||||||
|
item.get("category", ""),
|
||||||
|
item.get("trust", 0.5),
|
||||||
|
now, now, blob,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
ids.append(cur.lastrowid)
|
||||||
|
self._conn.commit()
|
||||||
|
return ids
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Search — hybrid FTS5 + HRR with Reciprocal Rank Fusion
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def search(
|
||||||
|
self,
|
||||||
|
query: str,
|
||||||
|
room: Optional[str] = None,
|
||||||
|
limit: int = 10,
|
||||||
|
min_trust: float = 0.0,
|
||||||
|
fts_weight: float = 0.5,
|
||||||
|
hrr_weight: float = 0.5,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Hybrid search: FTS5 keywords + HRR semantic similarity.
|
||||||
|
|
||||||
|
Uses Reciprocal Rank Fusion (RRF) to merge both rankings.
|
||||||
|
Returns list of dicts with content, room, score, trust_score.
|
||||||
|
"""
|
||||||
|
k_rrf = 60 # Standard RRF constant
|
||||||
|
|
||||||
|
# Stage 1: FTS5 candidates
|
||||||
|
fts_results = self._fts_search(query, room, min_trust, limit * 3)
|
||||||
|
|
||||||
|
# Stage 2: HRR candidates (scan top N by trust)
|
||||||
|
hrr_results = self._hrr_search(query, room, min_trust, limit * 3)
|
||||||
|
|
||||||
|
# Stage 3: RRF fusion
|
||||||
|
scores: dict[int, float] = {}
|
||||||
|
meta: dict[int, dict] = {}
|
||||||
|
|
||||||
|
for rank, row in enumerate(fts_results):
|
||||||
|
mid = row["memory_id"]
|
||||||
|
scores[mid] = scores.get(mid, 0) + fts_weight / (k_rrf + rank + 1)
|
||||||
|
meta[mid] = dict(row)
|
||||||
|
|
||||||
|
for rank, row in enumerate(hrr_results):
|
||||||
|
mid = row["memory_id"]
|
||||||
|
scores[mid] = scores.get(mid, 0) + hrr_weight / (k_rrf + rank + 1)
|
||||||
|
if mid not in meta:
|
||||||
|
meta[mid] = dict(row)
|
||||||
|
|
||||||
|
# Sort by fused score
|
||||||
|
ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)[:limit]
|
||||||
|
|
||||||
|
results = []
|
||||||
|
for mid, score in ranked:
|
||||||
|
m = meta[mid]
|
||||||
|
# Bump retrieval count
|
||||||
|
self._conn.execute(
|
||||||
|
"UPDATE memories SET retrieval_count = retrieval_count + 1 WHERE memory_id = ?",
|
||||||
|
(mid,),
|
||||||
|
)
|
||||||
|
results.append({
|
||||||
|
"memory_id": mid,
|
||||||
|
"content": m["content"],
|
||||||
|
"room": m["room"],
|
||||||
|
"category": m.get("category", ""),
|
||||||
|
"trust_score": m["trust_score"],
|
||||||
|
"score": round(score, 6),
|
||||||
|
})
|
||||||
|
|
||||||
|
if results:
|
||||||
|
self._conn.commit()
|
||||||
|
return results
|
||||||
|
|
||||||
|
def _fts_search(
|
||||||
|
self, query: str, room: Optional[str], min_trust: float, limit: int
|
||||||
|
) -> list[dict]:
|
||||||
|
"""FTS5 full-text search."""
|
||||||
|
try:
|
||||||
|
if room:
|
||||||
|
rows = self._conn.execute(
|
||||||
|
"""SELECT m.memory_id, m.content, m.room, m.category,
|
||||||
|
m.trust_score, m.retrieval_count
|
||||||
|
FROM memories_fts f
|
||||||
|
JOIN memories m ON f.rowid = m.memory_id
|
||||||
|
WHERE memories_fts MATCH ? AND m.room = ?
|
||||||
|
AND m.trust_score >= ?
|
||||||
|
ORDER BY rank LIMIT ?""",
|
||||||
|
(query, room, min_trust, limit),
|
||||||
|
).fetchall()
|
||||||
|
else:
|
||||||
|
rows = self._conn.execute(
|
||||||
|
"""SELECT m.memory_id, m.content, m.room, m.category,
|
||||||
|
m.trust_score, m.retrieval_count
|
||||||
|
FROM memories_fts f
|
||||||
|
JOIN memories m ON f.rowid = m.memory_id
|
||||||
|
WHERE memories_fts MATCH ?
|
||||||
|
AND m.trust_score >= ?
|
||||||
|
ORDER BY rank LIMIT ?""",
|
||||||
|
(query, min_trust, limit),
|
||||||
|
).fetchall()
|
||||||
|
return [dict(r) for r in rows]
|
||||||
|
except sqlite3.OperationalError:
|
||||||
|
# Bad FTS query syntax — degrade gracefully
|
||||||
|
return []
|
||||||
|
|
||||||
|
def _hrr_search(
|
||||||
|
self, query: str, room: Optional[str], min_trust: float, limit: int
|
||||||
|
) -> list[dict]:
|
||||||
|
"""HRR cosine similarity search (brute-force scan, fast for <100K facts)."""
|
||||||
|
query_vec = encode_text(query)
|
||||||
|
|
||||||
|
if room:
|
||||||
|
rows = self._conn.execute(
|
||||||
|
"""SELECT memory_id, content, room, category, trust_score,
|
||||||
|
retrieval_count, hrr_vector
|
||||||
|
FROM memories
|
||||||
|
WHERE room = ? AND trust_score >= ? AND hrr_vector IS NOT NULL""",
|
||||||
|
(room, min_trust),
|
||||||
|
).fetchall()
|
||||||
|
else:
|
||||||
|
rows = self._conn.execute(
|
||||||
|
"""SELECT memory_id, content, room, category, trust_score,
|
||||||
|
retrieval_count, hrr_vector
|
||||||
|
FROM memories
|
||||||
|
WHERE trust_score >= ? AND hrr_vector IS NOT NULL""",
|
||||||
|
(min_trust,),
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
|
scored = []
|
||||||
|
for r in rows:
|
||||||
|
stored_vec = deserialize_vector(r["hrr_vector"])
|
||||||
|
sim = cosine_similarity_phase(query_vec, stored_vec)
|
||||||
|
scored.append((sim, dict(r)))
|
||||||
|
|
||||||
|
scored.sort(key=lambda x: x[0], reverse=True)
|
||||||
|
return [item[1] for item in scored[:limit]]
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Trust management
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def boost_trust(self, memory_id: int, delta: float = 0.05) -> None:
|
||||||
|
"""Increase trust score when a memory proves useful."""
|
||||||
|
self._conn.execute(
|
||||||
|
"""UPDATE memories SET trust_score = MIN(1.0, trust_score + ?),
|
||||||
|
updated_at = ? WHERE memory_id = ?""",
|
||||||
|
(delta, time.time(), memory_id),
|
||||||
|
)
|
||||||
|
self._conn.commit()
|
||||||
|
|
||||||
|
def decay_trust(self, memory_id: int, delta: float = 0.02) -> None:
|
||||||
|
"""Decrease trust score when a memory is contradicted."""
|
||||||
|
self._conn.execute(
|
||||||
|
"""UPDATE memories SET trust_score = MAX(0.0, trust_score - ?),
|
||||||
|
updated_at = ? WHERE memory_id = ?""",
|
||||||
|
(delta, time.time(), memory_id),
|
||||||
|
)
|
||||||
|
self._conn.commit()
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Room operations
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def list_rooms(self) -> list[dict]:
|
||||||
|
"""List all rooms with fact counts."""
|
||||||
|
rows = self._conn.execute(
|
||||||
|
"""SELECT room, COUNT(*) as count,
|
||||||
|
AVG(trust_score) as avg_trust
|
||||||
|
FROM memories GROUP BY room ORDER BY count DESC"""
|
||||||
|
).fetchall()
|
||||||
|
return [dict(r) for r in rows]
|
||||||
|
|
||||||
|
def room_contents(self, room: str, limit: int = 50) -> list[dict]:
|
||||||
|
"""Get all facts in a room, ordered by trust."""
|
||||||
|
rows = self._conn.execute(
|
||||||
|
"""SELECT memory_id, content, category, trust_score,
|
||||||
|
retrieval_count, created_at
|
||||||
|
FROM memories WHERE room = ?
|
||||||
|
ORDER BY trust_score DESC, created_at DESC LIMIT ?""",
|
||||||
|
(room, limit),
|
||||||
|
).fetchall()
|
||||||
|
return [dict(r) for r in rows]
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Stats
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def stats(self) -> dict:
|
||||||
|
"""Return store statistics."""
|
||||||
|
row = self._conn.execute(
|
||||||
|
"""SELECT COUNT(*) as total,
|
||||||
|
AVG(trust_score) as avg_trust,
|
||||||
|
SUM(retrieval_count) as total_retrievals,
|
||||||
|
COUNT(DISTINCT room) as room_count
|
||||||
|
FROM memories"""
|
||||||
|
).fetchone()
|
||||||
|
return dict(row)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Promotion support (scratchpad → durable)
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def log_promotion(
|
||||||
|
self,
|
||||||
|
session_id: str,
|
||||||
|
scratch_key: str,
|
||||||
|
memory_id: int,
|
||||||
|
reason: str = "",
|
||||||
|
) -> None:
|
||||||
|
"""Record a scratchpad-to-palace promotion in the audit log."""
|
||||||
|
self._conn.execute(
|
||||||
|
"""INSERT INTO promotion_log
|
||||||
|
(session_id, scratch_key, memory_id, promoted_at, reason)
|
||||||
|
VALUES (?, ?, ?, ?, ?)""",
|
||||||
|
(session_id, scratch_key, memory_id, time.time(), reason),
|
||||||
|
)
|
||||||
|
self._conn.commit()
|
||||||
|
|
||||||
|
def recent_promotions(self, limit: int = 20) -> list[dict]:
|
||||||
|
"""Get recent promotion log entries."""
|
||||||
|
rows = self._conn.execute(
|
||||||
|
"""SELECT p.*, m.content, m.room
|
||||||
|
FROM promotion_log p
|
||||||
|
LEFT JOIN memories m ON p.memory_id = m.memory_id
|
||||||
|
ORDER BY p.promoted_at DESC LIMIT ?""",
|
||||||
|
(limit,),
|
||||||
|
).fetchall()
|
||||||
|
return [dict(r) for r in rows]
|
||||||
0
hermes-sovereign/mempalace/tests/__init__.py
Normal file
180
hermes-sovereign/mempalace/tests/test_mempalace.py
Normal file
@@ -0,0 +1,180 @@
|
|||||||
|
"""Tests for the mempalace skill.
|
||||||
|
|
||||||
|
Validates PalaceRoom, Mempalace class, factory constructors,
|
||||||
|
and the analyse_issues entry-point.
|
||||||
|
|
||||||
|
Refs: Epic #367, Sub-issue #368
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
# Ensure the package is importable from the repo layout
|
||||||
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
|
||||||
|
|
||||||
|
from mempalace.mempalace import Mempalace, PalaceRoom, analyse_issues
|
||||||
|
|
||||||
|
|
||||||
|
# ── PalaceRoom unit tests ─────────────────────────────────────────────────
|
||||||
|
|
||||||
|
class TestPalaceRoom:
|
||||||
|
def test_store_and_retrieve(self):
|
||||||
|
room = PalaceRoom(name="test", label="Test Room")
|
||||||
|
room.store("key1", 42)
|
||||||
|
assert room.retrieve("key1") == 42
|
||||||
|
|
||||||
|
def test_retrieve_default(self):
|
||||||
|
room = PalaceRoom(name="test", label="Test Room")
|
||||||
|
assert room.retrieve("missing") is None
|
||||||
|
assert room.retrieve("missing", "fallback") == "fallback"
|
||||||
|
|
||||||
|
def test_summary_format(self):
|
||||||
|
room = PalaceRoom(name="test", label="Test Room")
|
||||||
|
room.store("repos", 5)
|
||||||
|
summary = room.summary()
|
||||||
|
assert "## Test Room" in summary
|
||||||
|
assert "repos: 5" in summary
|
||||||
|
|
||||||
|
def test_contents_default_factory_isolation(self):
|
||||||
|
"""Each room gets its own dict — no shared mutable default."""
|
||||||
|
r1 = PalaceRoom(name="a", label="A")
|
||||||
|
r2 = PalaceRoom(name="b", label="B")
|
||||||
|
r1.store("x", 1)
|
||||||
|
assert r2.retrieve("x") is None
|
||||||
|
|
||||||
|
def test_entered_at_is_recent(self):
|
||||||
|
before = time.time()
|
||||||
|
room = PalaceRoom(name="t", label="T")
|
||||||
|
after = time.time()
|
||||||
|
assert before <= room.entered_at <= after
|
||||||
|
|
||||||
|
|
||||||
|
# ── Mempalace core tests ──────────────────────────────────────────────────
|
||||||
|
|
||||||
|
class TestMempalace:
|
||||||
|
def test_add_and_enter_room(self):
|
||||||
|
p = Mempalace(domain="test")
|
||||||
|
p.add_room("r1", "Room 1")
|
||||||
|
room = p.enter("r1")
|
||||||
|
assert room.name == "r1"
|
||||||
|
|
||||||
|
def test_enter_nonexistent_room_raises(self):
|
||||||
|
p = Mempalace()
|
||||||
|
with pytest.raises(KeyError, match="No room"):
|
||||||
|
p.enter("ghost")
|
||||||
|
|
||||||
|
def test_store_without_enter_raises(self):
|
||||||
|
p = Mempalace()
|
||||||
|
p.add_room("r", "R")
|
||||||
|
with pytest.raises(RuntimeError, match="Enter a room"):
|
||||||
|
p.store("k", "v")
|
||||||
|
|
||||||
|
def test_store_and_retrieve_via_palace(self):
|
||||||
|
p = Mempalace()
|
||||||
|
p.add_room("r", "R")
|
||||||
|
p.enter("r")
|
||||||
|
p.store("count", 10)
|
||||||
|
assert p.retrieve("r", "count") == 10
|
||||||
|
|
||||||
|
def test_retrieve_missing_room_returns_default(self):
|
||||||
|
p = Mempalace()
|
||||||
|
assert p.retrieve("nope", "key") is None
|
||||||
|
assert p.retrieve("nope", "key", 99) == 99
|
||||||
|
|
||||||
|
def test_render_includes_domain(self):
|
||||||
|
p = Mempalace(domain="audit")
|
||||||
|
p.add_room("r", "Room")
|
||||||
|
p.enter("r")
|
||||||
|
p.store("item", "value")
|
||||||
|
output = p.render()
|
||||||
|
assert "audit" in output
|
||||||
|
assert "Room" in output
|
||||||
|
|
||||||
|
def test_to_dict_structure(self):
|
||||||
|
p = Mempalace(domain="test")
|
||||||
|
p.add_room("r", "R")
|
||||||
|
p.enter("r")
|
||||||
|
p.store("a", 1)
|
||||||
|
d = p.to_dict()
|
||||||
|
assert d["domain"] == "test"
|
||||||
|
assert "elapsed_seconds" in d
|
||||||
|
assert d["rooms"]["r"] == {"a": 1}
|
||||||
|
|
||||||
|
def test_to_json_is_valid(self):
|
||||||
|
p = Mempalace(domain="j")
|
||||||
|
p.add_room("x", "X")
|
||||||
|
p.enter("x")
|
||||||
|
p.store("v", [1, 2, 3])
|
||||||
|
parsed = json.loads(p.to_json())
|
||||||
|
assert parsed["rooms"]["x"]["v"] == [1, 2, 3]
|
||||||
|
|
||||||
|
|
||||||
|
# ── Factory constructor tests ─────────────────────────────────────────────
|
||||||
|
|
||||||
|
class TestFactories:
|
||||||
|
def test_for_issue_analysis_rooms(self):
|
||||||
|
p = Mempalace.for_issue_analysis()
|
||||||
|
assert p.domain == "issue_analysis"
|
||||||
|
for key in ("repo_architecture", "assignment_status",
|
||||||
|
"triage_priority", "resolution_patterns"):
|
||||||
|
p.enter(key) # should not raise
|
||||||
|
|
||||||
|
def test_for_health_check_rooms(self):
|
||||||
|
p = Mempalace.for_health_check()
|
||||||
|
assert p.domain == "health_check"
|
||||||
|
for key in ("service_topology", "failure_signals", "recovery_history"):
|
||||||
|
p.enter(key)
|
||||||
|
|
||||||
|
def test_for_code_review_rooms(self):
|
||||||
|
p = Mempalace.for_code_review()
|
||||||
|
assert p.domain == "code_review"
|
||||||
|
for key in ("change_scope", "risk_surface",
|
||||||
|
"test_coverage", "reviewer_context"):
|
||||||
|
p.enter(key)
|
||||||
|
|
||||||
|
|
||||||
|
# ── analyse_issues entry-point tests ──────────────────────────────────────
|
||||||
|
|
||||||
|
class TestAnalyseIssues:
|
||||||
|
SAMPLE_DATA = [
|
||||||
|
{"repo": "the-nexus", "open_issues": 40, "assigned": 30, "unassigned": 10},
|
||||||
|
{"repo": "timmy-home", "open_issues": 30, "assigned": 25, "unassigned": 5},
|
||||||
|
{"repo": "hermes-agent", "open_issues": 20, "assigned": 15, "unassigned": 5},
|
||||||
|
{"repo": "empty-repo", "open_issues": 0, "assigned": 0, "unassigned": 0},
|
||||||
|
]
|
||||||
|
|
||||||
|
def test_returns_string(self):
|
||||||
|
result = analyse_issues(self.SAMPLE_DATA)
|
||||||
|
assert isinstance(result, str)
|
||||||
|
assert len(result) > 0
|
||||||
|
|
||||||
|
def test_contains_room_headers(self):
|
||||||
|
result = analyse_issues(self.SAMPLE_DATA)
|
||||||
|
assert "Repository Architecture" in result
|
||||||
|
assert "Assignment Status" in result
|
||||||
|
|
||||||
|
def test_coverage_below_target(self):
|
||||||
|
result = analyse_issues(self.SAMPLE_DATA, target_assignee_rate=0.90)
|
||||||
|
assert "BELOW TARGET" in result
|
||||||
|
|
||||||
|
def test_coverage_meets_target(self):
|
||||||
|
good_data = [
|
||||||
|
{"repo": "a", "open_issues": 10, "assigned": 10, "unassigned": 0},
|
||||||
|
]
|
||||||
|
result = analyse_issues(good_data, target_assignee_rate=0.80)
|
||||||
|
assert "OK" in result
|
||||||
|
|
||||||
|
def test_empty_repos_list(self):
|
||||||
|
result = analyse_issues([])
|
||||||
|
assert isinstance(result, str)
|
||||||
|
|
||||||
|
def test_single_repo(self):
|
||||||
|
data = [{"repo": "solo", "open_issues": 5, "assigned": 3, "unassigned": 2}]
|
||||||
|
result = analyse_issues(data)
|
||||||
|
assert "solo" in result or "issue_analysis" in result
|
||||||
143
hermes-sovereign/mempalace/tests/test_retrieval_enforcer.py
Normal file
@@ -0,0 +1,143 @@
|
|||||||
|
"""Tests for retrieval_enforcer.py.
|
||||||
|
|
||||||
|
Refs: Epic #367, Sub-issue #369
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import patch, MagicMock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
|
||||||
|
|
||||||
|
from mempalace.retrieval_enforcer import (
|
||||||
|
is_recall_query,
|
||||||
|
load_identity,
|
||||||
|
load_scratchpad,
|
||||||
|
enforce_retrieval_order,
|
||||||
|
search_skills,
|
||||||
|
RECALL_PATTERNS,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class TestRecallDetection:
|
||||||
|
"""Test the recall-query pattern matcher."""
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("query", [
|
||||||
|
"what did we work on yesterday",
|
||||||
|
"status of the mempalace integration",
|
||||||
|
"remember the fleet audit results",
|
||||||
|
"last time we deployed the nexus",
|
||||||
|
"previously you mentioned a CI fix",
|
||||||
|
"we discussed the sovereign deployment",
|
||||||
|
])
|
||||||
|
def test_recall_queries_detected(self, query):
|
||||||
|
assert is_recall_query(query) is True
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("query", [
|
||||||
|
"create a new file called test.py",
|
||||||
|
"run the test suite",
|
||||||
|
"deploy to production",
|
||||||
|
"write a function that sums numbers",
|
||||||
|
"install the package",
|
||||||
|
])
|
||||||
|
def test_non_recall_queries_skipped(self, query):
|
||||||
|
assert is_recall_query(query) is False
|
||||||
|
|
||||||
|
|
||||||
|
class TestLoadIdentity:
|
||||||
|
def test_loads_existing_identity(self, tmp_path):
|
||||||
|
identity_file = tmp_path / "identity.txt"
|
||||||
|
identity_file.write_text("I am Timmy. A sovereign AI.")
|
||||||
|
with patch("mempalace.retrieval_enforcer.IDENTITY_PATH", identity_file):
|
||||||
|
result = load_identity()
|
||||||
|
assert "Timmy" in result
|
||||||
|
|
||||||
|
def test_returns_empty_on_missing_file(self, tmp_path):
|
||||||
|
identity_file = tmp_path / "nonexistent.txt"
|
||||||
|
with patch("mempalace.retrieval_enforcer.IDENTITY_PATH", identity_file):
|
||||||
|
result = load_identity()
|
||||||
|
assert result == ""
|
||||||
|
|
||||||
|
def test_truncates_long_identity(self, tmp_path):
|
||||||
|
identity_file = tmp_path / "identity.txt"
|
||||||
|
identity_file.write_text(" ".join(["word"] * 300))
|
||||||
|
with patch("mempalace.retrieval_enforcer.IDENTITY_PATH", identity_file):
|
||||||
|
result = load_identity()
|
||||||
|
assert result.endswith("...")
|
||||||
|
assert len(result.split()) <= 201 # 200 words + "..."
|
||||||
|
|
||||||
|
|
||||||
|
class TestLoadScratchpad:
|
||||||
|
def test_loads_valid_scratchpad(self, tmp_path):
|
||||||
|
scratch_file = tmp_path / "session123.json"
|
||||||
|
scratch_file.write_text(json.dumps({"note": "test value", "key2": 42}))
|
||||||
|
with patch("mempalace.retrieval_enforcer.SCRATCHPAD_DIR", tmp_path):
|
||||||
|
result = load_scratchpad("session123")
|
||||||
|
assert "note: test value" in result
|
||||||
|
assert "key2: 42" in result
|
||||||
|
|
||||||
|
def test_returns_empty_on_missing_file(self, tmp_path):
|
||||||
|
with patch("mempalace.retrieval_enforcer.SCRATCHPAD_DIR", tmp_path):
|
||||||
|
result = load_scratchpad("nonexistent")
|
||||||
|
assert result == ""
|
||||||
|
|
||||||
|
def test_returns_empty_on_invalid_json(self, tmp_path):
|
||||||
|
scratch_file = tmp_path / "bad.json"
|
||||||
|
scratch_file.write_text("not valid json{{{")
|
||||||
|
with patch("mempalace.retrieval_enforcer.SCRATCHPAD_DIR", tmp_path):
|
||||||
|
result = load_scratchpad("bad")
|
||||||
|
assert result == ""
|
||||||
|
|
||||||
|
|
||||||
|
class TestEnforceRetrievalOrder:
|
||||||
|
def test_skips_non_recall_query(self):
|
||||||
|
result = enforce_retrieval_order("create a new file")
|
||||||
|
assert result["retrieved_from"] is None
|
||||||
|
assert result["tokens"] == 0
|
||||||
|
|
||||||
|
def test_runs_for_recall_query(self, tmp_path):
|
||||||
|
identity_file = tmp_path / "identity.txt"
|
||||||
|
identity_file.write_text("I am Timmy.")
|
||||||
|
with patch("mempalace.retrieval_enforcer.IDENTITY_PATH", identity_file), \
|
||||||
|
patch("mempalace.retrieval_enforcer.search_palace", return_value=""), \
|
||||||
|
patch("mempalace.retrieval_enforcer.search_gitea", return_value=""), \
|
||||||
|
patch("mempalace.retrieval_enforcer.search_skills", return_value=""):
|
||||||
|
result = enforce_retrieval_order("what did we work on yesterday")
|
||||||
|
assert "Identity" in result["context"]
|
||||||
|
assert "L0" in result["layers_checked"]
|
||||||
|
|
||||||
|
def test_palace_hit_sets_l1(self, tmp_path):
|
||||||
|
identity_file = tmp_path / "identity.txt"
|
||||||
|
identity_file.write_text("I am Timmy.")
|
||||||
|
with patch("mempalace.retrieval_enforcer.IDENTITY_PATH", identity_file), \
|
||||||
|
patch("mempalace.retrieval_enforcer.search_palace", return_value="Found: fleet audit results"), \
|
||||||
|
patch("mempalace.retrieval_enforcer.search_gitea", return_value=""):
|
||||||
|
result = enforce_retrieval_order("what did we discuss yesterday")
|
||||||
|
assert result["retrieved_from"] == "L1"
|
||||||
|
assert "Palace Memory" in result["context"]
|
||||||
|
|
||||||
|
def test_falls_through_to_l5(self, tmp_path):
|
||||||
|
identity_file = tmp_path / "nonexistent.txt"
|
||||||
|
with patch("mempalace.retrieval_enforcer.IDENTITY_PATH", identity_file), \
|
||||||
|
patch("mempalace.retrieval_enforcer.search_palace", return_value=""), \
|
||||||
|
patch("mempalace.retrieval_enforcer.search_gitea", return_value=""), \
|
||||||
|
patch("mempalace.retrieval_enforcer.search_skills", return_value=""):
|
||||||
|
result = enforce_retrieval_order("remember the old deployment", skip_if_not_recall=True)
|
||||||
|
assert result["retrieved_from"] == "L5"
|
||||||
|
|
||||||
|
def test_force_mode_skips_recall_check(self, tmp_path):
|
||||||
|
identity_file = tmp_path / "identity.txt"
|
||||||
|
identity_file.write_text("I am Timmy.")
|
||||||
|
with patch("mempalace.retrieval_enforcer.IDENTITY_PATH", identity_file), \
|
||||||
|
patch("mempalace.retrieval_enforcer.search_palace", return_value=""), \
|
||||||
|
patch("mempalace.retrieval_enforcer.search_gitea", return_value=""), \
|
||||||
|
patch("mempalace.retrieval_enforcer.search_skills", return_value=""):
|
||||||
|
result = enforce_retrieval_order("deploy now", skip_if_not_recall=False)
|
||||||
|
assert "Identity" in result["context"]
|
||||||