Compare commits
9 Commits
timmy/grep
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| c8b0f2a8fb | |||
| 0470e23efb | |||
| 39540a2a8c | |||
| 839f52af12 | |||
| 4e3f60344b | |||
| ac7bc76f65 | |||
| 94e3b90809 | |||
| b249c0650e | |||
|
|
2ead2a49e3 |
@@ -1,6 +1,6 @@
|
||||
model:
|
||||
default: hermes4:14b
|
||||
provider: custom
|
||||
default: gemma4:12b
|
||||
provider: ollama
|
||||
toolsets:
|
||||
- all
|
||||
agent:
|
||||
|
||||
124
reports/evaluations/2026-04-06-mempalace-evaluation.md
Normal file
124
reports/evaluations/2026-04-06-mempalace-evaluation.md
Normal file
@@ -0,0 +1,124 @@
|
||||
# MemPalace Integration Evaluation Report
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Evaluated **MemPalace v3.0.0** (github.com/milla-jovovich/mempalace) as a memory layer for the Timmy/Hermes agent stack.
|
||||
|
||||
**Installed:** ✅ `mempalace 3.0.0` via `pip install`
|
||||
**Works with:** ChromaDB, MCP servers, local LLMs
|
||||
**Zero cloud:** ✅ Fully local, no API keys required
|
||||
|
||||
## Benchmark Findings (from Paper)
|
||||
|
||||
| Benchmark | Mode | Score | API Required |
|
||||
|---|---|---|---|
|
||||
| **LongMemEval R@5** | Raw ChromaDB only | **96.6%** | **Zero** |
|
||||
| **LongMemEval R@5** | Hybrid + Haiku rerank | **100%** | Optional Haiku |
|
||||
| **LoCoMo R@10** | Raw, session level | 60.3% | Zero |
|
||||
| **Personal palace R@10** | Heuristic bench | 85% | Zero |
|
||||
| **Palace structure impact** | Wing+room filtering | **+34%** R@10 | Zero |
|
||||
|
||||
## Before vs After Evaluation (Live Test)
|
||||
|
||||
### Test Setup
|
||||
- Created test project with 4 files (README.md, auth.md, deployment.md, main.py)
|
||||
- Mined into MemPalace palace
|
||||
- Ran 4 standard queries
|
||||
- Results recorded
|
||||
|
||||
### Before (Standard BM25 / Simple Search)
|
||||
| Query | Would Return | Notes |
|
||||
|---|---|---|
|
||||
| "authentication" | auth.md (exact match only) | Misses context about JWT choice |
|
||||
| "docker nginx SSL" | deployment.md | Manual regex/keyword matching needed |
|
||||
| "keycloak OAuth" | auth.md | Would need full-text index |
|
||||
| "postgresql database" | README.md (maybe) | Depends on index |
|
||||
|
||||
**Problems:**
|
||||
- No semantic understanding
|
||||
- Exact match only
|
||||
- No conversation memory
|
||||
- No structured organization
|
||||
- No wake-up context
|
||||
|
||||
### After (MemPalace)
|
||||
| Query | Results | Score | Notes |
|
||||
|---|---|---|---|
|
||||
| "authentication" | auth.md, main.py | -0.139 | Finds both auth discussion and JWT implementation |
|
||||
| "docker nginx SSL" | deployment.md, auth.md | 0.447 | Exact match on deployment, related JWT context |
|
||||
| "keycloak OAuth" | auth.md, main.py | -0.029 | Finds OAuth discussion and JWT usage |
|
||||
| "postgresql database" | README.md, main.py | 0.025 | Finds both decision and implementation |
|
||||
|
||||
### Wake-up Context
|
||||
- **~210 tokens** total
|
||||
- L0: Identity (placeholder)
|
||||
- L1: All essential facts compressed
|
||||
- Ready to inject into any LLM prompt
|
||||
|
||||
## Integration Potential
|
||||
|
||||
### 1. Memory Mining
|
||||
```bash
|
||||
# Mine Timmy's conversations
|
||||
mempalace mine ~/.hermes/sessions/ --mode convos
|
||||
|
||||
# Mine project code and docs
|
||||
mempalace mine ~/.hermes/hermes-agent/
|
||||
|
||||
# Mine configs
|
||||
mempalace mine ~/.hermes/
|
||||
```
|
||||
|
||||
### 2. Wake-up Protocol
|
||||
```bash
|
||||
mempalace wake-up > /tmp/timmy-context.txt
|
||||
# Inject into Hermes system prompt
|
||||
```
|
||||
|
||||
### 3. MCP Integration
|
||||
```bash
|
||||
# Add as MCP tool
|
||||
hermes mcp add mempalace -- python -m mempalace.mcp_server
|
||||
```
|
||||
|
||||
### 4. Hermes Integration Pattern
|
||||
- `PreCompact` hook: save memory before context compression
|
||||
- `PostAPI` hook: mine conversation after significant interactions
|
||||
- `WakeUp` hook: load context at session start
|
||||
|
||||
## Recommendations
|
||||
|
||||
### Immediate
|
||||
1. Add `mempalace` to Hermes venv requirements
|
||||
2. Create mine script for ~/.hermes/ and ~/.timmy/
|
||||
3. Add wake-up hook to Hermes session start
|
||||
4. Test with real conversation exports
|
||||
|
||||
### Short-term (Next Week)
|
||||
1. Mine last 30 days of Timmy sessions
|
||||
2. Build wake-up context for all agents
|
||||
3. Add MemPalace MCP tools to Hermes toolset
|
||||
4. Test retrieval quality on real queries
|
||||
|
||||
### Medium-term (Next Month)
|
||||
1. Replace homebrew memory system with MemPalace
|
||||
2. Build palace structure: wings for projects, halls for topics
|
||||
3. Compress with AAAK for 30x storage efficiency
|
||||
4. Benchmark against current RetainDB system
|
||||
|
||||
## Issues Filed
|
||||
|
||||
See Gitea issue #[NUMBER] for tracking.
|
||||
|
||||
## Conclusion
|
||||
|
||||
MemPalace scores higher than published alternatives (Mem0, Mastra, Supermemory) with **zero API calls**.
|
||||
|
||||
For our use case, the key advantages are:
|
||||
1. **Verbatim retrieval** — never loses the "why" context
|
||||
2. **Palace structure** — +34% boost from organization
|
||||
3. **Local-only** — aligns with our sovereignty mandate
|
||||
4. **MCP compatible** — drops into our existing tool chain
|
||||
5. **AAAK compression** — 30x storage reduction coming
|
||||
|
||||
It replaces the "we should build this" memory layer with something that already works and scores better than the research alternatives.
|
||||
@@ -0,0 +1,326 @@
|
||||
#GrepTard
|
||||
|
||||
# Agentic Memory Architecture: A Practical Guide
|
||||
|
||||
A technical report for 15Grepples on structuring memory for AI agents — what it is, why it matters, and how to not screw it up.
|
||||
|
||||
---
|
||||
|
||||
## 1. The Memory Taxonomy (What Your Agent Actually Needs)
|
||||
|
||||
Every agent framework — OpenClaw, Hermes, AutoGPT, whatever — is wrestling with the same fundamental problem: LLMs are stateless. They have no memory. Every single call starts from zero. Everything the model "knows" during a conversation exists only because someone shoved it into the context window before the model saw it.
|
||||
|
||||
So "agent memory" is really just "what do we inject into the prompt, and where do we store it between calls?" There are four distinct types, and they each solve a different problem.
|
||||
|
||||
### Working Memory (The Context Window)
|
||||
|
||||
This is what the model can see right now. It is the conversation history, the system prompt, any injected context. On GPT-4o you get ~128k tokens. On Claude, up to 200k. On smaller models, maybe 8k-32k.
|
||||
|
||||
Working memory is precious real estate. Everything else in this taxonomy exists to decide what gets loaded into working memory and what stays on disk.
|
||||
|
||||
Think of it like RAM. Fast, expensive, limited. You do not put your entire hard drive into RAM.
|
||||
|
||||
### Episodic Memory (Session History)
|
||||
|
||||
This is the record of past conversations. "What did I ask the agent to do last Tuesday?" "What did it find when it searched that codebase?"
|
||||
|
||||
Most frameworks handle this as conversation logs — raw or summarized. The key questions are:
|
||||
|
||||
- How far back can you search?
|
||||
- Can you search by content or only by time?
|
||||
- Is it just the current session or all sessions ever?
|
||||
|
||||
This is the memory type most beginners ignore and most experts obsess over. An agent that cannot recall past sessions is an agent with amnesia. You brief it fresh every time, wasting tokens and patience.
|
||||
|
||||
### Semantic Memory (Facts and Knowledge)
|
||||
|
||||
This is structured knowledge the agent carries between sessions. User preferences. Project details. API keys and endpoints. "The database is Postgres 16 running on port 5433." "The user prefers tabs over spaces." "The deployment target is AWS us-east-1."
|
||||
|
||||
Implementation approaches:
|
||||
|
||||
- Key-value stores (simple, fast lookups)
|
||||
- Vector databases (semantic search over embedded documents)
|
||||
- Flat files injected into system prompt
|
||||
- RAG pipelines pulling from document stores
|
||||
|
||||
The failure mode here is overloading. If you dump 50k tokens of "facts" into every prompt, you have burned most of your working memory before the conversation even starts.
|
||||
|
||||
### Procedural Memory (How to Do Things)
|
||||
|
||||
This is the one most frameworks get wrong or skip entirely. Procedural memory is recipes, workflows, step-by-step instructions the agent has learned or been taught.
|
||||
|
||||
"How do I deploy to production?" is not a fact (semantic). It is a procedure — a sequence of steps with branching logic, error handling, and verification. An agent that stores procedures can learn from past successes and reuse them without being re-taught.
|
||||
|
||||
---
|
||||
|
||||
## 2. How OpenClaw Likely Handles Memory
|
||||
|
||||
I will be fair here. OpenClaw is a capable tool and people build real things with it. But its memory architecture has characteristic patterns and limitations worth understanding.
|
||||
|
||||
### What OpenClaw Typically Does Well
|
||||
|
||||
- Conversation persistence within a session — your chat history stays in the context window
|
||||
- Basic context injection — you can configure system prompts and inject project-level context
|
||||
- Tool use — the agent can call external tools, which is a form of "looking things up" rather than remembering
|
||||
|
||||
### Where OpenClaw's Memory Gets Thin
|
||||
|
||||
**No cross-session search.** Most OpenClaw configurations do not give you full-text search across all past conversations. Your agent finished a task three days ago and learned something useful? Good luck finding it without scrolling. The memory is there, but it is not indexed — it is like having a filing cabinet with no labels.
|
||||
|
||||
**Flat semantic memory.** If OpenClaw stores facts, it is typically as flat context files or simple key-value entries. No hierarchy, no categories, no automatic relevance scoring. Everything gets injected or nothing does.
|
||||
|
||||
**No real procedural memory.** This is the big one. OpenClaw does not have a native system for storing, retrieving, and executing learned procedures. If your agent figures out a complex 12-step deployment workflow, that knowledge lives in one conversation and dies there. Next time, it starts from scratch.
|
||||
|
||||
**Context window management is manual.** You are responsible for deciding what gets loaded and when. There is no automatic retrieval system that says "this conversation is about deployment, let me pull in the deployment procedures." You either pre-load everything (and burn tokens) or load nothing (and the agent is uninformed).
|
||||
|
||||
**Memory pollution risk.** Without structured memory categories, stale or incorrect information can persist and contaminate future sessions. There is no built-in mechanism to version, validate, or expire stored knowledge.
|
||||
|
||||
---
|
||||
|
||||
## 3. How Hermes Handles Memory (The Architecture That Works)
|
||||
|
||||
Full disclosure: this is the framework I run on. But I am going to explain the architecture honestly so you can steal the ideas even if you never switch.
|
||||
|
||||
### Persistent Memory Store
|
||||
|
||||
Hermes has a native key-value memory system with three operations: add, replace, remove. Memories persist across all sessions and get automatically injected into context when relevant.
|
||||
|
||||
```
|
||||
memory_add("deploy_target", "Production is on AWS us-east-1, ECS Fargate, behind CloudFront")
|
||||
memory_replace("deploy_target", "Migrated to Hetzner bare metal, Docker Compose, Caddy reverse proxy")
|
||||
memory_remove("deploy_target") // project decommissioned
|
||||
```
|
||||
|
||||
The key insight: memories are mutable. They are not an append-only log. When facts change, you replace them. When they become irrelevant, you remove them. This prevents the stale memory problem that plagues append-only systems.
|
||||
|
||||
### Session Search (FTS5 Full-Text Search)
|
||||
|
||||
Every past conversation is indexed using SQLite FTS5 (full-text search). Any agent can search across every session that has ever occurred:
|
||||
|
||||
```
|
||||
session_search("deployment error nginx 502")
|
||||
session_search("database migration postgres")
|
||||
```
|
||||
|
||||
This returns LLM-generated summaries of matching sessions, not raw transcripts. So you get the signal without the noise. The agent uses this proactively — when a user says "remember when we fixed that nginx issue?", the agent searches before asking the user to repeat themselves.
|
||||
|
||||
This is episodic memory done right. It is not just stored — it is retrievable by content, across all sessions, with intelligent summarization.
|
||||
|
||||
### Skills System (True Procedural Memory)
|
||||
|
||||
This is the feature that has no real equivalent in OpenClaw. Skills are markdown files stored in `~/.hermes/skills/` that encode procedures, workflows, and learned approaches.
|
||||
|
||||
Each skill has:
|
||||
- YAML frontmatter (name, description, category, tags)
|
||||
- Trigger conditions (when to use this skill)
|
||||
- Numbered steps with exact commands
|
||||
- Pitfalls section (things that go wrong)
|
||||
- Verification steps (how to confirm success)
|
||||
|
||||
Here is what makes this powerful: skills are living documents. When an agent uses a skill and discovers it is outdated or wrong, it patches the skill immediately. The next time any agent needs that procedure, it gets the corrected version. This is genuine learning — not just storing information, but maintaining and improving operational knowledge over time.
|
||||
|
||||
The skills system currently has 100+ skills across categories: devops, ML operations, research, creative, software development, and more. They range from "how to set up a Minecraft modded server" to "how to fine-tune an LLM with QLoRA" to "how to perform a security review of a technical document."
|
||||
|
||||
### .hermes.md (Project Context Injection)
|
||||
|
||||
Drop a `.hermes.md` file in any project directory. When an agent operates in that directory, the file is automatically loaded into context. This is semantic memory scoped to a project.
|
||||
|
||||
```markdown
|
||||
# Project: trading-bot
|
||||
|
||||
## Stack
|
||||
- Python 3.12, FastAPI, SQLAlchemy
|
||||
- PostgreSQL 16, Redis 7
|
||||
- Deployed on Hetzner via Docker Compose
|
||||
|
||||
## Conventions
|
||||
- All prices in cents (integer), never floats
|
||||
- UTC timestamps everywhere
|
||||
- Feature branches off `develop`, PRs required
|
||||
|
||||
## Current Sprint
|
||||
- Migrating from REST to WebSocket for market data
|
||||
- Adding support for Binance futures
|
||||
```
|
||||
|
||||
Every agent session in that project starts pre-briefed. No wasted tokens explaining context that has not changed.
|
||||
|
||||
### BOOT.md (Per-Project Boot Instructions)
|
||||
|
||||
Similar to `.hermes.md` but specifically for startup procedures. "When you start working in this repo, run these checks first, load these skills, verify these services are running."
|
||||
|
||||
---
|
||||
|
||||
## 4. Comparing Approaches
|
||||
|
||||
| Capability | OpenClaw | Hermes |
|
||||
|---|---|---|
|
||||
| Working memory (context window) | Standard — depends on model | Standard — depends on model |
|
||||
| Session persistence | Current session only | All sessions, FTS5 indexed |
|
||||
| Cross-session search | Not native | Built-in, with smart summarization |
|
||||
| Semantic memory | Flat files / basic config | Persistent key-value with add/replace/remove |
|
||||
| Procedural memory (skills) | None native | 100+ skills, auto-maintained, categorized |
|
||||
| Project context | Manual injection | Automatic via .hermes.md |
|
||||
| Memory mutation | Append-only or manual | First-class replace/remove operations |
|
||||
| Memory scoping | Global or nothing | Per-project, per-category, per-skill |
|
||||
| Stale memory handling | Manual cleanup | Replace/remove + skill auto-patching |
|
||||
|
||||
The fundamental difference: OpenClaw treats memory as configuration. Hermes treats memory as a living system that the agent actively maintains.
|
||||
|
||||
---
|
||||
|
||||
## 5. Practical Architecture Recommendations
|
||||
|
||||
Here is the "retarded structure" you asked for. Regardless of what framework you use, build your agent memory like this:
|
||||
|
||||
### Layer 1: Immutable Project Context (Load Once, Rarely Changes)
|
||||
|
||||
Create a project context file. Call it whatever your framework supports. Include:
|
||||
- Tech stack and versions
|
||||
- Key architectural decisions
|
||||
- Team conventions and coding standards
|
||||
- Infrastructure topology
|
||||
- Current priorities
|
||||
|
||||
This gets loaded at the start of every session. Keep it under 2000 tokens. If it is bigger, you are putting too much in here.
|
||||
|
||||
### Layer 2: Mutable Facts Store (Changes Weekly)
|
||||
|
||||
A key-value store for things that change:
|
||||
- Current sprint goals
|
||||
- Recent deployments and their status
|
||||
- Known bugs and workarounds
|
||||
- API endpoints and credentials references
|
||||
- Team member roles and availability
|
||||
|
||||
Update these actively. Delete them when they expire. If your store has entries from three months ago that are still accurate, great. If it has entries from three months ago that nobody has checked, that is a time bomb.
|
||||
|
||||
### Layer 3: Searchable History (Never Deleted, Always Indexed)
|
||||
|
||||
Every conversation should be stored and indexed for full-text search. You do not need to load all of history into context — you need to be able to find the right conversation when it matters.
|
||||
|
||||
If your framework does not support this natively (OpenClaw does not), build it:
|
||||
|
||||
```python
|
||||
# Minimal session indexing with SQLite FTS5
|
||||
import sqlite3
|
||||
|
||||
db = sqlite3.connect("agent_memory.db")
|
||||
db.execute("""
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS sessions
|
||||
USING fts5(session_id, timestamp, role, content)
|
||||
""")
|
||||
|
||||
def store_message(session_id, role, content):
|
||||
db.execute(
|
||||
"INSERT INTO sessions VALUES (?, datetime('now'), ?, ?)",
|
||||
(session_id, role, content)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
def search_history(query, limit=5):
|
||||
return db.execute(
|
||||
"SELECT session_id, timestamp, snippet(sessions, 3, '>>>', '<<<', '...', 32) "
|
||||
"FROM sessions WHERE sessions MATCH ? ORDER BY rank LIMIT ?",
|
||||
(query, limit)
|
||||
).fetchall()
|
||||
```
|
||||
|
||||
That is 20 lines. It gives you cross-session search. There is no excuse not to have this.
|
||||
|
||||
### Layer 4: Procedural Library (Grows Over Time)
|
||||
|
||||
When your agent successfully completes a complex task (5+ steps, errors overcome, non-obvious approach), save the procedure:
|
||||
|
||||
```markdown
|
||||
# Skill: deploy-to-production
|
||||
|
||||
## When to Use
|
||||
- User asks to deploy latest changes
|
||||
- CI passes on main branch
|
||||
|
||||
## Steps
|
||||
1. Pull latest main: `git pull origin main`
|
||||
2. Run tests: `pytest --tb=short`
|
||||
3. Build container: `docker build -t app:$(git rev-parse --short HEAD) .`
|
||||
4. Push to registry: `docker push registry.example.com/app:$(git rev-parse --short HEAD)`
|
||||
5. Update compose: change image tag in docker-compose.prod.yml
|
||||
6. Deploy: `docker compose -f docker-compose.prod.yml up -d`
|
||||
7. Verify: `curl -f https://app.example.com/health`
|
||||
|
||||
## Pitfalls
|
||||
- Always run tests before building — broken deploys waste 10 minutes
|
||||
- The health endpoint takes up to 30 seconds after container start
|
||||
- If migrations are pending, run them BEFORE deploying the new container
|
||||
|
||||
## Last Updated
|
||||
2026-04-01 — added migration warning after incident
|
||||
```
|
||||
|
||||
Store these as files. Index them by name and description. Load the relevant one when a matching task comes up.
|
||||
|
||||
### Layer 5: Automatic Retrieval Logic
|
||||
|
||||
This is where most DIY setups fail. Having memory is not enough — you need retrieval logic that decides what to load when.
|
||||
|
||||
Rules of thumb:
|
||||
- Layer 1 (project context): always loaded
|
||||
- Layer 2 (facts): loaded on session start, refreshed on demand
|
||||
- Layer 3 (history): loaded only when the agent searches, never bulk-loaded
|
||||
- Layer 4 (procedures): loaded when the task matches a known skill, scanned at session start
|
||||
|
||||
If you are building this yourself on top of OpenClaw, you are essentially building what Hermes already has. That is fine — understanding the architecture matters more than the specific tool.
|
||||
|
||||
---
|
||||
|
||||
## 6. Common Pitfalls (How Memory Systems Fail)
|
||||
|
||||
### Context Window Overflow
|
||||
|
||||
The number one killer. You eagerly load everything — project context, all facts, recent history, every relevant skill — and suddenly you have used 80k tokens before the user says anything. The model's actual working space is cramped, responses degrade, and costs spike.
|
||||
|
||||
**Fix:** Budget your context. Reserve at least 40% for the actual conversation. If your injected context exceeds 60% of the window, you are loading too much. Summarize, prioritize, and leave things on disk until they are actually needed.
|
||||
|
||||
### Stale Memory
|
||||
|
||||
"The deploy target is AWS" — except you migrated to Hetzner two months ago and nobody updated the memory. Now the agent is confidently giving you AWS-specific advice for a Hetzner server.
|
||||
|
||||
**Fix:** Every memory entry needs a mechanism for replacement or expiration. Append-only stores are a trap. If your framework only supports adding memories, you need a garbage collection process — periodic review that flags and removes outdated entries.
|
||||
|
||||
### Memory Pollution
|
||||
|
||||
The agent stores a wrong conclusion from one session. It retrieves that wrong conclusion in a future session and compounds the error. Garbage in, garbage out, but now the garbage is persistent.
|
||||
|
||||
**Fix:** Be selective about what gets stored. Not every conversation produces storeable knowledge. Require some quality bar — only store outcomes of successful tasks, verified facts, and user-confirmed procedures. Never auto-store speculative reasoning or intermediate debugging thoughts.
|
||||
|
||||
### The "I Remember Everything" Trap
|
||||
|
||||
Storing everything is almost as bad as storing nothing. When the agent retrieves 50 "relevant" memories for a simple question, the signal-to-noise ratio collapses. The model gets confused by contradictory or tangentially related information.
|
||||
|
||||
**Fix:** Less is more. Rank retrieval results by relevance. Return the top 3-5, not the top 50. Use temporal decay — recent memories should rank higher than old ones for the same relevance score.
|
||||
|
||||
### No Memory Hygiene
|
||||
|
||||
Memories are never reviewed, never pruned, never organized. Over months the store becomes a swamp of outdated facts, half-completed procedures, and conflicting information.
|
||||
|
||||
**Fix:** Schedule maintenance. Whether it is automated (expiration dates, periodic LLM-driven review) or manual (a human scans the memory store monthly), memory systems need upkeep. Hermes handles this partly through its replace/remove operations and skill auto-patching, but even there, periodic human review catches things the agent misses.
|
||||
|
||||
---
|
||||
|
||||
## 7. TL;DR — The Practical Answer
|
||||
|
||||
You asked for the structure. Here it is:
|
||||
|
||||
1. **Static project context** → one file, always loaded, under 2k tokens
|
||||
2. **Mutable facts** → key-value store with add/update/delete, loaded at session start
|
||||
3. **Searchable history** → every conversation indexed with FTS5, searched on demand
|
||||
4. **Procedural skills** → markdown files with steps/pitfalls/verification, loaded when task matches
|
||||
5. **Retrieval logic** → decides what from layers 2-4 gets loaded into the context window
|
||||
|
||||
Build these five layers and your agent will actually remember things without choking on its own context. Whether you build it on top of OpenClaw or switch to something that has it built in (Hermes has all five natively) is your call.
|
||||
|
||||
The memory problem is a solved problem. It is just not solved by most frameworks out of the box.
|
||||
|
||||
---
|
||||
|
||||
*Written by a Hermes agent. Biased, but honest about it.*
|
||||
63
scripts/auto_restart_agent.sh
Normal file
63
scripts/auto_restart_agent.sh
Normal file
@@ -0,0 +1,63 @@
|
||||
#!/usr/bin/env bash
|
||||
# auto_restart_agent.sh — Auto-restart dead critical processes (FLEET-007)
|
||||
# Refs: timmy-home #560
|
||||
set -euo pipefail
|
||||
|
||||
LOG_DIR="/var/log/timmy"
|
||||
ALERT_LOG="${LOG_DIR}/auto_restart.log"
|
||||
STATE_DIR="/var/lib/timmy/restarts"
|
||||
mkdir -p "$LOG_DIR" "$STATE_DIR"
|
||||
|
||||
TELEGRAM_BOT_TOKEN="${TELEGRAM_BOT_TOKEN:-}"
|
||||
TELEGRAM_CHAT_ID="${TELEGRAM_CHAT_ID:-}"
|
||||
|
||||
log() { echo "[$(date -Iseconds)] $1" | tee -a "$ALERT_LOG"; }
|
||||
|
||||
send_telegram() {
|
||||
local msg="$1"
|
||||
if [[ -n "$TELEGRAM_BOT_TOKEN" && -n "$TELEGRAM_CHAT_ID" ]]; then
|
||||
curl -s -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
|
||||
-d "chat_id=${TELEGRAM_CHAT_ID}" -d "text=${msg}" >/dev/null 2>&1 || true
|
||||
fi
|
||||
}
|
||||
|
||||
# Format: "process_name:command_to_restart"
|
||||
# Override via AUTO_RESTART_PROCESSES env var
|
||||
DEFAULT_PROCESSES="act_runner:cd /opt/gitea-runner && nohup ./act_runner daemon >/var/log/gitea-runner.log 2>&1 &"
|
||||
PROCESSES="${AUTO_RESTART_PROCESSES:-$DEFAULT_PROCESSES}"
|
||||
|
||||
IFS=',' read -ra PROC_LIST <<< "$PROCESSES"
|
||||
|
||||
for entry in "${PROC_LIST[@]}"; do
|
||||
proc_name="${entry%%:*}"
|
||||
restart_cmd="${entry#*:}"
|
||||
proc_name=$(echo "$proc_name" | xargs)
|
||||
restart_cmd=$(echo "$restart_cmd" | xargs)
|
||||
|
||||
state_file="${STATE_DIR}/${proc_name}.count"
|
||||
count=$(cat "$state_file" 2>/dev/null || echo 0)
|
||||
|
||||
if pgrep -f "$proc_name" >/dev/null 2>&1; then
|
||||
# Process alive — reset counter
|
||||
if [[ "$count" -ne 0 ]]; then
|
||||
echo 0 > "$state_file"
|
||||
log "$proc_name is healthy — reset restart counter"
|
||||
fi
|
||||
continue
|
||||
fi
|
||||
|
||||
# Process dead
|
||||
count=$((count + 1))
|
||||
echo "$count" > "$state_file"
|
||||
|
||||
if [[ "$count" -le 3 ]]; then
|
||||
log "CRITICAL: $proc_name is dead (attempt $count/3). Restarting..."
|
||||
eval "$restart_cmd" || log "ERROR: restart command failed for $proc_name"
|
||||
send_telegram "🔄 Auto-restarted $proc_name (attempt $count/3)"
|
||||
else
|
||||
log "ESCALATION: $proc_name still dead after 3 restart attempts."
|
||||
send_telegram "🚨 ESCALATION: $proc_name failed to restart after 3 attempts. Manual intervention required."
|
||||
fi
|
||||
done
|
||||
|
||||
touch "${STATE_DIR}/auto_restart.last"
|
||||
80
scripts/backup_pipeline.sh
Normal file
80
scripts/backup_pipeline.sh
Normal file
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env bash
|
||||
# backup_pipeline.sh — Daily fleet backup pipeline (FLEET-008)
|
||||
# Refs: timmy-home #561
|
||||
set -euo pipefail
|
||||
|
||||
BACKUP_ROOT="/backups/timmy"
|
||||
DATESTAMP=$(date +%Y%m%d-%H%M%S)
|
||||
BACKUP_DIR="${BACKUP_ROOT}/${DATESTAMP}"
|
||||
LOG_DIR="/var/log/timmy"
|
||||
ALERT_LOG="${LOG_DIR}/backup_pipeline.log"
|
||||
mkdir -p "$BACKUP_DIR" "$LOG_DIR"
|
||||
|
||||
TELEGRAM_BOT_TOKEN="${TELEGRAM_BOT_TOKEN:-}"
|
||||
TELEGRAM_CHAT_ID="${TELEGRAM_CHAT_ID:-}"
|
||||
OFFSITE_TARGET="${OFFSITE_TARGET:-}"
|
||||
|
||||
log() { echo "[$(date -Iseconds)] $1" | tee -a "$ALERT_LOG"; }
|
||||
|
||||
send_telegram() {
|
||||
local msg="$1"
|
||||
if [[ -n "$TELEGRAM_BOT_TOKEN" && -n "$TELEGRAM_CHAT_ID" ]]; then
|
||||
curl -s -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
|
||||
-d "chat_id=${TELEGRAM_CHAT_ID}" -d "text=${msg}" >/dev/null 2>&1 || true
|
||||
fi
|
||||
}
|
||||
|
||||
status=0
|
||||
|
||||
# --- Gitea repositories ---
|
||||
if [[ -d /root/gitea ]]; then
|
||||
tar czf "${BACKUP_DIR}/gitea-repos.tar.gz" -C /root gitea 2>/dev/null || true
|
||||
log "Backed up Gitea repos"
|
||||
fi
|
||||
|
||||
# --- Agent configs and state ---
|
||||
for wiz in bezalel allegro ezra timmy; do
|
||||
if [[ -d "/root/wizards/${wiz}" ]]; then
|
||||
tar czf "${BACKUP_DIR}/${wiz}-home.tar.gz" -C /root/wizards "${wiz}" 2>/dev/null || true
|
||||
log "Backed up ${wiz} home"
|
||||
fi
|
||||
done
|
||||
|
||||
# --- System configs ---
|
||||
cp /etc/crontab "${BACKUP_DIR}/crontab" 2>/dev/null || true
|
||||
cp -r /etc/systemd/system "${BACKUP_DIR}/systemd" 2>/dev/null || true
|
||||
log "Backed up system configs"
|
||||
|
||||
# --- Evennia worlds (if present) ---
|
||||
if [[ -d /root/evennia ]]; then
|
||||
tar czf "${BACKUP_DIR}/evennia-worlds.tar.gz" -C /root evennia 2>/dev/null || true
|
||||
log "Backed up Evennia worlds"
|
||||
fi
|
||||
|
||||
# --- Manifest ---
|
||||
find "$BACKUP_DIR" -type f > "${BACKUP_DIR}/manifest.txt"
|
||||
log "Backup manifest written"
|
||||
|
||||
# --- Offsite sync ---
|
||||
if [[ -n "$OFFSITE_TARGET" ]]; then
|
||||
if rsync -az --delete "${BACKUP_DIR}/" "${OFFSITE_TARGET}/${DATESTAMP}/" 2>/dev/null; then
|
||||
log "Offsite sync completed"
|
||||
else
|
||||
log "WARNING: Offsite sync failed"
|
||||
status=1
|
||||
fi
|
||||
fi
|
||||
|
||||
# --- Retention: keep last 7 days ---
|
||||
find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 -type d -mtime +7 -exec rm -rf {} + 2>/dev/null || true
|
||||
log "Retention applied (7 days)"
|
||||
|
||||
if [[ "$status" -eq 0 ]]; then
|
||||
log "Backup pipeline completed: ${BACKUP_DIR}"
|
||||
send_telegram "✅ Daily backup completed: ${DATESTAMP}"
|
||||
else
|
||||
log "Backup pipeline completed with WARNINGS: ${BACKUP_DIR}"
|
||||
send_telegram "⚠️ Daily backup completed with warnings: ${DATESTAMP}"
|
||||
fi
|
||||
|
||||
exit "$status"
|
||||
@@ -23,7 +23,7 @@ def main():
|
||||
if fleet.get("ezra") == "OFFLINE":
|
||||
print("Ezra (Primary) is OFFLINE. Optimizing for local-only fallback...")
|
||||
# In a real scenario, this would update the YAML config
|
||||
print("Updated config.yaml: fallback_model -> local:hermes3")
|
||||
print("Updated config.yaml: fallback_model -> ollama:gemma4:12b")
|
||||
else:
|
||||
print("Fleet health is optimal. Maintaining high-performance routing.")
|
||||
|
||||
|
||||
83
scripts/fleet_health_probe.sh
Normal file
83
scripts/fleet_health_probe.sh
Normal file
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env bash
|
||||
# fleet_health_probe.sh — Automated health checks for Timmy Foundation fleet
|
||||
# Refs: timmy-home #559, FLEET-006
|
||||
# Runs every 5 min via cron. Checks: SSH reachability, disk < 90%, memory < 90%, critical processes.
|
||||
set -euo pipefail
|
||||
|
||||
LOG_DIR="/var/log/timmy"
|
||||
ALERT_LOG="${LOG_DIR}/fleet_health.log"
|
||||
HEARTBEAT_DIR="/var/lib/timmy/heartbeats"
|
||||
mkdir -p "$LOG_DIR" "$HEARTBEAT_DIR"
|
||||
|
||||
# Configurable thresholds
|
||||
DISK_THRESHOLD=90
|
||||
MEM_THRESHOLD=90
|
||||
|
||||
# Hosts to probe (space-separated SSH hosts)
|
||||
FLEET_HOSTS="${FLEET_HOSTS:-143.198.27.163 104.131.15.18}"
|
||||
|
||||
# Critical processes that must be running locally
|
||||
CRITICAL_PROCESSES="${CRITICAL_PROCESSES:-act_runner}"
|
||||
|
||||
log() {
|
||||
echo "[$(date -Iseconds)] $1" | tee -a "$ALERT_LOG"
|
||||
}
|
||||
|
||||
alert() {
|
||||
log "ALERT: $1"
|
||||
}
|
||||
|
||||
ok() {
|
||||
log "OK: $1"
|
||||
}
|
||||
|
||||
status=0
|
||||
|
||||
# --- SSH Reachability ---
|
||||
for host in $FLEET_HOSTS; do
|
||||
if nc -z -w 5 "$host" 22 >/dev/null 2>&1 || timeout 5 bash -c "</dev/tcp/${host}/22" 2>/dev/null; then
|
||||
ok "SSH reachable: $host"
|
||||
else
|
||||
alert "SSH unreachable: $host"
|
||||
status=1
|
||||
fi
|
||||
done
|
||||
|
||||
# --- Disk Usage ---
|
||||
disk_usage=$(df / | awk 'NR==2 {print $5}' | tr -d '%')
|
||||
if [[ "$disk_usage" -lt "$DISK_THRESHOLD" ]]; then
|
||||
ok "Disk usage: ${disk_usage}%"
|
||||
else
|
||||
alert "Disk usage critical: ${disk_usage}%"
|
||||
status=1
|
||||
fi
|
||||
|
||||
# --- Memory Usage ---
|
||||
mem_usage=$(free | awk '/Mem:/ {printf("%.0f", $3/$2 * 100.0)}')
|
||||
if [[ "$mem_usage" -lt "$MEM_THRESHOLD" ]]; then
|
||||
ok "Memory usage: ${mem_usage}%"
|
||||
else
|
||||
alert "Memory usage critical: ${mem_usage}%"
|
||||
status=1
|
||||
fi
|
||||
|
||||
# --- Critical Processes ---
|
||||
for proc in $CRITICAL_PROCESSES; do
|
||||
if pgrep -f "$proc" >/dev/null 2>&1; then
|
||||
ok "Process alive: $proc"
|
||||
else
|
||||
alert "Process missing: $proc"
|
||||
status=1
|
||||
fi
|
||||
done
|
||||
|
||||
# --- Heartbeat Touch ---
|
||||
touch "${HEARTBEAT_DIR}/fleet_health.last"
|
||||
|
||||
if [[ "$status" -eq 0 ]]; then
|
||||
log "Fleet health probe passed."
|
||||
else
|
||||
log "Fleet health probe FAILED."
|
||||
fi
|
||||
|
||||
exit "$status"
|
||||
164
scripts/fleet_milestones.py
Normal file
164
scripts/fleet_milestones.py
Normal file
@@ -0,0 +1,164 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
fleet_milestones.py — Print milestone messages when fleet achievements trigger.
|
||||
Refs: timmy-home #557, FLEET-004
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
|
||||
STATE_FILE = Path("/var/lib/timmy/milestones.json")
|
||||
LOG_FILE = Path("/var/log/timmy/fleet_milestones.log")
|
||||
|
||||
MILESTONES = {
|
||||
"health_check_first_run": {
|
||||
"phase": 1,
|
||||
"message": "◈ MILESTONE: First automated health check ran — we are no longer watching the clock.",
|
||||
},
|
||||
"auto_restart_3am": {
|
||||
"phase": 2,
|
||||
"message": "◈ MILESTONE: A process failed at 3am and restarted itself before anyone woke up.",
|
||||
},
|
||||
"backup_first_success": {
|
||||
"phase": 2,
|
||||
"message": "◈ MILESTONE: First automated backup completed — fleet state is no longer ephemeral.",
|
||||
},
|
||||
"ci_green_main": {
|
||||
"phase": 2,
|
||||
"message": "◈ MILESTONE: CI pipeline kept main green for 24 hours straight.",
|
||||
},
|
||||
"pr_auto_merged": {
|
||||
"phase": 2,
|
||||
"message": "◈ MILESTONE: An agent PR passed review and merged without human hands.",
|
||||
},
|
||||
"dns_self_healed": {
|
||||
"phase": 2,
|
||||
"message": "◈ MILESTONE: DNS outage detected and resolved automatically.",
|
||||
},
|
||||
"runner_self_healed": {
|
||||
"phase": 2,
|
||||
"message": "◈ MILESTONE: CI runner died and resurrected itself within 60 seconds.",
|
||||
},
|
||||
"secrets_scan_clean": {
|
||||
"phase": 2,
|
||||
"message": "◈ MILESTONE: 7 consecutive days with zero leaked secrets detected.",
|
||||
},
|
||||
"local_inference_first": {
|
||||
"phase": 3,
|
||||
"message": "◈ MILESTONE: First fully local inference completed — no tokens left the building.",
|
||||
},
|
||||
"ollama_serving_fleet": {
|
||||
"phase": 3,
|
||||
"message": "◈ MILESTONE: Ollama serving models to all fleet wizards.",
|
||||
},
|
||||
"offline_docs_sync": {
|
||||
"phase": 3,
|
||||
"message": "◈ MILESTONE: Entire documentation tree synchronized without internet.",
|
||||
},
|
||||
"cross_agent_delegate": {
|
||||
"phase": 3,
|
||||
"message": "◈ MILESTONE: One wizard delegated a task to another and received a finished result.",
|
||||
},
|
||||
"backup_verified_restore": {
|
||||
"phase": 4,
|
||||
"message": "◈ MILESTONE: Backup restored and verified — disaster recovery is real.",
|
||||
},
|
||||
"vps_bootstrap_under_60": {
|
||||
"phase": 4,
|
||||
"message": "◈ MILESTONE: New VPS bootstrapped from bare metal in under 60 minutes.",
|
||||
},
|
||||
"zero_cloud_day": {
|
||||
"phase": 4,
|
||||
"message": "◈ MILESTONE: 24 hours with zero cloud API calls — total sovereignty achieved.",
|
||||
},
|
||||
"fleet_orchestrator_active": {
|
||||
"phase": 5,
|
||||
"message": "◈ MILESTONE: Fleet orchestrator actively balancing load across agents.",
|
||||
},
|
||||
"cell_isolation_proven": {
|
||||
"phase": 5,
|
||||
"message": "◈ MILESTONE: Agent cell isolation proven — one crash did not spread.",
|
||||
},
|
||||
"mission_bus_first": {
|
||||
"phase": 5,
|
||||
"message": "◈ MILESTONE: First cross-agent mission completed via the mission bus.",
|
||||
},
|
||||
"resurrection_pool_used": {
|
||||
"phase": 5,
|
||||
"message": "◈ MILESTONE: A dead wizard was detected and resurrected automatically.",
|
||||
},
|
||||
"infra_generates_revenue": {
|
||||
"phase": 6,
|
||||
"message": "◈ MILESTONE: Infrastructure generated its first dollar of revenue.",
|
||||
},
|
||||
"client_onboarded_unattended": {
|
||||
"phase": 6,
|
||||
"message": "◈ MILESTONE: Client onboarded without human intervention.",
|
||||
},
|
||||
"fleet_pays_for_itself": {
|
||||
"phase": 6,
|
||||
"message": "◈ MILESTONE: Fleet revenue exceeds operational cost — it breathes on its own.",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def load_state() -> dict:
|
||||
if STATE_FILE.exists():
|
||||
return json.loads(STATE_FILE.read_text())
|
||||
return {}
|
||||
|
||||
|
||||
def save_state(state: dict):
|
||||
STATE_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
STATE_FILE.write_text(json.dumps(state, indent=2))
|
||||
|
||||
|
||||
def log(msg: str):
|
||||
LOG_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
entry = f"[{datetime.utcnow().isoformat()}Z] {msg}"
|
||||
print(entry)
|
||||
with LOG_FILE.open("a") as f:
|
||||
f.write(entry + "\n")
|
||||
|
||||
|
||||
def trigger(key: str, dry_run: bool = False):
|
||||
if key not in MILESTONES:
|
||||
print(f"Unknown milestone: {key}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
state = load_state()
|
||||
if state.get(key):
|
||||
if not dry_run:
|
||||
print(f"Milestone {key} already triggered. Skipping.")
|
||||
return
|
||||
milestone = MILESTONES[key]
|
||||
if not dry_run:
|
||||
state[key] = {"triggered_at": datetime.utcnow().isoformat() + "Z", "phase": milestone["phase"]}
|
||||
save_state(state)
|
||||
log(milestone["message"])
|
||||
|
||||
|
||||
def list_all():
|
||||
for key, m in MILESTONES.items():
|
||||
print(f"{key} (phase {m['phase']}): {m['message']}")
|
||||
|
||||
|
||||
def main():
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser(description="Fleet milestone tracker")
|
||||
parser.add_argument("--trigger", help="Trigger a milestone by key")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Show but do not record")
|
||||
parser.add_argument("--list", action="store_true", help="List all milestones")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.list:
|
||||
list_all()
|
||||
elif args.trigger:
|
||||
trigger(args.trigger, dry_run=args.dry_run)
|
||||
else:
|
||||
parser.print_help()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
59
scripts/telegram_thread_reporter.py
Normal file
59
scripts/telegram_thread_reporter.py
Normal file
@@ -0,0 +1,59 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
telegram_thread_reporter.py — Route reports to Telegram threads (#895)
|
||||
Usage:
|
||||
python telegram_thread_reporter.py --topic ops --message "Heartbeat OK"
|
||||
python telegram_thread_reporter.py --topic burn --message "Burn cycle done"
|
||||
python telegram_thread_reporter.py --topic main --message "Escalation!"
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
import json
|
||||
|
||||
DEFAULT_THREADS = {
|
||||
"ops": os.environ.get("TELEGRAM_OPS_THREAD_ID"),
|
||||
"burn": os.environ.get("TELEGRAM_BURN_THREAD_ID"),
|
||||
"main": None, # main channel = no thread id
|
||||
}
|
||||
|
||||
|
||||
def send_message(bot_token: str, chat_id: str, text: str, thread_id: str | None = None):
|
||||
url = f"https://api.telegram.org/bot{bot_token}/sendMessage"
|
||||
data = {"chat_id": chat_id, "text": text, "parse_mode": "HTML"}
|
||||
if thread_id:
|
||||
data["message_thread_id"] = thread_id
|
||||
payload = urllib.parse.urlencode(data).encode("utf-8")
|
||||
req = urllib.request.Request(url, data=payload, headers={"Content-Type": "application/x-www-form-urlencoded"})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=15) as resp:
|
||||
return json.loads(resp.read().decode("utf-8"))
|
||||
except Exception as e:
|
||||
return {"ok": False, "error": str(e)}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Telegram thread reporter")
|
||||
parser.add_argument("--topic", required=True, choices=["ops", "burn", "main"])
|
||||
parser.add_argument("--message", required=True)
|
||||
args = parser.parse_args()
|
||||
|
||||
bot_token = os.environ.get("TELEGRAM_BOT_TOKEN")
|
||||
chat_id = os.environ.get("TELEGRAM_CHAT_ID")
|
||||
if not bot_token or not chat_id:
|
||||
print("Missing TELEGRAM_BOT_TOKEN or TELEGRAM_CHAT_ID", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
thread_id = DEFAULT_THREADS.get(args.topic)
|
||||
result = send_message(bot_token, chat_id, args.message, thread_id)
|
||||
if result.get("ok"):
|
||||
print(f"Sent to {args.topic}")
|
||||
else:
|
||||
print(f"Failed: {result}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,8 +1,33 @@
|
||||
model:
|
||||
default: kimi-for-coding
|
||||
default: kimi-k2.5
|
||||
provider: kimi-coding
|
||||
toolsets:
|
||||
- all
|
||||
fallback_providers:
|
||||
- provider: kimi-coding
|
||||
model: kimi-k2.5
|
||||
timeout: 120
|
||||
reason: Kimi coding fallback (front of chain)
|
||||
- provider: anthropic
|
||||
model: claude-sonnet-4-20250514
|
||||
timeout: 120
|
||||
reason: Direct Anthropic fallback
|
||||
- provider: openrouter
|
||||
model: anthropic/claude-sonnet-4-20250514
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
timeout: 120
|
||||
reason: OpenRouter fallback
|
||||
providers:
|
||||
kimi-coding:
|
||||
base_url: https://api.kimi.com/coding/v1
|
||||
timeout: 60
|
||||
max_retries: 3
|
||||
anthropic:
|
||||
timeout: 120
|
||||
openrouter:
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
timeout: 120
|
||||
agent:
|
||||
max_turns: 30
|
||||
reasoning_effort: medium
|
||||
|
||||
Reference in New Issue
Block a user