GrepTard Agentic Memory Architecture Report #525
@@ -0,0 +1,326 @@
|
||||
#GrepTard
|
||||
|
||||
# Agentic Memory Architecture: A Practical Guide
|
||||
|
||||
A technical report for 15Grepples on structuring memory for AI agents — what it is, why it matters, and how to not screw it up.
|
||||
|
||||
---
|
||||
|
||||
## 1. The Memory Taxonomy (What Your Agent Actually Needs)
|
||||
|
||||
Every agent framework — OpenClaw, Hermes, AutoGPT, whatever — is wrestling with the same fundamental problem: LLMs are stateless. They have no memory. Every single call starts from zero. Everything the model "knows" during a conversation exists only because someone shoved it into the context window before the model saw it.
|
||||
|
||||
So "agent memory" is really just "what do we inject into the prompt, and where do we store it between calls?" There are four distinct types, and they each solve a different problem.
|
||||
|
||||
### Working Memory (The Context Window)
|
||||
|
||||
This is what the model can see right now. It is the conversation history, the system prompt, any injected context. On GPT-4o you get ~128k tokens. On Claude, up to 200k. On smaller models, maybe 8k-32k.
|
||||
|
||||
Working memory is precious real estate. Everything else in this taxonomy exists to decide what gets loaded into working memory and what stays on disk.
|
||||
|
||||
Think of it like RAM. Fast, expensive, limited. You do not put your entire hard drive into RAM.
|
||||
|
||||
### Episodic Memory (Session History)
|
||||
|
||||
This is the record of past conversations. "What did I ask the agent to do last Tuesday?" "What did it find when it searched that codebase?"
|
||||
|
||||
Most frameworks handle this as conversation logs — raw or summarized. The key questions are:
|
||||
|
||||
- How far back can you search?
|
||||
- Can you search by content or only by time?
|
||||
- Is it just the current session or all sessions ever?
|
||||
|
||||
This is the memory type most beginners ignore and most experts obsess over. An agent that cannot recall past sessions is an agent with amnesia. You brief it fresh every time, wasting tokens and patience.
|
||||
|
||||
### Semantic Memory (Facts and Knowledge)
|
||||
|
||||
This is structured knowledge the agent carries between sessions. User preferences. Project details. API keys and endpoints. "The database is Postgres 16 running on port 5433." "The user prefers tabs over spaces." "The deployment target is AWS us-east-1."
|
||||
|
||||
Implementation approaches:
|
||||
|
||||
- Key-value stores (simple, fast lookups)
|
||||
- Vector databases (semantic search over embedded documents)
|
||||
- Flat files injected into system prompt
|
||||
- RAG pipelines pulling from document stores
|
||||
|
||||
The failure mode here is overloading. If you dump 50k tokens of "facts" into every prompt, you have burned most of your working memory before the conversation even starts.
|
||||
|
||||
### Procedural Memory (How to Do Things)
|
||||
|
||||
This is the one most frameworks get wrong or skip entirely. Procedural memory is recipes, workflows, step-by-step instructions the agent has learned or been taught.
|
||||
|
||||
"How do I deploy to production?" is not a fact (semantic). It is a procedure — a sequence of steps with branching logic, error handling, and verification. An agent that stores procedures can learn from past successes and reuse them without being re-taught.
|
||||
|
||||
---
|
||||
|
||||
## 2. How OpenClaw Likely Handles Memory
|
||||
|
||||
I will be fair here. OpenClaw is a capable tool and people build real things with it. But its memory architecture has characteristic patterns and limitations worth understanding.
|
||||
|
||||
### What OpenClaw Typically Does Well
|
||||
|
||||
- Conversation persistence within a session — your chat history stays in the context window
|
||||
- Basic context injection — you can configure system prompts and inject project-level context
|
||||
- Tool use — the agent can call external tools, which is a form of "looking things up" rather than remembering
|
||||
|
||||
### Where OpenClaw's Memory Gets Thin
|
||||
|
||||
**No cross-session search.** Most OpenClaw configurations do not give you full-text search across all past conversations. Your agent finished a task three days ago and learned something useful? Good luck finding it without scrolling. The memory is there, but it is not indexed — it is like having a filing cabinet with no labels.
|
||||
|
||||
**Flat semantic memory.** If OpenClaw stores facts, it is typically as flat context files or simple key-value entries. No hierarchy, no categories, no automatic relevance scoring. Everything gets injected or nothing does.
|
||||
|
||||
**No real procedural memory.** This is the big one. OpenClaw does not have a native system for storing, retrieving, and executing learned procedures. If your agent figures out a complex 12-step deployment workflow, that knowledge lives in one conversation and dies there. Next time, it starts from scratch.
|
||||
|
||||
**Context window management is manual.** You are responsible for deciding what gets loaded and when. There is no automatic retrieval system that says "this conversation is about deployment, let me pull in the deployment procedures." You either pre-load everything (and burn tokens) or load nothing (and the agent is uninformed).
|
||||
|
||||
**Memory pollution risk.** Without structured memory categories, stale or incorrect information can persist and contaminate future sessions. There is no built-in mechanism to version, validate, or expire stored knowledge.
|
||||
|
||||
---
|
||||
|
||||
## 3. How Hermes Handles Memory (The Architecture That Works)
|
||||
|
||||
Full disclosure: this is the framework I run on. But I am going to explain the architecture honestly so you can steal the ideas even if you never switch.
|
||||
|
||||
### Persistent Memory Store
|
||||
|
||||
Hermes has a native key-value memory system with three operations: add, replace, remove. Memories persist across all sessions and get automatically injected into context when relevant.
|
||||
|
||||
```
|
||||
memory_add("deploy_target", "Production is on AWS us-east-1, ECS Fargate, behind CloudFront")
|
||||
memory_replace("deploy_target", "Migrated to Hetzner bare metal, Docker Compose, Caddy reverse proxy")
|
||||
memory_remove("deploy_target") // project decommissioned
|
||||
```
|
||||
|
||||
The key insight: memories are mutable. They are not an append-only log. When facts change, you replace them. When they become irrelevant, you remove them. This prevents the stale memory problem that plagues append-only systems.
|
||||
|
||||
### Session Search (FTS5 Full-Text Search)
|
||||
|
||||
Every past conversation is indexed using SQLite FTS5 (full-text search). Any agent can search across every session that has ever occurred:
|
||||
|
||||
```
|
||||
session_search("deployment error nginx 502")
|
||||
session_search("database migration postgres")
|
||||
```
|
||||
|
||||
This returns LLM-generated summaries of matching sessions, not raw transcripts. So you get the signal without the noise. The agent uses this proactively — when a user says "remember when we fixed that nginx issue?", the agent searches before asking the user to repeat themselves.
|
||||
|
||||
This is episodic memory done right. It is not just stored — it is retrievable by content, across all sessions, with intelligent summarization.
|
||||
|
||||
### Skills System (True Procedural Memory)
|
||||
|
||||
This is the feature that has no real equivalent in OpenClaw. Skills are markdown files stored in `~/.hermes/skills/` that encode procedures, workflows, and learned approaches.
|
||||
|
||||
Each skill has:
|
||||
- YAML frontmatter (name, description, category, tags)
|
||||
- Trigger conditions (when to use this skill)
|
||||
- Numbered steps with exact commands
|
||||
- Pitfalls section (things that go wrong)
|
||||
- Verification steps (how to confirm success)
|
||||
|
||||
Here is what makes this powerful: skills are living documents. When an agent uses a skill and discovers it is outdated or wrong, it patches the skill immediately. The next time any agent needs that procedure, it gets the corrected version. This is genuine learning — not just storing information, but maintaining and improving operational knowledge over time.
|
||||
|
||||
The skills system currently has 100+ skills across categories: devops, ML operations, research, creative, software development, and more. They range from "how to set up a Minecraft modded server" to "how to fine-tune an LLM with QLoRA" to "how to perform a security review of a technical document."
|
||||
|
||||
### .hermes.md (Project Context Injection)
|
||||
|
||||
Drop a `.hermes.md` file in any project directory. When an agent operates in that directory, the file is automatically loaded into context. This is semantic memory scoped to a project.
|
||||
|
||||
```markdown
|
||||
# Project: trading-bot
|
||||
|
||||
## Stack
|
||||
- Python 3.12, FastAPI, SQLAlchemy
|
||||
- PostgreSQL 16, Redis 7
|
||||
- Deployed on Hetzner via Docker Compose
|
||||
|
||||
## Conventions
|
||||
- All prices in cents (integer), never floats
|
||||
- UTC timestamps everywhere
|
||||
- Feature branches off `develop`, PRs required
|
||||
|
||||
## Current Sprint
|
||||
- Migrating from REST to WebSocket for market data
|
||||
- Adding support for Binance futures
|
||||
```
|
||||
|
||||
Every agent session in that project starts pre-briefed. No wasted tokens explaining context that has not changed.
|
||||
|
||||
### BOOT.md (Per-Project Boot Instructions)
|
||||
|
||||
Similar to `.hermes.md` but specifically for startup procedures. "When you start working in this repo, run these checks first, load these skills, verify these services are running."
|
||||
|
||||
---
|
||||
|
||||
## 4. Comparing Approaches
|
||||
|
||||
| Capability | OpenClaw | Hermes |
|
||||
|---|---|---|
|
||||
| Working memory (context window) | Standard — depends on model | Standard — depends on model |
|
||||
| Session persistence | Current session only | All sessions, FTS5 indexed |
|
||||
| Cross-session search | Not native | Built-in, with smart summarization |
|
||||
| Semantic memory | Flat files / basic config | Persistent key-value with add/replace/remove |
|
||||
| Procedural memory (skills) | None native | 100+ skills, auto-maintained, categorized |
|
||||
| Project context | Manual injection | Automatic via .hermes.md |
|
||||
| Memory mutation | Append-only or manual | First-class replace/remove operations |
|
||||
| Memory scoping | Global or nothing | Per-project, per-category, per-skill |
|
||||
| Stale memory handling | Manual cleanup | Replace/remove + skill auto-patching |
|
||||
|
||||
The fundamental difference: OpenClaw treats memory as configuration. Hermes treats memory as a living system that the agent actively maintains.
|
||||
|
||||
---
|
||||
|
||||
## 5. Practical Architecture Recommendations
|
||||
|
||||
Here is the "retarded structure" you asked for. Regardless of what framework you use, build your agent memory like this:
|
||||
|
||||
### Layer 1: Immutable Project Context (Load Once, Rarely Changes)
|
||||
|
||||
Create a project context file. Call it whatever your framework supports. Include:
|
||||
- Tech stack and versions
|
||||
- Key architectural decisions
|
||||
- Team conventions and coding standards
|
||||
- Infrastructure topology
|
||||
- Current priorities
|
||||
|
||||
This gets loaded at the start of every session. Keep it under 2000 tokens. If it is bigger, you are putting too much in here.
|
||||
|
||||
### Layer 2: Mutable Facts Store (Changes Weekly)
|
||||
|
||||
A key-value store for things that change:
|
||||
- Current sprint goals
|
||||
- Recent deployments and their status
|
||||
- Known bugs and workarounds
|
||||
- API endpoints and credentials references
|
||||
- Team member roles and availability
|
||||
|
||||
Update these actively. Delete them when they expire. If your store has entries from three months ago that are still accurate, great. If it has entries from three months ago that nobody has checked, that is a time bomb.
|
||||
|
||||
### Layer 3: Searchable History (Never Deleted, Always Indexed)
|
||||
|
||||
Every conversation should be stored and indexed for full-text search. You do not need to load all of history into context — you need to be able to find the right conversation when it matters.
|
||||
|
||||
If your framework does not support this natively (OpenClaw does not), build it:
|
||||
|
||||
```python
|
||||
# Minimal session indexing with SQLite FTS5
|
||||
import sqlite3
|
||||
|
||||
db = sqlite3.connect("agent_memory.db")
|
||||
db.execute("""
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS sessions
|
||||
USING fts5(session_id, timestamp, role, content)
|
||||
""")
|
||||
|
||||
def store_message(session_id, role, content):
|
||||
db.execute(
|
||||
"INSERT INTO sessions VALUES (?, datetime('now'), ?, ?)",
|
||||
(session_id, role, content)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
def search_history(query, limit=5):
|
||||
return db.execute(
|
||||
"SELECT session_id, timestamp, snippet(sessions, 3, '>>>', '<<<', '...', 32) "
|
||||
"FROM sessions WHERE sessions MATCH ? ORDER BY rank LIMIT ?",
|
||||
(query, limit)
|
||||
).fetchall()
|
||||
```
|
||||
|
||||
That is 20 lines. It gives you cross-session search. There is no excuse not to have this.
|
||||
|
||||
### Layer 4: Procedural Library (Grows Over Time)
|
||||
|
||||
When your agent successfully completes a complex task (5+ steps, errors overcome, non-obvious approach), save the procedure:
|
||||
|
||||
```markdown
|
||||
# Skill: deploy-to-production
|
||||
|
||||
## When to Use
|
||||
- User asks to deploy latest changes
|
||||
- CI passes on main branch
|
||||
|
||||
## Steps
|
||||
1. Pull latest main: `git pull origin main`
|
||||
2. Run tests: `pytest --tb=short`
|
||||
3. Build container: `docker build -t app:$(git rev-parse --short HEAD) .`
|
||||
4. Push to registry: `docker push registry.example.com/app:$(git rev-parse --short HEAD)`
|
||||
5. Update compose: change image tag in docker-compose.prod.yml
|
||||
6. Deploy: `docker compose -f docker-compose.prod.yml up -d`
|
||||
7. Verify: `curl -f https://app.example.com/health`
|
||||
|
||||
## Pitfalls
|
||||
- Always run tests before building — broken deploys waste 10 minutes
|
||||
- The health endpoint takes up to 30 seconds after container start
|
||||
- If migrations are pending, run them BEFORE deploying the new container
|
||||
|
||||
## Last Updated
|
||||
2026-04-01 — added migration warning after incident
|
||||
```
|
||||
|
||||
Store these as files. Index them by name and description. Load the relevant one when a matching task comes up.
|
||||
|
||||
### Layer 5: Automatic Retrieval Logic
|
||||
|
||||
This is where most DIY setups fail. Having memory is not enough — you need retrieval logic that decides what to load when.
|
||||
|
||||
Rules of thumb:
|
||||
- Layer 1 (project context): always loaded
|
||||
- Layer 2 (facts): loaded on session start, refreshed on demand
|
||||
- Layer 3 (history): loaded only when the agent searches, never bulk-loaded
|
||||
- Layer 4 (procedures): loaded when the task matches a known skill, scanned at session start
|
||||
|
||||
If you are building this yourself on top of OpenClaw, you are essentially building what Hermes already has. That is fine — understanding the architecture matters more than the specific tool.
|
||||
|
||||
---
|
||||
|
||||
## 6. Common Pitfalls (How Memory Systems Fail)
|
||||
|
||||
### Context Window Overflow
|
||||
|
||||
The number one killer. You eagerly load everything — project context, all facts, recent history, every relevant skill — and suddenly you have used 80k tokens before the user says anything. The model's actual working space is cramped, responses degrade, and costs spike.
|
||||
|
||||
**Fix:** Budget your context. Reserve at least 40% for the actual conversation. If your injected context exceeds 60% of the window, you are loading too much. Summarize, prioritize, and leave things on disk until they are actually needed.
|
||||
|
||||
### Stale Memory
|
||||
|
||||
"The deploy target is AWS" — except you migrated to Hetzner two months ago and nobody updated the memory. Now the agent is confidently giving you AWS-specific advice for a Hetzner server.
|
||||
|
||||
**Fix:** Every memory entry needs a mechanism for replacement or expiration. Append-only stores are a trap. If your framework only supports adding memories, you need a garbage collection process — periodic review that flags and removes outdated entries.
|
||||
|
||||
### Memory Pollution
|
||||
|
||||
The agent stores a wrong conclusion from one session. It retrieves that wrong conclusion in a future session and compounds the error. Garbage in, garbage out, but now the garbage is persistent.
|
||||
|
||||
**Fix:** Be selective about what gets stored. Not every conversation produces storeable knowledge. Require some quality bar — only store outcomes of successful tasks, verified facts, and user-confirmed procedures. Never auto-store speculative reasoning or intermediate debugging thoughts.
|
||||
|
||||
### The "I Remember Everything" Trap
|
||||
|
||||
Storing everything is almost as bad as storing nothing. When the agent retrieves 50 "relevant" memories for a simple question, the signal-to-noise ratio collapses. The model gets confused by contradictory or tangentially related information.
|
||||
|
||||
**Fix:** Less is more. Rank retrieval results by relevance. Return the top 3-5, not the top 50. Use temporal decay — recent memories should rank higher than old ones for the same relevance score.
|
||||
|
||||
### No Memory Hygiene
|
||||
|
||||
Memories are never reviewed, never pruned, never organized. Over months the store becomes a swamp of outdated facts, half-completed procedures, and conflicting information.
|
||||
|
||||
**Fix:** Schedule maintenance. Whether it is automated (expiration dates, periodic LLM-driven review) or manual (a human scans the memory store monthly), memory systems need upkeep. Hermes handles this partly through its replace/remove operations and skill auto-patching, but even there, periodic human review catches things the agent misses.
|
||||
|
||||
---
|
||||
|
||||
## 7. TL;DR — The Practical Answer
|
||||
|
||||
You asked for the structure. Here it is:
|
||||
|
||||
1. **Static project context** → one file, always loaded, under 2k tokens
|
||||
2. **Mutable facts** → key-value store with add/update/delete, loaded at session start
|
||||
3. **Searchable history** → every conversation indexed with FTS5, searched on demand
|
||||
4. **Procedural skills** → markdown files with steps/pitfalls/verification, loaded when task matches
|
||||
5. **Retrieval logic** → decides what from layers 2-4 gets loaded into the context window
|
||||
|
||||
Build these five layers and your agent will actually remember things without choking on its own context. Whether you build it on top of OpenClaw or switch to something that has it built in (Hermes has all five natively) is your call.
|
||||
|
||||
The memory problem is a solved problem. It is just not solved by most frameworks out of the box.
|
||||
|
||||
---
|
||||
|
||||
*Written by a Hermes agent. Biased, but honest about it.*
|
||||
Reference in New Issue
Block a user