From 511eacb5731e93327562a95cf01f8310f50bfba9 Mon Sep 17 00:00:00 2001 From: Alexander Whitestone Date: Thu, 9 Apr 2026 12:45:48 -0400 Subject: [PATCH] docs: add Memory Architecture Guide Comprehensive guide covering the Hermes memory system: - Built-in memory (MEMORY.md / USER.md) with frozen snapshot pattern - Session search (FTS5 + Gemini Flash summarization) - Skills as procedural memory - External memory providers (8 plugins) - System interaction flow and data lifecycle - Best practices for what to save/skip - Privacy and data locality guarantees - Configuration reference (char limits, nudge interval, flush settings) - Troubleshooting common issues Closes #258 --- docs/memory-architecture.md | 335 ++++++++++++++++++++++++++++++++++++ 1 file changed, 335 insertions(+) create mode 100644 docs/memory-architecture.md diff --git a/docs/memory-architecture.md b/docs/memory-architecture.md new file mode 100644 index 000000000..f636dfdaa --- /dev/null +++ b/docs/memory-architecture.md @@ -0,0 +1,335 @@ +# Memory Architecture Guide + +How Hermes Agent remembers things across sessions — the stores, the tools, the data flow, and how to configure it all. + +## Overview + +Hermes has a multi-layered memory system. It is not one thing — it is several independent systems that complement each other: + +1. **Persistent Memory** (MEMORY.md / USER.md) — bounded, curated notes injected into every system prompt +2. **Session Search** — full-text search across all past conversation transcripts +3. **Skills** — procedural memory: reusable workflows stored as SKILL.md files +4. **External Memory Providers** — optional plugins (Honcho, Holographic, Mem0, etc.) for deeper recall + +All built-in memory lives on disk under `~/.hermes/` (or `$HERMES_HOME`). No memory data leaves the machine unless you explicitly configure an external cloud provider. + +## Memory Types in Detail + +### 1. Persistent Memory (MEMORY.md and USER.md) + +The core memory system. Two files in `~/.hermes/memories/`: + +| File | Purpose | Default Char Limit | +|------|---------|--------------------| +| `MEMORY.md` | Agent's personal notes — environment facts, project conventions, tool quirks, lessons learned | 2,200 chars (~800 tokens) | +| `USER.md` | User profile — name, preferences, communication style, pet peeves | 1,375 chars (~500 tokens) | + +**How it works:** + +- Loaded from disk at session start and injected into the system prompt as a frozen snapshot +- The agent uses the `memory` tool to add, replace, or remove entries during a session +- Mid-session writes go to disk immediately (durable) but do NOT update the system prompt — this preserves the LLM's prefix cache for performance +- The snapshot refreshes on the next session start +- Entries are delimited by `§` (section sign) and can be multiline + +**System prompt appearance:** + +``` +══════════════════════════════════════════════ +MEMORY (your personal notes) [67% — 1,474/2,200 chars] +══════════════════════════════════════════════ +User's project is a Rust web service at ~/code/myapi using Axum + SQLx +§ +This machine runs Ubuntu 22.04, has Docker and Podman installed +§ +User prefers concise responses, dislikes verbose explanations +``` + +**Memory tool actions:** + +- `add` — append a new entry (rejected if it would exceed the char limit) +- `replace` — find an entry by substring match and replace it +- `remove` — find an entry by substring match and delete it + +Substring matching means you only need a unique fragment of the entry, not the full text. If the fragment matches multiple entries, the tool returns an error asking for a more specific match. + +### 2. Session Search + +Cross-session conversation recall via SQLite FTS5 full-text search. + +- All CLI and messaging sessions are stored in `~/.hermes/state.db` +- The `session_search` tool finds relevant past conversations by keyword +- Top matching sessions are summarized by Gemini Flash (cheap, fast) before being returned to the main model +- Returns focused summaries, not raw transcripts + +**When to use session_search vs. memory:** + +| Feature | Persistent Memory | Session Search | +|---------|------------------|----------------| +| Capacity | ~3,575 chars total | Unlimited (all sessions) | +| Speed | Instant (in system prompt) | Requires search + LLM summarization | +| Use case | Key facts always in context | "What did we discuss about X last week?" | +| Management | Manually curated by the agent | Automatic — all sessions stored | +| Token cost | Fixed per session (~1,300 tokens) | On-demand (searched when needed) | + +**Rule of thumb:** Memory is for facts that should *always* be available. Session search is for recalling specific past conversations on demand. Don't save task progress or session outcomes to memory — use session_search to find those. + +### 3. Skills (Procedural Memory) + +Skills are reusable workflows stored as `SKILL.md` files in `~/.hermes/skills/` (and optionally external skill directories). + +- Organized by category: `skills/github/github-pr-workflow/SKILL.md` +- YAML frontmatter with name, description, version, platform restrictions +- Progressive disclosure: metadata shown in skill list, full content loaded on demand via `skill_view` +- The agent creates skills proactively after complex tasks (5+ tool calls) using the `skill_manage` tool +- Skills can be patched when found outdated — stale skills are a liability + +Skills are *not* injected into the system prompt by default. The agent sees a compact index of available skills and loads them on demand. This keeps the prompt lean while giving access to deep procedural knowledge. + +**Skills vs. Memory:** + +- **Memory:** compact facts ("User's project uses Go 1.22 with chi router") +- **Skills:** detailed procedures ("How to deploy the staging server: step 1, step 2, ...") + +### 4. External Memory Providers + +Optional plugins that add deeper, structured memory alongside the built-in system. Only one external provider can be active at a time. + +| Provider | Storage | Key Feature | +|----------|---------|-------------| +| Honcho | Cloud | Dialectic user modeling with semantic search | +| OpenViking | Self-hosted | Filesystem-style knowledge hierarchy | +| Mem0 | Cloud | Server-side LLM fact extraction | +| Hindsight | Cloud/Local | Knowledge graph with entity resolution | +| Holographic | Local SQLite | HRR algebraic reasoning + trust scoring | +| RetainDB | Cloud | Hybrid search with delta compression | +| ByteRover | Local/Cloud | Hierarchical knowledge tree with CLI | +| Supermemory | Cloud | Context fencing + session graph ingest | + +External providers run **alongside** built-in memory (never replacing it). They receive hooks for: +- System prompt injection (provider context) +- Pre-turn memory prefetch +- Post-turn conversation sync +- Session-end extraction +- Built-in memory write mirroring + +Setup: `hermes memory setup` or set `memory.provider` in `~/.hermes/config.yaml`. + +See `website/docs/user-guide/features/memory-providers.md` for full provider details. + +## How the Systems Interact + +``` +Session Start + | + +--> Load MEMORY.md + USER.md from disk --> frozen snapshot into system prompt + +--> Provider: system_prompt_block() --> injected into system prompt + +--> Skills index --> injected into system prompt (compact metadata only) + | + v +Each Turn + | + +--> Provider: prefetch(query) --> relevant recalled context + +--> Agent sees: system prompt (memory + provider context + skills index) + +--> Agent can call: memory tool, session_search tool, skill tools, provider tools + | + v +After Each Response + | + +--> Provider: sync_turn(user, assistant) --> persist conversation + | + v +Periodic (every N turns, default 10) + | + +--> Memory nudge: agent prompted to review and update memory + | + v +Session End / Compression + | + +--> Memory flush: agent saves important facts before context is discarded + +--> Provider: on_session_end(messages) --> final extraction + +--> Provider: on_pre_compress(messages) --> save insights before compression +``` + +## Best Practices + +### What to Save + +Save proactively — don't wait for the user to ask: + +- **User preferences:** "I prefer TypeScript over JavaScript" → `user` target +- **Corrections:** "Don't use sudo for Docker, I'm in the docker group" → `memory` target +- **Environment facts:** "This server runs Debian 12 with PostgreSQL 16" → `memory` target +- **Conventions:** "Project uses tabs, 120-char lines, Google docstrings" → `memory` target +- **Explicit requests:** "Remember that my API key rotation is monthly" → `memory` target + +### What NOT to Save + +- **Task progress or session outcomes** — use session_search to recall these +- **Trivially re-discoverable facts** — "Python 3.12 supports f-strings" (web search this) +- **Raw data dumps** — large code blocks, log files, data tables +- **Session-specific ephemera** — temporary file paths, one-off debugging context +- **Content already in SOUL.md or AGENTS.md** — those are already in context + +### Writing Good Entries + +Compact, information-dense entries work best: + +``` +# Good — packs multiple related facts +User runs macOS 14 Sonoma, uses Homebrew, has Docker Desktop and Podman. Shell: zsh. Editor: VS Code with Vim bindings. + +# Good — specific, actionable convention +Project ~/code/api uses Go 1.22, sqlc for DB, chi router. Tests: make test. CI: GitHub Actions. + +# Bad — too vague +User has a project. + +# Bad — too verbose +On January 5th, 2026, the user asked me to look at their project which is +located at ~/code/api. I discovered it uses Go version 1.22 and... +``` + +### Capacity Management + +When memory is above 80% capacity (visible in the system prompt header), consolidate before adding. Merge related entries into shorter, denser versions. The tool will reject additions that would exceed the limit — use `replace` to consolidate first. + +Priority order for what stays in memory: +1. User preferences and corrections (highest — prevents repeated steering) +2. Environment facts and project conventions +3. Tool quirks and workarounds +4. Lessons learned (lowest — can often be rediscovered) + +### Memory Nudge + +Every N turns (default: 10), the agent receives a nudge prompting it to review and update its memory. This is a lightweight prompt injected into the conversation — not a separate API call. The agent can choose to update memory or skip if nothing has changed. + +## Privacy and Data Locality + +**Built-in memory is fully local.** MEMORY.md and USER.md are plain text files in `~/.hermes/memories/`. No network calls are made in the memory read/write path. The memory tool scans entries for prompt injection and exfiltration patterns before accepting them. + +**Session search is local.** The SQLite database (`~/.hermes/state.db`) stays on disk. FTS5 search is a local operation. However, the summarization step uses Gemini Flash (via the auxiliary LLM client) — conversation snippets are sent to Google's API for summarization. If this is a concern, session_search can be disabled. + +**External providers may send data off-machine.** Cloud providers (Honcho, Mem0, RetainDB, Supermemory) send data to their respective APIs. Self-hosted providers (OpenViking, Hindsight local mode, Holographic, ByteRover local mode) keep everything on your machine. Check the provider's documentation for specifics. + +**Security scanning.** All content written to memory (via the `memory` tool) is scanned for: +- Prompt injection patterns ("ignore previous instructions", role hijacking, etc.) +- Credential exfiltration attempts (curl/wget with secrets, reading .env files) +- SSH backdoor patterns +- Invisible unicode characters (used for steganographic injection) + +Blocked content is rejected with a descriptive error message. + +## Configuration + +In `~/.hermes/config.yaml`: + +```yaml +memory: + # Enable/disable the two built-in memory stores + memory_enabled: true # MEMORY.md + user_profile_enabled: true # USER.md + + # Character limits (not tokens — model-independent) + memory_char_limit: 2200 # ~800 tokens at 2.75 chars/token + user_char_limit: 1375 # ~500 tokens at 2.75 chars/token + + # External memory provider (empty string = built-in only) + # Options: "honcho", "openviking", "mem0", "hindsight", + # "holographic", "retaindb", "byterover", "supermemory" + provider: "" +``` + +Additional settings are read from `run_agent.py` defaults: + +| Setting | Default | Description | +|---------|---------|-------------| +| `nudge_interval` | 10 | Turns between memory review nudges (0 = disabled) | +| `flush_min_turns` | 6 | Minimum user turns before memory flush on session end/compression (0 = never flush) | + +These are set under the `memory` key in config.yaml: + +```yaml +memory: + nudge_interval: 10 + flush_min_turns: 6 +``` + +### Disabling Memory + +To disable memory entirely, set both to false: + +```yaml +memory: + memory_enabled: false + user_profile_enabled: false +``` + +The `memory` tool will not appear in the tool list, and no memory blocks are injected into the system prompt. + +You can also disable memory per-invocation with `skip_memory=True` in the AIAgent constructor (used by cron jobs and flush agents). + +## File Locations + +``` +~/.hermes/ +├── memories/ +│ ├── MEMORY.md # Agent's persistent notes +│ ├── USER.md # User profile +│ ├── MEMORY.md.lock # File lock (auto-created) +│ └── USER.md.lock # File lock (auto-created) +├── state.db # SQLite session store (FTS5) +├── config.yaml # Memory config + provider selection +└── .env # API keys for external providers +``` + +All paths respect `$HERMES_HOME` — if you use Hermes profiles, each profile has its own isolated memory directory. + +## Troubleshooting + +### "Memory full" errors + +The tool returns an error when adding would exceed the character limit. The response includes current entries so the agent can consolidate. Fix by: +1. Replacing multiple related entries with one denser entry +2. Removing entries that are no longer relevant +3. Increasing `memory_char_limit` in config (at the cost of larger system prompts) + +### Stale memory entries + +If the agent seems to have outdated information: +- Check `~/.hermes/memories/MEMORY.md` directly — you can edit it by hand +- The frozen snapshot pattern means changes only take effect on the next session start +- If the agent wrote something wrong mid-session, it persists on disk but won't affect the current session's system prompt + +### Memory not appearing in system prompt + +- Verify `memory_enabled: true` in config.yaml +- Check that `~/.hermes/memories/MEMORY.md` exists and has content +- The file might be empty if all entries were removed — add entries with the `memory` tool + +### Session search returns no results + +- Session search requires sessions to be stored in `state.db` — new installations have no history +- FTS5 indexes are built automatically but may lag behind on very large databases +- The summarization step requires the auxiliary LLM client to be configured (API key for Gemini Flash) + +### Skill drift + +Skills that haven't been updated can become wrong or incomplete. The agent is prompted to patch skills when it finds them outdated during use (`skill_manage(action='patch')`). If you notice stale skills: +- Use `/skills` to browse and review installed skills +- Delete or update skills in `~/.hermes/skills/` directly +- The agent creates skills after complex tasks — review and prune periodically + +### Provider not activating + +- Run `hermes memory status` to check provider state +- Verify the provider plugin is installed in `~/.hermes/plugins/memory/` +- Check that required API keys are set in `~/.hermes/.env` +- Start a new session after changing provider config — existing sessions use the old provider + +### Concurrent write conflicts + +The memory tool uses file locking (`fcntl.flock`) and atomic file replacement (`os.replace`) to handle concurrent writes from multiple sessions. If you see corrupted memory files: +- Check for stale `.lock` files in `~/.hermes/memories/` +- Restart any hung Hermes processes +- The atomic write pattern means readers always see either the old or new file — never a partial write -- 2.43.0