timmy-config/docs/MEMORY_ARCHITECTURE.md

Memory Architecture

How Timmy remembers, recalls, and learns — without hallucinating.

Refs: Epic #367 | Sub-issues #368, #369, #370, #371, #372

Overview

Timmy's memory system uses a Memory Palace architecture — a structured, file-backed knowledge store organized into rooms and drawers. When faced with a recall question, the agent checks its palace before generating from scratch.

This document defines the retrieval order, storage layers, and data flow that make this work.

Retrieval Order (L0–L5)

When the agent receives a prompt that looks like a recall question ("what did we do?", "what's the status of X?"), the retrieval enforcer intercepts it and walks through layers in order:

Layer	Source	Question Answered	Short-circuits?
L0	identity.txt	Who am I? What are my mandates?	No (always loaded)
L1	Palace rooms/drawers	What do I know about this topic?	Yes, if hit
L2	Session scratchpad	What have I learned this session?	Yes, if hit
L3	Artifact retrieval (Gitea API)	Can I fetch the actual issue/file/log?	Yes, if hit
L4	Procedures/playbooks	Is there a documented way to do this?	Yes, if hit
L5	Free generation	(Only when L0–L4 are exhausted)	N/A

Key principle: The agent never reaches L5 (free generation) if any prior layer has relevant data. This eliminates hallucination for recall-style queries.

Storage Layout

~/.mempalace/
  identity.txt              # L0: Who I am, mandates, personality
  rooms/
    projects/
      timmy-config.md        # What I know about timmy-config
      hermes-agent.md        # What I know about hermes-agent
    people/
      alexander.md           # Working relationship context
    architecture/
      fleet.md               # Fleet system knowledge
      mempalace.md           # Self-knowledge about this system
  config/
    mempalace.yaml           # Palace configuration

~/.hermes/
  scratchpad/
    {session_id}.json        # L2: Ephemeral session context

Components

1. Memory Palace Skill (mempalace.py) — #368

Core data structures:

• PalaceRoom: A named collection of drawers (topics)
• Mempalace: The top-level palace with room management
• Factory constructors: for_issue_analysis(), for_health_check(), for_code_review()

2. Retrieval Enforcer (retrieval_enforcer.py) — #369

Middleware that intercepts recall-style prompts:

1. Detects recall patterns ("what did", "status of", "last time we")
2. Walks L0→L4 in order, short-circuiting on first hit
3. Only allows free generation (L5) when all layers return empty
4. Produces an honest fallback: "I don't have this in my memory palace."

3. Session Scratchpad (scratchpad.py) — #370

Ephemeral, session-scoped working memory:

• Write-append only during a session
• Entries have TTL (default: 1 hour)
• Queried at L2 in retrieval chain
• Never auto-promoted to palace

4. Memory Promotion — #371

Explicit promotion from scratchpad to palace:

• Agent must call promote_to_palace() with a reason
• Dedup check against target drawer
• Summary required (raw tool output never stored)
• Conflict detection when new memory contradicts existing

5. Wake-Up Protocol (wakeup.py) — #372

Boot sequence for new sessions:

Session Start
  │
  ├─ L0: Load identity.txt
  ├─ L1: Scan palace rooms for active context
  ├─ L1.5: Surface promoted memories from last session
  ├─ L2: Load surviving scratchpad entries
  │
  └─ Ready: agent knows who it is, what it was doing, what it learned

Data Flow

              ┌──────────────────┐
              │  User Prompt     │
              └────────┬─────────┘
                       │
              ┌────────┴─────────┐
              │ Recall Detector  │
              └────┬───────┬─────┘
                   │           │
            [recall]     [not recall]
                   │           │
          ┌───────┴────┐    ┌──┬─┴───────┐
          │ Retrieval  │    │ Normal Flow │
          │ Enforcer   │    └─────────────┘
          │ L0→L1→L2  │
          │ →L3→L4→L5│
          └──────┬─────┘
                 │
          ┌──────┴─────┐
          │  Response    │
          │ (grounded)  │
          └────────────┘

Anti-Patterns

Don't	Do Instead
Generate from vibes when palace has data	Check palace first (L1)
Auto-promote everything to palace	Require explicit promote_to_palace() with reason
Store raw API responses as memories	Summarize before storing
Hallucinate when palace is empty	Say "I don't have this in my memory palace"
Dump entire palace on wake-up	Selective loading based on session context

Status

Component	Issue	PR	Status
Skill port	#368	#374	In Review
Retrieval enforcer	#369	#374	In Review
Session scratchpad	#370	#374	In Review
Memory promotion	#371	—	Open
Wake-up protocol	#372	#374	In Review