[Core] Three-Tier Memory System — Working/Episodic/Semantic with Retrieval #873

New Issue

perplexity · 2026-03-21T23:40:05Z

perplexity commented

2026-03-21 23:40:05 +00:00

Why This Is High Leverage

Every research reference (Smallville, CHIM, the feasibility guide, the V2 roadmap) converges on the same memory architecture. Timmy already has SQLite storage and a memory system, but it's not structured into the three tiers that make long-running autonomous agents actually work. This is the difference between an agent that forgets everything after context window overflow and one that accumulates genuine experience.

Scope

Implement the three-tier memory system validated by Stanford's Smallville and refined by CHIM (~$5/month on 70B):

Working Memory

Current context window: active quest, nearby NPCs, inventory, last 5-10 actions, health/stats
Managed as a sliding window, always available to the LLM

Episodic Memory

Chronological log of key events with embeddings for retrieval
Scored by recency + importance + relevance (Smallville's retrieval formula)
Periodically summarized (compress old episodes into summaries)

Semantic Memory

Static knowledge: game lore, quest walkthroughs, NPC locations, item databases
Retrieved via RAG (sentence-transformers for embeddings, SQLite for storage)

Requirements

src/timmy/memory/working.py — Sliding window context manager
src/timmy/memory/episodic.py — Event log with embedding-based retrieval, importance scoring
src/timmy/memory/semantic.py — RAG-queryable knowledge store
src/timmy/memory/retriever.py — Unified retrieval: query all three tiers, rank, return top-k
Migration from existing memory system (preserve accumulated data)
Summarization job: compress episodes older than N hours into summaries

Acceptance Criteria

Store an event, retrieve it by semantic similarity
Working memory correctly maintains sliding window
Retrieval ranks recent+important events higher than old+mundane ones
Existing memory data migrated without loss

Assignee: Timmy

## Why This Is High Leverage Every research reference (Smallville, CHIM, the feasibility guide, the V2 roadmap) converges on the same memory architecture. Timmy already has SQLite storage and a memory system, but it's not structured into the three tiers that make long-running autonomous agents actually work. This is the difference between an agent that forgets everything after context window overflow and one that accumulates genuine experience. ## Scope Implement the three-tier memory system validated by Stanford's Smallville and refined by CHIM (~$5/month on 70B): ### Working Memory - Current context window: active quest, nearby NPCs, inventory, last 5-10 actions, health/stats - Managed as a sliding window, always available to the LLM ### Episodic Memory - Chronological log of key events with embeddings for retrieval - Scored by recency + importance + relevance (Smallville's retrieval formula) - Periodically summarized (compress old episodes into summaries) ### Semantic Memory - Static knowledge: game lore, quest walkthroughs, NPC locations, item databases - Retrieved via RAG (sentence-transformers for embeddings, SQLite for storage) ## Requirements - [ ] `src/timmy/memory/working.py` — Sliding window context manager - [ ] `src/timmy/memory/episodic.py` — Event log with embedding-based retrieval, importance scoring - [ ] `src/timmy/memory/semantic.py` — RAG-queryable knowledge store - [ ] `src/timmy/memory/retriever.py` — Unified retrieval: query all three tiers, rank, return top-k - [ ] Migration from existing memory system (preserve accumulated data) - [ ] Summarization job: compress episodes older than N hours into summaries ## Acceptance Criteria - Store an event, retrieve it by semantic similarity - Working memory correctly maintains sliding window - Retrieval ranks recent+important events higher than old+mundane ones - Existing memory data migrated without loss ## Assignee: Timmy

perplexity referenced this issue

2026-03-22 13:00:27 +00:00

[Study] State-of-the-Art Open Source for Sovereign Creative AI Agents — Q1 2026 Landscape Survey #903

perplexity referenced this issue

2026-03-22 13:00:55 +00:00

[Study] State-of-the-Art Open Source for Sovereign Creative AI Agents — Q1 2026 Landscape Survey #903

perplexity referenced this issue

2026-03-22 13:04:23 +00:00

[Study] Autoresearch Integration Proposal v2 — Karpathy's Self-Improvement Loop for Timmy Time #904

perplexity referenced this issue

2026-03-22 13:06:04 +00:00

[Autoresearch H1] Refactor autoresearch.py → SystemExperiment Class #906