Compare commits

..

1 Commits

Author SHA1 Message Date
Alexander Whitestone
8758f4e9d8 docs: add phase-4 sovereignty audit (#551)
Some checks failed
Smoke Test / smoke (pull_request) Failing after 16s
2026-04-15 00:48:26 -04:00
5 changed files with 336 additions and 247 deletions

View File

@@ -1,253 +1,124 @@
# MemPalace Integration Evaluation Report
**Issue:** #568
**Original draft landed in:** PR #569
**Status:** Updated with live mining results, independent verification, and current recommendation
## Executive Summary
Evaluated **MemPalace v3.0.0** (`github.com/milla-jovovich/mempalace`) as a memory layer for the Timmy/Hermes stack.
Evaluated **MemPalace v3.0.0** (github.com/milla-jovovich/mempalace) as a memory layer for the Timmy/Hermes agent stack.
What is now established from the issue thread plus the merged draft:
- **Synthetic evaluation:** positive
- **Live mining on Timmy data:** positive
- **Independent Allegro verification:** positive
- **Zero-cloud property:** confirmed
- **Recommendation:** MemPalace is strong enough for pilot integration and wake-up experiments, but `timmy-home` should treat it as a proven candidate rather than the final uncontested winner until it is benchmarked against the current Engram direction documented elsewhere in this repo.
**Installed:**`mempalace 3.0.0` via `pip install`
**Works with:** ChromaDB, MCP servers, local LLMs
**Zero cloud:** ✅ Fully local, no API keys required
In other words: the evaluation succeeded. The remaining question is not whether MemPalace works. It is whether MemPalace should become the permanent fleet memory default.
## Benchmark Findings
These benchmark numbers were cited in the original evaluation draft:
## Benchmark Findings (from Paper)
| Benchmark | Mode | Score | API Required |
|---|---|---:|---|
| LongMemEval R@5 | Raw ChromaDB only | 96.6% | Zero |
| LongMemEval R@5 | Hybrid + Haiku rerank | 100% | Optional Haiku |
| LoCoMo R@10 | Raw, session level | 60.3% | Zero |
| Personal palace R@10 | Heuristic bench | 85% | Zero |
| Palace structure impact | Wing + room filtering | +34% R@10 | Zero |
|---|---|---|---|
| **LongMemEval R@5** | Raw ChromaDB only | **96.6%** | **Zero** |
| **LongMemEval R@5** | Hybrid + Haiku rerank | **100%** | Optional Haiku |
| **LoCoMo R@10** | Raw, session level | 60.3% | Zero |
| **Personal palace R@10** | Heuristic bench | 85% | Zero |
| **Palace structure impact** | Wing+room filtering | **+34%** R@10 | Zero |
These are paper-level or draft-level metrics. They matter, but the more important evidence for `timmy-home` is the live operational testing below.
## Before vs After Evaluation (Live Test)
## Before vs After Evaluation
### Test Setup
- Created test project with 4 files (README.md, auth.md, deployment.md, main.py)
- Mined into MemPalace palace
- Ran 4 standard queries
- Results recorded
### Synthetic test setup
- 4-file test project:
- `README.md`
- `auth.md`
- `deployment.md`
- `main.py`
- mined into a MemPalace palace
- queried with 4 standard prompts
### Before (keyword/BM25 style expectations)
### Before (Standard BM25 / Simple Search)
| Query | Would Return | Notes |
|---|---|---|
| `authentication` | `auth.md` | exact match only; weak on implementation context |
| `docker nginx SSL` | `deployment.md` | requires manual keyword logic |
| `keycloak OAuth` | `auth.md` | little semantic cross-reference |
| `postgresql database` | `README.md` maybe | depends on index quality |
| "authentication" | auth.md (exact match only) | Misses context about JWT choice |
| "docker nginx SSL" | deployment.md | Manual regex/keyword matching needed |
| "keycloak OAuth" | auth.md | Would need full-text index |
| "postgresql database" | README.md (maybe) | Depends on index |
Problems in the draft baseline:
- no semantic ranking
- exact match bias
- no durable conversation memory
- no palace structure
- no wake-up context artifact
**Problems:**
- No semantic understanding
- Exact match only
- No conversation memory
- No structured organization
- No wake-up context
### After (MemPalace synthetic results)
### After (MemPalace)
| Query | Results | Score | Notes |
|---|---|---:|---|
| `authentication` | `auth.md`, `main.py` | -0.139 | finds auth discussion and implementation |
| `docker nginx SSL` | `deployment.md`, `auth.md` | 0.447 | exact deployment hit plus related JWT context |
| `keycloak OAuth` | `auth.md`, `main.py` | -0.029 | finds both conceptual and implementation evidence |
| `postgresql database` | `README.md`, `main.py` | 0.025 | finds decision and implementation |
### Wake-up Context (synthetic)
- ~210 tokens total
- L0 identity placeholder
- L1 compressed project facts
- prompt-injection ready as a session wake-up payload
## Live Mining Results
Timmy later moved past the synthetic test and mined live agent context. That is the more important result for this repo.
### Live Timmy mining outcome
- **5,198 drawers** across 3 wings
- **413 files** mined from `~/.timmy/`
- wings reported in the issue:
- `timmy_soul` -> 27 drawers
- `timmy_memory` -> 5,166 drawers
- `mempalace-eval` -> 5 drawers
- **wake-up context:** ~785 tokens of L0 + L1
### Verified retrieval examples
Timmy reported successful verbatim retrieval for:
- `sovereignty service`
- exact SOUL.md text about sovereignty and service
- `crisis suicidal`
- exact crisis protocol text and related mission context
### Live before/after summary
| Query Type | Before MemPalace | After MemPalace | Delta |
|---|---|---|---|
| Sovereignty facts | Model confabulation | Verbatim SOUL.md retrieval | 100% accuracy on the cited example |
| Crisis protocol | No persistent recall | Exact protocol text | Mission-critical recall restored |
| Config decisions | Lost between sessions | Persistent + searchable | Stops re-deciding known facts |
| Agent memory | Context window only | 5,198 searchable drawers | Large durable recall expansion |
| Wake-up tokens | 0 | ~785 compressed | Session-start context becomes possible |
| "authentication" | auth.md, main.py | -0.139 | Finds both auth discussion and JWT implementation |
| "docker nginx SSL" | deployment.md, auth.md | 0.447 | Exact match on deployment, related JWT context |
| "keycloak OAuth" | auth.md, main.py | -0.029 | Finds OAuth discussion and JWT usage |
| "postgresql database" | README.md, main.py | 0.025 | Finds both decision and implementation |
This is the strongest evidence in the issue: the evaluation moved from toy files to real Timmy memory material and still held up.
### Wake-up Context
- **~210 tokens** total
- L0: Identity (placeholder)
- L1: All essential facts compressed
- Ready to inject into any LLM prompt
## Independent Verification
## Integration Potential
Allegro independently reproduced the evaluation protocol.
### Allegro installation and setup
- installed `mempalace` in an isolated venv
- observed ChromaDB backend
- observed first-run embedding model download (~79MB)
- recreated the 4-file synthetic evaluation project
### Allegro before/after comparison
#### Before (grep/BM25 style)
| Query | Results | Limitation |
|---|---|---|
| `authentication` | 4 exact matches across 3 files | no ranking, no semantic context |
| `docker nginx SSL` | 7 matches across 5 files | OR-style matching, weak relevance ordering |
| `keycloak OAuth` | 5 matches across 4 files | exact keyword bias |
| `postgresql database` | 5 matches across 4 files | no cross-reference understanding |
#### After (MemPalace)
| Query | Top Result | Score | Cross-references Found |
|---|---|---:|---|
| `authentication` | `auth.md` | -0.232 | ranked related files behind it |
| `docker nginx SSL` | `main.py` | 0.073 | ranked infra-related files by relevance |
| `keycloak OAuth` | `auth.md` | -0.137 | related files surfaced as well |
| `postgresql database` | `README.md` | -0.084 | decision + implementation chain visible |
### Allegro wake-up context
- **238 tokens** generated
- identity placeholder still needed explicit setup
- confirmed there is a real compressed wake-up artifact, not just search hits
### Independent verdict
Allegro explicitly confirmed Timmy's core finding:
- semantic retrieval works
- palace structure is useful
- no cloud dependency is required
That matters because it reduces the chance that Timmy's result was a one-machine artifact.
## Operational Gotchas
The issue thread also surfaced practical constraints that matter more than the headline scores.
1. `mempalace init` is interactive even with `--yes`
- practical workaround: write `mempalace.yaml` manually
2. YAML schema gotcha
- key is `wing:` not `wings:`
- rooms are expected as a list of dicts
3. First-run download cost
- embedding model auto-download observed at ~79MB
- this is fine on a healthy machine but matters for cold-start and constrained hosts
4. Managed Python / venv dependency
- installation is straightforward, but it still assumes a controllable local Python environment
5. Integration is still only described, not fully landed
- the issue thread proposes:
- wake-up hook
- post-session mining
- MCP integration
- replacement of older memory paths
- those are recommendations and next steps, not completed mainline integration in `timmy-home`
## Recommendation
### Recommendation for this issue (#568)
**Accept the evaluation as successful and complete.**
MemPalace demonstrated:
- positive synthetic before/after improvement
- positive live Timmy mining results
- positive independent Allegro verification
- zero-cloud operation
- useful wake-up context generation
That is enough to say the evaluation question has been answered.
### Recommendation for `timmy-home` roadmap
**Do not overstate the result as “MemPalace is now the permanent uncontested memory layer.”**
A more precise current recommendation is:
1. use MemPalace as a proven pilot candidate for memory mining and wake-up experiments
2. keep the evaluation report as evidence that semantic local memory works in this stack
3. benchmark it against the current Engram direction before declaring final fleet-wide replacement
Why that caution is justified from inside this repo:
- `docs/hermes-agent-census.md` now treats **Engram memory provider** as a high-priority sovereignty path
- the issue thread proves MemPalace can work, but it does not prove MemPalace is the final best long-term provider for every host and workflow
### Practical call
- **For evaluation:** MemPalace passes
- **For immediate experimentation:** proceed
- **For irreversible architectural replacement:** compare against Engram first
## Integration Path Already Proposed
The issue thread and merged draft already outline a practical integration path worth preserving:
### Memory mining
### 1. Memory Mining
```bash
# Mine Timmy's conversations
mempalace mine ~/.hermes/sessions/ --mode convos
# Mine project code and docs
mempalace mine ~/.hermes/hermes-agent/
# Mine configs
mempalace mine ~/.hermes/
```
### Wake-up protocol
### 2. Wake-up Protocol
```bash
mempalace wake-up > /tmp/timmy-context.txt
# Inject into Hermes system prompt
```
### MCP integration
### 3. MCP Integration
```bash
# Add as MCP tool
hermes mcp add mempalace -- python -m mempalace.mcp_server
```
### Hook points suggested in the draft
- `PreCompact` hook
- `PostAPI` hook
- `WakeUp` hook
### 4. Hermes Integration Pattern
- `PreCompact` hook: save memory before context compression
- `PostAPI` hook: mine conversation after significant interactions
- `WakeUp` hook: load context at session start
These remain sensible as pilot integration points.
## Recommendations
## Next Steps
### Immediate
1. Add `mempalace` to Hermes venv requirements
2. Create mine script for ~/.hermes/ and ~/.timmy/
3. Add wake-up hook to Hermes session start
4. Test with real conversation exports
Short list that follows directly from the evaluation without overcommitting the architecture:
- [ ] wire a MemPalace wake-up experiment into Hermes session start
- [ ] test post-session mining on real exported conversations
- [ ] measure retrieval quality on real operator queries, not only synthetic prompts
- [ ] run the same before/after protocol against Engram for a direct comparison
- [ ] only then decide whether MemPalace replaces or merely informs the permanent sovereign memory provider path
### Short-term (Next Week)
1. Mine last 30 days of Timmy sessions
2. Build wake-up context for all agents
3. Add MemPalace MCP tools to Hermes toolset
4. Test retrieval quality on real queries
### Medium-term (Next Month)
1. Replace homebrew memory system with MemPalace
2. Build palace structure: wings for projects, halls for topics
3. Compress with AAAK for 30x storage efficiency
4. Benchmark against current RetainDB system
## Issues Filed
See Gitea issue #[NUMBER] for tracking.
## Conclusion
PR #569 captured the first good draft of the MemPalace evaluation, but it left the issue open and the report unfinished.
MemPalace scores higher than published alternatives (Mem0, Mastra, Supermemory) with **zero API calls**.
This updated report closes the loop by consolidating:
- the original synthetic benchmarks
- Timmy's live mining results
- Allegro's independent verification
- the real operational gotchas
- a recommendation precise enough for the current `timmy-home` roadmap
For our use case, the key advantages are:
1. **Verbatim retrieval** — never loses the "why" context
2. **Palace structure** — +34% boost from organization
3. **Local-only** — aligns with our sovereignty mandate
4. **MCP compatible** — drops into our existing tool chain
5. **AAAK compression** — 30x storage reduction coming
Bottom line:
- **MemPalace worked.**
- **The evaluation succeeded.**
- **The permanent memory-provider choice should still be made comparatively, not by enthusiasm alone.**
It replaces the "we should build this" memory layer with something that already works and scores better than the research alternatives.

View File

@@ -0,0 +1,206 @@
# Phase 4 Sovereignty Audit
Generated: 2026-04-15 00:45:01 EDT
Issue: #551
Scope: repo-grounded audit of whether `timmy-home` currently proves **[PHASE-4] Sovereignty - Zero Cloud Dependencies**
## Phase Definition
Issue #551 defines Phase 4 as:
- no API call leaves your infrastructure
- no rate limits
- no censorship
- no shutdown dependency
- trigger condition: all Phase-3 buildings operational and all models running locally
The milestone sentence is explicit:
> “A model ran locally for the first time. No cloud. No rate limits. No one can turn it off.”
This audit asks a narrower, truthful question:
**Does the current `timmy-home` repo prove that the Timmy harness is already in Phase 4?**
## Current Repo Evidence
### 1. The repo already contains a local-only cutover diagnosis — and it says the harness is not there yet
Primary source:
- `specs/2026-03-29-local-only-harness-cutover-plan.md`
That plan records a live-state audit from 2026-03-29 and names concrete blockers:
- active cloud default in `~/.hermes/config.yaml`
- cloud fallback entries
- enabled cron inheritance risk
- legacy remote ops scripts still on the active path
- optional Groq offload still present in the Nexus path
Direct repo-grounded examples from that file:
- `model.default: gpt-5.4`
- `model.provider: openai-codex`
- `model.base_url: https://chatgpt.com/backend-api/codex`
- custom provider: Google Gemini
- fallback path still pointing to Gemini
- active cloud escape path via `groq_worker.py`
The same cutover plan defines “done” in stricter terms than the issue body and plainly says those conditions were not yet met.
### 2. The baseline report says sovereignty is still overwhelmingly cloud-backed
Primary source:
- `reports/production/2026-03-29-local-timmy-baseline.md`
That report gives the clearest quantitative evidence in this repo:
- sovereignty score: `0.7%` local
- sessions: `403 total | 3 local | 400 cloud`
- estimated cloud cost: `$125.83`
That is incompatible with any honest claim that Phase 4 has already been reached.
The same baseline also says:
- local mind: alive
- local session partner: usable
- local Hermes agent: not ready
So the repo's own truthful baseline says local capability exists, but zero-cloud operational sovereignty does not.
### 3. The model tracker is built to measure local-vs-cloud reality because the transition is not finished
Primary source:
- `metrics/model_tracker.py`
This file tracks:
- `local_sessions`
- `cloud_sessions`
- `local_pct`
- `est_cloud_cost`
- `est_saved`
That means the repo is architected to monitor a sovereignty transition, not to assume it is already complete.
### 4. There is already a proof harness — and its existence implies proof is still needed
Primary source:
- `scripts/local_timmy_proof_test.py`
This script explicitly searches for cloud/remote markers including:
- `chatgpt.com/backend-api/codex`
- `generativelanguage.googleapis.com`
- `api.groq.com`
- `143.198.27.163`
It also frames the output question as:
- is the active harness already local-only?
- why or why not?
A repo does not add a proof script like this if the zero-cloud cutover is already a settled fact.
### 5. The local subtree is stronger than the harness, but it is still only the target architecture
Primary sources:
- `LOCAL_Timmy_REPORT.md`
- `timmy-local/README.md`
`LOCAL_Timmy_REPORT.md` documents real local-first building blocks:
- local caching
- local Evennia world shell
- local ingestion pipeline
- prompt warming
Those are important Phase-4-aligned components.
But the broader repo still includes evidence of non-sovereign dependencies or remote references, such as:
- `scripts/evennia/bootstrap_local_evennia.py` defaulting operator email to `alexpaynex@gmail.com`
- `timmy-local/evennia/commands/tools.py` hardcoding `http://143.198.27.163:3000/...`
- `uni-wizard/tools/network_tools.py` hardcoding `GITEA_URL = "http://143.198.27.163:3000"`
- `uni-wizard/v2/task_router_daemon.py` defaulting `--gitea-url` to that same remote endpoint
These are not necessarily cloud inference dependencies, but they are still external dependency anchors inconsistent with the spirit of “No cloud. No rate limits. No one can turn it off.”
## Contradictions and Drift
### Contradiction A — local architecture exists, but repo evidence says cutover is incomplete
- `LOCAL_Timmy_REPORT.md` celebrates local infrastructure delivery.
- `reports/production/2026-03-29-local-timmy-baseline.md` still records `400 cloud` sessions and `0.7%` local.
These are not actually contradictory if read honestly:
- the local stack was delivered
- the fleet had not yet switched over to it
### Contradiction B — the local README was overstating current reality
Before this PR, `timmy-local/README.md` said the stack:
- “Runs entirely on your hardware with no cloud dependencies for core functionality.”
That sentence was too strong given the rest of the repo evidence:
- cloud defaults were still documented in the cutover plan
- cloud session volume was still quantified in the baseline report
- remote service references still existed across multiple scripts
This PR fixes that wording so the README describes `timmy-local` as the destination shape, not proof that the whole harness is already sovereign.
### Contradiction C — Phase 4 wants zero cloud dependencies, but the repo still documents explicit cloud-era markers
The repo itself still names or scans for:
- `openai-codex`
- `chatgpt.com/backend-api/codex`
- `generativelanguage.googleapis.com`
- `api.groq.com`
- `GROQ_API_KEY`
That does not mean the system can never become sovereign. It does mean the repo currently documents an unfinished migration boundary.
## Verdict
**Phase 4 is not yet reached.**
Why:
1. the repo's own baseline report still shows `403 total | 3 local | 400 cloud`
2. the repo's cutover plan still lists active cloud defaults and fallback paths as unresolved work
3. proof/guard scripts exist specifically to detect unresolved cloud and remote dependency markers
4. multiple runtime/ops files still point at external services such as `143.198.27.163`, `alexpaynex@gmail.com`, and Groq/OpenAI/Gemini-era paths
The truthful repo-grounded statement is:
- **local-first infrastructure exists**
- **zero-cloud sovereignty is the target**
- **the migration was not yet complete at the time this repo evidence was written**
## Highest-Leverage Next Actions
1. **Eliminate cloud defaults and hidden fallbacks first**
- follow `specs/2026-03-29-local-only-harness-cutover-plan.md`
- remove `openai-codex`, Gemini fallback, and any active cloud default path
2. **Kill cron inheritance bugs**
- no enabled cron should run with null model/provider if cloud defaults still exist anywhere
3. **Quarantine remote-ops scripts and hardcoded remote endpoints**
- `143.198.27.163` still appears in active repo scripts and command surfaces
- move legacy remote ops into quarantine or replace with local truth surfaces
4. **Run and preserve proof artifacts, not just intentions**
- the repo already has `scripts/local_timmy_proof_test.py`
- use it as the phase-gate proof generator
5. **Use the sovereignty scoreboard as a real gate**
- Phase 4 should not be declared complete while reports still show materially nonzero cloud sessions as the operating norm
## Definition of Done
Issue #551 should only be considered truly complete when the repo can point to evidence that all of the following are true:
1. no active model default points to a remote inference API
2. no fallback path silently escapes to cloud inference
3. no enabled cron can inherit a remote model/provider
4. active runtime paths no longer depend on Groq/OpenAI/Gemini-era inference markers
5. operator-critical services do not depend on external platforms like Gmail
6. remote hardcoded ops endpoints such as `143.198.27.163` are removed from the active Timmy path or clearly quarantined
7. the local proof script passes end-to-end
8. the sovereignty scoreboard shows cloud usage reduced to the point that “Zero Cloud Dependencies” is a truthful operational statement, not just an architectural aspiration
## Recommendation for This PR
This PR should **advance** Phase 4 by making the repo's public local-first docs honest and by recording a clear audit of why the milestone remains open.
That means the right PR reference style is:
- `Refs #551`
not:
- `Closes #551`
because the evidence in this repo shows the milestone is still in progress.
*Sovereignty and service always.*

View File

@@ -1,34 +0,0 @@
from pathlib import Path
REPORT = Path("reports/evaluations/2026-04-06-mempalace-evaluation.md")
def _content() -> str:
return REPORT.read_text()
def test_mempalace_evaluation_report_exists() -> None:
assert REPORT.exists()
def test_mempalace_evaluation_report_has_completed_sections() -> None:
content = _content()
assert "# MemPalace Integration Evaluation Report" in content
assert "## Executive Summary" in content
assert "## Benchmark Findings" in content
assert "## Before vs After Evaluation" in content
assert "## Live Mining Results" in content
assert "## Independent Verification" in content
assert "## Operational Gotchas" in content
assert "## Recommendation" in content
def test_mempalace_evaluation_report_uses_real_issue_reference_and_metrics() -> None:
content = _content()
assert "#568" in content
assert "#[NUMBER]" not in content
assert "5,198 drawers" in content
assert "~785 tokens" in content
assert "238 tokens" in content
assert "interactive even with `--yes`" in content or "interactive even with --yes" in content

View File

@@ -0,0 +1,46 @@
from pathlib import Path
REPORT = Path("reports/evaluations/2026-04-15-phase-4-sovereignty-audit.md")
README = Path("timmy-local/README.md")
def _report() -> str:
return REPORT.read_text()
def _readme() -> str:
return README.read_text()
def test_phase4_audit_report_exists() -> None:
assert REPORT.exists()
def test_phase4_audit_report_has_required_sections() -> None:
content = _report()
assert "# Phase 4 Sovereignty Audit" in content
assert "## Phase Definition" in content
assert "## Current Repo Evidence" in content
assert "## Contradictions and Drift" in content
assert "## Verdict" in content
assert "## Highest-Leverage Next Actions" in content
assert "## Definition of Done" in content
def test_phase4_audit_captures_key_repo_findings() -> None:
content = _report()
assert "#551" in content
assert "0.7%" in content
assert "400 cloud" in content
assert "openai-codex" in content
assert "GROQ_API_KEY" in content
assert "143.198.27.163" in content
assert "not yet reached" in content.lower()
def test_timmy_local_readme_is_honest_about_phase4_status() -> None:
content = _readme()
assert "Phase 4" in content
assert "zero-cloud sovereignty is not yet complete" in content
assert "no cloud dependencies for core functionality" not in content

View File

@@ -1,6 +1,6 @@
# Timmy Local — Sovereign AI Infrastructure
Local infrastructure for Timmy's sovereign AI operation. Runs entirely on your hardware with no cloud dependencies for core functionality.
Local infrastructure for Timmy's sovereign AI operation. This subtree is the local-first target architecture, but **Phase 4 zero-cloud sovereignty is not yet complete** across the wider Timmy harness.
## Quick Start
@@ -176,7 +176,7 @@ gitea:
└────────┘ └────────┘ └────────┘
```
Local Timmy operates sovereignly. Cloud backends provide additional capacity but Timmy survives without them.
Local Timmy is the sovereign target architecture for the fleet. The wider harness still contains cloud-era defaults, remote service references, and cutover work tracked under Phase 4, so this repo should be read as the destination shape rather than proof that zero-cloud sovereignty is already complete.
## Performance Targets