docs: finalize MemPalace evaluation report (#568)

genomes/burn-fleet-GENOME.md

@@ -1,476 +0,0 @@
-# GENOME.md: burn-fleet
-
-**Generated:** 2026-04-15
-**Repo:** Timmy_Foundation/burn-fleet
-**Purpose:** Laned tmux dispatcher for sovereign burn operations across Mac and Allegro
-**Analyzed commit:** `2d4d9ab`
-**Size:** 5 top-level source/config files + README | 985 total lines (`fleet-dispatch.py` 320, `fleet-christen.py` 205, `fleet-status.py` 143, `fleet-launch.sh` 126, `fleet-spec.json` 98, `README.md` 93)
-
----
-
-## Project Overview
-
-`burn-fleet` is a compact control-plane repo for the Hundred-Pane Fleet.
-Its job is not model inference itself. Its job is to shape where inference runs, which panes wake up, which repos route to which windows, and how work is fanned out across Mac and VPS workers.
-
-The repo turns a narrative naming scheme into executable infrastructure:
-- Mac runs the local session (`BURN`) with windows like `CRUCIBLE`, `GNOMES`, `LOOM`, `FOUNDRY`, `WARD`, `COUNCIL`
-- Allegro runs a remote session (`BURN`) with windows like `FORGE`, `ANVIL`, `CRUCIBLE-2`, `SENTINEL`
-- `fleet-spec.json` is the single source of truth for pane counts, lanes, sublanes, glyphs, and names
-- `fleet-launch.sh` materializes the tmux topology
-- `fleet-christen.py` boots `hermes chat --yolo` in each pane and pushes identity prompts
-- `fleet-dispatch.py` consumes Gitea issues, maps repos to windows through `MAC_ROUTE` and `ALLEGRO_ROUTE`, and sends `/queue` work into the right panes
-- `fleet-status.py` inspects pane output and reports fleet health
-
-The repo is small, but it sits on a high-blast-radius operational seam:
-- it controls 100+ panes
-- it writes to live tmux sessions
-- it comments on live Gitea issues
-- it depends on SSH reachability to the VPS
-- it is effectively a narrative infrastructure orchestrator
-
-This means the right way to read it is as a dispatch kernel, not just a set of scripts.
-
----
-
-## Architecture
-
-```mermaid
-graph TD
-    A[fleet-spec.json] --> B[fleet-launch.sh]
-    A --> C[fleet-christen.py]
-    A --> D[fleet-dispatch.py]
-    A --> E[fleet-status.py]
-
-    B --> F[tmux session BURN on Mac]
-    B --> G[tmux session BURN on Allegro over SSH]
-
-    C --> F
-    C --> G
-    C --> H[hermes chat --yolo in every pane]
-    H --> I[identity + lane prompt]
-
-    J[Gitea issues on forge.alexanderwhitestone.com] --> D
-    D --> K[MAC_ROUTE]
-    D --> L[ALLEGRO_ROUTE]
-    D --> M[/queue prompt generation]
-    M --> F
-    M --> G
-    D --> N[comment_on_issue]
-    N --> J
-    D --> O[dispatch-state.json]
-
-    E --> F
-    E --> G
-    E --> P[get_pane_status]
-    P --> Q[fleet health summary]
-```
-
-### Structural reading
-
-The repo has one real architecture pattern:
-1. declarative topology in `fleet-spec.json`
-2. imperative realization scripts that consume that topology
-3. runtime state in `dispatch-state.json`
-4. external side effects in tmux, SSH, and Gitea
-
-That makes `fleet-spec.json` the nucleus and the four scripts adapters around it.
-
----
-
-## Entry Points
-
-| Entry point | Type | Role |
-|-------------|------|------|
-| `fleet-launch.sh [mac|allegro|both]` | Shell CLI | Creates tmux sessions and pane layouts from `fleet-spec.json` |
-| `python3 fleet-christen.py [mac|allegro|both]` | Python CLI | Starts Hermes workers and injects identity/lane prompts |
-| `python3 fleet-dispatch.py [--cycles N] [--interval S] [--machine mac|allegro|both]` | Python CLI | Pulls open Gitea issues, routes them, comments on issues, persists `dispatch-state.json` |
-| `python3 fleet-status.py [--machine mac|allegro|both]` | Python CLI | Samples pane output and reports working/idle/error/dead state |
-| `README.md` quick start | Human runbook | Documents the intended operator flow from launch to christening to dispatch to status |
-
-### Hidden operational entry points
-
-These are not CLI entry points, but they matter for behavior:
-- `MAC_ROUTE` in `fleet-dispatch.py`
-- `ALLEGRO_ROUTE` in `fleet-dispatch.py`
-- `SKIP_LABELS` and `INACTIVE` filtering in `fleet-dispatch.py`
-- `send_to_pane()` as the effectful dispatch primitive
-- `comment_on_issue()` as the visible acknowledgement primitive
-- `get_pane_status()` in `fleet-status.py` as the fleet health classifier
-
----
-
-## Data Flow
-
-### 1. Topology creation
-
-`fleet-launch.sh` reads `fleet-spec.json`, parses each window's pane count, and creates the tmux layout.
-
-Flow:
-- load spec file path from `SCRIPT_DIR/fleet-spec.json`
-- parse `machines.mac.windows` or `machines.allegro.windows`
-- create `BURN` session locally or remotely
-- create first window, then split panes, then create remaining windows
-- continuously tile after splits
-
-This script is layout-only. It does not launch Hermes.
-
-### 2. Agent wake-up / identity seeding
-
-`fleet-christen.py` reads the same `fleet-spec.json` and sends `hermes chat --yolo` into each pane.
-After a fixed wait window, it sends a second `/queue` identity message containing:
-- glyph
-- pane name
-- machine name
-- window name
-- pane number
-- sublane
-- sovereign operating instructions
-
-That identity message is the bridge from infrastructure to narrative.
-The worker is not just launched; it is assigned a mythic/operator identity with a lane.
-
-### 3. Issue harvest and lane dispatch
-
-`fleet-dispatch.py` is the center of the runtime.
-
-Flow:
-- load `fleet-spec.json`
-- load `dispatch-state.json`
-- load Gitea token
-- fetch open issues per repo with `requests`
-- filter PRs, meta labels, and previously dispatched issues
-- build a candidate pool per machine/window
-- assign issues pane-by-pane
-- call `send_to_pane()` to inject `/queue ...`
-- call `comment_on_issue()` to leave a visible burn dispatch comment
-- persist the issue assignment into `dispatch-state.json`
-
-Important: the data flow is not issue -> worker directly.
-It is:
-issue -> repo route table -> window -> pane -> `/queue` prompt -> worker.
-
-### 4. Health sampling
-
-`fleet-status.py` runs the inverse direction.
-It samples pane output through `tmux capture-pane` locally or over SSH and classifies the last visible signal as:
-- `working`
-- `idle`
-- `error`
-- `dead`
-
-It then summarizes by window, machine, and global fleet totals.
-
-### 5. Runtime state persistence
-
-`dispatch-state.json` is not checked in, but it is the only persistent memory of what the dispatcher already assigned.
-That means the runtime depends on a local mutable file rather than a centralized dispatch ledger.
-
----
-
-## Key Abstractions
-
-### 1. `fleet-spec.json`
-
-This is the primary abstraction in the repo.
-It encodes:
-- machine identity (`mac`, `allegro`)
-- host / SSH details
-- hardware metadata (`cores`, `ram_gb`)
-- tmux session names
-- default model/provider metadata
-- windows with `panes`, `lane`, `sublanes`, `glyphs`, `names`
-
-Everything else in the repo interprets this document.
-If the spec drifts from the route tables or runtime assumptions, the fleet silently degrades.
-
-### 2. Route tables: `MAC_ROUTE` and `ALLEGRO_ROUTE`
-
-These tables are the repo's second control nucleus.
-They map repo names to windows.
-This is how `timmy-home`, `the-nexus`, `the-door`, `fleet-ops`, and `the-beacon` land in different operational lanes.
-
-This split means routing logic is duplicated:
-- once in the topology spec
-- once in Python route dictionaries
-
-That duplication is one of the most important maintainability risks in the repo.
-
-### 3. Pane effect primitive: `send_to_pane()`
-
-`send_to_pane()` is the real actuator.
-It turns a dispatch decision into a tmux `send-keys` side effect.
-It handles both:
-- local tmux injection
-- remote SSH + tmux injection
-
-Everything operationally dangerous funnels through this function.
-It is therefore a critical path even though the repo has no tests around it.
-
-### 4. Issue acknowledgement primitive: `comment_on_issue()`
-
-This is the repo's social trace primitive.
-It posts a burn dispatch comment back to the issue so humans can see that the fleet claimed it.
-This is the visible heartbeat of autonomous dispatch.
-
-### 5. Runtime memory: `dispatch-state.json`
-
-This file is the anti-duplication ledger for dispatch cycles.
-Without it, the dispatcher would keep recycling the same issues every pass.
-Because it is local-file state instead of centralized state, machine locality matters.
-
-### 6. Health classifier: `get_pane_status()`
-
-`fleet-status.py` does not know the true worker state.
-It infers state from captured pane output using string heuristics.
-So `get_pane_status()` is effectively a lightweight log classifier.
-Its correctness depends on fragile output pattern matching.
-
----
-
-## API Surface
-
-The repo exposes CLI-level APIs rather than import-oriented libraries.
-
-### Shell API
-
-`fleet-launch.sh`
-- `./fleet-launch.sh mac`
-- `./fleet-launch.sh allegro`
-- `./fleet-launch.sh both`
-
-### Python CLIs
-
-`fleet-christen.py`
-- `python3 fleet-christen.py mac`
-- `python3 fleet-christen.py allegro`
-- `python3 fleet-christen.py both`
-
-`fleet-dispatch.py`
-- `python3 fleet-dispatch.py`
-- `python3 fleet-dispatch.py --cycles 10 --interval 60`
-- `python3 fleet-dispatch.py --machine mac`
-
-`fleet-status.py`
-- `python3 fleet-status.py`
-- `python3 fleet-status.py --machine allegro`
-
-### Internal function surface worth naming explicitly
-
-`fleet-launch.sh`
-- `parse_spec()`
-- `launch_local()`
-- `launch_remote()`
-
-`fleet-christen.py`
-- `send_keys()`
-- `christen_window()`
-- `christen_machine()`
-- `christen_remote()`
-
-`fleet-dispatch.py`
-- `load_token()`
-- `load_spec()`
-- `load_state()`
-- `save_state()`
-- `get_issues()`
-- `send_to_pane()`
-- `comment_on_issue()`
-- `build_prompt()`
-- `dispatch_cycle()`
-- `dispatch_council()`
-
-`fleet-status.py`
-- `get_pane_status()`
-- `check_machine()`
-
-These are the true API surface for future hardening and testing.
-
----
-
-## Test Coverage Gaps
-
-### Current state
-
-Grounded from the pipeline dry run on `/tmp/burn-fleet-genome`:
-- 0% estimated coverage
-- untested modules called out by pipeline: `fleet-christen`, `fleet-dispatch`, `fleet-status`
-- no checked-in automated test suite
-
-### Critical paths with no tests
-
-1. `send_to_pane()`
-   - local tmux command construction
-   - remote SSH command construction
-   - escaping of issue titles and prompts
-   - failure handling when tmux or SSH fails
-
-2. `comment_on_issue()`
-   - verifies Gitea comment formatting
-   - verifies non-200 responses do not silently disappear
-
-3. `get_issues()`
-   - PR filtering
-   - `SKIP_LABELS` filtering
-   - title-based meta filtering
-   - robustness when Gitea returns malformed or partial issue objects
-
-4. `dispatch_cycle()`
-   - correct pooling by window
-   - deduplication via `dispatch-state.json`
-   - pane recycling behavior
-   - correctness when one repo has zero issues and another has many
-
-5. `get_pane_status()`
-   - classification heuristics for working/idle/error/dead
-   - false positives from incidental strings like `error` in normal output
-
-6. `fleet-launch.sh`
-   - parse correctness for pane counts
-   - layout creation behavior across first vs later windows
-   - remote script generation for Allegro
-
-### Missing tests to generate next in the real target repo
-
-If the goal is to harden `burn-fleet` itself, the first tests to add should be:
-- `test_route_tables_cover_spec_windows`
-- `test_send_to_pane_escapes_single_quotes_and_special_chars`
-- `test_comment_on_issue_formats_machine_window_pane_body`
-- `test_get_issues_skips_prs_and_meta_labels`
-- `test_dispatch_cycle_persists_dispatch_state_once`
-- `test_get_pane_status_classifies_spinner_vs_traceback_vs_empty`
-
-These are the minimum critical-path tests.
-
----
-
-## Security Considerations
-
-### 1. Command injection surface
-
-`send_to_pane()` and the remote tmux/SSH command assembly are the biggest security surface.
-Even though single quotes are escaped in prompts, this remains a command injection boundary because untrusted issue titles and repo metadata cross into shell commands.
-
-This is why `command injection` is the right risk label for the repo.
-The risk is not hypothetical; the repo is literally translating issue text into shell transport.
-
-### 2. Credential handling
-
-The dispatcher uses a local token file for Gitea authentication.
-That is a credential handling concern because:
-- token locality is assumed
-- file path and host assumptions are embedded into runtime code
-- there is no retry / fallback / explicit missing-token UX beyond failure
-
-### 3. SSH trust boundary
-
-Remote pane control over `root@167.99.126.228` means the repo assumes a trusted SSH path to a root shell.
-That is operationally powerful and dangerous.
-A malformed remote command, stale known_hosts state, or wrong host mapping has fleet-wide consequences.
-
-### 4. Runtime state tampering
-
-`dispatch-state.json` is a local mutable state file with no locking, signing, or cross-machine reconciliation.
-If it is corrupted or lost, deduplication semantics fail.
-That can cause repeated dispatches or misleading status.
-
-### 5. Live-forge mutation
-
-`comment_on_issue()` mutates live issue threads on every dispatch cycle.
-That means any bug in deduplication or routing will create visible comment spam on the forge.
-
-### 6. Dependency risk
-
-The repo depends on `requests` for Gitea API access but has no pinned dependency metadata or environment contract in-repo.
-This is a small operational repo, but reproducibility is weak.
-
----
-
-## Dependency Picture
-
-### Runtime dependencies
-- Python 3
-- `requests`
-- tmux
-- SSH client
-- ssh trust boundary to `root@167.99.126.228`
-- access to a Gitea token file
-
-### Implied environment dependencies
-- active tmux sessions on Mac and Allegro
-- SSH trust / connectivity to the VPS
-- hermes available in pane environments
-- Gitea reachable at `https://forge.alexanderwhitestone.com`
-
-### Notably missing
-- no `requirements.txt`
-- no `pyproject.toml`
-- no explicit test harness
-- no schema validation for `fleet-spec.json`
-
----
-
-## Performance Characteristics
-
-For such a small repo, the performance question is not CPU time inside Python.
-It is orchestration fan-out latency.
-
-The main scaling costs are:
-- repeated Gitea issue fetches across repos
-- SSH round-trips to Allegro
-- tmux pane fan-out across 100+ panes
-- serialized `time.sleep(0.2)` dispatch staggering
-
-This means the bottleneck is control-plane coordination, not computation.
-The repo will scale until SSH / tmux / Gitea latency become dominant.
-
----
-
-## Dead Code / Drift Risks
-
-### 1. Spec vs route duplication
-
-`fleet-spec.json` defines windows and lanes, while `fleet-dispatch.py` separately defines `MAC_ROUTE` and `ALLEGRO_ROUTE`.
-That is the biggest drift risk.
-A window can exist in the spec and be missing from a route table, or vice versa.
-
-### 2. Runtime-generated files absent from repo contracts
-
-`dispatch-state.json` is operationally critical but not described as a first-class contract in code.
-The repo assumes it exists or can be created, but does not validate structure.
-
-### 3. README drift risk
-
-The README says "use fleet-christen.sh" in one place while the actual file is `fleet-christen.py`.
-That is a small but real operator-footgun and a sign the human runbook can drift from the executable surface.
-
----
-
-## Suggested Follow-up Work
-
-1. Move repo-to-window routing into `fleet-spec.json` and derive `MAC_ROUTE` / `ALLEGRO_ROUTE` programmatically.
-2. Add automated tests for `send_to_pane`, `get_issues`, `dispatch_cycle`, and `get_pane_status`.
-3. Add a schema validator for `fleet-spec.json`.
-4. Add explicit dependency metadata (`requirements.txt` or `pyproject.toml`).
-5. Add dry-run / no-side-effect mode for dispatch and christening.
-6. Add retry/backoff and error reporting around Gitea comments and SSH execution.
-
----
-
-## Bottom Line
-
-`burn-fleet` is a small repo with outsized operational leverage.
-Its genome is simple:
-- one declarative topology file
-- four operational adapters
-- one local runtime ledger
-- many side effects across tmux, SSH, and Gitea
-
-It already expresses the philosophy of narrative-driven infrastructure well.
-What it lacks is not architecture.
-What it lacks is hardening:
-- tests around the dangerous paths
-- centralization of duplicated routing truth
-- stronger command / credential / runtime-state safeguards
-
-That makes it a strong control-plane prototype and a weakly tested production surface.

reports/evaluations/2026-04-06-mempalace-evaluation.md

@@ -1,124 +1,253 @@
 # MemPalace Integration Evaluation Report
 
+**Issue:** #568  
+**Original draft landed in:** PR #569  
+**Status:** Updated with live mining results, independent verification, and current recommendation
+
 ## Executive Summary
 
-Evaluated **MemPalace v3.0.0** (github.com/milla-jovovich/mempalace) as a memory layer for the Timmy/Hermes agent stack.
+Evaluated **MemPalace v3.0.0** (`github.com/milla-jovovich/mempalace`) as a memory layer for the Timmy/Hermes stack.
 
-**Installed:** ✅ `mempalace 3.0.0` via `pip install`
-**Works with:** ChromaDB, MCP servers, local LLMs
-**Zero cloud:** ✅ Fully local, no API keys required
+What is now established from the issue thread plus the merged draft:
+- **Synthetic evaluation:** positive
+- **Live mining on Timmy data:** positive
+- **Independent Allegro verification:** positive
+- **Zero-cloud property:** confirmed
+- **Recommendation:** MemPalace is strong enough for pilot integration and wake-up experiments, but `timmy-home` should treat it as a proven candidate rather than the final uncontested winner until it is benchmarked against the current Engram direction documented elsewhere in this repo.
 
-## Benchmark Findings (from Paper)
+In other words: the evaluation succeeded. The remaining question is not whether MemPalace works. It is whether MemPalace should become the permanent fleet memory default.
+
+## Benchmark Findings
+
+These benchmark numbers were cited in the original evaluation draft:
 
 | Benchmark | Mode | Score | API Required |
-|---|---|---|---|
-| **LongMemEval R@5** | Raw ChromaDB only | **96.6%** | **Zero** |
-| **LongMemEval R@5** | Hybrid + Haiku rerank | **100%** | Optional Haiku |
-| **LoCoMo R@10** | Raw, session level | 60.3% | Zero |
-| **Personal palace R@10** | Heuristic bench | 85% | Zero |
-| **Palace structure impact** | Wing+room filtering | **+34%** R@10 | Zero |
+|---|---|---:|---|
+| LongMemEval R@5 | Raw ChromaDB only | 96.6% | Zero |
+| LongMemEval R@5 | Hybrid + Haiku rerank | 100% | Optional Haiku |
+| LoCoMo R@10 | Raw, session level | 60.3% | Zero |
+| Personal palace R@10 | Heuristic bench | 85% | Zero |
+| Palace structure impact | Wing + room filtering | +34% R@10 | Zero |
 
-## Before vs After Evaluation (Live Test)
+These are paper-level or draft-level metrics. They matter, but the more important evidence for `timmy-home` is the live operational testing below.
 
-### Test Setup
-- Created test project with 4 files (README.md, auth.md, deployment.md, main.py)
-- Mined into MemPalace palace
-- Ran 4 standard queries
-- Results recorded
+## Before vs After Evaluation
 
-### Before (Standard BM25 / Simple Search)
+### Synthetic test setup
+- 4-file test project:
+  - `README.md`
+  - `auth.md`
+  - `deployment.md`
+  - `main.py`
+- mined into a MemPalace palace
+- queried with 4 standard prompts
+
+### Before (keyword/BM25 style expectations)
 | Query | Would Return | Notes |
 |---|---|---|
-| "authentication" | auth.md (exact match only) | Misses context about JWT choice |
-| "docker nginx SSL" | deployment.md | Manual regex/keyword matching needed |
-| "keycloak OAuth" | auth.md | Would need full-text index |
-| "postgresql database" | README.md (maybe) | Depends on index |
+| `authentication` | `auth.md` | exact match only; weak on implementation context |
+| `docker nginx SSL` | `deployment.md` | requires manual keyword logic |
+| `keycloak OAuth` | `auth.md` | little semantic cross-reference |
+| `postgresql database` | `README.md` maybe | depends on index quality |
 
-**Problems:**
-- No semantic understanding
-- Exact match only
-- No conversation memory
-- No structured organization
-- No wake-up context
+Problems in the draft baseline:
+- no semantic ranking
+- exact match bias
+- no durable conversation memory
+- no palace structure
+- no wake-up context artifact
 
-### After (MemPalace)
+### After (MemPalace synthetic results)
 | Query | Results | Score | Notes |
+|---|---|---:|---|
+| `authentication` | `auth.md`, `main.py` | -0.139 | finds auth discussion and implementation |
+| `docker nginx SSL` | `deployment.md`, `auth.md` | 0.447 | exact deployment hit plus related JWT context |
+| `keycloak OAuth` | `auth.md`, `main.py` | -0.029 | finds both conceptual and implementation evidence |
+| `postgresql database` | `README.md`, `main.py` | 0.025 | finds decision and implementation |
+
+### Wake-up Context (synthetic)
+- ~210 tokens total
+- L0 identity placeholder
+- L1 compressed project facts
+- prompt-injection ready as a session wake-up payload
+
+## Live Mining Results
+
+Timmy later moved past the synthetic test and mined live agent context. That is the more important result for this repo.
+
+### Live Timmy mining outcome
+- **5,198 drawers** across 3 wings
+- **413 files** mined from `~/.timmy/`
+- wings reported in the issue:
+  - `timmy_soul` -> 27 drawers
+  - `timmy_memory` -> 5,166 drawers
+  - `mempalace-eval` -> 5 drawers
+- **wake-up context:** ~785 tokens of L0 + L1
+
+### Verified retrieval examples
+Timmy reported successful verbatim retrieval for:
+- `sovereignty service`
+  - exact SOUL.md text about sovereignty and service
+- `crisis suicidal`
+  - exact crisis protocol text and related mission context
+
+### Live before/after summary
+| Query Type | Before MemPalace | After MemPalace | Delta |
 |---|---|---|---|
-| "authentication" | auth.md, main.py | -0.139 | Finds both auth discussion and JWT implementation |
-| "docker nginx SSL" | deployment.md, auth.md | 0.447 | Exact match on deployment, related JWT context |
-| "keycloak OAuth" | auth.md, main.py | -0.029 | Finds OAuth discussion and JWT usage |
-| "postgresql database" | README.md, main.py | 0.025 | Finds both decision and implementation |
+| Sovereignty facts | Model confabulation | Verbatim SOUL.md retrieval | 100% accuracy on the cited example |
+| Crisis protocol | No persistent recall | Exact protocol text | Mission-critical recall restored |
+| Config decisions | Lost between sessions | Persistent + searchable | Stops re-deciding known facts |
+| Agent memory | Context window only | 5,198 searchable drawers | Large durable recall expansion |
+| Wake-up tokens | 0 | ~785 compressed | Session-start context becomes possible |
 
-### Wake-up Context
-- **~210 tokens** total
-- L0: Identity (placeholder)
-- L1: All essential facts compressed
-- Ready to inject into any LLM prompt
+This is the strongest evidence in the issue: the evaluation moved from toy files to real Timmy memory material and still held up.
 
-## Integration Potential
+## Independent Verification
 
-### 1. Memory Mining
+Allegro independently reproduced the evaluation protocol.
+
+### Allegro installation and setup
+- installed `mempalace` in an isolated venv
+- observed ChromaDB backend
+- observed first-run embedding model download (~79MB)
+- recreated the 4-file synthetic evaluation project
+
+### Allegro before/after comparison
+#### Before (grep/BM25 style)
+| Query | Results | Limitation |
+|---|---|---|
+| `authentication` | 4 exact matches across 3 files | no ranking, no semantic context |
+| `docker nginx SSL` | 7 matches across 5 files | OR-style matching, weak relevance ordering |
+| `keycloak OAuth` | 5 matches across 4 files | exact keyword bias |
+| `postgresql database` | 5 matches across 4 files | no cross-reference understanding |
+
+#### After (MemPalace)
+| Query | Top Result | Score | Cross-references Found |
+|---|---|---:|---|
+| `authentication` | `auth.md` | -0.232 | ranked related files behind it |
+| `docker nginx SSL` | `main.py` | 0.073 | ranked infra-related files by relevance |
+| `keycloak OAuth` | `auth.md` | -0.137 | related files surfaced as well |
+| `postgresql database` | `README.md` | -0.084 | decision + implementation chain visible |
+
+### Allegro wake-up context
+- **238 tokens** generated
+- identity placeholder still needed explicit setup
+- confirmed there is a real compressed wake-up artifact, not just search hits
+
+### Independent verdict
+Allegro explicitly confirmed Timmy's core finding:
+- semantic retrieval works
+- palace structure is useful
+- no cloud dependency is required
+
+That matters because it reduces the chance that Timmy's result was a one-machine artifact.
+
+## Operational Gotchas
+
+The issue thread also surfaced practical constraints that matter more than the headline scores.
+
+1. `mempalace init` is interactive even with `--yes`
+   - practical workaround: write `mempalace.yaml` manually
+
+2. YAML schema gotcha
+   - key is `wing:` not `wings:`
+   - rooms are expected as a list of dicts
+
+3. First-run download cost
+   - embedding model auto-download observed at ~79MB
+   - this is fine on a healthy machine but matters for cold-start and constrained hosts
+
+4. Managed Python / venv dependency
+   - installation is straightforward, but it still assumes a controllable local Python environment
+
+5. Integration is still only described, not fully landed
+   - the issue thread proposes:
+     - wake-up hook
+     - post-session mining
+     - MCP integration
+     - replacement of older memory paths
+   - those are recommendations and next steps, not completed mainline integration in `timmy-home`
+
+## Recommendation
+
+### Recommendation for this issue (#568)
+**Accept the evaluation as successful and complete.**
+
+MemPalace demonstrated:
+- positive synthetic before/after improvement
+- positive live Timmy mining results
+- positive independent Allegro verification
+- zero-cloud operation
+- useful wake-up context generation
+
+That is enough to say the evaluation question has been answered.
+
+### Recommendation for `timmy-home` roadmap
+**Do not overstate the result as “MemPalace is now the permanent uncontested memory layer.”**
+
+A more precise current recommendation is:
+1. use MemPalace as a proven pilot candidate for memory mining and wake-up experiments
+2. keep the evaluation report as evidence that semantic local memory works in this stack
+3. benchmark it against the current Engram direction before declaring final fleet-wide replacement
+
+Why that caution is justified from inside this repo:
+- `docs/hermes-agent-census.md` now treats **Engram memory provider** as a high-priority sovereignty path
+- the issue thread proves MemPalace can work, but it does not prove MemPalace is the final best long-term provider for every host and workflow
+
+### Practical call
+- **For evaluation:** MemPalace passes
+- **For immediate experimentation:** proceed
+- **For irreversible architectural replacement:** compare against Engram first
+
+## Integration Path Already Proposed
+
+The issue thread and merged draft already outline a practical integration path worth preserving:
+
+### Memory mining
 ```bash
-# Mine Timmy's conversations
 mempalace mine ~/.hermes/sessions/ --mode convos
-
-# Mine project code and docs
 mempalace mine ~/.hermes/hermes-agent/
-
-# Mine configs
 mempalace mine ~/.hermes/
 ```
 
-### 2. Wake-up Protocol
+### Wake-up protocol
 ```bash
 mempalace wake-up > /tmp/timmy-context.txt
-# Inject into Hermes system prompt
 ```
 
-### 3. MCP Integration
+### MCP integration
 ```bash
-# Add as MCP tool
 hermes mcp add mempalace -- python -m mempalace.mcp_server
 ```
 
-### 4. Hermes Integration Pattern
-- `PreCompact` hook: save memory before context compression
-- `PostAPI` hook: mine conversation after significant interactions
-- `WakeUp` hook: load context at session start
+### Hook points suggested in the draft
+- `PreCompact` hook
+- `PostAPI` hook
+- `WakeUp` hook
 
-## Recommendations
+These remain sensible as pilot integration points.
 
-### Immediate
-1. Add `mempalace` to Hermes venv requirements
-2. Create mine script for ~/.hermes/ and ~/.timmy/
-3. Add wake-up hook to Hermes session start
-4. Test with real conversation exports
+## Next Steps
 
-### Short-term (Next Week)
-1. Mine last 30 days of Timmy sessions
-2. Build wake-up context for all agents
-3. Add MemPalace MCP tools to Hermes toolset
-4. Test retrieval quality on real queries
-
-### Medium-term (Next Month)
-1. Replace homebrew memory system with MemPalace
-2. Build palace structure: wings for projects, halls for topics
-3. Compress with AAAK for 30x storage efficiency
-4. Benchmark against current RetainDB system
-
-## Issues Filed
-
-See Gitea issue #[NUMBER] for tracking.
+Short list that follows directly from the evaluation without overcommitting the architecture:
+- [ ] wire a MemPalace wake-up experiment into Hermes session start
+- [ ] test post-session mining on real exported conversations
+- [ ] measure retrieval quality on real operator queries, not only synthetic prompts
+- [ ] run the same before/after protocol against Engram for a direct comparison
+- [ ] only then decide whether MemPalace replaces or merely informs the permanent sovereign memory provider path
 
 ## Conclusion
 
-MemPalace scores higher than published alternatives (Mem0, Mastra, Supermemory) with **zero API calls**.
+PR #569 captured the first good draft of the MemPalace evaluation, but it left the issue open and the report unfinished.
 
-For our use case, the key advantages are:
-1. **Verbatim retrieval** — never loses the "why" context
-2. **Palace structure** — +34% boost from organization
-3. **Local-only** — aligns with our sovereignty mandate
-4. **MCP compatible** — drops into our existing tool chain
-5. **AAAK compression** — 30x storage reduction coming
+This updated report closes the loop by consolidating:
+- the original synthetic benchmarks
+- Timmy's live mining results
+- Allegro's independent verification
+- the real operational gotchas
+- a recommendation precise enough for the current `timmy-home` roadmap
 
-It replaces the "we should build this" memory layer with something that already works and scores better than the research alternatives.
+Bottom line:
+- **MemPalace worked.**
+- **The evaluation succeeded.**
+- **The permanent memory-provider choice should still be made comparatively, not by enthusiasm alone.**

tests/docs/test_mempalace_evaluation_report.py

@@ -0,0 +1,34 @@
+from pathlib import Path
+
+
+REPORT = Path("reports/evaluations/2026-04-06-mempalace-evaluation.md")
+
+
+def _content() -> str:
+    return REPORT.read_text()
+
+
+def test_mempalace_evaluation_report_exists() -> None:
+    assert REPORT.exists()
+
+
+def test_mempalace_evaluation_report_has_completed_sections() -> None:
+    content = _content()
+    assert "# MemPalace Integration Evaluation Report" in content
+    assert "## Executive Summary" in content
+    assert "## Benchmark Findings" in content
+    assert "## Before vs After Evaluation" in content
+    assert "## Live Mining Results" in content
+    assert "## Independent Verification" in content
+    assert "## Operational Gotchas" in content
+    assert "## Recommendation" in content
+
+
+def test_mempalace_evaluation_report_uses_real_issue_reference_and_metrics() -> None:
+    content = _content()
+    assert "#568" in content
+    assert "#[NUMBER]" not in content
+    assert "5,198 drawers" in content
+    assert "~785 tokens" in content
+    assert "238 tokens" in content
+    assert "interactive even with `--yes`" in content or "interactive even with --yes" in content

tests/test_burn_fleet_genome.py

@@ -1,70 +0,0 @@
-from pathlib import Path
-
-GENOME = Path('genomes/burn-fleet-GENOME.md')
-
-
-def read_genome() -> str:
-    assert GENOME.exists(), 'burn-fleet genome must exist at genomes/burn-fleet-GENOME.md'
-    return GENOME.read_text(encoding='utf-8')
-
-
-def test_genome_exists():
-    assert GENOME.exists(), 'burn-fleet genome must exist at genomes/burn-fleet-GENOME.md'
-
-
-def test_genome_has_required_sections():
-    text = read_genome()
-    for heading in [
-        '# GENOME.md: burn-fleet',
-        '## Project Overview',
-        '## Architecture',
-        '## Entry Points',
-        '## Data Flow',
-        '## Key Abstractions',
-        '## API Surface',
-        '## Test Coverage Gaps',
-        '## Security Considerations',
-    ]:
-        assert heading in text
-
-
-def test_genome_contains_mermaid_diagram():
-    text = read_genome()
-    assert '```mermaid' in text
-    assert 'graph TD' in text or 'flowchart TD' in text
-
-
-def test_genome_mentions_core_files_and_runtime_state():
-    text = read_genome()
-    for token in [
-        'fleet-spec.json',
-        'fleet-launch.sh',
-        'fleet-christen.py',
-        'fleet-dispatch.py',
-        'fleet-status.py',
-        'dispatch-state.json',
-        'tmux',
-        'ssh',
-        'MAC_ROUTE',
-        'ALLEGRO_ROUTE',
-    ]:
-        assert token in text
-
-
-def test_genome_mentions_test_gap_and_risk_findings():
-    text = read_genome()
-    for token in [
-        '0% estimated coverage',
-        'send_to_pane',
-        'comment_on_issue',
-        'get_pane_status',
-        'requests',
-        'command injection',
-        'credential handling',
-    ]:
-        assert token in text
-
-
-def test_genome_is_substantial():
-    text = read_genome()
-    assert len(text) >= 6000