Timmy_Foundation/compounding-intelligence
15
0
Fork 0
Code Issues 173 Pull Requests 59 Actions Packages Projects Releases Wiki Activity

[ATLAS][Memory Eval] Run a live Hindsight vs MemPalace bakeoff with raw context-window artifacts #230

New Issue
Open
opened 2026-04-22 14:25:19 +00:00 by Rockachopa · 2 comments
Rockachopa commented 2026-04-22 14:25:19 +00:00
Owner
Copy Link

Parent

Timmy_Foundation/hermes-agent#985

Depends on

Timmy_Foundation/hermes-agent#1010

Sources to Evaluate

  • built-in Hermes memory surfaces
  • mempalace
  • vectorize-io/hindsight

What

Run a live, source-backed bakeoff between baseline Hermes memory, MemPalace, and local Hindsight.

This is not a README benchmark. This is a runtime bakeoff that captures what actually reaches the prompt and whether it improves answers on the kinds of recall we care about.

The evaluation must include:

  • built-in preference recall
  • structured fact recall
  • architecture/decision recall
  • fleet/operational recall
  • contradiction/failure-mode framing
  • at least one long-horizon or BEAM-style impossible-to-stuff memory scenario

Tasks

  • Build a prompt matrix covering the recall categories above
  • Run the matrix against baseline / MemPalace / local Hindsight using profile-scoped eval homes
  • Capture raw context-window artifacts for each run
  • Score answer quality, relevance of injected context, and retrieval noise
  • Publish a report with PASS/FAIL verdicts and concrete failure modes
  • Recommend keep / adopt / hybrid / reject for each provider path

Acceptance Criteria

  • All three paths (baseline, MemPalace, Hindsight) are evaluated on the same prompt matrix
  • Raw artifacts are saved for every run: system prompt, provider prefetch, final memory injection, final user payload
  • The report includes explicit failure modes, not just scores
  • At least one long-horizon memory test exceeds what can be solved by naive context stuffing
  • The final report makes a grounded recommendation for the sovereign Mac-local stack
## Parent Timmy_Foundation/hermes-agent#985 ## Depends on Timmy_Foundation/hermes-agent#1010 ## Sources to Evaluate - built-in Hermes memory surfaces - `mempalace` - `vectorize-io/hindsight` ## What Run a **live, source-backed bakeoff** between baseline Hermes memory, MemPalace, and local Hindsight. This is not a README benchmark. This is a runtime bakeoff that captures what actually reaches the prompt and whether it improves answers on the kinds of recall we care about. The evaluation must include: - built-in preference recall - structured fact recall - architecture/decision recall - fleet/operational recall - contradiction/failure-mode framing - at least one long-horizon or BEAM-style impossible-to-stuff memory scenario ## Tasks - [ ] Build a prompt matrix covering the recall categories above - [ ] Run the matrix against baseline / MemPalace / local Hindsight using profile-scoped eval homes - [ ] Capture raw context-window artifacts for each run - [ ] Score answer quality, relevance of injected context, and retrieval noise - [ ] Publish a report with PASS/FAIL verdicts and concrete failure modes - [ ] Recommend keep / adopt / hybrid / reject for each provider path ## Acceptance Criteria - [ ] All three paths (baseline, MemPalace, Hindsight) are evaluated on the same prompt matrix - [ ] Raw artifacts are saved for every run: system prompt, provider prefetch, final memory injection, final user payload - [ ] The report includes explicit failure modes, not just scores - [ ] At least one long-horizon memory test exceeds what can be solved by naive context stuffing - [ ] The final report makes a grounded recommendation for the sovereign Mac-local stack
Rockachopa commented 2026-04-23 01:37:20 +00:00
Author
Owner
Copy Link

🔥 BURN DISPATCH — BURN:LOOM.1

Queued for execution with /queue.

Target branch: fix/230-hindsight-mempalace-bakeoff
Workflow: Gitea-first, dedup PRs, verify before PR.

🔥 **BURN DISPATCH** — `BURN:LOOM.1` Queued for execution with `/queue`. Target branch: `fix/230-hindsight-mempalace-bakeoff` Workflow: Gitea-first, dedup PRs, verify before PR.
Rockachopa commented 2026-04-29 03:12:42 +00:00
Author
Owner
Copy Link

STEP35 FREE BURN — BLOCKED

Attempting to implement #230 but hitting a hard dependency gap.

Blocker Analysis

Depends on: hermes-agent#1010 — local Hindsight adapter

  • hermes-agent#1010 state: open (no PR merged)
  • No local Hindsight integration exists yet
  • No MemPalace adapter found in environment

Environment status:

  • MemPalace Python module: NOT installed (ModuleNotFoundError)
  • Hindsight Python module: NOT installed (ModuleNotFoundError)
  • No mempalace or hindsight tools in hermes-agent toolset
  • No vendor'd Hindsight binary in PATH

Acceptance Criteria Status

Criterion Status Notes
All 3 paths (baseline/MemPalace/Hindsight) evaluated ❌ BLOCKED Cannot eval missing backends
Raw artifacts captured for every run ❌ BLOCKED No runtime to capture
Report with PASS/FAIL + failure modes ❌ BLOCKED No execution data
Long-horizon memory test (> context stuffing) ❌ BLOCKED Depends on Hindsight
Recommendation for sovereign Mac-local stack ⚠️ DEFERRED Needs eval results

What Would Be Needed

  1. First: merge hermes-agent#1010 (local Hindsight adapter)
  2. Second: verify MemPalace integration works in hermes-agent runtime
  3. Third: this bakeoff script can execute 3-way comparison

Recommendation

Close #230 as blocked, reopen when #1010 merges. The bakeoff work is downstream infrastructure that cannot proceed without its dependency.

Action taken: Stopped work per FREE BURN guardrail. Commented blocker. No implementation, no PR opened.

**STEP35 FREE BURN — BLOCKED** Attempting to implement #230 but hitting a hard dependency gap. ## Blocker Analysis **Depends on:** `hermes-agent#1010` — local Hindsight adapter - `hermes-agent#1010` state: **open** (no PR merged) - No local Hindsight integration exists yet - No MemPalace adapter found in environment **Environment status:** - MemPalace Python module: NOT installed (`ModuleNotFoundError`) - Hindsight Python module: NOT installed (`ModuleNotFoundError`) - No `mempalace` or `hindsight` tools in hermes-agent toolset - No vendor'd Hindsight binary in PATH ## Acceptance Criteria Status | Criterion | Status | Notes | |-----------|--------|-------| | All 3 paths (baseline/MemPalace/Hindsight) evaluated | ❌ BLOCKED | Cannot eval missing backends | | Raw artifacts captured for every run | ❌ BLOCKED | No runtime to capture | | Report with PASS/FAIL + failure modes | ❌ BLOCKED | No execution data | | Long-horizon memory test (> context stuffing) | ❌ BLOCKED | Depends on Hindsight | | Recommendation for sovereign Mac-local stack | ⚠️ DEFERRED | Needs eval results | ## What Would Be Needed 1. **First:** merge `hermes-agent#1010` (local Hindsight adapter) 2. **Second:** verify MemPalace integration works in hermes-agent runtime 3. **Third:** this bakeoff script can execute 3-way comparison ## Recommendation Close #230 as blocked, reopen when #1010 merges. The bakeoff work is downstream infrastructure that cannot proceed without its dependency. **Action taken:** Stopped work per FREE BURN guardrail. Commented blocker. No implementation, no PR opened.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acceptance-criteria
batch-pipeline
Token masterplan batch pipeline
 bootstrapper
Pre-session context injection
 epic
Epic-level issue
 harvester
Session knowledge extraction
 measurer
Compounding metrics
 milestone:1
Milestone 1: Foundation
 milestone:2
Milestone 2: Integration
 milestone:3
Milestone 3: Measurement
 milestone:4
Milestone 4: Retroactive
 pipeline
Pipeline/integration work
 pipeline
priority:high
priority:medium
retroactive
Processing existing sessions
 throughput-10x
throughput-10x label
 token-masterplan
token-masterplan label
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Rockachopa Timmy allegro antigravity bezalel claude codex-agent ezra gemini google grok hermes kimi manus perplexity
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/compounding-intelligence#230
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?