Compare commits
2 Commits
sprint/iss
...
fix/533
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e99b251b68 | ||
|
|
7beae447ba |
@@ -1,7 +1,7 @@
|
||||
# NH Broadband Install Packet
|
||||
|
||||
**Packet ID:** nh-bb-20260415-113232
|
||||
**Generated:** 2026-04-15T11:32:32.781304+00:00
|
||||
**Packet ID:** nh-bb-20260417-015617
|
||||
**Generated:** 2026-04-17T01:56:17.706676+00:00
|
||||
**Status:** pending_scheduling_call
|
||||
|
||||
## Contact
|
||||
@@ -19,6 +19,26 @@
|
||||
|
||||
residential-fiber
|
||||
|
||||
## Availability Check
|
||||
|
||||
- Status: pending
|
||||
- Checked at: pending
|
||||
- Notes: Pending exact-address lookup on nhbroadband.com
|
||||
|
||||
## Pricing
|
||||
|
||||
- Monthly cost: pending live quote
|
||||
- Install fee: pending live quote
|
||||
- Quoted at: pending
|
||||
- Notes: Pending live quote from NH Broadband
|
||||
|
||||
## Appointment
|
||||
|
||||
- Scheduled for: pending
|
||||
- Window: pending
|
||||
- Confirmation number: pending
|
||||
- Installer access notes: Driveway condition note pending
|
||||
|
||||
## Call Log
|
||||
|
||||
- **2026-04-15T14:30:00Z** — no_answer
|
||||
@@ -35,3 +55,11 @@ residential-fiber
|
||||
- [ ] Post-install: run speed test (fast.com / speedtest.net)
|
||||
- [ ] Log final speeds and appointment outcome
|
||||
|
||||
## Post-Install Verification
|
||||
|
||||
- Download: pending
|
||||
- Upload: pending
|
||||
- Latency: pending
|
||||
- Tested at: pending
|
||||
- Notes: Populate after installation completes
|
||||
|
||||
|
||||
@@ -11,6 +11,31 @@ service:
|
||||
|
||||
desired_plan: residential-fiber
|
||||
|
||||
availability:
|
||||
status: pending
|
||||
checked_at: ""
|
||||
notes: "Pending exact-address lookup on nhbroadband.com"
|
||||
|
||||
pricing:
|
||||
monthly_cost_usd:
|
||||
install_fee_usd:
|
||||
quoted_at: ""
|
||||
notes: "Pending live quote from NH Broadband"
|
||||
|
||||
appointment:
|
||||
scheduled_for: ""
|
||||
window: ""
|
||||
confirmation_number: ""
|
||||
installer_access_notes: "Driveway condition note pending"
|
||||
|
||||
post_install:
|
||||
speed_test:
|
||||
download_mbps:
|
||||
upload_mbps:
|
||||
latency_ms:
|
||||
tested_at: ""
|
||||
notes: "Populate after installation completes"
|
||||
|
||||
call_log:
|
||||
- timestamp: "2026-04-15T14:30:00Z"
|
||||
outcome: no_answer
|
||||
|
||||
@@ -1,111 +0,0 @@
|
||||
# MemPalace v3.0.0 Integration Evaluation — Before/After Report
|
||||
|
||||
**Closes:** #568
|
||||
**Date:** 2026-04-16
|
||||
**Status:** Formalized evaluation with before/after benchmarks
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Formalized evaluation report for **MemPalace v3.0.0** integration with the Timmy/Hermes stack, providing before/after benchmark data and integration recommendation.
|
||||
|
||||
**Key findings:**
|
||||
- 96.6% R@5 with zero API calls
|
||||
- +34% retrieval boost from palace structure
|
||||
- 210-token wake-up context
|
||||
- **Recommendation:** Integrate as primary memory layer
|
||||
|
||||
## Before vs After Benchmark Comparison
|
||||
|
||||
### Before Integration (Baseline)
|
||||
|
||||
| Metric | Value | Notes |
|
||||
|---|---:|---|
|
||||
| LongMemEval R@5 | ~62% | Standard ChromaDB without palace structure |
|
||||
| Retrieval latency | Variable | Dependent on embedding model |
|
||||
| API calls required | Multiple | Cloud-based reranking typical |
|
||||
| Wake-up context | None | No compressed state artifact |
|
||||
| Palace structure benefit | N/A | Not applicable |
|
||||
|
||||
### After Integration (MemPalace v3.0.0)
|
||||
|
||||
| Metric | Value | Notes |
|
||||
|---|---:|---|
|
||||
| LongMemEval R@5 | 96.6% | Raw ChromaDB with palace indexing |
|
||||
| Retrieval boost (palace structure) | +34% | Wing + room filtering |
|
||||
| API calls required | Zero | Fully local operation |
|
||||
| Wake-up context | 210 tokens | Compressed project state |
|
||||
| Palace structure | Enabled | Wing + room semantic organization |
|
||||
|
||||
## Benchmark Details
|
||||
|
||||
### Core Metrics
|
||||
|
||||
| Benchmark | Mode | Score | API Required |
|
||||
|---|---|---:|---|
|
||||
| LongMemEval R@5 | Raw ChromaDB only | 96.6% | Zero |
|
||||
| LongMemEval R@5 | Hybrid + Haiku rerank | 100% | Optional Haiku |
|
||||
| LoCoMo R@10 | Raw, session level | 60.3% | Zero |
|
||||
| Personal palace R@10 | Heuristic bench | 85% | Zero |
|
||||
| Palace structure impact | Wing + room filtering | +34% R@10 | Zero |
|
||||
|
||||
### Retrieval Performance Analysis
|
||||
|
||||
**Before palace structure:**
|
||||
- Flat vector search across all documents
|
||||
- No semantic organization
|
||||
- R@5 approximately 62% on standard benchmarks
|
||||
- Required cloud API for acceptable quality
|
||||
|
||||
**After palace structure:**
|
||||
- Hierarchical wing + room organization
|
||||
- Semantic filtering before vector search
|
||||
- R@5 improved to 96.6% (zero API calls)
|
||||
- +34% retrieval boost from structure alone
|
||||
- 100% achievable with optional Haiku rerank
|
||||
|
||||
## Wake-up Context Evaluation
|
||||
|
||||
### Before
|
||||
- No compressed state artifact
|
||||
- Full context reload required on each session
|
||||
- Higher token overhead for session initialization
|
||||
|
||||
### After
|
||||
- 210-token wake-up context
|
||||
- L0 identity placeholder
|
||||
- L1 compressed project state
|
||||
- L2 active memory pointers
|
||||
- Rapid session initialization
|
||||
|
||||
## Integration Recommendation
|
||||
|
||||
### Primary Finding
|
||||
MemPalace v3.0.0 demonstrates sufficient performance for production integration as the primary memory layer for Timmy/Hermes stack.
|
||||
|
||||
### Key Evidence
|
||||
1. **96.6% R@5 with zero API calls** — Meets sovereignty requirements
|
||||
2. **+34% retrieval boost** — Palace structure provides measurable improvement
|
||||
3. **210-token wake-up context** — Efficient session initialization
|
||||
4. **Zero-cloud operation** — Aligns with infrastructure constraints
|
||||
|
||||
### Recommendation
|
||||
**Integrate MemPalace v3.0.0 as primary memory layer.**
|
||||
|
||||
Rationale:
|
||||
- Performance exceeds baseline requirements
|
||||
- Zero API dependency maintains sovereignty
|
||||
- Palace structure provides semantic organization
|
||||
- Wake-up context enables efficient cold starts
|
||||
- Operational simplicity (local-only operation)
|
||||
|
||||
## Appendix: Test Configuration
|
||||
|
||||
- **MemPalace version:** v3.0.0
|
||||
- **Embedding model:** Local (no cloud dependency)
|
||||
- **Vector store:** ChromaDB (embedded)
|
||||
- **Palace structure:** Wing + room hierarchy
|
||||
- **Test dataset:** LongMemEval + LoCoMo benchmarks
|
||||
|
||||
---
|
||||
|
||||
*Report generated for issue #568 — MemPalace v3.0.0 integration evaluation with before/after comparison.*
|
||||
@@ -17,6 +17,27 @@ def load_request(path: str | Path) -> dict[str, Any]:
|
||||
data.setdefault("service", {})
|
||||
data.setdefault("call_log", [])
|
||||
data.setdefault("checklist", [])
|
||||
data.setdefault("availability", {})
|
||||
data.setdefault("pricing", {})
|
||||
data.setdefault("appointment", {})
|
||||
data.setdefault("post_install", {})
|
||||
data["availability"].setdefault("status", "pending")
|
||||
data["availability"].setdefault("checked_at", "")
|
||||
data["availability"].setdefault("notes", "")
|
||||
data["pricing"].setdefault("monthly_cost_usd", None)
|
||||
data["pricing"].setdefault("install_fee_usd", None)
|
||||
data["pricing"].setdefault("quoted_at", "")
|
||||
data["pricing"].setdefault("notes", "")
|
||||
data["appointment"].setdefault("scheduled_for", "")
|
||||
data["appointment"].setdefault("window", "")
|
||||
data["appointment"].setdefault("confirmation_number", "")
|
||||
data["appointment"].setdefault("installer_access_notes", "")
|
||||
data["post_install"].setdefault("speed_test", {})
|
||||
data["post_install"]["speed_test"].setdefault("download_mbps", None)
|
||||
data["post_install"]["speed_test"].setdefault("upload_mbps", None)
|
||||
data["post_install"]["speed_test"].setdefault("latency_ms", None)
|
||||
data["post_install"]["speed_test"].setdefault("tested_at", "")
|
||||
data["post_install"]["speed_test"].setdefault("notes", "")
|
||||
return data
|
||||
|
||||
|
||||
@@ -39,6 +60,19 @@ def build_packet(data: dict[str, Any]) -> dict[str, Any]:
|
||||
validate_request(data)
|
||||
contact = data["contact"]
|
||||
service = data["service"]
|
||||
availability = data.get("availability", {})
|
||||
pricing = data.get("pricing", {})
|
||||
appointment = data.get("appointment", {})
|
||||
post_install = data.get("post_install", {})
|
||||
speed_test = post_install.get("speed_test", {})
|
||||
|
||||
status = "pending_scheduling_call"
|
||||
if availability.get("status") == "unavailable":
|
||||
status = "blocked_unavailable"
|
||||
elif appointment.get("confirmation_number") or appointment.get("scheduled_for"):
|
||||
status = "scheduled_install"
|
||||
if speed_test.get("download_mbps") is not None:
|
||||
status = "post_install_verified"
|
||||
|
||||
return {
|
||||
"packet_id": f"nh-bb-{datetime.now(timezone.utc).strftime('%Y%m%d-%H%M%S')}",
|
||||
@@ -55,18 +89,69 @@ def build_packet(data: dict[str, Any]) -> dict[str, Any]:
|
||||
"zip": service.get("zip", ""),
|
||||
},
|
||||
"desired_plan": data.get("desired_plan", "residential-fiber"),
|
||||
"availability": {
|
||||
"status": availability.get("status", "pending"),
|
||||
"checked_at": availability.get("checked_at", ""),
|
||||
"notes": availability.get("notes", ""),
|
||||
},
|
||||
"pricing": {
|
||||
"monthly_cost_usd": pricing.get("monthly_cost_usd"),
|
||||
"install_fee_usd": pricing.get("install_fee_usd"),
|
||||
"quoted_at": pricing.get("quoted_at", ""),
|
||||
"notes": pricing.get("notes", ""),
|
||||
},
|
||||
"appointment": {
|
||||
"scheduled_for": appointment.get("scheduled_for", ""),
|
||||
"window": appointment.get("window", ""),
|
||||
"confirmation_number": appointment.get("confirmation_number", ""),
|
||||
"installer_access_notes": appointment.get("installer_access_notes", ""),
|
||||
},
|
||||
"call_log": data.get("call_log", []),
|
||||
"checklist": [
|
||||
{"item": item, "done": False} if isinstance(item, str) else item
|
||||
for item in data["checklist"]
|
||||
],
|
||||
"status": "pending_scheduling_call",
|
||||
"post_install": {
|
||||
"speed_test": {
|
||||
"download_mbps": speed_test.get("download_mbps"),
|
||||
"upload_mbps": speed_test.get("upload_mbps"),
|
||||
"latency_ms": speed_test.get("latency_ms"),
|
||||
"tested_at": speed_test.get("tested_at", ""),
|
||||
"notes": speed_test.get("notes", ""),
|
||||
}
|
||||
},
|
||||
"status": status,
|
||||
}
|
||||
|
||||
|
||||
def render_markdown(packet: dict[str, Any], data: dict[str, Any]) -> str:
|
||||
contact = packet["contact"]
|
||||
addr = packet["service_address"]
|
||||
availability = packet["availability"]
|
||||
pricing = packet["pricing"]
|
||||
appointment = packet["appointment"]
|
||||
speed_test = packet["post_install"]["speed_test"]
|
||||
|
||||
monthly_cost = pricing["monthly_cost_usd"]
|
||||
install_fee = pricing["install_fee_usd"]
|
||||
monthly_line = f"${monthly_cost:.2f}" if isinstance(monthly_cost, (int, float)) else "pending live quote"
|
||||
install_line = f"${install_fee:.2f}" if isinstance(install_fee, (int, float)) else "pending live quote"
|
||||
download_line = (
|
||||
f"{speed_test['download_mbps']} Mbps"
|
||||
if speed_test["download_mbps"] is not None else
|
||||
"pending"
|
||||
)
|
||||
upload_line = (
|
||||
f"{speed_test['upload_mbps']} Mbps"
|
||||
if speed_test["upload_mbps"] is not None else
|
||||
"pending"
|
||||
)
|
||||
latency_line = (
|
||||
f"{speed_test['latency_ms']} ms"
|
||||
if speed_test["latency_ms"] is not None else
|
||||
"pending"
|
||||
)
|
||||
|
||||
lines = [
|
||||
f"# NH Broadband Install Packet",
|
||||
"",
|
||||
@@ -89,6 +174,26 @@ def render_markdown(packet: dict[str, Any], data: dict[str, Any]) -> str:
|
||||
"",
|
||||
f"{packet['desired_plan']}",
|
||||
"",
|
||||
"## Availability Check",
|
||||
"",
|
||||
f"- Status: {availability['status']}",
|
||||
f"- Checked at: {availability['checked_at'] or 'pending'}",
|
||||
f"- Notes: {availability['notes'] or 'pending exact-address lookup'}",
|
||||
"",
|
||||
"## Pricing",
|
||||
"",
|
||||
f"- Monthly cost: {monthly_line}",
|
||||
f"- Install fee: {install_line}",
|
||||
f"- Quoted at: {pricing['quoted_at'] or 'pending'}",
|
||||
f"- Notes: {pricing['notes'] or 'pending live quote'}",
|
||||
"",
|
||||
"## Appointment",
|
||||
"",
|
||||
f"- Scheduled for: {appointment['scheduled_for'] or 'pending'}",
|
||||
f"- Window: {appointment['window'] or 'pending'}",
|
||||
f"- Confirmation number: {appointment['confirmation_number'] or 'pending'}",
|
||||
f"- Installer access notes: {appointment['installer_access_notes'] or 'pending'}",
|
||||
"",
|
||||
"## Call Log",
|
||||
"",
|
||||
]
|
||||
@@ -112,7 +217,17 @@ def render_markdown(packet: dict[str, Any], data: dict[str, Any]) -> str:
|
||||
mark = "x" if item.get("done") else " "
|
||||
lines.append(f"- [{mark}] {item['item']}")
|
||||
|
||||
lines.append("")
|
||||
lines.extend([
|
||||
"",
|
||||
"## Post-Install Verification",
|
||||
"",
|
||||
f"- Download: {download_line}",
|
||||
f"- Upload: {upload_line}",
|
||||
f"- Latency: {latency_line}",
|
||||
f"- Tested at: {speed_test['tested_at'] or 'pending'}",
|
||||
f"- Notes: {speed_test['notes'] or 'pending'}",
|
||||
"",
|
||||
])
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
|
||||
@@ -35,6 +35,11 @@ def test_load_and_build_packet() -> None:
|
||||
assert packet["status"] == "pending_scheduling_call"
|
||||
assert len(packet["checklist"]) == 8
|
||||
assert packet["checklist"][0]["done"] is False
|
||||
assert packet["availability"]["status"] == "pending"
|
||||
assert packet["pricing"]["monthly_cost_usd"] is None
|
||||
assert packet["pricing"]["install_fee_usd"] is None
|
||||
assert packet["appointment"]["confirmation_number"] == ""
|
||||
assert packet["post_install"]["speed_test"]["download_mbps"] is None
|
||||
|
||||
|
||||
def test_validate_rejects_missing_contact_name() -> None:
|
||||
@@ -86,8 +91,12 @@ def test_render_markdown_contains_key_sections() -> None:
|
||||
assert "# NH Broadband Install Packet" in md
|
||||
assert "## Contact" in md
|
||||
assert "## Service Address" in md
|
||||
assert "## Availability Check" in md
|
||||
assert "## Pricing" in md
|
||||
assert "## Appointment" in md
|
||||
assert "## Call Log" in md
|
||||
assert "## Appointment Checklist" in md
|
||||
assert "## Post-Install Verification" in md
|
||||
assert "Concord" in md
|
||||
assert "NH" in md
|
||||
|
||||
@@ -103,3 +112,16 @@ def test_example_yaml_is_valid() -> None:
|
||||
data = yaml.safe_load(Path("docs/nh-broadband-install-request.example.yaml").read_text())
|
||||
assert data["contact"]["name"] == "Timmy Operator"
|
||||
assert len(data["checklist"]) == 8
|
||||
assert data["availability"]["status"] == "pending"
|
||||
assert data["appointment"]["confirmation_number"] == ""
|
||||
|
||||
|
||||
def test_render_markdown_shows_pending_live_fields() -> None:
|
||||
data = load_request("docs/nh-broadband-install-request.example.yaml")
|
||||
packet = build_packet(data)
|
||||
md = render_markdown(packet, data)
|
||||
assert "Status: pending" in md
|
||||
assert "Monthly cost: pending live quote" in md
|
||||
assert "Install fee: pending live quote" in md
|
||||
assert "Confirmation number: pending" in md
|
||||
assert "Download: pending" in md
|
||||
|
||||
Reference in New Issue
Block a user