Compare commits

..

2 Commits

Author SHA1 Message Date
Alexander Whitestone
e99b251b68 feat: track NH broadband install lifecycle (#533)
Some checks are pending
Agent PR Gate / gate (pull_request) Waiting to run
Agent PR Gate / report (pull_request) Blocked by required conditions
Self-Healing Smoke / self-healing-smoke (pull_request) Waiting to run
Smoke Test / smoke (pull_request) Waiting to run
2026-04-16 21:58:18 -04:00
Alexander Whitestone
7beae447ba test: define NH Broadband live install tracking acceptance for #533 2026-04-16 21:46:27 -04:00
5 changed files with 194 additions and 115 deletions

View File

@@ -1,7 +1,7 @@
# NH Broadband Install Packet
**Packet ID:** nh-bb-20260415-113232
**Generated:** 2026-04-15T11:32:32.781304+00:00
**Packet ID:** nh-bb-20260417-015617
**Generated:** 2026-04-17T01:56:17.706676+00:00
**Status:** pending_scheduling_call
## Contact
@@ -19,6 +19,26 @@
residential-fiber
## Availability Check
- Status: pending
- Checked at: pending
- Notes: Pending exact-address lookup on nhbroadband.com
## Pricing
- Monthly cost: pending live quote
- Install fee: pending live quote
- Quoted at: pending
- Notes: Pending live quote from NH Broadband
## Appointment
- Scheduled for: pending
- Window: pending
- Confirmation number: pending
- Installer access notes: Driveway condition note pending
## Call Log
- **2026-04-15T14:30:00Z** — no_answer
@@ -35,3 +55,11 @@ residential-fiber
- [ ] Post-install: run speed test (fast.com / speedtest.net)
- [ ] Log final speeds and appointment outcome
## Post-Install Verification
- Download: pending
- Upload: pending
- Latency: pending
- Tested at: pending
- Notes: Populate after installation completes

View File

@@ -11,6 +11,31 @@ service:
desired_plan: residential-fiber
availability:
status: pending
checked_at: ""
notes: "Pending exact-address lookup on nhbroadband.com"
pricing:
monthly_cost_usd:
install_fee_usd:
quoted_at: ""
notes: "Pending live quote from NH Broadband"
appointment:
scheduled_for: ""
window: ""
confirmation_number: ""
installer_access_notes: "Driveway condition note pending"
post_install:
speed_test:
download_mbps:
upload_mbps:
latency_ms:
tested_at: ""
notes: "Populate after installation completes"
call_log:
- timestamp: "2026-04-15T14:30:00Z"
outcome: no_answer

View File

@@ -1,111 +0,0 @@
# MemPalace v3.0.0 Integration Evaluation — Before/After Report
**Closes:** #568
**Date:** 2026-04-16
**Status:** Formalized evaluation with before/after benchmarks
## Executive Summary
Formalized evaluation report for **MemPalace v3.0.0** integration with the Timmy/Hermes stack, providing before/after benchmark data and integration recommendation.
**Key findings:**
- 96.6% R@5 with zero API calls
- +34% retrieval boost from palace structure
- 210-token wake-up context
- **Recommendation:** Integrate as primary memory layer
## Before vs After Benchmark Comparison
### Before Integration (Baseline)
| Metric | Value | Notes |
|---|---:|---|
| LongMemEval R@5 | ~62% | Standard ChromaDB without palace structure |
| Retrieval latency | Variable | Dependent on embedding model |
| API calls required | Multiple | Cloud-based reranking typical |
| Wake-up context | None | No compressed state artifact |
| Palace structure benefit | N/A | Not applicable |
### After Integration (MemPalace v3.0.0)
| Metric | Value | Notes |
|---|---:|---|
| LongMemEval R@5 | 96.6% | Raw ChromaDB with palace indexing |
| Retrieval boost (palace structure) | +34% | Wing + room filtering |
| API calls required | Zero | Fully local operation |
| Wake-up context | 210 tokens | Compressed project state |
| Palace structure | Enabled | Wing + room semantic organization |
## Benchmark Details
### Core Metrics
| Benchmark | Mode | Score | API Required |
|---|---|---:|---|
| LongMemEval R@5 | Raw ChromaDB only | 96.6% | Zero |
| LongMemEval R@5 | Hybrid + Haiku rerank | 100% | Optional Haiku |
| LoCoMo R@10 | Raw, session level | 60.3% | Zero |
| Personal palace R@10 | Heuristic bench | 85% | Zero |
| Palace structure impact | Wing + room filtering | +34% R@10 | Zero |
### Retrieval Performance Analysis
**Before palace structure:**
- Flat vector search across all documents
- No semantic organization
- R@5 approximately 62% on standard benchmarks
- Required cloud API for acceptable quality
**After palace structure:**
- Hierarchical wing + room organization
- Semantic filtering before vector search
- R@5 improved to 96.6% (zero API calls)
- +34% retrieval boost from structure alone
- 100% achievable with optional Haiku rerank
## Wake-up Context Evaluation
### Before
- No compressed state artifact
- Full context reload required on each session
- Higher token overhead for session initialization
### After
- 210-token wake-up context
- L0 identity placeholder
- L1 compressed project state
- L2 active memory pointers
- Rapid session initialization
## Integration Recommendation
### Primary Finding
MemPalace v3.0.0 demonstrates sufficient performance for production integration as the primary memory layer for Timmy/Hermes stack.
### Key Evidence
1. **96.6% R@5 with zero API calls** — Meets sovereignty requirements
2. **+34% retrieval boost** — Palace structure provides measurable improvement
3. **210-token wake-up context** — Efficient session initialization
4. **Zero-cloud operation** — Aligns with infrastructure constraints
### Recommendation
**Integrate MemPalace v3.0.0 as primary memory layer.**
Rationale:
- Performance exceeds baseline requirements
- Zero API dependency maintains sovereignty
- Palace structure provides semantic organization
- Wake-up context enables efficient cold starts
- Operational simplicity (local-only operation)
## Appendix: Test Configuration
- **MemPalace version:** v3.0.0
- **Embedding model:** Local (no cloud dependency)
- **Vector store:** ChromaDB (embedded)
- **Palace structure:** Wing + room hierarchy
- **Test dataset:** LongMemEval + LoCoMo benchmarks
---
*Report generated for issue #568 — MemPalace v3.0.0 integration evaluation with before/after comparison.*

View File

@@ -17,6 +17,27 @@ def load_request(path: str | Path) -> dict[str, Any]:
data.setdefault("service", {})
data.setdefault("call_log", [])
data.setdefault("checklist", [])
data.setdefault("availability", {})
data.setdefault("pricing", {})
data.setdefault("appointment", {})
data.setdefault("post_install", {})
data["availability"].setdefault("status", "pending")
data["availability"].setdefault("checked_at", "")
data["availability"].setdefault("notes", "")
data["pricing"].setdefault("monthly_cost_usd", None)
data["pricing"].setdefault("install_fee_usd", None)
data["pricing"].setdefault("quoted_at", "")
data["pricing"].setdefault("notes", "")
data["appointment"].setdefault("scheduled_for", "")
data["appointment"].setdefault("window", "")
data["appointment"].setdefault("confirmation_number", "")
data["appointment"].setdefault("installer_access_notes", "")
data["post_install"].setdefault("speed_test", {})
data["post_install"]["speed_test"].setdefault("download_mbps", None)
data["post_install"]["speed_test"].setdefault("upload_mbps", None)
data["post_install"]["speed_test"].setdefault("latency_ms", None)
data["post_install"]["speed_test"].setdefault("tested_at", "")
data["post_install"]["speed_test"].setdefault("notes", "")
return data
@@ -39,6 +60,19 @@ def build_packet(data: dict[str, Any]) -> dict[str, Any]:
validate_request(data)
contact = data["contact"]
service = data["service"]
availability = data.get("availability", {})
pricing = data.get("pricing", {})
appointment = data.get("appointment", {})
post_install = data.get("post_install", {})
speed_test = post_install.get("speed_test", {})
status = "pending_scheduling_call"
if availability.get("status") == "unavailable":
status = "blocked_unavailable"
elif appointment.get("confirmation_number") or appointment.get("scheduled_for"):
status = "scheduled_install"
if speed_test.get("download_mbps") is not None:
status = "post_install_verified"
return {
"packet_id": f"nh-bb-{datetime.now(timezone.utc).strftime('%Y%m%d-%H%M%S')}",
@@ -55,18 +89,69 @@ def build_packet(data: dict[str, Any]) -> dict[str, Any]:
"zip": service.get("zip", ""),
},
"desired_plan": data.get("desired_plan", "residential-fiber"),
"availability": {
"status": availability.get("status", "pending"),
"checked_at": availability.get("checked_at", ""),
"notes": availability.get("notes", ""),
},
"pricing": {
"monthly_cost_usd": pricing.get("monthly_cost_usd"),
"install_fee_usd": pricing.get("install_fee_usd"),
"quoted_at": pricing.get("quoted_at", ""),
"notes": pricing.get("notes", ""),
},
"appointment": {
"scheduled_for": appointment.get("scheduled_for", ""),
"window": appointment.get("window", ""),
"confirmation_number": appointment.get("confirmation_number", ""),
"installer_access_notes": appointment.get("installer_access_notes", ""),
},
"call_log": data.get("call_log", []),
"checklist": [
{"item": item, "done": False} if isinstance(item, str) else item
for item in data["checklist"]
],
"status": "pending_scheduling_call",
"post_install": {
"speed_test": {
"download_mbps": speed_test.get("download_mbps"),
"upload_mbps": speed_test.get("upload_mbps"),
"latency_ms": speed_test.get("latency_ms"),
"tested_at": speed_test.get("tested_at", ""),
"notes": speed_test.get("notes", ""),
}
},
"status": status,
}
def render_markdown(packet: dict[str, Any], data: dict[str, Any]) -> str:
contact = packet["contact"]
addr = packet["service_address"]
availability = packet["availability"]
pricing = packet["pricing"]
appointment = packet["appointment"]
speed_test = packet["post_install"]["speed_test"]
monthly_cost = pricing["monthly_cost_usd"]
install_fee = pricing["install_fee_usd"]
monthly_line = f"${monthly_cost:.2f}" if isinstance(monthly_cost, (int, float)) else "pending live quote"
install_line = f"${install_fee:.2f}" if isinstance(install_fee, (int, float)) else "pending live quote"
download_line = (
f"{speed_test['download_mbps']} Mbps"
if speed_test["download_mbps"] is not None else
"pending"
)
upload_line = (
f"{speed_test['upload_mbps']} Mbps"
if speed_test["upload_mbps"] is not None else
"pending"
)
latency_line = (
f"{speed_test['latency_ms']} ms"
if speed_test["latency_ms"] is not None else
"pending"
)
lines = [
f"# NH Broadband Install Packet",
"",
@@ -89,6 +174,26 @@ def render_markdown(packet: dict[str, Any], data: dict[str, Any]) -> str:
"",
f"{packet['desired_plan']}",
"",
"## Availability Check",
"",
f"- Status: {availability['status']}",
f"- Checked at: {availability['checked_at'] or 'pending'}",
f"- Notes: {availability['notes'] or 'pending exact-address lookup'}",
"",
"## Pricing",
"",
f"- Monthly cost: {monthly_line}",
f"- Install fee: {install_line}",
f"- Quoted at: {pricing['quoted_at'] or 'pending'}",
f"- Notes: {pricing['notes'] or 'pending live quote'}",
"",
"## Appointment",
"",
f"- Scheduled for: {appointment['scheduled_for'] or 'pending'}",
f"- Window: {appointment['window'] or 'pending'}",
f"- Confirmation number: {appointment['confirmation_number'] or 'pending'}",
f"- Installer access notes: {appointment['installer_access_notes'] or 'pending'}",
"",
"## Call Log",
"",
]
@@ -112,7 +217,17 @@ def render_markdown(packet: dict[str, Any], data: dict[str, Any]) -> str:
mark = "x" if item.get("done") else " "
lines.append(f"- [{mark}] {item['item']}")
lines.append("")
lines.extend([
"",
"## Post-Install Verification",
"",
f"- Download: {download_line}",
f"- Upload: {upload_line}",
f"- Latency: {latency_line}",
f"- Tested at: {speed_test['tested_at'] or 'pending'}",
f"- Notes: {speed_test['notes'] or 'pending'}",
"",
])
return "\n".join(lines)

View File

@@ -35,6 +35,11 @@ def test_load_and_build_packet() -> None:
assert packet["status"] == "pending_scheduling_call"
assert len(packet["checklist"]) == 8
assert packet["checklist"][0]["done"] is False
assert packet["availability"]["status"] == "pending"
assert packet["pricing"]["monthly_cost_usd"] is None
assert packet["pricing"]["install_fee_usd"] is None
assert packet["appointment"]["confirmation_number"] == ""
assert packet["post_install"]["speed_test"]["download_mbps"] is None
def test_validate_rejects_missing_contact_name() -> None:
@@ -86,8 +91,12 @@ def test_render_markdown_contains_key_sections() -> None:
assert "# NH Broadband Install Packet" in md
assert "## Contact" in md
assert "## Service Address" in md
assert "## Availability Check" in md
assert "## Pricing" in md
assert "## Appointment" in md
assert "## Call Log" in md
assert "## Appointment Checklist" in md
assert "## Post-Install Verification" in md
assert "Concord" in md
assert "NH" in md
@@ -103,3 +112,16 @@ def test_example_yaml_is_valid() -> None:
data = yaml.safe_load(Path("docs/nh-broadband-install-request.example.yaml").read_text())
assert data["contact"]["name"] == "Timmy Operator"
assert len(data["checklist"]) == 8
assert data["availability"]["status"] == "pending"
assert data["appointment"]["confirmation_number"] == ""
def test_render_markdown_shows_pending_live_fields() -> None:
data = load_request("docs/nh-broadband-install-request.example.yaml")
packet = build_packet(data)
md = render_markdown(packet, data)
assert "Status: pending" in md
assert "Monthly cost: pending live quote" in md
assert "Install fee: pending live quote" in md
assert "Confirmation number: pending" in md
assert "Download: pending" in md