From b0fb85dac37586c7927a7a1a94e59db81cd3111a Mon Sep 17 00:00:00 2001 From: Perplexity Computer Date: Wed, 18 Mar 2026 22:39:13 +0000 Subject: [PATCH] docs: integration plan for Timmy Dashboard + Agent Token Economy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Covers: - WebSocket gateway adapter (backend → Matrix protocol) - LiveWebSocket client to replace MockWebSocket - Agent token economy display (balances, costs, treasury) - 3D economy visuals (budget stress glow, sat flow particles) - 5-phase implementation roadmap with file change matrix - Open questions on auth, serving model, multi-operator --- INTEGRATION.md | 355 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 355 insertions(+) create mode 100644 INTEGRATION.md diff --git a/INTEGRATION.md b/INTEGRATION.md new file mode 100644 index 0000000..91053b0 --- /dev/null +++ b/INTEGRATION.md @@ -0,0 +1,355 @@ +# Integration Plan: The Matrix × Timmy Time + +**Date:** 2026-03-18 +**Author:** Perplexity Computer +**Status:** Draft — ready for dev team review + +--- + +## Overview + +The Matrix is a standalone 3D world that visualizes and commands the Timmy Time agent swarm. It currently runs on a mock WebSocket. This document defines how it integrates with the two other modes of the Timmy Time system: + +1. **Timmy Dashboard** — the existing FastAPI/HTMX mission control UI (`/home`, `/briefing`, `/tasks`, `/swarm`, `/spark`, `/market`, `/tools`, etc.) +2. **Agent Token Economy** — the Lightning Network-based economic layer where agents earn/spend sats for task execution + +The Matrix does not replace either system. It is a third interface — a spatial, persistent, iPad-native way to interact with the same backend. + +--- + +## Architecture + +``` +┌─────────────────────────────────────────────────────────┐ +│ TIMMY TIME BACKEND │ +│ │ +│ FastAPI ─── SQLite ─── Ollama ─── Lightning (LND/CLN) │ +│ │ │ │ │ │ +│ ▼ ▼ ▼ ▼ │ +│ /api/tasks /api/agents /api/chat /api/economy │ +│ │ │ │ │ │ +│ └───────────┴──────────┴────────────┘ │ +│ │ │ +│ WebSocket Gateway │ +│ ws://tower:8080/ws │ +│ │ │ +└──────────────────────┼──────────────────────────────────┘ + │ + ┌────────────┼────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌──────────┐ ┌──────────┐ ┌──────────┐ + │ Timmy │ │ The │ │ Telegram │ + │Dashboard │ │ Matrix │ │ Bot │ + │(HTMX/WS)│ │(Three.js)│ │ (bridge) │ + └──────────┘ └──────────┘ └──────────┘ +``` + +All three interfaces share the same backend. The Matrix connects via the WebSocket gateway, same as the dashboard's live feeds. + +--- + +## Mode 1: Timmy Dashboard Integration + +### What Already Exists (Dashboard) + +- `/tasks` — three-column queue (Pending, Running, Completed) with Approve/Modify/Veto +- `/swarm` — agent cards with status, live WebSocket log +- `/briefing` — morning briefing with task summaries +- `/home` — agent overview with TASK buttons +- SQLite `tasks` table with full lifecycle (pending_approval → approved → running → completed/vetoed/failed) +- WebSocket live feeds for task updates and agent state + +### What The Matrix Needs + +The Matrix already speaks the right protocol (defined in `PROTOCOL.md`). The integration points: + +#### 1. WebSocket Gateway Adapter + +**File:** `backend/ws_gateway.py` (new) + +The backend needs a WebSocket endpoint that translates between the existing FastAPI internals and the Matrix protocol format: + +| Matrix Message (Client → Server) | Backend Action | +|---|---| +| `{"type": "chat_message", "agent_id": "timmy", "content": "..."}` | Forward to agent chat endpoint, return streaming response | +| `{"type": "task_action", "task_id": "...", "action": "approve"}` | Call `PATCH /api/tasks/{id}/approve` | +| `{"type": "task_action", "task_id": "...", "action": "veto"}` | Call `PATCH /api/tasks/{id}/veto` | +| `{"type": "subscribe", "channels": ["agents", "tasks"]}` | Register client for push events | + +| Matrix Message (Server → Client) | Backend Trigger | +|---|---| +| `{"type": "agent_state", "agent_id": "timmy", "state": "working", ...}` | Agent state change in swarm coordinator | +| `{"type": "task_created", "task_id": "...", ...}` | New row in tasks table | +| `{"type": "task_update", "task_id": "...", "status": "completed"}` | Task status change | +| `{"type": "agent_message", "agent_id": "timmy", "content": "..."}` | Agent chat response (streamed) | +| `{"type": "memory_event", "agent_id": "...", ...}` | New memory entry logged | +| `{"type": "system_status", ...}` | Periodic heartbeat (every 5s) | + +**Implementation:** Wrap the existing WebSocket infrastructure (same one powering `/swarm` live log) with a JSON message router that speaks the Matrix protocol. + +#### 2. Replace MockWebSocket + +**File:** `js/websocket.js` → `js/websocket-live.js` + +```javascript +// Current: MockWebSocket simulates everything locally +// Target: Real WebSocket to backend + +export class LiveWebSocket { + constructor(url) { + this.url = url || `ws://${location.hostname}:8080/ws/matrix`; + this.ws = null; + this.handlers = {}; + this.reconnectDelay = 1000; + this._connect(); + } + + _connect() { + this.ws = new WebSocket(this.url); + this.ws.onmessage = (e) => { + const msg = JSON.parse(e.data); + this._dispatch(msg); + }; + this.ws.onclose = () => { + setTimeout(() => this._connect(), this.reconnectDelay); + this.reconnectDelay = Math.min(this.reconnectDelay * 2, 30000); + }; + this.ws.onopen = () => { + this.reconnectDelay = 1000; + this.send({ type: 'subscribe', channels: ['agents', 'tasks', 'system'] }); + }; + } + + send(msg) { + if (this.ws?.readyState === WebSocket.OPEN) { + this.ws.send(JSON.stringify(msg)); + } + } + // ... event handling same interface as MockWebSocket +} +``` + +**Key:** The `LiveWebSocket` must expose the exact same interface as `MockWebSocket` — same `.on()`, `.send()`, `.getAgent()`, `.getAgentTasks()` methods. This keeps `ui.js`, `main.js`, and all other files untouched. + +#### 3. Connection Config + +**File:** `js/config.js` (new) + +```javascript +export const CONFIG = { + // Auto-detect: if served from the backend, use relative WS + // If standalone, prompt or use saved endpoint + WS_URL: localStorage.getItem('matrix_ws_url') + || (location.port === '8080' ? `ws://${location.host}/ws/matrix` : null), + + // Fallback to mock mode for demo/development + USE_MOCK: false, +}; +``` + +**`main.js` change:** Import config, instantiate `LiveWebSocket` or `MockWebSocket` based on `CONFIG.USE_MOCK`. + +#### 4. Agent Registry Sync + +Current: 4 agents hardcoded in `AGENT_DEFS` (`websocket.js`) + +Target: Agents discovered from backend at connect time. The handshake response should include: + +```json +{ + "type": "connection", + "status": "connected", + "agents": { + "timmy": { "name": "Timmy", "role": "Main Orchestrator", "color": "#00ff41" }, + "forge": { "name": "Forge", "role": "Builder Agent", "color": "#ff8c00" }, + ... + } +} +``` + +This lets the backend define which agents exist without changing frontend code. + +--- + +## Mode 2: Agent Token Economy Integration + +### What the Token Economy Is + +From the Timmy Time spec: +- Agents earn sats for completing tasks +- Agents spend sats to use tools, call APIs (L402 gating) +- The human operator sets budgets and approves high-cost operations +- Lightning Network is the settlement layer +- Per-agent sat limits and daily global caps + +### Integration Points + +#### 1. Economy Data in Agent State + +Extend the `agent_state` message to include economic fields: + +```json +{ + "type": "agent_state", + "agent_id": "forge", + "state": "working", + "glow_intensity": 0.8, + "economy": { + "balance_sats": 15420, + "earned_today": 8500, + "spent_today": 3200, + "daily_limit": 50000, + "pending_invoices": 2 + } +} +``` + +#### 2. Status Tab — Economy Section + +Add economy rows to the Status tab when data is present: + +| Label | Value | +|---|---| +| BALANCE | ⚡ 15,420 sats | +| EARNED TODAY | ⚡ 8,500 | +| SPENT TODAY | ⚡ 3,200 | +| DAILY LIMIT | ⚡ 50,000 | +| PENDING | 2 invoices | + +Style the lightning bolt in `--matrix-warning` (orange) for visual pop. + +#### 3. Task Cost Display + +Extend task items in the Tasks tab to show cost: + +```json +{ + "task_id": "...", + "title": "Run sentiment analysis", + "status": "pending_approval", + "priority": "normal", + "estimated_cost_sats": 500, + "agent_id": "seer" +} +``` + +The Approve/Veto buttons become more meaningful when you see: "This task will cost ⚡ 500 sats." + +#### 4. Core System Panel — Treasury + +The Core (system status) panel should show aggregate economy: + +| Label | Value | +|---|---| +| TREASURY | ⚡ 142,800 sats | +| DAILY BURN | ⚡ 12,400 | +| DAILY EARN | ⚡ 18,200 | +| NET TODAY | +⚡ 5,800 | +| INVOICES PAID | 47 | +| INVOICES PENDING | 3 | + +#### 5. Visual Economy Indicators in 3D World + +**Agent glow color shift:** Agents burning through their budget fast could shift from green → orange → red glow. This is already supported — `glow_intensity` controls brightness, and the agent's base color is set per-agent. Add a `budget_stress` field (0.0–1.0) that blends the glow toward red. + +**Task object color:** Floating task cubes near agents could be colored by cost: +- Free / low cost → green +- Medium cost → orange +- High cost → red +- Pending approval → pulsing white + +**Connection lines:** When agent A is paying agent B (inter-agent economy), the connection line between them could pulse brighter, with small particles flowing along it to represent sat flow. + +#### 6. New Message Types + +| Message | Direction | Purpose | +|---|---|---| +| `economy_update` | Server → Client | Agent balance changed | +| `invoice_created` | Server → Client | Agent generated a Lightning invoice | +| `invoice_paid` | Server → Client | Invoice settled | +| `budget_alert` | Server → Client | Agent approaching daily limit | +| `set_budget` | Client → Server | Operator adjusts agent's daily limit | + +--- + +## Implementation Phases + +### Phase 1: Live WebSocket (Backend Team) +**Effort:** 1–2 days +- [ ] Create `ws_gateway.py` with Matrix protocol router +- [ ] Wire to existing task and agent state events +- [ ] Agent chat forwarding (Ollama responses streamed as `agent_message`) +- [ ] Handshake with dynamic agent registry + +### Phase 2: Frontend Switch (Matrix Repo) +**Effort:** 1 day +- [ ] Create `js/websocket-live.js` with same interface as mock +- [ ] Create `js/config.js` with endpoint config + mock fallback +- [ ] Update `js/main.js` to use config +- [ ] Test with backend WebSocket + +### Phase 3: Economy Display (Both) +**Effort:** 2–3 days +- [ ] Backend: Add economy fields to agent_state and task messages +- [ ] Frontend: Extend Status tab with economy rows +- [ ] Frontend: Add cost display to task items +- [ ] Frontend: Extend Core panel with treasury view +- [ ] Style lightning amounts with ⚡ prefix and orange accent + +### Phase 4: 3D Economy Visuals (Matrix Repo) +**Effort:** 2–3 days +- [ ] Budget stress → glow color blending in `agents.js` +- [ ] Task object coloring by cost +- [ ] Sat flow particles on connection lines +- [ ] Budget alert toast/notification overlay + +### Phase 5: Full Economy Controls (Both) +**Effort:** 2–3 days +- [ ] Set budget from panel UI +- [ ] Invoice approval flow in Tasks tab +- [ ] Transaction history in Memory tab (reuse memory UI for econ log) +- [ ] Emergency kill switch — freeze all agent spending from Core panel + +--- + +## File Changes Summary + +| File | Change | Phase | +|---|---|---| +| `js/websocket.js` | Keep as-is (mock mode) | — | +| `js/websocket-live.js` | New — real WebSocket client | 2 | +| `js/config.js` | New — connection config | 2 | +| `js/main.js` | Import config, conditional WS | 2 | +| `js/ui.js` | Economy rows in Status, cost in Tasks, treasury in Core | 3 | +| `js/agents.js` | Budget stress glow, task cost coloring, sat flow particles | 4 | +| `style.css` | Economy styling, lightning bolt accent | 3 | +| `PROTOCOL.md` | Add economy message types | 3 | +| `backend/ws_gateway.py` | New — WebSocket gateway (Timmy Time repo) | 1 | + +--- + +## Open Questions + +1. **Serving model:** Should The Matrix be served by the FastAPI backend (as a static route), or stay standalone and connect remotely? Standalone is cleaner for iPad PWA deployment but needs CORS config on the backend WS. + +2. **Authentication:** The Matrix WS should require auth before accepting commands. Options: + - Token in WS URL query param: `ws://tower:8080/ws/matrix?token=...` + - L402 auth (pay-to-connect, on-brand with the sovereignty model) + - Simple shared secret for now, upgrade to L402 later + +3. **Multi-operator:** If multiple iPads connect, should they see each other's cursors/selections? Not needed for v1 but worth designing for. + +4. **Agent hot-add:** When a new agent joins the swarm at runtime, the backend should push an `agent_joined` event so The Matrix can spawn the 3D avatar dynamically without reload. + +--- + +## Dependencies + +- Timmy Time backend must expose a WebSocket endpoint speaking the protocol in `PROTOCOL.md` +- Lightning integration (LND or CLN) must expose balance/invoice APIs to the backend +- The Matrix has zero backend dependencies — it's pure static files + WebSocket + +--- + +*This document lives in the `perplexity/the-matrix` repo. Backend implementation tasks should be tracked in the main Timmy Time repo.*