Perplexity Computer
22daaade36
- Added Fork Coordination section documenting replit/the-matrix changes - Acknowledged Replit resolved issues #1 (Vite), #2 (agent-defs), #6 (UUID) - Updated architecture diagram to reflect Vite + ES module structure - Documented Replit's module exports for each file - Aligned field names to camelCase (matching Replit's codebase) - Updated Phase 2 ownership: Replit primary, Perplexity review - Added coordination strategy: code flows replit→canonical, docs stay canonical - Resolved WS path decision (#10): VITE_WS_URL env var
450 lines
18 KiB
Markdown
450 lines
18 KiB
Markdown
# Integration Plan: The Matrix × Timmy Time
|
||
|
||
**Date:** 2026-03-18 (v2 — updated against live codebase + Replit fork)
|
||
**Author:** Perplexity Computer
|
||
**Status:** Active — cross-referenced with issues across all three repos
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
The Matrix is a standalone 3D world that visualizes and commands the Timmy Time agent swarm. It currently runs on a `MockWebSocket`. This document defines how it integrates with the two other systems:
|
||
|
||
1. **Timmy Dashboard** — FastAPI/HTMX mission control (`rockachopa/Timmy-time-dashboard`)
|
||
2. **Token-Gated Economy** — Lightning-based agent economy (`replit/token-gated-economy`)
|
||
|
||
The Matrix does not replace either system. It is a third interface — spatial, persistent, iPad-native.
|
||
|
||
### Related Issues
|
||
|
||
| Repo | Key Issues |
|
||
|---|---|
|
||
| `perplexity/the-matrix` | #1–#17 (full backlog) |
|
||
| `replit/token-gated-economy` | #1 (EPIC), #2 (WebSocket), #3 (SSE), #7 (3D env), #9 (chat UI) |
|
||
| `rockachopa/Timmy-time-dashboard` | #325 (cognitive state → Matrix), #326 (Timmy hands), #324 (three-phase loop) |
|
||
|
||
---
|
||
|
||
## Fork Coordination: `perplexity/the-matrix` ↔ `replit/the-matrix`
|
||
|
||
Replit forked the canonical repo and is actively developing on branch `feat/vite-build-agent-defs`. This creates a two-track development model:
|
||
|
||
### What Replit Has Done (as of 2026-03-18)
|
||
|
||
| Change | Addresses Issue | Details |
|
||
|---|---|---|
|
||
| Vite build system | #1 | `package.json` + `vite.config.js`, Three.js as npm dep (`0.171.0`) |
|
||
| Consolidated agent defs | #2 | New `js/agent-defs.js` — single source of truth for agent id/color/role/position |
|
||
| `crypto.randomUUID()` | #6 | Used in `websocket.js` subscribe handshake |
|
||
| ES module refactor | — | All files use `import/export`, Three.js via npm not CDN |
|
||
| CSS inlined | — | Styles moved into `index.html <style>`, `style.css` removed |
|
||
| `PROTOCOL.md` removed | — | Working from clean slate on their fork |
|
||
| `INTEGRATION.md` removed | — | Working from clean slate on their fork |
|
||
| Simplified interaction | — | `OrbitControls` via `three/addons`, much cleaner |
|
||
| WS URL via env var | #7 partial | `import.meta.env.VITE_WS_URL` — configurable at build time |
|
||
|
||
### Coordination Strategy
|
||
|
||
**Replit = upstream for code.** Their Vite refactor is a clean break. The canonical `perplexity/the-matrix` keeps docs (`PROTOCOL.md`, `INTEGRATION.md`) and the issue tracker. Replit's code will be the production codebase.
|
||
|
||
**Workflow:**
|
||
1. Replit develops features on their fork (`replit/the-matrix`)
|
||
2. Replit opens PRs to canonical (`perplexity/the-matrix`) when features are stable
|
||
3. Perplexity (this account) reviews, tests, and merges upstream
|
||
4. Issues stay on the canonical repo — both teams reference them
|
||
5. `INTEGRATION.md` and `PROTOCOL.md` live in canonical only — Replit reads them here
|
||
|
||
**Sync cadence:** Replit syncs their fork from canonical before starting new work. Cross-fork PRs merge at feature boundaries, not every commit.
|
||
|
||
### Issues Resolved by Replit's Branch
|
||
|
||
When the `feat/vite-build-agent-defs` branch is merged upstream, the following issues can be closed:
|
||
|
||
- **#1** — Build system: Vite replaces esm.sh CDN ✓
|
||
- **#2** — Agent definition consolidation ✓
|
||
- **#6** — `crypto.randomUUID()` ✓
|
||
- **#11/#12** — partially addressed (commit mentions them, needs verification)
|
||
|
||
---
|
||
|
||
## Architecture (Current State)
|
||
|
||
```
|
||
┌─── Timmy Tower (FastAPI + SQLite + Ollama) ──────────────────────┐
|
||
│ │
|
||
│ src/dashboard/ → HTMX pages, WebSocket live feeds │
|
||
│ src/timmy/ → Agent core, agentic loop, memory │
|
||
│ src/timmy/adapters/ → gitea_adapter, time_adapter (NEW) │
|
||
│ src/infrastructure/ → EventBus (async, SQLite-backed, │
|
||
│ wildcard subscriptions, replay) │
|
||
│ src/dashboard/routes/chat_api_v1.py → SSE chat, v1 API (NEW) │
|
||
│ │
|
||
└───────────────────────────────────────────────────────────────────┘
|
||
|
||
┌─── Token Economy (Express + PostgreSQL + LNbits) ─────────────────┐
|
||
│ │
|
||
│ artifacts/api-server/src/routes/ │
|
||
│ jobs.ts → eval invoice → work invoice → execute → refund │
|
||
│ sessions.ts → deposit → macaroon auth → balance management │
|
||
│ ui.ts → Timmy chat interface │
|
||
│ lib/ │
|
||
│ pricing.ts → cost-based pricing, BTC oracle, model rates │
|
||
│ lnbits.ts → Lightning invoices (stub mode available) │
|
||
│ agent.ts → Anthropic eval + work execution │
|
||
│ event-bus.ts → typed EventEmitter (job:*, session:*) │
|
||
│ stream-registry.ts → SSE stream management (PR #20) │
|
||
│ │
|
||
└────────────────────────────────────────────────────────────────────┘
|
||
|
||
┌─── The Matrix (Three.js + Vite + WebSocket) ─────────────────────┐
|
||
│ │
|
||
│ js/agent-defs.js → AGENT_DEFS single source of truth (NEW) │
|
||
│ js/websocket.js → WS client, VITE_WS_URL env var │
|
||
│ js/agents.js → Agent class, 3D avatars, connection lines │
|
||
│ js/ui.js → HUD, agent list, chat panel │
|
||
│ js/effects.js → Matrix rain particles, starfield │
|
||
│ js/interaction.js → OrbitControls (three/addons) │
|
||
│ js/main.js → Init + render loop │
|
||
│ js/world.js → Scene, camera, renderer, grid │
|
||
│ vite.config.js → Build config (esnext target) │
|
||
│ package.json → three@0.171.0, vite@^5.4.0 │
|
||
│ │
|
||
│ Docs (canonical repo only): │
|
||
│ PROTOCOL.md → WebSocket message spec │
|
||
│ INTEGRATION.md → This document │
|
||
│ │
|
||
└────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### Module Structure (Replit's Vite Refactor)
|
||
|
||
```
|
||
js/
|
||
agent-defs.js → exports: AGENT_DEFS[], colorToCss()
|
||
agents.js → exports: initAgents(), updateAgents(), setAgentState(),
|
||
getAgentCount(), getAgentDefs()
|
||
effects.js → exports: initEffects(), updateEffects()
|
||
interaction.js → exports: initInteraction(), updateControls()
|
||
main.js → entry point, imports all modules, runs animate loop
|
||
ui.js → exports: initUI(), updateUI(), appendChatMessage()
|
||
websocket.js → exports: initWebSocket(), getConnectionState(), getJobCount()
|
||
world.js → exports: initWorld(), onWindowResize()
|
||
```
|
||
|
||
Key design: `agent-defs.js` is the single source of truth. To add an agent, append one entry to `AGENT_DEFS`. No other file needs editing. WebSocket handler maps `msg.agentId` → agent defs for state and chat rendering.
|
||
|
||
---
|
||
|
||
## The Hard Problem: Two Payment Models
|
||
|
||
> Tracked in: `the-matrix#9`
|
||
|
||
The Matrix protocol currently assumes simple state transitions:
|
||
```
|
||
agent_state → idle | active
|
||
job_started → increment counter
|
||
job_completed → decrement counter
|
||
```
|
||
|
||
The Token Economy has a multi-step payment flow:
|
||
```
|
||
POST /api/jobs → eval invoice (10 sats)
|
||
→ pay eval invoice → Timmy evaluates
|
||
→ work invoice (variable sats, cost-based)
|
||
→ pay work invoice → Timmy executes
|
||
→ honest accounting → refund if overpaid
|
||
```
|
||
|
||
### Resolution: Extend WebSocket message types
|
||
|
||
The Matrix message handler (`websocket.js handleMessage()`) currently handles: `agent_state`, `job_started`, `job_completed`, `chat`, `agent_count`. New message types needed:
|
||
|
||
| Economy State | New WS message type | Matrix behavior |
|
||
|---|---|---|
|
||
| `created` | `invoice_request` (type=eval) | Show eval cost in chat panel |
|
||
| `evaluating` | `agent_state` (state=active) | Agent glows active |
|
||
| `awaiting_work_payment` | `invoice_request` (type=work) | Show work cost + pay button |
|
||
| `work_paid` | `invoice_settled` | Log in chat, start work animation |
|
||
| `completed` | `job_completed` (extended) | Show result + actual cost + refund |
|
||
| `rejected` | `job_completed` (with error) | Log rejection |
|
||
|
||
New message schemas for `PROTOCOL.md`:
|
||
|
||
```jsonc
|
||
// Server → Client: invoice for operator to pay
|
||
{
|
||
"type": "invoice_request",
|
||
"jobId": "...",
|
||
"invoiceType": "eval" | "work",
|
||
"paymentRequest": "lnbc...", // BOLT11
|
||
"amountSats": 500,
|
||
"description": "Work fee for: analyze sentiment..."
|
||
}
|
||
|
||
// Server → Client: payment confirmed
|
||
{
|
||
"type": "invoice_settled",
|
||
"jobId": "...",
|
||
"invoiceType": "eval" | "work"
|
||
}
|
||
|
||
// Server → Client: job completed with accounting
|
||
{
|
||
"type": "job_completed",
|
||
"jobId": "...",
|
||
"agentId": "alpha",
|
||
"result": "...",
|
||
"actualCostSats": 380,
|
||
"estimatedCostSats": 500,
|
||
"refundSats": 120
|
||
}
|
||
```
|
||
|
||
Note: field names use camelCase to match Replit's refactored code (`agentId`, not `agent_id`).
|
||
|
||
---
|
||
|
||
## WebSocket Path Alignment
|
||
|
||
> Tracked in: `the-matrix#10`
|
||
|
||
Three systems, three WS paths:
|
||
- Matrix (Replit refactor): `import.meta.env.VITE_WS_URL` — fully configurable
|
||
- Economy PR #20: SSE on HTTP (no WS path yet); issue #2/#15 track WS
|
||
- Dashboard: `ws://tower:8080/ws`
|
||
|
||
**Resolved:** Replit's refactor uses `VITE_WS_URL` env var. No hardcoded path. Set it at build/dev time:
|
||
|
||
```bash
|
||
VITE_WS_URL=ws://tower:8080/ws/matrix npm run dev
|
||
```
|
||
|
||
If empty, the Matrix runs disconnected (no mock — just shows OFFLINE).
|
||
|
||
---
|
||
|
||
## Phase 1: WebSocket Gateway
|
||
|
||
> Tracked in: `the-matrix#8`, `dashboard#325`
|
||
|
||
**Owner:** Dashboard team (Kimi or Claude)
|
||
**Effort:** 2–3 days
|
||
|
||
The dashboard's `infrastructure/events/bus.py` already provides:
|
||
- Typed `Event` dataclass with source, type, data, timestamp
|
||
- Async pub/sub with wildcard matching (`agent.task.*`)
|
||
- SQLite persistence + replay
|
||
- Existing subscribers: gitea_adapter, time_adapter
|
||
|
||
What's needed: A WebSocket endpoint that bridges the EventBus to Matrix clients.
|
||
|
||
```python
|
||
# src/infrastructure/ws_gateway/matrix_handler.py
|
||
|
||
@router.websocket("/ws/matrix")
|
||
async def matrix_ws(websocket: WebSocket):
|
||
await websocket.accept()
|
||
|
||
# Send agent registry on connect
|
||
await websocket.send_json({
|
||
"type": "connection",
|
||
"status": "connected",
|
||
"agents": get_agent_registry()
|
||
})
|
||
|
||
# Subscribe to relevant bus events
|
||
@bus.subscribe("agent.*")
|
||
async def on_agent_event(event: Event):
|
||
await websocket.send_json(translate_to_matrix(event))
|
||
|
||
@bus.subscribe("task.*")
|
||
async def on_task_event(event: Event):
|
||
await websocket.send_json(translate_to_matrix(event))
|
||
|
||
# Handle inbound Matrix messages
|
||
async for msg in websocket.iter_json():
|
||
await handle_matrix_message(msg)
|
||
```
|
||
|
||
The `translate_to_matrix()` function maps EventBus events → Matrix protocol messages (using the camelCase field names from Replit's codebase). The `handle_matrix_message()` function routes Matrix commands → internal APIs.
|
||
|
||
### Cognitive State Signal
|
||
|
||
Dashboard issue #325 asks for Timmy's cognitive state to drive his 3D avatar behavior. This feeds into the gateway:
|
||
|
||
```jsonc
|
||
// New event on the bus: timmy.cognitive_state
|
||
{
|
||
"type": "agent_state",
|
||
"agentId": "timmy",
|
||
"state": "deep_focus",
|
||
"focusTopic": "analyzing PR #315",
|
||
"engagement": 0.9,
|
||
"coherence": 0.85
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Phase 2: Frontend Integration
|
||
|
||
> Tracked in: `the-matrix#7`
|
||
|
||
**Owner:** Replit (primary), Perplexity (review)
|
||
**Effort:** 1–2 days
|
||
**Blocked on:** Phase 1 (need a real WS endpoint to connect to)
|
||
|
||
Replit's refactor already handles most of this:
|
||
- `websocket.js` connects to `VITE_WS_URL` with auto-reconnect
|
||
- `handleMessage()` dispatches by `msg.type`
|
||
- `setAgentState()` drives 3D avatar glow/animation
|
||
|
||
What remains:
|
||
- Add `invoice_request` / `invoice_settled` / extended `job_completed` handlers to `handleMessage()`
|
||
- Expose economy data to `ui.js` for payment UI
|
||
- Add connection handshake → populate agent registry from server (not just `AGENT_DEFS`)
|
||
|
||
---
|
||
|
||
## Phase 3: Economy Display
|
||
|
||
> Tracked in: `the-matrix#15`, `#17`
|
||
|
||
**Owner:** Perplexity (this repo) or Replit
|
||
**Effort:** 2–3 days
|
||
**Blocked on:** Phase 1 (need real data)
|
||
|
||
### Status Panel — Economy Section
|
||
|
||
Extend `ui.js updateUI()` to show economy data when available:
|
||
|
||
```
|
||
┌─────────────────────────────────────────┐
|
||
│ BALANCE ⚡ 15,420 sats │
|
||
│ EARNED TODAY ⚡ 8,500 │
|
||
│ SPENT TODAY ⚡ 3,200 │
|
||
│ DAILY LIMIT ⚡ 50,000 │
|
||
│ PENDING 2 invoices │
|
||
└─────────────────────────────────────────┘
|
||
```
|
||
|
||
### Chat Panel — Payment Flow
|
||
|
||
Payment events render in the existing chat panel via `appendChatMessage()`:
|
||
|
||
```
|
||
[SYS] JOB a4f2... eval invoice: ⚡10 sats
|
||
[SYS] JOB a4f2... eval paid — evaluating...
|
||
[ALPHA] Task looks feasible. Work cost: ⚡500 sats
|
||
[SYS] JOB a4f2... work invoice: ⚡500 sats [PAY]
|
||
[SYS] JOB a4f2... work paid — executing...
|
||
[ALPHA] Analysis complete. Actual cost: ⚡380 sats, refund: ⚡120 sats
|
||
```
|
||
|
||
### Core Panel — Treasury
|
||
|
||
New DOM element in `index.html`, rendered by `ui.js`:
|
||
|
||
```
|
||
┌─────────────────────────────────────────┐
|
||
│ TREASURY ⚡ 142,800 sats │
|
||
│ DAILY BURN ⚡ 12,400 │
|
||
│ NET TODAY +⚡ 5,800 │
|
||
│ SESSION BALANCE ⚡ 2,340 │
|
||
└─────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Phase 4: 3D Economy Visuals
|
||
|
||
> Tracked in: `the-matrix#13`, `#15`
|
||
|
||
**Owner:** Perplexity or Replit
|
||
**Effort:** 2–3 days
|
||
**Blocked on:** Phase 3
|
||
|
||
### Budget Stress Glow
|
||
|
||
In `agents.js` — the Agent class `update()` method already blends `emissiveIntensity` based on state. Extend to include `budgetStress` (0.0–1.0):
|
||
- 0.0: normal agent color
|
||
- 0.5: shifts toward warning orange
|
||
- 1.0: emergency red
|
||
|
||
### Sat Flow Particles
|
||
|
||
On `invoice_settled` events, animate particles along connection lines from payer → payee. Small glowing dots traveling the path. Use `effects.js` pattern (BufferGeometry + Points).
|
||
|
||
### Dynamic Agent Hot-Add (#12)
|
||
|
||
When `agent_joined` fires via WS, create a new Agent instance from the message data and call `scene.add()`. The `AGENT_DEFS` array becomes the default set; live agents can extend it.
|
||
|
||
---
|
||
|
||
## Phase 5: Session Mode
|
||
|
||
> Maps to: `economy/sessions.ts`
|
||
|
||
**Owner:** Both repos
|
||
**Effort:** 2–3 days
|
||
**Blocked on:** Phase 2
|
||
|
||
The economy supports session mode: deposit sats upfront, get a macaroon, chat freely until balance runs out. This is the iPad-native experience.
|
||
|
||
### Flow in The Matrix
|
||
|
||
1. First connection: "Deposit ⚡100–10,000 sats to start a session" prompt
|
||
2. Show Lightning invoice QR (render in chat panel or overlay)
|
||
3. On payment: session activates, macaroon stored in localStorage
|
||
4. Chat freely — each message deducts from session balance
|
||
5. Balance display in status panel, warning at minimum threshold
|
||
6. Session expiry countdown
|
||
|
||
---
|
||
|
||
## File Changes Summary
|
||
|
||
### Matrix repo (`replit/the-matrix` → merged to `perplexity/the-matrix`)
|
||
|
||
| File | Change | Phase | Issue |
|
||
|---|---|---|---|
|
||
| `js/websocket.js` | Add economy message handlers | 2 | #7, #9 |
|
||
| `js/ui.js` | Economy rows, payment chat, treasury panel | 3 | #15, #17 |
|
||
| `js/agents.js` | Budget stress glow, dynamic hot-add | 4 | #12, #13, #15 |
|
||
| `js/effects.js` | Sat flow particles | 4 | #13 |
|
||
| `js/agent-defs.js` | May become runtime-extensible for hot-add | 4 | #12 |
|
||
| `index.html` | Treasury DOM, PWA manifest link | 3 | #5, #17 |
|
||
| `PROTOCOL.md` | Add economy message types, cognitive state | 2 | #9 |
|
||
|
||
### Dashboard repo (`rockachopa/Timmy-time-dashboard`)
|
||
|
||
| File | Change | Phase |
|
||
|---|---|---|
|
||
| `src/infrastructure/ws_gateway/` | New — Matrix protocol bridge | 1 |
|
||
| `src/timmy/cognitive_state.py` | New — observable cognitive state | 1 |
|
||
|
||
### Economy repo (`replit/token-gated-economy`)
|
||
|
||
| File | Change | Phase |
|
||
|---|---|---|
|
||
| WebSocket endpoint | New — payment events over WS | 1 |
|
||
| `lib/event-bus.ts` | Emit to Matrix-format WS channel | 1 |
|
||
|
||
---
|
||
|
||
## Open Decisions
|
||
|
||
| # | Question | Options | Status |
|
||
|---|---|---|---|
|
||
| 1 | WS path | `VITE_WS_URL` env var — resolved | ✓ Resolved by Replit |
|
||
| 2 | Auth | Token param for now, L402 later | Proposed in #11 |
|
||
| 3 | Which backend? | Dashboard (Tower) or Economy (Replit) | Depends on deployment |
|
||
| 4 | Multi-operator | Defer to v2 | — |
|
||
| 5 | Agent hot-add | Backend pushes `agent_joined` event | Tracked in #12 |
|
||
| 6 | Session vs per-task | Support both, session default on iPad | Phase 5 |
|
||
| 7 | Fork merge cadence | Feature-boundary PRs, not every commit | Proposed above |
|
||
|
||
---
|
||
|
||
*This document lives in `perplexity/the-matrix` (canonical) and is the source of truth for how the three systems connect. Replit's fork reads this doc; code PRs flow from `replit/the-matrix` → `perplexity/the-matrix`. Backend tasks are cross-referenced to their respective repos via issue numbers.*
|