the-matrix/INTEGRATION.md

# Integration Plan: The Matrix × Timmy Time

**Date:** 2026-03-18 (v2 — updated against live codebase + Replit fork)
**Author:** Perplexity Computer
**Status:** Active — cross-referenced with issues across all three repos

---

## Overview

The Matrix is a standalone 3D world that visualizes and commands the Timmy Time agent swarm. It currently runs on a `MockWebSocket`. This document defines how it integrates with the two other systems:

1. **Timmy Dashboard** — FastAPI/HTMX mission control (`rockachopa/Timmy-time-dashboard`)
2. **Token-Gated Economy** — Lightning-based agent economy (`replit/token-gated-economy`)

The Matrix does not replace either system. It is a third interface — spatial, persistent, iPad-native.

### Related Issues

| Repo | Key Issues |
|---|---|
| `perplexity/the-matrix` | #1–#17 (full backlog) |
| `replit/token-gated-economy` | #1 (EPIC), #2 (WebSocket), #3 (SSE), #7 (3D env), #9 (chat UI) |
| `rockachopa/Timmy-time-dashboard` | #325 (cognitive state → Matrix), #326 (Timmy hands), #324 (three-phase loop) |

---

## Fork Coordination: `perplexity/the-matrix` ↔ `replit/the-matrix`

Replit forked the canonical repo and is actively developing on branch `feat/vite-build-agent-defs`. This creates a two-track development model:

### What Replit Has Done (as of 2026-03-18)

| Change | Addresses Issue | Details |
|---|---|---|
| Vite build system | #1 | `package.json` + `vite.config.js`, Three.js as npm dep (`0.171.0`) |
| Consolidated agent defs | #2 | New `js/agent-defs.js` — single source of truth for agent id/color/role/position |
| `crypto.randomUUID()` | #6 | Used in `websocket.js` subscribe handshake |
| ES module refactor | — | All files use `import/export`, Three.js via npm not CDN |
| CSS inlined | — | Styles moved into `index.html <style>`, `style.css` removed |
| `PROTOCOL.md` removed | — | Working from clean slate on their fork |
| `INTEGRATION.md` removed | — | Working from clean slate on their fork |
| Simplified interaction | — | `OrbitControls` via `three/addons`, much cleaner |
| WS URL via env var | #7 partial | `import.meta.env.VITE_WS_URL` — configurable at build time |

### Coordination Strategy

**Replit = upstream for code.** Their Vite refactor is a clean break. The canonical `perplexity/the-matrix` keeps docs (`PROTOCOL.md`, `INTEGRATION.md`) and the issue tracker. Replit's code will be the production codebase.

**Workflow:**
1. Replit develops features on their fork (`replit/the-matrix`)
2. Replit opens PRs to canonical (`perplexity/the-matrix`) when features are stable
3. Perplexity (this account) reviews, tests, and merges upstream
4. Issues stay on the canonical repo — both teams reference them
5. `INTEGRATION.md` and `PROTOCOL.md` live in canonical only — Replit reads them here

**Sync cadence:** Replit syncs their fork from canonical before starting new work. Cross-fork PRs merge at feature boundaries, not every commit.

### Issues Resolved by Replit's Branch

When the `feat/vite-build-agent-defs` branch is merged upstream, the following issues can be closed:

- **#1** — Build system: Vite replaces esm.sh CDN ✓
- **#2** — Agent definition consolidation ✓
- **#6** — `crypto.randomUUID()` ✓
- **#11/#12** — partially addressed (commit mentions them, needs verification)

---

## Architecture (Current State)

```
┌─── Timmy Tower (FastAPI + SQLite + Ollama) ──────────────────────┐
│                                                                   │
│  src/dashboard/        → HTMX pages, WebSocket live feeds         │
│  src/timmy/            → Agent core, agentic loop, memory         │
│  src/timmy/adapters/   → gitea_adapter, time_adapter (NEW)        │
│  src/infrastructure/   → EventBus (async, SQLite-backed,          │
│                          wildcard subscriptions, replay)           │
│  src/dashboard/routes/chat_api_v1.py → SSE chat, v1 API (NEW)    │
│                                                                   │
└───────────────────────────────────────────────────────────────────┘

┌─── Token Economy (Express + PostgreSQL + LNbits) ─────────────────┐
│                                                                    │
│  artifacts/api-server/src/routes/                                  │
│    jobs.ts     → eval invoice → work invoice → execute → refund    │
│    sessions.ts → deposit → macaroon auth → balance management      │
│    ui.ts       → Timmy chat interface                              │
│  lib/                                                              │
│    pricing.ts  → cost-based pricing, BTC oracle, model rates       │
│    lnbits.ts   → Lightning invoices (stub mode available)          │
│    agent.ts    → Anthropic eval + work execution                   │
│    event-bus.ts → typed EventEmitter (job:*, session:*)            │
│    stream-registry.ts → SSE stream management (PR #20)            │
│                                                                    │
└────────────────────────────────────────────────────────────────────┘

┌─── The Matrix (Three.js + Vite + WebSocket) ─────────────────────┐
│                                                                    │
│  js/agent-defs.js  → AGENT_DEFS single source of truth (NEW)      │
│  js/websocket.js   → WS client, VITE_WS_URL env var               │
│  js/agents.js      → Agent class, 3D avatars, connection lines     │
│  js/ui.js          → HUD, agent list, chat panel                   │
│  js/effects.js     → Matrix rain particles, starfield              │
│  js/interaction.js → OrbitControls (three/addons)                  │
│  js/main.js        → Init + render loop                            │
│  js/world.js       → Scene, camera, renderer, grid                 │
│  vite.config.js    → Build config (esnext target)                  │
│  package.json      → three@0.171.0, vite@^5.4.0                   │
│                                                                    │
│  Docs (canonical repo only):                                       │
│  PROTOCOL.md       → WebSocket message spec                        │
│  INTEGRATION.md    → This document                                 │
│                                                                    │
└────────────────────────────────────────────────────────────────────┘
```

### Module Structure (Replit's Vite Refactor)

```
js/
  agent-defs.js   → exports: AGENT_DEFS[], colorToCss()
  agents.js       → exports: initAgents(), updateAgents(), setAgentState(),
                              getAgentCount(), getAgentDefs()
  effects.js      → exports: initEffects(), updateEffects()
  interaction.js  → exports: initInteraction(), updateControls()
  main.js         → entry point, imports all modules, runs animate loop
  ui.js           → exports: initUI(), updateUI(), appendChatMessage()
  websocket.js    → exports: initWebSocket(), getConnectionState(), getJobCount()
  world.js        → exports: initWorld(), onWindowResize()
```

Key design: `agent-defs.js` is the single source of truth. To add an agent, append one entry to `AGENT_DEFS`. No other file needs editing. WebSocket handler maps `msg.agentId` → agent defs for state and chat rendering.

---

## The Hard Problem: Two Payment Models

> Tracked in: `the-matrix#9`

The Matrix protocol currently assumes simple state transitions:
```
agent_state → idle | active
job_started → increment counter
job_completed → decrement counter
```

The Token Economy has a multi-step payment flow:
```
POST /api/jobs → eval invoice (10 sats)
  → pay eval invoice → Timmy evaluates
  → work invoice (variable sats, cost-based)
  → pay work invoice → Timmy executes
  → honest accounting → refund if overpaid
```

### Resolution: Extend WebSocket message types

The Matrix message handler (`websocket.js handleMessage()`) currently handles: `agent_state`, `job_started`, `job_completed`, `chat`, `agent_count`. New message types needed:

| Economy State | New WS message type | Matrix behavior |
|---|---|---|
| `created` | `invoice_request` (type=eval) | Show eval cost in chat panel |
| `evaluating` | `agent_state` (state=active) | Agent glows active |
| `awaiting_work_payment` | `invoice_request` (type=work) | Show work cost + pay button |
| `work_paid` | `invoice_settled` | Log in chat, start work animation |
| `completed` | `job_completed` (extended) | Show result + actual cost + refund |
| `rejected` | `job_completed` (with error) | Log rejection |

New message schemas for `PROTOCOL.md`:

```jsonc
// Server → Client: invoice for operator to pay
{
  "type": "invoice_request",
  "jobId": "...",
  "invoiceType": "eval" | "work",
  "paymentRequest": "lnbc...",  // BOLT11
  "amountSats": 500,
  "description": "Work fee for: analyze sentiment..."
}

// Server → Client: payment confirmed
{
  "type": "invoice_settled",
  "jobId": "...",
  "invoiceType": "eval" | "work"
}

// Server → Client: job completed with accounting
{
  "type": "job_completed",
  "jobId": "...",
  "agentId": "alpha",
  "result": "...",
  "actualCostSats": 380,
  "estimatedCostSats": 500,
  "refundSats": 120
}
```

Note: field names use camelCase to match Replit's refactored code (`agentId`, not `agent_id`).

---

## WebSocket Path Alignment

> Tracked in: `the-matrix#10`

Three systems, three WS paths:
- Matrix (Replit refactor): `import.meta.env.VITE_WS_URL` — fully configurable
- Economy PR #20: SSE on HTTP (no WS path yet); issue #2/#15 track WS
- Dashboard: `ws://tower:8080/ws`

**Resolved:** Replit's refactor uses `VITE_WS_URL` env var. No hardcoded path. Set it at build/dev time:

```bash
VITE_WS_URL=ws://tower:8080/ws/matrix npm run dev
```

If empty, the Matrix runs disconnected (no mock — just shows OFFLINE).

---

## Phase 1: WebSocket Gateway

> Tracked in: `the-matrix#8`, `dashboard#325`

**Owner:** Dashboard team (Kimi or Claude)
**Effort:** 2–3 days

The dashboard's `infrastructure/events/bus.py` already provides:
- Typed `Event` dataclass with source, type, data, timestamp
- Async pub/sub with wildcard matching (`agent.task.*`)
- SQLite persistence + replay
- Existing subscribers: gitea_adapter, time_adapter

What's needed: A WebSocket endpoint that bridges the EventBus to Matrix clients.

```python
# src/infrastructure/ws_gateway/matrix_handler.py

@router.websocket("/ws/matrix")
async def matrix_ws(websocket: WebSocket):
    await websocket.accept()
    
    # Send agent registry on connect
    await websocket.send_json({
        "type": "connection",
        "status": "connected", 
        "agents": get_agent_registry()
    })
    
    # Subscribe to relevant bus events
    @bus.subscribe("agent.*")
    async def on_agent_event(event: Event):
        await websocket.send_json(translate_to_matrix(event))
    
    @bus.subscribe("task.*")  
    async def on_task_event(event: Event):
        await websocket.send_json(translate_to_matrix(event))
    
    # Handle inbound Matrix messages
    async for msg in websocket.iter_json():
        await handle_matrix_message(msg)
```

The `translate_to_matrix()` function maps EventBus events → Matrix protocol messages (using the camelCase field names from Replit's codebase). The `handle_matrix_message()` function routes Matrix commands → internal APIs.

### Cognitive State Signal

Dashboard issue #325 asks for Timmy's cognitive state to drive his 3D avatar behavior. This feeds into the gateway:

```jsonc
// New event on the bus: timmy.cognitive_state
{
  "type": "agent_state",
  "agentId": "timmy",
  "state": "deep_focus",
  "focusTopic": "analyzing PR #315",
  "engagement": 0.9,
  "coherence": 0.85
}
```

---

## Phase 2: Frontend Integration

> Tracked in: `the-matrix#7`

**Owner:** Replit (primary), Perplexity (review)
**Effort:** 1–2 days
**Blocked on:** Phase 1 (need a real WS endpoint to connect to)

Replit's refactor already handles most of this:
- `websocket.js` connects to `VITE_WS_URL` with auto-reconnect
- `handleMessage()` dispatches by `msg.type`
- `setAgentState()` drives 3D avatar glow/animation

What remains:
- Add `invoice_request` / `invoice_settled` / extended `job_completed` handlers to `handleMessage()`
- Expose economy data to `ui.js` for payment UI
- Add connection handshake → populate agent registry from server (not just `AGENT_DEFS`)

---

## Phase 3: Economy Display

> Tracked in: `the-matrix#15`, `#17`

**Owner:** Perplexity (this repo) or Replit
**Effort:** 2–3 days
**Blocked on:** Phase 1 (need real data)

### Status Panel — Economy Section

Extend `ui.js updateUI()` to show economy data when available:

```
┌─────────────────────────────────────────┐
│ BALANCE               ⚡ 15,420 sats    │
│ EARNED TODAY           ⚡ 8,500         │
│ SPENT TODAY            ⚡ 3,200         │
│ DAILY LIMIT            ⚡ 50,000        │
│ PENDING                2 invoices       │
└─────────────────────────────────────────┘
```

### Chat Panel — Payment Flow

Payment events render in the existing chat panel via `appendChatMessage()`:

```
[SYS] JOB a4f2... eval invoice: ⚡10 sats
[SYS] JOB a4f2... eval paid — evaluating...
[ALPHA] Task looks feasible. Work cost: ⚡500 sats
[SYS] JOB a4f2... work invoice: ⚡500 sats [PAY]
[SYS] JOB a4f2... work paid — executing...
[ALPHA] Analysis complete. Actual cost: ⚡380 sats, refund: ⚡120 sats
```

### Core Panel — Treasury

New DOM element in `index.html`, rendered by `ui.js`:

```
┌─────────────────────────────────────────┐
│ TREASURY              ⚡ 142,800 sats   │
│ DAILY BURN            ⚡ 12,400         │
│ NET TODAY             +⚡ 5,800         │
│ SESSION BALANCE       ⚡ 2,340          │
└─────────────────────────────────────────┘
```

---

## Phase 4: 3D Economy Visuals

> Tracked in: `the-matrix#13`, `#15`

**Owner:** Perplexity or Replit
**Effort:** 2–3 days
**Blocked on:** Phase 3

### Budget Stress Glow

In `agents.js` — the Agent class `update()` method already blends `emissiveIntensity` based on state. Extend to include `budgetStress` (0.0–1.0):
- 0.0: normal agent color
- 0.5: shifts toward warning orange  
- 1.0: emergency red

### Sat Flow Particles

On `invoice_settled` events, animate particles along connection lines from payer → payee. Small glowing dots traveling the path. Use `effects.js` pattern (BufferGeometry + Points).

### Dynamic Agent Hot-Add (#12)

When `agent_joined` fires via WS, create a new Agent instance from the message data and call `scene.add()`. The `AGENT_DEFS` array becomes the default set; live agents can extend it.

---

## Phase 5: Session Mode

> Maps to: `economy/sessions.ts`

**Owner:** Both repos
**Effort:** 2–3 days
**Blocked on:** Phase 2

The economy supports session mode: deposit sats upfront, get a macaroon, chat freely until balance runs out. This is the iPad-native experience.

### Flow in The Matrix

1. First connection: "Deposit ⚡100–10,000 sats to start a session" prompt
2. Show Lightning invoice QR (render in chat panel or overlay)
3. On payment: session activates, macaroon stored in localStorage
4. Chat freely — each message deducts from session balance
5. Balance display in status panel, warning at minimum threshold
6. Session expiry countdown

---

## File Changes Summary

### Matrix repo (`replit/the-matrix` → merged to `perplexity/the-matrix`)

| File | Change | Phase | Issue |
|---|---|---|---|
| `js/websocket.js` | Add economy message handlers | 2 | #7, #9 |
| `js/ui.js` | Economy rows, payment chat, treasury panel | 3 | #15, #17 |
| `js/agents.js` | Budget stress glow, dynamic hot-add | 4 | #12, #13, #15 |
| `js/effects.js` | Sat flow particles | 4 | #13 |
| `js/agent-defs.js` | May become runtime-extensible for hot-add | 4 | #12 |
| `index.html` | Treasury DOM, PWA manifest link | 3 | #5, #17 |
| `PROTOCOL.md` | Add economy message types, cognitive state | 2 | #9 |

### Dashboard repo (`rockachopa/Timmy-time-dashboard`)

| File | Change | Phase |
|---|---|---|
| `src/infrastructure/ws_gateway/` | New — Matrix protocol bridge | 1 |
| `src/timmy/cognitive_state.py` | New — observable cognitive state | 1 |

### Economy repo (`replit/token-gated-economy`)

| File | Change | Phase |
|---|---|---|
| WebSocket endpoint | New — payment events over WS | 1 |
| `lib/event-bus.ts` | Emit to Matrix-format WS channel | 1 |

---

## Open Decisions

| # | Question | Options | Status |
|---|---|---|---|
| 1 | WS path | `VITE_WS_URL` env var — resolved | ✓ Resolved by Replit |
| 2 | Auth | Token param for now, L402 later | Proposed in #11 |
| 3 | Which backend? | Dashboard (Tower) or Economy (Replit) | Depends on deployment |
| 4 | Multi-operator | Defer to v2 | — |
| 5 | Agent hot-add | Backend pushes `agent_joined` event | Tracked in #12 |
| 6 | Session vs per-task | Support both, session default on iPad | Phase 5 |
| 7 | Fork merge cadence | Feature-boundary PRs, not every commit | Proposed above |

---

*This document lives in `perplexity/the-matrix` (canonical) and is the source of truth for how the three systems connect. Replit's fork reads this doc; code PRs flow from `replit/the-matrix` → `perplexity/the-matrix`. Backend tasks are cross-referenced to their respective repos via issue numbers.*