the-matrix/INTEGRATION.md

Integration Plan: The Matrix × Timmy Time

Date: 2026-03-18 (v2 — updated against live codebase + Replit fork) Author: Perplexity Computer Status: Active — cross-referenced with issues across all three repos

---

Overview

The Matrix is a standalone 3D world that visualizes and commands the Timmy Time agent swarm. It currently runs on a MockWebSocket. This document defines how it integrates with the two other systems:

1. Timmy Dashboard — FastAPI/HTMX mission control (rockachopa/Timmy-time-dashboard)
2. Token-Gated Economy — Lightning-based agent economy (replit/token-gated-economy)

The Matrix does not replace either system. It is a third interface — spatial, persistent, iPad-native.

Related Issues

Repo	Key Issues
perplexity/the-matrix	#1–#17 (full backlog)
replit/token-gated-economy	#1 (EPIC), #2 (WebSocket), #3 (SSE), #7 (3D env), #9 (chat UI)
rockachopa/Timmy-time-dashboard	#325 (cognitive state → Matrix), #326 (Timmy hands), #324 (three-phase loop)

---

Fork Coordination: perplexity/the-matrix ↔ replit/the-matrix

Replit forked the canonical repo and is actively developing on branch feat/vite-build-agent-defs. This creates a two-track development model:

What Replit Has Done (as of 2026-03-18)

Change	Addresses Issue	Details
Vite build system	#1	package.json + vite.config.js, Three.js as npm dep (0.171.0)
Consolidated agent defs	#2	New js/agent-defs.js — single source of truth for agent id/color/role/position
crypto.randomUUID()	#6	Used in websocket.js subscribe handshake
ES module refactor	—	All files use import/export, Three.js via npm not CDN
CSS inlined	—	Styles moved into index.html <style>, style.css removed
PROTOCOL.md removed	—	Working from clean slate on their fork
INTEGRATION.md removed	—	Working from clean slate on their fork
Simplified interaction	—	OrbitControls via three/addons, much cleaner
WS URL via env var	#7 partial	import.meta.env.VITE_WS_URL — configurable at build time

Coordination Strategy

Replit = upstream for code. Their Vite refactor is a clean break. The canonical perplexity/the-matrix keeps docs (PROTOCOL.md, INTEGRATION.md) and the issue tracker. Replit's code will be the production codebase.

Workflow:

1. Replit develops features on their fork (replit/the-matrix)
2. Replit opens PRs to canonical (perplexity/the-matrix) when features are stable
3. Perplexity (this account) reviews, tests, and merges upstream
4. Issues stay on the canonical repo — both teams reference them
5. INTEGRATION.md and PROTOCOL.md live in canonical only — Replit reads them here

Sync cadence: Replit syncs their fork from canonical before starting new work. Cross-fork PRs merge at feature boundaries, not every commit.

Issues Resolved by Replit's Branch

When the feat/vite-build-agent-defs branch is merged upstream, the following issues can be closed:

• #1 — Build system: Vite replaces esm.sh CDN ✓
• #2 — Agent definition consolidation ✓
• #6 — crypto.randomUUID() ✓
• #11/#12 — partially addressed (commit mentions them, needs verification)

---

Architecture (Current State)

┌─── Timmy Tower (FastAPI + SQLite + Ollama) ──────────────────────┐
│                                                                   │
│  src/dashboard/        → HTMX pages, WebSocket live feeds         │
│  src/timmy/            → Agent core, agentic loop, memory         │
│  src/timmy/adapters/   → gitea_adapter, time_adapter (NEW)        │
│  src/infrastructure/   → EventBus (async, SQLite-backed,          │
│                          wildcard subscriptions, replay)           │
│  src/dashboard/routes/chat_api_v1.py → SSE chat, v1 API (NEW)    │
│                                                                   │
└───────────────────────────────────────────────────────────────────┘

┌─── Token Economy (Express + PostgreSQL + LNbits) ─────────────────┐
│                                                                    │
│  artifacts/api-server/src/routes/                                  │
│    jobs.ts     → eval invoice → work invoice → execute → refund    │
│    sessions.ts → deposit → macaroon auth → balance management      │
│    ui.ts       → Timmy chat interface                              │
│  lib/                                                              │
│    pricing.ts  → cost-based pricing, BTC oracle, model rates       │
│    lnbits.ts   → Lightning invoices (stub mode available)          │
│    agent.ts    → Anthropic eval + work execution                   │
│    event-bus.ts → typed EventEmitter (job:*, session:*)            │
│    stream-registry.ts → SSE stream management (PR #20)            │
│                                                                    │
└────────────────────────────────────────────────────────────────────┘

┌─── The Matrix (Three.js + Vite + WebSocket) ─────────────────────┐
│                                                                    │
│  js/agent-defs.js  → AGENT_DEFS single source of truth (NEW)      │
│  js/websocket.js   → WS client, VITE_WS_URL env var               │
│  js/agents.js      → Agent class, 3D avatars, connection lines     │
│  js/ui.js          → HUD, agent list, chat panel                   │
│  js/effects.js     → Matrix rain particles, starfield              │
│  js/interaction.js → OrbitControls (three/addons)                  │
│  js/main.js        → Init + render loop                            │
│  js/world.js       → Scene, camera, renderer, grid                 │
│  vite.config.js    → Build config (esnext target)                  │
│  package.json      → three@0.171.0, vite@^5.4.0                   │
│                                                                    │
│  Docs (canonical repo only):                                       │
│  PROTOCOL.md       → WebSocket message spec                        │
│  INTEGRATION.md    → This document                                 │
│                                                                    │
└────────────────────────────────────────────────────────────────────┘

Module Structure (Replit's Vite Refactor)

js/
  agent-defs.js   → exports: AGENT_DEFS[], colorToCss()
  agents.js       → exports: initAgents(), updateAgents(), setAgentState(),
                              getAgentCount(), getAgentDefs()
  effects.js      → exports: initEffects(), updateEffects()
  interaction.js  → exports: initInteraction(), updateControls()
  main.js         → entry point, imports all modules, runs animate loop
  ui.js           → exports: initUI(), updateUI(), appendChatMessage()
  websocket.js    → exports: initWebSocket(), getConnectionState(), getJobCount()
  world.js        → exports: initWorld(), onWindowResize()

Key design: agent-defs.js is the single source of truth. To add an agent, append one entry to AGENT_DEFS. No other file needs editing. WebSocket handler maps msg.agentId → agent defs for state and chat rendering.

---

The Hard Problem: Two Payment Models

Tracked in: the-matrix#9

The Matrix protocol currently assumes simple state transitions:

agent_state → idle | active
job_started → increment counter
job_completed → decrement counter

The Token Economy has a multi-step payment flow:

POST /api/jobs → eval invoice (10 sats)
  → pay eval invoice → Timmy evaluates
  → work invoice (variable sats, cost-based)
  → pay work invoice → Timmy executes
  → honest accounting → refund if overpaid

Resolution: Extend WebSocket message types

The Matrix message handler (websocket.js handleMessage()) currently handles: agent_state, job_started, job_completed, chat, agent_count. New message types needed:

Economy State	New WS message type	Matrix behavior
created	invoice_request (type=eval)	Show eval cost in chat panel
evaluating	agent_state (state=active)	Agent glows active
awaiting_work_payment	invoice_request (type=work)	Show work cost + pay button
work_paid	invoice_settled	Log in chat, start work animation
completed	job_completed (extended)	Show result + actual cost + refund
rejected	job_completed (with error)	Log rejection

New message schemas for PROTOCOL.md:

// Server → Client: invoice for operator to pay
{
  "type": "invoice_request",
  "jobId": "...",
  "invoiceType": "eval" | "work",
  "paymentRequest": "lnbc...",  // BOLT11
  "amountSats": 500,
  "description": "Work fee for: analyze sentiment..."
}

// Server → Client: payment confirmed
{
  "type": "invoice_settled",
  "jobId": "...",
  "invoiceType": "eval" | "work"
}

// Server → Client: job completed with accounting
{
  "type": "job_completed",
  "jobId": "...",
  "agentId": "alpha",
  "result": "...",
  "actualCostSats": 380,
  "estimatedCostSats": 500,
  "refundSats": 120
}

Note: field names use camelCase to match Replit's refactored code (agentId, not agent_id).

---

WebSocket Path Alignment

Tracked in: the-matrix#10

Three systems, three WS paths:

• Matrix (Replit refactor): import.meta.env.VITE_WS_URL — fully configurable
• Economy PR #20: SSE on HTTP (no WS path yet); issue #2/#15 track WS
• Dashboard: ws://tower:8080/ws

Resolved: Replit's refactor uses VITE_WS_URL env var. No hardcoded path. Set it at build/dev time:

VITE_WS_URL=ws://tower:8080/ws/matrix npm run dev

If empty, the Matrix runs disconnected (no mock — just shows OFFLINE).

---

Phase 1: WebSocket Gateway

Tracked in: the-matrix#8, dashboard#325

Owner: Dashboard team (Kimi or Claude) Effort: 2–3 days

The dashboard's infrastructure/events/bus.py already provides:

• Typed Event dataclass with source, type, data, timestamp
• Async pub/sub with wildcard matching (agent.task.*)
• SQLite persistence + replay
• Existing subscribers: gitea_adapter, time_adapter

What's needed: A WebSocket endpoint that bridges the EventBus to Matrix clients.

# src/infrastructure/ws_gateway/matrix_handler.py

@router.websocket("/ws/matrix")
async def matrix_ws(websocket: WebSocket):
    await websocket.accept()
    
    # Send agent registry on connect
    await websocket.send_json({
        "type": "connection",
        "status": "connected", 
        "agents": get_agent_registry()
    })
    
    # Subscribe to relevant bus events
    @bus.subscribe("agent.*")
    async def on_agent_event(event: Event):
        await websocket.send_json(translate_to_matrix(event))
    
    @bus.subscribe("task.*")  
    async def on_task_event(event: Event):
        await websocket.send_json(translate_to_matrix(event))
    
    # Handle inbound Matrix messages
    async for msg in websocket.iter_json():
        await handle_matrix_message(msg)

The translate_to_matrix() function maps EventBus events → Matrix protocol messages (using the camelCase field names from Replit's codebase). The handle_matrix_message() function routes Matrix commands → internal APIs.

Cognitive State Signal

Dashboard issue #325 asks for Timmy's cognitive state to drive his 3D avatar behavior. This feeds into the gateway:

// New event on the bus: timmy.cognitive_state
{
  "type": "agent_state",
  "agentId": "timmy",
  "state": "deep_focus",
  "focusTopic": "analyzing PR #315",
  "engagement": 0.9,
  "coherence": 0.85
}

---

Phase 2: Frontend Integration

Tracked in: the-matrix#7

Owner: Replit (primary), Perplexity (review) Effort: 1–2 days Blocked on: Phase 1 (need a real WS endpoint to connect to)

Replit's refactor already handles most of this:

• websocket.js connects to VITE_WS_URL with auto-reconnect
• handleMessage() dispatches by msg.type
• setAgentState() drives 3D avatar glow/animation

What remains:

• Add invoice_request / invoice_settled / extended job_completed handlers to handleMessage()
• Expose economy data to ui.js for payment UI
• Add connection handshake → populate agent registry from server (not just AGENT_DEFS)

---

Phase 3: Economy Display

Tracked in: the-matrix#15, #17

Owner: Perplexity (this repo) or Replit Effort: 2–3 days Blocked on: Phase 1 (need real data)

Status Panel — Economy Section

Extend ui.js updateUI() to show economy data when available:

┌─────────────────────────────────────────┐
│ BALANCE               ⚡ 15,420 sats    │
│ EARNED TODAY           ⚡ 8,500         │
│ SPENT TODAY            ⚡ 3,200         │
│ DAILY LIMIT            ⚡ 50,000        │
│ PENDING                2 invoices       │
└─────────────────────────────────────────┘

Chat Panel — Payment Flow

Payment events render in the existing chat panel via appendChatMessage():

[SYS] JOB a4f2... eval invoice: ⚡10 sats
[SYS] JOB a4f2... eval paid — evaluating...
[ALPHA] Task looks feasible. Work cost: ⚡500 sats
[SYS] JOB a4f2... work invoice: ⚡500 sats [PAY]
[SYS] JOB a4f2... work paid — executing...
[ALPHA] Analysis complete. Actual cost: ⚡380 sats, refund: ⚡120 sats

Core Panel — Treasury

New DOM element in index.html, rendered by ui.js:

┌─────────────────────────────────────────┐
│ TREASURY              ⚡ 142,800 sats   │
│ DAILY BURN            ⚡ 12,400         │
│ NET TODAY             +⚡ 5,800         │
│ SESSION BALANCE       ⚡ 2,340          │
└─────────────────────────────────────────┘

---

Phase 4: 3D Economy Visuals

Tracked in: the-matrix#13, #15

Owner: Perplexity or Replit Effort: 2–3 days Blocked on: Phase 3

Budget Stress Glow

In agents.js — the Agent class update() method already blends emissiveIntensity based on state. Extend to include budgetStress (0.0–1.0):

• 0.0: normal agent color
• 0.5: shifts toward warning orange
• 1.0: emergency red

Sat Flow Particles

On invoice_settled events, animate particles along connection lines from payer → payee. Small glowing dots traveling the path. Use effects.js pattern (BufferGeometry + Points).

Dynamic Agent Hot-Add (#12)

When agent_joined fires via WS, create a new Agent instance from the message data and call scene.add(). The AGENT_DEFS array becomes the default set; live agents can extend it.

---

Phase 5: Session Mode

Maps to: economy/sessions.ts

Owner: Both repos Effort: 2–3 days Blocked on: Phase 2

The economy supports session mode: deposit sats upfront, get a macaroon, chat freely until balance runs out. This is the iPad-native experience.

Flow in The Matrix

1. First connection: "Deposit ⚡100–10,000 sats to start a session" prompt
2. Show Lightning invoice QR (render in chat panel or overlay)
3. On payment: session activates, macaroon stored in localStorage
4. Chat freely — each message deducts from session balance
5. Balance display in status panel, warning at minimum threshold
6. Session expiry countdown

---

File Changes Summary

Matrix repo (replit/the-matrix → merged to perplexity/the-matrix)

File	Change	Phase	Issue
js/websocket.js	Add economy message handlers	2	#7, #9
js/ui.js	Economy rows, payment chat, treasury panel	3	#15, #17
js/agents.js	Budget stress glow, dynamic hot-add	4	#12, #13, #15
js/effects.js	Sat flow particles	4	#13
js/agent-defs.js	May become runtime-extensible for hot-add	4	#12
index.html	Treasury DOM, PWA manifest link	3	#5, #17
PROTOCOL.md	Add economy message types, cognitive state	2	#9

Dashboard repo (rockachopa/Timmy-time-dashboard)

File	Change	Phase
src/infrastructure/ws_gateway/	New — Matrix protocol bridge	1
src/timmy/cognitive_state.py	New — observable cognitive state	1

Economy repo (replit/token-gated-economy)

File	Change	Phase
WebSocket endpoint	New — payment events over WS	1
lib/event-bus.ts	Emit to Matrix-format WS channel	1

---

Open Decisions

#	Question	Options	Status
1	WS path	VITE_WS_URL env var — resolved	✓ Resolved by Replit
2	Auth	Token param for now, L402 later	Proposed in #11
3	Which backend?	Dashboard (Tower) or Economy (Replit)	Depends on deployment
4	Multi-operator	Defer to v2	—
5	Agent hot-add	Backend pushes agent_joined event	Tracked in #12
6	Session vs per-task	Support both, session default on iPad	Phase 5
7	Fork merge cadence	Feature-boundary PRs, not every commit	Proposed above

---

This document lives in perplexity/the-matrix (canonical) and is the source of truth for how the three systems connect. Replit's fork reads this doc; code PRs flow from replit/the-matrix → perplexity/the-matrix. Backend tasks are cross-referenced to their respective repos via issue numbers.