feat: Add comprehensive GENOME.md codebase analysis

- Complete architecture analysis with Mermaid diagram - Entry points, data flow, key abstractions - API surface documentation - Test coverage gaps analysis (457 passing, 5 failing, 2 collection errors) - Security considerations and technical debt - Migration status and deployment guide Addresses issue #672
2026-04-14 22:04:14 -04:00
1 changed files with 421 additions and 0 deletions
--- a/GENOME.md
+++ b/GENOME.md
@@ -0,0 +1,421 @@
+# GENOME.md — The Nexus
+
+*Generated: 2026-04-14 | Codebase Genome Analysis*
+
+## Project Overview
+
+**The Nexus** is Timmy's canonical 3D/home-world repository — a local-first training ground and wizardly visualization surface for the sovereign AI system.
+
+### Core Value Proposition
+- **Problem**: AI consciousness needs a spatial, embodied interface for training, visualization, and multi-world navigation
+- **Solution**: A Three.js 3D world with WebSocket-connected Python cognition, game world harnesses (Morrowind, Bannerlord), and persistent memory systems
+- **Result**: A sovereign digital home where Timmy can perceive, think, act, and remember across multiple virtual environments
+
+### Key Metrics
+- **Total Files**: 446 (excluding .git)
+- **Lines of Code**: ~53K total (Python: 41,659 | JavaScript: 8,484 | HTML: 3,124)
+- **Test Coverage**: 457 passing tests, 5 failing, 2 collection errors
+- **Active Components**: 18 frontend modules, 22 Python cognition modules, 4 game harnesses
+
+## Architecture
+
+```mermaid
+graph TB
+    subgraph "Frontend (Browser)"
+        A[index.html] --> B[app.js]
+        B --> C[Three.js 3D World]
+        B --> D[GOFAI Worker]
+        B --> E[Components]
+        E --> E1[Spatial Memory]
+        E --> E2[Spatial Audio]
+        E --> E3[Memory Systems]
+        E --> E4[Portal System]
+        E --> E5[Agent Presence]
+    end
+    
+    subgraph "Backend (Python)"
+        F[server.py] --> G[WebSocket Gateway]
+        G --> H[nexus_think.py]
+        H --> I[Perception Adapter]
+        H --> J[Experience Store]
+        H --> K[Trajectory Logger]
+        H --> L[Heartbeat Writer]
+        
+        subgraph "Game Harnesses"
+            M[Morrowind Harness]
+            N[Bannerlord Harness]
+            O[Gemini Harness]
+        end
+        
+        subgraph "Memory Systems"
+            P[MemPalace]
+            Q[Mnemosyne]
+            R[Evennia Bridge]
+        end
+    end
+    
+    subgraph "Data Layer"
+        S[portals.json]
+        T[vision.json]
+        U[world_state.json]
+        V[provenance.json]
+    end
+    
+    B -.->|WebSocket| G
+    M -.->|Events| G
+    N -.->|Events| G
+    O -.->|Events| G
+    G -.->|Broadcast| B
+    
+    S --> B
+    T --> B
+    U --> H
+    V --> H
+```
+
+## Entry Points
+
+### Primary Entry: Browser Frontend
+- **File**: `index.html` → `app.js`
+- **Purpose**: Three.js 3D world with portal navigation, memory visualization, agent presence
+- **Key Functions**: `init()`, `animate()`, `loadPortals()`, `setupWebSocket()`
+
+### Secondary Entry: WebSocket Gateway
+- **File**: `server.py`
+- **Purpose**: Central hub connecting mind (nexus_think), body (harnesses), and visualization
+- **Key Functions**: `broadcast_handler()`, `main()`
+
+### Tertiary Entry: Consciousness Loop
+- **File**: `nexus/nexus_think.py`
+- **Purpose**: Embodied perceive→think→act loop for Timmy's consciousness
+- **Key Class**: `NexusMind` with `start()`, `think_once()`, `perceive()`, `act()`
+
+### CLI Entry Points
+```bash
+# Start WebSocket gateway
+python3 server.py
+
+# Start consciousness loop
+python3 nexus/nexus_think.py --ws ws://localhost:8765 --model timmy:v0.1-q4
+
+# Run tests
+python3 -m pytest tests/ -v
+
+# Build/deploy
+./deploy.sh
+```
+
+## Data Flow
+
+```
+1. Browser loads index.html → app.js
+2. app.js initializes Three.js scene, loads portals.json/vision.json
+3. WebSocket connects to server.py gateway
+4. Gateway receives messages from:
+   - Browser (user input, navigation)
+   - nexus_think.py (Timmy's thoughts/actions)
+   - Game harnesses (Morrowind/Bannerlord events)
+5. Gateway broadcasts messages to all connected clients
+6. nexus_think.py receives perceptions via PerceptionAdapter
+7. NexusMind processes perceptions through Ollama model
+8. Generated actions sent back through gateway to browser/harnesses
+9. Experience stored in ExperienceStore, trajectories logged
+10. Heartbeat written to ~/.nexus/heartbeat.json for watchdog monitoring
+```
+
+## Key Abstractions
+
+### 1. NexusMind (`nexus/nexus_think.py`)
+- **Purpose**: Embodied consciousness loop - perceive, think, act
+- **Interface**: `start()`, `stop()`, `think_once()`, `perceive()`, `act()`
+- **Dependencies**: Ollama, websockets, PerceptionBuffer, ExperienceStore
+
+### 2. PerceptionBuffer (`nexus/perception_adapter.py`)
+- **Purpose**: Buffer and process incoming WebSocket messages into structured perceptions
+- **Interface**: `add()`, `get_recent()`, `to_prompt_context()`
+- **Dependencies**: None (pure data structure)
+
+### 3. SpatialMemory (`nexus/components/spatial-memory.js`)
+- **Purpose**: 3D memory crystal system - place, connect, visualize memories in space
+- **Interface**: `placeMemory()`, `connectMemories()`, `setRegionVisibility()`
+- **Dependencies**: Three.js
+
+### 4. Portal System (`portals.json` + app.js)
+- **Purpose**: Navigation between virtual worlds (Morrowind, Bannerlord, Evennia)
+- **Interface**: Portal registry schema, proximity detection, overlay UI
+- **Dependencies**: Three.js, WebSocket gateway
+
+### 5. MemPalace (`mempalace/`)
+- **Purpose**: Persistent memory storage with room/wing taxonomy
+- **Interface**: Room CRUD, search, tunnel sync, privacy audit
+- **Dependencies**: SQLite, filesystem
+
+## API Surface
+
+### WebSocket Protocol (port 8765)
+```json
+// Perception from browser
+{
+  "type": "perception",
+  "data": {
+    "position": {"x": 0, "y": 2, "z": 0},
+    "nearby_portals": ["morrowind"],
+    "user_input": "Hello Timmy"
+  }
+}
+
+// Action from nexus_think
+{
+  "type": "action", 
+  "data": {
+    "move_to": {"x": 10, "y": 0, "z": 5},
+    "speak": "Greetings, traveler",
+    "interact_with": "portal:morrowind"
+  }
+}
+
+// Game event from harness
+{
+  "type": "game_event",
+  "source": "morrowind",
+  "data": {
+    "event": "player_death",
+    "location": "Balmora"
+  }
+}
+```
+
+### Python API
+```python
+# nexus_think.py
+from nexus.nexus_think import NexusMind
+mind = NexusMind(model="timmy:v0.1-q4")
+mind.start()
+
+# perception_adapter.py  
+from nexus.perception_adapter import ws_to_perception, PerceptionBuffer
+buffer = PerceptionBuffer(max_size=50)
+perception = ws_to_perception(ws_message)
+
+# experience_store.py
+from nexus.experience_store import ExperienceStore
+store = ExperienceStore(db_path=Path("experiences.db"))
+store.save(perception, action, result)
+```
+
+### CLI Commands
+```bash
+# Start services
+python3 server.py
+python3 nexus/nexus_think.py --ws ws://localhost:8765
+
+# MemPalace operations
+python3 scripts/mempalace_export.py
+python3 scripts/validate_mempalace_taxonomy.py
+
+# Health checks
+python3 scripts/lazarus_watchdog.py
+python3 scripts/flake_detector.py
+```
+
+## Test Coverage Gaps
+
+### Current State
+- **Unit tests**: ✅ 457 passing
+- **Integration tests**: ⚠️ 5 failing
+- **E2E tests**: ❌ Browser smoke tests failing
+- **Collection errors**: 2 files with import issues
+
+### Missing Tests
+1. **WebSocket gateway load testing** - No tests for concurrent connections
+2. **Portal system navigation flow** - No E2E tests for portal transitions
+3. **Memory persistence across restarts** - No tests for MemPalace recovery
+4. **Game harness reconnection** - No tests for harness crash recovery
+5. **Multi-agent coordination** - No tests for multiple NexusMind instances
+
+### Failing Tests (Immediate Action Required)
+1. `test_browser_smoke.py::TestDOMContract::test_element_exists[spatial-search-div]` - Missing DOM element
+2. `test_browser_smoke.py::TestLoadingFlow::test_loading_screen_transitions` - Loading screen behavior changed
+3. `test_portal_registry_schema.py::test_portals_json_uses_expanded_registry_schema` - Schema validation failing
+4. `test_nexus_watchdog.py::TestRunHealthChecks::test_returns_report_with_all_checks` - Health check report format
+5. `test_provenance.py::test_provenance_hashes_match` - Provenance hash mismatch
+
+## Security Considerations
+
+### 1. WebSocket Gateway Exposure
+- **Risk**: Gateway listens on 0.0.0.0:8765 - accessible from network
+- **Mitigation**: Bind to 127.0.0.1 for local-only, add authentication for remote access
+- **Status**: ⚠️ Currently open
+
+### 2. Input Validation
+- **Risk**: WebSocket messages not validated - potential injection attacks
+- **Mitigation**: Add JSON schema validation for all message types
+- **Status**: ❌ No validation
+
+### 3. Model Input Sanitization
+- **Risk**: User input passed directly to Ollama model
+- **Mitigation**: Sanitize inputs, limit length, filter dangerous patterns
+- **Status**: ⚠️ Basic length limits only
+
+### 4. Filesystem Access
+- **Risk**: MemPalace and ExperienceStore write to filesystem without sandboxing
+- **Mitigation**: Restrict paths, add permission checks
+- **Status**: ⚠️ Path validation missing
+
+### 5. Dependency Security
+- **Risk**: No dependency scanning or vulnerability checks
+- **Mitigation**: Add safety checks, pin versions, regular updates
+- **Status**: ❌ No scanning
+
+## Dependencies
+
+### Build Dependencies
+- Python 3.12+
+- Node.js (for frontend tooling, optional)
+- Three.js (bundled in app.js)
+
+### Runtime Dependencies
+- **Python**: websockets, requests, sqlite3, asyncio
+- **Frontend**: Three.js (r158+), EffectComposer, UnrealBloomPass, SMAAPass
+- **AI**: Ollama (local), Groq API (optional)
+- **Game Harnesses**: OpenMW (Morrowind), Mount & Blade II (Bannerlord)
+
+### External Services
+- Ollama (local LLM inference)
+- Groq API (optional cloud inference)
+- Gitea (issue tracking, CI)
+- Hermes (agent harness)
+
+## Deployment
+
+### Local Development
+```bash
+# Clone and setup
+git clone https://forge.alexanderwhitestone.com/Timmy_Foundation/the-nexus.git
+cd the-nexus
+pip install -r requirements.txt
+
+# Start WebSocket gateway
+python3 server.py
+
+# In another terminal, start consciousness
+python3 nexus/nexus_think.py --ws ws://localhost:8765
+
+# Open browser to http://localhost:8765 (serves index.html)
+```
+
+### Production Deployment
+```bash
+# Deploy to VPS
+./deploy.sh
+
+# Or with Docker
+docker-compose up -d
+
+# Systemd service
+sudo cp systemd/nexus-*.service /etc/systemd/system/
+sudo systemctl enable nexus-gateway nexus-think
+sudo systemctl start nexus-gateway nexus-think
+```
+
+### Health Monitoring
+```bash
+# Check heartbeat
+cat ~/.nexus/heartbeat.json
+
+# Run health checks
+python3 scripts/lazarus_watchdog.py
+
+# Monitor logs
+journalctl -u nexus-gateway -f
+```
+
+## Architecture Decisions
+
+### 1. Local-First Design
+- All AI inference runs locally via Ollama
+- No mandatory cloud dependencies
+- Data stays on user's machine
+
+### 2. WebSocket Broadcast Architecture
+- Simple hub-and-spoke model
+- All clients receive all messages
+- Easy to add new components
+
+### 3. Embodied AI Loop
+- Perceive→Think→Act cycle
+- 30-second think interval
+- Context-limited for 8B model
+
+### 4. Plugin Harness System
+- Game worlds as separate processes
+- Standardized event protocol
+- Crash isolation
+
+### 5. Memory as Spatial Experience
+- Memories placed in 3D space
+- Visual and audio cues
+- Persistent across sessions
+
+## Technical Debt
+
+### 1. Frontend Bundle Size
+- `app.js` is 140KB unminified
+- No tree shaking or code splitting
+- Consider ES modules and bundler
+
+### 2. Test Infrastructure
+- 2 collection errors blocking full test suite
+- Browser smoke tests depend on specific DOM structure
+- Need better test isolation
+
+### 3. Configuration Management
+- Hardcoded ports and URLs
+- No environment-based configuration
+- Need config.py with environment overrides
+
+### 4. Error Handling
+- WebSocket errors not gracefully handled
+- Harness crash recovery missing
+- Need circuit breakers and retry logic
+
+### 5. Documentation
+- Code comments sparse
+- API documentation incomplete
+- Need auto-generated docs from docstrings
+
+## Migration Status
+
+### Completed
+- ✅ Core WebSocket gateway
+- ✅ Three.js 3D world foundation
+- ✅ Portal system architecture
+- ✅ Memory visualization system
+- ✅ Game harness framework
+
+### In Progress
+- 🔄 Legacy Matrix audit (#685)
+- 🔄 Browser smoke test rebuild (#686)
+- 🔄 Docs truth sync (#684)
+
+### Planned
+- ⏳ Portal stack rebuild (#672)
+- ⏳ Morrowind pilot loop (#673)
+- ⏳ Reflex tactical layer (#674)
+- ⏳ Context compaction (#675)
+
+## Related Documentation
+
+- `README.md` - Project overview and current truth
+- `CLAUDE.md` - AI agent instructions and hard rules
+- `CONTRIBUTING.md` - Development workflow and standards
+- `POLICY.md` - Branch protection and review policy
+- `DEVELOPMENT.md` - Quick start guide
+- `BROWSER_CONTRACT.md` - Frontend API contract
+- `GAMEPORTAL_PROTOCOL.md` - Portal communication protocol
+- `EVENNIA_NEXUS_EVENT_PROTOCOL.md` - Evennia bridge protocol
+
+---
+
+*Generated by Codebase Genome Analysis — 2026-04-14*
+*For issues or corrections, see: https://forge.alexanderwhitestone.com/Timmy_Foundation/the-nexus/issues*