feat: wire llama health into night watch (#1123)

GENOME.md

@@ -1,421 +0,0 @@
-# GENOME.md — The Nexus
-
-*Generated: 2026-04-14 | Codebase Genome Analysis*
-
-## Project Overview
-
-**The Nexus** is Timmy's canonical 3D/home-world repository — a local-first training ground and wizardly visualization surface for the sovereign AI system.
-
-### Core Value Proposition
-- **Problem**: AI consciousness needs a spatial, embodied interface for training, visualization, and multi-world navigation
-- **Solution**: A Three.js 3D world with WebSocket-connected Python cognition, game world harnesses (Morrowind, Bannerlord), and persistent memory systems
-- **Result**: A sovereign digital home where Timmy can perceive, think, act, and remember across multiple virtual environments
-
-### Key Metrics
-- **Total Files**: 446 (excluding .git)
-- **Lines of Code**: ~53K total (Python: 41,659 | JavaScript: 8,484 | HTML: 3,124)
-- **Test Coverage**: 457 passing tests, 5 failing, 2 collection errors
-- **Active Components**: 18 frontend modules, 22 Python cognition modules, 4 game harnesses
-
-## Architecture
-
-```mermaid
-graph TB
-    subgraph "Frontend (Browser)"
-        A[index.html] --> B[app.js]
-        B --> C[Three.js 3D World]
-        B --> D[GOFAI Worker]
-        B --> E[Components]
-        E --> E1[Spatial Memory]
-        E --> E2[Spatial Audio]
-        E --> E3[Memory Systems]
-        E --> E4[Portal System]
-        E --> E5[Agent Presence]
-    end
-    
-    subgraph "Backend (Python)"
-        F[server.py] --> G[WebSocket Gateway]
-        G --> H[nexus_think.py]
-        H --> I[Perception Adapter]
-        H --> J[Experience Store]
-        H --> K[Trajectory Logger]
-        H --> L[Heartbeat Writer]
-        
-        subgraph "Game Harnesses"
-            M[Morrowind Harness]
-            N[Bannerlord Harness]
-            O[Gemini Harness]
-        end
-        
-        subgraph "Memory Systems"
-            P[MemPalace]
-            Q[Mnemosyne]
-            R[Evennia Bridge]
-        end
-    end
-    
-    subgraph "Data Layer"
-        S[portals.json]
-        T[vision.json]
-        U[world_state.json]
-        V[provenance.json]
-    end
-    
-    B -.->|WebSocket| G
-    M -.->|Events| G
-    N -.->|Events| G
-    O -.->|Events| G
-    G -.->|Broadcast| B
-    
-    S --> B
-    T --> B
-    U --> H
-    V --> H
-```
-
-## Entry Points
-
-### Primary Entry: Browser Frontend
-- **File**: `index.html` → `app.js`
-- **Purpose**: Three.js 3D world with portal navigation, memory visualization, agent presence
-- **Key Functions**: `init()`, `animate()`, `loadPortals()`, `setupWebSocket()`
-
-### Secondary Entry: WebSocket Gateway
-- **File**: `server.py`
-- **Purpose**: Central hub connecting mind (nexus_think), body (harnesses), and visualization
-- **Key Functions**: `broadcast_handler()`, `main()`
-
-### Tertiary Entry: Consciousness Loop
-- **File**: `nexus/nexus_think.py`
-- **Purpose**: Embodied perceive→think→act loop for Timmy's consciousness
-- **Key Class**: `NexusMind` with `start()`, `think_once()`, `perceive()`, `act()`
-
-### CLI Entry Points
-```bash
-# Start WebSocket gateway
-python3 server.py
-
-# Start consciousness loop
-python3 nexus/nexus_think.py --ws ws://localhost:8765 --model timmy:v0.1-q4
-
-# Run tests
-python3 -m pytest tests/ -v
-
-# Build/deploy
-./deploy.sh
-```
-
-## Data Flow
-
-```
-1. Browser loads index.html → app.js
-2. app.js initializes Three.js scene, loads portals.json/vision.json
-3. WebSocket connects to server.py gateway
-4. Gateway receives messages from:
-   - Browser (user input, navigation)
-   - nexus_think.py (Timmy's thoughts/actions)
-   - Game harnesses (Morrowind/Bannerlord events)
-5. Gateway broadcasts messages to all connected clients
-6. nexus_think.py receives perceptions via PerceptionAdapter
-7. NexusMind processes perceptions through Ollama model
-8. Generated actions sent back through gateway to browser/harnesses
-9. Experience stored in ExperienceStore, trajectories logged
-10. Heartbeat written to ~/.nexus/heartbeat.json for watchdog monitoring
-```
-
-## Key Abstractions
-
-### 1. NexusMind (`nexus/nexus_think.py`)
-- **Purpose**: Embodied consciousness loop - perceive, think, act
-- **Interface**: `start()`, `stop()`, `think_once()`, `perceive()`, `act()`
-- **Dependencies**: Ollama, websockets, PerceptionBuffer, ExperienceStore
-
-### 2. PerceptionBuffer (`nexus/perception_adapter.py`)
-- **Purpose**: Buffer and process incoming WebSocket messages into structured perceptions
-- **Interface**: `add()`, `get_recent()`, `to_prompt_context()`
-- **Dependencies**: None (pure data structure)
-
-### 3. SpatialMemory (`nexus/components/spatial-memory.js`)
-- **Purpose**: 3D memory crystal system - place, connect, visualize memories in space
-- **Interface**: `placeMemory()`, `connectMemories()`, `setRegionVisibility()`
-- **Dependencies**: Three.js
-
-### 4. Portal System (`portals.json` + app.js)
-- **Purpose**: Navigation between virtual worlds (Morrowind, Bannerlord, Evennia)
-- **Interface**: Portal registry schema, proximity detection, overlay UI
-- **Dependencies**: Three.js, WebSocket gateway
-
-### 5. MemPalace (`mempalace/`)
-- **Purpose**: Persistent memory storage with room/wing taxonomy
-- **Interface**: Room CRUD, search, tunnel sync, privacy audit
-- **Dependencies**: SQLite, filesystem
-
-## API Surface
-
-### WebSocket Protocol (port 8765)
-```json
-// Perception from browser
-{
-  "type": "perception",
-  "data": {
-    "position": {"x": 0, "y": 2, "z": 0},
-    "nearby_portals": ["morrowind"],
-    "user_input": "Hello Timmy"
-  }
-}
-
-// Action from nexus_think
-{
-  "type": "action", 
-  "data": {
-    "move_to": {"x": 10, "y": 0, "z": 5},
-    "speak": "Greetings, traveler",
-    "interact_with": "portal:morrowind"
-  }
-}
-
-// Game event from harness
-{
-  "type": "game_event",
-  "source": "morrowind",
-  "data": {
-    "event": "player_death",
-    "location": "Balmora"
-  }
-}
-```
-
-### Python API
-```python
-# nexus_think.py
-from nexus.nexus_think import NexusMind
-mind = NexusMind(model="timmy:v0.1-q4")
-mind.start()
-
-# perception_adapter.py  
-from nexus.perception_adapter import ws_to_perception, PerceptionBuffer
-buffer = PerceptionBuffer(max_size=50)
-perception = ws_to_perception(ws_message)
-
-# experience_store.py
-from nexus.experience_store import ExperienceStore
-store = ExperienceStore(db_path=Path("experiences.db"))
-store.save(perception, action, result)
-```
-
-### CLI Commands
-```bash
-# Start services
-python3 server.py
-python3 nexus/nexus_think.py --ws ws://localhost:8765
-
-# MemPalace operations
-python3 scripts/mempalace_export.py
-python3 scripts/validate_mempalace_taxonomy.py
-
-# Health checks
-python3 scripts/lazarus_watchdog.py
-python3 scripts/flake_detector.py
-```
-
-## Test Coverage Gaps
-
-### Current State
-- **Unit tests**: ✅ 457 passing
-- **Integration tests**: ⚠️ 5 failing
-- **E2E tests**: ❌ Browser smoke tests failing
-- **Collection errors**: 2 files with import issues
-
-### Missing Tests
-1. **WebSocket gateway load testing** - No tests for concurrent connections
-2. **Portal system navigation flow** - No E2E tests for portal transitions
-3. **Memory persistence across restarts** - No tests for MemPalace recovery
-4. **Game harness reconnection** - No tests for harness crash recovery
-5. **Multi-agent coordination** - No tests for multiple NexusMind instances
-
-### Failing Tests (Immediate Action Required)
-1. `test_browser_smoke.py::TestDOMContract::test_element_exists[spatial-search-div]` - Missing DOM element
-2. `test_browser_smoke.py::TestLoadingFlow::test_loading_screen_transitions` - Loading screen behavior changed
-3. `test_portal_registry_schema.py::test_portals_json_uses_expanded_registry_schema` - Schema validation failing
-4. `test_nexus_watchdog.py::TestRunHealthChecks::test_returns_report_with_all_checks` - Health check report format
-5. `test_provenance.py::test_provenance_hashes_match` - Provenance hash mismatch
-
-## Security Considerations
-
-### 1. WebSocket Gateway Exposure
-- **Risk**: Gateway listens on 0.0.0.0:8765 - accessible from network
-- **Mitigation**: Bind to 127.0.0.1 for local-only, add authentication for remote access
-- **Status**: ⚠️ Currently open
-
-### 2. Input Validation
-- **Risk**: WebSocket messages not validated - potential injection attacks
-- **Mitigation**: Add JSON schema validation for all message types
-- **Status**: ❌ No validation
-
-### 3. Model Input Sanitization
-- **Risk**: User input passed directly to Ollama model
-- **Mitigation**: Sanitize inputs, limit length, filter dangerous patterns
-- **Status**: ⚠️ Basic length limits only
-
-### 4. Filesystem Access
-- **Risk**: MemPalace and ExperienceStore write to filesystem without sandboxing
-- **Mitigation**: Restrict paths, add permission checks
-- **Status**: ⚠️ Path validation missing
-
-### 5. Dependency Security
-- **Risk**: No dependency scanning or vulnerability checks
-- **Mitigation**: Add safety checks, pin versions, regular updates
-- **Status**: ❌ No scanning
-
-## Dependencies
-
-### Build Dependencies
-- Python 3.12+
-- Node.js (for frontend tooling, optional)
-- Three.js (bundled in app.js)
-
-### Runtime Dependencies
-- **Python**: websockets, requests, sqlite3, asyncio
-- **Frontend**: Three.js (r158+), EffectComposer, UnrealBloomPass, SMAAPass
-- **AI**: Ollama (local), Groq API (optional)
-- **Game Harnesses**: OpenMW (Morrowind), Mount & Blade II (Bannerlord)
-
-### External Services
-- Ollama (local LLM inference)
-- Groq API (optional cloud inference)
-- Gitea (issue tracking, CI)
-- Hermes (agent harness)
-
-## Deployment
-
-### Local Development
-```bash
-# Clone and setup
-git clone https://forge.alexanderwhitestone.com/Timmy_Foundation/the-nexus.git
-cd the-nexus
-pip install -r requirements.txt
-
-# Start WebSocket gateway
-python3 server.py
-
-# In another terminal, start consciousness
-python3 nexus/nexus_think.py --ws ws://localhost:8765
-
-# Open browser to http://localhost:8765 (serves index.html)
-```
-
-### Production Deployment
-```bash
-# Deploy to VPS
-./deploy.sh
-
-# Or with Docker
-docker-compose up -d
-
-# Systemd service
-sudo cp systemd/nexus-*.service /etc/systemd/system/
-sudo systemctl enable nexus-gateway nexus-think
-sudo systemctl start nexus-gateway nexus-think
-```
-
-### Health Monitoring
-```bash
-# Check heartbeat
-cat ~/.nexus/heartbeat.json
-
-# Run health checks
-python3 scripts/lazarus_watchdog.py
-
-# Monitor logs
-journalctl -u nexus-gateway -f
-```
-
-## Architecture Decisions
-
-### 1. Local-First Design
-- All AI inference runs locally via Ollama
-- No mandatory cloud dependencies
-- Data stays on user's machine
-
-### 2. WebSocket Broadcast Architecture
-- Simple hub-and-spoke model
-- All clients receive all messages
-- Easy to add new components
-
-### 3. Embodied AI Loop
-- Perceive→Think→Act cycle
-- 30-second think interval
-- Context-limited for 8B model
-
-### 4. Plugin Harness System
-- Game worlds as separate processes
-- Standardized event protocol
-- Crash isolation
-
-### 5. Memory as Spatial Experience
-- Memories placed in 3D space
-- Visual and audio cues
-- Persistent across sessions
-
-## Technical Debt
-
-### 1. Frontend Bundle Size
-- `app.js` is 140KB unminified
-- No tree shaking or code splitting
-- Consider ES modules and bundler
-
-### 2. Test Infrastructure
-- 2 collection errors blocking full test suite
-- Browser smoke tests depend on specific DOM structure
-- Need better test isolation
-
-### 3. Configuration Management
-- Hardcoded ports and URLs
-- No environment-based configuration
-- Need config.py with environment overrides
-
-### 4. Error Handling
-- WebSocket errors not gracefully handled
-- Harness crash recovery missing
-- Need circuit breakers and retry logic
-
-### 5. Documentation
-- Code comments sparse
-- API documentation incomplete
-- Need auto-generated docs from docstrings
-
-## Migration Status
-
-### Completed
-- ✅ Core WebSocket gateway
-- ✅ Three.js 3D world foundation
-- ✅ Portal system architecture
-- ✅ Memory visualization system
-- ✅ Game harness framework
-
-### In Progress
-- 🔄 Legacy Matrix audit (#685)
-- 🔄 Browser smoke test rebuild (#686)
-- 🔄 Docs truth sync (#684)
-
-### Planned
-- ⏳ Portal stack rebuild (#672)
-- ⏳ Morrowind pilot loop (#673)
-- ⏳ Reflex tactical layer (#674)
-- ⏳ Context compaction (#675)
-
-## Related Documentation
-
-- `README.md` - Project overview and current truth
-- `CLAUDE.md` - AI agent instructions and hard rules
-- `CONTRIBUTING.md` - Development workflow and standards
-- `POLICY.md` - Branch protection and review policy
-- `DEVELOPMENT.md` - Quick start guide
-- `BROWSER_CONTRACT.md` - Frontend API contract
-- `GAMEPORTAL_PROTOCOL.md` - Portal communication protocol
-- `EVENNIA_NEXUS_EVENT_PROTOCOL.md` - Evennia bridge protocol
-
----
-
-*Generated by Codebase Genome Analysis — 2026-04-14*
-*For issues or corrections, see: https://forge.alexanderwhitestone.com/Timmy_Foundation/the-nexus/issues*

bin/night_watch.py

@@ -37,6 +37,8 @@ import shutil
 import subprocess
 import sys
 import time
+import urllib.error
+import urllib.request
 from datetime import datetime, timezone
 from pathlib import Path
 from typing import Optional
@@ -119,8 +121,6 @@ def _check_memory(threshold_pct: int = 90) -> tuple[str, str]:
 
 def _check_gitea_reachability(gitea_url: str = "https://forge.alexanderwhitestone.com") -> tuple[str, str]:
     """Return (status, detail) for Gitea HTTPS reachability."""
-    import urllib.request
-    import urllib.error
     try:
         with urllib.request.urlopen(gitea_url, timeout=10) as resp:
             code = resp.status
@@ -131,6 +131,21 @@ def _check_gitea_reachability(gitea_url: str = "https://forge.alexanderwhiteston
         return "WARN", f"Gitea unreachable: {exc}"
 
 
+def _check_llama_server(endpoint: str = "http://127.0.0.1:11435") -> tuple[str, str]:
+    """Return (status, detail) for the local llama.cpp server health endpoint."""
+    health_url = f"{endpoint.rstrip('/')}/health"
+    try:
+        req = urllib.request.Request(health_url, headers={"Accept": "application/json"})
+        with urllib.request.urlopen(req, timeout=10) as resp:
+            data = json.loads(resp.read().decode())
+        if data.get("status") == "ok":
+            model_name = Path(str(data.get("model_path", ""))).name or data.get("model", "unknown-model")
+            return "OK", f"llama-server healthy at {endpoint} ({model_name})"
+        return "WARN", f"llama-server unhealthy at {endpoint}: {data}"
+    except Exception as exc:
+        return "WARN", f"llama-server unreachable at {endpoint}: {exc}"
+
+
 def _check_world_readable_secrets() -> tuple[str, str]:
     """Return (status, detail) for world-readable sensitive files."""
     sensitive_patterns = ["*.key", "*.pem", "*.secret", ".env", "*.token"]
@@ -172,6 +187,9 @@ def generate_report(date_str: str, checker_mod) -> str:
     gitea_status, gitea_detail = _check_gitea_reachability()
     rows.append(("Alpha VPS", gitea_status, gitea_detail))
 
+    llama_status, llama_detail = _check_llama_server()
+    rows.append(("Local LLM", llama_status, llama_detail))
+
     sec_status, sec_detail = _check_world_readable_secrets()
     rows.append(("Security", sec_status, sec_detail))
 

docs/local-llm.md

@@ -40,6 +40,9 @@ Standardizes local LLM inference across the fleet using llama.cpp.
     curl -sf http://localhost:11435/health
     curl -s http://localhost:11435/v1/models
 
+Night Watch integration:
+- `bin/night_watch.py` probes the local llama.cpp `/health` endpoint and surfaces failures in the nightly report.
+
 ## Troubleshooting
 
 - Won't start → smaller model / lower quant

tests/test_night_watch_llama.py

@@ -0,0 +1,75 @@
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+from unittest.mock import patch
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+
+class _FakeResponse:
+    def __init__(self, payload: dict):
+        self.payload = json.dumps(payload).encode()
+
+    def read(self):
+        return self.payload
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc, tb):
+        return False
+
+
+class _FakeHeartbeatReport:
+    def to_panel_markdown(self):
+        return "## Heartbeat Panel\n\nAll jobs healthy."
+
+
+class _FakeChecker:
+    @staticmethod
+    def build_report():
+        return _FakeHeartbeatReport()
+
+
+def test_check_llama_server_reports_healthy_model():
+    import bin.night_watch as nw
+
+    with patch(
+        "bin.night_watch.urllib.request.urlopen",
+        return_value=_FakeResponse({"status": "ok", "model_path": "/opt/models/llama/Qwen2.5-7B-Instruct-Q4_K_M.gguf"}),
+    ):
+        status, detail = nw._check_llama_server("http://127.0.0.1:11435")
+
+    assert status == "OK"
+    assert "127.0.0.1:11435" in detail
+    assert "Qwen2.5-7B-Instruct-Q4_K_M.gguf" in detail
+
+
+def test_check_llama_server_reports_warning_on_failure():
+    import bin.night_watch as nw
+
+    with patch(
+        "bin.night_watch.urllib.request.urlopen",
+        side_effect=OSError("connection refused"),
+    ):
+        status, detail = nw._check_llama_server("http://127.0.0.1:11435")
+
+    assert status == "WARN"
+    assert "connection refused" in detail
+
+
+def test_generate_report_includes_local_llm_row():
+    import bin.night_watch as nw
+
+    with patch("bin.night_watch._check_service", return_value=("OK", "hermes-bezalel is active")), \
+         patch("bin.night_watch._check_disk", return_value=("OK", "disk usage 23%")), \
+         patch("bin.night_watch._check_memory", return_value=("OK", "memory usage 30%")), \
+         patch("bin.night_watch._check_gitea_reachability", return_value=("OK", "Gitea HTTPS is responding (200)")), \
+         patch("bin.night_watch._check_world_readable_secrets", return_value=("OK", "no sensitive recently-modified world-readable files found")), \
+         patch("bin.night_watch._check_llama_server", return_value=("OK", "llama-server healthy at http://127.0.0.1:11435 (Qwen2.5-7B-Instruct-Q4_K_M.gguf)")):
+        report = nw.generate_report("2026-04-15", _FakeChecker())
+
+    assert "| Local LLM | OK | llama-server healthy at http://127.0.0.1:11435 (Qwen2.5-7B-Instruct-Q4_K_M.gguf) |" in report
+    assert "## Heartbeat Panel" in report