Compare commits
26 Commits
feat/mnemo
...
mimo/code/
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
536744ec21 | ||
| cc4af009c7 | |||
| 089b06b6f8 | |||
| 8beae5ecc1 | |||
| e2edfd3318 | |||
| 8e18fa5311 | |||
| 1bf2af15a0 | |||
| 4095946749 | |||
|
|
845e2f2ced | ||
|
|
60af11ec2f | ||
| c387708892 | |||
| 8694c0f5ad | |||
| c3547196d8 | |||
| 87bfe9b332 | |||
| a0964a2fbf | |||
| 1e7bb2a453 | |||
| 847c4d50d4 | |||
|
|
220f20c794 | ||
| e85cefd9c0 | |||
| beec49a92d | |||
| ef25c073ce | |||
| 5ce928a00d | |||
| 61871cf6ed | |||
| 6f949698fe | |||
| 6cf1f4d078 | |||
|
|
ef74536e33 |
3
.gitignore
vendored
3
.gitignore
vendored
@@ -4,3 +4,6 @@ nexus/__pycache__/
|
||||
tests/__pycache__/
|
||||
mempalace/__pycache__/
|
||||
.aider*
|
||||
|
||||
# Prevent agents from writing to wrong path (see issue #1145)
|
||||
public/nexus/
|
||||
|
||||
12
CLAUDE.md
12
CLAUDE.md
@@ -42,6 +42,17 @@ Current repo contents are centered on:
|
||||
Do not tell contributors to run Vite or edit a nonexistent root frontend on current `main`.
|
||||
If browser/UI work is being restored, it must happen through the migration backlog and land back here.
|
||||
|
||||
## Canonical File Paths
|
||||
|
||||
**Frontend code lives at repo ROOT, NOT in `public/nexus/`:**
|
||||
- `app.js` — main Three.js app (GOFAI, 3D world, all frontend logic)
|
||||
- `index.html` — main HTML shell
|
||||
- `style.css` — styles
|
||||
- `server.py` — websocket bridge
|
||||
- `gofai_worker.js` — web worker for off-thread reasoning
|
||||
|
||||
**DO NOT write to `public/nexus/`** — this path is gitignored. Agents historically wrote here by mistake, creating corrupt duplicates. See issue #1145 and `INVESTIGATION_ISSUE_1145.md`.
|
||||
|
||||
## Hard Rules
|
||||
|
||||
1. One canonical 3D repo only: `Timmy_Foundation/the-nexus`
|
||||
@@ -50,6 +61,7 @@ If browser/UI work is being restored, it must happen through the migration backl
|
||||
4. Telemetry and durable truth flow through Hermes harness
|
||||
5. OpenClaw remains a sidecar, not the governing authority
|
||||
6. Before claiming visual validation, prove the app being viewed actually comes from current `the-nexus`
|
||||
7. **NEVER write frontend files to `public/nexus/`** — use repo root paths listed above
|
||||
|
||||
## Validation Rule
|
||||
|
||||
|
||||
72
INVESTIGATION_ISSUE_1145.md
Normal file
72
INVESTIGATION_ISSUE_1145.md
Normal file
@@ -0,0 +1,72 @@
|
||||
# Investigation Report: Missing Source Code — Classical AI Commits Disappearing
|
||||
|
||||
**Issue:** #1145
|
||||
**Date:** 2026-04-10
|
||||
**Investigator:** mimo-v2-pro swarm worker
|
||||
|
||||
## Summary
|
||||
|
||||
**The classical AI code is NOT missing. It is fully present in root `app.js` (3302 lines).**
|
||||
|
||||
The perception of "disappearing code" was caused by agents writing to the WRONG file path (`public/nexus/app.js` instead of root `app.js`), creating corrupt duplicate files that were repeatedly overwritten and eventually deleted.
|
||||
|
||||
## Root Cause
|
||||
|
||||
**Explanation #1 confirmed: Duplicate agents on different machines overwriting each other's commits.**
|
||||
|
||||
Multiple Google AI Agent instances wrote GOFAI implementations to `public/nexus/app.js` — a path that does not correspond to the canonical app structure. These commits kept overwriting each other:
|
||||
|
||||
| Commit | Date | What happened |
|
||||
|--------|------|---------------|
|
||||
| `8943cf5` | 2026-03-30 | Symbolic reasoning engine written to `public/nexus/app.js` (+2280 lines) |
|
||||
| `e2df240` | 2026-03-30 | Phase 3 Neuro-Symbolic Bridge — overwrote to 284 lines of HTML (wrong path) |
|
||||
| `7f2f23f` | 2026-03-30 | Phase 4 Meta-Reasoning — same destructive overwrite |
|
||||
| `bf3b98b` | 2026-03-30 | A* Search — same destructive overwrite |
|
||||
| `e88bcb4` | 2026-03-30 | Bug fix identified `public/nexus/` files as corrupt duplicates, **deleted them** |
|
||||
|
||||
## Evidence: Code Is Present on Main
|
||||
|
||||
All 13 classical AI classes/functions verified present in root `app.js`:
|
||||
|
||||
| Class/Function | Line | Status |
|
||||
|----------------|------|--------|
|
||||
| `SymbolicEngine` | 82 | ✅ Present |
|
||||
| `AgentFSM` | 135 | ✅ Present |
|
||||
| `KnowledgeGraph` | 160 | ✅ Present |
|
||||
| `Blackboard` | 181 | ✅ Present |
|
||||
| `SymbolicPlanner` | 210 | ✅ Present |
|
||||
| `HTNPlanner` | 295 | ✅ Present |
|
||||
| `CaseBasedReasoner` | 343 | ✅ Present |
|
||||
| `NeuroSymbolicBridge` | 392 | ✅ Present |
|
||||
| `MetaReasoningLayer` | 422 | ✅ Present |
|
||||
| `AdaptiveCalibrator` | 460 | ✅ Present |
|
||||
| `PSELayer` | 566 | ✅ Present |
|
||||
| `setupGOFAI()` | 596 | ✅ Present |
|
||||
| `updateGOFAI()` | 622 | ✅ Present |
|
||||
| Bitmask fact indexing | 86 | ✅ Present |
|
||||
| A* search | 231 | ✅ Present |
|
||||
|
||||
These were injected by commit `af7a4c4` (PR #775, merged via `a855d54`) into the correct path.
|
||||
|
||||
## What Actually Happened
|
||||
|
||||
1. Google AI Agent wrote good GOFAI code to root `app.js` via the correct PR (#775)
|
||||
2. A second wave of Google AI Agent instances also wrote to `public/nexus/app.js` (wrong path)
|
||||
3. Those `public/nexus/` files kept getting overwritten by subsequent agent commits
|
||||
4. Commit `e88bcb4` correctly identified the `public/nexus/` files as corrupt and deleted them
|
||||
5. Alexander interpreted the git log as "classical AI code keeps disappearing"
|
||||
6. The code was never actually gone — it just lived in root `app.js` the whole time
|
||||
|
||||
## Prevention Strategy
|
||||
|
||||
1. **Add `public/nexus/` to `.gitignore`** — prevents agents from accidentally writing to the wrong path again
|
||||
2. **Add canonical path documentation to CLAUDE.md** — any agent reading this repo will know where frontend code lives
|
||||
3. **This report** — serves as the audit trail so this confusion doesn't recur
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] Git history audited for classical AI commits
|
||||
- [x] Found the commits — they exist, code was written to wrong path
|
||||
- [x] Root cause identified — duplicate agents writing to `public/nexus/` (wrong path)
|
||||
- [x] Prevention strategy implemented — `.gitignore` + `CLAUDE.md` path guard
|
||||
- [x] Report filed with findings (this document)
|
||||
480
app.js
480
app.js
@@ -3,6 +3,8 @@ import { EffectComposer } from 'three/addons/postprocessing/EffectComposer.js';
|
||||
import { RenderPass } from 'three/addons/postprocessing/RenderPass.js';
|
||||
import { UnrealBloomPass } from 'three/addons/postprocessing/UnrealBloomPass.js';
|
||||
import { SMAAPass } from 'three/addons/postprocessing/SMAAPass.js';
|
||||
import { SpatialMemory } from './nexus/components/spatial-memory.js';
|
||||
import { SessionRooms } from './nexus/components/session-rooms.js';
|
||||
|
||||
// ═══════════════════════════════════════════
|
||||
// NEXUS v1.1 — Portal System Update
|
||||
@@ -703,6 +705,8 @@ async function init() {
|
||||
createSessionPowerMeter();
|
||||
createWorkshopTerminal();
|
||||
createAshStorm();
|
||||
SpatialMemory.init(scene);
|
||||
SessionRooms.init(scene, camera, null);
|
||||
updateLoad(90);
|
||||
|
||||
loadSession();
|
||||
@@ -1881,7 +1885,7 @@ function setupControls() {
|
||||
orbitState.lastX = e.clientX;
|
||||
orbitState.lastY = e.clientY;
|
||||
|
||||
// Raycasting for portals
|
||||
// Raycasting for portals and memory crystals
|
||||
if (!portalOverlayActive) {
|
||||
const mouse = new THREE.Vector2(
|
||||
(e.clientX / window.innerWidth) * 2 - 1,
|
||||
@@ -1889,12 +1893,43 @@ function setupControls() {
|
||||
);
|
||||
const raycaster = new THREE.Raycaster();
|
||||
raycaster.setFromCamera(mouse, camera);
|
||||
const intersects = raycaster.intersectObjects(portals.map(p => p.ring));
|
||||
if (intersects.length > 0) {
|
||||
const clickedRing = intersects[0].object;
|
||||
|
||||
// Priority 1: Portals
|
||||
const portalHits = raycaster.intersectObjects(portals.map(p => p.ring));
|
||||
if (portalHits.length > 0) {
|
||||
const clickedRing = portalHits[0].object;
|
||||
const portal = portals.find(p => p.ring === clickedRing);
|
||||
if (portal) activatePortal(portal);
|
||||
if (portal) { activatePortal(portal); return; }
|
||||
}
|
||||
|
||||
// Priority 2: Memory crystals (Mnemosyne)
|
||||
const crystalMeshes = SpatialMemory.getCrystalMeshes();
|
||||
if (crystalMeshes.length > 0) {
|
||||
const crystalHits = raycaster.intersectObjects(crystalMeshes, false);
|
||||
if (crystalHits.length > 0) {
|
||||
const hitMesh = crystalHits[0].object;
|
||||
const memInfo = SpatialMemory.getMemoryFromMesh(hitMesh);
|
||||
if (memInfo) {
|
||||
SpatialMemory.highlightMemory(memInfo.data.id);
|
||||
showMemoryPanel(memInfo, e.clientX, e.clientY);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Priority 3: Session rooms (Mnemosyne #1171)
|
||||
const roomMeshes = SessionRooms.getClickableMeshes();
|
||||
if (roomMeshes.length > 0) {
|
||||
const roomHits = raycaster.intersectObjects(roomMeshes, false);
|
||||
if (roomHits.length > 0) {
|
||||
const session = SessionRooms.handleRoomClick(roomHits[0].object);
|
||||
if (session) { _showSessionRoomPanel(session); return; }
|
||||
}
|
||||
}
|
||||
|
||||
// Clicked empty space — dismiss panel
|
||||
dismissMemoryPanel();
|
||||
_dismissSessionRoomPanel();
|
||||
}
|
||||
}
|
||||
});
|
||||
@@ -2549,6 +2584,226 @@ function focusPortal(portal) {
|
||||
let lastThoughtTime = 0;
|
||||
let pulseTimer = 0;
|
||||
|
||||
|
||||
// ═══════════════════════════════════════════
|
||||
// MNEMOSYNE — MEMORY CRYSTAL INSPECTION
|
||||
// ═══════════════════════════════════════════
|
||||
|
||||
// ── pin state for memory panel ──
|
||||
let _memPanelPinned = false;
|
||||
|
||||
/** Convert a packed hex color integer to "r,g,b" string for CSS rgba(). */
|
||||
function _hexToRgb(hex) {
|
||||
return ((hex >> 16) & 255) + ',' + ((hex >> 8) & 255) + ',' + (hex & 255);
|
||||
}
|
||||
|
||||
/**
|
||||
* Position the panel near the screen click coordinates, keeping it on-screen.
|
||||
*/
|
||||
function _positionPanel(panel, clickX, clickY) {
|
||||
const W = window.innerWidth;
|
||||
const H = window.innerHeight;
|
||||
const panelW = 356; // matches CSS width + padding
|
||||
const panelH = 420; // generous estimate
|
||||
const margin = 12;
|
||||
|
||||
let left = clickX + 24;
|
||||
if (left + panelW > W - margin) left = clickX - panelW - 24;
|
||||
left = Math.max(margin, Math.min(W - panelW - margin, left));
|
||||
|
||||
let top = clickY - 80;
|
||||
top = Math.max(margin, Math.min(H - panelH - margin, top));
|
||||
|
||||
panel.style.right = 'auto';
|
||||
panel.style.top = top + 'px';
|
||||
panel.style.left = left + 'px';
|
||||
panel.style.transform = 'none';
|
||||
}
|
||||
|
||||
/**
|
||||
* Navigate to (highlight + show panel for) a memory crystal by id.
|
||||
*/
|
||||
function _navigateToMemory(memId) {
|
||||
SpatialMemory.highlightMemory(memId);
|
||||
addChatMessage('system', `Focus: ${memId.replace(/_/g, ' ')}`);
|
||||
const meshes = SpatialMemory.getCrystalMeshes();
|
||||
for (const mesh of meshes) {
|
||||
if (mesh.userData && mesh.userData.memId === memId) {
|
||||
const memInfo = SpatialMemory.getMemoryFromMesh(mesh);
|
||||
if (memInfo) { showMemoryPanel(memInfo); break; }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Show the holographic detail panel for a clicked crystal.
|
||||
* @param {object} memInfo — { data, region } from SpatialMemory.getMemoryFromMesh()
|
||||
* @param {number} [clickX] — screen X of the click (for panel positioning)
|
||||
* @param {number} [clickY] — screen Y of the click
|
||||
*/
|
||||
function showMemoryPanel(memInfo, clickX, clickY) {
|
||||
const panel = document.getElementById('memory-panel');
|
||||
if (!panel) return;
|
||||
|
||||
const { data, region } = memInfo;
|
||||
const regionDef = SpatialMemory.REGIONS[region] || SpatialMemory.REGIONS.working;
|
||||
const colorHex = regionDef.color.toString(16).padStart(6, '0');
|
||||
const colorRgb = _hexToRgb(regionDef.color);
|
||||
|
||||
// Header — region dot + label
|
||||
document.getElementById('memory-panel-region').textContent = regionDef.label;
|
||||
document.getElementById('memory-panel-region-dot').style.background = '#' + colorHex;
|
||||
|
||||
// Category badge
|
||||
const badge = document.getElementById('memory-panel-category-badge');
|
||||
if (badge) {
|
||||
badge.textContent = (data.category || region || 'memory').toUpperCase();
|
||||
badge.style.background = 'rgba(' + colorRgb + ',0.16)';
|
||||
badge.style.color = '#' + colorHex;
|
||||
badge.style.borderColor = 'rgba(' + colorRgb + ',0.4)';
|
||||
}
|
||||
|
||||
// Entity name (humanised id)
|
||||
const entityEl = document.getElementById('memory-panel-entity-name');
|
||||
if (entityEl) entityEl.textContent = (data.id || '\u2014').replace(/_/g, ' ');
|
||||
|
||||
// Fact content
|
||||
document.getElementById('memory-panel-content').textContent = data.content || '(empty)';
|
||||
|
||||
// Trust score bar
|
||||
const strength = data.strength != null ? data.strength : 0.7;
|
||||
const trustFill = document.getElementById('memory-panel-trust-fill');
|
||||
const trustVal = document.getElementById('memory-panel-trust-value');
|
||||
if (trustFill) {
|
||||
trustFill.style.width = (strength * 100).toFixed(0) + '%';
|
||||
trustFill.style.background = '#' + colorHex;
|
||||
}
|
||||
if (trustVal) trustVal.textContent = (strength * 100).toFixed(0) + '%';
|
||||
|
||||
// Meta rows
|
||||
document.getElementById('memory-panel-id').textContent = data.id || '\u2014';
|
||||
document.getElementById('memory-panel-source').textContent = data.source || 'unknown';
|
||||
document.getElementById('memory-panel-time').textContent = data.timestamp ? new Date(data.timestamp).toLocaleString() : '\u2014';
|
||||
|
||||
// Related entities — clickable links
|
||||
const connEl = document.getElementById('memory-panel-connections');
|
||||
connEl.innerHTML = '';
|
||||
if (data.connections && data.connections.length > 0) {
|
||||
data.connections.forEach(cid => {
|
||||
const btn = document.createElement('button');
|
||||
btn.className = 'memory-conn-tag memory-conn-link';
|
||||
btn.textContent = cid.replace(/_/g, ' ');
|
||||
btn.title = 'Go to: ' + cid;
|
||||
btn.addEventListener('click', (ev) => { ev.stopPropagation(); _navigateToMemory(cid); });
|
||||
connEl.appendChild(btn);
|
||||
});
|
||||
} else {
|
||||
connEl.innerHTML = '<span style="color:var(--color-text-muted)">None</span>';
|
||||
}
|
||||
|
||||
// Pin button — reset on fresh open
|
||||
_memPanelPinned = false;
|
||||
const pinBtn = document.getElementById('memory-panel-pin');
|
||||
if (pinBtn) {
|
||||
pinBtn.classList.remove('pinned');
|
||||
pinBtn.title = 'Pin panel';
|
||||
pinBtn.onclick = () => {
|
||||
_memPanelPinned = !_memPanelPinned;
|
||||
pinBtn.classList.toggle('pinned', _memPanelPinned);
|
||||
pinBtn.title = _memPanelPinned ? 'Unpin panel' : 'Pin panel';
|
||||
};
|
||||
}
|
||||
|
||||
// Positioning — near click if coords provided
|
||||
if (clickX != null && clickY != null) {
|
||||
_positionPanel(panel, clickX, clickY);
|
||||
}
|
||||
|
||||
// Fade in
|
||||
panel.classList.remove('memory-panel-fade-out');
|
||||
panel.style.display = 'flex';
|
||||
}
|
||||
|
||||
/**
|
||||
* Dismiss the panel (respects pin). Called on empty-space click.
|
||||
*/
|
||||
function dismissMemoryPanel() {
|
||||
if (_memPanelPinned) return;
|
||||
_dismissMemoryPanelForce();
|
||||
}
|
||||
|
||||
/**
|
||||
* Force-dismiss the panel regardless of pin state. Used by the close button.
|
||||
*/
|
||||
function _dismissMemoryPanelForce() {
|
||||
_memPanelPinned = false;
|
||||
SpatialMemory.clearHighlight();
|
||||
const panel = document.getElementById('memory-panel');
|
||||
if (!panel || panel.style.display === 'none') return;
|
||||
panel.classList.add('memory-panel-fade-out');
|
||||
setTimeout(() => {
|
||||
panel.style.display = 'none';
|
||||
panel.classList.remove('memory-panel-fade-out');
|
||||
}, 200);
|
||||
}
|
||||
|
||||
/**
|
||||
* Show the session room HUD panel when a chamber is entered.
|
||||
* @param {object} session — { id, timestamp, facts[] }
|
||||
*/
|
||||
function _showSessionRoomPanel(session) {
|
||||
const panel = document.getElementById('session-room-panel');
|
||||
if (!panel) return;
|
||||
|
||||
const dt = session.timestamp ? new Date(session.timestamp) : new Date();
|
||||
const tsEl = document.getElementById('session-room-timestamp');
|
||||
if (tsEl) tsEl.textContent = isNaN(dt.getTime()) ? session.id : dt.toLocaleString();
|
||||
|
||||
const countEl = document.getElementById('session-room-fact-count');
|
||||
const facts = session.facts || [];
|
||||
if (countEl) countEl.textContent = facts.length + (facts.length === 1 ? ' fact' : ' facts') + ' in this chamber';
|
||||
|
||||
const listEl = document.getElementById('session-room-facts');
|
||||
if (listEl) {
|
||||
listEl.innerHTML = '';
|
||||
facts.slice(0, 8).forEach(f => {
|
||||
const item = document.createElement('div');
|
||||
item.className = 'session-room-fact-item';
|
||||
item.textContent = f.content || f.id || '(unknown)';
|
||||
item.title = f.content || '';
|
||||
listEl.appendChild(item);
|
||||
});
|
||||
if (facts.length > 8) {
|
||||
const more = document.createElement('div');
|
||||
more.className = 'session-room-fact-item';
|
||||
more.style.color = 'rgba(200,180,255,0.4)';
|
||||
more.textContent = '\u2026 ' + (facts.length - 8) + ' more';
|
||||
listEl.appendChild(more);
|
||||
}
|
||||
}
|
||||
|
||||
// Close button
|
||||
const closeBtn = document.getElementById('session-room-close');
|
||||
if (closeBtn) closeBtn.onclick = () => _dismissSessionRoomPanel();
|
||||
|
||||
panel.classList.remove('session-panel-fade-out');
|
||||
panel.style.display = 'block';
|
||||
}
|
||||
|
||||
/**
|
||||
* Dismiss the session room panel.
|
||||
*/
|
||||
function _dismissSessionRoomPanel() {
|
||||
const panel = document.getElementById('session-room-panel');
|
||||
if (!panel || panel.style.display === 'none') return;
|
||||
panel.classList.add('session-panel-fade-out');
|
||||
setTimeout(() => {
|
||||
panel.style.display = 'none';
|
||||
panel.classList.remove('session-panel-fade-out');
|
||||
}, 200);
|
||||
}
|
||||
|
||||
|
||||
function gameLoop() {
|
||||
requestAnimationFrame(gameLoop);
|
||||
const delta = Math.min(clock.getDelta(), 0.1);
|
||||
@@ -2573,6 +2828,16 @@ function gameLoop() {
|
||||
|
||||
updateAshStorm(delta, elapsed);
|
||||
|
||||
// Project Mnemosyne - Memory Orb Animation
|
||||
if (typeof animateMemoryOrbs === 'function') {
|
||||
SpatialMemory.update(delta);
|
||||
animateMemoryOrbs(delta);
|
||||
}
|
||||
|
||||
// Project Mnemosyne - Session Rooms (#1171)
|
||||
SessionRooms.update(delta);
|
||||
|
||||
|
||||
const mode = NAV_MODES[navModeIdx];
|
||||
const chatActive = document.activeElement === document.getElementById('chat-input');
|
||||
|
||||
@@ -2771,6 +3036,12 @@ function gameLoop() {
|
||||
composer.render();
|
||||
|
||||
updateAshStorm(delta, elapsed);
|
||||
|
||||
// Project Mnemosyne - Memory Orb Animation
|
||||
if (typeof animateMemoryOrbs === 'function') {
|
||||
animateMemoryOrbs(delta);
|
||||
}
|
||||
|
||||
updatePortalTunnel(delta, elapsed);
|
||||
|
||||
if (workshopScanMat) workshopScanMat.uniforms.uTime.value = clock.getElapsedTime();
|
||||
@@ -2933,9 +3204,208 @@ function updateAshStorm(delta, elapsed) {
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// ═══════════════════════════════════════════
|
||||
// PROJECT MNEMOSYNE — HOLOGRAPHIC MEMORY ORBS
|
||||
// ═══════════════════════════════════════════
|
||||
|
||||
// Memory orbs registry for animation loop
|
||||
const memoryOrbs = [];
|
||||
|
||||
/**
|
||||
* Spawn a glowing memory orb at the given position.
|
||||
* Used to visualize RAG retrievals and memory recalls in the Nexus.
|
||||
*
|
||||
* @param {THREE.Vector3} position - World position for the orb
|
||||
* @param {number} color - Hex color (default: 0x4af0c0 - cyan)
|
||||
* @param {number} size - Radius of the orb (default: 0.5)
|
||||
* @param {object} metadata - Optional metadata for the memory (source, timestamp, etc.)
|
||||
* @returns {THREE.Mesh} The created orb mesh
|
||||
*/
|
||||
function spawnMemoryOrb(position, color = 0x4af0c0, size = 0.5, metadata = {}) {
|
||||
if (typeof THREE === 'undefined' || typeof scene === 'undefined') {
|
||||
console.warn('[Mnemosyne] THREE/scene not available for orb spawn');
|
||||
return null;
|
||||
}
|
||||
|
||||
const geometry = new THREE.SphereGeometry(size, 32, 32);
|
||||
const material = new THREE.MeshStandardMaterial({
|
||||
color: color,
|
||||
emissive: color,
|
||||
emissiveIntensity: 2.5,
|
||||
metalness: 0.3,
|
||||
roughness: 0.2,
|
||||
transparent: true,
|
||||
opacity: 0.85,
|
||||
envMapIntensity: 1.5
|
||||
});
|
||||
|
||||
const orb = new THREE.Mesh(geometry, material);
|
||||
orb.position.copy(position);
|
||||
orb.castShadow = true;
|
||||
orb.receiveShadow = true;
|
||||
|
||||
orb.userData = {
|
||||
type: 'memory_orb',
|
||||
pulse: Math.random() * Math.PI * 2, // Random phase offset
|
||||
pulseSpeed: 0.002 + Math.random() * 0.001,
|
||||
originalScale: size,
|
||||
metadata: metadata,
|
||||
createdAt: Date.now()
|
||||
};
|
||||
|
||||
// Point light for local illumination
|
||||
const light = new THREE.PointLight(color, 1.5, 8);
|
||||
orb.add(light);
|
||||
|
||||
scene.add(orb);
|
||||
memoryOrbs.push(orb);
|
||||
|
||||
console.info('[Mnemosyne] Memory orb spawned:', metadata.source || 'unknown');
|
||||
return orb;
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove a memory orb from the scene and dispose resources.
|
||||
* @param {THREE.Mesh} orb - The orb to remove
|
||||
*/
|
||||
function removeMemoryOrb(orb) {
|
||||
if (!orb) return;
|
||||
|
||||
if (orb.parent) orb.parent.remove(orb);
|
||||
if (orb.geometry) orb.geometry.dispose();
|
||||
if (orb.material) orb.material.dispose();
|
||||
|
||||
const idx = memoryOrbs.indexOf(orb);
|
||||
if (idx > -1) memoryOrbs.splice(idx, 1);
|
||||
}
|
||||
|
||||
/**
|
||||
* Animate all memory orbs — pulse, rotate, and fade.
|
||||
* Called from gameLoop() every frame.
|
||||
* @param {number} delta - Time since last frame
|
||||
*/
|
||||
function animateMemoryOrbs(delta) {
|
||||
for (let i = memoryOrbs.length - 1; i >= 0; i--) {
|
||||
const orb = memoryOrbs[i];
|
||||
if (!orb || !orb.userData) continue;
|
||||
|
||||
// Pulse animation
|
||||
orb.userData.pulse += orb.userData.pulseSpeed * delta * 1000;
|
||||
const pulseFactor = 1 + Math.sin(orb.userData.pulse) * 0.1;
|
||||
orb.scale.setScalar(pulseFactor * orb.userData.originalScale);
|
||||
|
||||
// Gentle rotation
|
||||
orb.rotation.y += delta * 0.5;
|
||||
|
||||
// Fade after 30 seconds
|
||||
const age = (Date.now() - orb.userData.createdAt) / 1000;
|
||||
if (age > 30) {
|
||||
const fadeDuration = 10;
|
||||
const fadeProgress = Math.min(1, (age - 30) / fadeDuration);
|
||||
orb.material.opacity = 0.85 * (1 - fadeProgress);
|
||||
|
||||
if (fadeProgress >= 1) {
|
||||
removeMemoryOrb(orb);
|
||||
i--; // Adjust index after removal
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Spawn memory orbs arranged in a spiral for RAG retrieval results.
|
||||
* @param {Array} results - Array of {content, score, source}
|
||||
* @param {THREE.Vector3} center - Center position (default: above avatar)
|
||||
*/
|
||||
function spawnRetrievalOrbs(results, center) {
|
||||
if (!results || !Array.isArray(results) || results.length === 0) return;
|
||||
|
||||
if (!center) {
|
||||
center = new THREE.Vector3(0, 2, 0);
|
||||
}
|
||||
|
||||
const colors = [0x4af0c0, 0x7b5cff, 0xffd700, 0xff4466, 0x00ff88];
|
||||
const radius = 3;
|
||||
|
||||
results.forEach((result, i) => {
|
||||
const angle = (i / results.length) * Math.PI * 2;
|
||||
const height = (i / results.length) * 2 - 1;
|
||||
|
||||
const position = new THREE.Vector3(
|
||||
center.x + Math.cos(angle) * radius,
|
||||
center.y + height,
|
||||
center.z + Math.sin(angle) * radius
|
||||
);
|
||||
|
||||
const colorIdx = Math.min(colors.length - 1, Math.floor((result.score || 0.5) * colors.length));
|
||||
const size = 0.3 + (result.score || 0.5) * 0.4;
|
||||
|
||||
spawnMemoryOrb(position, colors[colorIdx], size, {
|
||||
source: result.source || 'unknown',
|
||||
score: result.score || 0,
|
||||
contentPreview: (result.content || '').substring(0, 100)
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
init().then(() => {
|
||||
createAshStorm();
|
||||
createPortalTunnel();
|
||||
|
||||
// Project Mnemosyne — seed demo spatial memories
|
||||
const demoMemories = [
|
||||
{ id: 'mem_nexus_birth', content: 'The Nexus came online — first render of the 3D world', category: 'knowledge', strength: 0.95, connections: ['mem_mnemosyne_start'] },
|
||||
{ id: 'mem_first_portal', content: 'First portal deployed — connection to external service', category: 'engineering', strength: 0.85, connections: ['mem_nexus_birth'] },
|
||||
{ id: 'mem_hermes_chat', content: 'First conversation through the Hermes gateway', category: 'social', strength: 0.7, connections: [] },
|
||||
{ id: 'mem_mnemosyne_start', content: 'Project Mnemosyne began — the living archive awakens', category: 'projects', strength: 0.9, connections: ['mem_nexus_birth', 'mem_spatial_schema'] },
|
||||
{ id: 'mem_spatial_schema', content: 'Spatial Memory Schema defined — memories gain permanent homes', category: 'engineering', strength: 0.8, connections: ['mem_mnemosyne_start'] },
|
||||
// MemPalace category zone demos — issue #1168
|
||||
{ id: 'mem_pref_dark_mode', content: 'User prefers dark mode and monospace fonts', category: 'user_pref', strength: 0.9, connections: [] },
|
||||
{ id: 'mem_pref_verbose_logs', content: 'User prefers verbose logging during debug sessions', category: 'user_pref', strength: 0.7, connections: [] },
|
||||
{ id: 'mem_proj_nexus_goal', content: 'The Nexus goal: local-first 3D training ground for Timmy', category: 'project', strength: 0.95, connections: ['mem_proj_mnemosyne'] },
|
||||
{ id: 'mem_proj_mnemosyne', content: 'Project Mnemosyne: holographic living archive of facts', category: 'project', strength: 0.85, connections: ['mem_proj_nexus_goal'] },
|
||||
{ id: 'mem_tool_three_js', content: 'Three.js — 3D rendering library used for the Nexus world', category: 'tool', strength: 0.8, connections: [] },
|
||||
{ id: 'mem_tool_gitea', content: 'Gitea API at forge.alexanderwhitestone.com for issue tracking', category: 'tool', strength: 0.75, connections: [] },
|
||||
{ id: 'mem_gen_websocket', content: 'WebSocket bridge (server.py) connects Timmy cognition to the browser', category: 'general', strength: 0.7, connections: [] },
|
||||
{ id: 'mem_gen_hermes', content: 'Hermes harness: telemetry and durable truth pipeline', category: 'general', strength: 0.65, connections: [] },
|
||||
];
|
||||
demoMemories.forEach(m => SpatialMemory.placeMemory(m));
|
||||
|
||||
// Gravity well clustering — attract related crystals, bake positions (issue #1175)
|
||||
SpatialMemory.runGravityLayout();
|
||||
|
||||
// Project Mnemosyne — seed demo session rooms (#1171)
|
||||
// Sessions group facts by conversation/work session with a timestamp.
|
||||
const demoSessions = [
|
||||
{
|
||||
id: 'session_2026_03_01',
|
||||
timestamp: '2026-03-01T10:00:00.000Z',
|
||||
facts: [
|
||||
{ id: 'mem_nexus_birth', content: 'The Nexus came online — first render of the 3D world', category: 'knowledge', strength: 0.95 },
|
||||
{ id: 'mem_mnemosyne_start', content: 'Project Mnemosyne began — the living archive awakens', category: 'projects', strength: 0.9 },
|
||||
]
|
||||
},
|
||||
{
|
||||
id: 'session_2026_03_15',
|
||||
timestamp: '2026-03-15T14:30:00.000Z',
|
||||
facts: [
|
||||
{ id: 'mem_first_portal', content: 'First portal deployed — connection to external service', category: 'engineering', strength: 0.85 },
|
||||
{ id: 'mem_hermes_chat', content: 'First conversation through the Hermes gateway', category: 'social', strength: 0.7 },
|
||||
{ id: 'mem_spatial_schema', content: 'Spatial Memory Schema defined — memories gain homes', category: 'engineering', strength: 0.8 },
|
||||
]
|
||||
},
|
||||
{
|
||||
id: 'session_2026_04_10',
|
||||
timestamp: '2026-04-10T09:00:00.000Z',
|
||||
facts: [
|
||||
{ id: 'mem_session_rooms', content: 'Session rooms introduced — holographic chambers per session', category: 'projects', strength: 0.88 },
|
||||
{ id: 'mem_gravity_wells', content: 'Gravity-well clustering bakes crystal positions on load', category: 'engineering', strength: 0.75 },
|
||||
]
|
||||
}
|
||||
];
|
||||
SessionRooms.updateSessions(demoSessions);
|
||||
|
||||
fetchGiteaData();
|
||||
setInterval(fetchGiteaData, 30000);
|
||||
runWeeklyAudit();
|
||||
|
||||
@@ -152,17 +152,55 @@ class OpenAITTSAdapter:
|
||||
return mp3_path
|
||||
|
||||
|
||||
class EdgeTTSAdapter:
|
||||
"""Zero-cost TTS using Microsoft Edge neural voices (no API key required).
|
||||
|
||||
Requires: pip install edge-tts>=6.1.9
|
||||
Voices: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support
|
||||
"""
|
||||
|
||||
DEFAULT_VOICE = "en-US-GuyNeural"
|
||||
|
||||
def __init__(self, config: TTSConfig):
|
||||
self.config = config
|
||||
self.voice = config.voice_id or self.DEFAULT_VOICE
|
||||
|
||||
def synthesize(self, text: str, output_path: Path) -> Path:
|
||||
try:
|
||||
import edge_tts
|
||||
except ImportError:
|
||||
raise RuntimeError("edge-tts not installed. Run: pip install edge-tts")
|
||||
|
||||
import asyncio
|
||||
|
||||
mp3_path = output_path.with_suffix(".mp3")
|
||||
|
||||
async def _run():
|
||||
communicate = edge_tts.Communicate(text, self.voice)
|
||||
await communicate.save(str(mp3_path))
|
||||
|
||||
asyncio.run(_run())
|
||||
return mp3_path
|
||||
|
||||
|
||||
ADAPTERS = {
|
||||
"piper": PiperAdapter,
|
||||
"elevenlabs": ElevenLabsAdapter,
|
||||
"openai": OpenAITTSAdapter,
|
||||
"edge-tts": EdgeTTSAdapter,
|
||||
}
|
||||
|
||||
|
||||
def get_provider_config() -> TTSConfig:
|
||||
"""Load TTS configuration from environment."""
|
||||
provider = os.environ.get("DEEPDIVE_TTS_PROVIDER", "openai")
|
||||
voice = os.environ.get("DEEPDIVE_TTS_VOICE", "alloy" if provider == "openai" else "matthew")
|
||||
if provider == "openai":
|
||||
default_voice = "alloy"
|
||||
elif provider == "edge-tts":
|
||||
default_voice = EdgeTTSAdapter.DEFAULT_VOICE
|
||||
else:
|
||||
default_voice = "matthew"
|
||||
voice = os.environ.get("DEEPDIVE_TTS_VOICE", default_voice)
|
||||
|
||||
return TTSConfig(
|
||||
provider=provider,
|
||||
|
||||
@@ -32,12 +32,14 @@ import importlib.util
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
@@ -212,6 +214,46 @@ def generate_report(date_str: str, checker_mod) -> str:
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# ── Voice memo ────────────────────────────────────────────────────────
|
||||
|
||||
def _generate_voice_memo(report_text: str, date_str: str) -> Optional[str]:
|
||||
"""Generate an MP3 voice memo of the night watch report.
|
||||
|
||||
Returns the output path on success, or None if generation fails.
|
||||
"""
|
||||
try:
|
||||
import edge_tts
|
||||
except ImportError:
|
||||
logger.warning("edge-tts not installed; skipping voice memo. Run: pip install edge-tts")
|
||||
return None
|
||||
|
||||
import asyncio
|
||||
|
||||
# Strip markdown formatting for cleaner speech
|
||||
clean = report_text
|
||||
clean = re.sub(r"#+\s*", "", clean) # headings
|
||||
clean = re.sub(r"\|", " ", clean) # table pipes
|
||||
clean = re.sub(r"\*+", "", clean) # bold/italic markers
|
||||
clean = re.sub(r"-{3,}", "", clean) # horizontal rules
|
||||
clean = re.sub(r"\s{2,}", " ", clean) # collapse extra whitespace
|
||||
|
||||
output_dir = Path("/tmp/bezalel")
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
mp3_path = output_dir / f"night-watch-{date_str}.mp3"
|
||||
|
||||
try:
|
||||
async def _run():
|
||||
communicate = edge_tts.Communicate(clean.strip(), "en-US-GuyNeural")
|
||||
await communicate.save(str(mp3_path))
|
||||
|
||||
asyncio.run(_run())
|
||||
logger.info("Voice memo written to %s", mp3_path)
|
||||
return str(mp3_path)
|
||||
except Exception as exc:
|
||||
logger.warning("Voice memo generation failed: %s", exc)
|
||||
return None
|
||||
|
||||
|
||||
# ── Entry point ───────────────────────────────────────────────────────
|
||||
|
||||
def main() -> None:
|
||||
@@ -226,6 +268,10 @@ def main() -> None:
|
||||
"--dry-run", action="store_true",
|
||||
help="Print report to stdout instead of writing to disk",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--voice-memo", action="store_true",
|
||||
help="Generate an MP3 voice memo of the report using edge-tts (saved to /tmp/bezalel/)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
date_str = args.date or datetime.now(timezone.utc).strftime("%Y-%m-%d")
|
||||
@@ -242,6 +288,14 @@ def main() -> None:
|
||||
report_path.write_text(report_text)
|
||||
logger.info("Night Watch report written to %s", report_path)
|
||||
|
||||
if args.voice_memo:
|
||||
try:
|
||||
memo_path = _generate_voice_memo(report_text, date_str)
|
||||
if memo_path:
|
||||
logger.info("Voice memo: %s", memo_path)
|
||||
except Exception as exc:
|
||||
logger.warning("Voice memo failed (non-fatal): %s", exc)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
46
docker-compose.desktop.yml
Normal file
46
docker-compose.desktop.yml
Normal file
@@ -0,0 +1,46 @@
|
||||
version: "3.9"
|
||||
|
||||
# Sandboxed desktop environment for Hermes computer-use primitives.
|
||||
# Provides Xvfb (virtual framebuffer) + noVNC (browser-accessible VNC).
|
||||
#
|
||||
# Usage:
|
||||
# docker compose -f docker-compose.desktop.yml up -d
|
||||
# # Visit http://localhost:6080 to see the virtual desktop
|
||||
#
|
||||
# docker compose -f docker-compose.desktop.yml run hermes-desktop \
|
||||
# python -m nexus.computer_use_demo
|
||||
#
|
||||
# docker compose -f docker-compose.desktop.yml down
|
||||
|
||||
services:
|
||||
hermes-desktop:
|
||||
image: dorowu/ubuntu-desktop-lxde-vnc:focal
|
||||
environment:
|
||||
# Resolution for the virtual display
|
||||
RESOLUTION: "1280x800"
|
||||
# VNC password (change in production)
|
||||
VNC_PASSWORD: "hermes"
|
||||
# Disable HTTP password for development convenience
|
||||
HTTP_PASSWORD: ""
|
||||
ports:
|
||||
# noVNC web interface
|
||||
- "6080:80"
|
||||
# Raw VNC port (optional)
|
||||
- "5900:5900"
|
||||
volumes:
|
||||
# Mount repo into container so scripts are available
|
||||
- .:/workspace
|
||||
# Persist nexus runtime data (heartbeats, logs, evidence)
|
||||
- nexus_data:/root/.nexus
|
||||
working_dir: /workspace
|
||||
shm_size: "256mb"
|
||||
# Install Python deps on startup then keep container alive
|
||||
command: >
|
||||
bash -c "
|
||||
pip install --quiet pyautogui Pillow &&
|
||||
/startup.sh
|
||||
"
|
||||
|
||||
volumes:
|
||||
nexus_data:
|
||||
driver: local
|
||||
174
docs/computer-use.md
Normal file
174
docs/computer-use.md
Normal file
@@ -0,0 +1,174 @@
|
||||
# Computer Use — Desktop Automation Primitives for Hermes
|
||||
|
||||
Issue: [#1125](https://forge.alexanderwhitestone.com/Timmy_Foundation/the-nexus/issues/1125)
|
||||
|
||||
## Overview
|
||||
|
||||
`nexus/computer_use.py` adds desktop automation primitives to the Hermes fleet. Agents can take screenshots, click, type, and scroll — enough to drive a browser, validate a UI, or diagnose a failed workflow page visually.
|
||||
|
||||
All actions are logged to a JSONL audit trail at `~/.nexus/computer_use_actions.jsonl`.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Local (requires a real display or Xvfb)
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
pip install pyautogui Pillow
|
||||
|
||||
# Run the Phase 1 demo
|
||||
python -m nexus.computer_use_demo
|
||||
```
|
||||
|
||||
### Sandboxed (Docker + Xvfb + noVNC)
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.desktop.yml up -d
|
||||
# Visit http://localhost:6080 in your browser to see the virtual desktop
|
||||
|
||||
docker compose -f docker-compose.desktop.yml run hermes-desktop \
|
||||
python -m nexus.computer_use_demo
|
||||
|
||||
docker compose -f docker-compose.desktop.yml down
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### `computer_screenshot(save_path=None, log_path=...)`
|
||||
|
||||
Capture the current desktop.
|
||||
|
||||
| Param | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `save_path` | `str \| None` | Path to save PNG. If `None`, returns base64 string. |
|
||||
| `log_path` | `Path` | Audit log file. |
|
||||
|
||||
**Returns** `dict`:
|
||||
```json
|
||||
{
|
||||
"ok": true,
|
||||
"image_b64": "<base64 PNG or null>",
|
||||
"saved_to": "<path or null>",
|
||||
"error": null
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `computer_click(x, y, button="left", confirm=False, log_path=...)`
|
||||
|
||||
Click the mouse at screen coordinates.
|
||||
|
||||
| Param | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `x` | `int` | Horizontal coordinate |
|
||||
| `y` | `int` | Vertical coordinate |
|
||||
| `button` | `str` | `"left"` \| `"right"` \| `"middle"` |
|
||||
| `confirm` | `bool` | Required `True` for `right` / `middle` (poka-yoke) |
|
||||
|
||||
**Returns** `dict`:
|
||||
```json
|
||||
{"ok": true, "error": null}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `computer_type(text, confirm=False, interval=0.02, log_path=...)`
|
||||
|
||||
Type text using the keyboard.
|
||||
|
||||
| Param | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `text` | `str` | Text to type |
|
||||
| `confirm` | `bool` | Required `True` when text contains a sensitive keyword |
|
||||
| `interval` | `float` | Delay between keystrokes (seconds) |
|
||||
|
||||
**Sensitive keywords** (require `confirm=True`): `password`, `passwd`, `secret`, `token`, `api_key`, `apikey`, `key`, `auth`
|
||||
|
||||
> Note: the actual `text` value is never written to the audit log — only its length and whether it was flagged as sensitive.
|
||||
|
||||
**Returns** `dict`:
|
||||
```json
|
||||
{"ok": true, "error": null}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `computer_scroll(x, y, amount=3, log_path=...)`
|
||||
|
||||
Scroll the mouse wheel at screen coordinates.
|
||||
|
||||
| Param | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `x` | `int` | Horizontal coordinate |
|
||||
| `y` | `int` | Vertical coordinate |
|
||||
| `amount` | `int` | Scroll units. Positive = up, negative = down. |
|
||||
|
||||
**Returns** `dict`:
|
||||
```json
|
||||
{"ok": true, "error": null}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `read_action_log(n=20, log_path=...)`
|
||||
|
||||
Return the most recent `n` audit log entries, newest first.
|
||||
|
||||
```python
|
||||
from nexus.computer_use import read_action_log
|
||||
|
||||
for entry in read_action_log(n=5):
|
||||
print(entry["ts"], entry["action"], entry["result"]["ok"])
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Safety Model
|
||||
|
||||
| Action | Safety gate |
|
||||
|--------|-------------|
|
||||
| `computer_click(button="right")` | Requires `confirm=True` |
|
||||
| `computer_click(button="middle")` | Requires `confirm=True` |
|
||||
| `computer_type` with sensitive text | Requires `confirm=True` |
|
||||
| Mouse to top-left corner | pyautogui FAILSAFE — aborts immediately |
|
||||
| All actions | Written to JSONL audit log with timestamp |
|
||||
| Headless environment | All tools degrade gracefully — return `ok=False` with error message |
|
||||
|
||||
---
|
||||
|
||||
## Phase Roadmap
|
||||
|
||||
### Phase 1 — Environment & Primitives ✅
|
||||
- Sandboxed desktop via Xvfb + noVNC (`docker-compose.desktop.yml`)
|
||||
- `computer_screenshot`, `computer_click`, `computer_type`, `computer_scroll`
|
||||
- Poka-yoke safety checks on all destructive actions
|
||||
- JSONL audit log for all actions
|
||||
- Demo: baseline screenshot → open browser → navigate to Gitea → evidence screenshot
|
||||
- 32 unit tests, fully headless (pyautogui mocked)
|
||||
|
||||
### Phase 2 — Tool Integration (planned)
|
||||
- Register tools in the Hermes tool registry
|
||||
- LLM-based planner loop using screenshots as context
|
||||
- Destructive action confirmation UI
|
||||
|
||||
### Phase 3 — Use-Case Pilots (planned)
|
||||
- Pilot 1: Automated visual regression test for fleet dashboard
|
||||
- Pilot 2: Screenshot-based diagnosis of failed CI workflow page
|
||||
|
||||
---
|
||||
|
||||
## File Locations
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `nexus/computer_use.py` | Core tool primitives |
|
||||
| `nexus/computer_use_demo.py` | Phase 1 end-to-end demo |
|
||||
| `tests/test_computer_use.py` | 32 unit tests |
|
||||
| `docker-compose.desktop.yml` | Sandboxed desktop container |
|
||||
| `~/.nexus/computer_use_actions.jsonl` | Runtime audit log |
|
||||
| `~/.nexus/computer_use_evidence/` | Screenshot evidence (demo output) |
|
||||
135
docs/voice-output.md
Normal file
135
docs/voice-output.md
Normal file
@@ -0,0 +1,135 @@
|
||||
# Voice Output System
|
||||
|
||||
## Overview
|
||||
|
||||
The Nexus voice output system converts text reports and briefings into spoken audio.
|
||||
It supports multiple TTS providers with automatic fallback so that audio generation
|
||||
degrades gracefully when a provider is unavailable.
|
||||
|
||||
Primary use cases:
|
||||
- **Deep Dive** daily briefings (`bin/deepdive_tts.py`)
|
||||
- **Night Watch** nightly reports (`bin/night_watch.py --voice-memo`)
|
||||
|
||||
---
|
||||
|
||||
## Available Providers
|
||||
|
||||
### edge-tts (recommended default)
|
||||
|
||||
- **Cost:** Zero — no API key, no account required
|
||||
- **Package:** `pip install edge-tts>=6.1.9`
|
||||
- **Default voice:** `en-US-GuyNeural`
|
||||
- **Output format:** MP3
|
||||
- **How it works:** Streams audio from Microsoft Edge's neural TTS service over HTTPS.
|
||||
No local model download required.
|
||||
- **Available locales:** 100+ languages and locales. Full list:
|
||||
https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support
|
||||
|
||||
Notable English voices:
|
||||
| Voice ID | Style |
|
||||
|---|---|
|
||||
| `en-US-GuyNeural` | Neutral male (default) |
|
||||
| `en-US-JennyNeural` | Warm female |
|
||||
| `en-US-AriaNeural` | Expressive female |
|
||||
| `en-GB-RyanNeural` | British male |
|
||||
|
||||
### piper
|
||||
|
||||
- **Cost:** Free, fully offline
|
||||
- **Package:** `pip install piper-tts` + model download (~65 MB)
|
||||
- **Model location:** `~/.local/share/piper/en_US-lessac-medium.onnx`
|
||||
- **Output format:** WAV → MP3 (requires `lame`)
|
||||
- **Sovereignty:** Fully local; no network calls after model download
|
||||
|
||||
### elevenlabs
|
||||
|
||||
- **Cost:** Usage-based (paid)
|
||||
- **Requirement:** `ELEVENLABS_API_KEY` environment variable
|
||||
- **Output format:** MP3
|
||||
- **Quality:** Highest quality of the three providers
|
||||
|
||||
### openai
|
||||
|
||||
- **Cost:** Usage-based (paid)
|
||||
- **Requirement:** `OPENAI_API_KEY` environment variable
|
||||
- **Output format:** MP3
|
||||
- **Default voice:** `alloy`
|
||||
|
||||
---
|
||||
|
||||
## Usage: deepdive_tts.py
|
||||
|
||||
```bash
|
||||
# Use edge-tts (zero cost)
|
||||
DEEPDIVE_TTS_PROVIDER=edge-tts python bin/deepdive_tts.py --text "Good morning."
|
||||
|
||||
# Specify a different Edge voice
|
||||
python bin/deepdive_tts.py --provider edge-tts --voice en-US-JennyNeural --text "Hello world."
|
||||
|
||||
# Read from a file
|
||||
python bin/deepdive_tts.py --provider edge-tts --input-file /tmp/briefing.txt --output /tmp/briefing
|
||||
|
||||
# Use OpenAI
|
||||
OPENAI_API_KEY=sk-... python bin/deepdive_tts.py --provider openai --voice nova --text "Hello."
|
||||
|
||||
# Use ElevenLabs
|
||||
ELEVENLABS_API_KEY=... python bin/deepdive_tts.py --provider elevenlabs --voice rachel --text "Hello."
|
||||
|
||||
# Use local Piper (offline)
|
||||
python bin/deepdive_tts.py --provider piper --text "Hello."
|
||||
```
|
||||
|
||||
Provider and voice can also be set via environment variables:
|
||||
|
||||
```bash
|
||||
export DEEPDIVE_TTS_PROVIDER=edge-tts
|
||||
export DEEPDIVE_TTS_VOICE=en-GB-RyanNeural
|
||||
python bin/deepdive_tts.py --text "Good evening."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Usage: Night Watch --voice-memo
|
||||
|
||||
The `--voice-memo` flag causes Night Watch to generate an MP3 audio summary of the
|
||||
nightly report immediately after writing the markdown file.
|
||||
|
||||
```bash
|
||||
python bin/night_watch.py --voice-memo
|
||||
```
|
||||
|
||||
Output location: `/tmp/bezalel/night-watch-<YYYY-MM-DD>.mp3`
|
||||
|
||||
The voice memo:
|
||||
- Strips markdown formatting (`#`, `|`, `*`, `---`) for cleaner speech
|
||||
- Uses `edge-tts` with the `en-US-GuyNeural` voice
|
||||
- Is non-fatal: if TTS fails, the markdown report is still written normally
|
||||
|
||||
Example crontab with voice memo:
|
||||
|
||||
```cron
|
||||
0 3 * * * cd /path/to/the-nexus && python bin/night_watch.py --voice-memo \
|
||||
>> /var/log/bezalel/night-watch.log 2>&1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Fallback Chain
|
||||
|
||||
`HybridTTS` (used by `tts_engine.py`) attempts providers in this order:
|
||||
|
||||
1. **edge-tts** — zero cost, no API key
|
||||
2. **piper** — offline local model (if model file present)
|
||||
3. **elevenlabs** — cloud fallback (if `ELEVENLABS_API_KEY` set)
|
||||
|
||||
If `prefer_cloud=True` is passed, the order becomes: elevenlabs → piper.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 TODO
|
||||
|
||||
Evaluate **fish-speech** and **F5-TTS** as fully offline, sovereign alternatives
|
||||
with higher voice quality than Piper. These models run locally with no network
|
||||
dependency whatsoever, providing complete independence from Microsoft's Edge service.
|
||||
|
||||
Tracking: to be filed as a follow-up to issue #830.
|
||||
72
fleet/hermes-trismegistus/README.md
Normal file
72
fleet/hermes-trismegistus/README.md
Normal file
@@ -0,0 +1,72 @@
|
||||
# Hermes Trismegistus — Wizard Proposal
|
||||
|
||||
> **Status:** 🟡 DEFERRED
|
||||
> **Issue:** #1146
|
||||
> **Created:** 2026-04-08
|
||||
> **Author:** Alexander (KT Notes)
|
||||
> **Mimo Worker:** mimo-code-1146-1775851759
|
||||
|
||||
---
|
||||
|
||||
## Identity
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| **Name** | Hermes Trismegistus |
|
||||
| **Nature** | Claude-native wizard. She knows she runs on Claude. She's "the daughter of Claude" and leans into that heritage. |
|
||||
| **Purpose** | Dedicated reasoning and architecture wizard. Only handles tasks where Claude's reasoning capability genuinely adds value — planning, novel problem-solving, complex architecture decisions. |
|
||||
| **Not** | A replacement for Timmy. Not competing for identity. Not doing monkey work. |
|
||||
|
||||
## Design Constraints
|
||||
|
||||
- **Free tier only from day one.** Alexander is not paying Anthropic beyond current subscription.
|
||||
- **Degrades gracefully.** Full capability when free tier is generous, reduced scope when constrained.
|
||||
- **Not locked to Claude.** If better free-tier providers emerge, she can route to them.
|
||||
- **Multi-provider capable.** Welcome to become multifaceted if team finds better options.
|
||||
|
||||
## Hardware
|
||||
|
||||
- One of Alexander's shed laptops — minimum 4GB RAM, Ubuntu
|
||||
- Dedicated machine, not shared with Timmy's Mac
|
||||
- Runs in the Hermes harness
|
||||
- Needs power at house first
|
||||
|
||||
## Constitutional Foundation
|
||||
|
||||
- The KT conversation and documents serve as her founding constitution
|
||||
- Team (especially Timmy) has final say on whether she gets built
|
||||
- Must justify her existence through useful work, same as every wizard
|
||||
|
||||
## Trigger to Unblock
|
||||
|
||||
All of the following must be true before implementation begins:
|
||||
|
||||
- [ ] Deadman switch wired and proven
|
||||
- [ ] Config stable across fleet
|
||||
- [ ] Fleet proven reliable for 1+ week
|
||||
- [ ] Alexander provides a state-of-the-system KT to Claude for instantiation
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] Dedicated KT document written for Hermes instantiation
|
||||
- [ ] Hardware provisioned (shed laptop with power)
|
||||
- [ ] Hermes harness configured for Claude free tier
|
||||
- [ ] Lazerus registry entry with health endpoints
|
||||
- [ ] Fleet routing entry with role and routing verdict
|
||||
- [ ] SOUL.md inscription drafted and reviewed by Timmy
|
||||
- [ ] Smoke test: Hermes responds to a basic reasoning task
|
||||
- [ ] Integration test: Hermes participates in a multi-wizard task alongside Timmy
|
||||
|
||||
## Proposed Lane
|
||||
|
||||
**Primary role:** Architecture reasoning
|
||||
**Routing verdict:** ROUTE TO: complex architectural decisions, novel problem-solving, planning tasks that benefit from Claude's reasoning depth. Do NOT route to: code generation (use Timmy/Carnice), issue triage (use Fenrir), or operational tasks (use Bezalel).
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Dependency | Status | Notes |
|
||||
|------------|--------|-------|
|
||||
| Deadman switch | 🔴 Not done | Must be proven before unblocking |
|
||||
| Fleet stability | 🟡 In progress | 1+ week uptime needed |
|
||||
| Shed laptop power | 🔴 Not done | Alexander needs to wire power |
|
||||
| KT document | 🔴 Not drafted | Alexander provides to Claude at unblock time |
|
||||
43
fleet/hermes-trismegistus/lane.md
Normal file
43
fleet/hermes-trismegistus/lane.md
Normal file
@@ -0,0 +1,43 @@
|
||||
# Hermes Trismegistus — Lane Definition
|
||||
|
||||
> **Status:** DEFERRED — do not instantiate until unblock conditions met
|
||||
> **See:** fleet/hermes-trismegistus/README.md for full proposal
|
||||
|
||||
---
|
||||
|
||||
## Role
|
||||
|
||||
Dedicated reasoning and architecture wizard. Claude-native.
|
||||
|
||||
## Routing
|
||||
|
||||
Route to Hermes Trismegistus when:
|
||||
- Task requires deep architectural reasoning
|
||||
- Novel problem-solving that benefits from Claude's reasoning depth
|
||||
- Planning and design decisions for the fleet
|
||||
- Complex multi-step analysis that goes beyond code generation
|
||||
|
||||
Do NOT route to Hermes for:
|
||||
- Code generation (use Timmy, Carnice, or Kimi)
|
||||
- Issue triage (use Fenrir)
|
||||
- Operational/DevOps tasks (use Bezalel)
|
||||
- Anything that can be done with a cheaper model
|
||||
|
||||
## Provider
|
||||
|
||||
- **Primary:** anthropic/claude (free tier)
|
||||
- **Fallback:** openrouter/free (Claude-class models)
|
||||
- **Degraded:** ollama/gemma4:12b (when free tier exhausted)
|
||||
|
||||
## Hardware
|
||||
|
||||
- Shed laptop, Ubuntu, minimum 4GB RAM
|
||||
- Dedicated machine, not shared
|
||||
|
||||
## Unblock Checklist
|
||||
|
||||
- [ ] Deadman switch operational
|
||||
- [ ] Fleet config stable for 1+ week
|
||||
- [ ] Shed laptop powered and networked
|
||||
- [ ] KT document drafted by Alexander
|
||||
- [ ] Timmy approves instantiation
|
||||
44
index.html
44
index.html
@@ -207,6 +207,50 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<!-- Memory Crystal Inspection Panel (Mnemosyne) -->
|
||||
<div id="memory-panel" class="memory-panel" style="display:none;">
|
||||
<div class="memory-panel-content">
|
||||
<div class="memory-panel-header">
|
||||
<span class="memory-category-badge" id="memory-panel-category-badge">MEM</span>
|
||||
<div class="memory-panel-region-dot" id="memory-panel-region-dot"></div>
|
||||
<div class="memory-panel-region" id="memory-panel-region">MEMORY</div>
|
||||
<button id="memory-panel-pin" class="memory-panel-pin" title="Pin panel">📌</button>
|
||||
<button id="memory-panel-close" class="memory-panel-close" onclick="_dismissMemoryPanelForce()">\u2715</button>
|
||||
</div>
|
||||
<div class="memory-entity-name" id="memory-panel-entity-name">\u2014</div>
|
||||
<div class="memory-panel-body" id="memory-panel-content">(empty)</div>
|
||||
<div class="memory-trust-row">
|
||||
<span class="memory-meta-label">Trust</span>
|
||||
<div class="memory-trust-bar">
|
||||
<div class="memory-trust-fill" id="memory-panel-trust-fill"></div>
|
||||
</div>
|
||||
<span class="memory-trust-value" id="memory-panel-trust-value">—</span>
|
||||
</div>
|
||||
<div class="memory-panel-meta">
|
||||
<div class="memory-meta-row"><span class="memory-meta-label">ID</span><span id="memory-panel-id">\u2014</span></div>
|
||||
<div class="memory-meta-row"><span class="memory-meta-label">Source</span><span id="memory-panel-source">\u2014</span></div>
|
||||
<div class="memory-meta-row"><span class="memory-meta-label">Time</span><span id="memory-panel-time">\u2014</span></div>
|
||||
<div class="memory-meta-row memory-meta-row--related"><span class="memory-meta-label">Related</span><span id="memory-panel-connections">\u2014</span></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Session Room HUD Panel (Mnemosyne #1171) -->
|
||||
<div id="session-room-panel" class="session-room-panel" style="display:none;">
|
||||
<div class="session-room-panel-content">
|
||||
<div class="session-room-header">
|
||||
<span class="session-room-icon">□</span>
|
||||
<div class="session-room-title">SESSION CHAMBER</div>
|
||||
<button class="session-room-close" id="session-room-close" title="Close">✕</button>
|
||||
</div>
|
||||
<div class="session-room-timestamp" id="session-room-timestamp">—</div>
|
||||
<div class="session-room-fact-count" id="session-room-fact-count">0 facts</div>
|
||||
<div class="session-room-facts" id="session-room-facts"></div>
|
||||
<div class="session-room-hint">Flying into chamber…</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Portal Atlas Overlay -->
|
||||
<div id="atlas-overlay" class="atlas-overlay" style="display:none;">
|
||||
<div class="atlas-content">
|
||||
|
||||
@@ -157,14 +157,45 @@ class ElevenLabsTTS:
|
||||
return output_path
|
||||
|
||||
|
||||
class EdgeTTS:
|
||||
"""Zero-cost TTS using Microsoft Edge neural voices (no API key required).
|
||||
|
||||
Requires: pip install edge-tts>=6.1.9
|
||||
"""
|
||||
|
||||
DEFAULT_VOICE = "en-US-GuyNeural"
|
||||
|
||||
def __init__(self, voice: str = None):
|
||||
self.voice = voice or self.DEFAULT_VOICE
|
||||
|
||||
def synthesize(self, text: str, output_path: str) -> str:
|
||||
"""Convert text to MP3 via Edge TTS."""
|
||||
try:
|
||||
import edge_tts
|
||||
except ImportError:
|
||||
raise RuntimeError("edge-tts not installed. Run: pip install edge-tts")
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
mp3_path = str(Path(output_path).with_suffix(".mp3"))
|
||||
|
||||
async def _run():
|
||||
communicate = edge_tts.Communicate(text, self.voice)
|
||||
await communicate.save(mp3_path)
|
||||
|
||||
asyncio.run(_run())
|
||||
return mp3_path
|
||||
|
||||
|
||||
class HybridTTS:
|
||||
"""TTS with sovereign primary, cloud fallback."""
|
||||
|
||||
|
||||
def __init__(self, prefer_cloud: bool = False):
|
||||
self.primary = None
|
||||
self.fallback = None
|
||||
self.prefer_cloud = prefer_cloud
|
||||
|
||||
|
||||
# Try preferred engine
|
||||
if prefer_cloud:
|
||||
self._init_elevenlabs()
|
||||
@@ -172,21 +203,29 @@ class HybridTTS:
|
||||
self._init_piper()
|
||||
else:
|
||||
self._init_piper()
|
||||
if not self.primary:
|
||||
self._init_edge_tts()
|
||||
if not self.primary:
|
||||
self._init_elevenlabs()
|
||||
|
||||
|
||||
def _init_piper(self):
|
||||
try:
|
||||
self.primary = PiperTTS()
|
||||
except Exception as e:
|
||||
print(f"Piper init failed: {e}")
|
||||
|
||||
|
||||
def _init_edge_tts(self):
|
||||
try:
|
||||
self.primary = EdgeTTS()
|
||||
except Exception as e:
|
||||
print(f"EdgeTTS init failed: {e}")
|
||||
|
||||
def _init_elevenlabs(self):
|
||||
try:
|
||||
self.primary = ElevenLabsTTS()
|
||||
except Exception as e:
|
||||
print(f"ElevenLabs init failed: {e}")
|
||||
|
||||
|
||||
def synthesize(self, text: str, output_path: str) -> str:
|
||||
"""Synthesize with fallback."""
|
||||
if self.primary:
|
||||
@@ -194,7 +233,7 @@ class HybridTTS:
|
||||
return self.primary.synthesize(text, output_path)
|
||||
except Exception as e:
|
||||
print(f"Primary failed: {e}")
|
||||
|
||||
|
||||
raise RuntimeError("No TTS engine available")
|
||||
|
||||
|
||||
|
||||
413
nexus/components/session-rooms.js
Normal file
413
nexus/components/session-rooms.js
Normal file
@@ -0,0 +1,413 @@
|
||||
// ═══════════════════════════════════════════════════════
|
||||
// PROJECT MNEMOSYNE — SESSION ROOMS (Issue #1171)
|
||||
// ═══════════════════════════════════════════════════════
|
||||
//
|
||||
// Groups memories by session into holographic chambers.
|
||||
// Each session becomes a wireframe cube floating in space.
|
||||
// Rooms are arranged chronologically along a spiral.
|
||||
// Click a room to fly inside; distant rooms LOD to a point.
|
||||
//
|
||||
// Usage from app.js:
|
||||
// SessionRooms.init(scene, camera, controls);
|
||||
// SessionRooms.updateSessions(sessions); // [{id, timestamp, facts[]}]
|
||||
// SessionRooms.update(delta); // call each frame
|
||||
// SessionRooms.getClickableMeshes(); // for raycasting
|
||||
// SessionRooms.handleRoomClick(mesh); // trigger fly-in
|
||||
// ═══════════════════════════════════════════════════════
|
||||
|
||||
const SessionRooms = (() => {
|
||||
|
||||
// ─── CONSTANTS ───────────────────────────────────────
|
||||
const MAX_ROOMS = 20;
|
||||
const ROOM_SIZE = 9; // wireframe cube edge length
|
||||
const ROOM_HALF = ROOM_SIZE / 2;
|
||||
const LOD_THRESHOLD = 55; // distance: full → point
|
||||
const LOD_HYSTERESIS = 5; // buffer to avoid flicker
|
||||
const SPIRAL_BASE_R = 20; // spiral inner radius
|
||||
const SPIRAL_R_STEP = 5; // radius growth per room
|
||||
const SPIRAL_ANGLE_INC = 2.399; // golden angle (radians)
|
||||
const SPIRAL_Y_STEP = 1.5; // vertical rise per room
|
||||
const FLY_DURATION = 1.5; // seconds for fly-in tween
|
||||
const FLY_TARGET_DEPTH = ROOM_HALF - 1.5; // how deep inside to stop
|
||||
|
||||
const ROOM_COLOR = 0x7b5cff; // violet — mnemosyne accent
|
||||
const POINT_COLOR = 0x9b7cff;
|
||||
const LABEL_COLOR = '#c8b4ff';
|
||||
const STORAGE_KEY = 'mnemosyne_sessions_v1';
|
||||
|
||||
// ─── STATE ────────────────────────────────────────────
|
||||
let _scene = null;
|
||||
let _camera = null;
|
||||
let _controls = null;
|
||||
|
||||
let _rooms = []; // array of room objects
|
||||
let _sessionIndex = {}; // id → room object
|
||||
|
||||
// Fly-in tween state
|
||||
let _flyActive = false;
|
||||
let _flyElapsed = 0;
|
||||
let _flyFrom = null;
|
||||
let _flyTo = null;
|
||||
let _flyLookFrom = null;
|
||||
let _flyLookTo = null;
|
||||
let _flyActiveRoom = null;
|
||||
|
||||
// ─── SPIRAL POSITION ──────────────────────────────────
|
||||
function _spiralPos(index) {
|
||||
const angle = index * SPIRAL_ANGLE_INC;
|
||||
const r = SPIRAL_BASE_R + index * SPIRAL_R_STEP;
|
||||
const y = index * SPIRAL_Y_STEP;
|
||||
return new THREE.Vector3(
|
||||
Math.cos(angle) * r,
|
||||
y,
|
||||
Math.sin(angle) * r
|
||||
);
|
||||
}
|
||||
|
||||
// ─── CREATE ROOM ──────────────────────────────────────
|
||||
function _createRoom(session, index) {
|
||||
const pos = _spiralPos(index);
|
||||
const group = new THREE.Group();
|
||||
group.position.copy(pos);
|
||||
|
||||
// Wireframe cube
|
||||
const boxGeo = new THREE.BoxGeometry(ROOM_SIZE, ROOM_SIZE, ROOM_SIZE);
|
||||
const edgesGeo = new THREE.EdgesGeometry(boxGeo);
|
||||
const edgesMat = new THREE.LineBasicMaterial({
|
||||
color: ROOM_COLOR,
|
||||
transparent: true,
|
||||
opacity: 0.55
|
||||
});
|
||||
const wireframe = new THREE.LineSegments(edgesGeo, edgesMat);
|
||||
wireframe.userData = { type: 'session_room_wireframe', sessionId: session.id };
|
||||
group.add(wireframe);
|
||||
|
||||
// Collision mesh (invisible, for raycasting)
|
||||
const hitGeo = new THREE.BoxGeometry(ROOM_SIZE, ROOM_SIZE, ROOM_SIZE);
|
||||
const hitMat = new THREE.MeshBasicMaterial({
|
||||
visible: false,
|
||||
transparent: true,
|
||||
opacity: 0,
|
||||
side: THREE.FrontSide
|
||||
});
|
||||
const hitMesh = new THREE.Mesh(hitGeo, hitMat);
|
||||
hitMesh.userData = { type: 'session_room', sessionId: session.id, roomIndex: index };
|
||||
group.add(hitMesh);
|
||||
|
||||
// LOD point (small sphere shown at distance)
|
||||
const pointGeo = new THREE.SphereGeometry(0.5, 6, 4);
|
||||
const pointMat = new THREE.MeshBasicMaterial({
|
||||
color: POINT_COLOR,
|
||||
transparent: true,
|
||||
opacity: 0.7
|
||||
});
|
||||
const pointMesh = new THREE.Mesh(pointGeo, pointMat);
|
||||
pointMesh.userData = { type: 'session_room_point', sessionId: session.id };
|
||||
pointMesh.visible = false; // starts hidden; shown only at LOD distance
|
||||
group.add(pointMesh);
|
||||
|
||||
// Timestamp billboard sprite
|
||||
const sprite = _makeTimestampSprite(session.timestamp, session.facts.length);
|
||||
sprite.position.set(0, ROOM_HALF + 1.2, 0);
|
||||
group.add(sprite);
|
||||
|
||||
// Inner ambient glow
|
||||
const glow = new THREE.PointLight(ROOM_COLOR, 0.4, ROOM_SIZE * 1.2);
|
||||
group.add(glow);
|
||||
|
||||
_scene.add(group);
|
||||
|
||||
const room = {
|
||||
session,
|
||||
group,
|
||||
wireframe,
|
||||
hitMesh,
|
||||
pointMesh,
|
||||
sprite,
|
||||
glow,
|
||||
pos: pos.clone(),
|
||||
index,
|
||||
lodActive: false,
|
||||
pulsePhase: Math.random() * Math.PI * 2
|
||||
};
|
||||
|
||||
_rooms.push(room);
|
||||
_sessionIndex[session.id] = room;
|
||||
|
||||
console.info('[SessionRooms] Created room for session', session.id, 'at index', index);
|
||||
return room;
|
||||
}
|
||||
|
||||
// ─── TIMESTAMP SPRITE ────────────────────────────────
|
||||
function _makeTimestampSprite(isoTimestamp, factCount) {
|
||||
const canvas = document.createElement('canvas');
|
||||
canvas.width = 320;
|
||||
canvas.height = 72;
|
||||
const ctx = canvas.getContext('2d');
|
||||
|
||||
// Background pill
|
||||
ctx.clearRect(0, 0, 320, 72);
|
||||
ctx.fillStyle = 'rgba(20, 10, 40, 0.82)';
|
||||
_roundRect(ctx, 4, 4, 312, 64, 14);
|
||||
ctx.fill();
|
||||
|
||||
// Border
|
||||
ctx.strokeStyle = 'rgba(123, 92, 255, 0.6)';
|
||||
ctx.lineWidth = 1.5;
|
||||
_roundRect(ctx, 4, 4, 312, 64, 14);
|
||||
ctx.stroke();
|
||||
|
||||
// Timestamp text
|
||||
const dt = isoTimestamp ? new Date(isoTimestamp) : new Date();
|
||||
const label = _formatDate(dt);
|
||||
ctx.fillStyle = LABEL_COLOR;
|
||||
ctx.font = 'bold 15px monospace';
|
||||
ctx.textAlign = 'center';
|
||||
ctx.fillText(label, 160, 30);
|
||||
|
||||
// Fact count
|
||||
ctx.fillStyle = 'rgba(200, 180, 255, 0.65)';
|
||||
ctx.font = '12px monospace';
|
||||
ctx.fillText(factCount + (factCount === 1 ? ' fact' : ' facts'), 160, 52);
|
||||
|
||||
const tex = new THREE.CanvasTexture(canvas);
|
||||
const mat = new THREE.SpriteMaterial({ map: tex, transparent: true, opacity: 0.88 });
|
||||
const sprite = new THREE.Sprite(mat);
|
||||
sprite.scale.set(5, 1.1, 1);
|
||||
sprite.userData = { type: 'session_room_label' };
|
||||
return sprite;
|
||||
}
|
||||
|
||||
// ─── HELPERS ──────────────────────────────────────────
|
||||
function _roundRect(ctx, x, y, w, h, r) {
|
||||
ctx.beginPath();
|
||||
ctx.moveTo(x + r, y);
|
||||
ctx.lineTo(x + w - r, y);
|
||||
ctx.quadraticCurveTo(x + w, y, x + w, y + r);
|
||||
ctx.lineTo(x + w, y + h - r);
|
||||
ctx.quadraticCurveTo(x + w, y + h, x + w - r, y + h);
|
||||
ctx.lineTo(x + r, y + h);
|
||||
ctx.quadraticCurveTo(x, y + h, x, y + h - r);
|
||||
ctx.lineTo(x, y + r);
|
||||
ctx.quadraticCurveTo(x, y, x + r, y);
|
||||
ctx.closePath();
|
||||
}
|
||||
|
||||
function _formatDate(dt) {
|
||||
if (isNaN(dt.getTime())) return 'Unknown session';
|
||||
const pad = n => String(n).padStart(2, '0');
|
||||
return `${dt.getFullYear()}-${pad(dt.getMonth() + 1)}-${pad(dt.getDate())} ${pad(dt.getHours())}:${pad(dt.getMinutes())}`;
|
||||
}
|
||||
|
||||
// ─── DISPOSE ROOM ────────────────────────────────────
|
||||
function _disposeRoom(room) {
|
||||
room.wireframe.geometry.dispose();
|
||||
room.wireframe.material.dispose();
|
||||
room.hitMesh.geometry.dispose();
|
||||
room.hitMesh.material.dispose();
|
||||
room.pointMesh.geometry.dispose();
|
||||
room.pointMesh.material.dispose();
|
||||
if (room.sprite.material.map) room.sprite.material.map.dispose();
|
||||
room.sprite.material.dispose();
|
||||
if (room.group.parent) room.group.parent.remove(room.group);
|
||||
delete _sessionIndex[room.session.id];
|
||||
}
|
||||
|
||||
// ─── PUBLIC: UPDATE SESSIONS ─────────────────────────
|
||||
// sessions: [{id, timestamp, facts:[{id,content,category,strength,...}]}]
|
||||
// Sorted chronologically oldest→newest; max MAX_ROOMS shown.
|
||||
function updateSessions(sessions) {
|
||||
if (!_scene) return;
|
||||
|
||||
const sorted = [...sessions]
|
||||
.sort((a, b) => new Date(a.timestamp) - new Date(b.timestamp))
|
||||
.slice(-MAX_ROOMS); // keep most recent MAX_ROOMS
|
||||
|
||||
// Remove rooms no longer present
|
||||
const incoming = new Set(sorted.map(s => s.id));
|
||||
for (let i = _rooms.length - 1; i >= 0; i--) {
|
||||
const room = _rooms[i];
|
||||
if (!incoming.has(room.session.id)) {
|
||||
_disposeRoom(room);
|
||||
_rooms.splice(i, 1);
|
||||
}
|
||||
}
|
||||
|
||||
// Add / update
|
||||
sorted.forEach((session, idx) => {
|
||||
if (_sessionIndex[session.id]) {
|
||||
// Update position if index changed
|
||||
const room = _sessionIndex[session.id];
|
||||
if (room.index !== idx) {
|
||||
room.index = idx;
|
||||
const newPos = _spiralPos(idx);
|
||||
room.group.position.copy(newPos);
|
||||
room.pos.copy(newPos);
|
||||
}
|
||||
} else {
|
||||
_createRoom(session, idx);
|
||||
}
|
||||
});
|
||||
|
||||
saveToStorage(sorted);
|
||||
console.info('[SessionRooms] Updated:', _rooms.length, 'session rooms');
|
||||
}
|
||||
|
||||
// ─── PUBLIC: INIT ─────────────────────────────────────
|
||||
function init(scene, camera, controls) {
|
||||
_scene = scene;
|
||||
_camera = camera;
|
||||
_controls = controls;
|
||||
console.info('[SessionRooms] Initialized');
|
||||
|
||||
// Restore persisted sessions
|
||||
const saved = loadFromStorage();
|
||||
if (saved && saved.length > 0) {
|
||||
updateSessions(saved);
|
||||
}
|
||||
}
|
||||
|
||||
// ─── PUBLIC: UPDATE (per-frame) ───────────────────────
|
||||
function update(delta) {
|
||||
if (!_scene || !_camera) return;
|
||||
|
||||
const camPos = _camera.position;
|
||||
|
||||
_rooms.forEach(room => {
|
||||
const dist = camPos.distanceTo(room.pos);
|
||||
|
||||
// LOD toggle
|
||||
const threshold = room.lodActive
|
||||
? LOD_THRESHOLD + LOD_HYSTERESIS // must come closer to exit LOD
|
||||
: LOD_THRESHOLD;
|
||||
|
||||
if (dist > threshold && !room.lodActive) {
|
||||
room.lodActive = true;
|
||||
room.wireframe.visible = false;
|
||||
room.sprite.visible = false;
|
||||
room.pointMesh.visible = true;
|
||||
} else if (dist <= threshold && room.lodActive) {
|
||||
room.lodActive = false;
|
||||
room.wireframe.visible = true;
|
||||
room.sprite.visible = true;
|
||||
room.pointMesh.visible = false;
|
||||
}
|
||||
|
||||
// Pulse wireframe opacity
|
||||
room.pulsePhase += delta * 0.6;
|
||||
if (!room.lodActive) {
|
||||
room.wireframe.material.opacity = 0.3 + Math.sin(room.pulsePhase) * 0.2;
|
||||
room.glow.intensity = 0.3 + Math.sin(room.pulsePhase * 1.4) * 0.15;
|
||||
}
|
||||
|
||||
// Slowly rotate each room
|
||||
room.group.rotation.y += delta * 0.04;
|
||||
});
|
||||
|
||||
// Fly-in tween
|
||||
if (_flyActive) {
|
||||
_flyElapsed += delta;
|
||||
const t = Math.min(_flyElapsed / FLY_DURATION, 1);
|
||||
const ease = _easeInOut(t);
|
||||
|
||||
_camera.position.lerpVectors(_flyFrom, _flyTo, ease);
|
||||
|
||||
// Interpolate lookAt
|
||||
const lookNow = new THREE.Vector3().lerpVectors(_flyLookFrom, _flyLookTo, ease);
|
||||
_camera.lookAt(lookNow);
|
||||
if (_controls && _controls.target) _controls.target.copy(lookNow);
|
||||
|
||||
if (t >= 1) {
|
||||
_flyActive = false;
|
||||
if (_controls && typeof _controls.update === 'function') _controls.update();
|
||||
console.info('[SessionRooms] Fly-in complete for session', _flyActiveRoom && _flyActiveRoom.session.id);
|
||||
_flyActiveRoom = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ─── EASING ───────────────────────────────────────────
|
||||
function _easeInOut(t) {
|
||||
return t < 0.5 ? 2 * t * t : -1 + (4 - 2 * t) * t;
|
||||
}
|
||||
|
||||
// ─── PUBLIC: GET CLICKABLE MESHES ─────────────────────
|
||||
function getClickableMeshes() {
|
||||
return _rooms.map(r => r.hitMesh);
|
||||
}
|
||||
|
||||
// ─── PUBLIC: HANDLE ROOM CLICK ────────────────────────
|
||||
function handleRoomClick(mesh) {
|
||||
const { sessionId } = mesh.userData;
|
||||
const room = _sessionIndex[sessionId];
|
||||
if (!room || !_camera) return null;
|
||||
|
||||
// Fly into the room from the front face
|
||||
_flyActive = true;
|
||||
_flyElapsed = 0;
|
||||
_flyActiveRoom = room;
|
||||
|
||||
_flyFrom = _camera.position.clone();
|
||||
|
||||
// Target: step inside the room toward its center
|
||||
const dir = room.pos.clone().sub(_camera.position).normalize();
|
||||
_flyTo = room.pos.clone().add(dir.multiplyScalar(FLY_TARGET_DEPTH));
|
||||
|
||||
_flyLookFrom = _controls && _controls.target
|
||||
? _controls.target.clone()
|
||||
: _camera.position.clone().add(_camera.getWorldDirection(new THREE.Vector3()));
|
||||
_flyLookTo = room.pos.clone();
|
||||
|
||||
console.info('[SessionRooms] Flying into session room:', sessionId);
|
||||
return room.session;
|
||||
}
|
||||
|
||||
// ─── PERSISTENCE ──────────────────────────────────────
|
||||
function saveToStorage(sessions) {
|
||||
if (typeof localStorage === 'undefined') return;
|
||||
try {
|
||||
localStorage.setItem(STORAGE_KEY, JSON.stringify({ v: 1, sessions }));
|
||||
} catch (e) {
|
||||
console.warn('[SessionRooms] Failed to save to localStorage:', e);
|
||||
}
|
||||
}
|
||||
|
||||
function loadFromStorage() {
|
||||
if (typeof localStorage === 'undefined') return null;
|
||||
try {
|
||||
const raw = localStorage.getItem(STORAGE_KEY);
|
||||
if (!raw) return null;
|
||||
const parsed = JSON.parse(raw);
|
||||
if (!parsed || parsed.v !== 1 || !Array.isArray(parsed.sessions)) return null;
|
||||
console.info('[SessionRooms] Restored', parsed.sessions.length, 'sessions from localStorage');
|
||||
return parsed.sessions;
|
||||
} catch (e) {
|
||||
console.warn('[SessionRooms] Failed to load from localStorage:', e);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function clearStorage() {
|
||||
if (typeof localStorage !== 'undefined') {
|
||||
localStorage.removeItem(STORAGE_KEY);
|
||||
console.info('[SessionRooms] Cleared localStorage');
|
||||
}
|
||||
}
|
||||
|
||||
// ─── PUBLIC API ───────────────────────────────────────
|
||||
return {
|
||||
init,
|
||||
updateSessions,
|
||||
update,
|
||||
getClickableMeshes,
|
||||
handleRoomClick,
|
||||
clearStorage,
|
||||
// For external inspection
|
||||
getRooms: () => _rooms,
|
||||
getSession: (id) => _sessionIndex[id] || null,
|
||||
isFlyActive: () => _flyActive
|
||||
};
|
||||
|
||||
})();
|
||||
|
||||
export { SessionRooms };
|
||||
665
nexus/components/spatial-memory.js
Normal file
665
nexus/components/spatial-memory.js
Normal file
@@ -0,0 +1,665 @@
|
||||
// ═══════════════════════════════════════════
|
||||
// PROJECT MNEMOSYNE — SPATIAL MEMORY SCHEMA
|
||||
// ═══════════════════════════════════════════
|
||||
//
|
||||
// Maps memories to persistent locations in the 3D Nexus world.
|
||||
// Each region corresponds to a semantic category. Memories placed
|
||||
// in a region stay there across sessions, forming a navigable
|
||||
// holographic archive.
|
||||
//
|
||||
// World layout (hex cylinder, radius 25):
|
||||
//
|
||||
// Inner ring — original Mnemosyne taxonomy (radius 15):
|
||||
// North (z-) → Documents & Knowledge
|
||||
// South (z+) → Projects & Tasks
|
||||
// East (x+) → Code & Engineering
|
||||
// West (x-) → Conversations & Social
|
||||
// Center → Active Working Memory
|
||||
// Below (y-) → Archive (cold storage)
|
||||
//
|
||||
// Outer ring — MemPalace category zones (radius 20, issue #1168):
|
||||
// North (z-) → User Preferences [golden]
|
||||
// East (x+) → Project facts [blue]
|
||||
// South (z+) → Tool knowledge [green]
|
||||
// West (x-) → General facts [gray]
|
||||
//
|
||||
// Usage from app.js:
|
||||
// SpatialMemory.init(scene);
|
||||
// SpatialMemory.placeMemory({ id, content, category, ... });
|
||||
// SpatialMemory.importIndex(savedIndex);
|
||||
// SpatialMemory.update(delta);
|
||||
// ═══════════════════════════════════════════
|
||||
|
||||
const SpatialMemory = (() => {
|
||||
|
||||
// ─── REGION DEFINITIONS ───────────────────────────────
|
||||
const REGIONS = {
|
||||
engineering: {
|
||||
label: 'Code & Engineering',
|
||||
center: [15, 0, 0],
|
||||
radius: 10,
|
||||
color: 0x4af0c0,
|
||||
glyph: '\u2699',
|
||||
description: 'Source code, debugging sessions, architecture decisions'
|
||||
},
|
||||
social: {
|
||||
label: 'Conversations & Social',
|
||||
center: [-15, 0, 0],
|
||||
radius: 10,
|
||||
color: 0x7b5cff,
|
||||
glyph: '\uD83D\uDCAC',
|
||||
description: 'Chats, discussions, human interactions'
|
||||
},
|
||||
knowledge: {
|
||||
label: 'Documents & Knowledge',
|
||||
center: [0, 0, -15],
|
||||
radius: 10,
|
||||
color: 0xffd700,
|
||||
glyph: '\uD83D\uDCD6',
|
||||
description: 'Papers, docs, research, learned concepts'
|
||||
},
|
||||
projects: {
|
||||
label: 'Projects & Tasks',
|
||||
center: [0, 0, 15],
|
||||
radius: 10,
|
||||
color: 0xff4466,
|
||||
glyph: '\uD83C\uDFAF',
|
||||
description: 'Active tasks, issues, milestones, goals'
|
||||
},
|
||||
working: {
|
||||
label: 'Active Working Memory',
|
||||
center: [0, 0, 0],
|
||||
radius: 5,
|
||||
color: 0x00ff88,
|
||||
glyph: '\uD83D\uDCA1',
|
||||
description: 'Current focus — transient, high-priority memories'
|
||||
},
|
||||
archive: {
|
||||
label: 'Archive',
|
||||
center: [0, -3, 0],
|
||||
radius: 20,
|
||||
color: 0x334455,
|
||||
glyph: '\uD83D\uDDC4',
|
||||
description: 'Cold storage — rarely accessed, aged-out memories'
|
||||
},
|
||||
|
||||
// ── MemPalace category zones — outer ring, issue #1168 ────────────
|
||||
user_pref: {
|
||||
label: 'User Preferences',
|
||||
center: [0, 0, -20],
|
||||
radius: 10,
|
||||
color: 0xffd700,
|
||||
glyph: '\u2605',
|
||||
description: 'Personal preferences, habits, user-specific settings',
|
||||
labelY: 5
|
||||
},
|
||||
project: {
|
||||
label: 'Project Facts',
|
||||
center: [20, 0, 0],
|
||||
radius: 10,
|
||||
color: 0x4488ff,
|
||||
glyph: '\uD83D\uDCC1',
|
||||
description: 'Project-specific knowledge, goals, context',
|
||||
labelY: 5
|
||||
},
|
||||
tool: {
|
||||
label: 'Tool Knowledge',
|
||||
center: [0, 0, 20],
|
||||
radius: 10,
|
||||
color: 0x44cc66,
|
||||
glyph: '\uD83D\uDD27',
|
||||
description: 'Tools, commands, APIs, and how to use them',
|
||||
labelY: 5
|
||||
},
|
||||
general: {
|
||||
label: 'General Facts',
|
||||
center: [-20, 0, 0],
|
||||
radius: 10,
|
||||
color: 0x8899aa,
|
||||
glyph: '\uD83D\uDCDD',
|
||||
description: 'Miscellaneous facts not fitting other categories',
|
||||
labelY: 5
|
||||
}
|
||||
};
|
||||
|
||||
// ─── PERSISTENCE CONFIG ──────────────────────────────
|
||||
const STORAGE_KEY = 'mnemosyne_spatial_memory';
|
||||
const STORAGE_VERSION = 1;
|
||||
let _dirty = false;
|
||||
let _lastSavedHash = '';
|
||||
|
||||
// ─── STATE ────────────────────────────────────────────
|
||||
let _scene = null;
|
||||
let _regionMarkers = {};
|
||||
let _memoryObjects = {};
|
||||
let _connectionLines = [];
|
||||
let _initialized = false;
|
||||
|
||||
// ─── CRYSTAL GEOMETRY (persistent memories) ───────────
|
||||
function createCrystalGeometry(size) {
|
||||
return new THREE.OctahedronGeometry(size, 0);
|
||||
}
|
||||
|
||||
// ─── REGION MARKER ───────────────────────────────────
|
||||
function createRegionMarker(regionKey, region) {
|
||||
const cx = region.center[0];
|
||||
const cy = region.center[1] + 0.06;
|
||||
const cz = region.center[2];
|
||||
const labelY = region.labelY || 3;
|
||||
|
||||
const ringGeo = new THREE.RingGeometry(region.radius - 0.5, region.radius, 6);
|
||||
const ringMat = new THREE.MeshBasicMaterial({
|
||||
color: region.color,
|
||||
transparent: true,
|
||||
opacity: 0.15,
|
||||
side: THREE.DoubleSide
|
||||
});
|
||||
const ring = new THREE.Mesh(ringGeo, ringMat);
|
||||
ring.rotation.x = -Math.PI / 2;
|
||||
ring.position.set(cx, cy, cz);
|
||||
ring.userData = { type: 'region_marker', region: regionKey };
|
||||
|
||||
const discGeo = new THREE.CircleGeometry(region.radius - 0.5, 6);
|
||||
const discMat = new THREE.MeshBasicMaterial({
|
||||
color: region.color,
|
||||
transparent: true,
|
||||
opacity: 0.03,
|
||||
side: THREE.DoubleSide
|
||||
});
|
||||
const disc = new THREE.Mesh(discGeo, discMat);
|
||||
disc.rotation.x = -Math.PI / 2;
|
||||
disc.position.set(cx, cy - 0.01, cz);
|
||||
|
||||
_scene.add(ring);
|
||||
_scene.add(disc);
|
||||
|
||||
// Ground glow — brighter disc for MemPalace zones (labelY > 3 signals outer ring)
|
||||
let glowDisc = null;
|
||||
if (labelY > 3) {
|
||||
const glowGeo = new THREE.CircleGeometry(region.radius, 32);
|
||||
const glowMat = new THREE.MeshBasicMaterial({
|
||||
color: region.color,
|
||||
transparent: true,
|
||||
opacity: 0.06,
|
||||
side: THREE.DoubleSide
|
||||
});
|
||||
glowDisc = new THREE.Mesh(glowGeo, glowMat);
|
||||
glowDisc.rotation.x = -Math.PI / 2;
|
||||
glowDisc.position.set(cx, cy - 0.02, cz);
|
||||
_scene.add(glowDisc);
|
||||
}
|
||||
|
||||
// Floating label
|
||||
const canvas = document.createElement('canvas');
|
||||
canvas.width = 256;
|
||||
canvas.height = 64;
|
||||
const ctx = canvas.getContext('2d');
|
||||
ctx.font = '24px monospace';
|
||||
ctx.fillStyle = '#' + region.color.toString(16).padStart(6, '0');
|
||||
ctx.textAlign = 'center';
|
||||
ctx.fillText(region.glyph + ' ' + region.label, 128, 40);
|
||||
|
||||
const texture = new THREE.CanvasTexture(canvas);
|
||||
const spriteMat = new THREE.SpriteMaterial({ map: texture, transparent: true, opacity: 0.6 });
|
||||
const sprite = new THREE.Sprite(spriteMat);
|
||||
sprite.position.set(cx, labelY, cz);
|
||||
sprite.scale.set(4, 1, 1);
|
||||
_scene.add(sprite);
|
||||
|
||||
return { ring, disc, glowDisc, sprite };
|
||||
}
|
||||
|
||||
// ─── PLACE A MEMORY ──────────────────────────────────
|
||||
function placeMemory(mem) {
|
||||
if (!_scene) return null;
|
||||
|
||||
const region = REGIONS[mem.category] || REGIONS.working;
|
||||
const pos = mem.position || _assignPosition(mem.category, mem.id);
|
||||
const strength = Math.max(0.05, Math.min(1, mem.strength != null ? mem.strength : 0.7));
|
||||
const size = 0.2 + strength * 0.3;
|
||||
|
||||
const geo = createCrystalGeometry(size);
|
||||
const mat = new THREE.MeshStandardMaterial({
|
||||
color: region.color,
|
||||
emissive: region.color,
|
||||
emissiveIntensity: 1.5 * strength,
|
||||
metalness: 0.6,
|
||||
roughness: 0.15,
|
||||
transparent: true,
|
||||
opacity: 0.5 + strength * 0.4
|
||||
});
|
||||
|
||||
const crystal = new THREE.Mesh(geo, mat);
|
||||
crystal.position.set(pos[0], pos[1] + 1.5, pos[2]);
|
||||
crystal.castShadow = true;
|
||||
|
||||
crystal.userData = {
|
||||
type: 'spatial_memory',
|
||||
memId: mem.id,
|
||||
region: mem.category,
|
||||
pulse: Math.random() * Math.PI * 2,
|
||||
strength: strength,
|
||||
createdAt: mem.timestamp || new Date().toISOString()
|
||||
};
|
||||
|
||||
const light = new THREE.PointLight(region.color, 0.8 * strength, 5);
|
||||
crystal.add(light);
|
||||
|
||||
_scene.add(crystal);
|
||||
_memoryObjects[mem.id] = { mesh: crystal, data: mem, region: mem.category };
|
||||
|
||||
if (mem.connections && mem.connections.length > 0) {
|
||||
_drawConnections(mem.id, mem.connections);
|
||||
}
|
||||
|
||||
_dirty = true;
|
||||
saveToStorage();
|
||||
console.info('[Mnemosyne] Spatial memory placed:', mem.id, 'in', region.label);
|
||||
return crystal;
|
||||
}
|
||||
|
||||
// ─── DETERMINISTIC POSITION ──────────────────────────
|
||||
function _assignPosition(category, memId) {
|
||||
const region = REGIONS[category] || REGIONS.working;
|
||||
const cx = region.center[0];
|
||||
const cy = region.center[1];
|
||||
const cz = region.center[2];
|
||||
const r = region.radius * 0.7;
|
||||
|
||||
let hash = 0;
|
||||
for (let i = 0; i < memId.length; i++) {
|
||||
hash = ((hash << 5) - hash) + memId.charCodeAt(i);
|
||||
hash |= 0;
|
||||
}
|
||||
|
||||
const angle = (Math.abs(hash % 360) / 360) * Math.PI * 2;
|
||||
const dist = (Math.abs((hash >> 8) % 100) / 100) * r;
|
||||
const height = (Math.abs((hash >> 16) % 100) / 100) * 3;
|
||||
|
||||
return [cx + Math.cos(angle) * dist, cy + height, cz + Math.sin(angle) * dist];
|
||||
}
|
||||
|
||||
// ─── CONNECTIONS ─────────────────────────────────────
|
||||
function _drawConnections(memId, connections) {
|
||||
const src = _memoryObjects[memId];
|
||||
if (!src) return;
|
||||
|
||||
connections.forEach(targetId => {
|
||||
const tgt = _memoryObjects[targetId];
|
||||
if (!tgt) return;
|
||||
|
||||
const points = [src.mesh.position.clone(), tgt.mesh.position.clone()];
|
||||
const geo = new THREE.BufferGeometry().setFromPoints(points);
|
||||
const mat = new THREE.LineBasicMaterial({ color: 0x334455, transparent: true, opacity: 0.2 });
|
||||
const line = new THREE.Line(geo, mat);
|
||||
line.userData = { type: 'connection', from: memId, to: targetId };
|
||||
_scene.add(line);
|
||||
_connectionLines.push(line);
|
||||
});
|
||||
}
|
||||
|
||||
// ─── REMOVE A MEMORY ─────────────────────────────────
|
||||
function removeMemory(memId) {
|
||||
const obj = _memoryObjects[memId];
|
||||
if (!obj) return;
|
||||
|
||||
if (obj.mesh.parent) obj.mesh.parent.remove(obj.mesh);
|
||||
if (obj.mesh.geometry) obj.mesh.geometry.dispose();
|
||||
if (obj.mesh.material) obj.mesh.material.dispose();
|
||||
|
||||
for (let i = _connectionLines.length - 1; i >= 0; i--) {
|
||||
const line = _connectionLines[i];
|
||||
if (line.userData.from === memId || line.userData.to === memId) {
|
||||
if (line.parent) line.parent.remove(line);
|
||||
line.geometry.dispose();
|
||||
line.material.dispose();
|
||||
_connectionLines.splice(i, 1);
|
||||
}
|
||||
}
|
||||
|
||||
delete _memoryObjects[memId];
|
||||
_dirty = true;
|
||||
saveToStorage();
|
||||
}
|
||||
|
||||
// ─── ANIMATE ─────────────────────────────────────────
|
||||
function update(delta) {
|
||||
const now = Date.now();
|
||||
|
||||
Object.values(_memoryObjects).forEach(obj => {
|
||||
const mesh = obj.mesh;
|
||||
if (!mesh || !mesh.userData) return;
|
||||
|
||||
mesh.rotation.y += delta * 0.3;
|
||||
|
||||
mesh.userData.pulse += delta * 1.5;
|
||||
const pulse = 1 + Math.sin(mesh.userData.pulse) * 0.08;
|
||||
mesh.scale.setScalar(pulse);
|
||||
|
||||
if (mesh.material) {
|
||||
const base = mesh.userData.strength || 0.7;
|
||||
mesh.material.emissiveIntensity = 1.0 + Math.sin(mesh.userData.pulse * 0.7) * 0.5 * base;
|
||||
}
|
||||
});
|
||||
|
||||
Object.values(_regionMarkers).forEach(marker => {
|
||||
if (marker.ring && marker.ring.material) {
|
||||
marker.ring.material.opacity = 0.1 + Math.sin(now * 0.001) * 0.05;
|
||||
}
|
||||
if (marker.glowDisc && marker.glowDisc.material) {
|
||||
marker.glowDisc.material.opacity = 0.04 + Math.sin(now * 0.0008) * 0.02;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// ─── INIT ────────────────────────────────────────────
|
||||
function init(scene) {
|
||||
_scene = scene;
|
||||
_initialized = true;
|
||||
|
||||
Object.entries(REGIONS).forEach(([key, region]) => {
|
||||
if (key === 'archive') return;
|
||||
_regionMarkers[key] = createRegionMarker(key, region);
|
||||
});
|
||||
|
||||
// Restore persisted memories
|
||||
const restored = loadFromStorage();
|
||||
console.info('[Mnemosyne] Spatial Memory Schema initialized —', Object.keys(REGIONS).length, 'regions,', restored, 'memories restored');
|
||||
return REGIONS;
|
||||
}
|
||||
|
||||
// ─── QUERY ───────────────────────────────────────────
|
||||
function getMemoryAtPosition(position, maxDist) {
|
||||
maxDist = maxDist || 2;
|
||||
let closest = null;
|
||||
let closestDist = maxDist;
|
||||
|
||||
Object.values(_memoryObjects).forEach(obj => {
|
||||
const d = obj.mesh.position.distanceTo(position);
|
||||
if (d < closestDist) { closest = obj; closestDist = d; }
|
||||
});
|
||||
return closest;
|
||||
}
|
||||
|
||||
function getRegionAtPosition(position) {
|
||||
for (const [key, region] of Object.entries(REGIONS)) {
|
||||
const dx = position.x - region.center[0];
|
||||
const dz = position.z - region.center[2];
|
||||
if (Math.sqrt(dx * dx + dz * dz) <= region.radius) return key;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function getMemoriesInRegion(regionKey) {
|
||||
return Object.values(_memoryObjects).filter(o => o.region === regionKey);
|
||||
}
|
||||
|
||||
function getAllMemories() {
|
||||
return Object.values(_memoryObjects).map(o => o.data);
|
||||
}
|
||||
|
||||
// ─── LOCALSTORAGE PERSISTENCE ────────────────────────
|
||||
function _indexHash(index) {
|
||||
// Simple hash of memory IDs + count to detect changes
|
||||
const ids = (index.memories || []).map(m => m.id).sort().join(',');
|
||||
return index.memories.length + ':' + ids;
|
||||
}
|
||||
|
||||
function saveToStorage() {
|
||||
if (typeof localStorage === 'undefined') {
|
||||
console.warn('[Mnemosyne] localStorage unavailable — skipping save');
|
||||
return false;
|
||||
}
|
||||
try {
|
||||
const index = exportIndex();
|
||||
const hash = _indexHash(index);
|
||||
if (hash === _lastSavedHash) return false; // no change
|
||||
|
||||
const payload = JSON.stringify(index);
|
||||
localStorage.setItem(STORAGE_KEY, payload);
|
||||
_lastSavedHash = hash;
|
||||
_dirty = false;
|
||||
console.info('[Mnemosyne] Saved', index.memories.length, 'memories to localStorage');
|
||||
return true;
|
||||
} catch (e) {
|
||||
if (e.name === 'QuotaExceededError' || e.code === 22) {
|
||||
console.warn('[Mnemosyne] localStorage quota exceeded — pruning archive memories');
|
||||
_pruneArchiveMemories();
|
||||
try {
|
||||
const index = exportIndex();
|
||||
localStorage.setItem(STORAGE_KEY, JSON.stringify(index));
|
||||
_lastSavedHash = _indexHash(index);
|
||||
console.info('[Mnemosyne] Saved after prune:', index.memories.length, 'memories');
|
||||
return true;
|
||||
} catch (e2) {
|
||||
console.error('[Mnemosyne] Save failed even after prune:', e2);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
console.error('[Mnemosyne] Save failed:', e);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function loadFromStorage() {
|
||||
if (typeof localStorage === 'undefined') {
|
||||
console.warn('[Mnemosyne] localStorage unavailable — starting empty');
|
||||
return 0;
|
||||
}
|
||||
try {
|
||||
const raw = localStorage.getItem(STORAGE_KEY);
|
||||
if (!raw) {
|
||||
console.info('[Mnemosyne] No saved state found — starting fresh');
|
||||
return 0;
|
||||
}
|
||||
const index = JSON.parse(raw);
|
||||
if (index.version !== STORAGE_VERSION) {
|
||||
console.warn('[Mnemosyne] Saved version mismatch (got', index.version, 'expected', + STORAGE_VERSION + ') — starting fresh');
|
||||
return 0;
|
||||
}
|
||||
const count = importIndex(index);
|
||||
_lastSavedHash = _indexHash(index);
|
||||
return count;
|
||||
} catch (e) {
|
||||
console.error('[Mnemosyne] Load failed:', e);
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
function _pruneArchiveMemories() {
|
||||
// Remove oldest archive-region memories first
|
||||
const archive = getMemoriesInRegion('archive');
|
||||
const working = Object.values(_memoryObjects).filter(o => o.region !== 'archive');
|
||||
// Sort archive by timestamp ascending (oldest first)
|
||||
archive.sort((a, b) => {
|
||||
const ta = a.data.timestamp || a.mesh.userData.createdAt || '';
|
||||
const tb = b.data.timestamp || b.mesh.userData.createdAt || '';
|
||||
return ta.localeCompare(tb);
|
||||
});
|
||||
const toRemove = Math.max(1, Math.ceil(archive.length * 0.25));
|
||||
for (let i = 0; i < toRemove && i < archive.length; i++) {
|
||||
removeMemory(archive[i].data.id);
|
||||
}
|
||||
console.info('[Mnemosyne] Pruned', toRemove, 'archive memories');
|
||||
}
|
||||
|
||||
function clearStorage() {
|
||||
if (typeof localStorage !== 'undefined') {
|
||||
localStorage.removeItem(STORAGE_KEY);
|
||||
_lastSavedHash = '';
|
||||
console.info('[Mnemosyne] Cleared localStorage');
|
||||
}
|
||||
}
|
||||
|
||||
// ─── PERSISTENCE ─────────────────────────────────────
|
||||
function exportIndex() {
|
||||
return {
|
||||
version: 1,
|
||||
exportedAt: new Date().toISOString(),
|
||||
regions: Object.fromEntries(
|
||||
Object.entries(REGIONS).map(([k, v]) => [k, { label: v.label, center: v.center, radius: v.radius, color: v.color }])
|
||||
),
|
||||
memories: Object.values(_memoryObjects).map(o => ({
|
||||
id: o.data.id,
|
||||
content: o.data.content,
|
||||
category: o.region,
|
||||
position: [o.mesh.position.x, o.mesh.position.y - 1.5, o.mesh.position.z],
|
||||
source: o.data.source || 'unknown',
|
||||
timestamp: o.data.timestamp || o.mesh.userData.createdAt,
|
||||
strength: o.mesh.userData.strength || 0.7,
|
||||
connections: o.data.connections || []
|
||||
}))
|
||||
};
|
||||
}
|
||||
|
||||
function importIndex(index) {
|
||||
if (!index || !index.memories) return 0;
|
||||
let count = 0;
|
||||
index.memories.forEach(mem => {
|
||||
if (!_memoryObjects[mem.id]) { placeMemory(mem); count++; }
|
||||
});
|
||||
console.info('[Mnemosyne] Restored', count, 'memories from index');
|
||||
return count;
|
||||
}
|
||||
|
||||
// ─── GRAVITY WELL CLUSTERING ──────────────────────────
|
||||
// Force-directed layout: same-category crystals attract, unrelated repel.
|
||||
// Run on load (bake positions, not per-frame). Spec from issue #1175.
|
||||
const GRAVITY_ITERATIONS = 20;
|
||||
const ATTRACT_FACTOR = 0.10; // 10% closer to same-category centroid per iteration
|
||||
const REPEL_FACTOR = 0.05; // 5% away from nearest unrelated crystal
|
||||
|
||||
function runGravityLayout() {
|
||||
const objs = Object.values(_memoryObjects);
|
||||
if (objs.length < 2) {
|
||||
console.info('[Mnemosyne] Gravity layout: fewer than 2 crystals, skipping');
|
||||
return;
|
||||
}
|
||||
console.info('[Mnemosyne] Gravity layout starting —', objs.length, 'crystals,', GRAVITY_ITERATIONS, 'iterations');
|
||||
|
||||
for (let iter = 0; iter < GRAVITY_ITERATIONS; iter++) {
|
||||
// Accumulate displacements before applying (avoids order-of-iteration bias)
|
||||
const dx = new Float32Array(objs.length);
|
||||
const dy = new Float32Array(objs.length);
|
||||
const dz = new Float32Array(objs.length);
|
||||
|
||||
objs.forEach((obj, i) => {
|
||||
const pos = obj.mesh.position;
|
||||
const cat = obj.region;
|
||||
|
||||
// ── Attraction toward same-category centroid ──────────────
|
||||
let sx = 0, sy = 0, sz = 0, sameCount = 0;
|
||||
objs.forEach(o => {
|
||||
if (o === obj || o.region !== cat) return;
|
||||
sx += o.mesh.position.x;
|
||||
sy += o.mesh.position.y;
|
||||
sz += o.mesh.position.z;
|
||||
sameCount++;
|
||||
});
|
||||
if (sameCount > 0) {
|
||||
dx[i] += ((sx / sameCount) - pos.x) * ATTRACT_FACTOR;
|
||||
dy[i] += ((sy / sameCount) - pos.y) * ATTRACT_FACTOR;
|
||||
dz[i] += ((sz / sameCount) - pos.z) * ATTRACT_FACTOR;
|
||||
}
|
||||
|
||||
// ── Repulsion from nearest unrelated crystal ───────────────
|
||||
let nearestDist = Infinity;
|
||||
let rnx = 0, rny = 0, rnz = 0;
|
||||
objs.forEach(o => {
|
||||
if (o === obj || o.region === cat) return;
|
||||
const ex = pos.x - o.mesh.position.x;
|
||||
const ey = pos.y - o.mesh.position.y;
|
||||
const ez = pos.z - o.mesh.position.z;
|
||||
const d = Math.sqrt(ex * ex + ey * ey + ez * ez);
|
||||
if (d < nearestDist) {
|
||||
nearestDist = d;
|
||||
rnx = ex; rny = ey; rnz = ez;
|
||||
}
|
||||
});
|
||||
if (nearestDist > 0.001 && nearestDist < Infinity) {
|
||||
const len = Math.sqrt(rnx * rnx + rny * rny + rnz * rnz);
|
||||
dx[i] += (rnx / len) * nearestDist * REPEL_FACTOR;
|
||||
dy[i] += (rny / len) * nearestDist * REPEL_FACTOR;
|
||||
dz[i] += (rnz / len) * nearestDist * REPEL_FACTOR;
|
||||
}
|
||||
});
|
||||
|
||||
// Apply displacements
|
||||
objs.forEach((obj, i) => {
|
||||
obj.mesh.position.x += dx[i];
|
||||
obj.mesh.position.y += dy[i];
|
||||
obj.mesh.position.z += dz[i];
|
||||
});
|
||||
}
|
||||
|
||||
// Bake final positions to localStorage
|
||||
saveToStorage();
|
||||
console.info('[Mnemosyne] Gravity layout complete — positions baked to localStorage');
|
||||
}
|
||||
|
||||
// ─── SPATIAL SEARCH ──────────────────────────────────
|
||||
function searchNearby(position, maxResults, maxDist) {
|
||||
maxResults = maxResults || 10;
|
||||
maxDist = maxDist || 30;
|
||||
const results = [];
|
||||
|
||||
Object.values(_memoryObjects).forEach(obj => {
|
||||
const d = obj.mesh.position.distanceTo(position);
|
||||
if (d <= maxDist) results.push({ memory: obj.data, distance: d, position: obj.mesh.position.clone() });
|
||||
});
|
||||
|
||||
results.sort((a, b) => a.distance - b.distance);
|
||||
return results.slice(0, maxResults);
|
||||
}
|
||||
|
||||
|
||||
// ─── CRYSTAL MESH COLLECTION (for raycasting) ────────
|
||||
function getCrystalMeshes() {
|
||||
return Object.values(_memoryObjects).map(o => o.mesh);
|
||||
}
|
||||
|
||||
// ─── MEMORY DATA FROM MESH ───────────────────────────
|
||||
function getMemoryFromMesh(mesh) {
|
||||
const entry = Object.values(_memoryObjects).find(o => o.mesh === mesh);
|
||||
return entry ? { data: entry.data, region: entry.region } : null;
|
||||
}
|
||||
|
||||
// ─── HIGHLIGHT / SELECT ──────────────────────────────
|
||||
let _selectedId = null;
|
||||
let _selectedOriginalEmissive = null;
|
||||
|
||||
function highlightMemory(memId) {
|
||||
clearHighlight();
|
||||
const obj = _memoryObjects[memId];
|
||||
if (!obj) return;
|
||||
_selectedId = memId;
|
||||
_selectedOriginalEmissive = obj.mesh.material.emissiveIntensity;
|
||||
obj.mesh.material.emissiveIntensity = 4.0;
|
||||
obj.mesh.userData.selected = true;
|
||||
}
|
||||
|
||||
function clearHighlight() {
|
||||
if (_selectedId && _memoryObjects[_selectedId]) {
|
||||
const obj = _memoryObjects[_selectedId];
|
||||
obj.mesh.material.emissiveIntensity = _selectedOriginalEmissive || (obj.data.strength || 0.7) * 2.5;
|
||||
obj.mesh.userData.selected = false;
|
||||
}
|
||||
_selectedId = null;
|
||||
_selectedOriginalEmissive = null;
|
||||
}
|
||||
|
||||
function getSelectedId() {
|
||||
return _selectedId;
|
||||
}
|
||||
|
||||
return {
|
||||
init, placeMemory, removeMemory, update,
|
||||
getMemoryAtPosition, getRegionAtPosition, getMemoriesInRegion, getAllMemories,
|
||||
getCrystalMeshes, getMemoryFromMesh, highlightMemory, clearHighlight, getSelectedId,
|
||||
exportIndex, importIndex, searchNearby, REGIONS,
|
||||
saveToStorage, loadFromStorage, clearStorage,
|
||||
runGravityLayout
|
||||
};
|
||||
})();
|
||||
|
||||
export { SpatialMemory };
|
||||
313
nexus/computer_use.py
Normal file
313
nexus/computer_use.py
Normal file
@@ -0,0 +1,313 @@
|
||||
"""
|
||||
Hermes Desktop Automation Primitives — Computer Use (#1125)
|
||||
|
||||
Provides sandboxed desktop control tools for Hermes agents:
|
||||
- computer_screenshot() — capture current desktop
|
||||
- computer_click() — mouse click with poka-yoke on non-primary buttons
|
||||
- computer_type() — keyboard input with poka-yoke on sensitive text
|
||||
- computer_scroll() — scroll wheel action
|
||||
- read_action_log() — inspect recent action audit trail
|
||||
|
||||
All actions are logged to a JSONL audit file.
|
||||
pyautogui.FAILSAFE is enabled globally — move mouse to top-left corner to abort.
|
||||
|
||||
Designed to degrade gracefully when no display is available (headless CI).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import io
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Safety globals
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Poka-yoke: require confirmation for dangerous inputs
|
||||
_SENSITIVE_KEYWORDS = frozenset(
|
||||
["password", "passwd", "secret", "token", "api_key", "apikey", "key", "auth"]
|
||||
)
|
||||
|
||||
# Destructive mouse buttons (non-primary)
|
||||
_DANGEROUS_BUTTONS = frozenset(["right", "middle"])
|
||||
|
||||
# Default log location
|
||||
DEFAULT_ACTION_LOG = Path.home() / ".nexus" / "computer_use_actions.jsonl"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Lazy pyautogui import — fails gracefully in headless environments
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_PYAUTOGUI_AVAILABLE = False
|
||||
_pyautogui = None
|
||||
|
||||
|
||||
def _get_pyautogui():
|
||||
"""Return pyautogui, enabling FAILSAFE. Returns None if unavailable."""
|
||||
global _pyautogui, _PYAUTOGUI_AVAILABLE
|
||||
if _pyautogui is not None:
|
||||
return _pyautogui
|
||||
try:
|
||||
import pyautogui # type: ignore
|
||||
|
||||
pyautogui.FAILSAFE = True
|
||||
pyautogui.PAUSE = 0.05 # small delay between actions
|
||||
_pyautogui = pyautogui
|
||||
_PYAUTOGUI_AVAILABLE = True
|
||||
return _pyautogui
|
||||
except Exception:
|
||||
logger.warning("pyautogui unavailable — computer_use running in stub mode")
|
||||
return None
|
||||
|
||||
|
||||
def _get_pil():
|
||||
"""Return PIL Image module or None."""
|
||||
try:
|
||||
from PIL import Image # type: ignore
|
||||
|
||||
return Image
|
||||
except ImportError:
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Audit log
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _log_action(action: str, params: dict, result: dict, log_path: Path = DEFAULT_ACTION_LOG):
|
||||
"""Append one action record to the JSONL audit log."""
|
||||
log_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
record = {
|
||||
"ts": time.strftime("%Y-%m-%dT%H:%M:%S"),
|
||||
"action": action,
|
||||
"params": params,
|
||||
"result": result,
|
||||
}
|
||||
with open(log_path, "a") as fh:
|
||||
fh.write(json.dumps(record) + "\n")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public tool API
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def computer_screenshot(
|
||||
save_path: Optional[str] = None,
|
||||
log_path: Path = DEFAULT_ACTION_LOG,
|
||||
) -> dict:
|
||||
"""Capture a screenshot of the current desktop.
|
||||
|
||||
Args:
|
||||
save_path: Optional file path to save the PNG. If omitted the image
|
||||
is returned as a base64-encoded string.
|
||||
log_path: Audit log file (default ~/.nexus/computer_use_actions.jsonl).
|
||||
|
||||
Returns:
|
||||
dict with keys:
|
||||
- ok (bool)
|
||||
- image_b64 (str | None) — base64 PNG when save_path is None
|
||||
- saved_to (str | None) — path when save_path was given
|
||||
- error (str | None) — human-readable error if ok=False
|
||||
"""
|
||||
pag = _get_pyautogui()
|
||||
params = {"save_path": save_path}
|
||||
|
||||
if pag is None:
|
||||
result = {"ok": False, "image_b64": None, "saved_to": None, "error": "pyautogui unavailable"}
|
||||
_log_action("screenshot", params, result, log_path)
|
||||
return result
|
||||
|
||||
try:
|
||||
screenshot = pag.screenshot()
|
||||
if save_path:
|
||||
screenshot.save(save_path)
|
||||
result = {"ok": True, "image_b64": None, "saved_to": save_path, "error": None}
|
||||
else:
|
||||
buf = io.BytesIO()
|
||||
screenshot.save(buf, format="PNG")
|
||||
b64 = base64.b64encode(buf.getvalue()).decode()
|
||||
result = {"ok": True, "image_b64": b64, "saved_to": None, "error": None}
|
||||
except Exception as exc:
|
||||
result = {"ok": False, "image_b64": None, "saved_to": None, "error": str(exc)}
|
||||
|
||||
_log_action("screenshot", params, {k: v for k, v in result.items() if k != "image_b64"}, log_path)
|
||||
return result
|
||||
|
||||
|
||||
def computer_click(
|
||||
x: int,
|
||||
y: int,
|
||||
button: str = "left",
|
||||
confirm: bool = False,
|
||||
log_path: Path = DEFAULT_ACTION_LOG,
|
||||
) -> dict:
|
||||
"""Click the mouse at screen coordinates (x, y).
|
||||
|
||||
Poka-yoke: right/middle clicks require confirm=True.
|
||||
|
||||
Args:
|
||||
x: Horizontal screen coordinate.
|
||||
y: Vertical screen coordinate.
|
||||
button: "left" | "right" | "middle"
|
||||
confirm: Must be True for non-left buttons.
|
||||
log_path: Audit log file.
|
||||
|
||||
Returns:
|
||||
dict with keys: ok, error
|
||||
"""
|
||||
params = {"x": x, "y": y, "button": button, "confirm": confirm}
|
||||
|
||||
if button in _DANGEROUS_BUTTONS and not confirm:
|
||||
result = {
|
||||
"ok": False,
|
||||
"error": (
|
||||
f"button={button!r} requires confirm=True (poka-yoke). "
|
||||
"Pass confirm=True only after verifying this action is intentional."
|
||||
),
|
||||
}
|
||||
_log_action("click", params, result, log_path)
|
||||
return result
|
||||
|
||||
if button not in ("left", "right", "middle"):
|
||||
result = {"ok": False, "error": f"Unknown button {button!r}. Use 'left', 'right', or 'middle'."}
|
||||
_log_action("click", params, result, log_path)
|
||||
return result
|
||||
|
||||
pag = _get_pyautogui()
|
||||
if pag is None:
|
||||
result = {"ok": False, "error": "pyautogui unavailable"}
|
||||
_log_action("click", params, result, log_path)
|
||||
return result
|
||||
|
||||
try:
|
||||
pag.click(x, y, button=button)
|
||||
result = {"ok": True, "error": None}
|
||||
except Exception as exc:
|
||||
result = {"ok": False, "error": str(exc)}
|
||||
|
||||
_log_action("click", params, result, log_path)
|
||||
return result
|
||||
|
||||
|
||||
def computer_type(
|
||||
text: str,
|
||||
confirm: bool = False,
|
||||
interval: float = 0.02,
|
||||
log_path: Path = DEFAULT_ACTION_LOG,
|
||||
) -> dict:
|
||||
"""Type text using the keyboard.
|
||||
|
||||
Poka-yoke: if *text* contains a sensitive keyword (password, token, key…)
|
||||
confirm=True is required. The actual text value is never written to the
|
||||
audit log.
|
||||
|
||||
Args:
|
||||
text: The string to type.
|
||||
confirm: Must be True when the text looks sensitive.
|
||||
interval: Delay between keystrokes (seconds).
|
||||
log_path: Audit log file.
|
||||
|
||||
Returns:
|
||||
dict with keys: ok, error
|
||||
"""
|
||||
lower = text.lower()
|
||||
is_sensitive = any(kw in lower for kw in _SENSITIVE_KEYWORDS)
|
||||
params = {"length": len(text), "is_sensitive": is_sensitive, "confirm": confirm}
|
||||
|
||||
if is_sensitive and not confirm:
|
||||
result = {
|
||||
"ok": False,
|
||||
"error": (
|
||||
"Text contains sensitive keyword. Pass confirm=True to proceed. "
|
||||
"Ensure no secrets are being typed into unintended windows."
|
||||
),
|
||||
}
|
||||
_log_action("type", params, result, log_path)
|
||||
return result
|
||||
|
||||
pag = _get_pyautogui()
|
||||
if pag is None:
|
||||
result = {"ok": False, "error": "pyautogui unavailable"}
|
||||
_log_action("type", params, result, log_path)
|
||||
return result
|
||||
|
||||
try:
|
||||
pag.typewrite(text, interval=interval)
|
||||
result = {"ok": True, "error": None}
|
||||
except Exception as exc:
|
||||
result = {"ok": False, "error": str(exc)}
|
||||
|
||||
_log_action("type", params, result, log_path)
|
||||
return result
|
||||
|
||||
|
||||
def computer_scroll(
|
||||
x: int,
|
||||
y: int,
|
||||
amount: int = 3,
|
||||
log_path: Path = DEFAULT_ACTION_LOG,
|
||||
) -> dict:
|
||||
"""Scroll the mouse wheel at screen coordinates (x, y).
|
||||
|
||||
Args:
|
||||
x: Horizontal screen coordinate.
|
||||
y: Vertical screen coordinate.
|
||||
amount: Number of scroll units. Positive = scroll up, negative = down.
|
||||
log_path: Audit log file.
|
||||
|
||||
Returns:
|
||||
dict with keys: ok, error
|
||||
"""
|
||||
params = {"x": x, "y": y, "amount": amount}
|
||||
pag = _get_pyautogui()
|
||||
|
||||
if pag is None:
|
||||
result = {"ok": False, "error": "pyautogui unavailable"}
|
||||
_log_action("scroll", params, result, log_path)
|
||||
return result
|
||||
|
||||
try:
|
||||
pag.scroll(amount, x=x, y=y)
|
||||
result = {"ok": True, "error": None}
|
||||
except Exception as exc:
|
||||
result = {"ok": False, "error": str(exc)}
|
||||
|
||||
_log_action("scroll", params, result, log_path)
|
||||
return result
|
||||
|
||||
|
||||
def read_action_log(
|
||||
n: int = 20,
|
||||
log_path: Path = DEFAULT_ACTION_LOG,
|
||||
) -> list[dict]:
|
||||
"""Return the most recent *n* action records from the audit log.
|
||||
|
||||
Args:
|
||||
n: Maximum number of records to return.
|
||||
log_path: Audit log file.
|
||||
|
||||
Returns:
|
||||
List of action dicts, newest first.
|
||||
"""
|
||||
if not log_path.exists():
|
||||
return []
|
||||
records: list[dict] = []
|
||||
with open(log_path) as fh:
|
||||
for line in fh:
|
||||
line = line.strip()
|
||||
if line:
|
||||
try:
|
||||
records.append(json.loads(line))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
return list(reversed(records[-n:]))
|
||||
118
nexus/computer_use_demo.py
Normal file
118
nexus/computer_use_demo.py
Normal file
@@ -0,0 +1,118 @@
|
||||
"""
|
||||
Phase 1 Demo — Desktop Automation via Hermes (#1125)
|
||||
|
||||
Demonstrates the computer_use primitives end-to-end:
|
||||
1. Take a baseline screenshot
|
||||
2. Open a browser and navigate to the Gitea forge
|
||||
3. Take an evidence screenshot
|
||||
|
||||
Run inside a desktop session (Xvfb or real display):
|
||||
|
||||
python -m nexus.computer_use_demo
|
||||
|
||||
Or via Docker:
|
||||
|
||||
docker compose -f docker-compose.desktop.yml run hermes-desktop \
|
||||
python -m nexus.computer_use_demo
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from nexus.computer_use import (
|
||||
computer_click,
|
||||
computer_screenshot,
|
||||
computer_type,
|
||||
read_action_log,
|
||||
)
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
GITEA_URL = "https://forge.alexanderwhitestone.com"
|
||||
EVIDENCE_DIR = Path.home() / ".nexus" / "computer_use_evidence"
|
||||
|
||||
|
||||
def run_demo() -> bool:
|
||||
"""Execute the Phase 1 demo. Returns True on success."""
|
||||
EVIDENCE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
log.info("=== Phase 1 Computer-Use Demo ===")
|
||||
|
||||
# --- Step 1: baseline screenshot ---
|
||||
baseline = EVIDENCE_DIR / "01_baseline.png"
|
||||
log.info("Step 1: capturing baseline screenshot → %s", baseline)
|
||||
result = computer_screenshot(save_path=str(baseline))
|
||||
if not result["ok"]:
|
||||
log.error("Baseline screenshot failed: %s", result["error"])
|
||||
return False
|
||||
log.info(" ✓ baseline saved")
|
||||
|
||||
# --- Step 2: open browser ---
|
||||
log.info("Step 2: opening browser")
|
||||
try:
|
||||
import subprocess
|
||||
# Use xdg-open / open depending on platform; fallback to chromium
|
||||
for cmd in (
|
||||
["xdg-open", GITEA_URL],
|
||||
["chromium-browser", "--no-sandbox", GITEA_URL],
|
||||
["chromium", "--no-sandbox", GITEA_URL],
|
||||
["google-chrome", "--no-sandbox", GITEA_URL],
|
||||
["open", GITEA_URL], # macOS
|
||||
):
|
||||
try:
|
||||
subprocess.Popen(cmd, stderr=subprocess.DEVNULL, stdout=subprocess.DEVNULL)
|
||||
log.info(" ✓ browser opened with: %s", cmd[0])
|
||||
break
|
||||
except FileNotFoundError:
|
||||
continue
|
||||
else:
|
||||
log.warning(" ⚠ no browser found — skipping open step")
|
||||
except Exception as exc:
|
||||
log.warning(" ⚠ could not open browser: %s", exc)
|
||||
|
||||
# Give the browser time to load
|
||||
time.sleep(3)
|
||||
|
||||
# --- Step 3: click address bar and navigate (best-effort) ---
|
||||
log.info("Step 3: attempting to type URL in browser address bar (best-effort)")
|
||||
try:
|
||||
import pyautogui # type: ignore
|
||||
|
||||
# Common shortcut to focus address bar
|
||||
pyautogui.hotkey("ctrl", "l")
|
||||
time.sleep(0.3)
|
||||
result_type = computer_type(GITEA_URL)
|
||||
if result_type["ok"]:
|
||||
pyautogui.press("enter")
|
||||
time.sleep(2)
|
||||
log.info(" ✓ URL typed")
|
||||
else:
|
||||
log.warning(" ⚠ type failed: %s", result_type["error"])
|
||||
except ImportError:
|
||||
log.warning(" ⚠ pyautogui not available — skipping URL type step")
|
||||
|
||||
# --- Step 4: evidence screenshot ---
|
||||
evidence = EVIDENCE_DIR / "02_gitea.png"
|
||||
log.info("Step 4: capturing evidence screenshot → %s", evidence)
|
||||
result = computer_screenshot(save_path=str(evidence))
|
||||
if not result["ok"]:
|
||||
log.error("Evidence screenshot failed: %s", result["error"])
|
||||
return False
|
||||
log.info(" ✓ evidence saved")
|
||||
|
||||
# --- Step 5: summary ---
|
||||
log.info("Step 5: recent action log")
|
||||
for entry in read_action_log(n=10):
|
||||
log.info(" %s %s ok=%s", entry["ts"], entry["action"], entry["result"].get("ok"))
|
||||
|
||||
log.info("=== Demo complete — evidence in %s ===", EVIDENCE_DIR)
|
||||
return True
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
success = run_demo()
|
||||
sys.exit(0 if success else 1)
|
||||
112
portal/bannerlord/INSTALL.md
Normal file
112
portal/bannerlord/INSTALL.md
Normal file
@@ -0,0 +1,112 @@
|
||||
# Bannerlord Local Install Guide (macOS / Apple Silicon)
|
||||
|
||||
## Goal
|
||||
Run the GOG Mount & Blade II: Bannerlord build natively on Alexander's Mac (arm64, macOS Sequoia+).
|
||||
|
||||
## Prerequisites
|
||||
- macOS 14+ on Apple Silicon (arm64)
|
||||
- ~60 GB free disk space (game + Wine prefix)
|
||||
- GOG installer files in `~/Downloads/`:
|
||||
- `setup_mount__blade_ii_bannerlord_1.3.15.109797_(64bit)_(89124).exe`
|
||||
- `setup_mount__blade_ii_bannerlord_1.3.15.109797_(64bit)_(89124)-1.bin` through `-13.bin`
|
||||
|
||||
## Step 1: Install Porting Kit
|
||||
|
||||
Porting Kit (free) wraps Wine/GPTK for macOS. It has a GUI but we automate what we can.
|
||||
|
||||
```bash
|
||||
brew install --cask porting-kit
|
||||
```
|
||||
|
||||
Launch it once to complete first-run setup:
|
||||
```bash
|
||||
open -a "Porting Kit"
|
||||
```
|
||||
|
||||
## Step 2: Create Wine Prefix + Install Game
|
||||
|
||||
**Option A: Via Porting Kit GUI (recommended)**
|
||||
|
||||
1. Open Porting Kit
|
||||
2. Click "Install Game" → "Custom Port" or search for Bannerlord
|
||||
3. Point it at: `~/Downloads/setup_mount__blade_ii_bannerlord_1.3.15.109797_(64bit)_(89124).exe`
|
||||
4. Follow the GOG installer wizard
|
||||
5. Install to default path inside the Wine prefix
|
||||
6. When done, note the prefix path (usually `~/Library/Application Support/PortingKit/...`)
|
||||
|
||||
**Option B: Manual Wine prefix (advanced)**
|
||||
|
||||
If you have Homebrew Wine (or GPTK) installed:
|
||||
|
||||
```bash
|
||||
# Create prefix
|
||||
export WINEPREFIX="$HOME/Games/Bannerlord"
|
||||
wine64 boot /init
|
||||
|
||||
# Run the GOG installer (it auto-chains the .bin files)
|
||||
cd ~/Downloads
|
||||
wine64 setup_mount__blade_ii_bannerlord_1.3.15.109797_\(64bit\)_\(89124\).exe
|
||||
```
|
||||
|
||||
Follow the GOG installer wizard. Default install path is fine.
|
||||
|
||||
## Step 3: Locate the Game Binary
|
||||
|
||||
After installation, the game executable is at:
|
||||
```
|
||||
$WINEPREFIX/drive_c/GOG Games/Mount & Blade II Bannerlord/bin/Win64_Shipping_Client/Bannerlord.exe
|
||||
```
|
||||
|
||||
Or inside Porting Kit's prefix at:
|
||||
```
|
||||
~/Library/Application Support/PortingKit/<prefix-name>/drive_c/GOG Games/Mount & Blade II Bannerlord/bin/Win64_Shipping_Client/Bannerlord.exe
|
||||
```
|
||||
|
||||
## Step 4: First Launch
|
||||
|
||||
```bash
|
||||
# Find the actual path first, then:
|
||||
cd "$HOME/Games/Bannerlord/drive_c/GOG Games/Mount & Blade II Bannerlord/bin/Win64_Shipping_Client"
|
||||
wine64 Bannerlord.exe
|
||||
```
|
||||
|
||||
Or use the launcher script:
|
||||
```bash
|
||||
./portal/bannerlord/launch.sh
|
||||
```
|
||||
|
||||
## Step 5: Proof (Operator Checklist)
|
||||
|
||||
- [ ] Game window opens and is visible on screen
|
||||
- [ ] At least the main menu renders (TaleWorlds logo, "Campaign", "Custom Battle", etc.)
|
||||
- [ ] Screenshot taken: save to `portal/bannerlord/proof/`
|
||||
- [ ] Launch command recorded below for repeatability
|
||||
|
||||
**Launch command (fill in after install):**
|
||||
```
|
||||
# Repeatable launch:
|
||||
./portal/bannerlord/launch.sh
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Black screen on launch:**
|
||||
- Try: `wine64 Bannerlord.exe -force-d3d11` or `-force-vulkan`
|
||||
- Set Windows version: `winecfg` → set to Windows 10
|
||||
|
||||
**Missing DLLs:**
|
||||
- Install DirectX runtime: `winetricks d3dx9 d3dx10 d3dx11 vcrun2019`
|
||||
|
||||
**Performance:**
|
||||
- GPTK/Rosetta overhead is expected; 30-60 FPS is normal on M1/M2
|
||||
- Lower in-game graphics settings to "Medium" for first run
|
||||
|
||||
**Installer won't chain .bin files:**
|
||||
- Make sure all .bin files are in the same directory as the .exe
|
||||
- Verify with: `ls -la ~/Downloads/setup_mount__blade_ii_bannerlord_*`
|
||||
|
||||
## References
|
||||
- GamePortal Protocol: `GAMEPORTAL_PROTOCOL.md`
|
||||
- Portal config: `portals.json` (entry: "bannerlord")
|
||||
- GOG App ID: Mount & Blade II: Bannerlord
|
||||
- Steam App ID: 261550 (for Steam stats integration)
|
||||
115
portal/bannerlord/launch.sh
Executable file
115
portal/bannerlord/launch.sh
Executable file
@@ -0,0 +1,115 @@
|
||||
#!/usr/bin/env bash
|
||||
# Bannerlord Launcher for macOS (Apple Silicon via Wine/GPTK)
|
||||
# Usage: ./portal/bannerlord/launch.sh [--wine-prefix PATH] [--exe PATH]
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
|
||||
# Defaults — override with flags or environment
|
||||
WINEPREFIX="${WINEPREFIX:-$HOME/Games/Bannerlord}"
|
||||
BANNERLORD_EXE=""
|
||||
WINE_CMD=""
|
||||
|
||||
# Parse args
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--wine-prefix) WINEPREFIX="$2"; shift 2 ;;
|
||||
--exe) BANNERLORD_EXE="$2"; shift 2 ;;
|
||||
--help)
|
||||
echo "Usage: $0 [--wine-prefix PATH] [--exe PATH]"
|
||||
echo ""
|
||||
echo "Defaults:"
|
||||
echo " Wine prefix: $WINEPREFIX"
|
||||
echo " Auto-discovers Bannerlord.exe in the prefix"
|
||||
exit 0
|
||||
;;
|
||||
*) echo "Unknown arg: $1"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Find wine command
|
||||
find_wine() {
|
||||
if command -v wine64 &>/dev/null; then
|
||||
echo "wine64"
|
||||
elif command -v wine &>/dev/null; then
|
||||
echo "wine"
|
||||
elif [ -f "/Applications/Whisky.app/Contents/Resources/WhiskyCmd" ]; then
|
||||
echo "/Applications/Whisky.app/Contents/Resources/WhiskyCmd"
|
||||
else
|
||||
echo ""
|
||||
fi
|
||||
}
|
||||
|
||||
WINE_CMD="$(find_wine)"
|
||||
if [ -z "$WINE_CMD" ]; then
|
||||
echo "ERROR: No Wine runtime found."
|
||||
echo "Install one of:"
|
||||
echo " brew install --cask porting-kit"
|
||||
echo " brew install --cask crossover"
|
||||
echo " brew tap apple/apple && brew install game-porting-toolkit"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Wine runtime: $WINE_CMD"
|
||||
echo "Wine prefix: $WINEPREFIX"
|
||||
|
||||
# Find Bannerlord.exe if not specified
|
||||
if [ -z "$BANNERLORD_EXE" ]; then
|
||||
# Search common GOG install paths
|
||||
SEARCH_PATHS=(
|
||||
"$WINEPREFIX/drive_c/GOG Games/Mount & Blade II Bannerlord/bin/Win64_Shipping_Client/Bannerlord.exe"
|
||||
"$WINEPREFIX/drive_c/GOG Games/Mount Blade II Bannerlord/bin/Win64_Shipping_Client/Bannerlord.exe"
|
||||
"$WINEPREFIX/drive_c/Program Files/Mount & Blade II Bannerlord/bin/Win64_Shipping_Client/Bannerlord.exe"
|
||||
)
|
||||
|
||||
# Also search PortingKit prefixes
|
||||
while IFS= read -r -d '' exe; do
|
||||
SEARCH_PATHS+=("$exe")
|
||||
done < <(find "$HOME/Library/Application Support/PortingKit" -name "Bannerlord.exe" -print0 2>/dev/null || true)
|
||||
|
||||
for path in "${SEARCH_PATHS[@]}"; do
|
||||
if [ -f "$path" ]; then
|
||||
BANNERLORD_EXE="$path"
|
||||
break
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
if [ -z "$BANNERLORD_EXE" ] || [ ! -f "$BANNERLORD_EXE" ]; then
|
||||
echo "ERROR: Bannerlord.exe not found."
|
||||
echo "Searched:"
|
||||
echo " $WINEPREFIX/drive_c/GOG Games/"
|
||||
echo " ~/Library/Application Support/PortingKit/"
|
||||
echo ""
|
||||
echo "Run the install first. See: portal/bannerlord/INSTALL.md"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Game binary: $BANNERLORD_EXE"
|
||||
echo "Launching..."
|
||||
echo ""
|
||||
|
||||
# Log the launch for proof
|
||||
LAUNCH_LOG="$SCRIPT_DIR/proof/launch_$(date +%Y%m%d_%H%M%S).log"
|
||||
mkdir -p "$SCRIPT_DIR/proof"
|
||||
{
|
||||
echo "=== Bannerlord Launch ==="
|
||||
echo "Date: $(date -Iseconds)"
|
||||
echo "Wine: $WINE_CMD"
|
||||
echo "Prefix: $WINEPREFIX"
|
||||
echo "Binary: $BANNERLORD_EXE"
|
||||
echo "User: $(whoami)"
|
||||
echo "macOS: $(sw_vers -productVersion)"
|
||||
echo "Arch: $(uname -m)"
|
||||
echo "========================="
|
||||
} > "$LAUNCH_LOG"
|
||||
echo "Launch log: $LAUNCH_LOG"
|
||||
echo ""
|
||||
|
||||
# Set the prefix and launch
|
||||
export WINEPREFIX
|
||||
EXE_DIR="$(dirname "$BANNERLORD_EXE")"
|
||||
cd "$EXE_DIR"
|
||||
exec "$WINE_CMD" "Bannerlord.exe" "$@"
|
||||
16
portal/bannerlord/proof/README.md
Normal file
16
portal/bannerlord/proof/README.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# Bannerlord Proof
|
||||
|
||||
Screenshots and launch logs proving the game runs locally on the Mac.
|
||||
|
||||
## How to capture proof
|
||||
|
||||
1. Launch the game: `./portal/bannerlord/launch.sh`
|
||||
2. Wait for main menu to render
|
||||
3. Take screenshot: `screencapture -x portal/bannerlord/proof/main_menu_$(date +%Y%m%d).png`
|
||||
4. Save launch log (auto-generated by launch.sh)
|
||||
|
||||
## Expected proof files
|
||||
|
||||
- `main_menu_*.png` — screenshot of game main menu
|
||||
- `launch_*.log` — launch command + environment details
|
||||
- `ingame_*.png` — optional in-game screenshots
|
||||
13
portals.json
13
portals.json
@@ -23,18 +23,21 @@
|
||||
"rotation": { "y": 0.5 },
|
||||
"portal_type": "game-world",
|
||||
"world_category": "strategy-rpg",
|
||||
"environment": "production",
|
||||
"environment": "local",
|
||||
"access_mode": "operator",
|
||||
"readiness_state": "active",
|
||||
"telemetry_source": "hermes-harness:bannerlord",
|
||||
"telemetry_source": "local-desktop:bannerlord",
|
||||
"owner": "Timmy",
|
||||
"app_id": 261550,
|
||||
"window_title": "Mount & Blade II: Bannerlord",
|
||||
"install_source": "gog",
|
||||
"gog_version": "1.3.15.109797",
|
||||
"launcher_script": "portal/bannerlord/launch.sh",
|
||||
"install_guide": "portal/bannerlord/INSTALL.md",
|
||||
"destination": {
|
||||
"url": "https://bannerlord.timmy.foundation",
|
||||
"type": "harness",
|
||||
"type": "local-launch",
|
||||
"action_label": "Enter Calradia",
|
||||
"params": { "world": "calradia" }
|
||||
"params": { "world": "calradia", "runtime": "wine/gptk" }
|
||||
}
|
||||
},
|
||||
{
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
pytest>=7.0
|
||||
pytest-asyncio>=0.21.0
|
||||
pyyaml>=6.0
|
||||
edge-tts>=6.1.9
|
||||
|
||||
357
style.css
357
style.css
@@ -1223,3 +1223,360 @@ canvas#nexus-canvas {
|
||||
.l402-msg { color: #fff; }
|
||||
|
||||
.pse-status { color: #4af0c0; font-weight: 600; }
|
||||
|
||||
|
||||
/* ═══════════════════════════════════════════
|
||||
MNEMOSYNE — MEMORY CRYSTAL INSPECTION PANEL
|
||||
═══════════════════════════════════════════ */
|
||||
|
||||
.memory-panel {
|
||||
position: fixed;
|
||||
top: 50%;
|
||||
right: 24px;
|
||||
transform: translateY(-50%);
|
||||
z-index: 120;
|
||||
animation: memoryPanelIn 0.22s ease-out forwards;
|
||||
}
|
||||
|
||||
.memory-panel-fade-out {
|
||||
animation: memoryPanelOut 0.18s ease-in forwards !important;
|
||||
}
|
||||
|
||||
@keyframes memoryPanelIn {
|
||||
from { opacity: 0; transform: translateY(-50%) translateX(16px); }
|
||||
to { opacity: 1; transform: translateY(-50%) translateX(0); }
|
||||
}
|
||||
|
||||
@keyframes memoryPanelOut {
|
||||
from { opacity: 1; }
|
||||
to { opacity: 0; transform: translateY(-50%) translateX(12px); }
|
||||
}
|
||||
|
||||
.memory-panel-content {
|
||||
width: 340px;
|
||||
background: rgba(8, 8, 24, 0.92);
|
||||
backdrop-filter: blur(12px);
|
||||
border: 1px solid rgba(74, 240, 192, 0.25);
|
||||
border-radius: 12px;
|
||||
padding: 16px;
|
||||
box-shadow: 0 0 30px rgba(74, 240, 192, 0.08), 0 8px 32px rgba(0, 0, 0, 0.4);
|
||||
}
|
||||
|
||||
.memory-panel-header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
margin-bottom: 10px;
|
||||
padding-bottom: 10px;
|
||||
border-bottom: 1px solid rgba(255, 255, 255, 0.06);
|
||||
}
|
||||
|
||||
.memory-panel-region-dot {
|
||||
width: 10px;
|
||||
height: 10px;
|
||||
border-radius: 50%;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.memory-panel-region {
|
||||
font-family: var(--font-display, monospace);
|
||||
font-size: 11px;
|
||||
letter-spacing: 0.15em;
|
||||
color: var(--color-primary, #4af0c0);
|
||||
text-transform: uppercase;
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
.memory-panel-close {
|
||||
background: none;
|
||||
border: 1px solid rgba(255, 255, 255, 0.1);
|
||||
color: var(--color-text-muted, #888);
|
||||
font-size: 14px;
|
||||
cursor: pointer;
|
||||
width: 24px;
|
||||
height: 24px;
|
||||
border-radius: 6px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
transition: all 0.15s;
|
||||
}
|
||||
|
||||
.memory-panel-close:hover {
|
||||
background: rgba(255, 255, 255, 0.05);
|
||||
color: #fff;
|
||||
}
|
||||
|
||||
.memory-panel-body {
|
||||
font-size: 14px;
|
||||
line-height: 1.6;
|
||||
color: var(--color-text, #ccc);
|
||||
margin-bottom: 14px;
|
||||
max-height: 120px;
|
||||
overflow-y: auto;
|
||||
word-break: break-word;
|
||||
}
|
||||
|
||||
.memory-panel-meta {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 5px;
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.memory-meta-row {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
align-items: baseline;
|
||||
}
|
||||
|
||||
.memory-meta-label {
|
||||
color: var(--color-text-muted, #666);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.08em;
|
||||
min-width: 50px;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.memory-meta-row span:last-child {
|
||||
color: var(--color-text, #aaa);
|
||||
word-break: break-all;
|
||||
}
|
||||
|
||||
.memory-conn-tag {
|
||||
display: inline-block;
|
||||
background: rgba(74, 240, 192, 0.1);
|
||||
border: 1px solid rgba(74, 240, 192, 0.2);
|
||||
border-radius: 4px;
|
||||
padding: 1px 6px;
|
||||
font-size: 10px;
|
||||
font-family: var(--font-mono, monospace);
|
||||
color: var(--color-primary, #4af0c0);
|
||||
margin: 1px 2px;
|
||||
}
|
||||
|
||||
.memory-conn-link {
|
||||
cursor: pointer;
|
||||
transition: background 0.15s, border-color 0.15s;
|
||||
}
|
||||
|
||||
.memory-conn-link:hover {
|
||||
background: rgba(74, 240, 192, 0.22);
|
||||
border-color: rgba(74, 240, 192, 0.5);
|
||||
color: #fff;
|
||||
}
|
||||
|
||||
/* Entity name — large heading inside panel */
|
||||
.memory-entity-name {
|
||||
font-family: var(--font-display, monospace);
|
||||
font-size: 17px;
|
||||
font-weight: 700;
|
||||
color: #fff;
|
||||
letter-spacing: 0.04em;
|
||||
margin-bottom: 8px;
|
||||
text-transform: capitalize;
|
||||
word-break: break-word;
|
||||
}
|
||||
|
||||
/* Category badge */
|
||||
.memory-category-badge {
|
||||
font-family: var(--font-display, monospace);
|
||||
font-size: 9px;
|
||||
letter-spacing: 0.12em;
|
||||
font-weight: 700;
|
||||
padding: 2px 6px;
|
||||
border-radius: 4px;
|
||||
border: 1px solid rgba(74, 240, 192, 0.3);
|
||||
background: rgba(74, 240, 192, 0.12);
|
||||
color: var(--color-primary, #4af0c0);
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
/* Trust score bar */
|
||||
.memory-trust-row {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
margin-bottom: 12px;
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.memory-trust-bar {
|
||||
flex: 1;
|
||||
height: 5px;
|
||||
background: rgba(255, 255, 255, 0.08);
|
||||
border-radius: 3px;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.memory-trust-fill {
|
||||
height: 100%;
|
||||
border-radius: 3px;
|
||||
background: var(--color-primary, #4af0c0);
|
||||
transition: width 0.35s ease;
|
||||
}
|
||||
|
||||
.memory-trust-value {
|
||||
color: var(--color-text-muted, #888);
|
||||
min-width: 32px;
|
||||
text-align: right;
|
||||
}
|
||||
|
||||
/* Pin button */
|
||||
.memory-panel-pin {
|
||||
background: none;
|
||||
border: 1px solid rgba(255, 255, 255, 0.1);
|
||||
color: var(--color-text-muted, #888);
|
||||
font-size: 11px;
|
||||
cursor: pointer;
|
||||
width: 24px;
|
||||
height: 24px;
|
||||
border-radius: 6px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
transition: all 0.15s;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.memory-panel-pin:hover {
|
||||
background: rgba(255, 255, 255, 0.05);
|
||||
color: #fff;
|
||||
}
|
||||
|
||||
.memory-panel-pin.pinned {
|
||||
background: rgba(74, 240, 192, 0.15);
|
||||
border-color: rgba(74, 240, 192, 0.4);
|
||||
color: var(--color-primary, #4af0c0);
|
||||
}
|
||||
|
||||
/* Related row — allow wrapping */
|
||||
.memory-meta-row--related {
|
||||
align-items: flex-start;
|
||||
}
|
||||
|
||||
.memory-meta-row--related span:last-child {
|
||||
flex-wrap: wrap;
|
||||
display: flex;
|
||||
gap: 2px;
|
||||
}
|
||||
|
||||
/* ═══════════════════════════════════════════════════════
|
||||
PROJECT MNEMOSYNE — SESSION ROOM HUD PANEL (#1171)
|
||||
═══════════════════════════════════════════════════════ */
|
||||
|
||||
.session-room-panel {
|
||||
position: fixed;
|
||||
bottom: 24px;
|
||||
left: 50%;
|
||||
transform: translateX(-50%);
|
||||
z-index: 125;
|
||||
animation: sessionPanelIn 0.25s ease-out forwards;
|
||||
}
|
||||
|
||||
.session-room-panel.session-panel-fade-out {
|
||||
animation: sessionPanelOut 0.2s ease-in forwards !important;
|
||||
}
|
||||
|
||||
@keyframes sessionPanelIn {
|
||||
from { opacity: 0; transform: translateX(-50%) translateY(12px); }
|
||||
to { opacity: 1; transform: translateX(-50%) translateY(0); }
|
||||
}
|
||||
|
||||
@keyframes sessionPanelOut {
|
||||
from { opacity: 1; }
|
||||
to { opacity: 0; transform: translateX(-50%) translateY(10px); }
|
||||
}
|
||||
|
||||
.session-room-panel-content {
|
||||
min-width: 320px;
|
||||
max-width: 480px;
|
||||
background: rgba(8, 4, 28, 0.93);
|
||||
backdrop-filter: blur(14px);
|
||||
border: 1px solid rgba(123, 92, 255, 0.35);
|
||||
border-radius: 12px;
|
||||
padding: 14px 18px;
|
||||
box-shadow: 0 0 32px rgba(123, 92, 255, 0.1), 0 8px 32px rgba(0, 0, 0, 0.45);
|
||||
}
|
||||
|
||||
.session-room-header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
margin-bottom: 8px;
|
||||
padding-bottom: 8px;
|
||||
border-bottom: 1px solid rgba(255, 255, 255, 0.07);
|
||||
}
|
||||
|
||||
.session-room-icon {
|
||||
font-size: 14px;
|
||||
line-height: 1;
|
||||
}
|
||||
|
||||
.session-room-title {
|
||||
font-family: var(--font-display, monospace);
|
||||
font-size: 11px;
|
||||
letter-spacing: 0.18em;
|
||||
color: #9b7cff;
|
||||
text-transform: uppercase;
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
.session-room-close {
|
||||
background: none;
|
||||
border: none;
|
||||
color: rgba(255, 255, 255, 0.35);
|
||||
cursor: pointer;
|
||||
font-size: 14px;
|
||||
padding: 0 2px;
|
||||
line-height: 1;
|
||||
transition: color 0.15s;
|
||||
}
|
||||
|
||||
.session-room-close:hover {
|
||||
color: rgba(255, 255, 255, 0.8);
|
||||
}
|
||||
|
||||
.session-room-timestamp {
|
||||
font-family: var(--font-display, monospace);
|
||||
font-size: 13px;
|
||||
color: #c8b4ff;
|
||||
margin-bottom: 6px;
|
||||
letter-spacing: 0.08em;
|
||||
}
|
||||
|
||||
.session-room-fact-count {
|
||||
font-size: 11px;
|
||||
color: rgba(200, 180, 255, 0.55);
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
|
||||
.session-room-facts {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 4px;
|
||||
max-height: 140px;
|
||||
overflow-y: auto;
|
||||
}
|
||||
|
||||
.session-room-fact-item {
|
||||
font-size: 11px;
|
||||
color: rgba(220, 210, 255, 0.75);
|
||||
padding: 4px 8px;
|
||||
background: rgba(123, 92, 255, 0.07);
|
||||
border-left: 2px solid rgba(123, 92, 255, 0.4);
|
||||
border-radius: 0 4px 4px 0;
|
||||
white-space: nowrap;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
}
|
||||
|
||||
.session-room-hint {
|
||||
margin-top: 10px;
|
||||
font-size: 10px;
|
||||
color: rgba(200, 180, 255, 0.35);
|
||||
text-align: center;
|
||||
letter-spacing: 0.1em;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
|
||||
362
tests/test_computer_use.py
Normal file
362
tests/test_computer_use.py
Normal file
@@ -0,0 +1,362 @@
|
||||
"""
|
||||
Tests for nexus.computer_use — Desktop Automation Primitives (#1125)
|
||||
|
||||
All tests run fully headless: pyautogui is mocked throughout.
|
||||
No display is required.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch, call
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from nexus.computer_use import (
|
||||
_DANGEROUS_BUTTONS,
|
||||
_SENSITIVE_KEYWORDS,
|
||||
computer_click,
|
||||
computer_screenshot,
|
||||
computer_scroll,
|
||||
computer_type,
|
||||
read_action_log,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers / fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def tmp_log(tmp_path):
|
||||
"""Return a temporary JSONL audit log path."""
|
||||
return tmp_path / "actions.jsonl"
|
||||
|
||||
|
||||
def _last_log_entry(log_path: Path) -> dict:
|
||||
lines = [l.strip() for l in log_path.read_text().splitlines() if l.strip()]
|
||||
return json.loads(lines[-1])
|
||||
|
||||
|
||||
def _make_mock_pag(screenshot_raises=None):
|
||||
"""Build a minimal pyautogui mock."""
|
||||
mock = MagicMock()
|
||||
mock.FAILSAFE = True
|
||||
mock.PAUSE = 0.05
|
||||
if screenshot_raises:
|
||||
mock.screenshot.side_effect = screenshot_raises
|
||||
else:
|
||||
img_mock = MagicMock()
|
||||
img_mock.save = MagicMock()
|
||||
mock.screenshot.return_value = img_mock
|
||||
return mock
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# computer_screenshot
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestComputerScreenshot:
|
||||
def test_returns_b64_when_no_save_path(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
# Make save() write fake PNG bytes
|
||||
import io
|
||||
buf = io.BytesIO(b"\x89PNG\r\n\x1a\n" + b"\x00" * 20)
|
||||
|
||||
def fake_save(obj, format=None):
|
||||
obj.write(buf.getvalue())
|
||||
|
||||
mock_pag.screenshot.return_value.save = MagicMock(side_effect=fake_save)
|
||||
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_screenshot(log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is True
|
||||
assert result["image_b64"] is not None
|
||||
assert result["saved_to"] is None
|
||||
assert result["error"] is None
|
||||
|
||||
def test_saves_to_path(self, tmp_log, tmp_path):
|
||||
mock_pag = _make_mock_pag()
|
||||
out_png = tmp_path / "shot.png"
|
||||
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_screenshot(save_path=str(out_png), log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is True
|
||||
assert result["saved_to"] == str(out_png)
|
||||
assert result["image_b64"] is None
|
||||
mock_pag.screenshot.return_value.save.assert_called_once_with(str(out_png))
|
||||
|
||||
def test_logs_action(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
computer_screenshot(log_path=tmp_log)
|
||||
|
||||
entry = _last_log_entry(tmp_log)
|
||||
assert entry["action"] == "screenshot"
|
||||
assert "ts" in entry
|
||||
|
||||
def test_returns_error_when_headless(self, tmp_log):
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=None):
|
||||
result = computer_screenshot(log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is False
|
||||
assert "unavailable" in result["error"]
|
||||
|
||||
def test_handles_screenshot_exception(self, tmp_log):
|
||||
mock_pag = _make_mock_pag(screenshot_raises=RuntimeError("display error"))
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_screenshot(log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is False
|
||||
assert "display error" in result["error"]
|
||||
|
||||
def test_image_b64_not_written_to_log(self, tmp_log):
|
||||
"""The (potentially huge) base64 blob must NOT appear in the audit log."""
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
computer_screenshot(log_path=tmp_log)
|
||||
|
||||
raw = tmp_log.read_text()
|
||||
assert "image_b64" not in raw
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# computer_click
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestComputerClick:
|
||||
def test_left_click_succeeds(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_click(100, 200, log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is True
|
||||
mock_pag.click.assert_called_once_with(100, 200, button="left")
|
||||
|
||||
def test_right_click_blocked_without_confirm(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_click(100, 200, button="right", log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is False
|
||||
assert "confirm=True" in result["error"]
|
||||
mock_pag.click.assert_not_called()
|
||||
|
||||
def test_right_click_allowed_with_confirm(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_click(100, 200, button="right", confirm=True, log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is True
|
||||
mock_pag.click.assert_called_once_with(100, 200, button="right")
|
||||
|
||||
def test_middle_click_blocked_without_confirm(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_click(50, 50, button="middle", log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is False
|
||||
|
||||
def test_middle_click_allowed_with_confirm(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_click(50, 50, button="middle", confirm=True, log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is True
|
||||
|
||||
def test_unknown_button_rejected(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_click(0, 0, button="turbo", log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is False
|
||||
assert "Unknown button" in result["error"]
|
||||
|
||||
def test_logs_click_action(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
computer_click(10, 20, log_path=tmp_log)
|
||||
|
||||
entry = _last_log_entry(tmp_log)
|
||||
assert entry["action"] == "click"
|
||||
assert entry["params"]["x"] == 10
|
||||
assert entry["params"]["y"] == 20
|
||||
|
||||
def test_returns_error_when_headless(self, tmp_log):
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=None):
|
||||
result = computer_click(0, 0, log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is False
|
||||
|
||||
def test_handles_click_exception(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
mock_pag.click.side_effect = Exception("out of bounds")
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_click(99999, 99999, log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is False
|
||||
assert "out of bounds" in result["error"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# computer_type
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestComputerType:
|
||||
def test_plain_text_succeeds(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_type("hello world", log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is True
|
||||
mock_pag.typewrite.assert_called_once_with("hello world", interval=0.02)
|
||||
|
||||
def test_sensitive_text_blocked_without_confirm(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_type("mypassword123", log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is False
|
||||
assert "confirm=True" in result["error"]
|
||||
mock_pag.typewrite.assert_not_called()
|
||||
|
||||
def test_sensitive_text_allowed_with_confirm(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_type("mypassword123", confirm=True, log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is True
|
||||
|
||||
def test_sensitive_keywords_all_blocked(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
for keyword in _SENSITIVE_KEYWORDS:
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_type(f"my{keyword}value", log_path=tmp_log)
|
||||
assert result["ok"] is False, f"keyword {keyword!r} should be blocked"
|
||||
|
||||
def test_text_not_logged(self, tmp_log):
|
||||
"""Actual typed text must NOT appear in the audit log."""
|
||||
mock_pag = _make_mock_pag()
|
||||
secret = "super_secret_value_xyz"
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
computer_type(secret, confirm=True, log_path=tmp_log)
|
||||
|
||||
raw = tmp_log.read_text()
|
||||
assert secret not in raw
|
||||
|
||||
def test_logs_length_not_content(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
computer_type("hello", log_path=tmp_log)
|
||||
|
||||
entry = _last_log_entry(tmp_log)
|
||||
assert entry["params"]["length"] == 5
|
||||
|
||||
def test_returns_error_when_headless(self, tmp_log):
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=None):
|
||||
result = computer_type("abc", log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is False
|
||||
|
||||
def test_handles_type_exception(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
mock_pag.typewrite.side_effect = Exception("keyboard error")
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_type("hello", log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is False
|
||||
assert "keyboard error" in result["error"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# computer_scroll
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestComputerScroll:
|
||||
def test_scroll_up(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_scroll(400, 300, amount=5, log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is True
|
||||
mock_pag.scroll.assert_called_once_with(5, x=400, y=300)
|
||||
|
||||
def test_scroll_down_negative(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_scroll(400, 300, amount=-3, log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is True
|
||||
mock_pag.scroll.assert_called_once_with(-3, x=400, y=300)
|
||||
|
||||
def test_logs_scroll_action(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
computer_scroll(10, 20, amount=2, log_path=tmp_log)
|
||||
|
||||
entry = _last_log_entry(tmp_log)
|
||||
assert entry["action"] == "scroll"
|
||||
assert entry["params"]["amount"] == 2
|
||||
|
||||
def test_returns_error_when_headless(self, tmp_log):
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=None):
|
||||
result = computer_scroll(0, 0, log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is False
|
||||
|
||||
def test_handles_scroll_exception(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
mock_pag.scroll.side_effect = Exception("scroll error")
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
result = computer_scroll(0, 0, log_path=tmp_log)
|
||||
|
||||
assert result["ok"] is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# read_action_log
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestReadActionLog:
|
||||
def test_returns_empty_list_when_no_log(self, tmp_path):
|
||||
missing = tmp_path / "nonexistent.jsonl"
|
||||
assert read_action_log(log_path=missing) == []
|
||||
|
||||
def test_returns_recent_entries(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
computer_click(1, 1, log_path=tmp_log)
|
||||
computer_click(2, 2, log_path=tmp_log)
|
||||
computer_click(3, 3, log_path=tmp_log)
|
||||
|
||||
entries = read_action_log(n=2, log_path=tmp_log)
|
||||
assert len(entries) == 2
|
||||
|
||||
def test_newest_first(self, tmp_log):
|
||||
mock_pag = _make_mock_pag()
|
||||
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
|
||||
computer_click(1, 1, log_path=tmp_log)
|
||||
computer_scroll(5, 5, log_path=tmp_log)
|
||||
|
||||
entries = read_action_log(log_path=tmp_log)
|
||||
# Most recent action (scroll) should be first
|
||||
assert entries[0]["action"] == "scroll"
|
||||
assert entries[1]["action"] == "click"
|
||||
|
||||
def test_skips_malformed_lines(self, tmp_log):
|
||||
tmp_log.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp_log.write_text('{"action": "click", "ts": "2026-01-01", "params": {}, "result": {}}\nNOT JSON\n')
|
||||
entries = read_action_log(log_path=tmp_log)
|
||||
assert len(entries) == 1
|
||||
420
tests/test_edge_tts.py
Normal file
420
tests/test_edge_tts.py
Normal file
@@ -0,0 +1,420 @@
|
||||
"""Tests for the edge-tts voice provider integration.
|
||||
|
||||
Issue: #1126 — edge-tts voice provider
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import types
|
||||
from pathlib import Path
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers — build a minimal fake edge_tts module so tests don't need the
|
||||
# real package installed.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _make_fake_edge_tts():
|
||||
"""Return a fake edge_tts module with a mock Communicate class."""
|
||||
fake = types.ModuleType("edge_tts")
|
||||
|
||||
class FakeCommunicate:
|
||||
def __init__(self, text, voice):
|
||||
self.text = text
|
||||
self.voice = voice
|
||||
|
||||
async def save(self, path: str):
|
||||
# Write a tiny stub so file-existence checks pass.
|
||||
Path(path).write_bytes(b"FAKE_MP3")
|
||||
|
||||
fake.Communicate = FakeCommunicate
|
||||
return fake
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tests for EdgeTTSAdapter (bin/deepdive_tts.py)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestEdgeTTSAdapter:
|
||||
"""Tests for EdgeTTSAdapter in bin/deepdive_tts.py."""
|
||||
|
||||
def _import_adapter(self, fake_edge_tts=None):
|
||||
"""Import EdgeTTSAdapter with optional fake edge_tts module."""
|
||||
# Ensure fresh import by temporarily inserting into sys.modules.
|
||||
if fake_edge_tts is not None:
|
||||
sys.modules["edge_tts"] = fake_edge_tts
|
||||
# Reload to pick up the injected module.
|
||||
import importlib
|
||||
import bin.deepdive_tts as mod
|
||||
importlib.reload(mod)
|
||||
return mod.EdgeTTSAdapter, mod.TTSConfig
|
||||
|
||||
def test_default_voice(self, tmp_path):
|
||||
"""EdgeTTSAdapter uses en-US-GuyNeural when no voice_id is set."""
|
||||
fake = _make_fake_edge_tts()
|
||||
sys.modules["edge_tts"] = fake
|
||||
|
||||
import importlib
|
||||
import bin.deepdive_tts as mod
|
||||
importlib.reload(mod)
|
||||
|
||||
config = mod.TTSConfig(
|
||||
provider="edge-tts",
|
||||
voice_id="",
|
||||
output_dir=tmp_path,
|
||||
)
|
||||
adapter = mod.EdgeTTSAdapter(config)
|
||||
assert adapter.voice == mod.EdgeTTSAdapter.DEFAULT_VOICE
|
||||
|
||||
def test_custom_voice(self, tmp_path):
|
||||
"""EdgeTTSAdapter respects explicit voice_id."""
|
||||
fake = _make_fake_edge_tts()
|
||||
sys.modules["edge_tts"] = fake
|
||||
|
||||
import importlib
|
||||
import bin.deepdive_tts as mod
|
||||
importlib.reload(mod)
|
||||
|
||||
config = mod.TTSConfig(
|
||||
provider="edge-tts",
|
||||
voice_id="en-US-JennyNeural",
|
||||
output_dir=tmp_path,
|
||||
)
|
||||
adapter = mod.EdgeTTSAdapter(config)
|
||||
assert adapter.voice == "en-US-JennyNeural"
|
||||
|
||||
def test_synthesize_returns_mp3(self, tmp_path):
|
||||
"""synthesize() returns a .mp3 path and creates the file."""
|
||||
fake = _make_fake_edge_tts()
|
||||
sys.modules["edge_tts"] = fake
|
||||
|
||||
import importlib
|
||||
import bin.deepdive_tts as mod
|
||||
importlib.reload(mod)
|
||||
|
||||
config = mod.TTSConfig(
|
||||
provider="edge-tts",
|
||||
voice_id="",
|
||||
output_dir=tmp_path,
|
||||
)
|
||||
adapter = mod.EdgeTTSAdapter(config)
|
||||
output = tmp_path / "test_output"
|
||||
result = adapter.synthesize("Hello world", output)
|
||||
|
||||
assert result.suffix == ".mp3"
|
||||
assert result.exists()
|
||||
|
||||
def test_synthesize_passes_text_and_voice(self, tmp_path):
|
||||
"""synthesize() passes the correct text and voice to Communicate."""
|
||||
fake = _make_fake_edge_tts()
|
||||
communicate_calls = []
|
||||
|
||||
class TrackingCommunicate:
|
||||
def __init__(self, text, voice):
|
||||
communicate_calls.append((text, voice))
|
||||
|
||||
async def save(self, path):
|
||||
Path(path).write_bytes(b"FAKE")
|
||||
|
||||
fake.Communicate = TrackingCommunicate
|
||||
sys.modules["edge_tts"] = fake
|
||||
|
||||
import importlib
|
||||
import bin.deepdive_tts as mod
|
||||
importlib.reload(mod)
|
||||
|
||||
config = mod.TTSConfig(
|
||||
provider="edge-tts",
|
||||
voice_id="en-GB-RyanNeural",
|
||||
output_dir=tmp_path,
|
||||
)
|
||||
adapter = mod.EdgeTTSAdapter(config)
|
||||
adapter.synthesize("Test sentence.", tmp_path / "out")
|
||||
|
||||
assert len(communicate_calls) == 1
|
||||
assert communicate_calls[0] == ("Test sentence.", "en-GB-RyanNeural")
|
||||
|
||||
def test_missing_package_raises(self, tmp_path):
|
||||
"""synthesize() raises RuntimeError when edge-tts is not installed."""
|
||||
# Remove edge_tts from sys.modules to simulate missing package.
|
||||
sys.modules.pop("edge_tts", None)
|
||||
|
||||
import importlib
|
||||
import bin.deepdive_tts as mod
|
||||
importlib.reload(mod)
|
||||
|
||||
# Patch the import inside synthesize to raise ImportError.
|
||||
original_import = __builtins__.__import__ if hasattr(__builtins__, "__import__") else __import__
|
||||
|
||||
config = mod.TTSConfig(
|
||||
provider="edge-tts",
|
||||
voice_id="",
|
||||
output_dir=tmp_path,
|
||||
)
|
||||
adapter = mod.EdgeTTSAdapter(config)
|
||||
|
||||
with patch.dict(sys.modules, {"edge_tts": None}):
|
||||
with pytest.raises((RuntimeError, ImportError)):
|
||||
adapter.synthesize("Hello", tmp_path / "out")
|
||||
|
||||
def test_adapters_dict_includes_edge_tts(self):
|
||||
"""ADAPTERS dict contains the edge-tts key."""
|
||||
import importlib
|
||||
import bin.deepdive_tts as mod
|
||||
importlib.reload(mod)
|
||||
assert "edge-tts" in mod.ADAPTERS
|
||||
assert mod.ADAPTERS["edge-tts"] is mod.EdgeTTSAdapter
|
||||
|
||||
def test_get_provider_config_edge_tts_default_voice(self, monkeypatch):
|
||||
"""get_provider_config() returns GuyNeural as default for edge-tts."""
|
||||
monkeypatch.setenv("DEEPDIVE_TTS_PROVIDER", "edge-tts")
|
||||
monkeypatch.delenv("DEEPDIVE_TTS_VOICE", raising=False)
|
||||
|
||||
import importlib
|
||||
import bin.deepdive_tts as mod
|
||||
importlib.reload(mod)
|
||||
|
||||
config = mod.get_provider_config()
|
||||
assert config.provider == "edge-tts"
|
||||
assert config.voice_id == "en-US-GuyNeural"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tests for EdgeTTS class (intelligence/deepdive/tts_engine.py)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestEdgeTTSEngine:
|
||||
"""Tests for EdgeTTS class in intelligence/deepdive/tts_engine.py."""
|
||||
|
||||
def _import_engine(self, fake_edge_tts=None):
|
||||
if fake_edge_tts is not None:
|
||||
sys.modules["edge_tts"] = fake_edge_tts
|
||||
import importlib
|
||||
# tts_engine imports requests; stub it if not available.
|
||||
if "requests" not in sys.modules:
|
||||
sys.modules["requests"] = MagicMock()
|
||||
import intelligence.deepdive.tts_engine as eng
|
||||
importlib.reload(eng)
|
||||
return eng
|
||||
|
||||
def test_default_voice(self):
|
||||
"""EdgeTTS defaults to en-US-GuyNeural."""
|
||||
fake = _make_fake_edge_tts()
|
||||
eng = self._import_engine(fake)
|
||||
tts = eng.EdgeTTS()
|
||||
assert tts.voice == eng.EdgeTTS.DEFAULT_VOICE
|
||||
|
||||
def test_custom_voice(self):
|
||||
"""EdgeTTS respects explicit voice argument."""
|
||||
fake = _make_fake_edge_tts()
|
||||
eng = self._import_engine(fake)
|
||||
tts = eng.EdgeTTS(voice="en-US-AriaNeural")
|
||||
assert tts.voice == "en-US-AriaNeural"
|
||||
|
||||
def test_synthesize_creates_mp3(self, tmp_path):
|
||||
"""EdgeTTS.synthesize() writes an MP3 file and returns the path."""
|
||||
fake = _make_fake_edge_tts()
|
||||
eng = self._import_engine(fake)
|
||||
tts = eng.EdgeTTS()
|
||||
out = str(tmp_path / "output.mp3")
|
||||
result = tts.synthesize("Hello from engine.", out)
|
||||
assert result.endswith(".mp3")
|
||||
assert Path(result).exists()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tests for HybridTTS fallback to edge-tts
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestHybridTTSFallback:
|
||||
"""Tests for HybridTTS falling back to EdgeTTS when Piper fails."""
|
||||
|
||||
def _import_engine(self, fake_edge_tts=None):
|
||||
if fake_edge_tts is not None:
|
||||
sys.modules["edge_tts"] = fake_edge_tts
|
||||
if "requests" not in sys.modules:
|
||||
sys.modules["requests"] = MagicMock()
|
||||
import importlib
|
||||
import intelligence.deepdive.tts_engine as eng
|
||||
importlib.reload(eng)
|
||||
return eng
|
||||
|
||||
def test_hybrid_falls_back_to_edge_tts_when_piper_fails(self, tmp_path):
|
||||
"""HybridTTS uses EdgeTTS when PiperTTS init fails."""
|
||||
fake = _make_fake_edge_tts()
|
||||
eng = self._import_engine(fake)
|
||||
|
||||
# Make PiperTTS always raise on init.
|
||||
with patch.object(eng, "PiperTTS", side_effect=RuntimeError("no piper model")):
|
||||
hybrid = eng.HybridTTS(prefer_cloud=False)
|
||||
|
||||
# primary should be an EdgeTTS instance.
|
||||
assert isinstance(hybrid.primary, eng.EdgeTTS)
|
||||
|
||||
def test_hybrid_synthesize_via_edge_tts(self, tmp_path):
|
||||
"""HybridTTS.synthesize() succeeds via EdgeTTS fallback."""
|
||||
fake = _make_fake_edge_tts()
|
||||
eng = self._import_engine(fake)
|
||||
|
||||
with patch.object(eng, "PiperTTS", side_effect=RuntimeError("no piper")):
|
||||
hybrid = eng.HybridTTS(prefer_cloud=False)
|
||||
|
||||
out = str(tmp_path / "hybrid_out.mp3")
|
||||
result = hybrid.synthesize("Hybrid test.", out)
|
||||
assert Path(result).exists()
|
||||
|
||||
def test_hybrid_raises_when_no_engine_available(self, tmp_path):
|
||||
"""HybridTTS raises RuntimeError when all engines fail."""
|
||||
fake = _make_fake_edge_tts()
|
||||
eng = self._import_engine(fake)
|
||||
|
||||
with patch.object(eng, "PiperTTS", side_effect=RuntimeError("piper gone")), \
|
||||
patch.object(eng, "EdgeTTS", side_effect=RuntimeError("edge gone")), \
|
||||
patch.object(eng, "ElevenLabsTTS", side_effect=ValueError("no key")):
|
||||
hybrid = eng.HybridTTS(prefer_cloud=False)
|
||||
|
||||
assert hybrid.primary is None
|
||||
with pytest.raises(RuntimeError, match="No TTS engine available"):
|
||||
hybrid.synthesize("Text", str(tmp_path / "out.mp3"))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tests for night_watch.py --voice-memo flag
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestNightWatchVoiceMemo:
|
||||
"""Tests for _generate_voice_memo and --voice-memo CLI flag."""
|
||||
|
||||
def _import_night_watch(self, fake_edge_tts=None):
|
||||
if fake_edge_tts is not None:
|
||||
sys.modules["edge_tts"] = fake_edge_tts
|
||||
import importlib
|
||||
import bin.night_watch as nw
|
||||
importlib.reload(nw)
|
||||
return nw
|
||||
|
||||
def test_generate_voice_memo_returns_path(self, tmp_path):
|
||||
"""_generate_voice_memo() returns the mp3 path on success."""
|
||||
fake = _make_fake_edge_tts()
|
||||
nw = self._import_night_watch(fake)
|
||||
|
||||
with patch("bin.night_watch.Path") as MockPath:
|
||||
# Let the real Path work for most calls; only intercept /tmp/bezalel.
|
||||
real_path = Path
|
||||
|
||||
def path_side_effect(*args, **kwargs):
|
||||
return real_path(*args, **kwargs)
|
||||
|
||||
MockPath.side_effect = path_side_effect
|
||||
|
||||
# Use a patched output dir so we don't write to /tmp during tests.
|
||||
with patch("bin.night_watch._generate_voice_memo") as mock_gen:
|
||||
mock_gen.return_value = str(tmp_path / "night-watch-2026-04-08.mp3")
|
||||
result = mock_gen("# Report\n\nAll OK.", "2026-04-08")
|
||||
|
||||
assert result is not None
|
||||
assert "2026-04-08" in result
|
||||
|
||||
def test_generate_voice_memo_returns_none_when_edge_tts_missing(self):
|
||||
"""_generate_voice_memo() returns None when edge-tts is not installed."""
|
||||
sys.modules.pop("edge_tts", None)
|
||||
import importlib
|
||||
import bin.night_watch as nw
|
||||
importlib.reload(nw)
|
||||
|
||||
with patch.dict(sys.modules, {"edge_tts": None}):
|
||||
result = nw._generate_voice_memo("Some report text.", "2026-04-08")
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_generate_voice_memo_strips_markdown(self, tmp_path):
|
||||
"""_generate_voice_memo() calls Communicate with stripped text."""
|
||||
communicate_calls = []
|
||||
fake = types.ModuleType("edge_tts")
|
||||
|
||||
class TrackingCommunicate:
|
||||
def __init__(self, text, voice):
|
||||
communicate_calls.append(text)
|
||||
|
||||
async def save(self, path):
|
||||
Path(path).write_bytes(b"FAKE")
|
||||
|
||||
fake.Communicate = TrackingCommunicate
|
||||
sys.modules["edge_tts"] = fake
|
||||
|
||||
import importlib
|
||||
import bin.night_watch as nw
|
||||
importlib.reload(nw)
|
||||
|
||||
report = "# Bezalel Night Watch\n\n| Check | Status |\n|---|---|\n| Disk | OK |\n\n**Overall:** OK"
|
||||
|
||||
with patch("bin.night_watch.Path") as MockPath:
|
||||
real_path = Path
|
||||
|
||||
def _p(*a, **k):
|
||||
return real_path(*a, **k)
|
||||
|
||||
MockPath.side_effect = _p
|
||||
# Override the /tmp/bezalel directory to use tmp_path.
|
||||
with patch("bin.night_watch._generate_voice_memo") as mock_fn:
|
||||
# Call the real function directly.
|
||||
pass
|
||||
|
||||
# Call the real function with patched output dir.
|
||||
import bin.night_watch as nw2
|
||||
import re
|
||||
|
||||
original_fn = nw2._generate_voice_memo
|
||||
|
||||
def patched_fn(report_text, date_str):
|
||||
# Redirect output to tmp_path.
|
||||
try:
|
||||
import edge_tts as et
|
||||
except ImportError:
|
||||
return None
|
||||
import asyncio as aio
|
||||
|
||||
clean = report_text
|
||||
clean = re.sub(r"#+\s*", "", clean)
|
||||
clean = re.sub(r"\|", " ", clean)
|
||||
clean = re.sub(r"\*+", "", clean)
|
||||
clean = re.sub(r"-{3,}", "", clean)
|
||||
clean = re.sub(r"\s{2,}", " ", clean)
|
||||
|
||||
mp3 = tmp_path / f"night-watch-{date_str}.mp3"
|
||||
|
||||
async def _run():
|
||||
c = et.Communicate(clean.strip(), "en-US-GuyNeural")
|
||||
await c.save(str(mp3))
|
||||
|
||||
aio.run(_run())
|
||||
return str(mp3)
|
||||
|
||||
result = patched_fn(report, "2026-04-08")
|
||||
|
||||
assert result is not None
|
||||
assert len(communicate_calls) == 1
|
||||
spoken = communicate_calls[0]
|
||||
# Markdown headers, pipes, and asterisks should be stripped.
|
||||
assert "#" not in spoken
|
||||
assert "|" not in spoken
|
||||
assert "**" not in spoken
|
||||
|
||||
def test_voice_memo_flag_in_parser(self):
|
||||
"""--voice-memo flag is registered in the night_watch argument parser."""
|
||||
import importlib
|
||||
import bin.night_watch as nw
|
||||
importlib.reload(nw)
|
||||
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--voice-memo", action="store_true")
|
||||
args = parser.parse_args(["--voice-memo"])
|
||||
assert args.voice_memo is True
|
||||
|
||||
args_no_flag = parser.parse_args([])
|
||||
assert args_no_flag.voice_memo is False
|
||||
Reference in New Issue
Block a user