Compare commits

...

19 Commits

Author SHA1 Message Date
Timmy Swarm (mimo-v2-pro)
43455e9c83 fix: [PANELS] Add heartbeat / morning briefing panel tied to Hermes state (closes #698) 2026-04-10 20:19:36 -04:00
cc4af009c7 [claude] Mnemosyne session rooms — holographic chambers per session (#1171) (#1178)
Some checks failed
Deploy Nexus / deploy (push) Failing after 3s
Staging Verification Gate / verify-staging (push) Failing after 3s
CI / test (pull_request) Failing after 9s
CI / validate (pull_request) Failing after 13s
Review Approval Gate / verify-review (pull_request) Failing after 3s
2026-04-10 22:45:10 +00:00
089b06b6f8 [claude] Mnemosyne category regions — spatial zones for fact types (#1168) (#1179)
Some checks failed
Deploy Nexus / deploy (push) Failing after 2s
Staging Verification Gate / verify-staging (push) Has been cancelled
2026-04-10 22:45:04 +00:00
8beae5ecc1 [claude] Mnemosyne holographic fact detail panel (#1172) (#1177)
Some checks failed
Deploy Nexus / deploy (push) Failing after 3s
Staging Verification Gate / verify-staging (push) Failing after 6s
2026-04-10 22:29:17 +00:00
e2edfd3318 [claude] Mnemosyne gravity well clustering — related memories attract in 3D (#1175) (#1176)
Some checks failed
Deploy Nexus / deploy (push) Failing after 4s
Staging Verification Gate / verify-staging (push) Failing after 4s
2026-04-10 22:18:46 +00:00
8e18fa5311 Merge pull request 'fix: Missing Source Code Investigation — Classical AI Commits Disappearing' (#1163) from mimo/code/issue-1145 into main
Some checks failed
Deploy Nexus / deploy (push) Failing after 2s
Staging Verification Gate / verify-staging (push) Failing after 3s
Auto-merged by Timmy
2026-04-10 21:00:40 +00:00
1bf2af15a0 Merge pull request 'fix: [DEFERRED] Hermes Trismegistus — New Wizard Proposal' (#1162) from mimo/code/issue-1146 into main
Some checks failed
Deploy Nexus / deploy (push) Has been cancelled
Staging Verification Gate / verify-staging (push) Has been cancelled
Auto-merged by Timmy
2026-04-10 21:00:37 +00:00
4095946749 Merge pull request '[Mnemosyne] Memory crystal click-to-inspect interaction' (#1161) from feat/mnemosyne-crystal-inspect into main
Some checks failed
Deploy Nexus / deploy (push) Failing after 3s
Staging Verification Gate / verify-staging (push) Failing after 3s
Auto-merged by Timmy
2026-04-10 21:00:24 +00:00
Alexander Whitestone
845e2f2ced fix: Missing Source Code Investigation — Classical AI Commits Disappearing (closes #1145)
Some checks failed
CI / test (pull_request) Failing after 8s
CI / validate (pull_request) Failing after 12s
Review Approval Gate / verify-review (pull_request) Failing after 3s
Root cause: duplicate agents wrote GOFAI code to public/nexus/app.js (wrong path)
instead of root app.js. The public/nexus/ files were corrupt duplicates that got
overwritten and eventually deleted, creating the illusion of disappearing code.

The classical AI code is fully present in root app.js — all 13 classes verified:
SymbolicEngine, AgentFSM, KnowledgeGraph, Blackboard, SymbolicPlanner,
HTNPlanner, CaseBasedReasoner, NeuroSymbolicBridge, MetaReasoningLayer,
AdaptiveCalibrator, PSELayer, plus A* search and bitmask fact indexing.

Prevention:
- Added public/nexus/ to .gitignore
- Added canonical file path documentation to CLAUDE.md
- Filed investigation report (INVESTIGATION_ISSUE_1145.md)
2026-04-10 16:25:57 -04:00
Mimo Swarm (mimo-code-1146)
60af11ec2f fix: [DEFERRED] Hermes Trismegistus — New Wizard Proposal (closes #1146)
Some checks failed
CI / test (pull_request) Failing after 9s
CI / validate (pull_request) Failing after 13s
Review Approval Gate / verify-review (pull_request) Failing after 3s
- Added fleet/hermes-trismegistus/README.md with full proposal
- Added fleet/hermes-trismegistus/lane.md with routing definition
- Filled in acceptance criteria from issue #1146
- Status remains DEFERRED — unblock conditions documented

Automated by mimo-v2-pro swarm.
2026-04-10 16:14:13 -04:00
c387708892 feat(mnemosyne): add memory crystal inspection panel styles
Some checks failed
CI / test (pull_request) Failing after 10s
CI / validate (pull_request) Failing after 13s
Review Approval Gate / verify-review (pull_request) Failing after 2s
2026-04-10 19:38:57 +00:00
8694c0f5ad feat(mnemosyne): add memory crystal inspection panel HTML overlay 2026-04-10 19:38:55 +00:00
c3547196d8 feat(mnemosyne): memory crystal click-to-inspect — raycast crystals, show panel, dismiss on empty click 2026-04-10 19:38:54 +00:00
87bfe9b332 feat(mnemosyne): add crystal mesh query + highlight/select API for click-to-inspect 2026-04-10 19:38:53 +00:00
a0964a2fbf auto-merge PR #1159
Some checks failed
Deploy Nexus / deploy (push) Failing after 3s
Staging Verification Gate / verify-staging (push) Failing after 2s
2026-04-10 19:03:41 +00:00
1e7bb2a453 feat: add localStorage persistence to Spatial Memory Schema
Some checks failed
CI / test (pull_request) Failing after 13s
CI / validate (pull_request) Failing after 13s
Review Approval Gate / verify-review (pull_request) Failing after 2s
Auto-saves on placeMemory/removeMemory. Auto-restores on init.
Prunes archive memories on quota exceeded.

Closes #1158
2026-04-10 12:43:00 +00:00
847c4d50d4 [auto-merge] Add desktop automation primitives to Hermes
Some checks failed
Deploy Nexus / deploy (push) Failing after 3s
Staging Verification Gate / verify-staging (push) Failing after 3s
Auto-merged by PR review bot: Add desktop automation primitives to Hermes
2026-04-10 11:48:25 +00:00
Alexander Whitestone
220f20c794 feat: add desktop automation primitives to Hermes (#1125)
Some checks failed
CI / test (pull_request) Failing after 8s
CI / validate (pull_request) Failing after 10s
Review Approval Gate / verify-review (pull_request) Failing after 2s
Implements Phase 1 and Phase 2 tooling from issue #1125:

- nexus/computer_use.py: four Hermes tools with poka-yoke safety
    * computer_screenshot() — capture & base64-encode desktop snapshot
    * computer_click(x, y, button, confirm) — right/middle require confirm=True
    * computer_type(text, confirm) — sensitive keywords blocked without confirm=True;
      text value is never written to audit log
    * computer_scroll(x, y, amount) — scroll wheel
    * read_action_log() — inspect recent JSONL audit entries
    * pyautogui.FAILSAFE=True; all tools degrade gracefully when headless

- nexus/computer_use_demo.py: Phase 1 demo (baseline screenshot →
  open browser → navigate to Gitea forge → evidence screenshot)

- tests/test_computer_use.py: 32 unit tests, fully headless
  (pyautogui mocked), all passing

- docs/computer-use.md: API reference, safety table, phase roadmap,
  pilot recipes

- docker-compose.desktop.yml: sandboxed Xvfb + noVNC container

Fixes #1125

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 05:45:27 -04:00
e85cefd9c0 Merge pull request #1130
Some checks failed
Deploy Nexus / deploy (push) Failing after 2s
Staging Verification Gate / verify-staging (push) Failing after 3s
Merged PR #1130
2026-04-10 09:37:52 +00:00
16 changed files with 3040 additions and 16 deletions

6
.gitignore vendored
View File

@@ -4,3 +4,9 @@ nexus/__pycache__/
tests/__pycache__/
mempalace/__pycache__/
.aider*
# Prevent agents from writing to wrong path (see issue #1145)
public/nexus/
__pycache__/
*.pyc

View File

@@ -42,6 +42,17 @@ Current repo contents are centered on:
Do not tell contributors to run Vite or edit a nonexistent root frontend on current `main`.
If browser/UI work is being restored, it must happen through the migration backlog and land back here.
## Canonical File Paths
**Frontend code lives at repo ROOT, NOT in `public/nexus/`:**
- `app.js` — main Three.js app (GOFAI, 3D world, all frontend logic)
- `index.html` — main HTML shell
- `style.css` — styles
- `server.py` — websocket bridge
- `gofai_worker.js` — web worker for off-thread reasoning
**DO NOT write to `public/nexus/`** — this path is gitignored. Agents historically wrote here by mistake, creating corrupt duplicates. See issue #1145 and `INVESTIGATION_ISSUE_1145.md`.
## Hard Rules
1. One canonical 3D repo only: `Timmy_Foundation/the-nexus`
@@ -50,6 +61,7 @@ If browser/UI work is being restored, it must happen through the migration backl
4. Telemetry and durable truth flow through Hermes harness
5. OpenClaw remains a sidecar, not the governing authority
6. Before claiming visual validation, prove the app being viewed actually comes from current `the-nexus`
7. **NEVER write frontend files to `public/nexus/`** — use repo root paths listed above
## Validation Rule

View File

@@ -0,0 +1,72 @@
# Investigation Report: Missing Source Code — Classical AI Commits Disappearing
**Issue:** #1145
**Date:** 2026-04-10
**Investigator:** mimo-v2-pro swarm worker
## Summary
**The classical AI code is NOT missing. It is fully present in root `app.js` (3302 lines).**
The perception of "disappearing code" was caused by agents writing to the WRONG file path (`public/nexus/app.js` instead of root `app.js`), creating corrupt duplicate files that were repeatedly overwritten and eventually deleted.
## Root Cause
**Explanation #1 confirmed: Duplicate agents on different machines overwriting each other's commits.**
Multiple Google AI Agent instances wrote GOFAI implementations to `public/nexus/app.js` — a path that does not correspond to the canonical app structure. These commits kept overwriting each other:
| Commit | Date | What happened |
|--------|------|---------------|
| `8943cf5` | 2026-03-30 | Symbolic reasoning engine written to `public/nexus/app.js` (+2280 lines) |
| `e2df240` | 2026-03-30 | Phase 3 Neuro-Symbolic Bridge — overwrote to 284 lines of HTML (wrong path) |
| `7f2f23f` | 2026-03-30 | Phase 4 Meta-Reasoning — same destructive overwrite |
| `bf3b98b` | 2026-03-30 | A* Search — same destructive overwrite |
| `e88bcb4` | 2026-03-30 | Bug fix identified `public/nexus/` files as corrupt duplicates, **deleted them** |
## Evidence: Code Is Present on Main
All 13 classical AI classes/functions verified present in root `app.js`:
| Class/Function | Line | Status |
|----------------|------|--------|
| `SymbolicEngine` | 82 | ✅ Present |
| `AgentFSM` | 135 | ✅ Present |
| `KnowledgeGraph` | 160 | ✅ Present |
| `Blackboard` | 181 | ✅ Present |
| `SymbolicPlanner` | 210 | ✅ Present |
| `HTNPlanner` | 295 | ✅ Present |
| `CaseBasedReasoner` | 343 | ✅ Present |
| `NeuroSymbolicBridge` | 392 | ✅ Present |
| `MetaReasoningLayer` | 422 | ✅ Present |
| `AdaptiveCalibrator` | 460 | ✅ Present |
| `PSELayer` | 566 | ✅ Present |
| `setupGOFAI()` | 596 | ✅ Present |
| `updateGOFAI()` | 622 | ✅ Present |
| Bitmask fact indexing | 86 | ✅ Present |
| A* search | 231 | ✅ Present |
These were injected by commit `af7a4c4` (PR #775, merged via `a855d54`) into the correct path.
## What Actually Happened
1. Google AI Agent wrote good GOFAI code to root `app.js` via the correct PR (#775)
2. A second wave of Google AI Agent instances also wrote to `public/nexus/app.js` (wrong path)
3. Those `public/nexus/` files kept getting overwritten by subsequent agent commits
4. Commit `e88bcb4` correctly identified the `public/nexus/` files as corrupt and deleted them
5. Alexander interpreted the git log as "classical AI code keeps disappearing"
6. The code was never actually gone — it just lived in root `app.js` the whole time
## Prevention Strategy
1. **Add `public/nexus/` to `.gitignore`** — prevents agents from accidentally writing to the wrong path again
2. **Add canonical path documentation to CLAUDE.md** — any agent reading this repo will know where frontend code lives
3. **This report** — serves as the audit trail so this confusion doesn't recur
## Acceptance Criteria
- [x] Git history audited for classical AI commits
- [x] Found the commits — they exist, code was written to wrong path
- [x] Root cause identified — duplicate agents writing to `public/nexus/` (wrong path)
- [x] Prevention strategy implemented — `.gitignore` + `CLAUDE.md` path guard
- [x] Report filed with findings (this document)

433
app.js
View File

@@ -4,6 +4,7 @@ import { RenderPass } from 'three/addons/postprocessing/RenderPass.js';
import { UnrealBloomPass } from 'three/addons/postprocessing/UnrealBloomPass.js';
import { SMAAPass } from 'three/addons/postprocessing/SMAAPass.js';
import { SpatialMemory } from './nexus/components/spatial-memory.js';
import { SessionRooms } from './nexus/components/session-rooms.js';
// ═══════════════════════════════════════════
// NEXUS v1.1 — Portal System Update
@@ -705,12 +706,15 @@ async function init() {
createWorkshopTerminal();
createAshStorm();
SpatialMemory.init(scene);
SessionRooms.init(scene, camera, null);
updateLoad(90);
loadSession();
connectHermes();
fetchGiteaData();
setInterval(fetchGiteaData, 30000); // Refresh every 30s
updateHeartbeatBriefing(); // Initial briefing load
setInterval(updateHeartbeatBriefing, 60000); // Refresh briefing every 60s
composer = new EffectComposer(renderer);
composer.addPass(new RenderPass(scene, camera));
@@ -1138,6 +1142,7 @@ async function fetchGiteaData() {
const worldState = JSON.parse(atob(content.content));
updateNexusCommand(worldState);
updateSovereignHealth();
updateHeartbeatBriefing();
}
} catch (e) {
console.error('Failed to fetch Gitea data:', e);
@@ -1233,6 +1238,127 @@ function updateNexusCommand(state) {
terminal.updatePanelText(lines);
}
// ═══ HEARTBEAT BRIEFING PANEL ═════════════════════════════════════════
async function updateHeartbeatBriefing() {
const container = document.getElementById('heartbeat-briefing-content');
const pulseDot = document.querySelector('.hb-pulse-dot');
if (!container) return;
let data = null;
try {
// Derive briefing endpoint from current location or fallback to localhost
const briefingUrl = window.location.protocol === 'file:'
? 'http://localhost:8766/api/briefing'
: `${window.location.protocol}//${window.location.hostname}:8766/api/briefing`;
const res = await fetch(briefingUrl);
if (res.ok) data = await res.json();
} catch (e) {
// Server not reachable — show honest offline state
}
if (!data) {
if (pulseDot) pulseDot.classList.add('offline');
container.innerHTML = '<div class="hb-empty">Briefing offline<br>No connection to Nexus gateway</div>';
return;
}
if (pulseDot) pulseDot.classList.remove('offline');
let html = '';
// ── Core Heartbeat ──
html += '<div class="briefing-section">';
html += '<div class="briefing-section-label">Nexus Core</div>';
const hb = data.core_heartbeat;
if (hb) {
const age = hb.age_secs != null ? hb.age_secs : '?';
const ageLabel = typeof age === 'number'
? (age < 60 ? `${age.toFixed(0)}s ago` : `${(age / 60).toFixed(1)}m ago`)
: age;
const isAlive = typeof age === 'number' && age < 120;
html += `<div class="hb-core-row">
<span class="hb-core-status ${isAlive ? 'alive' : 'dead'}">${isAlive ? '● ALIVE' : '○ STALE'}</span>
<span class="hb-core-meta">cycle ${hb.cycle || '?'} · ${hb.model || 'unknown'} · ${ageLabel}</span>
</div>`;
html += `<div class="hb-core-meta">status: ${hb.status || '?'}</div>`;
} else {
html += '<div class="hb-core-row"><span class="hb-core-status dead">○ NO DATA</span></div>';
}
html += '</div>';
// ── Cron Heartbeats ──
const cron = data.cron_heartbeat;
if (cron && cron.jobs && cron.jobs.length > 0) {
html += '<div class="briefing-section">';
html += '<div class="briefing-section-label">Cron Heartbeats</div>';
html += `<div class="hb-cron-row">
<span class="hb-cron-healthy">● ${cron.healthy_count} healthy</span>
${cron.stale_count > 0 ? `<span class="hb-cron-stale">⚠ ${cron.stale_count} stale</span>` : ''}
</div>`;
for (const job of cron.jobs.slice(0, 5)) {
const cls = job.healthy ? 'healthy' : 'stale';
html += `<div class="hb-cron-job">
<span class="hb-cron-job-name">${esc(job.job)}</span>
<span class="hb-cron-job-status ${cls}">${esc(job.message)}</span>
</div>`;
}
if (cron.jobs.length > 5) {
html += `<div class="hb-core-meta">+${cron.jobs.length - 5} more jobs</div>`;
}
html += '</div>';
}
// ── Morning Report ──
const report = data.morning_report;
if (report) {
html += '<div class="briefing-section">';
html += '<div class="briefing-section-label">Latest Report</div>';
// Aggregate stats
let totalClosed = 0, totalMerged = 0;
if (report.repos) {
for (const r of Object.values(report.repos)) {
totalClosed += r.closed_issues || 0;
totalMerged += r.merged_prs || 0;
}
}
html += `<div class="hb-stats-row">
<div class="hb-stat"><span class="hb-stat-value" style="color:#4af0c0">${totalClosed}</span><span class="hb-stat-label">Closed</span></div>
<div class="hb-stat"><span class="hb-stat-value" style="color:#7b5cff">${totalMerged}</span><span class="hb-stat-label">Merged</span></div>
<div class="hb-stat"><span class="hb-stat-value" style="color:#ffd700">${(report.blockers || []).length}</span><span class="hb-stat-label">Blockers</span></div>
</div>`;
// Highlights (up to 3)
if (report.highlights && report.highlights.length > 0) {
for (const h of report.highlights.slice(0, 3)) {
html += `<div class="hb-core-meta">+ ${esc(h)}</div>`;
}
}
// Blockers
if (report.blockers && report.blockers.length > 0) {
for (const b of report.blockers) {
html += `<div class="hb-blocker">⚠ ${esc(b)}</div>`;
}
}
html += `<div class="hb-timestamp">Report: ${esc(report.generated_at || '?')}</div>`;
html += '</div>';
}
// ── Timestamp ──
html += `<div class="hb-timestamp">Briefing updated: ${new Date().toLocaleTimeString('en-US', { hour12: false })}</div>`;
container.innerHTML = html;
}
function esc(str) {
if (!str) return '';
const d = document.createElement('div');
d.textContent = String(str);
return d.innerHTML;
}
// ═══ AGENT PRESENCE SYSTEM ═══
function createAgentPresences() {
const agentData = [
@@ -1883,7 +2009,7 @@ function setupControls() {
orbitState.lastX = e.clientX;
orbitState.lastY = e.clientY;
// Raycasting for portals
// Raycasting for portals and memory crystals
if (!portalOverlayActive) {
const mouse = new THREE.Vector2(
(e.clientX / window.innerWidth) * 2 - 1,
@@ -1891,12 +2017,43 @@ function setupControls() {
);
const raycaster = new THREE.Raycaster();
raycaster.setFromCamera(mouse, camera);
const intersects = raycaster.intersectObjects(portals.map(p => p.ring));
if (intersects.length > 0) {
const clickedRing = intersects[0].object;
// Priority 1: Portals
const portalHits = raycaster.intersectObjects(portals.map(p => p.ring));
if (portalHits.length > 0) {
const clickedRing = portalHits[0].object;
const portal = portals.find(p => p.ring === clickedRing);
if (portal) activatePortal(portal);
if (portal) { activatePortal(portal); return; }
}
// Priority 2: Memory crystals (Mnemosyne)
const crystalMeshes = SpatialMemory.getCrystalMeshes();
if (crystalMeshes.length > 0) {
const crystalHits = raycaster.intersectObjects(crystalMeshes, false);
if (crystalHits.length > 0) {
const hitMesh = crystalHits[0].object;
const memInfo = SpatialMemory.getMemoryFromMesh(hitMesh);
if (memInfo) {
SpatialMemory.highlightMemory(memInfo.data.id);
showMemoryPanel(memInfo, e.clientX, e.clientY);
return;
}
}
}
// Priority 3: Session rooms (Mnemosyne #1171)
const roomMeshes = SessionRooms.getClickableMeshes();
if (roomMeshes.length > 0) {
const roomHits = raycaster.intersectObjects(roomMeshes, false);
if (roomHits.length > 0) {
const session = SessionRooms.handleRoomClick(roomHits[0].object);
if (session) { _showSessionRoomPanel(session); return; }
}
}
// Clicked empty space — dismiss panel
dismissMemoryPanel();
_dismissSessionRoomPanel();
}
}
});
@@ -2551,6 +2708,226 @@ function focusPortal(portal) {
let lastThoughtTime = 0;
let pulseTimer = 0;
// ═══════════════════════════════════════════
// MNEMOSYNE — MEMORY CRYSTAL INSPECTION
// ═══════════════════════════════════════════
// ── pin state for memory panel ──
let _memPanelPinned = false;
/** Convert a packed hex color integer to "r,g,b" string for CSS rgba(). */
function _hexToRgb(hex) {
return ((hex >> 16) & 255) + ',' + ((hex >> 8) & 255) + ',' + (hex & 255);
}
/**
* Position the panel near the screen click coordinates, keeping it on-screen.
*/
function _positionPanel(panel, clickX, clickY) {
const W = window.innerWidth;
const H = window.innerHeight;
const panelW = 356; // matches CSS width + padding
const panelH = 420; // generous estimate
const margin = 12;
let left = clickX + 24;
if (left + panelW > W - margin) left = clickX - panelW - 24;
left = Math.max(margin, Math.min(W - panelW - margin, left));
let top = clickY - 80;
top = Math.max(margin, Math.min(H - panelH - margin, top));
panel.style.right = 'auto';
panel.style.top = top + 'px';
panel.style.left = left + 'px';
panel.style.transform = 'none';
}
/**
* Navigate to (highlight + show panel for) a memory crystal by id.
*/
function _navigateToMemory(memId) {
SpatialMemory.highlightMemory(memId);
addChatMessage('system', `Focus: ${memId.replace(/_/g, ' ')}`);
const meshes = SpatialMemory.getCrystalMeshes();
for (const mesh of meshes) {
if (mesh.userData && mesh.userData.memId === memId) {
const memInfo = SpatialMemory.getMemoryFromMesh(mesh);
if (memInfo) { showMemoryPanel(memInfo); break; }
}
}
}
/**
* Show the holographic detail panel for a clicked crystal.
* @param {object} memInfo — { data, region } from SpatialMemory.getMemoryFromMesh()
* @param {number} [clickX] — screen X of the click (for panel positioning)
* @param {number} [clickY] — screen Y of the click
*/
function showMemoryPanel(memInfo, clickX, clickY) {
const panel = document.getElementById('memory-panel');
if (!panel) return;
const { data, region } = memInfo;
const regionDef = SpatialMemory.REGIONS[region] || SpatialMemory.REGIONS.working;
const colorHex = regionDef.color.toString(16).padStart(6, '0');
const colorRgb = _hexToRgb(regionDef.color);
// Header — region dot + label
document.getElementById('memory-panel-region').textContent = regionDef.label;
document.getElementById('memory-panel-region-dot').style.background = '#' + colorHex;
// Category badge
const badge = document.getElementById('memory-panel-category-badge');
if (badge) {
badge.textContent = (data.category || region || 'memory').toUpperCase();
badge.style.background = 'rgba(' + colorRgb + ',0.16)';
badge.style.color = '#' + colorHex;
badge.style.borderColor = 'rgba(' + colorRgb + ',0.4)';
}
// Entity name (humanised id)
const entityEl = document.getElementById('memory-panel-entity-name');
if (entityEl) entityEl.textContent = (data.id || '\u2014').replace(/_/g, ' ');
// Fact content
document.getElementById('memory-panel-content').textContent = data.content || '(empty)';
// Trust score bar
const strength = data.strength != null ? data.strength : 0.7;
const trustFill = document.getElementById('memory-panel-trust-fill');
const trustVal = document.getElementById('memory-panel-trust-value');
if (trustFill) {
trustFill.style.width = (strength * 100).toFixed(0) + '%';
trustFill.style.background = '#' + colorHex;
}
if (trustVal) trustVal.textContent = (strength * 100).toFixed(0) + '%';
// Meta rows
document.getElementById('memory-panel-id').textContent = data.id || '\u2014';
document.getElementById('memory-panel-source').textContent = data.source || 'unknown';
document.getElementById('memory-panel-time').textContent = data.timestamp ? new Date(data.timestamp).toLocaleString() : '\u2014';
// Related entities — clickable links
const connEl = document.getElementById('memory-panel-connections');
connEl.innerHTML = '';
if (data.connections && data.connections.length > 0) {
data.connections.forEach(cid => {
const btn = document.createElement('button');
btn.className = 'memory-conn-tag memory-conn-link';
btn.textContent = cid.replace(/_/g, ' ');
btn.title = 'Go to: ' + cid;
btn.addEventListener('click', (ev) => { ev.stopPropagation(); _navigateToMemory(cid); });
connEl.appendChild(btn);
});
} else {
connEl.innerHTML = '<span style="color:var(--color-text-muted)">None</span>';
}
// Pin button — reset on fresh open
_memPanelPinned = false;
const pinBtn = document.getElementById('memory-panel-pin');
if (pinBtn) {
pinBtn.classList.remove('pinned');
pinBtn.title = 'Pin panel';
pinBtn.onclick = () => {
_memPanelPinned = !_memPanelPinned;
pinBtn.classList.toggle('pinned', _memPanelPinned);
pinBtn.title = _memPanelPinned ? 'Unpin panel' : 'Pin panel';
};
}
// Positioning — near click if coords provided
if (clickX != null && clickY != null) {
_positionPanel(panel, clickX, clickY);
}
// Fade in
panel.classList.remove('memory-panel-fade-out');
panel.style.display = 'flex';
}
/**
* Dismiss the panel (respects pin). Called on empty-space click.
*/
function dismissMemoryPanel() {
if (_memPanelPinned) return;
_dismissMemoryPanelForce();
}
/**
* Force-dismiss the panel regardless of pin state. Used by the close button.
*/
function _dismissMemoryPanelForce() {
_memPanelPinned = false;
SpatialMemory.clearHighlight();
const panel = document.getElementById('memory-panel');
if (!panel || panel.style.display === 'none') return;
panel.classList.add('memory-panel-fade-out');
setTimeout(() => {
panel.style.display = 'none';
panel.classList.remove('memory-panel-fade-out');
}, 200);
}
/**
* Show the session room HUD panel when a chamber is entered.
* @param {object} session — { id, timestamp, facts[] }
*/
function _showSessionRoomPanel(session) {
const panel = document.getElementById('session-room-panel');
if (!panel) return;
const dt = session.timestamp ? new Date(session.timestamp) : new Date();
const tsEl = document.getElementById('session-room-timestamp');
if (tsEl) tsEl.textContent = isNaN(dt.getTime()) ? session.id : dt.toLocaleString();
const countEl = document.getElementById('session-room-fact-count');
const facts = session.facts || [];
if (countEl) countEl.textContent = facts.length + (facts.length === 1 ? ' fact' : ' facts') + ' in this chamber';
const listEl = document.getElementById('session-room-facts');
if (listEl) {
listEl.innerHTML = '';
facts.slice(0, 8).forEach(f => {
const item = document.createElement('div');
item.className = 'session-room-fact-item';
item.textContent = f.content || f.id || '(unknown)';
item.title = f.content || '';
listEl.appendChild(item);
});
if (facts.length > 8) {
const more = document.createElement('div');
more.className = 'session-room-fact-item';
more.style.color = 'rgba(200,180,255,0.4)';
more.textContent = '\u2026 ' + (facts.length - 8) + ' more';
listEl.appendChild(more);
}
}
// Close button
const closeBtn = document.getElementById('session-room-close');
if (closeBtn) closeBtn.onclick = () => _dismissSessionRoomPanel();
panel.classList.remove('session-panel-fade-out');
panel.style.display = 'block';
}
/**
* Dismiss the session room panel.
*/
function _dismissSessionRoomPanel() {
const panel = document.getElementById('session-room-panel');
if (!panel || panel.style.display === 'none') return;
panel.classList.add('session-panel-fade-out');
setTimeout(() => {
panel.style.display = 'none';
panel.classList.remove('session-panel-fade-out');
}, 200);
}
function gameLoop() {
requestAnimationFrame(gameLoop);
const delta = Math.min(clock.getDelta(), 0.1);
@@ -2581,6 +2958,9 @@ function gameLoop() {
animateMemoryOrbs(delta);
}
// Project Mnemosyne - Session Rooms (#1171)
SessionRooms.update(delta);
const mode = NAV_MODES[navModeIdx];
const chatActive = document.activeElement === document.getElementById('chat-input');
@@ -3104,9 +3484,52 @@ init().then(() => {
{ id: 'mem_hermes_chat', content: 'First conversation through the Hermes gateway', category: 'social', strength: 0.7, connections: [] },
{ id: 'mem_mnemosyne_start', content: 'Project Mnemosyne began — the living archive awakens', category: 'projects', strength: 0.9, connections: ['mem_nexus_birth', 'mem_spatial_schema'] },
{ id: 'mem_spatial_schema', content: 'Spatial Memory Schema defined — memories gain permanent homes', category: 'engineering', strength: 0.8, connections: ['mem_mnemosyne_start'] },
// MemPalace category zone demos — issue #1168
{ id: 'mem_pref_dark_mode', content: 'User prefers dark mode and monospace fonts', category: 'user_pref', strength: 0.9, connections: [] },
{ id: 'mem_pref_verbose_logs', content: 'User prefers verbose logging during debug sessions', category: 'user_pref', strength: 0.7, connections: [] },
{ id: 'mem_proj_nexus_goal', content: 'The Nexus goal: local-first 3D training ground for Timmy', category: 'project', strength: 0.95, connections: ['mem_proj_mnemosyne'] },
{ id: 'mem_proj_mnemosyne', content: 'Project Mnemosyne: holographic living archive of facts', category: 'project', strength: 0.85, connections: ['mem_proj_nexus_goal'] },
{ id: 'mem_tool_three_js', content: 'Three.js — 3D rendering library used for the Nexus world', category: 'tool', strength: 0.8, connections: [] },
{ id: 'mem_tool_gitea', content: 'Gitea API at forge.alexanderwhitestone.com for issue tracking', category: 'tool', strength: 0.75, connections: [] },
{ id: 'mem_gen_websocket', content: 'WebSocket bridge (server.py) connects Timmy cognition to the browser', category: 'general', strength: 0.7, connections: [] },
{ id: 'mem_gen_hermes', content: 'Hermes harness: telemetry and durable truth pipeline', category: 'general', strength: 0.65, connections: [] },
];
demoMemories.forEach(m => SpatialMemory.placeMemory(m));
// Gravity well clustering — attract related crystals, bake positions (issue #1175)
SpatialMemory.runGravityLayout();
// Project Mnemosyne — seed demo session rooms (#1171)
// Sessions group facts by conversation/work session with a timestamp.
const demoSessions = [
{
id: 'session_2026_03_01',
timestamp: '2026-03-01T10:00:00.000Z',
facts: [
{ id: 'mem_nexus_birth', content: 'The Nexus came online — first render of the 3D world', category: 'knowledge', strength: 0.95 },
{ id: 'mem_mnemosyne_start', content: 'Project Mnemosyne began — the living archive awakens', category: 'projects', strength: 0.9 },
]
},
{
id: 'session_2026_03_15',
timestamp: '2026-03-15T14:30:00.000Z',
facts: [
{ id: 'mem_first_portal', content: 'First portal deployed — connection to external service', category: 'engineering', strength: 0.85 },
{ id: 'mem_hermes_chat', content: 'First conversation through the Hermes gateway', category: 'social', strength: 0.7 },
{ id: 'mem_spatial_schema', content: 'Spatial Memory Schema defined — memories gain homes', category: 'engineering', strength: 0.8 },
]
},
{
id: 'session_2026_04_10',
timestamp: '2026-04-10T09:00:00.000Z',
facts: [
{ id: 'mem_session_rooms', content: 'Session rooms introduced — holographic chambers per session', category: 'projects', strength: 0.88 },
{ id: 'mem_gravity_wells', content: 'Gravity-well clustering bakes crystal positions on load', category: 'engineering', strength: 0.75 },
]
}
];
SessionRooms.updateSessions(demoSessions);
fetchGiteaData();
setInterval(fetchGiteaData, 30000);
runWeeklyAudit();

View File

@@ -0,0 +1,46 @@
version: "3.9"
# Sandboxed desktop environment for Hermes computer-use primitives.
# Provides Xvfb (virtual framebuffer) + noVNC (browser-accessible VNC).
#
# Usage:
# docker compose -f docker-compose.desktop.yml up -d
# # Visit http://localhost:6080 to see the virtual desktop
#
# docker compose -f docker-compose.desktop.yml run hermes-desktop \
# python -m nexus.computer_use_demo
#
# docker compose -f docker-compose.desktop.yml down
services:
hermes-desktop:
image: dorowu/ubuntu-desktop-lxde-vnc:focal
environment:
# Resolution for the virtual display
RESOLUTION: "1280x800"
# VNC password (change in production)
VNC_PASSWORD: "hermes"
# Disable HTTP password for development convenience
HTTP_PASSWORD: ""
ports:
# noVNC web interface
- "6080:80"
# Raw VNC port (optional)
- "5900:5900"
volumes:
# Mount repo into container so scripts are available
- .:/workspace
# Persist nexus runtime data (heartbeats, logs, evidence)
- nexus_data:/root/.nexus
working_dir: /workspace
shm_size: "256mb"
# Install Python deps on startup then keep container alive
command: >
bash -c "
pip install --quiet pyautogui Pillow &&
/startup.sh
"
volumes:
nexus_data:
driver: local

174
docs/computer-use.md Normal file
View File

@@ -0,0 +1,174 @@
# Computer Use — Desktop Automation Primitives for Hermes
Issue: [#1125](https://forge.alexanderwhitestone.com/Timmy_Foundation/the-nexus/issues/1125)
## Overview
`nexus/computer_use.py` adds desktop automation primitives to the Hermes fleet. Agents can take screenshots, click, type, and scroll — enough to drive a browser, validate a UI, or diagnose a failed workflow page visually.
All actions are logged to a JSONL audit trail at `~/.nexus/computer_use_actions.jsonl`.
---
## Quick Start
### Local (requires a real display or Xvfb)
```bash
# Install dependencies
pip install pyautogui Pillow
# Run the Phase 1 demo
python -m nexus.computer_use_demo
```
### Sandboxed (Docker + Xvfb + noVNC)
```bash
docker compose -f docker-compose.desktop.yml up -d
# Visit http://localhost:6080 in your browser to see the virtual desktop
docker compose -f docker-compose.desktop.yml run hermes-desktop \
python -m nexus.computer_use_demo
docker compose -f docker-compose.desktop.yml down
```
---
## API Reference
### `computer_screenshot(save_path=None, log_path=...)`
Capture the current desktop.
| Param | Type | Description |
|-------|------|-------------|
| `save_path` | `str \| None` | Path to save PNG. If `None`, returns base64 string. |
| `log_path` | `Path` | Audit log file. |
**Returns** `dict`:
```json
{
"ok": true,
"image_b64": "<base64 PNG or null>",
"saved_to": "<path or null>",
"error": null
}
```
---
### `computer_click(x, y, button="left", confirm=False, log_path=...)`
Click the mouse at screen coordinates.
| Param | Type | Description |
|-------|------|-------------|
| `x` | `int` | Horizontal coordinate |
| `y` | `int` | Vertical coordinate |
| `button` | `str` | `"left"` \| `"right"` \| `"middle"` |
| `confirm` | `bool` | Required `True` for `right` / `middle` (poka-yoke) |
**Returns** `dict`:
```json
{"ok": true, "error": null}
```
---
### `computer_type(text, confirm=False, interval=0.02, log_path=...)`
Type text using the keyboard.
| Param | Type | Description |
|-------|------|-------------|
| `text` | `str` | Text to type |
| `confirm` | `bool` | Required `True` when text contains a sensitive keyword |
| `interval` | `float` | Delay between keystrokes (seconds) |
**Sensitive keywords** (require `confirm=True`): `password`, `passwd`, `secret`, `token`, `api_key`, `apikey`, `key`, `auth`
> Note: the actual `text` value is never written to the audit log — only its length and whether it was flagged as sensitive.
**Returns** `dict`:
```json
{"ok": true, "error": null}
```
---
### `computer_scroll(x, y, amount=3, log_path=...)`
Scroll the mouse wheel at screen coordinates.
| Param | Type | Description |
|-------|------|-------------|
| `x` | `int` | Horizontal coordinate |
| `y` | `int` | Vertical coordinate |
| `amount` | `int` | Scroll units. Positive = up, negative = down. |
**Returns** `dict`:
```json
{"ok": true, "error": null}
```
---
### `read_action_log(n=20, log_path=...)`
Return the most recent `n` audit log entries, newest first.
```python
from nexus.computer_use import read_action_log
for entry in read_action_log(n=5):
print(entry["ts"], entry["action"], entry["result"]["ok"])
```
---
## Safety Model
| Action | Safety gate |
|--------|-------------|
| `computer_click(button="right")` | Requires `confirm=True` |
| `computer_click(button="middle")` | Requires `confirm=True` |
| `computer_type` with sensitive text | Requires `confirm=True` |
| Mouse to top-left corner | pyautogui FAILSAFE — aborts immediately |
| All actions | Written to JSONL audit log with timestamp |
| Headless environment | All tools degrade gracefully — return `ok=False` with error message |
---
## Phase Roadmap
### Phase 1 — Environment & Primitives ✅
- Sandboxed desktop via Xvfb + noVNC (`docker-compose.desktop.yml`)
- `computer_screenshot`, `computer_click`, `computer_type`, `computer_scroll`
- Poka-yoke safety checks on all destructive actions
- JSONL audit log for all actions
- Demo: baseline screenshot → open browser → navigate to Gitea → evidence screenshot
- 32 unit tests, fully headless (pyautogui mocked)
### Phase 2 — Tool Integration (planned)
- Register tools in the Hermes tool registry
- LLM-based planner loop using screenshots as context
- Destructive action confirmation UI
### Phase 3 — Use-Case Pilots (planned)
- Pilot 1: Automated visual regression test for fleet dashboard
- Pilot 2: Screenshot-based diagnosis of failed CI workflow page
---
## File Locations
| File | Purpose |
|------|---------|
| `nexus/computer_use.py` | Core tool primitives |
| `nexus/computer_use_demo.py` | Phase 1 end-to-end demo |
| `tests/test_computer_use.py` | 32 unit tests |
| `docker-compose.desktop.yml` | Sandboxed desktop container |
| `~/.nexus/computer_use_actions.jsonl` | Runtime audit log |
| `~/.nexus/computer_use_evidence/` | Screenshot evidence (demo output) |

View File

@@ -0,0 +1,72 @@
# Hermes Trismegistus — Wizard Proposal
> **Status:** 🟡 DEFERRED
> **Issue:** #1146
> **Created:** 2026-04-08
> **Author:** Alexander (KT Notes)
> **Mimo Worker:** mimo-code-1146-1775851759
---
## Identity
| Field | Value |
|-------|-------|
| **Name** | Hermes Trismegistus |
| **Nature** | Claude-native wizard. She knows she runs on Claude. She's "the daughter of Claude" and leans into that heritage. |
| **Purpose** | Dedicated reasoning and architecture wizard. Only handles tasks where Claude's reasoning capability genuinely adds value — planning, novel problem-solving, complex architecture decisions. |
| **Not** | A replacement for Timmy. Not competing for identity. Not doing monkey work. |
## Design Constraints
- **Free tier only from day one.** Alexander is not paying Anthropic beyond current subscription.
- **Degrades gracefully.** Full capability when free tier is generous, reduced scope when constrained.
- **Not locked to Claude.** If better free-tier providers emerge, she can route to them.
- **Multi-provider capable.** Welcome to become multifaceted if team finds better options.
## Hardware
- One of Alexander's shed laptops — minimum 4GB RAM, Ubuntu
- Dedicated machine, not shared with Timmy's Mac
- Runs in the Hermes harness
- Needs power at house first
## Constitutional Foundation
- The KT conversation and documents serve as her founding constitution
- Team (especially Timmy) has final say on whether she gets built
- Must justify her existence through useful work, same as every wizard
## Trigger to Unblock
All of the following must be true before implementation begins:
- [ ] Deadman switch wired and proven
- [ ] Config stable across fleet
- [ ] Fleet proven reliable for 1+ week
- [ ] Alexander provides a state-of-the-system KT to Claude for instantiation
## Acceptance Criteria
- [ ] Dedicated KT document written for Hermes instantiation
- [ ] Hardware provisioned (shed laptop with power)
- [ ] Hermes harness configured for Claude free tier
- [ ] Lazerus registry entry with health endpoints
- [ ] Fleet routing entry with role and routing verdict
- [ ] SOUL.md inscription drafted and reviewed by Timmy
- [ ] Smoke test: Hermes responds to a basic reasoning task
- [ ] Integration test: Hermes participates in a multi-wizard task alongside Timmy
## Proposed Lane
**Primary role:** Architecture reasoning
**Routing verdict:** ROUTE TO: complex architectural decisions, novel problem-solving, planning tasks that benefit from Claude's reasoning depth. Do NOT route to: code generation (use Timmy/Carnice), issue triage (use Fenrir), or operational tasks (use Bezalel).
## Dependencies
| Dependency | Status | Notes |
|------------|--------|-------|
| Deadman switch | 🔴 Not done | Must be proven before unblocking |
| Fleet stability | 🟡 In progress | 1+ week uptime needed |
| Shed laptop power | 🔴 Not done | Alexander needs to wire power |
| KT document | 🔴 Not drafted | Alexander provides to Claude at unblock time |

View File

@@ -0,0 +1,43 @@
# Hermes Trismegistus — Lane Definition
> **Status:** DEFERRED — do not instantiate until unblock conditions met
> **See:** fleet/hermes-trismegistus/README.md for full proposal
---
## Role
Dedicated reasoning and architecture wizard. Claude-native.
## Routing
Route to Hermes Trismegistus when:
- Task requires deep architectural reasoning
- Novel problem-solving that benefits from Claude's reasoning depth
- Planning and design decisions for the fleet
- Complex multi-step analysis that goes beyond code generation
Do NOT route to Hermes for:
- Code generation (use Timmy, Carnice, or Kimi)
- Issue triage (use Fenrir)
- Operational/DevOps tasks (use Bezalel)
- Anything that can be done with a cheaper model
## Provider
- **Primary:** anthropic/claude (free tier)
- **Fallback:** openrouter/free (Claude-class models)
- **Degraded:** ollama/gemma4:12b (when free tier exhausted)
## Hardware
- Shed laptop, Ubuntu, minimum 4GB RAM
- Dedicated machine, not shared
## Unblock Checklist
- [ ] Deadman switch operational
- [ ] Fleet config stable for 1+ week
- [ ] Shed laptop powered and networked
- [ ] KT document drafted by Alexander
- [ ] Timmy approves instantiation

View File

@@ -96,6 +96,12 @@
<div class="panel-header">SOVEREIGN HEALTH</div>
<div id="sovereign-health-content" class="panel-content"></div>
</div>
<div class="hud-panel hud-panel-briefing" id="heartbeat-briefing-log">
<div class="panel-header">
<span class="hb-pulse-dot"></span> HEARTBEAT BRIEFING
</div>
<div id="heartbeat-briefing-content" class="panel-content"></div>
</div>
<div class="hud-panel" id="calibrator-log">
<div class="panel-header">ADAPTIVE CALIBRATOR</div>
<div id="calibrator-log-content" class="panel-content"></div>
@@ -207,6 +213,50 @@
</div>
</div>
<!-- Memory Crystal Inspection Panel (Mnemosyne) -->
<div id="memory-panel" class="memory-panel" style="display:none;">
<div class="memory-panel-content">
<div class="memory-panel-header">
<span class="memory-category-badge" id="memory-panel-category-badge">MEM</span>
<div class="memory-panel-region-dot" id="memory-panel-region-dot"></div>
<div class="memory-panel-region" id="memory-panel-region">MEMORY</div>
<button id="memory-panel-pin" class="memory-panel-pin" title="Pin panel">&#x1F4CC;</button>
<button id="memory-panel-close" class="memory-panel-close" onclick="_dismissMemoryPanelForce()">\u2715</button>
</div>
<div class="memory-entity-name" id="memory-panel-entity-name">\u2014</div>
<div class="memory-panel-body" id="memory-panel-content">(empty)</div>
<div class="memory-trust-row">
<span class="memory-meta-label">Trust</span>
<div class="memory-trust-bar">
<div class="memory-trust-fill" id="memory-panel-trust-fill"></div>
</div>
<span class="memory-trust-value" id="memory-panel-trust-value"></span>
</div>
<div class="memory-panel-meta">
<div class="memory-meta-row"><span class="memory-meta-label">ID</span><span id="memory-panel-id">\u2014</span></div>
<div class="memory-meta-row"><span class="memory-meta-label">Source</span><span id="memory-panel-source">\u2014</span></div>
<div class="memory-meta-row"><span class="memory-meta-label">Time</span><span id="memory-panel-time">\u2014</span></div>
<div class="memory-meta-row memory-meta-row--related"><span class="memory-meta-label">Related</span><span id="memory-panel-connections">\u2014</span></div>
</div>
</div>
</div>
<!-- Session Room HUD Panel (Mnemosyne #1171) -->
<div id="session-room-panel" class="session-room-panel" style="display:none;">
<div class="session-room-panel-content">
<div class="session-room-header">
<span class="session-room-icon">&#x25A1;</span>
<div class="session-room-title">SESSION CHAMBER</div>
<button class="session-room-close" id="session-room-close" title="Close">&#x2715;</button>
</div>
<div class="session-room-timestamp" id="session-room-timestamp">&mdash;</div>
<div class="session-room-fact-count" id="session-room-fact-count">0 facts</div>
<div class="session-room-facts" id="session-room-facts"></div>
<div class="session-room-hint">Flying into chamber&hellip;</div>
</div>
</div>
<!-- Portal Atlas Overlay -->
<div id="atlas-overlay" class="atlas-overlay" style="display:none;">
<div class="atlas-content">

View File

@@ -0,0 +1,413 @@
// ═══════════════════════════════════════════════════════
// PROJECT MNEMOSYNE — SESSION ROOMS (Issue #1171)
// ═══════════════════════════════════════════════════════
//
// Groups memories by session into holographic chambers.
// Each session becomes a wireframe cube floating in space.
// Rooms are arranged chronologically along a spiral.
// Click a room to fly inside; distant rooms LOD to a point.
//
// Usage from app.js:
// SessionRooms.init(scene, camera, controls);
// SessionRooms.updateSessions(sessions); // [{id, timestamp, facts[]}]
// SessionRooms.update(delta); // call each frame
// SessionRooms.getClickableMeshes(); // for raycasting
// SessionRooms.handleRoomClick(mesh); // trigger fly-in
// ═══════════════════════════════════════════════════════
const SessionRooms = (() => {
// ─── CONSTANTS ───────────────────────────────────────
const MAX_ROOMS = 20;
const ROOM_SIZE = 9; // wireframe cube edge length
const ROOM_HALF = ROOM_SIZE / 2;
const LOD_THRESHOLD = 55; // distance: full → point
const LOD_HYSTERESIS = 5; // buffer to avoid flicker
const SPIRAL_BASE_R = 20; // spiral inner radius
const SPIRAL_R_STEP = 5; // radius growth per room
const SPIRAL_ANGLE_INC = 2.399; // golden angle (radians)
const SPIRAL_Y_STEP = 1.5; // vertical rise per room
const FLY_DURATION = 1.5; // seconds for fly-in tween
const FLY_TARGET_DEPTH = ROOM_HALF - 1.5; // how deep inside to stop
const ROOM_COLOR = 0x7b5cff; // violet — mnemosyne accent
const POINT_COLOR = 0x9b7cff;
const LABEL_COLOR = '#c8b4ff';
const STORAGE_KEY = 'mnemosyne_sessions_v1';
// ─── STATE ────────────────────────────────────────────
let _scene = null;
let _camera = null;
let _controls = null;
let _rooms = []; // array of room objects
let _sessionIndex = {}; // id → room object
// Fly-in tween state
let _flyActive = false;
let _flyElapsed = 0;
let _flyFrom = null;
let _flyTo = null;
let _flyLookFrom = null;
let _flyLookTo = null;
let _flyActiveRoom = null;
// ─── SPIRAL POSITION ──────────────────────────────────
function _spiralPos(index) {
const angle = index * SPIRAL_ANGLE_INC;
const r = SPIRAL_BASE_R + index * SPIRAL_R_STEP;
const y = index * SPIRAL_Y_STEP;
return new THREE.Vector3(
Math.cos(angle) * r,
y,
Math.sin(angle) * r
);
}
// ─── CREATE ROOM ──────────────────────────────────────
function _createRoom(session, index) {
const pos = _spiralPos(index);
const group = new THREE.Group();
group.position.copy(pos);
// Wireframe cube
const boxGeo = new THREE.BoxGeometry(ROOM_SIZE, ROOM_SIZE, ROOM_SIZE);
const edgesGeo = new THREE.EdgesGeometry(boxGeo);
const edgesMat = new THREE.LineBasicMaterial({
color: ROOM_COLOR,
transparent: true,
opacity: 0.55
});
const wireframe = new THREE.LineSegments(edgesGeo, edgesMat);
wireframe.userData = { type: 'session_room_wireframe', sessionId: session.id };
group.add(wireframe);
// Collision mesh (invisible, for raycasting)
const hitGeo = new THREE.BoxGeometry(ROOM_SIZE, ROOM_SIZE, ROOM_SIZE);
const hitMat = new THREE.MeshBasicMaterial({
visible: false,
transparent: true,
opacity: 0,
side: THREE.FrontSide
});
const hitMesh = new THREE.Mesh(hitGeo, hitMat);
hitMesh.userData = { type: 'session_room', sessionId: session.id, roomIndex: index };
group.add(hitMesh);
// LOD point (small sphere shown at distance)
const pointGeo = new THREE.SphereGeometry(0.5, 6, 4);
const pointMat = new THREE.MeshBasicMaterial({
color: POINT_COLOR,
transparent: true,
opacity: 0.7
});
const pointMesh = new THREE.Mesh(pointGeo, pointMat);
pointMesh.userData = { type: 'session_room_point', sessionId: session.id };
pointMesh.visible = false; // starts hidden; shown only at LOD distance
group.add(pointMesh);
// Timestamp billboard sprite
const sprite = _makeTimestampSprite(session.timestamp, session.facts.length);
sprite.position.set(0, ROOM_HALF + 1.2, 0);
group.add(sprite);
// Inner ambient glow
const glow = new THREE.PointLight(ROOM_COLOR, 0.4, ROOM_SIZE * 1.2);
group.add(glow);
_scene.add(group);
const room = {
session,
group,
wireframe,
hitMesh,
pointMesh,
sprite,
glow,
pos: pos.clone(),
index,
lodActive: false,
pulsePhase: Math.random() * Math.PI * 2
};
_rooms.push(room);
_sessionIndex[session.id] = room;
console.info('[SessionRooms] Created room for session', session.id, 'at index', index);
return room;
}
// ─── TIMESTAMP SPRITE ────────────────────────────────
function _makeTimestampSprite(isoTimestamp, factCount) {
const canvas = document.createElement('canvas');
canvas.width = 320;
canvas.height = 72;
const ctx = canvas.getContext('2d');
// Background pill
ctx.clearRect(0, 0, 320, 72);
ctx.fillStyle = 'rgba(20, 10, 40, 0.82)';
_roundRect(ctx, 4, 4, 312, 64, 14);
ctx.fill();
// Border
ctx.strokeStyle = 'rgba(123, 92, 255, 0.6)';
ctx.lineWidth = 1.5;
_roundRect(ctx, 4, 4, 312, 64, 14);
ctx.stroke();
// Timestamp text
const dt = isoTimestamp ? new Date(isoTimestamp) : new Date();
const label = _formatDate(dt);
ctx.fillStyle = LABEL_COLOR;
ctx.font = 'bold 15px monospace';
ctx.textAlign = 'center';
ctx.fillText(label, 160, 30);
// Fact count
ctx.fillStyle = 'rgba(200, 180, 255, 0.65)';
ctx.font = '12px monospace';
ctx.fillText(factCount + (factCount === 1 ? ' fact' : ' facts'), 160, 52);
const tex = new THREE.CanvasTexture(canvas);
const mat = new THREE.SpriteMaterial({ map: tex, transparent: true, opacity: 0.88 });
const sprite = new THREE.Sprite(mat);
sprite.scale.set(5, 1.1, 1);
sprite.userData = { type: 'session_room_label' };
return sprite;
}
// ─── HELPERS ──────────────────────────────────────────
function _roundRect(ctx, x, y, w, h, r) {
ctx.beginPath();
ctx.moveTo(x + r, y);
ctx.lineTo(x + w - r, y);
ctx.quadraticCurveTo(x + w, y, x + w, y + r);
ctx.lineTo(x + w, y + h - r);
ctx.quadraticCurveTo(x + w, y + h, x + w - r, y + h);
ctx.lineTo(x + r, y + h);
ctx.quadraticCurveTo(x, y + h, x, y + h - r);
ctx.lineTo(x, y + r);
ctx.quadraticCurveTo(x, y, x + r, y);
ctx.closePath();
}
function _formatDate(dt) {
if (isNaN(dt.getTime())) return 'Unknown session';
const pad = n => String(n).padStart(2, '0');
return `${dt.getFullYear()}-${pad(dt.getMonth() + 1)}-${pad(dt.getDate())} ${pad(dt.getHours())}:${pad(dt.getMinutes())}`;
}
// ─── DISPOSE ROOM ────────────────────────────────────
function _disposeRoom(room) {
room.wireframe.geometry.dispose();
room.wireframe.material.dispose();
room.hitMesh.geometry.dispose();
room.hitMesh.material.dispose();
room.pointMesh.geometry.dispose();
room.pointMesh.material.dispose();
if (room.sprite.material.map) room.sprite.material.map.dispose();
room.sprite.material.dispose();
if (room.group.parent) room.group.parent.remove(room.group);
delete _sessionIndex[room.session.id];
}
// ─── PUBLIC: UPDATE SESSIONS ─────────────────────────
// sessions: [{id, timestamp, facts:[{id,content,category,strength,...}]}]
// Sorted chronologically oldest→newest; max MAX_ROOMS shown.
function updateSessions(sessions) {
if (!_scene) return;
const sorted = [...sessions]
.sort((a, b) => new Date(a.timestamp) - new Date(b.timestamp))
.slice(-MAX_ROOMS); // keep most recent MAX_ROOMS
// Remove rooms no longer present
const incoming = new Set(sorted.map(s => s.id));
for (let i = _rooms.length - 1; i >= 0; i--) {
const room = _rooms[i];
if (!incoming.has(room.session.id)) {
_disposeRoom(room);
_rooms.splice(i, 1);
}
}
// Add / update
sorted.forEach((session, idx) => {
if (_sessionIndex[session.id]) {
// Update position if index changed
const room = _sessionIndex[session.id];
if (room.index !== idx) {
room.index = idx;
const newPos = _spiralPos(idx);
room.group.position.copy(newPos);
room.pos.copy(newPos);
}
} else {
_createRoom(session, idx);
}
});
saveToStorage(sorted);
console.info('[SessionRooms] Updated:', _rooms.length, 'session rooms');
}
// ─── PUBLIC: INIT ─────────────────────────────────────
function init(scene, camera, controls) {
_scene = scene;
_camera = camera;
_controls = controls;
console.info('[SessionRooms] Initialized');
// Restore persisted sessions
const saved = loadFromStorage();
if (saved && saved.length > 0) {
updateSessions(saved);
}
}
// ─── PUBLIC: UPDATE (per-frame) ───────────────────────
function update(delta) {
if (!_scene || !_camera) return;
const camPos = _camera.position;
_rooms.forEach(room => {
const dist = camPos.distanceTo(room.pos);
// LOD toggle
const threshold = room.lodActive
? LOD_THRESHOLD + LOD_HYSTERESIS // must come closer to exit LOD
: LOD_THRESHOLD;
if (dist > threshold && !room.lodActive) {
room.lodActive = true;
room.wireframe.visible = false;
room.sprite.visible = false;
room.pointMesh.visible = true;
} else if (dist <= threshold && room.lodActive) {
room.lodActive = false;
room.wireframe.visible = true;
room.sprite.visible = true;
room.pointMesh.visible = false;
}
// Pulse wireframe opacity
room.pulsePhase += delta * 0.6;
if (!room.lodActive) {
room.wireframe.material.opacity = 0.3 + Math.sin(room.pulsePhase) * 0.2;
room.glow.intensity = 0.3 + Math.sin(room.pulsePhase * 1.4) * 0.15;
}
// Slowly rotate each room
room.group.rotation.y += delta * 0.04;
});
// Fly-in tween
if (_flyActive) {
_flyElapsed += delta;
const t = Math.min(_flyElapsed / FLY_DURATION, 1);
const ease = _easeInOut(t);
_camera.position.lerpVectors(_flyFrom, _flyTo, ease);
// Interpolate lookAt
const lookNow = new THREE.Vector3().lerpVectors(_flyLookFrom, _flyLookTo, ease);
_camera.lookAt(lookNow);
if (_controls && _controls.target) _controls.target.copy(lookNow);
if (t >= 1) {
_flyActive = false;
if (_controls && typeof _controls.update === 'function') _controls.update();
console.info('[SessionRooms] Fly-in complete for session', _flyActiveRoom && _flyActiveRoom.session.id);
_flyActiveRoom = null;
}
}
}
// ─── EASING ───────────────────────────────────────────
function _easeInOut(t) {
return t < 0.5 ? 2 * t * t : -1 + (4 - 2 * t) * t;
}
// ─── PUBLIC: GET CLICKABLE MESHES ─────────────────────
function getClickableMeshes() {
return _rooms.map(r => r.hitMesh);
}
// ─── PUBLIC: HANDLE ROOM CLICK ────────────────────────
function handleRoomClick(mesh) {
const { sessionId } = mesh.userData;
const room = _sessionIndex[sessionId];
if (!room || !_camera) return null;
// Fly into the room from the front face
_flyActive = true;
_flyElapsed = 0;
_flyActiveRoom = room;
_flyFrom = _camera.position.clone();
// Target: step inside the room toward its center
const dir = room.pos.clone().sub(_camera.position).normalize();
_flyTo = room.pos.clone().add(dir.multiplyScalar(FLY_TARGET_DEPTH));
_flyLookFrom = _controls && _controls.target
? _controls.target.clone()
: _camera.position.clone().add(_camera.getWorldDirection(new THREE.Vector3()));
_flyLookTo = room.pos.clone();
console.info('[SessionRooms] Flying into session room:', sessionId);
return room.session;
}
// ─── PERSISTENCE ──────────────────────────────────────
function saveToStorage(sessions) {
if (typeof localStorage === 'undefined') return;
try {
localStorage.setItem(STORAGE_KEY, JSON.stringify({ v: 1, sessions }));
} catch (e) {
console.warn('[SessionRooms] Failed to save to localStorage:', e);
}
}
function loadFromStorage() {
if (typeof localStorage === 'undefined') return null;
try {
const raw = localStorage.getItem(STORAGE_KEY);
if (!raw) return null;
const parsed = JSON.parse(raw);
if (!parsed || parsed.v !== 1 || !Array.isArray(parsed.sessions)) return null;
console.info('[SessionRooms] Restored', parsed.sessions.length, 'sessions from localStorage');
return parsed.sessions;
} catch (e) {
console.warn('[SessionRooms] Failed to load from localStorage:', e);
return null;
}
}
function clearStorage() {
if (typeof localStorage !== 'undefined') {
localStorage.removeItem(STORAGE_KEY);
console.info('[SessionRooms] Cleared localStorage');
}
}
// ─── PUBLIC API ───────────────────────────────────────
return {
init,
updateSessions,
update,
getClickableMeshes,
handleRoomClick,
clearStorage,
// For external inspection
getRooms: () => _rooms,
getSession: (id) => _sessionIndex[id] || null,
isFlyActive: () => _flyActive
};
})();
export { SessionRooms };

View File

@@ -8,12 +8,20 @@
// holographic archive.
//
// World layout (hex cylinder, radius 25):
// North (z-) → Documents & Knowledge
// South (z+) → Projects & Tasks
// East (x+) → Code & Engineering
// West (x-) → Conversations & Social
// Center → Active Working Memory
// Below (y-) → Archive (cold storage)
//
// Inner ring — original Mnemosyne taxonomy (radius 15):
// North (z-) → Documents & Knowledge
// South (z+) → Projects & Tasks
// East (x+) → Code & Engineering
// West (x-) → Conversations & Social
// Center → Active Working Memory
// Below (y-) → Archive (cold storage)
//
// Outer ring — MemPalace category zones (radius 20, issue #1168):
// North (z-) → User Preferences [golden]
// East (x+) → Project facts [blue]
// South (z+) → Tool knowledge [green]
// West (x-) → General facts [gray]
//
// Usage from app.js:
// SpatialMemory.init(scene);
@@ -73,9 +81,53 @@ const SpatialMemory = (() => {
color: 0x334455,
glyph: '\uD83D\uDDC4',
description: 'Cold storage — rarely accessed, aged-out memories'
},
// ── MemPalace category zones — outer ring, issue #1168 ────────────
user_pref: {
label: 'User Preferences',
center: [0, 0, -20],
radius: 10,
color: 0xffd700,
glyph: '\u2605',
description: 'Personal preferences, habits, user-specific settings',
labelY: 5
},
project: {
label: 'Project Facts',
center: [20, 0, 0],
radius: 10,
color: 0x4488ff,
glyph: '\uD83D\uDCC1',
description: 'Project-specific knowledge, goals, context',
labelY: 5
},
tool: {
label: 'Tool Knowledge',
center: [0, 0, 20],
radius: 10,
color: 0x44cc66,
glyph: '\uD83D\uDD27',
description: 'Tools, commands, APIs, and how to use them',
labelY: 5
},
general: {
label: 'General Facts',
center: [-20, 0, 0],
radius: 10,
color: 0x8899aa,
glyph: '\uD83D\uDCDD',
description: 'Miscellaneous facts not fitting other categories',
labelY: 5
}
};
// ─── PERSISTENCE CONFIG ──────────────────────────────
const STORAGE_KEY = 'mnemosyne_spatial_memory';
const STORAGE_VERSION = 1;
let _dirty = false;
let _lastSavedHash = '';
// ─── STATE ────────────────────────────────────────────
let _scene = null;
let _regionMarkers = {};
@@ -93,6 +145,7 @@ const SpatialMemory = (() => {
const cx = region.center[0];
const cy = region.center[1] + 0.06;
const cz = region.center[2];
const labelY = region.labelY || 3;
const ringGeo = new THREE.RingGeometry(region.radius - 0.5, region.radius, 6);
const ringMat = new THREE.MeshBasicMaterial({
@@ -120,6 +173,22 @@ const SpatialMemory = (() => {
_scene.add(ring);
_scene.add(disc);
// Ground glow — brighter disc for MemPalace zones (labelY > 3 signals outer ring)
let glowDisc = null;
if (labelY > 3) {
const glowGeo = new THREE.CircleGeometry(region.radius, 32);
const glowMat = new THREE.MeshBasicMaterial({
color: region.color,
transparent: true,
opacity: 0.06,
side: THREE.DoubleSide
});
glowDisc = new THREE.Mesh(glowGeo, glowMat);
glowDisc.rotation.x = -Math.PI / 2;
glowDisc.position.set(cx, cy - 0.02, cz);
_scene.add(glowDisc);
}
// Floating label
const canvas = document.createElement('canvas');
canvas.width = 256;
@@ -133,11 +202,11 @@ const SpatialMemory = (() => {
const texture = new THREE.CanvasTexture(canvas);
const spriteMat = new THREE.SpriteMaterial({ map: texture, transparent: true, opacity: 0.6 });
const sprite = new THREE.Sprite(spriteMat);
sprite.position.set(cx, 3, cz);
sprite.position.set(cx, labelY, cz);
sprite.scale.set(4, 1, 1);
_scene.add(sprite);
return { ring, disc, sprite };
return { ring, disc, glowDisc, sprite };
}
// ─── PLACE A MEMORY ──────────────────────────────────
@@ -183,6 +252,8 @@ const SpatialMemory = (() => {
_drawConnections(mem.id, mem.connections);
}
_dirty = true;
saveToStorage();
console.info('[Mnemosyne] Spatial memory placed:', mem.id, 'in', region.label);
return crystal;
}
@@ -247,6 +318,8 @@ const SpatialMemory = (() => {
}
delete _memoryObjects[memId];
_dirty = true;
saveToStorage();
}
// ─── ANIMATE ─────────────────────────────────────────
@@ -273,6 +346,9 @@ const SpatialMemory = (() => {
if (marker.ring && marker.ring.material) {
marker.ring.material.opacity = 0.1 + Math.sin(now * 0.001) * 0.05;
}
if (marker.glowDisc && marker.glowDisc.material) {
marker.glowDisc.material.opacity = 0.04 + Math.sin(now * 0.0008) * 0.02;
}
});
}
@@ -286,7 +362,9 @@ const SpatialMemory = (() => {
_regionMarkers[key] = createRegionMarker(key, region);
});
console.info('[Mnemosyne] Spatial Memory Schema initialized —', Object.keys(REGIONS).length, 'regions');
// Restore persisted memories
const restored = loadFromStorage();
console.info('[Mnemosyne] Spatial Memory Schema initialized —', Object.keys(REGIONS).length, 'regions,', restored, 'memories restored');
return REGIONS;
}
@@ -320,6 +398,99 @@ const SpatialMemory = (() => {
return Object.values(_memoryObjects).map(o => o.data);
}
// ─── LOCALSTORAGE PERSISTENCE ────────────────────────
function _indexHash(index) {
// Simple hash of memory IDs + count to detect changes
const ids = (index.memories || []).map(m => m.id).sort().join(',');
return index.memories.length + ':' + ids;
}
function saveToStorage() {
if (typeof localStorage === 'undefined') {
console.warn('[Mnemosyne] localStorage unavailable — skipping save');
return false;
}
try {
const index = exportIndex();
const hash = _indexHash(index);
if (hash === _lastSavedHash) return false; // no change
const payload = JSON.stringify(index);
localStorage.setItem(STORAGE_KEY, payload);
_lastSavedHash = hash;
_dirty = false;
console.info('[Mnemosyne] Saved', index.memories.length, 'memories to localStorage');
return true;
} catch (e) {
if (e.name === 'QuotaExceededError' || e.code === 22) {
console.warn('[Mnemosyne] localStorage quota exceeded — pruning archive memories');
_pruneArchiveMemories();
try {
const index = exportIndex();
localStorage.setItem(STORAGE_KEY, JSON.stringify(index));
_lastSavedHash = _indexHash(index);
console.info('[Mnemosyne] Saved after prune:', index.memories.length, 'memories');
return true;
} catch (e2) {
console.error('[Mnemosyne] Save failed even after prune:', e2);
return false;
}
}
console.error('[Mnemosyne] Save failed:', e);
return false;
}
}
function loadFromStorage() {
if (typeof localStorage === 'undefined') {
console.warn('[Mnemosyne] localStorage unavailable — starting empty');
return 0;
}
try {
const raw = localStorage.getItem(STORAGE_KEY);
if (!raw) {
console.info('[Mnemosyne] No saved state found — starting fresh');
return 0;
}
const index = JSON.parse(raw);
if (index.version !== STORAGE_VERSION) {
console.warn('[Mnemosyne] Saved version mismatch (got', index.version, 'expected', + STORAGE_VERSION + ') — starting fresh');
return 0;
}
const count = importIndex(index);
_lastSavedHash = _indexHash(index);
return count;
} catch (e) {
console.error('[Mnemosyne] Load failed:', e);
return 0;
}
}
function _pruneArchiveMemories() {
// Remove oldest archive-region memories first
const archive = getMemoriesInRegion('archive');
const working = Object.values(_memoryObjects).filter(o => o.region !== 'archive');
// Sort archive by timestamp ascending (oldest first)
archive.sort((a, b) => {
const ta = a.data.timestamp || a.mesh.userData.createdAt || '';
const tb = b.data.timestamp || b.mesh.userData.createdAt || '';
return ta.localeCompare(tb);
});
const toRemove = Math.max(1, Math.ceil(archive.length * 0.25));
for (let i = 0; i < toRemove && i < archive.length; i++) {
removeMemory(archive[i].data.id);
}
console.info('[Mnemosyne] Pruned', toRemove, 'archive memories');
}
function clearStorage() {
if (typeof localStorage !== 'undefined') {
localStorage.removeItem(STORAGE_KEY);
_lastSavedHash = '';
console.info('[Mnemosyne] Cleared localStorage');
}
}
// ─── PERSISTENCE ─────────────────────────────────────
function exportIndex() {
return {
@@ -351,6 +522,81 @@ const SpatialMemory = (() => {
return count;
}
// ─── GRAVITY WELL CLUSTERING ──────────────────────────
// Force-directed layout: same-category crystals attract, unrelated repel.
// Run on load (bake positions, not per-frame). Spec from issue #1175.
const GRAVITY_ITERATIONS = 20;
const ATTRACT_FACTOR = 0.10; // 10% closer to same-category centroid per iteration
const REPEL_FACTOR = 0.05; // 5% away from nearest unrelated crystal
function runGravityLayout() {
const objs = Object.values(_memoryObjects);
if (objs.length < 2) {
console.info('[Mnemosyne] Gravity layout: fewer than 2 crystals, skipping');
return;
}
console.info('[Mnemosyne] Gravity layout starting —', objs.length, 'crystals,', GRAVITY_ITERATIONS, 'iterations');
for (let iter = 0; iter < GRAVITY_ITERATIONS; iter++) {
// Accumulate displacements before applying (avoids order-of-iteration bias)
const dx = new Float32Array(objs.length);
const dy = new Float32Array(objs.length);
const dz = new Float32Array(objs.length);
objs.forEach((obj, i) => {
const pos = obj.mesh.position;
const cat = obj.region;
// ── Attraction toward same-category centroid ──────────────
let sx = 0, sy = 0, sz = 0, sameCount = 0;
objs.forEach(o => {
if (o === obj || o.region !== cat) return;
sx += o.mesh.position.x;
sy += o.mesh.position.y;
sz += o.mesh.position.z;
sameCount++;
});
if (sameCount > 0) {
dx[i] += ((sx / sameCount) - pos.x) * ATTRACT_FACTOR;
dy[i] += ((sy / sameCount) - pos.y) * ATTRACT_FACTOR;
dz[i] += ((sz / sameCount) - pos.z) * ATTRACT_FACTOR;
}
// ── Repulsion from nearest unrelated crystal ───────────────
let nearestDist = Infinity;
let rnx = 0, rny = 0, rnz = 0;
objs.forEach(o => {
if (o === obj || o.region === cat) return;
const ex = pos.x - o.mesh.position.x;
const ey = pos.y - o.mesh.position.y;
const ez = pos.z - o.mesh.position.z;
const d = Math.sqrt(ex * ex + ey * ey + ez * ez);
if (d < nearestDist) {
nearestDist = d;
rnx = ex; rny = ey; rnz = ez;
}
});
if (nearestDist > 0.001 && nearestDist < Infinity) {
const len = Math.sqrt(rnx * rnx + rny * rny + rnz * rnz);
dx[i] += (rnx / len) * nearestDist * REPEL_FACTOR;
dy[i] += (rny / len) * nearestDist * REPEL_FACTOR;
dz[i] += (rnz / len) * nearestDist * REPEL_FACTOR;
}
});
// Apply displacements
objs.forEach((obj, i) => {
obj.mesh.position.x += dx[i];
obj.mesh.position.y += dy[i];
obj.mesh.position.z += dz[i];
});
}
// Bake final positions to localStorage
saveToStorage();
console.info('[Mnemosyne] Gravity layout complete — positions baked to localStorage');
}
// ─── SPATIAL SEARCH ──────────────────────────────────
function searchNearby(position, maxResults, maxDist) {
maxResults = maxResults || 10;
@@ -366,10 +612,53 @@ const SpatialMemory = (() => {
return results.slice(0, maxResults);
}
// ─── CRYSTAL MESH COLLECTION (for raycasting) ────────
function getCrystalMeshes() {
return Object.values(_memoryObjects).map(o => o.mesh);
}
// ─── MEMORY DATA FROM MESH ───────────────────────────
function getMemoryFromMesh(mesh) {
const entry = Object.values(_memoryObjects).find(o => o.mesh === mesh);
return entry ? { data: entry.data, region: entry.region } : null;
}
// ─── HIGHLIGHT / SELECT ──────────────────────────────
let _selectedId = null;
let _selectedOriginalEmissive = null;
function highlightMemory(memId) {
clearHighlight();
const obj = _memoryObjects[memId];
if (!obj) return;
_selectedId = memId;
_selectedOriginalEmissive = obj.mesh.material.emissiveIntensity;
obj.mesh.material.emissiveIntensity = 4.0;
obj.mesh.userData.selected = true;
}
function clearHighlight() {
if (_selectedId && _memoryObjects[_selectedId]) {
const obj = _memoryObjects[_selectedId];
obj.mesh.material.emissiveIntensity = _selectedOriginalEmissive || (obj.data.strength || 0.7) * 2.5;
obj.mesh.userData.selected = false;
}
_selectedId = null;
_selectedOriginalEmissive = null;
}
function getSelectedId() {
return _selectedId;
}
return {
init, placeMemory, removeMemory, update,
getMemoryAtPosition, getRegionAtPosition, getMemoriesInRegion, getAllMemories,
exportIndex, importIndex, searchNearby, REGIONS
getCrystalMeshes, getMemoryFromMesh, highlightMemory, clearHighlight, getSelectedId,
exportIndex, importIndex, searchNearby, REGIONS,
saveToStorage, loadFromStorage, clearStorage,
runGravityLayout
};
})();

313
nexus/computer_use.py Normal file
View File

@@ -0,0 +1,313 @@
"""
Hermes Desktop Automation Primitives — Computer Use (#1125)
Provides sandboxed desktop control tools for Hermes agents:
- computer_screenshot() — capture current desktop
- computer_click() — mouse click with poka-yoke on non-primary buttons
- computer_type() — keyboard input with poka-yoke on sensitive text
- computer_scroll() — scroll wheel action
- read_action_log() — inspect recent action audit trail
All actions are logged to a JSONL audit file.
pyautogui.FAILSAFE is enabled globally — move mouse to top-left corner to abort.
Designed to degrade gracefully when no display is available (headless CI).
"""
from __future__ import annotations
import base64
import io
import json
import logging
import os
import time
from pathlib import Path
from typing import Optional
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Safety globals
# ---------------------------------------------------------------------------
# Poka-yoke: require confirmation for dangerous inputs
_SENSITIVE_KEYWORDS = frozenset(
["password", "passwd", "secret", "token", "api_key", "apikey", "key", "auth"]
)
# Destructive mouse buttons (non-primary)
_DANGEROUS_BUTTONS = frozenset(["right", "middle"])
# Default log location
DEFAULT_ACTION_LOG = Path.home() / ".nexus" / "computer_use_actions.jsonl"
# ---------------------------------------------------------------------------
# Lazy pyautogui import — fails gracefully in headless environments
# ---------------------------------------------------------------------------
_PYAUTOGUI_AVAILABLE = False
_pyautogui = None
def _get_pyautogui():
"""Return pyautogui, enabling FAILSAFE. Returns None if unavailable."""
global _pyautogui, _PYAUTOGUI_AVAILABLE
if _pyautogui is not None:
return _pyautogui
try:
import pyautogui # type: ignore
pyautogui.FAILSAFE = True
pyautogui.PAUSE = 0.05 # small delay between actions
_pyautogui = pyautogui
_PYAUTOGUI_AVAILABLE = True
return _pyautogui
except Exception:
logger.warning("pyautogui unavailable — computer_use running in stub mode")
return None
def _get_pil():
"""Return PIL Image module or None."""
try:
from PIL import Image # type: ignore
return Image
except ImportError:
return None
# ---------------------------------------------------------------------------
# Audit log
# ---------------------------------------------------------------------------
def _log_action(action: str, params: dict, result: dict, log_path: Path = DEFAULT_ACTION_LOG):
"""Append one action record to the JSONL audit log."""
log_path.parent.mkdir(parents=True, exist_ok=True)
record = {
"ts": time.strftime("%Y-%m-%dT%H:%M:%S"),
"action": action,
"params": params,
"result": result,
}
with open(log_path, "a") as fh:
fh.write(json.dumps(record) + "\n")
# ---------------------------------------------------------------------------
# Public tool API
# ---------------------------------------------------------------------------
def computer_screenshot(
save_path: Optional[str] = None,
log_path: Path = DEFAULT_ACTION_LOG,
) -> dict:
"""Capture a screenshot of the current desktop.
Args:
save_path: Optional file path to save the PNG. If omitted the image
is returned as a base64-encoded string.
log_path: Audit log file (default ~/.nexus/computer_use_actions.jsonl).
Returns:
dict with keys:
- ok (bool)
- image_b64 (str | None) — base64 PNG when save_path is None
- saved_to (str | None) — path when save_path was given
- error (str | None) — human-readable error if ok=False
"""
pag = _get_pyautogui()
params = {"save_path": save_path}
if pag is None:
result = {"ok": False, "image_b64": None, "saved_to": None, "error": "pyautogui unavailable"}
_log_action("screenshot", params, result, log_path)
return result
try:
screenshot = pag.screenshot()
if save_path:
screenshot.save(save_path)
result = {"ok": True, "image_b64": None, "saved_to": save_path, "error": None}
else:
buf = io.BytesIO()
screenshot.save(buf, format="PNG")
b64 = base64.b64encode(buf.getvalue()).decode()
result = {"ok": True, "image_b64": b64, "saved_to": None, "error": None}
except Exception as exc:
result = {"ok": False, "image_b64": None, "saved_to": None, "error": str(exc)}
_log_action("screenshot", params, {k: v for k, v in result.items() if k != "image_b64"}, log_path)
return result
def computer_click(
x: int,
y: int,
button: str = "left",
confirm: bool = False,
log_path: Path = DEFAULT_ACTION_LOG,
) -> dict:
"""Click the mouse at screen coordinates (x, y).
Poka-yoke: right/middle clicks require confirm=True.
Args:
x: Horizontal screen coordinate.
y: Vertical screen coordinate.
button: "left" | "right" | "middle"
confirm: Must be True for non-left buttons.
log_path: Audit log file.
Returns:
dict with keys: ok, error
"""
params = {"x": x, "y": y, "button": button, "confirm": confirm}
if button in _DANGEROUS_BUTTONS and not confirm:
result = {
"ok": False,
"error": (
f"button={button!r} requires confirm=True (poka-yoke). "
"Pass confirm=True only after verifying this action is intentional."
),
}
_log_action("click", params, result, log_path)
return result
if button not in ("left", "right", "middle"):
result = {"ok": False, "error": f"Unknown button {button!r}. Use 'left', 'right', or 'middle'."}
_log_action("click", params, result, log_path)
return result
pag = _get_pyautogui()
if pag is None:
result = {"ok": False, "error": "pyautogui unavailable"}
_log_action("click", params, result, log_path)
return result
try:
pag.click(x, y, button=button)
result = {"ok": True, "error": None}
except Exception as exc:
result = {"ok": False, "error": str(exc)}
_log_action("click", params, result, log_path)
return result
def computer_type(
text: str,
confirm: bool = False,
interval: float = 0.02,
log_path: Path = DEFAULT_ACTION_LOG,
) -> dict:
"""Type text using the keyboard.
Poka-yoke: if *text* contains a sensitive keyword (password, token, key…)
confirm=True is required. The actual text value is never written to the
audit log.
Args:
text: The string to type.
confirm: Must be True when the text looks sensitive.
interval: Delay between keystrokes (seconds).
log_path: Audit log file.
Returns:
dict with keys: ok, error
"""
lower = text.lower()
is_sensitive = any(kw in lower for kw in _SENSITIVE_KEYWORDS)
params = {"length": len(text), "is_sensitive": is_sensitive, "confirm": confirm}
if is_sensitive and not confirm:
result = {
"ok": False,
"error": (
"Text contains sensitive keyword. Pass confirm=True to proceed. "
"Ensure no secrets are being typed into unintended windows."
),
}
_log_action("type", params, result, log_path)
return result
pag = _get_pyautogui()
if pag is None:
result = {"ok": False, "error": "pyautogui unavailable"}
_log_action("type", params, result, log_path)
return result
try:
pag.typewrite(text, interval=interval)
result = {"ok": True, "error": None}
except Exception as exc:
result = {"ok": False, "error": str(exc)}
_log_action("type", params, result, log_path)
return result
def computer_scroll(
x: int,
y: int,
amount: int = 3,
log_path: Path = DEFAULT_ACTION_LOG,
) -> dict:
"""Scroll the mouse wheel at screen coordinates (x, y).
Args:
x: Horizontal screen coordinate.
y: Vertical screen coordinate.
amount: Number of scroll units. Positive = scroll up, negative = down.
log_path: Audit log file.
Returns:
dict with keys: ok, error
"""
params = {"x": x, "y": y, "amount": amount}
pag = _get_pyautogui()
if pag is None:
result = {"ok": False, "error": "pyautogui unavailable"}
_log_action("scroll", params, result, log_path)
return result
try:
pag.scroll(amount, x=x, y=y)
result = {"ok": True, "error": None}
except Exception as exc:
result = {"ok": False, "error": str(exc)}
_log_action("scroll", params, result, log_path)
return result
def read_action_log(
n: int = 20,
log_path: Path = DEFAULT_ACTION_LOG,
) -> list[dict]:
"""Return the most recent *n* action records from the audit log.
Args:
n: Maximum number of records to return.
log_path: Audit log file.
Returns:
List of action dicts, newest first.
"""
if not log_path.exists():
return []
records: list[dict] = []
with open(log_path) as fh:
for line in fh:
line = line.strip()
if line:
try:
records.append(json.loads(line))
except json.JSONDecodeError:
pass
return list(reversed(records[-n:]))

118
nexus/computer_use_demo.py Normal file
View File

@@ -0,0 +1,118 @@
"""
Phase 1 Demo — Desktop Automation via Hermes (#1125)
Demonstrates the computer_use primitives end-to-end:
1. Take a baseline screenshot
2. Open a browser and navigate to the Gitea forge
3. Take an evidence screenshot
Run inside a desktop session (Xvfb or real display):
python -m nexus.computer_use_demo
Or via Docker:
docker compose -f docker-compose.desktop.yml run hermes-desktop \
python -m nexus.computer_use_demo
"""
from __future__ import annotations
import logging
import sys
import time
from pathlib import Path
from nexus.computer_use import (
computer_click,
computer_screenshot,
computer_type,
read_action_log,
)
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger(__name__)
GITEA_URL = "https://forge.alexanderwhitestone.com"
EVIDENCE_DIR = Path.home() / ".nexus" / "computer_use_evidence"
def run_demo() -> bool:
"""Execute the Phase 1 demo. Returns True on success."""
EVIDENCE_DIR.mkdir(parents=True, exist_ok=True)
log.info("=== Phase 1 Computer-Use Demo ===")
# --- Step 1: baseline screenshot ---
baseline = EVIDENCE_DIR / "01_baseline.png"
log.info("Step 1: capturing baseline screenshot → %s", baseline)
result = computer_screenshot(save_path=str(baseline))
if not result["ok"]:
log.error("Baseline screenshot failed: %s", result["error"])
return False
log.info(" ✓ baseline saved")
# --- Step 2: open browser ---
log.info("Step 2: opening browser")
try:
import subprocess
# Use xdg-open / open depending on platform; fallback to chromium
for cmd in (
["xdg-open", GITEA_URL],
["chromium-browser", "--no-sandbox", GITEA_URL],
["chromium", "--no-sandbox", GITEA_URL],
["google-chrome", "--no-sandbox", GITEA_URL],
["open", GITEA_URL], # macOS
):
try:
subprocess.Popen(cmd, stderr=subprocess.DEVNULL, stdout=subprocess.DEVNULL)
log.info(" ✓ browser opened with: %s", cmd[0])
break
except FileNotFoundError:
continue
else:
log.warning(" ⚠ no browser found — skipping open step")
except Exception as exc:
log.warning(" ⚠ could not open browser: %s", exc)
# Give the browser time to load
time.sleep(3)
# --- Step 3: click address bar and navigate (best-effort) ---
log.info("Step 3: attempting to type URL in browser address bar (best-effort)")
try:
import pyautogui # type: ignore
# Common shortcut to focus address bar
pyautogui.hotkey("ctrl", "l")
time.sleep(0.3)
result_type = computer_type(GITEA_URL)
if result_type["ok"]:
pyautogui.press("enter")
time.sleep(2)
log.info(" ✓ URL typed")
else:
log.warning(" ⚠ type failed: %s", result_type["error"])
except ImportError:
log.warning(" ⚠ pyautogui not available — skipping URL type step")
# --- Step 4: evidence screenshot ---
evidence = EVIDENCE_DIR / "02_gitea.png"
log.info("Step 4: capturing evidence screenshot → %s", evidence)
result = computer_screenshot(save_path=str(evidence))
if not result["ok"]:
log.error("Evidence screenshot failed: %s", result["error"])
return False
log.info(" ✓ evidence saved")
# --- Step 5: summary ---
log.info("Step 5: recent action log")
for entry in read_action_log(n=10):
log.info(" %s %s ok=%s", entry["ts"], entry["action"], entry["result"].get("ok"))
log.info("=== Demo complete — evidence in %s ===", EVIDENCE_DIR)
return True
if __name__ == "__main__":
success = run_demo()
sys.exit(0 if success else 1)

144
server.py
View File

@@ -3,17 +3,156 @@
The Nexus WebSocket Gateway — Robust broadcast bridge for Timmy's consciousness.
This server acts as the central hub for the-nexus, connecting the mind (nexus_think.py),
the body (Evennia/Morrowind), and the visualization surface.
Serves HTTP alongside WebSocket:
GET /api/briefing — heartbeat + morning report data for the HUD briefing panel
"""
import asyncio
import json
import logging
import os
import signal
import sys
from typing import Set
from datetime import datetime, timezone
from http.server import HTTPServer, SimpleHTTPRequestHandler
from pathlib import Path
from threading import Thread
from typing import Any, Dict, Set
# Branch protected file - see POLICY.md
import websockets
# ── HTTP Briefing Endpoint ─────────────────────────────────────────────
HEARTBEAT_PATH = Path.home() / ".nexus" / "heartbeat.json"
REPORTS_DIR = Path.home() / ".local" / "timmy" / "reports"
CRON_HEARTBEAT_DIR_PRIMARY = Path("/var/run/bezalel/heartbeats")
CRON_HEARTBEAT_DIR_FALLBACK = Path.home() / ".bezalel" / "heartbeats"
def _read_json_file(path: Path) -> Any:
"""Read and parse a JSON file. Returns None on failure."""
try:
return json.loads(path.read_text(encoding="utf-8"))
except Exception:
return None
def _resolve_cron_dir() -> Path:
"""Return the first writable cron heartbeat directory."""
for d in [CRON_HEARTBEAT_DIR_PRIMARY, CRON_HEARTBEAT_DIR_FALLBACK]:
if d.exists() and os.access(str(d), os.R_OK):
return d
return CRON_HEARTBEAT_DIR_FALLBACK
def _read_cron_heartbeats() -> list:
"""Read all .last files from the cron heartbeat directory."""
hb_dir = _resolve_cron_dir()
if not hb_dir.exists():
return []
now = datetime.now(timezone.utc).timestamp()
jobs = []
for f in sorted(hb_dir.glob("*.last")):
data = _read_json_file(f)
if data is None:
jobs.append({"job": f.stem, "healthy": False, "message": "corrupt"})
continue
ts = float(data.get("timestamp", 0))
interval = int(data.get("interval", 3600))
age = now - ts
is_stale = age > (2 * interval)
jobs.append({
"job": f.stem,
"healthy": not is_stale,
"age_secs": round(age, 1),
"interval": interval,
"last_seen": datetime.fromtimestamp(ts, tz=timezone.utc).isoformat() if ts else None,
"message": f"{'STALE' if is_stale else 'OK'} ({age:.0f}s / {interval}s)" if ts else "no timestamp",
})
return jobs
def _latest_morning_report() -> Dict[str, Any] | None:
"""Find the most recent morning report file."""
if not REPORTS_DIR.exists():
return None
reports = sorted(REPORTS_DIR.glob("morning-*.json"), reverse=True)
if not reports:
return None
return _read_json_file(reports[0])
def _build_briefing() -> Dict[str, Any]:
"""Assemble the full briefing payload from real files."""
now = datetime.now(timezone.utc)
# Core heartbeat
core_hb = _read_json_file(HEARTBEAT_PATH)
if core_hb:
beat_ts = float(core_hb.get("timestamp", 0))
core_hb["age_secs"] = round(now.timestamp() - beat_ts, 1) if beat_ts else None
# Cron heartbeats
cron_jobs = _read_cron_heartbeats()
healthy_count = sum(1 for j in cron_jobs if j.get("healthy"))
stale_count = sum(1 for j in cron_jobs if not j.get("healthy"))
# Morning report
report = _latest_morning_report()
return {
"generated_at": now.isoformat(),
"core_heartbeat": core_hb,
"cron_heartbeat": {
"jobs": cron_jobs,
"healthy_count": healthy_count,
"stale_count": stale_count,
},
"morning_report": report,
}
class BriefingHandler(SimpleHTTPRequestHandler):
"""Minimal HTTP handler that only serves /api/briefing."""
def do_GET(self):
if self.path == "/api/briefing":
try:
data = _build_briefing()
body = json.dumps(data).encode("utf-8")
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.send_header("Access-Control-Allow-Origin", "*")
self.end_headers()
self.wfile.write(body)
except Exception as e:
self.send_error(500, str(e))
elif self.path == "/api/health":
body = json.dumps({"status": "ok"}).encode("utf-8")
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Access-Control-Allow-Origin", "*")
self.end_headers()
self.wfile.write(body)
else:
self.send_error(404)
def log_message(self, fmt, *args):
pass # Suppress HTTP access logs — WS gateway logs are enough
def start_http_server(port: int = 8766):
"""Run the HTTP server in a daemon thread."""
server = HTTPServer(("0.0.0.0", port), BriefingHandler)
thread = Thread(target=server.serve_forever, daemon=True)
thread.start()
logger = logging.getLogger("nexus-gateway")
logger.info(f"Briefing HTTP server started on http://0.0.0.0:{port}")
return server
# Configuration
PORT = 8765
HOST = "0.0.0.0" # Allow external connections if needed
@@ -80,6 +219,9 @@ async def broadcast_handler(websocket: websockets.WebSocketServerProtocol):
async def main():
"""Main server loop with graceful shutdown."""
# Start HTTP briefing endpoint alongside WS
http_server = start_http_server(port=8766)
logger.info(f"Starting Nexus WS gateway on ws://{HOST}:{PORT}")
# Set up signal handlers for graceful shutdown

489
style.css
View File

@@ -1223,3 +1223,492 @@ canvas#nexus-canvas {
.l402-msg { color: #fff; }
.pse-status { color: #4af0c0; font-weight: 600; }
/* ═══ HEARTBEAT BRIEFING PANEL ═════════════════════════════════════ */
.hud-panel-briefing {
width: 320px;
max-height: 420px;
border-left-color: #7b5cff;
}
.hud-panel-briefing .panel-header {
display: flex;
align-items: center;
gap: 6px;
color: #7b5cff;
}
.hud-panel-briefing .panel-content {
max-height: 360px;
overflow-y: auto;
}
/* Pulse dot */
.hb-pulse-dot {
width: 6px;
height: 6px;
border-radius: 50%;
background: #4af0c0;
box-shadow: 0 0 6px #4af0c0;
animation: hb-dot-pulse 2s ease-in-out infinite;
flex-shrink: 0;
}
.hb-pulse-dot.offline {
background: #ff4466;
box-shadow: 0 0 6px #ff4466;
animation: none;
}
@keyframes hb-dot-pulse {
0%, 100% { opacity: 0.4; }
50% { opacity: 1; }
}
/* Briefing sections */
.briefing-section {
margin-bottom: 8px;
padding-bottom: 6px;
border-bottom: 1px solid rgba(123, 92, 255, 0.12);
}
.briefing-section:last-child { border-bottom: none; margin-bottom: 0; }
.briefing-section-label {
font-size: 9px;
letter-spacing: 0.12em;
text-transform: uppercase;
color: #8a9ab8;
margin-bottom: 4px;
}
/* Core heartbeat row */
.hb-core-row {
display: flex;
gap: 10px;
align-items: center;
margin-bottom: 4px;
}
.hb-core-status {
font-weight: 700;
font-size: 11px;
}
.hb-core-status.alive { color: #4af0c0; }
.hb-core-status.dead { color: #ff4466; }
.hb-core-meta {
font-size: 10px;
color: #8a9ab8;
}
/* Cron jobs */
.hb-cron-row {
display: flex;
gap: 8px;
font-size: 10px;
margin-bottom: 2px;
}
.hb-cron-healthy { color: #4af0c0; }
.hb-cron-stale { color: #ff4466; font-weight: 700; }
.hb-cron-job {
display: flex;
justify-content: space-between;
font-size: 10px;
padding: 1px 0;
}
.hb-cron-job-name { color: #e0f0ff; }
.hb-cron-job-status.healthy { color: #4af0c0; }
.hb-cron-job-status.stale { color: #ff4466; font-weight: 700; }
/* Morning report stats */
.hb-stats-row {
display: flex;
gap: 12px;
font-size: 10px;
}
.hb-stat {
display: flex;
flex-direction: column;
gap: 1px;
}
.hb-stat-value {
font-family: 'Orbitron', sans-serif;
font-weight: 700;
font-size: 14px;
}
.hb-stat-label {
font-size: 9px;
color: #8a9ab8;
letter-spacing: 0.08em;
text-transform: uppercase;
}
/* Blockers */
.hb-blocker {
font-size: 10px;
color: #ff4466;
padding: 1px 0;
}
/* Narrative */
.hb-narrative {
font-size: 10px;
color: #8a9ab8;
line-height: 1.5;
font-style: italic;
}
/* Empty / offline state */
.hb-empty {
font-size: 10px;
color: #8a9ab8;
text-align: center;
padding: 12px 0;
}
/* Timestamp */
.hb-timestamp {
font-size: 9px;
color: rgba(138, 154, 184, 0.6);
margin-top: 4px;
}
/* ═══════════════════════════════════════════
MNEMOSYNE — MEMORY CRYSTAL INSPECTION PANEL
═══════════════════════════════════════════ */
.memory-panel {
position: fixed;
top: 50%;
right: 24px;
transform: translateY(-50%);
z-index: 120;
animation: memoryPanelIn 0.22s ease-out forwards;
}
.memory-panel-fade-out {
animation: memoryPanelOut 0.18s ease-in forwards !important;
}
@keyframes memoryPanelIn {
from { opacity: 0; transform: translateY(-50%) translateX(16px); }
to { opacity: 1; transform: translateY(-50%) translateX(0); }
}
@keyframes memoryPanelOut {
from { opacity: 1; }
to { opacity: 0; transform: translateY(-50%) translateX(12px); }
}
.memory-panel-content {
width: 340px;
background: rgba(8, 8, 24, 0.92);
backdrop-filter: blur(12px);
border: 1px solid rgba(74, 240, 192, 0.25);
border-radius: 12px;
padding: 16px;
box-shadow: 0 0 30px rgba(74, 240, 192, 0.08), 0 8px 32px rgba(0, 0, 0, 0.4);
}
.memory-panel-header {
display: flex;
align-items: center;
gap: 6px;
margin-bottom: 10px;
padding-bottom: 10px;
border-bottom: 1px solid rgba(255, 255, 255, 0.06);
}
.memory-panel-region-dot {
width: 10px;
height: 10px;
border-radius: 50%;
flex-shrink: 0;
}
.memory-panel-region {
font-family: var(--font-display, monospace);
font-size: 11px;
letter-spacing: 0.15em;
color: var(--color-primary, #4af0c0);
text-transform: uppercase;
flex: 1;
}
.memory-panel-close {
background: none;
border: 1px solid rgba(255, 255, 255, 0.1);
color: var(--color-text-muted, #888);
font-size: 14px;
cursor: pointer;
width: 24px;
height: 24px;
border-radius: 6px;
display: flex;
align-items: center;
justify-content: center;
transition: all 0.15s;
}
.memory-panel-close:hover {
background: rgba(255, 255, 255, 0.05);
color: #fff;
}
.memory-panel-body {
font-size: 14px;
line-height: 1.6;
color: var(--color-text, #ccc);
margin-bottom: 14px;
max-height: 120px;
overflow-y: auto;
word-break: break-word;
}
.memory-panel-meta {
display: flex;
flex-direction: column;
gap: 5px;
font-size: 11px;
}
.memory-meta-row {
display: flex;
gap: 8px;
align-items: baseline;
}
.memory-meta-label {
color: var(--color-text-muted, #666);
text-transform: uppercase;
letter-spacing: 0.08em;
min-width: 50px;
flex-shrink: 0;
}
.memory-meta-row span:last-child {
color: var(--color-text, #aaa);
word-break: break-all;
}
.memory-conn-tag {
display: inline-block;
background: rgba(74, 240, 192, 0.1);
border: 1px solid rgba(74, 240, 192, 0.2);
border-radius: 4px;
padding: 1px 6px;
font-size: 10px;
font-family: var(--font-mono, monospace);
color: var(--color-primary, #4af0c0);
margin: 1px 2px;
}
.memory-conn-link {
cursor: pointer;
transition: background 0.15s, border-color 0.15s;
}
.memory-conn-link:hover {
background: rgba(74, 240, 192, 0.22);
border-color: rgba(74, 240, 192, 0.5);
color: #fff;
}
/* Entity name — large heading inside panel */
.memory-entity-name {
font-family: var(--font-display, monospace);
font-size: 17px;
font-weight: 700;
color: #fff;
letter-spacing: 0.04em;
margin-bottom: 8px;
text-transform: capitalize;
word-break: break-word;
}
/* Category badge */
.memory-category-badge {
font-family: var(--font-display, monospace);
font-size: 9px;
letter-spacing: 0.12em;
font-weight: 700;
padding: 2px 6px;
border-radius: 4px;
border: 1px solid rgba(74, 240, 192, 0.3);
background: rgba(74, 240, 192, 0.12);
color: var(--color-primary, #4af0c0);
flex-shrink: 0;
}
/* Trust score bar */
.memory-trust-row {
display: flex;
align-items: center;
gap: 8px;
margin-bottom: 12px;
font-size: 11px;
}
.memory-trust-bar {
flex: 1;
height: 5px;
background: rgba(255, 255, 255, 0.08);
border-radius: 3px;
overflow: hidden;
}
.memory-trust-fill {
height: 100%;
border-radius: 3px;
background: var(--color-primary, #4af0c0);
transition: width 0.35s ease;
}
.memory-trust-value {
color: var(--color-text-muted, #888);
min-width: 32px;
text-align: right;
}
/* Pin button */
.memory-panel-pin {
background: none;
border: 1px solid rgba(255, 255, 255, 0.1);
color: var(--color-text-muted, #888);
font-size: 11px;
cursor: pointer;
width: 24px;
height: 24px;
border-radius: 6px;
display: flex;
align-items: center;
justify-content: center;
transition: all 0.15s;
flex-shrink: 0;
}
.memory-panel-pin:hover {
background: rgba(255, 255, 255, 0.05);
color: #fff;
}
.memory-panel-pin.pinned {
background: rgba(74, 240, 192, 0.15);
border-color: rgba(74, 240, 192, 0.4);
color: var(--color-primary, #4af0c0);
}
/* Related row — allow wrapping */
.memory-meta-row--related {
align-items: flex-start;
}
.memory-meta-row--related span:last-child {
flex-wrap: wrap;
display: flex;
gap: 2px;
}
/* ═══════════════════════════════════════════════════════
PROJECT MNEMOSYNE — SESSION ROOM HUD PANEL (#1171)
═══════════════════════════════════════════════════════ */
.session-room-panel {
position: fixed;
bottom: 24px;
left: 50%;
transform: translateX(-50%);
z-index: 125;
animation: sessionPanelIn 0.25s ease-out forwards;
}
.session-room-panel.session-panel-fade-out {
animation: sessionPanelOut 0.2s ease-in forwards !important;
}
@keyframes sessionPanelIn {
from { opacity: 0; transform: translateX(-50%) translateY(12px); }
to { opacity: 1; transform: translateX(-50%) translateY(0); }
}
@keyframes sessionPanelOut {
from { opacity: 1; }
to { opacity: 0; transform: translateX(-50%) translateY(10px); }
}
.session-room-panel-content {
min-width: 320px;
max-width: 480px;
background: rgba(8, 4, 28, 0.93);
backdrop-filter: blur(14px);
border: 1px solid rgba(123, 92, 255, 0.35);
border-radius: 12px;
padding: 14px 18px;
box-shadow: 0 0 32px rgba(123, 92, 255, 0.1), 0 8px 32px rgba(0, 0, 0, 0.45);
}
.session-room-header {
display: flex;
align-items: center;
gap: 8px;
margin-bottom: 8px;
padding-bottom: 8px;
border-bottom: 1px solid rgba(255, 255, 255, 0.07);
}
.session-room-icon {
font-size: 14px;
line-height: 1;
}
.session-room-title {
font-family: var(--font-display, monospace);
font-size: 11px;
letter-spacing: 0.18em;
color: #9b7cff;
text-transform: uppercase;
flex: 1;
}
.session-room-close {
background: none;
border: none;
color: rgba(255, 255, 255, 0.35);
cursor: pointer;
font-size: 14px;
padding: 0 2px;
line-height: 1;
transition: color 0.15s;
}
.session-room-close:hover {
color: rgba(255, 255, 255, 0.8);
}
.session-room-timestamp {
font-family: var(--font-display, monospace);
font-size: 13px;
color: #c8b4ff;
margin-bottom: 6px;
letter-spacing: 0.08em;
}
.session-room-fact-count {
font-size: 11px;
color: rgba(200, 180, 255, 0.55);
margin-bottom: 10px;
}
.session-room-facts {
display: flex;
flex-direction: column;
gap: 4px;
max-height: 140px;
overflow-y: auto;
}
.session-room-fact-item {
font-size: 11px;
color: rgba(220, 210, 255, 0.75);
padding: 4px 8px;
background: rgba(123, 92, 255, 0.07);
border-left: 2px solid rgba(123, 92, 255, 0.4);
border-radius: 0 4px 4px 0;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
.session-room-hint {
margin-top: 10px;
font-size: 10px;
color: rgba(200, 180, 255, 0.35);
text-align: center;
letter-spacing: 0.1em;
text-transform: uppercase;
}

362
tests/test_computer_use.py Normal file
View File

@@ -0,0 +1,362 @@
"""
Tests for nexus.computer_use — Desktop Automation Primitives (#1125)
All tests run fully headless: pyautogui is mocked throughout.
No display is required.
"""
from __future__ import annotations
import json
import sys
from pathlib import Path
from unittest.mock import MagicMock, patch, call
import pytest
sys.path.insert(0, str(Path(__file__).parent.parent))
from nexus.computer_use import (
_DANGEROUS_BUTTONS,
_SENSITIVE_KEYWORDS,
computer_click,
computer_screenshot,
computer_scroll,
computer_type,
read_action_log,
)
# ---------------------------------------------------------------------------
# Helpers / fixtures
# ---------------------------------------------------------------------------
@pytest.fixture
def tmp_log(tmp_path):
"""Return a temporary JSONL audit log path."""
return tmp_path / "actions.jsonl"
def _last_log_entry(log_path: Path) -> dict:
lines = [l.strip() for l in log_path.read_text().splitlines() if l.strip()]
return json.loads(lines[-1])
def _make_mock_pag(screenshot_raises=None):
"""Build a minimal pyautogui mock."""
mock = MagicMock()
mock.FAILSAFE = True
mock.PAUSE = 0.05
if screenshot_raises:
mock.screenshot.side_effect = screenshot_raises
else:
img_mock = MagicMock()
img_mock.save = MagicMock()
mock.screenshot.return_value = img_mock
return mock
# ---------------------------------------------------------------------------
# computer_screenshot
# ---------------------------------------------------------------------------
class TestComputerScreenshot:
def test_returns_b64_when_no_save_path(self, tmp_log):
mock_pag = _make_mock_pag()
# Make save() write fake PNG bytes
import io
buf = io.BytesIO(b"\x89PNG\r\n\x1a\n" + b"\x00" * 20)
def fake_save(obj, format=None):
obj.write(buf.getvalue())
mock_pag.screenshot.return_value.save = MagicMock(side_effect=fake_save)
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_screenshot(log_path=tmp_log)
assert result["ok"] is True
assert result["image_b64"] is not None
assert result["saved_to"] is None
assert result["error"] is None
def test_saves_to_path(self, tmp_log, tmp_path):
mock_pag = _make_mock_pag()
out_png = tmp_path / "shot.png"
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_screenshot(save_path=str(out_png), log_path=tmp_log)
assert result["ok"] is True
assert result["saved_to"] == str(out_png)
assert result["image_b64"] is None
mock_pag.screenshot.return_value.save.assert_called_once_with(str(out_png))
def test_logs_action(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
computer_screenshot(log_path=tmp_log)
entry = _last_log_entry(tmp_log)
assert entry["action"] == "screenshot"
assert "ts" in entry
def test_returns_error_when_headless(self, tmp_log):
with patch("nexus.computer_use._get_pyautogui", return_value=None):
result = computer_screenshot(log_path=tmp_log)
assert result["ok"] is False
assert "unavailable" in result["error"]
def test_handles_screenshot_exception(self, tmp_log):
mock_pag = _make_mock_pag(screenshot_raises=RuntimeError("display error"))
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_screenshot(log_path=tmp_log)
assert result["ok"] is False
assert "display error" in result["error"]
def test_image_b64_not_written_to_log(self, tmp_log):
"""The (potentially huge) base64 blob must NOT appear in the audit log."""
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
computer_screenshot(log_path=tmp_log)
raw = tmp_log.read_text()
assert "image_b64" not in raw
# ---------------------------------------------------------------------------
# computer_click
# ---------------------------------------------------------------------------
class TestComputerClick:
def test_left_click_succeeds(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_click(100, 200, log_path=tmp_log)
assert result["ok"] is True
mock_pag.click.assert_called_once_with(100, 200, button="left")
def test_right_click_blocked_without_confirm(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_click(100, 200, button="right", log_path=tmp_log)
assert result["ok"] is False
assert "confirm=True" in result["error"]
mock_pag.click.assert_not_called()
def test_right_click_allowed_with_confirm(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_click(100, 200, button="right", confirm=True, log_path=tmp_log)
assert result["ok"] is True
mock_pag.click.assert_called_once_with(100, 200, button="right")
def test_middle_click_blocked_without_confirm(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_click(50, 50, button="middle", log_path=tmp_log)
assert result["ok"] is False
def test_middle_click_allowed_with_confirm(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_click(50, 50, button="middle", confirm=True, log_path=tmp_log)
assert result["ok"] is True
def test_unknown_button_rejected(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_click(0, 0, button="turbo", log_path=tmp_log)
assert result["ok"] is False
assert "Unknown button" in result["error"]
def test_logs_click_action(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
computer_click(10, 20, log_path=tmp_log)
entry = _last_log_entry(tmp_log)
assert entry["action"] == "click"
assert entry["params"]["x"] == 10
assert entry["params"]["y"] == 20
def test_returns_error_when_headless(self, tmp_log):
with patch("nexus.computer_use._get_pyautogui", return_value=None):
result = computer_click(0, 0, log_path=tmp_log)
assert result["ok"] is False
def test_handles_click_exception(self, tmp_log):
mock_pag = _make_mock_pag()
mock_pag.click.side_effect = Exception("out of bounds")
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_click(99999, 99999, log_path=tmp_log)
assert result["ok"] is False
assert "out of bounds" in result["error"]
# ---------------------------------------------------------------------------
# computer_type
# ---------------------------------------------------------------------------
class TestComputerType:
def test_plain_text_succeeds(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_type("hello world", log_path=tmp_log)
assert result["ok"] is True
mock_pag.typewrite.assert_called_once_with("hello world", interval=0.02)
def test_sensitive_text_blocked_without_confirm(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_type("mypassword123", log_path=tmp_log)
assert result["ok"] is False
assert "confirm=True" in result["error"]
mock_pag.typewrite.assert_not_called()
def test_sensitive_text_allowed_with_confirm(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_type("mypassword123", confirm=True, log_path=tmp_log)
assert result["ok"] is True
def test_sensitive_keywords_all_blocked(self, tmp_log):
mock_pag = _make_mock_pag()
for keyword in _SENSITIVE_KEYWORDS:
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_type(f"my{keyword}value", log_path=tmp_log)
assert result["ok"] is False, f"keyword {keyword!r} should be blocked"
def test_text_not_logged(self, tmp_log):
"""Actual typed text must NOT appear in the audit log."""
mock_pag = _make_mock_pag()
secret = "super_secret_value_xyz"
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
computer_type(secret, confirm=True, log_path=tmp_log)
raw = tmp_log.read_text()
assert secret not in raw
def test_logs_length_not_content(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
computer_type("hello", log_path=tmp_log)
entry = _last_log_entry(tmp_log)
assert entry["params"]["length"] == 5
def test_returns_error_when_headless(self, tmp_log):
with patch("nexus.computer_use._get_pyautogui", return_value=None):
result = computer_type("abc", log_path=tmp_log)
assert result["ok"] is False
def test_handles_type_exception(self, tmp_log):
mock_pag = _make_mock_pag()
mock_pag.typewrite.side_effect = Exception("keyboard error")
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_type("hello", log_path=tmp_log)
assert result["ok"] is False
assert "keyboard error" in result["error"]
# ---------------------------------------------------------------------------
# computer_scroll
# ---------------------------------------------------------------------------
class TestComputerScroll:
def test_scroll_up(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_scroll(400, 300, amount=5, log_path=tmp_log)
assert result["ok"] is True
mock_pag.scroll.assert_called_once_with(5, x=400, y=300)
def test_scroll_down_negative(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_scroll(400, 300, amount=-3, log_path=tmp_log)
assert result["ok"] is True
mock_pag.scroll.assert_called_once_with(-3, x=400, y=300)
def test_logs_scroll_action(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
computer_scroll(10, 20, amount=2, log_path=tmp_log)
entry = _last_log_entry(tmp_log)
assert entry["action"] == "scroll"
assert entry["params"]["amount"] == 2
def test_returns_error_when_headless(self, tmp_log):
with patch("nexus.computer_use._get_pyautogui", return_value=None):
result = computer_scroll(0, 0, log_path=tmp_log)
assert result["ok"] is False
def test_handles_scroll_exception(self, tmp_log):
mock_pag = _make_mock_pag()
mock_pag.scroll.side_effect = Exception("scroll error")
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
result = computer_scroll(0, 0, log_path=tmp_log)
assert result["ok"] is False
# ---------------------------------------------------------------------------
# read_action_log
# ---------------------------------------------------------------------------
class TestReadActionLog:
def test_returns_empty_list_when_no_log(self, tmp_path):
missing = tmp_path / "nonexistent.jsonl"
assert read_action_log(log_path=missing) == []
def test_returns_recent_entries(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
computer_click(1, 1, log_path=tmp_log)
computer_click(2, 2, log_path=tmp_log)
computer_click(3, 3, log_path=tmp_log)
entries = read_action_log(n=2, log_path=tmp_log)
assert len(entries) == 2
def test_newest_first(self, tmp_log):
mock_pag = _make_mock_pag()
with patch("nexus.computer_use._get_pyautogui", return_value=mock_pag):
computer_click(1, 1, log_path=tmp_log)
computer_scroll(5, 5, log_path=tmp_log)
entries = read_action_log(log_path=tmp_log)
# Most recent action (scroll) should be first
assert entries[0]["action"] == "scroll"
assert entries[1]["action"] == "click"
def test_skips_malformed_lines(self, tmp_log):
tmp_log.parent.mkdir(parents=True, exist_ok=True)
tmp_log.write_text('{"action": "click", "ts": "2026-01-01", "params": {}, "result": {}}\nNOT JSON\n')
entries = read_action_log(log_path=tmp_log)
assert len(entries) == 1