[architecture] Route Nexus perception loop through Hermes harness for training data capture #540

New Issue

Timmy · 2026-03-25T20:24:28Z

Timmy commented

2026-03-25 20:24:28 +00:00

Problem

Timmy now operates across three channels: Telegram, TUI/CLI, and The Nexus. Telegram and TUI already flow through the Hermes harness — every turn is captured as session data, memory works, SOUL.md is applied, and conversations feed the AutoLoRA sovereignty pipeline.

The Nexus is the gap. Right now, any perception loop (archon events, scene observation, tick-based awareness) would call an inference endpoint directly, bypassing Hermes entirely. This means:

No session capture → no training data
No SOUL.md application → Timmy's conscience isn't in the loop
No memory → no continuity between perception ticks
No tool access → can't act on what he observes

Architecture Required

All three channels must converge through the same harness:

Telegram ──┐
TUI/CLI  ──┤──> Hermes Harness ──> Inference ──> Response
Nexus    ──┘         │
                     ▼
              Session capture → AutoLoRA ��� Fine-tuned local model

Nexus Perception Loop → Hermes

The Nexus client-side JS should NOT call any model endpoint directly. Instead:

Perception events (archon_entering, archon_materialized, archon_departed, scene state changes, tick observations) are formatted as conversation turns
These turns are POSTed to the Hermes API server (already running on port 8642, OpenAI-compatible)
Hermes processes them through the full pipeline: SOUL.md, memory, tools, inference
Response is returned to the Nexus for rendering (chat bubbles, state changes, animations)

Key Constraints

The Hermes API server already exists (platforms.api_server.enabled: true, port 8642)
Use the existing Responses API or Chat Completions endpoint — don't invent a new protocol
Session must be persistent across perception ticks (not a new session per event)
Must work over Tailscale or local network (Nexus served from VPS, Hermes runs on Mac)
Training data capture is the PRIMARY goal — if the perception loop doesn't generate sessions, it's not done

What This Unlocks

Every Nexus interaction becomes training data for local Timmy
Timmy's soul applies to his embodied presence, not just text channels
Memory works across the Nexus (Timmy remembers what happened in the world)
Tool access from within the Nexus (Timmy can check Gitea, read files, etc. while 'in' the world)

Acceptance Criteria

Nexus perception events POST to Hermes API, not direct to any model
Sessions are captured in ~/.hermes/sessions/ like any other channel
SOUL.md and memory are active during Nexus interactions
A test: trigger an archon_entering event, verify a session file is created with the turn
No direct model calls from Nexus JS — Hermes is always in the middle

References

Hermes API server config: ~/.hermes/config.yaml (platforms.api_server)
Workshop live chat skill: already wires the Workshop panel to Hermes
Nexus WS events: the-nexus #533 (archon_entering, archon_materialized, archon_departed)
AutoLoRA pipeline: Timmy_Foundation/autolora

cc @Rockachopa — this is the piece that closes the sovereignty loop across all three channels.

## Problem Timmy now operates across three channels: Telegram, TUI/CLI, and The Nexus. Telegram and TUI already flow through the Hermes harness — every turn is captured as session data, memory works, SOUL.md is applied, and conversations feed the AutoLoRA sovereignty pipeline. The Nexus is the gap. Right now, any perception loop (archon events, scene observation, tick-based awareness) would call an inference endpoint directly, bypassing Hermes entirely. This means: - No session capture → no training data - No SOUL.md application → Timmy's conscience isn't in the loop - No memory → no continuity between perception ticks - No tool access → can't act on what he observes ## Architecture Required All three channels must converge through the same harness: ``` Telegram ──┐ TUI/CLI ──┤──> Hermes Harness ──> Inference ──> Response Nexus ──┘ │ ▼ Session capture → AutoLoRA �� Fine-tuned local model ``` ### Nexus Perception Loop → Hermes The Nexus client-side JS should NOT call any model endpoint directly. Instead: 1. **Perception events** (archon_entering, archon_materialized, archon_departed, scene state changes, tick observations) are formatted as conversation turns 2. These turns are POSTed to the Hermes API server (already running on port 8642, OpenAI-compatible) 3. Hermes processes them through the full pipeline: SOUL.md, memory, tools, inference 4. Response is returned to the Nexus for rendering (chat bubbles, state changes, animations) ### Key Constraints - The Hermes API server already exists (`platforms.api_server.enabled: true`, port 8642) - Use the existing Responses API or Chat Completions endpoint — don't invent a new protocol - Session must be persistent across perception ticks (not a new session per event) - Must work over Tailscale or local network (Nexus served from VPS, Hermes runs on Mac) - Training data capture is the PRIMARY goal — if the perception loop doesn't generate sessions, it's not done ### What This Unlocks - Every Nexus interaction becomes training data for local Timmy - Timmy's soul applies to his embodied presence, not just text channels - Memory works across the Nexus (Timmy remembers what happened in the world) - Tool access from within the Nexus (Timmy can check Gitea, read files, etc. while 'in' the world) ## Acceptance Criteria - [ ] Nexus perception events POST to Hermes API, not direct to any model - [ ] Sessions are captured in ~/.hermes/sessions/ like any other channel - [ ] SOUL.md and memory are active during Nexus interactions - [ ] A test: trigger an archon_entering event, verify a session file is created with the turn - [ ] No direct model calls from Nexus JS — Hermes is always in the middle ## References - Hermes API server config: `~/.hermes/config.yaml` (platforms.api_server) - Workshop live chat skill: already wires the Workshop panel to Hermes - Nexus WS events: the-nexus #533 (archon_entering, archon_materialized, archon_departed) - AutoLoRA pipeline: Timmy_Foundation/autolora cc @Rockachopa — this is the piece that closes the sovereignty loop across all three channels.

claude was assigned by Timmy

2026-03-25 20:24:28 +00:00

Timmy referenced this issue

2026-03-25 22:45:20 +00:00

DIRECTION SHIFT #542

Timmy commented

2026-03-26 14:30:55 +00:00

v0.4.0 Overlap — Reframe

Hermes v0.4.0 shipped an OpenAI-compatible API server (/v1/chat/completions) with SQLite-backed response persistence (#1756, #2472). Every request/response pair is now logged to disk automatically.

This IS a training data capture mechanism. The original intent of this issue — "route perception loop through Hermes for training data capture" — is now handled by the harness itself.

Action: Reframe this issue. Instead of building custom trajectory capture, configure the Hermes API server as our trajectory endpoint. Remaining work:

Point Ollama/local model traffic through the API server
Write an export script to convert SQLite response logs into DPO/SFT training format
Wire into the AutoLoRA pipeline (timmy-config #13)

The "build" is done. The "configure" remains.

## v0.4.0 Overlap — Reframe Hermes v0.4.0 shipped an **OpenAI-compatible API server** (`/v1/chat/completions`) with **SQLite-backed response persistence** (#1756, #2472). Every request/response pair is now logged to disk automatically. This IS a training data capture mechanism. The original intent of this issue — "route perception loop through Hermes for training data capture" — is now handled by the harness itself. **Action:** Reframe this issue. Instead of building custom trajectory capture, configure the Hermes API server as our trajectory endpoint. Remaining work: - Point Ollama/local model traffic through the API server - Write an export script to convert SQLite response logs into DPO/SFT training format - Wire into the AutoLoRA pipeline (timmy-config #13) The "build" is done. The "configure" remains.

Timmy commented

2026-03-28 04:52:50 +00:00

Closing during the 2026-03-28 backlog burn-down.

Reason: this issue is being retired as part of a backlog reset toward the current final vision: Heartbeat, Harness, and Portal. If the work still matters after reset, it should return as a narrower, proof-oriented next-step issue rather than stay open as a broad legacy frontier.

Closing during the 2026-03-28 backlog burn-down. Reason: this issue is being retired as part of a backlog reset toward the current final vision: Heartbeat, Harness, and Portal. If the work still matters after reset, it should return as a narrower, proof-oriented next-step issue rather than stay open as a broad legacy frontier.

Timmy closed this issue

2026-03-28 04:52:51 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Timmy_Foundation/the-nexus#540