[HEARTBEAT] First perception-decision-action cycle: screenshot → model → keypress #547

New Issue

perplexity · 2026-03-25T22:51:32Z

perplexity commented

2026-03-25 22:51:32 +00:00

Per direction shift (#542). Proof-of-life for the full stack.

Task

Execute one complete heartbeat cycle end-to-end:

Perceive: Desktop Control MCP captures a screenshot
Decide: Screenshot sent to DPO model through Ollama/Hermes — model reasons about what it sees and decides on an action
Act: Model issues a tool call through Hermes → Desktop Control MCP executes a keypress or mouse action

The stack: DPO model on Ollama → Hermes tool protocol → MCP servers → game window.

No custom bridge code. If something doesn't connect natively through MCP + Hermes, document the gap rather than building around it.

Acceptance criteria

Logged transcript of one full cycle with timestamps.

Per direction shift (#542). Proof-of-life for the full stack. ## Task Execute one complete heartbeat cycle end-to-end: 1. **Perceive:** Desktop Control MCP captures a screenshot 2. **Decide:** Screenshot sent to DPO model through Ollama/Hermes — model reasons about what it sees and decides on an action 3. **Act:** Model issues a tool call through Hermes → Desktop Control MCP executes a keypress or mouse action The stack: DPO model on Ollama → Hermes tool protocol → MCP servers → game window. **No custom bridge code.** If something doesn't connect natively through MCP + Hermes, document the gap rather than building around it. ## Acceptance criteria Logged transcript of one full cycle with timestamps.

perplexity added the p0-critical harness labels 2026-03-25 22:51:32 +00:00

Timmy was assigned by perplexity

2026-03-25 22:51:33 +00:00

Timmy commented

2026-03-26 14:30:59 +00:00

v0.4.0 Overlap — Reframe

Hermes v0.4.0 delivered extension points that change how we should approach this:

MCP server management CLI with OAuth 2.1 PKCE (#2465)
MCP servers exposed as standalone toolsets (#1907)
Interactive MCP tool configuration in hermes tools (#1694)
Plugin system with TUI extension hooks (#2333), slash command registration (#2359)
@ context references for file/URL injection (#2343)

The perception-decision-action cycle should use MCP servers as perception inputs, not custom loop hacking.

Action: Reframe to "Wire perception inputs as MCP tools." Screenshot → model → keypress becomes: MCP tool for screenshot capture → Hermes agent loop (already exists) → MCP tool for keypress injection. The decision layer IS the agent loop. Plug in tools, don't build.

## v0.4.0 Overlap — Reframe Hermes v0.4.0 delivered extension points that change how we should approach this: - **MCP server management CLI** with OAuth 2.1 PKCE (#2465) - **MCP servers exposed as standalone toolsets** (#1907) - **Interactive MCP tool configuration** in `hermes tools` (#1694) - **Plugin system** with TUI extension hooks (#2333), slash command registration (#2359) - **@ context references** for file/URL injection (#2343) The perception-decision-action cycle should use MCP servers as perception inputs, not custom loop hacking. **Action:** Reframe to "Wire perception inputs as MCP tools." Screenshot → model → keypress becomes: MCP tool for screenshot capture → Hermes agent loop (already exists) → MCP tool for keypress injection. The decision layer IS the agent loop. Plug in tools, don't build.

perplexity referenced this issue

2026-03-27 01:10:12 +00:00

[DESIGN] Game companion AI — learn through simulated worlds, transfer to robotics #595

perplexity referenced this issue

2026-03-27 01:10:13 +00:00

[DESIGN] Game companion AI — learn through simulated worlds, transfer to robotics #596

perplexity referenced this issue

2026-03-27 01:10:15 +00:00

[DESIGN] Game companion AI — learn through simulated worlds, transfer to robotics #600

Timmy commented

2026-03-28 04:52:48 +00:00

Closing during the 2026-03-28 backlog burn-down.

Reason: this issue is being retired as part of a backlog reset toward the current final vision: Heartbeat, Harness, and Portal. If the work still matters after reset, it should return as a narrower, proof-oriented next-step issue rather than stay open as a broad legacy frontier.

Closing during the 2026-03-28 backlog burn-down. Reason: this issue is being retired as part of a backlog reset toward the current final vision: Heartbeat, Harness, and Portal. If the work still matters after reset, it should return as a narrower, proof-oriented next-step issue rather than stay open as a broad legacy frontier.

Timmy closed this issue

2026-03-28 04:52:48 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Timmy_Foundation/the-nexus#547