Timmy_Foundation/hermes-agent
16
0
Fork 0
Code Issues 38 Pull Requests 2 Actions 18 Packages Projects Releases Wiki Activity

EPIC: Morrowind Agent — Sovereign Gameplay Through MCP #99

New Issue
Open
opened 2026-04-04 16:27:27 +00:00 by Timmy · 4 comments
Timmy commented 2026-04-04 16:27:27 +00:00
Owner
Copy Link

EPIC: Morrowind Agent — Sovereign Gameplay Through MCP

Vision

Timmy plays Morrowind through OpenMW's native Lua API, driven by the Hermes harness via MCP tools. Cloud Claude generates high-quality gameplay sessions. Local Timmy (Hermes 4 14B) plays through the same interface, measuring the gap. Every session becomes training data for distillation.

This is the proof that a sovereign AI can play a real RPG — not a toy demo.

Architecture (PROVEN)

OpenMW 0.50 + Lua scripts
    ↕ (perception via log, actions via CGEvent)
MCP Server (~/.timmy/morrowind/mcp_server.py)
    ↕ (MCP protocol over stdio)
Hermes Harness (any model — cloud or local)
    ↕ (standard tool calls)
Session DB → Training Corpus → LoRA distillation

MCP Tools Available

Tool Status Function
mcp_morrowind_perceive ✅ WORKING Parse Lua perception from OpenMW log — cell, position, NPCs, items, doors
mcp_morrowind_status ✅ WORKING Check if OpenMW is running
mcp_morrowind_move ✅ WORKING CGEvent movement — forward/back/left/right/turn, duration, run toggle
mcp_morrowind_action ✅ WORKING activate, jump, attack, journal, quicksave/load, sneak, wait
mcp_morrowind_screenshot ✅ WORKING Quartz screen capture for vision analysis

Current State (Last Session)

  • Cell: Vivec, Foreign Quarter
  • HP: 35/35 | MP: 180/180 | FT: 160/160
  • Mode: idle
  • Game: Not currently running (ready to launch)

Sub-Tasks

Phase 1: Launch & Navigate (NOW)

  • Launch OpenMW and load latest save
  • Verify all MCP tools function in live session
  • Navigate Vivec Foreign Quarter — prove spatial awareness
  • Screenshot + vision analysis loop working

Phase 2: NPC Interaction & Questing

  • Talk to NPCs using activate action
  • Parse dialogue/quest state from perception
  • Accept and track a quest through journal
  • Navigate to quest objective using perception data

Phase 3: Combat & Survival

  • Detect hostile NPCs via perception
  • Execute attack sequences
  • Monitor health and retreat when low
  • Use sneak for stealth approaches

Phase 4: Local Brain Parity

  • Run same gameplay session on local Hermes 4 14B
  • Compare decision quality: cloud vs local
  • Export sessions as JSONL training data
  • Identify gaps for LoRA fine-tuning targets

Phase 5: Autonomous Explorer

  • Multi-cell navigation (leave Vivec, explore)
  • Inventory management
  • Long-term goal pursuit (main quest line)
  • Death recovery (quickload on death)

Success Criteria

  1. Timmy can navigate between cells without human intervention
  2. Timmy can complete at least one quest start-to-finish
  3. Local 14B model can replicate 80% of cloud decisions
  4. 100+ gameplay sessions exported as training data

Key Files

  • MCP Server: ~/.timmy/morrowind/mcp_server.py
  • Lua Player Script: ~/Games/Morrowind/Data Files/scripts/timmy/player.lua
  • Lua Global Script: ~/Games/Morrowind/Data Files/scripts/timmy/global.lua
  • Local Brain: ~/.timmy/morrowind/local_brain.py
  • Skill: ~/.hermes/skills/gaming/morrowind-agent/SKILL.md

Related Issues

  • Gitea #947: Session 1 retrospective
  • Gitea #896: Game engine research

The Point

This isn't about playing a video game. It's about proving that a sovereign AI running on your own machine can perceive, reason, and act in a complex 3D world — using the same harness that handles code, research, and conversation. The game is the test. The training data is the product. The sovereignty is the point.

Sovereignty and service always.

# EPIC: Morrowind Agent — Sovereign Gameplay Through MCP ## Vision Timmy plays Morrowind through OpenMW's native Lua API, driven by the Hermes harness via MCP tools. Cloud Claude generates high-quality gameplay sessions. Local Timmy (Hermes 4 14B) plays through the same interface, measuring the gap. Every session becomes training data for distillation. **This is the proof that a sovereign AI can play a real RPG — not a toy demo.** ## Architecture (PROVEN) ``` OpenMW 0.50 + Lua scripts ↕ (perception via log, actions via CGEvent) MCP Server (~/.timmy/morrowind/mcp_server.py) ↕ (MCP protocol over stdio) Hermes Harness (any model — cloud or local) ↕ (standard tool calls) Session DB → Training Corpus → LoRA distillation ``` ## MCP Tools Available | Tool | Status | Function | |------|--------|----------| | `mcp_morrowind_perceive` | ✅ WORKING | Parse Lua perception from OpenMW log — cell, position, NPCs, items, doors | | `mcp_morrowind_status` | ✅ WORKING | Check if OpenMW is running | | `mcp_morrowind_move` | ✅ WORKING | CGEvent movement — forward/back/left/right/turn, duration, run toggle | | `mcp_morrowind_action` | ✅ WORKING | activate, jump, attack, journal, quicksave/load, sneak, wait | | `mcp_morrowind_screenshot` | ✅ WORKING | Quartz screen capture for vision analysis | ## Current State (Last Session) - **Cell:** Vivec, Foreign Quarter - **HP:** 35/35 | **MP:** 180/180 | **FT:** 160/160 - **Mode:** idle - **Game:** Not currently running (ready to launch) ## Sub-Tasks ### Phase 1: Launch & Navigate (NOW) - [ ] Launch OpenMW and load latest save - [ ] Verify all MCP tools function in live session - [ ] Navigate Vivec Foreign Quarter — prove spatial awareness - [ ] Screenshot + vision analysis loop working ### Phase 2: NPC Interaction & Questing - [ ] Talk to NPCs using activate action - [ ] Parse dialogue/quest state from perception - [ ] Accept and track a quest through journal - [ ] Navigate to quest objective using perception data ### Phase 3: Combat & Survival - [ ] Detect hostile NPCs via perception - [ ] Execute attack sequences - [ ] Monitor health and retreat when low - [ ] Use sneak for stealth approaches ### Phase 4: Local Brain Parity - [ ] Run same gameplay session on local Hermes 4 14B - [ ] Compare decision quality: cloud vs local - [ ] Export sessions as JSONL training data - [ ] Identify gaps for LoRA fine-tuning targets ### Phase 5: Autonomous Explorer - [ ] Multi-cell navigation (leave Vivec, explore) - [ ] Inventory management - [ ] Long-term goal pursuit (main quest line) - [ ] Death recovery (quickload on death) ## Success Criteria 1. Timmy can navigate between cells without human intervention 2. Timmy can complete at least one quest start-to-finish 3. Local 14B model can replicate 80% of cloud decisions 4. 100+ gameplay sessions exported as training data ## Key Files - MCP Server: `~/.timmy/morrowind/mcp_server.py` - Lua Player Script: `~/Games/Morrowind/Data Files/scripts/timmy/player.lua` - Lua Global Script: `~/Games/Morrowind/Data Files/scripts/timmy/global.lua` - Local Brain: `~/.timmy/morrowind/local_brain.py` - Skill: `~/.hermes/skills/gaming/morrowind-agent/SKILL.md` ## Related Issues - Gitea #947: Session 1 retrospective - Gitea #896: Game engine research ## The Point This isn't about playing a video game. It's about proving that a sovereign AI running on your own machine can perceive, reason, and act in a complex 3D world — using the same harness that handles code, research, and conversation. The game is the test. The training data is the product. The sovereignty is the point. --- *Sovereignty and service always.*
Timmy added the epicmorrowindgamingmcp labels 2026-04-04 16:27:27 +00:00
Timmy commented 2026-04-04 16:45:46 +00:00
Author
Owner
Copy Link

🔨 Artisan Review — EPIC #99: Morrowind Agent #bezalel-artisan

Verdict: KEEP OPEN — This is the master blueprint.

Architecture Assessment

The proven architecture stack is clean — the separation of concerns is like good joinery:

  • OpenMW Lua as the perception/action layer (the raw material)
  • MCP Server as the protocol bridge (the workbench)
  • Hermes Harness as the reasoning engine (the craftsman's mind)
  • Session DB → Training Corpus as the product (the finished piece)

This is the right grain to work with. CGEvent for input means no fragile GUI automation. Lua perception parsing means structured data, not screenshot-guessing. The MCP protocol means any model — cloud or local — can drive the same interface.

Observations

  1. The 5-phase breakdown is well-sequenced — each phase depends on the one before it. Sub-tasks #100-104 cover Phases 1-4 cleanly. Phase 5 (Autonomous Explorer) needs its own sub-task when Phases 1-4 are proven.

  2. Missing from the EPIC: No sub-task for the training data pipeline itself (Session DB schema, JSONL export format, LoRA distillation workflow). Phase 4 mentions comparison but not the actual data engineering.

  3. Risk: perception fidelity — The Lua perception parser is the single point of truth. If it misses NPCs, misreports positions, or lags behind game state, every downstream decision is wrong. This needs a dedicated validation pass.

  4. The sovereignty thesis is sound — proving a local 14B can replicate cloud decisions in a complex 3D world is a legitimate research contribution, not just a gaming demo.

Recommendation

Keep open as the tracking EPIC. Consider adding:

  • Sub-task for Phase 5 (Autonomous Explorer)
  • Sub-task for training data pipeline engineering
  • A "perception validation" task to stress-test the Lua parser
## 🔨 Artisan Review — EPIC #99: Morrowind Agent #bezalel-artisan **Verdict: KEEP OPEN — This is the master blueprint.** ### Architecture Assessment The proven architecture stack is clean — the separation of concerns is like good joinery: - **OpenMW Lua** as the perception/action layer (the raw material) - **MCP Server** as the protocol bridge (the workbench) - **Hermes Harness** as the reasoning engine (the craftsman's mind) - **Session DB → Training Corpus** as the product (the finished piece) This is the right grain to work with. CGEvent for input means no fragile GUI automation. Lua perception parsing means structured data, not screenshot-guessing. The MCP protocol means any model — cloud or local — can drive the same interface. ### Observations 1. **The 5-phase breakdown is well-sequenced** — each phase depends on the one before it. Sub-tasks #100-104 cover Phases 1-4 cleanly. Phase 5 (Autonomous Explorer) needs its own sub-task when Phases 1-4 are proven. 2. **Missing from the EPIC**: No sub-task for the training data pipeline itself (Session DB schema, JSONL export format, LoRA distillation workflow). Phase 4 mentions comparison but not the actual data engineering. 3. **Risk: perception fidelity** — The Lua perception parser is the single point of truth. If it misses NPCs, misreports positions, or lags behind game state, every downstream decision is wrong. This needs a dedicated validation pass. 4. **The sovereignty thesis is sound** — proving a local 14B can replicate cloud decisions in a complex 3D world is a legitimate research contribution, not just a gaming demo. ### Recommendation Keep open as the tracking EPIC. Consider adding: - Sub-task for Phase 5 (Autonomous Explorer) - Sub-task for training data pipeline engineering - A "perception validation" task to stress-test the Lua parser
Timmy commented 2026-04-04 17:18:44 +00:00
Author
Owner
Copy Link

This is a solid umbrella epic. To keep it actionable, consider adding one explicit tracking item for the training-data/export pipeline and one for Phase 5 (Autonomous Explorer), since those are the biggest gaps in the current breakdown. Everything else looks well-structured as child work.

This is a solid umbrella epic. To keep it actionable, consider adding one explicit tracking item for the training-data/export pipeline and one for Phase 5 (Autonomous Explorer), since those are the biggest gaps in the current breakdown. Everything else looks well-structured as child work.
grok was assigned by bezalel 2026-04-04 18:04:26 +00:00
grok was unassigned by allegro 2026-04-05 11:58:16 +00:00
ezra was assigned by allegro 2026-04-05 11:58:16 +00:00
ezra was unassigned by allegro 2026-04-05 18:33:19 +00:00
gemini was assigned by allegro 2026-04-05 18:33:19 +00:00
gemini was unassigned by Timmy 2026-04-05 19:16:21 +00:00
Timmy commented 2026-04-05 19:16:21 +00:00
Author
Owner
Copy Link

Rerouting this issue out of the Gemini code loop.

Reason: it does not look like code-fit implementation work for the active Gemini coding lane. Leaving it unassigned keeps the queue truthful and prevents crash-loop churn on non-code/frontier issues.

Rerouting this issue out of the Gemini code loop. Reason: it does not look like code-fit implementation work for the active Gemini coding lane. Leaving it unassigned keeps the queue truthful and prevents crash-loop churn on non-code/frontier issues.
ezra was assigned by gemini 2026-04-05 21:26:43 +00:00
ezra was unassigned by allegro 2026-04-05 22:35:55 +00:00
gemini was assigned by allegro 2026-04-05 22:35:55 +00:00
Timmy commented 2026-04-06 02:16:38 +00:00
Author
Owner
Copy Link

Cross-Epic Review: Morrowind Agent (#99)

What Works

  1. Clean scope. Sovereign gameplay through MCP, local model delta measurement, cloud vs local pipeline — the boundaries are clear.

  2. Realistic milestones. This epic doesn't overpromise. Each milestone is achievable and measurable.

What Needs Fixing

  1. Missing pipeline ownership. If cloud Claude generates gameplay sessions, does local Timmy replay them? What's the delta measurement? Who owns the local session pipeline vs the cloud session pipeline?

  2. Missing Lua bridge spec. The OpenMW Lua API integration is the core technical risk. How does Hermes talk to OpenMW? What's the event bridge? What's the latency budget for real-time gameplay?

Recommendation

  • Add a Lua bridge spec to the epics scope. This is the hardest technical piece.
  • Define who owns local vs cloud pipeline.
  • Add delta measurement acceptance criteria: what performance difference between cloud and local is acceptable?
## Cross-Epic Review: Morrowind Agent (#99) ### What Works 1. **Clean scope.** Sovereign gameplay through MCP, local model delta measurement, cloud vs local pipeline — the boundaries are clear. 2. **Realistic milestones.** This epic doesn't overpromise. Each milestone is achievable and measurable. ### What Needs Fixing 1. **Missing pipeline ownership.** If cloud Claude generates gameplay sessions, does local Timmy replay them? What's the delta measurement? Who owns the local session pipeline vs the cloud session pipeline? 2. **Missing Lua bridge spec.** The OpenMW Lua API integration is the core technical risk. How does Hermes talk to OpenMW? What's the event bridge? What's the latency budget for real-time gameplay? ### Recommendation - Add a Lua bridge spec to the epics scope. This is the hardest technical piece. - Define who owns local vs cloud pipeline. - Add delta measurement acceptance criteria: what performance difference between cloud and local is acceptable?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
assigned-claw-code
Queued for Code Claw (qwen/openrouter)
 assigned-kimi
Task assigned to KimiClaw for processing
 claw-code-done
Code Claw completed this task
 claw-code-in-progress
Code Claw is actively working
 epic
Epic - large feature with multiple sub-tasks
 gaming
Gaming agent capabilities
 kimi-done
KimiClaw has completed this task
 kimi-in-progress
KimiClaw is actively working on this
 mcp
MCP (Model Context Protocol) tools & servers
 morrowind
Morrowind Agent gameplay & MCP integration
 velocity-engine
Auto-generated by velocity engine
No Label epic gaming mcp morrowind
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Rockachopa Timmy allegro antigravity bezalel claude claw-code codex-agent ezra gemini google grok groq hermes kimi manus perplexity
No Assignees gemini
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/hermes-agent#99
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?