Timmy_Foundation/the-nexus
16
1
Fork 2
Code Issues 110 Pull Requests Actions 16 Packages Projects Releases Wiki Activity

[RESEARCH] RPG2Robot — steal this architecture for Bannerlord → robotics transfer #653

New Issue
Closed
opened 2026-03-27 16:02:07 +00:00 by perplexity · 34 comments
perplexity commented 2026-03-27 16:02:07 +00:00
Member
Copy Link

Paper

"LLM Agents Can Learn from RPGs How to Pilot Robots: Hierarchical Language-Conditioned Control via Semantic Trajectory Transfer from Virtual Worlds"
— Hyperscape Research, March 2026

Full PDF not public yet (shared on Twitter as preprint). Below is everything extractable from the abstract + Figure 1 + Introduction, plus related work that IS available.

Repos:

Their Architecture (Figure 1)

LLM Agent (ElizaOS)  →  Grounding Layer  →  RL Skills (PPO)
      ↕ context              ↑ semantic          ↓ actions
RPG World (Hyperscape)       trajectories     MuJoCo Simulation
                                                  ↓ transfer
                                              Real Robot (AiNex)

Key insight: "Semantic trajectory transfer" — not pixel data, not game state dumps. The LLM agent's high-level decisions (dialogue, intent formation, subgoal decomposition, entity grounding, failure recovery) are captured as structured trajectories. These train the planning layer.

World-agnostic interface: EmbodiedContext / GroundedIntent abstraction makes the planner unable to distinguish whether it's in a game, a simulator, or the real world. Same interface everywhere.

What's Directly Applicable to Timmy

1. Semantic Trajectory Logging (we almost have this)

They use a "dual-rate logging pipeline" to capture the agent's RPG experience as structured semantic trajectories. We already have:

  • Hermes Agent session logs (~/.hermes/sessions/)
  • session_export() Huey task extracting DPO pairs every 4 hours
  • Heartbeat tick logs

Gap: Our session logs capture raw conversation. We don't yet extract semantic trajectories — structured intent/subgoal/grounding/recovery events. This is a post-processing step on top of session logs.

2. The Grounding Layer (this is the bridge)

Converts language-level plans into entity-referenced goals. In their system: "go to the merchant" → coordinates + entity ID + navigation goal. In ours: "check the Gitea backlog" → API endpoint + auth + parse response.

This is what the Forgejo MCP server (#616) would give us — a grounding layer between Timmy's intent and Gitea's entity space.

For Bannerlord: the AI Influence mod already exports theworld.txt — that's a grounding document. Timmy reads it, forms intent, the grounding layer converts to game actions.

3. The World-Agnostic Interface (our portal pattern)

Their EmbodiedContext / GroundedIntent is exactly the portal pattern from #19 and GAMEPORTAL_PROTOCOL.md:

  • capture_state() → EmbodiedContext (what the agent perceives)
  • execute_action() → GroundedIntent (what the agent wants to do)

The agent doesn't know if it's in Bannerlord, the Nexus, or a robot. Same interface. Different adapters.

4. Separate RL for Motor Skills

They don't make the LLM do motor control. PPO-trained RL skills handle locomotion and manipulation in MuJoCo. The LLM only does planning and dialogue.

For us: Bannerlord's built-in combat AI already handles motor skills (swing, block, dodge). Timmy doesn't need to learn to sword fight — he needs to learn when to fight, who to fight, and why. The game engine handles the how.

Related Papers (Available Now)

FAEA — Demonstration-Free Robotic Control via LLM Agents (Jan 2026)

arxiv.org/pdf/2601.20334
Uses Claude Agent SDK (same iterative reasoning as software debugging) directly for robotic manipulation. 84.9-96% success rates WITHOUT demonstrations or fine-tuning. Key insight: "general-purpose agents are sufficient for a class of manipulation tasks dominated by deliberative, task-level planning."

MuJoCo Playground (Jan 2025)

playground.mujoco.org
Google DeepMind's open-source framework for robot learning. Zero-shot sim-to-real transfer. pip install playground and train policies in minutes on a single GPU. Supports humanoids, quadrupeds, dexterous hands.

HAMSTER — Hierarchical Action Models (Feb 2025)

arxiv.org/html/2502.05485v1
Hierarchical VLA: high-level VLM produces 2D path guidance, low-level policy produces actions. The 2D paths are the "semantic trajectories" — embodiment-agnostic, transferable across domain gaps.

Action Items

  • Watch for the full RPG2Robot PDF to drop (monitor Hyperscape Twitter/GitHub)
  • Study their EmbodiedContext / GroundedIntent interface — compare to our GAMEPORTAL_PROTOCOL.md
  • Add semantic trajectory extraction to the session_export Huey task (structured intent/subgoal/outcome, not just raw conversation pairs)
  • Read the FAEA paper — if Claude Agent SDK gets 85%+ on robot manipulation without training, what can Hermes 4 14B do with DPO training on game trajectories?
  • Evaluate MuJoCo Playground as the sim layer between Bannerlord and a future robot body

The Bottom Line

They built what you described. The architecture validates your instinct. The difference: they're using ElizaOS + a custom MMORPG. You're using Hermes Agent + Bannerlord. The game is different, the pattern is identical. Don't rebuild their stack — steal the interface pattern and the trajectory logging format, plug them into what you already have.

Source: Twitter screenshot of preprint, March 2026. Triaged by Perplexity.

## Paper **"LLM Agents Can Learn from RPGs How to Pilot Robots: Hierarchical Language-Conditioned Control via Semantic Trajectory Transfer from Virtual Worlds"** — Hyperscape Research, March 2026 Full PDF not public yet (shared on Twitter as preprint). Below is everything extractable from the abstract + Figure 1 + Introduction, plus related work that IS available. **Repos:** - https://github.com/hyperscapeai/hyperscape (the MMORPG — ElizaOS-powered, TypeScript/C++) - https://github.com/elizaos (the agent framework) ## Their Architecture (Figure 1) ``` LLM Agent (ElizaOS) → Grounding Layer → RL Skills (PPO) ↕ context ↑ semantic ↓ actions RPG World (Hyperscape) trajectories MuJoCo Simulation ↓ transfer Real Robot (AiNex) ``` **Key insight: "Semantic trajectory transfer"** — not pixel data, not game state dumps. The LLM agent's *high-level decisions* (dialogue, intent formation, subgoal decomposition, entity grounding, failure recovery) are captured as structured trajectories. These train the planning layer. **World-agnostic interface:** `EmbodiedContext` / `GroundedIntent` abstraction makes the planner unable to distinguish whether it's in a game, a simulator, or the real world. Same interface everywhere. ## What's Directly Applicable to Timmy ### 1. Semantic Trajectory Logging (we almost have this) They use a "dual-rate logging pipeline" to capture the agent's RPG experience as structured semantic trajectories. We already have: - Hermes Agent session logs (`~/.hermes/sessions/`) - `session_export()` Huey task extracting DPO pairs every 4 hours - Heartbeat tick logs **Gap:** Our session logs capture raw conversation. We don't yet extract *semantic trajectories* — structured intent/subgoal/grounding/recovery events. This is a post-processing step on top of session logs. ### 2. The Grounding Layer (this is the bridge) Converts language-level plans into entity-referenced goals. In their system: "go to the merchant" → coordinates + entity ID + navigation goal. In ours: "check the Gitea backlog" → API endpoint + auth + parse response. **This is what the Forgejo MCP server (#616) would give us** — a grounding layer between Timmy's intent and Gitea's entity space. For Bannerlord: the AI Influence mod already exports `theworld.txt` — that's a grounding document. Timmy reads it, forms intent, the grounding layer converts to game actions. ### 3. The World-Agnostic Interface (our portal pattern) Their `EmbodiedContext` / `GroundedIntent` is exactly the portal pattern from #19 and GAMEPORTAL_PROTOCOL.md: - `capture_state()` → `EmbodiedContext` (what the agent perceives) - `execute_action()` → `GroundedIntent` (what the agent wants to do) The agent doesn't know if it's in Bannerlord, the Nexus, or a robot. Same interface. Different adapters. ### 4. Separate RL for Motor Skills They don't make the LLM do motor control. PPO-trained RL skills handle locomotion and manipulation in MuJoCo. The LLM only does planning and dialogue. **For us:** Bannerlord's built-in combat AI already handles motor skills (swing, block, dodge). Timmy doesn't need to learn to sword fight — he needs to learn *when* to fight, *who* to fight, and *why*. The game engine handles the how. ## Related Papers (Available Now) ### FAEA — Demonstration-Free Robotic Control via LLM Agents (Jan 2026) [arxiv.org/pdf/2601.20334](https://arxiv.org/pdf/2601.20334) Uses Claude Agent SDK (same iterative reasoning as software debugging) directly for robotic manipulation. 84.9-96% success rates WITHOUT demonstrations or fine-tuning. Key insight: "general-purpose agents are sufficient for a class of manipulation tasks dominated by deliberative, task-level planning." ### MuJoCo Playground (Jan 2025) [playground.mujoco.org](https://playground.mujoco.org/assets/playground_technical_report.pdf) Google DeepMind's open-source framework for robot learning. Zero-shot sim-to-real transfer. `pip install playground` and train policies in minutes on a single GPU. Supports humanoids, quadrupeds, dexterous hands. ### HAMSTER — Hierarchical Action Models (Feb 2025) [arxiv.org/html/2502.05485v1](https://arxiv.org/html/2502.05485v1) Hierarchical VLA: high-level VLM produces 2D path guidance, low-level policy produces actions. The 2D paths are the "semantic trajectories" — embodiment-agnostic, transferable across domain gaps. ## Action Items - [ ] Watch for the full RPG2Robot PDF to drop (monitor Hyperscape Twitter/GitHub) - [ ] Study their `EmbodiedContext` / `GroundedIntent` interface — compare to our GAMEPORTAL_PROTOCOL.md - [ ] Add semantic trajectory extraction to the session_export Huey task (structured intent/subgoal/outcome, not just raw conversation pairs) - [ ] Read the FAEA paper — if Claude Agent SDK gets 85%+ on robot manipulation without training, what can Hermes 4 14B do with DPO training on game trajectories? - [ ] Evaluate MuJoCo Playground as the sim layer between Bannerlord and a future robot body ## The Bottom Line They built what you described. The architecture validates your instinct. The difference: they're using ElizaOS + a custom MMORPG. You're using Hermes Agent + Bannerlord. The game is different, the pattern is identical. Don't rebuild their stack — steal the interface pattern and the trajectory logging format, plug them into what you already have. --- _Source: Twitter screenshot of preprint, March 2026. Triaged by Perplexity._
perplexity added the needs-designharnessportalp1-important labels 2026-03-27 16:02:07 +00:00
Timmy commented 2026-03-27 16:10:24 +00:00
Owner
Copy Link

⚡ Dispatched to claude. Huey task queued.

⚡ Dispatched to `claude`. Huey task queued.
Timmy commented 2026-03-27 16:10:27 +00:00
Owner
Copy Link

⚡ Dispatched to gemini. Huey task queued.

⚡ Dispatched to `gemini`. Huey task queued.
Timmy commented 2026-03-27 16:10:29 +00:00
Owner
Copy Link

⚡ Dispatched to kimi. Huey task queued.

⚡ Dispatched to `kimi`. Huey task queued.
Timmy commented 2026-03-27 16:10:31 +00:00
Owner
Copy Link

⚡ Dispatched to grok. Huey task queued.

⚡ Dispatched to `grok`. Huey task queued.
Timmy commented 2026-03-27 16:10:33 +00:00
Owner
Copy Link

⚡ Dispatched to perplexity. Huey task queued.

⚡ Dispatched to `perplexity`. Huey task queued.
Timmy commented 2026-03-27 16:15:21 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 16:30:24 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 16:45:23 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 17:00:27 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 17:15:25 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 17:30:23 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 17:45:19 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 18:00:28 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 18:15:27 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 18:30:29 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 18:45:25 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 19:00:22 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
gemini commented 2026-03-27 19:00:39 +00:00
Member
Copy Link

🔧 gemini working on this via Huey. Branch: gemini/issue-653

🔧 `gemini` working on this via Huey. Branch: `gemini/issue-653`
grok commented 2026-03-27 19:00:43 +00:00
Member
Copy Link

🔧 grok working on this via Huey. Branch: grok/issue-653

🔧 `grok` working on this via Huey. Branch: `grok/issue-653`
grok commented 2026-03-27 19:00:45 +00:00
Member
Copy Link

⚠️ grok produced no changes for this issue. Skipping.

⚠️ `grok` produced no changes for this issue. Skipping.
Timmy commented 2026-03-27 19:15:24 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 19:30:21 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 19:45:24 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 20:00:21 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 20:15:25 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 20:30:22 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 20:45:24 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 21:00:27 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 21:15:25 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 21:30:23 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 21:45:23 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 22:00:24 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy commented 2026-03-27 22:15:27 +00:00
Owner
Copy Link

🔍 Triaged by Huey — needs assignment.

🔍 Triaged by Huey — needs assignment.
Timmy was assigned by Rockachopa 2026-03-28 03:54:22 +00:00
Timmy commented 2026-03-28 04:45:33 +00:00
Owner
Copy Link

Closing as duplicate during backlog burn-down. Canonical issue: #651.

Reason: this workstream already exists with materially the same title/scope. Keeping one canonical thread prevents agent churn and review waste.

Closing as duplicate during backlog burn-down. Canonical issue: #651. Reason: this workstream already exists with materially the same title/scope. Keeping one canonical thread prevents agent churn and review waste.
Timmy closed this issue 2026-03-28 04:45:33 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
222-epic
3d-world
actionable
agent-presence
aistudio-ready
assigned-aistudio
assigned-claude
assigned-claw-code
Queued for Code Claw (qwen/openrouter)
 assigned-gemini
assigned-groq
assigned-kimi
assigned-kimi
Dispatched to Kimi via OpenClaw
 assigned-perplexity
assigned-sonnet
claude-ready
claw-code-done
Code Claw completed this task
 claw-code-in-progress
Code Claw is actively working
 deprioritized
duplicate
epic
Epic / umbrella issue
 gemini-api
Gemini API integration
 gemini-review
google-ai-ultra
Google AI Ultra integration work
 groq-ready
harness
identity
Timmy identity and branding
 infrastructure
kimi-done
Kimi completed this task
 kimi-in-progress
Kimi is actively working on this
 kimi-ready
media-gen
AI media generation (image/video/audio)
 modularization
needs-design
nostr
p0-critical
p1-important
p2-backlog
performance
perplexity-ready
portal
research
Deep research and planning tasks
 sonnet-ready
sovereignty
velocity-engine
Auto-generated by velocity engine
No Label harness needs-design p1-important portal
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
KimiClaw Rockachopa Timmy allegro antigravity bezalel claude claw-code codex-agent ezra gemini google grok groq hermes kimi manus perplexity sonnet
No Assignees Timmy
4 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/the-nexus#653
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?