From c469253c820763f65312d040fb003d1fb2dacde5 Mon Sep 17 00:00:00 2001
From: Perplexity Computer <perplexity@tower.local>
Date: Wed, 25 Mar 2026 23:34:47 +0000
Subject: [PATCH] Add MCP server configs (#545, #546)

---
 mcp/README.md | 144 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 144 insertions(+)
 create mode 100644 mcp/README.md

diff --git a/mcp/README.md b/mcp/README.md
new file mode 100644
index 00000000..ffa220ef
--- /dev/null
+++ b/mcp/README.md
@@ -0,0 +1,144 @@
+# MCP Servers — Timmy's Perception & Action Layer
+
+Two off-the-shelf MCP servers replace all custom perception and action code.
+Zero lines of infrastructure. `pip install` and config.
+
+## Architecture
+
+```
+Ollama (DPO model)
+    ↓ tool_calls (Hermes protocol)
+    ↓
+MCP Client (heartbeat loop)
+    ├── steam-info-mcp     → game perception (playtime, achievements, friends)
+    └── mcp-pyautogui      → desktop action (screenshot, keypress, mouse)
+```
+
+The heartbeat loop is the MCP client. It:
+1. Calls `tools/list` on each MCP server at startup to discover available tools
+2. Passes tool schemas to Ollama via the `tools` parameter
+3. When the model returns `tool_calls`, executes them via `tools/call` on the right server
+4. Feeds results back to the model as `tool` role messages
+
+## Servers
+
+### steam-info-mcp (#545)
+
+**What:** Steam Web API exposed as MCP tools.
+Timmy can see what games are installed, what's been played recently, achievements, friends, news.
+
+**Package:** [steam-info-mcp](https://pypi.org/project/steam-info-mcp/)
+
+**Tools available:**
+| Tool | Description |
+|------|-------------|
+| `steam-owned-games` | List all owned games |
+| `steam-recently-played` | Recently played games + hours |
+| `steam-player-achievements` | Achievements for a game |
+| `steam-user-stats` | Player stats for a game |
+| `steam-current-players` | Live player count for a game |
+| `steam-news` | Latest news for a game |
+| `steam-player-summaries` | Player profile info |
+| `steam-friend-list` | Friends list |
+| `steam-level` | Steam level |
+| `steam-badges` | Badge collection |
+
+**Requires:** `STEAM_API_KEY` env var. Get one at https://steamcommunity.com/dev/apikey
+
+**Run:** `steam-info-mcp` (stdio transport)
+
+### mcp-pyautogui (#546)
+
+**What:** Desktop control via PyAutoGUI exposed as MCP tools.
+This IS the `execute_action()` implementation — no wrapper needed.
+
+**Package:** [mcp-pyautogui](https://pypi.org/project/mcp-pyautogui/)
+
+**Tools available:**
+| Tool | Description |
+|------|-------------|
+| `take_screenshot` | Capture screen to file |
+| `click` | Left-click at (x, y) |
+| `right_click` | Right-click at (x, y) |
+| `double_click` | Double-click at (x, y) |
+| `move_to` | Move mouse to (x, y) |
+| `drag_to` | Drag mouse to (x, y) |
+| `type_text` | Type a string |
+| `press_key` | Press a single key |
+| `hotkey` | Key combo (e.g., "ctrl c") |
+| `scroll` | Scroll up/down |
+| `get_mouse_position` | Current mouse (x, y) |
+| `get_screen_size` | Screen resolution |
+| `pixel_color` | RGB at pixel (x, y) |
+| `get_os` | Current OS name |
+
+**Requires:** macOS Accessibility permissions for Terminal / Python process.
+System Settings → Privacy & Security → Accessibility.
+
+**Run:** `mcp-pyautogui` (stdio transport)
+
+## Setup
+
+```bash
+cd ~/.timmy/timmy-config/mcp
+bash setup.sh
+```
+
+## How Ollama Connects
+
+Both servers communicate over **stdio** — they read JSON-RPC from stdin and write to stdout.
+The heartbeat loop spawns each server as a subprocess and talks to it over pipes.
+
+Ollama's native tool-calling works like this:
+
+```python
+import ollama
+import subprocess, json
+
+# 1. Spawn MCP server
+proc = subprocess.Popen(
+    ["mcp-pyautogui"],
+    stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE
+)
+
+# 2. Discover tools (JSON-RPC over stdio)
+request = {"jsonrpc": "2.0", "id": 1, "method": "tools/list"}
+proc.stdin.write(json.dumps(request).encode() + b"\n")
+proc.stdin.flush()
+tools = json.loads(proc.stdout.readline())
+
+# 3. Pass tool schemas to Ollama
+response = ollama.chat(
+    model="timmy:v0.2-dpo",
+    messages=[{"role": "user", "content": "Take a screenshot"}],
+    tools=[...convert MCP tools to Ollama format...]
+)
+
+# 4. Execute tool calls via MCP
+for call in response["message"]["tool_calls"]:
+    mcp_request = {
+        "jsonrpc": "2.0", "id": 2,
+        "method": "tools/call",
+        "params": {"name": call["function"]["name"], "arguments": call["function"]["arguments"]}
+    }
+    proc.stdin.write(json.dumps(mcp_request).encode() + b"\n")
+    proc.stdin.flush()
+    result = json.loads(proc.stdout.readline())
+```
+
+This is pseudocode. The actual heartbeat loop (#547) will be ~30 lines of glue
+connecting Ollama's tool-calling API to MCP's stdio protocol. No custom infrastructure.
+
+## What We Don't Own
+
+- Steam API integration → `steam-info-mcp` (beta/steam-info-mcp on GitHub)
+- Desktop automation → `mcp-pyautogui` (PyAutoGUI wrapper)
+- MCP protocol → JSON-RPC 2.0 over stdio (industry standard)
+- Tool calling → Ollama native (Hermes protocol)
+- Model serving → Ollama
+
+## What We Own
+
+- `servers.json` — which servers to run and their env vars
+- `setup.sh` — one-command install
+- This README