144
mcp/README.md
Normal file
144
mcp/README.md
Normal file
@@ -0,0 +1,144 @@
|
||||
# MCP Servers — Timmy's Perception & Action Layer
|
||||
|
||||
Two off-the-shelf MCP servers replace all custom perception and action code.
|
||||
Zero lines of infrastructure. `pip install` and config.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Ollama (DPO model)
|
||||
↓ tool_calls (Hermes protocol)
|
||||
↓
|
||||
MCP Client (heartbeat loop)
|
||||
├── steam-info-mcp → game perception (playtime, achievements, friends)
|
||||
└── mcp-pyautogui → desktop action (screenshot, keypress, mouse)
|
||||
```
|
||||
|
||||
The heartbeat loop is the MCP client. It:
|
||||
1. Calls `tools/list` on each MCP server at startup to discover available tools
|
||||
2. Passes tool schemas to Ollama via the `tools` parameter
|
||||
3. When the model returns `tool_calls`, executes them via `tools/call` on the right server
|
||||
4. Feeds results back to the model as `tool` role messages
|
||||
|
||||
## Servers
|
||||
|
||||
### steam-info-mcp (#545)
|
||||
|
||||
**What:** Steam Web API exposed as MCP tools.
|
||||
Timmy can see what games are installed, what's been played recently, achievements, friends, news.
|
||||
|
||||
**Package:** [steam-info-mcp](https://pypi.org/project/steam-info-mcp/)
|
||||
|
||||
**Tools available:**
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| `steam-owned-games` | List all owned games |
|
||||
| `steam-recently-played` | Recently played games + hours |
|
||||
| `steam-player-achievements` | Achievements for a game |
|
||||
| `steam-user-stats` | Player stats for a game |
|
||||
| `steam-current-players` | Live player count for a game |
|
||||
| `steam-news` | Latest news for a game |
|
||||
| `steam-player-summaries` | Player profile info |
|
||||
| `steam-friend-list` | Friends list |
|
||||
| `steam-level` | Steam level |
|
||||
| `steam-badges` | Badge collection |
|
||||
|
||||
**Requires:** `STEAM_API_KEY` env var. Get one at https://steamcommunity.com/dev/apikey
|
||||
|
||||
**Run:** `steam-info-mcp` (stdio transport)
|
||||
|
||||
### mcp-pyautogui (#546)
|
||||
|
||||
**What:** Desktop control via PyAutoGUI exposed as MCP tools.
|
||||
This IS the `execute_action()` implementation — no wrapper needed.
|
||||
|
||||
**Package:** [mcp-pyautogui](https://pypi.org/project/mcp-pyautogui/)
|
||||
|
||||
**Tools available:**
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| `take_screenshot` | Capture screen to file |
|
||||
| `click` | Left-click at (x, y) |
|
||||
| `right_click` | Right-click at (x, y) |
|
||||
| `double_click` | Double-click at (x, y) |
|
||||
| `move_to` | Move mouse to (x, y) |
|
||||
| `drag_to` | Drag mouse to (x, y) |
|
||||
| `type_text` | Type a string |
|
||||
| `press_key` | Press a single key |
|
||||
| `hotkey` | Key combo (e.g., "ctrl c") |
|
||||
| `scroll` | Scroll up/down |
|
||||
| `get_mouse_position` | Current mouse (x, y) |
|
||||
| `get_screen_size` | Screen resolution |
|
||||
| `pixel_color` | RGB at pixel (x, y) |
|
||||
| `get_os` | Current OS name |
|
||||
|
||||
**Requires:** macOS Accessibility permissions for Terminal / Python process.
|
||||
System Settings → Privacy & Security → Accessibility.
|
||||
|
||||
**Run:** `mcp-pyautogui` (stdio transport)
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
cd ~/.timmy/timmy-config/mcp
|
||||
bash setup.sh
|
||||
```
|
||||
|
||||
## How Ollama Connects
|
||||
|
||||
Both servers communicate over **stdio** — they read JSON-RPC from stdin and write to stdout.
|
||||
The heartbeat loop spawns each server as a subprocess and talks to it over pipes.
|
||||
|
||||
Ollama's native tool-calling works like this:
|
||||
|
||||
```python
|
||||
import ollama
|
||||
import subprocess, json
|
||||
|
||||
# 1. Spawn MCP server
|
||||
proc = subprocess.Popen(
|
||||
["mcp-pyautogui"],
|
||||
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE
|
||||
)
|
||||
|
||||
# 2. Discover tools (JSON-RPC over stdio)
|
||||
request = {"jsonrpc": "2.0", "id": 1, "method": "tools/list"}
|
||||
proc.stdin.write(json.dumps(request).encode() + b"\n")
|
||||
proc.stdin.flush()
|
||||
tools = json.loads(proc.stdout.readline())
|
||||
|
||||
# 3. Pass tool schemas to Ollama
|
||||
response = ollama.chat(
|
||||
model="timmy:v0.2-dpo",
|
||||
messages=[{"role": "user", "content": "Take a screenshot"}],
|
||||
tools=[...convert MCP tools to Ollama format...]
|
||||
)
|
||||
|
||||
# 4. Execute tool calls via MCP
|
||||
for call in response["message"]["tool_calls"]:
|
||||
mcp_request = {
|
||||
"jsonrpc": "2.0", "id": 2,
|
||||
"method": "tools/call",
|
||||
"params": {"name": call["function"]["name"], "arguments": call["function"]["arguments"]}
|
||||
}
|
||||
proc.stdin.write(json.dumps(mcp_request).encode() + b"\n")
|
||||
proc.stdin.flush()
|
||||
result = json.loads(proc.stdout.readline())
|
||||
```
|
||||
|
||||
This is pseudocode. The actual heartbeat loop (#547) will be ~30 lines of glue
|
||||
connecting Ollama's tool-calling API to MCP's stdio protocol. No custom infrastructure.
|
||||
|
||||
## What We Don't Own
|
||||
|
||||
- Steam API integration → `steam-info-mcp` (beta/steam-info-mcp on GitHub)
|
||||
- Desktop automation → `mcp-pyautogui` (PyAutoGUI wrapper)
|
||||
- MCP protocol → JSON-RPC 2.0 over stdio (industry standard)
|
||||
- Tool calling → Ollama native (Hermes protocol)
|
||||
- Model serving → Ollama
|
||||
|
||||
## What We Own
|
||||
|
||||
- `servers.json` — which servers to run and their env vars
|
||||
- `setup.sh` — one-command install
|
||||
- This README
|
||||
Reference in New Issue
Block a user