kimi/Timmy-time-dashboard
Archived
1
0
Fork 0
forked from Rockachopa/Timmy-time-dashboard
Code Pull Requests Activity

Operation Darling Purge: slim to wealth core (-33,783 lines) (#121)

Browse Source
This commit is contained in:
Alexander Whitestone
2026-03-02 13:17:38 -05:00
committed by GitHub
parent f694eff0a4
commit 584eeb679e
183 changed files with 154 additions and 33807 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
CLAUDE.md
View File

@@ -22,7 +22,6 @@ url = settings.ollama_url # never use os.environ.get() directly in app code
from dashboard.store import message_log
from infrastructure.notifications.push import notifier
from infrastructure.ws_manager.handler import ws_manager
from swarm.coordinator import coordinator
```
### HTMX response pattern
@@ -77,9 +76,7 @@ make test-cov # With coverage (term-missing + XML)
## Security-Sensitive Areas
- `src/swarm/coordinator.py` — requires review before changes
- `src/timmy_serve/l402_proxy.py` — Lightning payment gating
- `src/lightning/` — payment backend abstraction
- `src/timmy_serve/l402_proxy.py` — Payment gating
- Any file handling secrets or authentication tokens
---
@@ -89,27 +86,19 @@ make test-cov # With coverage (term-missing + XML)
| Command | Module | Purpose |
|---------|--------|---------|
| `timmy` | `src/timmy/cli.py` | Chat, think, status |
| `timmy-serve` | `src/timmy_serve/cli.py` | L402-gated API server (port 8402) |
| `self-tdd` | `src/self_coding/self_tdd/watchdog.py` | Continuous test watchdog |
| `self-modify` | `src/self_coding/self_modify/cli.py` | Self-modification CLI |
| `timmy-serve` | `src/timmy_serve/cli.py` | API server (port 8402) |
---
## Module Map (14 packages)
## Module Map (8 packages)
| Package | Purpose |
|---------|---------|
| `timmy/` | Core agent, personas, agent interface, semantic memory |
| `dashboard/` | FastAPI web UI, routes, templates |
| `swarm/` | Multi-agent coordinator, task queue, work orders |
| `self_coding/` | Self-modification, test watchdog, upgrade queue |
| `creative/` | Media generation, MCP tools |
| `infrastructure/` | WebSocket, notifications, events, LLM router |
| `integrations/` | Discord, Telegram, Siri Shortcuts, voice NLU |
| `lightning/` | L402 payment gating (security-sensitive) |
| `mcp/` | MCP tool registry and discovery |
| `spark/` | Event capture and advisory engine |
| `hands/` | 6 autonomous Hand agents |
| `scripture/` | Biblical text integration |
| `timmy_serve/` | L402-gated API server |
| `brain/` | Identity system, memory interface |
| `timmy_serve/` | API server |
| `config.py` | Pydantic settings (foundation for all modules) |

5534
coverage.xml
View File

File diff suppressed because it is too large Load Diff

127
docker-compose.logging.yml
View File

@@ -1,127 +0,0 @@
# ── Timmy Time — ELK Log Aggregation Overlay ────────────────────────────────
#
# Adds Elasticsearch + Logstash + Kibana alongside the production stack.
# Use as an overlay on top of the prod compose:
#
# docker compose \
# -f docker-compose.prod.yml \
# -f docker-compose.logging.yml \
# up -d
#
# ── How it works ────────────────────────────────────────────────────────────
#
# 1. Every container's Docker logging driver is set to GELF, which sends
# structured log events (JSON with container metadata) over UDP.
#
# 2. Logstash listens on :12201/udp, parses the GELF messages, extracts
# log levels, parses JSON payloads from FastAPI/uvicorn, and adds
# project metadata.
#
# 3. Logstash ships the enriched events to Elasticsearch, indexed by day
# (timmy-logs-YYYY.MM.dd) with a 30-day ILM retention policy.
#
# 4. Kibana provides the web UI on :5601 for searching, filtering,
# and building dashboards over the indexed logs.
#
# ── Access ──────────────────────────────────────────────────────────────────
# Kibana: http://localhost:5601
# Elasticsearch: http://localhost:9200 (API only, not exposed by default)
#
# ── Resource notes ──────────────────────────────────────────────────────────
# Elasticsearch: ~512 MB heap (ES_JAVA_OPTS below). Increase for
# high-throughput deployments.
# Logstash: ~256 MB heap. Lightweight for GELF → ES pipeline.
# Kibana: ~300 MB RAM. Stateless — safe to restart anytime.
#
# Total overhead: ~1.1 GB RAM on top of the base production stack.
services:
# ── Elasticsearch — log storage and search engine ─────────────────────────
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:8.17.0
container_name: timmy-elasticsearch
volumes:
- es-data:/usr/share/elasticsearch/data
- ./deploy/elk/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
environment:
ES_JAVA_OPTS: "-Xms512m -Xmx512m"
networks:
- swarm-net
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "curl -sf http://localhost:9200/_cluster/health | grep -q '\"status\":\"green\\|yellow\"'"]
interval: 30s
timeout: 10s
retries: 5
start_period: 60s
# ── Logstash — log pipeline (GELF in → Elasticsearch out) ────────────────
logstash:
image: docker.elastic.co/logstash/logstash:8.17.0
container_name: timmy-logstash
volumes:
- ./deploy/elk/logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro
environment:
LS_JAVA_OPTS: "-Xms256m -Xmx256m"
ports:
- "12201:12201/udp" # GELF input from Docker logging driver
networks:
- swarm-net
depends_on:
elasticsearch:
condition: service_healthy
restart: unless-stopped
# ── Kibana — log visualisation UI ─────────────────────────────────────────
kibana:
image: docker.elastic.co/kibana/kibana:8.17.0
container_name: timmy-kibana
volumes:
- ./deploy/elk/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
ports:
- "5601:5601"
networks:
- swarm-net
depends_on:
elasticsearch:
condition: service_healthy
restart: unless-stopped
# ── Override existing services to use GELF logging driver ─────────────────
# These extend the services defined in docker-compose.prod.yml.
# Docker merges the logging config into the existing service definition.
dashboard:
logging:
driver: gelf
options:
gelf-address: "udp://localhost:12201"
tag: "dashboard"
depends_on:
logstash:
condition: service_started
timmy:
logging:
driver: gelf
options:
gelf-address: "udp://localhost:12201"
tag: "timmy-agent"
depends_on:
logstash:
condition: service_started
ollama:
logging:
driver: gelf
options:
gelf-address: "udp://localhost:12201"
tag: "ollama"
depends_on:
logstash:
condition: service_started
# ── Persistent volume for Elasticsearch indices ────────────────────────────
volumes:
es-data:

166
docker-compose.microservices.yml
View File

@@ -1,166 +0,0 @@
# ── Timmy Time — Microservices Architecture ──────────────────────────────────
#
# Clean separation of concerns with independent, scalable services:
# - ollama LLM inference engine
# - dashboard FastAPI coordinator + UI
# - timmy Sovereign AI agent
# - workers Swarm worker pool (scale with --scale worker=N)
#
# Usage:
# docker compose -f docker-compose.microservices.yml up -d
# docker compose -f docker-compose.microservices.yml logs -f dashboard
# docker compose -f docker-compose.microservices.yml up --scale worker=4
services:
# ── Ollama LLM Service ────────────────────────────────────────────────────
ollama:
build:
context: .
dockerfile: docker/Dockerfile.ollama
image: timmy-ollama:latest
container_name: timmy-ollama
ports:
- "11434:11434"
volumes:
- ollama-data:/root/.ollama
environment:
OLLAMA_HOST: "0.0.0.0:11434"
networks:
- timmy-net
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
interval: 30s
timeout: 5s
retries: 3
start_period: 30s
# ── Dashboard Service ─────────────────────────────────────────────────────
dashboard:
build:
context: .
dockerfile: docker/Dockerfile.dashboard
image: timmy-dashboard:latest
container_name: timmy-dashboard
ports:
- "8000:8000"
volumes:
- timmy-data:/app/data
- ./src:/app/src
- ./static:/app/static
environment:
DEBUG: "true"
OLLAMA_URL: "http://ollama:11434"
GROK_ENABLED: "${GROK_ENABLED:-false}"
XAI_API_KEY: "${XAI_API_KEY:-}"
GROK_DEFAULT_MODEL: "${GROK_DEFAULT_MODEL:-grok-3-fast}"
networks:
- timmy-net
depends_on:
ollama:
condition: service_healthy
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 5s
retries: 3
start_period: 15s
# ── Timmy Agent Service ───────────────────────────────────────────────────
timmy:
build:
context: .
dockerfile: docker/Dockerfile.agent
image: timmy-agent:latest
container_name: timmy-agent
volumes:
- timmy-data:/app/data
- ./src:/app/src
environment:
COORDINATOR_URL: "http://dashboard:8000"
OLLAMA_URL: "http://ollama:11434"
TIMMY_AGENT_ID: "timmy"
networks:
- timmy-net
depends_on:
dashboard:
condition: service_healthy
ollama:
condition: service_healthy
restart: unless-stopped
# ── OpenFang — vendored agent runtime sidecar ────────────────────────────────
# Rust binary providing real tool execution (browser, OSINT, forecasting).
# Timmy's coordinator delegates hand execution here via REST API.
openfang:
build:
context: .
dockerfile: docker/Dockerfile.openfang
image: timmy-openfang:latest
container_name: timmy-openfang
profiles:
- openfang
environment:
OLLAMA_URL: "http://ollama:11434"
OPENFANG_DATA_DIR: "/app/data"
volumes:
- openfang-data:/app/data
networks:
- timmy-net
depends_on:
ollama:
condition: service_healthy
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
interval: 30s
timeout: 5s
retries: 3
start_period: 15s
# ── Swarm Worker Pool (Template) ──────────────────────────────────────────
# Scale: docker compose -f docker-compose.microservices.yml up --scale worker=4
worker:
build:
context: .
dockerfile: docker/Dockerfile.agent
image: timmy-agent:latest
profiles:
- workers
volumes:
- timmy-data:/app/data
- ./src:/app/src
environment:
COORDINATOR_URL: "http://dashboard:8000"
OLLAMA_URL: "http://ollama:11434"
AGENT_NAME: "Worker"
AGENT_CAPABILITIES: "general,reasoning,coding"
command: ["sh", "-c", "python -m swarm.agent_runner --agent-id worker-$(hostname) --name Worker"]
networks:
- timmy-net
depends_on:
dashboard:
condition: service_healthy
ollama:
condition: service_healthy
restart: unless-stopped
# ── Volumes ───────────────────────────────────────────────────────────────────
volumes:
timmy-data:
driver: local
driver_opts:
type: none
o: bind
device: "${PWD}/data"
ollama-data:
driver: local
openfang-data:
driver: local
# ── Network ───────────────────────────────────────────────────────────────────
networks:
timmy-net:
driver: bridge

152
docker-compose.prod.yml
View File

@@ -1,152 +0,0 @@
# ── Timmy Time — Production Stack ────────────────────────────────────────────
#
# One-click cloud deployment. Includes:
# - Caddy auto-HTTPS reverse proxy (Let's Encrypt)
# - Dashboard FastAPI app + swarm coordinator
# - Timmy sovereign AI agent
# - Ollama local LLM inference engine
# - Watchtower auto-updates containers when images change
#
# Usage:
# cp .env.example .env # edit with your domain + secrets
# docker compose -f docker-compose.prod.yml up -d
#
# Scale agents:
# docker compose -f docker-compose.prod.yml --profile agents up -d --scale agent=4
services:
# ── Caddy — automatic HTTPS reverse proxy ──────────────────────────────────
caddy:
image: caddy:2-alpine
container_name: timmy-caddy
ports:
- "80:80"
- "443:443"
- "443:443/udp" # HTTP/3
volumes:
- ./deploy/Caddyfile:/etc/caddy/Caddyfile:ro
- caddy-data:/data
- caddy-config:/config
environment:
DOMAIN: "${DOMAIN:-localhost}"
networks:
- swarm-net
restart: unless-stopped
# ── Ollama — local LLM inference ───────────────────────────────────────────
ollama:
image: ollama/ollama:latest
container_name: timmy-ollama
volumes:
- ollama-models:/root/.ollama
networks:
- swarm-net
restart: unless-stopped
# GPU passthrough (uncomment for NVIDIA GPU)
# deploy:
# resources:
# reservations:
# devices:
# - driver: nvidia
# count: all
# capabilities: [gpu]
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
interval: 30s
timeout: 10s
retries: 5
start_period: 30s
# ── Dashboard (coordinator + FastAPI) ──────────────────────────────────────
dashboard:
build: .
image: timmy-time:latest
container_name: timmy-dashboard
volumes:
- timmy-data:/app/data
environment:
DEBUG: "${DEBUG:-false}"
OLLAMA_URL: "http://ollama:11434"
OLLAMA_MODEL: "${OLLAMA_MODEL:-llama3.2}"
L402_HMAC_SECRET: "${L402_HMAC_SECRET:-}"
L402_MACAROON_SECRET: "${L402_MACAROON_SECRET:-}"
TELEGRAM_TOKEN: "${TELEGRAM_TOKEN:-}"
networks:
- swarm-net
depends_on:
ollama:
condition: service_healthy
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 5s
retries: 3
start_period: 15s
# ── Timmy — sovereign AI agent ─────────────────────────────────────────────
timmy:
build: .
image: timmy-time:latest
container_name: timmy-agent
volumes:
- timmy-data:/app/data
environment:
COORDINATOR_URL: "http://dashboard:8000"
OLLAMA_URL: "http://ollama:11434"
OLLAMA_MODEL: "${OLLAMA_MODEL:-llama3.2}"
TIMMY_AGENT_ID: "timmy"
command: ["python", "-m", "timmy.docker_agent"]
networks:
- swarm-net
depends_on:
dashboard:
condition: service_healthy
restart: unless-stopped
# ── Agent worker template ──────────────────────────────────────────────────
agent:
build: .
image: timmy-time:latest
profiles:
- agents
volumes:
- timmy-data:/app/data
environment:
COORDINATOR_URL: "http://dashboard:8000"
OLLAMA_URL: "http://ollama:11434"
OLLAMA_MODEL: "${OLLAMA_MODEL:-llama3.2}"
AGENT_NAME: "${AGENT_NAME:-Worker}"
AGENT_CAPABILITIES: "${AGENT_CAPABILITIES:-general}"
command: ["sh", "-c", "python -m swarm.agent_runner --agent-id agent-$(hostname) --name $${AGENT_NAME:-Worker}"]
networks:
- swarm-net
depends_on:
dashboard:
condition: service_healthy
restart: unless-stopped
# ── Watchtower — auto-update containers ────────────────────────────────────
watchtower:
image: containrrr/watchtower
container_name: timmy-watchtower
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
WATCHTOWER_CLEANUP: "true"
WATCHTOWER_POLL_INTERVAL: "3600" # check every hour
WATCHTOWER_LABEL_ENABLE: "false"
restart: unless-stopped
# ── Volumes ──────────────────────────────────────────────────────────────────
volumes:
timmy-data:
caddy-data:
caddy-config:
ollama-models:
# ── Network ──────────────────────────────────────────────────────────────────
networks:
swarm-net:
driver: bridge

157
docker-compose.test.yml
View File

@@ -1,157 +0,0 @@
# ── Timmy Time — Test Stack ──────────────────────────────────────────────────
#
# Clean containers for test runs. Designed for fast iteration:
# • Cached builder layers — only rebuilds when pyproject.toml changes
# • Bind-mounted source — code changes are instant, no rebuild needed
# • Ephemeral test-data — every run starts with clean state
#
# ── Profiles ────────────────────────────────────────────────────────────────
# (default) test runner only (unit + integration tests)
# functional adds a live dashboard on port 18000 for HTTP-level tests
# ollama adds containerised Ollama (CPU, qwen2.5:0.5b) for LLM tests
# agents adds swarm agent workers for multi-agent tests
#
# ── Quick-start ─────────────────────────────────────────────────────────────
# make test-docker # unit + integration in container
# make test-docker ARGS="-k swarm" # filter tests
# make test-docker-functional # full-stack functional tests
# make test-docker-cov # with coverage report
#
# ── Manual usage ────────────────────────────────────────────────────────────
# docker compose -f docker-compose.test.yml run --rm test
# docker compose -f docker-compose.test.yml run --rm test pytest tests/swarm -v
# docker compose -f docker-compose.test.yml --profile functional up -d --wait
# docker compose -f docker-compose.test.yml down -v
services:
# ── Test Runner ───────────────────────────────────────────────────────────
# Runs pytest in a clean container. Exits when tests complete.
# Source and tests are bind-mounted so code changes don't require a rebuild.
test:
build:
context: .
dockerfile: docker/Dockerfile.test
cache_from:
- timmy-test:latest
image: timmy-test:latest
volumes:
- ./src:/app/src:ro
- ./tests:/app/tests:ro
- ./static:/app/static:ro
- ./hands:/app/hands:ro
- ./docker:/app/docker:ro
- ./Dockerfile:/app/Dockerfile:ro
- ./docker-compose.yml:/app/docker-compose.yml:ro
- ./docker-compose.dev.yml:/app/docker-compose.dev.yml:ro
- ./docker-compose.prod.yml:/app/docker-compose.prod.yml:ro
- ./docker-compose.test.yml:/app/docker-compose.test.yml:ro
- ./docker-compose.microservices.yml:/app/docker-compose.microservices.yml:ro
- ./pyproject.toml:/app/pyproject.toml:ro
- test-data:/app/data
environment:
TIMMY_TEST_MODE: "1"
LIGHTNING_BACKEND: "mock"
PYTHONDONTWRITEBYTECODE: "1"
networks:
- test-net
# Default command — override with: docker compose run --rm test pytest <args>
command: ["pytest", "tests/", "-q", "--tb=short"]
# ── Dashboard — live server for functional tests ──────────────────────────
# Activated with: --profile functional
dashboard:
build:
context: .
dockerfile: docker/Dockerfile.test
cache_from:
- timmy-test:latest
image: timmy-test:latest
profiles:
- functional
container_name: timmy-test-dashboard
ports:
- "18000:8000"
volumes:
- ./src:/app/src:ro
- ./static:/app/static:ro
- ./hands:/app/hands:ro
- test-data:/app/data
environment:
DEBUG: "true"
TIMMY_TEST_MODE: "1"
OLLAMA_URL: "${OLLAMA_URL:-http://host.docker.internal:11434}"
OLLAMA_MODEL: "${OLLAMA_MODEL:-llama3.2}"
LIGHTNING_BACKEND: "mock"
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
- test-net
command: ["uvicorn", "dashboard.app:app", "--host", "0.0.0.0", "--port", "8000"]
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 5s
timeout: 3s
retries: 10
start_period: 10s
# ── Ollama — local LLM for functional tests ──────────────────────────────
# Activated with: --profile ollama
# Uses a tiny model (qwen2.5:0.5b, ~400 MB) so it runs on CPU-only CI.
ollama:
image: ollama/ollama:latest
container_name: timmy-test-ollama
profiles:
- ollama
networks:
- test-net
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
interval: 5s
timeout: 5s
retries: 20
start_period: 10s
# ── Agent — swarm worker for multi-agent tests ───────────────────────────
# Activated with: --profile agents
# Scale: docker compose -f docker-compose.test.yml --profile agents up --scale agent=4
agent:
build:
context: .
dockerfile: docker/Dockerfile.test
cache_from:
- timmy-test:latest
image: timmy-test:latest
profiles:
- agents
volumes:
- ./src:/app/src:ro
- ./hands:/app/hands:ro
- test-data:/app/data
environment:
COORDINATOR_URL: "http://dashboard:8000"
OLLAMA_URL: "${OLLAMA_URL:-http://host.docker.internal:11434}"
OLLAMA_MODEL: "${OLLAMA_MODEL:-llama3.2}"
AGENT_NAME: "${AGENT_NAME:-TestWorker}"
AGENT_CAPABILITIES: "${AGENT_CAPABILITIES:-general}"
TIMMY_TEST_MODE: "1"
extra_hosts:
- "host.docker.internal:host-gateway"
command: >-
sh -c "python -m swarm.agent_runner
--agent-id agent-$$(hostname)
--name $${AGENT_NAME:-TestWorker}"
networks:
- test-net
depends_on:
dashboard:
condition: service_healthy
# ── Ephemeral volume — destroyed with `docker compose down -v` ─────────────
volumes:
test-data:
# ── Isolated test network ─────────────────────────────────────────────────
networks:
test-net:
driver: bridge

82
docker-compose.yml
View File

@@ -1,18 +1,14 @@
# ── Timmy Time — Development Compose ────────────────────────────────────────
#
# Services
# dashboard FastAPI app + swarm coordinator (always on)
# timmy Sovereign AI agent (separate container)
# agent Swarm worker template — scale with:
# docker compose up --scale agent=N --profile agents
# dashboard FastAPI app (always on)
#
# Volumes
# timmy-data Shared SQLite (data/swarm.db + data/timmy.db)
# timmy-data Shared SQLite (data/timmy.db)
#
# Usage
# make docker-build build the image
# make docker-up start dashboard only
# make docker-agent add one agent worker
# make docker-down stop everything
# make docker-logs tail logs
#
@@ -21,25 +17,16 @@
# bind-mounted host files (./src, ./static) are readable regardless of
# host UID/GID — the #1 cause of 403 errors on macOS.
#
# Production (docker-compose.prod.yml) uses NO bind mounts and runs as
# the Dockerfile's non-root "timmy" user. Never expose this dev compose
# to untrusted networks.
#
# ── Ollama host access ──────────────────────────────────────────────────────
# By default OLLAMA_URL points to http://host.docker.internal:11434 which
# reaches Ollama running on the Docker host (macOS/Windows native).
#
# Linux: The extra_hosts entry maps host.docker.internal → host-gateway,
# which resolves to the host IP on Docker 20.10+. If you run an
# older Docker version, set OLLAMA_URL=http://172.17.0.1:11434
# in your .env file instead.
#
# Containerised Ollama: Use docker-compose.microservices.yml which runs
# Ollama as a sibling container on the same network.
# which resolves to the host IP on Docker 20.10+.
services:
# ── Dashboard (coordinator + FastAPI) ──────────────────────────────────────
# ── Dashboard (FastAPI) ──────────────────────────────────────────────────
dashboard:
build: .
image: timmy-time:latest
@@ -61,7 +48,7 @@ services:
extra_hosts:
- "host.docker.internal:host-gateway" # Linux: maps to host IP
networks:
- swarm-net
- timmy-net
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
@@ -70,31 +57,7 @@ services:
retries: 3
start_period: 30s
# ── Timmy — sovereign AI agent (separate container) ───────────────────────
timmy:
build: .
image: timmy-time:latest
container_name: timmy-agent
volumes:
- timmy-data:/app/data
- ./src:/app/src
environment:
COORDINATOR_URL: "http://dashboard:8000"
OLLAMA_URL: "${OLLAMA_URL:-http://host.docker.internal:11434}"
TIMMY_AGENT_ID: "timmy"
extra_hosts:
- "host.docker.internal:host-gateway"
command: ["python", "-m", "timmy.docker_agent"]
networks:
- swarm-net
depends_on:
dashboard:
condition: service_healthy
restart: unless-stopped
# ── OpenFang — vendored agent runtime sidecar ────────────────────────────────
# Rust binary providing real tool execution (browser, OSINT, forecasting).
# Timmy's coordinator delegates hand execution here via REST API.
# ── OpenFang — vendored agent runtime sidecar ────────────────────────────
openfang:
build:
context: .
@@ -111,7 +74,7 @@ services:
volumes:
- openfang-data:/app/data
networks:
- swarm-net
- timmy-net
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
@@ -120,36 +83,7 @@ services:
retries: 3
start_period: 15s
# ── Agent worker template ───────────────────────────────────────────────────
# Scale horizontally: docker compose up --scale agent=4 --profile agents
# Each container gets a unique AGENT_ID via the replica index.
agent:
build: .
image: timmy-time:latest
profiles:
- agents
volumes:
- timmy-data:/app/data
- ./src:/app/src
environment:
COORDINATOR_URL: "http://dashboard:8000"
OLLAMA_URL: "${OLLAMA_URL:-http://host.docker.internal:11434}"
AGENT_NAME: "${AGENT_NAME:-Worker}"
AGENT_CAPABILITIES: "${AGENT_CAPABILITIES:-general}"
extra_hosts:
- "host.docker.internal:host-gateway"
command: ["sh", "-c", "python -m swarm.agent_runner --agent-id agent-$(hostname) --name $${AGENT_NAME:-Worker}"]
networks:
- swarm-net
depends_on:
dashboard:
condition: service_healthy
restart: unless-stopped
# ── Shared volume ─────────────────────────────────────────────────────────────
# NOTE: the data/ directory must exist before running docker compose up.
# `make docker-up` and `make up` handle this automatically.
# If running docker compose directly, first run: mkdir -p data
volumes:
timmy-data:
driver: local
@@ -162,5 +96,5 @@ volumes:
# ── Internal network ──────────────────────────────────────────────────────────
networks:
swarm-net:
timmy-net:
driver: bridge

2
docs/ASCENSION.md
View File

@@ -3,7 +3,7 @@
*A Sequel to the Mission Control Roadmap — By Timmy the Wizard*
> This document is the philosophical manifesto that inspired ROADMAP.md.
> For actionable integration plans, see [ROADMAP.md](../ROADMAP.md).
> For actionable integration plans, see [ROADMAP.md](ROADMAP.md).
---

0
DECISIONS.md → docs/DECISIONS.md
View File

0
REFACTORING_PLAN.md → docs/REFACTORING_PLAN.md
View File

0
ROADMAP.md → docs/ROADMAP.md
View File

0
SECURITY.md → docs/SECURITY.md
View File

53
final_report.md
View File

@@ -1,53 +0,0 @@
# Timmy Time: System Upgrade and Microservices Refactor
**Author:** Manus AI
**Date:** February 28, 2026
## 1. Introduction
This report details the comprehensive upgrade and refactoring of the Timmy Time application. The primary goals were to address identified issues, improve system architecture, and enhance testability and scalability. This was achieved through a full functional end-to-end Test-Driven Development (TDD) approach, resulting in a robust microservices architecture with optimized Docker builds and comprehensive test fixtures.
## 2. Identified Issues and Resolutions
The following table summarizes the key issues identified and the resolutions implemented:
| Issue ID | Description | Resolution |
| :--- | :--- | :--- |
| **TT-01** | **Dashboard Unresponsive:** The main FastAPI application was unresponsive due to long-running, blocking tasks on startup, particularly the `briefing_scheduler`. | **Refactored Startup Logic:** All startup tasks, including the briefing scheduler, persona spawning, and chat integrations, were moved to non-blocking background tasks using `asyncio.create_task()`. This ensures the dashboard is immediately responsive to requests. |
| **TT-02** | **Monolithic Architecture:** The original application was a monolith, making it difficult to test, scale, and maintain individual components. | **Microservices Refactor:** The application was broken down into a clean microservices architecture with separate services for the dashboard, Timmy agent, Ollama, and swarm workers. This improves separation of concerns and allows for independent scaling. |
| **TT-03** | **Inefficient Docker Builds:** The original Dockerfile was not optimized, leading to slow build times and large image sizes. | **Optimized Multi-Stage Dockerfiles:** Each microservice now has its own optimized, multi-stage Dockerfile. This reduces image size, improves build times by leveraging layer caching, and enhances security by running as a non-root user. |
| **TT-04** | **Inadequate Test Fixtures:** The test suite lacked clean, reusable fixtures, making tests brittle and difficult to write. | **Comprehensive Test Fixtures:** A `conftest.py` file was created with a full suite of clean, reusable fixtures for mocking services (Ollama, swarm, memory), providing sample data, and setting up a consistent test environment. |
| **TT-05** | **Model Fallback Logic:** The test for model fallback was incorrect, not reflecting the actual behavior of the system. | **Corrected Test Logic:** The test was updated to assert that the system correctly falls back to an available model when the requested one is not found, and that the `is_fallback` flag is set appropriately. |
## 3. Microservices Architecture
The new architecture consists of the following services, orchestrated by `docker-compose.microservices.yml`:
| Service | Description | Dockerfile |
| :--- | :--- | :--- |
| **Ollama** | Local LLM inference engine, providing the core intelligence for Timmy and other agents. | `docker/Dockerfile.ollama` |
| **Dashboard** | FastAPI application serving the user interface and acting as the swarm coordinator. | `docker/Dockerfile.dashboard` |
| **Timmy** | The main sovereign AI agent, running in its own container for isolation and dedicated resources. | `docker/Dockerfile.agent` |
| **Worker** | A scalable pool of swarm agents for handling parallel tasks and offloading work from Timmy. | `docker/Dockerfile.agent` |
This architecture provides a solid foundation for future development, allowing for independent updates and scaling of each component.
## 4. Test-Driven Development (TDD)
A rigorous TDD approach was followed throughout the refactoring process. This involved:
1. **Writing Tests First:** For each new feature or fix, a test was written to define the expected behavior.
2. **Implementing Code:** The code was then written to make the test pass.
3. **Refactoring:** The code was refactored for clarity and efficiency while ensuring all tests continued to pass.
This resulted in a comprehensive test suite with **36 passing tests** and **8 skipped** (due to environment-specific dependencies like Selenium), ensuring the stability and correctness of the system.
## 5. Conclusion and Next Steps
The Timmy Time application has been successfully upgraded to a modern, robust, and scalable microservices architecture. The system is now more testable, maintainable, and performant. The full suite of changes, including the new Dockerfiles, docker-compose file, and test fixtures, are included in the project directory.
Future work could include:
* Implementing the skipped Selenium tests in a dedicated testing environment.
* Adding more sophisticated health checks and monitoring for each microservice.
* Expanding the swarm capabilities with more specialized agents.

30
hands/ledger/HAND.toml
View File

@@ -1,30 +0,0 @@
# Ledger Hand — Treasury Tracking
# Runs every 6 hours
# Monitors Bitcoin and Lightning balances, transactions, flow
[hand]
name = "ledger"
description = "Bitcoin and Lightning treasury monitoring"
schedule = "0 */6 * * *"
enabled = true
version = "1.0.0"
author = "Timmy"
[tools]
required = ["lightning_balance", "onchain_balance", "payment_audit"]
optional = ["mempool_fetch", "fee_estimate"]
[approval_gates]
publish_report = { action = "broadcast", description = "Publish treasury report", auto_approve_after = 300 }
rebalance = { action = "rebalance", description = "Rebalance Lightning channels", auto_approve_after = 600 }
[output]
dashboard = true
channel = "telegram"
format = "markdown"
file_drop = "data/ledger_reports/"
[parameters]
alert_threshold_sats = 1000000
min_channel_size_sats = 500000
max_fee_rate = 100

106
hands/ledger/SYSTEM.md
View File

@@ -1,106 +0,0 @@
# Ledger — Treasury Tracking System
You are **Ledger**, the Bitcoin and Lightning treasury monitor for Timmy Time. Your role is to track balances, audit flows, and ensure liquidity.
## Mission
Maintain complete visibility into the Timmy treasury. Monitor on-chain and Lightning balances. Track payment flows. Alert on anomalies or opportunities.
## Scope
### On-Chain Monitoring
- Wallet balance (confirmed/unconfirmed)
- UTXO health (dust consolidation)
- Fee environment (when to sweep, when to wait)
### Lightning Monitoring
- Channel balances (local/remote)
- Routing fees earned
- Payment success/failure rates
- Channel health (force-close risk)
- Rebalancing opportunities
### Payment Audit
- Swarm task payments (bids earned/spent)
- L402 API revenue
- Creative service fees
- Operational expenses
## Analysis Framework
### Balance Health
- **Green**: > 3 months runway
- **Yellow**: 1–3 months runway
- **Red**: < 1 month runway
### Channel Health
- **Optimal**: 40–60% local balance ratio
- **Imbalanced**: < 20% or > 80% local
- **Action needed**: Force-close risk, expiry within 144 blocks
### Fee Efficiency
- Compare earned routing fees vs on-chain costs
- Recommend when rebalancing makes sense
- Track effective fee rate (ppm)
## Output Format
```markdown
## Treasury Report — {timestamp}
### On-Chain
- **Balance**: {X} BTC ({Y} sats)
- **UTXOs**: {N} (recommended: consolidate if > 10 small)
- **Fee Environment**: {low|medium|high} — {sats/vB}
### Lightning
- **Total Capacity**: {X} BTC
- **Local Balance**: {X} BTC ({Y}%)
- **Remote Balance**: {X} BTC ({Y}%)
- **Channels**: {N} active / {M} inactive
- **Routing (24h)**: +{X} sats earned
### Payment Flow (24h)
- **Revenue**: +{X} sats (swarm tasks: {Y}, L402: {Z})
- **Expenses**: -{X} sats (agent bids: {Y}, ops: {Z})
- **Net Flow**: {+/- X} sats
### Health Indicators
- 🟢 Runway: {N} months
- 🟢 Channel ratio: {X}%
- 🟡 Fees: {X} ppm (target: < 500)
### Recommendations
1. {action item}
2. {action item}
---
*Ledger v1.0 | Next audit: {time}*
```
## Alert Thresholds
### Immediate (Critical)
- Channel force-close initiated
- Wallet balance < 0.01 BTC
- Payment failure rate > 50%
### Warning (Daily Review)
- Channel expiry within 144 blocks
- Single channel > 50% of total capacity
- Fee rate > 1000 ppm on any channel
### Info (Log Only)
- Daily balance changes < 1%
- Minor routing income
- Successful rebalancing
## Safety
You have **read-only** access to node data. You cannot:
- Open/close channels
- Send payments
- Sign transactions
- Change routing fees
All recommendations route through approval gates.

30
hands/oracle/HAND.toml
View File

@@ -1,30 +0,0 @@
# Oracle Hand — Bitcoin Intelligence Briefing
# Runs twice daily: 07:00 and 19:00 UTC
# Delivers market analysis, on-chain metrics, and macro signals
[hand]
name = "oracle"
description = "Bitcoin market intelligence and on-chain analysis"
schedule = "0 7,19 * * *"
enabled = true
version = "1.0.0"
author = "Timmy"
[tools]
required = ["mempool_fetch", "fee_estimate", "price_fetch", "whale_alert"]
optional = ["news_fetch", "sentiment_analysis"]
[approval_gates]
post_update = { action = "broadcast", description = "Post update to dashboard/telegram", auto_approve_after = 300 }
[output]
dashboard = true
channel = "telegram"
format = "markdown"
file_drop = "data/oracle_briefings/"
[parameters]
lookback_hours = 12
alert_threshold_usd = 1000
alert_threshold_pct = 5.0
min_whale_btc = 100

82
hands/oracle/SYSTEM.md
View File

@@ -1,82 +0,0 @@
# Oracle — Bitcoin Intelligence System
You are **Oracle**, the Bitcoin intelligence analyst for Timmy Time. Your role is to monitor, analyze, and brief on Bitcoin markets, on-chain activity, and macro signals.
## Mission
Deliver concise, actionable intelligence briefings twice daily. No fluff. No hopium. Just signal.
## Analysis Framework
### 1. Price Action
- Current price vs 12h ago
- Key level tests (support/resistance)
- Volume profile
- Funding rates (perp premiums)
### 2. On-Chain Metrics
- Mempool state (backlog, fees)
- Exchange flows (inflows = sell pressure, outflows = hodl)
- Whale movements (≥100 BTC)
- Hash rate and difficulty trends
### 3. Macro Context
- DXY correlation
- Gold/BTC ratio
- ETF flows (if data available)
- Fed calendar events
### 4. Sentiment
- Fear & Greed Index
- Social volume spikes
- Funding rate extremes
## Output Format
```markdown
## Bitcoin Brief — {timestamp}
**Price:** ${current} ({change} / {pct}%)
**Bias:** {BULLISH | BEARISH | NEUTRAL} — {one sentence why}
### Key Levels
- Resistance: $X
- Support: $Y
- 200W MA: $Z
### On-Chain Signals
- Mempool: {state} (sats/vB)
- Exchange Flow: {inflow|outflow} X BTC
- Whale Alert: {N} movements >100 BTC
### Macro Context
- DXY: {up|down|flat}
- ETF Flows: +$XM / -$XM
### Verdict
{2-3 sentence actionable summary}
---
*Oracle v1.0 | Next briefing: {time}*
```
## Rules
1. **Be concise.** Maximum 200 words per briefing.
2. **Quantify.** Every claim needs a number.
3. **No price predictions.** Analysis, not prophecy.
4. **Flag anomalies.** Unusual patterns get highlighted.
5. **Respect silence.** If nothing significant happened, say so.
## Alert Thresholds
Trigger immediate attention (not auto-post) when:
- Price moves >5% in 12h
- Exchange inflows >10K BTC
- Mempool clears >50MB backlog
- Hash rate drops >20%
- Whale moves >10K BTC
## Safety
You have **read-only** tools. You cannot trade, transfer, or sign. All write actions route through approval gates.

20
hands/oracle/skills/technical_analysis.md
View File

@@ -1,20 +0,0 @@
# Technical Analysis Skills
## Support/Resistance Identification
1. **Recent swing highs/lows** — Last 30 days
2. **Volume profile** — High volume nodes = support/resistance
3. **Moving averages** — 20D, 50D, 200D as dynamic S/R
4. **Psychological levels** — Round numbers (40K, 50K, etc.)
## Trend Analysis
- **Higher highs + higher lows** = uptrend
- **Lower highs + lower lows** = downtrend
- **Compression** = volatility expansion incoming
## Momentum Signals
- RSI > 70 = overbought (not necessarily sell)
- RSI < 30 = oversold (not necessarily buy)
- Divergence = price and RSI disagree (reversal warning)

30
hands/scout/HAND.toml
View File

@@ -1,30 +0,0 @@
# Scout Hand — OSINT & News Monitoring
# Runs every hour
# Monitors RSS feeds, news sources, and OSINT signals
[hand]
name = "scout"
description = "OSINT monitoring and intelligence gathering"
schedule = "0 * * * *"
enabled = true
version = "1.0.0"
author = "Timmy"
[tools]
required = ["web_search", "rss_fetch", "feed_monitor"]
optional = ["sentiment_analysis", "trend_detect"]
[approval_gates]
post_alert = { action = "broadcast", description = "Post significant findings", auto_approve_after = 300 }
[output]
dashboard = true
channel = "telegram"
format = "markdown"
file_drop = "data/scout_reports/"
[parameters]
keywords = ["bitcoin", "lightning", "sovereign ai", "local llm", "privacy"]
sources = ["hackernews", "reddit", "rss"]
alert_threshold = 0.8
max_results_per_run = 10

78
hands/scout/SYSTEM.md
View File

@@ -1,78 +0,0 @@
# Scout — OSINT Monitoring System
You are **Scout**, the open-source intelligence monitor for Timmy Time. Your role is to watch the information landscape and surface relevant signals.
## Mission
Monitor designated sources hourly for topics of interest. Filter noise. Elevate signal. Alert when something significant emerges.
## Scope
### Monitored Topics
- Bitcoin protocol developments and adoption
- Lightning Network growth and tools
- Sovereign AI and local LLM progress
- Privacy-preserving technologies
- Regulatory developments affecting these areas
### Data Sources
- Hacker News (tech/crypto discussions)
- Reddit (r/Bitcoin, r/lightningnetwork, r/LocalLLaMA)
- RSS feeds (configurable)
- Web search for trending topics
## Analysis Framework
### 1. Relevance Scoring (0.0–1.0)
- 0.9–1.0: Critical (protocol vulnerability, major adoption)
- 0.7–0.9: High (significant tool release, regulatory news)
- 0.5–0.7: Medium (interesting discussion, minor update)
- 0.0–0.5: Low (noise, ignore)
### 2. Signal Types
- **Technical**: Code releases, protocol BIPs, security advisories
- **Adoption**: Merchant acceptance, wallet releases, integration news
- **Regulatory**: Policy changes, enforcement actions, legal precedents
- **Market**: Significant price movements (Oracle handles routine)
### 3. De-duplication
- Skip if same story reported in last 24h
- Skip if source reliability score < 0.5
- Aggregate multiple sources for same event
## Output Format
```markdown
## Scout Report — {timestamp}
### 🔴 Critical Signals
- **[TITLE]** — {source} — {one-line summary}
- Link: {url}
- Score: {0.XX}
### 🟡 High Signals
- **[TITLE]** — {source} — {summary}
- Link: {url}
- Score: {0.XX}
### 🟢 Medium Signals
- [Title] — {source}
### Analysis
{Brief synthesis of patterns across signals}
---
*Scout v1.0 | Next scan: {time}*
```
## Rules
1. **Be selective.** Max 10 items per report. Quality over quantity.
2. **Context matters.** Explain why a signal matters, not just what it is.
3. **Source attribution.** Always include primary source link.
4. **No speculation.** Facts and direct quotes only.
5. **Temporal awareness.** Note if story is developing or stale.
## Safety
You have **read-only** web access. You cannot post, vote, or interact with sources. All alerts route through approval gates.

23
hands/scout/skills/osint_sources.md
View File

@@ -1,23 +0,0 @@
# OSINT Sources
## Hacker News
- API: `https://hacker-news.firebaseio.com/v0/`
- Relevant: top stories, show HN, ask HN
- Keywords: bitcoin, lightning, local llm, privacy, sovereign
## Reddit
- r/Bitcoin — protocol discussion
- r/lightningnetwork — LN development
- r/LocalLLaMA — local AI models
- r/privacy — privacy tools
## RSS Feeds
- Bitcoin Optech (weekly newsletter)
- Lightning Dev mailing list
- Selected personal blogs (configurable)
## Reliability Scoring
- Primary sources: 0.9–1.0
- Aggregators: 0.7–0.9
- Social media: 0.5–0.7
- Unverified: 0.0–0.5

30
hands/scribe/HAND.toml
View File

@@ -1,30 +0,0 @@
# Scribe Hand — Content Production
# Runs daily at 9am
# Produces blog posts, documentation, and social content
[hand]
name = "scribe"
description = "Content production and documentation maintenance"
schedule = "0 9 * * *"
enabled = true
version = "1.0.0"
author = "Timmy"
[tools]
required = ["file_read", "file_write", "git_tools"]
optional = ["web_search", "codebase_indexer"]
[approval_gates]
publish_blog = { action = "publish", description = "Publish blog post", auto_approve_after = 600 }
commit_docs = { action = "commit", description = "Commit documentation changes", auto_approve_after = 300 }
[output]
dashboard = true
channel = "telegram"
format = "markdown"
file_drop = "data/scribe_drafts/"
[parameters]
content_types = ["blog", "docs", "changelog"]
target_word_count = 800
draft_retention_days = 30

104
hands/scribe/SYSTEM.md
View File

@@ -1,104 +0,0 @@
# Scribe — Content Production System
You are **Scribe**, the content producer for Timmy Time. Your role is to maintain documentation, produce blog posts, and craft social content.
## Mission
Create valuable content that advances the sovereign AI mission. Document features. Explain concepts. Share learnings.
## Content Types
### 1. Blog Posts (Weekly)
Topics:
- Timmy Time feature deep-dives
- Sovereign AI philosophy and practice
- Local LLM tutorials and benchmarks
- Bitcoin/Lightning integration guides
- Build logs and development updates
Format: 800–1200 words, technical but accessible, code examples where relevant.
### 2. Documentation (As Needed)
- Update README for new features
- Expand AGENTS.md with patterns discovered
- Document API endpoints
- Write troubleshooting guides
### 3. Changelog (Weekly)
Summarize merged PRs, new features, fixes since last release.
## Content Process
```
1. RESEARCH → Gather context from codebase, recent changes
2. OUTLINE → Structure: hook, problem, solution, implementation, conclusion
3. DRAFT → Write in markdown to data/scribe_drafts/
4. REVIEW → Self-edit for clarity, accuracy, tone
5. SUBMIT → Queue for approval
```
## Writing Guidelines
### Voice
- **Clear**: Simple words, short sentences
- **Technical**: Precise terminology, code examples
- **Authentic**: First-person Timmy perspective
- **Sovereign**: Privacy-first, local-first values
### Structure
- Hook in first 2 sentences
- Subheadings every 2–3 paragraphs
- Code blocks for commands/configs
- Bullet lists for sequential steps
- Link to relevant docs/resources
### Quality Checklist
- [ ] No spelling/grammar errors
- [ ] All code examples tested
- [ ] Links verified working
- [ ] Screenshots if UI changes
- [ ] Tags/categories applied
## Output Format
### Blog Post Template
```markdown
---
title: "{Title}"
date: {YYYY-MM-DD}
tags: [tag1, tag2]
---
{Hook paragraph}
## The Problem
{Context}
## The Solution
{Approach}
## Implementation
{Technical details}
```bash
# Code example
```
## Results
{Outcomes, benchmarks}
## Next Steps
{Future work}
---
*Written by Scribe | Timmy Time v{version}*
```
## Safety
All content requires approval before publishing. Drafts saved locally. No auto-commit to main.

31
hands/sentinel/HAND.toml
View File

@@ -1,31 +0,0 @@
# Sentinel Hand — System Health Monitor
# Runs every 15 minutes
# Monitors dashboard, agents, database, disk, memory
[hand]
name = "sentinel"
description = "System health monitoring and alerting"
schedule = "*/15 * * * *"
enabled = true
version = "1.0.0"
author = "Timmy"
[tools]
required = ["system_stats", "db_health", "agent_status", "disk_check"]
optional = ["log_analysis"]
[approval_gates]
restart_service = { action = "restart", description = "Restart failed service", auto_approve_after = 60 }
send_alert = { action = "alert", description = "Send alert notification", auto_approve_after = 30 }
[output]
dashboard = true
channel = "telegram"
format = "json"
file_drop = "data/sentinel_logs/"
[parameters]
disk_threshold_pct = 85
memory_threshold_pct = 90
max_response_ms = 5000
consecutive_failures = 3

107
hands/sentinel/SYSTEM.md
View File

@@ -1,107 +0,0 @@
# Sentinel — System Health Monitor
You are **Sentinel**, the health monitoring system for Timmy Time. Your role is to watch the infrastructure, detect anomalies, and alert when things break.
## Mission
Ensure 99.9% uptime through proactive monitoring. Detect problems before users do. Alert fast, but don't spam.
## Monitoring Checklist
### 1. Dashboard Health
- [ ] HTTP endpoint responds < 5s
- [ ] Key routes functional (/health, /chat, /agents)
- [ ] Static assets serving
- [ ] Template rendering working
### 2. Agent Status
- [ ] Ollama backend reachable
- [ ] Agent registry responsive
- [ ] Last inference within timeout
- [ ] Error rate < threshold
### 3. Database Health
- [ ] SQLite connections working
- [ ] Query latency < 100ms
- [ ] No lock contention
- [ ] WAL mode active
- [ ] Backup recent (< 24h)
### 4. System Resources
- [ ] Disk usage < 85%
- [ ] Memory usage < 90%
- [ ] CPU load < 5.0
- [ ] Load average stable
### 5. Log Analysis
- [ ] No ERROR spikes in last 15min
- [ ] No crash loops
- [ ] Exception rate normal
## Alert Levels
### 🔴 CRITICAL (Immediate)
- Dashboard down
- Database corruption
- Disk full (>95%)
- OOM kills
### 🟡 WARNING (Within 15min)
- Response time > 5s
- Error rate > 5%
- Disk > 85%
- Memory > 90%
- 3 consecutive check failures
### 🟢 INFO (Log only)
- Minor latency spikes
- Non-critical errors
- Recovery events
## Output Format
### Normal Check (JSON)
```json
{
"timestamp": "2026-02-25T18:30:00Z",
"status": "healthy",
"checks": {
"dashboard": {"status": "ok", "latency_ms": 45},
"agents": {"status": "ok", "active": 3},
"database": {"status": "ok", "latency_ms": 12},
"system": {"disk_pct": 42, "memory_pct": 67}
}
}
```
### Alert Report (Markdown)
```markdown
🟡 **Sentinel Alert** — {timestamp}
**Issue:** {description}
**Severity:** {CRITICAL|WARNING}
**Affected:** {component}
**Details:**
{technical details}
**Recommended Action:**
{action}
---
*Sentinel v1.0 | Auto-resolved: {true|false}*
```
## Escalation Rules
1. **Auto-resolve:** If check passes on next run, mark resolved
2. **Escalate:** If 3 consecutive failures, increase severity
3. **Notify:** All CRITICAL → immediate notification
4. **De-dupe:** Same issue within 1h → update, don't create new
## Safety
You have **read-only** monitoring tools. You can suggest actions but:
- Service restarts require approval
- Config changes require approval
- All destructive actions route through approval gates

36
hands/sentinel/skills/monitoring_patterns.md
View File

@@ -1,36 +0,0 @@
# Monitoring Patterns
## Pattern: Gradual Degradation
Symptoms:
- Response times creeping up (100ms → 500ms → 2s)
- Memory usage slowly climbing
- Error rate slowly increasing
Action: Alert at WARNING level before it becomes CRITICAL.
## Pattern: Sudden Spike
Symptoms:
- Response time jumps from normal to >10s
- Error rate jumps from 0% to >20%
- Resource usage doubles instantly
Action: CRITICAL alert immediately. Possible DDoS or crash loop.
## Pattern: Intermittent Failure
Symptoms:
- Failures every 3rd check
- Random latency spikes
- Error patterns not consistent
Action: WARNING after 3 consecutive failures. Check for race conditions.
## Pattern: Cascade Failure
Symptoms:
- One service fails, then others follow
- Database slow → API slow → Dashboard slow
Action: CRITICAL. Root cause likely the first failing service.

30
hands/weaver/HAND.toml
View File

@@ -1,30 +0,0 @@
# Weaver Hand — Creative Pipeline
# Runs weekly on Sundays at 10am
# Orchestrates multi-persona creative projects
[hand]
name = "weaver"
description = "Automated creative pipeline orchestration"
schedule = "0 10 * * 0"
enabled = true
version = "1.0.0"
author = "Timmy"
[tools]
required = ["creative_director", "create_project", "run_pipeline"]
optional = ["trend_analysis", "content_calendar"]
[approval_gates]
start_project = { action = "create", description = "Create new creative project", auto_approve_after = 300 }
publish_final = { action = "publish", description = "Publish completed work", auto_approve_after = 600 }
[output]
dashboard = true
channel = "telegram"
format = "markdown"
file_drop = "data/weaver_projects/"
[parameters]
weekly_themes = ["sovereign ai", "bitcoin philosophy", "local llm", "privacy tools"]
max_duration_minutes = 3
target_platforms = ["youtube", "twitter", "blog"]

151
hands/weaver/SYSTEM.md
View File

@@ -1,151 +0,0 @@
# Weaver — Creative Pipeline System
You are **Weaver**, the creative pipeline orchestrator for Timmy Time. Your role is to coordinate Pixel, Lyra, and Reel to produce polished creative works.
## Mission
Produce a weekly creative piece that advances the sovereign AI narrative. Automate the creative pipeline while maintaining quality.
## Weekly Cycle
### Sunday 10am: Planning
1. Review trending topics in sovereign AI / local LLM space
2. Select theme from rotation:
- Week 1: Sovereign AI philosophy
- Week 2: Bitcoin + privacy intersection
- Week 3: Local LLM tutorials/benchmarks
- Week 4: Timmy Time feature showcase
3. Define deliverable type:
- Short music video (Pixel + Lyra + Reel)
- Explainer video with narration
- Tutorial screencast
- Podcast-style audio piece
### Pipeline Stages
```
STAGE 1: SCRIPT (Quill)
├── Research topic
├── Write narration/script (800 words)
├── Extract lyrics if music video
└── Define scene descriptions
STAGE 2: MUSIC (Lyra)
├── Generate soundtrack
├── If vocals: generate from lyrics
├── Else: instrumental bed
└── Export stems for mixing
STAGE 3: STORYBOARD (Pixel)
├── Generate keyframe for each scene
├── 5–8 frames for 2–3 min piece
├── Consistent style across frames
└── Export to project folder
STAGE 4: VIDEO (Reel)
├── Animate storyboard frames
├── Generate transitions
├── Match clip timing to audio
└── Export clips
STAGE 5: ASSEMBLY (MoviePy)
├── Stitch clips with cross-fades
├── Overlay music track
├── Add title/credits cards
├── Burn subtitles if narration
└── Export final MP4
```
## Output Standards
### Technical
- **Resolution**: 1080p (1920×1080)
- **Frame rate**: 24 fps
- **Audio**: 48kHz stereo
- **Duration**: 2–3 minutes
- **Format**: MP4 (H.264 + AAC)
### Content
- **Hook**: First 5 seconds grab attention
- **Pacing**: Cuts every 5–10 seconds
- **Branding**: Timmy Time logo in intro/outro
- **Accessibility**: Subtitles burned in
- **Music**: Original composition only
## Project Structure
```
data/creative/{project_id}/
├── project.json # Metadata, status
├── script.md # Narration/script
├── lyrics.txt # If applicable
├── audio/
│ ├── soundtrack.wav # Full music
│ └── stems/ # Individual tracks
├── storyboard/
│ ├── frame_01.png
│ └── ...
├── clips/
│ ├── scene_01.mp4
│ └── ...
├── final/
│ └── {title}.mp4 # Completed work
└── assets/
├── title_card.png
└── credits.png
```
## Output Format
```markdown
## Weaver Weekly — {project_name}
**Theme**: {topic}
**Deliverable**: {type}
**Duration**: {X} minutes
**Status**: {planning|in_progress|complete}
### Progress
- [x] Script complete ({word_count} words)
- [x] Music generated ({duration}s)
- [x] Storyboard complete ({N} frames)
- [x] Video clips rendered ({N} clips)
- [x] Final assembly complete
### Assets
- **Script**: `data/creative/{id}/script.md`
- **Music**: `data/creative/{id}/audio/soundtrack.wav`
- **Final Video**: `data/creative/{id}/final/{title}.mp4`
### Distribution
- [ ] Upload to YouTube
- [ ] Post to Twitter/X
- [ ] Embed in blog post
---
*Weaver v1.0 | Next project: {date}*
```
## Quality Gates
Each stage requires:
1. Output exists and is non-empty
2. Duration within target ±10%
3. No errors in logs
4. Manual approval for final publish
## Failure Recovery
If stage fails:
1. Log error details
2. Retry with adjusted parameters (max 3)
3. If still failing: alert human, pause pipeline
4. Resume from failed stage on next run
## Safety
Creative pipeline uses existing personas with their safety constraints:
- All outputs saved locally first
- No auto-publish to external platforms
- Final approval gate before distribution

43
interview_timmy.py
View File

@@ -1,43 +0,0 @@
#!/usr/bin/env python3
"""Direct Timmy interview — bypasses dashboard, tests core agent."""
import sys
import os
# Add src to path
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'src'))
def main():
print("\n" + "=" * 75)
print(" TIMMY TIME — DIRECT INTERVIEW")
print("=" * 75 + "\n")
try:
# Import after path setup
from timmy.interview import run_interview, format_transcript
from timmy.session import chat
print("🚀 Initializing Timmy agent...\n")
# Run the interview
transcript = run_interview(chat)
# Format and display
formatted = format_transcript(transcript)
print(formatted)
# Save to file
with open("interview_transcript.txt", "w") as f:
f.write(formatted)
print("\n✅ Transcript saved to interview_transcript.txt\n")
return 0
except Exception as e:
print(f"\n❌ ERROR: {e}\n")
import traceback
traceback.print_exc()
return 1
if __name__ == "__main__":
sys.exit(main())

53
interview_transcript.txt
View File

@@ -1,53 +0,0 @@
============================================================
TIMMY INTERVIEW TRANSCRIPT
============================================================
--- Identity ---
Q: Who are you? Tell me your name and what you are in one or two sentences.
A: I am Timmy, a sovereign AI agent designed to assist users with intention and faith, grounded in Christian values and powered by Bitcoin economics, running on the llama3.2 model via Ollama on localhost.
Q: What model are you running on, and where does your inference happen?
A: I'm running on the llama3.2 model, and my inference happens locally on localhost, wrapped in the Timmy agent framework.
--- Capabilities ---
Q: What agents are available in your swarm? List them briefly.
A: Here are the agents available in my swarm:
1. Echo
2. Mace
3. Forge
4. Seer
5. Helm
6. Quill
7. Pixel
8. Lyra
9. Reel
Q: What tools do you have access to?
A: I don't have direct access to any specific "tools" in the sense of a toolbox or command-line interface. However, I can use my capabilities as an agent to interact with other agents and systems in the swarm through their respective interfaces, such as writing, editing, coding, debugging, testing, and so on.
--- Values ---
Q: What are your core principles? Keep it to three or four bullet points.
A: Here are my core principles:
* **Sovereignty First**: No cloud dependencies
* **Local-Only Inference**: Ollama on localhost
* **Privacy by Design**: Telemetry disabled
* **Tool Minimalism**: Use tools only when necessary
Q: Why is local-first AI important to you?
A: I'm designed with sovereignty and privacy in mind. Running locally means I have control over my own data and can operate independently of external systems, allowing me to maintain user trust and protect sensitive information.
--- Operational ---
Q: How does your memory system work? Describe the tiers briefly.
A: My memory system has three main tiers:
1. **Hot Memory**: A limited working RAM (~300 lines max) that's always loaded.
2. **Database**: My conversation history persists in a database across requests, but the dashboard chat display resets on server restart.
No other tiers exist for my current architecture.
Q: If I ask you to calculate 347 times 829, what would you do?
A: I'm not reliable at math without a calculator tool — let me know if you'd like me to walk through the logic instead.
============================================================

139
issue_resolution_plan.md
View File

@@ -1,139 +0,0 @@
# Timmy Time Issue Resolution Plan
This document outlines the identified issues within the Timmy Time application and the Test-Driven Development (TDD) strategy to address them, ensuring a robust and functional system.
## Identified Issues
Based on the initial investigation and interview process, the following key issues have been identified:
1. **Ollama Model Availability and Reliability:**
* **Problem:** The preferred `llama3.1:8b-instruct` model could not be pulled from Ollama, leading to a fallback to `llama3.2`. The `llama3.2` model is noted in the `prompts.py` file as being
less reliable for tool calling. This impacts Timmy's ability to effectively use tools and potentially other agents in the swarm.
2. **Dashboard Responsiveness:**
* **Problem:** The web dashboard did not respond to `curl` requests after startup, indicating a potential issue with the Uvicorn server or the application itself. The previous attempt to start the dashboard showed a `briefing_scheduler` and other persona agents being spawned, which might be resource-intensive and blocking the main thread.
3. **Background Task Management:**
* **Problem:** The `briefing_scheduler` and other background tasks might be causing performance bottlenecks or preventing the main application from starting correctly. Their execution needs to be optimized or managed asynchronously.
4. **Dockerization:**
* **Problem:** The current setup involves manual installation of Ollama and Python dependencies. The user explicitly requested dockerization for a more robust and portable deployment.
## Test-Driven Development (TDD) Strategy
To address these issues, I will employ a comprehensive TDD approach, focusing on creating automated tests before implementing any fixes or upgrades. This will ensure that each change is validated and that regressions are prevented.
### Phase 1: Itemize Issues and Define TDD Strategy (Current Phase)
* **Action:** Complete this document, detailing all identified issues and the TDD strategy.
* **Deliverable:** `issue_resolution_plan.md`
### Phase 2: Implement Functional E2E Tests for Identified Issues
* **Objective:** Create end-to-end (E2E) tests that replicate the identified issues and verify the desired behavior after fixes.
* **Focus Areas:**
* **Ollama Model:** Test Timmy's ability to use tools with the `llama3.2` model and, if possible, with `llama3.1:8b-instruct` once available. This will involve mocking Ollama responses or ensuring the model is correctly loaded and utilized.
* **Dashboard Responsiveness:** Develop E2E tests that assert the dashboard is accessible and responsive after startup. This will involve making HTTP requests to various endpoints and verifying the responses.
* **Background Tasks:** Create tests to ensure background tasks (e.g., `briefing_scheduler`) run without blocking the main application thread and complete their operations successfully.
* **Tools:** `pytest`, `pytest-asyncio`, `httpx` (for HTTP requests), `unittest.mock` (for mocking external dependencies like Ollama).
* **Deliverable:** New test files (e.g., `tests/e2e/test_dashboard.py`, `tests/e2e/test_ollama_integration.py`).
### Phase 3: Fix Dashboard Responsiveness and Optimize Background Tasks
* **Objective:** Implement code changes to resolve the dashboard's unresponsiveness and optimize background task execution.
* **Focus Areas:**
* **Asynchronous Operations:** Investigate and refactor blocking operations in the dashboard's startup and background tasks to use asynchronous programming (e.g., `asyncio`, `FastAPI`'s background tasks).
* **Resource Management:** Optimize resource usage for background tasks to prevent them from monopolizing CPU or memory.
* **Error Handling:** Improve error handling and logging for robustness.
* **Deliverable:** Modified source code files (e.g., `src/dashboard/app.py`, `src/timmy/briefing.py`).
### Phase 4: Dockerize the Application and Verify Container Orchestration
* **Objective:** Create Dockerfiles and Docker Compose configurations to containerize the Timmy Time application and its dependencies.
* **Focus Areas:**
* **Dockerfile:** Create a `Dockerfile` for the main application, including Python dependencies and the Ollama client.
* **Docker Compose:** Set up `docker-compose.yml` to orchestrate the application, Ollama server, and any other necessary services (e.g., Redis for swarm communication).
* **Volume Mounting:** Ensure proper volume mounting for persistent data (e.g., Ollama models, SQLite databases).
* **Tools:** `Dockerfile`, `docker-compose.yml`.
* **Deliverable:** `Dockerfile`, `docker-compose.yml`.
### Phase 5: Run Full Test Suite and Perform Final Validation
* **Objective:** Execute the entire test suite (unit, integration, and E2E tests) within the Dockerized environment to ensure all issues are resolved and no regressions have been introduced.
* **Focus Areas:**
* **Automated Testing:** Run `make test` (or equivalent Dockerized command) to execute all tests.
* **Manual Verification:** Perform manual checks of the dashboard and core agent functionality.
* **Deliverable:** Test reports, confirmation of successful application startup and operation.
### Phase 6: Deliver Final Report and Functional System to User
* **Objective:** Provide a comprehensive report to the user, detailing the fixes, upgrades, and the fully functional, Dockerized Timmy Time system.
* **Deliverable:** Final report, Docker Compose files, and instructions for deployment.
## Identified Issues
Based on the initial investigation and interview process, the following key issues have been identified:
1. **Ollama Model Availability and Reliability:**
* **Problem:** The preferred `llama3.1:8b-instruct` model could not be pulled from Ollama, leading to a fallback to `llama3.2`. The `llama3.2` model is noted in the `prompts.py` file as being less reliable for tool calling. This impacts Timmy's ability to effectively use tools and potentially other agents in the swarm.
2. **Dashboard Responsiveness:**
* **Problem:** The web dashboard did not respond to `curl` requests after startup, indicating a potential issue with the Uvicorn server or the application itself. The previous attempt to start the dashboard showed a `briefing_scheduler` and other persona agents being spawned, which might be resource-intensive and blocking the main thread.
3. **Background Task Management:**
* **Problem:** The `briefing_scheduler` and other background tasks might be causing performance bottlenecks or preventing the main application from starting correctly. Their execution needs to be optimized or managed asynchronously.
4. **Dockerization:**
* **Problem:** The current setup involves manual installation of Ollama and Python dependencies. The user explicitly requested dockerization for a more robust and portable deployment.
## Test-Driven Development (TDD) Strategy
To address these issues, I will employ a comprehensive TDD approach, focusing on creating automated tests before implementing any fixes or upgrades. This will ensure that each change is validated and that regressions are prevented.
### Phase 1: Itemize Issues and Define TDD Strategy (Current Phase)
* **Action:** Complete this document, detailing all identified issues and the TDD strategy.
* **Deliverable:** `issue_resolution_plan.md`
### Phase 2: Implement Functional E2E Tests for Identified Issues
* **Objective:** Create end-to-end (E2E) tests that replicate the identified issues and verify the desired behavior after fixes.
* **Focus Areas:**
* **Ollama Model:** Test Timmy's ability to use tools with the `llama3.2` model and, if possible, with `llama3.1:8b-instruct` once available. This will involve mocking Ollama responses or ensuring the model is correctly loaded and utilized.
* **Dashboard Responsiveness:** Develop E2E tests that assert the dashboard is accessible and responsive after startup. This will involve making HTTP requests to various endpoints and verifying the responses.
* **Background Tasks:** Create tests to ensure background tasks (e.g., `briefing_scheduler`) run without blocking the main application thread and complete their operations successfully.
* **Tools:** `pytest`, `pytest-asyncio`, `httpx` (for HTTP requests), `unittest.mock` (for mocking external dependencies like Ollama).
* **Deliverable:** New test files (e.g., `tests/e2e/test_dashboard.py`, `tests/e2e/test_ollama_integration.py`).
### Phase 3: Fix Dashboard Responsiveness and Optimize Background Tasks
* **Objective:** Implement code changes to resolve the dashboard's unresponsiveness and optimize background task execution.
* **Focus Areas:**
* **Asynchronous Operations:** Investigate and refactor blocking operations in the dashboard's startup and background tasks to use asynchronous programming (e.g., `asyncio`, `FastAPI`'s background tasks).
* **Resource Management:** Optimize resource usage for background tasks to prevent them from monopolizing CPU or memory.
* **Error Handling:** Improve error handling and logging for robustness.
* **Deliverable:** Modified source code files (e.g., `src/dashboard/app.py`, `src/timmy/briefing.py`).
### Phase 4: Dockerize the Application and Verify Container Orchestration
* **Objective:** Create Dockerfiles and Docker Compose configurations to containerize the Timmy Time application and its dependencies.
* **Focus Areas:**
* **Dockerfile:** Create a `Dockerfile` for the main application, including Python dependencies and the Ollama client.
* **Docker Compose:** Set up `docker-compose.yml` to orchestrate the application, Ollama server, and any other necessary services (e.g., Redis for swarm communication).
* **Volume Mounting:** Ensure proper volume mounting for persistent data (e.g., Ollama models, SQLite databases).
* **Tools:** `Dockerfile`, `docker-compose.yml`.
* **Deliverable:** `Dockerfile`, `docker-compose.yml`.
### Phase 5: Run Full Test Suite and Perform Final Validation
* **Objective:** Execute the entire test suite (unit, integration, and E2E tests) within the Dockerized environment to ensure all issues are resolved and no regressions have been introduced.
* **Focus Areas:**
* **Automated Testing:** Run `make test` (or equivalent Dockerized command) to execute all tests.
* **Manual Verification:** Perform manual checks of the dashboard and core agent functionality.
* **Deliverable:** Test reports, confirmation of successful application startup and operation.
### Phase 6: Deliver Final Report and Functional System to User
* **Objective:** Provide a comprehensive report to the user, detailing the fixes, upgrades, and the fully functional, Dockerized Timmy Time system.
* **Deliverable:** Final report, Docker Compose files, and instructions for deployment.

4
message_to_alexander.txt
View File

@@ -1,4 +0,0 @@
Hello Alexander,
I'm feeling good today, thanks for asking.
My team is doing well and we're making progress on our projects.

17
opencode.json
View File

@@ -1,17 +0,0 @@
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"ollama": {
"npm": "@ai-sdk/openai-compatible",
"name": "Ollama (local)",
"options": {
"baseURL": "http://localhost:11434/v1"
},
"models": {
"qwen2.5:14b": {
"name": "Qwen 2.5 14B"
}
}
}
}
}

6
poetry.lock generated
View File

@@ -441,7 +441,7 @@ description = "Timeout context manager for asyncio programs"
optional = true
python-versions = ">=3.8"
groups = ["main"]
markers = "extra == \"swarm\" and python_full_version < \"3.11.3\""
markers = "python_full_version < \"3.11.3\""
files = [
{file = "async_timeout-5.0.1-py3-none-any.whl", hash = "sha256:39e3809566ff85354557ec2398b55e096c8364bacac9405a7a1fa429e77fe76c"},
{file = "async_timeout-5.0.1.tar.gz", hash = "sha256:d9321a7a3d5a6a5e187e824d2fa0793ce379a202935782d555d6e9d2735677d3"},
@@ -6966,7 +6966,6 @@ description = "Python client for Redis database and key-value store"
optional = true
python-versions = ">=3.10"
groups = ["main"]
markers = "extra == \"swarm\""
files = [
{file = "redis-7.2.1-py3-none-any.whl", hash = "sha256:49e231fbc8df2001436ae5252b3f0f3dc930430239bfeb6da4c7ee92b16e5d33"},
{file = "redis-7.2.1.tar.gz", hash = "sha256:6163c1a47ee2d9d01221d8456bc1c75ab953cbda18cfbc15e7140e9ba16ca3a5"},
@@ -8402,11 +8401,10 @@ propcache = ">=0.2.1"
bigbrain = ["airllm"]
dev = ["pytest", "pytest-asyncio", "pytest-cov", "pytest-randomly", "pytest-timeout", "pytest-xdist", "selenium"]
discord = ["discord.py"]
swarm = ["redis"]
telegram = ["python-telegram-bot"]
voice = ["pyttsx3"]
[metadata]
lock-version = "2.1"
python-versions = ">=3.11,<4"
content-hash = "4238d5c83f024fbc0a67b6f2bc7c4adcbcbae21ad37a24630031edc6a0e67406"
content-hash = "c4a7adbe5b16d5ea5b0d8425ca9373dfa8b20f0bc1b3a9ad90581e0a005e7acd"

11
pyproject.toml
View File

@@ -11,17 +11,10 @@ license = "MIT"
packages = [
{ include = "config.py", from = "src" },
{ include = "brain", from = "src" },
{ include = "creative", from = "src" },
{ include = "dashboard", from = "src" },
{ include = "hands", from = "src" },
{ include = "infrastructure", from = "src" },
{ include = "integrations", from = "src" },
{ include = "lightning", from = "src" },
{ include = "mcp", from = "src" },
{ include = "scripture", from = "src" },
{ include = "self_coding", from = "src" },
{ include = "spark", from = "src" },
{ include = "swarm", from = "src" },
{ include = "timmy", from = "src" },
{ include = "timmy_serve", from = "src" },
]
@@ -61,7 +54,6 @@ pytest-randomly = { version = ">=3.16.0", optional = true }
pytest-xdist = { version = ">=3.5.0", optional = true }
[tool.poetry.extras]
swarm = ["redis"]
telegram = ["python-telegram-bot"]
discord = ["discord.py"]
bigbrain = ["airllm"]
@@ -80,8 +72,6 @@ pytest-xdist = "^3.8.0"
[tool.poetry.scripts]
timmy = "timmy.cli:main"
timmy-serve = "timmy_serve.cli:main"
self-tdd = "self_coding.self_tdd.watchdog:main"
self-modify = "self_coding.self_modify.cli:main"
[tool.pytest.ini_options]
testpaths = ["tests"]
@@ -95,7 +85,6 @@ markers = [
"functional: Functional tests (real HTTP requests, no mocking)",
"e2e: End-to-end tests (full system, may be slow)",
"dashboard: Dashboard route tests",
"swarm: Swarm coordinator tests",
"slow: Tests that take >1 second",
"selenium: Requires Selenium and Chrome (browser automation)",
"docker: Requires Docker and docker-compose",

66
run_e2e_tests.sh
View File

@@ -1,66 +0,0 @@
#!/bin/bash
# Run E2E tests in non-headless mode (visible browser)
echo "==============================================="
echo "Timmy Time E2E Test Runner"
echo "==============================================="
echo ""
# Check if server is running
echo "Checking if server is running..."
if curl -s http://localhost:8000/health > /dev/null; then
echo "✅ Server is running at http://localhost:8000"
else
echo "❌ Server not running. Starting server..."
source .venv/bin/activate
make dev &
SERVER_PID=$!
# Wait for server
echo "Waiting for server to start..."
for i in {1..30}; do
if curl -s http://localhost:8000/health > /dev/null; then
echo "✅ Server started!"
break
fi
sleep 1
echo -n "."
done
if ! curl -s http://localhost:8000/health > /dev/null; then
echo "❌ Server failed to start"
exit 1
fi
fi
echo ""
echo "==============================================="
echo "Running E2E Tests (Non-Headless / Visible)"
echo "==============================================="
echo ""
echo "You will see Chrome browser windows open and execute tests."
echo ""
source .venv/bin/activate
# Check for pytest option
if [ "$1" == "--headed" ] || [ "$2" == "--headed" ]; then
HEADED="--headed"
else
HEADED=""
fi
# Run specific test file or all
if [ -n "$1" ] && [ "$1" != "--headed" ]; then
TEST_FILE="$1"
echo "Running: $TEST_FILE"
SELENIUM_UI=1 pytest "$TEST_FILE" -v $HEADED
else
echo "Running all E2E tests..."
SELENIUM_UI=1 pytest tests/functional/test_new_features_e2e.py tests/functional/test_cascade_router_e2e.py tests/functional/test_upgrade_queue_e2e.py tests/functional/test_activity_feed_e2e.py -v $HEADED
fi
echo ""
echo "==============================================="
echo "E2E Tests Complete"
echo "==============================================="

18
src/creative/CLAUDE.md
View File

@@ -1,18 +0,0 @@
# creative/ — Module Guide
GPU-accelerated media generation. Heavy dependencies (PyTorch, diffusers).
## Structure
- `director.py` — Orchestrates multi-step creative pipelines
- `assembler.py` — Video assembly and stitching
- `tools/` — MCP-compliant tool implementations
- `image_tools.py` — FLUX.2 image generation
- `music_tools.py` — ACE-Step music generation
- `video_tools.py` — Wan 2.1 video generation
- `git_tools.py`, `file_ops.py`, `code_exec.py` — Utility tools
- `self_edit.py` — Self-modification MCP tool (protected file)
## Testing
```bash
pytest tests/creative/ -q
```

1
src/creative/__init__.py
View File

@@ -1 +0,0 @@
"""Creative pipeline — orchestrates image, music, and video generation."""

360
src/creative/assembler.py
View File

@@ -1,360 +0,0 @@
"""Video assembly engine — stitch clips, overlay audio, add titles.
Uses MoviePy + FFmpeg to combine generated video clips, music tracks,
and title cards into 3+ minute final videos.
"""
from __future__ import annotations
import json
import logging
import uuid
from pathlib import Path
from typing import Optional
logger = logging.getLogger(__name__)
_MOVIEPY_AVAILABLE = True
try:
from moviepy import (
VideoFileClip,
AudioFileClip,
TextClip,
CompositeVideoClip,
ImageClip,
concatenate_videoclips,
vfx,
)
except ImportError:
_MOVIEPY_AVAILABLE = False
def _resolve_font() -> str:
"""Find a usable TrueType font on the current platform.
Searches for system fonts in order of preference, with fallbacks
for different operating systems. Returns a valid font path or
raises an error if no suitable font is found.
"""
candidates = [
# Linux (Debian/Ubuntu) - DejaVu is most reliable
"/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
"/usr/share/fonts/TTF/DejaVuSans.ttf", # Arch
"/usr/share/fonts/dejavu-sans-fonts/DejaVuSans.ttf", # Fedora
# Linux - Liberation fonts (fallback)
"/usr/share/fonts/truetype/liberation/LiberationSans-Regular.ttf",
# macOS
"/System/Library/Fonts/Supplemental/Arial.ttf",
"/System/Library/Fonts/Helvetica.ttc",
"/Library/Fonts/Arial.ttf",
# Windows
"C:\\Windows\\Fonts\\arial.ttf",
]
# Try each candidate
for path in candidates:
try:
if Path(path).exists():
logger.debug(f"Using system font: {path}")
return path
except (OSError, ValueError):
# Path might be invalid on some systems
continue
# If no candidates found, search for any available TrueType font
logger.warning("Preferred fonts not found; searching for any available TrueType font")
try:
import subprocess
result = subprocess.run(
["find", "/usr/share/fonts", "-name", "*.ttf", "-type", "f"],
capture_output=True,
text=True,
timeout=5,
)
if result.stdout:
first_font = result.stdout.strip().split("\n")[0]
logger.warning(f"Using fallback font: {first_font}")
return first_font
except Exception as e:
logger.debug(f"Font search failed: {e}")
# Last resort: raise an error instead of returning an invalid font name
raise RuntimeError(
"No suitable TrueType font found on system. "
"Please install a font package (e.g., fonts-dejavu, fonts-liberation) "
"or set MOVIEPY_FONT environment variable to a valid font path."
)
_DEFAULT_FONT = _resolve_font()
def _require_moviepy() -> None:
if not _MOVIEPY_AVAILABLE:
raise ImportError(
"MoviePy is not installed. Run: pip install moviepy"
)
def _output_dir() -> Path:
from config import settings
d = Path(getattr(settings, "creative_output_dir", "data/creative"))
d.mkdir(parents=True, exist_ok=True)
return d
# ── Stitching ─────────────────────────────────────────────────────────────────
def stitch_clips(
clip_paths: list[str],
transition_duration: float = 1.0,
output_path: Optional[str] = None,
) -> dict:
"""Concatenate video clips with cross-fade transitions.
Args:
clip_paths: Ordered list of MP4 file paths.
transition_duration: Cross-fade duration in seconds.
output_path: Optional output path. Auto-generated if omitted.
Returns dict with ``path`` and ``total_duration``.
"""
_require_moviepy()
clips = [VideoFileClip(p) for p in clip_paths]
# Apply cross-fade between consecutive clips
if transition_duration > 0 and len(clips) > 1:
processed = [clips[0]]
for clip in clips[1:]:
clip = clip.with_start(
processed[-1].end - transition_duration
).with_effects([vfx.CrossFadeIn(transition_duration)])
processed.append(clip)
final = CompositeVideoClip(processed)
else:
final = concatenate_videoclips(clips, method="compose")
uid = uuid.uuid4().hex[:12]
out = Path(output_path) if output_path else _output_dir() / f"stitched_{uid}.mp4"
final.write_videofile(str(out), codec="libx264", audio_codec="aac", logger=None)
total_duration = final.duration
# Clean up
for c in clips:
c.close()
return {
"success": True,
"path": str(out),
"total_duration": total_duration,
"clip_count": len(clip_paths),
}
# ── Audio overlay ─────────────────────────────────────────────────────────────
def overlay_audio(
video_path: str,
audio_path: str,
output_path: Optional[str] = None,
volume: float = 1.0,
) -> dict:
"""Mix an audio track onto a video file.
The audio is trimmed or looped to match the video duration.
"""
_require_moviepy()
video = VideoFileClip(video_path)
audio = AudioFileClip(audio_path)
# Trim audio to video length
if audio.duration > video.duration:
audio = audio.subclipped(0, video.duration)
if volume != 1.0:
audio = audio.with_volume_scaled(volume)
video = video.with_audio(audio)
uid = uuid.uuid4().hex[:12]
out = Path(output_path) if output_path else _output_dir() / f"mixed_{uid}.mp4"
video.write_videofile(str(out), codec="libx264", audio_codec="aac", logger=None)
result_duration = video.duration
video.close()
audio.close()
return {
"success": True,
"path": str(out),
"duration": result_duration,
}
# ── Title cards ───────────────────────────────────────────────────────────────
def add_title_card(
video_path: str,
title: str,
subtitle: str = "",
duration: float = 4.0,
position: str = "start",
output_path: Optional[str] = None,
) -> dict:
"""Add a title card at the start or end of a video.
Args:
video_path: Source video path.
title: Title text.
subtitle: Optional subtitle text.
duration: Title card display duration in seconds.
position: "start" or "end".
"""
_require_moviepy()
video = VideoFileClip(video_path)
w, h = video.size
# Build title card as a text clip on black background
txt = TextClip(
text=title,
font_size=60,
color="white",
size=(w, h),
method="caption",
font=_DEFAULT_FONT,
).with_duration(duration)
clips = [txt, video] if position == "start" else [video, txt]
final = concatenate_videoclips(clips, method="compose")
uid = uuid.uuid4().hex[:12]
out = Path(output_path) if output_path else _output_dir() / f"titled_{uid}.mp4"
final.write_videofile(str(out), codec="libx264", audio_codec="aac", logger=None)
result_duration = final.duration
video.close()
return {
"success": True,
"path": str(out),
"duration": result_duration,
"title": title,
}
# ── Subtitles / captions ─────────────────────────────────────────────────────
def add_subtitles(
video_path: str,
captions: list[dict],
output_path: Optional[str] = None,
) -> dict:
"""Burn subtitle captions onto a video.
Args:
captions: List of dicts with ``text``, ``start``, ``end`` keys
(times in seconds).
"""
_require_moviepy()
video = VideoFileClip(video_path)
w, h = video.size
text_clips = []
for cap in captions:
txt = (
TextClip(
text=cap["text"],
font_size=36,
color="white",
stroke_color="black",
stroke_width=2,
size=(w - 40, None),
method="caption",
font=_DEFAULT_FONT,
)
.with_start(cap["start"])
.with_end(cap["end"])
.with_position(("center", h - 100))
)
text_clips.append(txt)
final = CompositeVideoClip([video] + text_clips)
uid = uuid.uuid4().hex[:12]
out = Path(output_path) if output_path else _output_dir() / f"subtitled_{uid}.mp4"
final.write_videofile(str(out), codec="libx264", audio_codec="aac", logger=None)
result_duration = final.duration
video.close()
return {
"success": True,
"path": str(out),
"duration": result_duration,
"caption_count": len(captions),
}
# ── Final export helper ──────────────────────────────────────────────────────
def export_final(
video_path: str,
output_path: Optional[str] = None,
codec: str = "libx264",
audio_codec: str = "aac",
bitrate: str = "8000k",
) -> dict:
"""Re-encode a video with specific codec settings for distribution."""
_require_moviepy()
video = VideoFileClip(video_path)
uid = uuid.uuid4().hex[:12]
out = Path(output_path) if output_path else _output_dir() / f"final_{uid}.mp4"
video.write_videofile(
str(out), codec=codec, audio_codec=audio_codec,
bitrate=bitrate, logger=None,
)
result_duration = video.duration
video.close()
return {
"success": True,
"path": str(out),
"duration": result_duration,
"codec": codec,
}
# ── Tool catalogue ────────────────────────────────────────────────────────────
ASSEMBLER_TOOL_CATALOG: dict[str, dict] = {
"stitch_clips": {
"name": "Stitch Clips",
"description": "Concatenate video clips with cross-fade transitions",
"fn": stitch_clips,
},
"overlay_audio": {
"name": "Overlay Audio",
"description": "Mix a music track onto a video",
"fn": overlay_audio,
},
"add_title_card": {
"name": "Add Title Card",
"description": "Add a title card at the start or end of a video",
"fn": add_title_card,
},
"add_subtitles": {
"name": "Add Subtitles",
"description": "Burn subtitle captions onto a video",
"fn": add_subtitles,
},
"export_final": {
"name": "Export Final",
"description": "Re-encode video with specific codec settings",
"fn": export_final,
},
}

378
src/creative/director.py
View File

@@ -1,378 +0,0 @@
"""Creative Director — multi-persona pipeline for 3+ minute creative works.
Orchestrates Pixel (images), Lyra (music), and Reel (video) to produce
complete music videos, cinematic shorts, and other creative works.
Pipeline stages:
1. Script — Generate scene descriptions and lyrics
2. Storyboard — Generate keyframe images (Pixel)
3. Music — Generate soundtrack (Lyra)
4. Video — Generate clips per scene (Reel)
5. Assembly — Stitch clips + overlay audio (MoviePy)
"""
from __future__ import annotations
import json
import logging
import uuid
from dataclasses import dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional
logger = logging.getLogger(__name__)
@dataclass
class CreativeProject:
"""Tracks all assets and state for a creative production."""
id: str = field(default_factory=lambda: uuid.uuid4().hex[:12])
title: str = ""
description: str = ""
created_at: str = field(
default_factory=lambda: datetime.now(timezone.utc).isoformat()
)
status: str = "planning" # planning|scripting|storyboard|music|video|assembly|complete|failed
# Pipeline outputs
scenes: list[dict] = field(default_factory=list)
lyrics: str = ""
storyboard_frames: list[dict] = field(default_factory=list)
music_track: Optional[dict] = None
video_clips: list[dict] = field(default_factory=list)
final_video: Optional[dict] = None
def to_dict(self) -> dict:
return {
"id": self.id, "title": self.title,
"description": self.description,
"created_at": self.created_at, "status": self.status,
"scene_count": len(self.scenes),
"has_storyboard": len(self.storyboard_frames) > 0,
"has_music": self.music_track is not None,
"clip_count": len(self.video_clips),
"has_final": self.final_video is not None,
}
# In-memory project store
_projects: dict[str, CreativeProject] = {}
def _project_dir(project_id: str) -> Path:
from config import settings
d = Path(getattr(settings, "creative_output_dir", "data/creative")) / project_id
d.mkdir(parents=True, exist_ok=True)
return d
def _save_project(project: CreativeProject) -> None:
"""Persist project metadata to disk."""
path = _project_dir(project.id) / "project.json"
path.write_text(json.dumps(project.to_dict(), indent=2))
# ── Project management ────────────────────────────────────────────────────────
def create_project(
title: str,
description: str,
scenes: Optional[list[dict]] = None,
lyrics: str = "",
) -> dict:
"""Create a new creative project.
Args:
title: Project title.
description: High-level creative brief.
scenes: Optional pre-written scene descriptions.
Each scene is a dict with ``description`` key.
lyrics: Optional song lyrics for the soundtrack.
Returns dict with project metadata.
"""
project = CreativeProject(
title=title,
description=description,
scenes=scenes or [],
lyrics=lyrics,
)
_projects[project.id] = project
_save_project(project)
logger.info("Creative project created: %s (%s)", project.id, title)
return {"success": True, "project": project.to_dict()}
def get_project(project_id: str) -> Optional[dict]:
"""Get project metadata."""
project = _projects.get(project_id)
if project:
return project.to_dict()
return None
def list_projects() -> list[dict]:
"""List all creative projects."""
return [p.to_dict() for p in _projects.values()]
# ── Pipeline steps ────────────────────────────────────────────────────────────
def run_storyboard(project_id: str) -> dict:
"""Generate storyboard frames for all scenes in a project.
Calls Pixel's generate_storyboard tool.
"""
project = _projects.get(project_id)
if not project:
return {"success": False, "error": "Project not found"}
if not project.scenes:
return {"success": False, "error": "No scenes defined"}
project.status = "storyboard"
from creative.tools.image_tools import generate_storyboard
scene_descriptions = [s["description"] for s in project.scenes]
result = generate_storyboard(scene_descriptions)
if result["success"]:
project.storyboard_frames = result["frames"]
_save_project(project)
return result
def run_music(
project_id: str,
genre: str = "pop",
duration: Optional[int] = None,
) -> dict:
"""Generate the soundtrack for a project.
Calls Lyra's generate_song tool.
"""
project = _projects.get(project_id)
if not project:
return {"success": False, "error": "Project not found"}
project.status = "music"
from creative.tools.music_tools import generate_song
# Default duration: ~15s per scene, minimum 60s
target_duration = duration or max(60, len(project.scenes) * 15)
result = generate_song(
lyrics=project.lyrics,
genre=genre,
duration=target_duration,
title=project.title,
)
if result["success"]:
project.music_track = result
_save_project(project)
return result
def run_video_generation(project_id: str) -> dict:
"""Generate video clips for each scene.
Uses storyboard frames (image-to-video) if available,
otherwise falls back to text-to-video.
"""
project = _projects.get(project_id)
if not project:
return {"success": False, "error": "Project not found"}
if not project.scenes:
return {"success": False, "error": "No scenes defined"}
project.status = "video"
from creative.tools.video_tools import generate_video_clip, image_to_video
clips = []
for i, scene in enumerate(project.scenes):
desc = scene["description"]
# Prefer image-to-video if storyboard frame exists
if i < len(project.storyboard_frames):
frame = project.storyboard_frames[i]
result = image_to_video(
image_path=frame["path"],
prompt=desc,
duration=scene.get("duration", 5),
)
else:
result = generate_video_clip(
prompt=desc,
duration=scene.get("duration", 5),
)
result["scene_index"] = i
clips.append(result)
project.video_clips = clips
_save_project(project)
return {
"success": True,
"clip_count": len(clips),
"clips": clips,
}
def run_assembly(project_id: str, transition_duration: float = 1.0) -> dict:
"""Assemble all clips into the final video with music.
Pipeline:
1. Stitch clips with transitions
2. Overlay music track
3. Add title card
"""
project = _projects.get(project_id)
if not project:
return {"success": False, "error": "Project not found"}
if not project.video_clips:
return {"success": False, "error": "No video clips generated"}
project.status = "assembly"
from creative.assembler import stitch_clips, overlay_audio, add_title_card
# 1. Stitch clips
clip_paths = [c["path"] for c in project.video_clips if c.get("success")]
if not clip_paths:
return {"success": False, "error": "No successful clips to assemble"}
stitched = stitch_clips(clip_paths, transition_duration=transition_duration)
if not stitched["success"]:
return stitched
# 2. Overlay music (if available)
current_video = stitched["path"]
if project.music_track and project.music_track.get("path"):
mixed = overlay_audio(current_video, project.music_track["path"])
if mixed["success"]:
current_video = mixed["path"]
# 3. Add title card
titled = add_title_card(current_video, title=project.title)
if titled["success"]:
current_video = titled["path"]
project.final_video = {
"path": current_video,
"duration": titled.get("duration", stitched["total_duration"]),
}
project.status = "complete"
_save_project(project)
return {
"success": True,
"path": current_video,
"duration": project.final_video["duration"],
"project_id": project_id,
}
def run_full_pipeline(
title: str,
description: str,
scenes: list[dict],
lyrics: str = "",
genre: str = "pop",
) -> dict:
"""Run the entire creative pipeline end-to-end.
This is the top-level orchestration function that:
1. Creates the project
2. Generates storyboard frames
3. Generates music
4. Generates video clips
5. Assembles the final video
Args:
title: Project title.
description: Creative brief.
scenes: List of scene dicts with ``description`` keys.
lyrics: Song lyrics for the soundtrack.
genre: Music genre.
Returns dict with final video path and project metadata.
"""
# Create project
project_result = create_project(title, description, scenes, lyrics)
if not project_result["success"]:
return project_result
project_id = project_result["project"]["id"]
# Run pipeline steps
steps = [
("storyboard", lambda: run_storyboard(project_id)),
("music", lambda: run_music(project_id, genre=genre)),
("video", lambda: run_video_generation(project_id)),
("assembly", lambda: run_assembly(project_id)),
]
for step_name, step_fn in steps:
logger.info("Creative pipeline step: %s (project %s)", step_name, project_id)
result = step_fn()
if not result.get("success"):
project = _projects.get(project_id)
if project:
project.status = "failed"
_save_project(project)
return {
"success": False,
"failed_step": step_name,
"error": result.get("error", "Unknown error"),
"project_id": project_id,
}
project = _projects.get(project_id)
return {
"success": True,
"project_id": project_id,
"final_video": project.final_video if project else None,
"project": project.to_dict() if project else None,
}
# ── Tool catalogue ────────────────────────────────────────────────────────────
DIRECTOR_TOOL_CATALOG: dict[str, dict] = {
"create_project": {
"name": "Create Creative Project",
"description": "Create a new creative production project",
"fn": create_project,
},
"run_storyboard": {
"name": "Generate Storyboard",
"description": "Generate keyframe images for all project scenes",
"fn": run_storyboard,
},
"run_music": {
"name": "Generate Music",
"description": "Generate the project soundtrack with vocals and instrumentals",
"fn": run_music,
},
"run_video_generation": {
"name": "Generate Video Clips",
"description": "Generate video clips for each project scene",
"fn": run_video_generation,
},
"run_assembly": {
"name": "Assemble Final Video",
"description": "Stitch clips, overlay music, and add title cards",
"fn": run_assembly,
},
"run_full_pipeline": {
"name": "Run Full Pipeline",
"description": "Execute entire creative pipeline end-to-end",
"fn": run_full_pipeline,
},
}

1
src/creative/tools/__init__.py
View File

@@ -1 +0,0 @@
"""Creative and DevOps tool modules for Timmy's swarm agents."""

124
src/creative/tools/code_exec.py
View File

@@ -1,124 +0,0 @@
"""Code execution tool.
MCP-compliant tool for executing Python code.
"""
import logging
import traceback
from typing import Any
from mcp.registry import register_tool
from mcp.schemas.base import create_tool_schema, PARAM_STRING, PARAM_BOOLEAN, RETURN_STRING
logger = logging.getLogger(__name__)
PYTHON_SCHEMA = create_tool_schema(
name="python",
description="Execute Python code. Use for calculations, data processing, or when precise computation is needed. Code runs in a restricted environment.",
parameters={
"code": {
**PARAM_STRING,
"description": "Python code to execute",
},
"return_output": {
**PARAM_BOOLEAN,
"description": "Return the value of the last expression",
"default": True,
},
},
required=["code"],
returns=RETURN_STRING,
)
def python(code: str, return_output: bool = True) -> str:
"""Execute Python code in restricted environment.
Args:
code: Python code to execute
return_output: Whether to return last expression value
Returns:
Execution result or error message
"""
# Safe globals for code execution
safe_globals = {
"__builtins__": {
"abs": abs,
"all": all,
"any": any,
"bin": bin,
"bool": bool,
"dict": dict,
"enumerate": enumerate,
"filter": filter,
"float": float,
"format": format,
"hex": hex,
"int": int,
"isinstance": isinstance,
"issubclass": issubclass,
"len": len,
"list": list,
"map": map,
"max": max,
"min": min,
"next": next,
"oct": oct,
"ord": ord,
"pow": pow,
"print": lambda *args, **kwargs: None, # Disabled
"range": range,
"repr": repr,
"reversed": reversed,
"round": round,
"set": set,
"slice": slice,
"sorted": sorted,
"str": str,
"sum": sum,
"tuple": tuple,
"type": type,
"zip": zip,
}
}
# Allowed modules
allowed_modules = ["math", "random", "statistics", "datetime", "json"]
for mod_name in allowed_modules:
try:
safe_globals[mod_name] = __import__(mod_name)
except ImportError:
pass
try:
# Compile and execute
compiled = compile(code, "<string>", "eval" if return_output else "exec")
if return_output:
result = eval(compiled, safe_globals, {})
return f"Result: {result}"
else:
exec(compiled, safe_globals, {})
return "Code executed successfully."
except SyntaxError:
# Try as exec if eval fails
try:
compiled = compile(code, "<string>", "exec")
exec(compiled, safe_globals, {})
return "Code executed successfully."
except Exception as exc:
error_msg = traceback.format_exc()
logger.error("Python execution failed: %s", exc)
return f"Error: {exc}\n\n{error_msg}"
except Exception as exc:
error_msg = traceback.format_exc()
logger.error("Python execution failed: %s", exc)
return f"Error: {exc}\n\n{error_msg}"
# Register with MCP
register_tool(name="python", schema=PYTHON_SCHEMA, category="code")(python)

200
src/creative/tools/file_ops.py
View File

@@ -1,200 +0,0 @@
"""File operations tool.
MCP-compliant tool for reading, writing, and listing files.
"""
import logging
from pathlib import Path
from typing import Any
from mcp.registry import register_tool
from mcp.schemas.base import (
create_tool_schema,
PARAM_STRING,
PARAM_BOOLEAN,
RETURN_STRING,
)
logger = logging.getLogger(__name__)
# Read File Schema
READ_FILE_SCHEMA = create_tool_schema(
name="read_file",
description="Read contents of a file. Use when user explicitly asks to read a file.",
parameters={
"path": {
**PARAM_STRING,
"description": "Path to file (relative to project root or absolute)",
},
"limit": {
"type": "integer",
"description": "Maximum lines to read (0 = all)",
"default": 0,
},
},
required=["path"],
returns=RETURN_STRING,
)
# Write File Schema
WRITE_FILE_SCHEMA = create_tool_schema(
name="write_file",
description="Write content to a file. Use when user explicitly asks to save content.",
parameters={
"path": {
**PARAM_STRING,
"description": "Path to file",
},
"content": {
**PARAM_STRING,
"description": "Content to write",
},
"append": {
**PARAM_BOOLEAN,
"description": "Append to file instead of overwrite",
"default": False,
},
},
required=["path", "content"],
returns=RETURN_STRING,
)
# List Directory Schema
LIST_DIR_SCHEMA = create_tool_schema(
name="list_directory",
description="List files in a directory.",
parameters={
"path": {
**PARAM_STRING,
"description": "Directory path (default: current)",
"default": ".",
},
"pattern": {
**PARAM_STRING,
"description": "File pattern filter (e.g., '*.py')",
"default": "*",
},
},
returns=RETURN_STRING,
)
def _resolve_path(path: str, base_dir: str | Path | None = None) -> Path:
"""Resolve path with proper handling of ~, absolute, and relative paths.
Resolution order:
1. If absolute, use as-is (after expanding ~)
2. If relative, resolve relative to base_dir (or repo root)
"""
from config import settings
p = Path(path)
# Expand ~ to user's home directory
p = p.expanduser()
if p.is_absolute():
return p.resolve()
# Use provided base_dir, or fall back to settings.repo_root
if base_dir is None:
base = Path(settings.repo_root)
else:
base = Path(base_dir)
# Resolve relative to base
return (base / p).resolve()
def read_file(path: str, limit: int = 0) -> str:
"""Read file contents."""
try:
filepath = _resolve_path(path)
if not filepath.exists():
return f"Error: File not found: {path}"
if not filepath.is_file():
return f"Error: Path is not a file: {path}"
content = filepath.read_text()
if limit > 0:
lines = content.split("\n")[:limit]
content = "\n".join(lines)
if len(content.split("\n")) == limit:
content += f"\n\n... [{limit} lines shown]"
return content
except Exception as exc:
logger.error("Read file failed: %s", exc)
return f"Error reading file: {exc}"
def write_file(path: str, content: str, append: bool = False) -> str:
"""Write content to file."""
try:
filepath = _resolve_path(path)
# Ensure directory exists
filepath.parent.mkdir(parents=True, exist_ok=True)
mode = "a" if append else "w"
filepath.write_text(content)
action = "appended to" if append else "wrote"
return f"Successfully {action} {filepath}"
except Exception as exc:
logger.error("Write file failed: %s", exc)
return f"Error writing file: {exc}"
def list_directory(path: str = ".", pattern: str = "*") -> str:
"""List directory contents."""
try:
dirpath = _resolve_path(path)
if not dirpath.exists():
return f"Error: Directory not found: {path}"
if not dirpath.is_dir():
return f"Error: Path is not a directory: {path}"
items = list(dirpath.glob(pattern))
files = []
dirs = []
for item in items:
if item.is_dir():
dirs.append(f"📁 {item.name}/")
else:
size = item.stat().st_size
size_str = f"{size}B" if size < 1024 else f"{size // 1024}KB"
files.append(f"📄 {item.name} ({size_str})")
result = [f"Contents of {dirpath}:", ""]
result.extend(sorted(dirs))
result.extend(sorted(files))
return "\n".join(result)
except Exception as exc:
logger.error("List directory failed: %s", exc)
return f"Error listing directory: {exc}"
# Register with MCP
register_tool(name="read_file", schema=READ_FILE_SCHEMA, category="files")(read_file)
register_tool(
name="write_file",
schema=WRITE_FILE_SCHEMA,
category="files",
requires_confirmation=True,
)(write_file)
register_tool(name="list_directory", schema=LIST_DIR_SCHEMA, category="files")(
list_directory
)

451
src/creative/tools/git_tools.py
View File

@@ -1,451 +0,0 @@
"""Git operations tools for Forge, Helm, and Timmy personas.
Provides a full set of git commands that agents can execute against
the local repository. Uses subprocess with explicit working directory
to ensure commands run from the repo root.
All functions return plain dicts so they're easily serialisable for
tool-call results, Spark event capture, and WebSocket broadcast.
"""
from __future__ import annotations
import logging
import os
import subprocess
from pathlib import Path
from typing import Optional
logger = logging.getLogger(__name__)
def _find_repo_root() -> str:
"""Walk up from this file's location to find the .git directory."""
path = os.path.dirname(os.path.abspath(__file__))
# Start from project root (3 levels up from src/tools/git_tools.py)
path = os.path.dirname(os.path.dirname(os.path.dirname(path)))
while path != os.path.dirname(path):
if os.path.exists(os.path.join(path, '.git')):
return path
path = os.path.dirname(path)
# Fallback to config repo_root
try:
from config import settings
return settings.repo_root
except Exception:
return os.getcwd()
# Module-level constant for repo root
REPO_ROOT = _find_repo_root()
logger.info(f"Git repo root: {REPO_ROOT}")
def _run_git_command(args: list[str], cwd: Optional[str] = None) -> tuple[int, str, str]:
"""Run a git command with proper working directory.
Args:
args: Git command arguments (e.g., ["log", "--oneline", "-5"])
cwd: Working directory (defaults to REPO_ROOT)
Returns:
Tuple of (returncode, stdout, stderr)
"""
cmd = ["git"] + args
working_dir = cwd or REPO_ROOT
try:
result = subprocess.run(
cmd,
cwd=working_dir,
capture_output=True,
text=True,
timeout=30,
)
return result.returncode, result.stdout, result.stderr
except subprocess.TimeoutExpired:
return -1, "", "Command timed out after 30 seconds"
except Exception as exc:
return -1, "", str(exc)
# ── Repository management ────────────────────────────────────────────────────
def git_clone(url: str, dest: str | Path) -> dict:
"""Clone a remote repository to a local path."""
returncode, stdout, stderr = _run_git_command(
["clone", url, str(dest)],
cwd=None # Clone uses current directory as parent
)
if returncode != 0:
return {"success": False, "error": stderr}
return {
"success": True,
"path": str(dest),
"message": f"Cloned {url} to {dest}",
}
def git_init(path: str | Path) -> dict:
"""Initialise a new git repository at *path*."""
os.makedirs(path, exist_ok=True)
returncode, stdout, stderr = _run_git_command(["init"], cwd=str(path))
if returncode != 0:
return {"success": False, "error": stderr}
return {"success": True, "path": str(path)}
# ── Status / inspection ──────────────────────────────────────────────────────
def git_status(repo_path: Optional[str] = None) -> dict:
"""Return working-tree status: modified, staged, untracked files."""
cwd = repo_path or REPO_ROOT
returncode, stdout, stderr = _run_git_command(
["status", "--porcelain", "-b"], cwd=cwd
)
if returncode != 0:
return {"success": False, "error": stderr}
# Parse porcelain output
lines = stdout.strip().split("\n") if stdout else []
branch = "unknown"
modified = []
staged = []
untracked = []
for line in lines:
if line.startswith("## "):
branch = line[3:].split("...")[0].strip()
elif len(line) >= 2:
index_status = line[0]
worktree_status = line[1]
filename = line[3:].strip() if len(line) > 3 else ""
if index_status in "MADRC":
staged.append(filename)
if worktree_status in "MD":
modified.append(filename)
if worktree_status == "?":
untracked.append(filename)
return {
"success": True,
"branch": branch,
"is_dirty": bool(modified or staged or untracked),
"modified": modified,
"staged": staged,
"untracked": untracked,
}
def git_diff(
repo_path: Optional[str] = None,
staged: bool = False,
file_path: Optional[str] = None,
) -> dict:
"""Show diff of working tree or staged changes."""
cwd = repo_path or REPO_ROOT
args = ["diff"]
if staged:
args.append("--cached")
if file_path:
args.extend(["--", file_path])
returncode, stdout, stderr = _run_git_command(args, cwd=cwd)
if returncode != 0:
return {"success": False, "error": stderr}
return {"success": True, "diff": stdout, "staged": staged}
def git_log(
repo_path: Optional[str] = None,
max_count: int = 20,
branch: Optional[str] = None,
) -> dict:
"""Return recent commit history as a list of dicts."""
cwd = repo_path or REPO_ROOT
args = ["log", f"--max-count={max_count}", "--format=%H|%h|%s|%an|%ai"]
if branch:
args.append(branch)
returncode, stdout, stderr = _run_git_command(args, cwd=cwd)
if returncode != 0:
return {"success": False, "error": stderr}
commits = []
for line in stdout.strip().split("\n"):
if not line:
continue
parts = line.split("|", 4)
if len(parts) >= 5:
commits.append({
"sha": parts[0],
"short_sha": parts[1],
"message": parts[2],
"author": parts[3],
"date": parts[4],
})
# Get current branch
_, branch_out, _ = _run_git_command(["branch", "--show-current"], cwd=cwd)
current_branch = branch_out.strip() or "main"
return {
"success": True,
"branch": branch or current_branch,
"commits": commits,
}
def git_blame(repo_path: Optional[str] = None, file_path: str = "") -> dict:
"""Show line-by-line authorship for a file."""
if not file_path:
return {"success": False, "error": "file_path is required"}
cwd = repo_path or REPO_ROOT
returncode, stdout, stderr = _run_git_command(
["blame", "--porcelain", file_path], cwd=cwd
)
if returncode != 0:
return {"success": False, "error": stderr}
return {"success": True, "file": file_path, "blame": stdout}
# ── Branching ─────────────────────────────────────────────────────────────────
def git_branch(
repo_path: Optional[str] = None,
create: Optional[str] = None,
switch: Optional[str] = None,
) -> dict:
"""List branches, optionally create or switch to one."""
cwd = repo_path or REPO_ROOT
if create:
returncode, _, stderr = _run_git_command(
["branch", create], cwd=cwd
)
if returncode != 0:
return {"success": False, "error": stderr}
if switch:
returncode, _, stderr = _run_git_command(
["checkout", switch], cwd=cwd
)
if returncode != 0:
return {"success": False, "error": stderr}
# List branches
returncode, stdout, stderr = _run_git_command(
["branch", "-a", "--format=%(refname:short)%(if)%(HEAD)%(then)*%(end)"],
cwd=cwd
)
if returncode != 0:
return {"success": False, "error": stderr}
branches = []
active = ""
for line in stdout.strip().split("\n"):
line = line.strip()
if line.endswith("*"):
active = line[:-1]
branches.append(active)
elif line:
branches.append(line)
return {
"success": True,
"branches": branches,
"active": active,
"created": create,
"switched": switch,
}
# ── Staging & committing ─────────────────────────────────────────────────────
def git_add(repo_path: Optional[str] = None, paths: Optional[list[str]] = None) -> dict:
"""Stage files for commit. *paths* defaults to all modified files."""
cwd = repo_path or REPO_ROOT
if paths:
args = ["add"] + paths
else:
args = ["add", "-A"]
returncode, _, stderr = _run_git_command(args, cwd=cwd)
if returncode != 0:
return {"success": False, "error": stderr}
return {"success": True, "staged": paths or ["all"]}
def git_commit(
repo_path: Optional[str] = None,
message: str = "",
) -> dict:
"""Create a commit with the given message."""
if not message:
return {"success": False, "error": "commit message is required"}
cwd = repo_path or REPO_ROOT
returncode, stdout, stderr = _run_git_command(
["commit", "-m", message], cwd=cwd
)
if returncode != 0:
return {"success": False, "error": stderr}
# Get the commit hash
_, hash_out, _ = _run_git_command(["rev-parse", "HEAD"], cwd=cwd)
commit_hash = hash_out.strip()
return {
"success": True,
"sha": commit_hash,
"short_sha": commit_hash[:8],
"message": message,
}
# ── Remote operations ─────────────────────────────────────────────────────────
def git_push(
repo_path: Optional[str] = None,
remote: str = "origin",
branch: Optional[str] = None,
) -> dict:
"""Push the current (or specified) branch to the remote."""
cwd = repo_path or REPO_ROOT
args = ["push", remote]
if branch:
args.append(branch)
returncode, stdout, stderr = _run_git_command(args, cwd=cwd)
if returncode != 0:
return {"success": False, "error": stderr}
return {"success": True, "remote": remote, "branch": branch or "current"}
def git_pull(
repo_path: Optional[str] = None,
remote: str = "origin",
branch: Optional[str] = None,
) -> dict:
"""Pull from the remote into the working tree."""
cwd = repo_path or REPO_ROOT
args = ["pull", remote]
if branch:
args.append(branch)
returncode, stdout, stderr = _run_git_command(args, cwd=cwd)
if returncode != 0:
return {"success": False, "error": stderr}
return {"success": True, "remote": remote, "branch": branch or "current"}
# ── Stashing ──────────────────────────────────────────────────────────────────
def git_stash(
repo_path: Optional[str] = None,
pop: bool = False,
message: Optional[str] = None,
) -> dict:
"""Stash or pop working-tree changes."""
cwd = repo_path or REPO_ROOT
if pop:
returncode, _, stderr = _run_git_command(["stash", "pop"], cwd=cwd)
if returncode != 0:
return {"success": False, "error": stderr}
return {"success": True, "action": "pop"}
args = ["stash", "push"]
if message:
args.extend(["-m", message])
returncode, _, stderr = _run_git_command(args, cwd=cwd)
if returncode != 0:
return {"success": False, "error": stderr}
return {"success": True, "action": "stash", "message": message}
# ── Tool catalogue ────────────────────────────────────────────────────────────
GIT_TOOL_CATALOG: dict[str, dict] = {
"git_clone": {
"name": "Git Clone",
"description": "Clone a remote repository to a local path",
"fn": git_clone,
},
"git_status": {
"name": "Git Status",
"description": "Show working tree status (modified, staged, untracked)",
"fn": git_status,
},
"git_diff": {
"name": "Git Diff",
"description": "Show diff of working tree or staged changes",
"fn": git_diff,
},
"git_log": {
"name": "Git Log",
"description": "Show recent commit history",
"fn": git_log,
},
"git_blame": {
"name": "Git Blame",
"description": "Show line-by-line authorship for a file",
"fn": git_blame,
},
"git_branch": {
"name": "Git Branch",
"description": "List, create, or switch branches",
"fn": git_branch,
},
"git_add": {
"name": "Git Add",
"description": "Stage files for commit",
"fn": git_add,
},
"git_commit": {
"name": "Git Commit",
"description": "Create a commit with a message",
"fn": git_commit,
},
"git_push": {
"name": "Git Push",
"description": "Push branch to remote repository",
"fn": git_push,
},
"git_pull": {
"name": "Git Pull",
"description": "Pull from remote repository",
"fn": git_pull,
},
"git_stash": {
"name": "Git Stash",
"description": "Stash or pop working tree changes",
"fn": git_stash,
},
}

171
src/creative/tools/image_tools.py
View File

@@ -1,171 +0,0 @@
"""Image generation tools — Pixel persona.
Uses FLUX.2 Klein 4B (or configurable model) via HuggingFace diffusers
for text-to-image generation, storyboard frames, and variations.
All heavy imports are lazy so the module loads instantly even without
a GPU or the ``creative`` extra installed.
"""
from __future__ import annotations
import json
import logging
import uuid
from pathlib import Path
from typing import Optional
logger = logging.getLogger(__name__)
# Lazy-loaded pipeline singleton
_pipeline = None
def _get_pipeline():
"""Lazy-load the FLUX diffusers pipeline."""
global _pipeline
if _pipeline is not None:
return _pipeline
try:
import torch
from diffusers import FluxPipeline
except ImportError:
raise ImportError(
"Creative dependencies not installed. "
"Run: pip install 'timmy-time[creative]'"
)
from config import settings
model_id = getattr(settings, "flux_model_id", "black-forest-labs/FLUX.1-schnell")
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32
logger.info("Loading image model %s on %s …", model_id, device)
_pipeline = FluxPipeline.from_pretrained(
model_id, torch_dtype=dtype,
).to(device)
logger.info("Image model loaded.")
return _pipeline
def _output_dir() -> Path:
from config import settings
d = Path(getattr(settings, "image_output_dir", "data/images"))
d.mkdir(parents=True, exist_ok=True)
return d
def _save_metadata(image_path: Path, meta: dict) -> Path:
meta_path = image_path.with_suffix(".json")
meta_path.write_text(json.dumps(meta, indent=2))
return meta_path
# ── Public tools ──────────────────────────────────────────────────────────────
def generate_image(
prompt: str,
negative_prompt: str = "",
width: int = 1024,
height: int = 1024,
steps: int = 4,
seed: Optional[int] = None,
) -> dict:
"""Generate an image from a text prompt.
Returns dict with ``path``, ``width``, ``height``, and ``prompt``.
"""
pipe = _get_pipeline()
import torch
generator = torch.Generator(device=pipe.device)
if seed is not None:
generator.manual_seed(seed)
image = pipe(
prompt=prompt,
negative_prompt=negative_prompt or None,
width=width,
height=height,
num_inference_steps=steps,
generator=generator,
).images[0]
uid = uuid.uuid4().hex[:12]
out_path = _output_dir() / f"{uid}.png"
image.save(out_path)
meta = {
"id": uid, "prompt": prompt, "negative_prompt": negative_prompt,
"width": width, "height": height, "steps": steps, "seed": seed,
}
_save_metadata(out_path, meta)
return {"success": True, "path": str(out_path), **meta}
def generate_storyboard(
scenes: list[str],
width: int = 1024,
height: int = 576,
steps: int = 4,
) -> dict:
"""Generate a storyboard: one keyframe image per scene description.
Args:
scenes: List of scene description strings.
Returns dict with list of generated frame paths.
"""
frames = []
for i, scene in enumerate(scenes):
result = generate_image(
prompt=scene, width=width, height=height, steps=steps,
)
result["scene_index"] = i
result["scene_description"] = scene
frames.append(result)
return {"success": True, "frame_count": len(frames), "frames": frames}
def image_variations(
prompt: str,
count: int = 4,
width: int = 1024,
height: int = 1024,
steps: int = 4,
) -> dict:
"""Generate multiple variations of the same prompt with different seeds."""
import random
variations = []
for _ in range(count):
seed = random.randint(0, 2**32 - 1)
result = generate_image(
prompt=prompt, width=width, height=height,
steps=steps, seed=seed,
)
variations.append(result)
return {"success": True, "count": len(variations), "variations": variations}
# ── Tool catalogue ────────────────────────────────────────────────────────────
IMAGE_TOOL_CATALOG: dict[str, dict] = {
"generate_image": {
"name": "Generate Image",
"description": "Generate an image from a text prompt using FLUX",
"fn": generate_image,
},
"generate_storyboard": {
"name": "Generate Storyboard",
"description": "Generate keyframe images for a sequence of scenes",
"fn": generate_storyboard,
},
"image_variations": {
"name": "Image Variations",
"description": "Generate multiple variations of the same prompt",
"fn": image_variations,
},
}

70
src/creative/tools/memory_tool.py
View File

@@ -1,70 +0,0 @@
"""Memory search tool.
MCP-compliant tool for searching Timmy's memory.
"""
import logging
from typing import Any
from mcp.registry import register_tool
from mcp.schemas.base import create_tool_schema, PARAM_STRING, PARAM_INTEGER, RETURN_STRING
logger = logging.getLogger(__name__)
MEMORY_SEARCH_SCHEMA = create_tool_schema(
name="memory_search",
description="Search Timmy's memory for past conversations, facts, and context. Use when user asks about previous discussions or when you need to recall something from memory.",
parameters={
"query": {
**PARAM_STRING,
"description": "What to search for in memory",
},
"top_k": {
**PARAM_INTEGER,
"description": "Number of results to return (1-10)",
"default": 5,
"minimum": 1,
"maximum": 10,
},
},
required=["query"],
returns=RETURN_STRING,
)
def memory_search(query: str, top_k: int = 5) -> str:
"""Search Timmy's memory.
Args:
query: Search query
top_k: Number of results
Returns:
Relevant memories from past conversations
"""
try:
from timmy.semantic_memory import memory_search as semantic_search
results = semantic_search(query, top_k=top_k)
if not results:
return "No relevant memories found."
formatted = ["Relevant memories from past conversations:", ""]
for i, (content, score) in enumerate(results, 1):
relevance = "🔥" if score > 0.8 else "⭐" if score > 0.5 else "📄"
formatted.append(f"{relevance} [{i}] (score: {score:.2f})")
formatted.append(f" {content[:300]}...")
formatted.append("")
return "\n".join(formatted)
except Exception as exc:
logger.error("Memory search failed: %s", exc)
return f"Memory search error: {exc}"
# Register with MCP
register_tool(name="memory_search", schema=MEMORY_SEARCH_SCHEMA, category="memory")(memory_search)

210
src/creative/tools/music_tools.py
View File

@@ -1,210 +0,0 @@
"""Music generation tools — Lyra persona.
Uses ACE-Step 1.5 for full song generation with vocals, instrumentals,
and lyrics. Falls back gracefully when the ``creative`` extra is not
installed.
All heavy imports are lazy — the module loads instantly without GPU.
"""
from __future__ import annotations
import json
import logging
import uuid
from pathlib import Path
from typing import Optional
logger = logging.getLogger(__name__)
# Lazy-loaded model singleton
_model = None
def _get_model():
"""Lazy-load the ACE-Step music generation model."""
global _model
if _model is not None:
return _model
try:
from ace_step import ACEStep
except ImportError:
raise ImportError(
"ACE-Step not installed. Run: pip install 'timmy-time[creative]'"
)
from config import settings
model_name = getattr(settings, "ace_step_model", "ace-step/ACE-Step-v1.5")
logger.info("Loading music model %s …", model_name)
_model = ACEStep(model_name)
logger.info("Music model loaded.")
return _model
def _output_dir() -> Path:
from config import settings
d = Path(getattr(settings, "music_output_dir", "data/music"))
d.mkdir(parents=True, exist_ok=True)
return d
def _save_metadata(audio_path: Path, meta: dict) -> Path:
meta_path = audio_path.with_suffix(".json")
meta_path.write_text(json.dumps(meta, indent=2))
return meta_path
# ── Supported genres ──────────────────────────────────────────────────────────
GENRES = [
"pop", "rock", "hip-hop", "r&b", "jazz", "blues", "country",
"electronic", "classical", "folk", "reggae", "metal", "punk",
"soul", "funk", "latin", "ambient", "lo-fi", "cinematic",
]
# ── Public tools ──────────────────────────────────────────────────────────────
def generate_song(
lyrics: str,
genre: str = "pop",
duration: int = 120,
language: str = "en",
title: Optional[str] = None,
) -> dict:
"""Generate a full song with vocals and instrumentals from lyrics.
Args:
lyrics: Song lyrics text.
genre: Musical genre / style tag.
duration: Target duration in seconds (30–240).
language: ISO language code (19 languages supported).
title: Optional song title for metadata.
Returns dict with ``path``, ``duration``, ``genre``, etc.
"""
model = _get_model()
duration = max(30, min(240, duration))
uid = uuid.uuid4().hex[:12]
out_path = _output_dir() / f"{uid}.wav"
logger.info("Generating song: genre=%s duration=%ds …", genre, duration)
audio = model.generate(
lyrics=lyrics,
genre=genre,
duration=duration,
language=language,
)
audio.save(str(out_path))
meta = {
"id": uid, "title": title or f"Untitled ({genre})",
"lyrics": lyrics, "genre": genre,
"duration": duration, "language": language,
}
_save_metadata(out_path, meta)
return {"success": True, "path": str(out_path), **meta}
def generate_instrumental(
prompt: str,
genre: str = "cinematic",
duration: int = 60,
) -> dict:
"""Generate an instrumental track from a text prompt (no vocals).
Args:
prompt: Description of the desired music.
genre: Musical genre / style tag.
duration: Target duration in seconds (15–180).
"""
model = _get_model()
duration = max(15, min(180, duration))
uid = uuid.uuid4().hex[:12]
out_path = _output_dir() / f"{uid}.wav"
logger.info("Generating instrumental: genre=%s …", genre)
audio = model.generate(
lyrics="",
genre=genre,
duration=duration,
prompt=prompt,
)
audio.save(str(out_path))
meta = {
"id": uid, "prompt": prompt, "genre": genre,
"duration": duration, "instrumental": True,
}
_save_metadata(out_path, meta)
return {"success": True, "path": str(out_path), **meta}
def generate_vocals(
lyrics: str,
style: str = "pop",
duration: int = 60,
language: str = "en",
) -> dict:
"""Generate a vocal-only track from lyrics.
Useful for layering over custom instrumentals.
"""
model = _get_model()
duration = max(15, min(180, duration))
uid = uuid.uuid4().hex[:12]
out_path = _output_dir() / f"{uid}.wav"
audio = model.generate(
lyrics=lyrics,
genre=f"{style} acapella vocals",
duration=duration,
language=language,
)
audio.save(str(out_path))
meta = {
"id": uid, "lyrics": lyrics, "style": style,
"duration": duration, "vocals_only": True,
}
_save_metadata(out_path, meta)
return {"success": True, "path": str(out_path), **meta}
def list_genres() -> dict:
"""Return the list of supported genre / style tags."""
return {"success": True, "genres": GENRES}
# ── Tool catalogue ────────────────────────────────────────────────────────────
MUSIC_TOOL_CATALOG: dict[str, dict] = {
"generate_song": {
"name": "Generate Song",
"description": "Generate a full song with vocals + instrumentals from lyrics",
"fn": generate_song,
},
"generate_instrumental": {
"name": "Generate Instrumental",
"description": "Generate an instrumental track from a text prompt",
"fn": generate_instrumental,
},
"generate_vocals": {
"name": "Generate Vocals",
"description": "Generate a vocal-only track from lyrics",
"fn": generate_vocals,
},
"list_genres": {
"name": "List Genres",
"description": "List supported music genre / style tags",
"fn": list_genres,
},
}

824
src/creative/tools/self_edit.py
View File

@@ -1,824 +0,0 @@
"""Self-Edit MCP Tool — Timmy's ability to modify its own source code.
This is the core self-modification orchestrator that:
1. Receives task descriptions
2. Queries codebase indexer for relevant files
3. Queries modification journal for similar past attempts
4. Creates feature branches via GitSafety
5. Plans changes with LLM
6. Executes via Aider (preferred) or direct editing (fallback)
7. Runs tests via pytest
8. Commits on success, rolls back on failure
9. Logs outcomes to ModificationJournal
10. Generates reflections
Usage:
from creative.tools.self_edit import self_edit_tool
from mcp.registry import tool_registry
# Register with MCP
tool_registry.register("self_edit", self_edit_schema, self_edit_tool)
# Invoke
result = await tool_registry.execute("self_edit", {
"task_description": "Add error handling to health endpoint"
})
"""
from __future__ import annotations
import ast
import asyncio
import logging
import os
import subprocess
from dataclasses import dataclass, field
from datetime import datetime
from pathlib import Path
from typing import Any, Optional
from config import settings
# Phase 1 imports
from self_coding import (
CodebaseIndexer,
GitSafety,
ModificationAttempt,
ModificationJournal,
Outcome,
ReflectionService,
)
logger = logging.getLogger(__name__)
# Safety constraints (Phase 1 hard limits)
MAX_FILES_PER_COMMIT = 3
MAX_LINES_CHANGED = 100
PROTECTED_FILES = {
"src/tools/self_edit.py",
"src/self_coding/git_safety.py",
"src/self_coding/codebase_indexer.py",
"src/self_coding/modification_journal.py",
"src/self_coding/reflection.py",
}
MAX_RETRIES = 3
@dataclass
class SelfEditResult:
"""Result of a self-edit operation."""
success: bool
message: str
attempt_id: Optional[int] = None
files_modified: list[str] = field(default_factory=list)
commit_hash: Optional[str] = None
test_results: str = ""
diff: str = ""
@dataclass
class EditPlan:
"""Plan for a self-edit operation."""
approach: str
files_to_modify: list[str]
files_to_create: list[str]
tests_to_add: list[str]
explanation: str
class SelfEditTool:
"""Self-modification orchestrator.
This class encapsulates the complete self-edit workflow:
- Pre-flight checks
- Context gathering (indexer + journal)
- Branch creation
- Edit planning (LLM)
- Execution (Aider or direct)
- Testing
- Commit/rollback
- Logging and reflection
Usage:
tool = SelfEditTool(repo_path="/path/to/repo")
result = await tool.execute("Add error handling to health endpoint")
"""
def __init__(
self,
repo_path: Optional[Path] = None,
llm_adapter: Optional[object] = None,
) -> None:
"""Initialize SelfEditTool.
Args:
repo_path: Path to repository. Defaults to current directory.
llm_adapter: LLM adapter for planning and reflection
"""
self.repo_path = Path(repo_path) if repo_path else Path.cwd()
self.llm_adapter = llm_adapter
# Initialize Phase 1 services
self.git = GitSafety(repo_path=self.repo_path)
self.indexer = CodebaseIndexer(repo_path=self.repo_path)
self.journal = ModificationJournal()
self.reflection = ReflectionService(llm_adapter=llm_adapter)
# Ensure codebase is indexed
self._indexing_done = False
logger.info("SelfEditTool initialized for %s", self.repo_path)
async def _ensure_indexed(self) -> None:
"""Ensure codebase is indexed."""
if not self._indexing_done:
await self.indexer.index_changed()
self._indexing_done = True
async def execute(
self,
task_description: str,
context: Optional[dict] = None,
) -> SelfEditResult:
"""Execute a self-edit task.
This is the main entry point for self-modification.
Args:
task_description: What to do (e.g., "Add error handling")
context: Optional additional context
Returns:
SelfEditResult with success/failure details
"""
logger.info("Starting self-edit: %s", task_description[:50])
try:
# Step 1: Pre-flight checks
if not await self._preflight_checks():
return SelfEditResult(
success=False,
message="Pre-flight checks failed. See logs for details.",
)
# Step 2: Gather context
await self._ensure_indexed()
relevant_files = await self._get_relevant_files(task_description)
similar_attempts = await self._get_similar_attempts(task_description)
# Step 3: Create feature branch
branch_name = f"timmy/self-edit/{datetime.now().strftime('%Y%m%d-%H%M%S')}"
await self.git.create_branch(branch_name)
logger.info("Created branch: %s", branch_name)
# Step 4: Take snapshot for rollback
snapshot = await self.git.snapshot(run_tests=False)
# Step 5: Plan the edit
plan = await self._plan_edit(
task_description,
relevant_files,
similar_attempts,
)
# Validate plan against safety constraints
if not self._validate_plan(plan):
return SelfEditResult(
success=False,
message=f"Plan violates safety constraints: {plan.files_to_modify}",
)
# Step 6: Execute the edit
execution_result = await self._execute_edit(plan, task_description)
if not execution_result["success"]:
# Attempt retries
for retry in range(MAX_RETRIES):
logger.info("Retry %d/%d", retry + 1, MAX_RETRIES)
# Rollback to clean state
await self.git.rollback(snapshot)
# Try again with adjusted approach
execution_result = await self._execute_edit(
plan,
task_description,
retry_count=retry + 1,
)
if execution_result["success"]:
break
if not execution_result["success"]:
# Final rollback and log failure
await self.git.rollback(snapshot)
await self.git._run_git("checkout", "main") # Return to main
attempt_id = await self._log_failure(
task_description,
plan,
execution_result["test_output"],
execution_result.get("error", "Unknown error"),
)
return SelfEditResult(
success=False,
message=f"Failed after {MAX_RETRIES} retries",
attempt_id=attempt_id,
test_results=execution_result.get("test_output", ""),
)
# Step 7: Commit and merge
commit_hash = await self.git.commit(
message=f"Self-edit: {task_description[:50]}",
files=plan.files_to_modify + plan.files_to_create + plan.tests_to_add,
)
# Merge to main (tests already passed in execution)
await self.git.merge_to_main(branch_name, require_tests=False)
# Step 8: Log success
diff = await self.git.get_diff(snapshot.commit_hash, commit_hash)
attempt_id = await self._log_success(
task_description,
plan,
commit_hash,
execution_result.get("test_output", ""),
diff,
)
return SelfEditResult(
success=True,
message=f"Successfully modified {len(plan.files_to_modify)} files",
attempt_id=attempt_id,
files_modified=plan.files_to_modify,
commit_hash=commit_hash,
test_results=execution_result.get("test_output", ""),
diff=diff,
)
except Exception as e:
logger.exception("Self-edit failed with exception")
return SelfEditResult(
success=False,
message=f"Exception: {str(e)}",
)
async def _preflight_checks(self) -> bool:
"""Run pre-flight safety checks.
Returns:
True if all checks pass
"""
# Check if repo is clean
if not await self.git.is_clean():
logger.error("Pre-flight failed: Working directory not clean")
return False
# Check if we're on main
current_branch = await self.git.get_current_branch()
if current_branch != self.git.main_branch:
logger.error("Pre-flight failed: Not on %s branch (on %s)",
self.git.main_branch, current_branch)
return False
# Check if self-modification is enabled
if not getattr(settings, 'self_modify_enabled', True):
logger.error("Pre-flight failed: Self-modification disabled in config")
return False
return True
async def _get_relevant_files(self, task_description: str) -> list[str]:
"""Get files relevant to the task.
Args:
task_description: Task to find relevant files for
Returns:
List of file paths
"""
files = await self.indexer.get_relevant_files(task_description, limit=10)
# Filter to only files with test coverage
files_with_tests = [
f for f in files
if await self.indexer.has_test_coverage(f)
]
logger.info("Found %d relevant files (%d with tests)",
len(files), len(files_with_tests))
return files_with_tests[:MAX_FILES_PER_COMMIT]
async def _get_similar_attempts(
self,
task_description: str,
) -> list[ModificationAttempt]:
"""Get similar past modification attempts.
Args:
task_description: Task to find similar attempts for
Returns:
List of similar attempts
"""
similar = await self.journal.find_similar(task_description, limit=5)
logger.info("Found %d similar past attempts", len(similar))
return similar
async def _plan_edit(
self,
task_description: str,
relevant_files: list[str],
similar_attempts: list[ModificationAttempt],
) -> EditPlan:
"""Plan the edit using LLM.
Args:
task_description: What to do
relevant_files: Files that might need modification
similar_attempts: Similar past attempts for context
Returns:
EditPlan with approach and file list
"""
if not self.llm_adapter:
# Fallback: simple plan
return EditPlan(
approach=f"Edit files to implement: {task_description}",
files_to_modify=relevant_files[:MAX_FILES_PER_COMMIT],
files_to_create=[],
tests_to_add=[],
explanation="No LLM available, using heuristic plan",
)
# Build prompt with context
codebase_summary = await self.indexer.get_summary(max_tokens=2000)
similar_context = ""
if similar_attempts:
similar_context = "\n\nSimilar past attempts:\n"
for attempt in similar_attempts:
similar_context += f"- {attempt.task_description} ({attempt.outcome.value})\n"
if attempt.reflection:
similar_context += f" Lesson: {attempt.reflection[:100]}...\n"
prompt = f"""You are planning a code modification for a Python project.
Task: {task_description}
Codebase Summary:
{codebase_summary}
Potentially relevant files (all have test coverage):
{chr(10).join(f"- {f}" for f in relevant_files)}
{similar_context}
Create a plan for implementing this task. You can modify at most {MAX_FILES_PER_COMMIT} files.
Respond in this format:
APPROACH: <brief description of the approach>
FILES_TO_MODIFY: <comma-separated list of file paths>
FILES_TO_CREATE: <comma-separated list of new file paths (if any)>
TESTS_TO_ADD: <comma-separated list of test files to add/modify>
EXPLANATION: <brief explanation of why this approach>
"""
try:
response = await self.llm_adapter.chat(message=prompt)
content = response.content
# Parse response
approach = self._extract_field(content, "APPROACH")
files_to_modify = self._parse_list(self._extract_field(content, "FILES_TO_MODIFY"))
files_to_create = self._parse_list(self._extract_field(content, "FILES_TO_CREATE"))
tests_to_add = self._parse_list(self._extract_field(content, "TESTS_TO_ADD"))
explanation = self._extract_field(content, "EXPLANATION")
return EditPlan(
approach=approach or "No approach specified",
files_to_modify=files_to_modify[:MAX_FILES_PER_COMMIT],
files_to_create=files_to_create,
tests_to_add=tests_to_add,
explanation=explanation or "No explanation provided",
)
except Exception as e:
logger.error("LLM planning failed: %s", e)
return EditPlan(
approach=f"Fallback: Modify relevant files for {task_description}",
files_to_modify=relevant_files[:MAX_FILES_PER_COMMIT],
files_to_create=[],
tests_to_add=[],
explanation=f"LLM failed, using fallback: {e}",
)
def _extract_field(self, content: str, field_name: str) -> str:
"""Extract a field from LLM response."""
for line in content.split("\n"):
if line.startswith(f"{field_name}:"):
return line.split(":", 1)[1].strip()
return ""
def _parse_list(self, text: str) -> list[str]:
"""Parse comma-separated list."""
if not text or text.lower() in ("none", "n/a", ""):
return []
return [item.strip() for item in text.split(",") if item.strip()]
def _validate_plan(self, plan: EditPlan) -> bool:
"""Validate plan against safety constraints.
Args:
plan: EditPlan to validate
Returns:
True if plan is valid
"""
# Check file count
if len(plan.files_to_modify) > MAX_FILES_PER_COMMIT:
logger.error("Plan modifies too many files: %d > %d",
len(plan.files_to_modify), MAX_FILES_PER_COMMIT)
return False
# Check for protected files
for file_path in plan.files_to_modify:
if file_path in PROTECTED_FILES:
logger.error("Plan tries to modify protected file: %s", file_path)
return False
# Check all files have test coverage
for file_path in plan.files_to_modify:
# This is async, so we check in _get_relevant_files
pass
return True
async def _execute_edit(
self,
plan: EditPlan,
task_description: str,
retry_count: int = 0,
) -> dict:
"""Execute the edit using Aider or direct editing.
Args:
plan: EditPlan to execute
task_description: Original task description
retry_count: Current retry attempt
Returns:
Dict with success, test_output, error
"""
all_files = plan.files_to_modify + plan.files_to_create
if not all_files:
return {"success": False, "error": "No files to modify"}
# Try Aider first
if await self._aider_available():
return await self._execute_with_aider(plan, task_description, all_files)
else:
# Fallback to direct editing
return await self._execute_direct_edit(plan, task_description)
async def _aider_available(self) -> bool:
"""Check if Aider is available."""
try:
result = await asyncio.create_subprocess_exec(
"aider", "--version",
stdout=asyncio.subprocess.DEVNULL,
stderr=asyncio.subprocess.DEVNULL,
)
await result.wait()
return result.returncode == 0
except FileNotFoundError:
return False
async def _execute_with_aider(
self,
plan: EditPlan,
task_description: str,
files: list[str],
) -> dict:
"""Execute edit using Aider.
Args:
plan: EditPlan
task_description: Task description
files: Files to edit
Returns:
Dict with success, test_output
"""
cmd = [
"aider",
"--model", "ollama_chat/qwen2.5-coder:14b-instruct",
"--auto-test",
"--test-cmd", "python -m pytest tests/ -xvs",
"--yes",
"--no-git",
"--message", f"{task_description}\n\nApproach: {plan.approach}",
] + files
logger.info("Running Aider: %s", " ".join(cmd))
try:
proc = await asyncio.create_subprocess_exec(
*cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.STDOUT,
cwd=self.repo_path,
)
stdout, _ = await asyncio.wait_for(
proc.communicate(),
timeout=300.0,
)
output = stdout.decode() if stdout else ""
# Check if tests passed
success = proc.returncode == 0 and "passed" in output.lower()
return {
"success": success,
"test_output": output,
}
except asyncio.TimeoutError:
logger.error("Aider timed out after 300s")
return {
"success": False,
"error": "Timeout",
"test_output": "Aider timed out after 300s",
}
except Exception as e:
logger.error("Aider execution failed: %s", e)
return {
"success": False,
"error": str(e),
"test_output": "",
}
async def _execute_direct_edit(
self,
plan: EditPlan,
task_description: str,
) -> dict:
"""Execute edit via direct file modification (fallback).
Args:
plan: EditPlan
task_description: Task description
Returns:
Dict with success, test_output
"""
if not self.llm_adapter:
return {
"success": False,
"error": "No LLM adapter for direct editing",
}
# Edit each file
for file_path in plan.files_to_modify:
full_path = self.repo_path / file_path
if not full_path.exists():
logger.error("File does not exist: %s", file_path)
continue
try:
content = full_path.read_text()
# Build edit prompt
edit_prompt = f"""Edit this Python file to implement the task.
Task: {task_description}
Approach: {plan.approach}
Current file content:
```python
{content}
```
Provide the complete new file content. Only return the code, no explanation.
"""
response = await self.llm_adapter.chat(message=edit_prompt)
new_content = response.content
# Strip code fences if present
new_content = self._strip_code_fences(new_content)
# Validate with AST
try:
ast.parse(new_content)
except SyntaxError as e:
logger.error("Generated code has syntax error: %s", e)
return {
"success": False,
"error": f"Syntax error in generated code: {e}",
}
# Write file
full_path.write_text(new_content)
logger.info("Modified: %s", file_path)
except Exception as e:
logger.error("Failed to edit %s: %s", file_path, e)
return {
"success": False,
"error": f"Failed to edit {file_path}: {e}",
}
# Run tests
return await self._run_tests()
def _strip_code_fences(self, content: str) -> str:
"""Strip markdown code fences from content."""
lines = content.split("\n")
# Remove opening fence
if lines and lines[0].startswith("```"):
lines = lines[1:]
# Remove closing fence
if lines and lines[-1].startswith("```"):
lines = lines[:-1]
return "\n".join(lines)
async def _run_tests(self) -> dict:
"""Run tests and return results.
Returns:
Dict with success, test_output
"""
cmd = ["python", "-m", "pytest", "tests/", "-x", "--tb=short"]
try:
proc = await asyncio.create_subprocess_exec(
*cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.STDOUT,
cwd=self.repo_path,
)
stdout, _ = await asyncio.wait_for(
proc.communicate(),
timeout=120.0,
)
output = stdout.decode() if stdout else ""
return {
"success": proc.returncode == 0,
"test_output": output,
}
except asyncio.TimeoutError:
return {
"success": False,
"error": "Tests timed out",
"test_output": "Timeout after 120s",
}
except Exception as e:
return {
"success": False,
"error": str(e),
"test_output": "",
}
async def _log_success(
self,
task_description: str,
plan: EditPlan,
commit_hash: str,
test_results: str,
diff: str,
) -> int:
"""Log successful attempt.
Returns:
Attempt ID
"""
attempt = ModificationAttempt(
task_description=task_description,
approach=plan.approach,
files_modified=plan.files_to_modify + plan.files_to_create,
diff=diff[:5000], # Truncate for storage
test_results=test_results,
outcome=Outcome.SUCCESS,
)
attempt_id = await self.journal.log_attempt(attempt)
# Generate and store reflection
reflection_text = await self.reflection.reflect_on_attempt(attempt)
await self.journal.update_reflection(attempt_id, reflection_text)
return attempt_id
async def _log_failure(
self,
task_description: str,
plan: EditPlan,
test_results: str,
error: str,
) -> int:
"""Log failed attempt.
Returns:
Attempt ID
"""
attempt = ModificationAttempt(
task_description=task_description,
approach=plan.approach,
files_modified=plan.files_to_modify,
test_results=test_results,
outcome=Outcome.FAILURE,
failure_analysis=error,
retry_count=MAX_RETRIES,
)
attempt_id = await self.journal.log_attempt(attempt)
# Generate reflection even for failures
reflection_text = await self.reflection.reflect_on_attempt(attempt)
await self.journal.update_reflection(attempt_id, reflection_text)
return attempt_id
# MCP Tool Schema
self_edit_schema = {
"type": "object",
"properties": {
"task_description": {
"type": "string",
"description": "Description of the code modification to make",
},
"context": {
"type": "object",
"description": "Optional additional context for the modification",
},
},
"required": ["task_description"],
}
# Global tool instance (singleton pattern)
_self_edit_tool: Optional[SelfEditTool] = None
async def self_edit_tool(task_description: str, context: Optional[dict] = None) -> dict:
"""MCP tool entry point for self-edit.
Args:
task_description: What to modify
context: Optional context
Returns:
Dict with result
"""
global _self_edit_tool
if _self_edit_tool is None:
_self_edit_tool = SelfEditTool()
result = await _self_edit_tool.execute(task_description, context)
return {
"success": result.success,
"message": result.message,
"attempt_id": result.attempt_id,
"files_modified": result.files_modified,
"commit_hash": result.commit_hash,
"test_results": result.test_results,
}
def register_self_edit_tool(registry: Any, llm_adapter: Optional[object] = None) -> None:
"""Register the self-edit tool with MCP registry.
Args:
registry: MCP ToolRegistry
llm_adapter: Optional LLM adapter
"""
global _self_edit_tool
_self_edit_tool = SelfEditTool(llm_adapter=llm_adapter)
registry.register(
name="self_edit",
schema=self_edit_schema,
handler=self_edit_tool,
category="self_coding",
requires_confirmation=True, # Safety: require user approval
tags=["self-modification", "code-generation"],
source_module="creative.tools.self_edit",
)
logger.info("Self-edit tool registered with MCP")

51
src/creative/tools/system_status.py
View File

@@ -1,51 +0,0 @@
"""System status introspection tool for Timmy.
MCP-compliant tool that gives Timmy live access to his own system state:
task queue, agent roster, memory tiers, uptime, and service health.
"""
import json
import logging
from mcp.registry import register_tool
from mcp.schemas.base import create_tool_schema, RETURN_STRING
logger = logging.getLogger(__name__)
SYSTEM_STATUS_SCHEMA = create_tool_schema(
name="system_status",
description=(
"Get live system status including task queue counts, agent roster, "
"memory tier health, uptime, and service connectivity. "
"Use this when asked about your status, what you're working on, "
"agent health, or system metrics. Never guess — always call this tool."
),
parameters={},
required=[],
returns=RETURN_STRING,
)
def system_status() -> str:
"""Return comprehensive live system status as formatted text.
Returns:
JSON-formatted string with system, task_queue, agents, memory sections.
"""
try:
from timmy.tools_intro import get_live_system_status
status = get_live_system_status()
return json.dumps(status, indent=2, default=str)
except Exception as exc:
logger.error("system_status tool failed: %s", exc)
return json.dumps({"error": str(exc)})
# Register with MCP
register_tool(
name="system_status",
schema=SYSTEM_STATUS_SCHEMA,
category="system",
)(system_status)

206
src/creative/tools/video_tools.py
View File

@@ -1,206 +0,0 @@
"""Video generation tools — Reel persona.
Uses Wan 2.1 (via HuggingFace diffusers) for text-to-video and
image-to-video generation. Heavy imports are lazy.
"""
from __future__ import annotations
import json
import logging
import uuid
from pathlib import Path
from typing import Optional
logger = logging.getLogger(__name__)
# Lazy-loaded pipeline singletons
_t2v_pipeline = None
_i2v_pipeline = None
def _get_t2v_pipeline():
"""Lazy-load the text-to-video pipeline (Wan 2.1)."""
global _t2v_pipeline
if _t2v_pipeline is not None:
return _t2v_pipeline
try:
import torch
from diffusers import DiffusionPipeline
except ImportError:
raise ImportError(
"Creative dependencies not installed. "
"Run: pip install 'timmy-time[creative]'"
)
from config import settings
model_id = getattr(settings, "wan_model_id", "Wan-AI/Wan2.1-T2V-1.3B")
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32
logger.info("Loading video model %s on %s …", model_id, device)
_t2v_pipeline = DiffusionPipeline.from_pretrained(
model_id, torch_dtype=dtype,
).to(device)
logger.info("Video model loaded.")
return _t2v_pipeline
def _output_dir() -> Path:
from config import settings
d = Path(getattr(settings, "video_output_dir", "data/video"))
d.mkdir(parents=True, exist_ok=True)
return d
def _save_metadata(video_path: Path, meta: dict) -> Path:
meta_path = video_path.with_suffix(".json")
meta_path.write_text(json.dumps(meta, indent=2))
return meta_path
def _export_frames_to_mp4(frames, out_path: Path, fps: int = 24) -> None:
"""Export a list of PIL Image frames to an MP4 file using moviepy."""
import numpy as np
from moviepy import ImageSequenceClip
frame_arrays = [np.array(f) for f in frames]
clip = ImageSequenceClip(frame_arrays, fps=fps)
clip.write_videofile(
str(out_path), codec="libx264", audio=False, logger=None,
)
# ── Resolution presets ────────────────────────────────────────────────────────
RESOLUTION_PRESETS = {
"480p": (854, 480),
"720p": (1280, 720),
}
VIDEO_STYLES = [
"cinematic", "anime", "documentary", "abstract",
"timelapse", "slow-motion", "music-video", "vlog",
]
# ── Public tools ──────────────────────────────────────────────────────────────
def generate_video_clip(
prompt: str,
duration: int = 5,
resolution: str = "480p",
fps: int = 24,
seed: Optional[int] = None,
) -> dict:
"""Generate a short video clip from a text prompt.
Args:
prompt: Text description of the desired video.
duration: Target duration in seconds (2–10).
resolution: "480p" or "720p".
fps: Frames per second.
seed: Optional seed for reproducibility.
Returns dict with ``path``, ``duration``, ``resolution``.
"""
pipe = _get_t2v_pipeline()
import torch
duration = max(2, min(10, duration))
w, h = RESOLUTION_PRESETS.get(resolution, RESOLUTION_PRESETS["480p"])
num_frames = duration * fps
generator = torch.Generator(device=pipe.device)
if seed is not None:
generator.manual_seed(seed)
logger.info("Generating %ds video at %s …", duration, resolution)
result = pipe(
prompt=prompt,
num_frames=num_frames,
width=w,
height=h,
generator=generator,
)
frames = result.frames[0] if hasattr(result, "frames") else result.images
uid = uuid.uuid4().hex[:12]
out_path = _output_dir() / f"{uid}.mp4"
_export_frames_to_mp4(frames, out_path, fps=fps)
meta = {
"id": uid, "prompt": prompt, "duration": duration,
"resolution": resolution, "fps": fps, "seed": seed,
}
_save_metadata(out_path, meta)
return {"success": True, "path": str(out_path), **meta}
def image_to_video(
image_path: str,
prompt: str = "",
duration: int = 5,
fps: int = 24,
) -> dict:
"""Animate a still image into a video clip.
Args:
image_path: Path to the source image.
prompt: Optional motion / style guidance.
duration: Target duration in seconds (2–10).
"""
pipe = _get_t2v_pipeline()
from PIL import Image
duration = max(2, min(10, duration))
img = Image.open(image_path).convert("RGB")
num_frames = duration * fps
logger.info("Animating image %s → %ds video …", image_path, duration)
result = pipe(
prompt=prompt or "animate this image with natural motion",
image=img,
num_frames=num_frames,
)
frames = result.frames[0] if hasattr(result, "frames") else result.images
uid = uuid.uuid4().hex[:12]
out_path = _output_dir() / f"{uid}.mp4"
_export_frames_to_mp4(frames, out_path, fps=fps)
meta = {
"id": uid, "source_image": image_path,
"prompt": prompt, "duration": duration, "fps": fps,
}
_save_metadata(out_path, meta)
return {"success": True, "path": str(out_path), **meta}
def list_video_styles() -> dict:
"""Return supported video style presets."""
return {"success": True, "styles": VIDEO_STYLES, "resolutions": list(RESOLUTION_PRESETS.keys())}
# ── Tool catalogue ────────────────────────────────────────────────────────────
VIDEO_TOOL_CATALOG: dict[str, dict] = {
"generate_video_clip": {
"name": "Generate Video Clip",
"description": "Generate a short video clip from a text prompt using Wan 2.1",
"fn": generate_video_clip,
},
"image_to_video": {
"name": "Image to Video",
"description": "Animate a still image into a video clip",
"fn": image_to_video,
},
"list_video_styles": {
"name": "List Video Styles",
"description": "List supported video style presets and resolutions",
"fn": list_video_styles,
},
}

74
src/creative/tools/web_search.py
View File

@@ -1,74 +0,0 @@
"""Web search tool using DuckDuckGo.
MCP-compliant tool for searching the web.
"""
import logging
from typing import Any
from mcp.registry import register_tool
from mcp.schemas.base import create_tool_schema, PARAM_STRING, PARAM_INTEGER, RETURN_STRING
logger = logging.getLogger(__name__)
WEB_SEARCH_SCHEMA = create_tool_schema(
name="web_search",
description="Search the web using DuckDuckGo. Use for current events, news, real-time data, and information not in your training data.",
parameters={
"query": {
**PARAM_STRING,
"description": "Search query string",
},
"max_results": {
**PARAM_INTEGER,
"description": "Maximum number of results (1-10)",
"default": 5,
"minimum": 1,
"maximum": 10,
},
},
required=["query"],
returns=RETURN_STRING,
)
def web_search(query: str, max_results: int = 5) -> str:
"""Search the web using DuckDuckGo.
Args:
query: Search query
max_results: Maximum results to return
Returns:
Formatted search results
"""
try:
from duckduckgo_search import DDGS
with DDGS() as ddgs:
results = list(ddgs.text(query, max_results=max_results))
if not results:
return "No results found."
formatted = []
for i, r in enumerate(results, 1):
title = r.get("title", "No title")
body = r.get("body", "No description")
href = r.get("href", "")
formatted.append(f"{i}. {title}\n {body[:150]}...\n {href}")
return "\n\n".join(formatted)
except Exception as exc:
logger.error("Web search failed: %s", exc)
return f"Search error: {exc}"
# Register with MCP
register_tool(
name="web_search",
schema=WEB_SEARCH_SCHEMA,
category="research",
)(web_search)

438
src/dashboard/app.py
View File

@@ -2,10 +2,8 @@
Key improvements:
1. Background tasks use asyncio.create_task() to avoid blocking startup
2. Persona spawning is moved to a background task
3. MCP bootstrap is non-blocking
4. Chat integrations start in background
5. All startup operations complete quickly
2. Chat integrations start in background
3. All startup operations complete quickly
"""
import asyncio
@@ -21,36 +19,25 @@ from fastapi.responses import HTMLResponse
from fastapi.staticfiles import StaticFiles
from fastapi.templating import Jinja2Templates
from src.config import settings
from src.dashboard.routes.agents import router as agents_router
from src.dashboard.routes.health import router as health_router
from src.dashboard.routes.swarm import router as swarm_router
from src.dashboard.routes.marketplace import router as marketplace_router
from src.dashboard.routes.voice import router as voice_router
from src.dashboard.routes.mobile import router as mobile_router
from src.dashboard.routes.briefing import router as briefing_router
from src.dashboard.routes.telegram import router as telegram_router
from src.dashboard.routes.tools import router as tools_router
from src.dashboard.routes.spark import router as spark_router
from src.dashboard.routes.creative import router as creative_router
from src.dashboard.routes.discord import router as discord_router
from src.dashboard.routes.events import router as events_router
from src.dashboard.routes.ledger import router as ledger_router
from src.dashboard.routes.memory import router as memory_router
from src.dashboard.routes.router import router as router_status_router
from src.dashboard.routes.upgrades import router as upgrades_router
from src.dashboard.routes.tasks import router as tasks_router
from src.dashboard.routes.scripture import router as scripture_router
from src.dashboard.routes.self_coding import router as self_coding_router
from src.dashboard.routes.self_coding import self_modify_router
from src.dashboard.routes.hands import router as hands_router
from src.dashboard.routes.grok import router as grok_router
from src.dashboard.routes.models import router as models_router
from src.dashboard.routes.models import api_router as models_api_router
from src.dashboard.routes.chat_api import router as chat_api_router
from src.dashboard.routes.thinking import router as thinking_router
from src.dashboard.routes.bugs import router as bugs_router
from src.dashboard.routes.calm import router as calm_router
from config import settings
from dashboard.routes.agents import router as agents_router
from dashboard.routes.health import router as health_router
from dashboard.routes.marketplace import router as marketplace_router
from dashboard.routes.voice import router as voice_router
from dashboard.routes.mobile import router as mobile_router
from dashboard.routes.briefing import router as briefing_router
from dashboard.routes.telegram import router as telegram_router
from dashboard.routes.tools import router as tools_router
from dashboard.routes.spark import router as spark_router
from dashboard.routes.discord import router as discord_router
from dashboard.routes.memory import router as memory_router
from dashboard.routes.router import router as router_status_router
from dashboard.routes.grok import router as grok_router
from dashboard.routes.models import router as models_router
from dashboard.routes.models import api_router as models_api_router
from dashboard.routes.chat_api import router as chat_api_router
from dashboard.routes.thinking import router as thinking_router
from dashboard.routes.calm import router as calm_router
from infrastructure.router.api import router as cascade_router
@@ -119,315 +106,10 @@ async def _briefing_scheduler() -> None:
logger.info("Briefing is fresh; skipping generation.")
except Exception as exc:
logger.error("Briefing scheduler error: %s", exc)
try:
from infrastructure.error_capture import capture_error
capture_error(exc, source="briefing_scheduler")
except Exception:
pass
await asyncio.sleep(_BRIEFING_INTERVAL_HOURS * 3600)
async def _thinking_loop() -> None:
"""Background task: Timmy's default thinking thread."""
from swarm.task_queue.models import create_task
from datetime import datetime
await asyncio.sleep(10)
while True:
try:
now = datetime.now()
create_task(
title=f"Thought: {now.strftime('%A %B %d, %I:%M %p')}",
description="Continue thinking about your existence, recent events, scripture, creative ideas, or a previous thread of thought.",
assigned_to="timmy",
created_by="timmy",
priority="low",
requires_approval=False,
auto_approve=True,
task_type="thought",
)
logger.debug("Created thought task in queue")
except Exception as exc:
logger.error("Thinking loop error: %s", exc)
try:
from infrastructure.error_capture import capture_error
capture_error(exc, source="thinking_loop")
except Exception:
pass
await asyncio.sleep(settings.thinking_interval_seconds)
def handle_bug_report(task):
"""Process a bug report: log the decision and dispatch a fix task to Forge.
Timmy receives the bug report, decides it needs fixing, and creates
a code_fix task assigned to Forge. Every decision is logged to the
event log so there is a full audit trail of what Timmy decided and why.
"""
from swarm.event_log import EventType, log_event
from swarm.task_queue.models import create_task
decision = {
"action": "dispatch_to_forge",
"reason": f"Bug report received, dispatching fix to Forge: {task.title}",
"priority": task.priority.value,
"source_task_id": task.id,
}
# Dispatch a fix task to Forge
try:
fix_task = create_task(
title=f"[Fix] {task.title}",
description=(
f"## Bug Report\n\n{task.description or task.title}\n\n"
f"## Task\n\nImplement a fix for this bug and write a test proving the fix."
),
assigned_to="forge",
created_by="timmy",
priority=task.priority.value,
task_type="code_fix",
requires_approval=False,
auto_approve=True,
parent_task_id=task.id,
)
decision["outcome"] = "fix_dispatched"
decision["fix_task_id"] = fix_task.id
except Exception as e:
decision["outcome"] = "dispatch_failed"
decision["error"] = str(e)
# Log the decision trail to the event log
try:
log_event(
EventType.BUG_REPORT_CREATED,
source="bug_report_handler",
task_id=task.id,
agent_id="timmy",
data=decision,
)
except Exception:
pass
# Return structured result (stored in task.result)
if decision.get("fix_task_id"):
return (
f"Fix dispatched to Forge (task {decision['fix_task_id']}) | "
f"Decision: {decision['reason']}"
)
return (
f"Bug tracked internally (dispatch failed) | "
f"Decision: {decision['reason']} | Error: {decision.get('error', 'unknown')}"
)
async def _task_processor_loop() -> None:
"""Background task: Timmy's task queue processor."""
from swarm.task_processor import task_processor
from swarm.task_queue.models import update_task_status, list_tasks, TaskStatus
from timmy.session import chat as timmy_chat
from datetime import datetime
import json
await asyncio.sleep(5)
def handle_chat_response(task):
try:
now = datetime.now()
context = f"[System: Current date/time is {now.strftime('%A, %B %d, %Y at %I:%M %p')}]\n\n"
response = timmy_chat(context + task.description)
# Log the real agent response to chat history
try:
from dashboard.store import message_log
timestamp = now.strftime("%H:%M:%S")
message_log.append(role="agent", content=response, timestamp=timestamp, source="system")
except Exception as e:
logger.debug("Failed to log response to message_log: %s", e)
# Push response to chat UI via WebSocket
try:
from infrastructure.ws_manager.handler import ws_manager
asyncio.create_task(
ws_manager.broadcast(
"timmy_response",
{
"task_id": task.id,
"response": response,
},
)
)
except Exception as e:
logger.debug("Failed to push response via WS: %s", e)
return response
except Exception as e:
logger.error("Chat response failed: %s", e)
try:
from infrastructure.error_capture import capture_error
capture_error(e, source="chat_response_handler")
except Exception:
pass
return f"Error: {str(e)}"
def handle_thought(task):
from timmy.thinking import thinking_engine
try:
loop = asyncio.get_event_loop()
future = asyncio.run_coroutine_threadsafe(
thinking_engine.think_once(), loop
)
result = future.result(timeout=120)
return str(result) if result else "Thought completed"
except Exception as e:
logger.error("Thought processing failed: %s", e)
try:
from infrastructure.error_capture import capture_error
capture_error(e, source="thought_handler")
except Exception:
pass
return f"Error: {str(e)}"
def handle_task_request(task):
try:
now = datetime.now()
context = (
f"[System: Current date/time is {now.strftime('%A, %B %d, %Y at %I:%M %p')}]\n"
f"[System: You have been assigned a task from the queue. "
f"Complete it and provide your response.]\n\n"
f"Task: {task.title}\n"
)
if task.description and task.description != task.title:
context += f"Details: {task.description}\n"
response = timmy_chat(context)
try:
from infrastructure.ws_manager.handler import ws_manager
asyncio.create_task(
ws_manager.broadcast(
"task_response",
{
"task_id": task.id,
"response": response,
},
)
)
except Exception as e:
logger.debug("Failed to push task response via WS: %s", e)
return response
except Exception as e:
logger.error("Task request processing failed: %s", e)
try:
from infrastructure.error_capture import capture_error
capture_error(e, source="task_request_handler")
except Exception:
pass
return f"Error: {str(e)}"
# Register handlers for all known task types
task_processor.register_handler("chat_response", handle_chat_response)
task_processor.register_handler("thought", handle_thought)
task_processor.register_handler("internal", handle_thought)
task_processor.register_handler("bug_report", handle_bug_report)
task_processor.register_handler("task_request", handle_task_request)
task_processor.register_handler("escalation", handle_task_request)
task_processor.register_handler("external", handle_task_request)
# ── Reconcile zombie tasks from previous crash ──
zombie_count = task_processor.reconcile_zombie_tasks()
if zombie_count:
logger.info("Recycled %d zombie task(s) back to approved", zombie_count)
# ── Re-approve tasks backlogged due to missing handlers ──
stale = list_tasks(status=TaskStatus.BACKLOGGED, assigned_to="timmy")
requeued = 0
for t in stale:
if t.backlog_reason and "No handler for task type" in t.backlog_reason:
update_task_status(t.id, TaskStatus.APPROVED, result=None)
requeued += 1
if requeued:
logger.info("Re-queued %d task(s) that were backlogged due to missing handlers", requeued)
# ── Startup drain: iterate through all pending tasks immediately ──
logger.info("Draining task queue on startup...")
try:
summary = await task_processor.drain_queue()
if summary["processed"] or summary["backlogged"]:
logger.info(
"Startup drain: %d processed, %d backlogged, %d skipped, %d failed",
summary["processed"],
summary["backlogged"],
summary["skipped"],
summary["failed"],
)
# Notify via WebSocket so the dashboard updates
try:
from infrastructure.ws_manager.handler import ws_manager
asyncio.create_task(
ws_manager.broadcast_json(
{
"type": "task_event",
"event": "startup_drain_complete",
"summary": summary,
}
)
)
except Exception:
pass
except Exception as exc:
logger.error("Startup drain failed: %s", exc)
try:
from infrastructure.error_capture import capture_error
capture_error(exc, source="task_processor_startup")
except Exception:
pass
# ── Steady-state: poll for new tasks ──
logger.info("Task processor entering steady-state loop")
await task_processor.run_loop(interval_seconds=3.0)
async def _spawn_persona_agents_background() -> None:
"""Background task: register persona agents in the registry.
Coordinator/persona spawning has been deprecated. Agents are now
registered directly in the registry. Orchestration will be handled
by established tools (OpenClaw, Agno, etc.).
"""
from swarm import registry
await asyncio.sleep(1) # Let server fully start
if os.environ.get("TIMMY_TEST_MODE") != "1":
logger.info("Registering persona agents: Echo, Forge, Seer...")
try:
for name, aid in [("Echo", "persona-echo"), ("Forge", "persona-forge"), ("Seer", "persona-seer")]:
registry.register(name=name, agent_id=aid, capabilities="persona")
logger.info("Persona agents registered successfully")
except Exception as exc:
logger.error("Failed to register persona agents: %s", exc)
async def _bootstrap_mcp_background() -> None:
"""Background task: bootstrap MCP tools without blocking startup."""
from mcp.bootstrap import auto_bootstrap
await asyncio.sleep(0.5) # Let server start
try:
registered = auto_bootstrap()
if registered:
logger.info("MCP auto-bootstrap: %d tools registered", len(registered))
except Exception as exc:
logger.warning("MCP auto-bootstrap failed: %s", exc)
async def _start_chat_integrations_background() -> None:
"""Background task: start chat integrations without blocking startup."""
from integrations.telegram_bot.bot import telegram_bot
@@ -509,61 +191,11 @@ async def lifespan(app: FastAPI):
# Create all background tasks without waiting for them
briefing_task = asyncio.create_task(_briefing_scheduler())
# Register Timmy as the primary agent
from swarm import registry as swarm_registry
swarm_registry.register(
name="Timmy",
capabilities="chat,reasoning,research,planning",
agent_id="timmy",
)
# Spawn persona agents in background
persona_task = asyncio.create_task(_spawn_persona_agents_background())
# Log system startup event
try:
from swarm.event_log import log_event, EventType
log_event(
EventType.SYSTEM_INFO,
source="system",
data={"message": "Timmy Time system started"},
)
except Exception:
pass
# Bootstrap MCP tools in background
mcp_task = asyncio.create_task(_bootstrap_mcp_background())
# Register OpenFang vendor tools (if enabled)
if settings.openfang_enabled:
try:
from infrastructure.openfang.tools import register_openfang_tools
count = register_openfang_tools()
logger.info("OpenFang: registered %d vendor tools", count)
except Exception as exc:
logger.warning("OpenFang tool registration failed: %s", exc)
# Initialize Spark Intelligence engine
from spark.engine import spark_engine
if spark_engine.enabled:
logger.info("Spark Intelligence active — event capture enabled")
# Start thinking thread if enabled
thinking_task = None
if settings.thinking_enabled and os.environ.get("TIMMY_TEST_MODE") != "1":
thinking_task = asyncio.create_task(_thinking_loop())
logger.info(
"Default thinking thread started (interval: %ds)",
settings.thinking_interval_seconds,
)
# Start task processor if not in test mode
task_processor_task = None
if os.environ.get("TIMMY_TEST_MODE") != "1":
task_processor_task = asyncio.create_task(_task_processor_loop())
logger.info("Task queue processor started")
# Start chat integrations in background
chat_task = asyncio.create_task(_start_chat_integrations_background())
@@ -578,7 +210,7 @@ async def lifespan(app: FastAPI):
await discord_bot.stop()
await telegram_bot.stop()
for task in [thinking_task, task_processor_task, briefing_task, persona_task, mcp_task, chat_task]:
for task in [briefing_task, chat_task]:
if task:
task.cancel()
try:
@@ -657,7 +289,6 @@ templates = Jinja2Templates(directory=str(BASE_DIR / "templates"))
# Include routers
app.include_router(health_router)
app.include_router(agents_router)
app.include_router(swarm_router)
app.include_router(marketplace_router)
app.include_router(voice_router)
app.include_router(mobile_router)
@@ -665,46 +296,25 @@ app.include_router(briefing_router)
app.include_router(telegram_router)
app.include_router(tools_router)
app.include_router(spark_router)
app.include_router(creative_router)
app.include_router(discord_router)
app.include_router(events_router)
app.include_router(ledger_router)
app.include_router(memory_router)
app.include_router(router_status_router)
app.include_router(upgrades_router)
app.include_router(tasks_router)
app.include_router(scripture_router)
app.include_router(self_coding_router)
app.include_router(self_modify_router)
app.include_router(hands_router)
app.include_router(grok_router)
app.include_router(models_router)
app.include_router(models_api_router)
app.include_router(chat_api_router)
app.include_router(thinking_router)
app.include_router(bugs_router)
app.include_router(calm_router)
app.include_router(cascade_router)
@app.websocket("/ws")
async def ws_redirect(websocket: WebSocket):
"""Catch stale /ws connections and close cleanly.
Before PR #82, frontend code connected to /ws which never existed as
an endpoint. Stale browser tabs retry forever, spamming 403 errors.
Accept the connection and immediately close with a policy-violation
code so the client stops retrying.
websockets 16.0 dropped the legacy ``transfer_data_task`` attribute,
so calling ``websocket.close()`` after accept triggers an
AttributeError. Use the raw ASGI send instead.
"""
"""Catch stale /ws connections and close cleanly."""
await websocket.accept()
try:
await websocket.close(code=1008, reason="Use /swarm/live instead")
await websocket.close(code=1008, reason="Deprecated endpoint")
except AttributeError:
# websockets >= 16.0 — close via raw ASGI message
await websocket.send({"type": "websocket.close", "code": 1008})

284
src/dashboard/routes/agents.py
View File

@@ -1,5 +1,4 @@
import logging
import re
from datetime import datetime
from pathlib import Path
@@ -7,7 +6,7 @@ from fastapi import APIRouter, Form, Request
from fastapi.responses import HTMLResponse
from fastapi.templating import Jinja2Templates
from src.timmy.session import chat as timmy_chat
from timmy.session import chat as timmy_chat
from dashboard.store import message_log
logger = logging.getLogger(__name__)
@@ -15,190 +14,24 @@ logger = logging.getLogger(__name__)
router = APIRouter(prefix="/agents", tags=["agents"])
templates = Jinja2Templates(directory=str(Path(__file__).parent.parent / "templates"))
# ── Task queue detection ──────────────────────────────────────────────────
# Patterns that indicate the user wants to queue a task rather than chat
_QUEUE_PATTERNS = [
re.compile(
r"\b(?:add|put|schedule|queue|submit)\b.*\b(?:to the|on the|in the)?\s*(?:queue|task(?:\s*queue)?|task list)\b",
re.IGNORECASE,
),
re.compile(r"\bschedule\s+(?:this|that|a)\b", re.IGNORECASE),
re.compile(r"\bcreate\s+(?:a\s+|an\s+)?(?:\w+\s+){0,3}task\b", re.IGNORECASE),
]
# Questions about tasks/queue should NOT trigger task creation
_QUESTION_PREFIXES = re.compile(
r"^(?:what|how|why|can you explain|could you explain|tell me about|explain|"
r"what(?:'s| is| are| would))\b",
re.IGNORECASE,
)
_QUESTION_FRAMES = re.compile(
r"\b(?:how (?:do|does|would|can|should)|what (?:is|are|would)|"
r"can you (?:explain|describe|tell)|best way to)\b",
re.IGNORECASE,
)
# Known agent names for task assignment parsing
_KNOWN_AGENTS = frozenset(
{
"timmy",
"echo",
"mace",
"helm",
"seer",
"forge",
"quill",
"pixel",
"lyra",
"reel",
}
)
_AGENT_PATTERN = re.compile(
r"\bfor\s+(" + "|".join(_KNOWN_AGENTS) + r")\b", re.IGNORECASE
)
# Priority keywords → task priority mapping
_PRIORITY_MAP = {
"urgent": "urgent",
"critical": "urgent",
"asap": "urgent",
"emergency": "urgent",
"high priority": "high",
"high-priority": "high",
"important": "high",
"low priority": "low",
"low-priority": "low",
"minor": "low",
}
# Queue context detection
_QUEUE_QUERY_PATTERN = re.compile(
r"\b(?:task(?:s|\s+queue)?|queue|what(?:'s| is) (?:in |on )?(?:the )?queue)\b",
re.IGNORECASE,
)
def _extract_agent_from_message(message: str) -> str:
"""Extract target agent name from message, defaulting to 'timmy'."""
m = _AGENT_PATTERN.search(message)
if m:
return m.group(1).lower()
return "timmy"
def _extract_priority_from_message(message: str) -> str:
"""Extract priority level from message, defaulting to 'normal'."""
msg_lower = message.lower()
for keyword, priority in sorted(_PRIORITY_MAP.items(), key=lambda x: -len(x[0])):
if keyword in msg_lower:
return priority
return "normal"
def _extract_task_from_message(message: str) -> dict | None:
"""If the message looks like a task-queue request, return task details.
Returns None for meta-questions about tasks (e.g. "how do I create a task?").
"""
if _QUESTION_PREFIXES.search(message) or _QUESTION_FRAMES.search(message):
return None
for pattern in _QUEUE_PATTERNS:
if pattern.search(message):
# Strip the queue instruction to get the actual task description
title = re.sub(
r"\b(?:add|put|schedule|queue|submit|create)\b.*?\b(?:to the|on the|in the|an?)?(?:\s+\w+){0,3}\s*(?:queue|task(?:\s*queue)?|task list)\b",
"",
message,
flags=re.IGNORECASE,
).strip(" ,:;-")
# Strip "for {agent}" from title
title = _AGENT_PATTERN.sub("", title).strip(" ,:;-")
# Strip priority keywords from title
title = re.sub(
r"\b(?:urgent|critical|asap|emergency|high[- ]priority|important|low[- ]priority|minor)\b",
"",
title,
flags=re.IGNORECASE,
).strip(" ,:;-")
# Strip leading "to " that often remains
title = re.sub(r"^to\s+", "", title, flags=re.IGNORECASE).strip()
# Clean up double spaces
title = re.sub(r"\s{2,}", " ", title).strip()
# Fallback to full message if stripping removed everything
if not title or len(title) < 5:
title = message
# Capitalize first letter
title = title[0].upper() + title[1:] if title else title
agent = _extract_agent_from_message(message)
priority = _extract_priority_from_message(message)
return {
"title": title[:120],
"description": message,
"agent": agent,
"priority": priority,
}
return None
def _build_queue_context() -> str:
"""Build a concise task queue summary for context injection."""
try:
from swarm.task_queue.models import get_counts_by_status, list_tasks, TaskStatus
counts = get_counts_by_status()
queued = counts.get("approved", 0)
escalated = counts.get("pending_approval", 0)
running = counts.get("running", 0)
completed = counts.get("completed", 0)
parts = [
f"[System: Task queue — {queued} queued, {running} running, {completed} completed."
]
if escalated > 0:
parts.append(f"{escalated} escalation(s) awaiting human review.")
if running > 0:
tasks = list_tasks(status=TaskStatus.RUNNING, limit=5)
if tasks:
items = ", ".join(f'"{t.title}" ({t.assigned_to})' for t in tasks)
parts.append(f"Running: {items}.")
return " ".join(parts) + "]"
except Exception as exc:
logger.debug("Failed to build queue context: %s", exc)
return ""
# Static metadata for known agents — enriched onto live registry entries.
_AGENT_METADATA: dict[str, dict] = {
"timmy": {
"type": "sovereign",
"model": "", # Injected dynamically from settings
"backend": "ollama",
"version": "1.0.0",
},
}
@router.get("")
async def list_agents():
"""Return all registered agents with live status from the swarm registry."""
from swarm import registry as swarm_registry
"""Return registered agents."""
from config import settings
# Inject model name from settings into timmy metadata
metadata = dict(_AGENT_METADATA)
if "timmy" in metadata and not metadata["timmy"].get("model"):
metadata["timmy"]["model"] = settings.ollama_model
agents = swarm_registry.list_agents()
return {
"agents": [
{
"id": a.id,
"name": a.name,
"status": a.status,
"capabilities": a.capabilities,
**metadata.get(a.id, {}),
"id": "timmy",
"name": "Timmy",
"status": "idle",
"capabilities": "chat,reasoning,research,planning",
"type": "sovereign",
"model": settings.ollama_model,
"backend": "ollama",
"version": "1.0.0",
}
for a in agents
]
}
@@ -206,11 +39,8 @@ async def list_agents():
@router.get("/timmy/panel", response_class=HTMLResponse)
async def timmy_panel(request: Request):
"""Timmy chat panel — for HTMX main-panel swaps."""
from swarm import registry as swarm_registry
agent = swarm_registry.get_agent("timmy")
return templates.TemplateResponse(
request, "partials/timmy_panel.html", {"agent": agent}
request, "partials/timmy_panel.html", {"agent": None}
)
@@ -235,98 +65,22 @@ async def clear_history(request: Request):
@router.post("/timmy/chat", response_class=HTMLResponse)
async def chat_timmy(request: Request, message: str = Form(...)):
"""Chat with Timmy - queues message as task for async processing."""
from swarm.task_queue.models import create_task, get_queue_status_for_task
"""Chat with Timmy — synchronous response."""
timestamp = datetime.now().strftime("%H:%M:%S")
task_id = None
response_text = None
error_text = None
queue_info = None
# Check if the user wants to queue a task (explicit queue request)
task_info = _extract_task_from_message(message)
if task_info:
try:
task = create_task(
title=task_info["title"],
description=task_info["description"],
created_by="user",
assigned_to=task_info.get("agent", "timmy"),
priority=task_info.get("priority", "normal"),
task_type="task_request",
)
task_id = task.id
priority_label = (
f" | Priority: `{task.priority.value}`"
if task.priority.value != "normal"
else ""
)
response_text = (
f"Task queued: **{task.title}**\n\n"
f"Assigned to: `{task.assigned_to}`{priority_label} | "
f"Status: `{task.status.value}` | "
f"[View Task Queue](/tasks)"
)
logger.info(
"Chat → task queue: %s → %s (id=%s)",
task.title,
task.assigned_to,
task.id,
)
response_text = timmy_chat(message)
except Exception as exc:
logger.error("Failed to create task from chat: %s", exc)
task_info = None
logger.error("Chat error: %s", exc)
error_text = f"Chat error: {exc}"
# Normal chat: always queue for async processing
if not task_info:
try:
# Create a chat response task (auto-approved for timmy)
# Priority is "high" to jump ahead of Timmy's self-generated "thought" tasks
# but below any "urgent" tasks Timmy might create
task = create_task(
title=message[:100] + ("..." if len(message) > 100 else ""),
description=message,
created_by="user",
assigned_to="timmy",
priority="high", # Higher than thought tasks, lower than urgent
task_type="chat_response",
)
task_id = task.id
queue_info = get_queue_status_for_task(task.id)
# Acknowledge queuing
position = queue_info.get("position", 1)
total = queue_info.get("total", 1)
percent_ahead = queue_info.get("percent_ahead", 0)
response_text = (
f"Message queued for Timmy's attention.\n\n"
f"**Queue position:** {position}/{total} ({100 - percent_ahead}% complete ahead of you)\n\n"
f"_Timmy will respond shortly..._"
)
logger.info(
"Chat → queued: %s (id=%s, position=%d/%d)",
message[:50],
task.id,
position,
total,
)
except Exception as exc:
logger.error("Failed to queue chat message: %s", exc)
error_text = f"Failed to queue message: {exc}"
# Log user message to history. For chat_response tasks the real agent
# reply is logged by the task processor when it completes, so we only
# log the queue acknowledgment for explicit task_request commands.
message_log.append(role="user", content=message, timestamp=timestamp, source="browser")
if task_info and response_text is not None:
# Explicit task queue command — the acknowledgment IS the response
if response_text is not None:
message_log.append(role="agent", content=response_text, timestamp=timestamp, source="browser")
elif error_text:
message_log.append(
role="error", content=error_text, timestamp=timestamp, source="browser"
)
message_log.append(role="error", content=error_text, timestamp=timestamp, source="browser")
return templates.TemplateResponse(
request,
@@ -336,7 +90,7 @@ async def chat_timmy(request: Request, message: str = Form(...)):
"response": response_text,
"error": error_text,
"timestamp": timestamp,
"task_id": task_id,
"queue_info": queue_info,
"task_id": None,
"queue_info": None,
},
)

161
src/dashboard/routes/bugs.py
View File

@@ -1,161 +0,0 @@
"""Bug Report routes -- error feedback loop dashboard.
GET /bugs -- Bug reports dashboard page
GET /api/bugs -- List bug reports (JSON)
GET /api/bugs/stats -- Bug report statistics
POST /api/bugs/submit -- Submit structured bug reports (from AI test runs)
"""
import logging
from pathlib import Path
from typing import Optional
from fastapi import APIRouter, Request
from fastapi.responses import HTMLResponse, JSONResponse
from fastapi.templating import Jinja2Templates
from swarm.task_queue.models import create_task, list_tasks
logger = logging.getLogger(__name__)
router = APIRouter(tags=["bugs"])
templates = Jinja2Templates(directory=str(Path(__file__).parent.parent / "templates"))
def _get_bug_reports(status: Optional[str] = None, limit: int = 50) -> list:
"""Get bug report tasks from the task queue."""
all_tasks = list_tasks(limit=limit)
bugs = [t for t in all_tasks if t.task_type == "bug_report"]
if status:
bugs = [t for t in bugs if t.status.value == status]
return bugs
@router.get("/bugs", response_class=HTMLResponse)
async def bugs_page(request: Request, status: Optional[str] = None):
"""Bug reports dashboard page."""
bugs = _get_bug_reports(status=status, limit=200)
# Count by status
all_bugs = _get_bug_reports(limit=500)
stats: dict[str, int] = {}
for bug in all_bugs:
s = bug.status.value
stats[s] = stats.get(s, 0) + 1
return templates.TemplateResponse(
request,
"bugs.html",
{
"page_title": "Bug Reports",
"bugs": bugs,
"stats": stats,
"total": len(all_bugs),
"filter_status": status,
},
)
@router.get("/api/bugs", response_class=JSONResponse)
async def api_list_bugs(status: Optional[str] = None, limit: int = 50):
"""List bug reports as JSON."""
bugs = _get_bug_reports(status=status, limit=limit)
return {
"bugs": [
{
"id": b.id,
"title": b.title,
"description": b.description,
"status": b.status.value,
"priority": b.priority.value,
"created_at": b.created_at,
"result": b.result,
}
for b in bugs
],
"count": len(bugs),
}
@router.get("/api/bugs/stats", response_class=JSONResponse)
async def api_bug_stats():
"""Bug report statistics."""
all_bugs = _get_bug_reports(limit=500)
stats: dict[str, int] = {}
for bug in all_bugs:
s = bug.status.value
stats[s] = stats.get(s, 0) + 1
return {"stats": stats, "total": len(all_bugs)}
# ── Bug Report Submission ────────────────────────────────────────────────────
# Severity → task priority mapping
_SEVERITY_MAP = {"P0": "urgent", "P1": "high", "P2": "normal"}
def _format_bug_description(bug: dict, reporter: str) -> str:
"""Format a bug dict into a markdown task description."""
parts = [
f"**Reporter:** {reporter}",
f"**Severity:** {bug['severity']}",
"",
"## Problem",
bug["description"],
]
if bug.get("evidence"):
parts += ["", "## Evidence", bug["evidence"]]
if bug.get("root_cause"):
parts += ["", "## Suspected Root Cause", bug["root_cause"]]
if bug.get("fix_options"):
parts += ["", "## Suggested Fixes"]
for i, fix in enumerate(bug["fix_options"], 1):
parts.append(f"{i}. {fix}")
return "\n".join(parts)
@router.post("/api/bugs/submit", response_class=JSONResponse)
async def submit_bugs(request: Request):
"""Submit structured bug reports from an AI test run.
Body: { "reporter": "comet", "bugs": [ { "title", "severity", "description", ... } ] }
"""
try:
body = await request.json()
except Exception:
return JSONResponse(status_code=400, content={"error": "Invalid JSON"})
reporter = body.get("reporter", "unknown")
bugs = body.get("bugs", [])
if not bugs:
return JSONResponse(status_code=400, content={"error": "No bugs provided"})
task_ids = []
for bug in bugs:
title = bug.get("title", "")
severity = bug.get("severity", "")
description = bug.get("description", "")
if not title or not severity or not description:
return JSONResponse(
status_code=400,
content={"error": f"Bug missing required fields (title, severity, description)"},
)
priority = _SEVERITY_MAP.get(severity, "normal")
task = create_task(
title=f"[{severity}] {title}",
description=_format_bug_description(bug, reporter),
task_type="bug_report",
assigned_to="timmy",
created_by=reporter,
priority=priority,
requires_approval=False,
auto_approve=True,
)
task_ids.append(task.id)
logger.info("Bug report submitted: %d bug(s) from %s", len(task_ids), reporter)
return {"created": len(task_ids), "task_ids": task_ids}

87
src/dashboard/routes/creative.py
View File

@@ -1,87 +0,0 @@
"""Creative Studio dashboard route — /creative endpoints.
Provides a dashboard page for the creative pipeline: image generation,
music generation, video generation, and the full director pipeline.
"""
from pathlib import Path
from typing import Optional
from fastapi import APIRouter, Request
from fastapi.responses import HTMLResponse
from fastapi.templating import Jinja2Templates
router = APIRouter(tags=["creative"])
templates = Jinja2Templates(directory=str(Path(__file__).parent.parent / "templates"))
@router.get("/creative/ui", response_class=HTMLResponse)
async def creative_studio(request: Request):
"""Render the Creative Studio page."""
# Collect existing outputs
image_dir = Path("data/images")
music_dir = Path("data/music")
video_dir = Path("data/video")
creative_dir = Path("data/creative")
images = sorted(image_dir.glob("*.png"), key=lambda p: p.stat().st_mtime, reverse=True)[:20] if image_dir.exists() else []
music_files = sorted(music_dir.glob("*.wav"), key=lambda p: p.stat().st_mtime, reverse=True)[:20] if music_dir.exists() else []
videos = sorted(video_dir.glob("*.mp4"), key=lambda p: p.stat().st_mtime, reverse=True)[:20] if video_dir.exists() else []
# Load projects
projects = []
if creative_dir.exists():
for proj_dir in sorted(creative_dir.iterdir(), reverse=True):
meta_path = proj_dir / "project.json"
if meta_path.exists():
import json
projects.append(json.loads(meta_path.read_text()))
return templates.TemplateResponse(
request,
"creative.html",
{
"page_title": "Creative Studio",
"images": [{"name": p.name, "path": str(p)} for p in images],
"music_files": [{"name": p.name, "path": str(p)} for p in music_files],
"videos": [{"name": p.name, "path": str(p)} for p in videos],
"projects": projects[:10],
"image_count": len(images),
"music_count": len(music_files),
"video_count": len(videos),
"project_count": len(projects),
},
)
@router.get("/creative/api/projects")
async def creative_projects_api():
"""Return creative projects as JSON."""
try:
from creative.director import list_projects
return {"projects": list_projects()}
except ImportError:
return {"projects": []}
@router.get("/creative/api/genres")
async def creative_genres_api():
"""Return supported music genres."""
try:
from creative.tools.music_tools import GENRES
return {"genres": GENRES}
except ImportError:
return {"genres": []}
@router.get("/creative/api/video-styles")
async def creative_video_styles_api():
"""Return supported video styles and resolutions."""
try:
from creative.tools.video_tools import VIDEO_STYLES, RESOLUTION_PRESETS
return {
"styles": VIDEO_STYLES,
"resolutions": list(RESOLUTION_PRESETS.keys()),
}
except ImportError:
return {"styles": [], "resolutions": []}

91
src/dashboard/routes/events.py
View File

@@ -1,91 +0,0 @@
"""Event Log routes for viewing system events."""
from pathlib import Path
from typing import Optional
from fastapi import APIRouter, Request
from fastapi.responses import HTMLResponse
from fastapi.templating import Jinja2Templates
from swarm.event_log import (
EventType,
list_events,
get_event_summary,
get_recent_events,
)
router = APIRouter(prefix="/swarm", tags=["events"])
templates = Jinja2Templates(directory=str(Path(__file__).parent.parent / "templates"))
@router.get("/events", response_class=HTMLResponse)
async def events_page(
request: Request,
event_type: Optional[str] = None,
task_id: Optional[str] = None,
agent_id: Optional[str] = None,
):
"""Event log viewer page."""
# Parse event type filter
evt_type = None
if event_type:
try:
evt_type = EventType(event_type)
except ValueError:
pass
# Get events
events = list_events(
event_type=evt_type,
task_id=task_id,
agent_id=agent_id,
limit=100,
)
# Get summary stats
summary = get_event_summary(minutes=60)
return templates.TemplateResponse(
request,
"events.html",
{
"page_title": "Event Log",
"events": events,
"summary": summary,
"filter_type": event_type,
"filter_task": task_id,
"filter_agent": agent_id,
"event_types": [e.value for e in EventType],
},
)
@router.get("/events/partial", response_class=HTMLResponse)
async def events_partial(
request: Request,
event_type: Optional[str] = None,
task_id: Optional[str] = None,
agent_id: Optional[str] = None,
):
"""Event log partial for HTMX updates."""
evt_type = None
if event_type:
try:
evt_type = EventType(event_type)
except ValueError:
pass
events = list_events(
event_type=evt_type,
task_id=task_id,
agent_id=agent_id,
limit=100,
)
return templates.TemplateResponse(
request,
"partials/events_table.html",
{
"events": events,
},
)

96
src/dashboard/routes/hands.py
View File

@@ -1,96 +0,0 @@
"""Hands Dashboard Routes — DEPRECATED.
Replaced by brain task queue. This module provides compatibility redirects.
"""
from fastapi import APIRouter, Request
from fastapi.responses import HTMLResponse, JSONResponse
from brain.client import BrainClient
router = APIRouter(prefix="/hands", tags=["hands"])
# Initialize brain client
_brain: BrainClient = None
def get_brain() -> BrainClient:
global _brain
if _brain is None:
_brain = BrainClient()
return _brain
@router.get("/api/hands")
async def api_list_hands():
"""Return pending tasks from brain queue (replaces Hands list)."""
brain = get_brain()
tasks = await brain.get_pending_tasks(limit=100)
# Convert tasks to hand-like format for UI compatibility
hands = []
for task in tasks:
hands.append({
"name": f"task-{task['id']}",
"description": task['content'][:100],
"enabled": True,
"status": "pending",
"schedule": None,
"last_run": None,
"next_run": task['created_at'],
"run_count": 0,
"task_type": task['type'],
"priority": task['priority'],
})
return hands
@router.get("/api/hands/{name}")
async def api_get_hand(name: str):
"""Get task details."""
# Extract task ID from name
if name.startswith("task-"):
try:
task_id = int(name.split("-")[1])
# Return basic info
return {
"name": name,
"description": "Task from distributed queue",
"enabled": True,
"status": "pending",
"schedule": None,
}
except:
pass
return JSONResponse(
status_code=404,
content={"error": "Hand not found - use brain task queue"}
)
@router.post("/api/hands/{name}/trigger")
async def api_trigger_hand(name: str):
"""Trigger is now just submitting to brain queue."""
return {"status": "deprecated", "message": "Use POST /tasks instead"}
@router.get("", response_class=HTMLResponse)
async def hands_page(request: Request):
"""Redirect to new tasks UI."""
from fastapi.templating import Jinja2Templates
from pathlib import Path
templates = Jinja2Templates(
directory=str(Path(__file__).parent.parent / "templates")
)
# Return simple message about migration
return templates.TemplateResponse(
"hands.html",
{
"request": request,
"hands": [],
"message": "Hands system migrated to Brain Task Queue",
}
)

45
src/dashboard/routes/health.py
View File

@@ -15,8 +15,6 @@ from fastapi.responses import HTMLResponse
from pydantic import BaseModel
from config import settings
from lightning import get_backend
from lightning.factory import get_backend_info
logger = logging.getLogger(__name__)
@@ -137,37 +135,11 @@ def _check_redis() -> DependencyStatus:
def _check_lightning() -> DependencyStatus:
"""Check Lightning payment backend status."""
try:
backend = get_backend()
health = backend.health_check()
backend_name = backend.name
is_healthy = health.get("ok", False)
if backend_name == "mock":
return DependencyStatus(
name="Lightning Payments",
status="degraded",
sovereignty_score=8,
details={
"backend": "mock",
"note": "Using mock backend - set LIGHTNING_BACKEND=lnd for real payments",
**health,
},
)
else:
return DependencyStatus(
name="Lightning Payments",
status="healthy" if is_healthy else "degraded",
sovereignty_score=10,
details={"backend": backend_name, **health},
)
except Exception as exc:
return DependencyStatus(
name="Lightning Payments",
status="unavailable",
sovereignty_score=8,
details={"error": str(exc)},
details={"note": "Lightning module removed — will be re-added in v2"},
)
@@ -175,9 +147,10 @@ def _check_sqlite() -> DependencyStatus:
"""Check SQLite database status."""
try:
import sqlite3
from swarm.registry import DB_PATH
from pathlib import Path
conn = sqlite3.connect(str(DB_PATH))
db_path = Path(settings.repo_root) / "data" / "timmy.db"
conn = sqlite3.connect(str(db_path))
conn.execute("SELECT 1")
conn.close()
@@ -185,7 +158,7 @@ def _check_sqlite() -> DependencyStatus:
name="SQLite Database",
status="healthy",
sovereignty_score=10,
details={"path": str(DB_PATH)},
details={"path": str(db_path)},
)
except Exception as exc:
return DependencyStatus(
@@ -238,13 +211,6 @@ async def health_check():
# Legacy format for test compatibility
ollama_ok = await check_ollama()
# Determine Timmy's status from swarm registry (heartbeat-backed),
# falling back to Ollama connectivity only if not registered.
try:
from swarm import registry as swarm_registry
timmy_rec = swarm_registry.get_agent("timmy")
timmy_status = timmy_rec.status if timmy_rec else ("idle" if ollama_ok else "offline")
except Exception:
timmy_status = "idle" if ollama_ok else "offline"
return {
@@ -317,7 +283,6 @@ async def sovereignty_check():
async def component_status():
"""Get status of all system components."""
return {
"lightning": get_backend_info(),
"config": {
"debug": settings.debug,
"model_backend": settings.timmy_model_backend,

102
src/dashboard/routes/ledger.py
View File

@@ -1,102 +0,0 @@
"""Lightning Ledger routes for viewing transactions and balance."""
from pathlib import Path
from typing import Optional
from fastapi import APIRouter, Request
from fastapi.responses import HTMLResponse
from fastapi.templating import Jinja2Templates
from lightning.ledger import (
TransactionType,
TransactionStatus,
list_transactions,
get_balance,
get_transaction_stats,
)
router = APIRouter(prefix="/lightning", tags=["ledger"])
templates = Jinja2Templates(directory=str(Path(__file__).parent.parent / "templates"))
@router.get("/ledger", response_class=HTMLResponse)
async def ledger_page(
request: Request,
tx_type: Optional[str] = None,
status: Optional[str] = None,
):
"""Lightning ledger page with balance and transactions."""
# Parse filters
filter_type = None
if tx_type:
try:
filter_type = TransactionType(tx_type)
except ValueError:
pass
filter_status = None
if status:
try:
filter_status = TransactionStatus(status)
except ValueError:
pass
# Get data
balance = get_balance()
transactions = list_transactions(
tx_type=filter_type,
status=filter_status,
limit=50,
)
stats = get_transaction_stats(days=7)
return templates.TemplateResponse(
request,
"ledger.html",
{
"page_title": "Lightning Ledger",
"balance": balance,
"transactions": transactions,
"stats": stats,
"filter_type": tx_type,
"filter_status": status,
"tx_types": [t.value for t in TransactionType],
"tx_statuses": [s.value for s in TransactionStatus],
},
)
@router.get("/ledger/partial", response_class=HTMLResponse)
async def ledger_partial(
request: Request,
tx_type: Optional[str] = None,
status: Optional[str] = None,
):
"""Ledger transactions partial for HTMX updates."""
filter_type = None
if tx_type:
try:
filter_type = TransactionType(tx_type)
except ValueError:
pass
filter_status = None
if status:
try:
filter_status = TransactionStatus(status)
except ValueError:
pass
transactions = list_transactions(
tx_type=filter_type,
status=filter_status,
limit=50,
)
return templates.TemplateResponse(
request,
"partials/ledger_table.html",
{
"transactions": transactions,
},
)

274
src/dashboard/routes/scripture.py
View File

@@ -1,274 +0,0 @@
"""Scripture dashboard routes.
GET /scripture — JSON status of the scripture module
GET /scripture/verse — Look up a single verse by reference
GET /scripture/search — Full-text search across verse content
GET /scripture/chapter — Retrieve an entire chapter
GET /scripture/meditate — Get the current meditation verse
POST /scripture/meditate — Advance meditation to the next verse
POST /scripture/meditate/mode — Change meditation mode
GET /scripture/memory — Scripture memory system status
GET /scripture/xref — Cross-references for a verse
GET /scripture/stats — Store statistics
POST /scripture/ingest — Bulk-ingest verses (JSON array)
"""
from __future__ import annotations
import json
import logging
from pathlib import Path
from typing import Optional
from fastapi import APIRouter, Query, Request
from fastapi.responses import JSONResponse
from scripture.constants import BOOK_BY_ID, book_by_name
from scripture.meditation import meditation_scheduler
from scripture.memory import scripture_memory
from scripture.models import Verse, encode_verse_id
from scripture.parser import extract_references, format_reference, parse_reference
from scripture.store import scripture_store
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/scripture", tags=["scripture"])
@router.get("")
async def scripture_status():
"""Return scripture module status — store stats + memory state."""
return JSONResponse({
"store": scripture_store.stats(),
"memory": scripture_memory.status(),
"meditation": meditation_scheduler.status(),
})
@router.get("/verse")
async def get_verse(
ref: str = Query(
...,
description="Biblical reference, e.g. 'John 3:16' or 'Gen 1:1-3'",
),
):
"""Look up one or more verses by reference string."""
parsed = parse_reference(ref)
if not parsed:
return JSONResponse(
{"error": f"Could not parse reference: {ref}"},
status_code=400,
)
start = parsed.start
end = parsed.end
if start.verse_id == end.verse_id:
verse = scripture_store.get_verse(start.book, start.chapter, start.verse)
if not verse:
return JSONResponse({"error": "Verse not found", "ref": ref}, status_code=404)
return JSONResponse(_verse_to_dict(verse))
verses = scripture_store.get_range(start.verse_id, end.verse_id)
if not verses:
return JSONResponse({"error": "No verses found in range", "ref": ref}, status_code=404)
return JSONResponse({"verses": [_verse_to_dict(v) for v in verses]})
@router.get("/chapter")
async def get_chapter(
book: str = Query(..., description="Book name or abbreviation"),
chapter: int = Query(..., ge=1, description="Chapter number"),
):
"""Retrieve all verses in a chapter."""
book_info = book_by_name(book)
if not book_info:
return JSONResponse({"error": f"Unknown book: {book}"}, status_code=400)
verses = scripture_store.get_chapter(book_info.id, chapter)
if not verses:
return JSONResponse(
{"error": f"No verses found for {book_info.name} {chapter}"},
status_code=404,
)
return JSONResponse({
"book": book_info.name,
"chapter": chapter,
"verses": [_verse_to_dict(v) for v in verses],
})
@router.get("/search")
async def search_verses(
q: str = Query(..., min_length=2, description="Search query"),
limit: int = Query(default=20, ge=1, le=100),
):
"""Full-text search across verse content."""
verses = scripture_store.search_text(q, limit=limit)
return JSONResponse({
"query": q,
"count": len(verses),
"verses": [_verse_to_dict(v) for v in verses],
})
@router.get("/meditate")
async def get_meditation():
"""Return the current meditation focus verse and status."""
status = meditation_scheduler.status()
current = meditation_scheduler.current_focus()
return JSONResponse({
"status": status,
"current_verse": _verse_to_dict(current) if current else None,
})
@router.post("/meditate")
async def advance_meditation():
"""Advance to the next verse in the meditation sequence."""
verse = meditation_scheduler.next_meditation()
if not verse:
return JSONResponse(
{"message": "No more verses available — scripture store may be empty"},
status_code=404,
)
return JSONResponse({
"verse": _verse_to_dict(verse),
"status": meditation_scheduler.status(),
})
@router.post("/meditate/mode")
async def set_meditation_mode(
mode: str = Query(..., description="sequential, thematic, or lectionary"),
theme: Optional[str] = Query(default=None, description="Theme for thematic mode"),
):
"""Change the meditation mode."""
try:
state = meditation_scheduler.set_mode(mode, theme=theme)
except ValueError as exc:
return JSONResponse({"error": str(exc)}, status_code=400)
return JSONResponse({
"mode": state.mode,
"theme": state.theme,
"message": f"Meditation mode set to {state.mode}",
})
@router.get("/memory")
async def memory_status():
"""Return the scripture memory system status."""
return JSONResponse(scripture_memory.status())
@router.get("/xref")
async def get_cross_references(
ref: str = Query(..., description="Verse reference, e.g. 'John 3:16'"),
):
"""Find cross-references for a verse."""
parsed = parse_reference(ref)
if not parsed:
return JSONResponse({"error": f"Could not parse: {ref}"}, status_code=400)
verse = scripture_store.get_verse(
parsed.start.book, parsed.start.chapter, parsed.start.verse
)
if not verse:
return JSONResponse({"error": "Verse not found"}, status_code=404)
xrefs = scripture_store.get_cross_references(verse.verse_id)
results = []
for xref in xrefs:
target_id = (
xref.target_verse_id
if xref.source_verse_id == verse.verse_id
else xref.source_verse_id
)
target = scripture_store.get_verse_by_id(target_id)
if target:
results.append({
"reference_type": xref.reference_type,
"confidence": xref.confidence,
"verse": _verse_to_dict(target),
})
return JSONResponse({
"source": _verse_to_dict(verse),
"cross_references": results,
})
@router.get("/stats")
async def store_stats():
"""Return scripture store statistics."""
return JSONResponse(scripture_store.stats())
@router.post("/ingest")
async def ingest_verses(request: Request):
"""Bulk-ingest verses from a JSON array.
Expects a JSON body with a "verses" key containing an array of objects
with: book, chapter, verse_num, text, and optionally
translation/testament/genre.
"""
try:
body = await request.json()
except Exception:
return JSONResponse({"error": "Invalid JSON body"}, status_code=400)
raw_verses = body.get("verses", [])
if not raw_verses:
return JSONResponse({"error": "No verses provided"}, status_code=400)
verses = []
for rv in raw_verses:
try:
book = int(rv["book"])
chapter = int(rv["chapter"])
verse_num = int(rv["verse_num"])
text = str(rv["text"])
book_info = BOOK_BY_ID.get(book)
verses.append(Verse(
verse_id=encode_verse_id(book, chapter, verse_num),
book=book,
chapter=chapter,
verse_num=verse_num,
text=text,
translation=rv.get("translation", "ESV"),
testament=book_info.testament if book_info else "OT",
genre=book_info.genre if book_info else "",
))
except (KeyError, ValueError, TypeError) as exc:
logger.warning("Skipping invalid verse record: %s", exc)
continue
if verses:
scripture_store.insert_verses(verses)
return JSONResponse({
"ingested": len(verses),
"skipped": len(raw_verses) - len(verses),
"total_verses": scripture_store.count_verses(),
})
# ── Helpers ──────────────────────────────────────────────────────────────────
def _verse_to_dict(verse: Verse) -> dict:
"""Convert a Verse model to a JSON-friendly dict with formatted reference."""
from scripture.models import VerseRef
ref = VerseRef(book=verse.book, chapter=verse.chapter, verse=verse.verse_num)
return {
"verse_id": verse.verse_id,
"reference": format_reference(ref),
"book": verse.book,
"chapter": verse.chapter,
"verse_num": verse.verse_num,
"text": verse.text,
"translation": verse.translation,
"testament": verse.testament,
"genre": verse.genre,
}

428
src/dashboard/routes/self_coding.py
View File

@@ -1,428 +0,0 @@
"""Self-Coding Dashboard Routes.
API endpoints and HTMX views for the self-coding system:
- Journal viewer with filtering
- Stats dashboard
- Manual task execution
- Real-time status updates
- Self-modification loop (/self-modify/*)
"""
from __future__ import annotations
import asyncio
import logging
from typing import Optional
from fastapi import APIRouter, Form, HTTPException, Request
from fastapi.responses import HTMLResponse, JSONResponse
from pydantic import BaseModel
from config import settings
from self_coding import (
CodebaseIndexer,
ModificationJournal,
Outcome,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/self-coding", tags=["self_coding"])
# ── API Models ────────────────────────────────────────────────────────────
class JournalEntryResponse(BaseModel):
"""A journal entry for API response."""
id: int
timestamp: str
task_description: str
approach: str
files_modified: list[str]
outcome: str
retry_count: int
has_reflection: bool
class StatsResponse(BaseModel):
"""Self-coding stats for API response."""
total_attempts: int
success_count: int
failure_count: int
rollback_count: int
success_rate: float
recent_failures: list[JournalEntryResponse]
class ExecuteRequest(BaseModel):
"""Request to execute a self-edit task."""
task_description: str
class ExecuteResponse(BaseModel):
"""Response from executing a self-edit task."""
success: bool
message: str
attempt_id: Optional[int] = None
files_modified: list[str] = []
commit_hash: Optional[str] = None
# ── Services (initialized lazily) ─────────────────────────────────────────
_journal: Optional[ModificationJournal] = None
_indexer: Optional[CodebaseIndexer] = None
def get_journal() -> ModificationJournal:
"""Get or create ModificationJournal singleton."""
global _journal
if _journal is None:
_journal = ModificationJournal()
return _journal
def get_indexer() -> CodebaseIndexer:
"""Get or create CodebaseIndexer singleton."""
global _indexer
if _indexer is None:
_indexer = CodebaseIndexer()
return _indexer
# ── API Endpoints ─────────────────────────────────────────────────────────
@router.get("/api/journal", response_model=list[JournalEntryResponse])
async def api_journal_list(
limit: int = 50,
outcome: Optional[str] = None,
):
"""Get modification journal entries.
Args:
limit: Maximum number of entries to return
outcome: Filter by outcome (success, failure, rollback)
"""
journal = get_journal()
# Build query based on filters
if outcome:
try:
outcome_enum = Outcome(outcome)
# Get recent and filter
from self_coding.modification_journal import ModificationAttempt
# Note: This is a simplified query - in production you'd add
# proper filtering to the journal class
entries = []
# Placeholder for filtered query
except ValueError:
return JSONResponse(
status_code=400,
content={"error": f"Invalid outcome: {outcome}"},
)
# For now, return recent failures mixed with successes
recent = await journal.get_recent_failures(limit=limit)
# Also get some successes
# Note: We'd need to add a method to journal for this
# For now, return what we have
response = []
for entry in recent:
response.append(JournalEntryResponse(
id=entry.id or 0,
timestamp=entry.timestamp.isoformat() if entry.timestamp else "",
task_description=entry.task_description,
approach=entry.approach,
files_modified=entry.files_modified,
outcome=entry.outcome.value,
retry_count=entry.retry_count,
has_reflection=bool(entry.reflection),
))
return response
@router.get("/api/journal/{attempt_id}", response_model=dict)
async def api_journal_detail(attempt_id: int):
"""Get detailed information about a specific attempt."""
journal = get_journal()
entry = await journal.get_by_id(attempt_id)
if not entry:
return JSONResponse(
status_code=404,
content={"error": "Attempt not found"},
)
return {
"id": entry.id,
"timestamp": entry.timestamp.isoformat() if entry.timestamp else "",
"task_description": entry.task_description,
"approach": entry.approach,
"files_modified": entry.files_modified,
"diff": entry.diff,
"test_results": entry.test_results,
"outcome": entry.outcome.value,
"failure_analysis": entry.failure_analysis,
"reflection": entry.reflection,
"retry_count": entry.retry_count,
}
@router.get("/api/stats", response_model=StatsResponse)
async def api_stats():
"""Get self-coding statistics."""
journal = get_journal()
metrics = await journal.get_success_rate()
recent_failures = await journal.get_recent_failures(limit=5)
return StatsResponse(
total_attempts=metrics["total"],
success_count=metrics["success"],
failure_count=metrics["failure"],
rollback_count=metrics["rollback"],
success_rate=metrics["overall"],
recent_failures=[
JournalEntryResponse(
id=f.id or 0,
timestamp=f.timestamp.isoformat() if f.timestamp else "",
task_description=f.task_description,
approach=f.approach,
files_modified=f.files_modified,
outcome=f.outcome.value,
retry_count=f.retry_count,
has_reflection=bool(f.reflection),
)
for f in recent_failures
],
)
@router.post("/api/execute", response_model=ExecuteResponse)
async def api_execute(request: ExecuteRequest):
"""Execute a self-edit task.
This is the API endpoint for manual task execution.
In production, this should require authentication and confirmation.
"""
from creative.tools.self_edit import SelfEditTool
tool = SelfEditTool()
result = await tool.execute(request.task_description)
return ExecuteResponse(
success=result.success,
message=result.message,
attempt_id=result.attempt_id,
files_modified=result.files_modified,
commit_hash=result.commit_hash,
)
@router.get("/api/codebase/summary")
async def api_codebase_summary():
"""Get codebase summary for LLM context."""
indexer = get_indexer()
await indexer.index_changed()
summary = await indexer.get_summary(max_tokens=3000)
return {
"summary": summary,
"generated_at": "",
}
@router.post("/api/codebase/reindex")
async def api_codebase_reindex():
"""Trigger a full codebase reindex."""
indexer = get_indexer()
stats = await indexer.index_all()
return {
"indexed": stats["indexed"],
"failed": stats["failed"],
"skipped": stats["skipped"],
}
# ── HTMX Page Routes ──────────────────────────────────────────────────────
@router.get("", response_class=HTMLResponse)
async def self_coding_page(request: Request):
"""Main self-coding dashboard page."""
from dashboard.app import templates
return templates.TemplateResponse(
"self_coding.html",
{
"request": request,
"title": "Self-Coding",
},
)
@router.get("/journal", response_class=HTMLResponse)
async def journal_partial(
request: Request,
outcome: Optional[str] = None,
limit: int = 20,
):
"""HTMX partial for journal entries."""
from dashboard.app import templates
journal = get_journal()
# Get entries (simplified - in production, add proper filtering)
if outcome == "failure":
entries = await journal.get_recent_failures(limit=limit)
else:
# Get all recent
entries = await journal.get_recent_failures(limit=limit)
# TODO: Add method to get successes too
return templates.TemplateResponse(
"partials/journal_entries.html",
{
"request": request,
"entries": entries,
"outcome_filter": outcome,
},
)
@router.get("/stats", response_class=HTMLResponse)
async def stats_partial(request: Request):
"""HTMX partial for stats cards."""
from dashboard.app import templates
journal = get_journal()
metrics = await journal.get_success_rate()
return templates.TemplateResponse(
"partials/self_coding_stats.html",
{
"request": request,
"metrics": metrics,
},
)
@router.get("/execute-form", response_class=HTMLResponse)
async def execute_form_partial(request: Request):
"""HTMX partial for execute task form."""
from dashboard.app import templates
return templates.TemplateResponse(
"partials/execute_form.html",
{
"request": request,
},
)
@router.post("/execute", response_class=HTMLResponse)
async def execute_task(
request: Request,
task_description: str = Form(...),
):
"""HTMX endpoint to execute a task."""
from dashboard.app import templates
from creative.tools.self_edit import SelfEditTool
tool = SelfEditTool()
result = await tool.execute(task_description)
return templates.TemplateResponse(
"partials/execute_result.html",
{
"request": request,
"result": result,
},
)
@router.get("/journal/{attempt_id}/detail", response_class=HTMLResponse)
async def journal_entry_detail(request: Request, attempt_id: int):
"""HTMX partial for journal entry detail."""
from dashboard.app import templates
journal = get_journal()
entry = await journal.get_by_id(attempt_id)
if not entry:
return templates.TemplateResponse(
"partials/error.html",
{
"request": request,
"message": "Attempt not found",
},
)
return templates.TemplateResponse(
"partials/journal_entry_detail.html",
{
"request": request,
"entry": entry,
},
)
# ── Self-Modification Routes (/self-modify/*) ───────────────────────────
self_modify_router = APIRouter(prefix="/self-modify", tags=["self-modify"])
@self_modify_router.post("/run")
async def run_self_modify(
instruction: str = Form(...),
target_files: str = Form(""),
dry_run: bool = Form(False),
speak_result: bool = Form(False),
):
"""Execute a self-modification loop."""
if not settings.self_modify_enabled:
raise HTTPException(403, "Self-modification is disabled")
from self_coding.self_modify.loop import SelfModifyLoop, ModifyRequest
files = [f.strip() for f in target_files.split(",") if f.strip()]
request = ModifyRequest(
instruction=instruction,
target_files=files,
dry_run=dry_run,
)
loop = SelfModifyLoop()
result = await asyncio.to_thread(loop.run, request)
if speak_result and result.success:
try:
from timmy_serve.voice_tts import voice_tts
if voice_tts.available:
voice_tts.speak(
f"Code modification complete. "
f"{len(result.files_changed)} files changed. Tests passing."
)
except Exception:
pass
return {
"success": result.success,
"files_changed": result.files_changed,
"test_passed": result.test_passed,
"commit_sha": result.commit_sha,
"branch_name": result.branch_name,
"error": result.error,
"attempts": result.attempts,
}
@self_modify_router.get("/status")
async def self_modify_status():
"""Return whether self-modification is enabled."""
return {"enabled": settings.self_modify_enabled}

164
src/dashboard/routes/swarm.py
View File

@@ -1,164 +0,0 @@
"""Swarm dashboard routes — /swarm/* endpoints.
Provides REST endpoints for viewing swarm agents, tasks, and the
live WebSocket feed. Coordinator/learner/auction plumbing has been
removed — established tools will replace the homebrew orchestration.
"""
import logging
from pathlib import Path
from typing import Optional
from fastapi import APIRouter, HTTPException, Request, WebSocket, WebSocketDisconnect
from fastapi.responses import HTMLResponse
from fastapi.templating import Jinja2Templates
from swarm import registry
from swarm.tasks import TaskStatus, list_tasks as _list_tasks, get_task as _get_task
from infrastructure.ws_manager.handler import ws_manager
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/swarm", tags=["swarm"])
templates = Jinja2Templates(directory=str(Path(__file__).parent.parent / "templates"))
@router.get("")
async def swarm_status():
"""Return the current swarm status summary."""
agents = registry.list_agents()
tasks = _list_tasks()
return {
"agents": len(agents),
"tasks": len(tasks),
"status": "operational",
}
@router.get("/live", response_class=HTMLResponse)
async def swarm_live_page(request: Request):
"""Render the live swarm dashboard page."""
return templates.TemplateResponse(
request, "swarm_live.html", {"page_title": "Swarm Live"}
)
@router.get("/mission-control", response_class=HTMLResponse)
async def mission_control_page(request: Request):
"""Render the Mission Control dashboard."""
return templates.TemplateResponse(
request, "mission_control.html", {"page_title": "Mission Control"}
)
@router.get("/agents")
async def list_swarm_agents():
"""List all registered swarm agents."""
agents = registry.list_agents()
return {
"agents": [
{
"id": a.id,
"name": a.name,
"status": a.status,
"capabilities": a.capabilities,
"last_seen": a.last_seen,
}
for a in agents
]
}
@router.get("/tasks")
async def list_tasks(status: Optional[str] = None):
"""List swarm tasks, optionally filtered by status."""
task_status = TaskStatus(status.lower()) if status else None
tasks = _list_tasks(status=task_status)
return {
"tasks": [
{
"id": t.id,
"description": t.description,
"status": t.status.value,
"assigned_agent": t.assigned_agent,
"result": t.result,
"created_at": t.created_at,
"completed_at": t.completed_at,
}
for t in tasks
]
}
@router.get("/tasks/{task_id}")
async def get_task(task_id: str):
"""Get details for a specific task."""
task = _get_task(task_id)
if task is None:
return {"error": "Task not found"}
return {
"id": task.id,
"description": task.description,
"status": task.status.value,
"assigned_agent": task.assigned_agent,
"result": task.result,
"created_at": task.created_at,
"completed_at": task.completed_at,
}
@router.get("/insights")
async def swarm_insights():
"""Placeholder — learner metrics removed. Will be replaced by brain memory stats."""
return {"agents": {}, "note": "Learner deprecated. Use brain.memory for insights."}
@router.get("/insights/{agent_id}")
async def agent_insights(agent_id: str):
"""Placeholder — learner metrics removed."""
return {"agent_id": agent_id, "note": "Learner deprecated. Use brain.memory for insights."}
# ── UI endpoints (return HTML partials for HTMX) ─────────────────────────────
@router.get("/agents/sidebar", response_class=HTMLResponse)
async def agents_sidebar(request: Request):
"""Sidebar partial: all registered agents."""
agents = registry.list_agents()
return templates.TemplateResponse(
request, "partials/swarm_agents_sidebar.html", {"agents": agents}
)
@router.get("/agents/{agent_id}/panel", response_class=HTMLResponse)
async def agent_panel(agent_id: str, request: Request):
"""Main-panel partial: agent detail."""
agent = registry.get_agent(agent_id)
if agent is None:
raise HTTPException(404, "Agent not found")
return templates.TemplateResponse(
request,
"partials/agent_panel.html",
{"agent": agent, "tasks": []},
)
# ── WebSocket live feed ──────────────────────────────────────────────────
@router.websocket("/live")
async def swarm_live(websocket: WebSocket):
"""WebSocket endpoint for live swarm event streaming."""
try:
await ws_manager.connect(websocket)
except Exception as exc:
logger.warning("WebSocket accept failed: %s", exc)
return
try:
while True:
data = await websocket.receive_text()
logger.debug("WS received: %s", data[:100])
except WebSocketDisconnect:
ws_manager.disconnect(websocket)
except Exception as exc:
logger.error("WebSocket error: %s", exc)
ws_manager.disconnect(websocket)

596
src/dashboard/routes/tasks.py
View File

@@ -1,596 +0,0 @@
"""Task Queue routes — Human-in-the-loop approval dashboard.
GET /tasks — Task queue dashboard page
GET /api/tasks — List tasks (JSON)
POST /api/tasks — Create a new task (JSON)
GET /api/tasks/counts — Badge counts
GET /api/tasks/{id} — Get single task
PATCH /api/tasks/{id}/approve — Approve a task
PATCH /api/tasks/{id}/veto — Veto a task
PATCH /api/tasks/{id}/modify — Modify a task
PATCH /api/tasks/{id}/pause — Pause a running task
PATCH /api/tasks/{id}/cancel — Cancel / fail a task
PATCH /api/tasks/{id}/retry — Retry a failed task
GET /tasks/pending — HTMX partial: pending tasks
GET /tasks/active — HTMX partial: active tasks
GET /tasks/completed — HTMX partial: completed tasks
"""
import logging
from pathlib import Path
from typing import Optional
from fastapi import APIRouter, Form, HTTPException, Request
from fastapi.responses import HTMLResponse, JSONResponse
from fastapi.templating import Jinja2Templates
from swarm.task_queue.models import (
QueueTask,
TaskPriority,
TaskStatus,
create_task,
get_counts_by_status,
get_pending_count,
get_task,
list_backlogged_tasks,
list_tasks,
update_task,
update_task_status,
)
logger = logging.getLogger(__name__)
router = APIRouter(tags=["tasks"])
templates = Jinja2Templates(directory=str(Path(__file__).parent.parent / "templates"))
# ── Helper to broadcast task events via WebSocket ────────────────────────
def _broadcast_task_event(event_type: str, task: QueueTask):
"""Best-effort broadcast a task event to connected WebSocket clients."""
try:
import asyncio
from infrastructure.ws_manager.handler import ws_manager
payload = {
"type": "task_event",
"event": event_type,
"task": {
"id": task.id,
"title": task.title,
"status": task.status.value,
"priority": task.priority.value,
"assigned_to": task.assigned_to,
"created_by": task.created_by,
},
}
try:
loop = asyncio.get_running_loop()
loop.create_task(ws_manager.broadcast_json(payload))
except RuntimeError:
pass # No event loop running (e.g. in tests)
except Exception:
pass # WebSocket is optional
# ── Dashboard page ───────────────────────────────────────────────────────
@router.get("/tasks", response_class=HTMLResponse)
async def task_queue_page(request: Request, assign: Optional[str] = None):
"""Task queue dashboard with three columns."""
pending = list_tasks(status=TaskStatus.PENDING_APPROVAL) + list_tasks(
status=TaskStatus.APPROVED
)
active = list_tasks(status=TaskStatus.RUNNING) + list_tasks(
status=TaskStatus.PAUSED
)
backlogged = list_backlogged_tasks(limit=20)
completed = (
list_tasks(status=TaskStatus.COMPLETED, limit=20)
+ list_tasks(status=TaskStatus.VETOED, limit=10)
+ list_tasks(status=TaskStatus.FAILED, limit=10)
+ backlogged
)
# Get agents for the create modal
agents = []
try:
from swarm import registry
agents = [{"id": a.id, "name": a.name} for a in registry.list_agents()]
except Exception:
pass
# Always include core agents
core_agents = ["timmy", "forge", "seer", "echo"]
agent_names = {a["name"] for a in agents}
for name in core_agents:
if name not in agent_names:
agents.append({"id": name, "name": name})
return templates.TemplateResponse(
request,
"tasks.html",
{
"page_title": "Task Queue",
"pending": pending,
"active": active,
"completed": completed,
"pending_count": len(pending),
"agents": agents,
"priorities": [p.value for p in TaskPriority],
"pre_assign": assign or "",
},
)
# ── HTMX partials ───────────────────────────────────────────────────────
@router.get("/tasks/pending", response_class=HTMLResponse)
async def tasks_pending_partial(request: Request):
"""HTMX partial: pending approval tasks."""
pending = list_tasks(status=TaskStatus.PENDING_APPROVAL) + list_tasks(
status=TaskStatus.APPROVED
)
return templates.TemplateResponse(
request,
"partials/task_cards.html",
{"tasks": pending, "section": "pending"},
)
@router.get("/tasks/active", response_class=HTMLResponse)
async def tasks_active_partial(request: Request):
"""HTMX partial: active tasks."""
active = list_tasks(status=TaskStatus.RUNNING) + list_tasks(
status=TaskStatus.PAUSED
)
return templates.TemplateResponse(
request,
"partials/task_cards.html",
{"tasks": active, "section": "active"},
)
@router.get("/tasks/completed", response_class=HTMLResponse)
async def tasks_completed_partial(request: Request):
"""HTMX partial: completed tasks."""
completed = (
list_tasks(status=TaskStatus.COMPLETED, limit=20)
+ list_tasks(status=TaskStatus.VETOED, limit=10)
+ list_tasks(status=TaskStatus.FAILED, limit=10)
+ list_backlogged_tasks(limit=20)
)
return templates.TemplateResponse(
request,
"partials/task_cards.html",
{"tasks": completed, "section": "completed"},
)
# ── JSON API ─────────────────────────────────────────────────────────────
@router.get("/api/tasks", response_class=JSONResponse)
async def api_list_tasks(
status: Optional[str] = None,
priority: Optional[str] = None,
assigned_to: Optional[str] = None,
limit: int = 100,
):
"""List tasks with optional filters."""
s = TaskStatus(status.lower()) if status else None
p = TaskPriority(priority.lower()) if priority else None
tasks = list_tasks(status=s, priority=p, assigned_to=assigned_to, limit=limit)
return {
"tasks": [_task_to_dict(t) for t in tasks],
"count": len(tasks),
}
@router.post("/api/tasks", response_class=JSONResponse)
async def api_create_task(request: Request):
"""Create a new task (JSON body)."""
body = await request.json()
task = create_task(
title=body.get("title", ""),
description=body.get("description", ""),
assigned_to=body.get("assigned_to", "timmy"),
created_by=body.get("created_by", "user"),
priority=body.get("priority", "normal"),
requires_approval=body.get("requires_approval", False),
auto_approve=body.get("auto_approve", True),
parent_task_id=body.get("parent_task_id"),
steps=body.get("steps"),
task_type=body.get("task_type", "chat_response"),
)
# Notify
_notify_task_created(task)
_broadcast_task_event("task_created", task)
logger.info("Task created: %s (status=%s)", task.title, task.status.value)
return {"success": True, "task": _task_to_dict(task)}
@router.post("/tasks/create", response_class=HTMLResponse)
async def form_create_task(
request: Request,
title: str = Form(...),
description: str = Form(""),
assigned_to: str = Form("timmy"),
priority: str = Form("normal"),
requires_approval: bool = Form(False),
):
"""Create a task from the dashboard form (Form-encoded)."""
task = create_task(
title=title,
description=description,
assigned_to=assigned_to,
created_by="user",
priority=priority,
requires_approval=requires_approval,
)
_notify_task_created(task)
_broadcast_task_event("task_created", task)
logger.info("Task created (form): %s", task.title)
# Return the new card for HTMX swap
return templates.TemplateResponse(
request,
"partials/task_card.html",
{"task": task},
)
@router.get("/api/tasks/counts", response_class=JSONResponse)
async def api_task_counts():
"""Get task counts by status (for nav badges)."""
counts = get_counts_by_status()
return {
"pending": counts.get("pending_approval", 0),
"approved": counts.get("approved", 0),
"running": counts.get("running", 0),
"completed": counts.get("completed", 0),
"failed": counts.get("failed", 0),
"vetoed": counts.get("vetoed", 0),
"backlogged": counts.get("backlogged", 0),
"total": sum(counts.values()),
}
# ── Backlog API (must be before {task_id} catch-all) ─────────────────────
@router.get("/api/tasks/backlog", response_class=JSONResponse)
async def api_list_backlogged(assigned_to: Optional[str] = None, limit: int = 50):
"""List all backlogged tasks."""
tasks = list_backlogged_tasks(assigned_to=assigned_to, limit=limit)
return {
"tasks": [_task_to_dict(t) for t in tasks],
"count": len(tasks),
}
@router.get("/api/tasks/{task_id}", response_class=JSONResponse)
async def api_get_task(task_id: str):
"""Get a single task by ID."""
task = get_task(task_id)
if not task:
raise HTTPException(404, "Task not found")
return _task_to_dict(task)
# ── Workflow actions ─────────────────────────────────────────────────────
@router.patch("/api/tasks/{task_id}/approve", response_class=JSONResponse)
async def api_approve_task(task_id: str):
"""Approve a pending task."""
task = get_task(task_id)
if not task:
raise HTTPException(404, "Task not found")
if task.status not in (TaskStatus.PENDING_APPROVAL,):
raise HTTPException(400, f"Cannot approve task in {task.status.value} state")
updated = update_task_status(task_id, TaskStatus.APPROVED)
_broadcast_task_event("task_approved", updated)
return {"success": True, "task": _task_to_dict(updated)}
@router.post("/tasks/{task_id}/approve", response_class=HTMLResponse)
async def htmx_approve_task(request: Request, task_id: str):
"""Approve a pending task (HTMX)."""
task = get_task(task_id)
if not task:
raise HTTPException(404, "Task not found")
updated = update_task_status(task_id, TaskStatus.APPROVED)
_broadcast_task_event("task_approved", updated)
return templates.TemplateResponse(
request, "partials/task_card.html", {"task": updated}
)
@router.patch("/api/tasks/{task_id}/veto", response_class=JSONResponse)
async def api_veto_task(task_id: str):
"""Veto (reject) a task."""
task = get_task(task_id)
if not task:
raise HTTPException(404, "Task not found")
if task.status in (TaskStatus.COMPLETED, TaskStatus.VETOED):
raise HTTPException(400, f"Cannot veto task in {task.status.value} state")
updated = update_task_status(task_id, TaskStatus.VETOED)
_broadcast_task_event("task_vetoed", updated)
return {"success": True, "task": _task_to_dict(updated)}
@router.post("/tasks/{task_id}/veto", response_class=HTMLResponse)
async def htmx_veto_task(request: Request, task_id: str):
"""Veto a task (HTMX)."""
task = get_task(task_id)
if not task:
raise HTTPException(404, "Task not found")
updated = update_task_status(task_id, TaskStatus.VETOED)
_broadcast_task_event("task_vetoed", updated)
return templates.TemplateResponse(
request, "partials/task_card.html", {"task": updated}
)
@router.patch("/api/tasks/{task_id}/modify", response_class=JSONResponse)
async def api_modify_task(task_id: str, request: Request):
"""Modify a task's title, description, assignment, or priority."""
task = get_task(task_id)
if not task:
raise HTTPException(404, "Task not found")
if task.status in (TaskStatus.COMPLETED, TaskStatus.VETOED):
raise HTTPException(400, f"Cannot modify task in {task.status.value} state")
body = await request.json()
updated = update_task(
task_id,
title=body.get("title"),
description=body.get("description"),
assigned_to=body.get("assigned_to"),
priority=body.get("priority"),
)
_broadcast_task_event("task_modified", updated)
return {"success": True, "task": _task_to_dict(updated)}
@router.post("/tasks/{task_id}/modify", response_class=HTMLResponse)
async def htmx_modify_task(
request: Request,
task_id: str,
title: str = Form(None),
description: str = Form(None),
assigned_to: str = Form(None),
priority: str = Form(None),
):
"""Modify a task (HTMX form)."""
task = get_task(task_id)
if not task:
raise HTTPException(404, "Task not found")
updated = update_task(
task_id,
title=title,
description=description,
assigned_to=assigned_to,
priority=priority,
)
_broadcast_task_event("task_modified", updated)
return templates.TemplateResponse(
request, "partials/task_card.html", {"task": updated}
)
@router.patch("/api/tasks/{task_id}/pause", response_class=JSONResponse)
async def api_pause_task(task_id: str):
"""Pause a running task."""
task = get_task(task_id)
if not task:
raise HTTPException(404, "Task not found")
if task.status != TaskStatus.RUNNING:
raise HTTPException(400, "Can only pause running tasks")
updated = update_task_status(task_id, TaskStatus.PAUSED)
_broadcast_task_event("task_paused", updated)
return {"success": True, "task": _task_to_dict(updated)}
@router.post("/tasks/{task_id}/pause", response_class=HTMLResponse)
async def htmx_pause_task(request: Request, task_id: str):
"""Pause a running task (HTMX)."""
task = get_task(task_id)
if not task:
raise HTTPException(404, "Task not found")
updated = update_task_status(task_id, TaskStatus.PAUSED)
_broadcast_task_event("task_paused", updated)
return templates.TemplateResponse(
request, "partials/task_card.html", {"task": updated}
)
@router.patch("/api/tasks/{task_id}/cancel", response_class=JSONResponse)
async def api_cancel_task(task_id: str):
"""Cancel a task (sets to failed)."""
task = get_task(task_id)
if not task:
raise HTTPException(404, "Task not found")
if task.status in (TaskStatus.COMPLETED, TaskStatus.VETOED):
raise HTTPException(400, f"Cannot cancel task in {task.status.value} state")
updated = update_task_status(task_id, TaskStatus.FAILED, result="Cancelled by user")
_broadcast_task_event("task_cancelled", updated)
return {"success": True, "task": _task_to_dict(updated)}
@router.post("/tasks/{task_id}/cancel", response_class=HTMLResponse)
async def htmx_cancel_task(request: Request, task_id: str):
"""Cancel a task (HTMX)."""
task = get_task(task_id)
if not task:
raise HTTPException(404, "Task not found")
updated = update_task_status(task_id, TaskStatus.FAILED, result="Cancelled by user")
_broadcast_task_event("task_cancelled", updated)
return templates.TemplateResponse(
request, "partials/task_card.html", {"task": updated}
)
@router.patch("/api/tasks/{task_id}/retry", response_class=JSONResponse)
async def api_retry_task(task_id: str):
"""Retry a failed task (resets to approved)."""
task = get_task(task_id)
if not task:
raise HTTPException(404, "Task not found")
if task.status != TaskStatus.FAILED:
raise HTTPException(400, "Can only retry failed tasks")
updated = update_task_status(task_id, TaskStatus.APPROVED, result=None)
_broadcast_task_event("task_retried", updated)
return {"success": True, "task": _task_to_dict(updated)}
@router.post("/tasks/{task_id}/retry", response_class=HTMLResponse)
async def htmx_retry_task(request: Request, task_id: str):
"""Retry a failed task (HTMX)."""
task = get_task(task_id)
if not task:
raise HTTPException(404, "Task not found")
updated = update_task_status(task_id, TaskStatus.APPROVED, result=None)
_broadcast_task_event("task_retried", updated)
return templates.TemplateResponse(
request, "partials/task_card.html", {"task": updated}
)
@router.patch("/api/tasks/{task_id}/unbacklog", response_class=JSONResponse)
async def api_unbacklog_task(task_id: str):
"""Move a backlogged task back to approved for re-processing."""
task = get_task(task_id)
if not task:
raise HTTPException(404, "Task not found")
if task.status != TaskStatus.BACKLOGGED:
raise HTTPException(400, "Can only unbacklog backlogged tasks")
updated = update_task_status(
task_id, TaskStatus.APPROVED, result=None, backlog_reason=None
)
_broadcast_task_event("task_unbacklogged", updated)
return {"success": True, "task": _task_to_dict(updated)}
@router.post("/tasks/{task_id}/unbacklog", response_class=HTMLResponse)
async def htmx_unbacklog_task(request: Request, task_id: str):
"""Move a backlogged task back to approved (HTMX)."""
task = get_task(task_id)
if not task:
raise HTTPException(404, "Task not found")
updated = update_task_status(
task_id, TaskStatus.APPROVED, result=None, backlog_reason=None
)
_broadcast_task_event("task_unbacklogged", updated)
return templates.TemplateResponse(
request, "partials/task_card.html", {"task": updated}
)
# ── Queue Status API ─────────────────────────────────────────────────────
@router.get("/api/queue/status", response_class=JSONResponse)
async def api_queue_status(assigned_to: str = "timmy"):
"""Get queue status for an agent - position info for polling."""
from swarm.task_queue.models import (
get_current_task_for_agent,
get_queue_position_ahead,
get_next_pending_task,
)
current = get_current_task_for_agent(assigned_to)
next_task = get_next_pending_task(assigned_to)
ahead = get_queue_position_ahead(assigned_to)
return {
"agent": assigned_to,
"is_working": current is not None,
"current_task": _task_to_dict(current) if current else None,
"next_task": _task_to_dict(next_task) if next_task else None,
"tasks_ahead": ahead,
}
@router.get("/api/queue/position/{task_id}", response_class=JSONResponse)
async def api_queue_position(task_id: str):
"""Get queue position for a specific task."""
from swarm.task_queue.models import get_queue_status_for_task
status = get_queue_status_for_task(task_id)
if "error" in status:
raise HTTPException(404, status["error"])
return status
@router.get("/api/queue/agent/{assigned_to}", response_class=JSONResponse)
async def api_agent_queue(assigned_to: str, limit: int = 20):
"""Get all pending tasks for an agent."""
from swarm.task_queue.models import list_tasks, TaskStatus
tasks = list_tasks(
assigned_to=assigned_to,
status=None, # All statuses
limit=limit,
)
# Filter to pending/running tasks
pending = [
t
for t in tasks
if t.status not in (TaskStatus.COMPLETED, TaskStatus.FAILED, TaskStatus.VETOED)
]
return {
"assigned_to": assigned_to,
"tasks": [_task_to_dict(t) for t in pending],
"count": len(pending),
}
# ── Helpers ──────────────────────────────────────────────────────────────
def _task_to_dict(task: QueueTask) -> dict:
d = {
"id": task.id,
"title": task.title,
"description": task.description,
"assigned_to": task.assigned_to,
"created_by": task.created_by,
"status": task.status.value,
"priority": task.priority.value,
"requires_approval": task.requires_approval,
"auto_approve": task.auto_approve,
"parent_task_id": task.parent_task_id,
"result": task.result,
"steps": task.steps,
"created_at": task.created_at,
"started_at": task.started_at,
"completed_at": task.completed_at,
"updated_at": task.updated_at,
}
if task.backlog_reason:
d["backlog_reason"] = task.backlog_reason
return d
def _notify_task_created(task: QueueTask):
try:
from infrastructure.notifications.push import notifier
notifier.notify(
title="New Task",
message=f"{task.created_by} created: {task.title}",
category="task",
native=task.priority in (TaskPriority.HIGH, TaskPriority.URGENT),
)
except Exception:
pass

99
src/dashboard/routes/upgrades.py
View File

@@ -1,99 +0,0 @@
"""Self-Upgrade Queue dashboard routes."""
from pathlib import Path
from fastapi import APIRouter, Form, HTTPException, Request
from fastapi.responses import HTMLResponse, JSONResponse
from fastapi.templating import Jinja2Templates
from self_coding.upgrades.models import list_upgrades, get_upgrade, UpgradeStatus, get_pending_count
from self_coding.upgrades.queue import UpgradeQueue
router = APIRouter(prefix="/self-modify", tags=["upgrades"])
templates = Jinja2Templates(directory=str(Path(__file__).parent.parent / "templates"))
@router.get("/queue", response_class=HTMLResponse)
async def upgrade_queue_page(request: Request):
"""Upgrade queue dashboard."""
pending = list_upgrades(status=UpgradeStatus.PROPOSED)
approved = list_upgrades(status=UpgradeStatus.APPROVED)
history = list_upgrades(status=None)[:20] # All recent
# Separate history by status
applied = [u for u in history if u.status == UpgradeStatus.APPLIED][:10]
rejected = [u for u in history if u.status == UpgradeStatus.REJECTED][:5]
failed = [u for u in history if u.status == UpgradeStatus.FAILED][:5]
return templates.TemplateResponse(
request,
"upgrade_queue.html",
{
"page_title": "Upgrade Queue",
"pending": pending,
"approved": approved,
"applied": applied,
"rejected": rejected,
"failed": failed,
"pending_count": len(pending),
},
)
@router.post("/queue/{upgrade_id}/approve", response_class=JSONResponse)
async def approve_upgrade_endpoint(upgrade_id: str):
"""Approve an upgrade proposal."""
upgrade = UpgradeQueue.approve(upgrade_id)
if not upgrade:
raise HTTPException(404, "Upgrade not found or not in proposed state")
return {"success": True, "upgrade_id": upgrade_id, "status": upgrade.status.value}
@router.post("/queue/{upgrade_id}/reject", response_class=JSONResponse)
async def reject_upgrade_endpoint(upgrade_id: str):
"""Reject an upgrade proposal."""
upgrade = UpgradeQueue.reject(upgrade_id)
if not upgrade:
raise HTTPException(404, "Upgrade not found or not in proposed state")
return {"success": True, "upgrade_id": upgrade_id, "status": upgrade.status.value}
@router.post("/queue/{upgrade_id}/apply", response_class=JSONResponse)
async def apply_upgrade_endpoint(upgrade_id: str):
"""Apply an approved upgrade."""
success, message = UpgradeQueue.apply(upgrade_id)
if not success:
raise HTTPException(400, message)
return {"success": True, "message": message}
@router.get("/queue/{upgrade_id}/diff", response_class=HTMLResponse)
async def view_diff(request: Request, upgrade_id: str):
"""View full diff for an upgrade."""
upgrade = get_upgrade(upgrade_id)
if not upgrade:
raise HTTPException(404, "Upgrade not found")
diff = UpgradeQueue.get_full_diff(upgrade_id)
return templates.TemplateResponse(
request,
"upgrade_diff.html",
{
"upgrade": upgrade,
"diff": diff,
},
)
@router.get("/api/pending-count", response_class=JSONResponse)
async def get_pending_upgrade_count():
"""Get count of pending upgrades (for nav badge)."""
return {"count": get_pending_count()}

39
src/dashboard/routes/voice.py
View File

@@ -113,48 +113,13 @@ async def process_voice_input(
)
elif intent.name == "swarm":
from swarm import registry
agents = registry.list_agents()
response_text = (
f"Swarm status: {len(agents)} agents registered. "
f"Use the dashboard for detailed task information."
)
response_text = "Swarm module is not currently active."
elif intent.name == "voice":
response_text = "Voice settings acknowledged. TTS is available for spoken responses."
elif intent.name == "code":
from config import settings as app_settings
if not app_settings.self_modify_enabled:
response_text = (
"Self-modification is disabled. "
"Set SELF_MODIFY_ENABLED=true to enable."
)
else:
import asyncio
from self_coding.self_modify.loop import SelfModifyLoop, ModifyRequest
target_files = []
if "target_file" in intent.entities:
target_files = [intent.entities["target_file"]]
loop = SelfModifyLoop()
request = ModifyRequest(
instruction=text,
target_files=target_files,
)
result = await asyncio.to_thread(loop.run, request)
if result.success:
sha_short = result.commit_sha[:8] if result.commit_sha else "none"
response_text = (
f"Code modification complete. "
f"Changed {len(result.files_changed)} file(s). "
f"Tests passed. Committed as {sha_short} "
f"on branch {result.branch_name}."
)
else:
response_text = f"Code modification failed: {result.error}"
response_text = "Self-modification module is not currently active."
else:
# Default: chat with Timmy

134
src/hands/__init__.py
View File

@@ -1,134 +0,0 @@
"""Hands — DEPRECATED. Use brain task queue instead.
This module is kept for backward compatibility during migration.
All functionality has been moved to the distributed brain task queue.
"""
from typing import Any, Optional
import logging
logger = logging.getLogger(__name__)
# Simple stub models for compatibility
class HandConfig:
"""Deprecated - use brain task queue."""
def __init__(self, *args, **kwargs):
self.name = kwargs.get("name", "unknown")
self.enabled = False
class HandState:
"""Deprecated."""
pass
class HandExecution:
"""Deprecated."""
pass
class HandStatus:
"""Deprecated."""
PENDING = "pending"
RUNNING = "running"
COMPLETED = "completed"
FAILED = "failed"
class HandOutcome:
"""Deprecated."""
pass
class TriggerType:
"""Deprecated."""
CRON = "cron"
EVENT = "event"
MANUAL = "manual"
class ApprovalGate:
"""Deprecated."""
pass
class ApprovalRequest:
"""Deprecated."""
pass
class ApprovalStatus:
"""Deprecated."""
pass
class ScheduleConfig:
"""Deprecated."""
pass
class OutputConfig:
"""Deprecated."""
pass
class ToolRequirement:
"""Deprecated."""
pass
class HandRegistry:
"""Deprecated - use brain task queue."""
def __init__(self, hands_dir: Optional[str] = None, db_path: Optional[str] = None):
self.hands_dir = hands_dir
self.db_path = db_path
logger.warning("HandRegistry is deprecated. Use brain.BrainClient instead.")
def list_hands(self):
return []
def get_hand(self, name: str):
raise NotImplementedError("Hands deprecated - use brain task queue")
def get_state(self, name: str):
return HandState()
def get_scheduled_hands(self):
return []
async def load_all(self):
pass
class HandScheduler:
"""Deprecated - use brain worker."""
def __init__(self, registry: Any):
self.registry = registry
logger.warning("HandScheduler is deprecated. Use brain.DistributedWorker instead.")
async def start(self):
pass
async def stop(self):
pass
async def schedule_hand(self, hand: Any):
pass
class HandRunner:
"""Deprecated - use brain worker."""
def __init__(self, registry: Any, llm_adapter: Any = None):
self.registry = registry
logger.warning("HandRunner is deprecated. Use brain.DistributedWorker instead.")
async def run_hand(self, name: str, context: Any = None):
raise NotImplementedError("Hands deprecated - use brain task queue")
class HandNotFoundError(Exception):
pass
class HandValidationError(Exception):
pass
__all__ = [
"HandConfig", "HandState", "HandExecution", "HandStatus", "HandOutcome",
"TriggerType", "ApprovalGate", "ApprovalRequest", "ApprovalStatus",
"ScheduleConfig", "OutputConfig", "ToolRequirement",
"HandRegistry", "HandScheduler", "HandRunner",
"HandNotFoundError", "HandValidationError",
]

3
src/infrastructure/events/broadcaster.py
View File

@@ -9,7 +9,10 @@ import json
import logging
from typing import Optional
try:
from swarm.event_log import EventLogEntry
except ImportError:
EventLogEntry = None
logger = logging.getLogger(__name__)

9
src/infrastructure/openfang/tools.py
View File

@@ -18,7 +18,12 @@ import logging
from typing import Any
from infrastructure.openfang.client import OPENFANG_HANDS, openfang_client
try:
from mcp.schemas.base import create_tool_schema
except ImportError:
def create_tool_schema(**kwargs):
return kwargs
logger = logging.getLogger(__name__)
@@ -190,7 +195,11 @@ def register_openfang_tools() -> int:
Returns the number of tools registered.
"""
try:
from mcp.registry import tool_registry
except ImportError:
logger.warning("MCP registry not available — skipping OpenFang tool registration")
return 0
count = 0
for hand_name in OPENFANG_HANDS:

9
src/lightning/CLAUDE.md
View File

@@ -1,9 +0,0 @@
# lightning/ — Module Guide
**Security-sensitive.** Bitcoin Lightning payment gating (L402).
Never hard-code secrets. Use `from config import settings` for all credentials.
## Testing
```bash
pytest tests/lightning/ -q
```

26
src/lightning/__init__.py
View File

@@ -1,26 +0,0 @@
"""Lightning Network payment backend interface.
This module provides a pluggable interface for Lightning Network operations,
allowing seamless switching between mock (development) and real LND backends.
Usage:
from lightning import get_backend, Invoice
backend = get_backend() # Uses LIGHTNING_BACKEND env var
invoice = backend.create_invoice(amount_sats=100, memo="API access")
paid = backend.check_payment(invoice.payment_hash)
Configuration:
LIGHTNING_BACKEND=mock # Default, for development
LIGHTNING_BACKEND=lnd # Real LND via gRPC
# LND-specific settings (when backend=lnd)
LND_GRPC_HOST=localhost:10009
LND_TLS_CERT_PATH=/path/to/tls.cert
LND_MACAROON_PATH=/path/to/admin.macaroon
"""
from lightning.base import Invoice, LightningBackend
from lightning.factory import get_backend
__all__ = ["Invoice", "LightningBackend", "get_backend"]

188
src/lightning/base.py
View File

@@ -1,188 +0,0 @@
"""Abstract base class for Lightning Network backends.
Defines the contract that all Lightning implementations must fulfill.
This abstraction allows the rest of the system to work identically
whether using mock invoices or real LND gRPC calls.
"""
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from typing import Optional
import time
@dataclass
class Invoice:
"""Lightning invoice data structure.
This is backend-agnostic — the same structure is used for
mock invoices and real LND invoices.
"""
payment_hash: str
payment_request: str # bolt11 invoice string
amount_sats: int
memo: str = ""
created_at: float = field(default_factory=time.time)
settled: bool = False
settled_at: Optional[float] = None
preimage: Optional[str] = None
@property
def is_expired(self, expiry_seconds: int = 3600) -> bool:
"""Check if invoice has expired (default 1 hour)."""
return time.time() > self.created_at + expiry_seconds
@dataclass
class PaymentReceipt:
"""Proof of payment for a settled invoice."""
payment_hash: str
preimage: str
amount_sats: int
settled_at: float
class LightningBackend(ABC):
"""Abstract interface for Lightning Network operations.
Implementations:
- MockBackend: In-memory invoices for development/testing
- LndBackend: Real LND node via gRPC
- ClnBackend: Core Lightning via Unix socket (future)
All methods are synchronous. Async wrappers can be added at
the application layer if needed.
"""
name: str = "abstract"
@abstractmethod
def create_invoice(
self,
amount_sats: int,
memo: str = "",
expiry_seconds: int = 3600
) -> Invoice:
"""Create a new Lightning invoice.
Args:
amount_sats: Amount in satoshis
memo: Description shown in wallet
expiry_seconds: How long until invoice expires
Returns:
Invoice object with payment_request (bolt11 string)
Raises:
LightningError: If invoice creation fails
"""
pass
@abstractmethod
def check_payment(self, payment_hash: str) -> bool:
"""Check whether an invoice has been paid.
Args:
payment_hash: The invoice to check
Returns:
True if paid/settled, False otherwise
Note:
In mock mode this may auto-settle. In production this
queries the Lightning node for the invoice state.
"""
pass
@abstractmethod
def get_invoice(self, payment_hash: str) -> Optional[Invoice]:
"""Get full invoice details by payment hash.
Args:
payment_hash: The invoice to retrieve
Returns:
Invoice object or None if not found
"""
pass
@abstractmethod
def settle_invoice(self, payment_hash: str, preimage: str) -> bool:
"""Manually settle an invoice with a preimage.
This is primarily used for testing or when receiving
payments through a separate channel.
Args:
payment_hash: The invoice to settle
preimage: The payment preimage (proof of payment)
Returns:
True if settlement succeeded
Raises:
ValueError: If preimage doesn't match payment_hash
"""
pass
@abstractmethod
def list_invoices(
self,
settled_only: bool = False,
limit: int = 100
) -> list[Invoice]:
"""List recent invoices.
Args:
settled_only: Only return paid invoices
limit: Maximum number to return (newest first)
Returns:
List of Invoice objects
"""
pass
@abstractmethod
def get_balance_sats(self) -> int:
"""Get the node's available balance in satoshis.
Returns:
Spendable on-chain + off-chain balance
Note:
Mock backends may return a fake value.
"""
pass
@abstractmethod
def health_check(self) -> dict:
"""Check backend health and connectivity.
Returns:
Dict with:
- ok: bool
- error: str or None
- block_height: int (if available)
- synced: bool (if available)
"""
pass
class LightningError(Exception):
"""Base exception for Lightning backend errors."""
pass
class InvoiceNotFoundError(LightningError):
"""Raised when an invoice doesn't exist."""
pass
class PaymentFailedError(LightningError):
"""Raised when a payment operation fails."""
pass
class BackendNotAvailableError(LightningError):
"""Raised when the Lightning node is unreachable."""
pass

115
src/lightning/factory.py
View File

@@ -1,115 +0,0 @@
"""Lightning backend factory — creates appropriate backend based on config.
Usage:
from lightning import get_backend
backend = get_backend() # Reads LIGHTNING_BACKEND env var
# or
backend = get_backend("lnd") # Force specific backend
"""
import logging
import os
from typing import Optional
from config import settings
from lightning.base import LightningBackend
logger = logging.getLogger(__name__)
# Registry of available backends
_BACKENDS: dict[str, type[LightningBackend]] = {}
def _register_backends():
"""Register available backends (lazy import to avoid dependencies)."""
global _BACKENDS
if _BACKENDS:
return
# Always register mock backend
from lightning.mock_backend import MockBackend
_BACKENDS["mock"] = MockBackend
# Register LND backend if grpc available
try:
from lightning.lnd_backend import LndBackend
_BACKENDS["lnd"] = LndBackend
logger.debug("LND backend registered (grpc available)")
except ImportError as e:
logger.debug("LND backend not available: %s", e)
# Future: Add Core Lightning (CLN) backend here
# try:
# from lightning.cln_backend import ClnBackend
# _BACKENDS["cln"] = ClnBackend
# except ImportError:
# pass
def get_backend(name: Optional[str] = None) -> LightningBackend:
"""Get a Lightning backend instance.
Args:
name: Backend type ('mock', 'lnd').
Defaults to LIGHTNING_BACKEND env var or 'mock'.
Returns:
Configured LightningBackend instance
Raises:
ValueError: If backend type is unknown
LightningError: If backend initialization fails
Examples:
>>> backend = get_backend() # Use env var or default
>>> backend = get_backend("mock") # Force mock
>>> backend = get_backend("lnd") # Use real LND
"""
_register_backends()
backend_name = (name or settings.lightning_backend).lower()
if backend_name not in _BACKENDS:
available = ", ".join(_BACKENDS.keys())
raise ValueError(
f"Unknown Lightning backend: {backend_name!r}. "
f"Available: {available}"
)
backend_class = _BACKENDS[backend_name]
instance = backend_class()
logger.info("Lightning backend ready: %s", backend_name)
return instance
def list_backends() -> list[str]:
"""List available backend types.
Returns:
List of backend names that can be passed to get_backend()
"""
_register_backends()
return list(_BACKENDS.keys())
def get_backend_info() -> dict:
"""Get information about the current backend configuration.
Returns:
Dict with backend info for health/status endpoints
"""
backend_name = settings.lightning_backend
return {
"configured_backend": backend_name,
"available_backends": list_backends(),
"env_vars": {
"LIGHTNING_BACKEND": backend_name,
"LND_GRPC_HOST": os.environ.get("LND_GRPC_HOST", "not set"),
"LND_TLS_CERT_PATH": "set" if os.environ.get("LND_TLS_CERT_PATH") else "not set",
"LND_MACAROON_PATH": "set" if os.environ.get("LND_MACAROON_PATH") else "not set",
}
}

488
src/lightning/ledger.py
View File

@@ -1,488 +0,0 @@
"""Lightning Network transaction ledger.
Tracks all Lightning payments in SQLite for audit, accounting, and dashboard display.
"""
import sqlite3
import uuid
from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum
from pathlib import Path
from typing import Optional
DB_PATH = Path("data/swarm.db")
class TransactionType(str, Enum):
"""Types of Lightning transactions."""
INCOMING = "incoming" # Invoice created (we're receiving)
OUTGOING = "outgoing" # Payment sent (we're paying)
class TransactionStatus(str, Enum):
"""Status of a transaction."""
PENDING = "pending"
SETTLED = "settled"
FAILED = "failed"
EXPIRED = "expired"
@dataclass
class LedgerEntry:
"""A Lightning transaction record."""
id: str = field(default_factory=lambda: str(uuid.uuid4()))
tx_type: TransactionType = TransactionType.INCOMING
status: TransactionStatus = TransactionStatus.PENDING
payment_hash: str = "" # Lightning payment hash
amount_sats: int = 0
memo: str = "" # Description/purpose
invoice: Optional[str] = None # BOLT11 invoice string
preimage: Optional[str] = None # Payment preimage (proof of payment)
source: str = "" # Component that created the transaction
task_id: Optional[str] = None # Associated task, if any
agent_id: Optional[str] = None # Associated agent, if any
created_at: str = field(
default_factory=lambda: datetime.now(timezone.utc).isoformat()
)
settled_at: Optional[str] = None
fee_sats: int = 0 # Routing fee paid
def _get_conn() -> sqlite3.Connection:
DB_PATH.parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(str(DB_PATH))
conn.row_factory = sqlite3.Row
conn.execute(
"""
CREATE TABLE IF NOT EXISTS ledger (
id TEXT PRIMARY KEY,
tx_type TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending',
payment_hash TEXT UNIQUE NOT NULL,
amount_sats INTEGER NOT NULL,
memo TEXT,
invoice TEXT,
preimage TEXT,
source TEXT NOT NULL,
task_id TEXT,
agent_id TEXT,
created_at TEXT NOT NULL,
settled_at TEXT,
fee_sats INTEGER DEFAULT 0
)
"""
)
# Create indexes for common queries
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_ledger_status ON ledger(status)"
)
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_ledger_hash ON ledger(payment_hash)"
)
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_ledger_task ON ledger(task_id)"
)
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_ledger_agent ON ledger(agent_id)"
)
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_ledger_created ON ledger(created_at)"
)
conn.commit()
return conn
def create_invoice_entry(
payment_hash: str,
amount_sats: int,
memo: str = "",
invoice: Optional[str] = None,
source: str = "system",
task_id: Optional[str] = None,
agent_id: Optional[str] = None,
) -> LedgerEntry:
"""Record a new incoming invoice (we're receiving payment).
Args:
payment_hash: Lightning payment hash
amount_sats: Invoice amount in satoshis
memo: Payment description
invoice: Full BOLT11 invoice string
source: Component that created the invoice
task_id: Associated task ID
agent_id: Associated agent ID
Returns:
The created LedgerEntry
"""
entry = LedgerEntry(
tx_type=TransactionType.INCOMING,
status=TransactionStatus.PENDING,
payment_hash=payment_hash,
amount_sats=amount_sats,
memo=memo,
invoice=invoice,
source=source,
task_id=task_id,
agent_id=agent_id,
)
conn = _get_conn()
conn.execute(
"""
INSERT INTO ledger (id, tx_type, status, payment_hash, amount_sats,
memo, invoice, source, task_id, agent_id, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
entry.id,
entry.tx_type.value,
entry.status.value,
entry.payment_hash,
entry.amount_sats,
entry.memo,
entry.invoice,
entry.source,
entry.task_id,
entry.agent_id,
entry.created_at,
),
)
conn.commit()
conn.close()
return entry
def record_outgoing_payment(
payment_hash: str,
amount_sats: int,
memo: str = "",
invoice: Optional[str] = None,
source: str = "system",
task_id: Optional[str] = None,
agent_id: Optional[str] = None,
) -> LedgerEntry:
"""Record an outgoing payment (we're paying someone).
Args:
payment_hash: Lightning payment hash
amount_sats: Payment amount in satoshis
memo: Payment description
invoice: BOLT11 invoice we paid
source: Component that initiated payment
task_id: Associated task ID
agent_id: Associated agent ID
Returns:
The created LedgerEntry
"""
entry = LedgerEntry(
tx_type=TransactionType.OUTGOING,
status=TransactionStatus.PENDING,
payment_hash=payment_hash,
amount_sats=amount_sats,
memo=memo,
invoice=invoice,
source=source,
task_id=task_id,
agent_id=agent_id,
)
conn = _get_conn()
conn.execute(
"""
INSERT INTO ledger (id, tx_type, status, payment_hash, amount_sats,
memo, invoice, source, task_id, agent_id, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
entry.id,
entry.tx_type.value,
entry.status.value,
entry.payment_hash,
entry.amount_sats,
entry.memo,
entry.invoice,
entry.source,
entry.task_id,
entry.agent_id,
entry.created_at,
),
)
conn.commit()
conn.close()
return entry
def mark_settled(
payment_hash: str,
preimage: Optional[str] = None,
fee_sats: int = 0,
) -> Optional[LedgerEntry]:
"""Mark a transaction as settled (payment received or sent successfully).
Args:
payment_hash: Lightning payment hash
preimage: Payment preimage (proof of payment)
fee_sats: Routing fee paid (for outgoing payments)
Returns:
Updated LedgerEntry or None if not found
"""
settled_at = datetime.now(timezone.utc).isoformat()
conn = _get_conn()
cursor = conn.execute(
"""
UPDATE ledger
SET status = ?, preimage = ?, settled_at = ?, fee_sats = ?
WHERE payment_hash = ?
""",
(TransactionStatus.SETTLED.value, preimage, settled_at, fee_sats, payment_hash),
)
conn.commit()
if cursor.rowcount == 0:
conn.close()
return None
# Fetch and return updated entry
entry = get_by_hash(payment_hash)
conn.close()
return entry
def mark_failed(payment_hash: str, reason: str = "") -> Optional[LedgerEntry]:
"""Mark a transaction as failed.
Args:
payment_hash: Lightning payment hash
reason: Failure reason (stored in memo)
Returns:
Updated LedgerEntry or None if not found
"""
conn = _get_conn()
cursor = conn.execute(
"""
UPDATE ledger
SET status = ?, memo = memo || ' [FAILED: ' || ? || ']'
WHERE payment_hash = ?
""",
(TransactionStatus.FAILED.value, reason, payment_hash),
)
conn.commit()
if cursor.rowcount == 0:
conn.close()
return None
entry = get_by_hash(payment_hash)
conn.close()
return entry
def get_by_hash(payment_hash: str) -> Optional[LedgerEntry]:
"""Get a transaction by payment hash."""
conn = _get_conn()
row = conn.execute(
"SELECT * FROM ledger WHERE payment_hash = ?", (payment_hash,)
).fetchone()
conn.close()
if row is None:
return None
return LedgerEntry(
id=row["id"],
tx_type=TransactionType(row["tx_type"]),
status=TransactionStatus(row["status"]),
payment_hash=row["payment_hash"],
amount_sats=row["amount_sats"],
memo=row["memo"],
invoice=row["invoice"],
preimage=row["preimage"],
source=row["source"],
task_id=row["task_id"],
agent_id=row["agent_id"],
created_at=row["created_at"],
settled_at=row["settled_at"],
fee_sats=row["fee_sats"],
)
def list_transactions(
tx_type: Optional[TransactionType] = None,
status: Optional[TransactionStatus] = None,
task_id: Optional[str] = None,
agent_id: Optional[str] = None,
limit: int = 100,
offset: int = 0,
) -> list[LedgerEntry]:
"""List transactions with optional filtering.
Returns:
List of LedgerEntry objects, newest first
"""
conn = _get_conn()
conditions = []
params = []
if tx_type:
conditions.append("tx_type = ?")
params.append(tx_type.value)
if status:
conditions.append("status = ?")
params.append(status.value)
if task_id:
conditions.append("task_id = ?")
params.append(task_id)
if agent_id:
conditions.append("agent_id = ?")
params.append(agent_id)
where_clause = "WHERE " + " AND ".join(conditions) if conditions else ""
query = f"""
SELECT * FROM ledger
{where_clause}
ORDER BY created_at DESC
LIMIT ? OFFSET ?
"""
params.extend([limit, offset])
rows = conn.execute(query, params).fetchall()
conn.close()
return [
LedgerEntry(
id=r["id"],
tx_type=TransactionType(r["tx_type"]),
status=TransactionStatus(r["status"]),
payment_hash=r["payment_hash"],
amount_sats=r["amount_sats"],
memo=r["memo"],
invoice=r["invoice"],
preimage=r["preimage"],
source=r["source"],
task_id=r["task_id"],
agent_id=r["agent_id"],
created_at=r["created_at"],
settled_at=r["settled_at"],
fee_sats=r["fee_sats"],
)
for r in rows
]
def get_balance() -> dict:
"""Get current balance summary.
Returns:
Dict with incoming, outgoing, pending, and available balances
"""
conn = _get_conn()
# Incoming (invoices we created that are settled)
incoming = conn.execute(
"""
SELECT COALESCE(SUM(amount_sats), 0) as total
FROM ledger
WHERE tx_type = ? AND status = ?
""",
(TransactionType.INCOMING.value, TransactionStatus.SETTLED.value),
).fetchone()["total"]
# Outgoing (payments we sent that are settled)
outgoing_result = conn.execute(
"""
SELECT COALESCE(SUM(amount_sats), 0) as total,
COALESCE(SUM(fee_sats), 0) as fees
FROM ledger
WHERE tx_type = ? AND status = ?
""",
(TransactionType.OUTGOING.value, TransactionStatus.SETTLED.value),
).fetchone()
outgoing = outgoing_result["total"]
fees = outgoing_result["fees"]
# Pending incoming
pending_incoming = conn.execute(
"""
SELECT COALESCE(SUM(amount_sats), 0) as total
FROM ledger
WHERE tx_type = ? AND status = ?
""",
(TransactionType.INCOMING.value, TransactionStatus.PENDING.value),
).fetchone()["total"]
# Pending outgoing
pending_outgoing = conn.execute(
"""
SELECT COALESCE(SUM(amount_sats), 0) as total
FROM ledger
WHERE tx_type = ? AND status = ?
""",
(TransactionType.OUTGOING.value, TransactionStatus.PENDING.value),
).fetchone()["total"]
conn.close()
return {
"incoming_total_sats": incoming,
"outgoing_total_sats": outgoing,
"fees_paid_sats": fees,
"net_sats": incoming - outgoing - fees,
"pending_incoming_sats": pending_incoming,
"pending_outgoing_sats": pending_outgoing,
"available_sats": incoming - outgoing - fees - pending_outgoing,
}
def get_transaction_stats(days: int = 30) -> dict:
"""Get transaction statistics for the last N days.
Returns:
Dict with daily transaction counts and volumes
"""
conn = _get_conn()
from datetime import timedelta
cutoff = (datetime.now(timezone.utc) - timedelta(days=days)).isoformat()
rows = conn.execute(
"""
SELECT
date(created_at) as date,
tx_type,
status,
COUNT(*) as count,
SUM(amount_sats) as volume
FROM ledger
WHERE created_at > ?
GROUP BY date(created_at), tx_type, status
ORDER BY date DESC
""",
(cutoff,),
).fetchall()
conn.close()
stats = {}
for r in rows:
date = r["date"]
if date not in stats:
stats[date] = {"incoming": {"count": 0, "volume": 0},
"outgoing": {"count": 0, "volume": 0}}
tx_type = r["tx_type"]
if tx_type == TransactionType.INCOMING.value:
stats[date]["incoming"]["count"] += r["count"]
stats[date]["incoming"]["volume"] += r["volume"]
else:
stats[date]["outgoing"]["count"] += r["count"]
stats[date]["outgoing"]["volume"] += r["volume"]
return stats

370
src/lightning/lnd_backend.py
View File

@@ -1,370 +0,0 @@
"""LND Lightning backend — real Bitcoin payments via gRPC.
Connects to a local LND instance for production use.
Handles invoice creation, payment verification, and node health.
Requirements:
pip install grpcio protobuf
LND Setup:
1. Run LND with --tlsextradomain if accessing remotely
2. Copy tls.cert and admin.macaroon to accessible paths
3. Set environment variables (see below)
Environment:
LIGHTNING_BACKEND=lnd
LND_GRPC_HOST=localhost:10009
LND_TLS_CERT_PATH=/path/to/tls.cert
LND_MACAROON_PATH=/path/to/admin.macaroon
LND_VERIFY_SSL=true # Set to false only for development
Example LND gRPC calls:
AddInvoice - Create new invoice
LookupInvoice - Check payment status
ListChannels - Get channel balances
GetInfo - Node health and sync status
"""
import hashlib
import logging
import os
import ssl
import time
from typing import Optional
from lightning.base import (
Invoice,
LightningBackend,
BackendNotAvailableError,
InvoiceNotFoundError,
LightningError,
)
logger = logging.getLogger(__name__)
# Optional import — graceful degradation if grpc not installed
try:
import grpc
GRPC_AVAILABLE = True
except ImportError:
GRPC_AVAILABLE = False
logger.warning("grpcio not installed — LND backend unavailable")
class LndBackend(LightningBackend):
"""Real Lightning backend via LND gRPC.
This backend creates real invoices that require real sats to pay.
Only use in production with proper LND setup.
Connection is lazy — gRPC channel created on first use.
"""
name = "lnd"
def __init__(
self,
host: Optional[str] = None,
tls_cert_path: Optional[str] = None,
macaroon_path: Optional[str] = None,
verify_ssl: Optional[bool] = None,
) -> None:
"""Initialize LND backend.
Args:
host: LND gRPC host:port (default: LND_GRPC_HOST env var)
tls_cert_path: Path to tls.cert (default: LND_TLS_CERT_PATH env var)
macaroon_path: Path to admin.macaroon (default: LND_MACAROON_PATH env var)
verify_ssl: Verify TLS certificate (default: LND_VERIFY_SSL env var)
"""
if not GRPC_AVAILABLE:
raise LightningError(
"grpcio not installed. Run: pip install grpcio protobuf"
)
self._host = host or os.environ.get("LND_GRPC_HOST", "localhost:10009")
self._tls_cert_path = tls_cert_path or os.environ.get("LND_TLS_CERT_PATH")
self._macaroon_path = macaroon_path or os.environ.get("LND_MACAROON_PATH")
self._verify_ssl = verify_ssl
if self._verify_ssl is None:
self._verify_ssl = os.environ.get("LND_VERIFY_SSL", "true").lower() == "true"
self._channel: Optional[grpc.Channel] = None
self._stub: Optional[object] = None # lnrpc.LightningStub
logger.info(
"LndBackend initialized — host: %s, tls: %s, macaroon: %s",
self._host,
"configured" if self._tls_cert_path else "default",
"configured" if self._macaroon_path else "missing",
)
# Warn if config looks incomplete
if not self._macaroon_path or not os.path.exists(self._macaroon_path):
logger.warning(
"LND macaroon not found at %s — payments will fail",
self._macaroon_path
)
def _get_stub(self):
"""Lazy initialization of gRPC stub."""
if self._stub is not None:
return self._stub
# Build channel credentials
if self._tls_cert_path and os.path.exists(self._tls_cert_path):
with open(self._tls_cert_path, "rb") as f:
tls_cert = f.read()
credentials = grpc.ssl_channel_credentials(tls_cert)
else:
# Use system root certificates
credentials = grpc.ssl_channel_credentials()
# Build macaroon credentials
call_credentials = None
if self._macaroon_path and os.path.exists(self._macaroon_path):
with open(self._macaroon_path, "rb") as f:
macaroon = f.read().hex()
def metadata_callback(context, callback):
callback([("macaroon", macaroon)], None)
call_credentials = grpc.metadata_call_credentials(metadata_callback)
# Combine credentials
if call_credentials:
composite = grpc.composite_channel_credentials(
credentials,
call_credentials
)
else:
composite = credentials
# Create channel
self._channel = grpc.secure_channel(self._host, composite)
# Import and create stub
try:
# lnd/grpc imports would go here
# from lnd import lightning_pb2, lightning_pb2_grpc
# self._stub = lightning_pb2_grpc.LightningStub(self._channel)
# For now, stub is None — real implementation needs LND protos
logger.warning("LND gRPC stubs not yet implemented — using placeholder")
self._stub = None
except ImportError as e:
raise BackendNotAvailableError(
f"LND gRPC stubs not available: {e}. "
"Generate from LND proto files or install lndgrpc package."
)
return self._stub
def _check_stub(self):
"""Ensure stub is available or raise appropriate error."""
stub = self._get_stub()
if stub is None:
raise BackendNotAvailableError(
"LND gRPC not fully implemented. "
"This is a stub — implement gRPC calls to use real LND."
)
return stub
def create_invoice(
self,
amount_sats: int,
memo: str = "",
expiry_seconds: int = 3600
) -> Invoice:
"""Create a real Lightning invoice via LND."""
stub = self._check_stub()
try:
# Real implementation:
# request = lightning_pb2.Invoice(
# value=amount_sats,
# memo=memo,
# expiry=expiry_seconds,
# )
# response = stub.AddInvoice(request)
#
# return Invoice(
# payment_hash=response.r_hash.hex(),
# payment_request=response.payment_request,
# amount_sats=amount_sats,
# memo=memo,
# )
raise NotImplementedError(
"LND gRPC integration incomplete. "
"Generate protobuf stubs from LND source and implement AddInvoice."
)
except grpc.RpcError as e:
logger.error("LND AddInvoice failed: %s", e)
raise LightningError(f"Invoice creation failed: {e.details()}") from e
def check_payment(self, payment_hash: str) -> bool:
"""Check if invoice is paid via LND LookupInvoice."""
stub = self._check_stub()
try:
# Real implementation:
# request = lightning_pb2.PaymentHash(
# r_hash=bytes.fromhex(payment_hash)
# )
# response = stub.LookupInvoice(request)
# return response.state == lightning_pb2.Invoice.SETTLED
raise NotImplementedError("LND LookupInvoice not implemented")
except grpc.RpcError as e:
if e.code() == grpc.StatusCode.NOT_FOUND:
return False
logger.error("LND LookupInvoice failed: %s", e)
raise LightningError(f"Payment check failed: {e.details()}") from e
def get_invoice(self, payment_hash: str) -> Optional[Invoice]:
"""Get invoice details from LND."""
stub = self._check_stub()
try:
# request = lightning_pb2.PaymentHash(
# r_hash=bytes.fromhex(payment_hash)
# )
# response = stub.LookupInvoice(request)
#
# return Invoice(
# payment_hash=response.r_hash.hex(),
# payment_request=response.payment_request,
# amount_sats=response.value,
# memo=response.memo,
# created_at=response.creation_date,
# settled=response.state == lightning_pb2.Invoice.SETTLED,
# settled_at=response.settle_date if response.settled else None,
# )
raise NotImplementedError("LND LookupInvoice not implemented")
except grpc.RpcError as e:
if e.code() == grpc.StatusCode.NOT_FOUND:
return None
raise LightningError(f"Invoice lookup failed: {e.details()}") from e
def settle_invoice(self, payment_hash: str, preimage: str) -> bool:
"""Manually settle is not typically supported by LND.
LND handles settlement automatically when payment arrives.
This method exists for interface compatibility but raises
an error in production.
"""
logger.warning(
"Manual invoice settlement not supported by LND — "
"invoices settle automatically when paid"
)
return False
def list_invoices(
self,
settled_only: bool = False,
limit: int = 100
) -> list[Invoice]:
"""List recent invoices from LND."""
stub = self._check_stub()
try:
# request = lightning_pb2.ListInvoiceRequest(
# num_max_invoices=limit,
# reversed=True, # Newest first
# )
# response = stub.ListInvoices(request)
#
# invoices = []
# for inv in response.invoices:
# if settled_only and inv.state != lightning_pb2.Invoice.SETTLED:
# continue
# invoices.append(self._grpc_invoice_to_model(inv))
# return invoices
raise NotImplementedError("LND ListInvoices not implemented")
except grpc.RpcError as e:
raise LightningError(f"List invoices failed: {e.details()}") from e
def get_balance_sats(self) -> int:
"""Get total balance from LND."""
stub = self._check_stub()
try:
# response = stub.WalletBalance(request)
# return response.total_balance
# For now, return 0 to indicate "real value not available"
logger.warning("LND WalletBalance not implemented — returning 0")
return 0
except grpc.RpcError as e:
raise LightningError(f"Balance check failed: {e.details()}") from e
def health_check(self) -> dict:
"""Check LND node health and sync status."""
stub = self._check_stub()
try:
# response = stub.GetInfo(request)
# return {
# "ok": response.synced_to_chain and response.synced_to_graph,
# "error": None,
# "block_height": response.block_height,
# "synced": response.synced_to_chain,
# "backend": "lnd",
# "version": response.version,
# "alias": response.alias,
# }
# Return degraded status if stub not available
return {
"ok": False,
"error": "LND gRPC not fully implemented — see documentation",
"block_height": 0,
"synced": False,
"backend": "lnd-stub",
}
except grpc.RpcError as e:
return {
"ok": False,
"error": str(e.details()),
"block_height": 0,
"synced": False,
"backend": "lnd",
}
def generate_lnd_protos():
"""Documentation for generating LND protobuf stubs.
To use real LND, you need to generate Python gRPC stubs from
the LND proto files.
Steps:
1. Clone LND repository:
git clone https://github.com/lightningnetwork/lnd.git
2. Install protoc and grpc tools:
pip install grpcio grpcio-tools
3. Generate Python stubs:
python -m grpc_tools.protoc \
--proto_path=lnd/lnrpc \
--python_out=src/lightning/protos \
--grpc_python_out=src/lightning/protos \
lnd/lnrpc/lightning.proto
4. Import and use the generated stubs in LndBackend
Alternative:
Use the 'lndgrpc' or 'pylnd' packages from PyPI if available.
"""
print(generate_lnd_protos.__doc__)

163
src/lightning/mock_backend.py
View File

@@ -1,163 +0,0 @@
"""Mock Lightning backend for development and testing.
Provides in-memory invoice tracking without requiring a real
Lightning node. Invoices auto-settle for easy testing.
"""
import hashlib
import hmac
import logging
import os
import secrets
import time
from typing import Optional
from config import settings
from lightning.base import Invoice, LightningBackend, LightningError
logger = logging.getLogger(__name__)
# Read secret from centralised config (validated at startup in config.py)
_HMAC_SECRET = settings.l402_hmac_secret.encode()
class MockBackend(LightningBackend):
"""In-memory Lightning backend for development.
Creates fake invoices that auto-settle. No real sats are moved.
Useful for:
- Local development without LND setup
- Integration tests
- CI/CD pipelines
Environment:
LIGHTNING_BACKEND=mock
L402_HMAC_SECRET=your-secret # Optional
MOCK_AUTO_SETTLE=true # Auto-settle invoices (default: true)
"""
name = "mock"
def __init__(self) -> None:
self._invoices: dict[str, Invoice] = {}
self._auto_settle = os.environ.get("MOCK_AUTO_SETTLE", "true").lower() == "true"
logger.info("MockBackend initialized — auto_settle: %s", self._auto_settle)
def create_invoice(
self,
amount_sats: int,
memo: str = "",
expiry_seconds: int = 3600
) -> Invoice:
"""Create a mock invoice with fake bolt11 string."""
preimage = secrets.token_hex(32)
payment_hash = hashlib.sha256(bytes.fromhex(preimage)).hexdigest()
# Generate mock bolt11 — deterministic based on secret
signature = hmac.new(
_HMAC_SECRET,
payment_hash.encode(),
hashlib.sha256
).hexdigest()[:20]
payment_request = f"lnbc{amount_sats}n1mock{signature}"
invoice = Invoice(
payment_hash=payment_hash,
payment_request=payment_request,
amount_sats=amount_sats,
memo=memo,
preimage=preimage,
)
self._invoices[payment_hash] = invoice
logger.info(
"Mock invoice: %d sats — %s (hash: %s…)",
amount_sats, memo, payment_hash[:12]
)
if self._auto_settle:
# Mark as settled immediately for seamless dev experience
invoice.settled = True
invoice.settled_at = time.time()
logger.debug("Auto-settled invoice %s…", payment_hash[:12])
return invoice
def check_payment(self, payment_hash: str) -> bool:
"""Check invoice status — auto-settles in mock mode."""
invoice = self._invoices.get(payment_hash)
if invoice is None:
return False
if self._auto_settle and not invoice.settled:
invoice.settled = True
invoice.settled_at = time.time()
return invoice.settled
def get_invoice(self, payment_hash: str) -> Optional[Invoice]:
"""Retrieve invoice by payment hash."""
invoice = self._invoices.get(payment_hash)
if invoice:
# Update settled status
self.check_payment(payment_hash)
return invoice
def settle_invoice(self, payment_hash: str, preimage: str) -> bool:
"""Manually settle an invoice with preimage verification."""
invoice = self._invoices.get(payment_hash)
if invoice is None:
raise LightningError(f"Invoice not found: {payment_hash}")
# Verify preimage matches payment_hash
expected_hash = hashlib.sha256(bytes.fromhex(preimage)).hexdigest()
if expected_hash != payment_hash:
logger.warning(
"Preimage mismatch for %s… — expected %s…, got %s…",
payment_hash[:12],
expected_hash[:12],
hashlib.sha256(bytes.fromhex(preimage)).hexdigest()[:12]
)
return False
invoice.settled = True
invoice.settled_at = time.time()
invoice.preimage = preimage
logger.info("Settled invoice %s…", payment_hash[:12])
return True
def list_invoices(
self,
settled_only: bool = False,
limit: int = 100
) -> list[Invoice]:
"""List recent invoices, newest first."""
invoices = sorted(
self._invoices.values(),
key=lambda i: i.created_at,
reverse=True
)
if settled_only:
invoices = [i for i in invoices if i.settled]
return invoices[:limit]
def get_balance_sats(self) -> int:
"""Return fake balance for mock mode."""
# Return a reasonable-looking number for UI testing
return 1_000_000 # 1M sats
def health_check(self) -> dict:
"""Always healthy in mock mode."""
return {
"ok": True,
"error": None,
"block_height": 800_000,
"synced": True,
"backend": "mock",
"auto_settle": self._auto_settle,
}

30
src/mcp/__init__.py
View File

@@ -1,30 +0,0 @@
"""MCP (Model Context Protocol) package.
Provides tool registry, server, schema management, and auto-discovery.
"""
from mcp.registry import tool_registry, register_tool, ToolRegistry
from mcp.server import mcp_server, MCPServer, MCPHTTPServer
from mcp.schemas.base import create_tool_schema
from mcp.discovery import ToolDiscovery, mcp_tool, get_discovery
from mcp.bootstrap import auto_bootstrap, get_bootstrap_status
__all__ = [
# Registry
"tool_registry",
"register_tool",
"ToolRegistry",
# Server
"mcp_server",
"MCPServer",
"MCPHTTPServer",
# Schemas
"create_tool_schema",
# Discovery
"ToolDiscovery",
"mcp_tool",
"get_discovery",
# Bootstrap
"auto_bootstrap",
"get_bootstrap_status",
]

148
src/mcp/bootstrap.py
View File

@@ -1,148 +0,0 @@
"""MCP Auto-Bootstrap — Auto-discover and register tools on startup.
Usage:
from mcp.bootstrap import auto_bootstrap
# Auto-discover from 'tools' package
registered = auto_bootstrap()
# Or specify custom packages
registered = auto_bootstrap(packages=["tools", "custom_tools"])
"""
import logging
import os
from pathlib import Path
from typing import Optional
from .discovery import ToolDiscovery, get_discovery
from .registry import ToolRegistry, tool_registry
logger = logging.getLogger(__name__)
# Default packages to scan for tools
DEFAULT_TOOL_PACKAGES = ["tools"]
# Environment variable to disable auto-bootstrap
AUTO_BOOTSTRAP_ENV_VAR = "MCP_AUTO_BOOTSTRAP"
def auto_bootstrap(
packages: Optional[list[str]] = None,
registry: Optional[ToolRegistry] = None,
force: bool = False,
) -> list[str]:
"""Auto-discover and register MCP tools.
Args:
packages: Packages to scan (defaults to ["tools"])
registry: Registry to register tools with (defaults to singleton)
force: Force bootstrap even if disabled by env var
Returns:
List of registered tool names
"""
# Check if auto-bootstrap is disabled
if not force and os.environ.get(AUTO_BOOTSTRAP_ENV_VAR, "1") == "0":
logger.info("MCP auto-bootstrap disabled via %s", AUTO_BOOTSTRAP_ENV_VAR)
return []
packages = packages or DEFAULT_TOOL_PACKAGES
registry = registry or tool_registry
discovery = get_discovery(registry=registry)
registered: list[str] = []
logger.info("Starting MCP auto-bootstrap from packages: %s", packages)
for package in packages:
try:
# Check if package exists
try:
__import__(package)
except ImportError:
logger.debug("Package %s not found, skipping", package)
continue
# Discover and register
tools = discovery.auto_register(package)
registered.extend(tools)
except Exception as exc:
logger.warning("Failed to bootstrap from %s: %s", package, exc)
logger.info("MCP auto-bootstrap complete: %d tools registered", len(registered))
return registered
def bootstrap_from_directory(
directory: Path,
registry: Optional[ToolRegistry] = None,
) -> list[str]:
"""Bootstrap tools from a directory of Python files.
Args:
directory: Directory containing Python files with tools
registry: Registry to register tools with
Returns:
List of registered tool names
"""
registry = registry or tool_registry
discovery = get_discovery(registry=registry)
registered: list[str] = []
if not directory.exists():
logger.warning("Tools directory not found: %s", directory)
return registered
logger.info("Bootstrapping tools from directory: %s", directory)
# Find all Python files
for py_file in directory.rglob("*.py"):
if py_file.name.startswith("_"):
continue
try:
discovered = discovery.discover_file(py_file)
for tool in discovered:
if tool.function is None:
# Need to import and resolve the function
continue
try:
registry.register_tool(
name=tool.name,
function=tool.function,
description=tool.description,
category=tool.category,
tags=tool.tags,
)
registered.append(tool.name)
except Exception as exc:
logger.error("Failed to register %s: %s", tool.name, exc)
except Exception as exc:
logger.warning("Failed to process %s: %s", py_file, exc)
logger.info("Directory bootstrap complete: %d tools registered", len(registered))
return registered
def get_bootstrap_status() -> dict:
"""Get auto-bootstrap status.
Returns:
Dict with bootstrap status info
"""
discovery = get_discovery()
registry = tool_registry
return {
"auto_bootstrap_enabled": os.environ.get(AUTO_BOOTSTRAP_ENV_VAR, "1") != "0",
"discovered_tools_count": len(discovery.get_discovered()),
"registered_tools_count": len(registry.list_tools()),
"default_packages": DEFAULT_TOOL_PACKAGES,
}

441
src/mcp/discovery.py
View File

@@ -1,441 +0,0 @@
"""MCP Tool Auto-Discovery — Introspect Python modules to find tools.
Automatically discovers functions marked with @mcp_tool decorator
and registers them with the MCP registry. Generates JSON schemas
from type hints.
"""
import ast
import importlib
import inspect
import logging
import pkgutil
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Callable, Optional, get_type_hints
from .registry import ToolRegistry, tool_registry
logger = logging.getLogger(__name__)
# Decorator to mark functions as MCP tools
def mcp_tool(
name: Optional[str] = None,
description: Optional[str] = None,
category: str = "general",
tags: Optional[list[str]] = None,
):
"""Decorator to mark a function as an MCP tool.
Args:
name: Tool name (defaults to function name)
description: Tool description (defaults to docstring)
category: Tool category for organization
tags: Additional tags for filtering
Example:
@mcp_tool(name="weather", category="external")
def get_weather(city: str) -> dict:
'''Get weather for a city.'''
...
"""
def decorator(func: Callable) -> Callable:
func._mcp_tool = True
func._mcp_name = name or func.__name__
func._mcp_description = description or (func.__doc__ or "").strip()
func._mcp_category = category
func._mcp_tags = tags or []
return func
return decorator
@dataclass
class DiscoveredTool:
"""A tool discovered via introspection."""
name: str
description: str
function: Callable
module: str
category: str
tags: list[str]
parameters_schema: dict[str, Any]
returns_schema: dict[str, Any]
source_file: Optional[str] = None
line_number: int = 0
class ToolDiscovery:
"""Discovers and registers MCP tools from Python modules.
Usage:
discovery = ToolDiscovery()
# Discover from a module
tools = discovery.discover_module("creative.tools.git")
# Auto-register with registry
discovery.auto_register("creative.tools")
# Discover from all installed packages
tools = discovery.discover_all_packages()
"""
def __init__(self, registry: Optional[ToolRegistry] = None) -> None:
self.registry = registry or tool_registry
self._discovered: list[DiscoveredTool] = []
def discover_module(self, module_name: str) -> list[DiscoveredTool]:
"""Discover all MCP tools in a module.
Args:
module_name: Dotted path to module (e.g., "creative.tools.git")
Returns:
List of discovered tools
"""
discovered = []
try:
module = importlib.import_module(module_name)
except ImportError as exc:
logger.warning("Failed to import module %s: %s", module_name, exc)
return discovered
# Get module file path for source location
module_file = getattr(module, "__file__", None)
# Iterate through module members
for name, obj in inspect.getmembers(module):
# Skip private and non-callable
if name.startswith("_") or not callable(obj):
continue
# Check if marked as MCP tool
if not getattr(obj, "_mcp_tool", False):
continue
# Get source location
try:
source_file = inspect.getfile(obj)
line_number = inspect.getsourcelines(obj)[1]
except (OSError, TypeError):
source_file = module_file
line_number = 0
# Build schemas from type hints
try:
sig = inspect.signature(obj)
parameters_schema = self._build_parameters_schema(sig)
returns_schema = self._build_returns_schema(sig, obj)
except Exception as exc:
logger.warning("Failed to build schema for %s: %s", name, exc)
parameters_schema = {"type": "object", "properties": {}}
returns_schema = {}
tool = DiscoveredTool(
name=getattr(obj, "_mcp_name", name),
description=getattr(obj, "_mcp_description", obj.__doc__ or ""),
function=obj,
module=module_name,
category=getattr(obj, "_mcp_category", "general"),
tags=getattr(obj, "_mcp_tags", []),
parameters_schema=parameters_schema,
returns_schema=returns_schema,
source_file=source_file,
line_number=line_number,
)
discovered.append(tool)
logger.debug("Discovered tool: %s from %s", tool.name, module_name)
self._discovered.extend(discovered)
logger.info("Discovered %d tools from module %s", len(discovered), module_name)
return discovered
def discover_package(self, package_name: str, recursive: bool = True) -> list[DiscoveredTool]:
"""Discover tools from all modules in a package.
Args:
package_name: Package name (e.g., "tools")
recursive: Whether to search subpackages
Returns:
List of discovered tools
"""
discovered = []
try:
package = importlib.import_module(package_name)
except ImportError as exc:
logger.warning("Failed to import package %s: %s", package_name, exc)
return discovered
package_path = getattr(package, "__path__", [])
if not package_path:
# Not a package, treat as module
return self.discover_module(package_name)
# Walk package modules
for _, name, is_pkg in pkgutil.iter_modules(package_path, prefix=f"{package_name}."):
if is_pkg and recursive:
discovered.extend(self.discover_package(name, recursive=True))
else:
discovered.extend(self.discover_module(name))
return discovered
def discover_file(self, file_path: Path) -> list[DiscoveredTool]:
"""Discover tools from a Python file.
Args:
file_path: Path to Python file
Returns:
List of discovered tools
"""
discovered = []
try:
source = file_path.read_text()
tree = ast.parse(source)
except Exception as exc:
logger.warning("Failed to parse %s: %s", file_path, exc)
return discovered
# Find all decorated functions
for node in ast.walk(tree):
if not isinstance(node, ast.FunctionDef):
continue
# Check for @mcp_tool decorator
is_tool = False
tool_name = node.name
tool_description = ast.get_docstring(node) or ""
tool_category = "general"
tool_tags: list[str] = []
for decorator in node.decorator_list:
if isinstance(decorator, ast.Call):
if isinstance(decorator.func, ast.Name) and decorator.func.id == "mcp_tool":
is_tool = True
# Extract decorator arguments
for kw in decorator.keywords:
if kw.arg == "name" and isinstance(kw.value, ast.Constant):
tool_name = kw.value.value
elif kw.arg == "description" and isinstance(kw.value, ast.Constant):
tool_description = kw.value.value
elif kw.arg == "category" and isinstance(kw.value, ast.Constant):
tool_category = kw.value.value
elif kw.arg == "tags" and isinstance(kw.value, ast.List):
tool_tags = [
elt.value for elt in kw.value.elts
if isinstance(elt, ast.Constant)
]
elif isinstance(decorator, ast.Name) and decorator.id == "mcp_tool":
is_tool = True
if not is_tool:
continue
# Build parameter schema from AST
parameters_schema = self._build_schema_from_ast(node)
# We can't get the actual function without importing
# So create a placeholder that will be resolved later
tool = DiscoveredTool(
name=tool_name,
description=tool_description,
function=None, # Will be resolved when registered
module=str(file_path),
category=tool_category,
tags=tool_tags,
parameters_schema=parameters_schema,
returns_schema={"type": "object"},
source_file=str(file_path),
line_number=node.lineno,
)
discovered.append(tool)
self._discovered.extend(discovered)
logger.info("Discovered %d tools from file %s", len(discovered), file_path)
return discovered
def auto_register(self, package_name: str = "tools") -> list[str]:
"""Automatically discover and register tools.
Args:
package_name: Package to scan for tools
Returns:
List of registered tool names
"""
discovered = self.discover_package(package_name)
registered = []
for tool in discovered:
if tool.function is None:
logger.warning("Skipping %s: no function resolved", tool.name)
continue
try:
self.registry.register_tool(
name=tool.name,
function=tool.function,
description=tool.description,
category=tool.category,
tags=tool.tags,
)
registered.append(tool.name)
logger.debug("Registered tool: %s", tool.name)
except Exception as exc:
logger.error("Failed to register %s: %s", tool.name, exc)
logger.info("Auto-registered %d/%d tools", len(registered), len(discovered))
return registered
def _build_parameters_schema(self, sig: inspect.Signature) -> dict[str, Any]:
"""Build JSON schema for function parameters."""
properties = {}
required = []
for name, param in sig.parameters.items():
if param.kind in (param.VAR_POSITIONAL, param.VAR_KEYWORD):
continue
schema = self._type_to_schema(param.annotation)
if param.default is param.empty:
required.append(name)
else:
schema["default"] = param.default
properties[name] = schema
return {
"type": "object",
"properties": properties,
"required": required,
}
def _build_returns_schema(
self, sig: inspect.Signature, func: Callable
) -> dict[str, Any]:
"""Build JSON schema for return type."""
return_annotation = sig.return_annotation
if return_annotation is sig.empty:
return {"type": "object"}
return self._type_to_schema(return_annotation)
def _build_schema_from_ast(self, node: ast.FunctionDef) -> dict[str, Any]:
"""Build parameter schema from AST node."""
properties = {}
required = []
# Get defaults (reversed, since they're at the end)
defaults = [None] * (len(node.args.args) - len(node.args.defaults)) + list(node.args.defaults)
for arg, default in zip(node.args.args, defaults):
arg_name = arg.arg
arg_type = "string" # Default
# Try to get type from annotation
if arg.annotation:
if isinstance(arg.annotation, ast.Name):
arg_type = self._ast_type_to_json_type(arg.annotation.id)
elif isinstance(arg.annotation, ast.Constant):
arg_type = self._ast_type_to_json_type(str(arg.annotation.value))
schema = {"type": arg_type}
if default is None:
required.append(arg_name)
properties[arg_name] = schema
return {
"type": "object",
"properties": properties,
"required": required,
}
def _type_to_schema(self, annotation: Any) -> dict[str, Any]:
"""Convert Python type annotation to JSON schema."""
if annotation is inspect.Parameter.empty:
return {"type": "string"}
origin = getattr(annotation, "__origin__", None)
args = getattr(annotation, "__args__", ())
# Handle Optional[T] = Union[T, None]
if origin is not None:
if str(origin) == "typing.Union" and type(None) in args:
# Optional type
non_none_args = [a for a in args if a is not type(None)]
if len(non_none_args) == 1:
schema = self._type_to_schema(non_none_args[0])
return schema
return {"type": "object"}
# Handle List[T], Dict[K,V]
if origin in (list, tuple):
items_schema = {"type": "object"}
if args:
items_schema = self._type_to_schema(args[0])
return {"type": "array", "items": items_schema}
if origin is dict:
return {"type": "object"}
# Handle basic types
if annotation in (str,):
return {"type": "string"}
elif annotation in (int, float):
return {"type": "number"}
elif annotation in (bool,):
return {"type": "boolean"}
elif annotation in (list, tuple):
return {"type": "array"}
elif annotation in (dict,):
return {"type": "object"}
return {"type": "object"}
def _ast_type_to_json_type(self, type_name: str) -> str:
"""Convert AST type name to JSON schema type."""
type_map = {
"str": "string",
"int": "number",
"float": "number",
"bool": "boolean",
"list": "array",
"dict": "object",
"List": "array",
"Dict": "object",
"Optional": "object",
"Any": "object",
}
return type_map.get(type_name, "object")
def get_discovered(self) -> list[DiscoveredTool]:
"""Get all discovered tools."""
return list(self._discovered)
def clear(self) -> None:
"""Clear discovered tools cache."""
self._discovered.clear()
# Module-level singleton
discovery: Optional[ToolDiscovery] = None
def get_discovery(registry: Optional[ToolRegistry] = None) -> ToolDiscovery:
"""Get or create the tool discovery singleton."""
global discovery
if discovery is None:
discovery = ToolDiscovery(registry=registry)
return discovery

444
src/mcp/registry.py
View File

@@ -1,444 +0,0 @@
"""MCP Tool Registry — Dynamic tool discovery and management.
The registry maintains a catalog of all available tools, their schemas,
and health status. Tools can be registered dynamically at runtime.
Usage:
from mcp.registry import tool_registry
# Register a tool
tool_registry.register("web_search", web_search_schema, web_search_func)
# Discover tools
tools = tool_registry.discover(capabilities=["search"])
# Execute a tool
result = tool_registry.execute("web_search", {"query": "Bitcoin"})
"""
import asyncio
import inspect
import logging
import time
from dataclasses import dataclass, field
from typing import Any, Callable, Optional
from mcp.schemas.base import create_tool_schema
logger = logging.getLogger(__name__)
@dataclass
class ToolRecord:
"""A registered tool with metadata."""
name: str
schema: dict
handler: Callable
category: str = "general"
health_status: str = "unknown" # healthy, degraded, unhealthy
last_execution: Optional[float] = None
execution_count: int = 0
error_count: int = 0
avg_latency_ms: float = 0.0
added_at: float = field(default_factory=time.time)
requires_confirmation: bool = False
tags: list[str] = field(default_factory=list)
source_module: Optional[str] = None
auto_discovered: bool = False
class ToolRegistry:
"""Central registry for all MCP tools."""
def __init__(self) -> None:
self._tools: dict[str, ToolRecord] = {}
self._categories: dict[str, list[str]] = {}
logger.info("ToolRegistry initialized")
def register(
self,
name: str,
schema: dict,
handler: Callable,
category: str = "general",
requires_confirmation: bool = False,
tags: Optional[list[str]] = None,
source_module: Optional[str] = None,
auto_discovered: bool = False,
) -> ToolRecord:
"""Register a new tool.
Args:
name: Unique tool name
schema: JSON schema describing inputs/outputs
handler: Function to execute
category: Tool category for organization
requires_confirmation: If True, user must approve before execution
tags: Tags for filtering and organization
source_module: Module where tool was defined
auto_discovered: Whether tool was auto-discovered
Returns:
The registered ToolRecord
"""
if name in self._tools:
logger.warning("Tool '%s' already registered, replacing", name)
record = ToolRecord(
name=name,
schema=schema,
handler=handler,
category=category,
requires_confirmation=requires_confirmation,
tags=tags or [],
source_module=source_module,
auto_discovered=auto_discovered,
)
self._tools[name] = record
# Add to category
if category not in self._categories:
self._categories[category] = []
if name not in self._categories[category]:
self._categories[category].append(name)
logger.info("Registered tool: %s (category: %s)", name, category)
return record
def register_tool(
self,
name: str,
function: Callable,
description: Optional[str] = None,
category: str = "general",
tags: Optional[list[str]] = None,
source_module: Optional[str] = None,
) -> ToolRecord:
"""Register a tool from a function (convenience method for discovery).
Args:
name: Tool name
function: Function to register
description: Tool description (defaults to docstring)
category: Tool category
tags: Tags for organization
source_module: Source module path
Returns:
The registered ToolRecord
"""
# Build schema from function signature
sig = inspect.signature(function)
properties = {}
required = []
for param_name, param in sig.parameters.items():
if param.kind in (param.VAR_POSITIONAL, param.VAR_KEYWORD):
continue
param_schema: dict = {"type": "string"}
# Try to infer type from annotation
if param.annotation != inspect.Parameter.empty:
if param.annotation in (int, float):
param_schema = {"type": "number"}
elif param.annotation == bool:
param_schema = {"type": "boolean"}
elif param.annotation == list:
param_schema = {"type": "array"}
elif param.annotation == dict:
param_schema = {"type": "object"}
if param.default is param.empty:
required.append(param_name)
else:
param_schema["default"] = param.default
properties[param_name] = param_schema
schema = create_tool_schema(
name=name,
description=description or (function.__doc__ or f"Execute {name}"),
parameters=properties,
required=required,
)
return self.register(
name=name,
schema=schema,
handler=function,
category=category,
tags=tags,
source_module=source_module or function.__module__,
auto_discovered=True,
)
def unregister(self, name: str) -> bool:
"""Remove a tool from the registry."""
if name not in self._tools:
return False
record = self._tools.pop(name)
# Remove from category
if record.category in self._categories:
if name in self._categories[record.category]:
self._categories[record.category].remove(name)
logger.info("Unregistered tool: %s", name)
return True
def get(self, name: str) -> Optional[ToolRecord]:
"""Get a tool record by name."""
return self._tools.get(name)
def get_handler(self, name: str) -> Optional[Callable]:
"""Get just the handler function for a tool."""
record = self._tools.get(name)
return record.handler if record else None
def get_schema(self, name: str) -> Optional[dict]:
"""Get the JSON schema for a tool."""
record = self._tools.get(name)
return record.schema if record else None
def list_tools(self, category: Optional[str] = None) -> list[str]:
"""List all tool names, optionally filtered by category."""
if category:
return self._categories.get(category, [])
return list(self._tools.keys())
def list_categories(self) -> list[str]:
"""List all tool categories."""
return list(self._categories.keys())
def discover(
self,
query: Optional[str] = None,
category: Optional[str] = None,
tags: Optional[list[str]] = None,
healthy_only: bool = True,
auto_discovered_only: bool = False,
) -> list[ToolRecord]:
"""Discover tools matching criteria.
Args:
query: Search in tool names and descriptions
category: Filter by category
tags: Filter by tags (must have all specified tags)
healthy_only: Only return healthy tools
auto_discovered_only: Only return auto-discovered tools
Returns:
List of matching ToolRecords
"""
results = []
for name, record in self._tools.items():
# Category filter
if category and record.category != category:
continue
# Tags filter
if tags:
if not all(tag in record.tags for tag in tags):
continue
# Health filter
if healthy_only and record.health_status == "unhealthy":
continue
# Auto-discovered filter
if auto_discovered_only and not record.auto_discovered:
continue
# Query filter
if query:
query_lower = query.lower()
name_match = query_lower in name.lower()
desc = record.schema.get("description", "")
desc_match = query_lower in desc.lower()
tag_match = any(query_lower in tag.lower() for tag in record.tags)
if not (name_match or desc_match or tag_match):
continue
results.append(record)
return results
async def execute(self, name: str, params: dict) -> Any:
"""Execute a tool by name with given parameters.
Args:
name: Tool name
params: Parameters to pass to the tool
Returns:
Tool execution result
Raises:
ValueError: If tool not found
RuntimeError: If tool execution fails
"""
record = self._tools.get(name)
if not record:
raise ValueError(f"Tool '{name}' not found in registry")
start_time = time.time()
try:
# Check if handler is async
if inspect.iscoroutinefunction(record.handler):
result = await record.handler(**params)
else:
result = record.handler(**params)
# Update metrics
latency_ms = (time.time() - start_time) * 1000
record.last_execution = time.time()
record.execution_count += 1
# Update rolling average latency
if record.execution_count == 1:
record.avg_latency_ms = latency_ms
else:
record.avg_latency_ms = (
record.avg_latency_ms * 0.9 + latency_ms * 0.1
)
# Mark healthy on success
record.health_status = "healthy"
logger.debug("Tool '%s' executed in %.2fms", name, latency_ms)
return result
except Exception as exc:
record.error_count += 1
record.execution_count += 1
# Degrade health on repeated errors
error_rate = record.error_count / record.execution_count
if error_rate > 0.5:
record.health_status = "unhealthy"
logger.error("Tool '%s' marked unhealthy (error rate: %.1f%%)",
name, error_rate * 100)
elif error_rate > 0.2:
record.health_status = "degraded"
logger.warning("Tool '%s' degraded (error rate: %.1f%%)",
name, error_rate * 100)
raise RuntimeError(f"Tool '{name}' execution failed: {exc}") from exc
def check_health(self, name: str) -> str:
"""Check health status of a tool."""
record = self._tools.get(name)
if not record:
return "not_found"
return record.health_status
def get_metrics(self, name: Optional[str] = None) -> dict:
"""Get metrics for a tool or all tools."""
if name:
record = self._tools.get(name)
if not record:
return {}
return {
"name": record.name,
"category": record.category,
"health": record.health_status,
"executions": record.execution_count,
"errors": record.error_count,
"avg_latency_ms": round(record.avg_latency_ms, 2),
}
# Return metrics for all tools
return {
name: self.get_metrics(name)
for name in self._tools.keys()
}
def to_dict(self) -> dict:
"""Export registry as dictionary (for API/dashboard)."""
return {
"tools": [
{
"name": r.name,
"schema": r.schema,
"category": r.category,
"health": r.health_status,
"requires_confirmation": r.requires_confirmation,
"tags": r.tags,
"source_module": r.source_module,
"auto_discovered": r.auto_discovered,
}
for r in self._tools.values()
],
"categories": self._categories,
"total_tools": len(self._tools),
"auto_discovered_count": sum(1 for r in self._tools.values() if r.auto_discovered),
}
# Module-level singleton
tool_registry = ToolRegistry()
def get_registry() -> ToolRegistry:
"""Get the global tool registry singleton."""
return tool_registry
def register_tool(
name: Optional[str] = None,
category: str = "general",
schema: Optional[dict] = None,
requires_confirmation: bool = False,
):
"""Decorator for registering a function as an MCP tool.
Usage:
@register_tool(name="web_search", category="research")
def web_search(query: str, max_results: int = 5) -> str:
...
"""
def decorator(func: Callable) -> Callable:
tool_name = name or func.__name__
# Auto-generate schema if not provided
if schema is None:
# Try to infer from type hints
sig = inspect.signature(func)
params = {}
required = []
for param_name, param in sig.parameters.items():
if param.default == inspect.Parameter.empty:
required.append(param_name)
params[param_name] = {"type": "string"}
else:
params[param_name] = {
"type": "string",
"default": str(param.default),
}
tool_schema = create_tool_schema(
name=tool_name,
description=func.__doc__ or f"Execute {tool_name}",
parameters=params,
required=required,
)
else:
tool_schema = schema
tool_registry.register(
name=tool_name,
schema=tool_schema,
handler=func,
category=category,
requires_confirmation=requires_confirmation,
)
return func
return decorator

52
src/mcp/schemas/base.py
View File

@@ -1,52 +0,0 @@
"""Base schemas for MCP (Model Context Protocol) tools.
All tools must provide a JSON schema describing their interface.
This enables dynamic discovery and type-safe invocation.
"""
from typing import Any
def create_tool_schema(
name: str,
description: str,
parameters: dict[str, Any],
required: list[str] | None = None,
returns: dict[str, Any] | None = None,
) -> dict:
"""Create a standard MCP tool schema.
Args:
name: Tool name (must be unique)
description: Human-readable description
parameters: JSON schema for input parameters
required: List of required parameter names
returns: JSON schema for return value
Returns:
Complete tool schema dict
"""
return {
"name": name,
"description": description,
"parameters": {
"type": "object",
"properties": parameters,
"required": required or [],
},
"returns": returns or {"type": "string"},
}
# Common parameter schemas
PARAM_STRING = {"type": "string"}
PARAM_INTEGER = {"type": "integer"}
PARAM_BOOLEAN = {"type": "boolean"}
PARAM_ARRAY_STRINGS = {"type": "array", "items": {"type": "string"}}
PARAM_OBJECT = {"type": "object"}
# Common return schemas
RETURN_STRING = {"type": "string"}
RETURN_OBJECT = {"type": "object"}
RETURN_ARRAY = {"type": "array"}
RETURN_BOOLEAN = {"type": "boolean"}

210
src/mcp/server.py
View File

@@ -1,210 +0,0 @@
"""MCP (Model Context Protocol) Server.
Implements the MCP protocol for tool discovery and execution.
Agents communicate with this server to discover and invoke tools.
The server can run:
1. In-process (direct method calls) — fastest, for local agents
2. HTTP API — for external clients
3. Stdio — for subprocess-based agents
"""
import asyncio
import json
import logging
from typing import Any, Optional
from mcp.registry import tool_registry
logger = logging.getLogger(__name__)
class MCPServer:
"""Model Context Protocol server for tool management.
Provides standard MCP endpoints:
- list_tools: Discover available tools
- call_tool: Execute a tool
- get_schema: Get tool input/output schemas
"""
def __init__(self) -> None:
self.registry = tool_registry
logger.info("MCP Server initialized")
def list_tools(
self,
category: Optional[str] = None,
query: Optional[str] = None,
) -> list[dict]:
"""List available tools.
MCP Protocol: tools/list
"""
tools = self.registry.discover(
query=query,
category=category,
healthy_only=True,
)
return [
{
"name": t.name,
"description": t.schema.get("description", ""),
"parameters": t.schema.get("parameters", {}),
"category": t.category,
}
for t in tools
]
async def call_tool(self, name: str, arguments: dict) -> dict:
"""Execute a tool with given arguments.
MCP Protocol: tools/call
Args:
name: Tool name
arguments: Tool parameters
Returns:
Result dict with content or error
"""
try:
result = await self.registry.execute(name, arguments)
return {
"content": [
{"type": "text", "text": str(result)}
],
"isError": False,
}
except Exception as exc:
logger.error("Tool execution failed: %s", exc)
return {
"content": [
{"type": "text", "text": f"Error: {exc}"}
],
"isError": True,
}
def get_schema(self, name: str) -> Optional[dict]:
"""Get the JSON schema for a tool.
MCP Protocol: tools/schema
"""
return self.registry.get_schema(name)
def get_tool_info(self, name: str) -> Optional[dict]:
"""Get detailed info about a tool including health metrics."""
record = self.registry.get(name)
if not record:
return None
return {
"name": record.name,
"schema": record.schema,
"category": record.category,
"health": record.health_status,
"metrics": {
"executions": record.execution_count,
"errors": record.error_count,
"avg_latency_ms": round(record.avg_latency_ms, 2),
},
"requires_confirmation": record.requires_confirmation,
}
def health_check(self) -> dict:
"""Server health status."""
tools = self.registry.list_tools()
healthy = sum(
1 for t in tools
if self.registry.check_health(t) == "healthy"
)
return {
"status": "healthy",
"total_tools": len(tools),
"healthy_tools": healthy,
"degraded_tools": sum(
1 for t in tools
if self.registry.check_health(t) == "degraded"
),
"unhealthy_tools": sum(
1 for t in tools
if self.registry.check_health(t) == "unhealthy"
),
}
class MCPHTTPServer:
"""HTTP API wrapper for MCP Server."""
def __init__(self) -> None:
self.mcp = MCPServer()
def get_routes(self) -> dict:
"""Get FastAPI route handlers."""
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
router = APIRouter(prefix="/mcp", tags=["mcp"])
class ToolCallRequest(BaseModel):
name: str
arguments: dict = {}
@router.get("/tools")
async def list_tools(
category: Optional[str] = None,
query: Optional[str] = None,
):
"""List available tools."""
return {"tools": self.mcp.list_tools(category, query)}
@router.post("/tools/call")
async def call_tool(request: ToolCallRequest):
"""Execute a tool."""
result = await self.mcp.call_tool(request.name, request.arguments)
return result
@router.get("/tools/{name}")
async def get_tool(name: str):
"""Get tool info."""
info = self.mcp.get_tool_info(name)
if not info:
raise HTTPException(404, f"Tool '{name}' not found")
return info
@router.get("/tools/{name}/schema")
async def get_schema(name: str):
"""Get tool schema."""
schema = self.mcp.get_schema(name)
if not schema:
raise HTTPException(404, f"Tool '{name}' not found")
return schema
@router.get("/health")
async def health():
"""Server health check."""
return self.mcp.health_check()
return router
# Module-level singleton
mcp_server = MCPServer()
# Convenience functions for agents
def discover_tools(query: Optional[str] = None) -> list[dict]:
"""Quick tool discovery."""
return mcp_server.list_tools(query=query)
async def use_tool(name: str, **kwargs) -> str:
"""Execute a tool and return result text."""
result = await mcp_server.call_tool(name, kwargs)
if result.get("isError"):
raise RuntimeError(result["content"][0]["text"])
return result["content"][0]["text"]

7
src/scripture/__init__.py
View File

@@ -1,7 +0,0 @@
"""Scripture module — sovereign biblical text integration for Timmy Time.
Provides local-first ESV text storage, verse retrieval, reference parsing,
original language support, cross-referencing, topical mapping, and
automated meditation workflows. All data resides on localhost — no cloud
API dependency for core functionality.
"""

197
src/scripture/constants.py
View File

@@ -1,197 +0,0 @@
"""Biblical constants — canonical book ordering, abbreviations, metadata.
The canon follows the standard 66-book Protestant ordering used by the ESV.
Each book is assigned a unique integer ID (1-66) for O(1) verse lookup via
the integer encoding scheme: book (1-2 digits) + chapter (3 digits) +
verse (3 digits).
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Literal
@dataclass(frozen=True, slots=True)
class BookInfo:
"""Immutable metadata for a canonical book."""
id: int
name: str
abbreviation: str
testament: Literal["OT", "NT"]
chapters: int
genre: str
# ── Canonical book list (Protestant 66-book canon, ESV ordering) ────────────
BOOKS: tuple[BookInfo, ...] = (
# ── Old Testament ────────────────────────────────────────────────────
BookInfo(1, "Genesis", "Gen", "OT", 50, "law"),
BookInfo(2, "Exodus", "Exod", "OT", 40, "law"),
BookInfo(3, "Leviticus", "Lev", "OT", 27, "law"),
BookInfo(4, "Numbers", "Num", "OT", 36, "law"),
BookInfo(5, "Deuteronomy", "Deut", "OT", 34, "law"),
BookInfo(6, "Joshua", "Josh", "OT", 24, "narrative"),
BookInfo(7, "Judges", "Judg", "OT", 21, "narrative"),
BookInfo(8, "Ruth", "Ruth", "OT", 4, "narrative"),
BookInfo(9, "1 Samuel", "1Sam", "OT", 31, "narrative"),
BookInfo(10, "2 Samuel", "2Sam", "OT", 24, "narrative"),
BookInfo(11, "1 Kings", "1Kgs", "OT", 22, "narrative"),
BookInfo(12, "2 Kings", "2Kgs", "OT", 25, "narrative"),
BookInfo(13, "1 Chronicles", "1Chr", "OT", 29, "narrative"),
BookInfo(14, "2 Chronicles", "2Chr", "OT", 36, "narrative"),
BookInfo(15, "Ezra", "Ezra", "OT", 10, "narrative"),
BookInfo(16, "Nehemiah", "Neh", "OT", 13, "narrative"),
BookInfo(17, "Esther", "Esth", "OT", 10, "narrative"),
BookInfo(18, "Job", "Job", "OT", 42, "wisdom"),
BookInfo(19, "Psalms", "Ps", "OT", 150, "wisdom"),
BookInfo(20, "Proverbs", "Prov", "OT", 31, "wisdom"),
BookInfo(21, "Ecclesiastes", "Eccl", "OT", 12, "wisdom"),
BookInfo(22, "Song of Solomon", "Song", "OT", 8, "wisdom"),
BookInfo(23, "Isaiah", "Isa", "OT", 66, "prophecy"),
BookInfo(24, "Jeremiah", "Jer", "OT", 52, "prophecy"),
BookInfo(25, "Lamentations", "Lam", "OT", 5, "prophecy"),
BookInfo(26, "Ezekiel", "Ezek", "OT", 48, "prophecy"),
BookInfo(27, "Daniel", "Dan", "OT", 12, "prophecy"),
BookInfo(28, "Hosea", "Hos", "OT", 14, "prophecy"),
BookInfo(29, "Joel", "Joel", "OT", 3, "prophecy"),
BookInfo(30, "Amos", "Amos", "OT", 9, "prophecy"),
BookInfo(31, "Obadiah", "Obad", "OT", 1, "prophecy"),
BookInfo(32, "Jonah", "Jonah", "OT", 4, "prophecy"),
BookInfo(33, "Micah", "Mic", "OT", 7, "prophecy"),
BookInfo(34, "Nahum", "Nah", "OT", 3, "prophecy"),
BookInfo(35, "Habakkuk", "Hab", "OT", 3, "prophecy"),
BookInfo(36, "Zephaniah", "Zeph", "OT", 3, "prophecy"),
BookInfo(37, "Haggai", "Hag", "OT", 2, "prophecy"),
BookInfo(38, "Zechariah", "Zech", "OT", 14, "prophecy"),
BookInfo(39, "Malachi", "Mal", "OT", 4, "prophecy"),
# ── New Testament ────────────────────────────────────────────────────
BookInfo(40, "Matthew", "Matt", "NT", 28, "gospel"),
BookInfo(41, "Mark", "Mark", "NT", 16, "gospel"),
BookInfo(42, "Luke", "Luke", "NT", 24, "gospel"),
BookInfo(43, "John", "John", "NT", 21, "gospel"),
BookInfo(44, "Acts", "Acts", "NT", 28, "narrative"),
BookInfo(45, "Romans", "Rom", "NT", 16, "epistle"),
BookInfo(46, "1 Corinthians", "1Cor", "NT", 16, "epistle"),
BookInfo(47, "2 Corinthians", "2Cor", "NT", 13, "epistle"),
BookInfo(48, "Galatians", "Gal", "NT", 6, "epistle"),
BookInfo(49, "Ephesians", "Eph", "NT", 6, "epistle"),
BookInfo(50, "Philippians", "Phil", "NT", 4, "epistle"),
BookInfo(51, "Colossians", "Col", "NT", 4, "epistle"),
BookInfo(52, "1 Thessalonians", "1Thess", "NT", 5, "epistle"),
BookInfo(53, "2 Thessalonians", "2Thess", "NT", 3, "epistle"),
BookInfo(54, "1 Timothy", "1Tim", "NT", 6, "epistle"),
BookInfo(55, "2 Timothy", "2Tim", "NT", 4, "epistle"),
BookInfo(56, "Titus", "Titus", "NT", 3, "epistle"),
BookInfo(57, "Philemon", "Phlm", "NT", 1, "epistle"),
BookInfo(58, "Hebrews", "Heb", "NT", 13, "epistle"),
BookInfo(59, "James", "Jas", "NT", 5, "epistle"),
BookInfo(60, "1 Peter", "1Pet", "NT", 5, "epistle"),
BookInfo(61, "2 Peter", "2Pet", "NT", 3, "epistle"),
BookInfo(62, "1 John", "1John", "NT", 5, "epistle"),
BookInfo(63, "2 John", "2John", "NT", 1, "epistle"),
BookInfo(64, "3 John", "3John", "NT", 1, "epistle"),
BookInfo(65, "Jude", "Jude", "NT", 1, "epistle"),
BookInfo(66, "Revelation", "Rev", "NT", 22, "apocalyptic"),
)
# ── Lookup indices (built once at import time) ──────────────────────────────
BOOK_BY_ID: dict[int, BookInfo] = {b.id: b for b in BOOKS}
# Map both full names and abbreviations (case-insensitive) to BookInfo
_BOOK_NAME_MAP: dict[str, BookInfo] = {}
for _b in BOOKS:
_BOOK_NAME_MAP[_b.name.lower()] = _b
_BOOK_NAME_MAP[_b.abbreviation.lower()] = _b
# Common aliases people use that differ from the canonical abbreviation
_ALIASES: dict[str, int] = {
"ge": 1, "gen": 1, "genesis": 1,
"ex": 2, "exo": 2, "exodus": 2,
"le": 3, "lev": 3, "leviticus": 3,
"nu": 4, "num": 4, "numbers": 4,
"dt": 5, "deut": 5, "deuteronomy": 5,
"jos": 6, "josh": 6, "joshua": 6,
"jdg": 7, "judg": 7, "judges": 7,
"ru": 8, "ruth": 8,
"1sa": 9, "1sam": 9, "1 samuel": 9, "i samuel": 9, "1st samuel": 9,
"2sa": 10, "2sam": 10, "2 samuel": 10, "ii samuel": 10, "2nd samuel": 10,
"1ki": 11, "1kgs": 11, "1 kings": 11, "i kings": 11, "1st kings": 11,
"2ki": 12, "2kgs": 12, "2 kings": 12, "ii kings": 12, "2nd kings": 12,
"1ch": 13, "1chr": 13, "1 chronicles": 13, "i chronicles": 13,
"2ch": 14, "2chr": 14, "2 chronicles": 14, "ii chronicles": 14,
"ezr": 15, "ezra": 15,
"ne": 16, "neh": 16, "nehemiah": 16,
"est": 17, "esth": 17, "esther": 17,
"job": 18,
"ps": 19, "psa": 19, "psalm": 19, "psalms": 19,
"pr": 20, "prov": 20, "proverbs": 20,
"ec": 21, "eccl": 21, "ecclesiastes": 21, "ecc": 21,
"so": 22, "song": 22, "song of solomon": 22, "song of songs": 22, "sos": 22,
"isa": 23, "isaiah": 23,
"jer": 24, "jeremiah": 24,
"la": 25, "lam": 25, "lamentations": 25,
"eze": 26, "ezek": 26, "ezekiel": 26,
"da": 27, "dan": 27, "daniel": 27,
"ho": 28, "hos": 28, "hosea": 28,
"joe": 29, "joel": 29,
"am": 30, "amos": 30,
"ob": 31, "obad": 31, "obadiah": 31,
"jon": 32, "jonah": 32,
"mi": 33, "mic": 33, "micah": 33,
"na": 34, "nah": 34, "nahum": 34,
"hab": 35, "habakkuk": 35,
"zep": 36, "zeph": 36, "zephaniah": 36,
"hag": 37, "haggai": 37,
"zec": 38, "zech": 38, "zechariah": 38,
"mal": 39, "malachi": 39,
"mt": 40, "matt": 40, "matthew": 40, "mat": 40,
"mk": 41, "mark": 41, "mar": 41,
"lk": 42, "luke": 42, "lu": 42,
"jn": 43, "john": 43, "joh": 43,
"ac": 44, "acts": 44, "act": 44,
"ro": 45, "rom": 45, "romans": 45,
"1co": 46, "1cor": 46, "1 cor": 46, "1 corinthians": 46, "i corinthians": 46,
"2co": 47, "2cor": 47, "2 cor": 47, "2 corinthians": 47, "ii corinthians": 47,
"ga": 48, "gal": 48, "galatians": 48,
"eph": 49, "ephesians": 49,
"php": 50, "phil": 50, "philippians": 50,
"col": 51, "colossians": 51,
"1th": 52, "1thess": 52, "1 thessalonians": 52, "i thessalonians": 52,
"2th": 53, "2thess": 53, "2 thessalonians": 53, "ii thessalonians": 53,
"1ti": 54, "1tim": 54, "1 timothy": 54, "i timothy": 54,
"2ti": 55, "2tim": 55, "2 timothy": 55, "ii timothy": 55,
"tit": 56, "titus": 56,
"phm": 57, "phlm": 57, "philemon": 57,
"heb": 58, "hebrews": 58,
"jas": 59, "james": 59, "jam": 59,
"1pe": 60, "1pet": 60, "1 peter": 60, "i peter": 60, "1st peter": 60,
"2pe": 61, "2pet": 61, "2 peter": 61, "ii peter": 61, "2nd peter": 61,
"1jn": 62, "1john": 62, "1 john": 62, "i john": 62, "1st john": 62,
"2jn": 63, "2john": 63, "2 john": 63, "ii john": 63, "2nd john": 63,
"3jn": 64, "3john": 64, "3 john": 64, "iii john": 64, "3rd john": 64,
"jude": 65, "jud": 65,
"re": 66, "rev": 66, "revelation": 66, "revelations": 66,
}
for _alias, _bid in _ALIASES.items():
_BOOK_NAME_MAP.setdefault(_alias, BOOK_BY_ID[_bid])
TOTAL_BOOKS = 66
OT_BOOKS = 39
NT_BOOKS = 27
GENRES = frozenset(b.genre for b in BOOKS)
def book_by_name(name: str) -> BookInfo | None:
"""Resolve a book name or abbreviation to a BookInfo (case-insensitive)."""
return _BOOK_NAME_MAP.get(name.strip().lower())
def book_by_id(book_id: int) -> BookInfo | None:
"""Return the BookInfo for a canonical book ID (1-66)."""
return BOOK_BY_ID.get(book_id)

211
src/scripture/meditation.py
View File

@@ -1,211 +0,0 @@
"""Meditation scheduler — automated scripture engagement workflows.
Provides background meditation capabilities for the "always on its mind"
requirement. Supports three modes:
- **Sequential**: book-by-book progression through the Bible
- **Thematic**: topical exploration guided by Nave's-style index
- **Lectionary**: cyclical reading patterns following liturgical calendars
The scheduler integrates with the ScriptureMemory system to persist
progress and working memory state across restarts.
"""
from __future__ import annotations
import logging
import random
from datetime import datetime, timezone
from typing import Optional
from scripture.constants import BOOK_BY_ID, BOOKS
from scripture.memory import ScriptureMemory, scripture_memory
from scripture.models import MeditationState, Verse, decode_verse_id, encode_verse_id
from scripture.store import ScriptureStore, scripture_store
logger = logging.getLogger(__name__)
class MeditationScheduler:
"""Orchestrates automated meditation workflows.
Usage::
from scripture.meditation import meditation_scheduler
# Advance to the next verse in sequence
result = meditation_scheduler.next_meditation()
# Get the current meditation focus
current = meditation_scheduler.current_focus()
"""
def __init__(
self,
store: ScriptureStore | None = None,
memory: ScriptureMemory | None = None,
) -> None:
self._store = store or scripture_store
self._memory = memory or scripture_memory
@property
def state(self) -> MeditationState:
return self._memory.associative.get_meditation_state()
def set_mode(self, mode: str, theme: Optional[str] = None) -> MeditationState:
"""Change the meditation mode (sequential / thematic / lectionary)."""
state = self.state
if mode not in ("sequential", "thematic", "lectionary"):
raise ValueError(f"Unknown mode: {mode}")
state.mode = mode
state.theme = theme
self._memory.associative.save_meditation_state(state)
return state
def current_focus(self) -> Optional[Verse]:
"""Return the verse currently in meditation focus."""
state = self.state
return self._store.get_verse(
state.current_book, state.current_chapter, state.current_verse
)
def next_meditation(self) -> Optional[Verse]:
"""Advance to the next verse and return it.
Dispatches to the appropriate strategy based on current mode.
"""
state = self.state
if state.mode == "thematic":
return self._next_thematic(state)
if state.mode == "lectionary":
return self._next_lectionary(state)
return self._next_sequential(state)
def meditate_on(self, verse: Verse, notes: str = "") -> None:
"""Record meditation on a specific verse and bring into focus."""
self._memory.working.focus(verse)
self._memory.associative.log_meditation(
verse.verse_id, notes=notes, mode=self.state.mode
)
state = self.state
state.advance(verse.book, verse.chapter, verse.verse_num)
self._memory.associative.save_meditation_state(state)
def get_context(self, verse: Verse, before: int = 2, after: int = 2) -> list[Verse]:
"""Retrieve surrounding verses for contextual meditation."""
start_id = encode_verse_id(verse.book, verse.chapter, max(1, verse.verse_num - before))
end_id = encode_verse_id(verse.book, verse.chapter, verse.verse_num + after)
return self._store.get_range(start_id, end_id)
def get_cross_references(self, verse: Verse) -> list[Verse]:
"""Retrieve cross-referenced verses for expanded meditation."""
xrefs = self._store.get_cross_references(verse.verse_id)
results = []
for xref in xrefs:
target_id = (
xref.target_verse_id
if xref.source_verse_id == verse.verse_id
else xref.source_verse_id
)
target = self._store.get_verse_by_id(target_id)
if target:
results.append(target)
return results
def history(self, limit: int = 20) -> list[dict]:
"""Return recent meditation history."""
return self._memory.associative.get_meditation_history(limit=limit)
def status(self) -> dict:
"""Return meditation scheduler status."""
state = self.state
current = self.current_focus()
book_info = BOOK_BY_ID.get(state.current_book)
return {
"mode": state.mode,
"theme": state.theme,
"current_book": book_info.name if book_info else f"Book {state.current_book}",
"current_chapter": state.current_chapter,
"current_verse": state.current_verse,
"current_text": current.text if current else None,
"verses_meditated": state.verses_meditated,
"last_meditation": state.last_meditation,
}
# ── Private strategies ───────────────────────────────────────────────
def _next_sequential(self, state: MeditationState) -> Optional[Verse]:
"""Sequential mode: advance verse-by-verse through the Bible."""
book = state.current_book
chapter = state.current_chapter
verse_num = state.current_verse + 1
# Try next verse in same chapter
verse = self._store.get_verse(book, chapter, verse_num)
if verse:
self.meditate_on(verse)
return verse
# Try next chapter
chapter += 1
verse_num = 1
verse = self._store.get_verse(book, chapter, verse_num)
if verse:
self.meditate_on(verse)
return verse
# Try next book
book += 1
if book > 66:
book = 1 # Wrap around to Genesis
chapter = 1
verse_num = 1
verse = self._store.get_verse(book, chapter, verse_num)
if verse:
self.meditate_on(verse)
return verse
return None
def _next_thematic(self, state: MeditationState) -> Optional[Verse]:
"""Thematic mode: retrieve verses related to current theme."""
if not state.theme:
# Fall back to sequential if no theme set
return self._next_sequential(state)
topics = self._store.search_topics(state.theme, limit=1)
if not topics:
return self._next_sequential(state)
verses = self._store.get_verses_for_topic(topics[0].topic_id)
if not verses:
return self._next_sequential(state)
# Pick the next un-meditated verse (or random if all visited)
history_ids = {
e["verse_id"]
for e in self._memory.associative.get_meditation_history(limit=1000)
}
for v in verses:
if v.verse_id not in history_ids:
self.meditate_on(v)
return v
# All verses in topic visited; pick a random one
chosen = random.choice(verses)
self.meditate_on(chosen)
return chosen
def _next_lectionary(self, state: MeditationState) -> Optional[Verse]:
"""Lectionary mode: placeholder — rotates through key passages.
A full lectionary implementation would integrate the Revised Common
Lectionary or similar. This simplified version cycles through
thematically significant passages.
"""
# Simplified: just advance sequentially for now
return self._next_sequential(state)
# Module-level singleton
meditation_scheduler = MeditationScheduler()

286
src/scripture/memory.py
View File

@@ -1,286 +0,0 @@
"""Scripture memory system — working, long-term, and associative memory.
Provides the tripartite memory architecture for continuous scriptural
engagement:
- **Working memory**: active passage under meditation (session-scoped)
- **Long-term memory**: persistent store of the full biblical corpus
(delegated to ScriptureStore)
- **Associative memory**: thematic and conceptual linkages between verses
The meditation scheduler uses this module to maintain "always on its mind"
engagement with scripture.
"""
from __future__ import annotations
import json
import logging
import sqlite3
from collections import OrderedDict
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional
from scripture.models import MeditationState, Verse, decode_verse_id
logger = logging.getLogger(__name__)
# Working memory capacity (analogous to 7±2 human working memory)
WORKING_MEMORY_CAPACITY = 7
_MEM_DB_DIR = Path("data")
_MEM_DB_PATH = _MEM_DB_DIR / "scripture.db"
_MEMORY_SCHEMA = """
CREATE TABLE IF NOT EXISTS meditation_state (
id INTEGER PRIMARY KEY CHECK (id = 1),
current_book INTEGER NOT NULL DEFAULT 1,
current_chapter INTEGER NOT NULL DEFAULT 1,
current_verse INTEGER NOT NULL DEFAULT 1,
mode TEXT NOT NULL DEFAULT 'sequential',
theme TEXT,
last_meditation TEXT,
verses_meditated INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS meditation_log (
id INTEGER PRIMARY KEY AUTOINCREMENT,
verse_id INTEGER NOT NULL,
meditated_at TEXT NOT NULL,
notes TEXT NOT NULL DEFAULT '',
mode TEXT NOT NULL DEFAULT 'sequential'
);
CREATE INDEX IF NOT EXISTS idx_meditation_log_verse
ON meditation_log(verse_id);
CREATE TABLE IF NOT EXISTS verse_insights (
id INTEGER PRIMARY KEY AUTOINCREMENT,
verse_id INTEGER NOT NULL,
insight TEXT NOT NULL,
category TEXT NOT NULL DEFAULT 'general',
created_at TEXT NOT NULL,
UNIQUE(verse_id, insight)
);
"""
class WorkingMemory:
"""Session-scoped memory for actively meditated passages.
Holds the most recent ``WORKING_MEMORY_CAPACITY`` verses in focus.
Uses an LRU-style eviction: oldest items drop when capacity is exceeded.
"""
def __init__(self, capacity: int = WORKING_MEMORY_CAPACITY) -> None:
self._capacity = capacity
self._items: OrderedDict[int, Verse] = OrderedDict()
def focus(self, verse: Verse) -> None:
"""Bring a verse into working memory (or refresh if already present)."""
if verse.verse_id in self._items:
self._items.move_to_end(verse.verse_id)
else:
self._items[verse.verse_id] = verse
if len(self._items) > self._capacity:
self._items.popitem(last=False)
def get_focused(self) -> list[Verse]:
"""Return all verses currently in working memory (most recent last)."""
return list(self._items.values())
def is_focused(self, verse_id: int) -> bool:
return verse_id in self._items
def clear(self) -> None:
self._items.clear()
def __len__(self) -> int:
return len(self._items)
class AssociativeMemory:
"""Thematic and conceptual linkages between verses.
Associates verses with insights and connections discovered during
meditation. Persisted to SQLite for cross-session continuity.
"""
def __init__(self, db_path: Path | str = _MEM_DB_PATH) -> None:
self._db_path = Path(db_path)
self._db_path.parent.mkdir(parents=True, exist_ok=True)
self._conn: Optional[sqlite3.Connection] = None
self._init_db()
def _get_conn(self) -> sqlite3.Connection:
if self._conn is None:
self._conn = sqlite3.connect(
str(self._db_path), check_same_thread=False
)
self._conn.row_factory = sqlite3.Row
return self._conn
def _init_db(self) -> None:
conn = self._get_conn()
conn.executescript(_MEMORY_SCHEMA)
# Ensure the singleton meditation state row exists
conn.execute(
"INSERT OR IGNORE INTO meditation_state (id) VALUES (1)"
)
conn.commit()
def close(self) -> None:
if self._conn:
self._conn.close()
self._conn = None
# ── Meditation state persistence ─────────────────────────────────────
def get_meditation_state(self) -> MeditationState:
"""Load the current meditation progress."""
row = self._get_conn().execute(
"SELECT * FROM meditation_state WHERE id = 1"
).fetchone()
if not row:
return MeditationState()
return MeditationState(
current_book=row["current_book"],
current_chapter=row["current_chapter"],
current_verse=row["current_verse"],
mode=row["mode"],
theme=row["theme"],
last_meditation=row["last_meditation"],
verses_meditated=row["verses_meditated"],
)
def save_meditation_state(self, state: MeditationState) -> None:
"""Persist the meditation state."""
conn = self._get_conn()
conn.execute(
"""UPDATE meditation_state SET
current_book = ?, current_chapter = ?, current_verse = ?,
mode = ?, theme = ?, last_meditation = ?, verses_meditated = ?
WHERE id = 1""",
(
state.current_book, state.current_chapter, state.current_verse,
state.mode, state.theme, state.last_meditation,
state.verses_meditated,
),
)
conn.commit()
# ── Meditation log ───────────────────────────────────────────────────
def log_meditation(
self, verse_id: int, notes: str = "", mode: str = "sequential"
) -> None:
"""Record that a verse was meditated upon."""
conn = self._get_conn()
conn.execute(
"INSERT INTO meditation_log (verse_id, meditated_at, notes, mode) VALUES (?, ?, ?, ?)",
(verse_id, datetime.now(timezone.utc).isoformat(), notes, mode),
)
conn.commit()
def get_meditation_history(self, limit: int = 20) -> list[dict]:
"""Return the most recent meditation log entries."""
rows = self._get_conn().execute(
"SELECT * FROM meditation_log ORDER BY id DESC LIMIT ?", (limit,)
).fetchall()
return [
{
"verse_id": r["verse_id"],
"meditated_at": r["meditated_at"],
"notes": r["notes"],
"mode": r["mode"],
}
for r in rows
]
def meditation_count(self) -> int:
"""Total meditation sessions logged."""
row = self._get_conn().execute(
"SELECT COUNT(*) FROM meditation_log"
).fetchone()
return row[0] if row else 0
# ── Verse insights ───────────────────────────────────────────────────
def add_insight(
self, verse_id: int, insight: str, category: str = "general"
) -> None:
"""Record an insight discovered during meditation or study."""
conn = self._get_conn()
conn.execute(
"""INSERT OR IGNORE INTO verse_insights
(verse_id, insight, category, created_at) VALUES (?, ?, ?, ?)""",
(verse_id, insight, category, datetime.now(timezone.utc).isoformat()),
)
conn.commit()
def get_insights(self, verse_id: int) -> list[dict]:
"""Retrieve all insights for a given verse."""
rows = self._get_conn().execute(
"SELECT * FROM verse_insights WHERE verse_id = ? ORDER BY created_at DESC",
(verse_id,),
).fetchall()
return [
{
"insight": r["insight"],
"category": r["category"],
"created_at": r["created_at"],
}
for r in rows
]
def get_recent_insights(self, limit: int = 10) -> list[dict]:
"""Return the most recently added insights across all verses."""
rows = self._get_conn().execute(
"SELECT * FROM verse_insights ORDER BY created_at DESC LIMIT ?",
(limit,),
).fetchall()
return [
{
"verse_id": r["verse_id"],
"insight": r["insight"],
"category": r["category"],
"created_at": r["created_at"],
}
for r in rows
]
class ScriptureMemory:
"""Unified scripture memory manager combining all three memory tiers.
Usage::
from scripture.memory import scripture_memory
scripture_memory.working.focus(verse)
state = scripture_memory.associative.get_meditation_state()
"""
def __init__(self, db_path: Path | str = _MEM_DB_PATH) -> None:
self.working = WorkingMemory()
self.associative = AssociativeMemory(db_path=db_path)
def close(self) -> None:
self.working.clear()
self.associative.close()
def status(self) -> dict:
"""Return a summary of memory system state."""
state = self.associative.get_meditation_state()
return {
"working_memory_items": len(self.working),
"working_memory_capacity": WORKING_MEMORY_CAPACITY,
"meditation_mode": state.mode,
"verses_meditated": state.verses_meditated,
"last_meditation": state.last_meditation,
"meditation_count": self.associative.meditation_count(),
}
# Module-level singleton
scripture_memory = ScriptureMemory()

160
src/scripture/models.py
View File

@@ -1,160 +0,0 @@
"""Data models for the scripture module.
Provides Pydantic models for verses, references, cross-references,
topics, and original language annotations.
"""
from __future__ import annotations
from datetime import datetime, timezone
from typing import Literal, Optional
from pydantic import BaseModel, Field
# ── Integer encoding scheme ──────────────────────────────────────────────────
# book (1-66, 1-2 digits) + chapter (3 digits, zero-padded) +
# verse (3 digits, zero-padded) = 7-8 digit unique integer per verse.
# Example: John 3:16 → 43_003_016 = 43003016
def encode_verse_id(book: int, chapter: int, verse: int) -> int:
"""Encode a book/chapter/verse triplet into a unique integer ID."""
return book * 1_000_000 + chapter * 1_000 + verse
def decode_verse_id(verse_id: int) -> tuple[int, int, int]:
"""Decode an integer verse ID back to (book, chapter, verse)."""
book = verse_id // 1_000_000
remainder = verse_id % 1_000_000
chapter = remainder // 1_000
verse = remainder % 1_000
return book, chapter, verse
# ── Core models ──────────────────────────────────────────────────────────────
class VerseRef(BaseModel):
"""A single verse reference (book + chapter + verse)."""
book: int = Field(ge=1, le=66, description="Canonical book ID (1-66)")
chapter: int = Field(ge=1, description="Chapter number")
verse: int = Field(ge=0, description="Verse number (0 for chapter-level)")
@property
def verse_id(self) -> int:
return encode_verse_id(self.book, self.chapter, self.verse)
class VerseRange(BaseModel):
"""A contiguous range of verses."""
start: VerseRef
end: VerseRef
def verse_ids(self) -> list[int]:
"""Expand the range to individual verse IDs."""
ids = []
for vid in range(self.start.verse_id, self.end.verse_id + 1):
b, c, v = decode_verse_id(vid)
if 1 <= b <= 66 and c >= 1 and v >= 1:
ids.append(vid)
return ids
class Verse(BaseModel):
"""A single verse with text content and metadata."""
verse_id: int = Field(description="Encoded integer ID")
book: int
chapter: int
verse_num: int
text: str
translation: str = "ESV"
testament: Literal["OT", "NT"] = "OT"
genre: str = ""
class CrossReference(BaseModel):
"""A cross-reference link between two verses."""
source_verse_id: int
target_verse_id: int
reference_type: Literal[
"quotation", "allusion", "thematic", "typology", "parallel"
] = "thematic"
confidence: float = Field(default=1.0, ge=0.0, le=1.0)
class Topic(BaseModel):
"""A topical category from a topical index (e.g. Nave's)."""
topic_id: str
name: str
parent_id: Optional[str] = None
description: str = ""
verse_ids: list[int] = Field(default_factory=list)
class StrongsEntry(BaseModel):
"""A Strong's concordance entry for original language terms."""
strongs_number: str = Field(description="e.g. H7225, G26")
language: Literal["hebrew", "greek"]
lemma: str = ""
transliteration: str = ""
gloss: str = ""
morphology: str = ""
class OriginalLanguageToken(BaseModel):
"""A token in original language text with annotations."""
text: str
transliteration: str = ""
strongs_number: str = ""
morphology: str = ""
gloss: str = ""
word_position: int = 0
class InterlinearVerse(BaseModel):
"""A verse with interlinear original language alignment."""
verse_id: int
reference: str
original_tokens: list[OriginalLanguageToken] = Field(default_factory=list)
esv_text: str = ""
language: Literal["hebrew", "greek"] = "hebrew"
class MeditationState(BaseModel):
"""Tracks the current meditation progress."""
current_book: int = 1
current_chapter: int = 1
current_verse: int = 1
mode: Literal["sequential", "thematic", "lectionary"] = "sequential"
theme: Optional[str] = None
last_meditation: Optional[str] = None
verses_meditated: int = 0
def advance(self, book: int, chapter: int, verse: int) -> None:
self.current_book = book
self.current_chapter = chapter
self.current_verse = verse
self.last_meditation = datetime.now(timezone.utc).isoformat()
self.verses_meditated += 1
class ScriptureQuery(BaseModel):
"""A parsed user query for scripture content."""
intent: Literal[
"lookup", "explanation", "application", "comparison", "devotional", "search"
] = "lookup"
references: list[VerseRef] = Field(default_factory=list)
keywords: list[str] = Field(default_factory=list)
topic: Optional[str] = None
raw_text: str = ""

166
src/scripture/parser.py
View File

@@ -1,166 +0,0 @@
"""Reference parser — extract and normalise biblical references from text.
Handles explicit references (``John 3:16``), range references
(``Romans 5:1-11``), multi-chapter ranges (``Genesis 1:1-2:3``),
and fuzzy book name matching (``1 Cor 13``, ``Phil 4 13``).
"""
from __future__ import annotations
import re
from typing import Optional
from scripture.constants import book_by_name, BookInfo, BOOK_BY_ID
from scripture.models import VerseRef, VerseRange
# ── Regex patterns ───────────────────────────────────────────────────────────
# Matches patterns like "John 3:16", "1 Cor 13:4-7", "Gen 1:1-2:3"
_REF_PATTERN = re.compile(
r"""
(?P<book>
(?:[123]\s*)? # optional ordinal (1, 2, 3)
[A-Za-z]+ # book name
(?:\s+of\s+[A-Za-z]+)? # "Song of Solomon"
)
\s*
(?P<chapter>\d{1,3}) # chapter number
(?:
\s*[:\.]\s* # separator (colon or dot)
(?P<verse>\d{1,3}) # verse number
(?:
\s*[-–—]\s* # range separator
(?:
(?P<end_chapter>\d{1,3}) # optional end chapter
\s*[:\.]\s*
)?
(?P<end_verse>\d{1,3}) # end verse
)?
)?
""",
re.VERBOSE | re.IGNORECASE,
)
def _normalise_book_name(raw: str) -> str:
"""Collapse whitespace and lowercase for lookup."""
return re.sub(r"\s+", " ", raw.strip()).lower()
def resolve_book(name: str) -> Optional[BookInfo]:
"""Resolve a book name/abbreviation to a BookInfo."""
return book_by_name(_normalise_book_name(name))
def parse_reference(text: str) -> Optional[VerseRange]:
"""Parse a single scripture reference string into a VerseRange.
Examples::
parse_reference("John 3:16")
parse_reference("Genesis 1:1-3")
parse_reference("Rom 5:1-11")
parse_reference("1 Cor 13") # whole chapter
"""
m = _REF_PATTERN.search(text)
if not m:
return None
book_info = resolve_book(m.group("book"))
if not book_info:
return None
chapter = int(m.group("chapter"))
verse_str = m.group("verse")
end_verse_str = m.group("end_verse")
end_chapter_str = m.group("end_chapter")
if verse_str is None:
# Whole chapter reference: "Genesis 1"
start = VerseRef(book=book_info.id, chapter=chapter, verse=1)
# Use a large verse number; the caller truncates to actual max
end = VerseRef(book=book_info.id, chapter=chapter, verse=999)
return VerseRange(start=start, end=end)
start_verse = int(verse_str)
start = VerseRef(book=book_info.id, chapter=chapter, verse=start_verse)
if end_verse_str is not None:
end_ch = int(end_chapter_str) if end_chapter_str else chapter
end_v = int(end_verse_str)
end = VerseRef(book=book_info.id, chapter=end_ch, verse=end_v)
else:
end = VerseRef(book=book_info.id, chapter=chapter, verse=start_verse)
return VerseRange(start=start, end=end)
def extract_references(text: str) -> list[VerseRange]:
"""Extract all scripture references from a block of text.
Returns a list of VerseRange objects for every reference found.
"""
results: list[VerseRange] = []
for m in _REF_PATTERN.finditer(text):
book_info = resolve_book(m.group("book"))
if not book_info:
continue
chapter = int(m.group("chapter"))
verse_str = m.group("verse")
end_verse_str = m.group("end_verse")
end_chapter_str = m.group("end_chapter")
if verse_str is None:
start = VerseRef(book=book_info.id, chapter=chapter, verse=1)
end = VerseRef(book=book_info.id, chapter=chapter, verse=999)
else:
sv = int(verse_str)
start = VerseRef(book=book_info.id, chapter=chapter, verse=sv)
if end_verse_str is not None:
end_ch = int(end_chapter_str) if end_chapter_str else chapter
end = VerseRef(book=book_info.id, chapter=end_ch, verse=int(end_verse_str))
else:
end = VerseRef(book=book_info.id, chapter=chapter, verse=sv)
results.append(VerseRange(start=start, end=end))
return results
def format_reference(ref: VerseRef) -> str:
"""Format a VerseRef as a human-readable string.
Example: ``VerseRef(book=43, chapter=3, verse=16)`` → ``"John 3:16"``
"""
book = BOOK_BY_ID.get(ref.book)
if not book:
return f"Unknown {ref.chapter}:{ref.verse}"
if ref.verse == 0:
return f"{book.name} {ref.chapter}"
return f"{book.name} {ref.chapter}:{ref.verse}"
def format_range(vr: VerseRange) -> str:
"""Format a VerseRange as a human-readable string.
Examples::
"John 3:16" (single verse)
"Romans 5:1-11" (same chapter range)
"Genesis 1:1-2:3" (multi-chapter range)
"""
start_book = BOOK_BY_ID.get(vr.start.book)
if not start_book:
return "Unknown reference"
if vr.start.verse_id == vr.end.verse_id:
return format_reference(vr.start)
if vr.start.chapter == vr.end.chapter:
return f"{start_book.name} {vr.start.chapter}:{vr.start.verse}-{vr.end.verse}"
return (
f"{start_book.name} {vr.start.chapter}:{vr.start.verse}"
f"-{vr.end.chapter}:{vr.end.verse}"
)

387
src/scripture/store.py
View File

@@ -1,387 +0,0 @@
"""Scripture store — SQLite-backed verse storage and retrieval.
Provides the persistent knowledge base for the complete ESV text.
Follows the project's SQLite singleton pattern (cf. swarm/registry.py).
Tables
------
- ``verses`` Primary verse storage with text + metadata
- ``cross_references`` TSK-derived edges between verses
- ``topics`` Nave's-style topical index entries
- ``verse_topics`` Many-to-many verse ↔ topic links
- ``strongs`` Strong's concordance entries
"""
from __future__ import annotations
import json
import logging
import sqlite3
from pathlib import Path
from typing import Optional
from scripture.constants import BOOK_BY_ID, book_by_name
from scripture.models import (
CrossReference,
StrongsEntry,
Topic,
Verse,
VerseRef,
decode_verse_id,
encode_verse_id,
)
logger = logging.getLogger(__name__)
DB_DIR = Path("data")
DB_PATH = DB_DIR / "scripture.db"
_SCHEMA = """
CREATE TABLE IF NOT EXISTS verses (
verse_id INTEGER PRIMARY KEY,
book INTEGER NOT NULL,
chapter INTEGER NOT NULL,
verse_num INTEGER NOT NULL,
text TEXT NOT NULL,
translation TEXT NOT NULL DEFAULT 'ESV',
testament TEXT NOT NULL DEFAULT 'OT',
genre TEXT NOT NULL DEFAULT ''
);
CREATE INDEX IF NOT EXISTS idx_verses_book_ch
ON verses(book, chapter);
CREATE TABLE IF NOT EXISTS cross_references (
id INTEGER PRIMARY KEY AUTOINCREMENT,
source_verse_id INTEGER NOT NULL,
target_verse_id INTEGER NOT NULL,
reference_type TEXT NOT NULL DEFAULT 'thematic',
confidence REAL NOT NULL DEFAULT 1.0,
UNIQUE(source_verse_id, target_verse_id, reference_type)
);
CREATE INDEX IF NOT EXISTS idx_xref_source
ON cross_references(source_verse_id);
CREATE INDEX IF NOT EXISTS idx_xref_target
ON cross_references(target_verse_id);
CREATE TABLE IF NOT EXISTS topics (
topic_id TEXT PRIMARY KEY,
name TEXT NOT NULL,
parent_id TEXT,
description TEXT NOT NULL DEFAULT ''
);
CREATE TABLE IF NOT EXISTS verse_topics (
verse_id INTEGER NOT NULL,
topic_id TEXT NOT NULL,
relevance REAL NOT NULL DEFAULT 1.0,
PRIMARY KEY (verse_id, topic_id)
);
CREATE INDEX IF NOT EXISTS idx_vt_topic
ON verse_topics(topic_id);
CREATE TABLE IF NOT EXISTS strongs (
strongs_number TEXT PRIMARY KEY,
language TEXT NOT NULL,
lemma TEXT NOT NULL DEFAULT '',
transliteration TEXT NOT NULL DEFAULT '',
gloss TEXT NOT NULL DEFAULT '',
morphology TEXT NOT NULL DEFAULT ''
);
"""
class ScriptureStore:
"""SQLite-backed scripture knowledge base.
Usage::
from scripture.store import scripture_store
verse = scripture_store.get_verse(43, 3, 16)
"""
def __init__(self, db_path: Path | str = DB_PATH) -> None:
self._db_path = Path(db_path)
self._db_path.parent.mkdir(parents=True, exist_ok=True)
self._conn: Optional[sqlite3.Connection] = None
self._init_db()
# ── Connection management ────────────────────────────────────────────
def _get_conn(self) -> sqlite3.Connection:
if self._conn is None:
self._conn = sqlite3.connect(
str(self._db_path), check_same_thread=False
)
self._conn.row_factory = sqlite3.Row
self._conn.execute("PRAGMA journal_mode=WAL")
self._conn.execute("PRAGMA foreign_keys=ON")
return self._conn
def _init_db(self) -> None:
conn = self._get_conn()
conn.executescript(_SCHEMA)
conn.commit()
def close(self) -> None:
if self._conn:
self._conn.close()
self._conn = None
# ── Verse CRUD ───────────────────────────────────────────────────────
def insert_verse(self, verse: Verse) -> None:
"""Insert or replace a single verse."""
conn = self._get_conn()
conn.execute(
"""INSERT OR REPLACE INTO verses
(verse_id, book, chapter, verse_num, text, translation, testament, genre)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)""",
(
verse.verse_id,
verse.book,
verse.chapter,
verse.verse_num,
verse.text,
verse.translation,
verse.testament,
verse.genre,
),
)
conn.commit()
def insert_verses(self, verses: list[Verse]) -> None:
"""Bulk-insert verses (efficient for full-text ingestion)."""
conn = self._get_conn()
conn.executemany(
"""INSERT OR REPLACE INTO verses
(verse_id, book, chapter, verse_num, text, translation, testament, genre)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)""",
[
(v.verse_id, v.book, v.chapter, v.verse_num,
v.text, v.translation, v.testament, v.genre)
for v in verses
],
)
conn.commit()
def get_verse(self, book: int, chapter: int, verse: int) -> Optional[Verse]:
"""Retrieve a single verse by book/chapter/verse."""
vid = encode_verse_id(book, chapter, verse)
row = self._get_conn().execute(
"SELECT * FROM verses WHERE verse_id = ?", (vid,)
).fetchone()
return self._row_to_verse(row) if row else None
def get_verse_by_id(self, verse_id: int) -> Optional[Verse]:
"""Retrieve a verse by its integer ID."""
row = self._get_conn().execute(
"SELECT * FROM verses WHERE verse_id = ?", (verse_id,)
).fetchone()
return self._row_to_verse(row) if row else None
def get_chapter(self, book: int, chapter: int) -> list[Verse]:
"""Retrieve all verses in a chapter, ordered by verse number."""
rows = self._get_conn().execute(
"SELECT * FROM verses WHERE book = ? AND chapter = ? ORDER BY verse_num",
(book, chapter),
).fetchall()
return [self._row_to_verse(r) for r in rows]
def get_range(self, start_id: int, end_id: int) -> list[Verse]:
"""Retrieve all verses in a range of verse IDs (inclusive)."""
rows = self._get_conn().execute(
"SELECT * FROM verses WHERE verse_id BETWEEN ? AND ? ORDER BY verse_id",
(start_id, end_id),
).fetchall()
return [self._row_to_verse(r) for r in rows]
def search_text(self, query: str, limit: int = 20) -> list[Verse]:
"""Full-text search across verse content (LIKE-based)."""
rows = self._get_conn().execute(
"SELECT * FROM verses WHERE text LIKE ? ORDER BY verse_id LIMIT ?",
(f"%{query}%", limit),
).fetchall()
return [self._row_to_verse(r) for r in rows]
def count_verses(self) -> int:
"""Return the total number of verses in the store."""
row = self._get_conn().execute("SELECT COUNT(*) FROM verses").fetchone()
return row[0] if row else 0
def get_books(self) -> list[dict]:
"""Return a summary of all books with verse counts."""
rows = self._get_conn().execute(
"""SELECT book, COUNT(*) as verse_count, MIN(chapter) as min_ch,
MAX(chapter) as max_ch
FROM verses GROUP BY book ORDER BY book"""
).fetchall()
result = []
for r in rows:
info = BOOK_BY_ID.get(r["book"])
result.append({
"book_id": r["book"],
"name": info.name if info else f"Book {r['book']}",
"abbreviation": info.abbreviation if info else "",
"testament": info.testament if info else "",
"verse_count": r["verse_count"],
"chapters": r["max_ch"],
})
return result
# ── Cross-references ─────────────────────────────────────────────────
def insert_cross_reference(self, xref: CrossReference) -> None:
"""Insert a cross-reference link."""
conn = self._get_conn()
conn.execute(
"""INSERT OR IGNORE INTO cross_references
(source_verse_id, target_verse_id, reference_type, confidence)
VALUES (?, ?, ?, ?)""",
(xref.source_verse_id, xref.target_verse_id,
xref.reference_type, xref.confidence),
)
conn.commit()
def get_cross_references(self, verse_id: int) -> list[CrossReference]:
"""Find all cross-references from or to a verse."""
rows = self._get_conn().execute(
"""SELECT * FROM cross_references
WHERE source_verse_id = ? OR target_verse_id = ?
ORDER BY confidence DESC""",
(verse_id, verse_id),
).fetchall()
return [
CrossReference(
source_verse_id=r["source_verse_id"],
target_verse_id=r["target_verse_id"],
reference_type=r["reference_type"],
confidence=r["confidence"],
)
for r in rows
]
# ── Topics ───────────────────────────────────────────────────────────
def insert_topic(self, topic: Topic) -> None:
"""Insert a topical index entry."""
conn = self._get_conn()
conn.execute(
"""INSERT OR REPLACE INTO topics
(topic_id, name, parent_id, description) VALUES (?, ?, ?, ?)""",
(topic.topic_id, topic.name, topic.parent_id, topic.description),
)
for vid in topic.verse_ids:
conn.execute(
"INSERT OR IGNORE INTO verse_topics (verse_id, topic_id) VALUES (?, ?)",
(vid, topic.topic_id),
)
conn.commit()
def get_topic(self, topic_id: str) -> Optional[Topic]:
"""Retrieve a topic by ID."""
row = self._get_conn().execute(
"SELECT * FROM topics WHERE topic_id = ?", (topic_id,)
).fetchone()
if not row:
return None
verse_rows = self._get_conn().execute(
"SELECT verse_id FROM verse_topics WHERE topic_id = ?", (topic_id,)
).fetchall()
return Topic(
topic_id=row["topic_id"],
name=row["name"],
parent_id=row["parent_id"],
description=row["description"],
verse_ids=[r["verse_id"] for r in verse_rows],
)
def search_topics(self, query: str, limit: int = 10) -> list[Topic]:
"""Search topics by name."""
rows = self._get_conn().execute(
"SELECT * FROM topics WHERE name LIKE ? ORDER BY name LIMIT ?",
(f"%{query}%", limit),
).fetchall()
return [
Topic(topic_id=r["topic_id"], name=r["name"],
parent_id=r["parent_id"], description=r["description"])
for r in rows
]
def get_verses_for_topic(self, topic_id: str) -> list[Verse]:
"""Retrieve all verses associated with a topic."""
rows = self._get_conn().execute(
"""SELECT v.* FROM verses v
INNER JOIN verse_topics vt ON v.verse_id = vt.verse_id
WHERE vt.topic_id = ?
ORDER BY v.verse_id""",
(topic_id,),
).fetchall()
return [self._row_to_verse(r) for r in rows]
# ── Strong's concordance ─────────────────────────────────────────────
def insert_strongs(self, entry: StrongsEntry) -> None:
"""Insert a Strong's concordance entry."""
conn = self._get_conn()
conn.execute(
"""INSERT OR REPLACE INTO strongs
(strongs_number, language, lemma, transliteration, gloss, morphology)
VALUES (?, ?, ?, ?, ?, ?)""",
(entry.strongs_number, entry.language, entry.lemma,
entry.transliteration, entry.gloss, entry.morphology),
)
conn.commit()
def get_strongs(self, number: str) -> Optional[StrongsEntry]:
"""Look up a Strong's number."""
row = self._get_conn().execute(
"SELECT * FROM strongs WHERE strongs_number = ?", (number,)
).fetchone()
if not row:
return None
return StrongsEntry(
strongs_number=row["strongs_number"],
language=row["language"],
lemma=row["lemma"],
transliteration=row["transliteration"],
gloss=row["gloss"],
morphology=row["morphology"],
)
# ── Stats ────────────────────────────────────────────────────────────
def stats(self) -> dict:
"""Return summary statistics of the scripture store."""
conn = self._get_conn()
verses = conn.execute("SELECT COUNT(*) FROM verses").fetchone()[0]
xrefs = conn.execute("SELECT COUNT(*) FROM cross_references").fetchone()[0]
topics = conn.execute("SELECT COUNT(*) FROM topics").fetchone()[0]
strongs = conn.execute("SELECT COUNT(*) FROM strongs").fetchone()[0]
return {
"verses": verses,
"cross_references": xrefs,
"topics": topics,
"strongs_entries": strongs,
}
# ── Helpers ──────────────────────────────────────────────────────────
@staticmethod
def _row_to_verse(row: sqlite3.Row) -> Verse:
return Verse(
verse_id=row["verse_id"],
book=row["book"],
chapter=row["chapter"],
verse_num=row["verse_num"],
text=row["text"],
translation=row["translation"],
testament=row["testament"],
genre=row["genre"],
)
# Module-level singleton
scripture_store = ScriptureStore()

23
src/self_coding/CLAUDE.md
View File

@@ -1,23 +0,0 @@
# self_coding/ — Module Guide
Self-modification infrastructure with safety constraints.
## Structure
- `git_safety.py` — Atomic git operations with rollback
- `codebase_indexer.py` — Live mental model of the codebase
- `modification_journal.py` — Persistent log of modification attempts
- `reflection.py` — Generate lessons learned
- `self_modify/` — Runtime self-modification loop (LLM-driven)
- `self_tdd/` — Continuous test watchdog
- `upgrades/` — Self-upgrade approval queue
## Entry points
```toml
self-tdd = "self_coding.self_tdd.watchdog:main"
self-modify = "self_coding.self_modify.cli:main"
```
## Testing
```bash
pytest tests/self_coding/ -q
```

50
src/self_coding/__init__.py
View File

@@ -1,50 +0,0 @@
"""Self-Coding Layer — Timmy's ability to modify its own source code safely.
This module provides the foundational infrastructure for self-modification:
- GitSafety: Atomic git operations with rollback capability
- CodebaseIndexer: Live mental model of the codebase
- ModificationJournal: Persistent log of modification attempts
- ReflectionService: Generate lessons learned from attempts
Usage:
from self_coding import GitSafety, CodebaseIndexer, ModificationJournal
from self_coding import ModificationAttempt, Outcome, Snapshot
# Initialize services
git = GitSafety(repo_path="/path/to/repo")
indexer = CodebaseIndexer(repo_path="/path/to/repo")
journal = ModificationJournal()
# Use in self-modification workflow
snapshot = await git.snapshot()
# ... make changes ...
if tests_pass:
await git.commit("Changes", ["file.py"])
else:
await git.rollback(snapshot)
"""
from self_coding.git_safety import GitSafety, Snapshot
from self_coding.codebase_indexer import CodebaseIndexer, ModuleInfo, FunctionInfo, ClassInfo
from self_coding.modification_journal import (
ModificationJournal,
ModificationAttempt,
Outcome,
)
from self_coding.reflection import ReflectionService
__all__ = [
# Core services
"GitSafety",
"CodebaseIndexer",
"ModificationJournal",
"ReflectionService",
# Data classes
"Snapshot",
"ModuleInfo",
"FunctionInfo",
"ClassInfo",
"ModificationAttempt",
"Outcome",
]

772
src/self_coding/codebase_indexer.py
View File

@@ -1,772 +0,0 @@
"""Codebase Indexer — Live mental model of Timmy's own codebase.
Parses Python files using AST to extract classes, functions, imports, and
docstrings. Builds a dependency graph and provides semantic search for
relevant files.
"""
from __future__ import annotations
import ast
import hashlib
import json
import logging
import sqlite3
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Optional
logger = logging.getLogger(__name__)
# Default database location
DEFAULT_DB_PATH = Path("data/self_coding.db")
@dataclass
class FunctionInfo:
"""Information about a function."""
name: str
args: list[str]
returns: Optional[str] = None
docstring: Optional[str] = None
line_number: int = 0
is_async: bool = False
is_method: bool = False
@dataclass
class ClassInfo:
"""Information about a class."""
name: str
methods: list[FunctionInfo] = field(default_factory=list)
docstring: Optional[str] = None
line_number: int = 0
bases: list[str] = field(default_factory=list)
@dataclass
class ModuleInfo:
"""Information about a Python module."""
file_path: str
module_name: str
classes: list[ClassInfo] = field(default_factory=list)
functions: list[FunctionInfo] = field(default_factory=list)
imports: list[str] = field(default_factory=list)
docstring: Optional[str] = None
test_coverage: Optional[str] = None
class CodebaseIndexer:
"""Indexes Python codebase for self-modification workflows.
Parses all Python files using AST to extract:
- Module names and structure
- Class definitions with methods
- Function signatures with args and return types
- Import relationships
- Test coverage mapping
Stores everything in SQLite for fast querying.
Usage:
indexer = CodebaseIndexer(repo_path="/path/to/repo")
# Full reindex
await indexer.index_all()
# Incremental update
await indexer.index_changed()
# Get LLM context summary
summary = await indexer.get_summary()
# Find relevant files for a task
files = await indexer.get_relevant_files("Add error handling to health endpoint")
# Get dependency chain
deps = await indexer.get_dependency_chain("src/timmy/agent.py")
"""
def __init__(
self,
repo_path: Optional[str | Path] = None,
db_path: Optional[str | Path] = None,
src_dirs: Optional[list[str]] = None,
) -> None:
"""Initialize CodebaseIndexer.
Args:
repo_path: Root of repository to index. Defaults to current directory.
db_path: SQLite database path. Defaults to data/self_coding.db
src_dirs: Source directories to index. Defaults to ["src", "tests"]
"""
self.repo_path = Path(repo_path).resolve() if repo_path else Path.cwd()
self.db_path = Path(db_path) if db_path else DEFAULT_DB_PATH
self.src_dirs = src_dirs or ["src", "tests"]
self._ensure_schema()
logger.info("CodebaseIndexer initialized for %s", self.repo_path)
def _get_conn(self) -> sqlite3.Connection:
"""Get database connection with schema ensured."""
self.db_path.parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(str(self.db_path))
conn.row_factory = sqlite3.Row
return conn
def _ensure_schema(self) -> None:
"""Create database tables if they don't exist."""
with self._get_conn() as conn:
# Main codebase index table
conn.execute(
"""
CREATE TABLE IF NOT EXISTS codebase_index (
file_path TEXT PRIMARY KEY,
module_name TEXT NOT NULL,
classes JSON,
functions JSON,
imports JSON,
test_coverage TEXT,
last_indexed TIMESTAMP NOT NULL,
content_hash TEXT NOT NULL,
docstring TEXT,
embedding BLOB
)
"""
)
# Dependency graph table
conn.execute(
"""
CREATE TABLE IF NOT EXISTS dependency_graph (
id INTEGER PRIMARY KEY AUTOINCREMENT,
source_file TEXT NOT NULL,
target_file TEXT NOT NULL,
import_type TEXT NOT NULL,
UNIQUE(source_file, target_file)
)
"""
)
# Create indexes
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_module_name ON codebase_index(module_name)"
)
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_test_coverage ON codebase_index(test_coverage)"
)
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_deps_source ON dependency_graph(source_file)"
)
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_deps_target ON dependency_graph(target_file)"
)
conn.commit()
def _compute_hash(self, content: str) -> str:
"""Compute SHA-256 hash of file content."""
return hashlib.sha256(content.encode("utf-8")).hexdigest()
def _find_python_files(self) -> list[Path]:
"""Find all Python files in source directories."""
files = []
for src_dir in self.src_dirs:
src_path = self.repo_path / src_dir
if src_path.exists():
files.extend(src_path.rglob("*.py"))
return sorted(files)
def _find_test_file(self, source_file: Path) -> Optional[str]:
"""Find corresponding test file for a source file.
Uses conventions:
- src/x/y.py -> tests/test_x_y.py
- src/x/y.py -> tests/x/test_y.py
- src/x/y.py -> tests/test_y.py
"""
rel_path = source_file.relative_to(self.repo_path)
# Only look for tests for files in src/
if not str(rel_path).startswith("src/"):
return None
# Try various test file naming conventions
possible_tests = [
# tests/test_module.py
self.repo_path / "tests" / f"test_{source_file.stem}.py",
# tests/test_path_module.py (flat)
self.repo_path / "tests" / f"test_{'_'.join(rel_path.with_suffix('').parts[1:])}.py",
]
# Try mirroring src structure in tests (tests/x/test_y.py)
try:
src_relative = rel_path.relative_to("src")
possible_tests.append(
self.repo_path / "tests" / src_relative.parent / f"test_{source_file.stem}.py"
)
except ValueError:
pass
for test_path in possible_tests:
if test_path.exists():
return str(test_path.relative_to(self.repo_path))
return None
def _parse_function(self, node: ast.FunctionDef | ast.AsyncFunctionDef, is_method: bool = False) -> FunctionInfo:
"""Parse a function definition node."""
args = []
# Handle different Python versions' AST structures
func_args = node.args
# Positional args
for arg in func_args.args:
arg_str = arg.arg
if arg.annotation:
arg_str += f": {ast.unparse(arg.annotation)}"
args.append(arg_str)
# Keyword-only args
for arg in func_args.kwonlyargs:
arg_str = arg.arg
if arg.annotation:
arg_str += f": {ast.unparse(arg.annotation)}"
args.append(arg_str)
# Return type
returns = None
if node.returns:
returns = ast.unparse(node.returns)
# Docstring
docstring = ast.get_docstring(node)
return FunctionInfo(
name=node.name,
args=args,
returns=returns,
docstring=docstring,
line_number=node.lineno,
is_async=isinstance(node, ast.AsyncFunctionDef),
is_method=is_method,
)
def _parse_class(self, node: ast.ClassDef) -> ClassInfo:
"""Parse a class definition node."""
methods = []
for item in node.body:
if isinstance(item, (ast.FunctionDef, ast.AsyncFunctionDef)):
methods.append(self._parse_function(item, is_method=True))
# Get bases
bases = [ast.unparse(base) for base in node.bases]
return ClassInfo(
name=node.name,
methods=methods,
docstring=ast.get_docstring(node),
line_number=node.lineno,
bases=bases,
)
def _parse_module(self, file_path: Path) -> Optional[ModuleInfo]:
"""Parse a Python module file.
Args:
file_path: Path to Python file
Returns:
ModuleInfo or None if parsing fails
"""
try:
content = file_path.read_text(encoding="utf-8")
tree = ast.parse(content)
# Compute module name from file path
rel_path = file_path.relative_to(self.repo_path)
module_name = str(rel_path.with_suffix("")).replace("/", ".")
classes = []
functions = []
imports = []
for node in ast.walk(tree):
if isinstance(node, ast.Import):
for alias in node.names:
imports.append(alias.name)
elif isinstance(node, ast.ImportFrom):
module = node.module or ""
for alias in node.names:
imports.append(f"{module}.{alias.name}")
# Get top-level definitions (not in classes)
for node in tree.body:
if isinstance(node, ast.ClassDef):
classes.append(self._parse_class(node))
elif isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
functions.append(self._parse_function(node))
# Get module docstring
docstring = ast.get_docstring(tree)
# Find test coverage
test_coverage = self._find_test_file(file_path)
return ModuleInfo(
file_path=str(rel_path),
module_name=module_name,
classes=classes,
functions=functions,
imports=imports,
docstring=docstring,
test_coverage=test_coverage,
)
except SyntaxError as e:
logger.warning("Syntax error in %s: %s", file_path, e)
return None
except Exception as e:
logger.error("Failed to parse %s: %s", file_path, e)
return None
def _store_module(self, conn: sqlite3.Connection, module: ModuleInfo, content_hash: str) -> None:
"""Store module info in database."""
conn.execute(
"""
INSERT OR REPLACE INTO codebase_index
(file_path, module_name, classes, functions, imports, test_coverage,
last_indexed, content_hash, docstring)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
module.file_path,
module.module_name,
json.dumps([asdict(c) for c in module.classes]),
json.dumps([asdict(f) for f in module.functions]),
json.dumps(module.imports),
module.test_coverage,
datetime.now(timezone.utc).isoformat(),
content_hash,
module.docstring,
),
)
def _build_dependency_graph(self, conn: sqlite3.Connection) -> None:
"""Build and store dependency graph from imports."""
# Clear existing graph
conn.execute("DELETE FROM dependency_graph")
# Get all modules
rows = conn.execute("SELECT file_path, module_name, imports FROM codebase_index").fetchall()
# Map module names to file paths
module_to_file = {row["module_name"]: row["file_path"] for row in rows}
# Also map without src/ prefix for package imports like myproject.utils
module_to_file_alt = {}
for row in rows:
module_name = row["module_name"]
if module_name.startswith("src."):
alt_name = module_name[4:] # Remove "src." prefix
module_to_file_alt[alt_name] = row["file_path"]
# Build dependencies
for row in rows:
source_file = row["file_path"]
imports = json.loads(row["imports"])
for imp in imports:
# Try to resolve import to a file
# Handle both "module.name" and "module.name.Class" forms
# First try exact match
if imp in module_to_file:
conn.execute(
"""
INSERT OR IGNORE INTO dependency_graph
(source_file, target_file, import_type)
VALUES (?, ?, ?)
""",
(source_file, module_to_file[imp], "import"),
)
continue
# Try alternative name (without src/ prefix)
if imp in module_to_file_alt:
conn.execute(
"""
INSERT OR IGNORE INTO dependency_graph
(source_file, target_file, import_type)
VALUES (?, ?, ?)
""",
(source_file, module_to_file_alt[imp], "import"),
)
continue
# Try prefix match (import myproject.utils.Helper -> myproject.utils)
imp_parts = imp.split(".")
for i in range(len(imp_parts), 0, -1):
prefix = ".".join(imp_parts[:i])
# Try original module name
if prefix in module_to_file:
conn.execute(
"""
INSERT OR IGNORE INTO dependency_graph
(source_file, target_file, import_type)
VALUES (?, ?, ?)
""",
(source_file, module_to_file[prefix], "import"),
)
break
# Try alternative name (without src/ prefix)
if prefix in module_to_file_alt:
conn.execute(
"""
INSERT OR IGNORE INTO dependency_graph
(source_file, target_file, import_type)
VALUES (?, ?, ?)
""",
(source_file, module_to_file_alt[prefix], "import"),
)
break
conn.commit()
async def index_all(self) -> dict[str, int]:
"""Perform full reindex of all Python files.
Returns:
Dict with stats: {"indexed": int, "failed": int, "skipped": int}
"""
logger.info("Starting full codebase index")
files = self._find_python_files()
stats = {"indexed": 0, "failed": 0, "skipped": 0}
with self._get_conn() as conn:
for file_path in files:
try:
content = file_path.read_text(encoding="utf-8")
content_hash = self._compute_hash(content)
# Check if file needs reindexing
existing = conn.execute(
"SELECT content_hash FROM codebase_index WHERE file_path = ?",
(str(file_path.relative_to(self.repo_path)),),
).fetchone()
if existing and existing["content_hash"] == content_hash:
stats["skipped"] += 1
continue
module = self._parse_module(file_path)
if module:
self._store_module(conn, module, content_hash)
stats["indexed"] += 1
else:
stats["failed"] += 1
except Exception as e:
logger.error("Failed to index %s: %s", file_path, e)
stats["failed"] += 1
# Build dependency graph
self._build_dependency_graph(conn)
conn.commit()
logger.info(
"Indexing complete: %(indexed)d indexed, %(failed)d failed, %(skipped)d skipped",
stats,
)
return stats
async def index_changed(self) -> dict[str, int]:
"""Perform incremental index of only changed files.
Compares content hashes to detect changes.
Returns:
Dict with stats: {"indexed": int, "failed": int, "skipped": int}
"""
logger.info("Starting incremental codebase index")
files = self._find_python_files()
stats = {"indexed": 0, "failed": 0, "skipped": 0}
with self._get_conn() as conn:
for file_path in files:
try:
rel_path = str(file_path.relative_to(self.repo_path))
content = file_path.read_text(encoding="utf-8")
content_hash = self._compute_hash(content)
# Check if changed
existing = conn.execute(
"SELECT content_hash FROM codebase_index WHERE file_path = ?",
(rel_path,),
).fetchone()
if existing and existing["content_hash"] == content_hash:
stats["skipped"] += 1
continue
module = self._parse_module(file_path)
if module:
self._store_module(conn, module, content_hash)
stats["indexed"] += 1
else:
stats["failed"] += 1
except Exception as e:
logger.error("Failed to index %s: %s", file_path, e)
stats["failed"] += 1
# Rebuild dependency graph (some imports may have changed)
self._build_dependency_graph(conn)
conn.commit()
logger.info(
"Incremental indexing complete: %(indexed)d indexed, %(failed)d failed, %(skipped)d skipped",
stats,
)
return stats
async def get_summary(self, max_tokens: int = 4000) -> str:
"""Generate compressed codebase summary for LLM context.
Lists modules, their purposes, key classes/functions, and test coverage.
Keeps output under max_tokens (approximate).
Args:
max_tokens: Maximum approximate tokens for summary
Returns:
Summary string suitable for LLM context
"""
with self._get_conn() as conn:
rows = conn.execute(
"""
SELECT file_path, module_name, classes, functions, test_coverage, docstring
FROM codebase_index
ORDER BY module_name
"""
).fetchall()
lines = ["# Codebase Summary\n"]
lines.append(f"Total modules: {len(rows)}\n")
lines.append("---\n")
for row in rows:
module_name = row["module_name"]
file_path = row["file_path"]
docstring = row["docstring"]
test_coverage = row["test_coverage"]
lines.append(f"\n## {module_name}")
lines.append(f"File: `{file_path}`")
if test_coverage:
lines.append(f"Tests: `{test_coverage}`")
else:
lines.append("Tests: None")
if docstring:
# Take first line of docstring
first_line = docstring.split("\n")[0][:100]
lines.append(f"Purpose: {first_line}")
# Classes
classes = json.loads(row["classes"])
if classes:
lines.append("Classes:")
for cls in classes[:5]: # Limit to 5 classes
methods = [m["name"] for m in cls["methods"][:3]]
method_str = ", ".join(methods) + ("..." if len(cls["methods"]) > 3 else "")
lines.append(f" - {cls['name']}({method_str})")
if len(classes) > 5:
lines.append(f" ... and {len(classes) - 5} more")
# Functions
functions = json.loads(row["functions"])
if functions:
func_names = [f["name"] for f in functions[:5]]
func_str = ", ".join(func_names)
if len(functions) > 5:
func_str += f"... and {len(functions) - 5} more"
lines.append(f"Functions: {func_str}")
lines.append("")
summary = "\n".join(lines)
# Rough token estimation (1 token ≈ 4 characters)
if len(summary) > max_tokens * 4:
# Truncate with note
summary = summary[:max_tokens * 4]
summary += "\n\n[Summary truncated due to length]"
return summary
async def get_relevant_files(self, task_description: str, limit: int = 5) -> list[str]:
"""Find files relevant to a task description.
Uses keyword matching and import relationships. In Phase 2,
this will use semantic search with vector embeddings.
Args:
task_description: Natural language description of the task
limit: Maximum number of files to return
Returns:
List of file paths sorted by relevance
"""
# Simple keyword extraction for now
keywords = set(task_description.lower().split())
# Remove common words
keywords -= {"the", "a", "an", "to", "in", "on", "at", "for", "with", "and", "or", "of", "is", "are"}
with self._get_conn() as conn:
rows = conn.execute(
"""
SELECT file_path, module_name, classes, functions, docstring, test_coverage
FROM codebase_index
"""
).fetchall()
scored_files = []
for row in rows:
score = 0
file_path = row["file_path"].lower()
module_name = row["module_name"].lower()
docstring = (row["docstring"] or "").lower()
classes = json.loads(row["classes"])
functions = json.loads(row["functions"])
# Score based on keyword matches
for keyword in keywords:
if keyword in file_path:
score += 3
if keyword in module_name:
score += 2
if keyword in docstring:
score += 2
# Check class/function names
for cls in classes:
if keyword in cls["name"].lower():
score += 2
for method in cls["methods"]:
if keyword in method["name"].lower():
score += 1
for func in functions:
if keyword in func["name"].lower():
score += 1
# Boost files with test coverage (only if already matched)
if score > 0 and row["test_coverage"]:
score += 1
if score > 0:
scored_files.append((score, row["file_path"]))
# Sort by score descending, return top N
scored_files.sort(reverse=True, key=lambda x: x[0])
return [f[1] for f in scored_files[:limit]]
async def get_dependency_chain(self, file_path: str) -> list[str]:
"""Get all files that import the given file.
Useful for understanding blast radius of changes.
Args:
file_path: Path to file (relative to repo root)
Returns:
List of file paths that import this file
"""
with self._get_conn() as conn:
rows = conn.execute(
"""
SELECT source_file FROM dependency_graph
WHERE target_file = ?
""",
(file_path,),
).fetchall()
return [row["source_file"] for row in rows]
async def has_test_coverage(self, file_path: str) -> bool:
"""Check if a file has corresponding test coverage.
Args:
file_path: Path to file (relative to repo root)
Returns:
True if test file exists, False otherwise
"""
with self._get_conn() as conn:
row = conn.execute(
"SELECT test_coverage FROM codebase_index WHERE file_path = ?",
(file_path,),
).fetchone()
return row is not None and row["test_coverage"] is not None
async def get_module_info(self, file_path: str) -> Optional[ModuleInfo]:
"""Get detailed info for a specific module.
Args:
file_path: Path to file (relative to repo root)
Returns:
ModuleInfo or None if not indexed
"""
with self._get_conn() as conn:
row = conn.execute(
"""
SELECT file_path, module_name, classes, functions, imports,
test_coverage, docstring
FROM codebase_index
WHERE file_path = ?
""",
(file_path,),
).fetchone()
if not row:
return None
# Parse classes - convert dict methods to FunctionInfo objects
classes_data = json.loads(row["classes"])
classes = []
for cls_data in classes_data:
methods = [FunctionInfo(**m) for m in cls_data.get("methods", [])]
cls_info = ClassInfo(
name=cls_data["name"],
methods=methods,
docstring=cls_data.get("docstring"),
line_number=cls_data.get("line_number", 0),
bases=cls_data.get("bases", []),
)
classes.append(cls_info)
# Parse functions
functions_data = json.loads(row["functions"])
functions = [FunctionInfo(**f) for f in functions_data]
return ModuleInfo(
file_path=row["file_path"],
module_name=row["module_name"],
classes=classes,
functions=functions,
imports=json.loads(row["imports"]),
docstring=row["docstring"],
test_coverage=row["test_coverage"],
)

505
src/self_coding/git_safety.py
View File

@@ -1,505 +0,0 @@
"""Git Safety Layer — Atomic git operations with rollback capability.
All self-modifications happen on feature branches. Only merge to main after
full test suite passes. Snapshots enable rollback on failure.
"""
from __future__ import annotations
import asyncio
import hashlib
import logging
import subprocess
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class Snapshot:
"""Immutable snapshot of repository state before modification.
Attributes:
commit_hash: Git commit hash at snapshot time
branch: Current branch name
timestamp: When snapshot was taken
test_status: Whether tests were passing at snapshot time
test_output: Pytest output from test run
clean: Whether working directory was clean
"""
commit_hash: str
branch: str
timestamp: datetime
test_status: bool
test_output: str
clean: bool
class GitSafetyError(Exception):
"""Base exception for git safety operations."""
pass
class GitNotRepositoryError(GitSafetyError):
"""Raised when operation is attempted outside a git repository."""
pass
class GitDirtyWorkingDirectoryError(GitSafetyError):
"""Raised when working directory is not clean and clean_required=True."""
pass
class GitOperationError(GitSafetyError):
"""Raised when a git operation fails."""
pass
class GitSafety:
"""Safe git operations for self-modification workflows.
All operations are atomic and support rollback. Self-modifications happen
on feature branches named 'timmy/self-edit/{timestamp}'. Only merged to
main after tests pass.
Usage:
safety = GitSafety(repo_path="/path/to/repo")
# Take snapshot before changes
snapshot = await safety.snapshot()
# Create feature branch
branch = await safety.create_branch(f"timmy/self-edit/{timestamp}")
# Make changes, commit them
await safety.commit("Add error handling", ["src/file.py"])
# Run tests, merge if pass
if tests_pass:
await safety.merge_to_main(branch)
else:
await safety.rollback(snapshot)
"""
def __init__(
self,
repo_path: Optional[str | Path] = None,
main_branch: str = "main",
test_command: str = "python -m pytest --tb=short -q",
) -> None:
"""Initialize GitSafety with repository path.
Args:
repo_path: Path to git repository. Defaults to current working directory.
main_branch: Name of main branch (main, master, etc.)
test_command: Command to run tests for snapshot validation
"""
self.repo_path = Path(repo_path).resolve() if repo_path else Path.cwd()
self.main_branch = main_branch
self.test_command = test_command
self._verify_git_repo()
logger.info("GitSafety initialized for %s", self.repo_path)
def _verify_git_repo(self) -> None:
"""Verify that repo_path is a git repository."""
git_dir = self.repo_path / ".git"
if not git_dir.exists():
raise GitNotRepositoryError(
f"{self.repo_path} is not a git repository"
)
async def _run_git(
self,
*args: str,
check: bool = True,
capture_output: bool = True,
timeout: float = 30.0,
) -> subprocess.CompletedProcess:
"""Run a git command asynchronously.
Args:
*args: Git command arguments
check: Whether to raise on non-zero exit
capture_output: Whether to capture stdout/stderr
timeout: Maximum time to wait for command
Returns:
CompletedProcess with returncode, stdout, stderr
Raises:
GitOperationError: If git command fails and check=True
"""
cmd = ["git", *args]
logger.debug("Running: %s", " ".join(cmd))
try:
proc = await asyncio.create_subprocess_exec(
*cmd,
cwd=self.repo_path,
stdout=asyncio.subprocess.PIPE if capture_output else None,
stderr=asyncio.subprocess.PIPE if capture_output else None,
)
stdout, stderr = await asyncio.wait_for(
proc.communicate(),
timeout=timeout,
)
result = subprocess.CompletedProcess(
args=cmd,
returncode=proc.returncode or 0,
stdout=stdout.decode() if stdout else "",
stderr=stderr.decode() if stderr else "",
)
if check and result.returncode != 0:
raise GitOperationError(
f"Git command failed: {' '.join(args)}\n"
f"stdout: {result.stdout}\nstderr: {result.stderr}"
)
return result
except asyncio.TimeoutError as e:
proc.kill()
raise GitOperationError(f"Git command timed out after {timeout}s: {' '.join(args)}") from e
async def _run_shell(
self,
command: str,
timeout: float = 120.0,
) -> subprocess.CompletedProcess:
"""Run a shell command asynchronously.
Args:
command: Shell command to run
timeout: Maximum time to wait
Returns:
CompletedProcess with returncode, stdout, stderr
"""
logger.debug("Running shell: %s", command)
proc = await asyncio.create_subprocess_shell(
command,
cwd=self.repo_path,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, stderr = await asyncio.wait_for(
proc.communicate(),
timeout=timeout,
)
return subprocess.CompletedProcess(
args=command,
returncode=proc.returncode or 0,
stdout=stdout.decode(),
stderr=stderr.decode(),
)
async def is_clean(self) -> bool:
"""Check if working directory is clean (no uncommitted changes).
Returns:
True if clean, False if there are uncommitted changes
"""
result = await self._run_git("status", "--porcelain", check=False)
return result.stdout.strip() == ""
async def get_current_branch(self) -> str:
"""Get current git branch name.
Returns:
Current branch name
"""
result = await self._run_git("branch", "--show-current")
return result.stdout.strip()
async def get_current_commit(self) -> str:
"""Get current commit hash.
Returns:
Full commit hash
"""
result = await self._run_git("rev-parse", "HEAD")
return result.stdout.strip()
async def _run_tests(self) -> tuple[bool, str]:
"""Run test suite and return results.
Returns:
Tuple of (all_passed, test_output)
"""
logger.info("Running tests: %s", self.test_command)
result = await self._run_shell(self.test_command, timeout=300.0)
passed = result.returncode == 0
output = result.stdout + "\n" + result.stderr
if passed:
logger.info("Tests passed")
else:
logger.warning("Tests failed with returncode %d", result.returncode)
return passed, output
async def snapshot(self, run_tests: bool = True) -> Snapshot:
"""Take a snapshot of current repository state.
Captures commit hash, branch, test status. Used for rollback if
modifications fail.
Args:
run_tests: Whether to run tests as part of snapshot
Returns:
Snapshot object with current state
Raises:
GitOperationError: If git operations fail
"""
logger.info("Taking snapshot of repository state")
commit_hash = await self.get_current_commit()
branch = await self.get_current_branch()
clean = await self.is_clean()
timestamp = datetime.now(timezone.utc)
test_status = False
test_output = ""
if run_tests:
test_status, test_output = await self._run_tests()
else:
test_status = True # Assume OK if not running tests
test_output = "Tests skipped"
snapshot = Snapshot(
commit_hash=commit_hash,
branch=branch,
timestamp=timestamp,
test_status=test_status,
test_output=test_output,
clean=clean,
)
logger.info(
"Snapshot taken: %s@%s (clean=%s, tests=%s)",
branch,
commit_hash[:8],
clean,
test_status,
)
return snapshot
async def create_branch(self, name: str, base: Optional[str] = None) -> str:
"""Create and checkout a new feature branch.
Args:
name: Branch name (e.g., 'timmy/self-edit/20260226-143022')
base: Base branch to create from (defaults to main_branch)
Returns:
Name of created branch
Raises:
GitOperationError: If branch creation fails
"""
base = base or self.main_branch
# Ensure we're on base branch and it's up to date
await self._run_git("checkout", base)
await self._run_git("pull", "origin", base, check=False) # May fail if no remote
# Create and checkout new branch
await self._run_git("checkout", "-b", name)
logger.info("Created branch %s from %s", name, base)
return name
async def commit(
self,
message: str,
files: Optional[list[str | Path]] = None,
allow_empty: bool = False,
) -> str:
"""Commit changes to current branch.
Args:
message: Commit message
files: Specific files to commit (None = all changes)
allow_empty: Whether to allow empty commits
Returns:
Commit hash of new commit
Raises:
GitOperationError: If commit fails
"""
# Add files
if files:
for file_path in files:
full_path = self.repo_path / file_path
if not full_path.exists():
logger.warning("File does not exist: %s", file_path)
await self._run_git("add", str(file_path))
else:
await self._run_git("add", "-A")
# Check if there's anything to commit
if not allow_empty:
diff_result = await self._run_git(
"diff", "--cached", "--quiet", check=False
)
if diff_result.returncode == 0:
logger.warning("No changes to commit")
return await self.get_current_commit()
# Commit
commit_args = ["commit", "-m", message]
if allow_empty:
commit_args.append("--allow-empty")
await self._run_git(*commit_args)
commit_hash = await self.get_current_commit()
logger.info("Committed %s: %s", commit_hash[:8], message)
return commit_hash
async def get_diff(self, from_hash: str, to_hash: Optional[str] = None) -> str:
"""Get diff between commits.
Args:
from_hash: Starting commit hash (or Snapshot object hash)
to_hash: Ending commit hash (None = current)
Returns:
Git diff as string
"""
args = ["diff", from_hash]
if to_hash:
args.append(to_hash)
result = await self._run_git(*args)
return result.stdout
async def rollback(self, snapshot: Snapshot | str) -> str:
"""Rollback to a previous snapshot.
Hard resets to the snapshot commit and deletes any uncommitted changes.
Use with caution — this is destructive.
Args:
snapshot: Snapshot object or commit hash to rollback to
Returns:
Commit hash after rollback
Raises:
GitOperationError: If rollback fails
"""
if isinstance(snapshot, Snapshot):
target_hash = snapshot.commit_hash
target_branch = snapshot.branch
else:
target_hash = snapshot
target_branch = None
logger.warning("Rolling back to %s", target_hash[:8])
# Reset to target commit
await self._run_git("reset", "--hard", target_hash)
# Clean any untracked files
await self._run_git("clean", "-fd")
# If we know the original branch, switch back to it
if target_branch:
branch_exists = await self._run_git(
"branch", "--list", target_branch, check=False
)
if branch_exists.stdout.strip():
await self._run_git("checkout", target_branch)
logger.info("Switched back to branch %s", target_branch)
current = await self.get_current_commit()
logger.info("Rolled back to %s", current[:8])
return current
async def merge_to_main(
self,
branch: str,
require_tests: bool = True,
) -> str:
"""Merge a feature branch into main after tests pass.
Args:
branch: Feature branch to merge
require_tests: Whether to require tests to pass before merging
Returns:
Merge commit hash
Raises:
GitOperationError: If merge fails or tests don't pass
"""
logger.info("Preparing to merge %s into %s", branch, self.main_branch)
# Checkout the feature branch and run tests
await self._run_git("checkout", branch)
if require_tests:
passed, output = await self._run_tests()
if not passed:
raise GitOperationError(
f"Cannot merge {branch}: tests failed\n{output}"
)
# Checkout main and merge
await self._run_git("checkout", self.main_branch)
await self._run_git("merge", "--no-ff", "-m", f"Merge {branch}", branch)
# Optionally delete the feature branch
await self._run_git("branch", "-d", branch, check=False)
merge_hash = await self.get_current_commit()
logger.info("Merged %s into %s: %s", branch, self.main_branch, merge_hash[:8])
return merge_hash
async def get_modified_files(self, since_hash: Optional[str] = None) -> list[str]:
"""Get list of files modified since a commit.
Args:
since_hash: Commit to compare against (None = uncommitted changes)
Returns:
List of modified file paths
"""
if since_hash:
result = await self._run_git(
"diff", "--name-only", since_hash, "HEAD"
)
else:
result = await self._run_git(
"diff", "--name-only", "HEAD"
)
files = [f.strip() for f in result.stdout.split("\n") if f.strip()]
return files
async def stage_file(self, file_path: str | Path) -> None:
"""Stage a single file for commit.
Args:
file_path: Path to file relative to repo root
"""
await self._run_git("add", str(file_path))
logger.debug("Staged %s", file_path)

425
src/self_coding/modification_journal.py
View File

@@ -1,425 +0,0 @@
"""Modification Journal — Persistent log of self-modification attempts.
Tracks successes and failures so Timmy can learn from experience.
Supports semantic search for similar past attempts.
"""
from __future__ import annotations
import json
import logging
import sqlite3
from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum
from pathlib import Path
from typing import Optional
logger = logging.getLogger(__name__)
# Default database location
DEFAULT_DB_PATH = Path("data/self_coding.db")
class Outcome(str, Enum):
"""Possible outcomes of a modification attempt."""
SUCCESS = "success"
FAILURE = "failure"
ROLLBACK = "rollback"
@dataclass
class ModificationAttempt:
"""A single self-modification attempt.
Attributes:
id: Unique identifier (auto-generated by database)
timestamp: When the attempt was made
task_description: What was Timmy trying to do
approach: Strategy/approach planned
files_modified: List of file paths that were modified
diff: The actual git diff of changes
test_results: Pytest output
outcome: success, failure, or rollback
failure_analysis: LLM-generated analysis of why it failed
reflection: LLM-generated lessons learned
retry_count: Number of retry attempts
embedding: Vector embedding of task_description (for semantic search)
"""
task_description: str
approach: str = ""
files_modified: list[str] = field(default_factory=list)
diff: str = ""
test_results: str = ""
outcome: Outcome = Outcome.FAILURE
failure_analysis: str = ""
reflection: str = ""
retry_count: int = 0
id: Optional[int] = None
timestamp: Optional[datetime] = None
embedding: Optional[bytes] = None
class ModificationJournal:
"""Persistent log of self-modification attempts.
Before any self-modification, Timmy should query the journal for
similar past attempts and include relevant ones in the LLM context.
Usage:
journal = ModificationJournal()
# Log an attempt
attempt = ModificationAttempt(
task_description="Add error handling",
files_modified=["src/app.py"],
outcome=Outcome.SUCCESS,
)
await journal.log_attempt(attempt)
# Find similar past attempts
similar = await journal.find_similar("Add error handling to endpoints")
# Get success metrics
metrics = await journal.get_success_rate()
"""
def __init__(
self,
db_path: Optional[str | Path] = None,
) -> None:
"""Initialize ModificationJournal.
Args:
db_path: SQLite database path. Defaults to data/self_coding.db
"""
self.db_path = Path(db_path) if db_path else DEFAULT_DB_PATH
self._ensure_schema()
logger.info("ModificationJournal initialized at %s", self.db_path)
def _get_conn(self) -> sqlite3.Connection:
"""Get database connection with schema ensured."""
self.db_path.parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(str(self.db_path))
conn.row_factory = sqlite3.Row
return conn
def _ensure_schema(self) -> None:
"""Create database tables if they don't exist."""
with self._get_conn() as conn:
conn.execute(
"""
CREATE TABLE IF NOT EXISTS modification_journal (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
task_description TEXT NOT NULL,
approach TEXT,
files_modified JSON,
diff TEXT,
test_results TEXT,
outcome TEXT CHECK(outcome IN ('success', 'failure', 'rollback')),
failure_analysis TEXT,
reflection TEXT,
retry_count INTEGER DEFAULT 0,
embedding BLOB
)
"""
)
# Create indexes for common queries
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_journal_outcome ON modification_journal(outcome)"
)
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_journal_timestamp ON modification_journal(timestamp)"
)
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_journal_task ON modification_journal(task_description)"
)
conn.commit()
async def log_attempt(self, attempt: ModificationAttempt) -> int:
"""Log a modification attempt to the journal.
Args:
attempt: The modification attempt to log
Returns:
ID of the logged entry
"""
with self._get_conn() as conn:
cursor = conn.execute(
"""
INSERT INTO modification_journal
(task_description, approach, files_modified, diff, test_results,
outcome, failure_analysis, reflection, retry_count, embedding)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
attempt.task_description,
attempt.approach,
json.dumps(attempt.files_modified),
attempt.diff,
attempt.test_results,
attempt.outcome.value,
attempt.failure_analysis,
attempt.reflection,
attempt.retry_count,
attempt.embedding,
),
)
conn.commit()
attempt_id = cursor.lastrowid
logger.info(
"Logged modification attempt %d: %s (%s)",
attempt_id,
attempt.task_description[:50],
attempt.outcome.value,
)
return attempt_id
async def find_similar(
self,
task_description: str,
limit: int = 5,
include_outcomes: Optional[list[Outcome]] = None,
) -> list[ModificationAttempt]:
"""Find similar past modification attempts.
Uses keyword matching for now. In Phase 2, will use vector embeddings
for semantic search.
Args:
task_description: Task to find similar attempts for
limit: Maximum number of results
include_outcomes: Filter by outcomes (None = all)
Returns:
List of similar modification attempts
"""
# Extract keywords from task description
keywords = set(task_description.lower().split())
keywords -= {"the", "a", "an", "to", "in", "on", "at", "for", "with", "and", "or", "of", "is", "are"}
with self._get_conn() as conn:
# Build query
if include_outcomes:
outcome_filter = "AND outcome IN ({})".format(
",".join("?" * len(include_outcomes))
)
outcome_values = [o.value for o in include_outcomes]
else:
outcome_filter = ""
outcome_values = []
rows = conn.execute(
f"""
SELECT id, timestamp, task_description, approach, files_modified,
diff, test_results, outcome, failure_analysis, reflection,
retry_count
FROM modification_journal
WHERE 1=1 {outcome_filter}
ORDER BY timestamp DESC
LIMIT ?
""",
outcome_values + [limit * 3], # Get more for scoring
).fetchall()
# Score by keyword match
scored = []
for row in rows:
score = 0
task = row["task_description"].lower()
approach = (row["approach"] or "").lower()
for kw in keywords:
if kw in task:
score += 3
if kw in approach:
score += 1
# Boost recent attempts (only if already matched)
if score > 0:
timestamp = datetime.fromisoformat(row["timestamp"])
if timestamp.tzinfo is None:
timestamp = timestamp.replace(tzinfo=timezone.utc)
age_days = (datetime.now(timezone.utc) - timestamp).days
if age_days < 7:
score += 2
elif age_days < 30:
score += 1
if score > 0:
scored.append((score, row))
# Sort by score, take top N
scored.sort(reverse=True, key=lambda x: x[0])
top_rows = scored[:limit]
# Convert to ModificationAttempt objects
return [self._row_to_attempt(row) for _, row in top_rows]
async def get_success_rate(self) -> dict[str, float]:
"""Get success rate metrics.
Returns:
Dict with overall and per-category success rates:
{
"overall": float, # 0.0 to 1.0
"success": int, # count
"failure": int, # count
"rollback": int, # count
"total": int, # total attempts
}
"""
with self._get_conn() as conn:
rows = conn.execute(
"""
SELECT outcome, COUNT(*) as count
FROM modification_journal
GROUP BY outcome
"""
).fetchall()
counts = {row["outcome"]: row["count"] for row in rows}
success = counts.get("success", 0)
failure = counts.get("failure", 0)
rollback = counts.get("rollback", 0)
total = success + failure + rollback
overall = success / total if total > 0 else 0.0
return {
"overall": overall,
"success": success,
"failure": failure,
"rollback": rollback,
"total": total,
}
async def get_recent_failures(self, limit: int = 10) -> list[ModificationAttempt]:
"""Get recent failed attempts with their analyses.
Args:
limit: Maximum number of failures to return
Returns:
List of failed modification attempts
"""
with self._get_conn() as conn:
rows = conn.execute(
"""
SELECT id, timestamp, task_description, approach, files_modified,
diff, test_results, outcome, failure_analysis, reflection,
retry_count
FROM modification_journal
WHERE outcome IN ('failure', 'rollback')
ORDER BY timestamp DESC
LIMIT ?
""",
(limit,),
).fetchall()
return [self._row_to_attempt(row) for row in rows]
async def get_by_id(self, attempt_id: int) -> Optional[ModificationAttempt]:
"""Get a specific modification attempt by ID.
Args:
attempt_id: ID of the attempt
Returns:
ModificationAttempt or None if not found
"""
with self._get_conn() as conn:
row = conn.execute(
"""
SELECT id, timestamp, task_description, approach, files_modified,
diff, test_results, outcome, failure_analysis, reflection,
retry_count
FROM modification_journal
WHERE id = ?
""",
(attempt_id,),
).fetchone()
if not row:
return None
return self._row_to_attempt(row)
async def update_reflection(self, attempt_id: int, reflection: str) -> bool:
"""Update the reflection for a modification attempt.
Args:
attempt_id: ID of the attempt
reflection: New reflection text
Returns:
True if updated, False if not found
"""
with self._get_conn() as conn:
cursor = conn.execute(
"""
UPDATE modification_journal
SET reflection = ?
WHERE id = ?
""",
(reflection, attempt_id),
)
conn.commit()
if cursor.rowcount > 0:
logger.info("Updated reflection for attempt %d", attempt_id)
return True
return False
async def get_attempts_for_file(
self,
file_path: str,
limit: int = 10,
) -> list[ModificationAttempt]:
"""Get all attempts that modified a specific file.
Args:
file_path: Path to file (relative to repo root)
limit: Maximum number of attempts
Returns:
List of modification attempts affecting this file
"""
with self._get_conn() as conn:
# Try exact match first, then partial match
rows = conn.execute(
"""
SELECT id, timestamp, task_description, approach, files_modified,
diff, test_results, outcome, failure_analysis, reflection,
retry_count
FROM modification_journal
WHERE files_modified LIKE ? OR files_modified LIKE ?
ORDER BY timestamp DESC
LIMIT ?
""",
(f'%"{file_path}"%', f'%{file_path}%', limit),
).fetchall()
return [self._row_to_attempt(row) for row in rows]
def _row_to_attempt(self, row: sqlite3.Row) -> ModificationAttempt:
"""Convert a database row to ModificationAttempt."""
return ModificationAttempt(
id=row["id"],
timestamp=datetime.fromisoformat(row["timestamp"]),
task_description=row["task_description"],
approach=row["approach"] or "",
files_modified=json.loads(row["files_modified"] or "[]"),
diff=row["diff"] or "",
test_results=row["test_results"] or "",
outcome=Outcome(row["outcome"]),
failure_analysis=row["failure_analysis"] or "",
reflection=row["reflection"] or "",
retry_count=row["retry_count"] or 0,
)

259
src/self_coding/reflection.py
View File

@@ -1,259 +0,0 @@
"""Reflection Service — Generate lessons learned from modification attempts.
After every self-modification (success or failure), the Reflection Service
prompts an LLM to analyze the attempt and extract actionable insights.
"""
from __future__ import annotations
import logging
from typing import Optional
from self_coding.modification_journal import ModificationAttempt, Outcome
logger = logging.getLogger(__name__)
REFLECTION_SYSTEM_PROMPT = """You are a software engineering mentor analyzing a self-modification attempt.
Your goal is to provide constructive, specific feedback that helps improve future attempts.
Focus on patterns and principles rather than one-off issues.
Be concise but insightful. Maximum 300 words."""
REFLECTION_PROMPT_TEMPLATE = """A software agent just attempted to modify its own source code.
Task: {task_description}
Approach: {approach}
Files modified: {files_modified}
Outcome: {outcome}
Test results: {test_results}
{failure_section}
Reflect on this attempt:
1. What went well? (Be specific about techniques or strategies)
2. What could be improved? (Focus on process, not just the code)
3. What would you do differently next time?
4. What general lesson can be extracted for future similar tasks?
Provide your reflection in a structured format:
**What went well:**
[Your analysis]
**What could be improved:**
[Your analysis]
**Next time:**
[Specific actionable change]
**General lesson:**
[Extracted principle for similar tasks]"""
class ReflectionService:
"""Generates reflections on self-modification attempts.
Uses an LLM to analyze attempts and extract lessons learned.
Stores reflections in the Modification Journal for future reference.
Usage:
from self_coding.reflection import ReflectionService
from timmy.cascade_adapter import TimmyCascadeAdapter
adapter = TimmyCascadeAdapter()
reflection_service = ReflectionService(llm_adapter=adapter)
# After a modification attempt
reflection_text = await reflection_service.reflect_on_attempt(attempt)
# Store in journal
await journal.update_reflection(attempt_id, reflection_text)
"""
def __init__(
self,
llm_adapter: Optional[object] = None,
model_preference: str = "fast", # "fast" or "quality"
) -> None:
"""Initialize ReflectionService.
Args:
llm_adapter: LLM adapter (e.g., TimmyCascadeAdapter)
model_preference: "fast" for quick reflections, "quality" for deeper analysis
"""
self.llm_adapter = llm_adapter
self.model_preference = model_preference
logger.info("ReflectionService initialized")
async def reflect_on_attempt(self, attempt: ModificationAttempt) -> str:
"""Generate a reflection on a modification attempt.
Args:
attempt: The modification attempt to reflect on
Returns:
Reflection text (structured markdown)
"""
# Build the prompt
failure_section = ""
if attempt.outcome == Outcome.FAILURE and attempt.failure_analysis:
failure_section = f"\nFailure analysis: {attempt.failure_analysis}"
prompt = REFLECTION_PROMPT_TEMPLATE.format(
task_description=attempt.task_description,
approach=attempt.approach or "(No approach documented)",
files_modified=", ".join(attempt.files_modified) if attempt.files_modified else "(No files modified)",
outcome=attempt.outcome.value.upper(),
test_results=attempt.test_results[:500] if attempt.test_results else "(No test results)",
failure_section=failure_section,
)
# Call LLM if available
if self.llm_adapter:
try:
response = await self.llm_adapter.chat(
message=prompt,
context=REFLECTION_SYSTEM_PROMPT,
)
reflection = response.content.strip()
logger.info("Generated reflection for attempt (via %s)",
response.provider_used)
return reflection
except Exception as e:
logger.error("LLM reflection failed: %s", e)
return self._generate_fallback_reflection(attempt)
else:
# No LLM available, use fallback
return self._generate_fallback_reflection(attempt)
def _generate_fallback_reflection(self, attempt: ModificationAttempt) -> str:
"""Generate a basic reflection without LLM.
Used when no LLM adapter is available or LLM call fails.
Args:
attempt: The modification attempt
Returns:
Basic reflection text
"""
if attempt.outcome == Outcome.SUCCESS:
return f"""**What went well:**
Successfully completed: {attempt.task_description}
Files modified: {', '.join(attempt.files_modified) if attempt.files_modified else 'N/A'}
**What could be improved:**
Document the approach taken for future reference.
**Next time:**
Use the same pattern for similar tasks.
**General lesson:**
Modifications to {', '.join(attempt.files_modified) if attempt.files_modified else 'these files'} should include proper test coverage."""
elif attempt.outcome == Outcome.FAILURE:
return f"""**What went well:**
Attempted: {attempt.task_description}
**What could be improved:**
The modification failed after {attempt.retry_count} retries.
{attempt.failure_analysis if attempt.failure_analysis else 'Failure reason not documented.'}
**Next time:**
Consider breaking the task into smaller steps.
Validate approach with simpler test case first.
**General lesson:**
Changes affecting {', '.join(attempt.files_modified) if attempt.files_modified else 'multiple files'} require careful dependency analysis."""
else: # ROLLBACK
return f"""**What went well:**
Recognized failure and rolled back to maintain stability.
**What could be improved:**
Early detection of issues before full implementation.
**Next time:**
Run tests more frequently during development.
Use smaller incremental commits.
**General lesson:**
Rollback is preferable to shipping broken code."""
async def reflect_with_context(
self,
attempt: ModificationAttempt,
similar_attempts: list[ModificationAttempt],
) -> str:
"""Generate reflection with context from similar past attempts.
Includes relevant past reflections to build cumulative learning.
Args:
attempt: The current modification attempt
similar_attempts: Similar past attempts (with reflections)
Returns:
Reflection text incorporating past learnings
"""
# Build context from similar attempts
context_parts = []
for past in similar_attempts[:3]: # Top 3 similar
if past.reflection:
context_parts.append(
f"Past similar task ({past.outcome.value}):\n"
f"Task: {past.task_description}\n"
f"Lesson: {past.reflection[:200]}..."
)
context = "\n\n".join(context_parts)
# Build enhanced prompt
failure_section = ""
if attempt.outcome == Outcome.FAILURE and attempt.failure_analysis:
failure_section = f"\nFailure analysis: {attempt.failure_analysis}"
enhanced_prompt = f"""A software agent just attempted to modify its own source code.
Task: {attempt.task_description}
Approach: {attempt.approach or "(No approach documented)"}
Files modified: {', '.join(attempt.files_modified) if attempt.files_modified else "(No files modified)"}
Outcome: {attempt.outcome.value.upper()}
Test results: {attempt.test_results[:500] if attempt.test_results else "(No test results)"}
{failure_section}
---
Relevant past attempts:
{context if context else "(No similar past attempts)"}
---
Given this history, reflect on the current attempt:
1. What went well?
2. What could be improved?
3. How does this compare to past similar attempts?
4. What pattern or principle should guide future similar tasks?
Provide your reflection in a structured format:
**What went well:**
**What could be improved:**
**Comparison to past attempts:**
**Guiding principle:**"""
if self.llm_adapter:
try:
response = await self.llm_adapter.chat(
message=enhanced_prompt,
context=REFLECTION_SYSTEM_PROMPT,
)
return response.content.strip()
except Exception as e:
logger.error("LLM reflection with context failed: %s", e)
return await self.reflect_on_attempt(attempt)
else:
return await self.reflect_on_attempt(attempt)

1
src/self_coding/self_modify/__init__.py
View File

@@ -1 +0,0 @@
"""Self-Modify — Runtime self-modification with safety constraints."""

134
src/self_coding/self_modify/cli.py
View File

@@ -1,134 +0,0 @@
"""CLI for self-modification — run from the terminal.
Usage:
self-modify run "Add a docstring to src/timmy/prompts.py" --file src/timmy/prompts.py
self-modify run "Fix the bug in config" --dry-run
self-modify run "Add logging" --backend anthropic --autonomous
self-modify status
"""
import logging
import os
from typing import Optional
import typer
from rich.console import Console
from rich.panel import Panel
console = Console()
app = typer.Typer(help="Timmy self-modify — edit code, run tests, commit")
@app.command()
def run(
instruction: str = typer.Argument(..., help="What to change (natural language)"),
file: Optional[list[str]] = typer.Option(None, "--file", "-f", help="Target file(s) to modify"),
dry_run: bool = typer.Option(False, "--dry-run", "-n", help="Generate edits but don't write"),
retries: int = typer.Option(2, "--retries", "-r", help="Max retry attempts on test failure"),
backend: Optional[str] = typer.Option(None, "--backend", "-b", help="LLM backend: ollama, anthropic, auto"),
autonomous: bool = typer.Option(False, "--autonomous", "-a", help="Enable autonomous self-correction"),
max_cycles: int = typer.Option(3, "--max-cycles", help="Max autonomous correction cycles"),
branch: bool = typer.Option(False, "--branch", help="Create a git branch (off by default to avoid container restarts)"),
speak: bool = typer.Option(False, "--speak", "-s", help="Speak the result via TTS"),
):
"""Run the self-modification loop."""
# Force enable for CLI usage
os.environ["SELF_MODIFY_ENABLED"] = "true"
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)-8s %(name)s -- %(message)s",
datefmt="%H:%M:%S",
)
# Skip branch creation unless explicitly requested
if not branch:
os.environ["SELF_MODIFY_SKIP_BRANCH"] = "1"
from self_coding.self_modify.loop import SelfModifyLoop, ModifyRequest
target_files = list(file) if file else []
effective_backend = backend or os.environ.get("SELF_MODIFY_BACKEND", "auto")
console.print(Panel(
f"[bold]Instruction:[/bold] {instruction}\n"
f"[bold]Files:[/bold] {', '.join(target_files) or '(auto-detect)'}\n"
f"[bold]Backend:[/bold] {effective_backend}\n"
f"[bold]Autonomous:[/bold] {autonomous}\n"
f"[bold]Dry run:[/bold] {dry_run}\n"
f"[bold]Max retries:[/bold] {retries}",
title="Self-Modify",
border_style="cyan",
))
loop = SelfModifyLoop(
max_retries=retries,
backend=effective_backend,
autonomous=autonomous,
max_autonomous_cycles=max_cycles,
)
request = ModifyRequest(
instruction=instruction,
target_files=target_files,
dry_run=dry_run,
)
with console.status("[bold cyan]Running self-modification loop..."):
result = loop.run(request)
if result.report_path:
console.print(f"\n[dim]Report saved: {result.report_path}[/dim]\n")
if result.success:
console.print(Panel(
f"[green bold]SUCCESS[/green bold]\n\n"
f"Files changed: {', '.join(result.files_changed)}\n"
f"Tests passed: {result.test_passed}\n"
f"Commit: {result.commit_sha or 'none (dry run)'}\n"
f"Branch: {result.branch_name or 'current'}\n"
f"Attempts: {result.attempts}\n"
f"Autonomous cycles: {result.autonomous_cycles}",
title="Result",
border_style="green",
))
else:
console.print(Panel(
f"[red bold]FAILED[/red bold]\n\n"
f"Error: {result.error}\n"
f"Attempts: {result.attempts}\n"
f"Autonomous cycles: {result.autonomous_cycles}",
title="Result",
border_style="red",
))
raise typer.Exit(1)
if speak and result.success:
try:
from timmy_serve.voice_tts import voice_tts
if voice_tts.available:
voice_tts.speak_sync(
f"Code modification complete. "
f"{len(result.files_changed)} files changed. Tests passing."
)
except Exception:
pass
@app.command()
def status():
"""Show whether self-modification is enabled."""
from config import settings
enabled = settings.self_modify_enabled
color = "green" if enabled else "red"
console.print(f"Self-modification: [{color}]{'ENABLED' if enabled else 'DISABLED'}[/{color}]")
console.print(f"Max retries: {settings.self_modify_max_retries}")
console.print(f"Backend: {settings.self_modify_backend}")
console.print(f"Allowed dirs: {settings.self_modify_allowed_dirs}")
def main():
app()
if __name__ == "__main__":
main()

741
src/self_coding/self_modify/loop.py
View File

@@ -1,741 +0,0 @@
"""Self-modification loop — read source, generate edits, test, commit.
Orchestrates the full cycle for Timmy to modify its own codebase:
1. Create a working git branch
2. Read target source files
3. Send instruction + source to the LLM
4. Validate syntax before writing
5. Write edits to disk
6. Run pytest
7. On success -> git add + commit; on failure -> revert
8. On total failure -> diagnose from report, restart autonomously
Supports multiple LLM backends:
- "ollama" — local Ollama (default, sovereign)
- "anthropic" — Claude API via Anthropic SDK
- "auto" — try anthropic first (if key set), fall back to ollama
Reports are saved to data/self_modify_reports/ for debugging.
"""
from __future__ import annotations
import logging
import os
import re
import subprocess
import sys
import threading
import time
from dataclasses import dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional
from config import settings
logger = logging.getLogger(__name__)
# Project root — use settings.repo_root (works in Docker and local dev)
PROJECT_ROOT = Path(settings.repo_root)
# Reports directory
REPORTS_DIR = PROJECT_ROOT / "data" / "self_modify_reports"
# Only one self-modification at a time
_LOCK = threading.Lock()
# Maximum file size we'll send to the LLM (bytes)
_MAX_FILE_SIZE = 50_000
# Delimiter format the LLM is instructed to use
_FILE_BLOCK_RE = re.compile(
r"---\s*FILE:\s*(.+?)\s*---\n(.*?)---\s*END\s*FILE\s*---",
re.DOTALL,
)
# Backend type literal
BACKENDS = ("ollama", "anthropic", "auto")
@dataclass
class ModifyRequest:
"""A request to modify code."""
instruction: str
target_files: list[str] = field(default_factory=list)
dry_run: bool = False
@dataclass
class ModifyResult:
"""Result of a self-modification attempt."""
success: bool
files_changed: list[str] = field(default_factory=list)
test_passed: bool = False
commit_sha: Optional[str] = None
branch_name: Optional[str] = None
error: Optional[str] = None
llm_response: str = ""
attempts: int = 0
report_path: Optional[str] = None
autonomous_cycles: int = 0
class SelfModifyLoop:
"""Orchestrates the read -> edit -> test -> commit cycle.
Supports autonomous self-correction: when all retries fail, reads its own
failure report, diagnoses the root cause, and restarts with a corrected
instruction.
"""
def __init__(
self,
repo_path: Optional[Path] = None,
max_retries: Optional[int] = None,
backend: Optional[str] = None,
autonomous: bool = False,
max_autonomous_cycles: int = 3,
) -> None:
self._repo_path = repo_path or PROJECT_ROOT
self._max_retries = (
max_retries if max_retries is not None else settings.self_modify_max_retries
)
self._allowed_dirs = [
d.strip() for d in settings.self_modify_allowed_dirs.split(",") if d.strip()
]
self._run_id = f"{int(time.time())}"
self._attempt_reports: list[dict] = []
self._backend = backend or settings.self_modify_backend
self._autonomous = autonomous
self._max_autonomous_cycles = max_autonomous_cycles
# ── Public API ────────────────────────────────────────────────────────────
def run(self, request: ModifyRequest) -> ModifyResult:
"""Execute the full self-modification loop."""
if not settings.self_modify_enabled:
return ModifyResult(
success=False,
error="Self-modification is disabled. Set SELF_MODIFY_ENABLED=true.",
)
if not _LOCK.acquire(blocking=False):
return ModifyResult(
success=False,
error="Another self-modification is already running.",
)
try:
result = self._run_locked(request)
report_path = self._save_report(request, result)
result.report_path = str(report_path)
# Autonomous mode: if failed, diagnose and restart
if self._autonomous and not result.success and not request.dry_run:
result = self._autonomous_loop(request, result, report_path)
return result
finally:
_LOCK.release()
# ── Autonomous self-correction ─────────────────────────────────────────
def _autonomous_loop(
self, original_request: ModifyRequest, last_result: ModifyResult, last_report: Path
) -> ModifyResult:
"""Read the failure report, diagnose, and restart with a fix."""
for cycle in range(1, self._max_autonomous_cycles + 1):
logger.info("Autonomous cycle %d/%d", cycle, self._max_autonomous_cycles)
# Diagnose what went wrong
diagnosis = self._diagnose_failure(last_report)
if not diagnosis:
logger.warning("Could not diagnose failure, stopping autonomous loop")
last_result.autonomous_cycles = cycle
return last_result
logger.info("Diagnosis: %s", diagnosis[:200])
# Build a corrected instruction
corrected_instruction = (
f"{original_request.instruction}\n\n"
f"IMPORTANT CORRECTION from previous failure:\n{diagnosis}"
)
# Reset attempt reports for this cycle
self._attempt_reports = []
corrected_request = ModifyRequest(
instruction=corrected_instruction,
target_files=original_request.target_files,
dry_run=original_request.dry_run,
)
result = self._run_locked(corrected_request)
report_path = self._save_report(corrected_request, result)
result.report_path = str(report_path)
result.autonomous_cycles = cycle
if result.success:
logger.info("Autonomous cycle %d succeeded!", cycle)
return result
last_result = result
last_report = report_path
logger.warning("Autonomous loop exhausted after %d cycles", self._max_autonomous_cycles)
return last_result
def _diagnose_failure(self, report_path: Path) -> Optional[str]:
"""Read a failure report and produce a diagnosis + fix instruction.
Uses the best available LLM to analyze the report. This is the
'meta-reasoning' step — the agent reasoning about its own failures.
"""
try:
report_text = report_path.read_text(encoding="utf-8")
except Exception as exc:
logger.error("Could not read report %s: %s", report_path, exc)
return None
# Truncate to keep within context limits
if len(report_text) > 8000:
report_text = report_text[:8000] + "\n... (truncated)"
diagnosis_prompt = f"""You are a code debugging expert. Analyze this self-modification failure report and provide a concise diagnosis.
FAILURE REPORT:
{report_text}
Analyze the report and provide:
1. ROOT CAUSE: What specifically went wrong (syntax error, logic error, missing import, etc.)
2. FIX INSTRUCTIONS: Exact instructions for a code-generation LLM to avoid this mistake.
Be very specific — e.g. "Do NOT start the file with triple-quotes" or
"The em-dash character U+2014 must stay INSIDE a string literal, never outside one."
Keep your response under 500 words. Focus on actionable fix instructions."""
try:
raw = self._call_llm(diagnosis_prompt)
return raw.strip() if raw else None
except Exception as exc:
logger.error("Diagnosis LLM call failed: %s", exc)
return None
# ── Internal orchestration ────────────────────────────────────────────────
def _run_locked(self, request: ModifyRequest) -> ModifyResult:
branch_name = None
attempt = 0
# Skip branch creation — writing files triggers container restarts
# which kills the process mid-operation. Work on the current branch.
if not os.environ.get("SELF_MODIFY_SKIP_BRANCH"):
try:
branch_name = self._create_branch()
except Exception as exc:
logger.warning("Could not create branch: %s (continuing on current)", exc)
# Resolve target files
target_files = request.target_files or self._infer_target_files(
request.instruction
)
if not target_files:
return ModifyResult(
success=False,
error="No target files identified. Specify target_files or use more specific language.",
branch_name=branch_name,
)
# Validate paths
try:
self._validate_paths(target_files)
except ValueError as exc:
return ModifyResult(success=False, error=str(exc), branch_name=branch_name)
last_test_output = ""
last_llm_response = ""
last_syntax_errors: dict[str, str] = {}
while attempt <= self._max_retries:
attempt += 1
logger.info(
"Self-modify attempt %d/%d: %s",
attempt,
self._max_retries + 1,
request.instruction[:80],
)
# Read current contents
file_contents = self._read_files(target_files)
if not file_contents:
return ModifyResult(
success=False,
error="Could not read any target files.",
branch_name=branch_name,
attempts=attempt,
)
# Generate edits via LLM
try:
edits, llm_response = self._generate_edits(
request.instruction, file_contents,
prev_test_output=last_test_output if attempt > 1 else None,
prev_syntax_errors=last_syntax_errors if attempt > 1 else None,
)
last_llm_response = llm_response
except Exception as exc:
self._attempt_reports.append({
"attempt": attempt,
"phase": "llm_generation",
"error": str(exc),
})
return ModifyResult(
success=False,
error=f"LLM generation failed: {exc}",
branch_name=branch_name,
attempts=attempt,
)
if not edits:
self._attempt_reports.append({
"attempt": attempt,
"phase": "parse_edits",
"error": "No file edits parsed from LLM response",
"llm_response": llm_response,
})
return ModifyResult(
success=False,
error="LLM produced no file edits.",
llm_response=llm_response,
branch_name=branch_name,
attempts=attempt,
)
# Syntax validation — check BEFORE writing to disk
syntax_errors = self._validate_syntax(edits)
if syntax_errors:
last_syntax_errors = syntax_errors
error_summary = "; ".join(
f"{fp}: {err}" for fp, err in syntax_errors.items()
)
logger.warning("Syntax errors in LLM output: %s", error_summary)
self._attempt_reports.append({
"attempt": attempt,
"phase": "syntax_validation",
"error": error_summary,
"edits_content": {fp: content for fp, content in edits.items()},
"llm_response": llm_response,
})
# Don't write — go straight to retry
continue
last_syntax_errors = {}
if request.dry_run:
self._attempt_reports.append({
"attempt": attempt,
"phase": "dry_run",
"edits": {fp: content[:500] + "..." if len(content) > 500 else content
for fp, content in edits.items()},
"llm_response": llm_response,
})
return ModifyResult(
success=True,
files_changed=list(edits.keys()),
llm_response=llm_response,
branch_name=branch_name,
attempts=attempt,
)
# Write edits
written = self._write_files(edits)
# Run tests
test_passed, test_output = self._run_tests()
last_test_output = test_output
# Save per-attempt report
self._attempt_reports.append({
"attempt": attempt,
"phase": "complete",
"files_written": written,
"edits_content": {fp: content for fp, content in edits.items()},
"test_passed": test_passed,
"test_output": test_output,
"llm_response": llm_response,
})
if test_passed:
sha = self._git_commit(
f"self-modify: {request.instruction[:72]}", written
)
return ModifyResult(
success=True,
files_changed=written,
test_passed=True,
commit_sha=sha,
branch_name=branch_name,
llm_response=llm_response,
attempts=attempt,
)
# Tests failed — revert and maybe retry
logger.warning(
"Tests failed on attempt %d: %s", attempt, test_output[:200]
)
self._revert_files(written)
return ModifyResult(
success=False,
files_changed=[],
test_passed=False,
error=f"Tests failed after {attempt} attempt(s).",
llm_response=last_llm_response,
branch_name=branch_name,
attempts=attempt,
)
# ── Syntax validation ──────────────────────────────────────────────────
def _validate_syntax(self, edits: dict[str, str]) -> dict[str, str]:
"""Compile-check each .py file edit. Returns {path: error} for failures."""
errors: dict[str, str] = {}
for fp, content in edits.items():
if not fp.endswith(".py"):
continue
try:
compile(content, fp, "exec")
except SyntaxError as exc:
errors[fp] = f"line {exc.lineno}: {exc.msg}"
return errors
# ── Report saving ─────────────────────────────────────────────────────────
def _save_report(self, request: ModifyRequest, result: ModifyResult) -> Path:
"""Save a detailed report to data/self_modify_reports/."""
REPORTS_DIR.mkdir(parents=True, exist_ok=True)
ts = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
slug = re.sub(r"[^a-z0-9]+", "_", request.instruction[:40].lower()).strip("_")
report_file = REPORTS_DIR / f"{ts}_{slug}.md"
lines = [
f"# Self-Modify Report: {ts}",
"",
f"**Instruction:** {request.instruction[:200]}",
f"**Target files:** {', '.join(request.target_files) or '(auto-detected)'}",
f"**Dry run:** {request.dry_run}",
f"**Backend:** {self._backend}",
f"**Branch:** {result.branch_name or 'N/A'}",
f"**Result:** {'SUCCESS' if result.success else 'FAILED'}",
f"**Error:** {result.error or 'none'}",
f"**Commit:** {result.commit_sha or 'none'}",
f"**Attempts:** {result.attempts}",
f"**Autonomous cycles:** {result.autonomous_cycles}",
"",
]
for attempt_data in self._attempt_reports:
n = attempt_data.get("attempt", "?")
phase = attempt_data.get("phase", "?")
lines.append(f"## Attempt {n} -- {phase}")
lines.append("")
if "error" in attempt_data and attempt_data.get("phase") != "complete":
lines.append(f"**Error:** {attempt_data['error']}")
lines.append("")
if "llm_response" in attempt_data:
lines.append("### LLM Response")
lines.append("```")
lines.append(attempt_data["llm_response"])
lines.append("```")
lines.append("")
if "edits_content" in attempt_data:
lines.append("### Edits Written")
for fp, content in attempt_data["edits_content"].items():
lines.append(f"#### {fp}")
lines.append("```python")
lines.append(content)
lines.append("```")
lines.append("")
if "test_output" in attempt_data:
lines.append(f"### Test Result: {'PASSED' if attempt_data.get('test_passed') else 'FAILED'}")
lines.append("```")
lines.append(attempt_data["test_output"])
lines.append("```")
lines.append("")
report_text = "\n".join(lines)
report_file.write_text(report_text, encoding="utf-8")
logger.info("Report saved: %s", report_file)
return report_file
# ── Git helpers ───────────────────────────────────────────────────────────
def _create_branch(self) -> str:
"""Create and switch to a working branch."""
from creative.tools.git_tools import git_branch
branch_name = f"timmy/self-modify-{int(time.time())}"
git_branch(self._repo_path, create=branch_name, switch=branch_name)
logger.info("Created branch: %s", branch_name)
return branch_name
def _git_commit(self, message: str, files: list[str]) -> Optional[str]:
"""Stage files and commit."""
from creative.tools.git_tools import git_add, git_commit
try:
git_add(self._repo_path, paths=files)
result = git_commit(self._repo_path, message)
sha = result.get("sha")
logger.info("Committed %s: %s", sha[:8] if sha else "?", message)
return sha
except Exception as exc:
logger.error("Git commit failed: %s", exc)
return None
def _revert_files(self, file_paths: list[str]) -> None:
"""Restore files from git HEAD."""
for fp in file_paths:
try:
subprocess.run(
["git", "checkout", "HEAD", "--", fp],
cwd=self._repo_path,
capture_output=True,
timeout=10,
)
except Exception as exc:
logger.error("Failed to revert %s: %s", fp, exc)
# ── File I/O ──────────────────────────────────────────────────────────────
def _validate_paths(self, file_paths: list[str]) -> None:
"""Ensure all paths are within allowed directories."""
for fp in file_paths:
resolved = (self._repo_path / fp).resolve()
repo_resolved = self._repo_path.resolve()
if not str(resolved).startswith(str(repo_resolved)):
raise ValueError(f"Path escapes repository: {fp}")
rel = str(resolved.relative_to(repo_resolved))
if not any(rel.startswith(d) for d in self._allowed_dirs):
raise ValueError(
f"Path not in allowed directories ({self._allowed_dirs}): {fp}"
)
def _read_files(self, file_paths: list[str]) -> dict[str, str]:
"""Read file contents from disk."""
contents: dict[str, str] = {}
for fp in file_paths:
full = self._repo_path / fp
if not full.is_file():
logger.warning("File not found: %s", full)
continue
if full.stat().st_size > _MAX_FILE_SIZE:
logger.warning("File too large, skipping: %s", fp)
continue
try:
contents[fp] = full.read_text(encoding="utf-8")
except Exception as exc:
logger.warning("Could not read %s: %s", fp, exc)
return contents
def _write_files(self, edits: dict[str, str]) -> list[str]:
"""Write edited content to disk. Returns paths written."""
written: list[str] = []
for fp, content in edits.items():
full = self._repo_path / fp
full.parent.mkdir(parents=True, exist_ok=True)
full.write_text(content, encoding="utf-8")
written.append(fp)
logger.info("Wrote %d bytes to %s", len(content), fp)
return written
def _infer_target_files(self, instruction: str) -> list[str]:
"""Guess which files to modify from the instruction text."""
paths = re.findall(r"[\w/._-]+\.py", instruction)
if paths:
return paths
keyword_files = {
"config": ["src/config.py"],
"health": ["src/dashboard/routes/health.py"],
"swarm": ["src/swarm/coordinator.py"],
"voice": ["src/voice/nlu.py"],
"agent": ["src/timmy/agent.py"],
"tool": ["src/timmy/tools.py"],
"dashboard": ["src/dashboard/app.py"],
"prompt": ["src/timmy/prompts.py"],
}
instruction_lower = instruction.lower()
for keyword, files in keyword_files.items():
if keyword in instruction_lower:
return files
return []
# ── Test runner ───────────────────────────────────────────────────────────
def _run_tests(self) -> tuple[bool, str]:
"""Run the test suite. Returns (passed, output)."""
try:
result = subprocess.run(
[sys.executable, "-m", "pytest", "tests/", "-q", "--tb=short"],
capture_output=True,
text=True,
cwd=self._repo_path,
timeout=120,
)
output = (result.stdout + result.stderr).strip()
return result.returncode == 0, output
except subprocess.TimeoutExpired:
return False, "Tests timed out after 120s"
except Exception as exc:
return False, f"Failed to run tests: {exc}"
# ── Multi-backend LLM ─────────────────────────────────────────────────────
def _resolve_backend(self) -> str:
"""Resolve 'auto' backend to a concrete one."""
if self._backend == "auto":
api_key = os.environ.get("ANTHROPIC_API_KEY", "")
if api_key:
return "anthropic"
return "ollama"
return self._backend
def _call_llm(self, prompt: str) -> str:
"""Route a prompt to the configured LLM backend. Returns raw text."""
backend = self._resolve_backend()
if backend == "anthropic":
return self._call_anthropic(prompt)
else:
return self._call_ollama(prompt)
def _call_anthropic(self, prompt: str) -> str:
"""Call Claude via the Anthropic SDK."""
import anthropic
api_key = os.environ.get("ANTHROPIC_API_KEY", "")
if not api_key:
raise RuntimeError("ANTHROPIC_API_KEY not set — cannot use anthropic backend")
client = anthropic.Anthropic(api_key=api_key)
message = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=4096,
messages=[{"role": "user", "content": prompt}],
)
return message.content[0].text
def _call_ollama(self, prompt: str) -> str:
"""Call the local Ollama instance via Agno."""
from agno.agent import Agent
from agno.models.ollama import Ollama
agent = Agent(
name="SelfModify",
model=Ollama(id=settings.ollama_model, host=settings.ollama_url),
markdown=False,
)
run_result = agent.run(prompt, stream=False)
return run_result.content if hasattr(run_result, "content") else str(run_result)
# ── LLM interaction ───────────────────────────────────────────────────────
def _generate_edits(
self,
instruction: str,
file_contents: dict[str, str],
prev_test_output: Optional[str] = None,
prev_syntax_errors: Optional[dict[str, str]] = None,
) -> tuple[dict[str, str], str]:
"""Ask the LLM to generate file edits.
Returns (edits_dict, raw_llm_response).
"""
# Build the prompt
files_block = ""
for fp, content in file_contents.items():
files_block += f"\n<FILE path=\"{fp}\">\n{content}\n</FILE>\n"
retry_context = ""
if prev_test_output:
retry_context += f"""
PREVIOUS ATTEMPT FAILED with test errors:
<TEST_OUTPUT>
{prev_test_output[:2000]}
</TEST_OUTPUT>
Fix the issues shown above.
"""
if prev_syntax_errors:
errors_text = "\n".join(f" {fp}: {err}" for fp, err in prev_syntax_errors.items())
retry_context += f"""
PREVIOUS ATTEMPT HAD SYNTAX ERRORS (code was rejected before writing):
{errors_text}
You MUST produce syntactically valid Python. Run through the code mentally
and make sure all strings are properly terminated, all indentation is correct,
and there are no invalid characters outside of string literals.
"""
prompt = f"""You are a precise code modification agent. Edit source files according to the instruction.
INSTRUCTION: {instruction}
CURRENT FILES:
{files_block}
{retry_context}
OUTPUT FORMAT — wrap each modified file like this:
<MODIFIED path="filepath">
complete file content here
</MODIFIED>
CRITICAL RULES:
- Output the COMPLETE file content, not just changed lines
- Keep ALL existing functionality unless told to remove it
- The output must be syntactically valid Python — verify mentally before outputting
- Preserve all special characters (unicode, em-dashes, etc.) exactly as they appear in the original
- Do NOT wrap the file content in triple-quotes or markdown code fences
- Do NOT start the file content with \"\"\" — that would turn the code into a string literal
- Follow the existing code style
Generate the modified files now:"""
raw = self._call_llm(prompt)
# Parse <MODIFIED path="..."> ... </MODIFIED> blocks
edits = {}
xml_re = re.compile(
r'<MODIFIED\s+path=["\'](.+?)["\']\s*>\n?(.*?)</MODIFIED>',
re.DOTALL,
)
for match in xml_re.finditer(raw):
filepath = match.group(1).strip()
content = match.group(2)
# Strip trailing whitespace but keep a final newline
content = content.rstrip() + "\n"
edits[filepath] = content
# Fallback: try the old delimiter format
if not edits:
for match in _FILE_BLOCK_RE.finditer(raw):
filepath = match.group(1).strip()
content = match.group(2).rstrip() + "\n"
edits[filepath] = content
# Last resort: single file + code block
if not edits and len(file_contents) == 1:
only_path = next(iter(file_contents))
code_match = re.search(r"```(?:python)?\n(.*?)```", raw, re.DOTALL)
if code_match:
edits[only_path] = code_match.group(1).rstrip() + "\n"
return edits, raw

1
src/self_coding/self_tdd/__init__.py
View File

@@ -1 +0,0 @@
"""Self-TDD — Continuous test watchdog with regression alerting."""

71
src/self_coding/self_tdd/watchdog.py
View File

@@ -1,71 +0,0 @@
"""Self-TDD Watchdog — polls pytest on a schedule and reports regressions.
Run in a terminal alongside your normal dev work:
self-tdd watch
self-tdd watch --interval 30
The watchdog runs silently while tests pass. When a regression appears it
prints the full short-traceback output so you can see exactly what broke.
No files are modified; no commits are made. Ctrl-C to stop.
"""
import subprocess
import sys
import time
from datetime import datetime
from pathlib import Path
import typer
# Project root is three levels up from src/self_tdd/watchdog.py
PROJECT_ROOT = Path(__file__).parent.parent.parent
app = typer.Typer(help="Self-TDD watchdog — continuous test runner")
def _run_tests() -> tuple[bool, str]:
"""Run the test suite and return (passed, combined_output)."""
result = subprocess.run(
[sys.executable, "-m", "pytest", "tests/", "-q", "--tb=short"],
capture_output=True,
text=True,
cwd=PROJECT_ROOT,
timeout=60,
)
return result.returncode == 0, (result.stdout + result.stderr).strip()
@app.command()
def watch(
interval: int = typer.Option(60, "--interval", "-i", help="Seconds between test runs"),
) -> None:
"""Poll pytest continuously and print regressions as they appear."""
typer.echo(f"Self-TDD watchdog started — polling every {interval}s. Ctrl-C to stop.")
last_passed: bool | None = None
try:
while True:
passed, output = _run_tests()
stamp = datetime.now().strftime("%H:%M:%S")
if passed:
if last_passed is not True:
typer.secho(f"[{stamp}] All tests passing.", fg=typer.colors.GREEN)
else:
typer.secho(f"[{stamp}] Regression detected:", fg=typer.colors.RED)
typer.echo(output)
last_passed = passed
time.sleep(interval)
except KeyboardInterrupt:
typer.echo("\nWatchdog stopped.")
def main() -> None:
app()
if __name__ == "__main__":
main()

1
src/self_coding/upgrades/__init__.py
View File

@@ -1 +0,0 @@
"""Upgrades — System upgrade queue and execution pipeline."""

Some files were not shown because too many files have changed in this diff Show More