d8d976aa60
This commit implements six major features: 1. Event Log System (src/swarm/event_log.py) - SQLite-based audit trail for all swarm events - Task lifecycle tracking (created, assigned, completed, failed) - Agent lifecycle tracking (joined, left, status changes) - Integrated with coordinator for automatic logging - Dashboard page at /swarm/events 2. Lightning Ledger (src/lightning/ledger.py) - Transaction tracking for Lightning Network payments - Balance calculations (incoming, outgoing, net, available) - Integrated with payment_handler for automatic logging - Dashboard page at /lightning/ledger 3. Semantic Memory / Vector Store (src/memory/vector_store.py) - Embedding-based similarity search for Echo agent - Fallback to keyword matching if sentence-transformers unavailable - Personal facts storage and retrieval - Dashboard page at /memory 4. Cascade Router Integration (src/timmy/cascade_adapter.py) - Automatic LLM failover between providers (Ollama → AirLLM → API) - Circuit breaker pattern for failing providers - Metrics tracking per provider (latency, error rates) - Dashboard status page at /router/status 5. Self-Upgrade Approval Queue (src/upgrades/) - State machine for self-modifications: proposed → approved/rejected → applied/failed - Human approval required before applying changes - Git integration for branch management - Dashboard queue at /self-modify/queue 6. Real-Time Activity Feed (src/events/broadcaster.py) - WebSocket-based live activity streaming - Bridges event_log to dashboard clients - Activity panel on /swarm/live Tests: - 101 unit tests passing - 4 new E2E test files for Selenium testing - Run with: SELENIUM_UI=1 pytest tests/functional/ -v --headed Documentation: - 6 ADRs (017-022) documenting architecture decisions - Implementation summary in docs/IMPLEMENTATION_SUMMARY.md - Architecture diagram in docs/architecture-v2.md
221 lines
12 KiB
Markdown
221 lines
12 KiB
Markdown
# Timmy Time Architecture v2
|
|
|
|
## Overview
|
|
This document describes how the 6 new features integrate with the existing architecture.
|
|
|
|
## Architecture Diagram
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ DASHBOARD UI │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │
|
|
│ │ Event Log │ │ Ledger │ │ Memory │ │ Upgrade Queue │ │
|
|
│ │ /swarm/events│ │/lightning/ledger│ │ /memory │ │ /self-modify/queue│ │
|
|
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └────────┬─────────┘ │
|
|
│ │ │ │ │ │
|
|
│ ┌──────┴───────┐ ┌──────┴───────┐ ┌──────┴───────┐ ┌────────┴─────────┐ │
|
|
│ │ WebSocket │ │ │ │ │ │ Real-Time │ │
|
|
│ │ Activity │ │ │ │ │ │ Activity Feed │ │
|
|
│ │ Feed │ │ │ │ │ │ │ │
|
|
│ └──────┬───────┘ └──────────────┘ └──────────────┘ └──────────────────┘ │
|
|
└─────────┼───────────────────────────────────────────────────────────────────┘
|
|
│ WebSocket
|
|
┌─────────┼───────────────────────────────────────────────────────────────────┐
|
|
│ │ API LAYER │
|
|
│ ┌──────┴───────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │
|
|
│ │ Events │ │ Ledger │ │ Memory │ │ Self-Modify │ │
|
|
│ │ Routes │ │ Routes │ │ Routes │ │ Routes │ │
|
|
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └────────┬─────────┘ │
|
|
└─────────┼────────────────┼────────────────┼──────────────────┼─────────────┘
|
|
│ │ │ │
|
|
┌─────────┼────────────────┼────────────────┼──────────────────┼─────────────┐
|
|
│ │ CORE SERVICES │
|
|
│ │ │ │ │ │
|
|
│ ┌──────┴───────┐ ┌──────┴───────┐ ┌──────┴───────┐ ┌────────┴─────────┐ │
|
|
│ │ Event Log │ │ Ledger │ │Vector Store │ │ Self-Modify Loop │ │
|
|
│ │ Service │ │ Service │ │ (Echo) │ │ with Queue │ │
|
|
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └────────┬─────────┘ │
|
|
│ │ │ │ │ │
|
|
│ └────────────────┴────────────────┴──────────────────┘ │
|
|
│ │ │
|
|
│ ┌─────┴─────┐ │
|
|
│ │ SQLite DB │ │
|
|
│ │ swarm.db │ │
|
|
│ └───────────┘ │
|
|
│ │
|
|
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ CASCADE ROUTER (New) │ │
|
|
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌─────────────────────┐ │ │
|
|
│ │ │ Ollama │→ │ AirLLM │→ │ API │→ │ Metrics & Health │ │ │
|
|
│ │ │(local) │ │ (local) │ │(optional)│ │ Dashboard │ │ │
|
|
│ │ └──────────┘ └──────────┘ └──────────┘ └─────────────────────┘ │ │
|
|
│ └─────────────────────────────────────────────────────────────────────┘ │
|
|
│ │ │
|
|
│ ┌─────┴─────┐ │
|
|
│ │ Timmy │ │
|
|
│ │ Agent │ │
|
|
│ └───────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Data Flow
|
|
|
|
### 1. Event Log System
|
|
```
|
|
Coordinator Action → log_event() → SQLite event_log table
|
|
↓
|
|
WebSocket Broadcast (ADR-022)
|
|
↓
|
|
Dashboard Activity Feed
|
|
```
|
|
|
|
### 2. Lightning Ledger
|
|
```
|
|
Payment Handler → create_invoice_entry() → SQLite ledger table
|
|
↓
|
|
mark_settled()
|
|
↓
|
|
Dashboard /lightning/ledger
|
|
```
|
|
|
|
### 3. Semantic Memory
|
|
```
|
|
Conversation → store_memory() → SQLite memory_entries (with embedding)
|
|
↓
|
|
search_memories(query)
|
|
↓
|
|
Dashboard /memory
|
|
```
|
|
|
|
### 4. Self-Upgrade Queue
|
|
```
|
|
Self-Modify Loop → Propose Change → SQLite upgrades table (status: proposed)
|
|
↓
|
|
Dashboard Review
|
|
↓
|
|
Approve → Apply → Git Commit
|
|
or
|
|
Reject → Cleanup
|
|
```
|
|
|
|
### 5. Cascade Router
|
|
```
|
|
User Request → Cascade Router → Ollama (try)
|
|
↓ fail
|
|
AirLLM (fallback)
|
|
↓ fail
|
|
API Provider (optional)
|
|
↓
|
|
Metrics Tracking
|
|
↓
|
|
Dashboard /router/status
|
|
```
|
|
|
|
### 6. Real-Time Activity Feed
|
|
```
|
|
Event Logged → EventBroadcaster → ws_manager.broadcast()
|
|
↓
|
|
WebSocket Clients
|
|
↓
|
|
Dashboard Activity Panel
|
|
```
|
|
|
|
## Database Schema
|
|
|
|
### Tables
|
|
|
|
| Table | Purpose | Feature |
|
|
|-------|---------|---------|
|
|
| `tasks` | Task management | Existing |
|
|
| `agents` | Agent registry | Existing |
|
|
| `event_log` | Audit trail | **New - ADR-017** |
|
|
| `ledger` | Lightning payments | **New - ADR-018** |
|
|
| `memory_entries` | Semantic memory | **New - ADR-019** |
|
|
| `upgrades` | Self-mod queue | **New - ADR-021** |
|
|
| `provider_metrics` | LLM metrics | **New - ADR-020** |
|
|
|
|
## Integration Points
|
|
|
|
### Existing → New
|
|
|
|
| Existing Component | Integrates With | How |
|
|
|-------------------|-----------------|-----|
|
|
| `coordinator.py` | Event Log | Calls `log_event()` for all lifecycle events |
|
|
| `payment_handler.py` | Ledger | Creates entries on invoice/settlement |
|
|
| `self_modify/loop.py` | Upgrade Queue | Stops at proposal, waits for approval |
|
|
| `timmy/agent.py` | Cascade Router | Uses router instead of direct backends |
|
|
| `ws_manager/handler.py` | Activity Feed | Broadcasts events to clients |
|
|
|
|
### New → Existing
|
|
|
|
| New Component | Uses Existing | How |
|
|
|---------------|---------------|-----|
|
|
| Event Log | `coordinator.py` | Receives all coordinator actions |
|
|
| Ledger | `payment_handler.py` | Integrated into invoice lifecycle |
|
|
| Memory | Personas | Echo agent queries for context |
|
|
| Upgrade Queue | `self_modify/loop.py` | Controls when changes apply |
|
|
| Cascade Router | `timmy/agent.py` | Provides LLM abstraction |
|
|
| Activity Feed | `ws_manager/handler.py` | Uses WebSocket infrastructure |
|
|
|
|
## Implementation Order
|
|
|
|
### Phase 1: Data Layer (Done)
|
|
1. ✅ Event Log table + integration
|
|
2. ✅ Ledger table + integration
|
|
3. ✅ Vector store table + functions
|
|
|
|
### Phase 2: UI Layer (Done)
|
|
1. ✅ Event Log dashboard page
|
|
2. ✅ Ledger dashboard page
|
|
3. ✅ Memory browser page
|
|
|
|
### Phase 3: Advanced Features (Planned)
|
|
1. 📝 Cascade Router integration (ADR-020)
|
|
- Create adapter layer
|
|
- Modify Timmy agent
|
|
- Provider status dashboard
|
|
|
|
2. 📝 Self-Upgrade Queue (ADR-021)
|
|
- Create `upgrades` table
|
|
- Modify self-modify loop
|
|
- Dashboard queue UI
|
|
|
|
3. 📝 Real-Time Activity Feed (ADR-022)
|
|
- EventBroadcaster bridge
|
|
- WebSocket integration
|
|
- Activity feed panel
|
|
|
|
### Phase 4: Testing
|
|
1. Unit tests for each service
|
|
2. E2E tests for full workflows
|
|
3. Load testing for WebSocket connections
|
|
|
|
## Configuration
|
|
|
|
New config options in `config.py`:
|
|
|
|
```python
|
|
# Cascade Router
|
|
cascade_providers: list[ProviderConfig]
|
|
circuit_breaker_threshold: int = 5
|
|
|
|
# Self-Upgrade
|
|
auto_approve_upgrades: bool = False
|
|
upgrade_timeout_hours: int = 24
|
|
|
|
# Activity Feed
|
|
websocket_event_throttle: int = 10 # events/sec
|
|
activity_feed_buffer: int = 100 # events to buffer
|
|
```
|
|
|
|
## Security Considerations
|
|
|
|
| Feature | Risk | Mitigation |
|
|
|---------|------|------------|
|
|
| Event Log | Log injection | Sanitize all data fields |
|
|
| Ledger | Payment forgery | Verify with Lightning node |
|
|
| Memory | Data exposure | Filter by user permissions |
|
|
| Upgrade Queue | Unauthorized changes | Require approval, audit log |
|
|
| Cascade Router | API key exposure | Use environment variables |
|
|
| Activity Feed | Data leak | Authenticate WebSocket |
|