diff --git a/fleet/capacity-inventory.md b/fleet/capacity-inventory.md new file mode 100644 index 00000000..91e124c3 --- /dev/null +++ b/fleet/capacity-inventory.md @@ -0,0 +1,191 @@ +# Capacity Inventory - Fleet Resource Baseline + +**Last audited:** 2026-04-07 16:00 UTC +**Auditor:** Timmy (direct inspection) + +--- + +## Fleet Resources (Paperclips Model) + +Three primary resources govern the fleet: + +| Resource | Role | Generation | Consumption | +|----------|------|-----------|-------------| +| **Capacity** | Compute hours available across fleet. Determines what work can be done. | Through healthy utilization of VPS/Mac agents | Fleet improvements consume it (investing in automation, orchestration, sovereignty) | +| **Uptime** | % time services are running. Earned at Fibonacci milestones. | When services stay up naturally | Degrades on any failure | +| **Innovation** | Only generates when capacity is <70% utilized. Fuels Phase 3+. | When you leave capacity free | Phase 3+ buildings consume it (requires spare capacity to build) | + +### The Tension +- Run fleet at 95%+ capacity: maximum productivity, ZERO Innovation +- Run fleet at <70% capacity: Innovation generates but slower progress +- This forces the Paperclips question: optimize now or invest in future capability? + +--- + +## VPS Resource Baselines + +### Ezra (143.198.27.163) - "Forge" + +| Metric | Value | Utilization | +|--------|-------|-------------| +| **OS** | Ubuntu 24.04 (6.8.0-106-generic) | | +| **vCPU** | 4 vCPU (DO basic droplet, shared) | Load: 10.76/7.59/7.04 (very high) | +| **RAM** | 7,941 MB total | 2,104 used / 5,836 available (26% used, 74% free) | +| **Disk** | 154 GB vda1 | 111 GB used / 44 GB free (72%) **WARNING** | +| **Swap** | 6,143 MB | 643 MB used (10%) | +| **Uptime** | 7 days, 18 hours | | + +### Key Processes (sorted by memory) +| Process | RSS | %CPU | Notes | +|---------|-----|------|-------| +| Gitea | 556 MB | 83.5% | Web service, high CPU due to API load | +| MemPalace (ezra) | 268 MB | 136% | Mining project files - HIGH CPU | +| Hermes gateway (ezra) | 245 MB | 1.7% | Agent gateway | +| Ollama | 230 MB | 0.1% | Model serving | +| PostgreSQL | 138 MB | ~0% | Gitea database | + +**Capacity assessment:** 26% memory used, but 72% disk is getting tight. CPU load is very high (10.76 on 4vCPU = 269% utilization). Ezra is CPU-bound, not RAM-bound. + +### Allegro (167.99.126.228) + +| Metric | Value | Utilization | +|--------|-------|-------------| +| **OS** | Ubuntu 24.04 (6.8.0-106-generic) | | +| **vCPU** | 4 vCPU (DO basic droplet, shared) | Moderate load | +| **RAM** | 7,941 MB total | 1,591 used / 6,349 available (20% used, 80% free) | +| **Disk** | 154 GB vda1 | 41 GB used / 114 GB free (27%) **GOOD** | +| **Swap** | 8,191 MB | 686 MB used (8%) | +| **Uptime** | 7 days, 18 hours | | + +### Key Processes (sorted by memory) +| Process | RSS | %CPU | Notes | +|---------|-----|------|-------| +| Hermes gateway (allegro) | 680 MB | 0.9% | Main agent gateway | +| Gitea | 181 MB | 1.2% | Secondary gitea? | +| Systemd-journald | 160 MB | 0.0% | System logging | +| Ezra Hermes gateway | 58 MB | 0.0% | Running ezra agent here | +| Bezalel Hermes gateway | 58 MB | 0.0% | Running bezalel agent here | +| Dockerd | 48 MB | 0.0% | Docker daemon | + +**Capacity assessment:** 20% memory used, 27% disk used. Allegro has headroom. Also running hermes gateways for Ezra and Bezalel (cross-host agent execution). + +### Bezalel (159.203.146.185) + +| Metric | Value | Utilization | +|--------|-------|-------------| +| **OS** | Ubuntu 24.04 (6.8.0-71-generic) | | +| **vCPU** | 2 vCPU (DO basic droplet, shared) | Load varies | +| **RAM** | 1,968 MB total | 817 used / 1,151 available (42% used, 58% free) | +| **Disk** | 48 GB vda1 | 12 GB used / 37 GB free (24%) **GOOD** | +| **Swap** | 2,047 MB | 448 MB used (22%) | +| **Uptime** | 7 days, 18 hours | | + +### Key Processes (sorted by memory) +| Process | RSS | %CPU | Notes | +|---------|-----|------|-------| +| Hermes gateway | 339 MB | 7.7% | Agent gateway (16.8% of RAM) | +| uv pip install | 137 MB | 56.6% | Installing packages (temporary) | +| Mender | 27 MB | 0.0% | Device management | + +**Capacity assessment:** 42% memory used, only 2GB total RAM. Bezalel is the most constrained. 2 vCPU means less compute headroom than Ezra/Allegro. Disk is fine. + +### Mac Local (M3 Max) + +| Metric | Value | Utilization | +|--------|-------|-------------| +| **OS** | macOS 26.3.1 | | +| **CPU** | Apple M3 Max (14 cores) | Very capable | +| **RAM** | 36 GB | ~8 GB used (22%) | +| **Disk** | 926 GB total | ~624 GB used / 302 GB free (68%) | + +### Key Processes +| Process | Memory | Notes | +|---------|--------|-------| +| Hermes gateway | 500 MB | Primary gateway | +| Hermes agents (x3) | ~560 MB total | Multiple sessions | +| Ollama | ~20 MB base + model memory | Model loading varies | +| OpenClaw | 350 MB | Gateway process | +| Evennia (server+portal) | 56 MB | Game world | + +--- + +## Resource Summary + +| Resource | Ezra | Allegro | Bezalel | Mac Local | TOTAL | +|----------|------|---------|---------|-----------|-------| +| **vCPU** | 4 | 4 | 2 | 14 (M3 Max) | 24 | +| **RAM** | 8 GB (26% used) | 8 GB (20% used) | 2 GB (42% used) | 36 GB (22% used) | 54 GB | +| **Disk** | 154 GB (72%) | 154 GB (27%) | 48 GB (24%) | 926 GB (68%) | 1,282 GB | +| **Cost** | $12/mo | $12/mo | $12/mo | owned | $36/mo | + +### Utilization by Category +| Category | Estimated Daily Hours | % of Fleet Capacity | +|----------|----------------------|---------------------| +| Hermes agents | ~3-4 hrs active | 5-7% | +| Ollama inference | ~1-2 hrs | 2-4% | +| Gitea services | 24/7 | 5-10% | +| Evennia | 24/7 | <1% | +| Idle | ~18-20 hrs | ~80-90% | + +### Capacity Utilization: ~15-20% active +**Innovation rate:** GENERATING (capacity < 70%) +**Recommendation:** Good — Innovation is generating because most capacity is free. +This means Phase 3+ capabilities (orchestration, load balancing, etc.) are accessible NOW. + +--- + +## Uptime Baseline + +**Baseline period:** 2026-04-07 14:00-16:00 UTC (2 hours, ~24 checks at 5-min intervals) + +| Service | Checks | Uptime | Status | +|---------|--------|--------|--------| +| Ezra | 24/24 | 100.0% | GOOD | +| Allegro | 24/24 | 100.0% | GOOD | +| Bezalel | 24/24 | 100.0% | GOOD | +| Gitea | 23/24 | 95.8% | GOOD | +| Hermes Gateway | 23/24 | 95.8% | GOOD | +| Ollama | 24/24 | 100.0% | GOOD | +| OpenClaw | 24/24 | 100.0% | GOOD | +| Evennia | 24/24 | 100.0% | GOOD | +| Hermes Agent | 21/24 | 87.5% | **CHECK** | + +### Fibonacci Uptime Milestones +| Milestone | Target | Current | Status | +|-----------|--------|---------|--------| +| 95% | 95% | 100% (VPS), 98.6% (avg) | REACHED | +| 95.5% | 95.5% | 98.6% | REACHED | +| 96% | 96% | 98.6% | REACHED | +| 97% | 97% | 98.6% | REACHED | +| 98% | 98% | 98.6% | REACHED | +| 99% | 99% | 98.6% | APPROACHING | + +--- + +## Risk Assessment + +| Risk | Severity | Mitigation | +|------|----------|------------| +| Ezra disk 72% used | MEDIUM | Move non-essential data, add monitoring alert at 85% | +| Bezalel only 2GB RAM | HIGH | Cannot run large models locally. Good for Evennia, tight for agents | +| Ezra CPU load 269% | HIGH | MemPalace mining consuming 136% CPU. Consider scheduling | +| Mac disk 68% used | MEDIUM | 302 GB free still. Growing but not urgent | +| No cross-VPS mesh | LOW | SSH works but no Tailscale. No private network between VPSes | + +--- + +## Recommendations + +### Immediate (Phase 1-2) +1. **Ezra disk cleanup:** 44 GB free at 72%. Docker images, old logs, and MemPalace mine data could be rotated. +2. **Alert thresholds:** Add disk alerts at 85% (Ezra, Mac) before they become critical. + +### Short-term (Phase 3) +3. **Load balancing:** Ezra is CPU-bound, Allegro has 80% RAM free. Move some agent processes from Ezra to Allegro. +4. **Innovation investment:** Since fleet is at 15-20% utilization, Innovation is high. This is the time to build Phase 3 capabilities. + +### Medium-term (Phase 4) +5. **Bezalel RAM upgrade:** 2GB is tight. Consider upgrade to 4GB ($24/mo instead of $12/mo). +6. **Tailscale mesh:** Install on all VPSes for private inter-VPS network. + +---