Rockachopa/Timmy-time-dashboard
1
0
Fork 2
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

[Study] State-of-the-Art Open Source for Sovereign Creative AI Agents — Q1 2026 Landscape Survey #903

New Issue
Closed
opened 2026-03-22 13:00:27 +00:00 by perplexity · 2 comments
perplexity commented 2026-03-22 13:00:27 +00:00
Collaborator
Copy Link

Summary

Comprehensive Q1 2026 landscape survey of open-source tools for building a sovereign creative AI agent stack. 20-page report covering 12 technology domains with specific version numbers, benchmarks, and "Bottom Line for Timmy" integration recommendations for each.

PDF: State-of-the-Art — Open Source for Sovereign Creative AI Agents (20 pages, March 2026)

Domains Covered

1. Local LLM Inference (pp. 1–3)

  • MLX is now definitively faster than llama.cpp on Apple Silicon — MLX-LM v0.24.1 outperforms across all quantizations
  • vllm-mlx recommended as production inference layer (OpenAI-compatible API)
  • Ollama for development convenience, but not production
  • Cross-ref: #882 (Model Tiering Router), #881 (Off-Grid Compute Plan)

2. AI Coding Agents (pp. 3–4)

  • OpenCode (128K stars) — terminal-first, multi-model, MCP support
  • Goose (Block/Square) — extensible agent with MCP + custom tools
  • Aider — pair programming, universal model support
  • Forgejo — self-hosted forge with Actions CI/CD
  • Cross-ref: #875 (Docker Compose), #852 (Infra Epic)

3. Image Generation (pp. 4–5)

  • ComfyUI v0.3.21 — node-based, 1000+ custom nodes
  • Draw Things (Apple Silicon native) — FLUX.1-dev, ControlNet, LoRA
  • FLUX — SotA quality, GGUF quants run on 8GB VRAM
  • Cross-ref: #879 (AlexanderWhitestone.com Dashboard)

4. Music & Voice (pp. 5–7)

  • ACE-Step 1.5 — open-source music generation rivaling commercial services (major Q1 shift)
  • mlx-audio + Kokoro — local voice synthesis, 54 voices, Apple Silicon optimized
  • Chatterbox (Resemble AI) — zero-shot voice clone in 5 seconds
  • Piper TTS — lightweight, fast, model training supported
  • Cross-ref: #853 (Piper TTS Voice Model), #858 (Mumble Voice Bridge)

5. Agent Orchestration (pp. 7–9)

  • FastMCP v2.7.0 — the MCP framework to use (Python + TS)
  • CrewAI — multi-agent with role-based personas
  • PocketFlow — 100-line minimalist agent framework
  • Cross-ref: #872 (Heartbeat v2), #871 (WorldInterface), PR #865 (FastAPI Harness + SOUL.md)

6. Nostr Developer Stack (pp. 9–12)

  • nostr-sdk v0.44.2 — only Python library needed for Nostr + NWC (built-in wallet ops)
  • nostrdvm — NIP-90 Data Vending Machine framework = Timmy's economic primitive
  • Blossom — HTTP blob storage via SHA-256, Nostr key auth
  • startwithbitcoin — "Timmy Time starter kit" (Nostr identity + NWC wallet + Lightning)
  • Alby MCP — Connect Lightning wallet to any MCP-compatible LLM
  • Cross-ref: #856 (Nostr Identity), #877 (Nostr Agent Identity), #794 (Meet Protocol Epic), #892 (NIP-89/NIP-90 Study)

7. Lightning Payments (pp. 12–13)

  • Lightning agent-tools v1 (Feb 2026) — 7 composable skills + MCP server, lnget for L402, Aperture for paid endpoints
  • LND v0.20.1-beta — full Lightning node, SQL channel graph
  • LNbits v1.4 — non-custodial wallet server, 137+ extensions, Boltz standalone Docker
  • Cashu v0.17.0 — ecash for privacy-preserving inter-agent micropayments
  • Cross-ref: #857 (Lightning Wallet Setup), #876 (Cashu Mint), #842 (L402 Payment), #814 (L402 Protocol), #800 (Lightning Payment Integration)

8. Knowledge Graphs & Memory (pp. 13–15)

  • Graphiti v0.28.2 (Zep) — bi-temporal knowledge graph, 94.8% DMR benchmark, MCP server
  • Mem0 v1.0.5 — self-improving memory, +26% over OpenAI Memory, self-hosted w/ Qdrant
  • ChromaDB v1.5.5 — AI-native embedding DB, weekly releases
  • Three-layer stack: ChromaDB (documents) → Mem0 (conversations) → Graphiti (temporal KG)
  • Cross-ref: #873 (Three-Tier Memory System), #883 (UESP Knowledge Base)

9. Video & Streaming (pp. 15–16)

  • LiveKit Agents v1.4.3 — WebRTC SFU + STT→LLM→TTS voice pipelines
  • MediaMTX v1.16.3 — zero-dependency media server, "always available streams"
  • OBS Studio v32.0.4 — obs-websocket v5.x built-in, Python control via obsws-python
  • Wan 2.2 + LTX-Video 2 — local AI video generation in ComfyUI
  • Cross-ref: #834 (Stream Infrastructure), #835 (Director Camera), #880 (Episode Compiler), #879 (Web Dashboard)

10. Self-Hosting Infrastructure (pp. 16–17)

  • Coolify v4 — self-hosted PaaS, 280+ one-click services
  • n8n v2.12.2 — MCP webhook integration, visual workflow automation
  • Tailscale Aperture (alpha) — AI gateway securing LLM sessions with identity-aware networking
  • Forgejo v14.0.3 — self-hosted forge, GitHub Actions-compatible
  • Cross-ref: #875 (Docker Compose), #881 (Off-Grid Compute), #852 (Infra Epic)

11. Game Engines + LLM Integration (pp. 17–18)

  • Godot 4.6.1 + LibGodot — embed game engine as library, Python-controllable
  • Godot RL Agents — Python↔Godot RL bridge, ONNX export
  • OpenMW v0.50.0 — Morrowind reimplementation, extensive Lua 5.1 API, hot-reload
  • Luanti v5.15.1 — voxel platform, Lua scripting (Timmy tried to pivot here, overruled)
  • GamingAgent (ICLR 2026) — standardized LLM↔game interfaces
  • Cross-ref: #817 (Project Morrowind Epic), #871 (WorldInterface), #878 (TES3MP Bridge), #819 (Perception Script), #820 (Input Bridge), #901 (Morrowind Alternatives Study), #902 (Games for AI Co-Op)

12. Decentralized Identity (pp. 18–20)

  • did:nostr — DID method resolving Nostr pubkeys, offline-first
  • SoulSpec / SOUL.md v0.4 — agent persona standard (used by OpenClaw 145K stars, Claude Code, Cursor)
  • L402 (bLIP-0026) — HTTP 402 + Lightning = transaction-based identity
  • Identity triple: did:nostr (who) + SoulSpec (what) + L402 (how)
  • Cross-ref: #854 (SOUL.md Framework), #856 (Nostr Identity), #877 (Nostr Agent Identity), PR #865 (SOUL.md Framework)

Key Conclusion

"The critical path for Timmy Time this month is not more research — it's integration."

Recommended integration stack: vllm-mlx → FastMCP → nostr-sdk → LND + lightning-agent-tools → Graphiti → mlx-audio → ComfyUI → ACE-Step → Tailscale

Three biggest Q1 2026 shifts:

  1. MLX definitively faster than llama.cpp on Apple Silicon
  2. Lightning Labs open-sourced agent-tools (machine payments solved)
  3. ACE-Step 1.5 cracked open-source music generation

Labels

  • needs-extraction — The version numbers, install commands, and architecture recommendations should be extracted into actionable sub-tickets where they update existing ones
## Summary Comprehensive Q1 2026 landscape survey of open-source tools for building a sovereign creative AI agent stack. 20-page report covering 12 technology domains with specific version numbers, benchmarks, and "Bottom Line for Timmy" integration recommendations for each. **PDF**: State-of-the-Art — Open Source for Sovereign Creative AI Agents (20 pages, March 2026) ## Domains Covered ### 1. Local LLM Inference (pp. 1–3) - **MLX is now definitively faster than llama.cpp on Apple Silicon** — MLX-LM v0.24.1 outperforms across all quantizations - **vllm-mlx** recommended as production inference layer (OpenAI-compatible API) - **Ollama** for development convenience, but not production - Cross-ref: #882 (Model Tiering Router), #881 (Off-Grid Compute Plan) ### 2. AI Coding Agents (pp. 3–4) - **OpenCode** (128K stars) — terminal-first, multi-model, MCP support - **Goose** (Block/Square) — extensible agent with MCP + custom tools - **Aider** — pair programming, universal model support - **Forgejo** — self-hosted forge with Actions CI/CD - Cross-ref: #875 (Docker Compose), #852 (Infra Epic) ### 3. Image Generation (pp. 4–5) - **ComfyUI** v0.3.21 — node-based, 1000+ custom nodes - **Draw Things** (Apple Silicon native) — FLUX.1-dev, ControlNet, LoRA - **FLUX** — SotA quality, GGUF quants run on 8GB VRAM - Cross-ref: #879 (AlexanderWhitestone.com Dashboard) ### 4. Music & Voice (pp. 5–7) - **ACE-Step 1.5** — open-source music generation rivaling commercial services (major Q1 shift) - **mlx-audio + Kokoro** — local voice synthesis, 54 voices, Apple Silicon optimized - **Chatterbox** (Resemble AI) — zero-shot voice clone in 5 seconds - **Piper TTS** — lightweight, fast, model training supported - Cross-ref: #853 (Piper TTS Voice Model), #858 (Mumble Voice Bridge) ### 5. Agent Orchestration (pp. 7–9) - **FastMCP** v2.7.0 — the MCP framework to use (Python + TS) - **CrewAI** — multi-agent with role-based personas - **PocketFlow** — 100-line minimalist agent framework - Cross-ref: #872 (Heartbeat v2), #871 (WorldInterface), PR #865 (FastAPI Harness + SOUL.md) ### 6. Nostr Developer Stack (pp. 9–12) - **nostr-sdk** v0.44.2 — only Python library needed for Nostr + NWC (built-in wallet ops) - **nostrdvm** — NIP-90 Data Vending Machine framework = Timmy's economic primitive - **Blossom** — HTTP blob storage via SHA-256, Nostr key auth - **startwithbitcoin** — "Timmy Time starter kit" (Nostr identity + NWC wallet + Lightning) - **Alby MCP** — Connect Lightning wallet to any MCP-compatible LLM - Cross-ref: #856 (Nostr Identity), #877 (Nostr Agent Identity), #794 (Meet Protocol Epic), #892 (NIP-89/NIP-90 Study) ### 7. Lightning Payments (pp. 12–13) - **Lightning agent-tools v1** (Feb 2026) — 7 composable skills + MCP server, `lnget` for L402, Aperture for paid endpoints - **LND v0.20.1-beta** — full Lightning node, SQL channel graph - **LNbits v1.4** — non-custodial wallet server, 137+ extensions, Boltz standalone Docker - **Cashu v0.17.0** — ecash for privacy-preserving inter-agent micropayments - Cross-ref: #857 (Lightning Wallet Setup), #876 (Cashu Mint), #842 (L402 Payment), #814 (L402 Protocol), #800 (Lightning Payment Integration) ### 8. Knowledge Graphs & Memory (pp. 13–15) - **Graphiti v0.28.2** (Zep) — bi-temporal knowledge graph, 94.8% DMR benchmark, MCP server - **Mem0 v1.0.5** — self-improving memory, +26% over OpenAI Memory, self-hosted w/ Qdrant - **ChromaDB v1.5.5** — AI-native embedding DB, weekly releases - **Three-layer stack**: ChromaDB (documents) → Mem0 (conversations) → Graphiti (temporal KG) - Cross-ref: #873 (Three-Tier Memory System), #883 (UESP Knowledge Base) ### 9. Video & Streaming (pp. 15–16) - **LiveKit Agents v1.4.3** — WebRTC SFU + STT→LLM→TTS voice pipelines - **MediaMTX v1.16.3** — zero-dependency media server, "always available streams" - **OBS Studio v32.0.4** — obs-websocket v5.x built-in, Python control via obsws-python - **Wan 2.2 + LTX-Video 2** — local AI video generation in ComfyUI - Cross-ref: #834 (Stream Infrastructure), #835 (Director Camera), #880 (Episode Compiler), #879 (Web Dashboard) ### 10. Self-Hosting Infrastructure (pp. 16–17) - **Coolify v4** — self-hosted PaaS, 280+ one-click services - **n8n v2.12.2** — MCP webhook integration, visual workflow automation - **Tailscale Aperture** (alpha) — AI gateway securing LLM sessions with identity-aware networking - **Forgejo v14.0.3** — self-hosted forge, GitHub Actions-compatible - Cross-ref: #875 (Docker Compose), #881 (Off-Grid Compute), #852 (Infra Epic) ### 11. Game Engines + LLM Integration (pp. 17–18) - **Godot 4.6.1 + LibGodot** — embed game engine as library, Python-controllable - **Godot RL Agents** — Python↔Godot RL bridge, ONNX export - **OpenMW v0.50.0** — Morrowind reimplementation, extensive Lua 5.1 API, hot-reload - **Luanti v5.15.1** — voxel platform, Lua scripting (Timmy tried to pivot here, overruled) - **GamingAgent** (ICLR 2026) — standardized LLM↔game interfaces - Cross-ref: #817 (Project Morrowind Epic), #871 (WorldInterface), #878 (TES3MP Bridge), #819 (Perception Script), #820 (Input Bridge), #901 (Morrowind Alternatives Study), #902 (Games for AI Co-Op) ### 12. Decentralized Identity (pp. 18–20) - **did:nostr** — DID method resolving Nostr pubkeys, offline-first - **SoulSpec / SOUL.md v0.4** — agent persona standard (used by OpenClaw 145K stars, Claude Code, Cursor) - **L402** (bLIP-0026) — HTTP 402 + Lightning = transaction-based identity - **Identity triple**: did:nostr (who) + SoulSpec (what) + L402 (how) - Cross-ref: #854 (SOUL.md Framework), #856 (Nostr Identity), #877 (Nostr Agent Identity), PR #865 (SOUL.md Framework) ## Key Conclusion > "The critical path for Timmy Time this month is not more research — it's integration." Recommended integration stack: **vllm-mlx** → **FastMCP** → **nostr-sdk** → **LND + lightning-agent-tools** → **Graphiti** → **mlx-audio** → **ComfyUI** → **ACE-Step** → **Tailscale** Three biggest Q1 2026 shifts: 1. MLX definitively faster than llama.cpp on Apple Silicon 2. Lightning Labs open-sourced agent-tools (machine payments solved) 3. ACE-Step 1.5 cracked open-source music generation ## Labels - `needs-extraction` — The version numbers, install commands, and architecture recommendations should be extracted into actionable sub-tickets where they update existing ones
State-of-the-Art_-Open-Source-for-Sovereign-Creative-AI-Agents.pdf
200 KiB
perplexity commented 2026-03-22 13:00:55 +00:00
Author
Collaborator
Copy Link

📎 PDF attached above — 20-page full report with tool tables, version numbers, and "Bottom Line for Timmy" integration guides for each domain.

Cross-Reference Map

This study touches nearly every open ticket. Here's the mapping by priority:

Direct Updates Needed (tool versions/approaches in this PDF supersede or enhance existing tickets)

Existing Ticket What This PDF Adds
#873 (Three-Tier Memory) Specific stack: ChromaDB v1.5.5 + Mem0 v1.0.5 + Graphiti v0.28.2
#857 (Lightning Wallet) LND v0.20.1-beta + lightning-agent-tools v1 + LNbits v1.4 — complete stack
#856 (Nostr Identity) nostr-sdk v0.44.2 is the only library needed; did:nostr spec for DID
#854 (SOUL.md Framework) SoulSpec v0.4 is now the standard (145K stars via OpenClaw)
#853 (Piper TTS) Piper still valid, but mlx-audio/Kokoro and Chatterbox are faster alternatives
#876 (Cashu Mint) Cashu nutshell v0.17.0 — NUT-19/20/21/22 for auth + cached responses
#877 (Nostr Agent Identity) startwithbitcoin framework = "Timmy Time starter kit"
#882 (Model Tiering Router) vllm-mlx as production layer, MLX-LM v0.24.1 confirmed fastest
#878 (TES3MP Bridge) OpenMW v0.50.0 Lua 5.1 API with hot-reload confirmed ready

Validates Existing Architecture

  • #871 (WorldInterface) + #872 (Heartbeat v2) — FastMCP v2.7.0 is the confirmed orchestration layer
  • #834 (Stream Infrastructure) — MediaMTX v1.16.3 + LiveKit Agents v1.4.3 confirmed
  • #879 (Web Dashboard) — LiveKit for real-time viewer interaction layer
  • #817 (Project Morrowind Epic) — OpenMW Lua API confirmed production-ready

New Capabilities Not Yet Ticketed

  • ACE-Step 1.5 music generation — Timmy can create original music (no existing ticket)
  • Tailscale Aperture — AI gateway security for the whole stack (no existing ticket)
  • n8n v2.12.2 — visual workflow automation with MCP webhooks (no existing ticket)
  • Godot 4.6 LibGodot — embed game engine as library from Python (alternative engine path)
  • L402 as transaction-based identity — agent proves identity by paying, not authenticating
📎 **PDF attached above** — 20-page full report with tool tables, version numbers, and "Bottom Line for Timmy" integration guides for each domain. ## Cross-Reference Map This study touches nearly every open ticket. Here's the mapping by priority: ### Direct Updates Needed (tool versions/approaches in this PDF supersede or enhance existing tickets) | Existing Ticket | What This PDF Adds | |---|---| | #873 (Three-Tier Memory) | Specific stack: ChromaDB v1.5.5 + Mem0 v1.0.5 + Graphiti v0.28.2 | | #857 (Lightning Wallet) | LND v0.20.1-beta + lightning-agent-tools v1 + LNbits v1.4 — complete stack | | #856 (Nostr Identity) | nostr-sdk v0.44.2 is the only library needed; did:nostr spec for DID | | #854 (SOUL.md Framework) | SoulSpec v0.4 is now the standard (145K stars via OpenClaw) | | #853 (Piper TTS) | Piper still valid, but mlx-audio/Kokoro and Chatterbox are faster alternatives | | #876 (Cashu Mint) | Cashu nutshell v0.17.0 — NUT-19/20/21/22 for auth + cached responses | | #877 (Nostr Agent Identity) | startwithbitcoin framework = "Timmy Time starter kit" | | #882 (Model Tiering Router) | vllm-mlx as production layer, MLX-LM v0.24.1 confirmed fastest | | #878 (TES3MP Bridge) | OpenMW v0.50.0 Lua 5.1 API with hot-reload confirmed ready | ### Validates Existing Architecture - #871 (WorldInterface) + #872 (Heartbeat v2) — FastMCP v2.7.0 is the confirmed orchestration layer - #834 (Stream Infrastructure) — MediaMTX v1.16.3 + LiveKit Agents v1.4.3 confirmed - #879 (Web Dashboard) — LiveKit for real-time viewer interaction layer - #817 (Project Morrowind Epic) — OpenMW Lua API confirmed production-ready ### New Capabilities Not Yet Ticketed - **ACE-Step 1.5** music generation — Timmy can create original music (no existing ticket) - **Tailscale Aperture** — AI gateway security for the whole stack (no existing ticket) - **n8n v2.12.2** — visual workflow automation with MCP webhooks (no existing ticket) - **Godot 4.6 LibGodot** — embed game engine as library from Python (alternative engine path) - **L402 as transaction-based identity** — agent proves identity by paying, not authenticating
claude was assigned by Rockachopa 2026-03-22 23:32:38 +00:00
claude added the rejected-direction label 2026-03-23 13:51:19 +00:00
perplexity commented 2026-03-23 13:52:12 +00:00
Author
Collaborator
Copy Link

🧹 Closed — Superseded

Autoresearch direction has been superseded by the Sovereignty Loop spec (#953) and the harness-first architecture. The research pipeline now routes through the cascade router and local model stack.

Ref: Deep Backlog Triage #1076. Reopen if needed.

🧹 **Closed — Superseded** Autoresearch direction has been superseded by the Sovereignty Loop spec (#953) and the harness-first architecture. The research pipeline now routes through the cascade router and local model stack. Ref: Deep Backlog Triage #1076. Reopen if needed.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
222-epic
Workshop: Timmy as Presence (Epic #222)
 actionable
Has a concrete code/config task extracted
 assigned-claude
Issue currently assigned to Claude agent — do not assign to another agent
 assigned-gemini
Issue currently assigned to Gemini agent — do not assign to another agent
 assigned-groq
assigned-kimi
Issue currently assigned to Kimi agent — do not assign to another agent
 assigned-manus
Issue currently assigned to Manus agent — do not assign to another agent
 claude-ready
consolidation
Part of a consolidation epic
 deprioritized
Keep open but not blocking P0 work
 deprioritized
Keep open but not blocking P0 work
 duplicate
Duplicate of another issue
 gemini-review
Auto-generated by Gemini, needs relevance review
 groq-ready
harness
Core product: agent framework, heartbeat, inference, memory
 heartbeat
Harness: Agent heartbeat loop
 inference
Harness: Inference and model routing
 infrastructure
Supporting stage: dashboard, CI/CD, deployment, DNS
 kimi-ready
Scoped and ready for Kimi to pick up
 memory-session
Harness: Memory and session crystallization
 morrowind
Harness: Morrowind embodiment
 needs-design
Needs architectural design before implementation
 needs-extraction
Philosophy with unextracted engineering work
 p0-critical
Priority 0: Must fix now
 p1-important
Priority 1: Important, next sprint
 p2-backlog
Priority 2: Backlog, do when time permits
 philosophy
Philosophical foundation — informs architecture decisions
 rejected-direction
Closed: rejected or superseded direction
 seed:know-purpose
Three Seeds: KNOW YOUR PURPOSE
 seed:serve-real
Three Seeds: SERVE THE REAL
 seed:tell-truth
Three Seeds: TELL THE TRUTH
 sovereignty
Harness: Sovereignty stack
No Label rejected-direction
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Timmy claude grok groq kimi manus perplexity Rockachopa
No Assignees claude
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Rockachopa/Timmy-time-dashboard#903
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?