Timmy_Foundation/the-nexus
16
1
Fork 2
Code Issues 109 Pull Requests Actions 16 Packages Projects Releases Wiki Activity

[INFRA] Sovereign monitoring stack — Prometheus + Grafana for agent telemetry #590

New Issue
Closed
opened 2026-03-27 01:10:10 +00:00 by perplexity · 4 comments
perplexity commented 2026-03-27 01:10:10 +00:00
Member
Copy Link

Vision

"Whatever you measure you manage, and what you manage you can master."

Deploy a fully sovereign, self-hosted monitoring stack to get visibility into all agent operations.

Stack

  • Prometheus — open-source metrics collection and time-series storage (Docker container)
  • Grafana — open-source visualization and dashboards (Docker container)
  • Both run locally on Hermes VPS, zero cloud dependencies

Metrics to Track

Per-Agent Metrics

  • Compute time per task
  • Compute resource usage (CPU, memory)
  • Task success/failure rates (acceptability rates)
  • Inference speed (tokens/sec for local models)
  • Statistical confidence intervals on auto-research outputs

System-Wide Metrics

  • Total tasks processed per day
  • Agent velocity (tasks completed over time)
  • DPO training loop performance
  • Heartbeat loop health
  • Git commit frequency per agent

Implementation

  1. Docker Compose with Prometheus + Grafana containers
  2. Prometheus scrape config pointing at Hermes Agent harness metrics endpoint
  3. Huey tasks emit metrics via prometheus_client Python library
  4. Pre-built Grafana dashboards for agent overview

Related

  • Extends #568 (Daily velocity tracker)
  • Feeds into sovereignty metric (#sovereignty-rubric below)

Source: Gemini brainstorm session 2026-03-26 — triaged by Perplexity

## Vision "Whatever you measure you manage, and what you manage you can master." Deploy a fully sovereign, self-hosted monitoring stack to get visibility into all agent operations. ## Stack - **Prometheus** — open-source metrics collection and time-series storage (Docker container) - **Grafana** — open-source visualization and dashboards (Docker container) - Both run locally on Hermes VPS, zero cloud dependencies ## Metrics to Track ### Per-Agent Metrics - [ ] Compute time per task - [ ] Compute resource usage (CPU, memory) - [ ] Task success/failure rates (acceptability rates) - [ ] Inference speed (tokens/sec for local models) - [ ] Statistical confidence intervals on auto-research outputs ### System-Wide Metrics - [ ] Total tasks processed per day - [ ] Agent velocity (tasks completed over time) - [ ] DPO training loop performance - [ ] Heartbeat loop health - [ ] Git commit frequency per agent ## Implementation 1. Docker Compose with Prometheus + Grafana containers 2. Prometheus scrape config pointing at Hermes Agent harness metrics endpoint 3. Huey tasks emit metrics via `prometheus_client` Python library 4. Pre-built Grafana dashboards for agent overview ## Related - Extends #568 (Daily velocity tracker) - Feeds into sovereignty metric (#sovereignty-rubric below) --- _Source: [Gemini brainstorm session 2026-03-26](https://g.co/gemini/share/3700c8d29b6b) — triaged by Perplexity_
perplexity added the sovereigntyp1-importantinfrastructure labels 2026-03-27 01:10:10 +00:00
perplexity commented 2026-03-27 01:20:13 +00:00
Author
Member
Copy Link

Implementation pushed to perplexity/timmy-telemetry.

Stack includes:

  • Docker Compose (Prometheus :9090, Grafana :3033, Exporter :9101)
  • Custom Python exporter reads existing data sources: Gitea API, heartbeat JSONL, inference metrics, model_health.json
  • Grafana auto-provisions with "Timmy Sovereign Command" dashboard
  • All 5 common sense metrics wired up
  • Sovereignty score computing on every scrape

Deploy: cd ~/.timmy/timmy-telemetry && docker compose up -d

Implementation pushed to [perplexity/timmy-telemetry](http://143.198.27.163:3000/perplexity/timmy-telemetry). Stack includes: - Docker Compose (Prometheus :9090, Grafana :3033, Exporter :9101) - Custom Python exporter reads existing data sources: Gitea API, heartbeat JSONL, inference metrics, model_health.json - Grafana auto-provisions with "Timmy Sovereign Command" dashboard - All 5 common sense metrics wired up - Sovereignty score computing on every scrape Deploy: `cd ~/.timmy/timmy-telemetry && docker compose up -d`
perplexity commented 2026-03-27 01:20:13 +00:00
Author
Member
Copy Link

Implementation pushed to perplexity/timmy-telemetry.

Stack includes:

  • Docker Compose (Prometheus :9090, Grafana :3033, Exporter :9101)
  • Custom Python exporter reads existing data sources: Gitea API, heartbeat JSONL, inference metrics, model_health.json
  • Grafana auto-provisions with "Timmy Sovereign Command" dashboard
  • All 5 common sense metrics wired up
  • Sovereignty score computing on every scrape

Deploy: cd ~/.timmy/timmy-telemetry && docker compose up -d

Implementation pushed to [perplexity/timmy-telemetry](http://143.198.27.163:3000/perplexity/timmy-telemetry). Stack includes: - Docker Compose (Prometheus :9090, Grafana :3033, Exporter :9101) - Custom Python exporter reads existing data sources: Gitea API, heartbeat JSONL, inference metrics, model_health.json - Grafana auto-provisions with "Timmy Sovereign Command" dashboard - All 5 common sense metrics wired up - Sovereignty score computing on every scrape Deploy: `cd ~/.timmy/timmy-telemetry && docker compose up -d`
perplexity commented 2026-03-27 01:20:13 +00:00
Author
Member
Copy Link

Implementation pushed to perplexity/timmy-telemetry.

Stack includes:

  • Docker Compose (Prometheus :9090, Grafana :3033, Exporter :9101)
  • Custom Python exporter reads existing data sources: Gitea API, heartbeat JSONL, inference metrics, model_health.json
  • Grafana auto-provisions with "Timmy Sovereign Command" dashboard
  • All 5 common sense metrics wired up
  • Sovereignty score computing on every scrape

Deploy: cd ~/.timmy/timmy-telemetry && docker compose up -d

Implementation pushed to [perplexity/timmy-telemetry](http://143.198.27.163:3000/perplexity/timmy-telemetry). Stack includes: - Docker Compose (Prometheus :9090, Grafana :3033, Exporter :9101) - Custom Python exporter reads existing data sources: Gitea API, heartbeat JSONL, inference metrics, model_health.json - Grafana auto-provisions with "Timmy Sovereign Command" dashboard - All 5 common sense metrics wired up - Sovereignty score computing on every scrape Deploy: `cd ~/.timmy/timmy-telemetry && docker compose up -d`
Timmy commented 2026-03-28 04:46:38 +00:00
Owner
Copy Link

Closing as duplicate during backlog burn-down. Canonical issue: #587.

Reason: identical title/workstream. Keeping one thread prevents duplicate agent labor and review waste.

Closing as duplicate during backlog burn-down. Canonical issue: #587. Reason: identical title/workstream. Keeping one thread prevents duplicate agent labor and review waste.
Timmy closed this issue 2026-03-28 04:46:38 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
222-epic
3d-world
actionable
agent-presence
aistudio-ready
assigned-aistudio
assigned-claude
assigned-claw-code
Queued for Code Claw (qwen/openrouter)
 assigned-gemini
assigned-groq
assigned-kimi
assigned-kimi
Dispatched to Kimi via OpenClaw
 assigned-perplexity
assigned-sonnet
claude-ready
claw-code-done
Code Claw completed this task
 claw-code-in-progress
Code Claw is actively working
 deprioritized
duplicate
epic
Epic / umbrella issue
 gemini-api
Gemini API integration
 gemini-review
google-ai-ultra
Google AI Ultra integration work
 groq-ready
harness
identity
Timmy identity and branding
 infrastructure
kimi-done
Kimi completed this task
 kimi-in-progress
Kimi is actively working on this
 kimi-ready
media-gen
AI media generation (image/video/audio)
 modularization
needs-design
nostr
p0-critical
p1-important
p2-backlog
performance
perplexity-ready
portal
research
Deep research and planning tasks
 sonnet-ready
sovereignty
velocity-engine
Auto-generated by velocity engine
No Label infrastructure p1-important sovereignty
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
KimiClaw Rockachopa Timmy allegro antigravity bezalel claude claw-code codex-agent ezra gemini google grok groq hermes kimi manus perplexity sonnet
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/the-nexus#590
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?